##// END OF EJS Templates
Fix submitted output in verbatim mode.
chebee7i -
Show More
@@ -1,1081 +1,1153 b''
1 1 # -*- coding: utf-8 -*-
2 2 """
3 3 Sphinx directive to support embedded IPython code.
4 4
5 5 This directive allows pasting of entire interactive IPython sessions, prompts
6 6 and all, and their code will actually get re-executed at doc build time, with
7 7 all prompts renumbered sequentially. It also allows you to input code as a pure
8 8 python input by giving the argument python to the directive. The output looks
9 9 like an interactive ipython section.
10 10
11 11 To enable this directive, simply list it in your Sphinx ``conf.py`` file
12 12 (making sure the directory where you placed it is visible to sphinx, as is
13 13 needed for all Sphinx directives). For example, to enable syntax highlighting
14 14 and the IPython directive::
15 15
16 16 extensions = ['IPython.sphinxext.ipython_console_highlighting',
17 17 'IPython.sphinxext.ipython_directive']
18 18
19 19 The IPython directive outputs code-blocks with the language 'ipython'. So
20 20 if you do not have the syntax highlighting extension enabled as well, then
21 21 all rendered code-blocks will be uncolored. By default this directive assumes
22 22 that your prompts are unchanged IPython ones, but this can be customized.
23 23 The configurable options that can be placed in conf.py are:
24 24
25 25 ipython_savefig_dir:
26 26 The directory in which to save the figures. This is relative to the
27 27 Sphinx source directory. The default is `html_static_path`.
28 28 ipython_rgxin:
29 29 The compiled regular expression to denote the start of IPython input
30 30 lines. The default is re.compile('In \[(\d+)\]:\s?(.*)\s*'). You
31 31 shouldn't need to change this.
32 32 ipython_rgxout:
33 33 The compiled regular expression to denote the start of IPython output
34 34 lines. The default is re.compile('Out\[(\d+)\]:\s?(.*)\s*'). You
35 35 shouldn't need to change this.
36 36 ipython_promptin:
37 37 The string to represent the IPython input prompt in the generated ReST.
38 38 The default is 'In [%d]:'. This expects that the line numbers are used
39 39 in the prompt.
40 40 ipython_promptout:
41 41 The string to represent the IPython prompt in the generated ReST. The
42 42 default is 'Out [%d]:'. This expects that the line numbers are used
43 43 in the prompt.
44 44 ipython_mplbackend:
45 45 The string which specifies if the embedded Sphinx shell should import
46 46 Matplotlib and set the backend. The value specifies a backend that is
47 47 passed to `matplotlib.use()` before any lines in `ipython_execlines` are
48 48 executed. If not specified in conf.py, then the default value of 'agg' is
49 49 used. To use the IPython directive without matplotlib as a dependency, set
50 50 the value to `None`. It may end up that matplotlib is still imported
51 51 if the user specifies so in `ipython_execlines` or makes use of the
52 52 @savefig pseudo decorator.
53 53 ipython_execlines:
54 54 A list of strings to be exec'd in the embedded Sphinx shell. Typical
55 55 usage is to make certain packages always available. Set this to an empty
56 56 list if you wish to have no imports always available. If specified in
57 57 conf.py as `None`, then it has the effect of making no imports available.
58 58 If omitted from conf.py altogether, then the default value of
59 59 ['import numpy as np', 'import matplotlib.pyplot as plt'] is used.
60 60 ipython_holdcount
61 61 When the @suppress pseudo-decorator is used, the execution count can be
62 62 incremented or not. The default behavior is to hold the execution count,
63 63 corresponding to a value of `True`. Set this to `False` to increment
64 64 the execution count after each suppressed command.
65 65
66 66 As an example, to use the IPython directive when `matplotlib` is not available,
67 67 one sets the backend to `None`::
68 68
69 69 ipython_mplbackend = None
70 70
71 71 An example usage of the directive is:
72 72
73 73 .. code-block:: rst
74 74
75 75 .. ipython::
76 76
77 77 In [1]: x = 1
78 78
79 79 In [2]: y = x**2
80 80
81 81 In [3]: print(y)
82 82
83 83 See http://matplotlib.org/sampledoc/ipython_directive.html for additional
84 84 documentation.
85 85
86 86 Pseudo-Decorators
87 87 =================
88 88
89 Note: Only one decorator is supported per input. If more than one decorator
90 is specified, then only the last one is used.
91
89 92 In addition to the Pseudo-Decorators/options described at the above link,
90 several enhancements have been made.
91 The directive will emit a message to the console at build-time if
92 code-execution resulted in an exception or warning. You can suppress these on
93 a per-block basis by specifying the :okexcept: or :okwarning: options:
93 several enhancements have been made. The directive will emit a message to the
94 console at build-time if code-execution resulted in an exception or warning.
95 You can suppress these on a per-block basis by specifying the :okexcept:
96 or :okwarning: options:
94 97
95 98 .. code-block:: rst
96 99
97 100 .. ipython::
98 101 :okexcept:
99 102 :okwarning:
100 103
101 104 In [1]: 1/0
102 105 In [2]: # raise warning.
103 106
104 107 ToDo
105 108 ----
106 109
107 110 - Turn the ad-hoc test() function into a real test suite.
108 111 - Break up ipython-specific functionality from matplotlib stuff into better
109 112 separated code.
110 113
111 114 Authors
112 115 -------
113 116
114 117 - John D Hunter: orignal author.
115 118 - Fernando Perez: refactoring, documentation, cleanups, port to 0.11.
116 119 - VΓ‘clavΕ milauer <eudoxos-AT-arcig.cz>: Prompt generalizations.
117 120 - Skipper Seabold, refactoring, cleanups, pure python addition
118 121 """
119 122 from __future__ import print_function
120 123
121 124 #-----------------------------------------------------------------------------
122 125 # Imports
123 126 #-----------------------------------------------------------------------------
124 127
125 128 # Stdlib
126 129 import os
127 130 import re
128 131 import sys
129 132 import tempfile
130 133 import ast
131 134 import warnings
132 135
133 136 # To keep compatibility with various python versions
134 137 try:
135 138 from hashlib import md5
136 139 except ImportError:
137 140 from md5 import md5
138 141
139 142 # Third-party
140 143 import sphinx
141 144 from docutils.parsers.rst import directives
142 145 from docutils import nodes
143 146 from sphinx.util.compat import Directive
144 147
145 148 # Our own
146 149 from IPython import Config, InteractiveShell
147 150 from IPython.core.profiledir import ProfileDir
148 151 from IPython.utils import io
149 152 from IPython.utils.py3compat import PY3
150 153
151 154 if PY3:
152 155 from io import StringIO
153 156 else:
154 157 from StringIO import StringIO
155 158
156 159 #-----------------------------------------------------------------------------
157 160 # Globals
158 161 #-----------------------------------------------------------------------------
159 162 # for tokenizing blocks
160 163 COMMENT, INPUT, OUTPUT = range(3)
161 164
162 165 #-----------------------------------------------------------------------------
163 166 # Functions and class declarations
164 167 #-----------------------------------------------------------------------------
165 168
166 169 def block_parser(part, rgxin, rgxout, fmtin, fmtout):
167 170 """
168 171 part is a string of ipython text, comprised of at most one
169 172 input, one ouput, comments, and blank lines. The block parser
170 173 parses the text into a list of::
171 174
172 175 blocks = [ (TOKEN0, data0), (TOKEN1, data1), ...]
173 176
174 177 where TOKEN is one of [COMMENT | INPUT | OUTPUT ] and
175 178 data is, depending on the type of token::
176 179
177 180 COMMENT : the comment string
178 181
179 182 INPUT: the (DECORATOR, INPUT_LINE, REST) where
180 183 DECORATOR: the input decorator (or None)
181 184 INPUT_LINE: the input as string (possibly multi-line)
182 185 REST : any stdout generated by the input line (not OUTPUT)
183 186
184 187 OUTPUT: the output string, possibly multi-line
185 188
186 189 """
187 190 block = []
188 191 lines = part.split('\n')
189 192 N = len(lines)
190 193 i = 0
191 194 decorator = None
192 195 while 1:
193 196
194 197 if i==N:
195 198 # nothing left to parse -- the last line
196 199 break
197 200
198 201 line = lines[i]
199 202 i += 1
200 203 line_stripped = line.strip()
201 204 if line_stripped.startswith('#'):
202 205 block.append((COMMENT, line))
203 206 continue
204 207
205 208 if line_stripped.startswith('@'):
206 # we're assuming at most one decorator -- may need to
207 # rethink
209 # Here is where we assume there is, at most, one decorator.
210 # Might need to rethink this.
208 211 decorator = line_stripped
209 212 continue
210 213
211 214 # does this look like an input line?
212 215 matchin = rgxin.match(line)
213 216 if matchin:
214 217 lineno, inputline = int(matchin.group(1)), matchin.group(2)
215 218
216 219 # the ....: continuation string
217 220 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2))
218 221 Nc = len(continuation)
219 222 # input lines can continue on for more than one line, if
220 223 # we have a '\' line continuation char or a function call
221 224 # echo line 'print'. The input line can only be
222 225 # terminated by the end of the block or an output line, so
223 226 # we parse out the rest of the input line if it is
224 227 # multiline as well as any echo text
225 228
226 229 rest = []
227 230 while i<N:
228 231
229 232 # look ahead; if the next line is blank, or a comment, or
230 233 # an output line, we're done
231 234
232 235 nextline = lines[i]
233 236 matchout = rgxout.match(nextline)
234 237 #print "nextline=%s, continuation=%s, starts=%s"%(nextline, continuation, nextline.startswith(continuation))
235 238 if matchout or nextline.startswith('#'):
236 239 break
237 240 elif nextline.startswith(continuation):
238 241 # The default ipython_rgx* treat the space following the colon as optional.
239 242 # However, If the space is there we must consume it or code
240 243 # employing the cython_magic extension will fail to execute.
241 244 #
242 245 # This works with the default ipython_rgx* patterns,
243 246 # If you modify them, YMMV.
244 247 nextline = nextline[Nc:]
245 248 if nextline and nextline[0] == ' ':
246 249 nextline = nextline[1:]
247 250
248 251 inputline += '\n' + nextline
249 252 else:
250 253 rest.append(nextline)
251 254 i+= 1
252 255
253 256 block.append((INPUT, (decorator, inputline, '\n'.join(rest))))
254 257 continue
255 258
256 259 # if it looks like an output line grab all the text to the end
257 260 # of the block
258 261 matchout = rgxout.match(line)
259 262 if matchout:
260 263 lineno, output = int(matchout.group(1)), matchout.group(2)
261 264 if i<N-1:
262 265 output = '\n'.join([output] + lines[i:])
263 266
264 267 block.append((OUTPUT, output))
265 268 break
266 269
267 270 return block
268 271
269 272
270 273 class EmbeddedSphinxShell(object):
271 274 """An embedded IPython instance to run inside Sphinx"""
272 275
273 276 def __init__(self, exec_lines=None):
274 277
275 278 self.cout = StringIO()
276 279
277 280 if exec_lines is None:
278 281 exec_lines = []
279 282
280 283 # Create config object for IPython
281 284 config = Config()
282 285 config.InteractiveShell.autocall = False
283 286 config.InteractiveShell.autoindent = False
284 287 config.InteractiveShell.colors = 'NoColor'
285 288
286 289 # create a profile so instance history isn't saved
287 290 tmp_profile_dir = tempfile.mkdtemp(prefix='profile_')
288 291 profname = 'auto_profile_sphinx_build'
289 292 pdir = os.path.join(tmp_profile_dir,profname)
290 293 profile = ProfileDir.create_profile_dir(pdir)
291 294
292 295 # Create and initialize global ipython, but don't start its mainloop.
293 296 # This will persist across different EmbededSphinxShell instances.
294 297 IP = InteractiveShell.instance(config=config, profile_dir=profile)
295 298
296 299 # io.stdout redirect must be done after instantiating InteractiveShell
297 300 io.stdout = self.cout
298 301 io.stderr = self.cout
299 302
300 303 # For debugging, so we can see normal output, use this:
301 304 #from IPython.utils.io import Tee
302 305 #io.stdout = Tee(self.cout, channel='stdout') # dbg
303 306 #io.stderr = Tee(self.cout, channel='stderr') # dbg
304 307
305 308 # Store a few parts of IPython we'll need.
306 309 self.IP = IP
307 310 self.user_ns = self.IP.user_ns
308 311 self.user_global_ns = self.IP.user_global_ns
309 312
310 313 self.input = ''
311 314 self.output = ''
312 315
313 316 self.is_verbatim = False
314 317 self.is_doctest = False
315 318 self.is_suppress = False
316 319
317 320 # Optionally, provide more detailed information to shell.
318 321 # this is assigned by the SetUp method of IPythonDirective
319 322 # to point at itself.
320 323 #
321 324 # So, you can access handy things at self.directive.state
322 325 self.directive = None
323 326
324 327 # on the first call to the savefig decorator, we'll import
325 328 # pyplot as plt so we can make a call to the plt.gcf().savefig
326 329 self._pyplot_imported = False
327 330
328 331 # Prepopulate the namespace.
329 332 for line in exec_lines:
330 333 self.process_input_line(line, store_history=False)
331 334
332 335 def clear_cout(self):
333 336 self.cout.seek(0)
334 337 self.cout.truncate(0)
335 338
336 339 def process_input_line(self, line, store_history=True):
337 340 """process the input, capturing stdout"""
338 341
339 342 stdout = sys.stdout
340 343 splitter = self.IP.input_splitter
341 344 try:
342 345 sys.stdout = self.cout
343 346 splitter.push(line)
344 347 more = splitter.push_accepts_more()
345 348 if not more:
346 349 source_raw = splitter.raw_reset()
347 350 self.IP.run_cell(source_raw, store_history=store_history)
348 351 finally:
349 352 sys.stdout = stdout
350 353
351 354 def process_image(self, decorator):
352 355 """
353 356 # build out an image directive like
354 357 # .. image:: somefile.png
355 358 # :width 4in
356 359 #
357 360 # from an input like
358 361 # savefig somefile.png width=4in
359 362 """
360 363 savefig_dir = self.savefig_dir
361 364 source_dir = self.source_dir
362 365 saveargs = decorator.split(' ')
363 366 filename = saveargs[1]
364 367 # insert relative path to image file in source
365 368 outfile = os.path.relpath(os.path.join(savefig_dir,filename),
366 369 source_dir)
367 370
368 371 imagerows = ['.. image:: %s'%outfile]
369 372
370 373 for kwarg in saveargs[2:]:
371 374 arg, val = kwarg.split('=')
372 375 arg = arg.strip()
373 376 val = val.strip()
374 377 imagerows.append(' :%s: %s'%(arg, val))
375 378
376 379 image_file = os.path.basename(outfile) # only return file name
377 380 image_directive = '\n'.join(imagerows)
378 381 return image_file, image_directive
379 382
380 383 # Callbacks for each type of token
381 384 def process_input(self, data, input_prompt, lineno):
382 385 """
383 386 Process data block for INPUT token.
384 387
385 388 """
386 389 decorator, input, rest = data
387 390 image_file = None
388 391 image_directive = None
389 392
390 393 is_verbatim = decorator=='@verbatim' or self.is_verbatim
391 394 is_doctest = (decorator is not None and \
392 395 decorator.startswith('@doctest')) or self.is_doctest
393 396 is_suppress = decorator=='@suppress' or self.is_suppress
394 397 is_okexcept = decorator=='@okexcept' or self.is_okexcept
395 398 is_okwarning = decorator=='@okwarning' or self.is_okwarning
396 399 is_savefig = decorator is not None and \
397 400 decorator.startswith('@savefig')
398 401
399 402 input_lines = input.split('\n')
400 403 if len(input_lines) > 1:
401 404 if input_lines[-1] != "":
402 405 input_lines.append('') # make sure there's a blank line
403 406 # so splitter buffer gets reset
404 407
405 408 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2))
406 409
407 410 if is_savefig:
408 411 image_file, image_directive = self.process_image(decorator)
409 412
410 413 ret = []
411 414 is_semicolon = False
412 415
413 416 # Hold the execution count, if requested to do so.
414 417 if is_suppress and self.hold_count:
415 418 store_history = False
416 419 else:
417 420 store_history = True
418 421
419 422 # Note: catch_warnings is not thread safe
420 423 with warnings.catch_warnings(record=True) as ws:
421 424 for i, line in enumerate(input_lines):
422 425 if line.endswith(';'):
423 426 is_semicolon = True
424 427
425 428 if i == 0:
426 429 # process the first input line
427 430 if is_verbatim:
428 431 self.process_input_line('')
429 432 self.IP.execution_count += 1 # increment it anyway
430 433 else:
431 434 # only submit the line in non-verbatim mode
432 435 self.process_input_line(line, store_history=store_history)
433 436 formatted_line = '%s %s'%(input_prompt, line)
434 437 else:
435 438 # process a continuation line
436 439 if not is_verbatim:
437 440 self.process_input_line(line, store_history=store_history)
438 441
439 442 formatted_line = '%s %s'%(continuation, line)
440 443
441 444 if not is_suppress:
442 445 ret.append(formatted_line)
443 446
444 447 if not is_suppress and len(rest.strip()) and is_verbatim:
445 # the "rest" is the standard output of the
446 # input, which needs to be added in
447 # verbatim mode
448 # The "rest" is the standard output of the input.
449 # This needs to be added, even in verbatim mode.
448 450 ret.append(rest)
449 451
452 # Fetch the processed output. (This is not the submitted output.)
450 453 self.cout.seek(0)
451 output = self.cout.read()
454 processed_output = self.cout.read()
452 455 if not is_suppress and not is_semicolon:
453 ret.append(output)
454 elif is_semicolon: # get spacing right
456 #
457 # In IPythonDirective.run, the elements of `ret` are eventually
458 # combined such that '' entries correspond to newlines. So if
459 # `processed_output` is equal to '', then the adding it to `ret`
460 # ensures that there is a blank line between consecutive inputs
461 # that have no outputs, as in:
462 #
463 # In [1]: x = 4
464 #
465 # In [2]: x = 5
466 #
467 # When there is processed output, it has a '\n' at the tail end. So
468 # adding the output to `ret` will provide the necessary spacing
469 # between consecutive input/output blocks, as in:
470 #
471 # In [1]: x
472 # Out[1]: 5
473 #
474 # In [2]: x
475 # Out[2]: 5
476 #
477 # When there is stdout from the input, it also has a '\n' at the
478 # tail end, and so this ensures proper spacing as well. E.g.:
479 #
480 # In [1]: print x
481 # 5
482 #
483 # In [2]: x = 5
484 #
485 # When in verbatim mode, `processed_output` is empty (because
486 # nothing was passed to IP. Sometimes the submitted code block has
487 # an Out[] portion and sometimes it does not. When it does not, we
488 # need to ensure proper spacing, so we have to add '' to `ret`.
489 # However, if there is an Out[] in the submitted code, then we do
490 # not want to add a newline as `process_output` has stuff to add.
491 # The difficulty is that `process_input` doesn't know if
492 # `process_output` will be called---so it doesn't know if there is
493 # Out[] in the code block. The requires that we include a hack in
494 # `process_block`. See the comments there.
495 #
496 ret.append(processed_output)
497 elif is_semicolon:
498 # Make sure there is a newline after the semicolon.
455 499 ret.append('')
456 500
457 501 # context information
458 502 filename = "Unknown"
459 503 lineno = 0
460 504 if self.directive.state:
461 505 filename = self.directive.state.document.current_source
462 506 lineno = self.directive.state.document.current_line
463 507
464 508 # output any exceptions raised during execution to stdout
465 509 # unless :okexcept: has been specified.
466 if not is_okexcept and "Traceback" in output:
510 if not is_okexcept and "Traceback" in processed_output:
467 511 s = "\nException in %s at block ending on line %s\n" % (filename, lineno)
468 512 s += "Specify :okexcept: as an option in the ipython:: block to suppress this message\n"
469 513 sys.stdout.write('\n\n>>>' + ('-' * 73))
470 514 sys.stdout.write(s)
471 sys.stdout.write(output)
515 sys.stdout.write(processed_output)
472 516 sys.stdout.write('<<<' + ('-' * 73) + '\n\n')
473 517
474 518 # output any warning raised during execution to stdout
475 519 # unless :okwarning: has been specified.
476 520 if not is_okwarning:
477 521 for w in ws:
478 522 s = "\nWarning in %s at block ending on line %s\n" % (filename, lineno)
479 523 s += "Specify :okwarning: as an option in the ipython:: block to suppress this message\n"
480 524 sys.stdout.write('\n\n>>>' + ('-' * 73))
481 525 sys.stdout.write(s)
482 526 sys.stdout.write(('-' * 76) + '\n')
483 527 s=warnings.formatwarning(w.message, w.category,
484 528 w.filename, w.lineno, w.line)
485 529 sys.stdout.write(s)
486 530 sys.stdout.write('<<<' + ('-' * 73) + '\n')
487 531
488 532 self.cout.truncate(0)
489 return (ret, input_lines, output, is_doctest, decorator, image_file,
490 image_directive)
533
534 return (ret, input_lines, processed_output,
535 is_doctest, decorator, image_file, image_directive)
491 536
492 537
493 def process_output(self, data, output_prompt,
494 input_lines, output, is_doctest, decorator, image_file):
538 def process_output(self, data, output_prompt, input_lines, output,
539 is_doctest, decorator, image_file):
495 540 """
496 541 Process data block for OUTPUT token.
497 542
498 543 """
544 # Recall: `data` is the submitted output, and `output` is the processed
545 # output from `input_lines`.
546
499 547 TAB = ' ' * 4
500 548
501 549 if is_doctest and output is not None:
502 550
503 found = output
551 found = output # This is the processed output
504 552 found = found.strip()
505 553 submitted = data.strip()
506 554
507 555 if self.directive is None:
508 556 source = 'Unavailable'
509 557 content = 'Unavailable'
510 558 else:
511 559 source = self.directive.state.document.current_source
512 560 content = self.directive.content
513 561 # Add tabs and join into a single string.
514 562 content = '\n'.join([TAB + line for line in content])
515 563
516 564 # Make sure the output contains the output prompt.
517 565 ind = found.find(output_prompt)
518 566 if ind < 0:
519 567 e = ('output does not contain output prompt\n\n'
520 568 'Document source: {0}\n\n'
521 569 'Raw content: \n{1}\n\n'
522 570 'Input line(s):\n{TAB}{2}\n\n'
523 571 'Output line(s):\n{TAB}{3}\n\n')
524 572 e = e.format(source, content, '\n'.join(input_lines),
525 573 repr(found), TAB=TAB)
526 574 raise RuntimeError(e)
527 575 found = found[len(output_prompt):].strip()
528 576
529 577 # Handle the actual doctest comparison.
530 578 if decorator.strip() == '@doctest':
531 579 # Standard doctest
532 580 if found != submitted:
533 581 e = ('doctest failure\n\n'
534 582 'Document source: {0}\n\n'
535 583 'Raw content: \n{1}\n\n'
536 584 'On input line(s):\n{TAB}{2}\n\n'
537 585 'we found output:\n{TAB}{3}\n\n'
538 586 'instead of the expected:\n{TAB}{4}\n\n')
539 587 e = e.format(source, content, '\n'.join(input_lines),
540 588 repr(found), repr(submitted), TAB=TAB)
541 589 raise RuntimeError(e)
542 590 else:
543 591 self.custom_doctest(decorator, input_lines, found, submitted)
544 592
545 # Make sure output is processed when in verbatim mode.
593 # When in verbatim mode, this holds additional submitted output
594 # to be written in the final Sphinx output.
546 595 # https://github.com/ipython/ipython/issues/5776
596 out_data = []
597
547 598 is_verbatim = decorator=='@verbatim' or self.is_verbatim
548 if is_verbatim:
549 out_data = data
550 else:
551 out_data = None
599 if is_verbatim and data.strip():
600 # Note that `ret` in `process_block` has '' as its last element if
601 # the code block was in verbatim mode. So if there is no submitted
602 # output, then we will have proper spacing only if we do not add
603 # an additional '' to `out_data`. This is why we condition on
604 # `and data.strip()`.
605
606 # The submitted output has no output prompt. If we want the
607 # prompt and the code to appear, we need to join them now
608 # instead of adding them separately---as this would create an
609 # undesired newline. How we do this ultimately depends on the
610 # format of the output regex. I'll do what works for the default
611 # prompt for now, and we might have to adjust if it doesn't work
612 # in other cases. Finally, the submitted output does not have
613 # a trailing newline, so we must add it manually.
614 out_data.append("{0} {1}\n".format(output_prompt, data))
552 615
553 616 return out_data
554 617
555 618 def process_comment(self, data):
556 619 """Process data fPblock for COMMENT token."""
557 620 if not self.is_suppress:
558 621 return [data]
559 622
560 623 def save_image(self, image_file):
561 624 """
562 625 Saves the image file to disk.
563 626 """
564 627 self.ensure_pyplot()
565 628 command = 'plt.gcf().savefig("%s")'%image_file
566 629 #print 'SAVEFIG', command # dbg
567 630 self.process_input_line('bookmark ipy_thisdir', store_history=False)
568 631 self.process_input_line('cd -b ipy_savedir', store_history=False)
569 632 self.process_input_line(command, store_history=False)
570 633 self.process_input_line('cd -b ipy_thisdir', store_history=False)
571 634 self.process_input_line('bookmark -d ipy_thisdir', store_history=False)
572 635 self.clear_cout()
573 636
574 637 def process_block(self, block):
575 638 """
576 639 process block from the block_parser and return a list of processed lines
577 640 """
578 641 ret = []
579 642 output = None
580 643 input_lines = None
581 644 lineno = self.IP.execution_count
582 645
583 646 input_prompt = self.promptin % lineno
584 647 output_prompt = self.promptout % lineno
585 648 image_file = None
586 649 image_directive = None
587 650
588 651 for token, data in block:
589 652 if token == COMMENT:
590 653 out_data = self.process_comment(data)
591 654 elif token == INPUT:
592 (out_data, input_lines, output, is_doctest, decorator,
593 image_file, image_directive) = \
655 (out_data, input_lines, output, is_doctest,
656 decorator, image_file, image_directive) = \
594 657 self.process_input(data, input_prompt, lineno)
595 658 elif token == OUTPUT:
596 659 out_data = \
597 self.process_output(data, output_prompt,
598 input_lines, output, is_doctest,
599 decorator, image_file)
660 self.process_output(data, output_prompt, input_lines,
661 output, is_doctest, decorator,
662 image_file)
663 if out_data:
664 # Then there was user submitted output in verbatim mode.
665 # We need to remove the last element of `ret` that was
666 # added in `process_input`, as it is '' and would introduce
667 # an undesirable newline.
668 assert(ret[-1] == '')
669 del ret[-1]
670
600 671 if out_data:
601 672 ret.extend(out_data)
602 673
603 674 # save the image files
604 675 if image_file is not None:
605 676 self.save_image(image_file)
606 677
607 678 return ret, image_directive
608 679
609 680 def ensure_pyplot(self):
610 681 """
611 682 Ensures that pyplot has been imported into the embedded IPython shell.
612 683
613 684 Also, makes sure to set the backend appropriately if not set already.
614 685
615 686 """
616 687 # We are here if the @figure pseudo decorator was used. Thus, it's
617 688 # possible that we could be here even if python_mplbackend were set to
618 689 # `None`. That's also strange and perhaps worthy of raising an
619 690 # exception, but for now, we just set the backend to 'agg'.
620 691
621 692 if not self._pyplot_imported:
622 693 if 'matplotlib.backends' not in sys.modules:
623 694 # Then ipython_matplotlib was set to None but there was a
624 695 # call to the @figure decorator (and ipython_execlines did
625 696 # not set a backend).
626 697 #raise Exception("No backend was set, but @figure was used!")
627 698 import matplotlib
628 699 matplotlib.use('agg')
629 700
630 701 # Always import pyplot into embedded shell.
631 702 self.process_input_line('import matplotlib.pyplot as plt',
632 703 store_history=False)
633 704 self._pyplot_imported = True
634 705
635 706 def process_pure_python(self, content):
636 707 """
637 708 content is a list of strings. it is unedited directive content
638 709
639 710 This runs it line by line in the InteractiveShell, prepends
640 711 prompts as needed capturing stderr and stdout, then returns
641 712 the content as a list as if it were ipython code
642 713 """
643 714 output = []
644 715 savefig = False # keep up with this to clear figure
645 716 multiline = False # to handle line continuation
646 717 multiline_start = None
647 718 fmtin = self.promptin
648 719
649 720 ct = 0
650 721
651 722 for lineno, line in enumerate(content):
652 723
653 724 line_stripped = line.strip()
654 725 if not len(line):
655 726 output.append(line)
656 727 continue
657 728
658 729 # handle decorators
659 730 if line_stripped.startswith('@'):
660 731 output.extend([line])
661 732 if 'savefig' in line:
662 733 savefig = True # and need to clear figure
663 734 continue
664 735
665 736 # handle comments
666 737 if line_stripped.startswith('#'):
667 738 output.extend([line])
668 739 continue
669 740
670 741 # deal with lines checking for multiline
671 742 continuation = u' %s:'% ''.join(['.']*(len(str(ct))+2))
672 743 if not multiline:
673 744 modified = u"%s %s" % (fmtin % ct, line_stripped)
674 745 output.append(modified)
675 746 ct += 1
676 747 try:
677 748 ast.parse(line_stripped)
678 749 output.append(u'')
679 750 except Exception: # on a multiline
680 751 multiline = True
681 752 multiline_start = lineno
682 753 else: # still on a multiline
683 754 modified = u'%s %s' % (continuation, line)
684 755 output.append(modified)
685 756
686 757 # if the next line is indented, it should be part of multiline
687 758 if len(content) > lineno + 1:
688 759 nextline = content[lineno + 1]
689 760 if len(nextline) - len(nextline.lstrip()) > 3:
690 761 continue
691 762 try:
692 763 mod = ast.parse(
693 764 '\n'.join(content[multiline_start:lineno+1]))
694 765 if isinstance(mod.body[0], ast.FunctionDef):
695 766 # check to see if we have the whole function
696 767 for element in mod.body[0].body:
697 768 if isinstance(element, ast.Return):
698 769 multiline = False
699 770 else:
700 771 output.append(u'')
701 772 multiline = False
702 773 except Exception:
703 774 pass
704 775
705 776 if savefig: # clear figure if plotted
706 777 self.ensure_pyplot()
707 778 self.process_input_line('plt.clf()', store_history=False)
708 779 self.clear_cout()
709 780 savefig = False
710 781
711 782 return output
712 783
713 784 def custom_doctest(self, decorator, input_lines, found, submitted):
714 785 """
715 786 Perform a specialized doctest.
716 787
717 788 """
718 789 from .custom_doctests import doctests
719 790
720 791 args = decorator.split()
721 792 doctest_type = args[1]
722 793 if doctest_type in doctests:
723 794 doctests[doctest_type](self, args, input_lines, found, submitted)
724 795 else:
725 796 e = "Invalid option to @doctest: {0}".format(doctest_type)
726 797 raise Exception(e)
727 798
728 799
729 800 class IPythonDirective(Directive):
730 801
731 802 has_content = True
732 803 required_arguments = 0
733 804 optional_arguments = 4 # python, suppress, verbatim, doctest
734 805 final_argumuent_whitespace = True
735 806 option_spec = { 'python': directives.unchanged,
736 807 'suppress' : directives.flag,
737 808 'verbatim' : directives.flag,
738 809 'doctest' : directives.flag,
739 810 'okexcept': directives.flag,
740 811 'okwarning': directives.flag
741 812 }
742 813
743 814 shell = None
744 815
745 816 seen_docs = set()
746 817
747 818 def get_config_options(self):
748 819 # contains sphinx configuration variables
749 820 config = self.state.document.settings.env.config
750 821
751 822 # get config variables to set figure output directory
752 823 confdir = self.state.document.settings.env.app.confdir
753 824 savefig_dir = config.ipython_savefig_dir
754 825 source_dir = os.path.dirname(self.state.document.current_source)
755 826 if savefig_dir is None:
756 827 savefig_dir = config.html_static_path
757 828 if isinstance(savefig_dir, list):
758 829 savefig_dir = savefig_dir[0] # safe to assume only one path?
759 830 savefig_dir = os.path.join(confdir, savefig_dir)
760 831
761 832 # get regex and prompt stuff
762 833 rgxin = config.ipython_rgxin
763 834 rgxout = config.ipython_rgxout
764 835 promptin = config.ipython_promptin
765 836 promptout = config.ipython_promptout
766 837 mplbackend = config.ipython_mplbackend
767 838 exec_lines = config.ipython_execlines
768 839 hold_count = config.ipython_holdcount
769 840
770 841 return (savefig_dir, source_dir, rgxin, rgxout,
771 842 promptin, promptout, mplbackend, exec_lines, hold_count)
772 843
773 844 def setup(self):
774 845 # Get configuration values.
775 846 (savefig_dir, source_dir, rgxin, rgxout, promptin, promptout,
776 847 mplbackend, exec_lines, hold_count) = self.get_config_options()
777 848
778 849 if self.shell is None:
779 850 # We will be here many times. However, when the
780 851 # EmbeddedSphinxShell is created, its interactive shell member
781 852 # is the same for each instance.
782 853
783 854 if mplbackend:
784 855 import matplotlib
785 856 # Repeated calls to use() will not hurt us since `mplbackend`
786 857 # is the same each time.
787 858 matplotlib.use(mplbackend)
788 859
789 860 # Must be called after (potentially) importing matplotlib and
790 861 # setting its backend since exec_lines might import pylab.
791 862 self.shell = EmbeddedSphinxShell(exec_lines)
792 863
793 864 # Store IPython directive to enable better error messages
794 865 self.shell.directive = self
795 866
796 867 # reset the execution count if we haven't processed this doc
797 868 #NOTE: this may be borked if there are multiple seen_doc tmp files
798 869 #check time stamp?
799 870 if not self.state.document.current_source in self.seen_docs:
800 871 self.shell.IP.history_manager.reset()
801 872 self.shell.IP.execution_count = 1
802 873 self.shell.IP.prompt_manager.width = 0
803 874 self.seen_docs.add(self.state.document.current_source)
804 875
805 876 # and attach to shell so we don't have to pass them around
806 877 self.shell.rgxin = rgxin
807 878 self.shell.rgxout = rgxout
808 879 self.shell.promptin = promptin
809 880 self.shell.promptout = promptout
810 881 self.shell.savefig_dir = savefig_dir
811 882 self.shell.source_dir = source_dir
812 883 self.shell.hold_count = hold_count
813 884
814 885 # setup bookmark for saving figures directory
815 886 self.shell.process_input_line('bookmark ipy_savedir %s'%savefig_dir,
816 887 store_history=False)
817 888 self.shell.clear_cout()
818 889
819 890 return rgxin, rgxout, promptin, promptout
820 891
821 892 def teardown(self):
822 893 # delete last bookmark
823 894 self.shell.process_input_line('bookmark -d ipy_savedir',
824 895 store_history=False)
825 896 self.shell.clear_cout()
826 897
827 898 def run(self):
828 899 debug = False
829 900
830 901 #TODO, any reason block_parser can't be a method of embeddable shell
831 902 # then we wouldn't have to carry these around
832 903 rgxin, rgxout, promptin, promptout = self.setup()
833 904
834 905 options = self.options
835 906 self.shell.is_suppress = 'suppress' in options
836 907 self.shell.is_doctest = 'doctest' in options
837 908 self.shell.is_verbatim = 'verbatim' in options
838 909 self.shell.is_okexcept = 'okexcept' in options
839 910 self.shell.is_okwarning = 'okwarning' in options
840 911
841 912 # handle pure python code
842 913 if 'python' in self.arguments:
843 914 content = self.content
844 915 self.content = self.shell.process_pure_python(content)
845 916
846 917 parts = '\n'.join(self.content).split('\n\n')
847 918
848 919 lines = ['.. code-block:: ipython', '']
849 920 figures = []
850 921
851 922 for part in parts:
852 923 block = block_parser(part, rgxin, rgxout, promptin, promptout)
853 924 if len(block):
854 925 rows, figure = self.shell.process_block(block)
855 926 for row in rows:
856 lines.extend([' %s'%line for line in row.split('\n')])
927 lines.extend([' {0}'.format(line)
928 for line in row.split('\n')])
857 929
858 930 if figure is not None:
859 931 figures.append(figure)
860 932
861 933 for figure in figures:
862 934 lines.append('')
863 935 lines.extend(figure.split('\n'))
864 936 lines.append('')
865 937
866 if len(lines)>2:
938 if len(lines) > 2:
867 939 if debug:
868 940 print('\n'.join(lines))
869 941 else:
870 942 # This has to do with input, not output. But if we comment
871 943 # these lines out, then no IPython code will appear in the
872 944 # final output.
873 945 self.state_machine.insert_input(
874 946 lines, self.state_machine.input_lines.source(0))
875 947
876 948 # cleanup
877 949 self.teardown()
878 950
879 951 return []
880 952
881 953 # Enable as a proper Sphinx directive
882 954 def setup(app):
883 955 setup.app = app
884 956
885 957 app.add_directive('ipython', IPythonDirective)
886 958 app.add_config_value('ipython_savefig_dir', None, 'env')
887 959 app.add_config_value('ipython_rgxin',
888 960 re.compile('In \[(\d+)\]:\s?(.*)\s*'), 'env')
889 961 app.add_config_value('ipython_rgxout',
890 962 re.compile('Out\[(\d+)\]:\s?(.*)\s*'), 'env')
891 963 app.add_config_value('ipython_promptin', 'In [%d]:', 'env')
892 964 app.add_config_value('ipython_promptout', 'Out[%d]:', 'env')
893 965
894 966 # We could just let matplotlib pick whatever is specified as the default
895 967 # backend in the matplotlibrc file, but this would cause issues if the
896 968 # backend didn't work in headless environments. For this reason, 'agg'
897 969 # is a good default backend choice.
898 970 app.add_config_value('ipython_mplbackend', 'agg', 'env')
899 971
900 972 # If the user sets this config value to `None`, then EmbeddedSphinxShell's
901 973 # __init__ method will treat it as [].
902 974 execlines = ['import numpy as np', 'import matplotlib.pyplot as plt']
903 975 app.add_config_value('ipython_execlines', execlines, 'env')
904 976
905 977 app.add_config_value('ipython_holdcount', True, 'env')
906 978
907 979 # Simple smoke test, needs to be converted to a proper automatic test.
908 980 def test():
909 981
910 982 examples = [
911 983 r"""
912 984 In [9]: pwd
913 985 Out[9]: '/home/jdhunter/py4science/book'
914 986
915 987 In [10]: cd bookdata/
916 988 /home/jdhunter/py4science/book/bookdata
917 989
918 990 In [2]: from pylab import *
919 991
920 992 In [2]: ion()
921 993
922 994 In [3]: im = imread('stinkbug.png')
923 995
924 996 @savefig mystinkbug.png width=4in
925 997 In [4]: imshow(im)
926 998 Out[4]: <matplotlib.image.AxesImage object at 0x39ea850>
927 999
928 1000 """,
929 1001 r"""
930 1002
931 1003 In [1]: x = 'hello world'
932 1004
933 1005 # string methods can be
934 1006 # used to alter the string
935 1007 @doctest
936 1008 In [2]: x.upper()
937 1009 Out[2]: 'HELLO WORLD'
938 1010
939 1011 @verbatim
940 1012 In [3]: x.st<TAB>
941 1013 x.startswith x.strip
942 1014 """,
943 1015 r"""
944 1016
945 1017 In [130]: url = 'http://ichart.finance.yahoo.com/table.csv?s=CROX\
946 1018 .....: &d=9&e=22&f=2009&g=d&a=1&br=8&c=2006&ignore=.csv'
947 1019
948 1020 In [131]: print url.split('&')
949 1021 ['http://ichart.finance.yahoo.com/table.csv?s=CROX', 'd=9', 'e=22', 'f=2009', 'g=d', 'a=1', 'b=8', 'c=2006', 'ignore=.csv']
950 1022
951 1023 In [60]: import urllib
952 1024
953 1025 """,
954 1026 r"""\
955 1027
956 1028 In [133]: import numpy.random
957 1029
958 1030 @suppress
959 1031 In [134]: numpy.random.seed(2358)
960 1032
961 1033 @doctest
962 1034 In [135]: numpy.random.rand(10,2)
963 1035 Out[135]:
964 1036 array([[ 0.64524308, 0.59943846],
965 1037 [ 0.47102322, 0.8715456 ],
966 1038 [ 0.29370834, 0.74776844],
967 1039 [ 0.99539577, 0.1313423 ],
968 1040 [ 0.16250302, 0.21103583],
969 1041 [ 0.81626524, 0.1312433 ],
970 1042 [ 0.67338089, 0.72302393],
971 1043 [ 0.7566368 , 0.07033696],
972 1044 [ 0.22591016, 0.77731835],
973 1045 [ 0.0072729 , 0.34273127]])
974 1046
975 1047 """,
976 1048
977 1049 r"""
978 1050 In [106]: print x
979 1051 jdh
980 1052
981 1053 In [109]: for i in range(10):
982 1054 .....: print i
983 1055 .....:
984 1056 .....:
985 1057 0
986 1058 1
987 1059 2
988 1060 3
989 1061 4
990 1062 5
991 1063 6
992 1064 7
993 1065 8
994 1066 9
995 1067 """,
996 1068
997 1069 r"""
998 1070
999 1071 In [144]: from pylab import *
1000 1072
1001 1073 In [145]: ion()
1002 1074
1003 1075 # use a semicolon to suppress the output
1004 1076 @savefig test_hist.png width=4in
1005 1077 In [151]: hist(np.random.randn(10000), 100);
1006 1078
1007 1079
1008 1080 @savefig test_plot.png width=4in
1009 1081 In [151]: plot(np.random.randn(10000), 'o');
1010 1082 """,
1011 1083
1012 1084 r"""
1013 1085 # use a semicolon to suppress the output
1014 1086 In [151]: plt.clf()
1015 1087
1016 1088 @savefig plot_simple.png width=4in
1017 1089 In [151]: plot([1,2,3])
1018 1090
1019 1091 @savefig hist_simple.png width=4in
1020 1092 In [151]: hist(np.random.randn(10000), 100);
1021 1093
1022 1094 """,
1023 1095 r"""
1024 1096 # update the current fig
1025 1097 In [151]: ylabel('number')
1026 1098
1027 1099 In [152]: title('normal distribution')
1028 1100
1029 1101
1030 1102 @savefig hist_with_text.png
1031 1103 In [153]: grid(True)
1032 1104
1033 1105 @doctest float
1034 1106 In [154]: 0.1 + 0.2
1035 1107 Out[154]: 0.3
1036 1108
1037 1109 @doctest float
1038 1110 In [155]: np.arange(16).reshape(4,4)
1039 1111 Out[155]:
1040 1112 array([[ 0, 1, 2, 3],
1041 1113 [ 4, 5, 6, 7],
1042 1114 [ 8, 9, 10, 11],
1043 1115 [12, 13, 14, 15]])
1044 1116
1045 1117 In [1]: x = np.arange(16, dtype=float).reshape(4,4)
1046 1118
1047 1119 In [2]: x[0,0] = np.inf
1048 1120
1049 1121 In [3]: x[0,1] = np.nan
1050 1122
1051 1123 @doctest float
1052 1124 In [4]: x
1053 1125 Out[4]:
1054 1126 array([[ inf, nan, 2., 3.],
1055 1127 [ 4., 5., 6., 7.],
1056 1128 [ 8., 9., 10., 11.],
1057 1129 [ 12., 13., 14., 15.]])
1058 1130
1059 1131
1060 1132 """,
1061 1133 ]
1062 1134 # skip local-file depending first example:
1063 1135 examples = examples[1:]
1064 1136
1065 1137 #ipython_directive.DEBUG = True # dbg
1066 1138 #options = dict(suppress=True) # dbg
1067 1139 options = dict()
1068 1140 for example in examples:
1069 1141 content = example.split('\n')
1070 1142 IPythonDirective('debug', arguments=None, options=options,
1071 1143 content=content, lineno=0,
1072 1144 content_offset=None, block_text=None,
1073 1145 state=None, state_machine=None,
1074 1146 )
1075 1147
1076 1148 # Run test suite as a script
1077 1149 if __name__=='__main__':
1078 1150 if not os.path.isdir('_static'):
1079 1151 os.mkdir('_static')
1080 1152 test()
1081 1153 print('All OK? Check figures in _static/')
General Comments 0
You need to be logged in to leave comments. Login now