##// END OF EJS Templates
Merge pull request #13030 from rhshadrach/ipython_warning...
Blazej Michalik -
r26627:6c01e9bd merge
parent child Browse files
Show More
@@ -0,0 +1,4 b''
1 IPython directive warnings
2 --------------------------
3
4 The IPython directive now uses Sphinx logging for warnings.
@@ -1,1251 +1,1260 b''
1 1 # -*- coding: utf-8 -*-
2 2 """
3 3 Sphinx directive to support embedded IPython code.
4 4
5 5 IPython provides an extension for `Sphinx <http://www.sphinx-doc.org/>`_ to
6 6 highlight and run code.
7 7
8 8 This directive allows pasting of entire interactive IPython sessions, prompts
9 9 and all, and their code will actually get re-executed at doc build time, with
10 10 all prompts renumbered sequentially. It also allows you to input code as a pure
11 11 python input by giving the argument python to the directive. The output looks
12 12 like an interactive ipython section.
13 13
14 14 Here is an example of how the IPython directive can
15 15 **run** python code, at build time.
16 16
17 17 .. ipython::
18 18
19 19 In [1]: 1+1
20 20
21 21 In [1]: import datetime
22 22 ...: datetime.datetime.now()
23 23
24 24 It supports IPython construct that plain
25 25 Python does not understand (like magics):
26 26
27 27 .. ipython::
28 28
29 29 In [0]: import time
30 30
31 31 In [0]: %timeit time.sleep(0.05)
32 32
33 33 This will also support top-level async when using IPython 7.0+
34 34
35 35 .. ipython::
36 36
37 37 In [2]: import asyncio
38 38 ...: print('before')
39 39 ...: await asyncio.sleep(1)
40 40 ...: print('after')
41 41
42 42
43 43 The namespace will persist across multiple code chucks, Let's define a variable:
44 44
45 45 .. ipython::
46 46
47 47 In [0]: who = "World"
48 48
49 49 And now say hello:
50 50
51 51 .. ipython::
52 52
53 53 In [0]: print('Hello,', who)
54 54
55 55 If the current section raises an exception, you can add the ``:okexcept:`` flag
56 56 to the current block, otherwise the build will fail.
57 57
58 58 .. ipython::
59 59 :okexcept:
60 60
61 61 In [1]: 1/0
62 62
63 63 IPython Sphinx directive module
64 64 ===============================
65 65
66 66 To enable this directive, simply list it in your Sphinx ``conf.py`` file
67 67 (making sure the directory where you placed it is visible to sphinx, as is
68 68 needed for all Sphinx directives). For example, to enable syntax highlighting
69 69 and the IPython directive::
70 70
71 71 extensions = ['IPython.sphinxext.ipython_console_highlighting',
72 72 'IPython.sphinxext.ipython_directive']
73 73
74 74 The IPython directive outputs code-blocks with the language 'ipython'. So
75 75 if you do not have the syntax highlighting extension enabled as well, then
76 76 all rendered code-blocks will be uncolored. By default this directive assumes
77 77 that your prompts are unchanged IPython ones, but this can be customized.
78 78 The configurable options that can be placed in conf.py are:
79 79
80 80 ipython_savefig_dir:
81 81 The directory in which to save the figures. This is relative to the
82 82 Sphinx source directory. The default is `html_static_path`.
83 83 ipython_rgxin:
84 84 The compiled regular expression to denote the start of IPython input
85 85 lines. The default is ``re.compile('In \\[(\\d+)\\]:\\s?(.*)\\s*')``. You
86 86 shouldn't need to change this.
87 87 ipython_warning_is_error: [default to True]
88 88 Fail the build if something unexpected happen, for example if a block raise
89 89 an exception but does not have the `:okexcept:` flag. The exact behavior of
90 90 what is considered strict, may change between the sphinx directive version.
91 91 ipython_rgxout:
92 92 The compiled regular expression to denote the start of IPython output
93 93 lines. The default is ``re.compile('Out\\[(\\d+)\\]:\\s?(.*)\\s*')``. You
94 94 shouldn't need to change this.
95 95 ipython_promptin:
96 96 The string to represent the IPython input prompt in the generated ReST.
97 97 The default is ``'In [%d]:'``. This expects that the line numbers are used
98 98 in the prompt.
99 99 ipython_promptout:
100 100 The string to represent the IPython prompt in the generated ReST. The
101 101 default is ``'Out [%d]:'``. This expects that the line numbers are used
102 102 in the prompt.
103 103 ipython_mplbackend:
104 104 The string which specifies if the embedded Sphinx shell should import
105 105 Matplotlib and set the backend. The value specifies a backend that is
106 106 passed to `matplotlib.use()` before any lines in `ipython_execlines` are
107 107 executed. If not specified in conf.py, then the default value of 'agg' is
108 108 used. To use the IPython directive without matplotlib as a dependency, set
109 109 the value to `None`. It may end up that matplotlib is still imported
110 110 if the user specifies so in `ipython_execlines` or makes use of the
111 111 @savefig pseudo decorator.
112 112 ipython_execlines:
113 113 A list of strings to be exec'd in the embedded Sphinx shell. Typical
114 114 usage is to make certain packages always available. Set this to an empty
115 115 list if you wish to have no imports always available. If specified in
116 116 ``conf.py`` as `None`, then it has the effect of making no imports available.
117 117 If omitted from conf.py altogether, then the default value of
118 118 ['import numpy as np', 'import matplotlib.pyplot as plt'] is used.
119 119 ipython_holdcount
120 120 When the @suppress pseudo-decorator is used, the execution count can be
121 121 incremented or not. The default behavior is to hold the execution count,
122 122 corresponding to a value of `True`. Set this to `False` to increment
123 123 the execution count after each suppressed command.
124 124
125 125 As an example, to use the IPython directive when `matplotlib` is not available,
126 126 one sets the backend to `None`::
127 127
128 128 ipython_mplbackend = None
129 129
130 130 An example usage of the directive is:
131 131
132 132 .. code-block:: rst
133 133
134 134 .. ipython::
135 135
136 136 In [1]: x = 1
137 137
138 138 In [2]: y = x**2
139 139
140 140 In [3]: print(y)
141 141
142 142 See http://matplotlib.org/sampledoc/ipython_directive.html for additional
143 143 documentation.
144 144
145 145 Pseudo-Decorators
146 146 =================
147 147
148 148 Note: Only one decorator is supported per input. If more than one decorator
149 149 is specified, then only the last one is used.
150 150
151 151 In addition to the Pseudo-Decorators/options described at the above link,
152 152 several enhancements have been made. The directive will emit a message to the
153 153 console at build-time if code-execution resulted in an exception or warning.
154 154 You can suppress these on a per-block basis by specifying the :okexcept:
155 155 or :okwarning: options:
156 156
157 157 .. code-block:: rst
158 158
159 159 .. ipython::
160 160 :okexcept:
161 161 :okwarning:
162 162
163 163 In [1]: 1/0
164 164 In [2]: # raise warning.
165 165
166 166 To Do
167 167 =====
168 168
169 169 - Turn the ad-hoc test() function into a real test suite.
170 170 - Break up ipython-specific functionality from matplotlib stuff into better
171 171 separated code.
172 172
173 173 """
174 174
175 175 # Authors
176 176 # =======
177 177 #
178 178 # - John D Hunter: original author.
179 179 # - Fernando Perez: refactoring, documentation, cleanups, port to 0.11.
180 180 # - VΓ‘clavΕ milauer <eudoxos-AT-arcig.cz>: Prompt generalizations.
181 181 # - Skipper Seabold, refactoring, cleanups, pure python addition
182 182
183 183 #-----------------------------------------------------------------------------
184 184 # Imports
185 185 #-----------------------------------------------------------------------------
186 186
187 187 # Stdlib
188 188 import atexit
189 189 import errno
190 190 import os
191 191 import pathlib
192 192 import re
193 193 import sys
194 194 import tempfile
195 195 import ast
196 196 import warnings
197 197 import shutil
198 198 from io import StringIO
199 199
200 200 # Third-party
201 201 from docutils.parsers.rst import directives
202 202 from docutils.parsers.rst import Directive
203 from sphinx.util import logging
203 204
204 205 # Our own
205 206 from traitlets.config import Config
206 207 from IPython import InteractiveShell
207 208 from IPython.core.profiledir import ProfileDir
208 209
209 210 use_matplotlib = False
210 211 try:
211 212 import matplotlib
212 213 use_matplotlib = True
213 214 except Exception:
214 215 pass
215 216
216 217 #-----------------------------------------------------------------------------
217 218 # Globals
218 219 #-----------------------------------------------------------------------------
219 220 # for tokenizing blocks
220 221 COMMENT, INPUT, OUTPUT = range(3)
221 222
222 223 #-----------------------------------------------------------------------------
223 224 # Functions and class declarations
224 225 #-----------------------------------------------------------------------------
225 226
226 227 def block_parser(part, rgxin, rgxout, fmtin, fmtout):
227 228 """
228 229 part is a string of ipython text, comprised of at most one
229 230 input, one output, comments, and blank lines. The block parser
230 231 parses the text into a list of::
231 232
232 233 blocks = [ (TOKEN0, data0), (TOKEN1, data1), ...]
233 234
234 235 where TOKEN is one of [COMMENT | INPUT | OUTPUT ] and
235 236 data is, depending on the type of token::
236 237
237 238 COMMENT : the comment string
238 239
239 240 INPUT: the (DECORATOR, INPUT_LINE, REST) where
240 241 DECORATOR: the input decorator (or None)
241 242 INPUT_LINE: the input as string (possibly multi-line)
242 243 REST : any stdout generated by the input line (not OUTPUT)
243 244
244 245 OUTPUT: the output string, possibly multi-line
245 246
246 247 """
247 248 block = []
248 249 lines = part.split('\n')
249 250 N = len(lines)
250 251 i = 0
251 252 decorator = None
252 253 while 1:
253 254
254 255 if i==N:
255 256 # nothing left to parse -- the last line
256 257 break
257 258
258 259 line = lines[i]
259 260 i += 1
260 261 line_stripped = line.strip()
261 262 if line_stripped.startswith('#'):
262 263 block.append((COMMENT, line))
263 264 continue
264 265
265 266 if line_stripped.startswith('@'):
266 267 # Here is where we assume there is, at most, one decorator.
267 268 # Might need to rethink this.
268 269 decorator = line_stripped
269 270 continue
270 271
271 272 # does this look like an input line?
272 273 matchin = rgxin.match(line)
273 274 if matchin:
274 275 lineno, inputline = int(matchin.group(1)), matchin.group(2)
275 276
276 277 # the ....: continuation string
277 278 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2))
278 279 Nc = len(continuation)
279 280 # input lines can continue on for more than one line, if
280 281 # we have a '\' line continuation char or a function call
281 282 # echo line 'print'. The input line can only be
282 283 # terminated by the end of the block or an output line, so
283 284 # we parse out the rest of the input line if it is
284 285 # multiline as well as any echo text
285 286
286 287 rest = []
287 288 while i<N:
288 289
289 290 # look ahead; if the next line is blank, or a comment, or
290 291 # an output line, we're done
291 292
292 293 nextline = lines[i]
293 294 matchout = rgxout.match(nextline)
294 295 #print "nextline=%s, continuation=%s, starts=%s"%(nextline, continuation, nextline.startswith(continuation))
295 296 if matchout or nextline.startswith('#'):
296 297 break
297 298 elif nextline.startswith(continuation):
298 299 # The default ipython_rgx* treat the space following the colon as optional.
299 300 # However, If the space is there we must consume it or code
300 301 # employing the cython_magic extension will fail to execute.
301 302 #
302 303 # This works with the default ipython_rgx* patterns,
303 304 # If you modify them, YMMV.
304 305 nextline = nextline[Nc:]
305 306 if nextline and nextline[0] == ' ':
306 307 nextline = nextline[1:]
307 308
308 309 inputline += '\n' + nextline
309 310 else:
310 311 rest.append(nextline)
311 312 i+= 1
312 313
313 314 block.append((INPUT, (decorator, inputline, '\n'.join(rest))))
314 315 continue
315 316
316 317 # if it looks like an output line grab all the text to the end
317 318 # of the block
318 319 matchout = rgxout.match(line)
319 320 if matchout:
320 321 lineno, output = int(matchout.group(1)), matchout.group(2)
321 322 if i<N-1:
322 323 output = '\n'.join([output] + lines[i:])
323 324
324 325 block.append((OUTPUT, output))
325 326 break
326 327
327 328 return block
328 329
329 330
330 331 class EmbeddedSphinxShell(object):
331 332 """An embedded IPython instance to run inside Sphinx"""
332 333
333 334 def __init__(self, exec_lines=None):
334 335
335 336 self.cout = StringIO()
336 337
337 338 if exec_lines is None:
338 339 exec_lines = []
339 340
340 341 # Create config object for IPython
341 342 config = Config()
342 343 config.HistoryManager.hist_file = ':memory:'
343 344 config.InteractiveShell.autocall = False
344 345 config.InteractiveShell.autoindent = False
345 346 config.InteractiveShell.colors = 'NoColor'
346 347
347 348 # create a profile so instance history isn't saved
348 349 tmp_profile_dir = tempfile.mkdtemp(prefix='profile_')
349 350 profname = 'auto_profile_sphinx_build'
350 351 pdir = os.path.join(tmp_profile_dir,profname)
351 352 profile = ProfileDir.create_profile_dir(pdir)
352 353
353 354 # Create and initialize global ipython, but don't start its mainloop.
354 355 # This will persist across different EmbeddedSphinxShell instances.
355 356 IP = InteractiveShell.instance(config=config, profile_dir=profile)
356 357 atexit.register(self.cleanup)
357 358
358 359 # Store a few parts of IPython we'll need.
359 360 self.IP = IP
360 361 self.user_ns = self.IP.user_ns
361 362 self.user_global_ns = self.IP.user_global_ns
362 363
363 364 self.input = ''
364 365 self.output = ''
365 366 self.tmp_profile_dir = tmp_profile_dir
366 367
367 368 self.is_verbatim = False
368 369 self.is_doctest = False
369 370 self.is_suppress = False
370 371
371 372 # Optionally, provide more detailed information to shell.
372 373 # this is assigned by the SetUp method of IPythonDirective
373 374 # to point at itself.
374 375 #
375 376 # So, you can access handy things at self.directive.state
376 377 self.directive = None
377 378
378 379 # on the first call to the savefig decorator, we'll import
379 380 # pyplot as plt so we can make a call to the plt.gcf().savefig
380 381 self._pyplot_imported = False
381 382
382 383 # Prepopulate the namespace.
383 384 for line in exec_lines:
384 385 self.process_input_line(line, store_history=False)
385 386
386 387 def cleanup(self):
387 388 shutil.rmtree(self.tmp_profile_dir, ignore_errors=True)
388 389
389 390 def clear_cout(self):
390 391 self.cout.seek(0)
391 392 self.cout.truncate(0)
392 393
393 394 def process_input_line(self, line, store_history):
394 395 return self.process_input_lines([line], store_history=store_history)
395 396
396 397 def process_input_lines(self, lines, store_history=True):
397 398 """process the input, capturing stdout"""
398 399 stdout = sys.stdout
399 400 source_raw = '\n'.join(lines)
400 401 try:
401 402 sys.stdout = self.cout
402 403 self.IP.run_cell(source_raw, store_history=store_history)
403 404 finally:
404 405 sys.stdout = stdout
405 406
406 407 def process_image(self, decorator):
407 408 """
408 409 # build out an image directive like
409 410 # .. image:: somefile.png
410 411 # :width 4in
411 412 #
412 413 # from an input like
413 414 # savefig somefile.png width=4in
414 415 """
415 416 savefig_dir = self.savefig_dir
416 417 source_dir = self.source_dir
417 418 saveargs = decorator.split(' ')
418 419 filename = saveargs[1]
419 420 # insert relative path to image file in source
420 421 # as absolute path for Sphinx
421 422 # sphinx expects a posix path, even on Windows
422 423 path = pathlib.Path(savefig_dir, filename)
423 424 outfile = '/' + path.relative_to(source_dir).as_posix()
424 425
425 426 imagerows = ['.. image:: %s' % outfile]
426 427
427 428 for kwarg in saveargs[2:]:
428 429 arg, val = kwarg.split('=')
429 430 arg = arg.strip()
430 431 val = val.strip()
431 432 imagerows.append(' :%s: %s'%(arg, val))
432 433
433 434 image_file = os.path.basename(outfile) # only return file name
434 435 image_directive = '\n'.join(imagerows)
435 436 return image_file, image_directive
436 437
437 438 # Callbacks for each type of token
438 439 def process_input(self, data, input_prompt, lineno):
439 440 """
440 441 Process data block for INPUT token.
441 442
442 443 """
443 444 decorator, input, rest = data
444 445 image_file = None
445 446 image_directive = None
446 447
447 448 is_verbatim = decorator=='@verbatim' or self.is_verbatim
448 449 is_doctest = (decorator is not None and \
449 450 decorator.startswith('@doctest')) or self.is_doctest
450 451 is_suppress = decorator=='@suppress' or self.is_suppress
451 452 is_okexcept = decorator=='@okexcept' or self.is_okexcept
452 453 is_okwarning = decorator=='@okwarning' or self.is_okwarning
453 454 is_savefig = decorator is not None and \
454 455 decorator.startswith('@savefig')
455 456
456 457 input_lines = input.split('\n')
457 458 if len(input_lines) > 1:
458 459 if input_lines[-1] != "":
459 460 input_lines.append('') # make sure there's a blank line
460 461 # so splitter buffer gets reset
461 462
462 463 continuation = ' %s:'%''.join(['.']*(len(str(lineno))+2))
463 464
464 465 if is_savefig:
465 466 image_file, image_directive = self.process_image(decorator)
466 467
467 468 ret = []
468 469 is_semicolon = False
469 470
470 471 # Hold the execution count, if requested to do so.
471 472 if is_suppress and self.hold_count:
472 473 store_history = False
473 474 else:
474 475 store_history = True
475 476
476 477 # Note: catch_warnings is not thread safe
477 478 with warnings.catch_warnings(record=True) as ws:
478 479 if input_lines[0].endswith(';'):
479 480 is_semicolon = True
480 481 #for i, line in enumerate(input_lines):
481 482
482 483 # process the first input line
483 484 if is_verbatim:
484 485 self.process_input_lines([''])
485 486 self.IP.execution_count += 1 # increment it anyway
486 487 else:
487 488 # only submit the line in non-verbatim mode
488 489 self.process_input_lines(input_lines, store_history=store_history)
489 490
490 491 if not is_suppress:
491 492 for i, line in enumerate(input_lines):
492 493 if i == 0:
493 494 formatted_line = '%s %s'%(input_prompt, line)
494 495 else:
495 496 formatted_line = '%s %s'%(continuation, line)
496 497 ret.append(formatted_line)
497 498
498 499 if not is_suppress and len(rest.strip()) and is_verbatim:
499 500 # The "rest" is the standard output of the input. This needs to be
500 501 # added when in verbatim mode. If there is no "rest", then we don't
501 502 # add it, as the new line will be added by the processed output.
502 503 ret.append(rest)
503 504
504 505 # Fetch the processed output. (This is not the submitted output.)
505 506 self.cout.seek(0)
506 507 processed_output = self.cout.read()
507 508 if not is_suppress and not is_semicolon:
508 509 #
509 510 # In IPythonDirective.run, the elements of `ret` are eventually
510 511 # combined such that '' entries correspond to newlines. So if
511 512 # `processed_output` is equal to '', then the adding it to `ret`
512 513 # ensures that there is a blank line between consecutive inputs
513 514 # that have no outputs, as in:
514 515 #
515 516 # In [1]: x = 4
516 517 #
517 518 # In [2]: x = 5
518 519 #
519 520 # When there is processed output, it has a '\n' at the tail end. So
520 521 # adding the output to `ret` will provide the necessary spacing
521 522 # between consecutive input/output blocks, as in:
522 523 #
523 524 # In [1]: x
524 525 # Out[1]: 5
525 526 #
526 527 # In [2]: x
527 528 # Out[2]: 5
528 529 #
529 530 # When there is stdout from the input, it also has a '\n' at the
530 531 # tail end, and so this ensures proper spacing as well. E.g.:
531 532 #
532 533 # In [1]: print x
533 534 # 5
534 535 #
535 536 # In [2]: x = 5
536 537 #
537 538 # When in verbatim mode, `processed_output` is empty (because
538 539 # nothing was passed to IP. Sometimes the submitted code block has
539 540 # an Out[] portion and sometimes it does not. When it does not, we
540 541 # need to ensure proper spacing, so we have to add '' to `ret`.
541 542 # However, if there is an Out[] in the submitted code, then we do
542 543 # not want to add a newline as `process_output` has stuff to add.
543 544 # The difficulty is that `process_input` doesn't know if
544 545 # `process_output` will be called---so it doesn't know if there is
545 546 # Out[] in the code block. The requires that we include a hack in
546 547 # `process_block`. See the comments there.
547 548 #
548 549 ret.append(processed_output)
549 550 elif is_semicolon:
550 551 # Make sure there is a newline after the semicolon.
551 552 ret.append('')
552 553
553 554 # context information
554 555 filename = "Unknown"
555 556 lineno = 0
556 557 if self.directive.state:
557 558 filename = self.directive.state.document.current_source
558 559 lineno = self.directive.state.document.current_line
559 560
561 # Use sphinx logger for warnings
562 logger = logging.getLogger(__name__)
563
560 564 # output any exceptions raised during execution to stdout
561 565 # unless :okexcept: has been specified.
562 if not is_okexcept and (("Traceback" in processed_output) or ("SyntaxError" in processed_output)):
563 s = "\nException in %s at block ending on line %s\n" % (filename, lineno)
566 if not is_okexcept and (
567 ("Traceback" in processed_output) or ("SyntaxError" in processed_output)
568 ):
569 s = "\n>>>" + ("-" * 73) + "\n"
570 s += "Exception in %s at block ending on line %s\n" % (filename, lineno)
564 571 s += "Specify :okexcept: as an option in the ipython:: block to suppress this message\n"
565 sys.stdout.write('\n\n>>>' + ('-' * 73))
566 sys.stdout.write(s)
567 sys.stdout.write(processed_output)
568 sys.stdout.write('<<<' + ('-' * 73) + '\n\n')
572 s += processed_output + "\n"
573 s += "<<<" + ("-" * 73)
574 logger.warning(s)
569 575 if self.warning_is_error:
570 576 raise RuntimeError('Non Expected exception in `{}` line {}'.format(filename, lineno))
571 577
572 578 # output any warning raised during execution to stdout
573 579 # unless :okwarning: has been specified.
574 580 if not is_okwarning:
575 581 for w in ws:
576 s = "\nWarning in %s at block ending on line %s\n" % (filename, lineno)
582 s = "\n>>>" + ("-" * 73) + "\n"
583 s += "Warning in %s at block ending on line %s\n" % (filename, lineno)
577 584 s += "Specify :okwarning: as an option in the ipython:: block to suppress this message\n"
578 sys.stdout.write('\n\n>>>' + ('-' * 73))
579 sys.stdout.write(s)
580 sys.stdout.write(('-' * 76) + '\n')
581 s=warnings.formatwarning(w.message, w.category,
582 w.filename, w.lineno, w.line)
583 sys.stdout.write(s)
584 sys.stdout.write('<<<' + ('-' * 73) + '\n')
585 s += ("-" * 76) + "\n"
586 s += warnings.formatwarning(
587 w.message, w.category, w.filename, w.lineno, w.line
588 )
589 s += "<<<" + ("-" * 73)
590 logger.warning(s)
585 591 if self.warning_is_error:
586 592 raise RuntimeError('Non Expected warning in `{}` line {}'.format(filename, lineno))
587 593
588 594 self.clear_cout()
589 595 return (ret, input_lines, processed_output,
590 596 is_doctest, decorator, image_file, image_directive)
591 597
592 598
593 599 def process_output(self, data, output_prompt, input_lines, output,
594 600 is_doctest, decorator, image_file):
595 601 """
596 602 Process data block for OUTPUT token.
597 603
598 604 """
599 605 # Recall: `data` is the submitted output, and `output` is the processed
600 606 # output from `input_lines`.
601 607
602 608 TAB = ' ' * 4
603 609
604 610 if is_doctest and output is not None:
605 611
606 612 found = output # This is the processed output
607 613 found = found.strip()
608 614 submitted = data.strip()
609 615
610 616 if self.directive is None:
611 617 source = 'Unavailable'
612 618 content = 'Unavailable'
613 619 else:
614 620 source = self.directive.state.document.current_source
615 621 content = self.directive.content
616 622 # Add tabs and join into a single string.
617 623 content = '\n'.join([TAB + line for line in content])
618 624
619 625 # Make sure the output contains the output prompt.
620 626 ind = found.find(output_prompt)
621 627 if ind < 0:
622 628 e = ('output does not contain output prompt\n\n'
623 629 'Document source: {0}\n\n'
624 630 'Raw content: \n{1}\n\n'
625 631 'Input line(s):\n{TAB}{2}\n\n'
626 632 'Output line(s):\n{TAB}{3}\n\n')
627 633 e = e.format(source, content, '\n'.join(input_lines),
628 634 repr(found), TAB=TAB)
629 635 raise RuntimeError(e)
630 636 found = found[len(output_prompt):].strip()
631 637
632 638 # Handle the actual doctest comparison.
633 639 if decorator.strip() == '@doctest':
634 640 # Standard doctest
635 641 if found != submitted:
636 642 e = ('doctest failure\n\n'
637 643 'Document source: {0}\n\n'
638 644 'Raw content: \n{1}\n\n'
639 645 'On input line(s):\n{TAB}{2}\n\n'
640 646 'we found output:\n{TAB}{3}\n\n'
641 647 'instead of the expected:\n{TAB}{4}\n\n')
642 648 e = e.format(source, content, '\n'.join(input_lines),
643 649 repr(found), repr(submitted), TAB=TAB)
644 650 raise RuntimeError(e)
645 651 else:
646 652 self.custom_doctest(decorator, input_lines, found, submitted)
647 653
648 654 # When in verbatim mode, this holds additional submitted output
649 655 # to be written in the final Sphinx output.
650 656 # https://github.com/ipython/ipython/issues/5776
651 657 out_data = []
652 658
653 659 is_verbatim = decorator=='@verbatim' or self.is_verbatim
654 660 if is_verbatim and data.strip():
655 661 # Note that `ret` in `process_block` has '' as its last element if
656 662 # the code block was in verbatim mode. So if there is no submitted
657 663 # output, then we will have proper spacing only if we do not add
658 664 # an additional '' to `out_data`. This is why we condition on
659 665 # `and data.strip()`.
660 666
661 667 # The submitted output has no output prompt. If we want the
662 668 # prompt and the code to appear, we need to join them now
663 669 # instead of adding them separately---as this would create an
664 670 # undesired newline. How we do this ultimately depends on the
665 671 # format of the output regex. I'll do what works for the default
666 672 # prompt for now, and we might have to adjust if it doesn't work
667 673 # in other cases. Finally, the submitted output does not have
668 674 # a trailing newline, so we must add it manually.
669 675 out_data.append("{0} {1}\n".format(output_prompt, data))
670 676
671 677 return out_data
672 678
673 679 def process_comment(self, data):
674 680 """Process data fPblock for COMMENT token."""
675 681 if not self.is_suppress:
676 682 return [data]
677 683
678 684 def save_image(self, image_file):
679 685 """
680 686 Saves the image file to disk.
681 687 """
682 688 self.ensure_pyplot()
683 689 command = 'plt.gcf().savefig("%s")'%image_file
684 690 #print 'SAVEFIG', command # dbg
685 691 self.process_input_line('bookmark ipy_thisdir', store_history=False)
686 692 self.process_input_line('cd -b ipy_savedir', store_history=False)
687 693 self.process_input_line(command, store_history=False)
688 694 self.process_input_line('cd -b ipy_thisdir', store_history=False)
689 695 self.process_input_line('bookmark -d ipy_thisdir', store_history=False)
690 696 self.clear_cout()
691 697
692 698 def process_block(self, block):
693 699 """
694 700 process block from the block_parser and return a list of processed lines
695 701 """
696 702 ret = []
697 703 output = None
698 704 input_lines = None
699 705 lineno = self.IP.execution_count
700 706
701 707 input_prompt = self.promptin % lineno
702 708 output_prompt = self.promptout % lineno
703 709 image_file = None
704 710 image_directive = None
705 711
706 712 found_input = False
707 713 for token, data in block:
708 714 if token == COMMENT:
709 715 out_data = self.process_comment(data)
710 716 elif token == INPUT:
711 717 found_input = True
712 718 (out_data, input_lines, output, is_doctest,
713 719 decorator, image_file, image_directive) = \
714 720 self.process_input(data, input_prompt, lineno)
715 721 elif token == OUTPUT:
716 722 if not found_input:
717 723
718 724 TAB = ' ' * 4
719 725 linenumber = 0
720 726 source = 'Unavailable'
721 727 content = 'Unavailable'
722 728 if self.directive:
723 729 linenumber = self.directive.state.document.current_line
724 730 source = self.directive.state.document.current_source
725 731 content = self.directive.content
726 732 # Add tabs and join into a single string.
727 733 content = '\n'.join([TAB + line for line in content])
728 734
729 735 e = ('\n\nInvalid block: Block contains an output prompt '
730 736 'without an input prompt.\n\n'
731 737 'Document source: {0}\n\n'
732 738 'Content begins at line {1}: \n\n{2}\n\n'
733 739 'Problematic block within content: \n\n{TAB}{3}\n\n')
734 740 e = e.format(source, linenumber, content, block, TAB=TAB)
735 741
736 742 # Write, rather than include in exception, since Sphinx
737 743 # will truncate tracebacks.
738 744 sys.stdout.write(e)
739 745 raise RuntimeError('An invalid block was detected.')
740 746 out_data = \
741 747 self.process_output(data, output_prompt, input_lines,
742 748 output, is_doctest, decorator,
743 749 image_file)
744 750 if out_data:
745 751 # Then there was user submitted output in verbatim mode.
746 752 # We need to remove the last element of `ret` that was
747 753 # added in `process_input`, as it is '' and would introduce
748 754 # an undesirable newline.
749 755 assert(ret[-1] == '')
750 756 del ret[-1]
751 757
752 758 if out_data:
753 759 ret.extend(out_data)
754 760
755 761 # save the image files
756 762 if image_file is not None:
757 763 self.save_image(image_file)
758 764
759 765 return ret, image_directive
760 766
761 767 def ensure_pyplot(self):
762 768 """
763 769 Ensures that pyplot has been imported into the embedded IPython shell.
764 770
765 771 Also, makes sure to set the backend appropriately if not set already.
766 772
767 773 """
768 774 # We are here if the @figure pseudo decorator was used. Thus, it's
769 775 # possible that we could be here even if python_mplbackend were set to
770 776 # `None`. That's also strange and perhaps worthy of raising an
771 777 # exception, but for now, we just set the backend to 'agg'.
772 778
773 779 if not self._pyplot_imported:
774 780 if 'matplotlib.backends' not in sys.modules:
775 781 # Then ipython_matplotlib was set to None but there was a
776 782 # call to the @figure decorator (and ipython_execlines did
777 783 # not set a backend).
778 784 #raise Exception("No backend was set, but @figure was used!")
779 785 import matplotlib
780 786 matplotlib.use('agg')
781 787
782 788 # Always import pyplot into embedded shell.
783 789 self.process_input_line('import matplotlib.pyplot as plt',
784 790 store_history=False)
785 791 self._pyplot_imported = True
786 792
787 793 def process_pure_python(self, content):
788 794 """
789 795 content is a list of strings. it is unedited directive content
790 796
791 797 This runs it line by line in the InteractiveShell, prepends
792 798 prompts as needed capturing stderr and stdout, then returns
793 799 the content as a list as if it were ipython code
794 800 """
795 801 output = []
796 802 savefig = False # keep up with this to clear figure
797 803 multiline = False # to handle line continuation
798 804 multiline_start = None
799 805 fmtin = self.promptin
800 806
801 807 ct = 0
802 808
803 809 for lineno, line in enumerate(content):
804 810
805 811 line_stripped = line.strip()
806 812 if not len(line):
807 813 output.append(line)
808 814 continue
809 815
810 816 # handle decorators
811 817 if line_stripped.startswith('@'):
812 818 output.extend([line])
813 819 if 'savefig' in line:
814 820 savefig = True # and need to clear figure
815 821 continue
816 822
817 823 # handle comments
818 824 if line_stripped.startswith('#'):
819 825 output.extend([line])
820 826 continue
821 827
822 828 # deal with lines checking for multiline
823 829 continuation = u' %s:'% ''.join(['.']*(len(str(ct))+2))
824 830 if not multiline:
825 831 modified = u"%s %s" % (fmtin % ct, line_stripped)
826 832 output.append(modified)
827 833 ct += 1
828 834 try:
829 835 ast.parse(line_stripped)
830 836 output.append(u'')
831 837 except Exception: # on a multiline
832 838 multiline = True
833 839 multiline_start = lineno
834 840 else: # still on a multiline
835 841 modified = u'%s %s' % (continuation, line)
836 842 output.append(modified)
837 843
838 844 # if the next line is indented, it should be part of multiline
839 845 if len(content) > lineno + 1:
840 846 nextline = content[lineno + 1]
841 847 if len(nextline) - len(nextline.lstrip()) > 3:
842 848 continue
843 849 try:
844 850 mod = ast.parse(
845 851 '\n'.join(content[multiline_start:lineno+1]))
846 852 if isinstance(mod.body[0], ast.FunctionDef):
847 853 # check to see if we have the whole function
848 854 for element in mod.body[0].body:
849 855 if isinstance(element, ast.Return):
850 856 multiline = False
851 857 else:
852 858 output.append(u'')
853 859 multiline = False
854 860 except Exception:
855 861 pass
856 862
857 863 if savefig: # clear figure if plotted
858 864 self.ensure_pyplot()
859 865 self.process_input_line('plt.clf()', store_history=False)
860 866 self.clear_cout()
861 867 savefig = False
862 868
863 869 return output
864 870
865 871 def custom_doctest(self, decorator, input_lines, found, submitted):
866 872 """
867 873 Perform a specialized doctest.
868 874
869 875 """
870 876 from .custom_doctests import doctests
871 877
872 878 args = decorator.split()
873 879 doctest_type = args[1]
874 880 if doctest_type in doctests:
875 881 doctests[doctest_type](self, args, input_lines, found, submitted)
876 882 else:
877 883 e = "Invalid option to @doctest: {0}".format(doctest_type)
878 884 raise Exception(e)
879 885
880 886
881 887 class IPythonDirective(Directive):
882 888
883 889 has_content = True
884 890 required_arguments = 0
885 891 optional_arguments = 4 # python, suppress, verbatim, doctest
886 892 final_argumuent_whitespace = True
887 893 option_spec = { 'python': directives.unchanged,
888 894 'suppress' : directives.flag,
889 895 'verbatim' : directives.flag,
890 896 'doctest' : directives.flag,
891 897 'okexcept': directives.flag,
892 898 'okwarning': directives.flag
893 899 }
894 900
895 901 shell = None
896 902
897 903 seen_docs = set()
898 904
899 905 def get_config_options(self):
900 906 # contains sphinx configuration variables
901 907 config = self.state.document.settings.env.config
902 908
903 909 # get config variables to set figure output directory
904 910 savefig_dir = config.ipython_savefig_dir
905 911 source_dir = self.state.document.settings.env.srcdir
906 912 savefig_dir = os.path.join(source_dir, savefig_dir)
907 913
908 914 # get regex and prompt stuff
909 915 rgxin = config.ipython_rgxin
910 916 rgxout = config.ipython_rgxout
911 917 warning_is_error= config.ipython_warning_is_error
912 918 promptin = config.ipython_promptin
913 919 promptout = config.ipython_promptout
914 920 mplbackend = config.ipython_mplbackend
915 921 exec_lines = config.ipython_execlines
916 922 hold_count = config.ipython_holdcount
917 923
918 924 return (savefig_dir, source_dir, rgxin, rgxout,
919 925 promptin, promptout, mplbackend, exec_lines, hold_count, warning_is_error)
920 926
921 927 def setup(self):
922 928 # Get configuration values.
923 929 (savefig_dir, source_dir, rgxin, rgxout, promptin, promptout,
924 930 mplbackend, exec_lines, hold_count, warning_is_error) = self.get_config_options()
925 931
926 932 try:
927 933 os.makedirs(savefig_dir)
928 934 except OSError as e:
929 935 if e.errno != errno.EEXIST:
930 936 raise
931 937
932 938 if self.shell is None:
933 939 # We will be here many times. However, when the
934 940 # EmbeddedSphinxShell is created, its interactive shell member
935 941 # is the same for each instance.
936 942
937 943 if mplbackend and 'matplotlib.backends' not in sys.modules and use_matplotlib:
938 944 import matplotlib
939 945 matplotlib.use(mplbackend)
940 946
941 947 # Must be called after (potentially) importing matplotlib and
942 948 # setting its backend since exec_lines might import pylab.
943 949 self.shell = EmbeddedSphinxShell(exec_lines)
944 950
945 951 # Store IPython directive to enable better error messages
946 952 self.shell.directive = self
947 953
948 954 # reset the execution count if we haven't processed this doc
949 955 #NOTE: this may be borked if there are multiple seen_doc tmp files
950 956 #check time stamp?
951 957 if not self.state.document.current_source in self.seen_docs:
952 958 self.shell.IP.history_manager.reset()
953 959 self.shell.IP.execution_count = 1
954 960 self.seen_docs.add(self.state.document.current_source)
955 961
956 962 # and attach to shell so we don't have to pass them around
957 963 self.shell.rgxin = rgxin
958 964 self.shell.rgxout = rgxout
959 965 self.shell.promptin = promptin
960 966 self.shell.promptout = promptout
961 967 self.shell.savefig_dir = savefig_dir
962 968 self.shell.source_dir = source_dir
963 969 self.shell.hold_count = hold_count
964 970 self.shell.warning_is_error = warning_is_error
965 971
966 972 # setup bookmark for saving figures directory
967 973 self.shell.process_input_line('bookmark ipy_savedir %s'%savefig_dir,
968 974 store_history=False)
969 975 self.shell.clear_cout()
970 976
971 977 return rgxin, rgxout, promptin, promptout
972 978
973 979 def teardown(self):
974 980 # delete last bookmark
975 981 self.shell.process_input_line('bookmark -d ipy_savedir',
976 982 store_history=False)
977 983 self.shell.clear_cout()
978 984
979 985 def run(self):
980 986 debug = False
981 987
982 988 #TODO, any reason block_parser can't be a method of embeddable shell
983 989 # then we wouldn't have to carry these around
984 990 rgxin, rgxout, promptin, promptout = self.setup()
985 991
986 992 options = self.options
987 993 self.shell.is_suppress = 'suppress' in options
988 994 self.shell.is_doctest = 'doctest' in options
989 995 self.shell.is_verbatim = 'verbatim' in options
990 996 self.shell.is_okexcept = 'okexcept' in options
991 997 self.shell.is_okwarning = 'okwarning' in options
992 998
993 999 # handle pure python code
994 1000 if 'python' in self.arguments:
995 1001 content = self.content
996 1002 self.content = self.shell.process_pure_python(content)
997 1003
998 1004 # parts consists of all text within the ipython-block.
999 1005 # Each part is an input/output block.
1000 1006 parts = '\n'.join(self.content).split('\n\n')
1001 1007
1002 1008 lines = ['.. code-block:: ipython', '']
1003 1009 figures = []
1004 1010
1011 # Use sphinx logger for warnings
1012 logger = logging.getLogger(__name__)
1013
1005 1014 for part in parts:
1006 1015 block = block_parser(part, rgxin, rgxout, promptin, promptout)
1007 1016 if len(block):
1008 1017 rows, figure = self.shell.process_block(block)
1009 1018 for row in rows:
1010 1019 lines.extend([' {0}'.format(line)
1011 1020 for line in row.split('\n')])
1012 1021
1013 1022 if figure is not None:
1014 1023 figures.append(figure)
1015 1024 else:
1016 1025 message = 'Code input with no code at {}, line {}'\
1017 1026 .format(
1018 1027 self.state.document.current_source,
1019 1028 self.state.document.current_line)
1020 1029 if self.shell.warning_is_error:
1021 1030 raise RuntimeError(message)
1022 1031 else:
1023 warnings.warn(message)
1032 logger.warning(message)
1024 1033
1025 1034 for figure in figures:
1026 1035 lines.append('')
1027 1036 lines.extend(figure.split('\n'))
1028 1037 lines.append('')
1029 1038
1030 1039 if len(lines) > 2:
1031 1040 if debug:
1032 1041 print('\n'.join(lines))
1033 1042 else:
1034 1043 # This has to do with input, not output. But if we comment
1035 1044 # these lines out, then no IPython code will appear in the
1036 1045 # final output.
1037 1046 self.state_machine.insert_input(
1038 1047 lines, self.state_machine.input_lines.source(0))
1039 1048
1040 1049 # cleanup
1041 1050 self.teardown()
1042 1051
1043 1052 return []
1044 1053
1045 1054 # Enable as a proper Sphinx directive
1046 1055 def setup(app):
1047 1056 setup.app = app
1048 1057
1049 1058 app.add_directive('ipython', IPythonDirective)
1050 1059 app.add_config_value('ipython_savefig_dir', 'savefig', 'env')
1051 1060 app.add_config_value('ipython_warning_is_error', True, 'env')
1052 1061 app.add_config_value('ipython_rgxin',
1053 1062 re.compile(r'In \[(\d+)\]:\s?(.*)\s*'), 'env')
1054 1063 app.add_config_value('ipython_rgxout',
1055 1064 re.compile(r'Out\[(\d+)\]:\s?(.*)\s*'), 'env')
1056 1065 app.add_config_value('ipython_promptin', 'In [%d]:', 'env')
1057 1066 app.add_config_value('ipython_promptout', 'Out[%d]:', 'env')
1058 1067
1059 1068 # We could just let matplotlib pick whatever is specified as the default
1060 1069 # backend in the matplotlibrc file, but this would cause issues if the
1061 1070 # backend didn't work in headless environments. For this reason, 'agg'
1062 1071 # is a good default backend choice.
1063 1072 app.add_config_value('ipython_mplbackend', 'agg', 'env')
1064 1073
1065 1074 # If the user sets this config value to `None`, then EmbeddedSphinxShell's
1066 1075 # __init__ method will treat it as [].
1067 1076 execlines = ['import numpy as np']
1068 1077 if use_matplotlib:
1069 1078 execlines.append('import matplotlib.pyplot as plt')
1070 1079 app.add_config_value('ipython_execlines', execlines, 'env')
1071 1080
1072 1081 app.add_config_value('ipython_holdcount', True, 'env')
1073 1082
1074 1083 metadata = {'parallel_read_safe': True, 'parallel_write_safe': True}
1075 1084 return metadata
1076 1085
1077 1086 # Simple smoke test, needs to be converted to a proper automatic test.
1078 1087 def test():
1079 1088
1080 1089 examples = [
1081 1090 r"""
1082 1091 In [9]: pwd
1083 1092 Out[9]: '/home/jdhunter/py4science/book'
1084 1093
1085 1094 In [10]: cd bookdata/
1086 1095 /home/jdhunter/py4science/book/bookdata
1087 1096
1088 1097 In [2]: from pylab import *
1089 1098
1090 1099 In [2]: ion()
1091 1100
1092 1101 In [3]: im = imread('stinkbug.png')
1093 1102
1094 1103 @savefig mystinkbug.png width=4in
1095 1104 In [4]: imshow(im)
1096 1105 Out[4]: <matplotlib.image.AxesImage object at 0x39ea850>
1097 1106
1098 1107 """,
1099 1108 r"""
1100 1109
1101 1110 In [1]: x = 'hello world'
1102 1111
1103 1112 # string methods can be
1104 1113 # used to alter the string
1105 1114 @doctest
1106 1115 In [2]: x.upper()
1107 1116 Out[2]: 'HELLO WORLD'
1108 1117
1109 1118 @verbatim
1110 1119 In [3]: x.st<TAB>
1111 1120 x.startswith x.strip
1112 1121 """,
1113 1122 r"""
1114 1123
1115 1124 In [130]: url = 'http://ichart.finance.yahoo.com/table.csv?s=CROX\
1116 1125 .....: &d=9&e=22&f=2009&g=d&a=1&br=8&c=2006&ignore=.csv'
1117 1126
1118 1127 In [131]: print url.split('&')
1119 1128 ['http://ichart.finance.yahoo.com/table.csv?s=CROX', 'd=9', 'e=22', 'f=2009', 'g=d', 'a=1', 'b=8', 'c=2006', 'ignore=.csv']
1120 1129
1121 1130 In [60]: import urllib
1122 1131
1123 1132 """,
1124 1133 r"""\
1125 1134
1126 1135 In [133]: import numpy.random
1127 1136
1128 1137 @suppress
1129 1138 In [134]: numpy.random.seed(2358)
1130 1139
1131 1140 @doctest
1132 1141 In [135]: numpy.random.rand(10,2)
1133 1142 Out[135]:
1134 1143 array([[ 0.64524308, 0.59943846],
1135 1144 [ 0.47102322, 0.8715456 ],
1136 1145 [ 0.29370834, 0.74776844],
1137 1146 [ 0.99539577, 0.1313423 ],
1138 1147 [ 0.16250302, 0.21103583],
1139 1148 [ 0.81626524, 0.1312433 ],
1140 1149 [ 0.67338089, 0.72302393],
1141 1150 [ 0.7566368 , 0.07033696],
1142 1151 [ 0.22591016, 0.77731835],
1143 1152 [ 0.0072729 , 0.34273127]])
1144 1153
1145 1154 """,
1146 1155
1147 1156 r"""
1148 1157 In [106]: print x
1149 1158 jdh
1150 1159
1151 1160 In [109]: for i in range(10):
1152 1161 .....: print i
1153 1162 .....:
1154 1163 .....:
1155 1164 0
1156 1165 1
1157 1166 2
1158 1167 3
1159 1168 4
1160 1169 5
1161 1170 6
1162 1171 7
1163 1172 8
1164 1173 9
1165 1174 """,
1166 1175
1167 1176 r"""
1168 1177
1169 1178 In [144]: from pylab import *
1170 1179
1171 1180 In [145]: ion()
1172 1181
1173 1182 # use a semicolon to suppress the output
1174 1183 @savefig test_hist.png width=4in
1175 1184 In [151]: hist(np.random.randn(10000), 100);
1176 1185
1177 1186
1178 1187 @savefig test_plot.png width=4in
1179 1188 In [151]: plot(np.random.randn(10000), 'o');
1180 1189 """,
1181 1190
1182 1191 r"""
1183 1192 # use a semicolon to suppress the output
1184 1193 In [151]: plt.clf()
1185 1194
1186 1195 @savefig plot_simple.png width=4in
1187 1196 In [151]: plot([1,2,3])
1188 1197
1189 1198 @savefig hist_simple.png width=4in
1190 1199 In [151]: hist(np.random.randn(10000), 100);
1191 1200
1192 1201 """,
1193 1202 r"""
1194 1203 # update the current fig
1195 1204 In [151]: ylabel('number')
1196 1205
1197 1206 In [152]: title('normal distribution')
1198 1207
1199 1208
1200 1209 @savefig hist_with_text.png
1201 1210 In [153]: grid(True)
1202 1211
1203 1212 @doctest float
1204 1213 In [154]: 0.1 + 0.2
1205 1214 Out[154]: 0.3
1206 1215
1207 1216 @doctest float
1208 1217 In [155]: np.arange(16).reshape(4,4)
1209 1218 Out[155]:
1210 1219 array([[ 0, 1, 2, 3],
1211 1220 [ 4, 5, 6, 7],
1212 1221 [ 8, 9, 10, 11],
1213 1222 [12, 13, 14, 15]])
1214 1223
1215 1224 In [1]: x = np.arange(16, dtype=float).reshape(4,4)
1216 1225
1217 1226 In [2]: x[0,0] = np.inf
1218 1227
1219 1228 In [3]: x[0,1] = np.nan
1220 1229
1221 1230 @doctest float
1222 1231 In [4]: x
1223 1232 Out[4]:
1224 1233 array([[ inf, nan, 2., 3.],
1225 1234 [ 4., 5., 6., 7.],
1226 1235 [ 8., 9., 10., 11.],
1227 1236 [ 12., 13., 14., 15.]])
1228 1237
1229 1238
1230 1239 """,
1231 1240 ]
1232 1241 # skip local-file depending first example:
1233 1242 examples = examples[1:]
1234 1243
1235 1244 #ipython_directive.DEBUG = True # dbg
1236 1245 #options = dict(suppress=True) # dbg
1237 1246 options = {}
1238 1247 for example in examples:
1239 1248 content = example.split('\n')
1240 1249 IPythonDirective('debug', arguments=None, options=options,
1241 1250 content=content, lineno=0,
1242 1251 content_offset=None, block_text=None,
1243 1252 state=None, state_machine=None,
1244 1253 )
1245 1254
1246 1255 # Run test suite as a script
1247 1256 if __name__=='__main__':
1248 1257 if not os.path.isdir('_static'):
1249 1258 os.mkdir('_static')
1250 1259 test()
1251 1260 print('All OK? Check figures in _static/')
General Comments 0
You need to be logged in to leave comments. Login now