##// END OF EJS Templates
Merge pull request #13388 from Carreau/reformat-docstring...
Matthias Bussonnier -
r27298:9065df74 merge
parent child Browse files
Show More

The requested changes are too big and content was truncated. Show full diff

@@ -1,2253 +1,2261 b''
1 1 """Completion for IPython.
2 2
3 3 This module started as fork of the rlcompleter module in the Python standard
4 4 library. The original enhancements made to rlcompleter have been sent
5 5 upstream and were accepted as of Python 2.3,
6 6
7 7 This module now support a wide variety of completion mechanism both available
8 8 for normal classic Python code, as well as completer for IPython specific
9 9 Syntax like magics.
10 10
11 11 Latex and Unicode completion
12 12 ============================
13 13
14 14 IPython and compatible frontends not only can complete your code, but can help
15 15 you to input a wide range of characters. In particular we allow you to insert
16 16 a unicode character using the tab completion mechanism.
17 17
18 18 Forward latex/unicode completion
19 19 --------------------------------
20 20
21 21 Forward completion allows you to easily type a unicode character using its latex
22 22 name, or unicode long description. To do so type a backslash follow by the
23 23 relevant name and press tab:
24 24
25 25
26 26 Using latex completion:
27 27
28 28 .. code::
29 29
30 30 \\alpha<tab>
31 31 α
32 32
33 33 or using unicode completion:
34 34
35 35
36 36 .. code::
37 37
38 38 \\GREEK SMALL LETTER ALPHA<tab>
39 39 α
40 40
41 41
42 42 Only valid Python identifiers will complete. Combining characters (like arrow or
43 43 dots) are also available, unlike latex they need to be put after the their
44 44 counterpart that is to say, `F\\\\vec<tab>` is correct, not `\\\\vec<tab>F`.
45 45
46 46 Some browsers are known to display combining characters incorrectly.
47 47
48 48 Backward latex completion
49 49 -------------------------
50 50
51 51 It is sometime challenging to know how to type a character, if you are using
52 52 IPython, or any compatible frontend you can prepend backslash to the character
53 53 and press `<tab>` to expand it to its latex form.
54 54
55 55 .. code::
56 56
57 57 \\α<tab>
58 58 \\alpha
59 59
60 60
61 61 Both forward and backward completions can be deactivated by setting the
62 62 ``Completer.backslash_combining_completions`` option to ``False``.
63 63
64 64
65 65 Experimental
66 66 ============
67 67
68 68 Starting with IPython 6.0, this module can make use of the Jedi library to
69 69 generate completions both using static analysis of the code, and dynamically
70 70 inspecting multiple namespaces. Jedi is an autocompletion and static analysis
71 71 for Python. The APIs attached to this new mechanism is unstable and will
72 72 raise unless use in an :any:`provisionalcompleter` context manager.
73 73
74 74 You will find that the following are experimental:
75 75
76 76 - :any:`provisionalcompleter`
77 77 - :any:`IPCompleter.completions`
78 78 - :any:`Completion`
79 79 - :any:`rectify_completions`
80 80
81 81 .. note::
82 82
83 83 better name for :any:`rectify_completions` ?
84 84
85 85 We welcome any feedback on these new API, and we also encourage you to try this
86 86 module in debug mode (start IPython with ``--Completer.debug=True``) in order
87 87 to have extra logging information if :any:`jedi` is crashing, or if current
88 88 IPython completer pending deprecations are returning results not yet handled
89 89 by :any:`jedi`
90 90
91 91 Using Jedi for tab completion allow snippets like the following to work without
92 92 having to execute any code:
93 93
94 94 >>> myvar = ['hello', 42]
95 95 ... myvar[1].bi<tab>
96 96
97 97 Tab completion will be able to infer that ``myvar[1]`` is a real number without
98 98 executing any code unlike the previously available ``IPCompleter.greedy``
99 99 option.
100 100
101 101 Be sure to update :any:`jedi` to the latest stable version or to try the
102 102 current development version to get better completions.
103 103 """
104 104
105 105
106 106 # Copyright (c) IPython Development Team.
107 107 # Distributed under the terms of the Modified BSD License.
108 108 #
109 109 # Some of this code originated from rlcompleter in the Python standard library
110 110 # Copyright (C) 2001 Python Software Foundation, www.python.org
111 111
112 112
113 113 import builtins as builtin_mod
114 114 import glob
115 115 import inspect
116 116 import itertools
117 117 import keyword
118 118 import os
119 119 import re
120 120 import string
121 121 import sys
122 122 import time
123 123 import unicodedata
124 124 import uuid
125 125 import warnings
126 126 from contextlib import contextmanager
127 127 from importlib import import_module
128 128 from types import SimpleNamespace
129 129 from typing import Iterable, Iterator, List, Tuple, Union, Any, Sequence, Dict, NamedTuple, Pattern, Optional
130 130
131 131 from IPython.core.error import TryNext
132 132 from IPython.core.inputtransformer2 import ESC_MAGIC
133 133 from IPython.core.latex_symbols import latex_symbols, reverse_latex_symbol
134 134 from IPython.core.oinspect import InspectColors
135 135 from IPython.testing.skipdoctest import skip_doctest
136 136 from IPython.utils import generics
137 137 from IPython.utils.dir2 import dir2, get_real_method
138 138 from IPython.utils.path import ensure_dir_exists
139 139 from IPython.utils.process import arg_split
140 140 from traitlets import Bool, Enum, Int, List as ListTrait, Unicode, default, observe
141 141 from traitlets.config.configurable import Configurable
142 142
143 143 import __main__
144 144
145 145 # skip module docstests
146 146 __skip_doctest__ = True
147 147
148 148 try:
149 149 import jedi
150 150 jedi.settings.case_insensitive_completion = False
151 151 import jedi.api.helpers
152 152 import jedi.api.classes
153 153 JEDI_INSTALLED = True
154 154 except ImportError:
155 155 JEDI_INSTALLED = False
156 156 #-----------------------------------------------------------------------------
157 157 # Globals
158 158 #-----------------------------------------------------------------------------
159 159
160 160 # ranges where we have most of the valid unicode names. We could be more finer
161 161 # grained but is it worth it for performance While unicode have character in the
162 162 # range 0, 0x110000, we seem to have name for about 10% of those. (131808 as I
163 163 # write this). With below range we cover them all, with a density of ~67%
164 164 # biggest next gap we consider only adds up about 1% density and there are 600
165 165 # gaps that would need hard coding.
166 166 _UNICODE_RANGES = [(32, 0x3134b), (0xe0001, 0xe01f0)]
167 167
168 168 # Public API
169 169 __all__ = ['Completer','IPCompleter']
170 170
171 171 if sys.platform == 'win32':
172 172 PROTECTABLES = ' '
173 173 else:
174 174 PROTECTABLES = ' ()[]{}?=\\|;:\'#*"^&'
175 175
176 176 # Protect against returning an enormous number of completions which the frontend
177 177 # may have trouble processing.
178 178 MATCHES_LIMIT = 500
179 179
180 180
181 181 class ProvisionalCompleterWarning(FutureWarning):
182 182 """
183 183 Exception raise by an experimental feature in this module.
184 184
185 185 Wrap code in :any:`provisionalcompleter` context manager if you
186 186 are certain you want to use an unstable feature.
187 187 """
188 188 pass
189 189
190 190 warnings.filterwarnings('error', category=ProvisionalCompleterWarning)
191 191
192 192
193 193 @skip_doctest
194 194 @contextmanager
195 195 def provisionalcompleter(action='ignore'):
196 196 """
197 197 This context manager has to be used in any place where unstable completer
198 198 behavior and API may be called.
199 199
200 200 >>> with provisionalcompleter():
201 201 ... completer.do_experimental_things() # works
202 202
203 203 >>> completer.do_experimental_things() # raises.
204 204
205 205 .. note::
206 206
207 207 Unstable
208 208
209 209 By using this context manager you agree that the API in use may change
210 210 without warning, and that you won't complain if they do so.
211 211
212 212 You also understand that, if the API is not to your liking, you should report
213 213 a bug to explain your use case upstream.
214 214
215 215 We'll be happy to get your feedback, feature requests, and improvements on
216 216 any of the unstable APIs!
217 217 """
218 218 with warnings.catch_warnings():
219 219 warnings.filterwarnings(action, category=ProvisionalCompleterWarning)
220 220 yield
221 221
222 222
223 223 def has_open_quotes(s):
224 224 """Return whether a string has open quotes.
225 225
226 226 This simply counts whether the number of quote characters of either type in
227 227 the string is odd.
228 228
229 229 Returns
230 230 -------
231 231 If there is an open quote, the quote character is returned. Else, return
232 232 False.
233 233 """
234 234 # We check " first, then ', so complex cases with nested quotes will get
235 235 # the " to take precedence.
236 236 if s.count('"') % 2:
237 237 return '"'
238 238 elif s.count("'") % 2:
239 239 return "'"
240 240 else:
241 241 return False
242 242
243 243
244 244 def protect_filename(s, protectables=PROTECTABLES):
245 245 """Escape a string to protect certain characters."""
246 246 if set(s) & set(protectables):
247 247 if sys.platform == "win32":
248 248 return '"' + s + '"'
249 249 else:
250 250 return "".join(("\\" + c if c in protectables else c) for c in s)
251 251 else:
252 252 return s
253 253
254 254
255 255 def expand_user(path:str) -> Tuple[str, bool, str]:
256 256 """Expand ``~``-style usernames in strings.
257 257
258 258 This is similar to :func:`os.path.expanduser`, but it computes and returns
259 259 extra information that will be useful if the input was being used in
260 260 computing completions, and you wish to return the completions with the
261 261 original '~' instead of its expanded value.
262 262
263 263 Parameters
264 264 ----------
265 265 path : str
266 266 String to be expanded. If no ~ is present, the output is the same as the
267 267 input.
268 268
269 269 Returns
270 270 -------
271 271 newpath : str
272 272 Result of ~ expansion in the input path.
273 273 tilde_expand : bool
274 274 Whether any expansion was performed or not.
275 275 tilde_val : str
276 276 The value that ~ was replaced with.
277 277 """
278 278 # Default values
279 279 tilde_expand = False
280 280 tilde_val = ''
281 281 newpath = path
282 282
283 283 if path.startswith('~'):
284 284 tilde_expand = True
285 285 rest = len(path)-1
286 286 newpath = os.path.expanduser(path)
287 287 if rest:
288 288 tilde_val = newpath[:-rest]
289 289 else:
290 290 tilde_val = newpath
291 291
292 292 return newpath, tilde_expand, tilde_val
293 293
294 294
295 295 def compress_user(path:str, tilde_expand:bool, tilde_val:str) -> str:
296 296 """Does the opposite of expand_user, with its outputs.
297 297 """
298 298 if tilde_expand:
299 299 return path.replace(tilde_val, '~')
300 300 else:
301 301 return path
302 302
303 303
304 304 def completions_sorting_key(word):
305 305 """key for sorting completions
306 306
307 307 This does several things:
308 308
309 309 - Demote any completions starting with underscores to the end
310 310 - Insert any %magic and %%cellmagic completions in the alphabetical order
311 311 by their name
312 312 """
313 313 prio1, prio2 = 0, 0
314 314
315 315 if word.startswith('__'):
316 316 prio1 = 2
317 317 elif word.startswith('_'):
318 318 prio1 = 1
319 319
320 320 if word.endswith('='):
321 321 prio1 = -1
322 322
323 323 if word.startswith('%%'):
324 324 # If there's another % in there, this is something else, so leave it alone
325 325 if not "%" in word[2:]:
326 326 word = word[2:]
327 327 prio2 = 2
328 328 elif word.startswith('%'):
329 329 if not "%" in word[1:]:
330 330 word = word[1:]
331 331 prio2 = 1
332 332
333 333 return prio1, word, prio2
334 334
335 335
336 336 class _FakeJediCompletion:
337 337 """
338 338 This is a workaround to communicate to the UI that Jedi has crashed and to
339 339 report a bug. Will be used only id :any:`IPCompleter.debug` is set to true.
340 340
341 341 Added in IPython 6.0 so should likely be removed for 7.0
342 342
343 343 """
344 344
345 345 def __init__(self, name):
346 346
347 347 self.name = name
348 348 self.complete = name
349 349 self.type = 'crashed'
350 350 self.name_with_symbols = name
351 351 self.signature = ''
352 352 self._origin = 'fake'
353 353
354 354 def __repr__(self):
355 355 return '<Fake completion object jedi has crashed>'
356 356
357 357
358 358 class Completion:
359 359 """
360 360 Completion object used and return by IPython completers.
361 361
362 362 .. warning::
363 363
364 364 Unstable
365 365
366 366 This function is unstable, API may change without warning.
367 367 It will also raise unless use in proper context manager.
368 368
369 369 This act as a middle ground :any:`Completion` object between the
370 370 :any:`jedi.api.classes.Completion` object and the Prompt Toolkit completion
371 371 object. While Jedi need a lot of information about evaluator and how the
372 372 code should be ran/inspected, PromptToolkit (and other frontend) mostly
373 373 need user facing information.
374 374
375 375 - Which range should be replaced replaced by what.
376 376 - Some metadata (like completion type), or meta information to displayed to
377 377 the use user.
378 378
379 379 For debugging purpose we can also store the origin of the completion (``jedi``,
380 380 ``IPython.python_matches``, ``IPython.magics_matches``...).
381 381 """
382 382
383 383 __slots__ = ['start', 'end', 'text', 'type', 'signature', '_origin']
384 384
385 385 def __init__(self, start: int, end: int, text: str, *, type: str=None, _origin='', signature='') -> None:
386 386 warnings.warn("``Completion`` is a provisional API (as of IPython 6.0). "
387 387 "It may change without warnings. "
388 388 "Use in corresponding context manager.",
389 389 category=ProvisionalCompleterWarning, stacklevel=2)
390 390
391 391 self.start = start
392 392 self.end = end
393 393 self.text = text
394 394 self.type = type
395 395 self.signature = signature
396 396 self._origin = _origin
397 397
398 398 def __repr__(self):
399 399 return '<Completion start=%s end=%s text=%r type=%r, signature=%r,>' % \
400 400 (self.start, self.end, self.text, self.type or '?', self.signature or '?')
401 401
402 402 def __eq__(self, other)->Bool:
403 403 """
404 404 Equality and hash do not hash the type (as some completer may not be
405 405 able to infer the type), but are use to (partially) de-duplicate
406 406 completion.
407 407
408 408 Completely de-duplicating completion is a bit tricker that just
409 409 comparing as it depends on surrounding text, which Completions are not
410 410 aware of.
411 411 """
412 412 return self.start == other.start and \
413 413 self.end == other.end and \
414 414 self.text == other.text
415 415
416 416 def __hash__(self):
417 417 return hash((self.start, self.end, self.text))
418 418
419 419
420 420 _IC = Iterable[Completion]
421 421
422 422
423 423 def _deduplicate_completions(text: str, completions: _IC)-> _IC:
424 424 """
425 425 Deduplicate a set of completions.
426 426
427 427 .. warning::
428 428
429 429 Unstable
430 430
431 431 This function is unstable, API may change without warning.
432 432
433 433 Parameters
434 434 ----------
435 435 text : str
436 436 text that should be completed.
437 437 completions : Iterator[Completion]
438 438 iterator over the completions to deduplicate
439 439
440 440 Yields
441 441 ------
442 442 `Completions` objects
443 443 Completions coming from multiple sources, may be different but end up having
444 444 the same effect when applied to ``text``. If this is the case, this will
445 445 consider completions as equal and only emit the first encountered.
446 446 Not folded in `completions()` yet for debugging purpose, and to detect when
447 447 the IPython completer does return things that Jedi does not, but should be
448 448 at some point.
449 449 """
450 450 completions = list(completions)
451 451 if not completions:
452 452 return
453 453
454 454 new_start = min(c.start for c in completions)
455 455 new_end = max(c.end for c in completions)
456 456
457 457 seen = set()
458 458 for c in completions:
459 459 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
460 460 if new_text not in seen:
461 461 yield c
462 462 seen.add(new_text)
463 463
464 464
465 def rectify_completions(text: str, completions: _IC, *, _debug=False)->_IC:
465 def rectify_completions(text: str, completions: _IC, *, _debug: bool = False) -> _IC:
466 466 """
467 467 Rectify a set of completions to all have the same ``start`` and ``end``
468 468
469 469 .. warning::
470 470
471 471 Unstable
472 472
473 473 This function is unstable, API may change without warning.
474 474 It will also raise unless use in proper context manager.
475 475
476 476 Parameters
477 477 ----------
478 478 text : str
479 479 text that should be completed.
480 480 completions : Iterator[Completion]
481 481 iterator over the completions to rectify
482 _debug : bool
483 Log failed completion
482 484
483 485 Notes
484 486 -----
485 487 :any:`jedi.api.classes.Completion` s returned by Jedi may not have the same start and end, though
486 488 the Jupyter Protocol requires them to behave like so. This will readjust
487 489 the completion to have the same ``start`` and ``end`` by padding both
488 490 extremities with surrounding text.
489 491
490 492 During stabilisation should support a ``_debug`` option to log which
491 493 completion are return by the IPython completer and not found in Jedi in
492 494 order to make upstream bug report.
493 495 """
494 496 warnings.warn("`rectify_completions` is a provisional API (as of IPython 6.0). "
495 497 "It may change without warnings. "
496 498 "Use in corresponding context manager.",
497 499 category=ProvisionalCompleterWarning, stacklevel=2)
498 500
499 501 completions = list(completions)
500 502 if not completions:
501 503 return
502 504 starts = (c.start for c in completions)
503 505 ends = (c.end for c in completions)
504 506
505 507 new_start = min(starts)
506 508 new_end = max(ends)
507 509
508 510 seen_jedi = set()
509 511 seen_python_matches = set()
510 512 for c in completions:
511 513 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
512 514 if c._origin == 'jedi':
513 515 seen_jedi.add(new_text)
514 516 elif c._origin == 'IPCompleter.python_matches':
515 517 seen_python_matches.add(new_text)
516 518 yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin, signature=c.signature)
517 519 diff = seen_python_matches.difference(seen_jedi)
518 520 if diff and _debug:
519 521 print('IPython.python matches have extras:', diff)
520 522
521 523
522 524 if sys.platform == 'win32':
523 525 DELIMS = ' \t\n`!@#$^&*()=+[{]}|;\'",<>?'
524 526 else:
525 527 DELIMS = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
526 528
527 529 GREEDY_DELIMS = ' =\r\n'
528 530
529 531
530 532 class CompletionSplitter(object):
531 533 """An object to split an input line in a manner similar to readline.
532 534
533 535 By having our own implementation, we can expose readline-like completion in
534 536 a uniform manner to all frontends. This object only needs to be given the
535 537 line of text to be split and the cursor position on said line, and it
536 538 returns the 'word' to be completed on at the cursor after splitting the
537 539 entire line.
538 540
539 541 What characters are used as splitting delimiters can be controlled by
540 542 setting the ``delims`` attribute (this is a property that internally
541 543 automatically builds the necessary regular expression)"""
542 544
543 545 # Private interface
544 546
545 547 # A string of delimiter characters. The default value makes sense for
546 548 # IPython's most typical usage patterns.
547 549 _delims = DELIMS
548 550
549 551 # The expression (a normal string) to be compiled into a regular expression
550 552 # for actual splitting. We store it as an attribute mostly for ease of
551 553 # debugging, since this type of code can be so tricky to debug.
552 554 _delim_expr = None
553 555
554 556 # The regular expression that does the actual splitting
555 557 _delim_re = None
556 558
557 559 def __init__(self, delims=None):
558 560 delims = CompletionSplitter._delims if delims is None else delims
559 561 self.delims = delims
560 562
561 563 @property
562 564 def delims(self):
563 565 """Return the string of delimiter characters."""
564 566 return self._delims
565 567
566 568 @delims.setter
567 569 def delims(self, delims):
568 570 """Set the delimiters for line splitting."""
569 571 expr = '[' + ''.join('\\'+ c for c in delims) + ']'
570 572 self._delim_re = re.compile(expr)
571 573 self._delims = delims
572 574 self._delim_expr = expr
573 575
574 576 def split_line(self, line, cursor_pos=None):
575 577 """Split a line of text with a cursor at the given position.
576 578 """
577 579 l = line if cursor_pos is None else line[:cursor_pos]
578 580 return self._delim_re.split(l)[-1]
579 581
580 582
581 583
582 584 class Completer(Configurable):
583 585
584 586 greedy = Bool(False,
585 587 help="""Activate greedy completion
586 588 PENDING DEPRECATION. this is now mostly taken care of with Jedi.
587 589
588 590 This will enable completion on elements of lists, results of function calls, etc.,
589 591 but can be unsafe because the code is actually evaluated on TAB.
590 592 """
591 593 ).tag(config=True)
592 594
593 595 use_jedi = Bool(default_value=JEDI_INSTALLED,
594 596 help="Experimental: Use Jedi to generate autocompletions. "
595 597 "Default to True if jedi is installed.").tag(config=True)
596 598
597 599 jedi_compute_type_timeout = Int(default_value=400,
598 600 help="""Experimental: restrict time (in milliseconds) during which Jedi can compute types.
599 601 Set to 0 to stop computing types. Non-zero value lower than 100ms may hurt
600 602 performance by preventing jedi to build its cache.
601 603 """).tag(config=True)
602 604
603 605 debug = Bool(default_value=False,
604 606 help='Enable debug for the Completer. Mostly print extra '
605 607 'information for experimental jedi integration.')\
606 608 .tag(config=True)
607 609
608 610 backslash_combining_completions = Bool(True,
609 611 help="Enable unicode completions, e.g. \\alpha<tab> . "
610 612 "Includes completion of latex commands, unicode names, and expanding "
611 613 "unicode characters back to latex commands.").tag(config=True)
612 614
613 615 def __init__(self, namespace=None, global_namespace=None, **kwargs):
614 616 """Create a new completer for the command line.
615 617
616 618 Completer(namespace=ns, global_namespace=ns2) -> completer instance.
617 619
618 620 If unspecified, the default namespace where completions are performed
619 621 is __main__ (technically, __main__.__dict__). Namespaces should be
620 622 given as dictionaries.
621 623
622 624 An optional second namespace can be given. This allows the completer
623 625 to handle cases where both the local and global scopes need to be
624 626 distinguished.
625 627 """
626 628
627 629 # Don't bind to namespace quite yet, but flag whether the user wants a
628 630 # specific namespace or to use __main__.__dict__. This will allow us
629 631 # to bind to __main__.__dict__ at completion time, not now.
630 632 if namespace is None:
631 633 self.use_main_ns = True
632 634 else:
633 635 self.use_main_ns = False
634 636 self.namespace = namespace
635 637
636 638 # The global namespace, if given, can be bound directly
637 639 if global_namespace is None:
638 640 self.global_namespace = {}
639 641 else:
640 642 self.global_namespace = global_namespace
641 643
642 644 self.custom_matchers = []
643 645
644 646 super(Completer, self).__init__(**kwargs)
645 647
646 648 def complete(self, text, state):
647 649 """Return the next possible completion for 'text'.
648 650
649 651 This is called successively with state == 0, 1, 2, ... until it
650 652 returns None. The completion should begin with 'text'.
651 653
652 654 """
653 655 if self.use_main_ns:
654 656 self.namespace = __main__.__dict__
655 657
656 658 if state == 0:
657 659 if "." in text:
658 660 self.matches = self.attr_matches(text)
659 661 else:
660 662 self.matches = self.global_matches(text)
661 663 try:
662 664 return self.matches[state]
663 665 except IndexError:
664 666 return None
665 667
666 668 def global_matches(self, text):
667 669 """Compute matches when text is a simple name.
668 670
669 671 Return a list of all keywords, built-in functions and names currently
670 672 defined in self.namespace or self.global_namespace that match.
671 673
672 674 """
673 675 matches = []
674 676 match_append = matches.append
675 677 n = len(text)
676 678 for lst in [keyword.kwlist,
677 679 builtin_mod.__dict__.keys(),
678 680 self.namespace.keys(),
679 681 self.global_namespace.keys()]:
680 682 for word in lst:
681 683 if word[:n] == text and word != "__builtins__":
682 684 match_append(word)
683 685
684 686 snake_case_re = re.compile(r"[^_]+(_[^_]+)+?\Z")
685 687 for lst in [self.namespace.keys(),
686 688 self.global_namespace.keys()]:
687 689 shortened = {"_".join([sub[0] for sub in word.split('_')]) : word
688 690 for word in lst if snake_case_re.match(word)}
689 691 for word in shortened.keys():
690 692 if word[:n] == text and word != "__builtins__":
691 693 match_append(shortened[word])
692 694 return matches
693 695
694 696 def attr_matches(self, text):
695 697 """Compute matches when text contains a dot.
696 698
697 699 Assuming the text is of the form NAME.NAME....[NAME], and is
698 700 evaluatable in self.namespace or self.global_namespace, it will be
699 701 evaluated and its attributes (as revealed by dir()) are used as
700 702 possible completions. (For class instances, class members are
701 703 also considered.)
702 704
703 705 WARNING: this can still invoke arbitrary C code, if an object
704 706 with a __getattr__ hook is evaluated.
705 707
706 708 """
707 709
708 710 # Another option, seems to work great. Catches things like ''.<tab>
709 711 m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
710 712
711 713 if m:
712 714 expr, attr = m.group(1, 3)
713 715 elif self.greedy:
714 716 m2 = re.match(r"(.+)\.(\w*)$", self.line_buffer)
715 717 if not m2:
716 718 return []
717 719 expr, attr = m2.group(1,2)
718 720 else:
719 721 return []
720 722
721 723 try:
722 724 obj = eval(expr, self.namespace)
723 725 except:
724 726 try:
725 727 obj = eval(expr, self.global_namespace)
726 728 except:
727 729 return []
728 730
729 731 if self.limit_to__all__ and hasattr(obj, '__all__'):
730 732 words = get__all__entries(obj)
731 733 else:
732 734 words = dir2(obj)
733 735
734 736 try:
735 737 words = generics.complete_object(obj, words)
736 738 except TryNext:
737 739 pass
738 740 except AssertionError:
739 741 raise
740 742 except Exception:
741 743 # Silence errors from completion function
742 744 #raise # dbg
743 745 pass
744 746 # Build match list to return
745 747 n = len(attr)
746 748 return [u"%s.%s" % (expr, w) for w in words if w[:n] == attr ]
747 749
748 750
749 751 def get__all__entries(obj):
750 752 """returns the strings in the __all__ attribute"""
751 753 try:
752 754 words = getattr(obj, '__all__')
753 755 except:
754 756 return []
755 757
756 758 return [w for w in words if isinstance(w, str)]
757 759
758 760
759 761 def match_dict_keys(keys: List[Union[str, bytes, Tuple[Union[str, bytes]]]], prefix: str, delims: str,
760 762 extra_prefix: Optional[Tuple[str, bytes]]=None) -> Tuple[str, int, List[str]]:
761 763 """Used by dict_key_matches, matching the prefix to a list of keys
762 764
763 765 Parameters
764 766 ----------
765 767 keys
766 768 list of keys in dictionary currently being completed.
767 769 prefix
768 770 Part of the text already typed by the user. E.g. `mydict[b'fo`
769 771 delims
770 772 String of delimiters to consider when finding the current key.
771 773 extra_prefix : optional
772 774 Part of the text already typed in multi-key index cases. E.g. for
773 775 `mydict['foo', "bar", 'b`, this would be `('foo', 'bar')`.
774 776
775 777 Returns
776 778 -------
777 779 A tuple of three elements: ``quote``, ``token_start``, ``matched``, with
778 780 ``quote`` being the quote that need to be used to close current string.
779 781 ``token_start`` the position where the replacement should start occurring,
780 782 ``matches`` a list of replacement/completion
781 783
782 784 """
783 785 prefix_tuple = extra_prefix if extra_prefix else ()
784 786 Nprefix = len(prefix_tuple)
785 787 def filter_prefix_tuple(key):
786 788 # Reject too short keys
787 789 if len(key) <= Nprefix:
788 790 return False
789 791 # Reject keys with non str/bytes in it
790 792 for k in key:
791 793 if not isinstance(k, (str, bytes)):
792 794 return False
793 795 # Reject keys that do not match the prefix
794 796 for k, pt in zip(key, prefix_tuple):
795 797 if k != pt:
796 798 return False
797 799 # All checks passed!
798 800 return True
799 801
800 802 filtered_keys:List[Union[str,bytes]] = []
801 803 def _add_to_filtered_keys(key):
802 804 if isinstance(key, (str, bytes)):
803 805 filtered_keys.append(key)
804 806
805 807 for k in keys:
806 808 if isinstance(k, tuple):
807 809 if filter_prefix_tuple(k):
808 810 _add_to_filtered_keys(k[Nprefix])
809 811 else:
810 812 _add_to_filtered_keys(k)
811 813
812 814 if not prefix:
813 815 return '', 0, [repr(k) for k in filtered_keys]
814 816 quote_match = re.search('["\']', prefix)
815 817 assert quote_match is not None # silence mypy
816 818 quote = quote_match.group()
817 819 try:
818 820 prefix_str = eval(prefix + quote, {})
819 821 except Exception:
820 822 return '', 0, []
821 823
822 824 pattern = '[^' + ''.join('\\' + c for c in delims) + ']*$'
823 825 token_match = re.search(pattern, prefix, re.UNICODE)
824 826 assert token_match is not None # silence mypy
825 827 token_start = token_match.start()
826 828 token_prefix = token_match.group()
827 829
828 830 matched:List[str] = []
829 831 for key in filtered_keys:
830 832 try:
831 833 if not key.startswith(prefix_str):
832 834 continue
833 835 except (AttributeError, TypeError, UnicodeError):
834 836 # Python 3+ TypeError on b'a'.startswith('a') or vice-versa
835 837 continue
836 838
837 839 # reformat remainder of key to begin with prefix
838 840 rem = key[len(prefix_str):]
839 841 # force repr wrapped in '
840 842 rem_repr = repr(rem + '"') if isinstance(rem, str) else repr(rem + b'"')
841 843 rem_repr = rem_repr[1 + rem_repr.index("'"):-2]
842 844 if quote == '"':
843 845 # The entered prefix is quoted with ",
844 846 # but the match is quoted with '.
845 847 # A contained " hence needs escaping for comparison:
846 848 rem_repr = rem_repr.replace('"', '\\"')
847 849
848 850 # then reinsert prefix from start of token
849 851 matched.append('%s%s' % (token_prefix, rem_repr))
850 852 return quote, token_start, matched
851 853
852 854
853 855 def cursor_to_position(text:str, line:int, column:int)->int:
854 856 """
855 857 Convert the (line,column) position of the cursor in text to an offset in a
856 858 string.
857 859
858 860 Parameters
859 861 ----------
860 862 text : str
861 863 The text in which to calculate the cursor offset
862 864 line : int
863 865 Line of the cursor; 0-indexed
864 866 column : int
865 867 Column of the cursor 0-indexed
866 868
867 869 Returns
868 870 -------
869 871 Position of the cursor in ``text``, 0-indexed.
870 872
871 873 See Also
872 874 --------
873 875 position_to_cursor : reciprocal of this function
874 876
875 877 """
876 878 lines = text.split('\n')
877 879 assert line <= len(lines), '{} <= {}'.format(str(line), str(len(lines)))
878 880
879 881 return sum(len(l) + 1 for l in lines[:line]) + column
880 882
881 883 def position_to_cursor(text:str, offset:int)->Tuple[int, int]:
882 884 """
883 885 Convert the position of the cursor in text (0 indexed) to a line
884 886 number(0-indexed) and a column number (0-indexed) pair
885 887
886 888 Position should be a valid position in ``text``.
887 889
888 890 Parameters
889 891 ----------
890 892 text : str
891 893 The text in which to calculate the cursor offset
892 894 offset : int
893 895 Position of the cursor in ``text``, 0-indexed.
894 896
895 897 Returns
896 898 -------
897 899 (line, column) : (int, int)
898 900 Line of the cursor; 0-indexed, column of the cursor 0-indexed
899 901
900 902 See Also
901 903 --------
902 904 cursor_to_position : reciprocal of this function
903 905
904 906 """
905 907
906 908 assert 0 <= offset <= len(text) , "0 <= %s <= %s" % (offset , len(text))
907 909
908 910 before = text[:offset]
909 911 blines = before.split('\n') # ! splitnes trim trailing \n
910 912 line = before.count('\n')
911 913 col = len(blines[-1])
912 914 return line, col
913 915
914 916
915 917 def _safe_isinstance(obj, module, class_name):
916 918 """Checks if obj is an instance of module.class_name if loaded
917 919 """
918 920 return (module in sys.modules and
919 921 isinstance(obj, getattr(import_module(module), class_name)))
920 922
921 923 def back_unicode_name_matches(text:str) -> Tuple[str, Sequence[str]]:
922 924 """Match Unicode characters back to Unicode name
923 925
924 926 This does ``☃`` -> ``\\snowman``
925 927
926 928 Note that snowman is not a valid python3 combining character but will be expanded.
927 929 Though it will not recombine back to the snowman character by the completion machinery.
928 930
929 931 This will not either back-complete standard sequences like \\n, \\b ...
930 932
931 933 Returns
932 934 =======
933 935
934 936 Return a tuple with two elements:
935 937
936 938 - The Unicode character that was matched (preceded with a backslash), or
937 939 empty string,
938 940 - a sequence (of 1), name for the match Unicode character, preceded by
939 941 backslash, or empty if no match.
940 942
941 943 """
942 944 if len(text)<2:
943 945 return '', ()
944 946 maybe_slash = text[-2]
945 947 if maybe_slash != '\\':
946 948 return '', ()
947 949
948 950 char = text[-1]
949 951 # no expand on quote for completion in strings.
950 952 # nor backcomplete standard ascii keys
951 953 if char in string.ascii_letters or char in ('"',"'"):
952 954 return '', ()
953 955 try :
954 956 unic = unicodedata.name(char)
955 957 return '\\'+char,('\\'+unic,)
956 958 except KeyError:
957 959 pass
958 960 return '', ()
959 961
960 962 def back_latex_name_matches(text:str) -> Tuple[str, Sequence[str]] :
961 963 """Match latex characters back to unicode name
962 964
963 965 This does ``\\ℵ`` -> ``\\aleph``
964 966
965 967 """
966 968 if len(text)<2:
967 969 return '', ()
968 970 maybe_slash = text[-2]
969 971 if maybe_slash != '\\':
970 972 return '', ()
971 973
972 974
973 975 char = text[-1]
974 976 # no expand on quote for completion in strings.
975 977 # nor backcomplete standard ascii keys
976 978 if char in string.ascii_letters or char in ('"',"'"):
977 979 return '', ()
978 980 try :
979 981 latex = reverse_latex_symbol[char]
980 982 # '\\' replace the \ as well
981 983 return '\\'+char,[latex]
982 984 except KeyError:
983 985 pass
984 986 return '', ()
985 987
986 988
987 989 def _formatparamchildren(parameter) -> str:
988 990 """
989 991 Get parameter name and value from Jedi Private API
990 992
991 993 Jedi does not expose a simple way to get `param=value` from its API.
992 994
993 995 Parameters
994 996 ----------
995 997 parameter
996 998 Jedi's function `Param`
997 999
998 1000 Returns
999 1001 -------
1000 1002 A string like 'a', 'b=1', '*args', '**kwargs'
1001 1003
1002 1004 """
1003 1005 description = parameter.description
1004 1006 if not description.startswith('param '):
1005 1007 raise ValueError('Jedi function parameter description have change format.'
1006 1008 'Expected "param ...", found %r".' % description)
1007 1009 return description[6:]
1008 1010
1009 1011 def _make_signature(completion)-> str:
1010 1012 """
1011 1013 Make the signature from a jedi completion
1012 1014
1013 1015 Parameters
1014 1016 ----------
1015 1017 completion : jedi.Completion
1016 1018 object does not complete a function type
1017 1019
1018 1020 Returns
1019 1021 -------
1020 1022 a string consisting of the function signature, with the parenthesis but
1021 1023 without the function name. example:
1022 1024 `(a, *args, b=1, **kwargs)`
1023 1025
1024 1026 """
1025 1027
1026 1028 # it looks like this might work on jedi 0.17
1027 1029 if hasattr(completion, 'get_signatures'):
1028 1030 signatures = completion.get_signatures()
1029 1031 if not signatures:
1030 1032 return '(?)'
1031 1033
1032 1034 c0 = completion.get_signatures()[0]
1033 1035 return '('+c0.to_string().split('(', maxsplit=1)[1]
1034 1036
1035 1037 return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for signature in completion.get_signatures()
1036 1038 for p in signature.defined_names()) if f])
1037 1039
1038 1040
1039 1041 class _CompleteResult(NamedTuple):
1040 1042 matched_text : str
1041 1043 matches: Sequence[str]
1042 1044 matches_origin: Sequence[str]
1043 1045 jedi_matches: Any
1044 1046
1045 1047
1046 1048 class IPCompleter(Completer):
1047 1049 """Extension of the completer class with IPython-specific features"""
1048 1050
1049 1051 __dict_key_regexps: Optional[Dict[bool,Pattern]] = None
1050 1052
1051 1053 @observe('greedy')
1052 1054 def _greedy_changed(self, change):
1053 1055 """update the splitter and readline delims when greedy is changed"""
1054 1056 if change['new']:
1055 1057 self.splitter.delims = GREEDY_DELIMS
1056 1058 else:
1057 1059 self.splitter.delims = DELIMS
1058 1060
1059 1061 dict_keys_only = Bool(False,
1060 1062 help="""Whether to show dict key matches only""")
1061 1063
1062 1064 merge_completions = Bool(True,
1063 1065 help="""Whether to merge completion results into a single list
1064 1066
1065 1067 If False, only the completion results from the first non-empty
1066 1068 completer will be returned.
1067 1069 """
1068 1070 ).tag(config=True)
1069 1071 omit__names = Enum((0,1,2), default_value=2,
1070 1072 help="""Instruct the completer to omit private method names
1071 1073
1072 1074 Specifically, when completing on ``object.<tab>``.
1073 1075
1074 1076 When 2 [default]: all names that start with '_' will be excluded.
1075 1077
1076 1078 When 1: all 'magic' names (``__foo__``) will be excluded.
1077 1079
1078 1080 When 0: nothing will be excluded.
1079 1081 """
1080 1082 ).tag(config=True)
1081 1083 limit_to__all__ = Bool(False,
1082 1084 help="""
1083 1085 DEPRECATED as of version 5.0.
1084 1086
1085 1087 Instruct the completer to use __all__ for the completion
1086 1088
1087 1089 Specifically, when completing on ``object.<tab>``.
1088 1090
1089 1091 When True: only those names in obj.__all__ will be included.
1090 1092
1091 1093 When False [default]: the __all__ attribute is ignored
1092 1094 """,
1093 1095 ).tag(config=True)
1094 1096
1095 1097 profile_completions = Bool(
1096 1098 default_value=False,
1097 1099 help="If True, emit profiling data for completion subsystem using cProfile."
1098 1100 ).tag(config=True)
1099 1101
1100 1102 profiler_output_dir = Unicode(
1101 1103 default_value=".completion_profiles",
1102 1104 help="Template for path at which to output profile data for completions."
1103 1105 ).tag(config=True)
1104 1106
1105 1107 @observe('limit_to__all__')
1106 1108 def _limit_to_all_changed(self, change):
1107 1109 warnings.warn('`IPython.core.IPCompleter.limit_to__all__` configuration '
1108 1110 'value has been deprecated since IPython 5.0, will be made to have '
1109 1111 'no effects and then removed in future version of IPython.',
1110 1112 UserWarning)
1111 1113
1112 1114 def __init__(
1113 1115 self, shell=None, namespace=None, global_namespace=None, config=None, **kwargs
1114 1116 ):
1115 1117 """IPCompleter() -> completer
1116 1118
1117 1119 Return a completer object.
1118 1120
1119 1121 Parameters
1120 1122 ----------
1121 1123 shell
1122 1124 a pointer to the ipython shell itself. This is needed
1123 1125 because this completer knows about magic functions, and those can
1124 1126 only be accessed via the ipython instance.
1125 1127 namespace : dict, optional
1126 1128 an optional dict where completions are performed.
1127 1129 global_namespace : dict, optional
1128 1130 secondary optional dict for completions, to
1129 1131 handle cases (such as IPython embedded inside functions) where
1130 1132 both Python scopes are visible.
1131 use_readline : bool, optional
1132 DEPRECATED, ignored since IPython 6.0, will have no effects
1133 config : Config
1134 traitlet's config object
1135 **kwargs
1136 passed to super class unmodified.
1133 1137 """
1134 1138
1135 1139 self.magic_escape = ESC_MAGIC
1136 1140 self.splitter = CompletionSplitter()
1137 1141
1138 1142 # _greedy_changed() depends on splitter and readline being defined:
1139 Completer.__init__(self, namespace=namespace, global_namespace=global_namespace,
1140 config=config, **kwargs)
1143 super().__init__(
1144 namespace=namespace,
1145 global_namespace=global_namespace,
1146 config=config,
1147 **kwargs
1148 )
1141 1149
1142 1150 # List where completion matches will be stored
1143 1151 self.matches = []
1144 1152 self.shell = shell
1145 1153 # Regexp to split filenames with spaces in them
1146 1154 self.space_name_re = re.compile(r'([^\\] )')
1147 1155 # Hold a local ref. to glob.glob for speed
1148 1156 self.glob = glob.glob
1149 1157
1150 1158 # Determine if we are running on 'dumb' terminals, like (X)Emacs
1151 1159 # buffers, to avoid completion problems.
1152 1160 term = os.environ.get('TERM','xterm')
1153 1161 self.dumb_terminal = term in ['dumb','emacs']
1154 1162
1155 1163 # Special handling of backslashes needed in win32 platforms
1156 1164 if sys.platform == "win32":
1157 1165 self.clean_glob = self._clean_glob_win32
1158 1166 else:
1159 1167 self.clean_glob = self._clean_glob
1160 1168
1161 1169 #regexp to parse docstring for function signature
1162 1170 self.docstring_sig_re = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1163 1171 self.docstring_kwd_re = re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1164 1172 #use this if positional argument name is also needed
1165 1173 #= re.compile(r'[\s|\[]*(\w+)(?:\s*=?\s*.*)')
1166 1174
1167 1175 self.magic_arg_matchers = [
1168 1176 self.magic_config_matches,
1169 1177 self.magic_color_matches,
1170 1178 ]
1171 1179
1172 1180 # This is set externally by InteractiveShell
1173 1181 self.custom_completers = None
1174 1182
1175 1183 # This is a list of names of unicode characters that can be completed
1176 1184 # into their corresponding unicode value. The list is large, so we
1177 1185 # laziliy initialize it on first use. Consuming code should access this
1178 1186 # attribute through the `@unicode_names` property.
1179 1187 self._unicode_names = None
1180 1188
1181 1189 @property
1182 1190 def matchers(self) -> List[Any]:
1183 1191 """All active matcher routines for completion"""
1184 1192 if self.dict_keys_only:
1185 1193 return [self.dict_key_matches]
1186 1194
1187 1195 if self.use_jedi:
1188 1196 return [
1189 1197 *self.custom_matchers,
1190 1198 self.dict_key_matches,
1191 1199 self.file_matches,
1192 1200 self.magic_matches,
1193 1201 ]
1194 1202 else:
1195 1203 return [
1196 1204 *self.custom_matchers,
1197 1205 self.dict_key_matches,
1198 1206 self.python_matches,
1199 1207 self.file_matches,
1200 1208 self.magic_matches,
1201 1209 self.python_func_kw_matches,
1202 1210 ]
1203 1211
1204 1212 def all_completions(self, text:str) -> List[str]:
1205 1213 """
1206 1214 Wrapper around the completion methods for the benefit of emacs.
1207 1215 """
1208 1216 prefix = text.rpartition('.')[0]
1209 1217 with provisionalcompleter():
1210 1218 return ['.'.join([prefix, c.text]) if prefix and self.use_jedi else c.text
1211 1219 for c in self.completions(text, len(text))]
1212 1220
1213 1221 return self.complete(text)[1]
1214 1222
1215 1223 def _clean_glob(self, text:str):
1216 1224 return self.glob("%s*" % text)
1217 1225
1218 1226 def _clean_glob_win32(self, text:str):
1219 1227 return [f.replace("\\","/")
1220 1228 for f in self.glob("%s*" % text)]
1221 1229
1222 1230 def file_matches(self, text:str)->List[str]:
1223 1231 """Match filenames, expanding ~USER type strings.
1224 1232
1225 1233 Most of the seemingly convoluted logic in this completer is an
1226 1234 attempt to handle filenames with spaces in them. And yet it's not
1227 1235 quite perfect, because Python's readline doesn't expose all of the
1228 1236 GNU readline details needed for this to be done correctly.
1229 1237
1230 1238 For a filename with a space in it, the printed completions will be
1231 1239 only the parts after what's already been typed (instead of the
1232 1240 full completions, as is normally done). I don't think with the
1233 1241 current (as of Python 2.3) Python readline it's possible to do
1234 1242 better."""
1235 1243
1236 1244 # chars that require escaping with backslash - i.e. chars
1237 1245 # that readline treats incorrectly as delimiters, but we
1238 1246 # don't want to treat as delimiters in filename matching
1239 1247 # when escaped with backslash
1240 1248 if text.startswith('!'):
1241 1249 text = text[1:]
1242 1250 text_prefix = u'!'
1243 1251 else:
1244 1252 text_prefix = u''
1245 1253
1246 1254 text_until_cursor = self.text_until_cursor
1247 1255 # track strings with open quotes
1248 1256 open_quotes = has_open_quotes(text_until_cursor)
1249 1257
1250 1258 if '(' in text_until_cursor or '[' in text_until_cursor:
1251 1259 lsplit = text
1252 1260 else:
1253 1261 try:
1254 1262 # arg_split ~ shlex.split, but with unicode bugs fixed by us
1255 1263 lsplit = arg_split(text_until_cursor)[-1]
1256 1264 except ValueError:
1257 1265 # typically an unmatched ", or backslash without escaped char.
1258 1266 if open_quotes:
1259 1267 lsplit = text_until_cursor.split(open_quotes)[-1]
1260 1268 else:
1261 1269 return []
1262 1270 except IndexError:
1263 1271 # tab pressed on empty line
1264 1272 lsplit = ""
1265 1273
1266 1274 if not open_quotes and lsplit != protect_filename(lsplit):
1267 1275 # if protectables are found, do matching on the whole escaped name
1268 1276 has_protectables = True
1269 1277 text0,text = text,lsplit
1270 1278 else:
1271 1279 has_protectables = False
1272 1280 text = os.path.expanduser(text)
1273 1281
1274 1282 if text == "":
1275 1283 return [text_prefix + protect_filename(f) for f in self.glob("*")]
1276 1284
1277 1285 # Compute the matches from the filesystem
1278 1286 if sys.platform == 'win32':
1279 1287 m0 = self.clean_glob(text)
1280 1288 else:
1281 1289 m0 = self.clean_glob(text.replace('\\', ''))
1282 1290
1283 1291 if has_protectables:
1284 1292 # If we had protectables, we need to revert our changes to the
1285 1293 # beginning of filename so that we don't double-write the part
1286 1294 # of the filename we have so far
1287 1295 len_lsplit = len(lsplit)
1288 1296 matches = [text_prefix + text0 +
1289 1297 protect_filename(f[len_lsplit:]) for f in m0]
1290 1298 else:
1291 1299 if open_quotes:
1292 1300 # if we have a string with an open quote, we don't need to
1293 1301 # protect the names beyond the quote (and we _shouldn't_, as
1294 1302 # it would cause bugs when the filesystem call is made).
1295 1303 matches = m0 if sys.platform == "win32" else\
1296 1304 [protect_filename(f, open_quotes) for f in m0]
1297 1305 else:
1298 1306 matches = [text_prefix +
1299 1307 protect_filename(f) for f in m0]
1300 1308
1301 1309 # Mark directories in input list by appending '/' to their names.
1302 1310 return [x+'/' if os.path.isdir(x) else x for x in matches]
1303 1311
1304 1312 def magic_matches(self, text:str):
1305 1313 """Match magics"""
1306 1314 # Get all shell magics now rather than statically, so magics loaded at
1307 1315 # runtime show up too.
1308 1316 lsm = self.shell.magics_manager.lsmagic()
1309 1317 line_magics = lsm['line']
1310 1318 cell_magics = lsm['cell']
1311 1319 pre = self.magic_escape
1312 1320 pre2 = pre+pre
1313 1321
1314 1322 explicit_magic = text.startswith(pre)
1315 1323
1316 1324 # Completion logic:
1317 1325 # - user gives %%: only do cell magics
1318 1326 # - user gives %: do both line and cell magics
1319 1327 # - no prefix: do both
1320 1328 # In other words, line magics are skipped if the user gives %% explicitly
1321 1329 #
1322 1330 # We also exclude magics that match any currently visible names:
1323 1331 # https://github.com/ipython/ipython/issues/4877, unless the user has
1324 1332 # typed a %:
1325 1333 # https://github.com/ipython/ipython/issues/10754
1326 1334 bare_text = text.lstrip(pre)
1327 1335 global_matches = self.global_matches(bare_text)
1328 1336 if not explicit_magic:
1329 1337 def matches(magic):
1330 1338 """
1331 1339 Filter magics, in particular remove magics that match
1332 1340 a name present in global namespace.
1333 1341 """
1334 1342 return ( magic.startswith(bare_text) and
1335 1343 magic not in global_matches )
1336 1344 else:
1337 1345 def matches(magic):
1338 1346 return magic.startswith(bare_text)
1339 1347
1340 1348 comp = [ pre2+m for m in cell_magics if matches(m)]
1341 1349 if not text.startswith(pre2):
1342 1350 comp += [ pre+m for m in line_magics if matches(m)]
1343 1351
1344 1352 return comp
1345 1353
1346 1354 def magic_config_matches(self, text:str) -> List[str]:
1347 1355 """ Match class names and attributes for %config magic """
1348 1356 texts = text.strip().split()
1349 1357
1350 1358 if len(texts) > 0 and (texts[0] == 'config' or texts[0] == '%config'):
1351 1359 # get all configuration classes
1352 1360 classes = sorted(set([ c for c in self.shell.configurables
1353 1361 if c.__class__.class_traits(config=True)
1354 1362 ]), key=lambda x: x.__class__.__name__)
1355 1363 classnames = [ c.__class__.__name__ for c in classes ]
1356 1364
1357 1365 # return all classnames if config or %config is given
1358 1366 if len(texts) == 1:
1359 1367 return classnames
1360 1368
1361 1369 # match classname
1362 1370 classname_texts = texts[1].split('.')
1363 1371 classname = classname_texts[0]
1364 1372 classname_matches = [ c for c in classnames
1365 1373 if c.startswith(classname) ]
1366 1374
1367 1375 # return matched classes or the matched class with attributes
1368 1376 if texts[1].find('.') < 0:
1369 1377 return classname_matches
1370 1378 elif len(classname_matches) == 1 and \
1371 1379 classname_matches[0] == classname:
1372 1380 cls = classes[classnames.index(classname)].__class__
1373 1381 help = cls.class_get_help()
1374 1382 # strip leading '--' from cl-args:
1375 1383 help = re.sub(re.compile(r'^--', re.MULTILINE), '', help)
1376 1384 return [ attr.split('=')[0]
1377 1385 for attr in help.strip().splitlines()
1378 1386 if attr.startswith(texts[1]) ]
1379 1387 return []
1380 1388
1381 1389 def magic_color_matches(self, text:str) -> List[str] :
1382 1390 """ Match color schemes for %colors magic"""
1383 1391 texts = text.split()
1384 1392 if text.endswith(' '):
1385 1393 # .split() strips off the trailing whitespace. Add '' back
1386 1394 # so that: '%colors ' -> ['%colors', '']
1387 1395 texts.append('')
1388 1396
1389 1397 if len(texts) == 2 and (texts[0] == 'colors' or texts[0] == '%colors'):
1390 1398 prefix = texts[1]
1391 1399 return [ color for color in InspectColors.keys()
1392 1400 if color.startswith(prefix) ]
1393 1401 return []
1394 1402
1395 1403 def _jedi_matches(self, cursor_column:int, cursor_line:int, text:str) -> Iterable[Any]:
1396 1404 """
1397 1405 Return a list of :any:`jedi.api.Completions` object from a ``text`` and
1398 1406 cursor position.
1399 1407
1400 1408 Parameters
1401 1409 ----------
1402 1410 cursor_column : int
1403 1411 column position of the cursor in ``text``, 0-indexed.
1404 1412 cursor_line : int
1405 1413 line position of the cursor in ``text``, 0-indexed
1406 1414 text : str
1407 1415 text to complete
1408 1416
1409 1417 Notes
1410 1418 -----
1411 1419 If ``IPCompleter.debug`` is ``True`` may return a :any:`_FakeJediCompletion`
1412 1420 object containing a string with the Jedi debug information attached.
1413 1421 """
1414 1422 namespaces = [self.namespace]
1415 1423 if self.global_namespace is not None:
1416 1424 namespaces.append(self.global_namespace)
1417 1425
1418 1426 completion_filter = lambda x:x
1419 1427 offset = cursor_to_position(text, cursor_line, cursor_column)
1420 1428 # filter output if we are completing for object members
1421 1429 if offset:
1422 1430 pre = text[offset-1]
1423 1431 if pre == '.':
1424 1432 if self.omit__names == 2:
1425 1433 completion_filter = lambda c:not c.name.startswith('_')
1426 1434 elif self.omit__names == 1:
1427 1435 completion_filter = lambda c:not (c.name.startswith('__') and c.name.endswith('__'))
1428 1436 elif self.omit__names == 0:
1429 1437 completion_filter = lambda x:x
1430 1438 else:
1431 1439 raise ValueError("Don't understand self.omit__names == {}".format(self.omit__names))
1432 1440
1433 1441 interpreter = jedi.Interpreter(text[:offset], namespaces)
1434 1442 try_jedi = True
1435 1443
1436 1444 try:
1437 1445 # find the first token in the current tree -- if it is a ' or " then we are in a string
1438 1446 completing_string = False
1439 1447 try:
1440 1448 first_child = next(c for c in interpreter._get_module().tree_node.children if hasattr(c, 'value'))
1441 1449 except StopIteration:
1442 1450 pass
1443 1451 else:
1444 1452 # note the value may be ', ", or it may also be ''' or """, or
1445 1453 # in some cases, """what/you/typed..., but all of these are
1446 1454 # strings.
1447 1455 completing_string = len(first_child.value) > 0 and first_child.value[0] in {"'", '"'}
1448 1456
1449 1457 # if we are in a string jedi is likely not the right candidate for
1450 1458 # now. Skip it.
1451 1459 try_jedi = not completing_string
1452 1460 except Exception as e:
1453 1461 # many of things can go wrong, we are using private API just don't crash.
1454 1462 if self.debug:
1455 1463 print("Error detecting if completing a non-finished string :", e, '|')
1456 1464
1457 1465 if not try_jedi:
1458 1466 return []
1459 1467 try:
1460 1468 return filter(completion_filter, interpreter.complete(column=cursor_column, line=cursor_line + 1))
1461 1469 except Exception as e:
1462 1470 if self.debug:
1463 1471 return [_FakeJediCompletion('Oops Jedi has crashed, please report a bug with the following:\n"""\n%s\ns"""' % (e))]
1464 1472 else:
1465 1473 return []
1466 1474
1467 1475 def python_matches(self, text:str)->List[str]:
1468 1476 """Match attributes or global python names"""
1469 1477 if "." in text:
1470 1478 try:
1471 1479 matches = self.attr_matches(text)
1472 1480 if text.endswith('.') and self.omit__names:
1473 1481 if self.omit__names == 1:
1474 1482 # true if txt is _not_ a __ name, false otherwise:
1475 1483 no__name = (lambda txt:
1476 1484 re.match(r'.*\.__.*?__',txt) is None)
1477 1485 else:
1478 1486 # true if txt is _not_ a _ name, false otherwise:
1479 1487 no__name = (lambda txt:
1480 1488 re.match(r'\._.*?',txt[txt.rindex('.'):]) is None)
1481 1489 matches = filter(no__name, matches)
1482 1490 except NameError:
1483 1491 # catches <undefined attributes>.<tab>
1484 1492 matches = []
1485 1493 else:
1486 1494 matches = self.global_matches(text)
1487 1495 return matches
1488 1496
1489 1497 def _default_arguments_from_docstring(self, doc):
1490 1498 """Parse the first line of docstring for call signature.
1491 1499
1492 1500 Docstring should be of the form 'min(iterable[, key=func])\n'.
1493 1501 It can also parse cython docstring of the form
1494 1502 'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)'.
1495 1503 """
1496 1504 if doc is None:
1497 1505 return []
1498 1506
1499 1507 #care only the firstline
1500 1508 line = doc.lstrip().splitlines()[0]
1501 1509
1502 1510 #p = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1503 1511 #'min(iterable[, key=func])\n' -> 'iterable[, key=func]'
1504 1512 sig = self.docstring_sig_re.search(line)
1505 1513 if sig is None:
1506 1514 return []
1507 1515 # iterable[, key=func]' -> ['iterable[' ,' key=func]']
1508 1516 sig = sig.groups()[0].split(',')
1509 1517 ret = []
1510 1518 for s in sig:
1511 1519 #re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1512 1520 ret += self.docstring_kwd_re.findall(s)
1513 1521 return ret
1514 1522
1515 1523 def _default_arguments(self, obj):
1516 1524 """Return the list of default arguments of obj if it is callable,
1517 1525 or empty list otherwise."""
1518 1526 call_obj = obj
1519 1527 ret = []
1520 1528 if inspect.isbuiltin(obj):
1521 1529 pass
1522 1530 elif not (inspect.isfunction(obj) or inspect.ismethod(obj)):
1523 1531 if inspect.isclass(obj):
1524 1532 #for cython embedsignature=True the constructor docstring
1525 1533 #belongs to the object itself not __init__
1526 1534 ret += self._default_arguments_from_docstring(
1527 1535 getattr(obj, '__doc__', ''))
1528 1536 # for classes, check for __init__,__new__
1529 1537 call_obj = (getattr(obj, '__init__', None) or
1530 1538 getattr(obj, '__new__', None))
1531 1539 # for all others, check if they are __call__able
1532 1540 elif hasattr(obj, '__call__'):
1533 1541 call_obj = obj.__call__
1534 1542 ret += self._default_arguments_from_docstring(
1535 1543 getattr(call_obj, '__doc__', ''))
1536 1544
1537 1545 _keeps = (inspect.Parameter.KEYWORD_ONLY,
1538 1546 inspect.Parameter.POSITIONAL_OR_KEYWORD)
1539 1547
1540 1548 try:
1541 1549 sig = inspect.signature(obj)
1542 1550 ret.extend(k for k, v in sig.parameters.items() if
1543 1551 v.kind in _keeps)
1544 1552 except ValueError:
1545 1553 pass
1546 1554
1547 1555 return list(set(ret))
1548 1556
1549 1557 def python_func_kw_matches(self, text):
1550 1558 """Match named parameters (kwargs) of the last open function"""
1551 1559
1552 1560 if "." in text: # a parameter cannot be dotted
1553 1561 return []
1554 1562 try: regexp = self.__funcParamsRegex
1555 1563 except AttributeError:
1556 1564 regexp = self.__funcParamsRegex = re.compile(r'''
1557 1565 '.*?(?<!\\)' | # single quoted strings or
1558 1566 ".*?(?<!\\)" | # double quoted strings or
1559 1567 \w+ | # identifier
1560 1568 \S # other characters
1561 1569 ''', re.VERBOSE | re.DOTALL)
1562 1570 # 1. find the nearest identifier that comes before an unclosed
1563 1571 # parenthesis before the cursor
1564 1572 # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
1565 1573 tokens = regexp.findall(self.text_until_cursor)
1566 1574 iterTokens = reversed(tokens); openPar = 0
1567 1575
1568 1576 for token in iterTokens:
1569 1577 if token == ')':
1570 1578 openPar -= 1
1571 1579 elif token == '(':
1572 1580 openPar += 1
1573 1581 if openPar > 0:
1574 1582 # found the last unclosed parenthesis
1575 1583 break
1576 1584 else:
1577 1585 return []
1578 1586 # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
1579 1587 ids = []
1580 1588 isId = re.compile(r'\w+$').match
1581 1589
1582 1590 while True:
1583 1591 try:
1584 1592 ids.append(next(iterTokens))
1585 1593 if not isId(ids[-1]):
1586 1594 ids.pop(); break
1587 1595 if not next(iterTokens) == '.':
1588 1596 break
1589 1597 except StopIteration:
1590 1598 break
1591 1599
1592 1600 # Find all named arguments already assigned to, as to avoid suggesting
1593 1601 # them again
1594 1602 usedNamedArgs = set()
1595 1603 par_level = -1
1596 1604 for token, next_token in zip(tokens, tokens[1:]):
1597 1605 if token == '(':
1598 1606 par_level += 1
1599 1607 elif token == ')':
1600 1608 par_level -= 1
1601 1609
1602 1610 if par_level != 0:
1603 1611 continue
1604 1612
1605 1613 if next_token != '=':
1606 1614 continue
1607 1615
1608 1616 usedNamedArgs.add(token)
1609 1617
1610 1618 argMatches = []
1611 1619 try:
1612 1620 callableObj = '.'.join(ids[::-1])
1613 1621 namedArgs = self._default_arguments(eval(callableObj,
1614 1622 self.namespace))
1615 1623
1616 1624 # Remove used named arguments from the list, no need to show twice
1617 1625 for namedArg in set(namedArgs) - usedNamedArgs:
1618 1626 if namedArg.startswith(text):
1619 1627 argMatches.append("%s=" %namedArg)
1620 1628 except:
1621 1629 pass
1622 1630
1623 1631 return argMatches
1624 1632
1625 1633 @staticmethod
1626 1634 def _get_keys(obj: Any) -> List[Any]:
1627 1635 # Objects can define their own completions by defining an
1628 1636 # _ipy_key_completions_() method.
1629 1637 method = get_real_method(obj, '_ipython_key_completions_')
1630 1638 if method is not None:
1631 1639 return method()
1632 1640
1633 1641 # Special case some common in-memory dict-like types
1634 1642 if isinstance(obj, dict) or\
1635 1643 _safe_isinstance(obj, 'pandas', 'DataFrame'):
1636 1644 try:
1637 1645 return list(obj.keys())
1638 1646 except Exception:
1639 1647 return []
1640 1648 elif _safe_isinstance(obj, 'numpy', 'ndarray') or\
1641 1649 _safe_isinstance(obj, 'numpy', 'void'):
1642 1650 return obj.dtype.names or []
1643 1651 return []
1644 1652
1645 1653 def dict_key_matches(self, text:str) -> List[str]:
1646 1654 "Match string keys in a dictionary, after e.g. 'foo[' "
1647 1655
1648 1656
1649 1657 if self.__dict_key_regexps is not None:
1650 1658 regexps = self.__dict_key_regexps
1651 1659 else:
1652 1660 dict_key_re_fmt = r'''(?x)
1653 1661 ( # match dict-referring expression wrt greedy setting
1654 1662 %s
1655 1663 )
1656 1664 \[ # open bracket
1657 1665 \s* # and optional whitespace
1658 1666 # Capture any number of str-like objects (e.g. "a", "b", 'c')
1659 1667 ((?:[uUbB]? # string prefix (r not handled)
1660 1668 (?:
1661 1669 '(?:[^']|(?<!\\)\\')*'
1662 1670 |
1663 1671 "(?:[^"]|(?<!\\)\\")*"
1664 1672 )
1665 1673 \s*,\s*
1666 1674 )*)
1667 1675 ([uUbB]? # string prefix (r not handled)
1668 1676 (?: # unclosed string
1669 1677 '(?:[^']|(?<!\\)\\')*
1670 1678 |
1671 1679 "(?:[^"]|(?<!\\)\\")*
1672 1680 )
1673 1681 )?
1674 1682 $
1675 1683 '''
1676 1684 regexps = self.__dict_key_regexps = {
1677 1685 False: re.compile(dict_key_re_fmt % r'''
1678 1686 # identifiers separated by .
1679 1687 (?!\d)\w+
1680 1688 (?:\.(?!\d)\w+)*
1681 1689 '''),
1682 1690 True: re.compile(dict_key_re_fmt % '''
1683 1691 .+
1684 1692 ''')
1685 1693 }
1686 1694
1687 1695 match = regexps[self.greedy].search(self.text_until_cursor)
1688 1696
1689 1697 if match is None:
1690 1698 return []
1691 1699
1692 1700 expr, prefix0, prefix = match.groups()
1693 1701 try:
1694 1702 obj = eval(expr, self.namespace)
1695 1703 except Exception:
1696 1704 try:
1697 1705 obj = eval(expr, self.global_namespace)
1698 1706 except Exception:
1699 1707 return []
1700 1708
1701 1709 keys = self._get_keys(obj)
1702 1710 if not keys:
1703 1711 return keys
1704 1712
1705 1713 extra_prefix = eval(prefix0) if prefix0 != '' else None
1706 1714
1707 1715 closing_quote, token_offset, matches = match_dict_keys(keys, prefix, self.splitter.delims, extra_prefix=extra_prefix)
1708 1716 if not matches:
1709 1717 return matches
1710 1718
1711 1719 # get the cursor position of
1712 1720 # - the text being completed
1713 1721 # - the start of the key text
1714 1722 # - the start of the completion
1715 1723 text_start = len(self.text_until_cursor) - len(text)
1716 1724 if prefix:
1717 1725 key_start = match.start(3)
1718 1726 completion_start = key_start + token_offset
1719 1727 else:
1720 1728 key_start = completion_start = match.end()
1721 1729
1722 1730 # grab the leading prefix, to make sure all completions start with `text`
1723 1731 if text_start > key_start:
1724 1732 leading = ''
1725 1733 else:
1726 1734 leading = text[text_start:completion_start]
1727 1735
1728 1736 # the index of the `[` character
1729 1737 bracket_idx = match.end(1)
1730 1738
1731 1739 # append closing quote and bracket as appropriate
1732 1740 # this is *not* appropriate if the opening quote or bracket is outside
1733 1741 # the text given to this method
1734 1742 suf = ''
1735 1743 continuation = self.line_buffer[len(self.text_until_cursor):]
1736 1744 if key_start > text_start and closing_quote:
1737 1745 # quotes were opened inside text, maybe close them
1738 1746 if continuation.startswith(closing_quote):
1739 1747 continuation = continuation[len(closing_quote):]
1740 1748 else:
1741 1749 suf += closing_quote
1742 1750 if bracket_idx > text_start:
1743 1751 # brackets were opened inside text, maybe close them
1744 1752 if not continuation.startswith(']'):
1745 1753 suf += ']'
1746 1754
1747 1755 return [leading + k + suf for k in matches]
1748 1756
1749 1757 @staticmethod
1750 1758 def unicode_name_matches(text:str) -> Tuple[str, List[str]] :
1751 1759 """Match Latex-like syntax for unicode characters base
1752 1760 on the name of the character.
1753 1761
1754 1762 This does ``\\GREEK SMALL LETTER ETA`` -> ``η``
1755 1763
1756 1764 Works only on valid python 3 identifier, or on combining characters that
1757 1765 will combine to form a valid identifier.
1758 1766 """
1759 1767 slashpos = text.rfind('\\')
1760 1768 if slashpos > -1:
1761 1769 s = text[slashpos+1:]
1762 1770 try :
1763 1771 unic = unicodedata.lookup(s)
1764 1772 # allow combining chars
1765 1773 if ('a'+unic).isidentifier():
1766 1774 return '\\'+s,[unic]
1767 1775 except KeyError:
1768 1776 pass
1769 1777 return '', []
1770 1778
1771 1779
1772 1780 def latex_matches(self, text:str) -> Tuple[str, Sequence[str]]:
1773 1781 """Match Latex syntax for unicode characters.
1774 1782
1775 1783 This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
1776 1784 """
1777 1785 slashpos = text.rfind('\\')
1778 1786 if slashpos > -1:
1779 1787 s = text[slashpos:]
1780 1788 if s in latex_symbols:
1781 1789 # Try to complete a full latex symbol to unicode
1782 1790 # \\alpha -> α
1783 1791 return s, [latex_symbols[s]]
1784 1792 else:
1785 1793 # If a user has partially typed a latex symbol, give them
1786 1794 # a full list of options \al -> [\aleph, \alpha]
1787 1795 matches = [k for k in latex_symbols if k.startswith(s)]
1788 1796 if matches:
1789 1797 return s, matches
1790 1798 return '', ()
1791 1799
1792 1800 def dispatch_custom_completer(self, text):
1793 1801 if not self.custom_completers:
1794 1802 return
1795 1803
1796 1804 line = self.line_buffer
1797 1805 if not line.strip():
1798 1806 return None
1799 1807
1800 1808 # Create a little structure to pass all the relevant information about
1801 1809 # the current completion to any custom completer.
1802 1810 event = SimpleNamespace()
1803 1811 event.line = line
1804 1812 event.symbol = text
1805 1813 cmd = line.split(None,1)[0]
1806 1814 event.command = cmd
1807 1815 event.text_until_cursor = self.text_until_cursor
1808 1816
1809 1817 # for foo etc, try also to find completer for %foo
1810 1818 if not cmd.startswith(self.magic_escape):
1811 1819 try_magic = self.custom_completers.s_matches(
1812 1820 self.magic_escape + cmd)
1813 1821 else:
1814 1822 try_magic = []
1815 1823
1816 1824 for c in itertools.chain(self.custom_completers.s_matches(cmd),
1817 1825 try_magic,
1818 1826 self.custom_completers.flat_matches(self.text_until_cursor)):
1819 1827 try:
1820 1828 res = c(event)
1821 1829 if res:
1822 1830 # first, try case sensitive match
1823 1831 withcase = [r for r in res if r.startswith(text)]
1824 1832 if withcase:
1825 1833 return withcase
1826 1834 # if none, then case insensitive ones are ok too
1827 1835 text_low = text.lower()
1828 1836 return [r for r in res if r.lower().startswith(text_low)]
1829 1837 except TryNext:
1830 1838 pass
1831 1839 except KeyboardInterrupt:
1832 1840 """
1833 1841 If custom completer take too long,
1834 1842 let keyboard interrupt abort and return nothing.
1835 1843 """
1836 1844 break
1837 1845
1838 1846 return None
1839 1847
1840 1848 def completions(self, text: str, offset: int)->Iterator[Completion]:
1841 1849 """
1842 1850 Returns an iterator over the possible completions
1843 1851
1844 1852 .. warning::
1845 1853
1846 1854 Unstable
1847 1855
1848 1856 This function is unstable, API may change without warning.
1849 1857 It will also raise unless use in proper context manager.
1850 1858
1851 1859 Parameters
1852 1860 ----------
1853 1861 text : str
1854 1862 Full text of the current input, multi line string.
1855 1863 offset : int
1856 1864 Integer representing the position of the cursor in ``text``. Offset
1857 1865 is 0-based indexed.
1858 1866
1859 1867 Yields
1860 1868 ------
1861 1869 Completion
1862 1870
1863 1871 Notes
1864 1872 -----
1865 1873 The cursor on a text can either be seen as being "in between"
1866 1874 characters or "On" a character depending on the interface visible to
1867 1875 the user. For consistency the cursor being on "in between" characters X
1868 1876 and Y is equivalent to the cursor being "on" character Y, that is to say
1869 1877 the character the cursor is on is considered as being after the cursor.
1870 1878
1871 1879 Combining characters may span more that one position in the
1872 1880 text.
1873 1881
1874 1882 .. note::
1875 1883
1876 1884 If ``IPCompleter.debug`` is :any:`True` will yield a ``--jedi/ipython--``
1877 1885 fake Completion token to distinguish completion returned by Jedi
1878 1886 and usual IPython completion.
1879 1887
1880 1888 .. note::
1881 1889
1882 1890 Completions are not completely deduplicated yet. If identical
1883 1891 completions are coming from different sources this function does not
1884 1892 ensure that each completion object will only be present once.
1885 1893 """
1886 1894 warnings.warn("_complete is a provisional API (as of IPython 6.0). "
1887 1895 "It may change without warnings. "
1888 1896 "Use in corresponding context manager.",
1889 1897 category=ProvisionalCompleterWarning, stacklevel=2)
1890 1898
1891 1899 seen = set()
1892 1900 profiler:Optional[cProfile.Profile]
1893 1901 try:
1894 1902 if self.profile_completions:
1895 1903 import cProfile
1896 1904 profiler = cProfile.Profile()
1897 1905 profiler.enable()
1898 1906 else:
1899 1907 profiler = None
1900 1908
1901 1909 for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
1902 1910 if c and (c in seen):
1903 1911 continue
1904 1912 yield c
1905 1913 seen.add(c)
1906 1914 except KeyboardInterrupt:
1907 1915 """if completions take too long and users send keyboard interrupt,
1908 1916 do not crash and return ASAP. """
1909 1917 pass
1910 1918 finally:
1911 1919 if profiler is not None:
1912 1920 profiler.disable()
1913 1921 ensure_dir_exists(self.profiler_output_dir)
1914 1922 output_path = os.path.join(self.profiler_output_dir, str(uuid.uuid4()))
1915 1923 print("Writing profiler output to", output_path)
1916 1924 profiler.dump_stats(output_path)
1917 1925
1918 1926 def _completions(self, full_text: str, offset: int, *, _timeout) -> Iterator[Completion]:
1919 1927 """
1920 1928 Core completion module.Same signature as :any:`completions`, with the
1921 1929 extra `timeout` parameter (in seconds).
1922 1930
1923 1931 Computing jedi's completion ``.type`` can be quite expensive (it is a
1924 1932 lazy property) and can require some warm-up, more warm up than just
1925 1933 computing the ``name`` of a completion. The warm-up can be :
1926 1934
1927 1935 - Long warm-up the first time a module is encountered after
1928 1936 install/update: actually build parse/inference tree.
1929 1937
1930 1938 - first time the module is encountered in a session: load tree from
1931 1939 disk.
1932 1940
1933 1941 We don't want to block completions for tens of seconds so we give the
1934 1942 completer a "budget" of ``_timeout`` seconds per invocation to compute
1935 1943 completions types, the completions that have not yet been computed will
1936 1944 be marked as "unknown" an will have a chance to be computed next round
1937 1945 are things get cached.
1938 1946
1939 1947 Keep in mind that Jedi is not the only thing treating the completion so
1940 1948 keep the timeout short-ish as if we take more than 0.3 second we still
1941 1949 have lots of processing to do.
1942 1950
1943 1951 """
1944 1952 deadline = time.monotonic() + _timeout
1945 1953
1946 1954
1947 1955 before = full_text[:offset]
1948 1956 cursor_line, cursor_column = position_to_cursor(full_text, offset)
1949 1957
1950 1958 matched_text, matches, matches_origin, jedi_matches = self._complete(
1951 1959 full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column)
1952 1960
1953 1961 iter_jm = iter(jedi_matches)
1954 1962 if _timeout:
1955 1963 for jm in iter_jm:
1956 1964 try:
1957 1965 type_ = jm.type
1958 1966 except Exception:
1959 1967 if self.debug:
1960 1968 print("Error in Jedi getting type of ", jm)
1961 1969 type_ = None
1962 1970 delta = len(jm.name_with_symbols) - len(jm.complete)
1963 1971 if type_ == 'function':
1964 1972 signature = _make_signature(jm)
1965 1973 else:
1966 1974 signature = ''
1967 1975 yield Completion(start=offset - delta,
1968 1976 end=offset,
1969 1977 text=jm.name_with_symbols,
1970 1978 type=type_,
1971 1979 signature=signature,
1972 1980 _origin='jedi')
1973 1981
1974 1982 if time.monotonic() > deadline:
1975 1983 break
1976 1984
1977 1985 for jm in iter_jm:
1978 1986 delta = len(jm.name_with_symbols) - len(jm.complete)
1979 1987 yield Completion(start=offset - delta,
1980 1988 end=offset,
1981 1989 text=jm.name_with_symbols,
1982 1990 type='<unknown>', # don't compute type for speed
1983 1991 _origin='jedi',
1984 1992 signature='')
1985 1993
1986 1994
1987 1995 start_offset = before.rfind(matched_text)
1988 1996
1989 1997 # TODO:
1990 1998 # Suppress this, right now just for debug.
1991 1999 if jedi_matches and matches and self.debug:
1992 2000 yield Completion(start=start_offset, end=offset, text='--jedi/ipython--',
1993 2001 _origin='debug', type='none', signature='')
1994 2002
1995 2003 # I'm unsure if this is always true, so let's assert and see if it
1996 2004 # crash
1997 2005 assert before.endswith(matched_text)
1998 2006 for m, t in zip(matches, matches_origin):
1999 2007 yield Completion(start=start_offset, end=offset, text=m, _origin=t, signature='', type='<unknown>')
2000 2008
2001 2009
2002 2010 def complete(self, text=None, line_buffer=None, cursor_pos=None) -> Tuple[str, Sequence[str]]:
2003 2011 """Find completions for the given text and line context.
2004 2012
2005 2013 Note that both the text and the line_buffer are optional, but at least
2006 2014 one of them must be given.
2007 2015
2008 2016 Parameters
2009 2017 ----------
2010 2018 text : string, optional
2011 2019 Text to perform the completion on. If not given, the line buffer
2012 2020 is split using the instance's CompletionSplitter object.
2013 2021 line_buffer : string, optional
2014 2022 If not given, the completer attempts to obtain the current line
2015 2023 buffer via readline. This keyword allows clients which are
2016 2024 requesting for text completions in non-readline contexts to inform
2017 2025 the completer of the entire text.
2018 2026 cursor_pos : int, optional
2019 2027 Index of the cursor in the full line buffer. Should be provided by
2020 2028 remote frontends where kernel has no access to frontend state.
2021 2029
2022 2030 Returns
2023 2031 -------
2024 2032 Tuple of two items:
2025 2033 text : str
2026 2034 Text that was actually used in the completion.
2027 2035 matches : list
2028 2036 A list of completion matches.
2029 2037
2030 2038 Notes
2031 2039 -----
2032 2040 This API is likely to be deprecated and replaced by
2033 2041 :any:`IPCompleter.completions` in the future.
2034 2042
2035 2043 """
2036 2044 warnings.warn('`Completer.complete` is pending deprecation since '
2037 2045 'IPython 6.0 and will be replaced by `Completer.completions`.',
2038 2046 PendingDeprecationWarning)
2039 2047 # potential todo, FOLD the 3rd throw away argument of _complete
2040 2048 # into the first 2 one.
2041 2049 return self._complete(line_buffer=line_buffer, cursor_pos=cursor_pos, text=text, cursor_line=0)[:2]
2042 2050
2043 2051 def _complete(self, *, cursor_line, cursor_pos, line_buffer=None, text=None,
2044 2052 full_text=None) -> _CompleteResult:
2045 2053 """
2046 2054 Like complete but can also returns raw jedi completions as well as the
2047 2055 origin of the completion text. This could (and should) be made much
2048 2056 cleaner but that will be simpler once we drop the old (and stateful)
2049 2057 :any:`complete` API.
2050 2058
2051 2059 With current provisional API, cursor_pos act both (depending on the
2052 2060 caller) as the offset in the ``text`` or ``line_buffer``, or as the
2053 2061 ``column`` when passing multiline strings this could/should be renamed
2054 2062 but would add extra noise.
2055 2063
2056 2064 Parameters
2057 2065 ----------
2058 cursor_line :
2066 cursor_line
2059 2067 Index of the line the cursor is on. 0 indexed.
2060 cursor_pos :
2068 cursor_pos
2061 2069 Position of the cursor in the current line/line_buffer/text. 0
2062 2070 indexed.
2063 2071 line_buffer : optional, str
2064 2072 The current line the cursor is in, this is mostly due to legacy
2065 2073 reason that readline coudl only give a us the single current line.
2066 2074 Prefer `full_text`.
2067 2075 text : str
2068 2076 The current "token" the cursor is in, mostly also for historical
2069 2077 reasons. as the completer would trigger only after the current line
2070 2078 was parsed.
2071 2079 full_text : str
2072 2080 Full text of the current cell.
2073 2081
2074 2082 Returns
2075 2083 -------
2076 2084 A tuple of N elements which are (likely):
2077 2085 matched_text: ? the text that the complete matched
2078 2086 matches: list of completions ?
2079 2087 matches_origin: ? list same length as matches, and where each completion came from
2080 2088 jedi_matches: list of Jedi matches, have it's own structure.
2081 2089 """
2082 2090
2083 2091
2084 2092 # if the cursor position isn't given, the only sane assumption we can
2085 2093 # make is that it's at the end of the line (the common case)
2086 2094 if cursor_pos is None:
2087 2095 cursor_pos = len(line_buffer) if text is None else len(text)
2088 2096
2089 2097 if self.use_main_ns:
2090 2098 self.namespace = __main__.__dict__
2091 2099
2092 2100 # if text is either None or an empty string, rely on the line buffer
2093 2101 if (not line_buffer) and full_text:
2094 2102 line_buffer = full_text.split('\n')[cursor_line]
2095 2103 if not text: # issue #11508: check line_buffer before calling split_line
2096 2104 text = self.splitter.split_line(line_buffer, cursor_pos) if line_buffer else ''
2097 2105
2098 2106 if self.backslash_combining_completions:
2099 2107 # allow deactivation of these on windows.
2100 2108 base_text = text if not line_buffer else line_buffer[:cursor_pos]
2101 2109
2102 2110 for meth in (self.latex_matches,
2103 2111 self.unicode_name_matches,
2104 2112 back_latex_name_matches,
2105 2113 back_unicode_name_matches,
2106 2114 self.fwd_unicode_match):
2107 2115 name_text, name_matches = meth(base_text)
2108 2116 if name_text:
2109 2117 return _CompleteResult(name_text, name_matches[:MATCHES_LIMIT], \
2110 2118 [meth.__qualname__]*min(len(name_matches), MATCHES_LIMIT), ())
2111 2119
2112 2120
2113 2121 # If no line buffer is given, assume the input text is all there was
2114 2122 if line_buffer is None:
2115 2123 line_buffer = text
2116 2124
2117 2125 self.line_buffer = line_buffer
2118 2126 self.text_until_cursor = self.line_buffer[:cursor_pos]
2119 2127
2120 2128 # Do magic arg matches
2121 2129 for matcher in self.magic_arg_matchers:
2122 2130 matches = list(matcher(line_buffer))[:MATCHES_LIMIT]
2123 2131 if matches:
2124 2132 origins = [matcher.__qualname__] * len(matches)
2125 2133 return _CompleteResult(text, matches, origins, ())
2126 2134
2127 2135 # Start with a clean slate of completions
2128 2136 matches = []
2129 2137
2130 2138 # FIXME: we should extend our api to return a dict with completions for
2131 2139 # different types of objects. The rlcomplete() method could then
2132 2140 # simply collapse the dict into a list for readline, but we'd have
2133 2141 # richer completion semantics in other environments.
2134 2142 completions:Iterable[Any] = []
2135 2143 if self.use_jedi:
2136 2144 if not full_text:
2137 2145 full_text = line_buffer
2138 2146 completions = self._jedi_matches(
2139 2147 cursor_pos, cursor_line, full_text)
2140 2148
2141 2149 if self.merge_completions:
2142 2150 matches = []
2143 2151 for matcher in self.matchers:
2144 2152 try:
2145 2153 matches.extend([(m, matcher.__qualname__)
2146 2154 for m in matcher(text)])
2147 2155 except:
2148 2156 # Show the ugly traceback if the matcher causes an
2149 2157 # exception, but do NOT crash the kernel!
2150 2158 sys.excepthook(*sys.exc_info())
2151 2159 else:
2152 2160 for matcher in self.matchers:
2153 2161 matches = [(m, matcher.__qualname__)
2154 2162 for m in matcher(text)]
2155 2163 if matches:
2156 2164 break
2157 2165
2158 2166 seen = set()
2159 2167 filtered_matches = set()
2160 2168 for m in matches:
2161 2169 t, c = m
2162 2170 if t not in seen:
2163 2171 filtered_matches.add(m)
2164 2172 seen.add(t)
2165 2173
2166 2174 _filtered_matches = sorted(filtered_matches, key=lambda x: completions_sorting_key(x[0]))
2167 2175
2168 2176 custom_res = [(m, 'custom') for m in self.dispatch_custom_completer(text) or []]
2169 2177
2170 2178 _filtered_matches = custom_res or _filtered_matches
2171 2179
2172 2180 _filtered_matches = _filtered_matches[:MATCHES_LIMIT]
2173 2181 _matches = [m[0] for m in _filtered_matches]
2174 2182 origins = [m[1] for m in _filtered_matches]
2175 2183
2176 2184 self.matches = _matches
2177 2185
2178 2186 return _CompleteResult(text, _matches, origins, completions)
2179 2187
2180 2188 def fwd_unicode_match(self, text:str) -> Tuple[str, Sequence[str]]:
2181 2189 """
2182 2190 Forward match a string starting with a backslash with a list of
2183 2191 potential Unicode completions.
2184 2192
2185 2193 Will compute list list of Unicode character names on first call and cache it.
2186 2194
2187 2195 Returns
2188 2196 -------
2189 2197 At tuple with:
2190 2198 - matched text (empty if no matches)
2191 2199 - list of potential completions, empty tuple otherwise)
2192 2200 """
2193 2201 # TODO: self.unicode_names is here a list we traverse each time with ~100k elements.
2194 2202 # We could do a faster match using a Trie.
2195 2203
2196 2204 # Using pygtrie the following seem to work:
2197 2205
2198 2206 # s = PrefixSet()
2199 2207
2200 2208 # for c in range(0,0x10FFFF + 1):
2201 2209 # try:
2202 2210 # s.add(unicodedata.name(chr(c)))
2203 2211 # except ValueError:
2204 2212 # pass
2205 2213 # [''.join(k) for k in s.iter(prefix)]
2206 2214
2207 2215 # But need to be timed and adds an extra dependency.
2208 2216
2209 2217 slashpos = text.rfind('\\')
2210 2218 # if text starts with slash
2211 2219 if slashpos > -1:
2212 2220 # PERF: It's important that we don't access self._unicode_names
2213 2221 # until we're inside this if-block. _unicode_names is lazily
2214 2222 # initialized, and it takes a user-noticeable amount of time to
2215 2223 # initialize it, so we don't want to initialize it unless we're
2216 2224 # actually going to use it.
2217 2225 s = text[slashpos+1:]
2218 2226 candidates = [x for x in self.unicode_names if x.startswith(s)]
2219 2227 if candidates:
2220 2228 return s, candidates
2221 2229 else:
2222 2230 return '', ()
2223 2231
2224 2232 # if text does not start with slash
2225 2233 else:
2226 2234 return '', ()
2227 2235
2228 2236 @property
2229 2237 def unicode_names(self) -> List[str]:
2230 2238 """List of names of unicode code points that can be completed.
2231 2239
2232 2240 The list is lazily initialized on first access.
2233 2241 """
2234 2242 if self._unicode_names is None:
2235 2243 names = []
2236 2244 for c in range(0,0x10FFFF + 1):
2237 2245 try:
2238 2246 names.append(unicodedata.name(chr(c)))
2239 2247 except ValueError:
2240 2248 pass
2241 2249 self._unicode_names = _unicode_name_compute(_UNICODE_RANGES)
2242 2250
2243 2251 return self._unicode_names
2244 2252
2245 2253 def _unicode_name_compute(ranges:List[Tuple[int,int]]) -> List[str]:
2246 2254 names = []
2247 2255 for start,stop in ranges:
2248 2256 for c in range(start, stop) :
2249 2257 try:
2250 2258 names.append(unicodedata.name(chr(c)))
2251 2259 except ValueError:
2252 2260 pass
2253 2261 return names
@@ -1,223 +1,237 b''
1 1 # encoding: utf-8
2 2 """sys.excepthook for IPython itself, leaves a detailed report on disk.
3 3
4 4 Authors:
5 5
6 6 * Fernando Perez
7 7 * Brian E. Granger
8 8 """
9 9
10 10 #-----------------------------------------------------------------------------
11 11 # Copyright (C) 2001-2007 Fernando Perez. <fperez@colorado.edu>
12 12 # Copyright (C) 2008-2011 The IPython Development Team
13 13 #
14 14 # Distributed under the terms of the BSD License. The full license is in
15 15 # the file COPYING, distributed as part of this software.
16 16 #-----------------------------------------------------------------------------
17 17
18 18 #-----------------------------------------------------------------------------
19 19 # Imports
20 20 #-----------------------------------------------------------------------------
21 21
22 22 import os
23 23 import sys
24 24 import traceback
25 25 from pprint import pformat
26 26 from pathlib import Path
27 27
28 28 from IPython.core import ultratb
29 29 from IPython.core.release import author_email
30 30 from IPython.utils.sysinfo import sys_info
31 31 from IPython.utils.py3compat import input
32 32
33 33 from IPython.core.release import __version__ as version
34 34
35 from typing import Optional
36
35 37 #-----------------------------------------------------------------------------
36 38 # Code
37 39 #-----------------------------------------------------------------------------
38 40
39 41 # Template for the user message.
40 42 _default_message_template = """\
41 43 Oops, {app_name} crashed. We do our best to make it stable, but...
42 44
43 45 A crash report was automatically generated with the following information:
44 46 - A verbatim copy of the crash traceback.
45 47 - A copy of your input history during this session.
46 48 - Data on your current {app_name} configuration.
47 49
48 50 It was left in the file named:
49 51 \t'{crash_report_fname}'
50 52 If you can email this file to the developers, the information in it will help
51 53 them in understanding and correcting the problem.
52 54
53 55 You can mail it to: {contact_name} at {contact_email}
54 56 with the subject '{app_name} Crash Report'.
55 57
56 58 If you want to do it now, the following command will work (under Unix):
57 59 mail -s '{app_name} Crash Report' {contact_email} < {crash_report_fname}
58 60
59 61 In your email, please also include information about:
60 62 - The operating system under which the crash happened: Linux, macOS, Windows,
61 63 other, and which exact version (for example: Ubuntu 16.04.3, macOS 10.13.2,
62 64 Windows 10 Pro), and whether it is 32-bit or 64-bit;
63 65 - How {app_name} was installed: using pip or conda, from GitHub, as part of
64 66 a Docker container, or other, providing more detail if possible;
65 67 - How to reproduce the crash: what exact sequence of instructions can one
66 68 input to get the same crash? Ideally, find a minimal yet complete sequence
67 69 of instructions that yields the crash.
68 70
69 71 To ensure accurate tracking of this issue, please file a report about it at:
70 72 {bug_tracker}
71 73 """
72 74
73 75 _lite_message_template = """
74 76 If you suspect this is an IPython {version} bug, please report it at:
75 77 https://github.com/ipython/ipython/issues
76 78 or send an email to the mailing list at {email}
77 79
78 80 You can print a more detailed traceback right now with "%tb", or use "%debug"
79 81 to interactively debug it.
80 82
81 83 Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
82 84 {config}Application.verbose_crash=True
83 85 """
84 86
85 87
86 88 class CrashHandler(object):
87 89 """Customizable crash handlers for IPython applications.
88 90
89 91 Instances of this class provide a :meth:`__call__` method which can be
90 92 used as a ``sys.excepthook``. The :meth:`__call__` signature is::
91 93
92 94 def __call__(self, etype, evalue, etb)
93 95 """
94 96
95 97 message_template = _default_message_template
96 98 section_sep = '\n\n'+'*'*75+'\n\n'
97 99
98 def __init__(self, app, contact_name=None, contact_email=None,
99 bug_tracker=None, show_crash_traceback=True, call_pdb=False):
100 def __init__(
101 self,
102 app,
103 contact_name: Optional[str] = None,
104 contact_email: Optional[str] = None,
105 bug_tracker: Optional[str] = None,
106 show_crash_traceback: bool = True,
107 call_pdb: bool = False,
108 ):
100 109 """Create a new crash handler
101 110
102 111 Parameters
103 112 ----------
104 113 app : Application
105 114 A running :class:`Application` instance, which will be queried at
106 115 crash time for internal information.
107 116 contact_name : str
108 117 A string with the name of the person to contact.
109 118 contact_email : str
110 119 A string with the email address of the contact.
111 120 bug_tracker : str
112 121 A string with the URL for your project's bug tracker.
113 122 show_crash_traceback : bool
114 123 If false, don't print the crash traceback on stderr, only generate
115 124 the on-disk report
116 Non-argument instance attributes
125 call_pdb
126 Whether to call pdb on crash
127
128 Attributes
129 ----------
117 130 These instances contain some non-argument attributes which allow for
118 131 further customization of the crash handler's behavior. Please see the
119 132 source for further details.
133
120 134 """
121 135 self.crash_report_fname = "Crash_report_%s.txt" % app.name
122 136 self.app = app
123 137 self.call_pdb = call_pdb
124 138 #self.call_pdb = True # dbg
125 139 self.show_crash_traceback = show_crash_traceback
126 140 self.info = dict(app_name = app.name,
127 141 contact_name = contact_name,
128 142 contact_email = contact_email,
129 143 bug_tracker = bug_tracker,
130 144 crash_report_fname = self.crash_report_fname)
131 145
132 146
133 147 def __call__(self, etype, evalue, etb):
134 148 """Handle an exception, call for compatible with sys.excepthook"""
135 149
136 150 # do not allow the crash handler to be called twice without reinstalling it
137 151 # this prevents unlikely errors in the crash handling from entering an
138 152 # infinite loop.
139 153 sys.excepthook = sys.__excepthook__
140 154
141 155 # Report tracebacks shouldn't use color in general (safer for users)
142 156 color_scheme = 'NoColor'
143 157
144 158 # Use this ONLY for developer debugging (keep commented out for release)
145 159 #color_scheme = 'Linux' # dbg
146 160 try:
147 161 rptdir = self.app.ipython_dir
148 162 except:
149 163 rptdir = Path.cwd()
150 164 if rptdir is None or not Path.is_dir(rptdir):
151 165 rptdir = Path.cwd()
152 166 report_name = rptdir / self.crash_report_fname
153 167 # write the report filename into the instance dict so it can get
154 168 # properly expanded out in the user message template
155 169 self.crash_report_fname = report_name
156 170 self.info['crash_report_fname'] = report_name
157 171 TBhandler = ultratb.VerboseTB(
158 172 color_scheme=color_scheme,
159 173 long_header=1,
160 174 call_pdb=self.call_pdb,
161 175 )
162 176 if self.call_pdb:
163 177 TBhandler(etype,evalue,etb)
164 178 return
165 179 else:
166 180 traceback = TBhandler.text(etype,evalue,etb,context=31)
167 181
168 182 # print traceback to screen
169 183 if self.show_crash_traceback:
170 184 print(traceback, file=sys.stderr)
171 185
172 186 # and generate a complete report on disk
173 187 try:
174 188 report = open(report_name,'w')
175 189 except:
176 190 print('Could not create crash report on disk.', file=sys.stderr)
177 191 return
178 192
179 193 with report:
180 194 # Inform user on stderr of what happened
181 195 print('\n'+'*'*70+'\n', file=sys.stderr)
182 196 print(self.message_template.format(**self.info), file=sys.stderr)
183 197
184 198 # Construct report on disk
185 199 report.write(self.make_report(traceback))
186 200
187 201 input("Hit <Enter> to quit (your terminal may close):")
188 202
189 203 def make_report(self,traceback):
190 204 """Return a string containing a crash report."""
191 205
192 206 sec_sep = self.section_sep
193 207
194 208 report = ['*'*75+'\n\n'+'IPython post-mortem report\n\n']
195 209 rpt_add = report.append
196 210 rpt_add(sys_info())
197 211
198 212 try:
199 213 config = pformat(self.app.config)
200 214 rpt_add(sec_sep)
201 215 rpt_add('Application name: %s\n\n' % self.app_name)
202 216 rpt_add('Current user configuration structure:\n\n')
203 217 rpt_add(config)
204 218 except:
205 219 pass
206 220 rpt_add(sec_sep+'Crash traceback:\n\n' + traceback)
207 221
208 222 return ''.join(report)
209 223
210 224
211 225 def crash_handler_lite(etype, evalue, tb):
212 226 """a light excepthook, adding a small message to the usual traceback"""
213 227 traceback.print_exception(etype, evalue, tb)
214 228
215 229 from IPython.core.interactiveshell import InteractiveShell
216 230 if InteractiveShell.initialized():
217 231 # we are in a Shell environment, give %magic example
218 232 config = "%config "
219 233 else:
220 234 # we are not in a shell, show generic config
221 235 config = "c."
222 236 print(_lite_message_template.format(email=author_email, config=config, version=version), file=sys.stderr)
223 237
@@ -1,1003 +1,1000 b''
1 1 # -*- coding: utf-8 -*-
2 2 """
3 3 Pdb debugger class.
4 4
5 5
6 6 This is an extension to PDB which adds a number of new features.
7 7 Note that there is also the `IPython.terminal.debugger` class which provides UI
8 8 improvements.
9 9
10 10 We also strongly recommend to use this via the `ipdb` package, which provides
11 11 extra configuration options.
12 12
13 13 Among other things, this subclass of PDB:
14 14 - supports many IPython magics like pdef/psource
15 15 - hide frames in tracebacks based on `__tracebackhide__`
16 16 - allows to skip frames based on `__debuggerskip__`
17 17
18 18 The skipping and hiding frames are configurable via the `skip_predicates`
19 19 command.
20 20
21 21 By default, frames from readonly files will be hidden, frames containing
22 22 ``__tracebackhide__=True`` will be hidden.
23 23
24 24 Frames containing ``__debuggerskip__`` will be stepped over, frames who's parent
25 25 frames value of ``__debuggerskip__`` is ``True`` will be skipped.
26 26
27 27 >>> def helpers_helper():
28 28 ... pass
29 29 ...
30 30 ... def helper_1():
31 31 ... print("don't step in me")
32 32 ... helpers_helpers() # will be stepped over unless breakpoint set.
33 33 ...
34 34 ...
35 35 ... def helper_2():
36 36 ... print("in me neither")
37 37 ...
38 38
39 39 One can define a decorator that wraps a function between the two helpers:
40 40
41 41 >>> def pdb_skipped_decorator(function):
42 42 ...
43 43 ...
44 44 ... def wrapped_fn(*args, **kwargs):
45 45 ... __debuggerskip__ = True
46 46 ... helper_1()
47 47 ... __debuggerskip__ = False
48 48 ... result = function(*args, **kwargs)
49 49 ... __debuggerskip__ = True
50 50 ... helper_2()
51 51 ... # setting __debuggerskip__ to False again is not necessary
52 52 ... return result
53 53 ...
54 54 ... return wrapped_fn
55 55
56 56 When decorating a function, ipdb will directly step into ``bar()`` by
57 57 default:
58 58
59 59 >>> @foo_decorator
60 60 ... def bar(x, y):
61 61 ... return x * y
62 62
63 63
64 64 You can toggle the behavior with
65 65
66 66 ipdb> skip_predicates debuggerskip false
67 67
68 68 or configure it in your ``.pdbrc``
69 69
70 70
71 71
72 72 License
73 73 -------
74 74
75 75 Modified from the standard pdb.Pdb class to avoid including readline, so that
76 76 the command line completion of other programs which include this isn't
77 77 damaged.
78 78
79 79 In the future, this class will be expanded with improvements over the standard
80 80 pdb.
81 81
82 82 The original code in this file is mainly lifted out of cmd.py in Python 2.2,
83 83 with minor changes. Licensing should therefore be under the standard Python
84 84 terms. For details on the PSF (Python Software Foundation) standard license,
85 85 see:
86 86
87 87 https://docs.python.org/2/license.html
88 88
89 89
90 90 All the changes since then are under the same license as IPython.
91 91
92 92 """
93 93
94 94 #*****************************************************************************
95 95 #
96 96 # This file is licensed under the PSF license.
97 97 #
98 98 # Copyright (C) 2001 Python Software Foundation, www.python.org
99 99 # Copyright (C) 2005-2006 Fernando Perez. <fperez@colorado.edu>
100 100 #
101 101 #
102 102 #*****************************************************************************
103 103
104 104 import bdb
105 105 import inspect
106 106 import linecache
107 107 import sys
108 108 import warnings
109 109 import re
110 110 import os
111 111
112 112 from IPython import get_ipython
113 113 from IPython.utils import PyColorize
114 114 from IPython.utils import coloransi, py3compat
115 115 from IPython.core.excolors import exception_colors
116 116
117 117 # skip module docstests
118 118 __skip_doctest__ = True
119 119
120 120 prompt = 'ipdb> '
121 121
122 122 # We have to check this directly from sys.argv, config struct not yet available
123 123 from pdb import Pdb as OldPdb
124 124
125 125 # Allow the set_trace code to operate outside of an ipython instance, even if
126 126 # it does so with some limitations. The rest of this support is implemented in
127 127 # the Tracer constructor.
128 128
129 129 DEBUGGERSKIP = "__debuggerskip__"
130 130
131 131
132 132 def make_arrow(pad):
133 133 """generate the leading arrow in front of traceback or debugger"""
134 134 if pad >= 2:
135 135 return '-'*(pad-2) + '> '
136 136 elif pad == 1:
137 137 return '>'
138 138 return ''
139 139
140 140
141 141 def BdbQuit_excepthook(et, ev, tb, excepthook=None):
142 142 """Exception hook which handles `BdbQuit` exceptions.
143 143
144 144 All other exceptions are processed using the `excepthook`
145 145 parameter.
146 146 """
147 147 raise ValueError(
148 148 "`BdbQuit_excepthook` is deprecated since version 5.1",
149 149 )
150 150
151 151
152 152 def BdbQuit_IPython_excepthook(self, et, ev, tb, tb_offset=None):
153 153 raise ValueError(
154 154 "`BdbQuit_IPython_excepthook` is deprecated since version 5.1",
155 155 DeprecationWarning, stacklevel=2)
156 156
157 157
158 158 RGX_EXTRA_INDENT = re.compile(r'(?<=\n)\s+')
159 159
160 160
161 161 def strip_indentation(multiline_string):
162 162 return RGX_EXTRA_INDENT.sub('', multiline_string)
163 163
164 164
165 165 def decorate_fn_with_doc(new_fn, old_fn, additional_text=""):
166 166 """Make new_fn have old_fn's doc string. This is particularly useful
167 167 for the ``do_...`` commands that hook into the help system.
168 168 Adapted from from a comp.lang.python posting
169 169 by Duncan Booth."""
170 170 def wrapper(*args, **kw):
171 171 return new_fn(*args, **kw)
172 172 if old_fn.__doc__:
173 173 wrapper.__doc__ = strip_indentation(old_fn.__doc__) + additional_text
174 174 return wrapper
175 175
176 176
177 177 class Pdb(OldPdb):
178 178 """Modified Pdb class, does not load readline.
179 179
180 180 for a standalone version that uses prompt_toolkit, see
181 181 `IPython.terminal.debugger.TerminalPdb` and
182 182 `IPython.terminal.debugger.set_trace()`
183 183
184 184
185 185 This debugger can hide and skip frames that are tagged according to some predicates.
186 186 See the `skip_predicates` commands.
187 187
188 188 """
189 189
190 190 default_predicates = {
191 191 "tbhide": True,
192 192 "readonly": False,
193 193 "ipython_internal": True,
194 194 "debuggerskip": True,
195 195 }
196 196
197 197 def __init__(self, completekey=None, stdin=None, stdout=None, context=5, **kwargs):
198 198 """Create a new IPython debugger.
199 199
200 200 Parameters
201 201 ----------
202 202 completekey : default None
203 203 Passed to pdb.Pdb.
204 204 stdin : default None
205 205 Passed to pdb.Pdb.
206 206 stdout : default None
207 207 Passed to pdb.Pdb.
208 208 context : int
209 209 Number of lines of source code context to show when
210 210 displaying stacktrace information.
211 211 **kwargs
212 212 Passed to pdb.Pdb.
213 213
214 214 Notes
215 215 -----
216 216 The possibilities are python version dependent, see the python
217 217 docs for more info.
218 218 """
219 219
220 220 # Parent constructor:
221 221 try:
222 222 self.context = int(context)
223 223 if self.context <= 0:
224 224 raise ValueError("Context must be a positive integer")
225 225 except (TypeError, ValueError) as e:
226 226 raise ValueError("Context must be a positive integer") from e
227 227
228 228 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
229 229 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
230 230
231 231 # IPython changes...
232 232 self.shell = get_ipython()
233 233
234 234 if self.shell is None:
235 235 save_main = sys.modules['__main__']
236 236 # No IPython instance running, we must create one
237 237 from IPython.terminal.interactiveshell import \
238 238 TerminalInteractiveShell
239 239 self.shell = TerminalInteractiveShell.instance()
240 240 # needed by any code which calls __import__("__main__") after
241 241 # the debugger was entered. See also #9941.
242 242 sys.modules["__main__"] = save_main
243 243
244 244
245 245 color_scheme = self.shell.colors
246 246
247 247 self.aliases = {}
248 248
249 249 # Create color table: we copy the default one from the traceback
250 250 # module and add a few attributes needed for debugging
251 251 self.color_scheme_table = exception_colors()
252 252
253 253 # shorthands
254 254 C = coloransi.TermColors
255 255 cst = self.color_scheme_table
256 256
257 257 cst['NoColor'].colors.prompt = C.NoColor
258 258 cst['NoColor'].colors.breakpoint_enabled = C.NoColor
259 259 cst['NoColor'].colors.breakpoint_disabled = C.NoColor
260 260
261 261 cst['Linux'].colors.prompt = C.Green
262 262 cst['Linux'].colors.breakpoint_enabled = C.LightRed
263 263 cst['Linux'].colors.breakpoint_disabled = C.Red
264 264
265 265 cst['LightBG'].colors.prompt = C.Blue
266 266 cst['LightBG'].colors.breakpoint_enabled = C.LightRed
267 267 cst['LightBG'].colors.breakpoint_disabled = C.Red
268 268
269 269 cst['Neutral'].colors.prompt = C.Blue
270 270 cst['Neutral'].colors.breakpoint_enabled = C.LightRed
271 271 cst['Neutral'].colors.breakpoint_disabled = C.Red
272 272
273 273 # Add a python parser so we can syntax highlight source while
274 274 # debugging.
275 275 self.parser = PyColorize.Parser(style=color_scheme)
276 276 self.set_colors(color_scheme)
277 277
278 278 # Set the prompt - the default prompt is '(Pdb)'
279 279 self.prompt = prompt
280 280 self.skip_hidden = True
281 281 self.report_skipped = True
282 282
283 283 # list of predicates we use to skip frames
284 284 self._predicates = self.default_predicates
285 285
286 286 #
287 287 def set_colors(self, scheme):
288 288 """Shorthand access to the color table scheme selector method."""
289 289 self.color_scheme_table.set_active_scheme(scheme)
290 290 self.parser.style = scheme
291 291
292 292 def set_trace(self, frame=None):
293 293 if frame is None:
294 294 frame = sys._getframe().f_back
295 295 self.initial_frame = frame
296 296 return super().set_trace(frame)
297 297
298 298 def _hidden_predicate(self, frame):
299 299 """
300 300 Given a frame return whether it it should be hidden or not by IPython.
301 301 """
302 302
303 303 if self._predicates["readonly"]:
304 304 fname = frame.f_code.co_filename
305 305 # we need to check for file existence and interactively define
306 306 # function would otherwise appear as RO.
307 307 if os.path.isfile(fname) and not os.access(fname, os.W_OK):
308 308 return True
309 309
310 310 if self._predicates["tbhide"]:
311 311 if frame in (self.curframe, getattr(self, "initial_frame", None)):
312 312 return False
313 313 frame_locals = self._get_frame_locals(frame)
314 314 if "__tracebackhide__" not in frame_locals:
315 315 return False
316 316 return frame_locals["__tracebackhide__"]
317 317 return False
318 318
319 319 def hidden_frames(self, stack):
320 320 """
321 321 Given an index in the stack return whether it should be skipped.
322 322
323 323 This is used in up/down and where to skip frames.
324 324 """
325 325 # The f_locals dictionary is updated from the actual frame
326 326 # locals whenever the .f_locals accessor is called, so we
327 327 # avoid calling it here to preserve self.curframe_locals.
328 328 # Furthermore, there is no good reason to hide the current frame.
329 329 ip_hide = [self._hidden_predicate(s[0]) for s in stack]
330 330 ip_start = [i for i, s in enumerate(ip_hide) if s == "__ipython_bottom__"]
331 331 if ip_start and self._predicates["ipython_internal"]:
332 332 ip_hide = [h if i > ip_start[0] else True for (i, h) in enumerate(ip_hide)]
333 333 return ip_hide
334 334
335 335 def interaction(self, frame, traceback):
336 336 try:
337 337 OldPdb.interaction(self, frame, traceback)
338 338 except KeyboardInterrupt:
339 339 self.stdout.write("\n" + self.shell.get_exception_only())
340 340
341 341 def precmd(self, line):
342 342 """Perform useful escapes on the command before it is executed."""
343 343
344 344 if line.endswith("??"):
345 345 line = "pinfo2 " + line[:-2]
346 346 elif line.endswith("?"):
347 347 line = "pinfo " + line[:-1]
348 348
349 349 line = super().precmd(line)
350 350
351 351 return line
352 352
353 353 def new_do_frame(self, arg):
354 354 OldPdb.do_frame(self, arg)
355 355
356 356 def new_do_quit(self, arg):
357 357
358 358 if hasattr(self, 'old_all_completions'):
359 359 self.shell.Completer.all_completions = self.old_all_completions
360 360
361 361 return OldPdb.do_quit(self, arg)
362 362
363 363 do_q = do_quit = decorate_fn_with_doc(new_do_quit, OldPdb.do_quit)
364 364
365 365 def new_do_restart(self, arg):
366 366 """Restart command. In the context of ipython this is exactly the same
367 367 thing as 'quit'."""
368 368 self.msg("Restart doesn't make sense here. Using 'quit' instead.")
369 369 return self.do_quit(arg)
370 370
371 371 def print_stack_trace(self, context=None):
372 372 Colors = self.color_scheme_table.active_colors
373 373 ColorsNormal = Colors.Normal
374 374 if context is None:
375 375 context = self.context
376 376 try:
377 377 context = int(context)
378 378 if context <= 0:
379 379 raise ValueError("Context must be a positive integer")
380 380 except (TypeError, ValueError) as e:
381 381 raise ValueError("Context must be a positive integer") from e
382 382 try:
383 383 skipped = 0
384 384 for hidden, frame_lineno in zip(self.hidden_frames(self.stack), self.stack):
385 385 if hidden and self.skip_hidden:
386 386 skipped += 1
387 387 continue
388 388 if skipped:
389 389 print(
390 390 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
391 391 )
392 392 skipped = 0
393 393 self.print_stack_entry(frame_lineno, context=context)
394 394 if skipped:
395 395 print(
396 396 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
397 397 )
398 398 except KeyboardInterrupt:
399 399 pass
400 400
401 401 def print_stack_entry(self, frame_lineno, prompt_prefix='\n-> ',
402 402 context=None):
403 403 if context is None:
404 404 context = self.context
405 405 try:
406 406 context = int(context)
407 407 if context <= 0:
408 408 raise ValueError("Context must be a positive integer")
409 409 except (TypeError, ValueError) as e:
410 410 raise ValueError("Context must be a positive integer") from e
411 411 print(self.format_stack_entry(frame_lineno, '', context), file=self.stdout)
412 412
413 413 # vds: >>
414 414 frame, lineno = frame_lineno
415 415 filename = frame.f_code.co_filename
416 416 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
417 417 # vds: <<
418 418
419 419 def _get_frame_locals(self, frame):
420 420 """ "
421 421 Accessing f_local of current frame reset the namespace, so we want to avoid
422 422 that or the following can happen
423 423
424 424 ipdb> foo
425 425 "old"
426 426 ipdb> foo = "new"
427 427 ipdb> foo
428 428 "new"
429 429 ipdb> where
430 430 ipdb> foo
431 431 "old"
432 432
433 433 So if frame is self.current_frame we instead return self.curframe_locals
434 434
435 435 """
436 436 if frame is self.curframe:
437 437 return self.curframe_locals
438 438 else:
439 439 return frame.f_locals
440 440
441 441 def format_stack_entry(self, frame_lineno, lprefix=': ', context=None):
442 442 if context is None:
443 443 context = self.context
444 444 try:
445 445 context = int(context)
446 446 if context <= 0:
447 447 print("Context must be a positive integer", file=self.stdout)
448 448 except (TypeError, ValueError):
449 449 print("Context must be a positive integer", file=self.stdout)
450 450
451 451 import reprlib
452 452
453 453 ret = []
454 454
455 455 Colors = self.color_scheme_table.active_colors
456 456 ColorsNormal = Colors.Normal
457 457 tpl_link = "%s%%s%s" % (Colors.filenameEm, ColorsNormal)
458 458 tpl_call = "%s%%s%s%%s%s" % (Colors.vName, Colors.valEm, ColorsNormal)
459 459 tpl_line = "%%s%s%%s %s%%s" % (Colors.lineno, ColorsNormal)
460 460 tpl_line_em = "%%s%s%%s %s%%s%s" % (Colors.linenoEm, Colors.line, ColorsNormal)
461 461
462 462 frame, lineno = frame_lineno
463 463
464 464 return_value = ''
465 465 loc_frame = self._get_frame_locals(frame)
466 466 if "__return__" in loc_frame:
467 467 rv = loc_frame["__return__"]
468 468 # return_value += '->'
469 469 return_value += reprlib.repr(rv) + "\n"
470 470 ret.append(return_value)
471 471
472 472 #s = filename + '(' + `lineno` + ')'
473 473 filename = self.canonic(frame.f_code.co_filename)
474 474 link = tpl_link % py3compat.cast_unicode(filename)
475 475
476 476 if frame.f_code.co_name:
477 477 func = frame.f_code.co_name
478 478 else:
479 479 func = "<lambda>"
480 480
481 481 call = ""
482 482 if func != "?":
483 483 if "__args__" in loc_frame:
484 484 args = reprlib.repr(loc_frame["__args__"])
485 485 else:
486 486 args = '()'
487 487 call = tpl_call % (func, args)
488 488
489 489 # The level info should be generated in the same format pdb uses, to
490 490 # avoid breaking the pdbtrack functionality of python-mode in *emacs.
491 491 if frame is self.curframe:
492 492 ret.append('> ')
493 493 else:
494 494 ret.append(" ")
495 495 ret.append("%s(%s)%s\n" % (link, lineno, call))
496 496
497 497 start = lineno - 1 - context//2
498 498 lines = linecache.getlines(filename)
499 499 start = min(start, len(lines) - context)
500 500 start = max(start, 0)
501 501 lines = lines[start : start + context]
502 502
503 503 for i, line in enumerate(lines):
504 504 show_arrow = start + 1 + i == lineno
505 505 linetpl = (frame is self.curframe or show_arrow) and tpl_line_em or tpl_line
506 506 ret.append(
507 507 self.__format_line(
508 508 linetpl, filename, start + 1 + i, line, arrow=show_arrow
509 509 )
510 510 )
511 511 return "".join(ret)
512 512
513 513 def __format_line(self, tpl_line, filename, lineno, line, arrow=False):
514 514 bp_mark = ""
515 515 bp_mark_color = ""
516 516
517 517 new_line, err = self.parser.format2(line, 'str')
518 518 if not err:
519 519 line = new_line
520 520
521 521 bp = None
522 522 if lineno in self.get_file_breaks(filename):
523 523 bps = self.get_breaks(filename, lineno)
524 524 bp = bps[-1]
525 525
526 526 if bp:
527 527 Colors = self.color_scheme_table.active_colors
528 528 bp_mark = str(bp.number)
529 529 bp_mark_color = Colors.breakpoint_enabled
530 530 if not bp.enabled:
531 531 bp_mark_color = Colors.breakpoint_disabled
532 532
533 533 numbers_width = 7
534 534 if arrow:
535 535 # This is the line with the error
536 536 pad = numbers_width - len(str(lineno)) - len(bp_mark)
537 537 num = '%s%s' % (make_arrow(pad), str(lineno))
538 538 else:
539 539 num = '%*s' % (numbers_width - len(bp_mark), str(lineno))
540 540
541 541 return tpl_line % (bp_mark_color + bp_mark, num, line)
542 542
543 543 def print_list_lines(self, filename, first, last):
544 544 """The printing (as opposed to the parsing part of a 'list'
545 545 command."""
546 546 try:
547 547 Colors = self.color_scheme_table.active_colors
548 548 ColorsNormal = Colors.Normal
549 549 tpl_line = '%%s%s%%s %s%%s' % (Colors.lineno, ColorsNormal)
550 550 tpl_line_em = '%%s%s%%s %s%%s%s' % (Colors.linenoEm, Colors.line, ColorsNormal)
551 551 src = []
552 552 if filename == "<string>" and hasattr(self, "_exec_filename"):
553 553 filename = self._exec_filename
554 554
555 555 for lineno in range(first, last+1):
556 556 line = linecache.getline(filename, lineno)
557 557 if not line:
558 558 break
559 559
560 560 if lineno == self.curframe.f_lineno:
561 561 line = self.__format_line(
562 562 tpl_line_em, filename, lineno, line, arrow=True
563 563 )
564 564 else:
565 565 line = self.__format_line(
566 566 tpl_line, filename, lineno, line, arrow=False
567 567 )
568 568
569 569 src.append(line)
570 570 self.lineno = lineno
571 571
572 572 print(''.join(src), file=self.stdout)
573 573
574 574 except KeyboardInterrupt:
575 575 pass
576 576
577 577 def do_skip_predicates(self, args):
578 578 """
579 579 Turn on/off individual predicates as to whether a frame should be hidden/skip.
580 580
581 581 The global option to skip (or not) hidden frames is set with skip_hidden
582 582
583 583 To change the value of a predicate
584 584
585 585 skip_predicates key [true|false]
586 586
587 587 Call without arguments to see the current values.
588 588
589 589 To permanently change the value of an option add the corresponding
590 590 command to your ``~/.pdbrc`` file. If you are programmatically using the
591 591 Pdb instance you can also change the ``default_predicates`` class
592 592 attribute.
593 593 """
594 594 if not args.strip():
595 595 print("current predicates:")
596 596 for (p, v) in self._predicates.items():
597 597 print(" ", p, ":", v)
598 598 return
599 599 type_value = args.strip().split(" ")
600 600 if len(type_value) != 2:
601 601 print(
602 602 f"Usage: skip_predicates <type> <value>, with <type> one of {set(self._predicates.keys())}"
603 603 )
604 604 return
605 605
606 606 type_, value = type_value
607 607 if type_ not in self._predicates:
608 608 print(f"{type_!r} not in {set(self._predicates.keys())}")
609 609 return
610 610 if value.lower() not in ("true", "yes", "1", "no", "false", "0"):
611 611 print(
612 612 f"{value!r} is invalid - use one of ('true', 'yes', '1', 'no', 'false', '0')"
613 613 )
614 614 return
615 615
616 616 self._predicates[type_] = value.lower() in ("true", "yes", "1")
617 617 if not any(self._predicates.values()):
618 618 print(
619 619 "Warning, all predicates set to False, skip_hidden may not have any effects."
620 620 )
621 621
622 622 def do_skip_hidden(self, arg):
623 623 """
624 624 Change whether or not we should skip frames with the
625 625 __tracebackhide__ attribute.
626 626 """
627 627 if not arg.strip():
628 628 print(
629 629 f"skip_hidden = {self.skip_hidden}, use 'yes','no', 'true', or 'false' to change."
630 630 )
631 631 elif arg.strip().lower() in ("true", "yes"):
632 632 self.skip_hidden = True
633 633 elif arg.strip().lower() in ("false", "no"):
634 634 self.skip_hidden = False
635 635 if not any(self._predicates.values()):
636 636 print(
637 637 "Warning, all predicates set to False, skip_hidden may not have any effects."
638 638 )
639 639
640 640 def do_list(self, arg):
641 641 """Print lines of code from the current stack frame
642 642 """
643 643 self.lastcmd = 'list'
644 644 last = None
645 645 if arg:
646 646 try:
647 647 x = eval(arg, {}, {})
648 648 if type(x) == type(()):
649 649 first, last = x
650 650 first = int(first)
651 651 last = int(last)
652 652 if last < first:
653 653 # Assume it's a count
654 654 last = first + last
655 655 else:
656 656 first = max(1, int(x) - 5)
657 657 except:
658 658 print('*** Error in argument:', repr(arg), file=self.stdout)
659 659 return
660 660 elif self.lineno is None:
661 661 first = max(1, self.curframe.f_lineno - 5)
662 662 else:
663 663 first = self.lineno + 1
664 664 if last is None:
665 665 last = first + 10
666 666 self.print_list_lines(self.curframe.f_code.co_filename, first, last)
667 667
668 668 # vds: >>
669 669 lineno = first
670 670 filename = self.curframe.f_code.co_filename
671 671 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
672 672 # vds: <<
673 673
674 674 do_l = do_list
675 675
676 676 def getsourcelines(self, obj):
677 677 lines, lineno = inspect.findsource(obj)
678 678 if inspect.isframe(obj) and obj.f_globals is self._get_frame_locals(obj):
679 679 # must be a module frame: do not try to cut a block out of it
680 680 return lines, 1
681 681 elif inspect.ismodule(obj):
682 682 return lines, 1
683 683 return inspect.getblock(lines[lineno:]), lineno+1
684 684
685 685 def do_longlist(self, arg):
686 686 """Print lines of code from the current stack frame.
687 687
688 688 Shows more lines than 'list' does.
689 689 """
690 690 self.lastcmd = 'longlist'
691 691 try:
692 692 lines, lineno = self.getsourcelines(self.curframe)
693 693 except OSError as err:
694 694 self.error(err)
695 695 return
696 696 last = lineno + len(lines)
697 697 self.print_list_lines(self.curframe.f_code.co_filename, lineno, last)
698 698 do_ll = do_longlist
699 699
700 700 def do_debug(self, arg):
701 701 """debug code
702 702 Enter a recursive debugger that steps through the code
703 703 argument (which is an arbitrary expression or statement to be
704 704 executed in the current environment).
705 705 """
706 706 trace_function = sys.gettrace()
707 707 sys.settrace(None)
708 708 globals = self.curframe.f_globals
709 709 locals = self.curframe_locals
710 710 p = self.__class__(completekey=self.completekey,
711 711 stdin=self.stdin, stdout=self.stdout)
712 712 p.use_rawinput = self.use_rawinput
713 713 p.prompt = "(%s) " % self.prompt.strip()
714 714 self.message("ENTERING RECURSIVE DEBUGGER")
715 715 sys.call_tracing(p.run, (arg, globals, locals))
716 716 self.message("LEAVING RECURSIVE DEBUGGER")
717 717 sys.settrace(trace_function)
718 718 self.lastcmd = p.lastcmd
719 719
720 720 def do_pdef(self, arg):
721 721 """Print the call signature for any callable object.
722 722
723 723 The debugger interface to %pdef"""
724 724 namespaces = [
725 725 ("Locals", self.curframe_locals),
726 726 ("Globals", self.curframe.f_globals),
727 727 ]
728 728 self.shell.find_line_magic("pdef")(arg, namespaces=namespaces)
729 729
730 730 def do_pdoc(self, arg):
731 731 """Print the docstring for an object.
732 732
733 733 The debugger interface to %pdoc."""
734 734 namespaces = [
735 735 ("Locals", self.curframe_locals),
736 736 ("Globals", self.curframe.f_globals),
737 737 ]
738 738 self.shell.find_line_magic("pdoc")(arg, namespaces=namespaces)
739 739
740 740 def do_pfile(self, arg):
741 741 """Print (or run through pager) the file where an object is defined.
742 742
743 743 The debugger interface to %pfile.
744 744 """
745 745 namespaces = [
746 746 ("Locals", self.curframe_locals),
747 747 ("Globals", self.curframe.f_globals),
748 748 ]
749 749 self.shell.find_line_magic("pfile")(arg, namespaces=namespaces)
750 750
751 751 def do_pinfo(self, arg):
752 752 """Provide detailed information about an object.
753 753
754 754 The debugger interface to %pinfo, i.e., obj?."""
755 755 namespaces = [
756 756 ("Locals", self.curframe_locals),
757 757 ("Globals", self.curframe.f_globals),
758 758 ]
759 759 self.shell.find_line_magic("pinfo")(arg, namespaces=namespaces)
760 760
761 761 def do_pinfo2(self, arg):
762 762 """Provide extra detailed information about an object.
763 763
764 764 The debugger interface to %pinfo2, i.e., obj??."""
765 765 namespaces = [
766 766 ("Locals", self.curframe_locals),
767 767 ("Globals", self.curframe.f_globals),
768 768 ]
769 769 self.shell.find_line_magic("pinfo2")(arg, namespaces=namespaces)
770 770
771 771 def do_psource(self, arg):
772 772 """Print (or run through pager) the source code for an object."""
773 773 namespaces = [
774 774 ("Locals", self.curframe_locals),
775 775 ("Globals", self.curframe.f_globals),
776 776 ]
777 777 self.shell.find_line_magic("psource")(arg, namespaces=namespaces)
778 778
779 779 def do_where(self, arg):
780 780 """w(here)
781 781 Print a stack trace, with the most recent frame at the bottom.
782 782 An arrow indicates the "current frame", which determines the
783 783 context of most commands. 'bt' is an alias for this command.
784 784
785 785 Take a number as argument as an (optional) number of context line to
786 786 print"""
787 787 if arg:
788 788 try:
789 789 context = int(arg)
790 790 except ValueError as err:
791 791 self.error(err)
792 792 return
793 793 self.print_stack_trace(context)
794 794 else:
795 795 self.print_stack_trace()
796 796
797 797 do_w = do_where
798 798
799 799 def break_anywhere(self, frame):
800 800 """
801
802 801 _stop_in_decorator_internals is overly restrictive, as we may still want
803 802 to trace function calls, so we need to also update break_anywhere so
804 803 that is we don't `stop_here`, because of debugger skip, we may still
805 804 stop at any point inside the function
806 805
807 806 """
808 807
809 808 sup = super().break_anywhere(frame)
810 809 if sup:
811 810 return sup
812 811 if self._predicates["debuggerskip"]:
813 812 if DEBUGGERSKIP in frame.f_code.co_varnames:
814 813 return True
815 814 if frame.f_back and self._get_frame_locals(frame.f_back).get(DEBUGGERSKIP):
816 815 return True
817 816 return False
818 817
819 818 def _is_in_decorator_internal_and_should_skip(self, frame):
820 819 """
821 820 Utility to tell us whether we are in a decorator internal and should stop.
822 821
823
824
825 822 """
826 823
827 824 # if we are disabled don't skip
828 825 if not self._predicates["debuggerskip"]:
829 826 return False
830 827
831 828 # if frame is tagged, skip by default.
832 829 if DEBUGGERSKIP in frame.f_code.co_varnames:
833 830 return True
834 831
835 832 # if one of the parent frame value set to True skip as well.
836 833
837 834 cframe = frame
838 835 while getattr(cframe, "f_back", None):
839 836 cframe = cframe.f_back
840 837 if self._get_frame_locals(cframe).get(DEBUGGERSKIP):
841 838 return True
842 839
843 840 return False
844 841
845 842 def stop_here(self, frame):
846 843
847 844 if self._is_in_decorator_internal_and_should_skip(frame) is True:
848 845 return False
849 846
850 847 hidden = False
851 848 if self.skip_hidden:
852 849 hidden = self._hidden_predicate(frame)
853 850 if hidden:
854 851 if self.report_skipped:
855 852 Colors = self.color_scheme_table.active_colors
856 853 ColorsNormal = Colors.Normal
857 854 print(
858 855 f"{Colors.excName} [... skipped 1 hidden frame]{ColorsNormal}\n"
859 856 )
860 857 return super().stop_here(frame)
861 858
862 859 def do_up(self, arg):
863 860 """u(p) [count]
864 861 Move the current frame count (default one) levels up in the
865 862 stack trace (to an older frame).
866 863
867 864 Will skip hidden frames.
868 865 """
869 866 # modified version of upstream that skips
870 867 # frames with __tracebackhide__
871 868 if self.curindex == 0:
872 869 self.error("Oldest frame")
873 870 return
874 871 try:
875 872 count = int(arg or 1)
876 873 except ValueError:
877 874 self.error("Invalid frame count (%s)" % arg)
878 875 return
879 876 skipped = 0
880 877 if count < 0:
881 878 _newframe = 0
882 879 else:
883 880 counter = 0
884 881 hidden_frames = self.hidden_frames(self.stack)
885 882 for i in range(self.curindex - 1, -1, -1):
886 883 if hidden_frames[i] and self.skip_hidden:
887 884 skipped += 1
888 885 continue
889 886 counter += 1
890 887 if counter >= count:
891 888 break
892 889 else:
893 890 # if no break occurred.
894 891 self.error(
895 892 "all frames above hidden, use `skip_hidden False` to get get into those."
896 893 )
897 894 return
898 895
899 896 Colors = self.color_scheme_table.active_colors
900 897 ColorsNormal = Colors.Normal
901 898 _newframe = i
902 899 self._select_frame(_newframe)
903 900 if skipped:
904 901 print(
905 902 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
906 903 )
907 904
908 905 def do_down(self, arg):
909 906 """d(own) [count]
910 907 Move the current frame count (default one) levels down in the
911 908 stack trace (to a newer frame).
912 909
913 910 Will skip hidden frames.
914 911 """
915 912 if self.curindex + 1 == len(self.stack):
916 913 self.error("Newest frame")
917 914 return
918 915 try:
919 916 count = int(arg or 1)
920 917 except ValueError:
921 918 self.error("Invalid frame count (%s)" % arg)
922 919 return
923 920 if count < 0:
924 921 _newframe = len(self.stack) - 1
925 922 else:
926 923 counter = 0
927 924 skipped = 0
928 925 hidden_frames = self.hidden_frames(self.stack)
929 926 for i in range(self.curindex + 1, len(self.stack)):
930 927 if hidden_frames[i] and self.skip_hidden:
931 928 skipped += 1
932 929 continue
933 930 counter += 1
934 931 if counter >= count:
935 932 break
936 933 else:
937 934 self.error(
938 935 "all frames below hidden, use `skip_hidden False` to get get into those."
939 936 )
940 937 return
941 938
942 939 Colors = self.color_scheme_table.active_colors
943 940 ColorsNormal = Colors.Normal
944 941 if skipped:
945 942 print(
946 943 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
947 944 )
948 945 _newframe = i
949 946
950 947 self._select_frame(_newframe)
951 948
952 949 do_d = do_down
953 950 do_u = do_up
954 951
955 952 def do_context(self, context):
956 953 """context number_of_lines
957 954 Set the number of lines of source code to show when displaying
958 955 stacktrace information.
959 956 """
960 957 try:
961 958 new_context = int(context)
962 959 if new_context <= 0:
963 960 raise ValueError()
964 961 self.context = new_context
965 962 except ValueError:
966 963 self.error("The 'context' command requires a positive integer argument.")
967 964
968 965
969 966 class InterruptiblePdb(Pdb):
970 967 """Version of debugger where KeyboardInterrupt exits the debugger altogether."""
971 968
972 969 def cmdloop(self, intro=None):
973 970 """Wrap cmdloop() such that KeyboardInterrupt stops the debugger."""
974 971 try:
975 972 return OldPdb.cmdloop(self, intro=intro)
976 973 except KeyboardInterrupt:
977 974 self.stop_here = lambda frame: False
978 975 self.do_quit("")
979 976 sys.settrace(None)
980 977 self.quitting = False
981 978 raise
982 979
983 980 def _cmdloop(self):
984 981 while True:
985 982 try:
986 983 # keyboard interrupts allow for an easy way to cancel
987 984 # the current command, so allow them during interactive input
988 985 self.allow_kbdint = True
989 986 self.cmdloop()
990 987 self.allow_kbdint = False
991 988 break
992 989 except KeyboardInterrupt:
993 990 self.message('--KeyboardInterrupt--')
994 991 raise
995 992
996 993
997 994 def set_trace(frame=None):
998 995 """
999 996 Start debugging from `frame`.
1000 997
1001 998 If frame is not specified, debugging starts from caller's frame.
1002 999 """
1003 1000 Pdb().set_trace(frame or sys._getframe().f_back)
@@ -1,1256 +1,1272 b''
1 1 # -*- coding: utf-8 -*-
2 2 """Top-level display functions for displaying object in different formats."""
3 3
4 4 # Copyright (c) IPython Development Team.
5 5 # Distributed under the terms of the Modified BSD License.
6 6
7 7
8 8 from binascii import b2a_base64, hexlify
9 9 import html
10 10 import json
11 11 import mimetypes
12 12 import os
13 13 import struct
14 14 import warnings
15 15 from copy import deepcopy
16 16 from os.path import splitext
17 17 from pathlib import Path, PurePath
18 18
19 19 from IPython.utils.py3compat import cast_unicode
20 20 from IPython.testing.skipdoctest import skip_doctest
21 21 from . import display_functions
22 22
23 23
24 24 __all__ = ['display_pretty', 'display_html', 'display_markdown',
25 25 'display_svg', 'display_png', 'display_jpeg', 'display_latex', 'display_json',
26 26 'display_javascript', 'display_pdf', 'DisplayObject', 'TextDisplayObject',
27 27 'Pretty', 'HTML', 'Markdown', 'Math', 'Latex', 'SVG', 'ProgressBar', 'JSON',
28 28 'GeoJSON', 'Javascript', 'Image', 'set_matplotlib_formats',
29 29 'set_matplotlib_close',
30 30 'Video']
31 31
32 32 _deprecated_names = ["display", "clear_output", "publish_display_data", "update_display", "DisplayHandle"]
33 33
34 34 __all__ = __all__ + _deprecated_names
35 35
36 36
37 37 # ----- warn to import from IPython.display -----
38 38
39 39 from warnings import warn
40 40
41 41
42 42 def __getattr__(name):
43 43 if name in _deprecated_names:
44 44 warn(f"Importing {name} from IPython.core.display is deprecated since IPython 7.14, please import from IPython display", DeprecationWarning, stacklevel=2)
45 45 return getattr(display_functions, name)
46 46
47 47 if name in globals().keys():
48 48 return globals()[name]
49 49 else:
50 50 raise AttributeError(f"module {__name__} has no attribute {name}")
51 51
52 52
53 53 #-----------------------------------------------------------------------------
54 54 # utility functions
55 55 #-----------------------------------------------------------------------------
56 56
57 57 def _safe_exists(path):
58 58 """Check path, but don't let exceptions raise"""
59 59 try:
60 60 return os.path.exists(path)
61 61 except Exception:
62 62 return False
63 63
64 64
65 65 def _display_mimetype(mimetype, objs, raw=False, metadata=None):
66 66 """internal implementation of all display_foo methods
67 67
68 68 Parameters
69 69 ----------
70 70 mimetype : str
71 71 The mimetype to be published (e.g. 'image/png')
72 72 *objs : object
73 73 The Python objects to display, or if raw=True raw text data to
74 74 display.
75 75 raw : bool
76 76 Are the data objects raw data or Python objects that need to be
77 77 formatted before display? [default: False]
78 78 metadata : dict (optional)
79 79 Metadata to be associated with the specific mimetype output.
80 80 """
81 81 if metadata:
82 82 metadata = {mimetype: metadata}
83 83 if raw:
84 84 # turn list of pngdata into list of { 'image/png': pngdata }
85 85 objs = [ {mimetype: obj} for obj in objs ]
86 86 display(*objs, raw=raw, metadata=metadata, include=[mimetype])
87 87
88 88 #-----------------------------------------------------------------------------
89 89 # Main functions
90 90 #-----------------------------------------------------------------------------
91 91
92 92
93 93 def display_pretty(*objs, **kwargs):
94 94 """Display the pretty (default) representation of an object.
95 95
96 96 Parameters
97 97 ----------
98 98 *objs : object
99 99 The Python objects to display, or if raw=True raw text data to
100 100 display.
101 101 raw : bool
102 102 Are the data objects raw data or Python objects that need to be
103 103 formatted before display? [default: False]
104 104 metadata : dict (optional)
105 105 Metadata to be associated with the specific mimetype output.
106 106 """
107 107 _display_mimetype('text/plain', objs, **kwargs)
108 108
109 109
110 110 def display_html(*objs, **kwargs):
111 111 """Display the HTML representation of an object.
112 112
113 113 Note: If raw=False and the object does not have a HTML
114 114 representation, no HTML will be shown.
115 115
116 116 Parameters
117 117 ----------
118 118 *objs : object
119 119 The Python objects to display, or if raw=True raw HTML data to
120 120 display.
121 121 raw : bool
122 122 Are the data objects raw data or Python objects that need to be
123 123 formatted before display? [default: False]
124 124 metadata : dict (optional)
125 125 Metadata to be associated with the specific mimetype output.
126 126 """
127 127 _display_mimetype('text/html', objs, **kwargs)
128 128
129 129
130 130 def display_markdown(*objs, **kwargs):
131 131 """Displays the Markdown representation of an object.
132 132
133 133 Parameters
134 134 ----------
135 135 *objs : object
136 136 The Python objects to display, or if raw=True raw markdown data to
137 137 display.
138 138 raw : bool
139 139 Are the data objects raw data or Python objects that need to be
140 140 formatted before display? [default: False]
141 141 metadata : dict (optional)
142 142 Metadata to be associated with the specific mimetype output.
143 143 """
144 144
145 145 _display_mimetype('text/markdown', objs, **kwargs)
146 146
147 147
148 148 def display_svg(*objs, **kwargs):
149 149 """Display the SVG representation of an object.
150 150
151 151 Parameters
152 152 ----------
153 153 *objs : object
154 154 The Python objects to display, or if raw=True raw svg data to
155 155 display.
156 156 raw : bool
157 157 Are the data objects raw data or Python objects that need to be
158 158 formatted before display? [default: False]
159 159 metadata : dict (optional)
160 160 Metadata to be associated with the specific mimetype output.
161 161 """
162 162 _display_mimetype('image/svg+xml', objs, **kwargs)
163 163
164 164
165 165 def display_png(*objs, **kwargs):
166 166 """Display the PNG representation of an object.
167 167
168 168 Parameters
169 169 ----------
170 170 *objs : object
171 171 The Python objects to display, or if raw=True raw png data to
172 172 display.
173 173 raw : bool
174 174 Are the data objects raw data or Python objects that need to be
175 175 formatted before display? [default: False]
176 176 metadata : dict (optional)
177 177 Metadata to be associated with the specific mimetype output.
178 178 """
179 179 _display_mimetype('image/png', objs, **kwargs)
180 180
181 181
182 182 def display_jpeg(*objs, **kwargs):
183 183 """Display the JPEG representation of an object.
184 184
185 185 Parameters
186 186 ----------
187 187 *objs : object
188 188 The Python objects to display, or if raw=True raw JPEG data to
189 189 display.
190 190 raw : bool
191 191 Are the data objects raw data or Python objects that need to be
192 192 formatted before display? [default: False]
193 193 metadata : dict (optional)
194 194 Metadata to be associated with the specific mimetype output.
195 195 """
196 196 _display_mimetype('image/jpeg', objs, **kwargs)
197 197
198 198
199 199 def display_latex(*objs, **kwargs):
200 200 """Display the LaTeX representation of an object.
201 201
202 202 Parameters
203 203 ----------
204 204 *objs : object
205 205 The Python objects to display, or if raw=True raw latex data to
206 206 display.
207 207 raw : bool
208 208 Are the data objects raw data or Python objects that need to be
209 209 formatted before display? [default: False]
210 210 metadata : dict (optional)
211 211 Metadata to be associated with the specific mimetype output.
212 212 """
213 213 _display_mimetype('text/latex', objs, **kwargs)
214 214
215 215
216 216 def display_json(*objs, **kwargs):
217 217 """Display the JSON representation of an object.
218 218
219 219 Note that not many frontends support displaying JSON.
220 220
221 221 Parameters
222 222 ----------
223 223 *objs : object
224 224 The Python objects to display, or if raw=True raw json data to
225 225 display.
226 226 raw : bool
227 227 Are the data objects raw data or Python objects that need to be
228 228 formatted before display? [default: False]
229 229 metadata : dict (optional)
230 230 Metadata to be associated with the specific mimetype output.
231 231 """
232 232 _display_mimetype('application/json', objs, **kwargs)
233 233
234 234
235 235 def display_javascript(*objs, **kwargs):
236 236 """Display the Javascript representation of an object.
237 237
238 238 Parameters
239 239 ----------
240 240 *objs : object
241 241 The Python objects to display, or if raw=True raw javascript data to
242 242 display.
243 243 raw : bool
244 244 Are the data objects raw data or Python objects that need to be
245 245 formatted before display? [default: False]
246 246 metadata : dict (optional)
247 247 Metadata to be associated with the specific mimetype output.
248 248 """
249 249 _display_mimetype('application/javascript', objs, **kwargs)
250 250
251 251
252 252 def display_pdf(*objs, **kwargs):
253 253 """Display the PDF representation of an object.
254 254
255 255 Parameters
256 256 ----------
257 257 *objs : object
258 258 The Python objects to display, or if raw=True raw javascript data to
259 259 display.
260 260 raw : bool
261 261 Are the data objects raw data or Python objects that need to be
262 262 formatted before display? [default: False]
263 263 metadata : dict (optional)
264 264 Metadata to be associated with the specific mimetype output.
265 265 """
266 266 _display_mimetype('application/pdf', objs, **kwargs)
267 267
268 268
269 269 #-----------------------------------------------------------------------------
270 270 # Smart classes
271 271 #-----------------------------------------------------------------------------
272 272
273 273
274 274 class DisplayObject(object):
275 275 """An object that wraps data to be displayed."""
276 276
277 277 _read_flags = 'r'
278 278 _show_mem_addr = False
279 279 metadata = None
280 280
281 281 def __init__(self, data=None, url=None, filename=None, metadata=None):
282 282 """Create a display object given raw data.
283 283
284 284 When this object is returned by an expression or passed to the
285 285 display function, it will result in the data being displayed
286 286 in the frontend. The MIME type of the data should match the
287 287 subclasses used, so the Png subclass should be used for 'image/png'
288 288 data. If the data is a URL, the data will first be downloaded
289 289 and then displayed. If
290 290
291 291 Parameters
292 292 ----------
293 293 data : unicode, str or bytes
294 294 The raw data or a URL or file to load the data from
295 295 url : unicode
296 296 A URL to download the data from.
297 297 filename : unicode
298 298 Path to a local file to load the data from.
299 299 metadata : dict
300 300 Dict of metadata associated to be the object when displayed
301 301 """
302 302 if isinstance(data, (Path, PurePath)):
303 303 data = str(data)
304 304
305 305 if data is not None and isinstance(data, str):
306 306 if data.startswith('http') and url is None:
307 307 url = data
308 308 filename = None
309 309 data = None
310 310 elif _safe_exists(data) and filename is None:
311 311 url = None
312 312 filename = data
313 313 data = None
314 314
315 315 self.url = url
316 316 self.filename = filename
317 317 # because of @data.setter methods in
318 318 # subclasses ensure url and filename are set
319 319 # before assigning to self.data
320 320 self.data = data
321 321
322 322 if metadata is not None:
323 323 self.metadata = metadata
324 324 elif self.metadata is None:
325 325 self.metadata = {}
326 326
327 327 self.reload()
328 328 self._check_data()
329 329
330 330 def __repr__(self):
331 331 if not self._show_mem_addr:
332 332 cls = self.__class__
333 333 r = "<%s.%s object>" % (cls.__module__, cls.__name__)
334 334 else:
335 335 r = super(DisplayObject, self).__repr__()
336 336 return r
337 337
338 338 def _check_data(self):
339 339 """Override in subclasses if there's something to check."""
340 340 pass
341 341
342 342 def _data_and_metadata(self):
343 343 """shortcut for returning metadata with shape information, if defined"""
344 344 if self.metadata:
345 345 return self.data, deepcopy(self.metadata)
346 346 else:
347 347 return self.data
348 348
349 349 def reload(self):
350 350 """Reload the raw data from file or URL."""
351 351 if self.filename is not None:
352 352 with open(self.filename, self._read_flags) as f:
353 353 self.data = f.read()
354 354 elif self.url is not None:
355 355 # Deferred import
356 356 from urllib.request import urlopen
357 357 response = urlopen(self.url)
358 358 data = response.read()
359 359 # extract encoding from header, if there is one:
360 360 encoding = None
361 361 if 'content-type' in response.headers:
362 362 for sub in response.headers['content-type'].split(';'):
363 363 sub = sub.strip()
364 364 if sub.startswith('charset'):
365 365 encoding = sub.split('=')[-1].strip()
366 366 break
367 367 if 'content-encoding' in response.headers:
368 368 # TODO: do deflate?
369 369 if 'gzip' in response.headers['content-encoding']:
370 370 import gzip
371 371 from io import BytesIO
372 372 with gzip.open(BytesIO(data), 'rt', encoding=encoding) as fp:
373 373 encoding = None
374 374 data = fp.read()
375 375
376 376 # decode data, if an encoding was specified
377 377 # We only touch self.data once since
378 378 # subclasses such as SVG have @data.setter methods
379 379 # that transform self.data into ... well svg.
380 380 if encoding:
381 381 self.data = data.decode(encoding, 'replace')
382 382 else:
383 383 self.data = data
384 384
385 385
386 386 class TextDisplayObject(DisplayObject):
387 387 """Validate that display data is text"""
388 388 def _check_data(self):
389 389 if self.data is not None and not isinstance(self.data, str):
390 390 raise TypeError("%s expects text, not %r" % (self.__class__.__name__, self.data))
391 391
392 392 class Pretty(TextDisplayObject):
393 393
394 394 def _repr_pretty_(self, pp, cycle):
395 395 return pp.text(self.data)
396 396
397 397
398 398 class HTML(TextDisplayObject):
399 399
400 400 def __init__(self, data=None, url=None, filename=None, metadata=None):
401 401 def warn():
402 402 if not data:
403 403 return False
404 404
405 405 #
406 406 # Avoid calling lower() on the entire data, because it could be a
407 407 # long string and we're only interested in its beginning and end.
408 408 #
409 409 prefix = data[:10].lower()
410 410 suffix = data[-10:].lower()
411 411 return prefix.startswith("<iframe ") and suffix.endswith("</iframe>")
412 412
413 413 if warn():
414 414 warnings.warn("Consider using IPython.display.IFrame instead")
415 415 super(HTML, self).__init__(data=data, url=url, filename=filename, metadata=metadata)
416 416
417 417 def _repr_html_(self):
418 418 return self._data_and_metadata()
419 419
420 420 def __html__(self):
421 421 """
422 422 This method exists to inform other HTML-using modules (e.g. Markupsafe,
423 423 htmltag, etc) that this object is HTML and does not need things like
424 424 special characters (<>&) escaped.
425 425 """
426 426 return self._repr_html_()
427 427
428 428
429 429 class Markdown(TextDisplayObject):
430 430
431 431 def _repr_markdown_(self):
432 432 return self._data_and_metadata()
433 433
434 434
435 435 class Math(TextDisplayObject):
436 436
437 437 def _repr_latex_(self):
438 438 s = r"$\displaystyle %s$" % self.data.strip('$')
439 439 if self.metadata:
440 440 return s, deepcopy(self.metadata)
441 441 else:
442 442 return s
443 443
444 444
445 445 class Latex(TextDisplayObject):
446 446
447 447 def _repr_latex_(self):
448 448 return self._data_and_metadata()
449 449
450 450
451 451 class SVG(DisplayObject):
452 452 """Embed an SVG into the display.
453 453
454 454 Note if you just want to view a svg image via a URL use `:class:Image` with
455 455 a url=URL keyword argument.
456 456 """
457 457
458 458 _read_flags = 'rb'
459 459 # wrap data in a property, which extracts the <svg> tag, discarding
460 460 # document headers
461 461 _data = None
462 462
463 463 @property
464 464 def data(self):
465 465 return self._data
466 466
467 467 @data.setter
468 468 def data(self, svg):
469 469 if svg is None:
470 470 self._data = None
471 471 return
472 472 # parse into dom object
473 473 from xml.dom import minidom
474 474 x = minidom.parseString(svg)
475 475 # get svg tag (should be 1)
476 476 found_svg = x.getElementsByTagName('svg')
477 477 if found_svg:
478 478 svg = found_svg[0].toxml()
479 479 else:
480 480 # fallback on the input, trust the user
481 481 # but this is probably an error.
482 482 pass
483 483 svg = cast_unicode(svg)
484 484 self._data = svg
485 485
486 486 def _repr_svg_(self):
487 487 return self._data_and_metadata()
488 488
489 489 class ProgressBar(DisplayObject):
490 490 """Progressbar supports displaying a progressbar like element
491 491 """
492 492 def __init__(self, total):
493 493 """Creates a new progressbar
494 494
495 495 Parameters
496 496 ----------
497 497 total : int
498 498 maximum size of the progressbar
499 499 """
500 500 self.total = total
501 501 self._progress = 0
502 502 self.html_width = '60ex'
503 503 self.text_width = 60
504 504 self._display_id = hexlify(os.urandom(8)).decode('ascii')
505 505
506 506 def __repr__(self):
507 507 fraction = self.progress / self.total
508 508 filled = '=' * int(fraction * self.text_width)
509 509 rest = ' ' * (self.text_width - len(filled))
510 510 return '[{}{}] {}/{}'.format(
511 511 filled, rest,
512 512 self.progress, self.total,
513 513 )
514 514
515 515 def _repr_html_(self):
516 516 return "<progress style='width:{}' max='{}' value='{}'></progress>".format(
517 517 self.html_width, self.total, self.progress)
518 518
519 519 def display(self):
520 520 display(self, display_id=self._display_id)
521 521
522 522 def update(self):
523 523 display(self, display_id=self._display_id, update=True)
524 524
525 525 @property
526 526 def progress(self):
527 527 return self._progress
528 528
529 529 @progress.setter
530 530 def progress(self, value):
531 531 self._progress = value
532 532 self.update()
533 533
534 534 def __iter__(self):
535 535 self.display()
536 536 self._progress = -1 # First iteration is 0
537 537 return self
538 538
539 539 def __next__(self):
540 540 """Returns current value and increments display by one."""
541 541 self.progress += 1
542 542 if self.progress < self.total:
543 543 return self.progress
544 544 else:
545 545 raise StopIteration()
546 546
547 547 class JSON(DisplayObject):
548 548 """JSON expects a JSON-able dict or list
549 549
550 550 not an already-serialized JSON string.
551 551
552 552 Scalar types (None, number, string) are not allowed, only dict or list containers.
553 553 """
554 554 # wrap data in a property, which warns about passing already-serialized JSON
555 555 _data = None
556 556 def __init__(self, data=None, url=None, filename=None, expanded=False, metadata=None, root='root', **kwargs):
557 557 """Create a JSON display object given raw data.
558 558
559 559 Parameters
560 560 ----------
561 561 data : dict or list
562 562 JSON data to display. Not an already-serialized JSON string.
563 563 Scalar types (None, number, string) are not allowed, only dict
564 564 or list containers.
565 565 url : unicode
566 566 A URL to download the data from.
567 567 filename : unicode
568 568 Path to a local file to load the data from.
569 569 expanded : boolean
570 570 Metadata to control whether a JSON display component is expanded.
571 571 metadata : dict
572 572 Specify extra metadata to attach to the json display object.
573 573 root : str
574 574 The name of the root element of the JSON tree
575 575 """
576 576 self.metadata = {
577 577 'expanded': expanded,
578 578 'root': root,
579 579 }
580 580 if metadata:
581 581 self.metadata.update(metadata)
582 582 if kwargs:
583 583 self.metadata.update(kwargs)
584 584 super(JSON, self).__init__(data=data, url=url, filename=filename)
585 585
586 586 def _check_data(self):
587 587 if self.data is not None and not isinstance(self.data, (dict, list)):
588 588 raise TypeError("%s expects JSONable dict or list, not %r" % (self.__class__.__name__, self.data))
589 589
590 590 @property
591 591 def data(self):
592 592 return self._data
593 593
594 594 @data.setter
595 595 def data(self, data):
596 596 if isinstance(data, (Path, PurePath)):
597 597 data = str(data)
598 598
599 599 if isinstance(data, str):
600 600 if self.filename is None and self.url is None:
601 601 warnings.warn("JSON expects JSONable dict or list, not JSON strings")
602 602 data = json.loads(data)
603 603 self._data = data
604 604
605 605 def _data_and_metadata(self):
606 606 return self.data, self.metadata
607 607
608 608 def _repr_json_(self):
609 609 return self._data_and_metadata()
610 610
611 611 _css_t = """var link = document.createElement("link");
612 612 link.ref = "stylesheet";
613 613 link.type = "text/css";
614 614 link.href = "%s";
615 615 document.head.appendChild(link);
616 616 """
617 617
618 618 _lib_t1 = """new Promise(function(resolve, reject) {
619 619 var script = document.createElement("script");
620 620 script.onload = resolve;
621 621 script.onerror = reject;
622 622 script.src = "%s";
623 623 document.head.appendChild(script);
624 624 }).then(() => {
625 625 """
626 626
627 627 _lib_t2 = """
628 628 });"""
629 629
630 630 class GeoJSON(JSON):
631 631 """GeoJSON expects JSON-able dict
632 632
633 633 not an already-serialized JSON string.
634 634
635 635 Scalar types (None, number, string) are not allowed, only dict containers.
636 636 """
637 637
638 638 def __init__(self, *args, **kwargs):
639 639 """Create a GeoJSON display object given raw data.
640 640
641 641 Parameters
642 642 ----------
643 643 data : dict or list
644 644 VegaLite data. Not an already-serialized JSON string.
645 645 Scalar types (None, number, string) are not allowed, only dict
646 646 or list containers.
647 647 url_template : string
648 648 Leaflet TileLayer URL template: http://leafletjs.com/reference.html#url-template
649 649 layer_options : dict
650 650 Leaflet TileLayer options: http://leafletjs.com/reference.html#tilelayer-options
651 651 url : unicode
652 652 A URL to download the data from.
653 653 filename : unicode
654 654 Path to a local file to load the data from.
655 655 metadata : dict
656 656 Specify extra metadata to attach to the json display object.
657 657
658 658 Examples
659 659 --------
660 660 The following will display an interactive map of Mars with a point of
661 661 interest on frontend that do support GeoJSON display.
662 662
663 663 >>> from IPython.display import GeoJSON
664 664
665 665 >>> GeoJSON(data={
666 666 ... "type": "Feature",
667 667 ... "geometry": {
668 668 ... "type": "Point",
669 669 ... "coordinates": [-81.327, 296.038]
670 670 ... }
671 671 ... },
672 672 ... url_template="http://s3-eu-west-1.amazonaws.com/whereonmars.cartodb.net/{basemap_id}/{z}/{x}/{y}.png",
673 673 ... layer_options={
674 674 ... "basemap_id": "celestia_mars-shaded-16k_global",
675 675 ... "attribution" : "Celestia/praesepe",
676 676 ... "minZoom" : 0,
677 677 ... "maxZoom" : 18,
678 678 ... })
679 679 <IPython.core.display.GeoJSON object>
680 680
681 681 In the terminal IPython, you will only see the text representation of
682 682 the GeoJSON object.
683 683
684 684 """
685 685
686 686 super(GeoJSON, self).__init__(*args, **kwargs)
687 687
688 688
689 689 def _ipython_display_(self):
690 690 bundle = {
691 691 'application/geo+json': self.data,
692 692 'text/plain': '<IPython.display.GeoJSON object>'
693 693 }
694 694 metadata = {
695 695 'application/geo+json': self.metadata
696 696 }
697 697 display(bundle, metadata=metadata, raw=True)
698 698
699 699 class Javascript(TextDisplayObject):
700 700
701 701 def __init__(self, data=None, url=None, filename=None, lib=None, css=None):
702 702 """Create a Javascript display object given raw data.
703 703
704 704 When this object is returned by an expression or passed to the
705 705 display function, it will result in the data being displayed
706 706 in the frontend. If the data is a URL, the data will first be
707 707 downloaded and then displayed.
708 708
709 709 In the Notebook, the containing element will be available as `element`,
710 710 and jQuery will be available. Content appended to `element` will be
711 711 visible in the output area.
712 712
713 713 Parameters
714 714 ----------
715 715 data : unicode, str or bytes
716 716 The Javascript source code or a URL to download it from.
717 717 url : unicode
718 718 A URL to download the data from.
719 719 filename : unicode
720 720 Path to a local file to load the data from.
721 721 lib : list or str
722 722 A sequence of Javascript library URLs to load asynchronously before
723 723 running the source code. The full URLs of the libraries should
724 724 be given. A single Javascript library URL can also be given as a
725 725 string.
726 726 css : list or str
727 727 A sequence of css files to load before running the source code.
728 728 The full URLs of the css files should be given. A single css URL
729 729 can also be given as a string.
730 730 """
731 731 if isinstance(lib, str):
732 732 lib = [lib]
733 733 elif lib is None:
734 734 lib = []
735 735 if isinstance(css, str):
736 736 css = [css]
737 737 elif css is None:
738 738 css = []
739 739 if not isinstance(lib, (list,tuple)):
740 740 raise TypeError('expected sequence, got: %r' % lib)
741 741 if not isinstance(css, (list,tuple)):
742 742 raise TypeError('expected sequence, got: %r' % css)
743 743 self.lib = lib
744 744 self.css = css
745 745 super(Javascript, self).__init__(data=data, url=url, filename=filename)
746 746
747 747 def _repr_javascript_(self):
748 748 r = ''
749 749 for c in self.css:
750 750 r += _css_t % c
751 751 for l in self.lib:
752 752 r += _lib_t1 % l
753 753 r += self.data
754 754 r += _lib_t2*len(self.lib)
755 755 return r
756 756
757 757 # constants for identifying png/jpeg data
758 758 _PNG = b'\x89PNG\r\n\x1a\n'
759 759 _JPEG = b'\xff\xd8'
760 760
761 761 def _pngxy(data):
762 762 """read the (width, height) from a PNG header"""
763 763 ihdr = data.index(b'IHDR')
764 764 # next 8 bytes are width/height
765 765 return struct.unpack('>ii', data[ihdr+4:ihdr+12])
766 766
767 767 def _jpegxy(data):
768 768 """read the (width, height) from a JPEG header"""
769 769 # adapted from http://www.64lines.com/jpeg-width-height
770 770
771 771 idx = 4
772 772 while True:
773 773 block_size = struct.unpack('>H', data[idx:idx+2])[0]
774 774 idx = idx + block_size
775 775 if data[idx:idx+2] == b'\xFF\xC0':
776 776 # found Start of Frame
777 777 iSOF = idx
778 778 break
779 779 else:
780 780 # read another block
781 781 idx += 2
782 782
783 783 h, w = struct.unpack('>HH', data[iSOF+5:iSOF+9])
784 784 return w, h
785 785
786 786 def _gifxy(data):
787 787 """read the (width, height) from a GIF header"""
788 788 return struct.unpack('<HH', data[6:10])
789 789
790 790
791 791 class Image(DisplayObject):
792 792
793 793 _read_flags = 'rb'
794 794 _FMT_JPEG = u'jpeg'
795 795 _FMT_PNG = u'png'
796 796 _FMT_GIF = u'gif'
797 797 _ACCEPTABLE_EMBEDDINGS = [_FMT_JPEG, _FMT_PNG, _FMT_GIF]
798 798 _MIMETYPES = {
799 799 _FMT_PNG: 'image/png',
800 800 _FMT_JPEG: 'image/jpeg',
801 801 _FMT_GIF: 'image/gif',
802 802 }
803 803
804 804 def __init__(
805 805 self,
806 806 data=None,
807 807 url=None,
808 808 filename=None,
809 809 format=None,
810 810 embed=None,
811 811 width=None,
812 812 height=None,
813 813 retina=False,
814 814 unconfined=False,
815 815 metadata=None,
816 816 alt=None,
817 817 ):
818 818 """Create a PNG/JPEG/GIF image object given raw data.
819 819
820 820 When this object is returned by an input cell or passed to the
821 821 display function, it will result in the image being displayed
822 822 in the frontend.
823 823
824 824 Parameters
825 825 ----------
826 826 data : unicode, str or bytes
827 827 The raw image data or a URL or filename to load the data from.
828 828 This always results in embedded image data.
829
829 830 url : unicode
830 831 A URL to download the data from. If you specify `url=`,
831 832 the image data will not be embedded unless you also specify `embed=True`.
833
832 834 filename : unicode
833 835 Path to a local file to load the data from.
834 836 Images from a file are always embedded.
837
835 838 format : unicode
836 839 The format of the image data (png/jpeg/jpg/gif). If a filename or URL is given
837 840 for format will be inferred from the filename extension.
841
838 842 embed : bool
839 843 Should the image data be embedded using a data URI (True) or be
840 844 loaded using an <img> tag. Set this to True if you want the image
841 845 to be viewable later with no internet connection in the notebook.
842 846
843 847 Default is `True`, unless the keyword argument `url` is set, then
844 848 default value is `False`.
845 849
846 850 Note that QtConsole is not able to display images if `embed` is set to `False`
851
847 852 width : int
848 853 Width in pixels to which to constrain the image in html
854
849 855 height : int
850 856 Height in pixels to which to constrain the image in html
857
851 858 retina : bool
852 859 Automatically set the width and height to half of the measured
853 860 width and height.
854 861 This only works for embedded images because it reads the width/height
855 862 from image data.
856 863 For non-embedded images, you can just set the desired display width
857 864 and height directly.
865
858 866 unconfined : bool
859 867 Set unconfined=True to disable max-width confinement of the image.
868
860 869 metadata : dict
861 870 Specify extra metadata to attach to the image.
871
862 872 alt : unicode
863 873 Alternative text for the image, for use by screen readers.
864 874
865 875 Examples
866 876 --------
867 877 embedded image data, works in qtconsole and notebook
868 878 when passed positionally, the first arg can be any of raw image data,
869 879 a URL, or a filename from which to load image data.
870 880 The result is always embedding image data for inline images.
871 881
872 882 >>> Image('http://www.google.fr/images/srpr/logo3w.png')
873 883 <IPython.core.display.Image object>
874 884
875 885 >>> Image('/path/to/image.jpg')
876 886 <IPython.core.display.Image object>
877 887
878 888 >>> Image(b'RAW_PNG_DATA...')
879 889 <IPython.core.display.Image object>
880 890
881 891 Specifying Image(url=...) does not embed the image data,
882 892 it only generates ``<img>`` tag with a link to the source.
883 893 This will not work in the qtconsole or offline.
884 894
885 895 >>> Image(url='http://www.google.fr/images/srpr/logo3w.png')
886 896 <IPython.core.display.Image object>
887 897
888 898 """
889 899 if isinstance(data, (Path, PurePath)):
890 900 data = str(data)
891 901
892 902 if filename is not None:
893 903 ext = self._find_ext(filename)
894 904 elif url is not None:
895 905 ext = self._find_ext(url)
896 906 elif data is None:
897 907 raise ValueError("No image data found. Expecting filename, url, or data.")
898 908 elif isinstance(data, str) and (
899 909 data.startswith('http') or _safe_exists(data)
900 910 ):
901 911 ext = self._find_ext(data)
902 912 else:
903 913 ext = None
904 914
905 915 if format is None:
906 916 if ext is not None:
907 917 if ext == u'jpg' or ext == u'jpeg':
908 918 format = self._FMT_JPEG
909 919 elif ext == u'png':
910 920 format = self._FMT_PNG
911 921 elif ext == u'gif':
912 922 format = self._FMT_GIF
913 923 else:
914 924 format = ext.lower()
915 925 elif isinstance(data, bytes):
916 926 # infer image type from image data header,
917 927 # only if format has not been specified.
918 928 if data[:2] == _JPEG:
919 929 format = self._FMT_JPEG
920 930
921 931 # failed to detect format, default png
922 932 if format is None:
923 933 format = self._FMT_PNG
924 934
925 935 if format.lower() == 'jpg':
926 936 # jpg->jpeg
927 937 format = self._FMT_JPEG
928 938
929 939 self.format = format.lower()
930 940 self.embed = embed if embed is not None else (url is None)
931 941
932 942 if self.embed and self.format not in self._ACCEPTABLE_EMBEDDINGS:
933 943 raise ValueError("Cannot embed the '%s' image format" % (self.format))
934 944 if self.embed:
935 945 self._mimetype = self._MIMETYPES.get(self.format)
936 946
937 947 self.width = width
938 948 self.height = height
939 949 self.retina = retina
940 950 self.unconfined = unconfined
941 951 self.alt = alt
942 952 super(Image, self).__init__(data=data, url=url, filename=filename,
943 953 metadata=metadata)
944 954
945 955 if self.width is None and self.metadata.get('width', {}):
946 956 self.width = metadata['width']
947 957
948 958 if self.height is None and self.metadata.get('height', {}):
949 959 self.height = metadata['height']
950 960
951 961 if self.alt is None and self.metadata.get("alt", {}):
952 962 self.alt = metadata["alt"]
953 963
954 964 if retina:
955 965 self._retina_shape()
956 966
957 967
958 968 def _retina_shape(self):
959 969 """load pixel-doubled width and height from image data"""
960 970 if not self.embed:
961 971 return
962 972 if self.format == self._FMT_PNG:
963 973 w, h = _pngxy(self.data)
964 974 elif self.format == self._FMT_JPEG:
965 975 w, h = _jpegxy(self.data)
966 976 elif self.format == self._FMT_GIF:
967 977 w, h = _gifxy(self.data)
968 978 else:
969 979 # retina only supports png
970 980 return
971 981 self.width = w // 2
972 982 self.height = h // 2
973 983
974 984 def reload(self):
975 985 """Reload the raw data from file or URL."""
976 986 if self.embed:
977 987 super(Image,self).reload()
978 988 if self.retina:
979 989 self._retina_shape()
980 990
981 991 def _repr_html_(self):
982 992 if not self.embed:
983 993 width = height = klass = alt = ""
984 994 if self.width:
985 995 width = ' width="%d"' % self.width
986 996 if self.height:
987 997 height = ' height="%d"' % self.height
988 998 if self.unconfined:
989 999 klass = ' class="unconfined"'
990 1000 if self.alt:
991 1001 alt = ' alt="%s"' % html.escape(self.alt)
992 1002 return '<img src="{url}"{width}{height}{klass}{alt}/>'.format(
993 1003 url=self.url,
994 1004 width=width,
995 1005 height=height,
996 1006 klass=klass,
997 1007 alt=alt,
998 1008 )
999 1009
1000 1010 def _repr_mimebundle_(self, include=None, exclude=None):
1001 1011 """Return the image as a mimebundle
1002 1012
1003 1013 Any new mimetype support should be implemented here.
1004 1014 """
1005 1015 if self.embed:
1006 1016 mimetype = self._mimetype
1007 1017 data, metadata = self._data_and_metadata(always_both=True)
1008 1018 if metadata:
1009 1019 metadata = {mimetype: metadata}
1010 1020 return {mimetype: data}, metadata
1011 1021 else:
1012 1022 return {'text/html': self._repr_html_()}
1013 1023
1014 1024 def _data_and_metadata(self, always_both=False):
1015 1025 """shortcut for returning metadata with shape information, if defined"""
1016 1026 try:
1017 1027 b64_data = b2a_base64(self.data).decode('ascii')
1018 1028 except TypeError as e:
1019 1029 raise FileNotFoundError(
1020 1030 "No such file or directory: '%s'" % (self.data)) from e
1021 1031 md = {}
1022 1032 if self.metadata:
1023 1033 md.update(self.metadata)
1024 1034 if self.width:
1025 1035 md['width'] = self.width
1026 1036 if self.height:
1027 1037 md['height'] = self.height
1028 1038 if self.unconfined:
1029 1039 md['unconfined'] = self.unconfined
1030 1040 if self.alt:
1031 1041 md["alt"] = self.alt
1032 1042 if md or always_both:
1033 1043 return b64_data, md
1034 1044 else:
1035 1045 return b64_data
1036 1046
1037 1047 def _repr_png_(self):
1038 1048 if self.embed and self.format == self._FMT_PNG:
1039 1049 return self._data_and_metadata()
1040 1050
1041 1051 def _repr_jpeg_(self):
1042 1052 if self.embed and self.format == self._FMT_JPEG:
1043 1053 return self._data_and_metadata()
1044 1054
1045 1055 def _find_ext(self, s):
1046 1056 base, ext = splitext(s)
1047 1057
1048 1058 if not ext:
1049 1059 return base
1050 1060
1051 1061 # `splitext` includes leading period, so we skip it
1052 1062 return ext[1:].lower()
1053 1063
1054 1064
1055 1065 class Video(DisplayObject):
1056 1066
1057 1067 def __init__(self, data=None, url=None, filename=None, embed=False,
1058 1068 mimetype=None, width=None, height=None, html_attributes="controls"):
1059 1069 """Create a video object given raw data or an URL.
1060 1070
1061 1071 When this object is returned by an input cell or passed to the
1062 1072 display function, it will result in the video being displayed
1063 1073 in the frontend.
1064 1074
1065 1075 Parameters
1066 1076 ----------
1067 1077 data : unicode, str or bytes
1068 1078 The raw video data or a URL or filename to load the data from.
1069 1079 Raw data will require passing ``embed=True``.
1080
1070 1081 url : unicode
1071 1082 A URL for the video. If you specify ``url=``,
1072 1083 the image data will not be embedded.
1084
1073 1085 filename : unicode
1074 1086 Path to a local file containing the video.
1075 1087 Will be interpreted as a local URL unless ``embed=True``.
1088
1076 1089 embed : bool
1077 1090 Should the video be embedded using a data URI (True) or be
1078 1091 loaded using a <video> tag (False).
1079 1092
1080 1093 Since videos are large, embedding them should be avoided, if possible.
1081 1094 You must confirm embedding as your intention by passing ``embed=True``.
1082 1095
1083 1096 Local files can be displayed with URLs without embedding the content, via::
1084 1097
1085 1098 Video('./video.mp4')
1099
1086 1100 mimetype : unicode
1087 1101 Specify the mimetype for embedded videos.
1088 1102 Default will be guessed from file extension, if available.
1103
1089 1104 width : int
1090 1105 Width in pixels to which to constrain the video in HTML.
1091 1106 If not supplied, defaults to the width of the video.
1107
1092 1108 height : int
1093 1109 Height in pixels to which to constrain the video in html.
1094 1110 If not supplied, defaults to the height of the video.
1111
1095 1112 html_attributes : str
1096 1113 Attributes for the HTML ``<video>`` block.
1097 1114 Default: ``"controls"`` to get video controls.
1098 1115 Other examples: ``"controls muted"`` for muted video with controls,
1099 1116 ``"loop autoplay"`` for looping autoplaying video without controls.
1100 1117
1101 1118 Examples
1102 1119 --------
1103 1120 ::
1104 1121
1105 1122 Video('https://archive.org/download/Sita_Sings_the_Blues/Sita_Sings_the_Blues_small.mp4')
1106 1123 Video('path/to/video.mp4')
1107 1124 Video('path/to/video.mp4', embed=True)
1108 1125 Video('path/to/video.mp4', embed=True, html_attributes="controls muted autoplay")
1109 1126 Video(b'raw-videodata', embed=True)
1110 1127 """
1111 1128 if isinstance(data, (Path, PurePath)):
1112 1129 data = str(data)
1113 1130
1114 1131 if url is None and isinstance(data, str) and data.startswith(('http:', 'https:')):
1115 1132 url = data
1116 1133 data = None
1117 1134 elif data is not None and os.path.exists(data):
1118 1135 filename = data
1119 1136 data = None
1120 1137
1121 1138 if data and not embed:
1122 1139 msg = ''.join([
1123 1140 "To embed videos, you must pass embed=True ",
1124 1141 "(this may make your notebook files huge)\n",
1125 1142 "Consider passing Video(url='...')",
1126 1143 ])
1127 1144 raise ValueError(msg)
1128 1145
1129 1146 self.mimetype = mimetype
1130 1147 self.embed = embed
1131 1148 self.width = width
1132 1149 self.height = height
1133 1150 self.html_attributes = html_attributes
1134 1151 super(Video, self).__init__(data=data, url=url, filename=filename)
1135 1152
1136 1153 def _repr_html_(self):
1137 1154 width = height = ''
1138 1155 if self.width:
1139 1156 width = ' width="%d"' % self.width
1140 1157 if self.height:
1141 1158 height = ' height="%d"' % self.height
1142 1159
1143 1160 # External URLs and potentially local files are not embedded into the
1144 1161 # notebook output.
1145 1162 if not self.embed:
1146 1163 url = self.url if self.url is not None else self.filename
1147 1164 output = """<video src="{0}" {1} {2} {3}>
1148 1165 Your browser does not support the <code>video</code> element.
1149 1166 </video>""".format(url, self.html_attributes, width, height)
1150 1167 return output
1151 1168
1152 1169 # Embedded videos are base64-encoded.
1153 1170 mimetype = self.mimetype
1154 1171 if self.filename is not None:
1155 1172 if not mimetype:
1156 1173 mimetype, _ = mimetypes.guess_type(self.filename)
1157 1174
1158 1175 with open(self.filename, 'rb') as f:
1159 1176 video = f.read()
1160 1177 else:
1161 1178 video = self.data
1162 1179 if isinstance(video, str):
1163 1180 # unicode input is already b64-encoded
1164 1181 b64_video = video
1165 1182 else:
1166 1183 b64_video = b2a_base64(video).decode('ascii').rstrip()
1167 1184
1168 1185 output = """<video {0} {1} {2}>
1169 1186 <source src="data:{3};base64,{4}" type="{3}">
1170 1187 Your browser does not support the video tag.
1171 1188 </video>""".format(self.html_attributes, width, height, mimetype, b64_video)
1172 1189 return output
1173 1190
1174 1191 def reload(self):
1175 1192 # TODO
1176 1193 pass
1177 1194
1178 1195
1179 1196 @skip_doctest
1180 1197 def set_matplotlib_formats(*formats, **kwargs):
1181 1198 """
1182 1199 .. deprecated:: 7.23
1183 1200
1184 1201 use `matplotlib_inline.backend_inline.set_matplotlib_formats()`
1185 1202
1186 1203 Select figure formats for the inline backend. Optionally pass quality for JPEG.
1187 1204
1188 1205 For example, this enables PNG and JPEG output with a JPEG quality of 90%::
1189 1206
1190 1207 In [1]: set_matplotlib_formats('png', 'jpeg', quality=90)
1191 1208
1192 1209 To set this in your config files use the following::
1193 1210
1194 1211 c.InlineBackend.figure_formats = {'png', 'jpeg'}
1195 1212 c.InlineBackend.print_figure_kwargs.update({'quality' : 90})
1196 1213
1197 1214 Parameters
1198 1215 ----------
1199 1216 *formats : strs
1200 1217 One or more figure formats to enable: 'png', 'retina', 'jpeg', 'svg', 'pdf'.
1201 1218 **kwargs
1202 1219 Keyword args will be relayed to ``figure.canvas.print_figure``.
1203 1220 """
1204 1221 warnings.warn(
1205 1222 "`set_matplotlib_formats` is deprecated since IPython 7.23, directly "
1206 1223 "use `matplotlib_inline.backend_inline.set_matplotlib_formats()`",
1207 1224 DeprecationWarning,
1208 1225 stacklevel=2,
1209 1226 )
1210 1227
1211 1228 from matplotlib_inline.backend_inline import (
1212 1229 set_matplotlib_formats as set_matplotlib_formats_orig,
1213 1230 )
1214 1231
1215 1232 set_matplotlib_formats_orig(*formats, **kwargs)
1216 1233
1217 1234 @skip_doctest
1218 1235 def set_matplotlib_close(close=True):
1219 1236 """
1220 1237 .. deprecated:: 7.23
1221 1238
1222 1239 use `matplotlib_inline.backend_inline.set_matplotlib_close()`
1223 1240
1224
1225 1241 Set whether the inline backend closes all figures automatically or not.
1226 1242
1227 1243 By default, the inline backend used in the IPython Notebook will close all
1228 1244 matplotlib figures automatically after each cell is run. This means that
1229 1245 plots in different cells won't interfere. Sometimes, you may want to make
1230 1246 a plot in one cell and then refine it in later cells. This can be accomplished
1231 1247 by::
1232 1248
1233 1249 In [1]: set_matplotlib_close(False)
1234 1250
1235 1251 To set this in your config files use the following::
1236 1252
1237 1253 c.InlineBackend.close_figures = False
1238 1254
1239 1255 Parameters
1240 1256 ----------
1241 1257 close : bool
1242 1258 Should all matplotlib figures be automatically closed after each cell is
1243 1259 run?
1244 1260 """
1245 1261 warnings.warn(
1246 1262 "`set_matplotlib_close` is deprecated since IPython 7.23, directly "
1247 1263 "use `matplotlib_inline.backend_inline.set_matplotlib_close()`",
1248 1264 DeprecationWarning,
1249 1265 stacklevel=2,
1250 1266 )
1251 1267
1252 1268 from matplotlib_inline.backend_inline import (
1253 1269 set_matplotlib_close as set_matplotlib_close_orig,
1254 1270 )
1255 1271
1256 1272 set_matplotlib_close_orig(close)
@@ -1,1024 +1,1027 b''
1 1 # -*- coding: utf-8 -*-
2 2 """Display formatters.
3 3
4 4 Inheritance diagram:
5 5
6 6 .. inheritance-diagram:: IPython.core.formatters
7 7 :parts: 3
8 8 """
9 9
10 10 # Copyright (c) IPython Development Team.
11 11 # Distributed under the terms of the Modified BSD License.
12 12
13 13 import abc
14 14 import json
15 15 import sys
16 16 import traceback
17 17 import warnings
18 18 from io import StringIO
19 19
20 20 from decorator import decorator
21 21
22 22 from traitlets.config.configurable import Configurable
23 23 from .getipython import get_ipython
24 24 from ..utils.sentinel import Sentinel
25 25 from ..utils.dir2 import get_real_method
26 26 from ..lib import pretty
27 27 from traitlets import (
28 28 Bool, Dict, Integer, Unicode, CUnicode, ObjectName, List,
29 29 ForwardDeclaredInstance,
30 30 default, observe,
31 31 )
32 32
33 33
34 34 class DisplayFormatter(Configurable):
35 35
36 36 active_types = List(Unicode(),
37 37 help="""List of currently active mime-types to display.
38 38 You can use this to set a white-list for formats to display.
39 39
40 40 Most users will not need to change this value.
41 41 """).tag(config=True)
42 42
43 43 @default('active_types')
44 44 def _active_types_default(self):
45 45 return self.format_types
46 46
47 47 @observe('active_types')
48 48 def _active_types_changed(self, change):
49 49 for key, formatter in self.formatters.items():
50 50 if key in change['new']:
51 51 formatter.enabled = True
52 52 else:
53 53 formatter.enabled = False
54 54
55 55 ipython_display_formatter = ForwardDeclaredInstance('FormatterABC')
56 56 @default('ipython_display_formatter')
57 57 def _default_formatter(self):
58 58 return IPythonDisplayFormatter(parent=self)
59 59
60 60 mimebundle_formatter = ForwardDeclaredInstance('FormatterABC')
61 61 @default('mimebundle_formatter')
62 62 def _default_mime_formatter(self):
63 63 return MimeBundleFormatter(parent=self)
64 64
65 65 # A dict of formatter whose keys are format types (MIME types) and whose
66 66 # values are subclasses of BaseFormatter.
67 67 formatters = Dict()
68 68 @default('formatters')
69 69 def _formatters_default(self):
70 70 """Activate the default formatters."""
71 71 formatter_classes = [
72 72 PlainTextFormatter,
73 73 HTMLFormatter,
74 74 MarkdownFormatter,
75 75 SVGFormatter,
76 76 PNGFormatter,
77 77 PDFFormatter,
78 78 JPEGFormatter,
79 79 LatexFormatter,
80 80 JSONFormatter,
81 81 JavascriptFormatter
82 82 ]
83 83 d = {}
84 84 for cls in formatter_classes:
85 85 f = cls(parent=self)
86 86 d[f.format_type] = f
87 87 return d
88 88
89 89 def format(self, obj, include=None, exclude=None):
90 90 """Return a format data dict for an object.
91 91
92 92 By default all format types will be computed.
93 93
94 94 The following MIME types are usually implemented:
95 95
96 96 * text/plain
97 97 * text/html
98 98 * text/markdown
99 99 * text/latex
100 100 * application/json
101 101 * application/javascript
102 102 * application/pdf
103 103 * image/png
104 104 * image/jpeg
105 105 * image/svg+xml
106 106
107 107 Parameters
108 108 ----------
109 109 obj : object
110 110 The Python object whose format data will be computed.
111 111 include : list, tuple or set; optional
112 112 A list of format type strings (MIME types) to include in the
113 113 format data dict. If this is set *only* the format types included
114 114 in this list will be computed.
115 115 exclude : list, tuple or set; optional
116 116 A list of format type string (MIME types) to exclude in the format
117 117 data dict. If this is set all format types will be computed,
118 118 except for those included in this argument.
119 119 Mimetypes present in exclude will take precedence over the ones in include
120 120
121 121 Returns
122 122 -------
123 123 (format_dict, metadata_dict) : tuple of two dicts
124 124 format_dict is a dictionary of key/value pairs, one of each format that was
125 125 generated for the object. The keys are the format types, which
126 126 will usually be MIME type strings and the values and JSON'able
127 127 data structure containing the raw data for the representation in
128 128 that format.
129 129
130 130 metadata_dict is a dictionary of metadata about each mime-type output.
131 131 Its keys will be a strict subset of the keys in format_dict.
132 132
133 133 Notes
134 134 -----
135 135 If an object implement `_repr_mimebundle_` as well as various
136 136 `_repr_*_`, the data returned by `_repr_mimebundle_` will take
137 137 precedence and the corresponding `_repr_*_` for this mimetype will
138 138 not be called.
139 139
140 140 """
141 141 format_dict = {}
142 142 md_dict = {}
143 143
144 144 if self.ipython_display_formatter(obj):
145 145 # object handled itself, don't proceed
146 146 return {}, {}
147 147
148 148 format_dict, md_dict = self.mimebundle_formatter(obj, include=include, exclude=exclude)
149 149
150 150 if format_dict or md_dict:
151 151 if include:
152 152 format_dict = {k:v for k,v in format_dict.items() if k in include}
153 153 md_dict = {k:v for k,v in md_dict.items() if k in include}
154 154 if exclude:
155 155 format_dict = {k:v for k,v in format_dict.items() if k not in exclude}
156 156 md_dict = {k:v for k,v in md_dict.items() if k not in exclude}
157 157
158 158 for format_type, formatter in self.formatters.items():
159 159 if format_type in format_dict:
160 160 # already got it from mimebundle, maybe don't render again.
161 161 # exception: manually registered per-mime renderer
162 162 # check priority:
163 163 # 1. user-registered per-mime formatter
164 164 # 2. mime-bundle (user-registered or repr method)
165 165 # 3. default per-mime formatter (e.g. repr method)
166 166 try:
167 167 formatter.lookup(obj)
168 168 except KeyError:
169 169 # no special formatter, use mime-bundle-provided value
170 170 continue
171 171 if include and format_type not in include:
172 172 continue
173 173 if exclude and format_type in exclude:
174 174 continue
175 175
176 176 md = None
177 177 try:
178 178 data = formatter(obj)
179 179 except:
180 180 # FIXME: log the exception
181 181 raise
182 182
183 183 # formatters can return raw data or (data, metadata)
184 184 if isinstance(data, tuple) and len(data) == 2:
185 185 data, md = data
186 186
187 187 if data is not None:
188 188 format_dict[format_type] = data
189 189 if md is not None:
190 190 md_dict[format_type] = md
191 191 return format_dict, md_dict
192 192
193 193 @property
194 194 def format_types(self):
195 195 """Return the format types (MIME types) of the active formatters."""
196 196 return list(self.formatters.keys())
197 197
198 198
199 199 #-----------------------------------------------------------------------------
200 200 # Formatters for specific format types (text, html, svg, etc.)
201 201 #-----------------------------------------------------------------------------
202 202
203 203
204 204 def _safe_repr(obj):
205 205 """Try to return a repr of an object
206 206
207 207 always returns a string, at least.
208 208 """
209 209 try:
210 210 return repr(obj)
211 211 except Exception as e:
212 212 return "un-repr-able object (%r)" % e
213 213
214 214
215 215 class FormatterWarning(UserWarning):
216 216 """Warning class for errors in formatters"""
217 217
218 218 @decorator
219 219 def catch_format_error(method, self, *args, **kwargs):
220 220 """show traceback on failed format call"""
221 221 try:
222 222 r = method(self, *args, **kwargs)
223 223 except NotImplementedError:
224 224 # don't warn on NotImplementedErrors
225 225 return self._check_return(None, args[0])
226 226 except Exception:
227 227 exc_info = sys.exc_info()
228 228 ip = get_ipython()
229 229 if ip is not None:
230 230 ip.showtraceback(exc_info)
231 231 else:
232 232 traceback.print_exception(*exc_info)
233 233 return self._check_return(None, args[0])
234 234 return self._check_return(r, args[0])
235 235
236 236
237 237 class FormatterABC(metaclass=abc.ABCMeta):
238 238 """ Abstract base class for Formatters.
239 239
240 240 A formatter is a callable class that is responsible for computing the
241 241 raw format data for a particular format type (MIME type). For example,
242 242 an HTML formatter would have a format type of `text/html` and would return
243 243 the HTML representation of the object when called.
244 244 """
245 245
246 246 # The format type of the data returned, usually a MIME type.
247 247 format_type = 'text/plain'
248 248
249 249 # Is the formatter enabled...
250 250 enabled = True
251 251
252 252 @abc.abstractmethod
253 253 def __call__(self, obj):
254 254 """Return a JSON'able representation of the object.
255 255
256 256 If the object cannot be formatted by this formatter,
257 257 warn and return None.
258 258 """
259 259 return repr(obj)
260 260
261 261
262 262 def _mod_name_key(typ):
263 263 """Return a (__module__, __name__) tuple for a type.
264 264
265 265 Used as key in Formatter.deferred_printers.
266 266 """
267 267 module = getattr(typ, '__module__', None)
268 268 name = getattr(typ, '__name__', None)
269 269 return (module, name)
270 270
271 271
272 272 def _get_type(obj):
273 273 """Return the type of an instance (old and new-style)"""
274 274 return getattr(obj, '__class__', None) or type(obj)
275 275
276 276
277 277 _raise_key_error = Sentinel('_raise_key_error', __name__,
278 278 """
279 279 Special value to raise a KeyError
280 280
281 281 Raise KeyError in `BaseFormatter.pop` if passed as the default value to `pop`
282 282 """)
283 283
284 284
285 285 class BaseFormatter(Configurable):
286 286 """A base formatter class that is configurable.
287 287
288 288 This formatter should usually be used as the base class of all formatters.
289 289 It is a traited :class:`Configurable` class and includes an extensible
290 290 API for users to determine how their objects are formatted. The following
291 291 logic is used to find a function to format an given object.
292 292
293 293 1. The object is introspected to see if it has a method with the name
294 294 :attr:`print_method`. If is does, that object is passed to that method
295 295 for formatting.
296 296 2. If no print method is found, three internal dictionaries are consulted
297 297 to find print method: :attr:`singleton_printers`, :attr:`type_printers`
298 298 and :attr:`deferred_printers`.
299 299
300 300 Users should use these dictionaries to register functions that will be
301 301 used to compute the format data for their objects (if those objects don't
302 302 have the special print methods). The easiest way of using these
303 303 dictionaries is through the :meth:`for_type` and :meth:`for_type_by_name`
304 304 methods.
305 305
306 306 If no function/callable is found to compute the format data, ``None`` is
307 307 returned and this format type is not used.
308 308 """
309 309
310 310 format_type = Unicode('text/plain')
311 311 _return_type = str
312 312
313 313 enabled = Bool(True).tag(config=True)
314 314
315 315 print_method = ObjectName('__repr__')
316 316
317 317 # The singleton printers.
318 318 # Maps the IDs of the builtin singleton objects to the format functions.
319 319 singleton_printers = Dict().tag(config=True)
320 320
321 321 # The type-specific printers.
322 322 # Map type objects to the format functions.
323 323 type_printers = Dict().tag(config=True)
324 324
325 325 # The deferred-import type-specific printers.
326 326 # Map (modulename, classname) pairs to the format functions.
327 327 deferred_printers = Dict().tag(config=True)
328 328
329 329 @catch_format_error
330 330 def __call__(self, obj):
331 331 """Compute the format for an object."""
332 332 if self.enabled:
333 333 # lookup registered printer
334 334 try:
335 335 printer = self.lookup(obj)
336 336 except KeyError:
337 337 pass
338 338 else:
339 339 return printer(obj)
340 340 # Finally look for special method names
341 341 method = get_real_method(obj, self.print_method)
342 342 if method is not None:
343 343 return method()
344 344 return None
345 345 else:
346 346 return None
347 347
348 348 def __contains__(self, typ):
349 349 """map in to lookup_by_type"""
350 350 try:
351 351 self.lookup_by_type(typ)
352 352 except KeyError:
353 353 return False
354 354 else:
355 355 return True
356 356
357 357 def _check_return(self, r, obj):
358 358 """Check that a return value is appropriate
359 359
360 360 Return the value if so, None otherwise, warning if invalid.
361 361 """
362 362 if r is None or isinstance(r, self._return_type) or \
363 363 (isinstance(r, tuple) and r and isinstance(r[0], self._return_type)):
364 364 return r
365 365 else:
366 366 warnings.warn(
367 367 "%s formatter returned invalid type %s (expected %s) for object: %s" % \
368 368 (self.format_type, type(r), self._return_type, _safe_repr(obj)),
369 369 FormatterWarning
370 370 )
371 371
372 372 def lookup(self, obj):
373 373 """Look up the formatter for a given instance.
374 374
375 375 Parameters
376 376 ----------
377 377 obj : object instance
378 378
379 379 Returns
380 380 -------
381 381 f : callable
382 382 The registered formatting callable for the type.
383 383
384 384 Raises
385 385 ------
386 386 KeyError if the type has not been registered.
387 387 """
388 388 # look for singleton first
389 389 obj_id = id(obj)
390 390 if obj_id in self.singleton_printers:
391 391 return self.singleton_printers[obj_id]
392 392 # then lookup by type
393 393 return self.lookup_by_type(_get_type(obj))
394 394
395 395 def lookup_by_type(self, typ):
396 396 """Look up the registered formatter for a type.
397 397
398 398 Parameters
399 399 ----------
400 400 typ : type or '__module__.__name__' string for a type
401 401
402 402 Returns
403 403 -------
404 404 f : callable
405 405 The registered formatting callable for the type.
406 406
407 407 Raises
408 408 ------
409 409 KeyError if the type has not been registered.
410 410 """
411 411 if isinstance(typ, str):
412 412 typ_key = tuple(typ.rsplit('.',1))
413 413 if typ_key not in self.deferred_printers:
414 414 # We may have it cached in the type map. We will have to
415 415 # iterate over all of the types to check.
416 416 for cls in self.type_printers:
417 417 if _mod_name_key(cls) == typ_key:
418 418 return self.type_printers[cls]
419 419 else:
420 420 return self.deferred_printers[typ_key]
421 421 else:
422 422 for cls in pretty._get_mro(typ):
423 423 if cls in self.type_printers or self._in_deferred_types(cls):
424 424 return self.type_printers[cls]
425 425
426 426 # If we have reached here, the lookup failed.
427 427 raise KeyError("No registered printer for {0!r}".format(typ))
428 428
429 429 def for_type(self, typ, func=None):
430 430 """Add a format function for a given type.
431 431
432 432 Parameters
433 433 ----------
434 434 typ : type or '__module__.__name__' string for a type
435 435 The class of the object that will be formatted using `func`.
436
436 437 func : callable
437 438 A callable for computing the format data.
438 439 `func` will be called with the object to be formatted,
439 440 and will return the raw data in this formatter's format.
440 441 Subclasses may use a different call signature for the
441 442 `func` argument.
442 443
443 444 If `func` is None or not specified, there will be no change,
444 445 only returning the current value.
445 446
446 447 Returns
447 448 -------
448 449 oldfunc : callable
449 450 The currently registered callable.
450 451 If you are registering a new formatter,
451 452 this will be the previous value (to enable restoring later).
452 453 """
453 454 # if string given, interpret as 'pkg.module.class_name'
454 455 if isinstance(typ, str):
455 456 type_module, type_name = typ.rsplit('.', 1)
456 457 return self.for_type_by_name(type_module, type_name, func)
457 458
458 459 try:
459 460 oldfunc = self.lookup_by_type(typ)
460 461 except KeyError:
461 462 oldfunc = None
462 463
463 464 if func is not None:
464 465 self.type_printers[typ] = func
465 466
466 467 return oldfunc
467 468
468 469 def for_type_by_name(self, type_module, type_name, func=None):
469 470 """Add a format function for a type specified by the full dotted
470 471 module and name of the type, rather than the type of the object.
471 472
472 473 Parameters
473 474 ----------
474 475 type_module : str
475 476 The full dotted name of the module the type is defined in, like
476 477 ``numpy``.
478
477 479 type_name : str
478 480 The name of the type (the class name), like ``dtype``
481
479 482 func : callable
480 483 A callable for computing the format data.
481 484 `func` will be called with the object to be formatted,
482 485 and will return the raw data in this formatter's format.
483 486 Subclasses may use a different call signature for the
484 487 `func` argument.
485 488
486 489 If `func` is None or unspecified, there will be no change,
487 490 only returning the current value.
488 491
489 492 Returns
490 493 -------
491 494 oldfunc : callable
492 495 The currently registered callable.
493 496 If you are registering a new formatter,
494 497 this will be the previous value (to enable restoring later).
495 498 """
496 499 key = (type_module, type_name)
497 500
498 501 try:
499 502 oldfunc = self.lookup_by_type("%s.%s" % key)
500 503 except KeyError:
501 504 oldfunc = None
502 505
503 506 if func is not None:
504 507 self.deferred_printers[key] = func
505 508 return oldfunc
506 509
507 510 def pop(self, typ, default=_raise_key_error):
508 511 """Pop a formatter for the given type.
509 512
510 513 Parameters
511 514 ----------
512 515 typ : type or '__module__.__name__' string for a type
513 516 default : object
514 517 value to be returned if no formatter is registered for typ.
515 518
516 519 Returns
517 520 -------
518 521 obj : object
519 522 The last registered object for the type.
520 523
521 524 Raises
522 525 ------
523 526 KeyError if the type is not registered and default is not specified.
524 527 """
525 528
526 529 if isinstance(typ, str):
527 530 typ_key = tuple(typ.rsplit('.',1))
528 531 if typ_key not in self.deferred_printers:
529 532 # We may have it cached in the type map. We will have to
530 533 # iterate over all of the types to check.
531 534 for cls in self.type_printers:
532 535 if _mod_name_key(cls) == typ_key:
533 536 old = self.type_printers.pop(cls)
534 537 break
535 538 else:
536 539 old = default
537 540 else:
538 541 old = self.deferred_printers.pop(typ_key)
539 542 else:
540 543 if typ in self.type_printers:
541 544 old = self.type_printers.pop(typ)
542 545 else:
543 546 old = self.deferred_printers.pop(_mod_name_key(typ), default)
544 547 if old is _raise_key_error:
545 548 raise KeyError("No registered value for {0!r}".format(typ))
546 549 return old
547 550
548 551 def _in_deferred_types(self, cls):
549 552 """
550 553 Check if the given class is specified in the deferred type registry.
551 554
552 555 Successful matches will be moved to the regular type registry for future use.
553 556 """
554 557 mod = getattr(cls, '__module__', None)
555 558 name = getattr(cls, '__name__', None)
556 559 key = (mod, name)
557 560 if key in self.deferred_printers:
558 561 # Move the printer over to the regular registry.
559 562 printer = self.deferred_printers.pop(key)
560 563 self.type_printers[cls] = printer
561 564 return True
562 565 return False
563 566
564 567
565 568 class PlainTextFormatter(BaseFormatter):
566 569 """The default pretty-printer.
567 570
568 571 This uses :mod:`IPython.lib.pretty` to compute the format data of
569 572 the object. If the object cannot be pretty printed, :func:`repr` is used.
570 573 See the documentation of :mod:`IPython.lib.pretty` for details on
571 574 how to write pretty printers. Here is a simple example::
572 575
573 576 def dtype_pprinter(obj, p, cycle):
574 577 if cycle:
575 578 return p.text('dtype(...)')
576 579 if hasattr(obj, 'fields'):
577 580 if obj.fields is None:
578 581 p.text(repr(obj))
579 582 else:
580 583 p.begin_group(7, 'dtype([')
581 584 for i, field in enumerate(obj.descr):
582 585 if i > 0:
583 586 p.text(',')
584 587 p.breakable()
585 588 p.pretty(field)
586 589 p.end_group(7, '])')
587 590 """
588 591
589 592 # The format type of data returned.
590 593 format_type = Unicode('text/plain')
591 594
592 595 # This subclass ignores this attribute as it always need to return
593 596 # something.
594 597 enabled = Bool(True).tag(config=False)
595 598
596 599 max_seq_length = Integer(pretty.MAX_SEQ_LENGTH,
597 600 help="""Truncate large collections (lists, dicts, tuples, sets) to this size.
598 601
599 602 Set to 0 to disable truncation.
600 603 """
601 604 ).tag(config=True)
602 605
603 606 # Look for a _repr_pretty_ methods to use for pretty printing.
604 607 print_method = ObjectName('_repr_pretty_')
605 608
606 609 # Whether to pretty-print or not.
607 610 pprint = Bool(True).tag(config=True)
608 611
609 612 # Whether to be verbose or not.
610 613 verbose = Bool(False).tag(config=True)
611 614
612 615 # The maximum width.
613 616 max_width = Integer(79).tag(config=True)
614 617
615 618 # The newline character.
616 619 newline = Unicode('\n').tag(config=True)
617 620
618 621 # format-string for pprinting floats
619 622 float_format = Unicode('%r')
620 623 # setter for float precision, either int or direct format-string
621 624 float_precision = CUnicode('').tag(config=True)
622 625
623 626 @observe('float_precision')
624 627 def _float_precision_changed(self, change):
625 628 """float_precision changed, set float_format accordingly.
626 629
627 630 float_precision can be set by int or str.
628 631 This will set float_format, after interpreting input.
629 632 If numpy has been imported, numpy print precision will also be set.
630 633
631 634 integer `n` sets format to '%.nf', otherwise, format set directly.
632 635
633 636 An empty string returns to defaults (repr for float, 8 for numpy).
634 637
635 638 This parameter can be set via the '%precision' magic.
636 639 """
637 640 new = change['new']
638 641 if '%' in new:
639 642 # got explicit format string
640 643 fmt = new
641 644 try:
642 645 fmt%3.14159
643 646 except Exception as e:
644 647 raise ValueError("Precision must be int or format string, not %r"%new) from e
645 648 elif new:
646 649 # otherwise, should be an int
647 650 try:
648 651 i = int(new)
649 652 assert i >= 0
650 653 except ValueError as e:
651 654 raise ValueError("Precision must be int or format string, not %r"%new) from e
652 655 except AssertionError as e:
653 656 raise ValueError("int precision must be non-negative, not %r"%i) from e
654 657
655 658 fmt = '%%.%if'%i
656 659 if 'numpy' in sys.modules:
657 660 # set numpy precision if it has been imported
658 661 import numpy
659 662 numpy.set_printoptions(precision=i)
660 663 else:
661 664 # default back to repr
662 665 fmt = '%r'
663 666 if 'numpy' in sys.modules:
664 667 import numpy
665 668 # numpy default is 8
666 669 numpy.set_printoptions(precision=8)
667 670 self.float_format = fmt
668 671
669 672 # Use the default pretty printers from IPython.lib.pretty.
670 673 @default('singleton_printers')
671 674 def _singleton_printers_default(self):
672 675 return pretty._singleton_pprinters.copy()
673 676
674 677 @default('type_printers')
675 678 def _type_printers_default(self):
676 679 d = pretty._type_pprinters.copy()
677 680 d[float] = lambda obj,p,cycle: p.text(self.float_format%obj)
678 681 # if NumPy is used, set precision for its float64 type
679 682 if "numpy" in sys.modules:
680 683 import numpy
681 684
682 685 d[numpy.float64] = lambda obj, p, cycle: p.text(self.float_format % obj)
683 686 return d
684 687
685 688 @default('deferred_printers')
686 689 def _deferred_printers_default(self):
687 690 return pretty._deferred_type_pprinters.copy()
688 691
689 692 #### FormatterABC interface ####
690 693
691 694 @catch_format_error
692 695 def __call__(self, obj):
693 696 """Compute the pretty representation of the object."""
694 697 if not self.pprint:
695 698 return repr(obj)
696 699 else:
697 700 stream = StringIO()
698 701 printer = pretty.RepresentationPrinter(stream, self.verbose,
699 702 self.max_width, self.newline,
700 703 max_seq_length=self.max_seq_length,
701 704 singleton_pprinters=self.singleton_printers,
702 705 type_pprinters=self.type_printers,
703 706 deferred_pprinters=self.deferred_printers)
704 707 printer.pretty(obj)
705 708 printer.flush()
706 709 return stream.getvalue()
707 710
708 711
709 712 class HTMLFormatter(BaseFormatter):
710 713 """An HTML formatter.
711 714
712 715 To define the callables that compute the HTML representation of your
713 716 objects, define a :meth:`_repr_html_` method or use the :meth:`for_type`
714 717 or :meth:`for_type_by_name` methods to register functions that handle
715 718 this.
716 719
717 720 The return value of this formatter should be a valid HTML snippet that
718 721 could be injected into an existing DOM. It should *not* include the
719 722 ```<html>`` or ```<body>`` tags.
720 723 """
721 724 format_type = Unicode('text/html')
722 725
723 726 print_method = ObjectName('_repr_html_')
724 727
725 728
726 729 class MarkdownFormatter(BaseFormatter):
727 730 """A Markdown formatter.
728 731
729 732 To define the callables that compute the Markdown representation of your
730 733 objects, define a :meth:`_repr_markdown_` method or use the :meth:`for_type`
731 734 or :meth:`for_type_by_name` methods to register functions that handle
732 735 this.
733 736
734 737 The return value of this formatter should be a valid Markdown.
735 738 """
736 739 format_type = Unicode('text/markdown')
737 740
738 741 print_method = ObjectName('_repr_markdown_')
739 742
740 743 class SVGFormatter(BaseFormatter):
741 744 """An SVG formatter.
742 745
743 746 To define the callables that compute the SVG representation of your
744 747 objects, define a :meth:`_repr_svg_` method or use the :meth:`for_type`
745 748 or :meth:`for_type_by_name` methods to register functions that handle
746 749 this.
747 750
748 751 The return value of this formatter should be valid SVG enclosed in
749 752 ```<svg>``` tags, that could be injected into an existing DOM. It should
750 753 *not* include the ```<html>`` or ```<body>`` tags.
751 754 """
752 755 format_type = Unicode('image/svg+xml')
753 756
754 757 print_method = ObjectName('_repr_svg_')
755 758
756 759
757 760 class PNGFormatter(BaseFormatter):
758 761 """A PNG formatter.
759 762
760 763 To define the callables that compute the PNG representation of your
761 764 objects, define a :meth:`_repr_png_` method or use the :meth:`for_type`
762 765 or :meth:`for_type_by_name` methods to register functions that handle
763 766 this.
764 767
765 768 The return value of this formatter should be raw PNG data, *not*
766 769 base64 encoded.
767 770 """
768 771 format_type = Unicode('image/png')
769 772
770 773 print_method = ObjectName('_repr_png_')
771 774
772 775 _return_type = (bytes, str)
773 776
774 777
775 778 class JPEGFormatter(BaseFormatter):
776 779 """A JPEG formatter.
777 780
778 781 To define the callables that compute the JPEG representation of your
779 782 objects, define a :meth:`_repr_jpeg_` method or use the :meth:`for_type`
780 783 or :meth:`for_type_by_name` methods to register functions that handle
781 784 this.
782 785
783 786 The return value of this formatter should be raw JPEG data, *not*
784 787 base64 encoded.
785 788 """
786 789 format_type = Unicode('image/jpeg')
787 790
788 791 print_method = ObjectName('_repr_jpeg_')
789 792
790 793 _return_type = (bytes, str)
791 794
792 795
793 796 class LatexFormatter(BaseFormatter):
794 797 """A LaTeX formatter.
795 798
796 799 To define the callables that compute the LaTeX representation of your
797 800 objects, define a :meth:`_repr_latex_` method or use the :meth:`for_type`
798 801 or :meth:`for_type_by_name` methods to register functions that handle
799 802 this.
800 803
801 804 The return value of this formatter should be a valid LaTeX equation,
802 805 enclosed in either ```$```, ```$$``` or another LaTeX equation
803 806 environment.
804 807 """
805 808 format_type = Unicode('text/latex')
806 809
807 810 print_method = ObjectName('_repr_latex_')
808 811
809 812
810 813 class JSONFormatter(BaseFormatter):
811 814 """A JSON string formatter.
812 815
813 816 To define the callables that compute the JSONable representation of
814 817 your objects, define a :meth:`_repr_json_` method or use the :meth:`for_type`
815 818 or :meth:`for_type_by_name` methods to register functions that handle
816 819 this.
817 820
818 821 The return value of this formatter should be a JSONable list or dict.
819 822 JSON scalars (None, number, string) are not allowed, only dict or list containers.
820 823 """
821 824 format_type = Unicode('application/json')
822 825 _return_type = (list, dict)
823 826
824 827 print_method = ObjectName('_repr_json_')
825 828
826 829 def _check_return(self, r, obj):
827 830 """Check that a return value is appropriate
828 831
829 832 Return the value if so, None otherwise, warning if invalid.
830 833 """
831 834 if r is None:
832 835 return
833 836 md = None
834 837 if isinstance(r, tuple):
835 838 # unpack data, metadata tuple for type checking on first element
836 839 r, md = r
837 840
838 841 assert not isinstance(
839 842 r, str
840 843 ), "JSON-as-string has been deprecated since IPython < 3"
841 844
842 845 if md is not None:
843 846 # put the tuple back together
844 847 r = (r, md)
845 848 return super(JSONFormatter, self)._check_return(r, obj)
846 849
847 850
848 851 class JavascriptFormatter(BaseFormatter):
849 852 """A Javascript formatter.
850 853
851 854 To define the callables that compute the Javascript representation of
852 855 your objects, define a :meth:`_repr_javascript_` method or use the
853 856 :meth:`for_type` or :meth:`for_type_by_name` methods to register functions
854 857 that handle this.
855 858
856 859 The return value of this formatter should be valid Javascript code and
857 860 should *not* be enclosed in ```<script>``` tags.
858 861 """
859 862 format_type = Unicode('application/javascript')
860 863
861 864 print_method = ObjectName('_repr_javascript_')
862 865
863 866
864 867 class PDFFormatter(BaseFormatter):
865 868 """A PDF formatter.
866 869
867 870 To define the callables that compute the PDF representation of your
868 871 objects, define a :meth:`_repr_pdf_` method or use the :meth:`for_type`
869 872 or :meth:`for_type_by_name` methods to register functions that handle
870 873 this.
871 874
872 875 The return value of this formatter should be raw PDF data, *not*
873 876 base64 encoded.
874 877 """
875 878 format_type = Unicode('application/pdf')
876 879
877 880 print_method = ObjectName('_repr_pdf_')
878 881
879 882 _return_type = (bytes, str)
880 883
881 884 class IPythonDisplayFormatter(BaseFormatter):
882 885 """An escape-hatch Formatter for objects that know how to display themselves.
883 886
884 887 To define the callables that compute the representation of your
885 888 objects, define a :meth:`_ipython_display_` method or use the :meth:`for_type`
886 889 or :meth:`for_type_by_name` methods to register functions that handle
887 890 this. Unlike mime-type displays, this method should not return anything,
888 891 instead calling any appropriate display methods itself.
889 892
890 893 This display formatter has highest priority.
891 894 If it fires, no other display formatter will be called.
892 895
893 896 Prior to IPython 6.1, `_ipython_display_` was the only way to display custom mime-types
894 897 without registering a new Formatter.
895 898
896 899 IPython 6.1 introduces `_repr_mimebundle_` for displaying custom mime-types,
897 900 so `_ipython_display_` should only be used for objects that require unusual
898 901 display patterns, such as multiple display calls.
899 902 """
900 903 print_method = ObjectName('_ipython_display_')
901 904 _return_type = (type(None), bool)
902 905
903 906 @catch_format_error
904 907 def __call__(self, obj):
905 908 """Compute the format for an object."""
906 909 if self.enabled:
907 910 # lookup registered printer
908 911 try:
909 912 printer = self.lookup(obj)
910 913 except KeyError:
911 914 pass
912 915 else:
913 916 printer(obj)
914 917 return True
915 918 # Finally look for special method names
916 919 method = get_real_method(obj, self.print_method)
917 920 if method is not None:
918 921 method()
919 922 return True
920 923
921 924
922 925 class MimeBundleFormatter(BaseFormatter):
923 926 """A Formatter for arbitrary mime-types.
924 927
925 928 Unlike other `_repr_<mimetype>_` methods,
926 929 `_repr_mimebundle_` should return mime-bundle data,
927 930 either the mime-keyed `data` dictionary or the tuple `(data, metadata)`.
928 931 Any mime-type is valid.
929 932
930 933 To define the callables that compute the mime-bundle representation of your
931 934 objects, define a :meth:`_repr_mimebundle_` method or use the :meth:`for_type`
932 935 or :meth:`for_type_by_name` methods to register functions that handle
933 936 this.
934 937
935 938 .. versionadded:: 6.1
936 939 """
937 940 print_method = ObjectName('_repr_mimebundle_')
938 941 _return_type = dict
939 942
940 943 def _check_return(self, r, obj):
941 944 r = super(MimeBundleFormatter, self)._check_return(r, obj)
942 945 # always return (data, metadata):
943 946 if r is None:
944 947 return {}, {}
945 948 if not isinstance(r, tuple):
946 949 return r, {}
947 950 return r
948 951
949 952 @catch_format_error
950 953 def __call__(self, obj, include=None, exclude=None):
951 954 """Compute the format for an object.
952 955
953 956 Identical to parent's method but we pass extra parameters to the method.
954 957
955 958 Unlike other _repr_*_ `_repr_mimebundle_` should allow extra kwargs, in
956 959 particular `include` and `exclude`.
957 960 """
958 961 if self.enabled:
959 962 # lookup registered printer
960 963 try:
961 964 printer = self.lookup(obj)
962 965 except KeyError:
963 966 pass
964 967 else:
965 968 return printer(obj)
966 969 # Finally look for special method names
967 970 method = get_real_method(obj, self.print_method)
968 971
969 972 if method is not None:
970 973 return method(include=include, exclude=exclude)
971 974 return None
972 975 else:
973 976 return None
974 977
975 978
976 979 FormatterABC.register(BaseFormatter)
977 980 FormatterABC.register(PlainTextFormatter)
978 981 FormatterABC.register(HTMLFormatter)
979 982 FormatterABC.register(MarkdownFormatter)
980 983 FormatterABC.register(SVGFormatter)
981 984 FormatterABC.register(PNGFormatter)
982 985 FormatterABC.register(PDFFormatter)
983 986 FormatterABC.register(JPEGFormatter)
984 987 FormatterABC.register(LatexFormatter)
985 988 FormatterABC.register(JSONFormatter)
986 989 FormatterABC.register(JavascriptFormatter)
987 990 FormatterABC.register(IPythonDisplayFormatter)
988 991 FormatterABC.register(MimeBundleFormatter)
989 992
990 993
991 994 def format_display_data(obj, include=None, exclude=None):
992 995 """Return a format data dict for an object.
993 996
994 997 By default all format types will be computed.
995 998
996 999 Parameters
997 1000 ----------
998 1001 obj : object
999 1002 The Python object whose format data will be computed.
1000 1003
1001 1004 Returns
1002 1005 -------
1003 1006 format_dict : dict
1004 1007 A dictionary of key/value pairs, one or each format that was
1005 1008 generated for the object. The keys are the format types, which
1006 1009 will usually be MIME type strings and the values and JSON'able
1007 1010 data structure containing the raw data for the representation in
1008 1011 that format.
1009 1012 include : list or tuple, optional
1010 1013 A list of format type strings (MIME types) to include in the
1011 1014 format data dict. If this is set *only* the format types included
1012 1015 in this list will be computed.
1013 1016 exclude : list or tuple, optional
1014 1017 A list of format type string (MIME types) to exclude in the format
1015 1018 data dict. If this is set all format types will be computed,
1016 1019 except for those included in this argument.
1017 1020 """
1018 1021 from .interactiveshell import InteractiveShell
1019 1022
1020 1023 return InteractiveShell.instance().display_formatter.format(
1021 1024 obj,
1022 1025 include,
1023 1026 exclude
1024 1027 )
@@ -1,24 +1,24 b''
1 1 # encoding: utf-8
2 2 """Simple function to call to get the current InteractiveShell instance
3 3 """
4 4
5 5 #-----------------------------------------------------------------------------
6 6 # Copyright (C) 2013 The IPython Development Team
7 7 #
8 8 # Distributed under the terms of the BSD License. The full license is in
9 9 # the file COPYING, distributed as part of this software.
10 10 #-----------------------------------------------------------------------------
11 11
12 12 #-----------------------------------------------------------------------------
13 13 # Classes and functions
14 14 #-----------------------------------------------------------------------------
15 15
16 16
17 17 def get_ipython():
18 18 """Get the global InteractiveShell instance.
19
19
20 20 Returns None if no InteractiveShell instance is registered.
21 21 """
22 22 from IPython.core.interactiveshell import InteractiveShell
23 23 if InteractiveShell.initialized():
24 24 return InteractiveShell.instance()
@@ -1,913 +1,907 b''
1 1 """ History related magics and functionality """
2 2
3 3 # Copyright (c) IPython Development Team.
4 4 # Distributed under the terms of the Modified BSD License.
5 5
6 6
7 7 import atexit
8 8 import datetime
9 9 from pathlib import Path
10 10 import re
11 11 import sqlite3
12 12 import threading
13 13
14 14 from traitlets.config.configurable import LoggingConfigurable
15 15 from decorator import decorator
16 16 from IPython.utils.decorators import undoc
17 17 from IPython.paths import locate_profile
18 18 from traitlets import (
19 19 Any,
20 20 Bool,
21 21 Dict,
22 22 Instance,
23 23 Integer,
24 24 List,
25 25 Unicode,
26 26 Union,
27 27 TraitError,
28 28 default,
29 29 observe,
30 30 )
31 31
32 32 #-----------------------------------------------------------------------------
33 33 # Classes and functions
34 34 #-----------------------------------------------------------------------------
35 35
36 36 @undoc
37 37 class DummyDB(object):
38 38 """Dummy DB that will act as a black hole for history.
39 39
40 40 Only used in the absence of sqlite"""
41 41 def execute(*args, **kwargs):
42 42 return []
43 43
44 44 def commit(self, *args, **kwargs):
45 45 pass
46 46
47 47 def __enter__(self, *args, **kwargs):
48 48 pass
49 49
50 50 def __exit__(self, *args, **kwargs):
51 51 pass
52 52
53 53
54 54 @decorator
55 55 def only_when_enabled(f, self, *a, **kw):
56 56 """Decorator: return an empty list in the absence of sqlite."""
57 57 if not self.enabled:
58 58 return []
59 59 else:
60 60 return f(self, *a, **kw)
61 61
62 62
63 63 # use 16kB as threshold for whether a corrupt history db should be saved
64 64 # that should be at least 100 entries or so
65 65 _SAVE_DB_SIZE = 16384
66 66
67 67 @decorator
68 68 def catch_corrupt_db(f, self, *a, **kw):
69 69 """A decorator which wraps HistoryAccessor method calls to catch errors from
70 70 a corrupt SQLite database, move the old database out of the way, and create
71 71 a new one.
72 72
73 73 We avoid clobbering larger databases because this may be triggered due to filesystem issues,
74 74 not just a corrupt file.
75 75 """
76 76 try:
77 77 return f(self, *a, **kw)
78 78 except (sqlite3.DatabaseError, sqlite3.OperationalError) as e:
79 79 self._corrupt_db_counter += 1
80 80 self.log.error("Failed to open SQLite history %s (%s).", self.hist_file, e)
81 81 if self.hist_file != ':memory:':
82 82 if self._corrupt_db_counter > self._corrupt_db_limit:
83 83 self.hist_file = ':memory:'
84 84 self.log.error("Failed to load history too many times, history will not be saved.")
85 85 elif self.hist_file.is_file():
86 86 # move the file out of the way
87 87 base = str(self.hist_file.parent / self.hist_file.stem)
88 88 ext = self.hist_file.suffix
89 89 size = self.hist_file.stat().st_size
90 90 if size >= _SAVE_DB_SIZE:
91 91 # if there's significant content, avoid clobbering
92 92 now = datetime.datetime.now().isoformat().replace(':', '.')
93 93 newpath = base + '-corrupt-' + now + ext
94 94 # don't clobber previous corrupt backups
95 95 for i in range(100):
96 96 if not Path(newpath).exists():
97 97 break
98 98 else:
99 99 newpath = base + '-corrupt-' + now + (u'-%i' % i) + ext
100 100 else:
101 101 # not much content, possibly empty; don't worry about clobbering
102 102 # maybe we should just delete it?
103 103 newpath = base + '-corrupt' + ext
104 104 self.hist_file.rename(newpath)
105 105 self.log.error("History file was moved to %s and a new file created.", newpath)
106 106 self.init_db()
107 107 return []
108 108 else:
109 109 # Failed with :memory:, something serious is wrong
110 110 raise
111 111
112 112
113 113 class HistoryAccessorBase(LoggingConfigurable):
114 114 """An abstract class for History Accessors """
115 115
116 116 def get_tail(self, n=10, raw=True, output=False, include_latest=False):
117 117 raise NotImplementedError
118 118
119 119 def search(self, pattern="*", raw=True, search_raw=True,
120 120 output=False, n=None, unique=False):
121 121 raise NotImplementedError
122 122
123 123 def get_range(self, session, start=1, stop=None, raw=True,output=False):
124 124 raise NotImplementedError
125 125
126 126 def get_range_by_str(self, rangestr, raw=True, output=False):
127 127 raise NotImplementedError
128 128
129 129
130 130 class HistoryAccessor(HistoryAccessorBase):
131 131 """Access the history database without adding to it.
132 132
133 133 This is intended for use by standalone history tools. IPython shells use
134 134 HistoryManager, below, which is a subclass of this."""
135 135
136 136 # counter for init_db retries, so we don't keep trying over and over
137 137 _corrupt_db_counter = 0
138 138 # after two failures, fallback on :memory:
139 139 _corrupt_db_limit = 2
140 140
141 141 # String holding the path to the history file
142 142 hist_file = Union(
143 143 [Instance(Path), Unicode()],
144 144 help="""Path to file to use for SQLite history database.
145 145
146 146 By default, IPython will put the history database in the IPython
147 147 profile directory. If you would rather share one history among
148 148 profiles, you can set this value in each, so that they are consistent.
149 149
150 150 Due to an issue with fcntl, SQLite is known to misbehave on some NFS
151 151 mounts. If you see IPython hanging, try setting this to something on a
152 152 local disk, e.g::
153 153
154 154 ipython --HistoryManager.hist_file=/tmp/ipython_hist.sqlite
155 155
156 156 you can also use the specific value `:memory:` (including the colon
157 157 at both end but not the back ticks), to avoid creating an history file.
158 158
159 159 """,
160 160 ).tag(config=True)
161 161
162 162 enabled = Bool(True,
163 163 help="""enable the SQLite history
164 164
165 165 set enabled=False to disable the SQLite history,
166 166 in which case there will be no stored history, no SQLite connection,
167 167 and no background saving thread. This may be necessary in some
168 168 threaded environments where IPython is embedded.
169 169 """
170 170 ).tag(config=True)
171 171
172 172 connection_options = Dict(
173 173 help="""Options for configuring the SQLite connection
174 174
175 175 These options are passed as keyword args to sqlite3.connect
176 176 when establishing database connections.
177 177 """
178 178 ).tag(config=True)
179 179
180 180 # The SQLite database
181 181 db = Any()
182 182 @observe('db')
183 183 def _db_changed(self, change):
184 184 """validate the db, since it can be an Instance of two different types"""
185 185 new = change['new']
186 186 connection_types = (DummyDB, sqlite3.Connection)
187 187 if not isinstance(new, connection_types):
188 188 msg = "%s.db must be sqlite3 Connection or DummyDB, not %r" % \
189 189 (self.__class__.__name__, new)
190 190 raise TraitError(msg)
191 191
192 192 def __init__(self, profile="default", hist_file="", **traits):
193 193 """Create a new history accessor.
194 194
195 195 Parameters
196 196 ----------
197 197 profile : str
198 The name of the profile from which to open history.
198 The name of the profile from which to open history.
199 199 hist_file : str
200 Path to an SQLite history database stored by IPython. If specified,
201 hist_file overrides profile.
200 Path to an SQLite history database stored by IPython. If specified,
201 hist_file overrides profile.
202 202 config : :class:`~traitlets.config.loader.Config`
203 Config object. hist_file can also be set through this.
203 Config object. hist_file can also be set through this.
204 204 """
205 205 # We need a pointer back to the shell for various tasks.
206 206 super(HistoryAccessor, self).__init__(**traits)
207 207 # defer setting hist_file from kwarg until after init,
208 208 # otherwise the default kwarg value would clobber any value
209 209 # set by config
210 210 if hist_file:
211 211 self.hist_file = hist_file
212 212
213 213 try:
214 214 self.hist_file
215 215 except TraitError:
216 216 # No one has set the hist_file, yet.
217 217 self.hist_file = self._get_hist_file_name(profile)
218 218
219 219 self.init_db()
220 220
221 221 def _get_hist_file_name(self, profile='default'):
222 222 """Find the history file for the given profile name.
223 223
224 224 This is overridden by the HistoryManager subclass, to use the shell's
225 225 active profile.
226 226
227 227 Parameters
228 228 ----------
229 229 profile : str
230 The name of a profile which has a history file.
230 The name of a profile which has a history file.
231 231 """
232 232 return Path(locate_profile(profile)) / "history.sqlite"
233 233
234 234 @catch_corrupt_db
235 235 def init_db(self):
236 236 """Connect to the database, and create tables if necessary."""
237 237 if not self.enabled:
238 238 self.db = DummyDB()
239 239 return
240 240
241 241 # use detect_types so that timestamps return datetime objects
242 242 kwargs = dict(detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
243 243 kwargs.update(self.connection_options)
244 244 self.db = sqlite3.connect(str(self.hist_file), **kwargs)
245 245 self.db.execute("""CREATE TABLE IF NOT EXISTS sessions (session integer
246 246 primary key autoincrement, start timestamp,
247 247 end timestamp, num_cmds integer, remark text)""")
248 248 self.db.execute("""CREATE TABLE IF NOT EXISTS history
249 249 (session integer, line integer, source text, source_raw text,
250 250 PRIMARY KEY (session, line))""")
251 251 # Output history is optional, but ensure the table's there so it can be
252 252 # enabled later.
253 253 self.db.execute("""CREATE TABLE IF NOT EXISTS output_history
254 254 (session integer, line integer, output text,
255 255 PRIMARY KEY (session, line))""")
256 256 self.db.commit()
257 257 # success! reset corrupt db count
258 258 self._corrupt_db_counter = 0
259 259
260 260 def writeout_cache(self):
261 261 """Overridden by HistoryManager to dump the cache before certain
262 262 database lookups."""
263 263 pass
264 264
265 265 ## -------------------------------
266 266 ## Methods for retrieving history:
267 267 ## -------------------------------
268 268 def _run_sql(self, sql, params, raw=True, output=False, latest=False):
269 269 """Prepares and runs an SQL query for the history database.
270 270
271 271 Parameters
272 272 ----------
273 273 sql : str
274 Any filtering expressions to go after SELECT ... FROM ...
274 Any filtering expressions to go after SELECT ... FROM ...
275 275 params : tuple
276 Parameters passed to the SQL query (to replace "?")
276 Parameters passed to the SQL query (to replace "?")
277 277 raw, output : bool
278 See :meth:`get_range`
278 See :meth:`get_range`
279 279 latest : bool
280 Select rows with max (session, line)
280 Select rows with max (session, line)
281 281
282 282 Returns
283 283 -------
284 284 Tuples as :meth:`get_range`
285 285 """
286 286 toget = 'source_raw' if raw else 'source'
287 287 sqlfrom = "history"
288 288 if output:
289 289 sqlfrom = "history LEFT JOIN output_history USING (session, line)"
290 290 toget = "history.%s, output_history.output" % toget
291 291 if latest:
292 292 toget += ", MAX(session * 128 * 1024 + line)"
293 293 cur = self.db.execute("SELECT session, line, %s FROM %s " %\
294 294 (toget, sqlfrom) + sql, params)
295 295 if latest:
296 296 cur = (row[:-1] for row in cur)
297 297 if output: # Regroup into 3-tuples, and parse JSON
298 298 return ((ses, lin, (inp, out)) for ses, lin, inp, out in cur)
299 299 return cur
300 300
301 301 @only_when_enabled
302 302 @catch_corrupt_db
303 303 def get_session_info(self, session):
304 304 """Get info about a session.
305 305
306 306 Parameters
307 307 ----------
308
309 308 session : int
310 309 Session number to retrieve.
311 310
312 311 Returns
313 312 -------
314
315 313 session_id : int
316 Session ID number
314 Session ID number
317 315 start : datetime
318 Timestamp for the start of the session.
316 Timestamp for the start of the session.
319 317 end : datetime
320 Timestamp for the end of the session, or None if IPython crashed.
318 Timestamp for the end of the session, or None if IPython crashed.
321 319 num_cmds : int
322 Number of commands run, or None if IPython crashed.
320 Number of commands run, or None if IPython crashed.
323 321 remark : unicode
324 A manually set description.
322 A manually set description.
325 323 """
326 324 query = "SELECT * from sessions where session == ?"
327 325 return self.db.execute(query, (session,)).fetchone()
328 326
329 327 @catch_corrupt_db
330 328 def get_last_session_id(self):
331 329 """Get the last session ID currently in the database.
332 330
333 331 Within IPython, this should be the same as the value stored in
334 332 :attr:`HistoryManager.session_number`.
335 333 """
336 334 for record in self.get_tail(n=1, include_latest=True):
337 335 return record[0]
338 336
339 337 @catch_corrupt_db
340 338 def get_tail(self, n=10, raw=True, output=False, include_latest=False):
341 339 """Get the last n lines from the history database.
342 340
343 341 Parameters
344 342 ----------
345 343 n : int
346 The number of lines to get
344 The number of lines to get
347 345 raw, output : bool
348 See :meth:`get_range`
346 See :meth:`get_range`
349 347 include_latest : bool
350 If False (default), n+1 lines are fetched, and the latest one
351 is discarded. This is intended to be used where the function
352 is called by a user command, which it should not return.
348 If False (default), n+1 lines are fetched, and the latest one
349 is discarded. This is intended to be used where the function
350 is called by a user command, which it should not return.
353 351
354 352 Returns
355 353 -------
356 354 Tuples as :meth:`get_range`
357 355 """
358 356 self.writeout_cache()
359 357 if not include_latest:
360 358 n += 1
361 359 cur = self._run_sql("ORDER BY session DESC, line DESC LIMIT ?",
362 360 (n,), raw=raw, output=output)
363 361 if not include_latest:
364 362 return reversed(list(cur)[1:])
365 363 return reversed(list(cur))
366 364
367 365 @catch_corrupt_db
368 366 def search(self, pattern="*", raw=True, search_raw=True,
369 367 output=False, n=None, unique=False):
370 368 """Search the database using unix glob-style matching (wildcards
371 369 * and ?).
372 370
373 371 Parameters
374 372 ----------
375 373 pattern : str
376 The wildcarded pattern to match when searching
374 The wildcarded pattern to match when searching
377 375 search_raw : bool
378 If True, search the raw input, otherwise, the parsed input
376 If True, search the raw input, otherwise, the parsed input
379 377 raw, output : bool
380 See :meth:`get_range`
378 See :meth:`get_range`
381 379 n : None or int
382 If an integer is given, it defines the limit of
383 returned entries.
380 If an integer is given, it defines the limit of
381 returned entries.
384 382 unique : bool
385 When it is true, return only unique entries.
383 When it is true, return only unique entries.
386 384
387 385 Returns
388 386 -------
389 387 Tuples as :meth:`get_range`
390 388 """
391 389 tosearch = "source_raw" if search_raw else "source"
392 390 if output:
393 391 tosearch = "history." + tosearch
394 392 self.writeout_cache()
395 393 sqlform = "WHERE %s GLOB ?" % tosearch
396 394 params = (pattern,)
397 395 if unique:
398 396 sqlform += ' GROUP BY {0}'.format(tosearch)
399 397 if n is not None:
400 398 sqlform += " ORDER BY session DESC, line DESC LIMIT ?"
401 399 params += (n,)
402 400 elif unique:
403 401 sqlform += " ORDER BY session, line"
404 402 cur = self._run_sql(sqlform, params, raw=raw, output=output, latest=unique)
405 403 if n is not None:
406 404 return reversed(list(cur))
407 405 return cur
408 406
409 407 @catch_corrupt_db
410 408 def get_range(self, session, start=1, stop=None, raw=True,output=False):
411 409 """Retrieve input by session.
412 410
413 411 Parameters
414 412 ----------
415 413 session : int
416 414 Session number to retrieve.
417 415 start : int
418 416 First line to retrieve.
419 417 stop : int
420 418 End of line range (excluded from output itself). If None, retrieve
421 419 to the end of the session.
422 420 raw : bool
423 421 If True, return untranslated input
424 422 output : bool
425 423 If True, attempt to include output. This will be 'real' Python
426 424 objects for the current session, or text reprs from previous
427 425 sessions if db_log_output was enabled at the time. Where no output
428 426 is found, None is used.
429 427
430 428 Returns
431 429 -------
432 430 entries
433 An iterator over the desired lines. Each line is a 3-tuple, either
434 (session, line, input) if output is False, or
435 (session, line, (input, output)) if output is True.
431 An iterator over the desired lines. Each line is a 3-tuple, either
432 (session, line, input) if output is False, or
433 (session, line, (input, output)) if output is True.
436 434 """
437 435 if stop:
438 436 lineclause = "line >= ? AND line < ?"
439 437 params = (session, start, stop)
440 438 else:
441 439 lineclause = "line>=?"
442 440 params = (session, start)
443 441
444 442 return self._run_sql("WHERE session==? AND %s" % lineclause,
445 443 params, raw=raw, output=output)
446 444
447 445 def get_range_by_str(self, rangestr, raw=True, output=False):
448 446 """Get lines of history from a string of ranges, as used by magic
449 447 commands %hist, %save, %macro, etc.
450 448
451 449 Parameters
452 450 ----------
453 451 rangestr : str
454 A string specifying ranges, e.g. "5 ~2/1-4". If empty string is used,
455 this will return everything from current session's history.
452 A string specifying ranges, e.g. "5 ~2/1-4". If empty string is used,
453 this will return everything from current session's history.
456 454
457 See the documentation of :func:`%history` for the full details.
455 See the documentation of :func:`%history` for the full details.
458 456
459 457 raw, output : bool
460 As :meth:`get_range`
458 As :meth:`get_range`
461 459
462 460 Returns
463 461 -------
464 462 Tuples as :meth:`get_range`
465 463 """
466 464 for sess, s, e in extract_hist_ranges(rangestr):
467 465 for line in self.get_range(sess, s, e, raw=raw, output=output):
468 466 yield line
469 467
470 468
471 469 class HistoryManager(HistoryAccessor):
472 470 """A class to organize all history-related functionality in one place.
473 471 """
474 472 # Public interface
475 473
476 474 # An instance of the IPython shell we are attached to
477 475 shell = Instance('IPython.core.interactiveshell.InteractiveShellABC',
478 476 allow_none=True)
479 477 # Lists to hold processed and raw history. These start with a blank entry
480 478 # so that we can index them starting from 1
481 479 input_hist_parsed = List([""])
482 480 input_hist_raw = List([""])
483 481 # A list of directories visited during session
484 482 dir_hist = List()
485 483 @default('dir_hist')
486 484 def _dir_hist_default(self):
487 485 try:
488 486 return [Path.cwd()]
489 487 except OSError:
490 488 return []
491 489
492 490 # A dict of output history, keyed with ints from the shell's
493 491 # execution count.
494 492 output_hist = Dict()
495 493 # The text/plain repr of outputs.
496 494 output_hist_reprs = Dict()
497 495
498 496 # The number of the current session in the history database
499 497 session_number = Integer()
500 498
501 499 db_log_output = Bool(False,
502 500 help="Should the history database include output? (default: no)"
503 501 ).tag(config=True)
504 502 db_cache_size = Integer(0,
505 503 help="Write to database every x commands (higher values save disk access & power).\n"
506 504 "Values of 1 or less effectively disable caching."
507 505 ).tag(config=True)
508 506 # The input and output caches
509 507 db_input_cache = List()
510 508 db_output_cache = List()
511 509
512 510 # History saving in separate thread
513 511 save_thread = Instance('IPython.core.history.HistorySavingThread',
514 512 allow_none=True)
515 513 save_flag = Instance(threading.Event, allow_none=True)
516 514
517 515 # Private interface
518 516 # Variables used to store the three last inputs from the user. On each new
519 517 # history update, we populate the user's namespace with these, shifted as
520 518 # necessary.
521 519 _i00 = Unicode(u'')
522 520 _i = Unicode(u'')
523 521 _ii = Unicode(u'')
524 522 _iii = Unicode(u'')
525 523
526 524 # A regex matching all forms of the exit command, so that we don't store
527 525 # them in the history (it's annoying to rewind the first entry and land on
528 526 # an exit call).
529 527 _exit_re = re.compile(r"(exit|quit)(\s*\(.*\))?$")
530 528
531 529 def __init__(self, shell=None, config=None, **traits):
532 530 """Create a new history manager associated with a shell instance.
533 531 """
534 532 # We need a pointer back to the shell for various tasks.
535 533 super(HistoryManager, self).__init__(shell=shell, config=config,
536 534 **traits)
537 535 self.save_flag = threading.Event()
538 536 self.db_input_cache_lock = threading.Lock()
539 537 self.db_output_cache_lock = threading.Lock()
540 538
541 539 try:
542 540 self.new_session()
543 541 except sqlite3.OperationalError:
544 542 self.log.error("Failed to create history session in %s. History will not be saved.",
545 543 self.hist_file, exc_info=True)
546 544 self.hist_file = ':memory:'
547 545
548 546 if self.enabled and self.hist_file != ':memory:':
549 547 self.save_thread = HistorySavingThread(self)
550 548 self.save_thread.start()
551 549
552 550 def _get_hist_file_name(self, profile=None):
553 551 """Get default history file name based on the Shell's profile.
554 552
555 553 The profile parameter is ignored, but must exist for compatibility with
556 554 the parent class."""
557 555 profile_dir = self.shell.profile_dir.location
558 556 return Path(profile_dir) / "history.sqlite"
559 557
560 558 @only_when_enabled
561 559 def new_session(self, conn=None):
562 560 """Get a new session number."""
563 561 if conn is None:
564 562 conn = self.db
565 563
566 564 with conn:
567 565 cur = conn.execute("""INSERT INTO sessions VALUES (NULL, ?, NULL,
568 566 NULL, "") """, (datetime.datetime.now(),))
569 567 self.session_number = cur.lastrowid
570 568
571 569 def end_session(self):
572 570 """Close the database session, filling in the end time and line count."""
573 571 self.writeout_cache()
574 572 with self.db:
575 573 self.db.execute("""UPDATE sessions SET end=?, num_cmds=? WHERE
576 574 session==?""", (datetime.datetime.now(),
577 575 len(self.input_hist_parsed)-1, self.session_number))
578 576 self.session_number = 0
579 577
580 578 def name_session(self, name):
581 579 """Give the current session a name in the history database."""
582 580 with self.db:
583 581 self.db.execute("UPDATE sessions SET remark=? WHERE session==?",
584 582 (name, self.session_number))
585 583
586 584 def reset(self, new_session=True):
587 585 """Clear the session history, releasing all object references, and
588 586 optionally open a new session."""
589 587 self.output_hist.clear()
590 588 # The directory history can't be completely empty
591 589 self.dir_hist[:] = [Path.cwd()]
592 590
593 591 if new_session:
594 592 if self.session_number:
595 593 self.end_session()
596 594 self.input_hist_parsed[:] = [""]
597 595 self.input_hist_raw[:] = [""]
598 596 self.new_session()
599 597
600 598 # ------------------------------
601 599 # Methods for retrieving history
602 600 # ------------------------------
603 601 def get_session_info(self, session=0):
604 602 """Get info about a session.
605 603
606 604 Parameters
607 605 ----------
608
609 606 session : int
610 607 Session number to retrieve. The current session is 0, and negative
611 608 numbers count back from current session, so -1 is the previous session.
612 609
613 610 Returns
614 611 -------
615
616 612 session_id : int
617 Session ID number
613 Session ID number
618 614 start : datetime
619 Timestamp for the start of the session.
615 Timestamp for the start of the session.
620 616 end : datetime
621 Timestamp for the end of the session, or None if IPython crashed.
617 Timestamp for the end of the session, or None if IPython crashed.
622 618 num_cmds : int
623 Number of commands run, or None if IPython crashed.
619 Number of commands run, or None if IPython crashed.
624 620 remark : unicode
625 A manually set description.
621 A manually set description.
626 622 """
627 623 if session <= 0:
628 624 session += self.session_number
629 625
630 626 return super(HistoryManager, self).get_session_info(session=session)
631 627
632 628 def _get_range_session(self, start=1, stop=None, raw=True, output=False):
633 629 """Get input and output history from the current session. Called by
634 630 get_range, and takes similar parameters."""
635 631 input_hist = self.input_hist_raw if raw else self.input_hist_parsed
636 632
637 633 n = len(input_hist)
638 634 if start < 0:
639 635 start += n
640 636 if not stop or (stop > n):
641 637 stop = n
642 638 elif stop < 0:
643 639 stop += n
644 640
645 641 for i in range(start, stop):
646 642 if output:
647 643 line = (input_hist[i], self.output_hist_reprs.get(i))
648 644 else:
649 645 line = input_hist[i]
650 646 yield (0, i, line)
651 647
652 648 def get_range(self, session=0, start=1, stop=None, raw=True,output=False):
653 649 """Retrieve input by session.
654 650
655 651 Parameters
656 652 ----------
657 653 session : int
658 654 Session number to retrieve. The current session is 0, and negative
659 655 numbers count back from current session, so -1 is previous session.
660 656 start : int
661 657 First line to retrieve.
662 658 stop : int
663 659 End of line range (excluded from output itself). If None, retrieve
664 660 to the end of the session.
665 661 raw : bool
666 662 If True, return untranslated input
667 663 output : bool
668 664 If True, attempt to include output. This will be 'real' Python
669 665 objects for the current session, or text reprs from previous
670 666 sessions if db_log_output was enabled at the time. Where no output
671 667 is found, None is used.
672 668
673 669 Returns
674 670 -------
675 671 entries
676 An iterator over the desired lines. Each line is a 3-tuple, either
677 (session, line, input) if output is False, or
678 (session, line, (input, output)) if output is True.
672 An iterator over the desired lines. Each line is a 3-tuple, either
673 (session, line, input) if output is False, or
674 (session, line, (input, output)) if output is True.
679 675 """
680 676 if session <= 0:
681 677 session += self.session_number
682 678 if session==self.session_number: # Current session
683 679 return self._get_range_session(start, stop, raw, output)
684 680 return super(HistoryManager, self).get_range(session, start, stop, raw,
685 681 output)
686 682
687 683 ## ----------------------------
688 684 ## Methods for storing history:
689 685 ## ----------------------------
690 686 def store_inputs(self, line_num, source, source_raw=None):
691 687 """Store source and raw input in history and create input cache
692 688 variables ``_i*``.
693 689
694 690 Parameters
695 691 ----------
696 692 line_num : int
697 The prompt number of this input.
698
693 The prompt number of this input.
699 694 source : str
700 Python input.
701
695 Python input.
702 696 source_raw : str, optional
703 If given, this is the raw input without any IPython transformations
704 applied to it. If not given, ``source`` is used.
697 If given, this is the raw input without any IPython transformations
698 applied to it. If not given, ``source`` is used.
705 699 """
706 700 if source_raw is None:
707 701 source_raw = source
708 702 source = source.rstrip('\n')
709 703 source_raw = source_raw.rstrip('\n')
710 704
711 705 # do not store exit/quit commands
712 706 if self._exit_re.match(source_raw.strip()):
713 707 return
714 708
715 709 self.input_hist_parsed.append(source)
716 710 self.input_hist_raw.append(source_raw)
717 711
718 712 with self.db_input_cache_lock:
719 713 self.db_input_cache.append((line_num, source, source_raw))
720 714 # Trigger to flush cache and write to DB.
721 715 if len(self.db_input_cache) >= self.db_cache_size:
722 716 self.save_flag.set()
723 717
724 718 # update the auto _i variables
725 719 self._iii = self._ii
726 720 self._ii = self._i
727 721 self._i = self._i00
728 722 self._i00 = source_raw
729 723
730 724 # hackish access to user namespace to create _i1,_i2... dynamically
731 725 new_i = '_i%s' % line_num
732 726 to_main = {'_i': self._i,
733 727 '_ii': self._ii,
734 728 '_iii': self._iii,
735 729 new_i : self._i00 }
736 730
737 731 if self.shell is not None:
738 732 self.shell.push(to_main, interactive=False)
739 733
740 734 def store_output(self, line_num):
741 735 """If database output logging is enabled, this saves all the
742 736 outputs from the indicated prompt number to the database. It's
743 737 called by run_cell after code has been executed.
744 738
745 739 Parameters
746 740 ----------
747 741 line_num : int
748 The line number from which to save outputs
742 The line number from which to save outputs
749 743 """
750 744 if (not self.db_log_output) or (line_num not in self.output_hist_reprs):
751 745 return
752 746 output = self.output_hist_reprs[line_num]
753 747
754 748 with self.db_output_cache_lock:
755 749 self.db_output_cache.append((line_num, output))
756 750 if self.db_cache_size <= 1:
757 751 self.save_flag.set()
758 752
759 753 def _writeout_input_cache(self, conn):
760 754 with conn:
761 755 for line in self.db_input_cache:
762 756 conn.execute("INSERT INTO history VALUES (?, ?, ?, ?)",
763 757 (self.session_number,)+line)
764 758
765 759 def _writeout_output_cache(self, conn):
766 760 with conn:
767 761 for line in self.db_output_cache:
768 762 conn.execute("INSERT INTO output_history VALUES (?, ?, ?)",
769 763 (self.session_number,)+line)
770 764
771 765 @only_when_enabled
772 766 def writeout_cache(self, conn=None):
773 767 """Write any entries in the cache to the database."""
774 768 if conn is None:
775 769 conn = self.db
776 770
777 771 with self.db_input_cache_lock:
778 772 try:
779 773 self._writeout_input_cache(conn)
780 774 except sqlite3.IntegrityError:
781 775 self.new_session(conn)
782 776 print("ERROR! Session/line number was not unique in",
783 777 "database. History logging moved to new session",
784 778 self.session_number)
785 779 try:
786 780 # Try writing to the new session. If this fails, don't
787 781 # recurse
788 782 self._writeout_input_cache(conn)
789 783 except sqlite3.IntegrityError:
790 784 pass
791 785 finally:
792 786 self.db_input_cache = []
793 787
794 788 with self.db_output_cache_lock:
795 789 try:
796 790 self._writeout_output_cache(conn)
797 791 except sqlite3.IntegrityError:
798 792 print("!! Session/line number for output was not unique",
799 793 "in database. Output will not be stored.")
800 794 finally:
801 795 self.db_output_cache = []
802 796
803 797
804 798 class HistorySavingThread(threading.Thread):
805 799 """This thread takes care of writing history to the database, so that
806 800 the UI isn't held up while that happens.
807 801
808 802 It waits for the HistoryManager's save_flag to be set, then writes out
809 803 the history cache. The main thread is responsible for setting the flag when
810 804 the cache size reaches a defined threshold."""
811 805 daemon = True
812 806 stop_now = False
813 807 enabled = True
814 808 def __init__(self, history_manager):
815 809 super(HistorySavingThread, self).__init__(name="IPythonHistorySavingThread")
816 810 self.history_manager = history_manager
817 811 self.enabled = history_manager.enabled
818 812 atexit.register(self.stop)
819 813
820 814 @only_when_enabled
821 815 def run(self):
822 816 # We need a separate db connection per thread:
823 817 try:
824 818 self.db = sqlite3.connect(
825 819 str(self.history_manager.hist_file),
826 820 **self.history_manager.connection_options,
827 821 )
828 822 while True:
829 823 self.history_manager.save_flag.wait()
830 824 if self.stop_now:
831 825 self.db.close()
832 826 return
833 827 self.history_manager.save_flag.clear()
834 828 self.history_manager.writeout_cache(self.db)
835 829 except Exception as e:
836 830 print(("The history saving thread hit an unexpected error (%s)."
837 831 "History will not be written to the database.") % repr(e))
838 832
839 833 def stop(self):
840 834 """This can be called from the main thread to safely stop this thread.
841 835
842 836 Note that it does not attempt to write out remaining history before
843 837 exiting. That should be done by calling the HistoryManager's
844 838 end_session method."""
845 839 self.stop_now = True
846 840 self.history_manager.save_flag.set()
847 841 self.join()
848 842
849 843
850 844 # To match, e.g. ~5/8-~2/3
851 845 range_re = re.compile(r"""
852 846 ((?P<startsess>~?\d+)/)?
853 847 (?P<start>\d+)?
854 848 ((?P<sep>[\-:])
855 849 ((?P<endsess>~?\d+)/)?
856 850 (?P<end>\d+))?
857 851 $""", re.VERBOSE)
858 852
859 853
860 854 def extract_hist_ranges(ranges_str):
861 855 """Turn a string of history ranges into 3-tuples of (session, start, stop).
862 856
863 857 Empty string results in a `[(0, 1, None)]`, i.e. "everything from current
864 858 session".
865 859
866 860 Examples
867 861 --------
868 862 >>> list(extract_hist_ranges("~8/5-~7/4 2"))
869 863 [(-8, 5, None), (-7, 1, 5), (0, 2, 3)]
870 864 """
871 865 if ranges_str == "":
872 866 yield (0, 1, None) # Everything from current session
873 867 return
874 868
875 869 for range_str in ranges_str.split():
876 870 rmatch = range_re.match(range_str)
877 871 if not rmatch:
878 872 continue
879 873 start = rmatch.group("start")
880 874 if start:
881 875 start = int(start)
882 876 end = rmatch.group("end")
883 877 # If no end specified, get (a, a + 1)
884 878 end = int(end) if end else start + 1
885 879 else: # start not specified
886 880 if not rmatch.group('startsess'): # no startsess
887 881 continue
888 882 start = 1
889 883 end = None # provide the entire session hist
890 884
891 885 if rmatch.group("sep") == "-": # 1-3 == 1:4 --> [1, 2, 3]
892 886 end += 1
893 887 startsess = rmatch.group("startsess") or "0"
894 888 endsess = rmatch.group("endsess") or startsess
895 889 startsess = int(startsess.replace("~","-"))
896 890 endsess = int(endsess.replace("~","-"))
897 891 assert endsess >= startsess, "start session must be earlier than end session"
898 892
899 893 if endsess == startsess:
900 894 yield (startsess, start, end)
901 895 continue
902 896 # Multiple sessions in one range:
903 897 yield (startsess, start, None)
904 898 for sess in range(startsess+1, endsess):
905 899 yield (sess, 1, None)
906 900 yield (endsess, 1, end)
907 901
908 902
909 903 def _format_lineno(session, line):
910 904 """Helper function to format line numbers properly."""
911 905 if session == 0:
912 906 return str(line)
913 907 return "%s#%s" % (session, line)
@@ -1,772 +1,772 b''
1 1 """DEPRECATED: Input handling and transformation machinery.
2 2
3 3 This module was deprecated in IPython 7.0, in favour of inputtransformer2.
4 4
5 5 The first class in this module, :class:`InputSplitter`, is designed to tell when
6 6 input from a line-oriented frontend is complete and should be executed, and when
7 7 the user should be prompted for another line of code instead. The name 'input
8 8 splitter' is largely for historical reasons.
9 9
10 10 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
11 11 with full support for the extended IPython syntax (magics, system calls, etc).
12 12 The code to actually do these transformations is in :mod:`IPython.core.inputtransformer`.
13 13 :class:`IPythonInputSplitter` feeds the raw code to the transformers in order
14 14 and stores the results.
15 15
16 16 For more details, see the class docstrings below.
17 17 """
18 18
19 19 from warnings import warn
20 20
21 21 warn('IPython.core.inputsplitter is deprecated since IPython 7 in favor of `IPython.core.inputtransformer2`',
22 22 DeprecationWarning)
23 23
24 24 # Copyright (c) IPython Development Team.
25 25 # Distributed under the terms of the Modified BSD License.
26 26 import ast
27 27 import codeop
28 28 import io
29 29 import re
30 30 import sys
31 31 import tokenize
32 32 import warnings
33 33
34 34 from IPython.core.inputtransformer import (leading_indent,
35 35 classic_prompt,
36 36 ipy_prompt,
37 37 cellmagic,
38 38 assemble_logical_lines,
39 39 help_end,
40 40 escaped_commands,
41 41 assign_from_magic,
42 42 assign_from_system,
43 43 assemble_python_lines,
44 44 )
45 45
46 46 # These are available in this module for backwards compatibility.
47 47 from IPython.core.inputtransformer import (ESC_SHELL, ESC_SH_CAP, ESC_HELP,
48 48 ESC_HELP2, ESC_MAGIC, ESC_MAGIC2,
49 49 ESC_QUOTE, ESC_QUOTE2, ESC_PAREN, ESC_SEQUENCES)
50 50
51 51 #-----------------------------------------------------------------------------
52 52 # Utilities
53 53 #-----------------------------------------------------------------------------
54 54
55 55 # FIXME: These are general-purpose utilities that later can be moved to the
56 56 # general ward. Kept here for now because we're being very strict about test
57 57 # coverage with this code, and this lets us ensure that we keep 100% coverage
58 58 # while developing.
59 59
60 60 # compiled regexps for autoindent management
61 61 dedent_re = re.compile('|'.join([
62 62 r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
63 63 r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
64 64 r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
65 65 r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
66 66 r'^\s+pass\s*$', # pass (optionally followed by trailing spaces)
67 67 r'^\s+break\s*$', # break (optionally followed by trailing spaces)
68 68 r'^\s+continue\s*$', # continue (optionally followed by trailing spaces)
69 69 ]))
70 70 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
71 71
72 72 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
73 73 # before pure comments
74 74 comment_line_re = re.compile(r'^\s*\#')
75 75
76 76
77 77 def num_ini_spaces(s):
78 78 """Return the number of initial spaces in a string.
79 79
80 80 Note that tabs are counted as a single space. For now, we do *not* support
81 81 mixing of tabs and spaces in the user's input.
82 82
83 83 Parameters
84 84 ----------
85 85 s : string
86 86
87 87 Returns
88 88 -------
89 89 n : int
90 90 """
91 91
92 92 ini_spaces = ini_spaces_re.match(s)
93 93 if ini_spaces:
94 94 return ini_spaces.end()
95 95 else:
96 96 return 0
97 97
98 98 # Fake token types for partial_tokenize:
99 99 INCOMPLETE_STRING = tokenize.N_TOKENS
100 100 IN_MULTILINE_STATEMENT = tokenize.N_TOKENS + 1
101 101
102 102 # The 2 classes below have the same API as TokenInfo, but don't try to look up
103 103 # a token type name that they won't find.
104 104 class IncompleteString:
105 105 type = exact_type = INCOMPLETE_STRING
106 106 def __init__(self, s, start, end, line):
107 107 self.s = s
108 108 self.start = start
109 109 self.end = end
110 110 self.line = line
111 111
112 112 class InMultilineStatement:
113 113 type = exact_type = IN_MULTILINE_STATEMENT
114 114 def __init__(self, pos, line):
115 115 self.s = ''
116 116 self.start = self.end = pos
117 117 self.line = line
118 118
119 119 def partial_tokens(s):
120 120 """Iterate over tokens from a possibly-incomplete string of code.
121 121
122 122 This adds two special token types: INCOMPLETE_STRING and
123 123 IN_MULTILINE_STATEMENT. These can only occur as the last token yielded, and
124 124 represent the two main ways for code to be incomplete.
125 125 """
126 126 readline = io.StringIO(s).readline
127 127 token = tokenize.TokenInfo(tokenize.NEWLINE, '', (1, 0), (1, 0), '')
128 128 try:
129 129 for token in tokenize.generate_tokens(readline):
130 130 yield token
131 131 except tokenize.TokenError as e:
132 132 # catch EOF error
133 133 lines = s.splitlines(keepends=True)
134 134 end = len(lines), len(lines[-1])
135 135 if 'multi-line string' in e.args[0]:
136 136 l, c = start = token.end
137 137 s = lines[l-1][c:] + ''.join(lines[l:])
138 138 yield IncompleteString(s, start, end, lines[-1])
139 139 elif 'multi-line statement' in e.args[0]:
140 140 yield InMultilineStatement(end, lines[-1])
141 141 else:
142 142 raise
143 143
144 144 def find_next_indent(code):
145 145 """Find the number of spaces for the next line of indentation"""
146 146 tokens = list(partial_tokens(code))
147 147 if tokens[-1].type == tokenize.ENDMARKER:
148 148 tokens.pop()
149 149 if not tokens:
150 150 return 0
151 151 while (tokens[-1].type in {tokenize.DEDENT, tokenize.NEWLINE, tokenize.COMMENT}):
152 152 tokens.pop()
153 153
154 154 if tokens[-1].type == INCOMPLETE_STRING:
155 155 # Inside a multiline string
156 156 return 0
157 157
158 158 # Find the indents used before
159 159 prev_indents = [0]
160 160 def _add_indent(n):
161 161 if n != prev_indents[-1]:
162 162 prev_indents.append(n)
163 163
164 164 tokiter = iter(tokens)
165 165 for tok in tokiter:
166 166 if tok.type in {tokenize.INDENT, tokenize.DEDENT}:
167 167 _add_indent(tok.end[1])
168 168 elif (tok.type == tokenize.NL):
169 169 try:
170 170 _add_indent(next(tokiter).start[1])
171 171 except StopIteration:
172 172 break
173 173
174 174 last_indent = prev_indents.pop()
175 175
176 176 # If we've just opened a multiline statement (e.g. 'a = ['), indent more
177 177 if tokens[-1].type == IN_MULTILINE_STATEMENT:
178 178 if tokens[-2].exact_type in {tokenize.LPAR, tokenize.LSQB, tokenize.LBRACE}:
179 179 return last_indent + 4
180 180 return last_indent
181 181
182 182 if tokens[-1].exact_type == tokenize.COLON:
183 183 # Line ends with colon - indent
184 184 return last_indent + 4
185 185
186 186 if last_indent:
187 187 # Examine the last line for dedent cues - statements like return or
188 188 # raise which normally end a block of code.
189 189 last_line_starts = 0
190 190 for i, tok in enumerate(tokens):
191 191 if tok.type == tokenize.NEWLINE:
192 192 last_line_starts = i + 1
193 193
194 194 last_line_tokens = tokens[last_line_starts:]
195 195 names = [t.string for t in last_line_tokens if t.type == tokenize.NAME]
196 196 if names and names[0] in {'raise', 'return', 'pass', 'break', 'continue'}:
197 197 # Find the most recent indentation less than the current level
198 198 for indent in reversed(prev_indents):
199 199 if indent < last_indent:
200 200 return indent
201 201
202 202 return last_indent
203 203
204 204
205 205 def last_blank(src):
206 206 """Determine if the input source ends in a blank.
207 207
208 208 A blank is either a newline or a line consisting of whitespace.
209 209
210 210 Parameters
211 211 ----------
212 212 src : string
213 A single or multiline string.
213 A single or multiline string.
214 214 """
215 215 if not src: return False
216 216 ll = src.splitlines()[-1]
217 217 return (ll == '') or ll.isspace()
218 218
219 219
220 220 last_two_blanks_re = re.compile(r'\n\s*\n\s*$', re.MULTILINE)
221 221 last_two_blanks_re2 = re.compile(r'.+\n\s*\n\s+$', re.MULTILINE)
222 222
223 223 def last_two_blanks(src):
224 224 """Determine if the input source ends in two blanks.
225 225
226 226 A blank is either a newline or a line consisting of whitespace.
227 227
228 228 Parameters
229 229 ----------
230 230 src : string
231 A single or multiline string.
231 A single or multiline string.
232 232 """
233 233 if not src: return False
234 234 # The logic here is tricky: I couldn't get a regexp to work and pass all
235 235 # the tests, so I took a different approach: split the source by lines,
236 236 # grab the last two and prepend '###\n' as a stand-in for whatever was in
237 237 # the body before the last two lines. Then, with that structure, it's
238 238 # possible to analyze with two regexps. Not the most elegant solution, but
239 239 # it works. If anyone tries to change this logic, make sure to validate
240 240 # the whole test suite first!
241 241 new_src = '\n'.join(['###\n'] + src.splitlines()[-2:])
242 242 return (bool(last_two_blanks_re.match(new_src)) or
243 243 bool(last_two_blanks_re2.match(new_src)) )
244 244
245 245
246 246 def remove_comments(src):
247 247 """Remove all comments from input source.
248 248
249 249 Note: comments are NOT recognized inside of strings!
250 250
251 251 Parameters
252 252 ----------
253 253 src : string
254 A single or multiline input string.
254 A single or multiline input string.
255 255
256 256 Returns
257 257 -------
258 258 String with all Python comments removed.
259 259 """
260 260
261 261 return re.sub('#.*', '', src)
262 262
263 263
264 264 def get_input_encoding():
265 265 """Return the default standard input encoding.
266 266
267 267 If sys.stdin has no encoding, 'ascii' is returned."""
268 268 # There are strange environments for which sys.stdin.encoding is None. We
269 269 # ensure that a valid encoding is returned.
270 270 encoding = getattr(sys.stdin, 'encoding', None)
271 271 if encoding is None:
272 272 encoding = 'ascii'
273 273 return encoding
274 274
275 275 #-----------------------------------------------------------------------------
276 276 # Classes and functions for normal Python syntax handling
277 277 #-----------------------------------------------------------------------------
278 278
279 279 class InputSplitter(object):
280 280 r"""An object that can accumulate lines of Python source before execution.
281 281
282 282 This object is designed to be fed python source line-by-line, using
283 283 :meth:`push`. It will return on each push whether the currently pushed
284 284 code could be executed already. In addition, it provides a method called
285 285 :meth:`push_accepts_more` that can be used to query whether more input
286 286 can be pushed into a single interactive block.
287 287
288 288 This is a simple example of how an interactive terminal-based client can use
289 289 this tool::
290 290
291 291 isp = InputSplitter()
292 292 while isp.push_accepts_more():
293 293 indent = ' '*isp.indent_spaces
294 294 prompt = '>>> ' + indent
295 295 line = indent + raw_input(prompt)
296 296 isp.push(line)
297 297 print 'Input source was:\n', isp.source_reset(),
298 298 """
299 299 # A cache for storing the current indentation
300 300 # The first value stores the most recently processed source input
301 301 # The second value is the number of spaces for the current indentation
302 302 # If self.source matches the first value, the second value is a valid
303 303 # current indentation. Otherwise, the cache is invalid and the indentation
304 304 # must be recalculated.
305 305 _indent_spaces_cache = None, None
306 306 # String, indicating the default input encoding. It is computed by default
307 307 # at initialization time via get_input_encoding(), but it can be reset by a
308 308 # client with specific knowledge of the encoding.
309 309 encoding = ''
310 310 # String where the current full source input is stored, properly encoded.
311 311 # Reading this attribute is the normal way of querying the currently pushed
312 312 # source code, that has been properly encoded.
313 313 source = ''
314 314 # Code object corresponding to the current source. It is automatically
315 315 # synced to the source, so it can be queried at any time to obtain the code
316 316 # object; it will be None if the source doesn't compile to valid Python.
317 317 code = None
318 318
319 319 # Private attributes
320 320
321 321 # List with lines of input accumulated so far
322 322 _buffer = None
323 323 # Command compiler
324 324 _compile = None
325 325 # Boolean indicating whether the current block is complete
326 326 _is_complete = None
327 327 # Boolean indicating whether the current block has an unrecoverable syntax error
328 328 _is_invalid = False
329 329
330 330 def __init__(self):
331 331 """Create a new InputSplitter instance.
332 332 """
333 333 self._buffer = []
334 334 self._compile = codeop.CommandCompiler()
335 335 self.encoding = get_input_encoding()
336 336
337 337 def reset(self):
338 338 """Reset the input buffer and associated state."""
339 339 self._buffer[:] = []
340 340 self.source = ''
341 341 self.code = None
342 342 self._is_complete = False
343 343 self._is_invalid = False
344 344
345 345 def source_reset(self):
346 346 """Return the input source and perform a full reset.
347 347 """
348 348 out = self.source
349 349 self.reset()
350 350 return out
351 351
352 352 def check_complete(self, source):
353 353 """Return whether a block of code is ready to execute, or should be continued
354
354
355 355 This is a non-stateful API, and will reset the state of this InputSplitter.
356
356
357 357 Parameters
358 358 ----------
359 359 source : string
360 Python input code, which can be multiline.
361
360 Python input code, which can be multiline.
361
362 362 Returns
363 363 -------
364 364 status : str
365 One of 'complete', 'incomplete', or 'invalid' if source is not a
366 prefix of valid code.
365 One of 'complete', 'incomplete', or 'invalid' if source is not a
366 prefix of valid code.
367 367 indent_spaces : int or None
368 The number of spaces by which to indent the next line of code. If
369 status is not 'incomplete', this is None.
368 The number of spaces by which to indent the next line of code. If
369 status is not 'incomplete', this is None.
370 370 """
371 371 self.reset()
372 372 try:
373 373 self.push(source)
374 374 except SyntaxError:
375 375 # Transformers in IPythonInputSplitter can raise SyntaxError,
376 376 # which push() will not catch.
377 377 return 'invalid', None
378 378 else:
379 379 if self._is_invalid:
380 380 return 'invalid', None
381 381 elif self.push_accepts_more():
382 382 return 'incomplete', self.get_indent_spaces()
383 383 else:
384 384 return 'complete', None
385 385 finally:
386 386 self.reset()
387 387
388 388 def push(self, lines:str) -> bool:
389 389 """Push one or more lines of input.
390 390
391 391 This stores the given lines and returns a status code indicating
392 392 whether the code forms a complete Python block or not.
393 393
394 394 Any exceptions generated in compilation are swallowed, but if an
395 395 exception was produced, the method returns True.
396 396
397 397 Parameters
398 398 ----------
399 399 lines : string
400 One or more lines of Python input.
400 One or more lines of Python input.
401 401
402 402 Returns
403 403 -------
404 404 is_complete : boolean
405 True if the current input source (the result of the current input
406 plus prior inputs) forms a complete Python execution block. Note that
407 this value is also stored as a private attribute (``_is_complete``), so it
408 can be queried at any time.
405 True if the current input source (the result of the current input
406 plus prior inputs) forms a complete Python execution block. Note that
407 this value is also stored as a private attribute (``_is_complete``), so it
408 can be queried at any time.
409 409 """
410 410 assert isinstance(lines, str)
411 411 self._store(lines)
412 412 source = self.source
413 413
414 414 # Before calling _compile(), reset the code object to None so that if an
415 415 # exception is raised in compilation, we don't mislead by having
416 416 # inconsistent code/source attributes.
417 417 self.code, self._is_complete = None, None
418 418 self._is_invalid = False
419 419
420 420 # Honor termination lines properly
421 421 if source.endswith('\\\n'):
422 422 return False
423 423
424 424 try:
425 425 with warnings.catch_warnings():
426 426 warnings.simplefilter('error', SyntaxWarning)
427 427 self.code = self._compile(source, symbol="exec")
428 428 # Invalid syntax can produce any of a number of different errors from
429 429 # inside the compiler, so we have to catch them all. Syntax errors
430 430 # immediately produce a 'ready' block, so the invalid Python can be
431 431 # sent to the kernel for evaluation with possible ipython
432 432 # special-syntax conversion.
433 433 except (SyntaxError, OverflowError, ValueError, TypeError,
434 434 MemoryError, SyntaxWarning):
435 435 self._is_complete = True
436 436 self._is_invalid = True
437 437 else:
438 438 # Compilation didn't produce any exceptions (though it may not have
439 439 # given a complete code object)
440 440 self._is_complete = self.code is not None
441 441
442 442 return self._is_complete
443 443
444 444 def push_accepts_more(self):
445 445 """Return whether a block of interactive input can accept more input.
446 446
447 447 This method is meant to be used by line-oriented frontends, who need to
448 448 guess whether a block is complete or not based solely on prior and
449 449 current input lines. The InputSplitter considers it has a complete
450 450 interactive block and will not accept more input when either:
451
451
452 452 * A SyntaxError is raised
453 453
454 454 * The code is complete and consists of a single line or a single
455 455 non-compound statement
456 456
457 457 * The code is complete and has a blank line at the end
458 458
459 459 If the current input produces a syntax error, this method immediately
460 460 returns False but does *not* raise the syntax error exception, as
461 461 typically clients will want to send invalid syntax to an execution
462 462 backend which might convert the invalid syntax into valid Python via
463 463 one of the dynamic IPython mechanisms.
464 464 """
465 465
466 466 # With incomplete input, unconditionally accept more
467 467 # A syntax error also sets _is_complete to True - see push()
468 468 if not self._is_complete:
469 469 #print("Not complete") # debug
470 470 return True
471 471
472 472 # The user can make any (complete) input execute by leaving a blank line
473 473 last_line = self.source.splitlines()[-1]
474 474 if (not last_line) or last_line.isspace():
475 475 #print("Blank line") # debug
476 476 return False
477 477
478 478 # If there's just a single line or AST node, and we're flush left, as is
479 479 # the case after a simple statement such as 'a=1', we want to execute it
480 480 # straight away.
481 481 if self.get_indent_spaces() == 0:
482 482 if len(self.source.splitlines()) <= 1:
483 483 return False
484 484
485 485 try:
486 486 code_ast = ast.parse(u''.join(self._buffer))
487 487 except Exception:
488 488 #print("Can't parse AST") # debug
489 489 return False
490 490 else:
491 491 if len(code_ast.body) == 1 and \
492 492 not hasattr(code_ast.body[0], 'body'):
493 493 #print("Simple statement") # debug
494 494 return False
495 495
496 496 # General fallback - accept more code
497 497 return True
498 498
499 499 def get_indent_spaces(self):
500 500 sourcefor, n = self._indent_spaces_cache
501 501 if sourcefor == self.source:
502 502 return n
503 503
504 504 # self.source always has a trailing newline
505 505 n = find_next_indent(self.source[:-1])
506 506 self._indent_spaces_cache = (self.source, n)
507 507 return n
508 508
509 509 # Backwards compatibility. I think all code that used .indent_spaces was
510 510 # inside IPython, but we can leave this here until IPython 7 in case any
511 511 # other modules are using it. -TK, November 2017
512 512 indent_spaces = property(get_indent_spaces)
513 513
514 514 def _store(self, lines, buffer=None, store='source'):
515 515 """Store one or more lines of input.
516 516
517 517 If input lines are not newline-terminated, a newline is automatically
518 518 appended."""
519 519
520 520 if buffer is None:
521 521 buffer = self._buffer
522 522
523 523 if lines.endswith('\n'):
524 524 buffer.append(lines)
525 525 else:
526 526 buffer.append(lines+'\n')
527 527 setattr(self, store, self._set_source(buffer))
528 528
529 529 def _set_source(self, buffer):
530 530 return u''.join(buffer)
531 531
532 532
533 533 class IPythonInputSplitter(InputSplitter):
534 534 """An input splitter that recognizes all of IPython's special syntax."""
535 535
536 536 # String with raw, untransformed input.
537 537 source_raw = ''
538 538
539 539 # Flag to track when a transformer has stored input that it hasn't given
540 540 # back yet.
541 541 transformer_accumulating = False
542 542
543 543 # Flag to track when assemble_python_lines has stored input that it hasn't
544 544 # given back yet.
545 545 within_python_line = False
546 546
547 547 # Private attributes
548 548
549 549 # List with lines of raw input accumulated so far.
550 550 _buffer_raw = None
551 551
552 552 def __init__(self, line_input_checker=True, physical_line_transforms=None,
553 553 logical_line_transforms=None, python_line_transforms=None):
554 554 super(IPythonInputSplitter, self).__init__()
555 555 self._buffer_raw = []
556 556 self._validate = True
557 557
558 558 if physical_line_transforms is not None:
559 559 self.physical_line_transforms = physical_line_transforms
560 560 else:
561 561 self.physical_line_transforms = [
562 562 leading_indent(),
563 563 classic_prompt(),
564 564 ipy_prompt(),
565 565 cellmagic(end_on_blank_line=line_input_checker),
566 566 ]
567 567
568 568 self.assemble_logical_lines = assemble_logical_lines()
569 569 if logical_line_transforms is not None:
570 570 self.logical_line_transforms = logical_line_transforms
571 571 else:
572 572 self.logical_line_transforms = [
573 573 help_end(),
574 574 escaped_commands(),
575 575 assign_from_magic(),
576 576 assign_from_system(),
577 577 ]
578 578
579 579 self.assemble_python_lines = assemble_python_lines()
580 580 if python_line_transforms is not None:
581 581 self.python_line_transforms = python_line_transforms
582 582 else:
583 583 # We don't use any of these at present
584 584 self.python_line_transforms = []
585 585
586 586 @property
587 587 def transforms(self):
588 588 "Quick access to all transformers."
589 589 return self.physical_line_transforms + \
590 590 [self.assemble_logical_lines] + self.logical_line_transforms + \
591 591 [self.assemble_python_lines] + self.python_line_transforms
592 592
593 593 @property
594 594 def transforms_in_use(self):
595 595 """Transformers, excluding logical line transformers if we're in a
596 596 Python line."""
597 597 t = self.physical_line_transforms[:]
598 598 if not self.within_python_line:
599 599 t += [self.assemble_logical_lines] + self.logical_line_transforms
600 600 return t + [self.assemble_python_lines] + self.python_line_transforms
601 601
602 602 def reset(self):
603 603 """Reset the input buffer and associated state."""
604 604 super(IPythonInputSplitter, self).reset()
605 605 self._buffer_raw[:] = []
606 606 self.source_raw = ''
607 607 self.transformer_accumulating = False
608 608 self.within_python_line = False
609 609
610 610 for t in self.transforms:
611 611 try:
612 612 t.reset()
613 613 except SyntaxError:
614 614 # Nothing that calls reset() expects to handle transformer
615 615 # errors
616 616 pass
617 617
618 618 def flush_transformers(self):
619 619 def _flush(transform, outs):
620 620 """yield transformed lines
621
621
622 622 always strings, never None
623
623
624 624 transform: the current transform
625 625 outs: an iterable of previously transformed inputs.
626 626 Each may be multiline, which will be passed
627 627 one line at a time to transform.
628 628 """
629 629 for out in outs:
630 630 for line in out.splitlines():
631 631 # push one line at a time
632 632 tmp = transform.push(line)
633 633 if tmp is not None:
634 634 yield tmp
635 635
636 636 # reset the transform
637 637 tmp = transform.reset()
638 638 if tmp is not None:
639 639 yield tmp
640 640
641 641 out = []
642 642 for t in self.transforms_in_use:
643 643 out = _flush(t, out)
644 644
645 645 out = list(out)
646 646 if out:
647 647 self._store('\n'.join(out))
648 648
649 649 def raw_reset(self):
650 650 """Return raw input only and perform a full reset.
651 651 """
652 652 out = self.source_raw
653 653 self.reset()
654 654 return out
655 655
656 656 def source_reset(self):
657 657 try:
658 658 self.flush_transformers()
659 659 return self.source
660 660 finally:
661 661 self.reset()
662 662
663 663 def push_accepts_more(self):
664 664 if self.transformer_accumulating:
665 665 return True
666 666 else:
667 667 return super(IPythonInputSplitter, self).push_accepts_more()
668 668
669 669 def transform_cell(self, cell):
670 670 """Process and translate a cell of input.
671 671 """
672 672 self.reset()
673 673 try:
674 674 self.push(cell)
675 675 self.flush_transformers()
676 676 return self.source
677 677 finally:
678 678 self.reset()
679 679
680 680 def push(self, lines:str) -> bool:
681 681 """Push one or more lines of IPython input.
682 682
683 683 This stores the given lines and returns a status code indicating
684 684 whether the code forms a complete Python block or not, after processing
685 685 all input lines for special IPython syntax.
686 686
687 687 Any exceptions generated in compilation are swallowed, but if an
688 688 exception was produced, the method returns True.
689 689
690 690 Parameters
691 691 ----------
692 692 lines : string
693 One or more lines of Python input.
693 One or more lines of Python input.
694 694
695 695 Returns
696 696 -------
697 697 is_complete : boolean
698 True if the current input source (the result of the current input
699 plus prior inputs) forms a complete Python execution block. Note that
700 this value is also stored as a private attribute (_is_complete), so it
701 can be queried at any time.
698 True if the current input source (the result of the current input
699 plus prior inputs) forms a complete Python execution block. Note that
700 this value is also stored as a private attribute (_is_complete), so it
701 can be queried at any time.
702 702 """
703 703 assert isinstance(lines, str)
704 704 # We must ensure all input is pure unicode
705 705 # ''.splitlines() --> [], but we need to push the empty line to transformers
706 706 lines_list = lines.splitlines()
707 707 if not lines_list:
708 708 lines_list = ['']
709 709
710 710 # Store raw source before applying any transformations to it. Note
711 711 # that this must be done *after* the reset() call that would otherwise
712 712 # flush the buffer.
713 713 self._store(lines, self._buffer_raw, 'source_raw')
714 714
715 715 transformed_lines_list = []
716 716 for line in lines_list:
717 717 transformed = self._transform_line(line)
718 718 if transformed is not None:
719 719 transformed_lines_list.append(transformed)
720 720
721 721 if transformed_lines_list:
722 722 transformed_lines = '\n'.join(transformed_lines_list)
723 723 return super(IPythonInputSplitter, self).push(transformed_lines)
724 724 else:
725 725 # Got nothing back from transformers - they must be waiting for
726 726 # more input.
727 727 return False
728 728
729 729 def _transform_line(self, line):
730 730 """Push a line of input code through the various transformers.
731
731
732 732 Returns any output from the transformers, or None if a transformer
733 733 is accumulating lines.
734
734
735 735 Sets self.transformer_accumulating as a side effect.
736 736 """
737 737 def _accumulating(dbg):
738 738 #print(dbg)
739 739 self.transformer_accumulating = True
740 740 return None
741 741
742 742 for transformer in self.physical_line_transforms:
743 743 line = transformer.push(line)
744 744 if line is None:
745 745 return _accumulating(transformer)
746 746
747 747 if not self.within_python_line:
748 748 line = self.assemble_logical_lines.push(line)
749 749 if line is None:
750 750 return _accumulating('acc logical line')
751 751
752 752 for transformer in self.logical_line_transforms:
753 753 line = transformer.push(line)
754 754 if line is None:
755 755 return _accumulating(transformer)
756 756
757 757 line = self.assemble_python_lines.push(line)
758 758 if line is None:
759 759 self.within_python_line = True
760 760 return _accumulating('acc python line')
761 761 else:
762 762 self.within_python_line = False
763 763
764 764 for transformer in self.python_line_transforms:
765 765 line = transformer.push(line)
766 766 if line is None:
767 767 return _accumulating(transformer)
768 768
769 769 #print("transformers clear") #debug
770 770 self.transformer_accumulating = False
771 771 return line
772 772
@@ -1,536 +1,535 b''
1 1 """DEPRECATED: Input transformer classes to support IPython special syntax.
2 2
3 3 This module was deprecated in IPython 7.0, in favour of inputtransformer2.
4 4
5 5 This includes the machinery to recognise and transform ``%magic`` commands,
6 6 ``!system`` commands, ``help?`` querying, prompt stripping, and so forth.
7 7 """
8 8 import abc
9 9 import functools
10 10 import re
11 11 import tokenize
12 12 from tokenize import generate_tokens, untokenize, TokenError
13 13 from io import StringIO
14 14
15 15 from IPython.core.splitinput import LineInfo
16 16
17 17 #-----------------------------------------------------------------------------
18 18 # Globals
19 19 #-----------------------------------------------------------------------------
20 20
21 21 # The escape sequences that define the syntax transformations IPython will
22 22 # apply to user input. These can NOT be just changed here: many regular
23 23 # expressions and other parts of the code may use their hardcoded values, and
24 24 # for all intents and purposes they constitute the 'IPython syntax', so they
25 25 # should be considered fixed.
26 26
27 27 ESC_SHELL = '!' # Send line to underlying system shell
28 28 ESC_SH_CAP = '!!' # Send line to system shell and capture output
29 29 ESC_HELP = '?' # Find information about object
30 30 ESC_HELP2 = '??' # Find extra-detailed information about object
31 31 ESC_MAGIC = '%' # Call magic function
32 32 ESC_MAGIC2 = '%%' # Call cell-magic function
33 33 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
34 34 ESC_QUOTE2 = ';' # Quote all args as a single string, call
35 35 ESC_PAREN = '/' # Call first argument with rest of line as arguments
36 36
37 37 ESC_SEQUENCES = [ESC_SHELL, ESC_SH_CAP, ESC_HELP ,\
38 38 ESC_HELP2, ESC_MAGIC, ESC_MAGIC2,\
39 39 ESC_QUOTE, ESC_QUOTE2, ESC_PAREN ]
40 40
41 41
42 42 class InputTransformer(metaclass=abc.ABCMeta):
43 43 """Abstract base class for line-based input transformers."""
44 44
45 45 @abc.abstractmethod
46 46 def push(self, line):
47 47 """Send a line of input to the transformer, returning the transformed
48 48 input or None if the transformer is waiting for more input.
49
49
50 50 Must be overridden by subclasses.
51 51
52 52 Implementations may raise ``SyntaxError`` if the input is invalid. No
53 53 other exceptions may be raised.
54 54 """
55 55 pass
56 56
57 57 @abc.abstractmethod
58 58 def reset(self):
59 59 """Return, transformed any lines that the transformer has accumulated,
60 60 and reset its internal state.
61
61
62 62 Must be overridden by subclasses.
63 63 """
64 64 pass
65 65
66 66 @classmethod
67 67 def wrap(cls, func):
68 68 """Can be used by subclasses as a decorator, to return a factory that
69 69 will allow instantiation with the decorated object.
70 70 """
71 71 @functools.wraps(func)
72 72 def transformer_factory(**kwargs):
73 73 return cls(func, **kwargs)
74 74
75 75 return transformer_factory
76 76
77 77 class StatelessInputTransformer(InputTransformer):
78 78 """Wrapper for a stateless input transformer implemented as a function."""
79 79 def __init__(self, func):
80 80 self.func = func
81 81
82 82 def __repr__(self):
83 83 return "StatelessInputTransformer(func={0!r})".format(self.func)
84 84
85 85 def push(self, line):
86 86 """Send a line of input to the transformer, returning the
87 87 transformed input."""
88 88 return self.func(line)
89 89
90 90 def reset(self):
91 91 """No-op - exists for compatibility."""
92 92 pass
93 93
94 94 class CoroutineInputTransformer(InputTransformer):
95 95 """Wrapper for an input transformer implemented as a coroutine."""
96 96 def __init__(self, coro, **kwargs):
97 97 # Prime it
98 98 self.coro = coro(**kwargs)
99 99 next(self.coro)
100 100
101 101 def __repr__(self):
102 102 return "CoroutineInputTransformer(coro={0!r})".format(self.coro)
103 103
104 104 def push(self, line):
105 105 """Send a line of input to the transformer, returning the
106 106 transformed input or None if the transformer is waiting for more
107 107 input.
108 108 """
109 109 return self.coro.send(line)
110 110
111 111 def reset(self):
112 112 """Return, transformed any lines that the transformer has
113 113 accumulated, and reset its internal state.
114 114 """
115 115 return self.coro.send(None)
116 116
117 117 class TokenInputTransformer(InputTransformer):
118 118 """Wrapper for a token-based input transformer.
119 119
120 120 func should accept a list of tokens (5-tuples, see tokenize docs), and
121 121 return an iterable which can be passed to tokenize.untokenize().
122 122 """
123 123 def __init__(self, func):
124 124 self.func = func
125 125 self.buf = []
126 126 self.reset_tokenizer()
127 127
128 128 def reset_tokenizer(self):
129 129 it = iter(self.buf)
130 130 self.tokenizer = generate_tokens(it.__next__)
131 131
132 132 def push(self, line):
133 133 self.buf.append(line + '\n')
134 134 if all(l.isspace() for l in self.buf):
135 135 return self.reset()
136 136
137 137 tokens = []
138 138 stop_at_NL = False
139 139 try:
140 140 for intok in self.tokenizer:
141 141 tokens.append(intok)
142 142 t = intok[0]
143 143 if t == tokenize.NEWLINE or (stop_at_NL and t == tokenize.NL):
144 144 # Stop before we try to pull a line we don't have yet
145 145 break
146 146 elif t == tokenize.ERRORTOKEN:
147 147 stop_at_NL = True
148 148 except TokenError:
149 149 # Multi-line statement - stop and try again with the next line
150 150 self.reset_tokenizer()
151 151 return None
152 152
153 153 return self.output(tokens)
154 154
155 155 def output(self, tokens):
156 156 self.buf.clear()
157 157 self.reset_tokenizer()
158 158 return untokenize(self.func(tokens)).rstrip('\n')
159 159
160 160 def reset(self):
161 161 l = ''.join(self.buf)
162 162 self.buf.clear()
163 163 self.reset_tokenizer()
164 164 if l:
165 165 return l.rstrip('\n')
166 166
167 167 class assemble_python_lines(TokenInputTransformer):
168 168 def __init__(self):
169 169 super(assemble_python_lines, self).__init__(None)
170 170
171 171 def output(self, tokens):
172 172 return self.reset()
173 173
174 174 @CoroutineInputTransformer.wrap
175 175 def assemble_logical_lines():
176 176 r"""Join lines following explicit line continuations (\)"""
177 177 line = ''
178 178 while True:
179 179 line = (yield line)
180 180 if not line or line.isspace():
181 181 continue
182 182
183 183 parts = []
184 184 while line is not None:
185 185 if line.endswith('\\') and (not has_comment(line)):
186 186 parts.append(line[:-1])
187 187 line = (yield None) # Get another line
188 188 else:
189 189 parts.append(line)
190 190 break
191 191
192 192 # Output
193 193 line = ''.join(parts)
194 194
195 195 # Utilities
196 196 def _make_help_call(target, esc, lspace, next_input=None):
197 197 """Prepares a pinfo(2)/psearch call from a target name and the escape
198 198 (i.e. ? or ??)"""
199 199 method = 'pinfo2' if esc == '??' \
200 200 else 'psearch' if '*' in target \
201 201 else 'pinfo'
202 202 arg = " ".join([method, target])
203 203 #Prepare arguments for get_ipython().run_line_magic(magic_name, magic_args)
204 204 t_magic_name, _, t_magic_arg_s = arg.partition(' ')
205 205 t_magic_name = t_magic_name.lstrip(ESC_MAGIC)
206 206 if next_input is None:
207 207 return '%sget_ipython().run_line_magic(%r, %r)' % (lspace, t_magic_name, t_magic_arg_s)
208 208 else:
209 209 return '%sget_ipython().set_next_input(%r);get_ipython().run_line_magic(%r, %r)' % \
210 210 (lspace, next_input, t_magic_name, t_magic_arg_s)
211 211
212 212 # These define the transformations for the different escape characters.
213 213 def _tr_system(line_info):
214 214 "Translate lines escaped with: !"
215 215 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
216 216 return '%sget_ipython().system(%r)' % (line_info.pre, cmd)
217 217
218 218 def _tr_system2(line_info):
219 219 "Translate lines escaped with: !!"
220 220 cmd = line_info.line.lstrip()[2:]
221 221 return '%sget_ipython().getoutput(%r)' % (line_info.pre, cmd)
222 222
223 223 def _tr_help(line_info):
224 224 "Translate lines escaped with: ?/??"
225 225 # A naked help line should just fire the intro help screen
226 226 if not line_info.line[1:]:
227 227 return 'get_ipython().show_usage()'
228 228
229 229 return _make_help_call(line_info.ifun, line_info.esc, line_info.pre)
230 230
231 231 def _tr_magic(line_info):
232 232 "Translate lines escaped with: %"
233 233 tpl = '%sget_ipython().run_line_magic(%r, %r)'
234 234 if line_info.line.startswith(ESC_MAGIC2):
235 235 return line_info.line
236 236 cmd = ' '.join([line_info.ifun, line_info.the_rest]).strip()
237 237 #Prepare arguments for get_ipython().run_line_magic(magic_name, magic_args)
238 238 t_magic_name, _, t_magic_arg_s = cmd.partition(' ')
239 239 t_magic_name = t_magic_name.lstrip(ESC_MAGIC)
240 240 return tpl % (line_info.pre, t_magic_name, t_magic_arg_s)
241 241
242 242 def _tr_quote(line_info):
243 243 "Translate lines escaped with: ,"
244 244 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
245 245 '", "'.join(line_info.the_rest.split()) )
246 246
247 247 def _tr_quote2(line_info):
248 248 "Translate lines escaped with: ;"
249 249 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
250 250 line_info.the_rest)
251 251
252 252 def _tr_paren(line_info):
253 253 "Translate lines escaped with: /"
254 254 return '%s%s(%s)' % (line_info.pre, line_info.ifun,
255 255 ", ".join(line_info.the_rest.split()))
256 256
257 257 tr = { ESC_SHELL : _tr_system,
258 258 ESC_SH_CAP : _tr_system2,
259 259 ESC_HELP : _tr_help,
260 260 ESC_HELP2 : _tr_help,
261 261 ESC_MAGIC : _tr_magic,
262 262 ESC_QUOTE : _tr_quote,
263 263 ESC_QUOTE2 : _tr_quote2,
264 264 ESC_PAREN : _tr_paren }
265 265
266 266 @StatelessInputTransformer.wrap
267 267 def escaped_commands(line):
268 268 """Transform escaped commands - %magic, !system, ?help + various autocalls.
269 269 """
270 270 if not line or line.isspace():
271 271 return line
272 272 lineinf = LineInfo(line)
273 273 if lineinf.esc not in tr:
274 274 return line
275 275
276 276 return tr[lineinf.esc](lineinf)
277 277
278 278 _initial_space_re = re.compile(r'\s*')
279 279
280 280 _help_end_re = re.compile(r"""(%{0,2}
281 281 (?!\d)[\w*]+ # Variable name
282 282 (\.(?!\d)[\w*]+)* # .etc.etc
283 283 )
284 284 (\?\??)$ # ? or ??
285 285 """,
286 286 re.VERBOSE)
287 287
288 288 # Extra pseudotokens for multiline strings and data structures
289 289 _MULTILINE_STRING = object()
290 290 _MULTILINE_STRUCTURE = object()
291 291
292 292 def _line_tokens(line):
293 293 """Helper for has_comment and ends_in_comment_or_string."""
294 294 readline = StringIO(line).readline
295 295 toktypes = set()
296 296 try:
297 297 for t in generate_tokens(readline):
298 298 toktypes.add(t[0])
299 299 except TokenError as e:
300 300 # There are only two cases where a TokenError is raised.
301 301 if 'multi-line string' in e.args[0]:
302 302 toktypes.add(_MULTILINE_STRING)
303 303 else:
304 304 toktypes.add(_MULTILINE_STRUCTURE)
305 305 return toktypes
306 306
307 307 def has_comment(src):
308 308 """Indicate whether an input line has (i.e. ends in, or is) a comment.
309 309
310 310 This uses tokenize, so it can distinguish comments from # inside strings.
311 311
312 312 Parameters
313 313 ----------
314 314 src : string
315 A single line input string.
315 A single line input string.
316 316
317 317 Returns
318 318 -------
319 319 comment : bool
320 320 True if source has a comment.
321 321 """
322 322 return (tokenize.COMMENT in _line_tokens(src))
323 323
324 324 def ends_in_comment_or_string(src):
325 325 """Indicates whether or not an input line ends in a comment or within
326 326 a multiline string.
327
327
328 328 Parameters
329 329 ----------
330 330 src : string
331 A single line input string.
331 A single line input string.
332 332
333 333 Returns
334 334 -------
335 335 comment : bool
336 336 True if source ends in a comment or multiline string.
337 337 """
338 338 toktypes = _line_tokens(src)
339 339 return (tokenize.COMMENT in toktypes) or (_MULTILINE_STRING in toktypes)
340 340
341 341
342 342 @StatelessInputTransformer.wrap
343 343 def help_end(line):
344 344 """Translate lines with ?/?? at the end"""
345 345 m = _help_end_re.search(line)
346 346 if m is None or ends_in_comment_or_string(line):
347 347 return line
348 348 target = m.group(1)
349 349 esc = m.group(3)
350 350 lspace = _initial_space_re.match(line).group(0)
351 351
352 352 # If we're mid-command, put it back on the next prompt for the user.
353 353 next_input = line.rstrip('?') if line.strip() != m.group(0) else None
354 354
355 355 return _make_help_call(target, esc, lspace, next_input)
356 356
357 357
358 358 @CoroutineInputTransformer.wrap
359 359 def cellmagic(end_on_blank_line=False):
360 360 """Captures & transforms cell magics.
361
361
362 362 After a cell magic is started, this stores up any lines it gets until it is
363 363 reset (sent None).
364 364 """
365 365 tpl = 'get_ipython().run_cell_magic(%r, %r, %r)'
366 366 cellmagic_help_re = re.compile(r'%%\w+\?')
367 367 line = ''
368 368 while True:
369 369 line = (yield line)
370 370 # consume leading empty lines
371 371 while not line:
372 372 line = (yield line)
373 373
374 374 if not line.startswith(ESC_MAGIC2):
375 375 # This isn't a cell magic, idle waiting for reset then start over
376 376 while line is not None:
377 377 line = (yield line)
378 378 continue
379 379
380 380 if cellmagic_help_re.match(line):
381 381 # This case will be handled by help_end
382 382 continue
383 383
384 384 first = line
385 385 body = []
386 386 line = (yield None)
387 387 while (line is not None) and \
388 388 ((line.strip() != '') or not end_on_blank_line):
389 389 body.append(line)
390 390 line = (yield None)
391 391
392 392 # Output
393 393 magic_name, _, first = first.partition(' ')
394 394 magic_name = magic_name.lstrip(ESC_MAGIC2)
395 395 line = tpl % (magic_name, first, u'\n'.join(body))
396 396
397 397
398 398 def _strip_prompts(prompt_re, initial_re=None, turnoff_re=None):
399 399 """Remove matching input prompts from a block of input.
400
400
401 401 Parameters
402 402 ----------
403 403 prompt_re : regular expression
404 404 A regular expression matching any input prompt (including continuation)
405 405 initial_re : regular expression, optional
406 406 A regular expression matching only the initial prompt, but not continuation.
407 407 If no initial expression is given, prompt_re will be used everywhere.
408 408 Used mainly for plain Python prompts, where the continuation prompt
409 409 ``...`` is a valid Python expression in Python 3, so shouldn't be stripped.
410
411 410 If initial_re and prompt_re differ,
412 411 only initial_re will be tested against the first line.
413 412 If any prompt is found on the first two lines,
414 413 prompts will be stripped from the rest of the block.
415 414 """
416 415 if initial_re is None:
417 416 initial_re = prompt_re
418 417 line = ''
419 418 while True:
420 419 line = (yield line)
421 420
422 421 # First line of cell
423 422 if line is None:
424 423 continue
425 424 out, n1 = initial_re.subn('', line, count=1)
426 425 if turnoff_re and not n1:
427 426 if turnoff_re.match(line):
428 427 # We're in e.g. a cell magic; disable this transformer for
429 428 # the rest of the cell.
430 429 while line is not None:
431 430 line = (yield line)
432 431 continue
433 432
434 433 line = (yield out)
435 434
436 435 if line is None:
437 436 continue
438 437 # check for any prompt on the second line of the cell,
439 438 # because people often copy from just after the first prompt,
440 439 # so we might not see it in the first line.
441 440 out, n2 = prompt_re.subn('', line, count=1)
442 441 line = (yield out)
443 442
444 443 if n1 or n2:
445 444 # Found a prompt in the first two lines - check for it in
446 445 # the rest of the cell as well.
447 446 while line is not None:
448 447 line = (yield prompt_re.sub('', line, count=1))
449 448
450 449 else:
451 450 # Prompts not in input - wait for reset
452 451 while line is not None:
453 452 line = (yield line)
454 453
455 454 @CoroutineInputTransformer.wrap
456 455 def classic_prompt():
457 456 """Strip the >>>/... prompts of the Python interactive shell."""
458 457 # FIXME: non-capturing version (?:...) usable?
459 458 prompt_re = re.compile(r'^(>>>|\.\.\.)( |$)')
460 459 initial_re = re.compile(r'^>>>( |$)')
461 460 # Any %magic/!system is IPython syntax, so we needn't look for >>> prompts
462 461 turnoff_re = re.compile(r'^[%!]')
463 462 return _strip_prompts(prompt_re, initial_re, turnoff_re)
464 463
465 464 @CoroutineInputTransformer.wrap
466 465 def ipy_prompt():
467 466 """Strip IPython's In [1]:/...: prompts."""
468 467 # FIXME: non-capturing version (?:...) usable?
469 468 prompt_re = re.compile(r'^(In \[\d+\]: |\s*\.{3,}: ?)')
470 469 # Disable prompt stripping inside cell magics
471 470 turnoff_re = re.compile(r'^%%')
472 471 return _strip_prompts(prompt_re, turnoff_re=turnoff_re)
473 472
474 473
475 474 @CoroutineInputTransformer.wrap
476 475 def leading_indent():
477 476 """Remove leading indentation.
478
477
479 478 If the first line starts with a spaces or tabs, the same whitespace will be
480 479 removed from each following line until it is reset.
481 480 """
482 481 space_re = re.compile(r'^[ \t]+')
483 482 line = ''
484 483 while True:
485 484 line = (yield line)
486 485
487 486 if line is None:
488 487 continue
489 488
490 489 m = space_re.match(line)
491 490 if m:
492 491 space = m.group(0)
493 492 while line is not None:
494 493 if line.startswith(space):
495 494 line = line[len(space):]
496 495 line = (yield line)
497 496 else:
498 497 # No leading spaces - wait for reset
499 498 while line is not None:
500 499 line = (yield line)
501 500
502 501
503 502 _assign_pat = \
504 503 r'''(?P<lhs>(\s*)
505 504 ([\w\.]+) # Initial identifier
506 505 (\s*,\s*
507 506 \*?[\w\.]+)* # Further identifiers for unpacking
508 507 \s*?,? # Trailing comma
509 508 )
510 509 \s*=\s*
511 510 '''
512 511
513 512 assign_system_re = re.compile(r'{}!\s*(?P<cmd>.*)'.format(_assign_pat), re.VERBOSE)
514 513 assign_system_template = '%s = get_ipython().getoutput(%r)'
515 514 @StatelessInputTransformer.wrap
516 515 def assign_from_system(line):
517 516 """Transform assignment from system commands (e.g. files = !ls)"""
518 517 m = assign_system_re.match(line)
519 518 if m is None:
520 519 return line
521 520
522 521 return assign_system_template % m.group('lhs', 'cmd')
523 522
524 523 assign_magic_re = re.compile(r'{}%\s*(?P<cmd>.*)'.format(_assign_pat), re.VERBOSE)
525 524 assign_magic_template = '%s = get_ipython().run_line_magic(%r, %r)'
526 525 @StatelessInputTransformer.wrap
527 526 def assign_from_magic(line):
528 527 """Transform assignment from magic commands (e.g. a = %who_ls)"""
529 528 m = assign_magic_re.match(line)
530 529 if m is None:
531 530 return line
532 531 #Prepare arguments for get_ipython().run_line_magic(magic_name, magic_args)
533 532 m_lhs, m_cmd = m.group('lhs', 'cmd')
534 533 t_magic_name, _, t_magic_arg_s = m_cmd.partition(' ')
535 534 t_magic_name = t_magic_name.lstrip(ESC_MAGIC)
536 535 return assign_magic_template % (m_lhs, t_magic_name, t_magic_arg_s)
@@ -1,796 +1,796 b''
1 1 """Input transformer machinery to support IPython special syntax.
2 2
3 3 This includes the machinery to recognise and transform ``%magic`` commands,
4 4 ``!system`` commands, ``help?`` querying, prompt stripping, and so forth.
5 5
6 6 Added: IPython 7.0. Replaces inputsplitter and inputtransformer which were
7 7 deprecated in 7.0.
8 8 """
9 9
10 10 # Copyright (c) IPython Development Team.
11 11 # Distributed under the terms of the Modified BSD License.
12 12
13 13 import ast
14 14 import sys
15 15 from codeop import CommandCompiler, Compile
16 16 import re
17 17 import tokenize
18 18 from typing import List, Tuple, Optional, Any
19 19 import warnings
20 20
21 21 _indent_re = re.compile(r'^[ \t]+')
22 22
23 23 def leading_empty_lines(lines):
24 24 """Remove leading empty lines
25 25
26 26 If the leading lines are empty or contain only whitespace, they will be
27 27 removed.
28 28 """
29 29 if not lines:
30 30 return lines
31 31 for i, line in enumerate(lines):
32 32 if line and not line.isspace():
33 33 return lines[i:]
34 34 return lines
35 35
36 36 def leading_indent(lines):
37 37 """Remove leading indentation.
38 38
39 39 If the first line starts with a spaces or tabs, the same whitespace will be
40 40 removed from each following line in the cell.
41 41 """
42 42 if not lines:
43 43 return lines
44 44 m = _indent_re.match(lines[0])
45 45 if not m:
46 46 return lines
47 47 space = m.group(0)
48 48 n = len(space)
49 49 return [l[n:] if l.startswith(space) else l
50 50 for l in lines]
51 51
52 52 class PromptStripper:
53 53 """Remove matching input prompts from a block of input.
54 54
55 55 Parameters
56 56 ----------
57 57 prompt_re : regular expression
58 58 A regular expression matching any input prompt (including continuation,
59 59 e.g. ``...``)
60 60 initial_re : regular expression, optional
61 61 A regular expression matching only the initial prompt, but not continuation.
62 62 If no initial expression is given, prompt_re will be used everywhere.
63 63 Used mainly for plain Python prompts (``>>>``), where the continuation prompt
64 64 ``...`` is a valid Python expression in Python 3, so shouldn't be stripped.
65 65
66 66 Notes
67 67 -----
68 68
69 69 If initial_re and prompt_re differ,
70 70 only initial_re will be tested against the first line.
71 71 If any prompt is found on the first two lines,
72 72 prompts will be stripped from the rest of the block.
73 73 """
74 74 def __init__(self, prompt_re, initial_re=None):
75 75 self.prompt_re = prompt_re
76 76 self.initial_re = initial_re or prompt_re
77 77
78 78 def _strip(self, lines):
79 79 return [self.prompt_re.sub('', l, count=1) for l in lines]
80 80
81 81 def __call__(self, lines):
82 82 if not lines:
83 83 return lines
84 84 if self.initial_re.match(lines[0]) or \
85 85 (len(lines) > 1 and self.prompt_re.match(lines[1])):
86 86 return self._strip(lines)
87 87 return lines
88 88
89 89 classic_prompt = PromptStripper(
90 90 prompt_re=re.compile(r'^(>>>|\.\.\.)( |$)'),
91 91 initial_re=re.compile(r'^>>>( |$)')
92 92 )
93 93
94 94 ipython_prompt = PromptStripper(
95 95 re.compile(
96 96 r"""
97 97 ^( # Match from the beginning of a line, either:
98 98
99 99 # 1. First-line prompt:
100 100 ((\[nav\]|\[ins\])?\ )? # Vi editing mode prompt, if it's there
101 101 In\ # The 'In' of the prompt, with a space
102 102 \[\d+\]: # Command index, as displayed in the prompt
103 103 \ # With a mandatory trailing space
104 104
105 105 | # ... or ...
106 106
107 107 # 2. The three dots of the multiline prompt
108 108 \s* # All leading whitespace characters
109 109 \.{3,}: # The three (or more) dots
110 110 \ ? # With an optional trailing space
111 111
112 112 )
113 113 """,
114 114 re.VERBOSE,
115 115 )
116 116 )
117 117
118 118
119 119 def cell_magic(lines):
120 120 if not lines or not lines[0].startswith('%%'):
121 121 return lines
122 122 if re.match(r'%%\w+\?', lines[0]):
123 123 # This case will be handled by help_end
124 124 return lines
125 125 magic_name, _, first_line = lines[0][2:].rstrip().partition(' ')
126 126 body = ''.join(lines[1:])
127 127 return ['get_ipython().run_cell_magic(%r, %r, %r)\n'
128 128 % (magic_name, first_line, body)]
129 129
130 130
131 131 def _find_assign_op(token_line) -> Optional[int]:
132 132 """Get the index of the first assignment in the line ('=' not inside brackets)
133 133
134 134 Note: We don't try to support multiple special assignment (a = b = %foo)
135 135 """
136 136 paren_level = 0
137 137 for i, ti in enumerate(token_line):
138 138 s = ti.string
139 139 if s == '=' and paren_level == 0:
140 140 return i
141 141 if s in {'(','[','{'}:
142 142 paren_level += 1
143 143 elif s in {')', ']', '}'}:
144 144 if paren_level > 0:
145 145 paren_level -= 1
146 146 return None
147 147
148 148 def find_end_of_continued_line(lines, start_line: int):
149 149 """Find the last line of a line explicitly extended using backslashes.
150 150
151 151 Uses 0-indexed line numbers.
152 152 """
153 153 end_line = start_line
154 154 while lines[end_line].endswith('\\\n'):
155 155 end_line += 1
156 156 if end_line >= len(lines):
157 157 break
158 158 return end_line
159 159
160 160 def assemble_continued_line(lines, start: Tuple[int, int], end_line: int):
161 161 r"""Assemble a single line from multiple continued line pieces
162 162
163 163 Continued lines are lines ending in ``\``, and the line following the last
164 164 ``\`` in the block.
165 165
166 166 For example, this code continues over multiple lines::
167 167
168 168 if (assign_ix is not None) \
169 169 and (len(line) >= assign_ix + 2) \
170 170 and (line[assign_ix+1].string == '%') \
171 171 and (line[assign_ix+2].type == tokenize.NAME):
172 172
173 173 This statement contains four continued line pieces.
174 174 Assembling these pieces into a single line would give::
175 175
176 176 if (assign_ix is not None) and (len(line) >= assign_ix + 2) and (line[...
177 177
178 178 This uses 0-indexed line numbers. *start* is (lineno, colno).
179 179
180 180 Used to allow ``%magic`` and ``!system`` commands to be continued over
181 181 multiple lines.
182 182 """
183 183 parts = [lines[start[0]][start[1]:]] + lines[start[0]+1:end_line+1]
184 184 return ' '.join([p.rstrip()[:-1] for p in parts[:-1]] # Strip backslash+newline
185 185 + [parts[-1].rstrip()]) # Strip newline from last line
186 186
187 187 class TokenTransformBase:
188 188 """Base class for transformations which examine tokens.
189 189
190 190 Special syntax should not be transformed when it occurs inside strings or
191 191 comments. This is hard to reliably avoid with regexes. The solution is to
192 192 tokenise the code as Python, and recognise the special syntax in the tokens.
193 193
194 194 IPython's special syntax is not valid Python syntax, so tokenising may go
195 195 wrong after the special syntax starts. These classes therefore find and
196 196 transform *one* instance of special syntax at a time into regular Python
197 197 syntax. After each transformation, tokens are regenerated to find the next
198 198 piece of special syntax.
199 199
200 200 Subclasses need to implement one class method (find)
201 201 and one regular method (transform).
202 202
203 203 The priority attribute can select which transformation to apply if multiple
204 204 transformers match in the same place. Lower numbers have higher priority.
205 205 This allows "%magic?" to be turned into a help call rather than a magic call.
206 206 """
207 207 # Lower numbers -> higher priority (for matches in the same location)
208 208 priority = 10
209 209
210 210 def sortby(self):
211 211 return self.start_line, self.start_col, self.priority
212 212
213 213 def __init__(self, start):
214 214 self.start_line = start[0] - 1 # Shift from 1-index to 0-index
215 215 self.start_col = start[1]
216 216
217 217 @classmethod
218 218 def find(cls, tokens_by_line):
219 219 """Find one instance of special syntax in the provided tokens.
220 220
221 221 Tokens are grouped into logical lines for convenience,
222 222 so it is easy to e.g. look at the first token of each line.
223 223 *tokens_by_line* is a list of lists of tokenize.TokenInfo objects.
224 224
225 225 This should return an instance of its class, pointing to the start
226 226 position it has found, or None if it found no match.
227 227 """
228 228 raise NotImplementedError
229 229
230 230 def transform(self, lines: List[str]):
231 231 """Transform one instance of special syntax found by ``find()``
232 232
233 233 Takes a list of strings representing physical lines,
234 234 returns a similar list of transformed lines.
235 235 """
236 236 raise NotImplementedError
237 237
238 238 class MagicAssign(TokenTransformBase):
239 239 """Transformer for assignments from magics (a = %foo)"""
240 240 @classmethod
241 241 def find(cls, tokens_by_line):
242 242 """Find the first magic assignment (a = %foo) in the cell.
243 243 """
244 244 for line in tokens_by_line:
245 245 assign_ix = _find_assign_op(line)
246 246 if (assign_ix is not None) \
247 247 and (len(line) >= assign_ix + 2) \
248 248 and (line[assign_ix+1].string == '%') \
249 249 and (line[assign_ix+2].type == tokenize.NAME):
250 250 return cls(line[assign_ix+1].start)
251 251
252 252 def transform(self, lines: List[str]):
253 253 """Transform a magic assignment found by the ``find()`` classmethod.
254 254 """
255 255 start_line, start_col = self.start_line, self.start_col
256 256 lhs = lines[start_line][:start_col]
257 257 end_line = find_end_of_continued_line(lines, start_line)
258 258 rhs = assemble_continued_line(lines, (start_line, start_col), end_line)
259 259 assert rhs.startswith('%'), rhs
260 260 magic_name, _, args = rhs[1:].partition(' ')
261 261
262 262 lines_before = lines[:start_line]
263 263 call = "get_ipython().run_line_magic({!r}, {!r})".format(magic_name, args)
264 264 new_line = lhs + call + '\n'
265 265 lines_after = lines[end_line+1:]
266 266
267 267 return lines_before + [new_line] + lines_after
268 268
269 269
270 270 class SystemAssign(TokenTransformBase):
271 271 """Transformer for assignments from system commands (a = !foo)"""
272 272 @classmethod
273 273 def find(cls, tokens_by_line):
274 274 """Find the first system assignment (a = !foo) in the cell.
275 275 """
276 276 for line in tokens_by_line:
277 277 assign_ix = _find_assign_op(line)
278 278 if (assign_ix is not None) \
279 279 and not line[assign_ix].line.strip().startswith('=') \
280 280 and (len(line) >= assign_ix + 2) \
281 281 and (line[assign_ix + 1].type == tokenize.ERRORTOKEN):
282 282 ix = assign_ix + 1
283 283
284 284 while ix < len(line) and line[ix].type == tokenize.ERRORTOKEN:
285 285 if line[ix].string == '!':
286 286 return cls(line[ix].start)
287 287 elif not line[ix].string.isspace():
288 288 break
289 289 ix += 1
290 290
291 291 def transform(self, lines: List[str]):
292 292 """Transform a system assignment found by the ``find()`` classmethod.
293 293 """
294 294 start_line, start_col = self.start_line, self.start_col
295 295
296 296 lhs = lines[start_line][:start_col]
297 297 end_line = find_end_of_continued_line(lines, start_line)
298 298 rhs = assemble_continued_line(lines, (start_line, start_col), end_line)
299 299 assert rhs.startswith('!'), rhs
300 300 cmd = rhs[1:]
301 301
302 302 lines_before = lines[:start_line]
303 303 call = "get_ipython().getoutput({!r})".format(cmd)
304 304 new_line = lhs + call + '\n'
305 305 lines_after = lines[end_line + 1:]
306 306
307 307 return lines_before + [new_line] + lines_after
308 308
309 309 # The escape sequences that define the syntax transformations IPython will
310 310 # apply to user input. These can NOT be just changed here: many regular
311 311 # expressions and other parts of the code may use their hardcoded values, and
312 312 # for all intents and purposes they constitute the 'IPython syntax', so they
313 313 # should be considered fixed.
314 314
315 315 ESC_SHELL = '!' # Send line to underlying system shell
316 316 ESC_SH_CAP = '!!' # Send line to system shell and capture output
317 317 ESC_HELP = '?' # Find information about object
318 318 ESC_HELP2 = '??' # Find extra-detailed information about object
319 319 ESC_MAGIC = '%' # Call magic function
320 320 ESC_MAGIC2 = '%%' # Call cell-magic function
321 321 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
322 322 ESC_QUOTE2 = ';' # Quote all args as a single string, call
323 323 ESC_PAREN = '/' # Call first argument with rest of line as arguments
324 324
325 325 ESCAPE_SINGLES = {'!', '?', '%', ',', ';', '/'}
326 326 ESCAPE_DOUBLES = {'!!', '??'} # %% (cell magic) is handled separately
327 327
328 328 def _make_help_call(target, esc, next_input=None):
329 329 """Prepares a pinfo(2)/psearch call from a target name and the escape
330 330 (i.e. ? or ??)"""
331 331 method = 'pinfo2' if esc == '??' \
332 332 else 'psearch' if '*' in target \
333 333 else 'pinfo'
334 334 arg = " ".join([method, target])
335 335 #Prepare arguments for get_ipython().run_line_magic(magic_name, magic_args)
336 336 t_magic_name, _, t_magic_arg_s = arg.partition(' ')
337 337 t_magic_name = t_magic_name.lstrip(ESC_MAGIC)
338 338 if next_input is None:
339 339 return 'get_ipython().run_line_magic(%r, %r)' % (t_magic_name, t_magic_arg_s)
340 340 else:
341 341 return 'get_ipython().set_next_input(%r);get_ipython().run_line_magic(%r, %r)' % \
342 342 (next_input, t_magic_name, t_magic_arg_s)
343 343
344 344 def _tr_help(content):
345 345 """Translate lines escaped with: ?
346 346
347 347 A naked help line should fire the intro help screen (shell.show_usage())
348 348 """
349 349 if not content:
350 350 return 'get_ipython().show_usage()'
351 351
352 352 return _make_help_call(content, '?')
353 353
354 354 def _tr_help2(content):
355 355 """Translate lines escaped with: ??
356 356
357 357 A naked help line should fire the intro help screen (shell.show_usage())
358 358 """
359 359 if not content:
360 360 return 'get_ipython().show_usage()'
361 361
362 362 return _make_help_call(content, '??')
363 363
364 364 def _tr_magic(content):
365 365 "Translate lines escaped with a percent sign: %"
366 366 name, _, args = content.partition(' ')
367 367 return 'get_ipython().run_line_magic(%r, %r)' % (name, args)
368 368
369 369 def _tr_quote(content):
370 370 "Translate lines escaped with a comma: ,"
371 371 name, _, args = content.partition(' ')
372 372 return '%s("%s")' % (name, '", "'.join(args.split()) )
373 373
374 374 def _tr_quote2(content):
375 375 "Translate lines escaped with a semicolon: ;"
376 376 name, _, args = content.partition(' ')
377 377 return '%s("%s")' % (name, args)
378 378
379 379 def _tr_paren(content):
380 380 "Translate lines escaped with a slash: /"
381 381 name, _, args = content.partition(' ')
382 382 return '%s(%s)' % (name, ", ".join(args.split()))
383 383
384 384 tr = { ESC_SHELL : 'get_ipython().system({!r})'.format,
385 385 ESC_SH_CAP : 'get_ipython().getoutput({!r})'.format,
386 386 ESC_HELP : _tr_help,
387 387 ESC_HELP2 : _tr_help2,
388 388 ESC_MAGIC : _tr_magic,
389 389 ESC_QUOTE : _tr_quote,
390 390 ESC_QUOTE2 : _tr_quote2,
391 391 ESC_PAREN : _tr_paren }
392 392
393 393 class EscapedCommand(TokenTransformBase):
394 394 """Transformer for escaped commands like %foo, !foo, or /foo"""
395 395 @classmethod
396 396 def find(cls, tokens_by_line):
397 397 """Find the first escaped command (%foo, !foo, etc.) in the cell.
398 398 """
399 399 for line in tokens_by_line:
400 400 if not line:
401 401 continue
402 402 ix = 0
403 403 ll = len(line)
404 404 while ll > ix and line[ix].type in {tokenize.INDENT, tokenize.DEDENT}:
405 405 ix += 1
406 406 if ix >= ll:
407 407 continue
408 408 if line[ix].string in ESCAPE_SINGLES:
409 409 return cls(line[ix].start)
410 410
411 411 def transform(self, lines):
412 412 """Transform an escaped line found by the ``find()`` classmethod.
413 413 """
414 414 start_line, start_col = self.start_line, self.start_col
415 415
416 416 indent = lines[start_line][:start_col]
417 417 end_line = find_end_of_continued_line(lines, start_line)
418 418 line = assemble_continued_line(lines, (start_line, start_col), end_line)
419 419
420 420 if len(line) > 1 and line[:2] in ESCAPE_DOUBLES:
421 421 escape, content = line[:2], line[2:]
422 422 else:
423 423 escape, content = line[:1], line[1:]
424 424
425 425 if escape in tr:
426 426 call = tr[escape](content)
427 427 else:
428 428 call = ''
429 429
430 430 lines_before = lines[:start_line]
431 431 new_line = indent + call + '\n'
432 432 lines_after = lines[end_line + 1:]
433 433
434 434 return lines_before + [new_line] + lines_after
435 435
436 436 _help_end_re = re.compile(r"""(%{0,2}
437 437 (?!\d)[\w*]+ # Variable name
438 438 (\.(?!\d)[\w*]+)* # .etc.etc
439 439 )
440 440 (\?\??)$ # ? or ??
441 441 """,
442 442 re.VERBOSE)
443 443
444 444 class HelpEnd(TokenTransformBase):
445 445 """Transformer for help syntax: obj? and obj??"""
446 446 # This needs to be higher priority (lower number) than EscapedCommand so
447 447 # that inspecting magics (%foo?) works.
448 448 priority = 5
449 449
450 450 def __init__(self, start, q_locn):
451 451 super().__init__(start)
452 452 self.q_line = q_locn[0] - 1 # Shift from 1-indexed to 0-indexed
453 453 self.q_col = q_locn[1]
454 454
455 455 @classmethod
456 456 def find(cls, tokens_by_line):
457 457 """Find the first help command (foo?) in the cell.
458 458 """
459 459 for line in tokens_by_line:
460 460 # Last token is NEWLINE; look at last but one
461 461 if len(line) > 2 and line[-2].string == '?':
462 462 # Find the first token that's not INDENT/DEDENT
463 463 ix = 0
464 464 while line[ix].type in {tokenize.INDENT, tokenize.DEDENT}:
465 465 ix += 1
466 466 return cls(line[ix].start, line[-2].start)
467 467
468 468 def transform(self, lines):
469 469 """Transform a help command found by the ``find()`` classmethod.
470 470 """
471 471 piece = ''.join(lines[self.start_line:self.q_line+1])
472 472 indent, content = piece[:self.start_col], piece[self.start_col:]
473 473 lines_before = lines[:self.start_line]
474 474 lines_after = lines[self.q_line + 1:]
475 475
476 476 m = _help_end_re.search(content)
477 477 if not m:
478 478 raise SyntaxError(content)
479 479 assert m is not None, content
480 480 target = m.group(1)
481 481 esc = m.group(3)
482 482
483 483 # If we're mid-command, put it back on the next prompt for the user.
484 484 next_input = None
485 485 if (not lines_before) and (not lines_after) \
486 486 and content.strip() != m.group(0):
487 487 next_input = content.rstrip('?\n')
488 488
489 489 call = _make_help_call(target, esc, next_input=next_input)
490 490 new_line = indent + call + '\n'
491 491
492 492 return lines_before + [new_line] + lines_after
493 493
494 494 def make_tokens_by_line(lines:List[str]):
495 495 """Tokenize a series of lines and group tokens by line.
496 496
497 497 The tokens for a multiline Python string or expression are grouped as one
498 498 line. All lines except the last lines should keep their line ending ('\\n',
499 499 '\\r\\n') for this to properly work. Use `.splitlines(keeplineending=True)`
500 500 for example when passing block of text to this function.
501 501
502 502 """
503 503 # NL tokens are used inside multiline expressions, but also after blank
504 504 # lines or comments. This is intentional - see https://bugs.python.org/issue17061
505 505 # We want to group the former case together but split the latter, so we
506 506 # track parentheses level, similar to the internals of tokenize.
507 507
508 508 # reexported from token on 3.7+
509 509 NEWLINE, NL = tokenize.NEWLINE, tokenize.NL # type: ignore
510 510 tokens_by_line:List[List[Any]] = [[]]
511 511 if len(lines) > 1 and not lines[0].endswith(('\n', '\r', '\r\n', '\x0b', '\x0c')):
512 512 warnings.warn("`make_tokens_by_line` received a list of lines which do not have lineending markers ('\\n', '\\r', '\\r\\n', '\\x0b', '\\x0c'), behavior will be unspecified")
513 513 parenlev = 0
514 514 try:
515 515 for token in tokenize.generate_tokens(iter(lines).__next__):
516 516 tokens_by_line[-1].append(token)
517 517 if (token.type == NEWLINE) \
518 518 or ((token.type == NL) and (parenlev <= 0)):
519 519 tokens_by_line.append([])
520 520 elif token.string in {'(', '[', '{'}:
521 521 parenlev += 1
522 522 elif token.string in {')', ']', '}'}:
523 523 if parenlev > 0:
524 524 parenlev -= 1
525 525 except tokenize.TokenError:
526 526 # Input ended in a multiline string or expression. That's OK for us.
527 527 pass
528 528
529 529
530 530 if not tokens_by_line[-1]:
531 531 tokens_by_line.pop()
532 532
533 533
534 534 return tokens_by_line
535 535
536 536
537 537 def has_sunken_brackets(tokens: List[tokenize.TokenInfo]):
538 538 """Check if the depth of brackets in the list of tokens drops below 0"""
539 539 parenlev = 0
540 540 for token in tokens:
541 541 if token.string in {"(", "[", "{"}:
542 542 parenlev += 1
543 543 elif token.string in {")", "]", "}"}:
544 544 parenlev -= 1
545 545 if parenlev < 0:
546 546 return True
547 547 return False
548 548
549 549
550 550 def show_linewise_tokens(s: str):
551 551 """For investigation and debugging"""
552 552 if not s.endswith('\n'):
553 553 s += '\n'
554 554 lines = s.splitlines(keepends=True)
555 555 for line in make_tokens_by_line(lines):
556 556 print("Line -------")
557 557 for tokinfo in line:
558 558 print(" ", tokinfo)
559 559
560 560 # Arbitrary limit to prevent getting stuck in infinite loops
561 561 TRANSFORM_LOOP_LIMIT = 500
562 562
563 563 class TransformerManager:
564 564 """Applies various transformations to a cell or code block.
565 565
566 566 The key methods for external use are ``transform_cell()``
567 567 and ``check_complete()``.
568 568 """
569 569 def __init__(self):
570 570 self.cleanup_transforms = [
571 571 leading_empty_lines,
572 572 leading_indent,
573 573 classic_prompt,
574 574 ipython_prompt,
575 575 ]
576 576 self.line_transforms = [
577 577 cell_magic,
578 578 ]
579 579 self.token_transformers = [
580 580 MagicAssign,
581 581 SystemAssign,
582 582 EscapedCommand,
583 583 HelpEnd,
584 584 ]
585 585
586 586 def do_one_token_transform(self, lines):
587 587 """Find and run the transform earliest in the code.
588 588
589 589 Returns (changed, lines).
590 590
591 591 This method is called repeatedly until changed is False, indicating
592 592 that all available transformations are complete.
593 593
594 594 The tokens following IPython special syntax might not be valid, so
595 595 the transformed code is retokenised every time to identify the next
596 596 piece of special syntax. Hopefully long code cells are mostly valid
597 597 Python, not using lots of IPython special syntax, so this shouldn't be
598 598 a performance issue.
599 599 """
600 600 tokens_by_line = make_tokens_by_line(lines)
601 601 candidates = []
602 602 for transformer_cls in self.token_transformers:
603 603 transformer = transformer_cls.find(tokens_by_line)
604 604 if transformer:
605 605 candidates.append(transformer)
606 606
607 607 if not candidates:
608 608 # Nothing to transform
609 609 return False, lines
610 610 ordered_transformers = sorted(candidates, key=TokenTransformBase.sortby)
611 611 for transformer in ordered_transformers:
612 612 try:
613 613 return True, transformer.transform(lines)
614 614 except SyntaxError:
615 615 pass
616 616 return False, lines
617 617
618 618 def do_token_transforms(self, lines):
619 619 for _ in range(TRANSFORM_LOOP_LIMIT):
620 620 changed, lines = self.do_one_token_transform(lines)
621 621 if not changed:
622 622 return lines
623 623
624 624 raise RuntimeError("Input transformation still changing after "
625 625 "%d iterations. Aborting." % TRANSFORM_LOOP_LIMIT)
626 626
627 627 def transform_cell(self, cell: str) -> str:
628 628 """Transforms a cell of input code"""
629 629 if not cell.endswith('\n'):
630 630 cell += '\n' # Ensure the cell has a trailing newline
631 631 lines = cell.splitlines(keepends=True)
632 632 for transform in self.cleanup_transforms + self.line_transforms:
633 633 lines = transform(lines)
634 634
635 635 lines = self.do_token_transforms(lines)
636 636 return ''.join(lines)
637 637
638 638 def check_complete(self, cell: str):
639 639 """Return whether a block of code is ready to execute, or should be continued
640 640
641 641 Parameters
642 642 ----------
643 source : string
644 Python input code, which can be multiline.
643 cell : string
644 Python input code, which can be multiline.
645 645
646 646 Returns
647 647 -------
648 648 status : str
649 One of 'complete', 'incomplete', or 'invalid' if source is not a
650 prefix of valid code.
649 One of 'complete', 'incomplete', or 'invalid' if source is not a
650 prefix of valid code.
651 651 indent_spaces : int or None
652 The number of spaces by which to indent the next line of code. If
653 status is not 'incomplete', this is None.
652 The number of spaces by which to indent the next line of code. If
653 status is not 'incomplete', this is None.
654 654 """
655 655 # Remember if the lines ends in a new line.
656 656 ends_with_newline = False
657 657 for character in reversed(cell):
658 658 if character == '\n':
659 659 ends_with_newline = True
660 660 break
661 661 elif character.strip():
662 662 break
663 663 else:
664 664 continue
665 665
666 666 if not ends_with_newline:
667 667 # Append an newline for consistent tokenization
668 668 # See https://bugs.python.org/issue33899
669 669 cell += '\n'
670 670
671 671 lines = cell.splitlines(keepends=True)
672 672
673 673 if not lines:
674 674 return 'complete', None
675 675
676 676 if lines[-1].endswith('\\'):
677 677 # Explicit backslash continuation
678 678 return 'incomplete', find_last_indent(lines)
679 679
680 680 try:
681 681 for transform in self.cleanup_transforms:
682 682 if not getattr(transform, 'has_side_effects', False):
683 683 lines = transform(lines)
684 684 except SyntaxError:
685 685 return 'invalid', None
686 686
687 687 if lines[0].startswith('%%'):
688 688 # Special case for cell magics - completion marked by blank line
689 689 if lines[-1].strip():
690 690 return 'incomplete', find_last_indent(lines)
691 691 else:
692 692 return 'complete', None
693 693
694 694 try:
695 695 for transform in self.line_transforms:
696 696 if not getattr(transform, 'has_side_effects', False):
697 697 lines = transform(lines)
698 698 lines = self.do_token_transforms(lines)
699 699 except SyntaxError:
700 700 return 'invalid', None
701 701
702 702 tokens_by_line = make_tokens_by_line(lines)
703 703
704 704 # Bail if we got one line and there are more closing parentheses than
705 705 # the opening ones
706 706 if (
707 707 len(lines) == 1
708 708 and tokens_by_line
709 709 and has_sunken_brackets(tokens_by_line[0])
710 710 ):
711 711 return "invalid", None
712 712
713 713 if not tokens_by_line:
714 714 return 'incomplete', find_last_indent(lines)
715 715
716 716 if tokens_by_line[-1][-1].type != tokenize.ENDMARKER:
717 717 # We're in a multiline string or expression
718 718 return 'incomplete', find_last_indent(lines)
719 719
720 720 newline_types = {tokenize.NEWLINE, tokenize.COMMENT, tokenize.ENDMARKER} # type: ignore
721 721
722 722 # Pop the last line which only contains DEDENTs and ENDMARKER
723 723 last_token_line = None
724 724 if {t.type for t in tokens_by_line[-1]} in [
725 725 {tokenize.DEDENT, tokenize.ENDMARKER},
726 726 {tokenize.ENDMARKER}
727 727 ] and len(tokens_by_line) > 1:
728 728 last_token_line = tokens_by_line.pop()
729 729
730 730 while tokens_by_line[-1] and tokens_by_line[-1][-1].type in newline_types:
731 731 tokens_by_line[-1].pop()
732 732
733 733 if not tokens_by_line[-1]:
734 734 return 'incomplete', find_last_indent(lines)
735 735
736 736 if tokens_by_line[-1][-1].string == ':':
737 737 # The last line starts a block (e.g. 'if foo:')
738 738 ix = 0
739 739 while tokens_by_line[-1][ix].type in {tokenize.INDENT, tokenize.DEDENT}:
740 740 ix += 1
741 741
742 742 indent = tokens_by_line[-1][ix].start[1]
743 743 return 'incomplete', indent + 4
744 744
745 745 if tokens_by_line[-1][0].line.endswith('\\'):
746 746 return 'incomplete', None
747 747
748 748 # At this point, our checks think the code is complete (or invalid).
749 749 # We'll use codeop.compile_command to check this with the real parser
750 750 try:
751 751 with warnings.catch_warnings():
752 752 warnings.simplefilter('error', SyntaxWarning)
753 753 res = compile_command(''.join(lines), symbol='exec')
754 754 except (SyntaxError, OverflowError, ValueError, TypeError,
755 755 MemoryError, SyntaxWarning):
756 756 return 'invalid', None
757 757 else:
758 758 if res is None:
759 759 return 'incomplete', find_last_indent(lines)
760 760
761 761 if last_token_line and last_token_line[0].type == tokenize.DEDENT:
762 762 if ends_with_newline:
763 763 return 'complete', None
764 764 return 'incomplete', find_last_indent(lines)
765 765
766 766 # If there's a blank line at the end, assume we're ready to execute
767 767 if not lines[-1].strip():
768 768 return 'complete', None
769 769
770 770 return 'complete', None
771 771
772 772
773 773 def find_last_indent(lines):
774 774 m = _indent_re.match(lines[-1])
775 775 if not m:
776 776 return 0
777 777 return len(m.group(0).replace('\t', ' '*4))
778 778
779 779
780 780 class MaybeAsyncCompile(Compile):
781 781 def __init__(self, extra_flags=0):
782 782 super().__init__()
783 783 self.flags |= extra_flags
784 784
785 785 def __call__(self, *args, **kwds):
786 786 return compile(*args, **kwds)
787 787
788 788
789 789 class MaybeAsyncCommandCompiler(CommandCompiler):
790 790 def __init__(self, extra_flags=0):
791 791 self.compiler = MaybeAsyncCompile(extra_flags=extra_flags)
792 792
793 793
794 794 _extra_flags = ast.PyCF_ALLOW_TOP_LEVEL_AWAIT
795 795
796 796 compile_command = MaybeAsyncCommandCompiler(extra_flags=_extra_flags)
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
@@ -1,712 +1,700 b''
1 1 # encoding: utf-8
2 2 """Magic functions for InteractiveShell.
3 3 """
4 4
5 5 #-----------------------------------------------------------------------------
6 6 # Copyright (C) 2001 Janko Hauser <jhauser@zscout.de> and
7 7 # Copyright (C) 2001 Fernando Perez <fperez@colorado.edu>
8 8 # Copyright (C) 2008 The IPython Development Team
9 9
10 10 # Distributed under the terms of the BSD License. The full license is in
11 11 # the file COPYING, distributed as part of this software.
12 12 #-----------------------------------------------------------------------------
13 13
14 14 import os
15 15 import re
16 16 import sys
17 17 from getopt import getopt, GetoptError
18 18
19 19 from traitlets.config.configurable import Configurable
20 20 from . import oinspect
21 21 from .error import UsageError
22 22 from .inputtransformer2 import ESC_MAGIC, ESC_MAGIC2
23 23 from ..utils.ipstruct import Struct
24 24 from ..utils.process import arg_split
25 25 from ..utils.text import dedent
26 26 from traitlets import Bool, Dict, Instance, observe
27 27 from logging import error
28 28
29 29 #-----------------------------------------------------------------------------
30 30 # Globals
31 31 #-----------------------------------------------------------------------------
32 32
33 33 # A dict we'll use for each class that has magics, used as temporary storage to
34 34 # pass information between the @line/cell_magic method decorators and the
35 35 # @magics_class class decorator, because the method decorators have no
36 36 # access to the class when they run. See for more details:
37 37 # http://stackoverflow.com/questions/2366713/can-a-python-decorator-of-an-instance-method-access-the-class
38 38
39 39 magics = dict(line={}, cell={})
40 40
41 41 magic_kinds = ('line', 'cell')
42 42 magic_spec = ('line', 'cell', 'line_cell')
43 43 magic_escapes = dict(line=ESC_MAGIC, cell=ESC_MAGIC2)
44 44
45 45 #-----------------------------------------------------------------------------
46 46 # Utility classes and functions
47 47 #-----------------------------------------------------------------------------
48 48
49 49 class Bunch: pass
50 50
51 51
52 52 def on_off(tag):
53 53 """Return an ON/OFF string for a 1/0 input. Simple utility function."""
54 54 return ['OFF','ON'][tag]
55 55
56 56
57 57 def compress_dhist(dh):
58 58 """Compress a directory history into a new one with at most 20 entries.
59 59
60 60 Return a new list made from the first and last 10 elements of dhist after
61 61 removal of duplicates.
62 62 """
63 63 head, tail = dh[:-10], dh[-10:]
64 64
65 65 newhead = []
66 66 done = set()
67 67 for h in head:
68 68 if h in done:
69 69 continue
70 70 newhead.append(h)
71 71 done.add(h)
72 72
73 73 return newhead + tail
74 74
75 75
76 76 def needs_local_scope(func):
77 77 """Decorator to mark magic functions which need to local scope to run."""
78 78 func.needs_local_scope = True
79 79 return func
80 80
81 81 #-----------------------------------------------------------------------------
82 82 # Class and method decorators for registering magics
83 83 #-----------------------------------------------------------------------------
84 84
85 85 def magics_class(cls):
86 86 """Class decorator for all subclasses of the main Magics class.
87 87
88 88 Any class that subclasses Magics *must* also apply this decorator, to
89 89 ensure that all the methods that have been decorated as line/cell magics
90 90 get correctly registered in the class instance. This is necessary because
91 91 when method decorators run, the class does not exist yet, so they
92 92 temporarily store their information into a module global. Application of
93 93 this class decorator copies that global data to the class instance and
94 94 clears the global.
95 95
96 96 Obviously, this mechanism is not thread-safe, which means that the
97 97 *creation* of subclasses of Magic should only be done in a single-thread
98 98 context. Instantiation of the classes has no restrictions. Given that
99 99 these classes are typically created at IPython startup time and before user
100 100 application code becomes active, in practice this should not pose any
101 101 problems.
102 102 """
103 103 cls.registered = True
104 104 cls.magics = dict(line = magics['line'],
105 105 cell = magics['cell'])
106 106 magics['line'] = {}
107 107 magics['cell'] = {}
108 108 return cls
109 109
110 110
111 111 def record_magic(dct, magic_kind, magic_name, func):
112 112 """Utility function to store a function as a magic of a specific kind.
113 113
114 114 Parameters
115 115 ----------
116 116 dct : dict
117 A dictionary with 'line' and 'cell' subdicts.
118
117 A dictionary with 'line' and 'cell' subdicts.
119 118 magic_kind : str
120 Kind of magic to be stored.
121
119 Kind of magic to be stored.
122 120 magic_name : str
123 Key to store the magic as.
124
121 Key to store the magic as.
125 122 func : function
126 Callable object to store.
123 Callable object to store.
127 124 """
128 125 if magic_kind == 'line_cell':
129 126 dct['line'][magic_name] = dct['cell'][magic_name] = func
130 127 else:
131 128 dct[magic_kind][magic_name] = func
132 129
133 130
134 131 def validate_type(magic_kind):
135 132 """Ensure that the given magic_kind is valid.
136 133
137 134 Check that the given magic_kind is one of the accepted spec types (stored
138 135 in the global `magic_spec`), raise ValueError otherwise.
139 136 """
140 137 if magic_kind not in magic_spec:
141 138 raise ValueError('magic_kind must be one of %s, %s given' %
142 139 magic_kinds, magic_kind)
143 140
144 141
145 142 # The docstrings for the decorator below will be fairly similar for the two
146 143 # types (method and function), so we generate them here once and reuse the
147 144 # templates below.
148 145 _docstring_template = \
149 146 """Decorate the given {0} as {1} magic.
150 147
151 148 The decorator can be used with or without arguments, as follows.
152 149
153 150 i) without arguments: it will create a {1} magic named as the {0} being
154 151 decorated::
155 152
156 153 @deco
157 154 def foo(...)
158 155
159 156 will create a {1} magic named `foo`.
160 157
161 158 ii) with one string argument: which will be used as the actual name of the
162 159 resulting magic::
163 160
164 161 @deco('bar')
165 162 def foo(...)
166 163
167 164 will create a {1} magic named `bar`.
168 165
169 166 To register a class magic use ``Interactiveshell.register_magic(class or instance)``.
170 167 """
171 168
172 169 # These two are decorator factories. While they are conceptually very similar,
173 170 # there are enough differences in the details that it's simpler to have them
174 171 # written as completely standalone functions rather than trying to share code
175 172 # and make a single one with convoluted logic.
176 173
177 174 def _method_magic_marker(magic_kind):
178 175 """Decorator factory for methods in Magics subclasses.
179 176 """
180 177
181 178 validate_type(magic_kind)
182 179
183 180 # This is a closure to capture the magic_kind. We could also use a class,
184 181 # but it's overkill for just that one bit of state.
185 182 def magic_deco(arg):
186 183 if callable(arg):
187 184 # "Naked" decorator call (just @foo, no args)
188 185 func = arg
189 186 name = func.__name__
190 187 retval = arg
191 188 record_magic(magics, magic_kind, name, name)
192 189 elif isinstance(arg, str):
193 190 # Decorator called with arguments (@foo('bar'))
194 191 name = arg
195 192 def mark(func, *a, **kw):
196 193 record_magic(magics, magic_kind, name, func.__name__)
197 194 return func
198 195 retval = mark
199 196 else:
200 197 raise TypeError("Decorator can only be called with "
201 198 "string or function")
202 199 return retval
203 200
204 201 # Ensure the resulting decorator has a usable docstring
205 202 magic_deco.__doc__ = _docstring_template.format('method', magic_kind)
206 203 return magic_deco
207 204
208 205
209 206 def _function_magic_marker(magic_kind):
210 207 """Decorator factory for standalone functions.
211 208 """
212 209 validate_type(magic_kind)
213 210
214 211 # This is a closure to capture the magic_kind. We could also use a class,
215 212 # but it's overkill for just that one bit of state.
216 213 def magic_deco(arg):
217 214 # Find get_ipython() in the caller's namespace
218 215 caller = sys._getframe(1)
219 216 for ns in ['f_locals', 'f_globals', 'f_builtins']:
220 217 get_ipython = getattr(caller, ns).get('get_ipython')
221 218 if get_ipython is not None:
222 219 break
223 220 else:
224 221 raise NameError('Decorator can only run in context where '
225 222 '`get_ipython` exists')
226 223
227 224 ip = get_ipython()
228 225
229 226 if callable(arg):
230 227 # "Naked" decorator call (just @foo, no args)
231 228 func = arg
232 229 name = func.__name__
233 230 ip.register_magic_function(func, magic_kind, name)
234 231 retval = arg
235 232 elif isinstance(arg, str):
236 233 # Decorator called with arguments (@foo('bar'))
237 234 name = arg
238 235 def mark(func, *a, **kw):
239 236 ip.register_magic_function(func, magic_kind, name)
240 237 return func
241 238 retval = mark
242 239 else:
243 240 raise TypeError("Decorator can only be called with "
244 241 "string or function")
245 242 return retval
246 243
247 244 # Ensure the resulting decorator has a usable docstring
248 245 ds = _docstring_template.format('function', magic_kind)
249 246
250 247 ds += dedent("""
251 248 Note: this decorator can only be used in a context where IPython is already
252 249 active, so that the `get_ipython()` call succeeds. You can therefore use
253 250 it in your startup files loaded after IPython initializes, but *not* in the
254 251 IPython configuration file itself, which is executed before IPython is
255 252 fully up and running. Any file located in the `startup` subdirectory of
256 253 your configuration profile will be OK in this sense.
257 254 """)
258 255
259 256 magic_deco.__doc__ = ds
260 257 return magic_deco
261 258
262 259
263 260 MAGIC_NO_VAR_EXPAND_ATTR = '_ipython_magic_no_var_expand'
264 261
265 262
266 263 def no_var_expand(magic_func):
267 264 """Mark a magic function as not needing variable expansion
268 265
269 266 By default, IPython interprets `{a}` or `$a` in the line passed to magics
270 267 as variables that should be interpolated from the interactive namespace
271 268 before passing the line to the magic function.
272 269 This is not always desirable, e.g. when the magic executes Python code
273 270 (%timeit, %time, etc.).
274 271 Decorate magics with `@no_var_expand` to opt-out of variable expansion.
275 272
276 273 .. versionadded:: 7.3
277 274 """
278 275 setattr(magic_func, MAGIC_NO_VAR_EXPAND_ATTR, True)
279 276 return magic_func
280 277
281 278
282 279 # Create the actual decorators for public use
283 280
284 281 # These three are used to decorate methods in class definitions
285 282 line_magic = _method_magic_marker('line')
286 283 cell_magic = _method_magic_marker('cell')
287 284 line_cell_magic = _method_magic_marker('line_cell')
288 285
289 286 # These three decorate standalone functions and perform the decoration
290 287 # immediately. They can only run where get_ipython() works
291 288 register_line_magic = _function_magic_marker('line')
292 289 register_cell_magic = _function_magic_marker('cell')
293 290 register_line_cell_magic = _function_magic_marker('line_cell')
294 291
295 292 #-----------------------------------------------------------------------------
296 293 # Core Magic classes
297 294 #-----------------------------------------------------------------------------
298 295
299 296 class MagicsManager(Configurable):
300 297 """Object that handles all magic-related functionality for IPython.
301 298 """
302 299 # Non-configurable class attributes
303 300
304 301 # A two-level dict, first keyed by magic type, then by magic function, and
305 302 # holding the actual callable object as value. This is the dict used for
306 303 # magic function dispatch
307 304 magics = Dict()
308 305
309 306 # A registry of the original objects that we've been given holding magics.
310 307 registry = Dict()
311 308
312 309 shell = Instance('IPython.core.interactiveshell.InteractiveShellABC', allow_none=True)
313 310
314 311 auto_magic = Bool(True, help=
315 312 "Automatically call line magics without requiring explicit % prefix"
316 313 ).tag(config=True)
317 314 @observe('auto_magic')
318 315 def _auto_magic_changed(self, change):
319 316 self.shell.automagic = change['new']
320 317
321 318 _auto_status = [
322 319 'Automagic is OFF, % prefix IS needed for line magics.',
323 320 'Automagic is ON, % prefix IS NOT needed for line magics.']
324 321
325 322 user_magics = Instance('IPython.core.magics.UserMagics', allow_none=True)
326 323
327 324 def __init__(self, shell=None, config=None, user_magics=None, **traits):
328 325
329 326 super(MagicsManager, self).__init__(shell=shell, config=config,
330 327 user_magics=user_magics, **traits)
331 328 self.magics = dict(line={}, cell={})
332 329 # Let's add the user_magics to the registry for uniformity, so *all*
333 330 # registered magic containers can be found there.
334 331 self.registry[user_magics.__class__.__name__] = user_magics
335 332
336 333 def auto_status(self):
337 334 """Return descriptive string with automagic status."""
338 335 return self._auto_status[self.auto_magic]
339 336
340 337 def lsmagic(self):
341 338 """Return a dict of currently available magic functions.
342 339
343 340 The return dict has the keys 'line' and 'cell', corresponding to the
344 341 two types of magics we support. Each value is a list of names.
345 342 """
346 343 return self.magics
347 344
348 345 def lsmagic_docs(self, brief=False, missing=''):
349 346 """Return dict of documentation of magic functions.
350 347
351 348 The return dict has the keys 'line' and 'cell', corresponding to the
352 349 two types of magics we support. Each value is a dict keyed by magic
353 350 name whose value is the function docstring. If a docstring is
354 351 unavailable, the value of `missing` is used instead.
355 352
356 353 If brief is True, only the first line of each docstring will be returned.
357 354 """
358 355 docs = {}
359 356 for m_type in self.magics:
360 357 m_docs = {}
361 358 for m_name, m_func in self.magics[m_type].items():
362 359 if m_func.__doc__:
363 360 if brief:
364 361 m_docs[m_name] = m_func.__doc__.split('\n', 1)[0]
365 362 else:
366 363 m_docs[m_name] = m_func.__doc__.rstrip()
367 364 else:
368 365 m_docs[m_name] = missing
369 366 docs[m_type] = m_docs
370 367 return docs
371 368
372 369 def register(self, *magic_objects):
373 370 """Register one or more instances of Magics.
374 371
375 Take one or more classes or instances of classes that subclass the main
372 Take one or more classes or instances of classes that subclass the main
376 373 `core.Magic` class, and register them with IPython to use the magic
377 374 functions they provide. The registration process will then ensure that
378 375 any methods that have decorated to provide line and/or cell magics will
379 376 be recognized with the `%x`/`%%x` syntax as a line/cell magic
380 377 respectively.
381 378
382 379 If classes are given, they will be instantiated with the default
383 380 constructor. If your classes need a custom constructor, you should
384 381 instanitate them first and pass the instance.
385 382
386 383 The provided arguments can be an arbitrary mix of classes and instances.
387 384
388 385 Parameters
389 386 ----------
390 magic_objects : one or more classes or instances
387 *magic_objects : one or more classes or instances
391 388 """
392 389 # Start by validating them to ensure they have all had their magic
393 390 # methods registered at the instance level
394 391 for m in magic_objects:
395 392 if not m.registered:
396 393 raise ValueError("Class of magics %r was constructed without "
397 394 "the @register_magics class decorator")
398 395 if isinstance(m, type):
399 396 # If we're given an uninstantiated class
400 397 m = m(shell=self.shell)
401 398
402 399 # Now that we have an instance, we can register it and update the
403 400 # table of callables
404 401 self.registry[m.__class__.__name__] = m
405 402 for mtype in magic_kinds:
406 403 self.magics[mtype].update(m.magics[mtype])
407 404
408 405 def register_function(self, func, magic_kind='line', magic_name=None):
409 406 """Expose a standalone function as magic function for IPython.
410 407
411 408 This will create an IPython magic (line, cell or both) from a
412 409 standalone function. The functions should have the following
413 signatures:
410 signatures:
414 411
415 412 * For line magics: `def f(line)`
416 413 * For cell magics: `def f(line, cell)`
417 414 * For a function that does both: `def f(line, cell=None)`
418 415
419 416 In the latter case, the function will be called with `cell==None` when
420 417 invoked as `%f`, and with cell as a string when invoked as `%%f`.
421 418
422 419 Parameters
423 420 ----------
424 421 func : callable
425 Function to be registered as a magic.
426
422 Function to be registered as a magic.
427 423 magic_kind : str
428 Kind of magic, one of 'line', 'cell' or 'line_cell'
429
424 Kind of magic, one of 'line', 'cell' or 'line_cell'
430 425 magic_name : optional str
431 If given, the name the magic will have in the IPython namespace. By
432 default, the name of the function itself is used.
426 If given, the name the magic will have in the IPython namespace. By
427 default, the name of the function itself is used.
433 428 """
434 429
435 430 # Create the new method in the user_magics and register it in the
436 431 # global table
437 432 validate_type(magic_kind)
438 433 magic_name = func.__name__ if magic_name is None else magic_name
439 434 setattr(self.user_magics, magic_name, func)
440 435 record_magic(self.magics, magic_kind, magic_name, func)
441 436
442 437 def register_alias(self, alias_name, magic_name, magic_kind='line', magic_params=None):
443 438 """Register an alias to a magic function.
444 439
445 440 The alias is an instance of :class:`MagicAlias`, which holds the
446 441 name and kind of the magic it should call. Binding is done at
447 442 call time, so if the underlying magic function is changed the alias
448 443 will call the new function.
449 444
450 445 Parameters
451 446 ----------
452 447 alias_name : str
453 The name of the magic to be registered.
454
448 The name of the magic to be registered.
455 449 magic_name : str
456 The name of an existing magic.
457
450 The name of an existing magic.
458 451 magic_kind : str
459 Kind of magic, one of 'line' or 'cell'
452 Kind of magic, one of 'line' or 'cell'
460 453 """
461 454
462 455 # `validate_type` is too permissive, as it allows 'line_cell'
463 456 # which we do not handle.
464 457 if magic_kind not in magic_kinds:
465 458 raise ValueError('magic_kind must be one of %s, %s given' %
466 459 magic_kinds, magic_kind)
467 460
468 461 alias = MagicAlias(self.shell, magic_name, magic_kind, magic_params)
469 462 setattr(self.user_magics, alias_name, alias)
470 463 record_magic(self.magics, magic_kind, alias_name, alias)
471 464
472 465 # Key base class that provides the central functionality for magics.
473 466
474 467
475 468 class Magics(Configurable):
476 469 """Base class for implementing magic functions.
477 470
478 471 Shell functions which can be reached as %function_name. All magic
479 472 functions should accept a string, which they can parse for their own
480 473 needs. This can make some functions easier to type, eg `%cd ../`
481 474 vs. `%cd("../")`
482 475
483 476 Classes providing magic functions need to subclass this class, and they
484 477 MUST:
485 478
486 479 - Use the method decorators `@line_magic` and `@cell_magic` to decorate
487 480 individual methods as magic functions, AND
488 481
489 482 - Use the class decorator `@magics_class` to ensure that the magic
490 483 methods are properly registered at the instance level upon instance
491 484 initialization.
492 485
493 486 See :mod:`magic_functions` for examples of actual implementation classes.
494 487 """
495 488 # Dict holding all command-line options for each magic.
496 489 options_table = None
497 490 # Dict for the mapping of magic names to methods, set by class decorator
498 491 magics = None
499 492 # Flag to check that the class decorator was properly applied
500 493 registered = False
501 494 # Instance of IPython shell
502 495 shell = None
503 496
504 497 def __init__(self, shell=None, **kwargs):
505 498 if not(self.__class__.registered):
506 499 raise ValueError('Magics subclass without registration - '
507 500 'did you forget to apply @magics_class?')
508 501 if shell is not None:
509 502 if hasattr(shell, 'configurables'):
510 503 shell.configurables.append(self)
511 504 if hasattr(shell, 'config'):
512 505 kwargs.setdefault('parent', shell)
513 506
514 507 self.shell = shell
515 508 self.options_table = {}
516 509 # The method decorators are run when the instance doesn't exist yet, so
517 510 # they can only record the names of the methods they are supposed to
518 511 # grab. Only now, that the instance exists, can we create the proper
519 512 # mapping to bound methods. So we read the info off the original names
520 513 # table and replace each method name by the actual bound method.
521 514 # But we mustn't clobber the *class* mapping, in case of multiple instances.
522 515 class_magics = self.magics
523 516 self.magics = {}
524 517 for mtype in magic_kinds:
525 518 tab = self.magics[mtype] = {}
526 519 cls_tab = class_magics[mtype]
527 520 for magic_name, meth_name in cls_tab.items():
528 521 if isinstance(meth_name, str):
529 522 # it's a method name, grab it
530 523 tab[magic_name] = getattr(self, meth_name)
531 524 else:
532 525 # it's the real thing
533 526 tab[magic_name] = meth_name
534 527 # Configurable **needs** to be initiated at the end or the config
535 528 # magics get screwed up.
536 529 super(Magics, self).__init__(**kwargs)
537 530
538 531 def arg_err(self,func):
539 532 """Print docstring if incorrect arguments were passed"""
540 533 print('Error in arguments:')
541 534 print(oinspect.getdoc(func))
542 535
543 536 def format_latex(self, strng):
544 537 """Format a string for latex inclusion."""
545 538
546 539 # Characters that need to be escaped for latex:
547 540 escape_re = re.compile(r'(%|_|\$|#|&)',re.MULTILINE)
548 541 # Magic command names as headers:
549 542 cmd_name_re = re.compile(r'^(%s.*?):' % ESC_MAGIC,
550 543 re.MULTILINE)
551 544 # Magic commands
552 545 cmd_re = re.compile(r'(?P<cmd>%s.+?\b)(?!\}\}:)' % ESC_MAGIC,
553 546 re.MULTILINE)
554 547 # Paragraph continue
555 548 par_re = re.compile(r'\\$',re.MULTILINE)
556 549
557 550 # The "\n" symbol
558 551 newline_re = re.compile(r'\\n')
559 552
560 553 # Now build the string for output:
561 554 #strng = cmd_name_re.sub(r'\n\\texttt{\\textsl{\\large \1}}:',strng)
562 555 strng = cmd_name_re.sub(r'\n\\bigskip\n\\texttt{\\textbf{ \1}}:',
563 556 strng)
564 557 strng = cmd_re.sub(r'\\texttt{\g<cmd>}',strng)
565 558 strng = par_re.sub(r'\\\\',strng)
566 559 strng = escape_re.sub(r'\\\1',strng)
567 560 strng = newline_re.sub(r'\\textbackslash{}n',strng)
568 561 return strng
569 562
570 563 def parse_options(self, arg_str, opt_str, *long_opts, **kw):
571 564 """Parse options passed to an argument string.
572 565
573 566 The interface is similar to that of :func:`getopt.getopt`, but it
574 567 returns a :class:`~IPython.utils.struct.Struct` with the options as keys
575 568 and the stripped argument string still as a string.
576 569
577 570 arg_str is quoted as a true sys.argv vector by using shlex.split.
578 571 This allows us to easily expand variables, glob files, quote
579 572 arguments, etc.
580 573
581 574 Parameters
582 575 ----------
583
584 576 arg_str : str
585 The arguments to parse.
586
577 The arguments to parse.
587 578 opt_str : str
588 The options specification.
589
579 The options specification.
590 580 mode : str, default 'string'
591 If given as 'list', the argument string is returned as a list (split
592 on whitespace) instead of a string.
593
581 If given as 'list', the argument string is returned as a list (split
582 on whitespace) instead of a string.
594 583 list_all : bool, default False
595 Put all option values in lists. Normally only options
596 appearing more than once are put in a list.
597
584 Put all option values in lists. Normally only options
585 appearing more than once are put in a list.
598 586 posix : bool, default True
599 Whether to split the input line in POSIX mode or not, as per the
600 conventions outlined in the :mod:`shlex` module from the standard
601 library.
587 Whether to split the input line in POSIX mode or not, as per the
588 conventions outlined in the :mod:`shlex` module from the standard
589 library.
602 590 """
603 591
604 592 # inject default options at the beginning of the input line
605 593 caller = sys._getframe(1).f_code.co_name
606 594 arg_str = '%s %s' % (self.options_table.get(caller,''),arg_str)
607 595
608 596 mode = kw.get('mode','string')
609 597 if mode not in ['string','list']:
610 598 raise ValueError('incorrect mode given: %s' % mode)
611 599 # Get options
612 600 list_all = kw.get('list_all',0)
613 601 posix = kw.get('posix', os.name == 'posix')
614 602 strict = kw.get('strict', True)
615 603
616 604 preserve_non_opts = kw.get("preserve_non_opts", False)
617 605 remainder_arg_str = arg_str
618 606
619 607 # Check if we have more than one argument to warrant extra processing:
620 608 odict = {} # Dictionary with options
621 609 args = arg_str.split()
622 610 if len(args) >= 1:
623 611 # If the list of inputs only has 0 or 1 thing in it, there's no
624 612 # need to look for options
625 613 argv = arg_split(arg_str, posix, strict)
626 614 # Do regular option processing
627 615 try:
628 616 opts,args = getopt(argv, opt_str, long_opts)
629 617 except GetoptError as e:
630 618 raise UsageError(
631 619 '%s ( allowed: "%s" %s)' % (e.msg, opt_str, " ".join(long_opts))
632 620 ) from e
633 621 for o, a in opts:
634 622 if mode == "string" and preserve_non_opts:
635 623 # remove option-parts from the original args-string and preserve remaining-part.
636 624 # This relies on the arg_split(...) and getopt(...)'s impl spec, that the parsed options are
637 625 # returned in the original order.
638 626 remainder_arg_str = remainder_arg_str.replace(o, "", 1).replace(
639 627 a, "", 1
640 628 )
641 629 if o.startswith("--"):
642 630 o = o[2:]
643 631 else:
644 632 o = o[1:]
645 633 try:
646 634 odict[o].append(a)
647 635 except AttributeError:
648 636 odict[o] = [odict[o],a]
649 637 except KeyError:
650 638 if list_all:
651 639 odict[o] = [a]
652 640 else:
653 641 odict[o] = a
654 642
655 643 # Prepare opts,args for return
656 644 opts = Struct(odict)
657 645 if mode == 'string':
658 646 if preserve_non_opts:
659 647 args = remainder_arg_str.lstrip()
660 648 else:
661 649 args = " ".join(args)
662 650
663 651 return opts,args
664 652
665 653 def default_option(self, fn, optstr):
666 654 """Make an entry in the options_table for fn, with value optstr"""
667 655
668 656 if fn not in self.lsmagic():
669 657 error("%s is not a magic function" % fn)
670 658 self.options_table[fn] = optstr
671 659
672 660
673 661 class MagicAlias(object):
674 662 """An alias to another magic function.
675 663
676 664 An alias is determined by its magic name and magic kind. Lookup
677 665 is done at call time, so if the underlying magic changes the alias
678 666 will call the new function.
679 667
680 668 Use the :meth:`MagicsManager.register_alias` method or the
681 669 `%alias_magic` magic function to create and register a new alias.
682 670 """
683 671 def __init__(self, shell, magic_name, magic_kind, magic_params=None):
684 672 self.shell = shell
685 673 self.magic_name = magic_name
686 674 self.magic_params = magic_params
687 675 self.magic_kind = magic_kind
688 676
689 677 self.pretty_target = '%s%s' % (magic_escapes[self.magic_kind], self.magic_name)
690 678 self.__doc__ = "Alias for `%s`." % self.pretty_target
691 679
692 680 self._in_call = False
693 681
694 682 def __call__(self, *args, **kwargs):
695 683 """Call the magic alias."""
696 684 fn = self.shell.find_magic(self.magic_name, self.magic_kind)
697 685 if fn is None:
698 686 raise UsageError("Magic `%s` not found." % self.pretty_target)
699 687
700 688 # Protect against infinite recursion.
701 689 if self._in_call:
702 690 raise UsageError("Infinite recursion detected; "
703 691 "magic aliases cannot call themselves.")
704 692 self._in_call = True
705 693 try:
706 694 if self.magic_params:
707 695 args_list = list(args)
708 696 args_list[0] = self.magic_params + " " + args[0]
709 697 args = tuple(args_list)
710 698 return fn(*args, **kwargs)
711 699 finally:
712 700 self._in_call = False
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
General Comments 0
You need to be logged in to leave comments. Login now