##// END OF EJS Templates
Merge pull request #13205 from DimitriPapadopoulos/codespell...
Matthias Bussonnier -
r26877:28633241 merge
parent child Browse files
Show More

The requested changes are too big and content was truncated. Show full diff

@@ -1,2239 +1,2239 b''
1 """Completion for IPython.
1 """Completion for IPython.
2
2
3 This module started as fork of the rlcompleter module in the Python standard
3 This module started as fork of the rlcompleter module in the Python standard
4 library. The original enhancements made to rlcompleter have been sent
4 library. The original enhancements made to rlcompleter have been sent
5 upstream and were accepted as of Python 2.3,
5 upstream and were accepted as of Python 2.3,
6
6
7 This module now support a wide variety of completion mechanism both available
7 This module now support a wide variety of completion mechanism both available
8 for normal classic Python code, as well as completer for IPython specific
8 for normal classic Python code, as well as completer for IPython specific
9 Syntax like magics.
9 Syntax like magics.
10
10
11 Latex and Unicode completion
11 Latex and Unicode completion
12 ============================
12 ============================
13
13
14 IPython and compatible frontends not only can complete your code, but can help
14 IPython and compatible frontends not only can complete your code, but can help
15 you to input a wide range of characters. In particular we allow you to insert
15 you to input a wide range of characters. In particular we allow you to insert
16 a unicode character using the tab completion mechanism.
16 a unicode character using the tab completion mechanism.
17
17
18 Forward latex/unicode completion
18 Forward latex/unicode completion
19 --------------------------------
19 --------------------------------
20
20
21 Forward completion allows you to easily type a unicode character using its latex
21 Forward completion allows you to easily type a unicode character using its latex
22 name, or unicode long description. To do so type a backslash follow by the
22 name, or unicode long description. To do so type a backslash follow by the
23 relevant name and press tab:
23 relevant name and press tab:
24
24
25
25
26 Using latex completion:
26 Using latex completion:
27
27
28 .. code::
28 .. code::
29
29
30 \\alpha<tab>
30 \\alpha<tab>
31 α
31 α
32
32
33 or using unicode completion:
33 or using unicode completion:
34
34
35
35
36 .. code::
36 .. code::
37
37
38 \\GREEK SMALL LETTER ALPHA<tab>
38 \\GREEK SMALL LETTER ALPHA<tab>
39 α
39 α
40
40
41
41
42 Only valid Python identifiers will complete. Combining characters (like arrow or
42 Only valid Python identifiers will complete. Combining characters (like arrow or
43 dots) are also available, unlike latex they need to be put after the their
43 dots) are also available, unlike latex they need to be put after the their
44 counterpart that is to say, `F\\\\vec<tab>` is correct, not `\\\\vec<tab>F`.
44 counterpart that is to say, `F\\\\vec<tab>` is correct, not `\\\\vec<tab>F`.
45
45
46 Some browsers are known to display combining characters incorrectly.
46 Some browsers are known to display combining characters incorrectly.
47
47
48 Backward latex completion
48 Backward latex completion
49 -------------------------
49 -------------------------
50
50
51 It is sometime challenging to know how to type a character, if you are using
51 It is sometime challenging to know how to type a character, if you are using
52 IPython, or any compatible frontend you can prepend backslash to the character
52 IPython, or any compatible frontend you can prepend backslash to the character
53 and press `<tab>` to expand it to its latex form.
53 and press `<tab>` to expand it to its latex form.
54
54
55 .. code::
55 .. code::
56
56
57 \\α<tab>
57 \\α<tab>
58 \\alpha
58 \\alpha
59
59
60
60
61 Both forward and backward completions can be deactivated by setting the
61 Both forward and backward completions can be deactivated by setting the
62 ``Completer.backslash_combining_completions`` option to ``False``.
62 ``Completer.backslash_combining_completions`` option to ``False``.
63
63
64
64
65 Experimental
65 Experimental
66 ============
66 ============
67
67
68 Starting with IPython 6.0, this module can make use of the Jedi library to
68 Starting with IPython 6.0, this module can make use of the Jedi library to
69 generate completions both using static analysis of the code, and dynamically
69 generate completions both using static analysis of the code, and dynamically
70 inspecting multiple namespaces. Jedi is an autocompletion and static analysis
70 inspecting multiple namespaces. Jedi is an autocompletion and static analysis
71 for Python. The APIs attached to this new mechanism is unstable and will
71 for Python. The APIs attached to this new mechanism is unstable and will
72 raise unless use in an :any:`provisionalcompleter` context manager.
72 raise unless use in an :any:`provisionalcompleter` context manager.
73
73
74 You will find that the following are experimental:
74 You will find that the following are experimental:
75
75
76 - :any:`provisionalcompleter`
76 - :any:`provisionalcompleter`
77 - :any:`IPCompleter.completions`
77 - :any:`IPCompleter.completions`
78 - :any:`Completion`
78 - :any:`Completion`
79 - :any:`rectify_completions`
79 - :any:`rectify_completions`
80
80
81 .. note::
81 .. note::
82
82
83 better name for :any:`rectify_completions` ?
83 better name for :any:`rectify_completions` ?
84
84
85 We welcome any feedback on these new API, and we also encourage you to try this
85 We welcome any feedback on these new API, and we also encourage you to try this
86 module in debug mode (start IPython with ``--Completer.debug=True``) in order
86 module in debug mode (start IPython with ``--Completer.debug=True``) in order
87 to have extra logging information if :any:`jedi` is crashing, or if current
87 to have extra logging information if :any:`jedi` is crashing, or if current
88 IPython completer pending deprecations are returning results not yet handled
88 IPython completer pending deprecations are returning results not yet handled
89 by :any:`jedi`
89 by :any:`jedi`
90
90
91 Using Jedi for tab completion allow snippets like the following to work without
91 Using Jedi for tab completion allow snippets like the following to work without
92 having to execute any code:
92 having to execute any code:
93
93
94 >>> myvar = ['hello', 42]
94 >>> myvar = ['hello', 42]
95 ... myvar[1].bi<tab>
95 ... myvar[1].bi<tab>
96
96
97 Tab completion will be able to infer that ``myvar[1]`` is a real number without
97 Tab completion will be able to infer that ``myvar[1]`` is a real number without
98 executing any code unlike the previously available ``IPCompleter.greedy``
98 executing any code unlike the previously available ``IPCompleter.greedy``
99 option.
99 option.
100
100
101 Be sure to update :any:`jedi` to the latest stable version or to try the
101 Be sure to update :any:`jedi` to the latest stable version or to try the
102 current development version to get better completions.
102 current development version to get better completions.
103 """
103 """
104
104
105
105
106 # Copyright (c) IPython Development Team.
106 # Copyright (c) IPython Development Team.
107 # Distributed under the terms of the Modified BSD License.
107 # Distributed under the terms of the Modified BSD License.
108 #
108 #
109 # Some of this code originated from rlcompleter in the Python standard library
109 # Some of this code originated from rlcompleter in the Python standard library
110 # Copyright (C) 2001 Python Software Foundation, www.python.org
110 # Copyright (C) 2001 Python Software Foundation, www.python.org
111
111
112
112
113 import builtins as builtin_mod
113 import builtins as builtin_mod
114 import glob
114 import glob
115 import inspect
115 import inspect
116 import itertools
116 import itertools
117 import keyword
117 import keyword
118 import os
118 import os
119 import re
119 import re
120 import string
120 import string
121 import sys
121 import sys
122 import time
122 import time
123 import unicodedata
123 import unicodedata
124 import uuid
124 import uuid
125 import warnings
125 import warnings
126 from contextlib import contextmanager
126 from contextlib import contextmanager
127 from importlib import import_module
127 from importlib import import_module
128 from types import SimpleNamespace
128 from types import SimpleNamespace
129 from typing import Iterable, Iterator, List, Tuple, Union, Any, Sequence, Dict, NamedTuple, Pattern, Optional
129 from typing import Iterable, Iterator, List, Tuple, Union, Any, Sequence, Dict, NamedTuple, Pattern, Optional
130
130
131 from IPython.core.error import TryNext
131 from IPython.core.error import TryNext
132 from IPython.core.inputtransformer2 import ESC_MAGIC
132 from IPython.core.inputtransformer2 import ESC_MAGIC
133 from IPython.core.latex_symbols import latex_symbols, reverse_latex_symbol
133 from IPython.core.latex_symbols import latex_symbols, reverse_latex_symbol
134 from IPython.core.oinspect import InspectColors
134 from IPython.core.oinspect import InspectColors
135 from IPython.utils import generics
135 from IPython.utils import generics
136 from IPython.utils.dir2 import dir2, get_real_method
136 from IPython.utils.dir2 import dir2, get_real_method
137 from IPython.utils.path import ensure_dir_exists
137 from IPython.utils.path import ensure_dir_exists
138 from IPython.utils.process import arg_split
138 from IPython.utils.process import arg_split
139 from traitlets import Bool, Enum, Int, List as ListTrait, Unicode, default, observe
139 from traitlets import Bool, Enum, Int, List as ListTrait, Unicode, default, observe
140 from traitlets.config.configurable import Configurable
140 from traitlets.config.configurable import Configurable
141
141
142 import __main__
142 import __main__
143
143
144 # skip module docstests
144 # skip module docstests
145 __skip_doctest__ = True
145 __skip_doctest__ = True
146
146
147 try:
147 try:
148 import jedi
148 import jedi
149 jedi.settings.case_insensitive_completion = False
149 jedi.settings.case_insensitive_completion = False
150 import jedi.api.helpers
150 import jedi.api.helpers
151 import jedi.api.classes
151 import jedi.api.classes
152 JEDI_INSTALLED = True
152 JEDI_INSTALLED = True
153 except ImportError:
153 except ImportError:
154 JEDI_INSTALLED = False
154 JEDI_INSTALLED = False
155 #-----------------------------------------------------------------------------
155 #-----------------------------------------------------------------------------
156 # Globals
156 # Globals
157 #-----------------------------------------------------------------------------
157 #-----------------------------------------------------------------------------
158
158
159 # ranges where we have most of the valid unicode names. We could be more finer
159 # ranges where we have most of the valid unicode names. We could be more finer
160 # grained but is it worth it for performace While unicode have character in the
160 # grained but is it worth it for performance While unicode have character in the
161 # rage 0, 0x110000, we seem to have name for about 10% of those. (131808 as I
161 # range 0, 0x110000, we seem to have name for about 10% of those. (131808 as I
162 # write this). With below range we cover them all, with a density of ~67%
162 # write this). With below range we cover them all, with a density of ~67%
163 # biggest next gap we consider only adds up about 1% density and there are 600
163 # biggest next gap we consider only adds up about 1% density and there are 600
164 # gaps that would need hard coding.
164 # gaps that would need hard coding.
165 _UNICODE_RANGES = [(32, 0x3134b), (0xe0001, 0xe01f0)]
165 _UNICODE_RANGES = [(32, 0x3134b), (0xe0001, 0xe01f0)]
166
166
167 # Public API
167 # Public API
168 __all__ = ['Completer','IPCompleter']
168 __all__ = ['Completer','IPCompleter']
169
169
170 if sys.platform == 'win32':
170 if sys.platform == 'win32':
171 PROTECTABLES = ' '
171 PROTECTABLES = ' '
172 else:
172 else:
173 PROTECTABLES = ' ()[]{}?=\\|;:\'#*"^&'
173 PROTECTABLES = ' ()[]{}?=\\|;:\'#*"^&'
174
174
175 # Protect against returning an enormous number of completions which the frontend
175 # Protect against returning an enormous number of completions which the frontend
176 # may have trouble processing.
176 # may have trouble processing.
177 MATCHES_LIMIT = 500
177 MATCHES_LIMIT = 500
178
178
179 _deprecation_readline_sentinel = object()
179 _deprecation_readline_sentinel = object()
180
180
181
181
182 class ProvisionalCompleterWarning(FutureWarning):
182 class ProvisionalCompleterWarning(FutureWarning):
183 """
183 """
184 Exception raise by an experimental feature in this module.
184 Exception raise by an experimental feature in this module.
185
185
186 Wrap code in :any:`provisionalcompleter` context manager if you
186 Wrap code in :any:`provisionalcompleter` context manager if you
187 are certain you want to use an unstable feature.
187 are certain you want to use an unstable feature.
188 """
188 """
189 pass
189 pass
190
190
191 warnings.filterwarnings('error', category=ProvisionalCompleterWarning)
191 warnings.filterwarnings('error', category=ProvisionalCompleterWarning)
192
192
193 @contextmanager
193 @contextmanager
194 def provisionalcompleter(action='ignore'):
194 def provisionalcompleter(action='ignore'):
195 """
195 """
196 This context manager has to be used in any place where unstable completer
196 This context manager has to be used in any place where unstable completer
197 behavior and API may be called.
197 behavior and API may be called.
198
198
199 >>> with provisionalcompleter():
199 >>> with provisionalcompleter():
200 ... completer.do_experimental_things() # works
200 ... completer.do_experimental_things() # works
201
201
202 >>> completer.do_experimental_things() # raises.
202 >>> completer.do_experimental_things() # raises.
203
203
204 .. note::
204 .. note::
205
205
206 Unstable
206 Unstable
207
207
208 By using this context manager you agree that the API in use may change
208 By using this context manager you agree that the API in use may change
209 without warning, and that you won't complain if they do so.
209 without warning, and that you won't complain if they do so.
210
210
211 You also understand that, if the API is not to your liking, you should report
211 You also understand that, if the API is not to your liking, you should report
212 a bug to explain your use case upstream.
212 a bug to explain your use case upstream.
213
213
214 We'll be happy to get your feedback, feature requests, and improvements on
214 We'll be happy to get your feedback, feature requests, and improvements on
215 any of the unstable APIs!
215 any of the unstable APIs!
216 """
216 """
217 with warnings.catch_warnings():
217 with warnings.catch_warnings():
218 warnings.filterwarnings(action, category=ProvisionalCompleterWarning)
218 warnings.filterwarnings(action, category=ProvisionalCompleterWarning)
219 yield
219 yield
220
220
221
221
222 def has_open_quotes(s):
222 def has_open_quotes(s):
223 """Return whether a string has open quotes.
223 """Return whether a string has open quotes.
224
224
225 This simply counts whether the number of quote characters of either type in
225 This simply counts whether the number of quote characters of either type in
226 the string is odd.
226 the string is odd.
227
227
228 Returns
228 Returns
229 -------
229 -------
230 If there is an open quote, the quote character is returned. Else, return
230 If there is an open quote, the quote character is returned. Else, return
231 False.
231 False.
232 """
232 """
233 # We check " first, then ', so complex cases with nested quotes will get
233 # We check " first, then ', so complex cases with nested quotes will get
234 # the " to take precedence.
234 # the " to take precedence.
235 if s.count('"') % 2:
235 if s.count('"') % 2:
236 return '"'
236 return '"'
237 elif s.count("'") % 2:
237 elif s.count("'") % 2:
238 return "'"
238 return "'"
239 else:
239 else:
240 return False
240 return False
241
241
242
242
243 def protect_filename(s, protectables=PROTECTABLES):
243 def protect_filename(s, protectables=PROTECTABLES):
244 """Escape a string to protect certain characters."""
244 """Escape a string to protect certain characters."""
245 if set(s) & set(protectables):
245 if set(s) & set(protectables):
246 if sys.platform == "win32":
246 if sys.platform == "win32":
247 return '"' + s + '"'
247 return '"' + s + '"'
248 else:
248 else:
249 return "".join(("\\" + c if c in protectables else c) for c in s)
249 return "".join(("\\" + c if c in protectables else c) for c in s)
250 else:
250 else:
251 return s
251 return s
252
252
253
253
254 def expand_user(path:str) -> Tuple[str, bool, str]:
254 def expand_user(path:str) -> Tuple[str, bool, str]:
255 """Expand ``~``-style usernames in strings.
255 """Expand ``~``-style usernames in strings.
256
256
257 This is similar to :func:`os.path.expanduser`, but it computes and returns
257 This is similar to :func:`os.path.expanduser`, but it computes and returns
258 extra information that will be useful if the input was being used in
258 extra information that will be useful if the input was being used in
259 computing completions, and you wish to return the completions with the
259 computing completions, and you wish to return the completions with the
260 original '~' instead of its expanded value.
260 original '~' instead of its expanded value.
261
261
262 Parameters
262 Parameters
263 ----------
263 ----------
264 path : str
264 path : str
265 String to be expanded. If no ~ is present, the output is the same as the
265 String to be expanded. If no ~ is present, the output is the same as the
266 input.
266 input.
267
267
268 Returns
268 Returns
269 -------
269 -------
270 newpath : str
270 newpath : str
271 Result of ~ expansion in the input path.
271 Result of ~ expansion in the input path.
272 tilde_expand : bool
272 tilde_expand : bool
273 Whether any expansion was performed or not.
273 Whether any expansion was performed or not.
274 tilde_val : str
274 tilde_val : str
275 The value that ~ was replaced with.
275 The value that ~ was replaced with.
276 """
276 """
277 # Default values
277 # Default values
278 tilde_expand = False
278 tilde_expand = False
279 tilde_val = ''
279 tilde_val = ''
280 newpath = path
280 newpath = path
281
281
282 if path.startswith('~'):
282 if path.startswith('~'):
283 tilde_expand = True
283 tilde_expand = True
284 rest = len(path)-1
284 rest = len(path)-1
285 newpath = os.path.expanduser(path)
285 newpath = os.path.expanduser(path)
286 if rest:
286 if rest:
287 tilde_val = newpath[:-rest]
287 tilde_val = newpath[:-rest]
288 else:
288 else:
289 tilde_val = newpath
289 tilde_val = newpath
290
290
291 return newpath, tilde_expand, tilde_val
291 return newpath, tilde_expand, tilde_val
292
292
293
293
294 def compress_user(path:str, tilde_expand:bool, tilde_val:str) -> str:
294 def compress_user(path:str, tilde_expand:bool, tilde_val:str) -> str:
295 """Does the opposite of expand_user, with its outputs.
295 """Does the opposite of expand_user, with its outputs.
296 """
296 """
297 if tilde_expand:
297 if tilde_expand:
298 return path.replace(tilde_val, '~')
298 return path.replace(tilde_val, '~')
299 else:
299 else:
300 return path
300 return path
301
301
302
302
303 def completions_sorting_key(word):
303 def completions_sorting_key(word):
304 """key for sorting completions
304 """key for sorting completions
305
305
306 This does several things:
306 This does several things:
307
307
308 - Demote any completions starting with underscores to the end
308 - Demote any completions starting with underscores to the end
309 - Insert any %magic and %%cellmagic completions in the alphabetical order
309 - Insert any %magic and %%cellmagic completions in the alphabetical order
310 by their name
310 by their name
311 """
311 """
312 prio1, prio2 = 0, 0
312 prio1, prio2 = 0, 0
313
313
314 if word.startswith('__'):
314 if word.startswith('__'):
315 prio1 = 2
315 prio1 = 2
316 elif word.startswith('_'):
316 elif word.startswith('_'):
317 prio1 = 1
317 prio1 = 1
318
318
319 if word.endswith('='):
319 if word.endswith('='):
320 prio1 = -1
320 prio1 = -1
321
321
322 if word.startswith('%%'):
322 if word.startswith('%%'):
323 # If there's another % in there, this is something else, so leave it alone
323 # If there's another % in there, this is something else, so leave it alone
324 if not "%" in word[2:]:
324 if not "%" in word[2:]:
325 word = word[2:]
325 word = word[2:]
326 prio2 = 2
326 prio2 = 2
327 elif word.startswith('%'):
327 elif word.startswith('%'):
328 if not "%" in word[1:]:
328 if not "%" in word[1:]:
329 word = word[1:]
329 word = word[1:]
330 prio2 = 1
330 prio2 = 1
331
331
332 return prio1, word, prio2
332 return prio1, word, prio2
333
333
334
334
335 class _FakeJediCompletion:
335 class _FakeJediCompletion:
336 """
336 """
337 This is a workaround to communicate to the UI that Jedi has crashed and to
337 This is a workaround to communicate to the UI that Jedi has crashed and to
338 report a bug. Will be used only id :any:`IPCompleter.debug` is set to true.
338 report a bug. Will be used only id :any:`IPCompleter.debug` is set to true.
339
339
340 Added in IPython 6.0 so should likely be removed for 7.0
340 Added in IPython 6.0 so should likely be removed for 7.0
341
341
342 """
342 """
343
343
344 def __init__(self, name):
344 def __init__(self, name):
345
345
346 self.name = name
346 self.name = name
347 self.complete = name
347 self.complete = name
348 self.type = 'crashed'
348 self.type = 'crashed'
349 self.name_with_symbols = name
349 self.name_with_symbols = name
350 self.signature = ''
350 self.signature = ''
351 self._origin = 'fake'
351 self._origin = 'fake'
352
352
353 def __repr__(self):
353 def __repr__(self):
354 return '<Fake completion object jedi has crashed>'
354 return '<Fake completion object jedi has crashed>'
355
355
356
356
357 class Completion:
357 class Completion:
358 """
358 """
359 Completion object used and return by IPython completers.
359 Completion object used and return by IPython completers.
360
360
361 .. warning::
361 .. warning::
362
362
363 Unstable
363 Unstable
364
364
365 This function is unstable, API may change without warning.
365 This function is unstable, API may change without warning.
366 It will also raise unless use in proper context manager.
366 It will also raise unless use in proper context manager.
367
367
368 This act as a middle ground :any:`Completion` object between the
368 This act as a middle ground :any:`Completion` object between the
369 :any:`jedi.api.classes.Completion` object and the Prompt Toolkit completion
369 :any:`jedi.api.classes.Completion` object and the Prompt Toolkit completion
370 object. While Jedi need a lot of information about evaluator and how the
370 object. While Jedi need a lot of information about evaluator and how the
371 code should be ran/inspected, PromptToolkit (and other frontend) mostly
371 code should be ran/inspected, PromptToolkit (and other frontend) mostly
372 need user facing information.
372 need user facing information.
373
373
374 - Which range should be replaced replaced by what.
374 - Which range should be replaced replaced by what.
375 - Some metadata (like completion type), or meta information to displayed to
375 - Some metadata (like completion type), or meta information to displayed to
376 the use user.
376 the use user.
377
377
378 For debugging purpose we can also store the origin of the completion (``jedi``,
378 For debugging purpose we can also store the origin of the completion (``jedi``,
379 ``IPython.python_matches``, ``IPython.magics_matches``...).
379 ``IPython.python_matches``, ``IPython.magics_matches``...).
380 """
380 """
381
381
382 __slots__ = ['start', 'end', 'text', 'type', 'signature', '_origin']
382 __slots__ = ['start', 'end', 'text', 'type', 'signature', '_origin']
383
383
384 def __init__(self, start: int, end: int, text: str, *, type: str=None, _origin='', signature='') -> None:
384 def __init__(self, start: int, end: int, text: str, *, type: str=None, _origin='', signature='') -> None:
385 warnings.warn("``Completion`` is a provisional API (as of IPython 6.0). "
385 warnings.warn("``Completion`` is a provisional API (as of IPython 6.0). "
386 "It may change without warnings. "
386 "It may change without warnings. "
387 "Use in corresponding context manager.",
387 "Use in corresponding context manager.",
388 category=ProvisionalCompleterWarning, stacklevel=2)
388 category=ProvisionalCompleterWarning, stacklevel=2)
389
389
390 self.start = start
390 self.start = start
391 self.end = end
391 self.end = end
392 self.text = text
392 self.text = text
393 self.type = type
393 self.type = type
394 self.signature = signature
394 self.signature = signature
395 self._origin = _origin
395 self._origin = _origin
396
396
397 def __repr__(self):
397 def __repr__(self):
398 return '<Completion start=%s end=%s text=%r type=%r, signature=%r,>' % \
398 return '<Completion start=%s end=%s text=%r type=%r, signature=%r,>' % \
399 (self.start, self.end, self.text, self.type or '?', self.signature or '?')
399 (self.start, self.end, self.text, self.type or '?', self.signature or '?')
400
400
401 def __eq__(self, other)->Bool:
401 def __eq__(self, other)->Bool:
402 """
402 """
403 Equality and hash do not hash the type (as some completer may not be
403 Equality and hash do not hash the type (as some completer may not be
404 able to infer the type), but are use to (partially) de-duplicate
404 able to infer the type), but are use to (partially) de-duplicate
405 completion.
405 completion.
406
406
407 Completely de-duplicating completion is a bit tricker that just
407 Completely de-duplicating completion is a bit tricker that just
408 comparing as it depends on surrounding text, which Completions are not
408 comparing as it depends on surrounding text, which Completions are not
409 aware of.
409 aware of.
410 """
410 """
411 return self.start == other.start and \
411 return self.start == other.start and \
412 self.end == other.end and \
412 self.end == other.end and \
413 self.text == other.text
413 self.text == other.text
414
414
415 def __hash__(self):
415 def __hash__(self):
416 return hash((self.start, self.end, self.text))
416 return hash((self.start, self.end, self.text))
417
417
418
418
419 _IC = Iterable[Completion]
419 _IC = Iterable[Completion]
420
420
421
421
422 def _deduplicate_completions(text: str, completions: _IC)-> _IC:
422 def _deduplicate_completions(text: str, completions: _IC)-> _IC:
423 """
423 """
424 Deduplicate a set of completions.
424 Deduplicate a set of completions.
425
425
426 .. warning::
426 .. warning::
427
427
428 Unstable
428 Unstable
429
429
430 This function is unstable, API may change without warning.
430 This function is unstable, API may change without warning.
431
431
432 Parameters
432 Parameters
433 ----------
433 ----------
434 text : str
434 text : str
435 text that should be completed.
435 text that should be completed.
436 completions : Iterator[Completion]
436 completions : Iterator[Completion]
437 iterator over the completions to deduplicate
437 iterator over the completions to deduplicate
438
438
439 Yields
439 Yields
440 ------
440 ------
441 `Completions` objects
441 `Completions` objects
442 Completions coming from multiple sources, may be different but end up having
442 Completions coming from multiple sources, may be different but end up having
443 the same effect when applied to ``text``. If this is the case, this will
443 the same effect when applied to ``text``. If this is the case, this will
444 consider completions as equal and only emit the first encountered.
444 consider completions as equal and only emit the first encountered.
445 Not folded in `completions()` yet for debugging purpose, and to detect when
445 Not folded in `completions()` yet for debugging purpose, and to detect when
446 the IPython completer does return things that Jedi does not, but should be
446 the IPython completer does return things that Jedi does not, but should be
447 at some point.
447 at some point.
448 """
448 """
449 completions = list(completions)
449 completions = list(completions)
450 if not completions:
450 if not completions:
451 return
451 return
452
452
453 new_start = min(c.start for c in completions)
453 new_start = min(c.start for c in completions)
454 new_end = max(c.end for c in completions)
454 new_end = max(c.end for c in completions)
455
455
456 seen = set()
456 seen = set()
457 for c in completions:
457 for c in completions:
458 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
458 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
459 if new_text not in seen:
459 if new_text not in seen:
460 yield c
460 yield c
461 seen.add(new_text)
461 seen.add(new_text)
462
462
463
463
464 def rectify_completions(text: str, completions: _IC, *, _debug=False)->_IC:
464 def rectify_completions(text: str, completions: _IC, *, _debug=False)->_IC:
465 """
465 """
466 Rectify a set of completions to all have the same ``start`` and ``end``
466 Rectify a set of completions to all have the same ``start`` and ``end``
467
467
468 .. warning::
468 .. warning::
469
469
470 Unstable
470 Unstable
471
471
472 This function is unstable, API may change without warning.
472 This function is unstable, API may change without warning.
473 It will also raise unless use in proper context manager.
473 It will also raise unless use in proper context manager.
474
474
475 Parameters
475 Parameters
476 ----------
476 ----------
477 text : str
477 text : str
478 text that should be completed.
478 text that should be completed.
479 completions : Iterator[Completion]
479 completions : Iterator[Completion]
480 iterator over the completions to rectify
480 iterator over the completions to rectify
481
481
482 Notes
482 Notes
483 -----
483 -----
484 :any:`jedi.api.classes.Completion` s returned by Jedi may not have the same start and end, though
484 :any:`jedi.api.classes.Completion` s returned by Jedi may not have the same start and end, though
485 the Jupyter Protocol requires them to behave like so. This will readjust
485 the Jupyter Protocol requires them to behave like so. This will readjust
486 the completion to have the same ``start`` and ``end`` by padding both
486 the completion to have the same ``start`` and ``end`` by padding both
487 extremities with surrounding text.
487 extremities with surrounding text.
488
488
489 During stabilisation should support a ``_debug`` option to log which
489 During stabilisation should support a ``_debug`` option to log which
490 completion are return by the IPython completer and not found in Jedi in
490 completion are return by the IPython completer and not found in Jedi in
491 order to make upstream bug report.
491 order to make upstream bug report.
492 """
492 """
493 warnings.warn("`rectify_completions` is a provisional API (as of IPython 6.0). "
493 warnings.warn("`rectify_completions` is a provisional API (as of IPython 6.0). "
494 "It may change without warnings. "
494 "It may change without warnings. "
495 "Use in corresponding context manager.",
495 "Use in corresponding context manager.",
496 category=ProvisionalCompleterWarning, stacklevel=2)
496 category=ProvisionalCompleterWarning, stacklevel=2)
497
497
498 completions = list(completions)
498 completions = list(completions)
499 if not completions:
499 if not completions:
500 return
500 return
501 starts = (c.start for c in completions)
501 starts = (c.start for c in completions)
502 ends = (c.end for c in completions)
502 ends = (c.end for c in completions)
503
503
504 new_start = min(starts)
504 new_start = min(starts)
505 new_end = max(ends)
505 new_end = max(ends)
506
506
507 seen_jedi = set()
507 seen_jedi = set()
508 seen_python_matches = set()
508 seen_python_matches = set()
509 for c in completions:
509 for c in completions:
510 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
510 new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
511 if c._origin == 'jedi':
511 if c._origin == 'jedi':
512 seen_jedi.add(new_text)
512 seen_jedi.add(new_text)
513 elif c._origin == 'IPCompleter.python_matches':
513 elif c._origin == 'IPCompleter.python_matches':
514 seen_python_matches.add(new_text)
514 seen_python_matches.add(new_text)
515 yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin, signature=c.signature)
515 yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin, signature=c.signature)
516 diff = seen_python_matches.difference(seen_jedi)
516 diff = seen_python_matches.difference(seen_jedi)
517 if diff and _debug:
517 if diff and _debug:
518 print('IPython.python matches have extras:', diff)
518 print('IPython.python matches have extras:', diff)
519
519
520
520
521 if sys.platform == 'win32':
521 if sys.platform == 'win32':
522 DELIMS = ' \t\n`!@#$^&*()=+[{]}|;\'",<>?'
522 DELIMS = ' \t\n`!@#$^&*()=+[{]}|;\'",<>?'
523 else:
523 else:
524 DELIMS = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
524 DELIMS = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
525
525
526 GREEDY_DELIMS = ' =\r\n'
526 GREEDY_DELIMS = ' =\r\n'
527
527
528
528
529 class CompletionSplitter(object):
529 class CompletionSplitter(object):
530 """An object to split an input line in a manner similar to readline.
530 """An object to split an input line in a manner similar to readline.
531
531
532 By having our own implementation, we can expose readline-like completion in
532 By having our own implementation, we can expose readline-like completion in
533 a uniform manner to all frontends. This object only needs to be given the
533 a uniform manner to all frontends. This object only needs to be given the
534 line of text to be split and the cursor position on said line, and it
534 line of text to be split and the cursor position on said line, and it
535 returns the 'word' to be completed on at the cursor after splitting the
535 returns the 'word' to be completed on at the cursor after splitting the
536 entire line.
536 entire line.
537
537
538 What characters are used as splitting delimiters can be controlled by
538 What characters are used as splitting delimiters can be controlled by
539 setting the ``delims`` attribute (this is a property that internally
539 setting the ``delims`` attribute (this is a property that internally
540 automatically builds the necessary regular expression)"""
540 automatically builds the necessary regular expression)"""
541
541
542 # Private interface
542 # Private interface
543
543
544 # A string of delimiter characters. The default value makes sense for
544 # A string of delimiter characters. The default value makes sense for
545 # IPython's most typical usage patterns.
545 # IPython's most typical usage patterns.
546 _delims = DELIMS
546 _delims = DELIMS
547
547
548 # The expression (a normal string) to be compiled into a regular expression
548 # The expression (a normal string) to be compiled into a regular expression
549 # for actual splitting. We store it as an attribute mostly for ease of
549 # for actual splitting. We store it as an attribute mostly for ease of
550 # debugging, since this type of code can be so tricky to debug.
550 # debugging, since this type of code can be so tricky to debug.
551 _delim_expr = None
551 _delim_expr = None
552
552
553 # The regular expression that does the actual splitting
553 # The regular expression that does the actual splitting
554 _delim_re = None
554 _delim_re = None
555
555
556 def __init__(self, delims=None):
556 def __init__(self, delims=None):
557 delims = CompletionSplitter._delims if delims is None else delims
557 delims = CompletionSplitter._delims if delims is None else delims
558 self.delims = delims
558 self.delims = delims
559
559
560 @property
560 @property
561 def delims(self):
561 def delims(self):
562 """Return the string of delimiter characters."""
562 """Return the string of delimiter characters."""
563 return self._delims
563 return self._delims
564
564
565 @delims.setter
565 @delims.setter
566 def delims(self, delims):
566 def delims(self, delims):
567 """Set the delimiters for line splitting."""
567 """Set the delimiters for line splitting."""
568 expr = '[' + ''.join('\\'+ c for c in delims) + ']'
568 expr = '[' + ''.join('\\'+ c for c in delims) + ']'
569 self._delim_re = re.compile(expr)
569 self._delim_re = re.compile(expr)
570 self._delims = delims
570 self._delims = delims
571 self._delim_expr = expr
571 self._delim_expr = expr
572
572
573 def split_line(self, line, cursor_pos=None):
573 def split_line(self, line, cursor_pos=None):
574 """Split a line of text with a cursor at the given position.
574 """Split a line of text with a cursor at the given position.
575 """
575 """
576 l = line if cursor_pos is None else line[:cursor_pos]
576 l = line if cursor_pos is None else line[:cursor_pos]
577 return self._delim_re.split(l)[-1]
577 return self._delim_re.split(l)[-1]
578
578
579
579
580
580
581 class Completer(Configurable):
581 class Completer(Configurable):
582
582
583 greedy = Bool(False,
583 greedy = Bool(False,
584 help="""Activate greedy completion
584 help="""Activate greedy completion
585 PENDING DEPRECTION. this is now mostly taken care of with Jedi.
585 PENDING DEPRECTION. this is now mostly taken care of with Jedi.
586
586
587 This will enable completion on elements of lists, results of function calls, etc.,
587 This will enable completion on elements of lists, results of function calls, etc.,
588 but can be unsafe because the code is actually evaluated on TAB.
588 but can be unsafe because the code is actually evaluated on TAB.
589 """
589 """
590 ).tag(config=True)
590 ).tag(config=True)
591
591
592 use_jedi = Bool(default_value=JEDI_INSTALLED,
592 use_jedi = Bool(default_value=JEDI_INSTALLED,
593 help="Experimental: Use Jedi to generate autocompletions. "
593 help="Experimental: Use Jedi to generate autocompletions. "
594 "Default to True if jedi is installed.").tag(config=True)
594 "Default to True if jedi is installed.").tag(config=True)
595
595
596 jedi_compute_type_timeout = Int(default_value=400,
596 jedi_compute_type_timeout = Int(default_value=400,
597 help="""Experimental: restrict time (in milliseconds) during which Jedi can compute types.
597 help="""Experimental: restrict time (in milliseconds) during which Jedi can compute types.
598 Set to 0 to stop computing types. Non-zero value lower than 100ms may hurt
598 Set to 0 to stop computing types. Non-zero value lower than 100ms may hurt
599 performance by preventing jedi to build its cache.
599 performance by preventing jedi to build its cache.
600 """).tag(config=True)
600 """).tag(config=True)
601
601
602 debug = Bool(default_value=False,
602 debug = Bool(default_value=False,
603 help='Enable debug for the Completer. Mostly print extra '
603 help='Enable debug for the Completer. Mostly print extra '
604 'information for experimental jedi integration.')\
604 'information for experimental jedi integration.')\
605 .tag(config=True)
605 .tag(config=True)
606
606
607 backslash_combining_completions = Bool(True,
607 backslash_combining_completions = Bool(True,
608 help="Enable unicode completions, e.g. \\alpha<tab> . "
608 help="Enable unicode completions, e.g. \\alpha<tab> . "
609 "Includes completion of latex commands, unicode names, and expanding "
609 "Includes completion of latex commands, unicode names, and expanding "
610 "unicode characters back to latex commands.").tag(config=True)
610 "unicode characters back to latex commands.").tag(config=True)
611
611
612
612
613
613
614 def __init__(self, namespace=None, global_namespace=None, **kwargs):
614 def __init__(self, namespace=None, global_namespace=None, **kwargs):
615 """Create a new completer for the command line.
615 """Create a new completer for the command line.
616
616
617 Completer(namespace=ns, global_namespace=ns2) -> completer instance.
617 Completer(namespace=ns, global_namespace=ns2) -> completer instance.
618
618
619 If unspecified, the default namespace where completions are performed
619 If unspecified, the default namespace where completions are performed
620 is __main__ (technically, __main__.__dict__). Namespaces should be
620 is __main__ (technically, __main__.__dict__). Namespaces should be
621 given as dictionaries.
621 given as dictionaries.
622
622
623 An optional second namespace can be given. This allows the completer
623 An optional second namespace can be given. This allows the completer
624 to handle cases where both the local and global scopes need to be
624 to handle cases where both the local and global scopes need to be
625 distinguished.
625 distinguished.
626 """
626 """
627
627
628 # Don't bind to namespace quite yet, but flag whether the user wants a
628 # Don't bind to namespace quite yet, but flag whether the user wants a
629 # specific namespace or to use __main__.__dict__. This will allow us
629 # specific namespace or to use __main__.__dict__. This will allow us
630 # to bind to __main__.__dict__ at completion time, not now.
630 # to bind to __main__.__dict__ at completion time, not now.
631 if namespace is None:
631 if namespace is None:
632 self.use_main_ns = True
632 self.use_main_ns = True
633 else:
633 else:
634 self.use_main_ns = False
634 self.use_main_ns = False
635 self.namespace = namespace
635 self.namespace = namespace
636
636
637 # The global namespace, if given, can be bound directly
637 # The global namespace, if given, can be bound directly
638 if global_namespace is None:
638 if global_namespace is None:
639 self.global_namespace = {}
639 self.global_namespace = {}
640 else:
640 else:
641 self.global_namespace = global_namespace
641 self.global_namespace = global_namespace
642
642
643 self.custom_matchers = []
643 self.custom_matchers = []
644
644
645 super(Completer, self).__init__(**kwargs)
645 super(Completer, self).__init__(**kwargs)
646
646
647 def complete(self, text, state):
647 def complete(self, text, state):
648 """Return the next possible completion for 'text'.
648 """Return the next possible completion for 'text'.
649
649
650 This is called successively with state == 0, 1, 2, ... until it
650 This is called successively with state == 0, 1, 2, ... until it
651 returns None. The completion should begin with 'text'.
651 returns None. The completion should begin with 'text'.
652
652
653 """
653 """
654 if self.use_main_ns:
654 if self.use_main_ns:
655 self.namespace = __main__.__dict__
655 self.namespace = __main__.__dict__
656
656
657 if state == 0:
657 if state == 0:
658 if "." in text:
658 if "." in text:
659 self.matches = self.attr_matches(text)
659 self.matches = self.attr_matches(text)
660 else:
660 else:
661 self.matches = self.global_matches(text)
661 self.matches = self.global_matches(text)
662 try:
662 try:
663 return self.matches[state]
663 return self.matches[state]
664 except IndexError:
664 except IndexError:
665 return None
665 return None
666
666
667 def global_matches(self, text):
667 def global_matches(self, text):
668 """Compute matches when text is a simple name.
668 """Compute matches when text is a simple name.
669
669
670 Return a list of all keywords, built-in functions and names currently
670 Return a list of all keywords, built-in functions and names currently
671 defined in self.namespace or self.global_namespace that match.
671 defined in self.namespace or self.global_namespace that match.
672
672
673 """
673 """
674 matches = []
674 matches = []
675 match_append = matches.append
675 match_append = matches.append
676 n = len(text)
676 n = len(text)
677 for lst in [keyword.kwlist,
677 for lst in [keyword.kwlist,
678 builtin_mod.__dict__.keys(),
678 builtin_mod.__dict__.keys(),
679 self.namespace.keys(),
679 self.namespace.keys(),
680 self.global_namespace.keys()]:
680 self.global_namespace.keys()]:
681 for word in lst:
681 for word in lst:
682 if word[:n] == text and word != "__builtins__":
682 if word[:n] == text and word != "__builtins__":
683 match_append(word)
683 match_append(word)
684
684
685 snake_case_re = re.compile(r"[^_]+(_[^_]+)+?\Z")
685 snake_case_re = re.compile(r"[^_]+(_[^_]+)+?\Z")
686 for lst in [self.namespace.keys(),
686 for lst in [self.namespace.keys(),
687 self.global_namespace.keys()]:
687 self.global_namespace.keys()]:
688 shortened = {"_".join([sub[0] for sub in word.split('_')]) : word
688 shortened = {"_".join([sub[0] for sub in word.split('_')]) : word
689 for word in lst if snake_case_re.match(word)}
689 for word in lst if snake_case_re.match(word)}
690 for word in shortened.keys():
690 for word in shortened.keys():
691 if word[:n] == text and word != "__builtins__":
691 if word[:n] == text and word != "__builtins__":
692 match_append(shortened[word])
692 match_append(shortened[word])
693 return matches
693 return matches
694
694
695 def attr_matches(self, text):
695 def attr_matches(self, text):
696 """Compute matches when text contains a dot.
696 """Compute matches when text contains a dot.
697
697
698 Assuming the text is of the form NAME.NAME....[NAME], and is
698 Assuming the text is of the form NAME.NAME....[NAME], and is
699 evaluatable in self.namespace or self.global_namespace, it will be
699 evaluatable in self.namespace or self.global_namespace, it will be
700 evaluated and its attributes (as revealed by dir()) are used as
700 evaluated and its attributes (as revealed by dir()) are used as
701 possible completions. (For class instances, class members are
701 possible completions. (For class instances, class members are
702 also considered.)
702 also considered.)
703
703
704 WARNING: this can still invoke arbitrary C code, if an object
704 WARNING: this can still invoke arbitrary C code, if an object
705 with a __getattr__ hook is evaluated.
705 with a __getattr__ hook is evaluated.
706
706
707 """
707 """
708
708
709 # Another option, seems to work great. Catches things like ''.<tab>
709 # Another option, seems to work great. Catches things like ''.<tab>
710 m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
710 m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
711
711
712 if m:
712 if m:
713 expr, attr = m.group(1, 3)
713 expr, attr = m.group(1, 3)
714 elif self.greedy:
714 elif self.greedy:
715 m2 = re.match(r"(.+)\.(\w*)$", self.line_buffer)
715 m2 = re.match(r"(.+)\.(\w*)$", self.line_buffer)
716 if not m2:
716 if not m2:
717 return []
717 return []
718 expr, attr = m2.group(1,2)
718 expr, attr = m2.group(1,2)
719 else:
719 else:
720 return []
720 return []
721
721
722 try:
722 try:
723 obj = eval(expr, self.namespace)
723 obj = eval(expr, self.namespace)
724 except:
724 except:
725 try:
725 try:
726 obj = eval(expr, self.global_namespace)
726 obj = eval(expr, self.global_namespace)
727 except:
727 except:
728 return []
728 return []
729
729
730 if self.limit_to__all__ and hasattr(obj, '__all__'):
730 if self.limit_to__all__ and hasattr(obj, '__all__'):
731 words = get__all__entries(obj)
731 words = get__all__entries(obj)
732 else:
732 else:
733 words = dir2(obj)
733 words = dir2(obj)
734
734
735 try:
735 try:
736 words = generics.complete_object(obj, words)
736 words = generics.complete_object(obj, words)
737 except TryNext:
737 except TryNext:
738 pass
738 pass
739 except AssertionError:
739 except AssertionError:
740 raise
740 raise
741 except Exception:
741 except Exception:
742 # Silence errors from completion function
742 # Silence errors from completion function
743 #raise # dbg
743 #raise # dbg
744 pass
744 pass
745 # Build match list to return
745 # Build match list to return
746 n = len(attr)
746 n = len(attr)
747 return [u"%s.%s" % (expr, w) for w in words if w[:n] == attr ]
747 return [u"%s.%s" % (expr, w) for w in words if w[:n] == attr ]
748
748
749
749
750 def get__all__entries(obj):
750 def get__all__entries(obj):
751 """returns the strings in the __all__ attribute"""
751 """returns the strings in the __all__ attribute"""
752 try:
752 try:
753 words = getattr(obj, '__all__')
753 words = getattr(obj, '__all__')
754 except:
754 except:
755 return []
755 return []
756
756
757 return [w for w in words if isinstance(w, str)]
757 return [w for w in words if isinstance(w, str)]
758
758
759
759
760 def match_dict_keys(keys: List[Union[str, bytes, Tuple[Union[str, bytes]]]], prefix: str, delims: str,
760 def match_dict_keys(keys: List[Union[str, bytes, Tuple[Union[str, bytes]]]], prefix: str, delims: str,
761 extra_prefix: Optional[Tuple[str, bytes]]=None) -> Tuple[str, int, List[str]]:
761 extra_prefix: Optional[Tuple[str, bytes]]=None) -> Tuple[str, int, List[str]]:
762 """Used by dict_key_matches, matching the prefix to a list of keys
762 """Used by dict_key_matches, matching the prefix to a list of keys
763
763
764 Parameters
764 Parameters
765 ----------
765 ----------
766 keys
766 keys
767 list of keys in dictionary currently being completed.
767 list of keys in dictionary currently being completed.
768 prefix
768 prefix
769 Part of the text already typed by the user. E.g. `mydict[b'fo`
769 Part of the text already typed by the user. E.g. `mydict[b'fo`
770 delims
770 delims
771 String of delimiters to consider when finding the current key.
771 String of delimiters to consider when finding the current key.
772 extra_prefix : optional
772 extra_prefix : optional
773 Part of the text already typed in multi-key index cases. E.g. for
773 Part of the text already typed in multi-key index cases. E.g. for
774 `mydict['foo', "bar", 'b`, this would be `('foo', 'bar')`.
774 `mydict['foo', "bar", 'b`, this would be `('foo', 'bar')`.
775
775
776 Returns
776 Returns
777 -------
777 -------
778 A tuple of three elements: ``quote``, ``token_start``, ``matched``, with
778 A tuple of three elements: ``quote``, ``token_start``, ``matched``, with
779 ``quote`` being the quote that need to be used to close current string.
779 ``quote`` being the quote that need to be used to close current string.
780 ``token_start`` the position where the replacement should start occurring,
780 ``token_start`` the position where the replacement should start occurring,
781 ``matches`` a list of replacement/completion
781 ``matches`` a list of replacement/completion
782
782
783 """
783 """
784 prefix_tuple = extra_prefix if extra_prefix else ()
784 prefix_tuple = extra_prefix if extra_prefix else ()
785 Nprefix = len(prefix_tuple)
785 Nprefix = len(prefix_tuple)
786 def filter_prefix_tuple(key):
786 def filter_prefix_tuple(key):
787 # Reject too short keys
787 # Reject too short keys
788 if len(key) <= Nprefix:
788 if len(key) <= Nprefix:
789 return False
789 return False
790 # Reject keys with non str/bytes in it
790 # Reject keys with non str/bytes in it
791 for k in key:
791 for k in key:
792 if not isinstance(k, (str, bytes)):
792 if not isinstance(k, (str, bytes)):
793 return False
793 return False
794 # Reject keys that do not match the prefix
794 # Reject keys that do not match the prefix
795 for k, pt in zip(key, prefix_tuple):
795 for k, pt in zip(key, prefix_tuple):
796 if k != pt:
796 if k != pt:
797 return False
797 return False
798 # All checks passed!
798 # All checks passed!
799 return True
799 return True
800
800
801 filtered_keys:List[Union[str,bytes]] = []
801 filtered_keys:List[Union[str,bytes]] = []
802 def _add_to_filtered_keys(key):
802 def _add_to_filtered_keys(key):
803 if isinstance(key, (str, bytes)):
803 if isinstance(key, (str, bytes)):
804 filtered_keys.append(key)
804 filtered_keys.append(key)
805
805
806 for k in keys:
806 for k in keys:
807 if isinstance(k, tuple):
807 if isinstance(k, tuple):
808 if filter_prefix_tuple(k):
808 if filter_prefix_tuple(k):
809 _add_to_filtered_keys(k[Nprefix])
809 _add_to_filtered_keys(k[Nprefix])
810 else:
810 else:
811 _add_to_filtered_keys(k)
811 _add_to_filtered_keys(k)
812
812
813 if not prefix:
813 if not prefix:
814 return '', 0, [repr(k) for k in filtered_keys]
814 return '', 0, [repr(k) for k in filtered_keys]
815 quote_match = re.search('["\']', prefix)
815 quote_match = re.search('["\']', prefix)
816 assert quote_match is not None # silence mypy
816 assert quote_match is not None # silence mypy
817 quote = quote_match.group()
817 quote = quote_match.group()
818 try:
818 try:
819 prefix_str = eval(prefix + quote, {})
819 prefix_str = eval(prefix + quote, {})
820 except Exception:
820 except Exception:
821 return '', 0, []
821 return '', 0, []
822
822
823 pattern = '[^' + ''.join('\\' + c for c in delims) + ']*$'
823 pattern = '[^' + ''.join('\\' + c for c in delims) + ']*$'
824 token_match = re.search(pattern, prefix, re.UNICODE)
824 token_match = re.search(pattern, prefix, re.UNICODE)
825 assert token_match is not None # silence mypy
825 assert token_match is not None # silence mypy
826 token_start = token_match.start()
826 token_start = token_match.start()
827 token_prefix = token_match.group()
827 token_prefix = token_match.group()
828
828
829 matched:List[str] = []
829 matched:List[str] = []
830 for key in filtered_keys:
830 for key in filtered_keys:
831 try:
831 try:
832 if not key.startswith(prefix_str):
832 if not key.startswith(prefix_str):
833 continue
833 continue
834 except (AttributeError, TypeError, UnicodeError):
834 except (AttributeError, TypeError, UnicodeError):
835 # Python 3+ TypeError on b'a'.startswith('a') or vice-versa
835 # Python 3+ TypeError on b'a'.startswith('a') or vice-versa
836 continue
836 continue
837
837
838 # reformat remainder of key to begin with prefix
838 # reformat remainder of key to begin with prefix
839 rem = key[len(prefix_str):]
839 rem = key[len(prefix_str):]
840 # force repr wrapped in '
840 # force repr wrapped in '
841 rem_repr = repr(rem + '"') if isinstance(rem, str) else repr(rem + b'"')
841 rem_repr = repr(rem + '"') if isinstance(rem, str) else repr(rem + b'"')
842 rem_repr = rem_repr[1 + rem_repr.index("'"):-2]
842 rem_repr = rem_repr[1 + rem_repr.index("'"):-2]
843 if quote == '"':
843 if quote == '"':
844 # The entered prefix is quoted with ",
844 # The entered prefix is quoted with ",
845 # but the match is quoted with '.
845 # but the match is quoted with '.
846 # A contained " hence needs escaping for comparison:
846 # A contained " hence needs escaping for comparison:
847 rem_repr = rem_repr.replace('"', '\\"')
847 rem_repr = rem_repr.replace('"', '\\"')
848
848
849 # then reinsert prefix from start of token
849 # then reinsert prefix from start of token
850 matched.append('%s%s' % (token_prefix, rem_repr))
850 matched.append('%s%s' % (token_prefix, rem_repr))
851 return quote, token_start, matched
851 return quote, token_start, matched
852
852
853
853
854 def cursor_to_position(text:str, line:int, column:int)->int:
854 def cursor_to_position(text:str, line:int, column:int)->int:
855 """
855 """
856 Convert the (line,column) position of the cursor in text to an offset in a
856 Convert the (line,column) position of the cursor in text to an offset in a
857 string.
857 string.
858
858
859 Parameters
859 Parameters
860 ----------
860 ----------
861 text : str
861 text : str
862 The text in which to calculate the cursor offset
862 The text in which to calculate the cursor offset
863 line : int
863 line : int
864 Line of the cursor; 0-indexed
864 Line of the cursor; 0-indexed
865 column : int
865 column : int
866 Column of the cursor 0-indexed
866 Column of the cursor 0-indexed
867
867
868 Returns
868 Returns
869 -------
869 -------
870 Position of the cursor in ``text``, 0-indexed.
870 Position of the cursor in ``text``, 0-indexed.
871
871
872 See Also
872 See Also
873 --------
873 --------
874 position_to_cursor : reciprocal of this function
874 position_to_cursor : reciprocal of this function
875
875
876 """
876 """
877 lines = text.split('\n')
877 lines = text.split('\n')
878 assert line <= len(lines), '{} <= {}'.format(str(line), str(len(lines)))
878 assert line <= len(lines), '{} <= {}'.format(str(line), str(len(lines)))
879
879
880 return sum(len(l) + 1 for l in lines[:line]) + column
880 return sum(len(l) + 1 for l in lines[:line]) + column
881
881
882 def position_to_cursor(text:str, offset:int)->Tuple[int, int]:
882 def position_to_cursor(text:str, offset:int)->Tuple[int, int]:
883 """
883 """
884 Convert the position of the cursor in text (0 indexed) to a line
884 Convert the position of the cursor in text (0 indexed) to a line
885 number(0-indexed) and a column number (0-indexed) pair
885 number(0-indexed) and a column number (0-indexed) pair
886
886
887 Position should be a valid position in ``text``.
887 Position should be a valid position in ``text``.
888
888
889 Parameters
889 Parameters
890 ----------
890 ----------
891 text : str
891 text : str
892 The text in which to calculate the cursor offset
892 The text in which to calculate the cursor offset
893 offset : int
893 offset : int
894 Position of the cursor in ``text``, 0-indexed.
894 Position of the cursor in ``text``, 0-indexed.
895
895
896 Returns
896 Returns
897 -------
897 -------
898 (line, column) : (int, int)
898 (line, column) : (int, int)
899 Line of the cursor; 0-indexed, column of the cursor 0-indexed
899 Line of the cursor; 0-indexed, column of the cursor 0-indexed
900
900
901 See Also
901 See Also
902 --------
902 --------
903 cursor_to_position : reciprocal of this function
903 cursor_to_position : reciprocal of this function
904
904
905 """
905 """
906
906
907 assert 0 <= offset <= len(text) , "0 <= %s <= %s" % (offset , len(text))
907 assert 0 <= offset <= len(text) , "0 <= %s <= %s" % (offset , len(text))
908
908
909 before = text[:offset]
909 before = text[:offset]
910 blines = before.split('\n') # ! splitnes trim trailing \n
910 blines = before.split('\n') # ! splitnes trim trailing \n
911 line = before.count('\n')
911 line = before.count('\n')
912 col = len(blines[-1])
912 col = len(blines[-1])
913 return line, col
913 return line, col
914
914
915
915
916 def _safe_isinstance(obj, module, class_name):
916 def _safe_isinstance(obj, module, class_name):
917 """Checks if obj is an instance of module.class_name if loaded
917 """Checks if obj is an instance of module.class_name if loaded
918 """
918 """
919 return (module in sys.modules and
919 return (module in sys.modules and
920 isinstance(obj, getattr(import_module(module), class_name)))
920 isinstance(obj, getattr(import_module(module), class_name)))
921
921
922 def back_unicode_name_matches(text:str) -> Tuple[str, Sequence[str]]:
922 def back_unicode_name_matches(text:str) -> Tuple[str, Sequence[str]]:
923 """Match Unicode characters back to Unicode name
923 """Match Unicode characters back to Unicode name
924
924
925 This does ``☃`` -> ``\\snowman``
925 This does ``☃`` -> ``\\snowman``
926
926
927 Note that snowman is not a valid python3 combining character but will be expanded.
927 Note that snowman is not a valid python3 combining character but will be expanded.
928 Though it will not recombine back to the snowman character by the completion machinery.
928 Though it will not recombine back to the snowman character by the completion machinery.
929
929
930 This will not either back-complete standard sequences like \\n, \\b ...
930 This will not either back-complete standard sequences like \\n, \\b ...
931
931
932 Returns
932 Returns
933 =======
933 =======
934
934
935 Return a tuple with two elements:
935 Return a tuple with two elements:
936
936
937 - The Unicode character that was matched (preceded with a backslash), or
937 - The Unicode character that was matched (preceded with a backslash), or
938 empty string,
938 empty string,
939 - a sequence (of 1), name for the match Unicode character, preceded by
939 - a sequence (of 1), name for the match Unicode character, preceded by
940 backslash, or empty if no match.
940 backslash, or empty if no match.
941
941
942 """
942 """
943 if len(text)<2:
943 if len(text)<2:
944 return '', ()
944 return '', ()
945 maybe_slash = text[-2]
945 maybe_slash = text[-2]
946 if maybe_slash != '\\':
946 if maybe_slash != '\\':
947 return '', ()
947 return '', ()
948
948
949 char = text[-1]
949 char = text[-1]
950 # no expand on quote for completion in strings.
950 # no expand on quote for completion in strings.
951 # nor backcomplete standard ascii keys
951 # nor backcomplete standard ascii keys
952 if char in string.ascii_letters or char in ('"',"'"):
952 if char in string.ascii_letters or char in ('"',"'"):
953 return '', ()
953 return '', ()
954 try :
954 try :
955 unic = unicodedata.name(char)
955 unic = unicodedata.name(char)
956 return '\\'+char,('\\'+unic,)
956 return '\\'+char,('\\'+unic,)
957 except KeyError:
957 except KeyError:
958 pass
958 pass
959 return '', ()
959 return '', ()
960
960
961 def back_latex_name_matches(text:str) -> Tuple[str, Sequence[str]] :
961 def back_latex_name_matches(text:str) -> Tuple[str, Sequence[str]] :
962 """Match latex characters back to unicode name
962 """Match latex characters back to unicode name
963
963
964 This does ``\\ℵ`` -> ``\\aleph``
964 This does ``\\ℵ`` -> ``\\aleph``
965
965
966 """
966 """
967 if len(text)<2:
967 if len(text)<2:
968 return '', ()
968 return '', ()
969 maybe_slash = text[-2]
969 maybe_slash = text[-2]
970 if maybe_slash != '\\':
970 if maybe_slash != '\\':
971 return '', ()
971 return '', ()
972
972
973
973
974 char = text[-1]
974 char = text[-1]
975 # no expand on quote for completion in strings.
975 # no expand on quote for completion in strings.
976 # nor backcomplete standard ascii keys
976 # nor backcomplete standard ascii keys
977 if char in string.ascii_letters or char in ('"',"'"):
977 if char in string.ascii_letters or char in ('"',"'"):
978 return '', ()
978 return '', ()
979 try :
979 try :
980 latex = reverse_latex_symbol[char]
980 latex = reverse_latex_symbol[char]
981 # '\\' replace the \ as well
981 # '\\' replace the \ as well
982 return '\\'+char,[latex]
982 return '\\'+char,[latex]
983 except KeyError:
983 except KeyError:
984 pass
984 pass
985 return '', ()
985 return '', ()
986
986
987
987
988 def _formatparamchildren(parameter) -> str:
988 def _formatparamchildren(parameter) -> str:
989 """
989 """
990 Get parameter name and value from Jedi Private API
990 Get parameter name and value from Jedi Private API
991
991
992 Jedi does not expose a simple way to get `param=value` from its API.
992 Jedi does not expose a simple way to get `param=value` from its API.
993
993
994 Parameters
994 Parameters
995 ----------
995 ----------
996 parameter
996 parameter
997 Jedi's function `Param`
997 Jedi's function `Param`
998
998
999 Returns
999 Returns
1000 -------
1000 -------
1001 A string like 'a', 'b=1', '*args', '**kwargs'
1001 A string like 'a', 'b=1', '*args', '**kwargs'
1002
1002
1003 """
1003 """
1004 description = parameter.description
1004 description = parameter.description
1005 if not description.startswith('param '):
1005 if not description.startswith('param '):
1006 raise ValueError('Jedi function parameter description have change format.'
1006 raise ValueError('Jedi function parameter description have change format.'
1007 'Expected "param ...", found %r".' % description)
1007 'Expected "param ...", found %r".' % description)
1008 return description[6:]
1008 return description[6:]
1009
1009
1010 def _make_signature(completion)-> str:
1010 def _make_signature(completion)-> str:
1011 """
1011 """
1012 Make the signature from a jedi completion
1012 Make the signature from a jedi completion
1013
1013
1014 Parameters
1014 Parameters
1015 ----------
1015 ----------
1016 completion : jedi.Completion
1016 completion : jedi.Completion
1017 object does not complete a function type
1017 object does not complete a function type
1018
1018
1019 Returns
1019 Returns
1020 -------
1020 -------
1021 a string consisting of the function signature, with the parenthesis but
1021 a string consisting of the function signature, with the parenthesis but
1022 without the function name. example:
1022 without the function name. example:
1023 `(a, *args, b=1, **kwargs)`
1023 `(a, *args, b=1, **kwargs)`
1024
1024
1025 """
1025 """
1026
1026
1027 # it looks like this might work on jedi 0.17
1027 # it looks like this might work on jedi 0.17
1028 if hasattr(completion, 'get_signatures'):
1028 if hasattr(completion, 'get_signatures'):
1029 signatures = completion.get_signatures()
1029 signatures = completion.get_signatures()
1030 if not signatures:
1030 if not signatures:
1031 return '(?)'
1031 return '(?)'
1032
1032
1033 c0 = completion.get_signatures()[0]
1033 c0 = completion.get_signatures()[0]
1034 return '('+c0.to_string().split('(', maxsplit=1)[1]
1034 return '('+c0.to_string().split('(', maxsplit=1)[1]
1035
1035
1036 return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for signature in completion.get_signatures()
1036 return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for signature in completion.get_signatures()
1037 for p in signature.defined_names()) if f])
1037 for p in signature.defined_names()) if f])
1038
1038
1039
1039
1040 class _CompleteResult(NamedTuple):
1040 class _CompleteResult(NamedTuple):
1041 matched_text : str
1041 matched_text : str
1042 matches: Sequence[str]
1042 matches: Sequence[str]
1043 matches_origin: Sequence[str]
1043 matches_origin: Sequence[str]
1044 jedi_matches: Any
1044 jedi_matches: Any
1045
1045
1046
1046
1047 class IPCompleter(Completer):
1047 class IPCompleter(Completer):
1048 """Extension of the completer class with IPython-specific features"""
1048 """Extension of the completer class with IPython-specific features"""
1049
1049
1050 __dict_key_regexps: Optional[Dict[bool,Pattern]] = None
1050 __dict_key_regexps: Optional[Dict[bool,Pattern]] = None
1051
1051
1052 @observe('greedy')
1052 @observe('greedy')
1053 def _greedy_changed(self, change):
1053 def _greedy_changed(self, change):
1054 """update the splitter and readline delims when greedy is changed"""
1054 """update the splitter and readline delims when greedy is changed"""
1055 if change['new']:
1055 if change['new']:
1056 self.splitter.delims = GREEDY_DELIMS
1056 self.splitter.delims = GREEDY_DELIMS
1057 else:
1057 else:
1058 self.splitter.delims = DELIMS
1058 self.splitter.delims = DELIMS
1059
1059
1060 dict_keys_only = Bool(False,
1060 dict_keys_only = Bool(False,
1061 help="""Whether to show dict key matches only""")
1061 help="""Whether to show dict key matches only""")
1062
1062
1063 merge_completions = Bool(True,
1063 merge_completions = Bool(True,
1064 help="""Whether to merge completion results into a single list
1064 help="""Whether to merge completion results into a single list
1065
1065
1066 If False, only the completion results from the first non-empty
1066 If False, only the completion results from the first non-empty
1067 completer will be returned.
1067 completer will be returned.
1068 """
1068 """
1069 ).tag(config=True)
1069 ).tag(config=True)
1070 omit__names = Enum((0,1,2), default_value=2,
1070 omit__names = Enum((0,1,2), default_value=2,
1071 help="""Instruct the completer to omit private method names
1071 help="""Instruct the completer to omit private method names
1072
1072
1073 Specifically, when completing on ``object.<tab>``.
1073 Specifically, when completing on ``object.<tab>``.
1074
1074
1075 When 2 [default]: all names that start with '_' will be excluded.
1075 When 2 [default]: all names that start with '_' will be excluded.
1076
1076
1077 When 1: all 'magic' names (``__foo__``) will be excluded.
1077 When 1: all 'magic' names (``__foo__``) will be excluded.
1078
1078
1079 When 0: nothing will be excluded.
1079 When 0: nothing will be excluded.
1080 """
1080 """
1081 ).tag(config=True)
1081 ).tag(config=True)
1082 limit_to__all__ = Bool(False,
1082 limit_to__all__ = Bool(False,
1083 help="""
1083 help="""
1084 DEPRECATED as of version 5.0.
1084 DEPRECATED as of version 5.0.
1085
1085
1086 Instruct the completer to use __all__ for the completion
1086 Instruct the completer to use __all__ for the completion
1087
1087
1088 Specifically, when completing on ``object.<tab>``.
1088 Specifically, when completing on ``object.<tab>``.
1089
1089
1090 When True: only those names in obj.__all__ will be included.
1090 When True: only those names in obj.__all__ will be included.
1091
1091
1092 When False [default]: the __all__ attribute is ignored
1092 When False [default]: the __all__ attribute is ignored
1093 """,
1093 """,
1094 ).tag(config=True)
1094 ).tag(config=True)
1095
1095
1096 profile_completions = Bool(
1096 profile_completions = Bool(
1097 default_value=False,
1097 default_value=False,
1098 help="If True, emit profiling data for completion subsystem using cProfile."
1098 help="If True, emit profiling data for completion subsystem using cProfile."
1099 ).tag(config=True)
1099 ).tag(config=True)
1100
1100
1101 profiler_output_dir = Unicode(
1101 profiler_output_dir = Unicode(
1102 default_value=".completion_profiles",
1102 default_value=".completion_profiles",
1103 help="Template for path at which to output profile data for completions."
1103 help="Template for path at which to output profile data for completions."
1104 ).tag(config=True)
1104 ).tag(config=True)
1105
1105
1106 @observe('limit_to__all__')
1106 @observe('limit_to__all__')
1107 def _limit_to_all_changed(self, change):
1107 def _limit_to_all_changed(self, change):
1108 warnings.warn('`IPython.core.IPCompleter.limit_to__all__` configuration '
1108 warnings.warn('`IPython.core.IPCompleter.limit_to__all__` configuration '
1109 'value has been deprecated since IPython 5.0, will be made to have '
1109 'value has been deprecated since IPython 5.0, will be made to have '
1110 'no effects and then removed in future version of IPython.',
1110 'no effects and then removed in future version of IPython.',
1111 UserWarning)
1111 UserWarning)
1112
1112
1113 def __init__(self, shell=None, namespace=None, global_namespace=None,
1113 def __init__(self, shell=None, namespace=None, global_namespace=None,
1114 use_readline=_deprecation_readline_sentinel, config=None, **kwargs):
1114 use_readline=_deprecation_readline_sentinel, config=None, **kwargs):
1115 """IPCompleter() -> completer
1115 """IPCompleter() -> completer
1116
1116
1117 Return a completer object.
1117 Return a completer object.
1118
1118
1119 Parameters
1119 Parameters
1120 ----------
1120 ----------
1121 shell
1121 shell
1122 a pointer to the ipython shell itself. This is needed
1122 a pointer to the ipython shell itself. This is needed
1123 because this completer knows about magic functions, and those can
1123 because this completer knows about magic functions, and those can
1124 only be accessed via the ipython instance.
1124 only be accessed via the ipython instance.
1125 namespace : dict, optional
1125 namespace : dict, optional
1126 an optional dict where completions are performed.
1126 an optional dict where completions are performed.
1127 global_namespace : dict, optional
1127 global_namespace : dict, optional
1128 secondary optional dict for completions, to
1128 secondary optional dict for completions, to
1129 handle cases (such as IPython embedded inside functions) where
1129 handle cases (such as IPython embedded inside functions) where
1130 both Python scopes are visible.
1130 both Python scopes are visible.
1131 use_readline : bool, optional
1131 use_readline : bool, optional
1132 DEPRECATED, ignored since IPython 6.0, will have no effects
1132 DEPRECATED, ignored since IPython 6.0, will have no effects
1133 """
1133 """
1134
1134
1135 self.magic_escape = ESC_MAGIC
1135 self.magic_escape = ESC_MAGIC
1136 self.splitter = CompletionSplitter()
1136 self.splitter = CompletionSplitter()
1137
1137
1138 if use_readline is not _deprecation_readline_sentinel:
1138 if use_readline is not _deprecation_readline_sentinel:
1139 warnings.warn('The `use_readline` parameter is deprecated and ignored since IPython 6.0.',
1139 warnings.warn('The `use_readline` parameter is deprecated and ignored since IPython 6.0.',
1140 DeprecationWarning, stacklevel=2)
1140 DeprecationWarning, stacklevel=2)
1141
1141
1142 # _greedy_changed() depends on splitter and readline being defined:
1142 # _greedy_changed() depends on splitter and readline being defined:
1143 Completer.__init__(self, namespace=namespace, global_namespace=global_namespace,
1143 Completer.__init__(self, namespace=namespace, global_namespace=global_namespace,
1144 config=config, **kwargs)
1144 config=config, **kwargs)
1145
1145
1146 # List where completion matches will be stored
1146 # List where completion matches will be stored
1147 self.matches = []
1147 self.matches = []
1148 self.shell = shell
1148 self.shell = shell
1149 # Regexp to split filenames with spaces in them
1149 # Regexp to split filenames with spaces in them
1150 self.space_name_re = re.compile(r'([^\\] )')
1150 self.space_name_re = re.compile(r'([^\\] )')
1151 # Hold a local ref. to glob.glob for speed
1151 # Hold a local ref. to glob.glob for speed
1152 self.glob = glob.glob
1152 self.glob = glob.glob
1153
1153
1154 # Determine if we are running on 'dumb' terminals, like (X)Emacs
1154 # Determine if we are running on 'dumb' terminals, like (X)Emacs
1155 # buffers, to avoid completion problems.
1155 # buffers, to avoid completion problems.
1156 term = os.environ.get('TERM','xterm')
1156 term = os.environ.get('TERM','xterm')
1157 self.dumb_terminal = term in ['dumb','emacs']
1157 self.dumb_terminal = term in ['dumb','emacs']
1158
1158
1159 # Special handling of backslashes needed in win32 platforms
1159 # Special handling of backslashes needed in win32 platforms
1160 if sys.platform == "win32":
1160 if sys.platform == "win32":
1161 self.clean_glob = self._clean_glob_win32
1161 self.clean_glob = self._clean_glob_win32
1162 else:
1162 else:
1163 self.clean_glob = self._clean_glob
1163 self.clean_glob = self._clean_glob
1164
1164
1165 #regexp to parse docstring for function signature
1165 #regexp to parse docstring for function signature
1166 self.docstring_sig_re = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1166 self.docstring_sig_re = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1167 self.docstring_kwd_re = re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1167 self.docstring_kwd_re = re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1168 #use this if positional argument name is also needed
1168 #use this if positional argument name is also needed
1169 #= re.compile(r'[\s|\[]*(\w+)(?:\s*=?\s*.*)')
1169 #= re.compile(r'[\s|\[]*(\w+)(?:\s*=?\s*.*)')
1170
1170
1171 self.magic_arg_matchers = [
1171 self.magic_arg_matchers = [
1172 self.magic_config_matches,
1172 self.magic_config_matches,
1173 self.magic_color_matches,
1173 self.magic_color_matches,
1174 ]
1174 ]
1175
1175
1176 # This is set externally by InteractiveShell
1176 # This is set externally by InteractiveShell
1177 self.custom_completers = None
1177 self.custom_completers = None
1178
1178
1179 # This is a list of names of unicode characters that can be completed
1179 # This is a list of names of unicode characters that can be completed
1180 # into their corresponding unicode value. The list is large, so we
1180 # into their corresponding unicode value. The list is large, so we
1181 # laziliy initialize it on first use. Consuming code should access this
1181 # laziliy initialize it on first use. Consuming code should access this
1182 # attribute through the `@unicode_names` property.
1182 # attribute through the `@unicode_names` property.
1183 self._unicode_names = None
1183 self._unicode_names = None
1184
1184
1185 @property
1185 @property
1186 def matchers(self) -> List[Any]:
1186 def matchers(self) -> List[Any]:
1187 """All active matcher routines for completion"""
1187 """All active matcher routines for completion"""
1188 if self.dict_keys_only:
1188 if self.dict_keys_only:
1189 return [self.dict_key_matches]
1189 return [self.dict_key_matches]
1190
1190
1191 if self.use_jedi:
1191 if self.use_jedi:
1192 return [
1192 return [
1193 *self.custom_matchers,
1193 *self.custom_matchers,
1194 self.file_matches,
1194 self.file_matches,
1195 self.magic_matches,
1195 self.magic_matches,
1196 self.dict_key_matches,
1196 self.dict_key_matches,
1197 ]
1197 ]
1198 else:
1198 else:
1199 return [
1199 return [
1200 *self.custom_matchers,
1200 *self.custom_matchers,
1201 self.python_matches,
1201 self.python_matches,
1202 self.file_matches,
1202 self.file_matches,
1203 self.magic_matches,
1203 self.magic_matches,
1204 self.python_func_kw_matches,
1204 self.python_func_kw_matches,
1205 self.dict_key_matches,
1205 self.dict_key_matches,
1206 ]
1206 ]
1207
1207
1208 def all_completions(self, text:str) -> List[str]:
1208 def all_completions(self, text:str) -> List[str]:
1209 """
1209 """
1210 Wrapper around the completion methods for the benefit of emacs.
1210 Wrapper around the completion methods for the benefit of emacs.
1211 """
1211 """
1212 prefix = text.rpartition('.')[0]
1212 prefix = text.rpartition('.')[0]
1213 with provisionalcompleter():
1213 with provisionalcompleter():
1214 return ['.'.join([prefix, c.text]) if prefix and self.use_jedi else c.text
1214 return ['.'.join([prefix, c.text]) if prefix and self.use_jedi else c.text
1215 for c in self.completions(text, len(text))]
1215 for c in self.completions(text, len(text))]
1216
1216
1217 return self.complete(text)[1]
1217 return self.complete(text)[1]
1218
1218
1219 def _clean_glob(self, text:str):
1219 def _clean_glob(self, text:str):
1220 return self.glob("%s*" % text)
1220 return self.glob("%s*" % text)
1221
1221
1222 def _clean_glob_win32(self, text:str):
1222 def _clean_glob_win32(self, text:str):
1223 return [f.replace("\\","/")
1223 return [f.replace("\\","/")
1224 for f in self.glob("%s*" % text)]
1224 for f in self.glob("%s*" % text)]
1225
1225
1226 def file_matches(self, text:str)->List[str]:
1226 def file_matches(self, text:str)->List[str]:
1227 """Match filenames, expanding ~USER type strings.
1227 """Match filenames, expanding ~USER type strings.
1228
1228
1229 Most of the seemingly convoluted logic in this completer is an
1229 Most of the seemingly convoluted logic in this completer is an
1230 attempt to handle filenames with spaces in them. And yet it's not
1230 attempt to handle filenames with spaces in them. And yet it's not
1231 quite perfect, because Python's readline doesn't expose all of the
1231 quite perfect, because Python's readline doesn't expose all of the
1232 GNU readline details needed for this to be done correctly.
1232 GNU readline details needed for this to be done correctly.
1233
1233
1234 For a filename with a space in it, the printed completions will be
1234 For a filename with a space in it, the printed completions will be
1235 only the parts after what's already been typed (instead of the
1235 only the parts after what's already been typed (instead of the
1236 full completions, as is normally done). I don't think with the
1236 full completions, as is normally done). I don't think with the
1237 current (as of Python 2.3) Python readline it's possible to do
1237 current (as of Python 2.3) Python readline it's possible to do
1238 better."""
1238 better."""
1239
1239
1240 # chars that require escaping with backslash - i.e. chars
1240 # chars that require escaping with backslash - i.e. chars
1241 # that readline treats incorrectly as delimiters, but we
1241 # that readline treats incorrectly as delimiters, but we
1242 # don't want to treat as delimiters in filename matching
1242 # don't want to treat as delimiters in filename matching
1243 # when escaped with backslash
1243 # when escaped with backslash
1244 if text.startswith('!'):
1244 if text.startswith('!'):
1245 text = text[1:]
1245 text = text[1:]
1246 text_prefix = u'!'
1246 text_prefix = u'!'
1247 else:
1247 else:
1248 text_prefix = u''
1248 text_prefix = u''
1249
1249
1250 text_until_cursor = self.text_until_cursor
1250 text_until_cursor = self.text_until_cursor
1251 # track strings with open quotes
1251 # track strings with open quotes
1252 open_quotes = has_open_quotes(text_until_cursor)
1252 open_quotes = has_open_quotes(text_until_cursor)
1253
1253
1254 if '(' in text_until_cursor or '[' in text_until_cursor:
1254 if '(' in text_until_cursor or '[' in text_until_cursor:
1255 lsplit = text
1255 lsplit = text
1256 else:
1256 else:
1257 try:
1257 try:
1258 # arg_split ~ shlex.split, but with unicode bugs fixed by us
1258 # arg_split ~ shlex.split, but with unicode bugs fixed by us
1259 lsplit = arg_split(text_until_cursor)[-1]
1259 lsplit = arg_split(text_until_cursor)[-1]
1260 except ValueError:
1260 except ValueError:
1261 # typically an unmatched ", or backslash without escaped char.
1261 # typically an unmatched ", or backslash without escaped char.
1262 if open_quotes:
1262 if open_quotes:
1263 lsplit = text_until_cursor.split(open_quotes)[-1]
1263 lsplit = text_until_cursor.split(open_quotes)[-1]
1264 else:
1264 else:
1265 return []
1265 return []
1266 except IndexError:
1266 except IndexError:
1267 # tab pressed on empty line
1267 # tab pressed on empty line
1268 lsplit = ""
1268 lsplit = ""
1269
1269
1270 if not open_quotes and lsplit != protect_filename(lsplit):
1270 if not open_quotes and lsplit != protect_filename(lsplit):
1271 # if protectables are found, do matching on the whole escaped name
1271 # if protectables are found, do matching on the whole escaped name
1272 has_protectables = True
1272 has_protectables = True
1273 text0,text = text,lsplit
1273 text0,text = text,lsplit
1274 else:
1274 else:
1275 has_protectables = False
1275 has_protectables = False
1276 text = os.path.expanduser(text)
1276 text = os.path.expanduser(text)
1277
1277
1278 if text == "":
1278 if text == "":
1279 return [text_prefix + protect_filename(f) for f in self.glob("*")]
1279 return [text_prefix + protect_filename(f) for f in self.glob("*")]
1280
1280
1281 # Compute the matches from the filesystem
1281 # Compute the matches from the filesystem
1282 if sys.platform == 'win32':
1282 if sys.platform == 'win32':
1283 m0 = self.clean_glob(text)
1283 m0 = self.clean_glob(text)
1284 else:
1284 else:
1285 m0 = self.clean_glob(text.replace('\\', ''))
1285 m0 = self.clean_glob(text.replace('\\', ''))
1286
1286
1287 if has_protectables:
1287 if has_protectables:
1288 # If we had protectables, we need to revert our changes to the
1288 # If we had protectables, we need to revert our changes to the
1289 # beginning of filename so that we don't double-write the part
1289 # beginning of filename so that we don't double-write the part
1290 # of the filename we have so far
1290 # of the filename we have so far
1291 len_lsplit = len(lsplit)
1291 len_lsplit = len(lsplit)
1292 matches = [text_prefix + text0 +
1292 matches = [text_prefix + text0 +
1293 protect_filename(f[len_lsplit:]) for f in m0]
1293 protect_filename(f[len_lsplit:]) for f in m0]
1294 else:
1294 else:
1295 if open_quotes:
1295 if open_quotes:
1296 # if we have a string with an open quote, we don't need to
1296 # if we have a string with an open quote, we don't need to
1297 # protect the names beyond the quote (and we _shouldn't_, as
1297 # protect the names beyond the quote (and we _shouldn't_, as
1298 # it would cause bugs when the filesystem call is made).
1298 # it would cause bugs when the filesystem call is made).
1299 matches = m0 if sys.platform == "win32" else\
1299 matches = m0 if sys.platform == "win32" else\
1300 [protect_filename(f, open_quotes) for f in m0]
1300 [protect_filename(f, open_quotes) for f in m0]
1301 else:
1301 else:
1302 matches = [text_prefix +
1302 matches = [text_prefix +
1303 protect_filename(f) for f in m0]
1303 protect_filename(f) for f in m0]
1304
1304
1305 # Mark directories in input list by appending '/' to their names.
1305 # Mark directories in input list by appending '/' to their names.
1306 return [x+'/' if os.path.isdir(x) else x for x in matches]
1306 return [x+'/' if os.path.isdir(x) else x for x in matches]
1307
1307
1308 def magic_matches(self, text:str):
1308 def magic_matches(self, text:str):
1309 """Match magics"""
1309 """Match magics"""
1310 # Get all shell magics now rather than statically, so magics loaded at
1310 # Get all shell magics now rather than statically, so magics loaded at
1311 # runtime show up too.
1311 # runtime show up too.
1312 lsm = self.shell.magics_manager.lsmagic()
1312 lsm = self.shell.magics_manager.lsmagic()
1313 line_magics = lsm['line']
1313 line_magics = lsm['line']
1314 cell_magics = lsm['cell']
1314 cell_magics = lsm['cell']
1315 pre = self.magic_escape
1315 pre = self.magic_escape
1316 pre2 = pre+pre
1316 pre2 = pre+pre
1317
1317
1318 explicit_magic = text.startswith(pre)
1318 explicit_magic = text.startswith(pre)
1319
1319
1320 # Completion logic:
1320 # Completion logic:
1321 # - user gives %%: only do cell magics
1321 # - user gives %%: only do cell magics
1322 # - user gives %: do both line and cell magics
1322 # - user gives %: do both line and cell magics
1323 # - no prefix: do both
1323 # - no prefix: do both
1324 # In other words, line magics are skipped if the user gives %% explicitly
1324 # In other words, line magics are skipped if the user gives %% explicitly
1325 #
1325 #
1326 # We also exclude magics that match any currently visible names:
1326 # We also exclude magics that match any currently visible names:
1327 # https://github.com/ipython/ipython/issues/4877, unless the user has
1327 # https://github.com/ipython/ipython/issues/4877, unless the user has
1328 # typed a %:
1328 # typed a %:
1329 # https://github.com/ipython/ipython/issues/10754
1329 # https://github.com/ipython/ipython/issues/10754
1330 bare_text = text.lstrip(pre)
1330 bare_text = text.lstrip(pre)
1331 global_matches = self.global_matches(bare_text)
1331 global_matches = self.global_matches(bare_text)
1332 if not explicit_magic:
1332 if not explicit_magic:
1333 def matches(magic):
1333 def matches(magic):
1334 """
1334 """
1335 Filter magics, in particular remove magics that match
1335 Filter magics, in particular remove magics that match
1336 a name present in global namespace.
1336 a name present in global namespace.
1337 """
1337 """
1338 return ( magic.startswith(bare_text) and
1338 return ( magic.startswith(bare_text) and
1339 magic not in global_matches )
1339 magic not in global_matches )
1340 else:
1340 else:
1341 def matches(magic):
1341 def matches(magic):
1342 return magic.startswith(bare_text)
1342 return magic.startswith(bare_text)
1343
1343
1344 comp = [ pre2+m for m in cell_magics if matches(m)]
1344 comp = [ pre2+m for m in cell_magics if matches(m)]
1345 if not text.startswith(pre2):
1345 if not text.startswith(pre2):
1346 comp += [ pre+m for m in line_magics if matches(m)]
1346 comp += [ pre+m for m in line_magics if matches(m)]
1347
1347
1348 return comp
1348 return comp
1349
1349
1350 def magic_config_matches(self, text:str) -> List[str]:
1350 def magic_config_matches(self, text:str) -> List[str]:
1351 """ Match class names and attributes for %config magic """
1351 """ Match class names and attributes for %config magic """
1352 texts = text.strip().split()
1352 texts = text.strip().split()
1353
1353
1354 if len(texts) > 0 and (texts[0] == 'config' or texts[0] == '%config'):
1354 if len(texts) > 0 and (texts[0] == 'config' or texts[0] == '%config'):
1355 # get all configuration classes
1355 # get all configuration classes
1356 classes = sorted(set([ c for c in self.shell.configurables
1356 classes = sorted(set([ c for c in self.shell.configurables
1357 if c.__class__.class_traits(config=True)
1357 if c.__class__.class_traits(config=True)
1358 ]), key=lambda x: x.__class__.__name__)
1358 ]), key=lambda x: x.__class__.__name__)
1359 classnames = [ c.__class__.__name__ for c in classes ]
1359 classnames = [ c.__class__.__name__ for c in classes ]
1360
1360
1361 # return all classnames if config or %config is given
1361 # return all classnames if config or %config is given
1362 if len(texts) == 1:
1362 if len(texts) == 1:
1363 return classnames
1363 return classnames
1364
1364
1365 # match classname
1365 # match classname
1366 classname_texts = texts[1].split('.')
1366 classname_texts = texts[1].split('.')
1367 classname = classname_texts[0]
1367 classname = classname_texts[0]
1368 classname_matches = [ c for c in classnames
1368 classname_matches = [ c for c in classnames
1369 if c.startswith(classname) ]
1369 if c.startswith(classname) ]
1370
1370
1371 # return matched classes or the matched class with attributes
1371 # return matched classes or the matched class with attributes
1372 if texts[1].find('.') < 0:
1372 if texts[1].find('.') < 0:
1373 return classname_matches
1373 return classname_matches
1374 elif len(classname_matches) == 1 and \
1374 elif len(classname_matches) == 1 and \
1375 classname_matches[0] == classname:
1375 classname_matches[0] == classname:
1376 cls = classes[classnames.index(classname)].__class__
1376 cls = classes[classnames.index(classname)].__class__
1377 help = cls.class_get_help()
1377 help = cls.class_get_help()
1378 # strip leading '--' from cl-args:
1378 # strip leading '--' from cl-args:
1379 help = re.sub(re.compile(r'^--', re.MULTILINE), '', help)
1379 help = re.sub(re.compile(r'^--', re.MULTILINE), '', help)
1380 return [ attr.split('=')[0]
1380 return [ attr.split('=')[0]
1381 for attr in help.strip().splitlines()
1381 for attr in help.strip().splitlines()
1382 if attr.startswith(texts[1]) ]
1382 if attr.startswith(texts[1]) ]
1383 return []
1383 return []
1384
1384
1385 def magic_color_matches(self, text:str) -> List[str] :
1385 def magic_color_matches(self, text:str) -> List[str] :
1386 """ Match color schemes for %colors magic"""
1386 """ Match color schemes for %colors magic"""
1387 texts = text.split()
1387 texts = text.split()
1388 if text.endswith(' '):
1388 if text.endswith(' '):
1389 # .split() strips off the trailing whitespace. Add '' back
1389 # .split() strips off the trailing whitespace. Add '' back
1390 # so that: '%colors ' -> ['%colors', '']
1390 # so that: '%colors ' -> ['%colors', '']
1391 texts.append('')
1391 texts.append('')
1392
1392
1393 if len(texts) == 2 and (texts[0] == 'colors' or texts[0] == '%colors'):
1393 if len(texts) == 2 and (texts[0] == 'colors' or texts[0] == '%colors'):
1394 prefix = texts[1]
1394 prefix = texts[1]
1395 return [ color for color in InspectColors.keys()
1395 return [ color for color in InspectColors.keys()
1396 if color.startswith(prefix) ]
1396 if color.startswith(prefix) ]
1397 return []
1397 return []
1398
1398
1399 def _jedi_matches(self, cursor_column:int, cursor_line:int, text:str) -> Iterable[Any]:
1399 def _jedi_matches(self, cursor_column:int, cursor_line:int, text:str) -> Iterable[Any]:
1400 """
1400 """
1401 Return a list of :any:`jedi.api.Completions` object from a ``text`` and
1401 Return a list of :any:`jedi.api.Completions` object from a ``text`` and
1402 cursor position.
1402 cursor position.
1403
1403
1404 Parameters
1404 Parameters
1405 ----------
1405 ----------
1406 cursor_column : int
1406 cursor_column : int
1407 column position of the cursor in ``text``, 0-indexed.
1407 column position of the cursor in ``text``, 0-indexed.
1408 cursor_line : int
1408 cursor_line : int
1409 line position of the cursor in ``text``, 0-indexed
1409 line position of the cursor in ``text``, 0-indexed
1410 text : str
1410 text : str
1411 text to complete
1411 text to complete
1412
1412
1413 Notes
1413 Notes
1414 -----
1414 -----
1415 If ``IPCompleter.debug`` is ``True`` may return a :any:`_FakeJediCompletion`
1415 If ``IPCompleter.debug`` is ``True`` may return a :any:`_FakeJediCompletion`
1416 object containing a string with the Jedi debug information attached.
1416 object containing a string with the Jedi debug information attached.
1417 """
1417 """
1418 namespaces = [self.namespace]
1418 namespaces = [self.namespace]
1419 if self.global_namespace is not None:
1419 if self.global_namespace is not None:
1420 namespaces.append(self.global_namespace)
1420 namespaces.append(self.global_namespace)
1421
1421
1422 completion_filter = lambda x:x
1422 completion_filter = lambda x:x
1423 offset = cursor_to_position(text, cursor_line, cursor_column)
1423 offset = cursor_to_position(text, cursor_line, cursor_column)
1424 # filter output if we are completing for object members
1424 # filter output if we are completing for object members
1425 if offset:
1425 if offset:
1426 pre = text[offset-1]
1426 pre = text[offset-1]
1427 if pre == '.':
1427 if pre == '.':
1428 if self.omit__names == 2:
1428 if self.omit__names == 2:
1429 completion_filter = lambda c:not c.name.startswith('_')
1429 completion_filter = lambda c:not c.name.startswith('_')
1430 elif self.omit__names == 1:
1430 elif self.omit__names == 1:
1431 completion_filter = lambda c:not (c.name.startswith('__') and c.name.endswith('__'))
1431 completion_filter = lambda c:not (c.name.startswith('__') and c.name.endswith('__'))
1432 elif self.omit__names == 0:
1432 elif self.omit__names == 0:
1433 completion_filter = lambda x:x
1433 completion_filter = lambda x:x
1434 else:
1434 else:
1435 raise ValueError("Don't understand self.omit__names == {}".format(self.omit__names))
1435 raise ValueError("Don't understand self.omit__names == {}".format(self.omit__names))
1436
1436
1437 interpreter = jedi.Interpreter(text[:offset], namespaces)
1437 interpreter = jedi.Interpreter(text[:offset], namespaces)
1438 try_jedi = True
1438 try_jedi = True
1439
1439
1440 try:
1440 try:
1441 # find the first token in the current tree -- if it is a ' or " then we are in a string
1441 # find the first token in the current tree -- if it is a ' or " then we are in a string
1442 completing_string = False
1442 completing_string = False
1443 try:
1443 try:
1444 first_child = next(c for c in interpreter._get_module().tree_node.children if hasattr(c, 'value'))
1444 first_child = next(c for c in interpreter._get_module().tree_node.children if hasattr(c, 'value'))
1445 except StopIteration:
1445 except StopIteration:
1446 pass
1446 pass
1447 else:
1447 else:
1448 # note the value may be ', ", or it may also be ''' or """, or
1448 # note the value may be ', ", or it may also be ''' or """, or
1449 # in some cases, """what/you/typed..., but all of these are
1449 # in some cases, """what/you/typed..., but all of these are
1450 # strings.
1450 # strings.
1451 completing_string = len(first_child.value) > 0 and first_child.value[0] in {"'", '"'}
1451 completing_string = len(first_child.value) > 0 and first_child.value[0] in {"'", '"'}
1452
1452
1453 # if we are in a string jedi is likely not the right candidate for
1453 # if we are in a string jedi is likely not the right candidate for
1454 # now. Skip it.
1454 # now. Skip it.
1455 try_jedi = not completing_string
1455 try_jedi = not completing_string
1456 except Exception as e:
1456 except Exception as e:
1457 # many of things can go wrong, we are using private API just don't crash.
1457 # many of things can go wrong, we are using private API just don't crash.
1458 if self.debug:
1458 if self.debug:
1459 print("Error detecting if completing a non-finished string :", e, '|')
1459 print("Error detecting if completing a non-finished string :", e, '|')
1460
1460
1461 if not try_jedi:
1461 if not try_jedi:
1462 return []
1462 return []
1463 try:
1463 try:
1464 return filter(completion_filter, interpreter.complete(column=cursor_column, line=cursor_line + 1))
1464 return filter(completion_filter, interpreter.complete(column=cursor_column, line=cursor_line + 1))
1465 except Exception as e:
1465 except Exception as e:
1466 if self.debug:
1466 if self.debug:
1467 return [_FakeJediCompletion('Oops Jedi has crashed, please report a bug with the following:\n"""\n%s\ns"""' % (e))]
1467 return [_FakeJediCompletion('Oops Jedi has crashed, please report a bug with the following:\n"""\n%s\ns"""' % (e))]
1468 else:
1468 else:
1469 return []
1469 return []
1470
1470
1471 def python_matches(self, text:str)->List[str]:
1471 def python_matches(self, text:str)->List[str]:
1472 """Match attributes or global python names"""
1472 """Match attributes or global python names"""
1473 if "." in text:
1473 if "." in text:
1474 try:
1474 try:
1475 matches = self.attr_matches(text)
1475 matches = self.attr_matches(text)
1476 if text.endswith('.') and self.omit__names:
1476 if text.endswith('.') and self.omit__names:
1477 if self.omit__names == 1:
1477 if self.omit__names == 1:
1478 # true if txt is _not_ a __ name, false otherwise:
1478 # true if txt is _not_ a __ name, false otherwise:
1479 no__name = (lambda txt:
1479 no__name = (lambda txt:
1480 re.match(r'.*\.__.*?__',txt) is None)
1480 re.match(r'.*\.__.*?__',txt) is None)
1481 else:
1481 else:
1482 # true if txt is _not_ a _ name, false otherwise:
1482 # true if txt is _not_ a _ name, false otherwise:
1483 no__name = (lambda txt:
1483 no__name = (lambda txt:
1484 re.match(r'\._.*?',txt[txt.rindex('.'):]) is None)
1484 re.match(r'\._.*?',txt[txt.rindex('.'):]) is None)
1485 matches = filter(no__name, matches)
1485 matches = filter(no__name, matches)
1486 except NameError:
1486 except NameError:
1487 # catches <undefined attributes>.<tab>
1487 # catches <undefined attributes>.<tab>
1488 matches = []
1488 matches = []
1489 else:
1489 else:
1490 matches = self.global_matches(text)
1490 matches = self.global_matches(text)
1491 return matches
1491 return matches
1492
1492
1493 def _default_arguments_from_docstring(self, doc):
1493 def _default_arguments_from_docstring(self, doc):
1494 """Parse the first line of docstring for call signature.
1494 """Parse the first line of docstring for call signature.
1495
1495
1496 Docstring should be of the form 'min(iterable[, key=func])\n'.
1496 Docstring should be of the form 'min(iterable[, key=func])\n'.
1497 It can also parse cython docstring of the form
1497 It can also parse cython docstring of the form
1498 'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)'.
1498 'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)'.
1499 """
1499 """
1500 if doc is None:
1500 if doc is None:
1501 return []
1501 return []
1502
1502
1503 #care only the firstline
1503 #care only the firstline
1504 line = doc.lstrip().splitlines()[0]
1504 line = doc.lstrip().splitlines()[0]
1505
1505
1506 #p = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1506 #p = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
1507 #'min(iterable[, key=func])\n' -> 'iterable[, key=func]'
1507 #'min(iterable[, key=func])\n' -> 'iterable[, key=func]'
1508 sig = self.docstring_sig_re.search(line)
1508 sig = self.docstring_sig_re.search(line)
1509 if sig is None:
1509 if sig is None:
1510 return []
1510 return []
1511 # iterable[, key=func]' -> ['iterable[' ,' key=func]']
1511 # iterable[, key=func]' -> ['iterable[' ,' key=func]']
1512 sig = sig.groups()[0].split(',')
1512 sig = sig.groups()[0].split(',')
1513 ret = []
1513 ret = []
1514 for s in sig:
1514 for s in sig:
1515 #re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1515 #re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
1516 ret += self.docstring_kwd_re.findall(s)
1516 ret += self.docstring_kwd_re.findall(s)
1517 return ret
1517 return ret
1518
1518
1519 def _default_arguments(self, obj):
1519 def _default_arguments(self, obj):
1520 """Return the list of default arguments of obj if it is callable,
1520 """Return the list of default arguments of obj if it is callable,
1521 or empty list otherwise."""
1521 or empty list otherwise."""
1522 call_obj = obj
1522 call_obj = obj
1523 ret = []
1523 ret = []
1524 if inspect.isbuiltin(obj):
1524 if inspect.isbuiltin(obj):
1525 pass
1525 pass
1526 elif not (inspect.isfunction(obj) or inspect.ismethod(obj)):
1526 elif not (inspect.isfunction(obj) or inspect.ismethod(obj)):
1527 if inspect.isclass(obj):
1527 if inspect.isclass(obj):
1528 #for cython embedsignature=True the constructor docstring
1528 #for cython embedsignature=True the constructor docstring
1529 #belongs to the object itself not __init__
1529 #belongs to the object itself not __init__
1530 ret += self._default_arguments_from_docstring(
1530 ret += self._default_arguments_from_docstring(
1531 getattr(obj, '__doc__', ''))
1531 getattr(obj, '__doc__', ''))
1532 # for classes, check for __init__,__new__
1532 # for classes, check for __init__,__new__
1533 call_obj = (getattr(obj, '__init__', None) or
1533 call_obj = (getattr(obj, '__init__', None) or
1534 getattr(obj, '__new__', None))
1534 getattr(obj, '__new__', None))
1535 # for all others, check if they are __call__able
1535 # for all others, check if they are __call__able
1536 elif hasattr(obj, '__call__'):
1536 elif hasattr(obj, '__call__'):
1537 call_obj = obj.__call__
1537 call_obj = obj.__call__
1538 ret += self._default_arguments_from_docstring(
1538 ret += self._default_arguments_from_docstring(
1539 getattr(call_obj, '__doc__', ''))
1539 getattr(call_obj, '__doc__', ''))
1540
1540
1541 _keeps = (inspect.Parameter.KEYWORD_ONLY,
1541 _keeps = (inspect.Parameter.KEYWORD_ONLY,
1542 inspect.Parameter.POSITIONAL_OR_KEYWORD)
1542 inspect.Parameter.POSITIONAL_OR_KEYWORD)
1543
1543
1544 try:
1544 try:
1545 sig = inspect.signature(obj)
1545 sig = inspect.signature(obj)
1546 ret.extend(k for k, v in sig.parameters.items() if
1546 ret.extend(k for k, v in sig.parameters.items() if
1547 v.kind in _keeps)
1547 v.kind in _keeps)
1548 except ValueError:
1548 except ValueError:
1549 pass
1549 pass
1550
1550
1551 return list(set(ret))
1551 return list(set(ret))
1552
1552
1553 def python_func_kw_matches(self, text):
1553 def python_func_kw_matches(self, text):
1554 """Match named parameters (kwargs) of the last open function"""
1554 """Match named parameters (kwargs) of the last open function"""
1555
1555
1556 if "." in text: # a parameter cannot be dotted
1556 if "." in text: # a parameter cannot be dotted
1557 return []
1557 return []
1558 try: regexp = self.__funcParamsRegex
1558 try: regexp = self.__funcParamsRegex
1559 except AttributeError:
1559 except AttributeError:
1560 regexp = self.__funcParamsRegex = re.compile(r'''
1560 regexp = self.__funcParamsRegex = re.compile(r'''
1561 '.*?(?<!\\)' | # single quoted strings or
1561 '.*?(?<!\\)' | # single quoted strings or
1562 ".*?(?<!\\)" | # double quoted strings or
1562 ".*?(?<!\\)" | # double quoted strings or
1563 \w+ | # identifier
1563 \w+ | # identifier
1564 \S # other characters
1564 \S # other characters
1565 ''', re.VERBOSE | re.DOTALL)
1565 ''', re.VERBOSE | re.DOTALL)
1566 # 1. find the nearest identifier that comes before an unclosed
1566 # 1. find the nearest identifier that comes before an unclosed
1567 # parenthesis before the cursor
1567 # parenthesis before the cursor
1568 # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
1568 # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
1569 tokens = regexp.findall(self.text_until_cursor)
1569 tokens = regexp.findall(self.text_until_cursor)
1570 iterTokens = reversed(tokens); openPar = 0
1570 iterTokens = reversed(tokens); openPar = 0
1571
1571
1572 for token in iterTokens:
1572 for token in iterTokens:
1573 if token == ')':
1573 if token == ')':
1574 openPar -= 1
1574 openPar -= 1
1575 elif token == '(':
1575 elif token == '(':
1576 openPar += 1
1576 openPar += 1
1577 if openPar > 0:
1577 if openPar > 0:
1578 # found the last unclosed parenthesis
1578 # found the last unclosed parenthesis
1579 break
1579 break
1580 else:
1580 else:
1581 return []
1581 return []
1582 # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
1582 # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
1583 ids = []
1583 ids = []
1584 isId = re.compile(r'\w+$').match
1584 isId = re.compile(r'\w+$').match
1585
1585
1586 while True:
1586 while True:
1587 try:
1587 try:
1588 ids.append(next(iterTokens))
1588 ids.append(next(iterTokens))
1589 if not isId(ids[-1]):
1589 if not isId(ids[-1]):
1590 ids.pop(); break
1590 ids.pop(); break
1591 if not next(iterTokens) == '.':
1591 if not next(iterTokens) == '.':
1592 break
1592 break
1593 except StopIteration:
1593 except StopIteration:
1594 break
1594 break
1595
1595
1596 # Find all named arguments already assigned to, as to avoid suggesting
1596 # Find all named arguments already assigned to, as to avoid suggesting
1597 # them again
1597 # them again
1598 usedNamedArgs = set()
1598 usedNamedArgs = set()
1599 par_level = -1
1599 par_level = -1
1600 for token, next_token in zip(tokens, tokens[1:]):
1600 for token, next_token in zip(tokens, tokens[1:]):
1601 if token == '(':
1601 if token == '(':
1602 par_level += 1
1602 par_level += 1
1603 elif token == ')':
1603 elif token == ')':
1604 par_level -= 1
1604 par_level -= 1
1605
1605
1606 if par_level != 0:
1606 if par_level != 0:
1607 continue
1607 continue
1608
1608
1609 if next_token != '=':
1609 if next_token != '=':
1610 continue
1610 continue
1611
1611
1612 usedNamedArgs.add(token)
1612 usedNamedArgs.add(token)
1613
1613
1614 argMatches = []
1614 argMatches = []
1615 try:
1615 try:
1616 callableObj = '.'.join(ids[::-1])
1616 callableObj = '.'.join(ids[::-1])
1617 namedArgs = self._default_arguments(eval(callableObj,
1617 namedArgs = self._default_arguments(eval(callableObj,
1618 self.namespace))
1618 self.namespace))
1619
1619
1620 # Remove used named arguments from the list, no need to show twice
1620 # Remove used named arguments from the list, no need to show twice
1621 for namedArg in set(namedArgs) - usedNamedArgs:
1621 for namedArg in set(namedArgs) - usedNamedArgs:
1622 if namedArg.startswith(text):
1622 if namedArg.startswith(text):
1623 argMatches.append("%s=" %namedArg)
1623 argMatches.append("%s=" %namedArg)
1624 except:
1624 except:
1625 pass
1625 pass
1626
1626
1627 return argMatches
1627 return argMatches
1628
1628
1629 @staticmethod
1629 @staticmethod
1630 def _get_keys(obj: Any) -> List[Any]:
1630 def _get_keys(obj: Any) -> List[Any]:
1631 # Objects can define their own completions by defining an
1631 # Objects can define their own completions by defining an
1632 # _ipy_key_completions_() method.
1632 # _ipy_key_completions_() method.
1633 method = get_real_method(obj, '_ipython_key_completions_')
1633 method = get_real_method(obj, '_ipython_key_completions_')
1634 if method is not None:
1634 if method is not None:
1635 return method()
1635 return method()
1636
1636
1637 # Special case some common in-memory dict-like types
1637 # Special case some common in-memory dict-like types
1638 if isinstance(obj, dict) or\
1638 if isinstance(obj, dict) or\
1639 _safe_isinstance(obj, 'pandas', 'DataFrame'):
1639 _safe_isinstance(obj, 'pandas', 'DataFrame'):
1640 try:
1640 try:
1641 return list(obj.keys())
1641 return list(obj.keys())
1642 except Exception:
1642 except Exception:
1643 return []
1643 return []
1644 elif _safe_isinstance(obj, 'numpy', 'ndarray') or\
1644 elif _safe_isinstance(obj, 'numpy', 'ndarray') or\
1645 _safe_isinstance(obj, 'numpy', 'void'):
1645 _safe_isinstance(obj, 'numpy', 'void'):
1646 return obj.dtype.names or []
1646 return obj.dtype.names or []
1647 return []
1647 return []
1648
1648
1649 def dict_key_matches(self, text:str) -> List[str]:
1649 def dict_key_matches(self, text:str) -> List[str]:
1650 "Match string keys in a dictionary, after e.g. 'foo[' "
1650 "Match string keys in a dictionary, after e.g. 'foo[' "
1651
1651
1652
1652
1653 if self.__dict_key_regexps is not None:
1653 if self.__dict_key_regexps is not None:
1654 regexps = self.__dict_key_regexps
1654 regexps = self.__dict_key_regexps
1655 else:
1655 else:
1656 dict_key_re_fmt = r'''(?x)
1656 dict_key_re_fmt = r'''(?x)
1657 ( # match dict-referring expression wrt greedy setting
1657 ( # match dict-referring expression wrt greedy setting
1658 %s
1658 %s
1659 )
1659 )
1660 \[ # open bracket
1660 \[ # open bracket
1661 \s* # and optional whitespace
1661 \s* # and optional whitespace
1662 # Capture any number of str-like objects (e.g. "a", "b", 'c')
1662 # Capture any number of str-like objects (e.g. "a", "b", 'c')
1663 ((?:[uUbB]? # string prefix (r not handled)
1663 ((?:[uUbB]? # string prefix (r not handled)
1664 (?:
1664 (?:
1665 '(?:[^']|(?<!\\)\\')*'
1665 '(?:[^']|(?<!\\)\\')*'
1666 |
1666 |
1667 "(?:[^"]|(?<!\\)\\")*"
1667 "(?:[^"]|(?<!\\)\\")*"
1668 )
1668 )
1669 \s*,\s*
1669 \s*,\s*
1670 )*)
1670 )*)
1671 ([uUbB]? # string prefix (r not handled)
1671 ([uUbB]? # string prefix (r not handled)
1672 (?: # unclosed string
1672 (?: # unclosed string
1673 '(?:[^']|(?<!\\)\\')*
1673 '(?:[^']|(?<!\\)\\')*
1674 |
1674 |
1675 "(?:[^"]|(?<!\\)\\")*
1675 "(?:[^"]|(?<!\\)\\")*
1676 )
1676 )
1677 )?
1677 )?
1678 $
1678 $
1679 '''
1679 '''
1680 regexps = self.__dict_key_regexps = {
1680 regexps = self.__dict_key_regexps = {
1681 False: re.compile(dict_key_re_fmt % r'''
1681 False: re.compile(dict_key_re_fmt % r'''
1682 # identifiers separated by .
1682 # identifiers separated by .
1683 (?!\d)\w+
1683 (?!\d)\w+
1684 (?:\.(?!\d)\w+)*
1684 (?:\.(?!\d)\w+)*
1685 '''),
1685 '''),
1686 True: re.compile(dict_key_re_fmt % '''
1686 True: re.compile(dict_key_re_fmt % '''
1687 .+
1687 .+
1688 ''')
1688 ''')
1689 }
1689 }
1690
1690
1691 match = regexps[self.greedy].search(self.text_until_cursor)
1691 match = regexps[self.greedy].search(self.text_until_cursor)
1692
1692
1693 if match is None:
1693 if match is None:
1694 return []
1694 return []
1695
1695
1696 expr, prefix0, prefix = match.groups()
1696 expr, prefix0, prefix = match.groups()
1697 try:
1697 try:
1698 obj = eval(expr, self.namespace)
1698 obj = eval(expr, self.namespace)
1699 except Exception:
1699 except Exception:
1700 try:
1700 try:
1701 obj = eval(expr, self.global_namespace)
1701 obj = eval(expr, self.global_namespace)
1702 except Exception:
1702 except Exception:
1703 return []
1703 return []
1704
1704
1705 keys = self._get_keys(obj)
1705 keys = self._get_keys(obj)
1706 if not keys:
1706 if not keys:
1707 return keys
1707 return keys
1708
1708
1709 extra_prefix = eval(prefix0) if prefix0 != '' else None
1709 extra_prefix = eval(prefix0) if prefix0 != '' else None
1710
1710
1711 closing_quote, token_offset, matches = match_dict_keys(keys, prefix, self.splitter.delims, extra_prefix=extra_prefix)
1711 closing_quote, token_offset, matches = match_dict_keys(keys, prefix, self.splitter.delims, extra_prefix=extra_prefix)
1712 if not matches:
1712 if not matches:
1713 return matches
1713 return matches
1714
1714
1715 # get the cursor position of
1715 # get the cursor position of
1716 # - the text being completed
1716 # - the text being completed
1717 # - the start of the key text
1717 # - the start of the key text
1718 # - the start of the completion
1718 # - the start of the completion
1719 text_start = len(self.text_until_cursor) - len(text)
1719 text_start = len(self.text_until_cursor) - len(text)
1720 if prefix:
1720 if prefix:
1721 key_start = match.start(3)
1721 key_start = match.start(3)
1722 completion_start = key_start + token_offset
1722 completion_start = key_start + token_offset
1723 else:
1723 else:
1724 key_start = completion_start = match.end()
1724 key_start = completion_start = match.end()
1725
1725
1726 # grab the leading prefix, to make sure all completions start with `text`
1726 # grab the leading prefix, to make sure all completions start with `text`
1727 if text_start > key_start:
1727 if text_start > key_start:
1728 leading = ''
1728 leading = ''
1729 else:
1729 else:
1730 leading = text[text_start:completion_start]
1730 leading = text[text_start:completion_start]
1731
1731
1732 # the index of the `[` character
1732 # the index of the `[` character
1733 bracket_idx = match.end(1)
1733 bracket_idx = match.end(1)
1734
1734
1735 # append closing quote and bracket as appropriate
1735 # append closing quote and bracket as appropriate
1736 # this is *not* appropriate if the opening quote or bracket is outside
1736 # this is *not* appropriate if the opening quote or bracket is outside
1737 # the text given to this method
1737 # the text given to this method
1738 suf = ''
1738 suf = ''
1739 continuation = self.line_buffer[len(self.text_until_cursor):]
1739 continuation = self.line_buffer[len(self.text_until_cursor):]
1740 if key_start > text_start and closing_quote:
1740 if key_start > text_start and closing_quote:
1741 # quotes were opened inside text, maybe close them
1741 # quotes were opened inside text, maybe close them
1742 if continuation.startswith(closing_quote):
1742 if continuation.startswith(closing_quote):
1743 continuation = continuation[len(closing_quote):]
1743 continuation = continuation[len(closing_quote):]
1744 else:
1744 else:
1745 suf += closing_quote
1745 suf += closing_quote
1746 if bracket_idx > text_start:
1746 if bracket_idx > text_start:
1747 # brackets were opened inside text, maybe close them
1747 # brackets were opened inside text, maybe close them
1748 if not continuation.startswith(']'):
1748 if not continuation.startswith(']'):
1749 suf += ']'
1749 suf += ']'
1750
1750
1751 return [leading + k + suf for k in matches]
1751 return [leading + k + suf for k in matches]
1752
1752
1753 @staticmethod
1753 @staticmethod
1754 def unicode_name_matches(text:str) -> Tuple[str, List[str]] :
1754 def unicode_name_matches(text:str) -> Tuple[str, List[str]] :
1755 """Match Latex-like syntax for unicode characters base
1755 """Match Latex-like syntax for unicode characters base
1756 on the name of the character.
1756 on the name of the character.
1757
1757
1758 This does ``\\GREEK SMALL LETTER ETA`` -> ``η``
1758 This does ``\\GREEK SMALL LETTER ETA`` -> ``η``
1759
1759
1760 Works only on valid python 3 identifier, or on combining characters that
1760 Works only on valid python 3 identifier, or on combining characters that
1761 will combine to form a valid identifier.
1761 will combine to form a valid identifier.
1762 """
1762 """
1763 slashpos = text.rfind('\\')
1763 slashpos = text.rfind('\\')
1764 if slashpos > -1:
1764 if slashpos > -1:
1765 s = text[slashpos+1:]
1765 s = text[slashpos+1:]
1766 try :
1766 try :
1767 unic = unicodedata.lookup(s)
1767 unic = unicodedata.lookup(s)
1768 # allow combining chars
1768 # allow combining chars
1769 if ('a'+unic).isidentifier():
1769 if ('a'+unic).isidentifier():
1770 return '\\'+s,[unic]
1770 return '\\'+s,[unic]
1771 except KeyError:
1771 except KeyError:
1772 pass
1772 pass
1773 return '', []
1773 return '', []
1774
1774
1775
1775
1776 def latex_matches(self, text:str) -> Tuple[str, Sequence[str]]:
1776 def latex_matches(self, text:str) -> Tuple[str, Sequence[str]]:
1777 """Match Latex syntax for unicode characters.
1777 """Match Latex syntax for unicode characters.
1778
1778
1779 This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
1779 This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
1780 """
1780 """
1781 slashpos = text.rfind('\\')
1781 slashpos = text.rfind('\\')
1782 if slashpos > -1:
1782 if slashpos > -1:
1783 s = text[slashpos:]
1783 s = text[slashpos:]
1784 if s in latex_symbols:
1784 if s in latex_symbols:
1785 # Try to complete a full latex symbol to unicode
1785 # Try to complete a full latex symbol to unicode
1786 # \\alpha -> α
1786 # \\alpha -> α
1787 return s, [latex_symbols[s]]
1787 return s, [latex_symbols[s]]
1788 else:
1788 else:
1789 # If a user has partially typed a latex symbol, give them
1789 # If a user has partially typed a latex symbol, give them
1790 # a full list of options \al -> [\aleph, \alpha]
1790 # a full list of options \al -> [\aleph, \alpha]
1791 matches = [k for k in latex_symbols if k.startswith(s)]
1791 matches = [k for k in latex_symbols if k.startswith(s)]
1792 if matches:
1792 if matches:
1793 return s, matches
1793 return s, matches
1794 return '', ()
1794 return '', ()
1795
1795
1796 def dispatch_custom_completer(self, text):
1796 def dispatch_custom_completer(self, text):
1797 if not self.custom_completers:
1797 if not self.custom_completers:
1798 return
1798 return
1799
1799
1800 line = self.line_buffer
1800 line = self.line_buffer
1801 if not line.strip():
1801 if not line.strip():
1802 return None
1802 return None
1803
1803
1804 # Create a little structure to pass all the relevant information about
1804 # Create a little structure to pass all the relevant information about
1805 # the current completion to any custom completer.
1805 # the current completion to any custom completer.
1806 event = SimpleNamespace()
1806 event = SimpleNamespace()
1807 event.line = line
1807 event.line = line
1808 event.symbol = text
1808 event.symbol = text
1809 cmd = line.split(None,1)[0]
1809 cmd = line.split(None,1)[0]
1810 event.command = cmd
1810 event.command = cmd
1811 event.text_until_cursor = self.text_until_cursor
1811 event.text_until_cursor = self.text_until_cursor
1812
1812
1813 # for foo etc, try also to find completer for %foo
1813 # for foo etc, try also to find completer for %foo
1814 if not cmd.startswith(self.magic_escape):
1814 if not cmd.startswith(self.magic_escape):
1815 try_magic = self.custom_completers.s_matches(
1815 try_magic = self.custom_completers.s_matches(
1816 self.magic_escape + cmd)
1816 self.magic_escape + cmd)
1817 else:
1817 else:
1818 try_magic = []
1818 try_magic = []
1819
1819
1820 for c in itertools.chain(self.custom_completers.s_matches(cmd),
1820 for c in itertools.chain(self.custom_completers.s_matches(cmd),
1821 try_magic,
1821 try_magic,
1822 self.custom_completers.flat_matches(self.text_until_cursor)):
1822 self.custom_completers.flat_matches(self.text_until_cursor)):
1823 try:
1823 try:
1824 res = c(event)
1824 res = c(event)
1825 if res:
1825 if res:
1826 # first, try case sensitive match
1826 # first, try case sensitive match
1827 withcase = [r for r in res if r.startswith(text)]
1827 withcase = [r for r in res if r.startswith(text)]
1828 if withcase:
1828 if withcase:
1829 return withcase
1829 return withcase
1830 # if none, then case insensitive ones are ok too
1830 # if none, then case insensitive ones are ok too
1831 text_low = text.lower()
1831 text_low = text.lower()
1832 return [r for r in res if r.lower().startswith(text_low)]
1832 return [r for r in res if r.lower().startswith(text_low)]
1833 except TryNext:
1833 except TryNext:
1834 pass
1834 pass
1835 except KeyboardInterrupt:
1835 except KeyboardInterrupt:
1836 """
1836 """
1837 If custom completer take too long,
1837 If custom completer take too long,
1838 let keyboard interrupt abort and return nothing.
1838 let keyboard interrupt abort and return nothing.
1839 """
1839 """
1840 break
1840 break
1841
1841
1842 return None
1842 return None
1843
1843
1844 def completions(self, text: str, offset: int)->Iterator[Completion]:
1844 def completions(self, text: str, offset: int)->Iterator[Completion]:
1845 """
1845 """
1846 Returns an iterator over the possible completions
1846 Returns an iterator over the possible completions
1847
1847
1848 .. warning::
1848 .. warning::
1849
1849
1850 Unstable
1850 Unstable
1851
1851
1852 This function is unstable, API may change without warning.
1852 This function is unstable, API may change without warning.
1853 It will also raise unless use in proper context manager.
1853 It will also raise unless use in proper context manager.
1854
1854
1855 Parameters
1855 Parameters
1856 ----------
1856 ----------
1857 text : str
1857 text : str
1858 Full text of the current input, multi line string.
1858 Full text of the current input, multi line string.
1859 offset : int
1859 offset : int
1860 Integer representing the position of the cursor in ``text``. Offset
1860 Integer representing the position of the cursor in ``text``. Offset
1861 is 0-based indexed.
1861 is 0-based indexed.
1862
1862
1863 Yields
1863 Yields
1864 ------
1864 ------
1865 Completion
1865 Completion
1866
1866
1867 Notes
1867 Notes
1868 -----
1868 -----
1869 The cursor on a text can either be seen as being "in between"
1869 The cursor on a text can either be seen as being "in between"
1870 characters or "On" a character depending on the interface visible to
1870 characters or "On" a character depending on the interface visible to
1871 the user. For consistency the cursor being on "in between" characters X
1871 the user. For consistency the cursor being on "in between" characters X
1872 and Y is equivalent to the cursor being "on" character Y, that is to say
1872 and Y is equivalent to the cursor being "on" character Y, that is to say
1873 the character the cursor is on is considered as being after the cursor.
1873 the character the cursor is on is considered as being after the cursor.
1874
1874
1875 Combining characters may span more that one position in the
1875 Combining characters may span more that one position in the
1876 text.
1876 text.
1877
1877
1878 .. note::
1878 .. note::
1879
1879
1880 If ``IPCompleter.debug`` is :any:`True` will yield a ``--jedi/ipython--``
1880 If ``IPCompleter.debug`` is :any:`True` will yield a ``--jedi/ipython--``
1881 fake Completion token to distinguish completion returned by Jedi
1881 fake Completion token to distinguish completion returned by Jedi
1882 and usual IPython completion.
1882 and usual IPython completion.
1883
1883
1884 .. note::
1884 .. note::
1885
1885
1886 Completions are not completely deduplicated yet. If identical
1886 Completions are not completely deduplicated yet. If identical
1887 completions are coming from different sources this function does not
1887 completions are coming from different sources this function does not
1888 ensure that each completion object will only be present once.
1888 ensure that each completion object will only be present once.
1889 """
1889 """
1890 warnings.warn("_complete is a provisional API (as of IPython 6.0). "
1890 warnings.warn("_complete is a provisional API (as of IPython 6.0). "
1891 "It may change without warnings. "
1891 "It may change without warnings. "
1892 "Use in corresponding context manager.",
1892 "Use in corresponding context manager.",
1893 category=ProvisionalCompleterWarning, stacklevel=2)
1893 category=ProvisionalCompleterWarning, stacklevel=2)
1894
1894
1895 seen = set()
1895 seen = set()
1896 profiler:Optional[cProfile.Profile]
1896 profiler:Optional[cProfile.Profile]
1897 try:
1897 try:
1898 if self.profile_completions:
1898 if self.profile_completions:
1899 import cProfile
1899 import cProfile
1900 profiler = cProfile.Profile()
1900 profiler = cProfile.Profile()
1901 profiler.enable()
1901 profiler.enable()
1902 else:
1902 else:
1903 profiler = None
1903 profiler = None
1904
1904
1905 for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
1905 for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
1906 if c and (c in seen):
1906 if c and (c in seen):
1907 continue
1907 continue
1908 yield c
1908 yield c
1909 seen.add(c)
1909 seen.add(c)
1910 except KeyboardInterrupt:
1910 except KeyboardInterrupt:
1911 """if completions take too long and users send keyboard interrupt,
1911 """if completions take too long and users send keyboard interrupt,
1912 do not crash and return ASAP. """
1912 do not crash and return ASAP. """
1913 pass
1913 pass
1914 finally:
1914 finally:
1915 if profiler is not None:
1915 if profiler is not None:
1916 profiler.disable()
1916 profiler.disable()
1917 ensure_dir_exists(self.profiler_output_dir)
1917 ensure_dir_exists(self.profiler_output_dir)
1918 output_path = os.path.join(self.profiler_output_dir, str(uuid.uuid4()))
1918 output_path = os.path.join(self.profiler_output_dir, str(uuid.uuid4()))
1919 print("Writing profiler output to", output_path)
1919 print("Writing profiler output to", output_path)
1920 profiler.dump_stats(output_path)
1920 profiler.dump_stats(output_path)
1921
1921
1922 def _completions(self, full_text: str, offset: int, *, _timeout) -> Iterator[Completion]:
1922 def _completions(self, full_text: str, offset: int, *, _timeout) -> Iterator[Completion]:
1923 """
1923 """
1924 Core completion module.Same signature as :any:`completions`, with the
1924 Core completion module.Same signature as :any:`completions`, with the
1925 extra `timeout` parameter (in seconds).
1925 extra `timeout` parameter (in seconds).
1926
1926
1927 Computing jedi's completion ``.type`` can be quite expensive (it is a
1927 Computing jedi's completion ``.type`` can be quite expensive (it is a
1928 lazy property) and can require some warm-up, more warm up than just
1928 lazy property) and can require some warm-up, more warm up than just
1929 computing the ``name`` of a completion. The warm-up can be :
1929 computing the ``name`` of a completion. The warm-up can be :
1930
1930
1931 - Long warm-up the first time a module is encountered after
1931 - Long warm-up the first time a module is encountered after
1932 install/update: actually build parse/inference tree.
1932 install/update: actually build parse/inference tree.
1933
1933
1934 - first time the module is encountered in a session: load tree from
1934 - first time the module is encountered in a session: load tree from
1935 disk.
1935 disk.
1936
1936
1937 We don't want to block completions for tens of seconds so we give the
1937 We don't want to block completions for tens of seconds so we give the
1938 completer a "budget" of ``_timeout`` seconds per invocation to compute
1938 completer a "budget" of ``_timeout`` seconds per invocation to compute
1939 completions types, the completions that have not yet been computed will
1939 completions types, the completions that have not yet been computed will
1940 be marked as "unknown" an will have a chance to be computed next round
1940 be marked as "unknown" an will have a chance to be computed next round
1941 are things get cached.
1941 are things get cached.
1942
1942
1943 Keep in mind that Jedi is not the only thing treating the completion so
1943 Keep in mind that Jedi is not the only thing treating the completion so
1944 keep the timeout short-ish as if we take more than 0.3 second we still
1944 keep the timeout short-ish as if we take more than 0.3 second we still
1945 have lots of processing to do.
1945 have lots of processing to do.
1946
1946
1947 """
1947 """
1948 deadline = time.monotonic() + _timeout
1948 deadline = time.monotonic() + _timeout
1949
1949
1950
1950
1951 before = full_text[:offset]
1951 before = full_text[:offset]
1952 cursor_line, cursor_column = position_to_cursor(full_text, offset)
1952 cursor_line, cursor_column = position_to_cursor(full_text, offset)
1953
1953
1954 matched_text, matches, matches_origin, jedi_matches = self._complete(
1954 matched_text, matches, matches_origin, jedi_matches = self._complete(
1955 full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column)
1955 full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column)
1956
1956
1957 iter_jm = iter(jedi_matches)
1957 iter_jm = iter(jedi_matches)
1958 if _timeout:
1958 if _timeout:
1959 for jm in iter_jm:
1959 for jm in iter_jm:
1960 try:
1960 try:
1961 type_ = jm.type
1961 type_ = jm.type
1962 except Exception:
1962 except Exception:
1963 if self.debug:
1963 if self.debug:
1964 print("Error in Jedi getting type of ", jm)
1964 print("Error in Jedi getting type of ", jm)
1965 type_ = None
1965 type_ = None
1966 delta = len(jm.name_with_symbols) - len(jm.complete)
1966 delta = len(jm.name_with_symbols) - len(jm.complete)
1967 if type_ == 'function':
1967 if type_ == 'function':
1968 signature = _make_signature(jm)
1968 signature = _make_signature(jm)
1969 else:
1969 else:
1970 signature = ''
1970 signature = ''
1971 yield Completion(start=offset - delta,
1971 yield Completion(start=offset - delta,
1972 end=offset,
1972 end=offset,
1973 text=jm.name_with_symbols,
1973 text=jm.name_with_symbols,
1974 type=type_,
1974 type=type_,
1975 signature=signature,
1975 signature=signature,
1976 _origin='jedi')
1976 _origin='jedi')
1977
1977
1978 if time.monotonic() > deadline:
1978 if time.monotonic() > deadline:
1979 break
1979 break
1980
1980
1981 for jm in iter_jm:
1981 for jm in iter_jm:
1982 delta = len(jm.name_with_symbols) - len(jm.complete)
1982 delta = len(jm.name_with_symbols) - len(jm.complete)
1983 yield Completion(start=offset - delta,
1983 yield Completion(start=offset - delta,
1984 end=offset,
1984 end=offset,
1985 text=jm.name_with_symbols,
1985 text=jm.name_with_symbols,
1986 type='<unknown>', # don't compute type for speed
1986 type='<unknown>', # don't compute type for speed
1987 _origin='jedi',
1987 _origin='jedi',
1988 signature='')
1988 signature='')
1989
1989
1990
1990
1991 start_offset = before.rfind(matched_text)
1991 start_offset = before.rfind(matched_text)
1992
1992
1993 # TODO:
1993 # TODO:
1994 # Suppress this, right now just for debug.
1994 # Suppress this, right now just for debug.
1995 if jedi_matches and matches and self.debug:
1995 if jedi_matches and matches and self.debug:
1996 yield Completion(start=start_offset, end=offset, text='--jedi/ipython--',
1996 yield Completion(start=start_offset, end=offset, text='--jedi/ipython--',
1997 _origin='debug', type='none', signature='')
1997 _origin='debug', type='none', signature='')
1998
1998
1999 # I'm unsure if this is always true, so let's assert and see if it
1999 # I'm unsure if this is always true, so let's assert and see if it
2000 # crash
2000 # crash
2001 assert before.endswith(matched_text)
2001 assert before.endswith(matched_text)
2002 for m, t in zip(matches, matches_origin):
2002 for m, t in zip(matches, matches_origin):
2003 yield Completion(start=start_offset, end=offset, text=m, _origin=t, signature='', type='<unknown>')
2003 yield Completion(start=start_offset, end=offset, text=m, _origin=t, signature='', type='<unknown>')
2004
2004
2005
2005
2006 def complete(self, text=None, line_buffer=None, cursor_pos=None) -> Tuple[str, Sequence[str]]:
2006 def complete(self, text=None, line_buffer=None, cursor_pos=None) -> Tuple[str, Sequence[str]]:
2007 """Find completions for the given text and line context.
2007 """Find completions for the given text and line context.
2008
2008
2009 Note that both the text and the line_buffer are optional, but at least
2009 Note that both the text and the line_buffer are optional, but at least
2010 one of them must be given.
2010 one of them must be given.
2011
2011
2012 Parameters
2012 Parameters
2013 ----------
2013 ----------
2014 text : string, optional
2014 text : string, optional
2015 Text to perform the completion on. If not given, the line buffer
2015 Text to perform the completion on. If not given, the line buffer
2016 is split using the instance's CompletionSplitter object.
2016 is split using the instance's CompletionSplitter object.
2017 line_buffer : string, optional
2017 line_buffer : string, optional
2018 If not given, the completer attempts to obtain the current line
2018 If not given, the completer attempts to obtain the current line
2019 buffer via readline. This keyword allows clients which are
2019 buffer via readline. This keyword allows clients which are
2020 requesting for text completions in non-readline contexts to inform
2020 requesting for text completions in non-readline contexts to inform
2021 the completer of the entire text.
2021 the completer of the entire text.
2022 cursor_pos : int, optional
2022 cursor_pos : int, optional
2023 Index of the cursor in the full line buffer. Should be provided by
2023 Index of the cursor in the full line buffer. Should be provided by
2024 remote frontends where kernel has no access to frontend state.
2024 remote frontends where kernel has no access to frontend state.
2025
2025
2026 Returns
2026 Returns
2027 -------
2027 -------
2028 Tuple of two items:
2028 Tuple of two items:
2029 text : str
2029 text : str
2030 Text that was actually used in the completion.
2030 Text that was actually used in the completion.
2031 matches : list
2031 matches : list
2032 A list of completion matches.
2032 A list of completion matches.
2033
2033
2034 Notes
2034 Notes
2035 -----
2035 -----
2036 This API is likely to be deprecated and replaced by
2036 This API is likely to be deprecated and replaced by
2037 :any:`IPCompleter.completions` in the future.
2037 :any:`IPCompleter.completions` in the future.
2038
2038
2039 """
2039 """
2040 warnings.warn('`Completer.complete` is pending deprecation since '
2040 warnings.warn('`Completer.complete` is pending deprecation since '
2041 'IPython 6.0 and will be replaced by `Completer.completions`.',
2041 'IPython 6.0 and will be replaced by `Completer.completions`.',
2042 PendingDeprecationWarning)
2042 PendingDeprecationWarning)
2043 # potential todo, FOLD the 3rd throw away argument of _complete
2043 # potential todo, FOLD the 3rd throw away argument of _complete
2044 # into the first 2 one.
2044 # into the first 2 one.
2045 return self._complete(line_buffer=line_buffer, cursor_pos=cursor_pos, text=text, cursor_line=0)[:2]
2045 return self._complete(line_buffer=line_buffer, cursor_pos=cursor_pos, text=text, cursor_line=0)[:2]
2046
2046
2047 def _complete(self, *, cursor_line, cursor_pos, line_buffer=None, text=None,
2047 def _complete(self, *, cursor_line, cursor_pos, line_buffer=None, text=None,
2048 full_text=None) -> _CompleteResult:
2048 full_text=None) -> _CompleteResult:
2049 """
2049 """
2050 Like complete but can also returns raw jedi completions as well as the
2050 Like complete but can also returns raw jedi completions as well as the
2051 origin of the completion text. This could (and should) be made much
2051 origin of the completion text. This could (and should) be made much
2052 cleaner but that will be simpler once we drop the old (and stateful)
2052 cleaner but that will be simpler once we drop the old (and stateful)
2053 :any:`complete` API.
2053 :any:`complete` API.
2054
2054
2055 With current provisional API, cursor_pos act both (depending on the
2055 With current provisional API, cursor_pos act both (depending on the
2056 caller) as the offset in the ``text`` or ``line_buffer``, or as the
2056 caller) as the offset in the ``text`` or ``line_buffer``, or as the
2057 ``column`` when passing multiline strings this could/should be renamed
2057 ``column`` when passing multiline strings this could/should be renamed
2058 but would add extra noise.
2058 but would add extra noise.
2059
2059
2060 Returns
2060 Returns
2061 -------
2061 -------
2062 A tuple of N elements which are (likely):
2062 A tuple of N elements which are (likely):
2063 matched_text: ? the text that the complete matched
2063 matched_text: ? the text that the complete matched
2064 matches: list of completions ?
2064 matches: list of completions ?
2065 matches_origin: ? list same lenght as matches, and where each completion came from
2065 matches_origin: ? list same length as matches, and where each completion came from
2066 jedi_matches: list of Jedi matches, have it's own structure.
2066 jedi_matches: list of Jedi matches, have it's own structure.
2067 """
2067 """
2068
2068
2069
2069
2070 # if the cursor position isn't given, the only sane assumption we can
2070 # if the cursor position isn't given, the only sane assumption we can
2071 # make is that it's at the end of the line (the common case)
2071 # make is that it's at the end of the line (the common case)
2072 if cursor_pos is None:
2072 if cursor_pos is None:
2073 cursor_pos = len(line_buffer) if text is None else len(text)
2073 cursor_pos = len(line_buffer) if text is None else len(text)
2074
2074
2075 if self.use_main_ns:
2075 if self.use_main_ns:
2076 self.namespace = __main__.__dict__
2076 self.namespace = __main__.__dict__
2077
2077
2078 # if text is either None or an empty string, rely on the line buffer
2078 # if text is either None or an empty string, rely on the line buffer
2079 if (not line_buffer) and full_text:
2079 if (not line_buffer) and full_text:
2080 line_buffer = full_text.split('\n')[cursor_line]
2080 line_buffer = full_text.split('\n')[cursor_line]
2081 if not text: # issue #11508: check line_buffer before calling split_line
2081 if not text: # issue #11508: check line_buffer before calling split_line
2082 text = self.splitter.split_line(line_buffer, cursor_pos) if line_buffer else ''
2082 text = self.splitter.split_line(line_buffer, cursor_pos) if line_buffer else ''
2083
2083
2084 if self.backslash_combining_completions:
2084 if self.backslash_combining_completions:
2085 # allow deactivation of these on windows.
2085 # allow deactivation of these on windows.
2086 base_text = text if not line_buffer else line_buffer[:cursor_pos]
2086 base_text = text if not line_buffer else line_buffer[:cursor_pos]
2087
2087
2088 for meth in (self.latex_matches,
2088 for meth in (self.latex_matches,
2089 self.unicode_name_matches,
2089 self.unicode_name_matches,
2090 back_latex_name_matches,
2090 back_latex_name_matches,
2091 back_unicode_name_matches,
2091 back_unicode_name_matches,
2092 self.fwd_unicode_match):
2092 self.fwd_unicode_match):
2093 name_text, name_matches = meth(base_text)
2093 name_text, name_matches = meth(base_text)
2094 if name_text:
2094 if name_text:
2095 return _CompleteResult(name_text, name_matches[:MATCHES_LIMIT], \
2095 return _CompleteResult(name_text, name_matches[:MATCHES_LIMIT], \
2096 [meth.__qualname__]*min(len(name_matches), MATCHES_LIMIT), ())
2096 [meth.__qualname__]*min(len(name_matches), MATCHES_LIMIT), ())
2097
2097
2098
2098
2099 # If no line buffer is given, assume the input text is all there was
2099 # If no line buffer is given, assume the input text is all there was
2100 if line_buffer is None:
2100 if line_buffer is None:
2101 line_buffer = text
2101 line_buffer = text
2102
2102
2103 self.line_buffer = line_buffer
2103 self.line_buffer = line_buffer
2104 self.text_until_cursor = self.line_buffer[:cursor_pos]
2104 self.text_until_cursor = self.line_buffer[:cursor_pos]
2105
2105
2106 # Do magic arg matches
2106 # Do magic arg matches
2107 for matcher in self.magic_arg_matchers:
2107 for matcher in self.magic_arg_matchers:
2108 matches = list(matcher(line_buffer))[:MATCHES_LIMIT]
2108 matches = list(matcher(line_buffer))[:MATCHES_LIMIT]
2109 if matches:
2109 if matches:
2110 origins = [matcher.__qualname__] * len(matches)
2110 origins = [matcher.__qualname__] * len(matches)
2111 return _CompleteResult(text, matches, origins, ())
2111 return _CompleteResult(text, matches, origins, ())
2112
2112
2113 # Start with a clean slate of completions
2113 # Start with a clean slate of completions
2114 matches = []
2114 matches = []
2115
2115
2116 # FIXME: we should extend our api to return a dict with completions for
2116 # FIXME: we should extend our api to return a dict with completions for
2117 # different types of objects. The rlcomplete() method could then
2117 # different types of objects. The rlcomplete() method could then
2118 # simply collapse the dict into a list for readline, but we'd have
2118 # simply collapse the dict into a list for readline, but we'd have
2119 # richer completion semantics in other environments.
2119 # richer completion semantics in other environments.
2120 completions:Iterable[Any] = []
2120 completions:Iterable[Any] = []
2121 if self.use_jedi:
2121 if self.use_jedi:
2122 if not full_text:
2122 if not full_text:
2123 full_text = line_buffer
2123 full_text = line_buffer
2124 completions = self._jedi_matches(
2124 completions = self._jedi_matches(
2125 cursor_pos, cursor_line, full_text)
2125 cursor_pos, cursor_line, full_text)
2126
2126
2127 if self.merge_completions:
2127 if self.merge_completions:
2128 matches = []
2128 matches = []
2129 for matcher in self.matchers:
2129 for matcher in self.matchers:
2130 try:
2130 try:
2131 matches.extend([(m, matcher.__qualname__)
2131 matches.extend([(m, matcher.__qualname__)
2132 for m in matcher(text)])
2132 for m in matcher(text)])
2133 except:
2133 except:
2134 # Show the ugly traceback if the matcher causes an
2134 # Show the ugly traceback if the matcher causes an
2135 # exception, but do NOT crash the kernel!
2135 # exception, but do NOT crash the kernel!
2136 sys.excepthook(*sys.exc_info())
2136 sys.excepthook(*sys.exc_info())
2137 else:
2137 else:
2138 for matcher in self.matchers:
2138 for matcher in self.matchers:
2139 matches = [(m, matcher.__qualname__)
2139 matches = [(m, matcher.__qualname__)
2140 for m in matcher(text)]
2140 for m in matcher(text)]
2141 if matches:
2141 if matches:
2142 break
2142 break
2143
2143
2144 seen = set()
2144 seen = set()
2145 filtered_matches = set()
2145 filtered_matches = set()
2146 for m in matches:
2146 for m in matches:
2147 t, c = m
2147 t, c = m
2148 if t not in seen:
2148 if t not in seen:
2149 filtered_matches.add(m)
2149 filtered_matches.add(m)
2150 seen.add(t)
2150 seen.add(t)
2151
2151
2152 _filtered_matches = sorted(filtered_matches, key=lambda x: completions_sorting_key(x[0]))
2152 _filtered_matches = sorted(filtered_matches, key=lambda x: completions_sorting_key(x[0]))
2153
2153
2154 custom_res = [(m, 'custom') for m in self.dispatch_custom_completer(text) or []]
2154 custom_res = [(m, 'custom') for m in self.dispatch_custom_completer(text) or []]
2155
2155
2156 _filtered_matches = custom_res or _filtered_matches
2156 _filtered_matches = custom_res or _filtered_matches
2157
2157
2158 _filtered_matches = _filtered_matches[:MATCHES_LIMIT]
2158 _filtered_matches = _filtered_matches[:MATCHES_LIMIT]
2159 _matches = [m[0] for m in _filtered_matches]
2159 _matches = [m[0] for m in _filtered_matches]
2160 origins = [m[1] for m in _filtered_matches]
2160 origins = [m[1] for m in _filtered_matches]
2161
2161
2162 self.matches = _matches
2162 self.matches = _matches
2163
2163
2164 return _CompleteResult(text, _matches, origins, completions)
2164 return _CompleteResult(text, _matches, origins, completions)
2165
2165
2166 def fwd_unicode_match(self, text:str) -> Tuple[str, Sequence[str]]:
2166 def fwd_unicode_match(self, text:str) -> Tuple[str, Sequence[str]]:
2167 """
2167 """
2168 Forward match a string starting with a backslash with a list of
2168 Forward match a string starting with a backslash with a list of
2169 potential Unicode completions.
2169 potential Unicode completions.
2170
2170
2171 Will compute list list of Unicode character names on first call and cache it.
2171 Will compute list list of Unicode character names on first call and cache it.
2172
2172
2173 Returns
2173 Returns
2174 -------
2174 -------
2175 At tuple with:
2175 At tuple with:
2176 - matched text (empty if no matches)
2176 - matched text (empty if no matches)
2177 - list of potential completions, empty tuple otherwise)
2177 - list of potential completions, empty tuple otherwise)
2178 """
2178 """
2179 # TODO: self.unicode_names is here a list we traverse each time with ~100k elements.
2179 # TODO: self.unicode_names is here a list we traverse each time with ~100k elements.
2180 # We could do a faster match using a Trie.
2180 # We could do a faster match using a Trie.
2181
2181
2182 # Using pygtrie the follwing seem to work:
2182 # Using pygtrie the following seem to work:
2183
2183
2184 # s = PrefixSet()
2184 # s = PrefixSet()
2185
2185
2186 # for c in range(0,0x10FFFF + 1):
2186 # for c in range(0,0x10FFFF + 1):
2187 # try:
2187 # try:
2188 # s.add(unicodedata.name(chr(c)))
2188 # s.add(unicodedata.name(chr(c)))
2189 # except ValueError:
2189 # except ValueError:
2190 # pass
2190 # pass
2191 # [''.join(k) for k in s.iter(prefix)]
2191 # [''.join(k) for k in s.iter(prefix)]
2192
2192
2193 # But need to be timed and adds an extra dependency.
2193 # But need to be timed and adds an extra dependency.
2194
2194
2195 slashpos = text.rfind('\\')
2195 slashpos = text.rfind('\\')
2196 # if text starts with slash
2196 # if text starts with slash
2197 if slashpos > -1:
2197 if slashpos > -1:
2198 # PERF: It's important that we don't access self._unicode_names
2198 # PERF: It's important that we don't access self._unicode_names
2199 # until we're inside this if-block. _unicode_names is lazily
2199 # until we're inside this if-block. _unicode_names is lazily
2200 # initialized, and it takes a user-noticeable amount of time to
2200 # initialized, and it takes a user-noticeable amount of time to
2201 # initialize it, so we don't want to initialize it unless we're
2201 # initialize it, so we don't want to initialize it unless we're
2202 # actually going to use it.
2202 # actually going to use it.
2203 s = text[slashpos+1:]
2203 s = text[slashpos+1:]
2204 candidates = [x for x in self.unicode_names if x.startswith(s)]
2204 candidates = [x for x in self.unicode_names if x.startswith(s)]
2205 if candidates:
2205 if candidates:
2206 return s, candidates
2206 return s, candidates
2207 else:
2207 else:
2208 return '', ()
2208 return '', ()
2209
2209
2210 # if text does not start with slash
2210 # if text does not start with slash
2211 else:
2211 else:
2212 return '', ()
2212 return '', ()
2213
2213
2214 @property
2214 @property
2215 def unicode_names(self) -> List[str]:
2215 def unicode_names(self) -> List[str]:
2216 """List of names of unicode code points that can be completed.
2216 """List of names of unicode code points that can be completed.
2217
2217
2218 The list is lazily initialized on first access.
2218 The list is lazily initialized on first access.
2219 """
2219 """
2220 if self._unicode_names is None:
2220 if self._unicode_names is None:
2221 names = []
2221 names = []
2222 for c in range(0,0x10FFFF + 1):
2222 for c in range(0,0x10FFFF + 1):
2223 try:
2223 try:
2224 names.append(unicodedata.name(chr(c)))
2224 names.append(unicodedata.name(chr(c)))
2225 except ValueError:
2225 except ValueError:
2226 pass
2226 pass
2227 self._unicode_names = _unicode_name_compute(_UNICODE_RANGES)
2227 self._unicode_names = _unicode_name_compute(_UNICODE_RANGES)
2228
2228
2229 return self._unicode_names
2229 return self._unicode_names
2230
2230
2231 def _unicode_name_compute(ranges:List[Tuple[int,int]]) -> List[str]:
2231 def _unicode_name_compute(ranges:List[Tuple[int,int]]) -> List[str]:
2232 names = []
2232 names = []
2233 for start,stop in ranges:
2233 for start,stop in ranges:
2234 for c in range(start, stop) :
2234 for c in range(start, stop) :
2235 try:
2235 try:
2236 names.append(unicodedata.name(chr(c)))
2236 names.append(unicodedata.name(chr(c)))
2237 except ValueError:
2237 except ValueError:
2238 pass
2238 pass
2239 return names
2239 return names
@@ -1,1108 +1,1108 b''
1 # -*- coding: utf-8 -*-
1 # -*- coding: utf-8 -*-
2 """
2 """
3 Pdb debugger class.
3 Pdb debugger class.
4
4
5
5
6 This is an extension to PDB which adds a number of new features.
6 This is an extension to PDB which adds a number of new features.
7 Note that there is also the `IPython.terminal.debugger` class which provides UI
7 Note that there is also the `IPython.terminal.debugger` class which provides UI
8 improvements.
8 improvements.
9
9
10 We also strongly recommend to use this via the `ipdb` package, which provides
10 We also strongly recommend to use this via the `ipdb` package, which provides
11 extra configuration options.
11 extra configuration options.
12
12
13 Among other things, this subclass of PDB:
13 Among other things, this subclass of PDB:
14 - supports many IPython magics like pdef/psource
14 - supports many IPython magics like pdef/psource
15 - hide frames in tracebacks based on `__tracebackhide__`
15 - hide frames in tracebacks based on `__tracebackhide__`
16 - allows to skip frames based on `__debuggerskip__`
16 - allows to skip frames based on `__debuggerskip__`
17
17
18 The skipping and hiding frames are configurable via the `skip_predicates`
18 The skipping and hiding frames are configurable via the `skip_predicates`
19 command.
19 command.
20
20
21 By default, frames from readonly files will be hidden, frames containing
21 By default, frames from readonly files will be hidden, frames containing
22 ``__tracebackhide__=True`` will be hidden.
22 ``__tracebackhide__=True`` will be hidden.
23
23
24 Frames containing ``__debuggerskip__`` will be stepped over, frames who's parent
24 Frames containing ``__debuggerskip__`` will be stepped over, frames who's parent
25 frames value of ``__debuggerskip__`` is ``True`` will be skipped.
25 frames value of ``__debuggerskip__`` is ``True`` will be skipped.
26
26
27 >>> def helpers_helper():
27 >>> def helpers_helper():
28 ... pass
28 ... pass
29 ...
29 ...
30 ... def helper_1():
30 ... def helper_1():
31 ... print("don't step in me")
31 ... print("don't step in me")
32 ... helpers_helpers() # will be stepped over unless breakpoint set.
32 ... helpers_helpers() # will be stepped over unless breakpoint set.
33 ...
33 ...
34 ...
34 ...
35 ... def helper_2():
35 ... def helper_2():
36 ... print("in me neither")
36 ... print("in me neither")
37 ...
37 ...
38
38
39 One can define a decorator that wraps a function between the two helpers:
39 One can define a decorator that wraps a function between the two helpers:
40
40
41 >>> def pdb_skipped_decorator(function):
41 >>> def pdb_skipped_decorator(function):
42 ...
42 ...
43 ...
43 ...
44 ... def wrapped_fn(*args, **kwargs):
44 ... def wrapped_fn(*args, **kwargs):
45 ... __debuggerskip__ = True
45 ... __debuggerskip__ = True
46 ... helper_1()
46 ... helper_1()
47 ... __debuggerskip__ = False
47 ... __debuggerskip__ = False
48 ... result = function(*args, **kwargs)
48 ... result = function(*args, **kwargs)
49 ... __debuggerskip__ = True
49 ... __debuggerskip__ = True
50 ... helper_2()
50 ... helper_2()
51 ... # setting __debuggerskip__ to False again is not necessary
51 ... # setting __debuggerskip__ to False again is not necessary
52 ... return result
52 ... return result
53 ...
53 ...
54 ... return wrapped_fn
54 ... return wrapped_fn
55
55
56 When decorating a function, ipdb will directly step into ``bar()`` by
56 When decorating a function, ipdb will directly step into ``bar()`` by
57 default:
57 default:
58
58
59 >>> @foo_decorator
59 >>> @foo_decorator
60 ... def bar(x, y):
60 ... def bar(x, y):
61 ... return x * y
61 ... return x * y
62
62
63
63
64 You can toggle the behavior with
64 You can toggle the behavior with
65
65
66 ipdb> skip_predicates debuggerskip false
66 ipdb> skip_predicates debuggerskip false
67
67
68 or configure it in your ``.pdbrc``
68 or configure it in your ``.pdbrc``
69
69
70
70
71
71
72 Licencse
72 License
73 --------
73 -------
74
74
75 Modified from the standard pdb.Pdb class to avoid including readline, so that
75 Modified from the standard pdb.Pdb class to avoid including readline, so that
76 the command line completion of other programs which include this isn't
76 the command line completion of other programs which include this isn't
77 damaged.
77 damaged.
78
78
79 In the future, this class will be expanded with improvements over the standard
79 In the future, this class will be expanded with improvements over the standard
80 pdb.
80 pdb.
81
81
82 The original code in this file is mainly lifted out of cmd.py in Python 2.2,
82 The original code in this file is mainly lifted out of cmd.py in Python 2.2,
83 with minor changes. Licensing should therefore be under the standard Python
83 with minor changes. Licensing should therefore be under the standard Python
84 terms. For details on the PSF (Python Software Foundation) standard license,
84 terms. For details on the PSF (Python Software Foundation) standard license,
85 see:
85 see:
86
86
87 https://docs.python.org/2/license.html
87 https://docs.python.org/2/license.html
88
88
89
89
90 All the changes since then are under the same license as IPython.
90 All the changes since then are under the same license as IPython.
91
91
92 """
92 """
93
93
94 #*****************************************************************************
94 #*****************************************************************************
95 #
95 #
96 # This file is licensed under the PSF license.
96 # This file is licensed under the PSF license.
97 #
97 #
98 # Copyright (C) 2001 Python Software Foundation, www.python.org
98 # Copyright (C) 2001 Python Software Foundation, www.python.org
99 # Copyright (C) 2005-2006 Fernando Perez. <fperez@colorado.edu>
99 # Copyright (C) 2005-2006 Fernando Perez. <fperez@colorado.edu>
100 #
100 #
101 #
101 #
102 #*****************************************************************************
102 #*****************************************************************************
103
103
104 import bdb
104 import bdb
105 import functools
105 import functools
106 import inspect
106 import inspect
107 import linecache
107 import linecache
108 import sys
108 import sys
109 import warnings
109 import warnings
110 import re
110 import re
111 import os
111 import os
112
112
113 from IPython import get_ipython
113 from IPython import get_ipython
114 from IPython.utils import PyColorize
114 from IPython.utils import PyColorize
115 from IPython.utils import coloransi, py3compat
115 from IPython.utils import coloransi, py3compat
116 from IPython.core.excolors import exception_colors
116 from IPython.core.excolors import exception_colors
117 from IPython.testing.skipdoctest import skip_doctest
117 from IPython.testing.skipdoctest import skip_doctest
118
118
119 # skip module docstests
119 # skip module docstests
120 __skip_doctest__ = True
120 __skip_doctest__ = True
121
121
122 prompt = 'ipdb> '
122 prompt = 'ipdb> '
123
123
124 # We have to check this directly from sys.argv, config struct not yet available
124 # We have to check this directly from sys.argv, config struct not yet available
125 from pdb import Pdb as OldPdb
125 from pdb import Pdb as OldPdb
126
126
127 # Allow the set_trace code to operate outside of an ipython instance, even if
127 # Allow the set_trace code to operate outside of an ipython instance, even if
128 # it does so with some limitations. The rest of this support is implemented in
128 # it does so with some limitations. The rest of this support is implemented in
129 # the Tracer constructor.
129 # the Tracer constructor.
130
130
131 DEBUGGERSKIP = "__debuggerskip__"
131 DEBUGGERSKIP = "__debuggerskip__"
132
132
133
133
134 def make_arrow(pad):
134 def make_arrow(pad):
135 """generate the leading arrow in front of traceback or debugger"""
135 """generate the leading arrow in front of traceback or debugger"""
136 if pad >= 2:
136 if pad >= 2:
137 return '-'*(pad-2) + '> '
137 return '-'*(pad-2) + '> '
138 elif pad == 1:
138 elif pad == 1:
139 return '>'
139 return '>'
140 return ''
140 return ''
141
141
142
142
143 def BdbQuit_excepthook(et, ev, tb, excepthook=None):
143 def BdbQuit_excepthook(et, ev, tb, excepthook=None):
144 """Exception hook which handles `BdbQuit` exceptions.
144 """Exception hook which handles `BdbQuit` exceptions.
145
145
146 All other exceptions are processed using the `excepthook`
146 All other exceptions are processed using the `excepthook`
147 parameter.
147 parameter.
148 """
148 """
149 warnings.warn("`BdbQuit_excepthook` is deprecated since version 5.1",
149 warnings.warn("`BdbQuit_excepthook` is deprecated since version 5.1",
150 DeprecationWarning, stacklevel=2)
150 DeprecationWarning, stacklevel=2)
151 if et == bdb.BdbQuit:
151 if et == bdb.BdbQuit:
152 print('Exiting Debugger.')
152 print('Exiting Debugger.')
153 elif excepthook is not None:
153 elif excepthook is not None:
154 excepthook(et, ev, tb)
154 excepthook(et, ev, tb)
155 else:
155 else:
156 # Backwards compatibility. Raise deprecation warning?
156 # Backwards compatibility. Raise deprecation warning?
157 BdbQuit_excepthook.excepthook_ori(et, ev, tb)
157 BdbQuit_excepthook.excepthook_ori(et, ev, tb)
158
158
159
159
160 def BdbQuit_IPython_excepthook(self, et, ev, tb, tb_offset=None):
160 def BdbQuit_IPython_excepthook(self, et, ev, tb, tb_offset=None):
161 warnings.warn(
161 warnings.warn(
162 "`BdbQuit_IPython_excepthook` is deprecated since version 5.1",
162 "`BdbQuit_IPython_excepthook` is deprecated since version 5.1",
163 DeprecationWarning, stacklevel=2)
163 DeprecationWarning, stacklevel=2)
164 print('Exiting Debugger.')
164 print('Exiting Debugger.')
165
165
166
166
167 class Tracer(object):
167 class Tracer(object):
168 """
168 """
169 DEPRECATED
169 DEPRECATED
170
170
171 Class for local debugging, similar to pdb.set_trace.
171 Class for local debugging, similar to pdb.set_trace.
172
172
173 Instances of this class, when called, behave like pdb.set_trace, but
173 Instances of this class, when called, behave like pdb.set_trace, but
174 providing IPython's enhanced capabilities.
174 providing IPython's enhanced capabilities.
175
175
176 This is implemented as a class which must be initialized in your own code
176 This is implemented as a class which must be initialized in your own code
177 and not as a standalone function because we need to detect at runtime
177 and not as a standalone function because we need to detect at runtime
178 whether IPython is already active or not. That detection is done in the
178 whether IPython is already active or not. That detection is done in the
179 constructor, ensuring that this code plays nicely with a running IPython,
179 constructor, ensuring that this code plays nicely with a running IPython,
180 while functioning acceptably (though with limitations) if outside of it.
180 while functioning acceptably (though with limitations) if outside of it.
181 """
181 """
182
182
183 @skip_doctest
183 @skip_doctest
184 def __init__(self, colors=None):
184 def __init__(self, colors=None):
185 """
185 """
186 DEPRECATED
186 DEPRECATED
187
187
188 Create a local debugger instance.
188 Create a local debugger instance.
189
189
190 Parameters
190 Parameters
191 ----------
191 ----------
192 colors : str, optional
192 colors : str, optional
193 The name of the color scheme to use, it must be one of IPython's
193 The name of the color scheme to use, it must be one of IPython's
194 valid color schemes. If not given, the function will default to
194 valid color schemes. If not given, the function will default to
195 the current IPython scheme when running inside IPython, and to
195 the current IPython scheme when running inside IPython, and to
196 'NoColor' otherwise.
196 'NoColor' otherwise.
197
197
198 Examples
198 Examples
199 --------
199 --------
200 ::
200 ::
201
201
202 from IPython.core.debugger import Tracer; debug_here = Tracer()
202 from IPython.core.debugger import Tracer; debug_here = Tracer()
203
203
204 Later in your code::
204 Later in your code::
205
205
206 debug_here() # -> will open up the debugger at that point.
206 debug_here() # -> will open up the debugger at that point.
207
207
208 Once the debugger activates, you can use all of its regular commands to
208 Once the debugger activates, you can use all of its regular commands to
209 step through code, set breakpoints, etc. See the pdb documentation
209 step through code, set breakpoints, etc. See the pdb documentation
210 from the Python standard library for usage details.
210 from the Python standard library for usage details.
211 """
211 """
212 warnings.warn("`Tracer` is deprecated since version 5.1, directly use "
212 warnings.warn("`Tracer` is deprecated since version 5.1, directly use "
213 "`IPython.core.debugger.Pdb.set_trace()`",
213 "`IPython.core.debugger.Pdb.set_trace()`",
214 DeprecationWarning, stacklevel=2)
214 DeprecationWarning, stacklevel=2)
215
215
216 ip = get_ipython()
216 ip = get_ipython()
217 if ip is None:
217 if ip is None:
218 # Outside of ipython, we set our own exception hook manually
218 # Outside of ipython, we set our own exception hook manually
219 sys.excepthook = functools.partial(BdbQuit_excepthook,
219 sys.excepthook = functools.partial(BdbQuit_excepthook,
220 excepthook=sys.excepthook)
220 excepthook=sys.excepthook)
221 def_colors = 'NoColor'
221 def_colors = 'NoColor'
222 else:
222 else:
223 # In ipython, we use its custom exception handler mechanism
223 # In ipython, we use its custom exception handler mechanism
224 def_colors = ip.colors
224 def_colors = ip.colors
225 ip.set_custom_exc((bdb.BdbQuit,), BdbQuit_IPython_excepthook)
225 ip.set_custom_exc((bdb.BdbQuit,), BdbQuit_IPython_excepthook)
226
226
227 if colors is None:
227 if colors is None:
228 colors = def_colors
228 colors = def_colors
229
229
230 # The stdlib debugger internally uses a modified repr from the `repr`
230 # The stdlib debugger internally uses a modified repr from the `repr`
231 # module, that limits the length of printed strings to a hardcoded
231 # module, that limits the length of printed strings to a hardcoded
232 # limit of 30 characters. That much trimming is too aggressive, let's
232 # limit of 30 characters. That much trimming is too aggressive, let's
233 # at least raise that limit to 80 chars, which should be enough for
233 # at least raise that limit to 80 chars, which should be enough for
234 # most interactive uses.
234 # most interactive uses.
235 try:
235 try:
236 from reprlib import aRepr
236 from reprlib import aRepr
237 aRepr.maxstring = 80
237 aRepr.maxstring = 80
238 except:
238 except:
239 # This is only a user-facing convenience, so any error we encounter
239 # This is only a user-facing convenience, so any error we encounter
240 # here can be warned about but can be otherwise ignored. These
240 # here can be warned about but can be otherwise ignored. These
241 # printouts will tell us about problems if this API changes
241 # printouts will tell us about problems if this API changes
242 import traceback
242 import traceback
243 traceback.print_exc()
243 traceback.print_exc()
244
244
245 self.debugger = Pdb(colors)
245 self.debugger = Pdb(colors)
246
246
247 def __call__(self):
247 def __call__(self):
248 """Starts an interactive debugger at the point where called.
248 """Starts an interactive debugger at the point where called.
249
249
250 This is similar to the pdb.set_trace() function from the std lib, but
250 This is similar to the pdb.set_trace() function from the std lib, but
251 using IPython's enhanced debugger."""
251 using IPython's enhanced debugger."""
252
252
253 self.debugger.set_trace(sys._getframe().f_back)
253 self.debugger.set_trace(sys._getframe().f_back)
254
254
255
255
256 RGX_EXTRA_INDENT = re.compile(r'(?<=\n)\s+')
256 RGX_EXTRA_INDENT = re.compile(r'(?<=\n)\s+')
257
257
258
258
259 def strip_indentation(multiline_string):
259 def strip_indentation(multiline_string):
260 return RGX_EXTRA_INDENT.sub('', multiline_string)
260 return RGX_EXTRA_INDENT.sub('', multiline_string)
261
261
262
262
263 def decorate_fn_with_doc(new_fn, old_fn, additional_text=""):
263 def decorate_fn_with_doc(new_fn, old_fn, additional_text=""):
264 """Make new_fn have old_fn's doc string. This is particularly useful
264 """Make new_fn have old_fn's doc string. This is particularly useful
265 for the ``do_...`` commands that hook into the help system.
265 for the ``do_...`` commands that hook into the help system.
266 Adapted from from a comp.lang.python posting
266 Adapted from from a comp.lang.python posting
267 by Duncan Booth."""
267 by Duncan Booth."""
268 def wrapper(*args, **kw):
268 def wrapper(*args, **kw):
269 return new_fn(*args, **kw)
269 return new_fn(*args, **kw)
270 if old_fn.__doc__:
270 if old_fn.__doc__:
271 wrapper.__doc__ = strip_indentation(old_fn.__doc__) + additional_text
271 wrapper.__doc__ = strip_indentation(old_fn.__doc__) + additional_text
272 return wrapper
272 return wrapper
273
273
274
274
275 class Pdb(OldPdb):
275 class Pdb(OldPdb):
276 """Modified Pdb class, does not load readline.
276 """Modified Pdb class, does not load readline.
277
277
278 for a standalone version that uses prompt_toolkit, see
278 for a standalone version that uses prompt_toolkit, see
279 `IPython.terminal.debugger.TerminalPdb` and
279 `IPython.terminal.debugger.TerminalPdb` and
280 `IPython.terminal.debugger.set_trace()`
280 `IPython.terminal.debugger.set_trace()`
281
281
282
282
283 This debugger can hide and skip frames that are tagged according to some predicates.
283 This debugger can hide and skip frames that are tagged according to some predicates.
284 See the `skip_predicates` commands.
284 See the `skip_predicates` commands.
285
285
286 """
286 """
287
287
288 default_predicates = {
288 default_predicates = {
289 "tbhide": True,
289 "tbhide": True,
290 "readonly": False,
290 "readonly": False,
291 "ipython_internal": True,
291 "ipython_internal": True,
292 "debuggerskip": True,
292 "debuggerskip": True,
293 }
293 }
294
294
295 def __init__(self, color_scheme=None, completekey=None,
295 def __init__(self, color_scheme=None, completekey=None,
296 stdin=None, stdout=None, context=5, **kwargs):
296 stdin=None, stdout=None, context=5, **kwargs):
297 """Create a new IPython debugger.
297 """Create a new IPython debugger.
298
298
299 Parameters
299 Parameters
300 ----------
300 ----------
301 color_scheme : default None
301 color_scheme : default None
302 Deprecated, do not use.
302 Deprecated, do not use.
303 completekey : default None
303 completekey : default None
304 Passed to pdb.Pdb.
304 Passed to pdb.Pdb.
305 stdin : default None
305 stdin : default None
306 Passed to pdb.Pdb.
306 Passed to pdb.Pdb.
307 stdout : default None
307 stdout : default None
308 Passed to pdb.Pdb.
308 Passed to pdb.Pdb.
309 context : int
309 context : int
310 Number of lines of source code context to show when
310 Number of lines of source code context to show when
311 displaying stacktrace information.
311 displaying stacktrace information.
312 **kwargs
312 **kwargs
313 Passed to pdb.Pdb.
313 Passed to pdb.Pdb.
314
314
315 Notes
315 Notes
316 -----
316 -----
317 The possibilities are python version dependent, see the python
317 The possibilities are python version dependent, see the python
318 docs for more info.
318 docs for more info.
319 """
319 """
320
320
321 # Parent constructor:
321 # Parent constructor:
322 try:
322 try:
323 self.context = int(context)
323 self.context = int(context)
324 if self.context <= 0:
324 if self.context <= 0:
325 raise ValueError("Context must be a positive integer")
325 raise ValueError("Context must be a positive integer")
326 except (TypeError, ValueError) as e:
326 except (TypeError, ValueError) as e:
327 raise ValueError("Context must be a positive integer") from e
327 raise ValueError("Context must be a positive integer") from e
328
328
329 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
329 # `kwargs` ensures full compatibility with stdlib's `pdb.Pdb`.
330 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
330 OldPdb.__init__(self, completekey, stdin, stdout, **kwargs)
331
331
332 # IPython changes...
332 # IPython changes...
333 self.shell = get_ipython()
333 self.shell = get_ipython()
334
334
335 if self.shell is None:
335 if self.shell is None:
336 save_main = sys.modules['__main__']
336 save_main = sys.modules['__main__']
337 # No IPython instance running, we must create one
337 # No IPython instance running, we must create one
338 from IPython.terminal.interactiveshell import \
338 from IPython.terminal.interactiveshell import \
339 TerminalInteractiveShell
339 TerminalInteractiveShell
340 self.shell = TerminalInteractiveShell.instance()
340 self.shell = TerminalInteractiveShell.instance()
341 # needed by any code which calls __import__("__main__") after
341 # needed by any code which calls __import__("__main__") after
342 # the debugger was entered. See also #9941.
342 # the debugger was entered. See also #9941.
343 sys.modules["__main__"] = save_main
343 sys.modules["__main__"] = save_main
344
344
345 if color_scheme is not None:
345 if color_scheme is not None:
346 warnings.warn(
346 warnings.warn(
347 "The `color_scheme` argument is deprecated since version 5.1",
347 "The `color_scheme` argument is deprecated since version 5.1",
348 DeprecationWarning, stacklevel=2)
348 DeprecationWarning, stacklevel=2)
349 else:
349 else:
350 color_scheme = self.shell.colors
350 color_scheme = self.shell.colors
351
351
352 self.aliases = {}
352 self.aliases = {}
353
353
354 # Create color table: we copy the default one from the traceback
354 # Create color table: we copy the default one from the traceback
355 # module and add a few attributes needed for debugging
355 # module and add a few attributes needed for debugging
356 self.color_scheme_table = exception_colors()
356 self.color_scheme_table = exception_colors()
357
357
358 # shorthands
358 # shorthands
359 C = coloransi.TermColors
359 C = coloransi.TermColors
360 cst = self.color_scheme_table
360 cst = self.color_scheme_table
361
361
362 cst['NoColor'].colors.prompt = C.NoColor
362 cst['NoColor'].colors.prompt = C.NoColor
363 cst['NoColor'].colors.breakpoint_enabled = C.NoColor
363 cst['NoColor'].colors.breakpoint_enabled = C.NoColor
364 cst['NoColor'].colors.breakpoint_disabled = C.NoColor
364 cst['NoColor'].colors.breakpoint_disabled = C.NoColor
365
365
366 cst['Linux'].colors.prompt = C.Green
366 cst['Linux'].colors.prompt = C.Green
367 cst['Linux'].colors.breakpoint_enabled = C.LightRed
367 cst['Linux'].colors.breakpoint_enabled = C.LightRed
368 cst['Linux'].colors.breakpoint_disabled = C.Red
368 cst['Linux'].colors.breakpoint_disabled = C.Red
369
369
370 cst['LightBG'].colors.prompt = C.Blue
370 cst['LightBG'].colors.prompt = C.Blue
371 cst['LightBG'].colors.breakpoint_enabled = C.LightRed
371 cst['LightBG'].colors.breakpoint_enabled = C.LightRed
372 cst['LightBG'].colors.breakpoint_disabled = C.Red
372 cst['LightBG'].colors.breakpoint_disabled = C.Red
373
373
374 cst['Neutral'].colors.prompt = C.Blue
374 cst['Neutral'].colors.prompt = C.Blue
375 cst['Neutral'].colors.breakpoint_enabled = C.LightRed
375 cst['Neutral'].colors.breakpoint_enabled = C.LightRed
376 cst['Neutral'].colors.breakpoint_disabled = C.Red
376 cst['Neutral'].colors.breakpoint_disabled = C.Red
377
377
378 # Add a python parser so we can syntax highlight source while
378 # Add a python parser so we can syntax highlight source while
379 # debugging.
379 # debugging.
380 self.parser = PyColorize.Parser(style=color_scheme)
380 self.parser = PyColorize.Parser(style=color_scheme)
381 self.set_colors(color_scheme)
381 self.set_colors(color_scheme)
382
382
383 # Set the prompt - the default prompt is '(Pdb)'
383 # Set the prompt - the default prompt is '(Pdb)'
384 self.prompt = prompt
384 self.prompt = prompt
385 self.skip_hidden = True
385 self.skip_hidden = True
386 self.report_skipped = True
386 self.report_skipped = True
387
387
388 # list of predicates we use to skip frames
388 # list of predicates we use to skip frames
389 self._predicates = self.default_predicates
389 self._predicates = self.default_predicates
390
390
391 #
391 #
392 def set_colors(self, scheme):
392 def set_colors(self, scheme):
393 """Shorthand access to the color table scheme selector method."""
393 """Shorthand access to the color table scheme selector method."""
394 self.color_scheme_table.set_active_scheme(scheme)
394 self.color_scheme_table.set_active_scheme(scheme)
395 self.parser.style = scheme
395 self.parser.style = scheme
396
396
397 def set_trace(self, frame=None):
397 def set_trace(self, frame=None):
398 if frame is None:
398 if frame is None:
399 frame = sys._getframe().f_back
399 frame = sys._getframe().f_back
400 self.initial_frame = frame
400 self.initial_frame = frame
401 return super().set_trace(frame)
401 return super().set_trace(frame)
402
402
403 def _hidden_predicate(self, frame):
403 def _hidden_predicate(self, frame):
404 """
404 """
405 Given a frame return whether it it should be hidden or not by IPython.
405 Given a frame return whether it it should be hidden or not by IPython.
406 """
406 """
407
407
408 if self._predicates["readonly"]:
408 if self._predicates["readonly"]:
409 fname = frame.f_code.co_filename
409 fname = frame.f_code.co_filename
410 # we need to check for file existence and interactively define
410 # we need to check for file existence and interactively define
411 # function would otherwise appear as RO.
411 # function would otherwise appear as RO.
412 if os.path.isfile(fname) and not os.access(fname, os.W_OK):
412 if os.path.isfile(fname) and not os.access(fname, os.W_OK):
413 return True
413 return True
414
414
415 if self._predicates["tbhide"]:
415 if self._predicates["tbhide"]:
416 if frame in (self.curframe, getattr(self, "initial_frame", None)):
416 if frame in (self.curframe, getattr(self, "initial_frame", None)):
417 return False
417 return False
418 else:
418 else:
419 return self._get_frame_locals(frame).get("__tracebackhide__", False)
419 return self._get_frame_locals(frame).get("__tracebackhide__", False)
420
420
421 return False
421 return False
422
422
423 def hidden_frames(self, stack):
423 def hidden_frames(self, stack):
424 """
424 """
425 Given an index in the stack return whether it should be skipped.
425 Given an index in the stack return whether it should be skipped.
426
426
427 This is used in up/down and where to skip frames.
427 This is used in up/down and where to skip frames.
428 """
428 """
429 # The f_locals dictionary is updated from the actual frame
429 # The f_locals dictionary is updated from the actual frame
430 # locals whenever the .f_locals accessor is called, so we
430 # locals whenever the .f_locals accessor is called, so we
431 # avoid calling it here to preserve self.curframe_locals.
431 # avoid calling it here to preserve self.curframe_locals.
432 # Futhermore, there is no good reason to hide the current frame.
432 # Furthermore, there is no good reason to hide the current frame.
433 ip_hide = [self._hidden_predicate(s[0]) for s in stack]
433 ip_hide = [self._hidden_predicate(s[0]) for s in stack]
434 ip_start = [i for i, s in enumerate(ip_hide) if s == "__ipython_bottom__"]
434 ip_start = [i for i, s in enumerate(ip_hide) if s == "__ipython_bottom__"]
435 if ip_start and self._predicates["ipython_internal"]:
435 if ip_start and self._predicates["ipython_internal"]:
436 ip_hide = [h if i > ip_start[0] else True for (i, h) in enumerate(ip_hide)]
436 ip_hide = [h if i > ip_start[0] else True for (i, h) in enumerate(ip_hide)]
437 return ip_hide
437 return ip_hide
438
438
439 def interaction(self, frame, traceback):
439 def interaction(self, frame, traceback):
440 try:
440 try:
441 OldPdb.interaction(self, frame, traceback)
441 OldPdb.interaction(self, frame, traceback)
442 except KeyboardInterrupt:
442 except KeyboardInterrupt:
443 self.stdout.write("\n" + self.shell.get_exception_only())
443 self.stdout.write("\n" + self.shell.get_exception_only())
444
444
445 def precmd(self, line):
445 def precmd(self, line):
446 """Perform useful escapes on the command before it is executed."""
446 """Perform useful escapes on the command before it is executed."""
447
447
448 if line.endswith("??"):
448 if line.endswith("??"):
449 line = "pinfo2 " + line[:-2]
449 line = "pinfo2 " + line[:-2]
450 elif line.endswith("?"):
450 elif line.endswith("?"):
451 line = "pinfo " + line[:-1]
451 line = "pinfo " + line[:-1]
452
452
453 line = super().precmd(line)
453 line = super().precmd(line)
454
454
455 return line
455 return line
456
456
457 def new_do_frame(self, arg):
457 def new_do_frame(self, arg):
458 OldPdb.do_frame(self, arg)
458 OldPdb.do_frame(self, arg)
459
459
460 def new_do_quit(self, arg):
460 def new_do_quit(self, arg):
461
461
462 if hasattr(self, 'old_all_completions'):
462 if hasattr(self, 'old_all_completions'):
463 self.shell.Completer.all_completions = self.old_all_completions
463 self.shell.Completer.all_completions = self.old_all_completions
464
464
465 return OldPdb.do_quit(self, arg)
465 return OldPdb.do_quit(self, arg)
466
466
467 do_q = do_quit = decorate_fn_with_doc(new_do_quit, OldPdb.do_quit)
467 do_q = do_quit = decorate_fn_with_doc(new_do_quit, OldPdb.do_quit)
468
468
469 def new_do_restart(self, arg):
469 def new_do_restart(self, arg):
470 """Restart command. In the context of ipython this is exactly the same
470 """Restart command. In the context of ipython this is exactly the same
471 thing as 'quit'."""
471 thing as 'quit'."""
472 self.msg("Restart doesn't make sense here. Using 'quit' instead.")
472 self.msg("Restart doesn't make sense here. Using 'quit' instead.")
473 return self.do_quit(arg)
473 return self.do_quit(arg)
474
474
475 def print_stack_trace(self, context=None):
475 def print_stack_trace(self, context=None):
476 Colors = self.color_scheme_table.active_colors
476 Colors = self.color_scheme_table.active_colors
477 ColorsNormal = Colors.Normal
477 ColorsNormal = Colors.Normal
478 if context is None:
478 if context is None:
479 context = self.context
479 context = self.context
480 try:
480 try:
481 context = int(context)
481 context = int(context)
482 if context <= 0:
482 if context <= 0:
483 raise ValueError("Context must be a positive integer")
483 raise ValueError("Context must be a positive integer")
484 except (TypeError, ValueError) as e:
484 except (TypeError, ValueError) as e:
485 raise ValueError("Context must be a positive integer") from e
485 raise ValueError("Context must be a positive integer") from e
486 try:
486 try:
487 skipped = 0
487 skipped = 0
488 for hidden, frame_lineno in zip(self.hidden_frames(self.stack), self.stack):
488 for hidden, frame_lineno in zip(self.hidden_frames(self.stack), self.stack):
489 if hidden and self.skip_hidden:
489 if hidden and self.skip_hidden:
490 skipped += 1
490 skipped += 1
491 continue
491 continue
492 if skipped:
492 if skipped:
493 print(
493 print(
494 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
494 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
495 )
495 )
496 skipped = 0
496 skipped = 0
497 self.print_stack_entry(frame_lineno, context=context)
497 self.print_stack_entry(frame_lineno, context=context)
498 if skipped:
498 if skipped:
499 print(
499 print(
500 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
500 f"{Colors.excName} [... skipping {skipped} hidden frame(s)]{ColorsNormal}\n"
501 )
501 )
502 except KeyboardInterrupt:
502 except KeyboardInterrupt:
503 pass
503 pass
504
504
505 def print_stack_entry(self, frame_lineno, prompt_prefix='\n-> ',
505 def print_stack_entry(self, frame_lineno, prompt_prefix='\n-> ',
506 context=None):
506 context=None):
507 if context is None:
507 if context is None:
508 context = self.context
508 context = self.context
509 try:
509 try:
510 context = int(context)
510 context = int(context)
511 if context <= 0:
511 if context <= 0:
512 raise ValueError("Context must be a positive integer")
512 raise ValueError("Context must be a positive integer")
513 except (TypeError, ValueError) as e:
513 except (TypeError, ValueError) as e:
514 raise ValueError("Context must be a positive integer") from e
514 raise ValueError("Context must be a positive integer") from e
515 print(self.format_stack_entry(frame_lineno, '', context), file=self.stdout)
515 print(self.format_stack_entry(frame_lineno, '', context), file=self.stdout)
516
516
517 # vds: >>
517 # vds: >>
518 frame, lineno = frame_lineno
518 frame, lineno = frame_lineno
519 filename = frame.f_code.co_filename
519 filename = frame.f_code.co_filename
520 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
520 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
521 # vds: <<
521 # vds: <<
522
522
523 def _get_frame_locals(self, frame):
523 def _get_frame_locals(self, frame):
524 """ "
524 """ "
525 Acessing f_local of current frame reset the namespace, so we want to avoid
525 Accessing f_local of current frame reset the namespace, so we want to avoid
526 that or the following can happend
526 that or the following can happen
527
527
528 ipdb> foo
528 ipdb> foo
529 "old"
529 "old"
530 ipdb> foo = "new"
530 ipdb> foo = "new"
531 ipdb> foo
531 ipdb> foo
532 "new"
532 "new"
533 ipdb> where
533 ipdb> where
534 ipdb> foo
534 ipdb> foo
535 "old"
535 "old"
536
536
537 So if frame is self.current_frame we instead return self.curframe_locals
537 So if frame is self.current_frame we instead return self.curframe_locals
538
538
539 """
539 """
540 if frame is self.curframe:
540 if frame is self.curframe:
541 return self.curframe_locals
541 return self.curframe_locals
542 else:
542 else:
543 return frame.f_locals
543 return frame.f_locals
544
544
545 def format_stack_entry(self, frame_lineno, lprefix=': ', context=None):
545 def format_stack_entry(self, frame_lineno, lprefix=': ', context=None):
546 if context is None:
546 if context is None:
547 context = self.context
547 context = self.context
548 try:
548 try:
549 context = int(context)
549 context = int(context)
550 if context <= 0:
550 if context <= 0:
551 print("Context must be a positive integer", file=self.stdout)
551 print("Context must be a positive integer", file=self.stdout)
552 except (TypeError, ValueError):
552 except (TypeError, ValueError):
553 print("Context must be a positive integer", file=self.stdout)
553 print("Context must be a positive integer", file=self.stdout)
554
554
555 import reprlib
555 import reprlib
556
556
557 ret = []
557 ret = []
558
558
559 Colors = self.color_scheme_table.active_colors
559 Colors = self.color_scheme_table.active_colors
560 ColorsNormal = Colors.Normal
560 ColorsNormal = Colors.Normal
561 tpl_link = "%s%%s%s" % (Colors.filenameEm, ColorsNormal)
561 tpl_link = "%s%%s%s" % (Colors.filenameEm, ColorsNormal)
562 tpl_call = "%s%%s%s%%s%s" % (Colors.vName, Colors.valEm, ColorsNormal)
562 tpl_call = "%s%%s%s%%s%s" % (Colors.vName, Colors.valEm, ColorsNormal)
563 tpl_line = "%%s%s%%s %s%%s" % (Colors.lineno, ColorsNormal)
563 tpl_line = "%%s%s%%s %s%%s" % (Colors.lineno, ColorsNormal)
564 tpl_line_em = "%%s%s%%s %s%%s%s" % (Colors.linenoEm, Colors.line, ColorsNormal)
564 tpl_line_em = "%%s%s%%s %s%%s%s" % (Colors.linenoEm, Colors.line, ColorsNormal)
565
565
566 frame, lineno = frame_lineno
566 frame, lineno = frame_lineno
567
567
568 return_value = ''
568 return_value = ''
569 loc_frame = self._get_frame_locals(frame)
569 loc_frame = self._get_frame_locals(frame)
570 if "__return__" in loc_frame:
570 if "__return__" in loc_frame:
571 rv = loc_frame["__return__"]
571 rv = loc_frame["__return__"]
572 # return_value += '->'
572 # return_value += '->'
573 return_value += reprlib.repr(rv) + "\n"
573 return_value += reprlib.repr(rv) + "\n"
574 ret.append(return_value)
574 ret.append(return_value)
575
575
576 #s = filename + '(' + `lineno` + ')'
576 #s = filename + '(' + `lineno` + ')'
577 filename = self.canonic(frame.f_code.co_filename)
577 filename = self.canonic(frame.f_code.co_filename)
578 link = tpl_link % py3compat.cast_unicode(filename)
578 link = tpl_link % py3compat.cast_unicode(filename)
579
579
580 if frame.f_code.co_name:
580 if frame.f_code.co_name:
581 func = frame.f_code.co_name
581 func = frame.f_code.co_name
582 else:
582 else:
583 func = "<lambda>"
583 func = "<lambda>"
584
584
585 call = ""
585 call = ""
586 if func != "?":
586 if func != "?":
587 if "__args__" in loc_frame:
587 if "__args__" in loc_frame:
588 args = reprlib.repr(loc_frame["__args__"])
588 args = reprlib.repr(loc_frame["__args__"])
589 else:
589 else:
590 args = '()'
590 args = '()'
591 call = tpl_call % (func, args)
591 call = tpl_call % (func, args)
592
592
593 # The level info should be generated in the same format pdb uses, to
593 # The level info should be generated in the same format pdb uses, to
594 # avoid breaking the pdbtrack functionality of python-mode in *emacs.
594 # avoid breaking the pdbtrack functionality of python-mode in *emacs.
595 if frame is self.curframe:
595 if frame is self.curframe:
596 ret.append('> ')
596 ret.append('> ')
597 else:
597 else:
598 ret.append(" ")
598 ret.append(" ")
599 ret.append("%s(%s)%s\n" % (link, lineno, call))
599 ret.append("%s(%s)%s\n" % (link, lineno, call))
600
600
601 start = lineno - 1 - context//2
601 start = lineno - 1 - context//2
602 lines = linecache.getlines(filename)
602 lines = linecache.getlines(filename)
603 start = min(start, len(lines) - context)
603 start = min(start, len(lines) - context)
604 start = max(start, 0)
604 start = max(start, 0)
605 lines = lines[start : start + context]
605 lines = lines[start : start + context]
606
606
607 for i, line in enumerate(lines):
607 for i, line in enumerate(lines):
608 show_arrow = start + 1 + i == lineno
608 show_arrow = start + 1 + i == lineno
609 linetpl = (frame is self.curframe or show_arrow) and tpl_line_em or tpl_line
609 linetpl = (frame is self.curframe or show_arrow) and tpl_line_em or tpl_line
610 ret.append(
610 ret.append(
611 self.__format_line(
611 self.__format_line(
612 linetpl, filename, start + 1 + i, line, arrow=show_arrow
612 linetpl, filename, start + 1 + i, line, arrow=show_arrow
613 )
613 )
614 )
614 )
615 return "".join(ret)
615 return "".join(ret)
616
616
617 def __format_line(self, tpl_line, filename, lineno, line, arrow=False):
617 def __format_line(self, tpl_line, filename, lineno, line, arrow=False):
618 bp_mark = ""
618 bp_mark = ""
619 bp_mark_color = ""
619 bp_mark_color = ""
620
620
621 new_line, err = self.parser.format2(line, 'str')
621 new_line, err = self.parser.format2(line, 'str')
622 if not err:
622 if not err:
623 line = new_line
623 line = new_line
624
624
625 bp = None
625 bp = None
626 if lineno in self.get_file_breaks(filename):
626 if lineno in self.get_file_breaks(filename):
627 bps = self.get_breaks(filename, lineno)
627 bps = self.get_breaks(filename, lineno)
628 bp = bps[-1]
628 bp = bps[-1]
629
629
630 if bp:
630 if bp:
631 Colors = self.color_scheme_table.active_colors
631 Colors = self.color_scheme_table.active_colors
632 bp_mark = str(bp.number)
632 bp_mark = str(bp.number)
633 bp_mark_color = Colors.breakpoint_enabled
633 bp_mark_color = Colors.breakpoint_enabled
634 if not bp.enabled:
634 if not bp.enabled:
635 bp_mark_color = Colors.breakpoint_disabled
635 bp_mark_color = Colors.breakpoint_disabled
636
636
637 numbers_width = 7
637 numbers_width = 7
638 if arrow:
638 if arrow:
639 # This is the line with the error
639 # This is the line with the error
640 pad = numbers_width - len(str(lineno)) - len(bp_mark)
640 pad = numbers_width - len(str(lineno)) - len(bp_mark)
641 num = '%s%s' % (make_arrow(pad), str(lineno))
641 num = '%s%s' % (make_arrow(pad), str(lineno))
642 else:
642 else:
643 num = '%*s' % (numbers_width - len(bp_mark), str(lineno))
643 num = '%*s' % (numbers_width - len(bp_mark), str(lineno))
644
644
645 return tpl_line % (bp_mark_color + bp_mark, num, line)
645 return tpl_line % (bp_mark_color + bp_mark, num, line)
646
646
647 def print_list_lines(self, filename, first, last):
647 def print_list_lines(self, filename, first, last):
648 """The printing (as opposed to the parsing part of a 'list'
648 """The printing (as opposed to the parsing part of a 'list'
649 command."""
649 command."""
650 try:
650 try:
651 Colors = self.color_scheme_table.active_colors
651 Colors = self.color_scheme_table.active_colors
652 ColorsNormal = Colors.Normal
652 ColorsNormal = Colors.Normal
653 tpl_line = '%%s%s%%s %s%%s' % (Colors.lineno, ColorsNormal)
653 tpl_line = '%%s%s%%s %s%%s' % (Colors.lineno, ColorsNormal)
654 tpl_line_em = '%%s%s%%s %s%%s%s' % (Colors.linenoEm, Colors.line, ColorsNormal)
654 tpl_line_em = '%%s%s%%s %s%%s%s' % (Colors.linenoEm, Colors.line, ColorsNormal)
655 src = []
655 src = []
656 if filename == "<string>" and hasattr(self, "_exec_filename"):
656 if filename == "<string>" and hasattr(self, "_exec_filename"):
657 filename = self._exec_filename
657 filename = self._exec_filename
658
658
659 for lineno in range(first, last+1):
659 for lineno in range(first, last+1):
660 line = linecache.getline(filename, lineno)
660 line = linecache.getline(filename, lineno)
661 if not line:
661 if not line:
662 break
662 break
663
663
664 if lineno == self.curframe.f_lineno:
664 if lineno == self.curframe.f_lineno:
665 line = self.__format_line(
665 line = self.__format_line(
666 tpl_line_em, filename, lineno, line, arrow=True
666 tpl_line_em, filename, lineno, line, arrow=True
667 )
667 )
668 else:
668 else:
669 line = self.__format_line(
669 line = self.__format_line(
670 tpl_line, filename, lineno, line, arrow=False
670 tpl_line, filename, lineno, line, arrow=False
671 )
671 )
672
672
673 src.append(line)
673 src.append(line)
674 self.lineno = lineno
674 self.lineno = lineno
675
675
676 print(''.join(src), file=self.stdout)
676 print(''.join(src), file=self.stdout)
677
677
678 except KeyboardInterrupt:
678 except KeyboardInterrupt:
679 pass
679 pass
680
680
681 def do_skip_predicates(self, args):
681 def do_skip_predicates(self, args):
682 """
682 """
683 Turn on/off individual predicates as to whether a frame should be hidden/skip.
683 Turn on/off individual predicates as to whether a frame should be hidden/skip.
684
684
685 The global option to skip (or not) hidden frames is set with skip_hidden
685 The global option to skip (or not) hidden frames is set with skip_hidden
686
686
687 To change the value of a predicate
687 To change the value of a predicate
688
688
689 skip_predicates key [true|false]
689 skip_predicates key [true|false]
690
690
691 Call without arguments to see the current values.
691 Call without arguments to see the current values.
692
692
693 To permanently change the value of an option add the corresponding
693 To permanently change the value of an option add the corresponding
694 command to your ``~/.pdbrc`` file. If you are programmatically using the
694 command to your ``~/.pdbrc`` file. If you are programmatically using the
695 Pdb instance you can also change the ``default_predicates`` class
695 Pdb instance you can also change the ``default_predicates`` class
696 attribute.
696 attribute.
697 """
697 """
698 if not args.strip():
698 if not args.strip():
699 print("current predicates:")
699 print("current predicates:")
700 for (p, v) in self._predicates.items():
700 for (p, v) in self._predicates.items():
701 print(" ", p, ":", v)
701 print(" ", p, ":", v)
702 return
702 return
703 type_value = args.strip().split(" ")
703 type_value = args.strip().split(" ")
704 if len(type_value) != 2:
704 if len(type_value) != 2:
705 print(
705 print(
706 f"Usage: skip_predicates <type> <value>, with <type> one of {set(self._predicates.keys())}"
706 f"Usage: skip_predicates <type> <value>, with <type> one of {set(self._predicates.keys())}"
707 )
707 )
708 return
708 return
709
709
710 type_, value = type_value
710 type_, value = type_value
711 if type_ not in self._predicates:
711 if type_ not in self._predicates:
712 print(f"{type_!r} not in {set(self._predicates.keys())}")
712 print(f"{type_!r} not in {set(self._predicates.keys())}")
713 return
713 return
714 if value.lower() not in ("true", "yes", "1", "no", "false", "0"):
714 if value.lower() not in ("true", "yes", "1", "no", "false", "0"):
715 print(
715 print(
716 f"{value!r} is invalid - use one of ('true', 'yes', '1', 'no', 'false', '0')"
716 f"{value!r} is invalid - use one of ('true', 'yes', '1', 'no', 'false', '0')"
717 )
717 )
718 return
718 return
719
719
720 self._predicates[type_] = value.lower() in ("true", "yes", "1")
720 self._predicates[type_] = value.lower() in ("true", "yes", "1")
721 if not any(self._predicates.values()):
721 if not any(self._predicates.values()):
722 print(
722 print(
723 "Warning, all predicates set to False, skip_hidden may not have any effects."
723 "Warning, all predicates set to False, skip_hidden may not have any effects."
724 )
724 )
725
725
726 def do_skip_hidden(self, arg):
726 def do_skip_hidden(self, arg):
727 """
727 """
728 Change whether or not we should skip frames with the
728 Change whether or not we should skip frames with the
729 __tracebackhide__ attribute.
729 __tracebackhide__ attribute.
730 """
730 """
731 if not arg.strip():
731 if not arg.strip():
732 print(
732 print(
733 f"skip_hidden = {self.skip_hidden}, use 'yes','no', 'true', or 'false' to change."
733 f"skip_hidden = {self.skip_hidden}, use 'yes','no', 'true', or 'false' to change."
734 )
734 )
735 elif arg.strip().lower() in ("true", "yes"):
735 elif arg.strip().lower() in ("true", "yes"):
736 self.skip_hidden = True
736 self.skip_hidden = True
737 elif arg.strip().lower() in ("false", "no"):
737 elif arg.strip().lower() in ("false", "no"):
738 self.skip_hidden = False
738 self.skip_hidden = False
739 if not any(self._predicates.values()):
739 if not any(self._predicates.values()):
740 print(
740 print(
741 "Warning, all predicates set to False, skip_hidden may not have any effects."
741 "Warning, all predicates set to False, skip_hidden may not have any effects."
742 )
742 )
743
743
744 def do_list(self, arg):
744 def do_list(self, arg):
745 """Print lines of code from the current stack frame
745 """Print lines of code from the current stack frame
746 """
746 """
747 self.lastcmd = 'list'
747 self.lastcmd = 'list'
748 last = None
748 last = None
749 if arg:
749 if arg:
750 try:
750 try:
751 x = eval(arg, {}, {})
751 x = eval(arg, {}, {})
752 if type(x) == type(()):
752 if type(x) == type(()):
753 first, last = x
753 first, last = x
754 first = int(first)
754 first = int(first)
755 last = int(last)
755 last = int(last)
756 if last < first:
756 if last < first:
757 # Assume it's a count
757 # Assume it's a count
758 last = first + last
758 last = first + last
759 else:
759 else:
760 first = max(1, int(x) - 5)
760 first = max(1, int(x) - 5)
761 except:
761 except:
762 print('*** Error in argument:', repr(arg), file=self.stdout)
762 print('*** Error in argument:', repr(arg), file=self.stdout)
763 return
763 return
764 elif self.lineno is None:
764 elif self.lineno is None:
765 first = max(1, self.curframe.f_lineno - 5)
765 first = max(1, self.curframe.f_lineno - 5)
766 else:
766 else:
767 first = self.lineno + 1
767 first = self.lineno + 1
768 if last is None:
768 if last is None:
769 last = first + 10
769 last = first + 10
770 self.print_list_lines(self.curframe.f_code.co_filename, first, last)
770 self.print_list_lines(self.curframe.f_code.co_filename, first, last)
771
771
772 # vds: >>
772 # vds: >>
773 lineno = first
773 lineno = first
774 filename = self.curframe.f_code.co_filename
774 filename = self.curframe.f_code.co_filename
775 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
775 self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
776 # vds: <<
776 # vds: <<
777
777
778 do_l = do_list
778 do_l = do_list
779
779
780 def getsourcelines(self, obj):
780 def getsourcelines(self, obj):
781 lines, lineno = inspect.findsource(obj)
781 lines, lineno = inspect.findsource(obj)
782 if inspect.isframe(obj) and obj.f_globals is self._get_frame_locals(obj):
782 if inspect.isframe(obj) and obj.f_globals is self._get_frame_locals(obj):
783 # must be a module frame: do not try to cut a block out of it
783 # must be a module frame: do not try to cut a block out of it
784 return lines, 1
784 return lines, 1
785 elif inspect.ismodule(obj):
785 elif inspect.ismodule(obj):
786 return lines, 1
786 return lines, 1
787 return inspect.getblock(lines[lineno:]), lineno+1
787 return inspect.getblock(lines[lineno:]), lineno+1
788
788
789 def do_longlist(self, arg):
789 def do_longlist(self, arg):
790 """Print lines of code from the current stack frame.
790 """Print lines of code from the current stack frame.
791
791
792 Shows more lines than 'list' does.
792 Shows more lines than 'list' does.
793 """
793 """
794 self.lastcmd = 'longlist'
794 self.lastcmd = 'longlist'
795 try:
795 try:
796 lines, lineno = self.getsourcelines(self.curframe)
796 lines, lineno = self.getsourcelines(self.curframe)
797 except OSError as err:
797 except OSError as err:
798 self.error(err)
798 self.error(err)
799 return
799 return
800 last = lineno + len(lines)
800 last = lineno + len(lines)
801 self.print_list_lines(self.curframe.f_code.co_filename, lineno, last)
801 self.print_list_lines(self.curframe.f_code.co_filename, lineno, last)
802 do_ll = do_longlist
802 do_ll = do_longlist
803
803
804 def do_debug(self, arg):
804 def do_debug(self, arg):
805 """debug code
805 """debug code
806 Enter a recursive debugger that steps through the code
806 Enter a recursive debugger that steps through the code
807 argument (which is an arbitrary expression or statement to be
807 argument (which is an arbitrary expression or statement to be
808 executed in the current environment).
808 executed in the current environment).
809 """
809 """
810 trace_function = sys.gettrace()
810 trace_function = sys.gettrace()
811 sys.settrace(None)
811 sys.settrace(None)
812 globals = self.curframe.f_globals
812 globals = self.curframe.f_globals
813 locals = self.curframe_locals
813 locals = self.curframe_locals
814 p = self.__class__(completekey=self.completekey,
814 p = self.__class__(completekey=self.completekey,
815 stdin=self.stdin, stdout=self.stdout)
815 stdin=self.stdin, stdout=self.stdout)
816 p.use_rawinput = self.use_rawinput
816 p.use_rawinput = self.use_rawinput
817 p.prompt = "(%s) " % self.prompt.strip()
817 p.prompt = "(%s) " % self.prompt.strip()
818 self.message("ENTERING RECURSIVE DEBUGGER")
818 self.message("ENTERING RECURSIVE DEBUGGER")
819 sys.call_tracing(p.run, (arg, globals, locals))
819 sys.call_tracing(p.run, (arg, globals, locals))
820 self.message("LEAVING RECURSIVE DEBUGGER")
820 self.message("LEAVING RECURSIVE DEBUGGER")
821 sys.settrace(trace_function)
821 sys.settrace(trace_function)
822 self.lastcmd = p.lastcmd
822 self.lastcmd = p.lastcmd
823
823
824 def do_pdef(self, arg):
824 def do_pdef(self, arg):
825 """Print the call signature for any callable object.
825 """Print the call signature for any callable object.
826
826
827 The debugger interface to %pdef"""
827 The debugger interface to %pdef"""
828 namespaces = [
828 namespaces = [
829 ("Locals", self.curframe_locals),
829 ("Locals", self.curframe_locals),
830 ("Globals", self.curframe.f_globals),
830 ("Globals", self.curframe.f_globals),
831 ]
831 ]
832 self.shell.find_line_magic("pdef")(arg, namespaces=namespaces)
832 self.shell.find_line_magic("pdef")(arg, namespaces=namespaces)
833
833
834 def do_pdoc(self, arg):
834 def do_pdoc(self, arg):
835 """Print the docstring for an object.
835 """Print the docstring for an object.
836
836
837 The debugger interface to %pdoc."""
837 The debugger interface to %pdoc."""
838 namespaces = [
838 namespaces = [
839 ("Locals", self.curframe_locals),
839 ("Locals", self.curframe_locals),
840 ("Globals", self.curframe.f_globals),
840 ("Globals", self.curframe.f_globals),
841 ]
841 ]
842 self.shell.find_line_magic("pdoc")(arg, namespaces=namespaces)
842 self.shell.find_line_magic("pdoc")(arg, namespaces=namespaces)
843
843
844 def do_pfile(self, arg):
844 def do_pfile(self, arg):
845 """Print (or run through pager) the file where an object is defined.
845 """Print (or run through pager) the file where an object is defined.
846
846
847 The debugger interface to %pfile.
847 The debugger interface to %pfile.
848 """
848 """
849 namespaces = [
849 namespaces = [
850 ("Locals", self.curframe_locals),
850 ("Locals", self.curframe_locals),
851 ("Globals", self.curframe.f_globals),
851 ("Globals", self.curframe.f_globals),
852 ]
852 ]
853 self.shell.find_line_magic("pfile")(arg, namespaces=namespaces)
853 self.shell.find_line_magic("pfile")(arg, namespaces=namespaces)
854
854
855 def do_pinfo(self, arg):
855 def do_pinfo(self, arg):
856 """Provide detailed information about an object.
856 """Provide detailed information about an object.
857
857
858 The debugger interface to %pinfo, i.e., obj?."""
858 The debugger interface to %pinfo, i.e., obj?."""
859 namespaces = [
859 namespaces = [
860 ("Locals", self.curframe_locals),
860 ("Locals", self.curframe_locals),
861 ("Globals", self.curframe.f_globals),
861 ("Globals", self.curframe.f_globals),
862 ]
862 ]
863 self.shell.find_line_magic("pinfo")(arg, namespaces=namespaces)
863 self.shell.find_line_magic("pinfo")(arg, namespaces=namespaces)
864
864
865 def do_pinfo2(self, arg):
865 def do_pinfo2(self, arg):
866 """Provide extra detailed information about an object.
866 """Provide extra detailed information about an object.
867
867
868 The debugger interface to %pinfo2, i.e., obj??."""
868 The debugger interface to %pinfo2, i.e., obj??."""
869 namespaces = [
869 namespaces = [
870 ("Locals", self.curframe_locals),
870 ("Locals", self.curframe_locals),
871 ("Globals", self.curframe.f_globals),
871 ("Globals", self.curframe.f_globals),
872 ]
872 ]
873 self.shell.find_line_magic("pinfo2")(arg, namespaces=namespaces)
873 self.shell.find_line_magic("pinfo2")(arg, namespaces=namespaces)
874
874
875 def do_psource(self, arg):
875 def do_psource(self, arg):
876 """Print (or run through pager) the source code for an object."""
876 """Print (or run through pager) the source code for an object."""
877 namespaces = [
877 namespaces = [
878 ("Locals", self.curframe_locals),
878 ("Locals", self.curframe_locals),
879 ("Globals", self.curframe.f_globals),
879 ("Globals", self.curframe.f_globals),
880 ]
880 ]
881 self.shell.find_line_magic("psource")(arg, namespaces=namespaces)
881 self.shell.find_line_magic("psource")(arg, namespaces=namespaces)
882
882
883 def do_where(self, arg):
883 def do_where(self, arg):
884 """w(here)
884 """w(here)
885 Print a stack trace, with the most recent frame at the bottom.
885 Print a stack trace, with the most recent frame at the bottom.
886 An arrow indicates the "current frame", which determines the
886 An arrow indicates the "current frame", which determines the
887 context of most commands. 'bt' is an alias for this command.
887 context of most commands. 'bt' is an alias for this command.
888
888
889 Take a number as argument as an (optional) number of context line to
889 Take a number as argument as an (optional) number of context line to
890 print"""
890 print"""
891 if arg:
891 if arg:
892 try:
892 try:
893 context = int(arg)
893 context = int(arg)
894 except ValueError as err:
894 except ValueError as err:
895 self.error(err)
895 self.error(err)
896 return
896 return
897 self.print_stack_trace(context)
897 self.print_stack_trace(context)
898 else:
898 else:
899 self.print_stack_trace()
899 self.print_stack_trace()
900
900
901 do_w = do_where
901 do_w = do_where
902
902
903 def break_anywhere(self, frame):
903 def break_anywhere(self, frame):
904 """
904 """
905
905
906 _stop_in_decorator_internals is overly restrictive, as we may still want
906 _stop_in_decorator_internals is overly restrictive, as we may still want
907 to trace function calls, so we need to also update break_anywhere so
907 to trace function calls, so we need to also update break_anywhere so
908 that is we don't `stop_here`, because of debugger skip, we may still
908 that is we don't `stop_here`, because of debugger skip, we may still
909 stop at any point inside the function
909 stop at any point inside the function
910
910
911 """
911 """
912
912
913 sup = super().break_anywhere(frame)
913 sup = super().break_anywhere(frame)
914 if sup:
914 if sup:
915 return sup
915 return sup
916 if self._predicates["debuggerskip"]:
916 if self._predicates["debuggerskip"]:
917 if DEBUGGERSKIP in frame.f_code.co_varnames:
917 if DEBUGGERSKIP in frame.f_code.co_varnames:
918 return True
918 return True
919 if frame.f_back and self._get_frame_locals(frame.f_back).get(DEBUGGERSKIP):
919 if frame.f_back and self._get_frame_locals(frame.f_back).get(DEBUGGERSKIP):
920 return True
920 return True
921 return False
921 return False
922
922
923 @skip_doctest
923 @skip_doctest
924 def _is_in_decorator_internal_and_should_skip(self, frame):
924 def _is_in_decorator_internal_and_should_skip(self, frame):
925 """
925 """
926 Utility to tell us whether we are in a decorator internal and should stop.
926 Utility to tell us whether we are in a decorator internal and should stop.
927
927
928
928
929
929
930 """
930 """
931
931
932 # if we are disabled don't skip
932 # if we are disabled don't skip
933 if not self._predicates["debuggerskip"]:
933 if not self._predicates["debuggerskip"]:
934 return False
934 return False
935
935
936 # if frame is tagged, skip by default.
936 # if frame is tagged, skip by default.
937 if DEBUGGERSKIP in frame.f_code.co_varnames:
937 if DEBUGGERSKIP in frame.f_code.co_varnames:
938 return True
938 return True
939
939
940 # if one of the parent frame value set to True skip as well.
940 # if one of the parent frame value set to True skip as well.
941
941
942 cframe = frame
942 cframe = frame
943 while getattr(cframe, "f_back", None):
943 while getattr(cframe, "f_back", None):
944 cframe = cframe.f_back
944 cframe = cframe.f_back
945 if self._get_frame_locals(cframe).get(DEBUGGERSKIP):
945 if self._get_frame_locals(cframe).get(DEBUGGERSKIP):
946 return True
946 return True
947
947
948 return False
948 return False
949
949
950 def stop_here(self, frame):
950 def stop_here(self, frame):
951
951
952 if self._is_in_decorator_internal_and_should_skip(frame) is True:
952 if self._is_in_decorator_internal_and_should_skip(frame) is True:
953 return False
953 return False
954
954
955 hidden = False
955 hidden = False
956 if self.skip_hidden:
956 if self.skip_hidden:
957 hidden = self._hidden_predicate(frame)
957 hidden = self._hidden_predicate(frame)
958 if hidden:
958 if hidden:
959 if self.report_skipped:
959 if self.report_skipped:
960 Colors = self.color_scheme_table.active_colors
960 Colors = self.color_scheme_table.active_colors
961 ColorsNormal = Colors.Normal
961 ColorsNormal = Colors.Normal
962 print(
962 print(
963 f"{Colors.excName} [... skipped 1 hidden frame]{ColorsNormal}\n"
963 f"{Colors.excName} [... skipped 1 hidden frame]{ColorsNormal}\n"
964 )
964 )
965 return super().stop_here(frame)
965 return super().stop_here(frame)
966
966
967 def do_up(self, arg):
967 def do_up(self, arg):
968 """u(p) [count]
968 """u(p) [count]
969 Move the current frame count (default one) levels up in the
969 Move the current frame count (default one) levels up in the
970 stack trace (to an older frame).
970 stack trace (to an older frame).
971
971
972 Will skip hidden frames.
972 Will skip hidden frames.
973 """
973 """
974 # modified version of upstream that skips
974 # modified version of upstream that skips
975 # frames with __tracebackhide__
975 # frames with __tracebackhide__
976 if self.curindex == 0:
976 if self.curindex == 0:
977 self.error("Oldest frame")
977 self.error("Oldest frame")
978 return
978 return
979 try:
979 try:
980 count = int(arg or 1)
980 count = int(arg or 1)
981 except ValueError:
981 except ValueError:
982 self.error("Invalid frame count (%s)" % arg)
982 self.error("Invalid frame count (%s)" % arg)
983 return
983 return
984 skipped = 0
984 skipped = 0
985 if count < 0:
985 if count < 0:
986 _newframe = 0
986 _newframe = 0
987 else:
987 else:
988 counter = 0
988 counter = 0
989 hidden_frames = self.hidden_frames(self.stack)
989 hidden_frames = self.hidden_frames(self.stack)
990 for i in range(self.curindex - 1, -1, -1):
990 for i in range(self.curindex - 1, -1, -1):
991 if hidden_frames[i] and self.skip_hidden:
991 if hidden_frames[i] and self.skip_hidden:
992 skipped += 1
992 skipped += 1
993 continue
993 continue
994 counter += 1
994 counter += 1
995 if counter >= count:
995 if counter >= count:
996 break
996 break
997 else:
997 else:
998 # if no break occured.
998 # if no break occurred.
999 self.error(
999 self.error(
1000 "all frames above hidden, use `skip_hidden False` to get get into those."
1000 "all frames above hidden, use `skip_hidden False` to get get into those."
1001 )
1001 )
1002 return
1002 return
1003
1003
1004 Colors = self.color_scheme_table.active_colors
1004 Colors = self.color_scheme_table.active_colors
1005 ColorsNormal = Colors.Normal
1005 ColorsNormal = Colors.Normal
1006 _newframe = i
1006 _newframe = i
1007 self._select_frame(_newframe)
1007 self._select_frame(_newframe)
1008 if skipped:
1008 if skipped:
1009 print(
1009 print(
1010 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
1010 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
1011 )
1011 )
1012
1012
1013 def do_down(self, arg):
1013 def do_down(self, arg):
1014 """d(own) [count]
1014 """d(own) [count]
1015 Move the current frame count (default one) levels down in the
1015 Move the current frame count (default one) levels down in the
1016 stack trace (to a newer frame).
1016 stack trace (to a newer frame).
1017
1017
1018 Will skip hidden frames.
1018 Will skip hidden frames.
1019 """
1019 """
1020 if self.curindex + 1 == len(self.stack):
1020 if self.curindex + 1 == len(self.stack):
1021 self.error("Newest frame")
1021 self.error("Newest frame")
1022 return
1022 return
1023 try:
1023 try:
1024 count = int(arg or 1)
1024 count = int(arg or 1)
1025 except ValueError:
1025 except ValueError:
1026 self.error("Invalid frame count (%s)" % arg)
1026 self.error("Invalid frame count (%s)" % arg)
1027 return
1027 return
1028 if count < 0:
1028 if count < 0:
1029 _newframe = len(self.stack) - 1
1029 _newframe = len(self.stack) - 1
1030 else:
1030 else:
1031 counter = 0
1031 counter = 0
1032 skipped = 0
1032 skipped = 0
1033 hidden_frames = self.hidden_frames(self.stack)
1033 hidden_frames = self.hidden_frames(self.stack)
1034 for i in range(self.curindex + 1, len(self.stack)):
1034 for i in range(self.curindex + 1, len(self.stack)):
1035 if hidden_frames[i] and self.skip_hidden:
1035 if hidden_frames[i] and self.skip_hidden:
1036 skipped += 1
1036 skipped += 1
1037 continue
1037 continue
1038 counter += 1
1038 counter += 1
1039 if counter >= count:
1039 if counter >= count:
1040 break
1040 break
1041 else:
1041 else:
1042 self.error(
1042 self.error(
1043 "all frames bellow hidden, use `skip_hidden False` to get get into those."
1043 "all frames below hidden, use `skip_hidden False` to get get into those."
1044 )
1044 )
1045 return
1045 return
1046
1046
1047 Colors = self.color_scheme_table.active_colors
1047 Colors = self.color_scheme_table.active_colors
1048 ColorsNormal = Colors.Normal
1048 ColorsNormal = Colors.Normal
1049 if skipped:
1049 if skipped:
1050 print(
1050 print(
1051 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
1051 f"{Colors.excName} [... skipped {skipped} hidden frame(s)]{ColorsNormal}\n"
1052 )
1052 )
1053 _newframe = i
1053 _newframe = i
1054
1054
1055 self._select_frame(_newframe)
1055 self._select_frame(_newframe)
1056
1056
1057 do_d = do_down
1057 do_d = do_down
1058 do_u = do_up
1058 do_u = do_up
1059
1059
1060 def do_context(self, context):
1060 def do_context(self, context):
1061 """context number_of_lines
1061 """context number_of_lines
1062 Set the number of lines of source code to show when displaying
1062 Set the number of lines of source code to show when displaying
1063 stacktrace information.
1063 stacktrace information.
1064 """
1064 """
1065 try:
1065 try:
1066 new_context = int(context)
1066 new_context = int(context)
1067 if new_context <= 0:
1067 if new_context <= 0:
1068 raise ValueError()
1068 raise ValueError()
1069 self.context = new_context
1069 self.context = new_context
1070 except ValueError:
1070 except ValueError:
1071 self.error("The 'context' command requires a positive integer argument.")
1071 self.error("The 'context' command requires a positive integer argument.")
1072
1072
1073
1073
1074 class InterruptiblePdb(Pdb):
1074 class InterruptiblePdb(Pdb):
1075 """Version of debugger where KeyboardInterrupt exits the debugger altogether."""
1075 """Version of debugger where KeyboardInterrupt exits the debugger altogether."""
1076
1076
1077 def cmdloop(self, intro=None):
1077 def cmdloop(self, intro=None):
1078 """Wrap cmdloop() such that KeyboardInterrupt stops the debugger."""
1078 """Wrap cmdloop() such that KeyboardInterrupt stops the debugger."""
1079 try:
1079 try:
1080 return OldPdb.cmdloop(self, intro=intro)
1080 return OldPdb.cmdloop(self, intro=intro)
1081 except KeyboardInterrupt:
1081 except KeyboardInterrupt:
1082 self.stop_here = lambda frame: False
1082 self.stop_here = lambda frame: False
1083 self.do_quit("")
1083 self.do_quit("")
1084 sys.settrace(None)
1084 sys.settrace(None)
1085 self.quitting = False
1085 self.quitting = False
1086 raise
1086 raise
1087
1087
1088 def _cmdloop(self):
1088 def _cmdloop(self):
1089 while True:
1089 while True:
1090 try:
1090 try:
1091 # keyboard interrupts allow for an easy way to cancel
1091 # keyboard interrupts allow for an easy way to cancel
1092 # the current command, so allow them during interactive input
1092 # the current command, so allow them during interactive input
1093 self.allow_kbdint = True
1093 self.allow_kbdint = True
1094 self.cmdloop()
1094 self.cmdloop()
1095 self.allow_kbdint = False
1095 self.allow_kbdint = False
1096 break
1096 break
1097 except KeyboardInterrupt:
1097 except KeyboardInterrupt:
1098 self.message('--KeyboardInterrupt--')
1098 self.message('--KeyboardInterrupt--')
1099 raise
1099 raise
1100
1100
1101
1101
1102 def set_trace(frame=None):
1102 def set_trace(frame=None):
1103 """
1103 """
1104 Start debugging from `frame`.
1104 Start debugging from `frame`.
1105
1105
1106 If frame is not specified, debugging starts from caller's frame.
1106 If frame is not specified, debugging starts from caller's frame.
1107 """
1107 """
1108 Pdb().set_trace(frame or sys._getframe().f_back)
1108 Pdb().set_trace(frame or sys._getframe().f_back)
@@ -1,66 +1,66 b''
1 from IPython.utils.capture import capture_output
1 from IPython.utils.capture import capture_output
2
2
3 import pytest
3 import pytest
4
4
5 def test_alias_lifecycle():
5 def test_alias_lifecycle():
6 name = 'test_alias1'
6 name = 'test_alias1'
7 cmd = 'echo "Hello"'
7 cmd = 'echo "Hello"'
8 am = _ip.alias_manager
8 am = _ip.alias_manager
9 am.clear_aliases()
9 am.clear_aliases()
10 am.define_alias(name, cmd)
10 am.define_alias(name, cmd)
11 assert am.is_alias(name)
11 assert am.is_alias(name)
12 assert am.retrieve_alias(name) == cmd
12 assert am.retrieve_alias(name) == cmd
13 assert (name, cmd) in am.aliases
13 assert (name, cmd) in am.aliases
14
14
15 # Test running the alias
15 # Test running the alias
16 orig_system = _ip.system
16 orig_system = _ip.system
17 result = []
17 result = []
18 _ip.system = result.append
18 _ip.system = result.append
19 try:
19 try:
20 _ip.run_cell('%{}'.format(name))
20 _ip.run_cell('%{}'.format(name))
21 result = [c.strip() for c in result]
21 result = [c.strip() for c in result]
22 assert result == [cmd]
22 assert result == [cmd]
23 finally:
23 finally:
24 _ip.system = orig_system
24 _ip.system = orig_system
25
25
26 # Test removing the alias
26 # Test removing the alias
27 am.undefine_alias(name)
27 am.undefine_alias(name)
28 assert not am.is_alias(name)
28 assert not am.is_alias(name)
29 with pytest.raises(ValueError):
29 with pytest.raises(ValueError):
30 am.retrieve_alias(name)
30 am.retrieve_alias(name)
31 assert (name, cmd) not in am.aliases
31 assert (name, cmd) not in am.aliases
32
32
33
33
34 def test_alias_args_error():
34 def test_alias_args_error():
35 """Error expanding with wrong number of arguments"""
35 """Error expanding with wrong number of arguments"""
36 _ip.alias_manager.define_alias('parts', 'echo first %s second %s')
36 _ip.alias_manager.define_alias('parts', 'echo first %s second %s')
37 # capture stderr:
37 # capture stderr:
38 with capture_output() as cap:
38 with capture_output() as cap:
39 _ip.run_cell('parts 1')
39 _ip.run_cell('parts 1')
40
40
41 assert cap.stderr.split(":")[0] == "UsageError"
41 assert cap.stderr.split(":")[0] == "UsageError"
42
42
43
43
44 def test_alias_args_commented():
44 def test_alias_args_commented():
45 """Check that alias correctly ignores 'commented out' args"""
45 """Check that alias correctly ignores 'commented out' args"""
46 _ip.magic('alias commetarg echo this is %%s a commented out arg')
46 _ip.magic('alias commetarg echo this is %%s a commented out arg')
47
47
48 with capture_output() as cap:
48 with capture_output() as cap:
49 _ip.run_cell('commetarg')
49 _ip.run_cell('commetarg')
50
50
51 # strip() is for pytest compat; testing via iptest patch IPython shell
51 # strip() is for pytest compat; testing via iptest patch IPython shell
52 # in testin.globalipapp and replace the system call which messed up the
52 # in testing.globalipapp and replace the system call which messed up the
53 # \r\n
53 # \r\n
54 assert cap.stdout.strip() == 'this is %s a commented out arg'
54 assert cap.stdout.strip() == 'this is %s a commented out arg'
55
55
56 def test_alias_args_commented_nargs():
56 def test_alias_args_commented_nargs():
57 """Check that alias correctly counts args, excluding those commented out"""
57 """Check that alias correctly counts args, excluding those commented out"""
58 am = _ip.alias_manager
58 am = _ip.alias_manager
59 alias_name = 'comargcount'
59 alias_name = 'comargcount'
60 cmd = 'echo this is %%s a commented out arg and this is not %s'
60 cmd = 'echo this is %%s a commented out arg and this is not %s'
61
61
62 am.define_alias(alias_name, cmd)
62 am.define_alias(alias_name, cmd)
63 assert am.is_alias(alias_name)
63 assert am.is_alias(alias_name)
64
64
65 thealias = am.get_alias(alias_name)
65 thealias = am.get_alias(alias_name)
66 assert thealias.nargs == 1
66 assert thealias.nargs == 1
@@ -1,1278 +1,1278 b''
1 # encoding: utf-8
1 # encoding: utf-8
2 """Tests for the IPython tab-completion machinery."""
2 """Tests for the IPython tab-completion machinery."""
3
3
4 # Copyright (c) IPython Development Team.
4 # Copyright (c) IPython Development Team.
5 # Distributed under the terms of the Modified BSD License.
5 # Distributed under the terms of the Modified BSD License.
6
6
7 import os
7 import os
8 import sys
8 import sys
9 import textwrap
9 import textwrap
10 import unittest
10 import unittest
11
11
12 from contextlib import contextmanager
12 from contextlib import contextmanager
13
13
14 import nose.tools as nt
14 import nose.tools as nt
15
15
16 from traitlets.config.loader import Config
16 from traitlets.config.loader import Config
17 from IPython import get_ipython
17 from IPython import get_ipython
18 from IPython.core import completer
18 from IPython.core import completer
19 from IPython.external import decorators
19 from IPython.external import decorators
20 from IPython.utils.tempdir import TemporaryDirectory, TemporaryWorkingDirectory
20 from IPython.utils.tempdir import TemporaryDirectory, TemporaryWorkingDirectory
21 from IPython.utils.generics import complete_object
21 from IPython.utils.generics import complete_object
22 from IPython.testing import decorators as dec
22 from IPython.testing import decorators as dec
23
23
24 from IPython.core.completer import (
24 from IPython.core.completer import (
25 Completion,
25 Completion,
26 provisionalcompleter,
26 provisionalcompleter,
27 match_dict_keys,
27 match_dict_keys,
28 _deduplicate_completions,
28 _deduplicate_completions,
29 )
29 )
30 from nose.tools import assert_in, assert_not_in
30 from nose.tools import assert_in, assert_not_in
31
31
32 # -----------------------------------------------------------------------------
32 # -----------------------------------------------------------------------------
33 # Test functions
33 # Test functions
34 # -----------------------------------------------------------------------------
34 # -----------------------------------------------------------------------------
35
35
36 def recompute_unicode_ranges():
36 def recompute_unicode_ranges():
37 """
37 """
38 utility to recompute the largest unicode range without any characters
38 utility to recompute the largest unicode range without any characters
39
39
40 use to recompute the gap in the global _UNICODE_RANGES of completer.py
40 use to recompute the gap in the global _UNICODE_RANGES of completer.py
41 """
41 """
42 import itertools
42 import itertools
43 import unicodedata
43 import unicodedata
44 valid = []
44 valid = []
45 for c in range(0,0x10FFFF + 1):
45 for c in range(0,0x10FFFF + 1):
46 try:
46 try:
47 unicodedata.name(chr(c))
47 unicodedata.name(chr(c))
48 except ValueError:
48 except ValueError:
49 continue
49 continue
50 valid.append(c)
50 valid.append(c)
51
51
52 def ranges(i):
52 def ranges(i):
53 for a, b in itertools.groupby(enumerate(i), lambda pair: pair[1] - pair[0]):
53 for a, b in itertools.groupby(enumerate(i), lambda pair: pair[1] - pair[0]):
54 b = list(b)
54 b = list(b)
55 yield b[0][1], b[-1][1]
55 yield b[0][1], b[-1][1]
56
56
57 rg = list(ranges(valid))
57 rg = list(ranges(valid))
58 lens = []
58 lens = []
59 gap_lens = []
59 gap_lens = []
60 pstart, pstop = 0,0
60 pstart, pstop = 0,0
61 for start, stop in rg:
61 for start, stop in rg:
62 lens.append(stop-start)
62 lens.append(stop-start)
63 gap_lens.append((start - pstop, hex(pstop), hex(start), f'{round((start - pstop)/0xe01f0*100)}%'))
63 gap_lens.append((start - pstop, hex(pstop), hex(start), f'{round((start - pstop)/0xe01f0*100)}%'))
64 pstart, pstop = start, stop
64 pstart, pstop = start, stop
65
65
66 return sorted(gap_lens)[-1]
66 return sorted(gap_lens)[-1]
67
67
68
68
69
69
70 def test_unicode_range():
70 def test_unicode_range():
71 """
71 """
72 Test that the ranges we test for unicode names give the same number of
72 Test that the ranges we test for unicode names give the same number of
73 results than testing the full length.
73 results than testing the full length.
74 """
74 """
75 from IPython.core.completer import _unicode_name_compute, _UNICODE_RANGES
75 from IPython.core.completer import _unicode_name_compute, _UNICODE_RANGES
76
76
77 expected_list = _unicode_name_compute([(0, 0x110000)])
77 expected_list = _unicode_name_compute([(0, 0x110000)])
78 test = _unicode_name_compute(_UNICODE_RANGES)
78 test = _unicode_name_compute(_UNICODE_RANGES)
79 len_exp = len(expected_list)
79 len_exp = len(expected_list)
80 len_test = len(test)
80 len_test = len(test)
81
81
82 # do not inline the len() or on error pytest will try to print the 130 000 +
82 # do not inline the len() or on error pytest will try to print the 130 000 +
83 # elements.
83 # elements.
84 message = None
84 message = None
85 if len_exp != len_test or len_exp > 131808:
85 if len_exp != len_test or len_exp > 131808:
86 size, start, stop, prct = recompute_unicode_ranges()
86 size, start, stop, prct = recompute_unicode_ranges()
87 message = f"""_UNICODE_RANGES likely wrong and need updating. This is
87 message = f"""_UNICODE_RANGES likely wrong and need updating. This is
88 likely due to a new release of Python. We've find that the biggest gap
88 likely due to a new release of Python. We've find that the biggest gap
89 in unicode characters has reduces in size to be {size} charaters
89 in unicode characters has reduces in size to be {size} characters
90 ({prct}), from {start}, to {stop}. In completer.py likely update to
90 ({prct}), from {start}, to {stop}. In completer.py likely update to
91
91
92 _UNICODE_RANGES = [(32, {start}), ({stop}, 0xe01f0)]
92 _UNICODE_RANGES = [(32, {start}), ({stop}, 0xe01f0)]
93
93
94 And update the assertion below to use
94 And update the assertion below to use
95
95
96 len_exp <= {len_exp}
96 len_exp <= {len_exp}
97 """
97 """
98 assert len_exp == len_test, message
98 assert len_exp == len_test, message
99
99
100 # fail if new unicode symbols have been added.
100 # fail if new unicode symbols have been added.
101 assert len_exp <= 137714, message
101 assert len_exp <= 137714, message
102
102
103
103
104 @contextmanager
104 @contextmanager
105 def greedy_completion():
105 def greedy_completion():
106 ip = get_ipython()
106 ip = get_ipython()
107 greedy_original = ip.Completer.greedy
107 greedy_original = ip.Completer.greedy
108 try:
108 try:
109 ip.Completer.greedy = True
109 ip.Completer.greedy = True
110 yield
110 yield
111 finally:
111 finally:
112 ip.Completer.greedy = greedy_original
112 ip.Completer.greedy = greedy_original
113
113
114
114
115 def test_protect_filename():
115 def test_protect_filename():
116 if sys.platform == "win32":
116 if sys.platform == "win32":
117 pairs = [
117 pairs = [
118 ("abc", "abc"),
118 ("abc", "abc"),
119 (" abc", '" abc"'),
119 (" abc", '" abc"'),
120 ("a bc", '"a bc"'),
120 ("a bc", '"a bc"'),
121 ("a bc", '"a bc"'),
121 ("a bc", '"a bc"'),
122 (" bc", '" bc"'),
122 (" bc", '" bc"'),
123 ]
123 ]
124 else:
124 else:
125 pairs = [
125 pairs = [
126 ("abc", "abc"),
126 ("abc", "abc"),
127 (" abc", r"\ abc"),
127 (" abc", r"\ abc"),
128 ("a bc", r"a\ bc"),
128 ("a bc", r"a\ bc"),
129 ("a bc", r"a\ \ bc"),
129 ("a bc", r"a\ \ bc"),
130 (" bc", r"\ \ bc"),
130 (" bc", r"\ \ bc"),
131 # On posix, we also protect parens and other special characters.
131 # On posix, we also protect parens and other special characters.
132 ("a(bc", r"a\(bc"),
132 ("a(bc", r"a\(bc"),
133 ("a)bc", r"a\)bc"),
133 ("a)bc", r"a\)bc"),
134 ("a( )bc", r"a\(\ \)bc"),
134 ("a( )bc", r"a\(\ \)bc"),
135 ("a[1]bc", r"a\[1\]bc"),
135 ("a[1]bc", r"a\[1\]bc"),
136 ("a{1}bc", r"a\{1\}bc"),
136 ("a{1}bc", r"a\{1\}bc"),
137 ("a#bc", r"a\#bc"),
137 ("a#bc", r"a\#bc"),
138 ("a?bc", r"a\?bc"),
138 ("a?bc", r"a\?bc"),
139 ("a=bc", r"a\=bc"),
139 ("a=bc", r"a\=bc"),
140 ("a\\bc", r"a\\bc"),
140 ("a\\bc", r"a\\bc"),
141 ("a|bc", r"a\|bc"),
141 ("a|bc", r"a\|bc"),
142 ("a;bc", r"a\;bc"),
142 ("a;bc", r"a\;bc"),
143 ("a:bc", r"a\:bc"),
143 ("a:bc", r"a\:bc"),
144 ("a'bc", r"a\'bc"),
144 ("a'bc", r"a\'bc"),
145 ("a*bc", r"a\*bc"),
145 ("a*bc", r"a\*bc"),
146 ('a"bc', r"a\"bc"),
146 ('a"bc', r"a\"bc"),
147 ("a^bc", r"a\^bc"),
147 ("a^bc", r"a\^bc"),
148 ("a&bc", r"a\&bc"),
148 ("a&bc", r"a\&bc"),
149 ]
149 ]
150 # run the actual tests
150 # run the actual tests
151 for s1, s2 in pairs:
151 for s1, s2 in pairs:
152 s1p = completer.protect_filename(s1)
152 s1p = completer.protect_filename(s1)
153 nt.assert_equal(s1p, s2)
153 nt.assert_equal(s1p, s2)
154
154
155
155
156 def check_line_split(splitter, test_specs):
156 def check_line_split(splitter, test_specs):
157 for part1, part2, split in test_specs:
157 for part1, part2, split in test_specs:
158 cursor_pos = len(part1)
158 cursor_pos = len(part1)
159 line = part1 + part2
159 line = part1 + part2
160 out = splitter.split_line(line, cursor_pos)
160 out = splitter.split_line(line, cursor_pos)
161 nt.assert_equal(out, split)
161 nt.assert_equal(out, split)
162
162
163
163
164 def test_line_split():
164 def test_line_split():
165 """Basic line splitter test with default specs."""
165 """Basic line splitter test with default specs."""
166 sp = completer.CompletionSplitter()
166 sp = completer.CompletionSplitter()
167 # The format of the test specs is: part1, part2, expected answer. Parts 1
167 # The format of the test specs is: part1, part2, expected answer. Parts 1
168 # and 2 are joined into the 'line' sent to the splitter, as if the cursor
168 # and 2 are joined into the 'line' sent to the splitter, as if the cursor
169 # was at the end of part1. So an empty part2 represents someone hitting
169 # was at the end of part1. So an empty part2 represents someone hitting
170 # tab at the end of the line, the most common case.
170 # tab at the end of the line, the most common case.
171 t = [
171 t = [
172 ("run some/scrip", "", "some/scrip"),
172 ("run some/scrip", "", "some/scrip"),
173 ("run scripts/er", "ror.py foo", "scripts/er"),
173 ("run scripts/er", "ror.py foo", "scripts/er"),
174 ("echo $HOM", "", "HOM"),
174 ("echo $HOM", "", "HOM"),
175 ("print sys.pa", "", "sys.pa"),
175 ("print sys.pa", "", "sys.pa"),
176 ("print(sys.pa", "", "sys.pa"),
176 ("print(sys.pa", "", "sys.pa"),
177 ("execfile('scripts/er", "", "scripts/er"),
177 ("execfile('scripts/er", "", "scripts/er"),
178 ("a[x.", "", "x."),
178 ("a[x.", "", "x."),
179 ("a[x.", "y", "x."),
179 ("a[x.", "y", "x."),
180 ('cd "some_file/', "", "some_file/"),
180 ('cd "some_file/', "", "some_file/"),
181 ]
181 ]
182 check_line_split(sp, t)
182 check_line_split(sp, t)
183 # Ensure splitting works OK with unicode by re-running the tests with
183 # Ensure splitting works OK with unicode by re-running the tests with
184 # all inputs turned into unicode
184 # all inputs turned into unicode
185 check_line_split(sp, [map(str, p) for p in t])
185 check_line_split(sp, [map(str, p) for p in t])
186
186
187
187
188 class NamedInstanceMetaclass(type):
188 class NamedInstanceMetaclass(type):
189 def __getitem__(cls, item):
189 def __getitem__(cls, item):
190 return cls.get_instance(item)
190 return cls.get_instance(item)
191
191
192
192
193 class NamedInstanceClass(metaclass=NamedInstanceMetaclass):
193 class NamedInstanceClass(metaclass=NamedInstanceMetaclass):
194 def __init__(self, name):
194 def __init__(self, name):
195 if not hasattr(self.__class__, "instances"):
195 if not hasattr(self.__class__, "instances"):
196 self.__class__.instances = {}
196 self.__class__.instances = {}
197 self.__class__.instances[name] = self
197 self.__class__.instances[name] = self
198
198
199 @classmethod
199 @classmethod
200 def _ipython_key_completions_(cls):
200 def _ipython_key_completions_(cls):
201 return cls.instances.keys()
201 return cls.instances.keys()
202
202
203 @classmethod
203 @classmethod
204 def get_instance(cls, name):
204 def get_instance(cls, name):
205 return cls.instances[name]
205 return cls.instances[name]
206
206
207
207
208 class KeyCompletable:
208 class KeyCompletable:
209 def __init__(self, things=()):
209 def __init__(self, things=()):
210 self.things = things
210 self.things = things
211
211
212 def _ipython_key_completions_(self):
212 def _ipython_key_completions_(self):
213 return list(self.things)
213 return list(self.things)
214
214
215
215
216 class TestCompleter(unittest.TestCase):
216 class TestCompleter(unittest.TestCase):
217 def setUp(self):
217 def setUp(self):
218 """
218 """
219 We want to silence all PendingDeprecationWarning when testing the completer
219 We want to silence all PendingDeprecationWarning when testing the completer
220 """
220 """
221 self._assertwarns = self.assertWarns(PendingDeprecationWarning)
221 self._assertwarns = self.assertWarns(PendingDeprecationWarning)
222 self._assertwarns.__enter__()
222 self._assertwarns.__enter__()
223
223
224 def tearDown(self):
224 def tearDown(self):
225 try:
225 try:
226 self._assertwarns.__exit__(None, None, None)
226 self._assertwarns.__exit__(None, None, None)
227 except AssertionError:
227 except AssertionError:
228 pass
228 pass
229
229
230 def test_custom_completion_error(self):
230 def test_custom_completion_error(self):
231 """Test that errors from custom attribute completers are silenced."""
231 """Test that errors from custom attribute completers are silenced."""
232 ip = get_ipython()
232 ip = get_ipython()
233
233
234 class A:
234 class A:
235 pass
235 pass
236
236
237 ip.user_ns["x"] = A()
237 ip.user_ns["x"] = A()
238
238
239 @complete_object.register(A)
239 @complete_object.register(A)
240 def complete_A(a, existing_completions):
240 def complete_A(a, existing_completions):
241 raise TypeError("this should be silenced")
241 raise TypeError("this should be silenced")
242
242
243 ip.complete("x.")
243 ip.complete("x.")
244
244
245 def test_custom_completion_ordering(self):
245 def test_custom_completion_ordering(self):
246 """Test that errors from custom attribute completers are silenced."""
246 """Test that errors from custom attribute completers are silenced."""
247 ip = get_ipython()
247 ip = get_ipython()
248
248
249 _, matches = ip.complete('in')
249 _, matches = ip.complete('in')
250 assert matches.index('input') < matches.index('int')
250 assert matches.index('input') < matches.index('int')
251
251
252 def complete_example(a):
252 def complete_example(a):
253 return ['example2', 'example1']
253 return ['example2', 'example1']
254
254
255 ip.Completer.custom_completers.add_re('ex*', complete_example)
255 ip.Completer.custom_completers.add_re('ex*', complete_example)
256 _, matches = ip.complete('ex')
256 _, matches = ip.complete('ex')
257 assert matches.index('example2') < matches.index('example1')
257 assert matches.index('example2') < matches.index('example1')
258
258
259 def test_unicode_completions(self):
259 def test_unicode_completions(self):
260 ip = get_ipython()
260 ip = get_ipython()
261 # Some strings that trigger different types of completion. Check them both
261 # Some strings that trigger different types of completion. Check them both
262 # in str and unicode forms
262 # in str and unicode forms
263 s = ["ru", "%ru", "cd /", "floa", "float(x)/"]
263 s = ["ru", "%ru", "cd /", "floa", "float(x)/"]
264 for t in s + list(map(str, s)):
264 for t in s + list(map(str, s)):
265 # We don't need to check exact completion values (they may change
265 # We don't need to check exact completion values (they may change
266 # depending on the state of the namespace, but at least no exceptions
266 # depending on the state of the namespace, but at least no exceptions
267 # should be thrown and the return value should be a pair of text, list
267 # should be thrown and the return value should be a pair of text, list
268 # values.
268 # values.
269 text, matches = ip.complete(t)
269 text, matches = ip.complete(t)
270 nt.assert_true(isinstance(text, str))
270 nt.assert_true(isinstance(text, str))
271 nt.assert_true(isinstance(matches, list))
271 nt.assert_true(isinstance(matches, list))
272
272
273 def test_latex_completions(self):
273 def test_latex_completions(self):
274 from IPython.core.latex_symbols import latex_symbols
274 from IPython.core.latex_symbols import latex_symbols
275 import random
275 import random
276
276
277 ip = get_ipython()
277 ip = get_ipython()
278 # Test some random unicode symbols
278 # Test some random unicode symbols
279 keys = random.sample(latex_symbols.keys(), 10)
279 keys = random.sample(latex_symbols.keys(), 10)
280 for k in keys:
280 for k in keys:
281 text, matches = ip.complete(k)
281 text, matches = ip.complete(k)
282 nt.assert_equal(text, k)
282 nt.assert_equal(text, k)
283 nt.assert_equal(matches, [latex_symbols[k]])
283 nt.assert_equal(matches, [latex_symbols[k]])
284 # Test a more complex line
284 # Test a more complex line
285 text, matches = ip.complete("print(\\alpha")
285 text, matches = ip.complete("print(\\alpha")
286 nt.assert_equal(text, "\\alpha")
286 nt.assert_equal(text, "\\alpha")
287 nt.assert_equal(matches[0], latex_symbols["\\alpha"])
287 nt.assert_equal(matches[0], latex_symbols["\\alpha"])
288 # Test multiple matching latex symbols
288 # Test multiple matching latex symbols
289 text, matches = ip.complete("\\al")
289 text, matches = ip.complete("\\al")
290 nt.assert_in("\\alpha", matches)
290 nt.assert_in("\\alpha", matches)
291 nt.assert_in("\\aleph", matches)
291 nt.assert_in("\\aleph", matches)
292
292
293 def test_latex_no_results(self):
293 def test_latex_no_results(self):
294 """
294 """
295 forward latex should really return nothing in either field if nothing is found.
295 forward latex should really return nothing in either field if nothing is found.
296 """
296 """
297 ip = get_ipython()
297 ip = get_ipython()
298 text, matches = ip.Completer.latex_matches("\\really_i_should_match_nothing")
298 text, matches = ip.Completer.latex_matches("\\really_i_should_match_nothing")
299 nt.assert_equal(text, "")
299 nt.assert_equal(text, "")
300 nt.assert_equal(matches, ())
300 nt.assert_equal(matches, ())
301
301
302 def test_back_latex_completion(self):
302 def test_back_latex_completion(self):
303 ip = get_ipython()
303 ip = get_ipython()
304
304
305 # do not return more than 1 matches fro \beta, only the latex one.
305 # do not return more than 1 matches for \beta, only the latex one.
306 name, matches = ip.complete("\\β")
306 name, matches = ip.complete("\\β")
307 nt.assert_equal(matches, ['\\beta'])
307 nt.assert_equal(matches, ['\\beta'])
308
308
309 def test_back_unicode_completion(self):
309 def test_back_unicode_completion(self):
310 ip = get_ipython()
310 ip = get_ipython()
311
311
312 name, matches = ip.complete("\\Ⅴ")
312 name, matches = ip.complete("\\Ⅴ")
313 nt.assert_equal(matches, ("\\ROMAN NUMERAL FIVE",))
313 nt.assert_equal(matches, ("\\ROMAN NUMERAL FIVE",))
314
314
315 def test_forward_unicode_completion(self):
315 def test_forward_unicode_completion(self):
316 ip = get_ipython()
316 ip = get_ipython()
317
317
318 name, matches = ip.complete("\\ROMAN NUMERAL FIVE")
318 name, matches = ip.complete("\\ROMAN NUMERAL FIVE")
319 nt.assert_equal(matches, ["Ⅴ"] ) # This is not a V
319 nt.assert_equal(matches, ["Ⅴ"] ) # This is not a V
320 nt.assert_equal(matches, ["\u2164"] ) # same as above but explicit.
320 nt.assert_equal(matches, ["\u2164"] ) # same as above but explicit.
321
321
322 @nt.nottest # now we have a completion for \jmath
322 @nt.nottest # now we have a completion for \jmath
323 @decorators.knownfailureif(
323 @decorators.knownfailureif(
324 sys.platform == "win32", "Fails if there is a C:\\j... path"
324 sys.platform == "win32", "Fails if there is a C:\\j... path"
325 )
325 )
326 def test_no_ascii_back_completion(self):
326 def test_no_ascii_back_completion(self):
327 ip = get_ipython()
327 ip = get_ipython()
328 with TemporaryWorkingDirectory(): # Avoid any filename completions
328 with TemporaryWorkingDirectory(): # Avoid any filename completions
329 # single ascii letter that don't have yet completions
329 # single ascii letter that don't have yet completions
330 for letter in "jJ":
330 for letter in "jJ":
331 name, matches = ip.complete("\\" + letter)
331 name, matches = ip.complete("\\" + letter)
332 nt.assert_equal(matches, [])
332 nt.assert_equal(matches, [])
333
333
334 class CompletionSplitterTestCase(unittest.TestCase):
334 class CompletionSplitterTestCase(unittest.TestCase):
335 def setUp(self):
335 def setUp(self):
336 self.sp = completer.CompletionSplitter()
336 self.sp = completer.CompletionSplitter()
337
337
338 def test_delim_setting(self):
338 def test_delim_setting(self):
339 self.sp.delims = " "
339 self.sp.delims = " "
340 nt.assert_equal(self.sp.delims, " ")
340 nt.assert_equal(self.sp.delims, " ")
341 nt.assert_equal(self.sp._delim_expr, r"[\ ]")
341 nt.assert_equal(self.sp._delim_expr, r"[\ ]")
342
342
343 def test_spaces(self):
343 def test_spaces(self):
344 """Test with only spaces as split chars."""
344 """Test with only spaces as split chars."""
345 self.sp.delims = " "
345 self.sp.delims = " "
346 t = [("foo", "", "foo"), ("run foo", "", "foo"), ("run foo", "bar", "foo")]
346 t = [("foo", "", "foo"), ("run foo", "", "foo"), ("run foo", "bar", "foo")]
347 check_line_split(self.sp, t)
347 check_line_split(self.sp, t)
348
348
349 def test_has_open_quotes1(self):
349 def test_has_open_quotes1(self):
350 for s in ["'", "'''", "'hi' '"]:
350 for s in ["'", "'''", "'hi' '"]:
351 nt.assert_equal(completer.has_open_quotes(s), "'")
351 nt.assert_equal(completer.has_open_quotes(s), "'")
352
352
353 def test_has_open_quotes2(self):
353 def test_has_open_quotes2(self):
354 for s in ['"', '"""', '"hi" "']:
354 for s in ['"', '"""', '"hi" "']:
355 nt.assert_equal(completer.has_open_quotes(s), '"')
355 nt.assert_equal(completer.has_open_quotes(s), '"')
356
356
357 def test_has_open_quotes3(self):
357 def test_has_open_quotes3(self):
358 for s in ["''", "''' '''", "'hi' 'ipython'"]:
358 for s in ["''", "''' '''", "'hi' 'ipython'"]:
359 nt.assert_false(completer.has_open_quotes(s))
359 nt.assert_false(completer.has_open_quotes(s))
360
360
361 def test_has_open_quotes4(self):
361 def test_has_open_quotes4(self):
362 for s in ['""', '""" """', '"hi" "ipython"']:
362 for s in ['""', '""" """', '"hi" "ipython"']:
363 nt.assert_false(completer.has_open_quotes(s))
363 nt.assert_false(completer.has_open_quotes(s))
364
364
365 @decorators.knownfailureif(
365 @decorators.knownfailureif(
366 sys.platform == "win32", "abspath completions fail on Windows"
366 sys.platform == "win32", "abspath completions fail on Windows"
367 )
367 )
368 def test_abspath_file_completions(self):
368 def test_abspath_file_completions(self):
369 ip = get_ipython()
369 ip = get_ipython()
370 with TemporaryDirectory() as tmpdir:
370 with TemporaryDirectory() as tmpdir:
371 prefix = os.path.join(tmpdir, "foo")
371 prefix = os.path.join(tmpdir, "foo")
372 suffixes = ["1", "2"]
372 suffixes = ["1", "2"]
373 names = [prefix + s for s in suffixes]
373 names = [prefix + s for s in suffixes]
374 for n in names:
374 for n in names:
375 open(n, "w").close()
375 open(n, "w").close()
376
376
377 # Check simple completion
377 # Check simple completion
378 c = ip.complete(prefix)[1]
378 c = ip.complete(prefix)[1]
379 nt.assert_equal(c, names)
379 nt.assert_equal(c, names)
380
380
381 # Now check with a function call
381 # Now check with a function call
382 cmd = 'a = f("%s' % prefix
382 cmd = 'a = f("%s' % prefix
383 c = ip.complete(prefix, cmd)[1]
383 c = ip.complete(prefix, cmd)[1]
384 comp = [prefix + s for s in suffixes]
384 comp = [prefix + s for s in suffixes]
385 nt.assert_equal(c, comp)
385 nt.assert_equal(c, comp)
386
386
387 def test_local_file_completions(self):
387 def test_local_file_completions(self):
388 ip = get_ipython()
388 ip = get_ipython()
389 with TemporaryWorkingDirectory():
389 with TemporaryWorkingDirectory():
390 prefix = "./foo"
390 prefix = "./foo"
391 suffixes = ["1", "2"]
391 suffixes = ["1", "2"]
392 names = [prefix + s for s in suffixes]
392 names = [prefix + s for s in suffixes]
393 for n in names:
393 for n in names:
394 open(n, "w").close()
394 open(n, "w").close()
395
395
396 # Check simple completion
396 # Check simple completion
397 c = ip.complete(prefix)[1]
397 c = ip.complete(prefix)[1]
398 nt.assert_equal(c, names)
398 nt.assert_equal(c, names)
399
399
400 # Now check with a function call
400 # Now check with a function call
401 cmd = 'a = f("%s' % prefix
401 cmd = 'a = f("%s' % prefix
402 c = ip.complete(prefix, cmd)[1]
402 c = ip.complete(prefix, cmd)[1]
403 comp = {prefix + s for s in suffixes}
403 comp = {prefix + s for s in suffixes}
404 nt.assert_true(comp.issubset(set(c)))
404 nt.assert_true(comp.issubset(set(c)))
405
405
406 def test_quoted_file_completions(self):
406 def test_quoted_file_completions(self):
407 ip = get_ipython()
407 ip = get_ipython()
408 with TemporaryWorkingDirectory():
408 with TemporaryWorkingDirectory():
409 name = "foo'bar"
409 name = "foo'bar"
410 open(name, "w").close()
410 open(name, "w").close()
411
411
412 # Don't escape Windows
412 # Don't escape Windows
413 escaped = name if sys.platform == "win32" else "foo\\'bar"
413 escaped = name if sys.platform == "win32" else "foo\\'bar"
414
414
415 # Single quote matches embedded single quote
415 # Single quote matches embedded single quote
416 text = "open('foo"
416 text = "open('foo"
417 c = ip.Completer._complete(
417 c = ip.Completer._complete(
418 cursor_line=0, cursor_pos=len(text), full_text=text
418 cursor_line=0, cursor_pos=len(text), full_text=text
419 )[1]
419 )[1]
420 nt.assert_equal(c, [escaped])
420 nt.assert_equal(c, [escaped])
421
421
422 # Double quote requires no escape
422 # Double quote requires no escape
423 text = 'open("foo'
423 text = 'open("foo'
424 c = ip.Completer._complete(
424 c = ip.Completer._complete(
425 cursor_line=0, cursor_pos=len(text), full_text=text
425 cursor_line=0, cursor_pos=len(text), full_text=text
426 )[1]
426 )[1]
427 nt.assert_equal(c, [name])
427 nt.assert_equal(c, [name])
428
428
429 # No quote requires an escape
429 # No quote requires an escape
430 text = "%ls foo"
430 text = "%ls foo"
431 c = ip.Completer._complete(
431 c = ip.Completer._complete(
432 cursor_line=0, cursor_pos=len(text), full_text=text
432 cursor_line=0, cursor_pos=len(text), full_text=text
433 )[1]
433 )[1]
434 nt.assert_equal(c, [escaped])
434 nt.assert_equal(c, [escaped])
435
435
436 def test_all_completions_dups(self):
436 def test_all_completions_dups(self):
437 """
437 """
438 Make sure the output of `IPCompleter.all_completions` does not have
438 Make sure the output of `IPCompleter.all_completions` does not have
439 duplicated prefixes.
439 duplicated prefixes.
440 """
440 """
441 ip = get_ipython()
441 ip = get_ipython()
442 c = ip.Completer
442 c = ip.Completer
443 ip.ex("class TestClass():\n\ta=1\n\ta1=2")
443 ip.ex("class TestClass():\n\ta=1\n\ta1=2")
444 for jedi_status in [True, False]:
444 for jedi_status in [True, False]:
445 with provisionalcompleter():
445 with provisionalcompleter():
446 ip.Completer.use_jedi = jedi_status
446 ip.Completer.use_jedi = jedi_status
447 matches = c.all_completions("TestCl")
447 matches = c.all_completions("TestCl")
448 assert matches == ['TestClass'], jedi_status
448 assert matches == ['TestClass'], jedi_status
449 matches = c.all_completions("TestClass.")
449 matches = c.all_completions("TestClass.")
450 assert len(matches) > 2, jedi_status
450 assert len(matches) > 2, jedi_status
451 matches = c.all_completions("TestClass.a")
451 matches = c.all_completions("TestClass.a")
452 assert matches == ['TestClass.a', 'TestClass.a1'], jedi_status
452 assert matches == ['TestClass.a', 'TestClass.a1'], jedi_status
453
453
454 def test_jedi(self):
454 def test_jedi(self):
455 """
455 """
456 A couple of issue we had with Jedi
456 A couple of issue we had with Jedi
457 """
457 """
458 ip = get_ipython()
458 ip = get_ipython()
459
459
460 def _test_complete(reason, s, comp, start=None, end=None):
460 def _test_complete(reason, s, comp, start=None, end=None):
461 l = len(s)
461 l = len(s)
462 start = start if start is not None else l
462 start = start if start is not None else l
463 end = end if end is not None else l
463 end = end if end is not None else l
464 with provisionalcompleter():
464 with provisionalcompleter():
465 ip.Completer.use_jedi = True
465 ip.Completer.use_jedi = True
466 completions = set(ip.Completer.completions(s, l))
466 completions = set(ip.Completer.completions(s, l))
467 ip.Completer.use_jedi = False
467 ip.Completer.use_jedi = False
468 assert_in(Completion(start, end, comp), completions, reason)
468 assert_in(Completion(start, end, comp), completions, reason)
469
469
470 def _test_not_complete(reason, s, comp):
470 def _test_not_complete(reason, s, comp):
471 l = len(s)
471 l = len(s)
472 with provisionalcompleter():
472 with provisionalcompleter():
473 ip.Completer.use_jedi = True
473 ip.Completer.use_jedi = True
474 completions = set(ip.Completer.completions(s, l))
474 completions = set(ip.Completer.completions(s, l))
475 ip.Completer.use_jedi = False
475 ip.Completer.use_jedi = False
476 assert_not_in(Completion(l, l, comp), completions, reason)
476 assert_not_in(Completion(l, l, comp), completions, reason)
477
477
478 import jedi
478 import jedi
479
479
480 jedi_version = tuple(int(i) for i in jedi.__version__.split(".")[:3])
480 jedi_version = tuple(int(i) for i in jedi.__version__.split(".")[:3])
481 if jedi_version > (0, 10):
481 if jedi_version > (0, 10):
482 yield _test_complete, "jedi >0.9 should complete and not crash", "a=1;a.", "real"
482 yield _test_complete, "jedi >0.9 should complete and not crash", "a=1;a.", "real"
483 yield _test_complete, "can infer first argument", 'a=(1,"foo");a[0].', "real"
483 yield _test_complete, "can infer first argument", 'a=(1,"foo");a[0].', "real"
484 yield _test_complete, "can infer second argument", 'a=(1,"foo");a[1].', "capitalize"
484 yield _test_complete, "can infer second argument", 'a=(1,"foo");a[1].', "capitalize"
485 yield _test_complete, "cover duplicate completions", "im", "import", 0, 2
485 yield _test_complete, "cover duplicate completions", "im", "import", 0, 2
486
486
487 yield _test_not_complete, "does not mix types", 'a=(1,"foo");a[0].', "capitalize"
487 yield _test_not_complete, "does not mix types", 'a=(1,"foo");a[0].', "capitalize"
488
488
489 def test_completion_have_signature(self):
489 def test_completion_have_signature(self):
490 """
490 """
491 Lets make sure jedi is capable of pulling out the signature of the function we are completing.
491 Lets make sure jedi is capable of pulling out the signature of the function we are completing.
492 """
492 """
493 ip = get_ipython()
493 ip = get_ipython()
494 with provisionalcompleter():
494 with provisionalcompleter():
495 ip.Completer.use_jedi = True
495 ip.Completer.use_jedi = True
496 completions = ip.Completer.completions("ope", 3)
496 completions = ip.Completer.completions("ope", 3)
497 c = next(completions) # should be `open`
497 c = next(completions) # should be `open`
498 ip.Completer.use_jedi = False
498 ip.Completer.use_jedi = False
499 assert "file" in c.signature, "Signature of function was not found by completer"
499 assert "file" in c.signature, "Signature of function was not found by completer"
500 assert (
500 assert (
501 "encoding" in c.signature
501 "encoding" in c.signature
502 ), "Signature of function was not found by completer"
502 ), "Signature of function was not found by completer"
503
503
504 def test_deduplicate_completions(self):
504 def test_deduplicate_completions(self):
505 """
505 """
506 Test that completions are correctly deduplicated (even if ranges are not the same)
506 Test that completions are correctly deduplicated (even if ranges are not the same)
507 """
507 """
508 ip = get_ipython()
508 ip = get_ipython()
509 ip.ex(
509 ip.ex(
510 textwrap.dedent(
510 textwrap.dedent(
511 """
511 """
512 class Z:
512 class Z:
513 zoo = 1
513 zoo = 1
514 """
514 """
515 )
515 )
516 )
516 )
517 with provisionalcompleter():
517 with provisionalcompleter():
518 ip.Completer.use_jedi = True
518 ip.Completer.use_jedi = True
519 l = list(
519 l = list(
520 _deduplicate_completions("Z.z", ip.Completer.completions("Z.z", 3))
520 _deduplicate_completions("Z.z", ip.Completer.completions("Z.z", 3))
521 )
521 )
522 ip.Completer.use_jedi = False
522 ip.Completer.use_jedi = False
523
523
524 assert len(l) == 1, "Completions (Z.z<tab>) correctly deduplicate: %s " % l
524 assert len(l) == 1, "Completions (Z.z<tab>) correctly deduplicate: %s " % l
525 assert l[0].text == "zoo" # and not `it.accumulate`
525 assert l[0].text == "zoo" # and not `it.accumulate`
526
526
527 def test_greedy_completions(self):
527 def test_greedy_completions(self):
528 """
528 """
529 Test the capability of the Greedy completer.
529 Test the capability of the Greedy completer.
530
530
531 Most of the test here does not really show off the greedy completer, for proof
531 Most of the test here does not really show off the greedy completer, for proof
532 each of the text below now pass with Jedi. The greedy completer is capable of more.
532 each of the text below now pass with Jedi. The greedy completer is capable of more.
533
533
534 See the :any:`test_dict_key_completion_contexts`
534 See the :any:`test_dict_key_completion_contexts`
535
535
536 """
536 """
537 ip = get_ipython()
537 ip = get_ipython()
538 ip.ex("a=list(range(5))")
538 ip.ex("a=list(range(5))")
539 _, c = ip.complete(".", line="a[0].")
539 _, c = ip.complete(".", line="a[0].")
540 nt.assert_false(".real" in c, "Shouldn't have completed on a[0]: %s" % c)
540 nt.assert_false(".real" in c, "Shouldn't have completed on a[0]: %s" % c)
541
541
542 def _(line, cursor_pos, expect, message, completion):
542 def _(line, cursor_pos, expect, message, completion):
543 with greedy_completion(), provisionalcompleter():
543 with greedy_completion(), provisionalcompleter():
544 ip.Completer.use_jedi = False
544 ip.Completer.use_jedi = False
545 _, c = ip.complete(".", line=line, cursor_pos=cursor_pos)
545 _, c = ip.complete(".", line=line, cursor_pos=cursor_pos)
546 nt.assert_in(expect, c, message % c)
546 nt.assert_in(expect, c, message % c)
547
547
548 ip.Completer.use_jedi = True
548 ip.Completer.use_jedi = True
549 with provisionalcompleter():
549 with provisionalcompleter():
550 completions = ip.Completer.completions(line, cursor_pos)
550 completions = ip.Completer.completions(line, cursor_pos)
551 nt.assert_in(completion, completions)
551 nt.assert_in(completion, completions)
552
552
553 with provisionalcompleter():
553 with provisionalcompleter():
554 yield _, "a[0].", 5, "a[0].real", "Should have completed on a[0].: %s", Completion(
554 yield _, "a[0].", 5, "a[0].real", "Should have completed on a[0].: %s", Completion(
555 5, 5, "real"
555 5, 5, "real"
556 )
556 )
557 yield _, "a[0].r", 6, "a[0].real", "Should have completed on a[0].r: %s", Completion(
557 yield _, "a[0].r", 6, "a[0].real", "Should have completed on a[0].r: %s", Completion(
558 5, 6, "real"
558 5, 6, "real"
559 )
559 )
560
560
561 yield _, "a[0].from_", 10, "a[0].from_bytes", "Should have completed on a[0].from_: %s", Completion(
561 yield _, "a[0].from_", 10, "a[0].from_bytes", "Should have completed on a[0].from_: %s", Completion(
562 5, 10, "from_bytes"
562 5, 10, "from_bytes"
563 )
563 )
564
564
565 def test_omit__names(self):
565 def test_omit__names(self):
566 # also happens to test IPCompleter as a configurable
566 # also happens to test IPCompleter as a configurable
567 ip = get_ipython()
567 ip = get_ipython()
568 ip._hidden_attr = 1
568 ip._hidden_attr = 1
569 ip._x = {}
569 ip._x = {}
570 c = ip.Completer
570 c = ip.Completer
571 ip.ex("ip=get_ipython()")
571 ip.ex("ip=get_ipython()")
572 cfg = Config()
572 cfg = Config()
573 cfg.IPCompleter.omit__names = 0
573 cfg.IPCompleter.omit__names = 0
574 c.update_config(cfg)
574 c.update_config(cfg)
575 with provisionalcompleter():
575 with provisionalcompleter():
576 c.use_jedi = False
576 c.use_jedi = False
577 s, matches = c.complete("ip.")
577 s, matches = c.complete("ip.")
578 nt.assert_in("ip.__str__", matches)
578 nt.assert_in("ip.__str__", matches)
579 nt.assert_in("ip._hidden_attr", matches)
579 nt.assert_in("ip._hidden_attr", matches)
580
580
581 # c.use_jedi = True
581 # c.use_jedi = True
582 # completions = set(c.completions('ip.', 3))
582 # completions = set(c.completions('ip.', 3))
583 # nt.assert_in(Completion(3, 3, '__str__'), completions)
583 # nt.assert_in(Completion(3, 3, '__str__'), completions)
584 # nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
584 # nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
585
585
586 cfg = Config()
586 cfg = Config()
587 cfg.IPCompleter.omit__names = 1
587 cfg.IPCompleter.omit__names = 1
588 c.update_config(cfg)
588 c.update_config(cfg)
589 with provisionalcompleter():
589 with provisionalcompleter():
590 c.use_jedi = False
590 c.use_jedi = False
591 s, matches = c.complete("ip.")
591 s, matches = c.complete("ip.")
592 nt.assert_not_in("ip.__str__", matches)
592 nt.assert_not_in("ip.__str__", matches)
593 # nt.assert_in('ip._hidden_attr', matches)
593 # nt.assert_in('ip._hidden_attr', matches)
594
594
595 # c.use_jedi = True
595 # c.use_jedi = True
596 # completions = set(c.completions('ip.', 3))
596 # completions = set(c.completions('ip.', 3))
597 # nt.assert_not_in(Completion(3,3,'__str__'), completions)
597 # nt.assert_not_in(Completion(3,3,'__str__'), completions)
598 # nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
598 # nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
599
599
600 cfg = Config()
600 cfg = Config()
601 cfg.IPCompleter.omit__names = 2
601 cfg.IPCompleter.omit__names = 2
602 c.update_config(cfg)
602 c.update_config(cfg)
603 with provisionalcompleter():
603 with provisionalcompleter():
604 c.use_jedi = False
604 c.use_jedi = False
605 s, matches = c.complete("ip.")
605 s, matches = c.complete("ip.")
606 nt.assert_not_in("ip.__str__", matches)
606 nt.assert_not_in("ip.__str__", matches)
607 nt.assert_not_in("ip._hidden_attr", matches)
607 nt.assert_not_in("ip._hidden_attr", matches)
608
608
609 # c.use_jedi = True
609 # c.use_jedi = True
610 # completions = set(c.completions('ip.', 3))
610 # completions = set(c.completions('ip.', 3))
611 # nt.assert_not_in(Completion(3,3,'__str__'), completions)
611 # nt.assert_not_in(Completion(3,3,'__str__'), completions)
612 # nt.assert_not_in(Completion(3,3, "_hidden_attr"), completions)
612 # nt.assert_not_in(Completion(3,3, "_hidden_attr"), completions)
613
613
614 with provisionalcompleter():
614 with provisionalcompleter():
615 c.use_jedi = False
615 c.use_jedi = False
616 s, matches = c.complete("ip._x.")
616 s, matches = c.complete("ip._x.")
617 nt.assert_in("ip._x.keys", matches)
617 nt.assert_in("ip._x.keys", matches)
618
618
619 # c.use_jedi = True
619 # c.use_jedi = True
620 # completions = set(c.completions('ip._x.', 6))
620 # completions = set(c.completions('ip._x.', 6))
621 # nt.assert_in(Completion(6,6, "keys"), completions)
621 # nt.assert_in(Completion(6,6, "keys"), completions)
622
622
623 del ip._hidden_attr
623 del ip._hidden_attr
624 del ip._x
624 del ip._x
625
625
626 def test_limit_to__all__False_ok(self):
626 def test_limit_to__all__False_ok(self):
627 """
627 """
628 Limit to all is deprecated, once we remove it this test can go away.
628 Limit to all is deprecated, once we remove it this test can go away.
629 """
629 """
630 ip = get_ipython()
630 ip = get_ipython()
631 c = ip.Completer
631 c = ip.Completer
632 c.use_jedi = False
632 c.use_jedi = False
633 ip.ex("class D: x=24")
633 ip.ex("class D: x=24")
634 ip.ex("d=D()")
634 ip.ex("d=D()")
635 cfg = Config()
635 cfg = Config()
636 cfg.IPCompleter.limit_to__all__ = False
636 cfg.IPCompleter.limit_to__all__ = False
637 c.update_config(cfg)
637 c.update_config(cfg)
638 s, matches = c.complete("d.")
638 s, matches = c.complete("d.")
639 nt.assert_in("d.x", matches)
639 nt.assert_in("d.x", matches)
640
640
641 def test_get__all__entries_ok(self):
641 def test_get__all__entries_ok(self):
642 class A:
642 class A:
643 __all__ = ["x", 1]
643 __all__ = ["x", 1]
644
644
645 words = completer.get__all__entries(A())
645 words = completer.get__all__entries(A())
646 nt.assert_equal(words, ["x"])
646 nt.assert_equal(words, ["x"])
647
647
648 def test_get__all__entries_no__all__ok(self):
648 def test_get__all__entries_no__all__ok(self):
649 class A:
649 class A:
650 pass
650 pass
651
651
652 words = completer.get__all__entries(A())
652 words = completer.get__all__entries(A())
653 nt.assert_equal(words, [])
653 nt.assert_equal(words, [])
654
654
655 def test_func_kw_completions(self):
655 def test_func_kw_completions(self):
656 ip = get_ipython()
656 ip = get_ipython()
657 c = ip.Completer
657 c = ip.Completer
658 c.use_jedi = False
658 c.use_jedi = False
659 ip.ex("def myfunc(a=1,b=2): return a+b")
659 ip.ex("def myfunc(a=1,b=2): return a+b")
660 s, matches = c.complete(None, "myfunc(1,b")
660 s, matches = c.complete(None, "myfunc(1,b")
661 nt.assert_in("b=", matches)
661 nt.assert_in("b=", matches)
662 # Simulate completing with cursor right after b (pos==10):
662 # Simulate completing with cursor right after b (pos==10):
663 s, matches = c.complete(None, "myfunc(1,b)", 10)
663 s, matches = c.complete(None, "myfunc(1,b)", 10)
664 nt.assert_in("b=", matches)
664 nt.assert_in("b=", matches)
665 s, matches = c.complete(None, 'myfunc(a="escaped\\")string",b')
665 s, matches = c.complete(None, 'myfunc(a="escaped\\")string",b')
666 nt.assert_in("b=", matches)
666 nt.assert_in("b=", matches)
667 # builtin function
667 # builtin function
668 s, matches = c.complete(None, "min(k, k")
668 s, matches = c.complete(None, "min(k, k")
669 nt.assert_in("key=", matches)
669 nt.assert_in("key=", matches)
670
670
671 def test_default_arguments_from_docstring(self):
671 def test_default_arguments_from_docstring(self):
672 ip = get_ipython()
672 ip = get_ipython()
673 c = ip.Completer
673 c = ip.Completer
674 kwd = c._default_arguments_from_docstring("min(iterable[, key=func]) -> value")
674 kwd = c._default_arguments_from_docstring("min(iterable[, key=func]) -> value")
675 nt.assert_equal(kwd, ["key"])
675 nt.assert_equal(kwd, ["key"])
676 # with cython type etc
676 # with cython type etc
677 kwd = c._default_arguments_from_docstring(
677 kwd = c._default_arguments_from_docstring(
678 "Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n"
678 "Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n"
679 )
679 )
680 nt.assert_equal(kwd, ["ncall", "resume", "nsplit"])
680 nt.assert_equal(kwd, ["ncall", "resume", "nsplit"])
681 # white spaces
681 # white spaces
682 kwd = c._default_arguments_from_docstring(
682 kwd = c._default_arguments_from_docstring(
683 "\n Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n"
683 "\n Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n"
684 )
684 )
685 nt.assert_equal(kwd, ["ncall", "resume", "nsplit"])
685 nt.assert_equal(kwd, ["ncall", "resume", "nsplit"])
686
686
687 def test_line_magics(self):
687 def test_line_magics(self):
688 ip = get_ipython()
688 ip = get_ipython()
689 c = ip.Completer
689 c = ip.Completer
690 s, matches = c.complete(None, "lsmag")
690 s, matches = c.complete(None, "lsmag")
691 nt.assert_in("%lsmagic", matches)
691 nt.assert_in("%lsmagic", matches)
692 s, matches = c.complete(None, "%lsmag")
692 s, matches = c.complete(None, "%lsmag")
693 nt.assert_in("%lsmagic", matches)
693 nt.assert_in("%lsmagic", matches)
694
694
695 def test_cell_magics(self):
695 def test_cell_magics(self):
696 from IPython.core.magic import register_cell_magic
696 from IPython.core.magic import register_cell_magic
697
697
698 @register_cell_magic
698 @register_cell_magic
699 def _foo_cellm(line, cell):
699 def _foo_cellm(line, cell):
700 pass
700 pass
701
701
702 ip = get_ipython()
702 ip = get_ipython()
703 c = ip.Completer
703 c = ip.Completer
704
704
705 s, matches = c.complete(None, "_foo_ce")
705 s, matches = c.complete(None, "_foo_ce")
706 nt.assert_in("%%_foo_cellm", matches)
706 nt.assert_in("%%_foo_cellm", matches)
707 s, matches = c.complete(None, "%%_foo_ce")
707 s, matches = c.complete(None, "%%_foo_ce")
708 nt.assert_in("%%_foo_cellm", matches)
708 nt.assert_in("%%_foo_cellm", matches)
709
709
710 def test_line_cell_magics(self):
710 def test_line_cell_magics(self):
711 from IPython.core.magic import register_line_cell_magic
711 from IPython.core.magic import register_line_cell_magic
712
712
713 @register_line_cell_magic
713 @register_line_cell_magic
714 def _bar_cellm(line, cell):
714 def _bar_cellm(line, cell):
715 pass
715 pass
716
716
717 ip = get_ipython()
717 ip = get_ipython()
718 c = ip.Completer
718 c = ip.Completer
719
719
720 # The policy here is trickier, see comments in completion code. The
720 # The policy here is trickier, see comments in completion code. The
721 # returned values depend on whether the user passes %% or not explicitly,
721 # returned values depend on whether the user passes %% or not explicitly,
722 # and this will show a difference if the same name is both a line and cell
722 # and this will show a difference if the same name is both a line and cell
723 # magic.
723 # magic.
724 s, matches = c.complete(None, "_bar_ce")
724 s, matches = c.complete(None, "_bar_ce")
725 nt.assert_in("%_bar_cellm", matches)
725 nt.assert_in("%_bar_cellm", matches)
726 nt.assert_in("%%_bar_cellm", matches)
726 nt.assert_in("%%_bar_cellm", matches)
727 s, matches = c.complete(None, "%_bar_ce")
727 s, matches = c.complete(None, "%_bar_ce")
728 nt.assert_in("%_bar_cellm", matches)
728 nt.assert_in("%_bar_cellm", matches)
729 nt.assert_in("%%_bar_cellm", matches)
729 nt.assert_in("%%_bar_cellm", matches)
730 s, matches = c.complete(None, "%%_bar_ce")
730 s, matches = c.complete(None, "%%_bar_ce")
731 nt.assert_not_in("%_bar_cellm", matches)
731 nt.assert_not_in("%_bar_cellm", matches)
732 nt.assert_in("%%_bar_cellm", matches)
732 nt.assert_in("%%_bar_cellm", matches)
733
733
734 def test_magic_completion_order(self):
734 def test_magic_completion_order(self):
735 ip = get_ipython()
735 ip = get_ipython()
736 c = ip.Completer
736 c = ip.Completer
737
737
738 # Test ordering of line and cell magics.
738 # Test ordering of line and cell magics.
739 text, matches = c.complete("timeit")
739 text, matches = c.complete("timeit")
740 nt.assert_equal(matches, ["%timeit", "%%timeit"])
740 nt.assert_equal(matches, ["%timeit", "%%timeit"])
741
741
742 def test_magic_completion_shadowing(self):
742 def test_magic_completion_shadowing(self):
743 ip = get_ipython()
743 ip = get_ipython()
744 c = ip.Completer
744 c = ip.Completer
745 c.use_jedi = False
745 c.use_jedi = False
746
746
747 # Before importing matplotlib, %matplotlib magic should be the only option.
747 # Before importing matplotlib, %matplotlib magic should be the only option.
748 text, matches = c.complete("mat")
748 text, matches = c.complete("mat")
749 nt.assert_equal(matches, ["%matplotlib"])
749 nt.assert_equal(matches, ["%matplotlib"])
750
750
751 # The newly introduced name should shadow the magic.
751 # The newly introduced name should shadow the magic.
752 ip.run_cell("matplotlib = 1")
752 ip.run_cell("matplotlib = 1")
753 text, matches = c.complete("mat")
753 text, matches = c.complete("mat")
754 nt.assert_equal(matches, ["matplotlib"])
754 nt.assert_equal(matches, ["matplotlib"])
755
755
756 # After removing matplotlib from namespace, the magic should again be
756 # After removing matplotlib from namespace, the magic should again be
757 # the only option.
757 # the only option.
758 del ip.user_ns["matplotlib"]
758 del ip.user_ns["matplotlib"]
759 text, matches = c.complete("mat")
759 text, matches = c.complete("mat")
760 nt.assert_equal(matches, ["%matplotlib"])
760 nt.assert_equal(matches, ["%matplotlib"])
761
761
762 def test_magic_completion_shadowing_explicit(self):
762 def test_magic_completion_shadowing_explicit(self):
763 """
763 """
764 If the user try to complete a shadowed magic, and explicit % start should
764 If the user try to complete a shadowed magic, and explicit % start should
765 still return the completions.
765 still return the completions.
766 """
766 """
767 ip = get_ipython()
767 ip = get_ipython()
768 c = ip.Completer
768 c = ip.Completer
769
769
770 # Before importing matplotlib, %matplotlib magic should be the only option.
770 # Before importing matplotlib, %matplotlib magic should be the only option.
771 text, matches = c.complete("%mat")
771 text, matches = c.complete("%mat")
772 nt.assert_equal(matches, ["%matplotlib"])
772 nt.assert_equal(matches, ["%matplotlib"])
773
773
774 ip.run_cell("matplotlib = 1")
774 ip.run_cell("matplotlib = 1")
775
775
776 # After removing matplotlib from namespace, the magic should still be
776 # After removing matplotlib from namespace, the magic should still be
777 # the only option.
777 # the only option.
778 text, matches = c.complete("%mat")
778 text, matches = c.complete("%mat")
779 nt.assert_equal(matches, ["%matplotlib"])
779 nt.assert_equal(matches, ["%matplotlib"])
780
780
781 def test_magic_config(self):
781 def test_magic_config(self):
782 ip = get_ipython()
782 ip = get_ipython()
783 c = ip.Completer
783 c = ip.Completer
784
784
785 s, matches = c.complete(None, "conf")
785 s, matches = c.complete(None, "conf")
786 nt.assert_in("%config", matches)
786 nt.assert_in("%config", matches)
787 s, matches = c.complete(None, "conf")
787 s, matches = c.complete(None, "conf")
788 nt.assert_not_in("AliasManager", matches)
788 nt.assert_not_in("AliasManager", matches)
789 s, matches = c.complete(None, "config ")
789 s, matches = c.complete(None, "config ")
790 nt.assert_in("AliasManager", matches)
790 nt.assert_in("AliasManager", matches)
791 s, matches = c.complete(None, "%config ")
791 s, matches = c.complete(None, "%config ")
792 nt.assert_in("AliasManager", matches)
792 nt.assert_in("AliasManager", matches)
793 s, matches = c.complete(None, "config Ali")
793 s, matches = c.complete(None, "config Ali")
794 nt.assert_list_equal(["AliasManager"], matches)
794 nt.assert_list_equal(["AliasManager"], matches)
795 s, matches = c.complete(None, "%config Ali")
795 s, matches = c.complete(None, "%config Ali")
796 nt.assert_list_equal(["AliasManager"], matches)
796 nt.assert_list_equal(["AliasManager"], matches)
797 s, matches = c.complete(None, "config AliasManager")
797 s, matches = c.complete(None, "config AliasManager")
798 nt.assert_list_equal(["AliasManager"], matches)
798 nt.assert_list_equal(["AliasManager"], matches)
799 s, matches = c.complete(None, "%config AliasManager")
799 s, matches = c.complete(None, "%config AliasManager")
800 nt.assert_list_equal(["AliasManager"], matches)
800 nt.assert_list_equal(["AliasManager"], matches)
801 s, matches = c.complete(None, "config AliasManager.")
801 s, matches = c.complete(None, "config AliasManager.")
802 nt.assert_in("AliasManager.default_aliases", matches)
802 nt.assert_in("AliasManager.default_aliases", matches)
803 s, matches = c.complete(None, "%config AliasManager.")
803 s, matches = c.complete(None, "%config AliasManager.")
804 nt.assert_in("AliasManager.default_aliases", matches)
804 nt.assert_in("AliasManager.default_aliases", matches)
805 s, matches = c.complete(None, "config AliasManager.de")
805 s, matches = c.complete(None, "config AliasManager.de")
806 nt.assert_list_equal(["AliasManager.default_aliases"], matches)
806 nt.assert_list_equal(["AliasManager.default_aliases"], matches)
807 s, matches = c.complete(None, "config AliasManager.de")
807 s, matches = c.complete(None, "config AliasManager.de")
808 nt.assert_list_equal(["AliasManager.default_aliases"], matches)
808 nt.assert_list_equal(["AliasManager.default_aliases"], matches)
809
809
810 def test_magic_color(self):
810 def test_magic_color(self):
811 ip = get_ipython()
811 ip = get_ipython()
812 c = ip.Completer
812 c = ip.Completer
813
813
814 s, matches = c.complete(None, "colo")
814 s, matches = c.complete(None, "colo")
815 nt.assert_in("%colors", matches)
815 nt.assert_in("%colors", matches)
816 s, matches = c.complete(None, "colo")
816 s, matches = c.complete(None, "colo")
817 nt.assert_not_in("NoColor", matches)
817 nt.assert_not_in("NoColor", matches)
818 s, matches = c.complete(None, "%colors") # No trailing space
818 s, matches = c.complete(None, "%colors") # No trailing space
819 nt.assert_not_in("NoColor", matches)
819 nt.assert_not_in("NoColor", matches)
820 s, matches = c.complete(None, "colors ")
820 s, matches = c.complete(None, "colors ")
821 nt.assert_in("NoColor", matches)
821 nt.assert_in("NoColor", matches)
822 s, matches = c.complete(None, "%colors ")
822 s, matches = c.complete(None, "%colors ")
823 nt.assert_in("NoColor", matches)
823 nt.assert_in("NoColor", matches)
824 s, matches = c.complete(None, "colors NoCo")
824 s, matches = c.complete(None, "colors NoCo")
825 nt.assert_list_equal(["NoColor"], matches)
825 nt.assert_list_equal(["NoColor"], matches)
826 s, matches = c.complete(None, "%colors NoCo")
826 s, matches = c.complete(None, "%colors NoCo")
827 nt.assert_list_equal(["NoColor"], matches)
827 nt.assert_list_equal(["NoColor"], matches)
828
828
829 def test_match_dict_keys(self):
829 def test_match_dict_keys(self):
830 """
830 """
831 Test that match_dict_keys works on a couple of use case does return what
831 Test that match_dict_keys works on a couple of use case does return what
832 expected, and does not crash
832 expected, and does not crash
833 """
833 """
834 delims = " \t\n`!@#$^&*()=+[{]}\\|;:'\",<>?"
834 delims = " \t\n`!@#$^&*()=+[{]}\\|;:'\",<>?"
835
835
836 keys = ["foo", b"far"]
836 keys = ["foo", b"far"]
837 assert match_dict_keys(keys, "b'", delims=delims) == ("'", 2, ["far"])
837 assert match_dict_keys(keys, "b'", delims=delims) == ("'", 2, ["far"])
838 assert match_dict_keys(keys, "b'f", delims=delims) == ("'", 2, ["far"])
838 assert match_dict_keys(keys, "b'f", delims=delims) == ("'", 2, ["far"])
839 assert match_dict_keys(keys, 'b"', delims=delims) == ('"', 2, ["far"])
839 assert match_dict_keys(keys, 'b"', delims=delims) == ('"', 2, ["far"])
840 assert match_dict_keys(keys, 'b"f', delims=delims) == ('"', 2, ["far"])
840 assert match_dict_keys(keys, 'b"f', delims=delims) == ('"', 2, ["far"])
841
841
842 assert match_dict_keys(keys, "'", delims=delims) == ("'", 1, ["foo"])
842 assert match_dict_keys(keys, "'", delims=delims) == ("'", 1, ["foo"])
843 assert match_dict_keys(keys, "'f", delims=delims) == ("'", 1, ["foo"])
843 assert match_dict_keys(keys, "'f", delims=delims) == ("'", 1, ["foo"])
844 assert match_dict_keys(keys, '"', delims=delims) == ('"', 1, ["foo"])
844 assert match_dict_keys(keys, '"', delims=delims) == ('"', 1, ["foo"])
845 assert match_dict_keys(keys, '"f', delims=delims) == ('"', 1, ["foo"])
845 assert match_dict_keys(keys, '"f', delims=delims) == ('"', 1, ["foo"])
846
846
847 match_dict_keys
847 match_dict_keys
848
848
849 def test_match_dict_keys_tuple(self):
849 def test_match_dict_keys_tuple(self):
850 """
850 """
851 Test that match_dict_keys called with extra prefix works on a couple of use case,
851 Test that match_dict_keys called with extra prefix works on a couple of use case,
852 does return what expected, and does not crash.
852 does return what expected, and does not crash.
853 """
853 """
854 delims = " \t\n`!@#$^&*()=+[{]}\\|;:'\",<>?"
854 delims = " \t\n`!@#$^&*()=+[{]}\\|;:'\",<>?"
855
855
856 keys = [("foo", "bar"), ("foo", "oof"), ("foo", b"bar"), ('other', 'test')]
856 keys = [("foo", "bar"), ("foo", "oof"), ("foo", b"bar"), ('other', 'test')]
857
857
858 # Completion on first key == "foo"
858 # Completion on first key == "foo"
859 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("foo",)) == ("'", 1, ["bar", "oof"])
859 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("foo",)) == ("'", 1, ["bar", "oof"])
860 assert match_dict_keys(keys, "\"", delims=delims, extra_prefix=("foo",)) == ("\"", 1, ["bar", "oof"])
860 assert match_dict_keys(keys, "\"", delims=delims, extra_prefix=("foo",)) == ("\"", 1, ["bar", "oof"])
861 assert match_dict_keys(keys, "'o", delims=delims, extra_prefix=("foo",)) == ("'", 1, ["oof"])
861 assert match_dict_keys(keys, "'o", delims=delims, extra_prefix=("foo",)) == ("'", 1, ["oof"])
862 assert match_dict_keys(keys, "\"o", delims=delims, extra_prefix=("foo",)) == ("\"", 1, ["oof"])
862 assert match_dict_keys(keys, "\"o", delims=delims, extra_prefix=("foo",)) == ("\"", 1, ["oof"])
863 assert match_dict_keys(keys, "b'", delims=delims, extra_prefix=("foo",)) == ("'", 2, ["bar"])
863 assert match_dict_keys(keys, "b'", delims=delims, extra_prefix=("foo",)) == ("'", 2, ["bar"])
864 assert match_dict_keys(keys, "b\"", delims=delims, extra_prefix=("foo",)) == ("\"", 2, ["bar"])
864 assert match_dict_keys(keys, "b\"", delims=delims, extra_prefix=("foo",)) == ("\"", 2, ["bar"])
865 assert match_dict_keys(keys, "b'b", delims=delims, extra_prefix=("foo",)) == ("'", 2, ["bar"])
865 assert match_dict_keys(keys, "b'b", delims=delims, extra_prefix=("foo",)) == ("'", 2, ["bar"])
866 assert match_dict_keys(keys, "b\"b", delims=delims, extra_prefix=("foo",)) == ("\"", 2, ["bar"])
866 assert match_dict_keys(keys, "b\"b", delims=delims, extra_prefix=("foo",)) == ("\"", 2, ["bar"])
867
867
868 # No Completion
868 # No Completion
869 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("no_foo",)) == ("'", 1, [])
869 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("no_foo",)) == ("'", 1, [])
870 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("fo",)) == ("'", 1, [])
870 assert match_dict_keys(keys, "'", delims=delims, extra_prefix=("fo",)) == ("'", 1, [])
871
871
872 keys = [('foo1', 'foo2', 'foo3', 'foo4'), ('foo1', 'foo2', 'bar', 'foo4')]
872 keys = [('foo1', 'foo2', 'foo3', 'foo4'), ('foo1', 'foo2', 'bar', 'foo4')]
873 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1',)) == ("'", 1, ["foo2", "foo2"])
873 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1',)) == ("'", 1, ["foo2", "foo2"])
874 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2')) == ("'", 1, ["foo3"])
874 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2')) == ("'", 1, ["foo3"])
875 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2', 'foo3')) == ("'", 1, ["foo4"])
875 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2', 'foo3')) == ("'", 1, ["foo4"])
876 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2', 'foo3', 'foo4')) == ("'", 1, [])
876 assert match_dict_keys(keys, "'foo", delims=delims, extra_prefix=('foo1', 'foo2', 'foo3', 'foo4')) == ("'", 1, [])
877
877
878 def test_dict_key_completion_string(self):
878 def test_dict_key_completion_string(self):
879 """Test dictionary key completion for string keys"""
879 """Test dictionary key completion for string keys"""
880 ip = get_ipython()
880 ip = get_ipython()
881 complete = ip.Completer.complete
881 complete = ip.Completer.complete
882
882
883 ip.user_ns["d"] = {"abc": None}
883 ip.user_ns["d"] = {"abc": None}
884
884
885 # check completion at different stages
885 # check completion at different stages
886 _, matches = complete(line_buffer="d[")
886 _, matches = complete(line_buffer="d[")
887 nt.assert_in("'abc'", matches)
887 nt.assert_in("'abc'", matches)
888 nt.assert_not_in("'abc']", matches)
888 nt.assert_not_in("'abc']", matches)
889
889
890 _, matches = complete(line_buffer="d['")
890 _, matches = complete(line_buffer="d['")
891 nt.assert_in("abc", matches)
891 nt.assert_in("abc", matches)
892 nt.assert_not_in("abc']", matches)
892 nt.assert_not_in("abc']", matches)
893
893
894 _, matches = complete(line_buffer="d['a")
894 _, matches = complete(line_buffer="d['a")
895 nt.assert_in("abc", matches)
895 nt.assert_in("abc", matches)
896 nt.assert_not_in("abc']", matches)
896 nt.assert_not_in("abc']", matches)
897
897
898 # check use of different quoting
898 # check use of different quoting
899 _, matches = complete(line_buffer='d["')
899 _, matches = complete(line_buffer='d["')
900 nt.assert_in("abc", matches)
900 nt.assert_in("abc", matches)
901 nt.assert_not_in('abc"]', matches)
901 nt.assert_not_in('abc"]', matches)
902
902
903 _, matches = complete(line_buffer='d["a')
903 _, matches = complete(line_buffer='d["a')
904 nt.assert_in("abc", matches)
904 nt.assert_in("abc", matches)
905 nt.assert_not_in('abc"]', matches)
905 nt.assert_not_in('abc"]', matches)
906
906
907 # check sensitivity to following context
907 # check sensitivity to following context
908 _, matches = complete(line_buffer="d[]", cursor_pos=2)
908 _, matches = complete(line_buffer="d[]", cursor_pos=2)
909 nt.assert_in("'abc'", matches)
909 nt.assert_in("'abc'", matches)
910
910
911 _, matches = complete(line_buffer="d['']", cursor_pos=3)
911 _, matches = complete(line_buffer="d['']", cursor_pos=3)
912 nt.assert_in("abc", matches)
912 nt.assert_in("abc", matches)
913 nt.assert_not_in("abc'", matches)
913 nt.assert_not_in("abc'", matches)
914 nt.assert_not_in("abc']", matches)
914 nt.assert_not_in("abc']", matches)
915
915
916 # check multiple solutions are correctly returned and that noise is not
916 # check multiple solutions are correctly returned and that noise is not
917 ip.user_ns["d"] = {
917 ip.user_ns["d"] = {
918 "abc": None,
918 "abc": None,
919 "abd": None,
919 "abd": None,
920 "bad": None,
920 "bad": None,
921 object(): None,
921 object(): None,
922 5: None,
922 5: None,
923 ("abe", None): None,
923 ("abe", None): None,
924 (None, "abf"): None
924 (None, "abf"): None
925 }
925 }
926
926
927 _, matches = complete(line_buffer="d['a")
927 _, matches = complete(line_buffer="d['a")
928 nt.assert_in("abc", matches)
928 nt.assert_in("abc", matches)
929 nt.assert_in("abd", matches)
929 nt.assert_in("abd", matches)
930 nt.assert_not_in("bad", matches)
930 nt.assert_not_in("bad", matches)
931 nt.assert_not_in("abe", matches)
931 nt.assert_not_in("abe", matches)
932 nt.assert_not_in("abf", matches)
932 nt.assert_not_in("abf", matches)
933 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
933 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
934
934
935 # check escaping and whitespace
935 # check escaping and whitespace
936 ip.user_ns["d"] = {"a\nb": None, "a'b": None, 'a"b': None, "a word": None}
936 ip.user_ns["d"] = {"a\nb": None, "a'b": None, 'a"b': None, "a word": None}
937 _, matches = complete(line_buffer="d['a")
937 _, matches = complete(line_buffer="d['a")
938 nt.assert_in("a\\nb", matches)
938 nt.assert_in("a\\nb", matches)
939 nt.assert_in("a\\'b", matches)
939 nt.assert_in("a\\'b", matches)
940 nt.assert_in('a"b', matches)
940 nt.assert_in('a"b', matches)
941 nt.assert_in("a word", matches)
941 nt.assert_in("a word", matches)
942 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
942 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
943
943
944 # - can complete on non-initial word of the string
944 # - can complete on non-initial word of the string
945 _, matches = complete(line_buffer="d['a w")
945 _, matches = complete(line_buffer="d['a w")
946 nt.assert_in("word", matches)
946 nt.assert_in("word", matches)
947
947
948 # - understands quote escaping
948 # - understands quote escaping
949 _, matches = complete(line_buffer="d['a\\'")
949 _, matches = complete(line_buffer="d['a\\'")
950 nt.assert_in("b", matches)
950 nt.assert_in("b", matches)
951
951
952 # - default quoting should work like repr
952 # - default quoting should work like repr
953 _, matches = complete(line_buffer="d[")
953 _, matches = complete(line_buffer="d[")
954 nt.assert_in('"a\'b"', matches)
954 nt.assert_in('"a\'b"', matches)
955
955
956 # - when opening quote with ", possible to match with unescaped apostrophe
956 # - when opening quote with ", possible to match with unescaped apostrophe
957 _, matches = complete(line_buffer="d[\"a'")
957 _, matches = complete(line_buffer="d[\"a'")
958 nt.assert_in("b", matches)
958 nt.assert_in("b", matches)
959
959
960 # need to not split at delims that readline won't split at
960 # need to not split at delims that readline won't split at
961 if "-" not in ip.Completer.splitter.delims:
961 if "-" not in ip.Completer.splitter.delims:
962 ip.user_ns["d"] = {"before-after": None}
962 ip.user_ns["d"] = {"before-after": None}
963 _, matches = complete(line_buffer="d['before-af")
963 _, matches = complete(line_buffer="d['before-af")
964 nt.assert_in("before-after", matches)
964 nt.assert_in("before-after", matches)
965
965
966 # check completion on tuple-of-string keys at different stage - on first key
966 # check completion on tuple-of-string keys at different stage - on first key
967 ip.user_ns["d"] = {('foo', 'bar'): None}
967 ip.user_ns["d"] = {('foo', 'bar'): None}
968 _, matches = complete(line_buffer="d[")
968 _, matches = complete(line_buffer="d[")
969 nt.assert_in("'foo'", matches)
969 nt.assert_in("'foo'", matches)
970 nt.assert_not_in("'foo']", matches)
970 nt.assert_not_in("'foo']", matches)
971 nt.assert_not_in("'bar'", matches)
971 nt.assert_not_in("'bar'", matches)
972 nt.assert_not_in("foo", matches)
972 nt.assert_not_in("foo", matches)
973 nt.assert_not_in("bar", matches)
973 nt.assert_not_in("bar", matches)
974
974
975 # - match the prefix
975 # - match the prefix
976 _, matches = complete(line_buffer="d['f")
976 _, matches = complete(line_buffer="d['f")
977 nt.assert_in("foo", matches)
977 nt.assert_in("foo", matches)
978 nt.assert_not_in("foo']", matches)
978 nt.assert_not_in("foo']", matches)
979 nt.assert_not_in("foo\"]", matches)
979 nt.assert_not_in("foo\"]", matches)
980 _, matches = complete(line_buffer="d['foo")
980 _, matches = complete(line_buffer="d['foo")
981 nt.assert_in("foo", matches)
981 nt.assert_in("foo", matches)
982
982
983 # - can complete on second key
983 # - can complete on second key
984 _, matches = complete(line_buffer="d['foo', ")
984 _, matches = complete(line_buffer="d['foo', ")
985 nt.assert_in("'bar'", matches)
985 nt.assert_in("'bar'", matches)
986 _, matches = complete(line_buffer="d['foo', 'b")
986 _, matches = complete(line_buffer="d['foo', 'b")
987 nt.assert_in("bar", matches)
987 nt.assert_in("bar", matches)
988 nt.assert_not_in("foo", matches)
988 nt.assert_not_in("foo", matches)
989
989
990 # - does not propose missing keys
990 # - does not propose missing keys
991 _, matches = complete(line_buffer="d['foo', 'f")
991 _, matches = complete(line_buffer="d['foo', 'f")
992 nt.assert_not_in("bar", matches)
992 nt.assert_not_in("bar", matches)
993 nt.assert_not_in("foo", matches)
993 nt.assert_not_in("foo", matches)
994
994
995 # check sensitivity to following context
995 # check sensitivity to following context
996 _, matches = complete(line_buffer="d['foo',]", cursor_pos=8)
996 _, matches = complete(line_buffer="d['foo',]", cursor_pos=8)
997 nt.assert_in("'bar'", matches)
997 nt.assert_in("'bar'", matches)
998 nt.assert_not_in("bar", matches)
998 nt.assert_not_in("bar", matches)
999 nt.assert_not_in("'foo'", matches)
999 nt.assert_not_in("'foo'", matches)
1000 nt.assert_not_in("foo", matches)
1000 nt.assert_not_in("foo", matches)
1001
1001
1002 _, matches = complete(line_buffer="d['']", cursor_pos=3)
1002 _, matches = complete(line_buffer="d['']", cursor_pos=3)
1003 nt.assert_in("foo", matches)
1003 nt.assert_in("foo", matches)
1004 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1004 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1005
1005
1006 _, matches = complete(line_buffer='d[""]', cursor_pos=3)
1006 _, matches = complete(line_buffer='d[""]', cursor_pos=3)
1007 nt.assert_in("foo", matches)
1007 nt.assert_in("foo", matches)
1008 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1008 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1009
1009
1010 _, matches = complete(line_buffer='d["foo","]', cursor_pos=9)
1010 _, matches = complete(line_buffer='d["foo","]', cursor_pos=9)
1011 nt.assert_in("bar", matches)
1011 nt.assert_in("bar", matches)
1012 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1012 assert not any(m.endswith(("]", '"', "'")) for m in matches), matches
1013
1013
1014 _, matches = complete(line_buffer='d["foo",]', cursor_pos=8)
1014 _, matches = complete(line_buffer='d["foo",]', cursor_pos=8)
1015 nt.assert_in("'bar'", matches)
1015 nt.assert_in("'bar'", matches)
1016 nt.assert_not_in("bar", matches)
1016 nt.assert_not_in("bar", matches)
1017
1017
1018 # Can complete with longer tuple keys
1018 # Can complete with longer tuple keys
1019 ip.user_ns["d"] = {('foo', 'bar', 'foobar'): None}
1019 ip.user_ns["d"] = {('foo', 'bar', 'foobar'): None}
1020
1020
1021 # - can complete second key
1021 # - can complete second key
1022 _, matches = complete(line_buffer="d['foo', 'b")
1022 _, matches = complete(line_buffer="d['foo', 'b")
1023 nt.assert_in('bar', matches)
1023 nt.assert_in('bar', matches)
1024 nt.assert_not_in('foo', matches)
1024 nt.assert_not_in('foo', matches)
1025 nt.assert_not_in('foobar', matches)
1025 nt.assert_not_in('foobar', matches)
1026
1026
1027 # - can complete third key
1027 # - can complete third key
1028 _, matches = complete(line_buffer="d['foo', 'bar', 'fo")
1028 _, matches = complete(line_buffer="d['foo', 'bar', 'fo")
1029 nt.assert_in('foobar', matches)
1029 nt.assert_in('foobar', matches)
1030 nt.assert_not_in('foo', matches)
1030 nt.assert_not_in('foo', matches)
1031 nt.assert_not_in('bar', matches)
1031 nt.assert_not_in('bar', matches)
1032
1032
1033
1033
1034 def test_dict_key_completion_contexts(self):
1034 def test_dict_key_completion_contexts(self):
1035 """Test expression contexts in which dict key completion occurs"""
1035 """Test expression contexts in which dict key completion occurs"""
1036 ip = get_ipython()
1036 ip = get_ipython()
1037 complete = ip.Completer.complete
1037 complete = ip.Completer.complete
1038 d = {"abc": None}
1038 d = {"abc": None}
1039 ip.user_ns["d"] = d
1039 ip.user_ns["d"] = d
1040
1040
1041 class C:
1041 class C:
1042 data = d
1042 data = d
1043
1043
1044 ip.user_ns["C"] = C
1044 ip.user_ns["C"] = C
1045 ip.user_ns["get"] = lambda: d
1045 ip.user_ns["get"] = lambda: d
1046
1046
1047 def assert_no_completion(**kwargs):
1047 def assert_no_completion(**kwargs):
1048 _, matches = complete(**kwargs)
1048 _, matches = complete(**kwargs)
1049 nt.assert_not_in("abc", matches)
1049 nt.assert_not_in("abc", matches)
1050 nt.assert_not_in("abc'", matches)
1050 nt.assert_not_in("abc'", matches)
1051 nt.assert_not_in("abc']", matches)
1051 nt.assert_not_in("abc']", matches)
1052 nt.assert_not_in("'abc'", matches)
1052 nt.assert_not_in("'abc'", matches)
1053 nt.assert_not_in("'abc']", matches)
1053 nt.assert_not_in("'abc']", matches)
1054
1054
1055 def assert_completion(**kwargs):
1055 def assert_completion(**kwargs):
1056 _, matches = complete(**kwargs)
1056 _, matches = complete(**kwargs)
1057 nt.assert_in("'abc'", matches)
1057 nt.assert_in("'abc'", matches)
1058 nt.assert_not_in("'abc']", matches)
1058 nt.assert_not_in("'abc']", matches)
1059
1059
1060 # no completion after string closed, even if reopened
1060 # no completion after string closed, even if reopened
1061 assert_no_completion(line_buffer="d['a'")
1061 assert_no_completion(line_buffer="d['a'")
1062 assert_no_completion(line_buffer='d["a"')
1062 assert_no_completion(line_buffer='d["a"')
1063 assert_no_completion(line_buffer="d['a' + ")
1063 assert_no_completion(line_buffer="d['a' + ")
1064 assert_no_completion(line_buffer="d['a' + '")
1064 assert_no_completion(line_buffer="d['a' + '")
1065
1065
1066 # completion in non-trivial expressions
1066 # completion in non-trivial expressions
1067 assert_completion(line_buffer="+ d[")
1067 assert_completion(line_buffer="+ d[")
1068 assert_completion(line_buffer="(d[")
1068 assert_completion(line_buffer="(d[")
1069 assert_completion(line_buffer="C.data[")
1069 assert_completion(line_buffer="C.data[")
1070
1070
1071 # greedy flag
1071 # greedy flag
1072 def assert_completion(**kwargs):
1072 def assert_completion(**kwargs):
1073 _, matches = complete(**kwargs)
1073 _, matches = complete(**kwargs)
1074 nt.assert_in("get()['abc']", matches)
1074 nt.assert_in("get()['abc']", matches)
1075
1075
1076 assert_no_completion(line_buffer="get()[")
1076 assert_no_completion(line_buffer="get()[")
1077 with greedy_completion():
1077 with greedy_completion():
1078 assert_completion(line_buffer="get()[")
1078 assert_completion(line_buffer="get()[")
1079 assert_completion(line_buffer="get()['")
1079 assert_completion(line_buffer="get()['")
1080 assert_completion(line_buffer="get()['a")
1080 assert_completion(line_buffer="get()['a")
1081 assert_completion(line_buffer="get()['ab")
1081 assert_completion(line_buffer="get()['ab")
1082 assert_completion(line_buffer="get()['abc")
1082 assert_completion(line_buffer="get()['abc")
1083
1083
1084 def test_dict_key_completion_bytes(self):
1084 def test_dict_key_completion_bytes(self):
1085 """Test handling of bytes in dict key completion"""
1085 """Test handling of bytes in dict key completion"""
1086 ip = get_ipython()
1086 ip = get_ipython()
1087 complete = ip.Completer.complete
1087 complete = ip.Completer.complete
1088
1088
1089 ip.user_ns["d"] = {"abc": None, b"abd": None}
1089 ip.user_ns["d"] = {"abc": None, b"abd": None}
1090
1090
1091 _, matches = complete(line_buffer="d[")
1091 _, matches = complete(line_buffer="d[")
1092 nt.assert_in("'abc'", matches)
1092 nt.assert_in("'abc'", matches)
1093 nt.assert_in("b'abd'", matches)
1093 nt.assert_in("b'abd'", matches)
1094
1094
1095 if False: # not currently implemented
1095 if False: # not currently implemented
1096 _, matches = complete(line_buffer="d[b")
1096 _, matches = complete(line_buffer="d[b")
1097 nt.assert_in("b'abd'", matches)
1097 nt.assert_in("b'abd'", matches)
1098 nt.assert_not_in("b'abc'", matches)
1098 nt.assert_not_in("b'abc'", matches)
1099
1099
1100 _, matches = complete(line_buffer="d[b'")
1100 _, matches = complete(line_buffer="d[b'")
1101 nt.assert_in("abd", matches)
1101 nt.assert_in("abd", matches)
1102 nt.assert_not_in("abc", matches)
1102 nt.assert_not_in("abc", matches)
1103
1103
1104 _, matches = complete(line_buffer="d[B'")
1104 _, matches = complete(line_buffer="d[B'")
1105 nt.assert_in("abd", matches)
1105 nt.assert_in("abd", matches)
1106 nt.assert_not_in("abc", matches)
1106 nt.assert_not_in("abc", matches)
1107
1107
1108 _, matches = complete(line_buffer="d['")
1108 _, matches = complete(line_buffer="d['")
1109 nt.assert_in("abc", matches)
1109 nt.assert_in("abc", matches)
1110 nt.assert_not_in("abd", matches)
1110 nt.assert_not_in("abd", matches)
1111
1111
1112 def test_dict_key_completion_unicode_py3(self):
1112 def test_dict_key_completion_unicode_py3(self):
1113 """Test handling of unicode in dict key completion"""
1113 """Test handling of unicode in dict key completion"""
1114 ip = get_ipython()
1114 ip = get_ipython()
1115 complete = ip.Completer.complete
1115 complete = ip.Completer.complete
1116
1116
1117 ip.user_ns["d"] = {"a\u05d0": None}
1117 ip.user_ns["d"] = {"a\u05d0": None}
1118
1118
1119 # query using escape
1119 # query using escape
1120 if sys.platform != "win32":
1120 if sys.platform != "win32":
1121 # Known failure on Windows
1121 # Known failure on Windows
1122 _, matches = complete(line_buffer="d['a\\u05d0")
1122 _, matches = complete(line_buffer="d['a\\u05d0")
1123 nt.assert_in("u05d0", matches) # tokenized after \\
1123 nt.assert_in("u05d0", matches) # tokenized after \\
1124
1124
1125 # query using character
1125 # query using character
1126 _, matches = complete(line_buffer="d['a\u05d0")
1126 _, matches = complete(line_buffer="d['a\u05d0")
1127 nt.assert_in("a\u05d0", matches)
1127 nt.assert_in("a\u05d0", matches)
1128
1128
1129 with greedy_completion():
1129 with greedy_completion():
1130 # query using escape
1130 # query using escape
1131 _, matches = complete(line_buffer="d['a\\u05d0")
1131 _, matches = complete(line_buffer="d['a\\u05d0")
1132 nt.assert_in("d['a\\u05d0']", matches) # tokenized after \\
1132 nt.assert_in("d['a\\u05d0']", matches) # tokenized after \\
1133
1133
1134 # query using character
1134 # query using character
1135 _, matches = complete(line_buffer="d['a\u05d0")
1135 _, matches = complete(line_buffer="d['a\u05d0")
1136 nt.assert_in("d['a\u05d0']", matches)
1136 nt.assert_in("d['a\u05d0']", matches)
1137
1137
1138 @dec.skip_without("numpy")
1138 @dec.skip_without("numpy")
1139 def test_struct_array_key_completion(self):
1139 def test_struct_array_key_completion(self):
1140 """Test dict key completion applies to numpy struct arrays"""
1140 """Test dict key completion applies to numpy struct arrays"""
1141 import numpy
1141 import numpy
1142
1142
1143 ip = get_ipython()
1143 ip = get_ipython()
1144 complete = ip.Completer.complete
1144 complete = ip.Completer.complete
1145 ip.user_ns["d"] = numpy.array([], dtype=[("hello", "f"), ("world", "f")])
1145 ip.user_ns["d"] = numpy.array([], dtype=[("hello", "f"), ("world", "f")])
1146 _, matches = complete(line_buffer="d['")
1146 _, matches = complete(line_buffer="d['")
1147 nt.assert_in("hello", matches)
1147 nt.assert_in("hello", matches)
1148 nt.assert_in("world", matches)
1148 nt.assert_in("world", matches)
1149 # complete on the numpy struct itself
1149 # complete on the numpy struct itself
1150 dt = numpy.dtype(
1150 dt = numpy.dtype(
1151 [("my_head", [("my_dt", ">u4"), ("my_df", ">u4")]), ("my_data", ">f4", 5)]
1151 [("my_head", [("my_dt", ">u4"), ("my_df", ">u4")]), ("my_data", ">f4", 5)]
1152 )
1152 )
1153 x = numpy.zeros(2, dtype=dt)
1153 x = numpy.zeros(2, dtype=dt)
1154 ip.user_ns["d"] = x[1]
1154 ip.user_ns["d"] = x[1]
1155 _, matches = complete(line_buffer="d['")
1155 _, matches = complete(line_buffer="d['")
1156 nt.assert_in("my_head", matches)
1156 nt.assert_in("my_head", matches)
1157 nt.assert_in("my_data", matches)
1157 nt.assert_in("my_data", matches)
1158 # complete on a nested level
1158 # complete on a nested level
1159 with greedy_completion():
1159 with greedy_completion():
1160 ip.user_ns["d"] = numpy.zeros(2, dtype=dt)
1160 ip.user_ns["d"] = numpy.zeros(2, dtype=dt)
1161 _, matches = complete(line_buffer="d[1]['my_head']['")
1161 _, matches = complete(line_buffer="d[1]['my_head']['")
1162 nt.assert_true(any(["my_dt" in m for m in matches]))
1162 nt.assert_true(any(["my_dt" in m for m in matches]))
1163 nt.assert_true(any(["my_df" in m for m in matches]))
1163 nt.assert_true(any(["my_df" in m for m in matches]))
1164
1164
1165 @dec.skip_without("pandas")
1165 @dec.skip_without("pandas")
1166 def test_dataframe_key_completion(self):
1166 def test_dataframe_key_completion(self):
1167 """Test dict key completion applies to pandas DataFrames"""
1167 """Test dict key completion applies to pandas DataFrames"""
1168 import pandas
1168 import pandas
1169
1169
1170 ip = get_ipython()
1170 ip = get_ipython()
1171 complete = ip.Completer.complete
1171 complete = ip.Completer.complete
1172 ip.user_ns["d"] = pandas.DataFrame({"hello": [1], "world": [2]})
1172 ip.user_ns["d"] = pandas.DataFrame({"hello": [1], "world": [2]})
1173 _, matches = complete(line_buffer="d['")
1173 _, matches = complete(line_buffer="d['")
1174 nt.assert_in("hello", matches)
1174 nt.assert_in("hello", matches)
1175 nt.assert_in("world", matches)
1175 nt.assert_in("world", matches)
1176
1176
1177 def test_dict_key_completion_invalids(self):
1177 def test_dict_key_completion_invalids(self):
1178 """Smoke test cases dict key completion can't handle"""
1178 """Smoke test cases dict key completion can't handle"""
1179 ip = get_ipython()
1179 ip = get_ipython()
1180 complete = ip.Completer.complete
1180 complete = ip.Completer.complete
1181
1181
1182 ip.user_ns["no_getitem"] = None
1182 ip.user_ns["no_getitem"] = None
1183 ip.user_ns["no_keys"] = []
1183 ip.user_ns["no_keys"] = []
1184 ip.user_ns["cant_call_keys"] = dict
1184 ip.user_ns["cant_call_keys"] = dict
1185 ip.user_ns["empty"] = {}
1185 ip.user_ns["empty"] = {}
1186 ip.user_ns["d"] = {"abc": 5}
1186 ip.user_ns["d"] = {"abc": 5}
1187
1187
1188 _, matches = complete(line_buffer="no_getitem['")
1188 _, matches = complete(line_buffer="no_getitem['")
1189 _, matches = complete(line_buffer="no_keys['")
1189 _, matches = complete(line_buffer="no_keys['")
1190 _, matches = complete(line_buffer="cant_call_keys['")
1190 _, matches = complete(line_buffer="cant_call_keys['")
1191 _, matches = complete(line_buffer="empty['")
1191 _, matches = complete(line_buffer="empty['")
1192 _, matches = complete(line_buffer="name_error['")
1192 _, matches = complete(line_buffer="name_error['")
1193 _, matches = complete(line_buffer="d['\\") # incomplete escape
1193 _, matches = complete(line_buffer="d['\\") # incomplete escape
1194
1194
1195 def test_object_key_completion(self):
1195 def test_object_key_completion(self):
1196 ip = get_ipython()
1196 ip = get_ipython()
1197 ip.user_ns["key_completable"] = KeyCompletable(["qwerty", "qwick"])
1197 ip.user_ns["key_completable"] = KeyCompletable(["qwerty", "qwick"])
1198
1198
1199 _, matches = ip.Completer.complete(line_buffer="key_completable['qw")
1199 _, matches = ip.Completer.complete(line_buffer="key_completable['qw")
1200 nt.assert_in("qwerty", matches)
1200 nt.assert_in("qwerty", matches)
1201 nt.assert_in("qwick", matches)
1201 nt.assert_in("qwick", matches)
1202
1202
1203 def test_class_key_completion(self):
1203 def test_class_key_completion(self):
1204 ip = get_ipython()
1204 ip = get_ipython()
1205 NamedInstanceClass("qwerty")
1205 NamedInstanceClass("qwerty")
1206 NamedInstanceClass("qwick")
1206 NamedInstanceClass("qwick")
1207 ip.user_ns["named_instance_class"] = NamedInstanceClass
1207 ip.user_ns["named_instance_class"] = NamedInstanceClass
1208
1208
1209 _, matches = ip.Completer.complete(line_buffer="named_instance_class['qw")
1209 _, matches = ip.Completer.complete(line_buffer="named_instance_class['qw")
1210 nt.assert_in("qwerty", matches)
1210 nt.assert_in("qwerty", matches)
1211 nt.assert_in("qwick", matches)
1211 nt.assert_in("qwick", matches)
1212
1212
1213 def test_tryimport(self):
1213 def test_tryimport(self):
1214 """
1214 """
1215 Test that try-import don't crash on trailing dot, and import modules before
1215 Test that try-import don't crash on trailing dot, and import modules before
1216 """
1216 """
1217 from IPython.core.completerlib import try_import
1217 from IPython.core.completerlib import try_import
1218
1218
1219 assert try_import("IPython.")
1219 assert try_import("IPython.")
1220
1220
1221 def test_aimport_module_completer(self):
1221 def test_aimport_module_completer(self):
1222 ip = get_ipython()
1222 ip = get_ipython()
1223 _, matches = ip.complete("i", "%aimport i")
1223 _, matches = ip.complete("i", "%aimport i")
1224 nt.assert_in("io", matches)
1224 nt.assert_in("io", matches)
1225 nt.assert_not_in("int", matches)
1225 nt.assert_not_in("int", matches)
1226
1226
1227 def test_nested_import_module_completer(self):
1227 def test_nested_import_module_completer(self):
1228 ip = get_ipython()
1228 ip = get_ipython()
1229 _, matches = ip.complete(None, "import IPython.co", 17)
1229 _, matches = ip.complete(None, "import IPython.co", 17)
1230 nt.assert_in("IPython.core", matches)
1230 nt.assert_in("IPython.core", matches)
1231 nt.assert_not_in("import IPython.core", matches)
1231 nt.assert_not_in("import IPython.core", matches)
1232 nt.assert_not_in("IPython.display", matches)
1232 nt.assert_not_in("IPython.display", matches)
1233
1233
1234 def test_import_module_completer(self):
1234 def test_import_module_completer(self):
1235 ip = get_ipython()
1235 ip = get_ipython()
1236 _, matches = ip.complete("i", "import i")
1236 _, matches = ip.complete("i", "import i")
1237 nt.assert_in("io", matches)
1237 nt.assert_in("io", matches)
1238 nt.assert_not_in("int", matches)
1238 nt.assert_not_in("int", matches)
1239
1239
1240 def test_from_module_completer(self):
1240 def test_from_module_completer(self):
1241 ip = get_ipython()
1241 ip = get_ipython()
1242 _, matches = ip.complete("B", "from io import B", 16)
1242 _, matches = ip.complete("B", "from io import B", 16)
1243 nt.assert_in("BytesIO", matches)
1243 nt.assert_in("BytesIO", matches)
1244 nt.assert_not_in("BaseException", matches)
1244 nt.assert_not_in("BaseException", matches)
1245
1245
1246 def test_snake_case_completion(self):
1246 def test_snake_case_completion(self):
1247 ip = get_ipython()
1247 ip = get_ipython()
1248 ip.Completer.use_jedi = False
1248 ip.Completer.use_jedi = False
1249 ip.user_ns["some_three"] = 3
1249 ip.user_ns["some_three"] = 3
1250 ip.user_ns["some_four"] = 4
1250 ip.user_ns["some_four"] = 4
1251 _, matches = ip.complete("s_", "print(s_f")
1251 _, matches = ip.complete("s_", "print(s_f")
1252 nt.assert_in("some_three", matches)
1252 nt.assert_in("some_three", matches)
1253 nt.assert_in("some_four", matches)
1253 nt.assert_in("some_four", matches)
1254
1254
1255 def test_mix_terms(self):
1255 def test_mix_terms(self):
1256 ip = get_ipython()
1256 ip = get_ipython()
1257 from textwrap import dedent
1257 from textwrap import dedent
1258
1258
1259 ip.Completer.use_jedi = False
1259 ip.Completer.use_jedi = False
1260 ip.ex(
1260 ip.ex(
1261 dedent(
1261 dedent(
1262 """
1262 """
1263 class Test:
1263 class Test:
1264 def meth(self, meth_arg1):
1264 def meth(self, meth_arg1):
1265 print("meth")
1265 print("meth")
1266
1266
1267 def meth_1(self, meth1_arg1, meth1_arg2):
1267 def meth_1(self, meth1_arg1, meth1_arg2):
1268 print("meth1")
1268 print("meth1")
1269
1269
1270 def meth_2(self, meth2_arg1, meth2_arg2):
1270 def meth_2(self, meth2_arg1, meth2_arg2):
1271 print("meth2")
1271 print("meth2")
1272 test = Test()
1272 test = Test()
1273 """
1273 """
1274 )
1274 )
1275 )
1275 )
1276 _, matches = ip.complete(None, "test.meth(")
1276 _, matches = ip.complete(None, "test.meth(")
1277 nt.assert_in("meth_arg1=", matches)
1277 nt.assert_in("meth_arg1=", matches)
1278 nt.assert_not_in("meth2_arg1=", matches)
1278 nt.assert_not_in("meth2_arg1=", matches)
@@ -1,490 +1,490 b''
1 # Copyright (c) IPython Development Team.
1 # Copyright (c) IPython Development Team.
2 # Distributed under the terms of the Modified BSD License.
2 # Distributed under the terms of the Modified BSD License.
3
3
4 import json
4 import json
5 import os
5 import os
6 import warnings
6 import warnings
7
7
8 from unittest import mock
8 from unittest import mock
9
9
10 import nose.tools as nt
10 import nose.tools as nt
11
11
12 from IPython import display
12 from IPython import display
13 from IPython.core.getipython import get_ipython
13 from IPython.core.getipython import get_ipython
14 from IPython.utils.io import capture_output
14 from IPython.utils.io import capture_output
15 from IPython.utils.tempdir import NamedFileInTemporaryDirectory
15 from IPython.utils.tempdir import NamedFileInTemporaryDirectory
16 from IPython import paths as ipath
16 from IPython import paths as ipath
17 from IPython.testing.tools import AssertNotPrints
17 from IPython.testing.tools import AssertNotPrints
18
18
19 import IPython.testing.decorators as dec
19 import IPython.testing.decorators as dec
20
20
21 def test_image_size():
21 def test_image_size():
22 """Simple test for display.Image(args, width=x,height=y)"""
22 """Simple test for display.Image(args, width=x,height=y)"""
23 thisurl = 'http://www.google.fr/images/srpr/logo3w.png'
23 thisurl = 'http://www.google.fr/images/srpr/logo3w.png'
24 img = display.Image(url=thisurl, width=200, height=200)
24 img = display.Image(url=thisurl, width=200, height=200)
25 nt.assert_equal(u'<img src="%s" width="200" height="200"/>' % (thisurl), img._repr_html_())
25 nt.assert_equal(u'<img src="%s" width="200" height="200"/>' % (thisurl), img._repr_html_())
26 img = display.Image(url=thisurl, metadata={'width':200, 'height':200})
26 img = display.Image(url=thisurl, metadata={'width':200, 'height':200})
27 nt.assert_equal(u'<img src="%s" width="200" height="200"/>' % (thisurl), img._repr_html_())
27 nt.assert_equal(u'<img src="%s" width="200" height="200"/>' % (thisurl), img._repr_html_())
28 img = display.Image(url=thisurl, width=200)
28 img = display.Image(url=thisurl, width=200)
29 nt.assert_equal(u'<img src="%s" width="200"/>' % (thisurl), img._repr_html_())
29 nt.assert_equal(u'<img src="%s" width="200"/>' % (thisurl), img._repr_html_())
30 img = display.Image(url=thisurl)
30 img = display.Image(url=thisurl)
31 nt.assert_equal(u'<img src="%s"/>' % (thisurl), img._repr_html_())
31 nt.assert_equal(u'<img src="%s"/>' % (thisurl), img._repr_html_())
32 img = display.Image(url=thisurl, unconfined=True)
32 img = display.Image(url=thisurl, unconfined=True)
33 nt.assert_equal(u'<img src="%s" class="unconfined"/>' % (thisurl), img._repr_html_())
33 nt.assert_equal(u'<img src="%s" class="unconfined"/>' % (thisurl), img._repr_html_())
34
34
35
35
36 def test_image_mimes():
36 def test_image_mimes():
37 fmt = get_ipython().display_formatter.format
37 fmt = get_ipython().display_formatter.format
38 for format in display.Image._ACCEPTABLE_EMBEDDINGS:
38 for format in display.Image._ACCEPTABLE_EMBEDDINGS:
39 mime = display.Image._MIMETYPES[format]
39 mime = display.Image._MIMETYPES[format]
40 img = display.Image(b'garbage', format=format)
40 img = display.Image(b'garbage', format=format)
41 data, metadata = fmt(img)
41 data, metadata = fmt(img)
42 nt.assert_equal(sorted(data), sorted([mime, 'text/plain']))
42 nt.assert_equal(sorted(data), sorted([mime, 'text/plain']))
43
43
44
44
45 def test_geojson():
45 def test_geojson():
46
46
47 gj = display.GeoJSON(data={
47 gj = display.GeoJSON(data={
48 "type": "Feature",
48 "type": "Feature",
49 "geometry": {
49 "geometry": {
50 "type": "Point",
50 "type": "Point",
51 "coordinates": [-81.327, 296.038]
51 "coordinates": [-81.327, 296.038]
52 },
52 },
53 "properties": {
53 "properties": {
54 "name": "Inca City"
54 "name": "Inca City"
55 }
55 }
56 },
56 },
57 url_template="http://s3-eu-west-1.amazonaws.com/whereonmars.cartodb.net/{basemap_id}/{z}/{x}/{y}.png",
57 url_template="http://s3-eu-west-1.amazonaws.com/whereonmars.cartodb.net/{basemap_id}/{z}/{x}/{y}.png",
58 layer_options={
58 layer_options={
59 "basemap_id": "celestia_mars-shaded-16k_global",
59 "basemap_id": "celestia_mars-shaded-16k_global",
60 "attribution": "Celestia/praesepe",
60 "attribution": "Celestia/praesepe",
61 "minZoom": 0,
61 "minZoom": 0,
62 "maxZoom": 18,
62 "maxZoom": 18,
63 })
63 })
64 nt.assert_equal(u'<IPython.core.display.GeoJSON object>', str(gj))
64 nt.assert_equal(u'<IPython.core.display.GeoJSON object>', str(gj))
65
65
66 def test_retina_png():
66 def test_retina_png():
67 here = os.path.dirname(__file__)
67 here = os.path.dirname(__file__)
68 img = display.Image(os.path.join(here, "2x2.png"), retina=True)
68 img = display.Image(os.path.join(here, "2x2.png"), retina=True)
69 nt.assert_equal(img.height, 1)
69 nt.assert_equal(img.height, 1)
70 nt.assert_equal(img.width, 1)
70 nt.assert_equal(img.width, 1)
71 data, md = img._repr_png_()
71 data, md = img._repr_png_()
72 nt.assert_equal(md['width'], 1)
72 nt.assert_equal(md['width'], 1)
73 nt.assert_equal(md['height'], 1)
73 nt.assert_equal(md['height'], 1)
74
74
75 def test_embed_svg_url():
75 def test_embed_svg_url():
76 import gzip
76 import gzip
77 from io import BytesIO
77 from io import BytesIO
78 svg_data = b'<svg><circle x="0" y="0" r="1"/></svg>'
78 svg_data = b'<svg><circle x="0" y="0" r="1"/></svg>'
79 url = 'http://test.com/circle.svg'
79 url = 'http://test.com/circle.svg'
80
80
81 gzip_svg = BytesIO()
81 gzip_svg = BytesIO()
82 with gzip.open(gzip_svg, 'wb') as fp:
82 with gzip.open(gzip_svg, 'wb') as fp:
83 fp.write(svg_data)
83 fp.write(svg_data)
84 gzip_svg = gzip_svg.getvalue()
84 gzip_svg = gzip_svg.getvalue()
85
85
86 def mocked_urlopen(*args, **kwargs):
86 def mocked_urlopen(*args, **kwargs):
87 class MockResponse:
87 class MockResponse:
88 def __init__(self, svg):
88 def __init__(self, svg):
89 self._svg_data = svg
89 self._svg_data = svg
90 self.headers = {'content-type': 'image/svg+xml'}
90 self.headers = {'content-type': 'image/svg+xml'}
91
91
92 def read(self):
92 def read(self):
93 return self._svg_data
93 return self._svg_data
94
94
95 if args[0] == url:
95 if args[0] == url:
96 return MockResponse(svg_data)
96 return MockResponse(svg_data)
97 elif args[0] == url + "z":
97 elif args[0] == url + "z":
98 ret = MockResponse(gzip_svg)
98 ret = MockResponse(gzip_svg)
99 ret.headers["content-encoding"] = "gzip"
99 ret.headers["content-encoding"] = "gzip"
100 return ret
100 return ret
101 return MockResponse(None)
101 return MockResponse(None)
102
102
103 with mock.patch('urllib.request.urlopen', side_effect=mocked_urlopen):
103 with mock.patch('urllib.request.urlopen', side_effect=mocked_urlopen):
104 svg = display.SVG(url=url)
104 svg = display.SVG(url=url)
105 nt.assert_true(svg._repr_svg_().startswith('<svg'))
105 nt.assert_true(svg._repr_svg_().startswith('<svg'))
106 svg = display.SVG(url=url + 'z')
106 svg = display.SVG(url=url + 'z')
107 nt.assert_true(svg._repr_svg_().startswith('<svg'))
107 nt.assert_true(svg._repr_svg_().startswith('<svg'))
108
108
109 def test_retina_jpeg():
109 def test_retina_jpeg():
110 here = os.path.dirname(__file__)
110 here = os.path.dirname(__file__)
111 img = display.Image(os.path.join(here, "2x2.jpg"), retina=True)
111 img = display.Image(os.path.join(here, "2x2.jpg"), retina=True)
112 nt.assert_equal(img.height, 1)
112 nt.assert_equal(img.height, 1)
113 nt.assert_equal(img.width, 1)
113 nt.assert_equal(img.width, 1)
114 data, md = img._repr_jpeg_()
114 data, md = img._repr_jpeg_()
115 nt.assert_equal(md['width'], 1)
115 nt.assert_equal(md['width'], 1)
116 nt.assert_equal(md['height'], 1)
116 nt.assert_equal(md['height'], 1)
117
117
118 def test_base64image():
118 def test_base64image():
119 display.Image("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKAAAAA1BMVEUAAACnej3aAAAAAWJLR0QAiAUdSAAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB94BCRQnOqNu0b4AAAAKSURBVAjXY2AAAAACAAHiIbwzAAAAAElFTkSuQmCC")
119 display.Image("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKAAAAA1BMVEUAAACnej3aAAAAAWJLR0QAiAUdSAAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB94BCRQnOqNu0b4AAAAKSURBVAjXY2AAAAACAAHiIbwzAAAAAElFTkSuQmCC")
120
120
121 def test_image_filename_defaults():
121 def test_image_filename_defaults():
122 '''test format constraint, and validity of jpeg and png'''
122 '''test format constraint, and validity of jpeg and png'''
123 tpath = ipath.get_ipython_package_dir()
123 tpath = ipath.get_ipython_package_dir()
124 nt.assert_raises(ValueError, display.Image, filename=os.path.join(tpath, 'testing/tests/badformat.zip'),
124 nt.assert_raises(ValueError, display.Image, filename=os.path.join(tpath, 'testing/tests/badformat.zip'),
125 embed=True)
125 embed=True)
126 nt.assert_raises(ValueError, display.Image)
126 nt.assert_raises(ValueError, display.Image)
127 nt.assert_raises(ValueError, display.Image, data='this is not an image', format='badformat', embed=True)
127 nt.assert_raises(ValueError, display.Image, data='this is not an image', format='badformat', embed=True)
128 # check boths paths to allow packages to test at build and install time
128 # check both paths to allow packages to test at build and install time
129 imgfile = os.path.join(tpath, 'core/tests/2x2.png')
129 imgfile = os.path.join(tpath, 'core/tests/2x2.png')
130 img = display.Image(filename=imgfile)
130 img = display.Image(filename=imgfile)
131 nt.assert_equal('png', img.format)
131 nt.assert_equal('png', img.format)
132 nt.assert_is_not_none(img._repr_png_())
132 nt.assert_is_not_none(img._repr_png_())
133 img = display.Image(filename=os.path.join(tpath, 'testing/tests/logo.jpg'), embed=False)
133 img = display.Image(filename=os.path.join(tpath, 'testing/tests/logo.jpg'), embed=False)
134 nt.assert_equal('jpeg', img.format)
134 nt.assert_equal('jpeg', img.format)
135 nt.assert_is_none(img._repr_jpeg_())
135 nt.assert_is_none(img._repr_jpeg_())
136
136
137 def _get_inline_config():
137 def _get_inline_config():
138 from matplotlib_inline.config import InlineBackend
138 from matplotlib_inline.config import InlineBackend
139 return InlineBackend.instance()
139 return InlineBackend.instance()
140
140
141
141
142 @dec.skip_without("ipykernel")
142 @dec.skip_without("ipykernel")
143 @dec.skip_without("matplotlib")
143 @dec.skip_without("matplotlib")
144 def test_set_matplotlib_close():
144 def test_set_matplotlib_close():
145 cfg = _get_inline_config()
145 cfg = _get_inline_config()
146 cfg.close_figures = False
146 cfg.close_figures = False
147 display.set_matplotlib_close()
147 display.set_matplotlib_close()
148 assert cfg.close_figures
148 assert cfg.close_figures
149 display.set_matplotlib_close(False)
149 display.set_matplotlib_close(False)
150 assert not cfg.close_figures
150 assert not cfg.close_figures
151
151
152 _fmt_mime_map = {
152 _fmt_mime_map = {
153 'png': 'image/png',
153 'png': 'image/png',
154 'jpeg': 'image/jpeg',
154 'jpeg': 'image/jpeg',
155 'pdf': 'application/pdf',
155 'pdf': 'application/pdf',
156 'retina': 'image/png',
156 'retina': 'image/png',
157 'svg': 'image/svg+xml',
157 'svg': 'image/svg+xml',
158 }
158 }
159
159
160 @dec.skip_without('matplotlib')
160 @dec.skip_without('matplotlib')
161 def test_set_matplotlib_formats():
161 def test_set_matplotlib_formats():
162 from matplotlib.figure import Figure
162 from matplotlib.figure import Figure
163 formatters = get_ipython().display_formatter.formatters
163 formatters = get_ipython().display_formatter.formatters
164 for formats in [
164 for formats in [
165 ('png',),
165 ('png',),
166 ('pdf', 'svg'),
166 ('pdf', 'svg'),
167 ('jpeg', 'retina', 'png'),
167 ('jpeg', 'retina', 'png'),
168 (),
168 (),
169 ]:
169 ]:
170 active_mimes = {_fmt_mime_map[fmt] for fmt in formats}
170 active_mimes = {_fmt_mime_map[fmt] for fmt in formats}
171 display.set_matplotlib_formats(*formats)
171 display.set_matplotlib_formats(*formats)
172 for mime, f in formatters.items():
172 for mime, f in formatters.items():
173 if mime in active_mimes:
173 if mime in active_mimes:
174 nt.assert_in(Figure, f)
174 nt.assert_in(Figure, f)
175 else:
175 else:
176 nt.assert_not_in(Figure, f)
176 nt.assert_not_in(Figure, f)
177
177
178
178
179 @dec.skip_without("ipykernel")
179 @dec.skip_without("ipykernel")
180 @dec.skip_without("matplotlib")
180 @dec.skip_without("matplotlib")
181 def test_set_matplotlib_formats_kwargs():
181 def test_set_matplotlib_formats_kwargs():
182 from matplotlib.figure import Figure
182 from matplotlib.figure import Figure
183 ip = get_ipython()
183 ip = get_ipython()
184 cfg = _get_inline_config()
184 cfg = _get_inline_config()
185 cfg.print_figure_kwargs.update(dict(foo='bar'))
185 cfg.print_figure_kwargs.update(dict(foo='bar'))
186 kwargs = dict(dpi=150)
186 kwargs = dict(dpi=150)
187 display.set_matplotlib_formats('png', **kwargs)
187 display.set_matplotlib_formats('png', **kwargs)
188 formatter = ip.display_formatter.formatters['image/png']
188 formatter = ip.display_formatter.formatters['image/png']
189 f = formatter.lookup_by_type(Figure)
189 f = formatter.lookup_by_type(Figure)
190 formatter_kwargs = f.keywords
190 formatter_kwargs = f.keywords
191 expected = kwargs
191 expected = kwargs
192 expected["base64"] = True
192 expected["base64"] = True
193 expected["fmt"] = "png"
193 expected["fmt"] = "png"
194 expected.update(cfg.print_figure_kwargs)
194 expected.update(cfg.print_figure_kwargs)
195 nt.assert_equal(formatter_kwargs, expected)
195 nt.assert_equal(formatter_kwargs, expected)
196
196
197 def test_display_available():
197 def test_display_available():
198 """
198 """
199 Test that display is available without import
199 Test that display is available without import
200
200
201 We don't really care if it's in builtin or anything else, but it should
201 We don't really care if it's in builtin or anything else, but it should
202 always be available.
202 always be available.
203 """
203 """
204 ip = get_ipython()
204 ip = get_ipython()
205 with AssertNotPrints('NameError'):
205 with AssertNotPrints('NameError'):
206 ip.run_cell('display')
206 ip.run_cell('display')
207 try:
207 try:
208 ip.run_cell('del display')
208 ip.run_cell('del display')
209 except NameError:
209 except NameError:
210 pass # it's ok, it might be in builtins
210 pass # it's ok, it might be in builtins
211 # even if deleted it should be back
211 # even if deleted it should be back
212 with AssertNotPrints('NameError'):
212 with AssertNotPrints('NameError'):
213 ip.run_cell('display')
213 ip.run_cell('display')
214
214
215 def test_textdisplayobj_pretty_repr():
215 def test_textdisplayobj_pretty_repr():
216 p = display.Pretty("This is a simple test")
216 p = display.Pretty("This is a simple test")
217 nt.assert_equal(repr(p), '<IPython.core.display.Pretty object>')
217 nt.assert_equal(repr(p), '<IPython.core.display.Pretty object>')
218 nt.assert_equal(p.data, 'This is a simple test')
218 nt.assert_equal(p.data, 'This is a simple test')
219
219
220 p._show_mem_addr = True
220 p._show_mem_addr = True
221 nt.assert_equal(repr(p), object.__repr__(p))
221 nt.assert_equal(repr(p), object.__repr__(p))
222
222
223 def test_displayobject_repr():
223 def test_displayobject_repr():
224 h = display.HTML('<br />')
224 h = display.HTML('<br />')
225 nt.assert_equal(repr(h), '<IPython.core.display.HTML object>')
225 nt.assert_equal(repr(h), '<IPython.core.display.HTML object>')
226 h._show_mem_addr = True
226 h._show_mem_addr = True
227 nt.assert_equal(repr(h), object.__repr__(h))
227 nt.assert_equal(repr(h), object.__repr__(h))
228 h._show_mem_addr = False
228 h._show_mem_addr = False
229 nt.assert_equal(repr(h), '<IPython.core.display.HTML object>')
229 nt.assert_equal(repr(h), '<IPython.core.display.HTML object>')
230
230
231 j = display.Javascript('')
231 j = display.Javascript('')
232 nt.assert_equal(repr(j), '<IPython.core.display.Javascript object>')
232 nt.assert_equal(repr(j), '<IPython.core.display.Javascript object>')
233 j._show_mem_addr = True
233 j._show_mem_addr = True
234 nt.assert_equal(repr(j), object.__repr__(j))
234 nt.assert_equal(repr(j), object.__repr__(j))
235 j._show_mem_addr = False
235 j._show_mem_addr = False
236 nt.assert_equal(repr(j), '<IPython.core.display.Javascript object>')
236 nt.assert_equal(repr(j), '<IPython.core.display.Javascript object>')
237
237
238 @mock.patch('warnings.warn')
238 @mock.patch('warnings.warn')
239 def test_encourage_iframe_over_html(m_warn):
239 def test_encourage_iframe_over_html(m_warn):
240 display.HTML()
240 display.HTML()
241 m_warn.assert_not_called()
241 m_warn.assert_not_called()
242
242
243 display.HTML('<br />')
243 display.HTML('<br />')
244 m_warn.assert_not_called()
244 m_warn.assert_not_called()
245
245
246 display.HTML('<html><p>Lots of content here</p><iframe src="http://a.com"></iframe>')
246 display.HTML('<html><p>Lots of content here</p><iframe src="http://a.com"></iframe>')
247 m_warn.assert_not_called()
247 m_warn.assert_not_called()
248
248
249 display.HTML('<iframe src="http://a.com"></iframe>')
249 display.HTML('<iframe src="http://a.com"></iframe>')
250 m_warn.assert_called_with('Consider using IPython.display.IFrame instead')
250 m_warn.assert_called_with('Consider using IPython.display.IFrame instead')
251
251
252 m_warn.reset_mock()
252 m_warn.reset_mock()
253 display.HTML('<IFRAME SRC="http://a.com"></IFRAME>')
253 display.HTML('<IFRAME SRC="http://a.com"></IFRAME>')
254 m_warn.assert_called_with('Consider using IPython.display.IFrame instead')
254 m_warn.assert_called_with('Consider using IPython.display.IFrame instead')
255
255
256 def test_progress():
256 def test_progress():
257 p = display.ProgressBar(10)
257 p = display.ProgressBar(10)
258 nt.assert_in('0/10',repr(p))
258 nt.assert_in('0/10',repr(p))
259 p.html_width = '100%'
259 p.html_width = '100%'
260 p.progress = 5
260 p.progress = 5
261 nt.assert_equal(p._repr_html_(), "<progress style='width:100%' max='10' value='5'></progress>")
261 nt.assert_equal(p._repr_html_(), "<progress style='width:100%' max='10' value='5'></progress>")
262
262
263 def test_progress_iter():
263 def test_progress_iter():
264 with capture_output(display=False) as captured:
264 with capture_output(display=False) as captured:
265 for i in display.ProgressBar(5):
265 for i in display.ProgressBar(5):
266 out = captured.stdout
266 out = captured.stdout
267 nt.assert_in('{0}/5'.format(i), out)
267 nt.assert_in('{0}/5'.format(i), out)
268 out = captured.stdout
268 out = captured.stdout
269 nt.assert_in('5/5', out)
269 nt.assert_in('5/5', out)
270
270
271 def test_json():
271 def test_json():
272 d = {'a': 5}
272 d = {'a': 5}
273 lis = [d]
273 lis = [d]
274 metadata = [
274 metadata = [
275 {'expanded': False, 'root': 'root'},
275 {'expanded': False, 'root': 'root'},
276 {'expanded': True, 'root': 'root'},
276 {'expanded': True, 'root': 'root'},
277 {'expanded': False, 'root': 'custom'},
277 {'expanded': False, 'root': 'custom'},
278 {'expanded': True, 'root': 'custom'},
278 {'expanded': True, 'root': 'custom'},
279 ]
279 ]
280 json_objs = [
280 json_objs = [
281 display.JSON(d),
281 display.JSON(d),
282 display.JSON(d, expanded=True),
282 display.JSON(d, expanded=True),
283 display.JSON(d, root='custom'),
283 display.JSON(d, root='custom'),
284 display.JSON(d, expanded=True, root='custom'),
284 display.JSON(d, expanded=True, root='custom'),
285 ]
285 ]
286 for j, md in zip(json_objs, metadata):
286 for j, md in zip(json_objs, metadata):
287 nt.assert_equal(j._repr_json_(), (d, md))
287 nt.assert_equal(j._repr_json_(), (d, md))
288
288
289 with warnings.catch_warnings(record=True) as w:
289 with warnings.catch_warnings(record=True) as w:
290 warnings.simplefilter("always")
290 warnings.simplefilter("always")
291 j = display.JSON(json.dumps(d))
291 j = display.JSON(json.dumps(d))
292 nt.assert_equal(len(w), 1)
292 nt.assert_equal(len(w), 1)
293 nt.assert_equal(j._repr_json_(), (d, metadata[0]))
293 nt.assert_equal(j._repr_json_(), (d, metadata[0]))
294
294
295 json_objs = [
295 json_objs = [
296 display.JSON(lis),
296 display.JSON(lis),
297 display.JSON(lis, expanded=True),
297 display.JSON(lis, expanded=True),
298 display.JSON(lis, root='custom'),
298 display.JSON(lis, root='custom'),
299 display.JSON(lis, expanded=True, root='custom'),
299 display.JSON(lis, expanded=True, root='custom'),
300 ]
300 ]
301 for j, md in zip(json_objs, metadata):
301 for j, md in zip(json_objs, metadata):
302 nt.assert_equal(j._repr_json_(), (lis, md))
302 nt.assert_equal(j._repr_json_(), (lis, md))
303
303
304 with warnings.catch_warnings(record=True) as w:
304 with warnings.catch_warnings(record=True) as w:
305 warnings.simplefilter("always")
305 warnings.simplefilter("always")
306 j = display.JSON(json.dumps(lis))
306 j = display.JSON(json.dumps(lis))
307 nt.assert_equal(len(w), 1)
307 nt.assert_equal(len(w), 1)
308 nt.assert_equal(j._repr_json_(), (lis, metadata[0]))
308 nt.assert_equal(j._repr_json_(), (lis, metadata[0]))
309
309
310 def test_video_embedding():
310 def test_video_embedding():
311 """use a tempfile, with dummy-data, to ensure that video embedding doesn't crash"""
311 """use a tempfile, with dummy-data, to ensure that video embedding doesn't crash"""
312 v = display.Video("http://ignored")
312 v = display.Video("http://ignored")
313 assert not v.embed
313 assert not v.embed
314 html = v._repr_html_()
314 html = v._repr_html_()
315 nt.assert_not_in('src="data:', html)
315 nt.assert_not_in('src="data:', html)
316 nt.assert_in('src="http://ignored"', html)
316 nt.assert_in('src="http://ignored"', html)
317
317
318 with nt.assert_raises(ValueError):
318 with nt.assert_raises(ValueError):
319 v = display.Video(b'abc')
319 v = display.Video(b'abc')
320
320
321 with NamedFileInTemporaryDirectory('test.mp4') as f:
321 with NamedFileInTemporaryDirectory('test.mp4') as f:
322 f.write(b'abc')
322 f.write(b'abc')
323 f.close()
323 f.close()
324
324
325 v = display.Video(f.name)
325 v = display.Video(f.name)
326 assert not v.embed
326 assert not v.embed
327 html = v._repr_html_()
327 html = v._repr_html_()
328 nt.assert_not_in('src="data:', html)
328 nt.assert_not_in('src="data:', html)
329
329
330 v = display.Video(f.name, embed=True)
330 v = display.Video(f.name, embed=True)
331 html = v._repr_html_()
331 html = v._repr_html_()
332 nt.assert_in('src="data:video/mp4;base64,YWJj"',html)
332 nt.assert_in('src="data:video/mp4;base64,YWJj"',html)
333
333
334 v = display.Video(f.name, embed=True, mimetype='video/other')
334 v = display.Video(f.name, embed=True, mimetype='video/other')
335 html = v._repr_html_()
335 html = v._repr_html_()
336 nt.assert_in('src="data:video/other;base64,YWJj"',html)
336 nt.assert_in('src="data:video/other;base64,YWJj"',html)
337
337
338 v = display.Video(b'abc', embed=True, mimetype='video/mp4')
338 v = display.Video(b'abc', embed=True, mimetype='video/mp4')
339 html = v._repr_html_()
339 html = v._repr_html_()
340 nt.assert_in('src="data:video/mp4;base64,YWJj"',html)
340 nt.assert_in('src="data:video/mp4;base64,YWJj"',html)
341
341
342 v = display.Video(u'YWJj', embed=True, mimetype='video/xyz')
342 v = display.Video(u'YWJj', embed=True, mimetype='video/xyz')
343 html = v._repr_html_()
343 html = v._repr_html_()
344 nt.assert_in('src="data:video/xyz;base64,YWJj"',html)
344 nt.assert_in('src="data:video/xyz;base64,YWJj"',html)
345
345
346 def test_html_metadata():
346 def test_html_metadata():
347 s = "<h1>Test</h1>"
347 s = "<h1>Test</h1>"
348 h = display.HTML(s, metadata={"isolated": True})
348 h = display.HTML(s, metadata={"isolated": True})
349 nt.assert_equal(h._repr_html_(), (s, {"isolated": True}))
349 nt.assert_equal(h._repr_html_(), (s, {"isolated": True}))
350
350
351 def test_display_id():
351 def test_display_id():
352 ip = get_ipython()
352 ip = get_ipython()
353 with mock.patch.object(ip.display_pub, 'publish') as pub:
353 with mock.patch.object(ip.display_pub, 'publish') as pub:
354 handle = display.display('x')
354 handle = display.display('x')
355 nt.assert_is(handle, None)
355 nt.assert_is(handle, None)
356 handle = display.display('y', display_id='secret')
356 handle = display.display('y', display_id='secret')
357 nt.assert_is_instance(handle, display.DisplayHandle)
357 nt.assert_is_instance(handle, display.DisplayHandle)
358 handle2 = display.display('z', display_id=True)
358 handle2 = display.display('z', display_id=True)
359 nt.assert_is_instance(handle2, display.DisplayHandle)
359 nt.assert_is_instance(handle2, display.DisplayHandle)
360 nt.assert_not_equal(handle.display_id, handle2.display_id)
360 nt.assert_not_equal(handle.display_id, handle2.display_id)
361
361
362 nt.assert_equal(pub.call_count, 3)
362 nt.assert_equal(pub.call_count, 3)
363 args, kwargs = pub.call_args_list[0]
363 args, kwargs = pub.call_args_list[0]
364 nt.assert_equal(args, ())
364 nt.assert_equal(args, ())
365 nt.assert_equal(kwargs, {
365 nt.assert_equal(kwargs, {
366 'data': {
366 'data': {
367 'text/plain': repr('x')
367 'text/plain': repr('x')
368 },
368 },
369 'metadata': {},
369 'metadata': {},
370 })
370 })
371 args, kwargs = pub.call_args_list[1]
371 args, kwargs = pub.call_args_list[1]
372 nt.assert_equal(args, ())
372 nt.assert_equal(args, ())
373 nt.assert_equal(kwargs, {
373 nt.assert_equal(kwargs, {
374 'data': {
374 'data': {
375 'text/plain': repr('y')
375 'text/plain': repr('y')
376 },
376 },
377 'metadata': {},
377 'metadata': {},
378 'transient': {
378 'transient': {
379 'display_id': handle.display_id,
379 'display_id': handle.display_id,
380 },
380 },
381 })
381 })
382 args, kwargs = pub.call_args_list[2]
382 args, kwargs = pub.call_args_list[2]
383 nt.assert_equal(args, ())
383 nt.assert_equal(args, ())
384 nt.assert_equal(kwargs, {
384 nt.assert_equal(kwargs, {
385 'data': {
385 'data': {
386 'text/plain': repr('z')
386 'text/plain': repr('z')
387 },
387 },
388 'metadata': {},
388 'metadata': {},
389 'transient': {
389 'transient': {
390 'display_id': handle2.display_id,
390 'display_id': handle2.display_id,
391 },
391 },
392 })
392 })
393
393
394
394
395 def test_update_display():
395 def test_update_display():
396 ip = get_ipython()
396 ip = get_ipython()
397 with mock.patch.object(ip.display_pub, 'publish') as pub:
397 with mock.patch.object(ip.display_pub, 'publish') as pub:
398 with nt.assert_raises(TypeError):
398 with nt.assert_raises(TypeError):
399 display.update_display('x')
399 display.update_display('x')
400 display.update_display('x', display_id='1')
400 display.update_display('x', display_id='1')
401 display.update_display('y', display_id='2')
401 display.update_display('y', display_id='2')
402 args, kwargs = pub.call_args_list[0]
402 args, kwargs = pub.call_args_list[0]
403 nt.assert_equal(args, ())
403 nt.assert_equal(args, ())
404 nt.assert_equal(kwargs, {
404 nt.assert_equal(kwargs, {
405 'data': {
405 'data': {
406 'text/plain': repr('x')
406 'text/plain': repr('x')
407 },
407 },
408 'metadata': {},
408 'metadata': {},
409 'transient': {
409 'transient': {
410 'display_id': '1',
410 'display_id': '1',
411 },
411 },
412 'update': True,
412 'update': True,
413 })
413 })
414 args, kwargs = pub.call_args_list[1]
414 args, kwargs = pub.call_args_list[1]
415 nt.assert_equal(args, ())
415 nt.assert_equal(args, ())
416 nt.assert_equal(kwargs, {
416 nt.assert_equal(kwargs, {
417 'data': {
417 'data': {
418 'text/plain': repr('y')
418 'text/plain': repr('y')
419 },
419 },
420 'metadata': {},
420 'metadata': {},
421 'transient': {
421 'transient': {
422 'display_id': '2',
422 'display_id': '2',
423 },
423 },
424 'update': True,
424 'update': True,
425 })
425 })
426
426
427
427
428 def test_display_handle():
428 def test_display_handle():
429 ip = get_ipython()
429 ip = get_ipython()
430 handle = display.DisplayHandle()
430 handle = display.DisplayHandle()
431 nt.assert_is_instance(handle.display_id, str)
431 nt.assert_is_instance(handle.display_id, str)
432 handle = display.DisplayHandle('my-id')
432 handle = display.DisplayHandle('my-id')
433 nt.assert_equal(handle.display_id, 'my-id')
433 nt.assert_equal(handle.display_id, 'my-id')
434 with mock.patch.object(ip.display_pub, 'publish') as pub:
434 with mock.patch.object(ip.display_pub, 'publish') as pub:
435 handle.display('x')
435 handle.display('x')
436 handle.update('y')
436 handle.update('y')
437
437
438 args, kwargs = pub.call_args_list[0]
438 args, kwargs = pub.call_args_list[0]
439 nt.assert_equal(args, ())
439 nt.assert_equal(args, ())
440 nt.assert_equal(kwargs, {
440 nt.assert_equal(kwargs, {
441 'data': {
441 'data': {
442 'text/plain': repr('x')
442 'text/plain': repr('x')
443 },
443 },
444 'metadata': {},
444 'metadata': {},
445 'transient': {
445 'transient': {
446 'display_id': handle.display_id,
446 'display_id': handle.display_id,
447 }
447 }
448 })
448 })
449 args, kwargs = pub.call_args_list[1]
449 args, kwargs = pub.call_args_list[1]
450 nt.assert_equal(args, ())
450 nt.assert_equal(args, ())
451 nt.assert_equal(kwargs, {
451 nt.assert_equal(kwargs, {
452 'data': {
452 'data': {
453 'text/plain': repr('y')
453 'text/plain': repr('y')
454 },
454 },
455 'metadata': {},
455 'metadata': {},
456 'transient': {
456 'transient': {
457 'display_id': handle.display_id,
457 'display_id': handle.display_id,
458 },
458 },
459 'update': True,
459 'update': True,
460 })
460 })
461
461
462
462
463 def test_image_alt_tag():
463 def test_image_alt_tag():
464 """Simple test for display.Image(args, alt=x,)"""
464 """Simple test for display.Image(args, alt=x,)"""
465 thisurl = "http://example.com/image.png"
465 thisurl = "http://example.com/image.png"
466 img = display.Image(url=thisurl, alt="an image")
466 img = display.Image(url=thisurl, alt="an image")
467 nt.assert_equal(u'<img src="%s" alt="an image"/>' % (thisurl), img._repr_html_())
467 nt.assert_equal(u'<img src="%s" alt="an image"/>' % (thisurl), img._repr_html_())
468 img = display.Image(url=thisurl, unconfined=True, alt="an image")
468 img = display.Image(url=thisurl, unconfined=True, alt="an image")
469 nt.assert_equal(
469 nt.assert_equal(
470 u'<img src="%s" class="unconfined" alt="an image"/>' % (thisurl),
470 u'<img src="%s" class="unconfined" alt="an image"/>' % (thisurl),
471 img._repr_html_(),
471 img._repr_html_(),
472 )
472 )
473 img = display.Image(url=thisurl, alt='>"& <')
473 img = display.Image(url=thisurl, alt='>"& <')
474 nt.assert_equal(
474 nt.assert_equal(
475 u'<img src="%s" alt="&gt;&quot;&amp; &lt;"/>' % (thisurl), img._repr_html_()
475 u'<img src="%s" alt="&gt;&quot;&amp; &lt;"/>' % (thisurl), img._repr_html_()
476 )
476 )
477
477
478 img = display.Image(url=thisurl, metadata={"alt": "an image"})
478 img = display.Image(url=thisurl, metadata={"alt": "an image"})
479 nt.assert_equal(img.alt, "an image")
479 nt.assert_equal(img.alt, "an image")
480
480
481 here = os.path.dirname(__file__)
481 here = os.path.dirname(__file__)
482 img = display.Image(os.path.join(here, "2x2.png"), alt="an image")
482 img = display.Image(os.path.join(here, "2x2.png"), alt="an image")
483 nt.assert_equal(img.alt, "an image")
483 nt.assert_equal(img.alt, "an image")
484 _, md = img._repr_png_()
484 _, md = img._repr_png_()
485 nt.assert_equal(md["alt"], "an image")
485 nt.assert_equal(md["alt"], "an image")
486
486
487
487
488 @nt.raises(FileNotFoundError)
488 @nt.raises(FileNotFoundError)
489 def test_image_bad_filename_raises_proper_exception():
489 def test_image_bad_filename_raises_proper_exception():
490 display.Image("/this/file/does/not/exist/")._repr_png_()
490 display.Image("/this/file/does/not/exist/")._repr_png_()
@@ -1,355 +1,355 b''
1 """Tests for the token-based transformers in IPython.core.inputtransformer2
1 """Tests for the token-based transformers in IPython.core.inputtransformer2
2
2
3 Line-based transformers are the simpler ones; token-based transformers are
3 Line-based transformers are the simpler ones; token-based transformers are
4 more complex. See test_inputtransformer2_line for tests for line-based
4 more complex. See test_inputtransformer2_line for tests for line-based
5 transformations.
5 transformations.
6 """
6 """
7 import nose.tools as nt
7 import nose.tools as nt
8 import string
8 import string
9
9
10 from IPython.core import inputtransformer2 as ipt2
10 from IPython.core import inputtransformer2 as ipt2
11 from IPython.core.inputtransformer2 import make_tokens_by_line, _find_assign_op
11 from IPython.core.inputtransformer2 import make_tokens_by_line, _find_assign_op
12
12
13 from textwrap import dedent
13 from textwrap import dedent
14
14
15 MULTILINE_MAGIC = ("""\
15 MULTILINE_MAGIC = ("""\
16 a = f()
16 a = f()
17 %foo \\
17 %foo \\
18 bar
18 bar
19 g()
19 g()
20 """.splitlines(keepends=True), (2, 0), """\
20 """.splitlines(keepends=True), (2, 0), """\
21 a = f()
21 a = f()
22 get_ipython().run_line_magic('foo', ' bar')
22 get_ipython().run_line_magic('foo', ' bar')
23 g()
23 g()
24 """.splitlines(keepends=True))
24 """.splitlines(keepends=True))
25
25
26 INDENTED_MAGIC = ("""\
26 INDENTED_MAGIC = ("""\
27 for a in range(5):
27 for a in range(5):
28 %ls
28 %ls
29 """.splitlines(keepends=True), (2, 4), """\
29 """.splitlines(keepends=True), (2, 4), """\
30 for a in range(5):
30 for a in range(5):
31 get_ipython().run_line_magic('ls', '')
31 get_ipython().run_line_magic('ls', '')
32 """.splitlines(keepends=True))
32 """.splitlines(keepends=True))
33
33
34 CRLF_MAGIC = ([
34 CRLF_MAGIC = ([
35 "a = f()\n",
35 "a = f()\n",
36 "%ls\r\n",
36 "%ls\r\n",
37 "g()\n"
37 "g()\n"
38 ], (2, 0), [
38 ], (2, 0), [
39 "a = f()\n",
39 "a = f()\n",
40 "get_ipython().run_line_magic('ls', '')\n",
40 "get_ipython().run_line_magic('ls', '')\n",
41 "g()\n"
41 "g()\n"
42 ])
42 ])
43
43
44 MULTILINE_MAGIC_ASSIGN = ("""\
44 MULTILINE_MAGIC_ASSIGN = ("""\
45 a = f()
45 a = f()
46 b = %foo \\
46 b = %foo \\
47 bar
47 bar
48 g()
48 g()
49 """.splitlines(keepends=True), (2, 4), """\
49 """.splitlines(keepends=True), (2, 4), """\
50 a = f()
50 a = f()
51 b = get_ipython().run_line_magic('foo', ' bar')
51 b = get_ipython().run_line_magic('foo', ' bar')
52 g()
52 g()
53 """.splitlines(keepends=True))
53 """.splitlines(keepends=True))
54
54
55 MULTILINE_SYSTEM_ASSIGN = ("""\
55 MULTILINE_SYSTEM_ASSIGN = ("""\
56 a = f()
56 a = f()
57 b = !foo \\
57 b = !foo \\
58 bar
58 bar
59 g()
59 g()
60 """.splitlines(keepends=True), (2, 4), """\
60 """.splitlines(keepends=True), (2, 4), """\
61 a = f()
61 a = f()
62 b = get_ipython().getoutput('foo bar')
62 b = get_ipython().getoutput('foo bar')
63 g()
63 g()
64 """.splitlines(keepends=True))
64 """.splitlines(keepends=True))
65
65
66 #####
66 #####
67
67
68 MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT = ("""\
68 MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT = ("""\
69 def test():
69 def test():
70 for i in range(1):
70 for i in range(1):
71 print(i)
71 print(i)
72 res =! ls
72 res =! ls
73 """.splitlines(keepends=True), (4, 7), '''\
73 """.splitlines(keepends=True), (4, 7), '''\
74 def test():
74 def test():
75 for i in range(1):
75 for i in range(1):
76 print(i)
76 print(i)
77 res =get_ipython().getoutput(\' ls\')
77 res =get_ipython().getoutput(\' ls\')
78 '''.splitlines(keepends=True))
78 '''.splitlines(keepends=True))
79
79
80 ######
80 ######
81
81
82 AUTOCALL_QUOTE = (
82 AUTOCALL_QUOTE = (
83 [",f 1 2 3\n"], (1, 0),
83 [",f 1 2 3\n"], (1, 0),
84 ['f("1", "2", "3")\n']
84 ['f("1", "2", "3")\n']
85 )
85 )
86
86
87 AUTOCALL_QUOTE2 = (
87 AUTOCALL_QUOTE2 = (
88 [";f 1 2 3\n"], (1, 0),
88 [";f 1 2 3\n"], (1, 0),
89 ['f("1 2 3")\n']
89 ['f("1 2 3")\n']
90 )
90 )
91
91
92 AUTOCALL_PAREN = (
92 AUTOCALL_PAREN = (
93 ["/f 1 2 3\n"], (1, 0),
93 ["/f 1 2 3\n"], (1, 0),
94 ['f(1, 2, 3)\n']
94 ['f(1, 2, 3)\n']
95 )
95 )
96
96
97 SIMPLE_HELP = (
97 SIMPLE_HELP = (
98 ["foo?\n"], (1, 0),
98 ["foo?\n"], (1, 0),
99 ["get_ipython().run_line_magic('pinfo', 'foo')\n"]
99 ["get_ipython().run_line_magic('pinfo', 'foo')\n"]
100 )
100 )
101
101
102 DETAILED_HELP = (
102 DETAILED_HELP = (
103 ["foo??\n"], (1, 0),
103 ["foo??\n"], (1, 0),
104 ["get_ipython().run_line_magic('pinfo2', 'foo')\n"]
104 ["get_ipython().run_line_magic('pinfo2', 'foo')\n"]
105 )
105 )
106
106
107 MAGIC_HELP = (
107 MAGIC_HELP = (
108 ["%foo?\n"], (1, 0),
108 ["%foo?\n"], (1, 0),
109 ["get_ipython().run_line_magic('pinfo', '%foo')\n"]
109 ["get_ipython().run_line_magic('pinfo', '%foo')\n"]
110 )
110 )
111
111
112 HELP_IN_EXPR = (
112 HELP_IN_EXPR = (
113 ["a = b + c?\n"], (1, 0),
113 ["a = b + c?\n"], (1, 0),
114 ["get_ipython().set_next_input('a = b + c');"
114 ["get_ipython().set_next_input('a = b + c');"
115 "get_ipython().run_line_magic('pinfo', 'c')\n"]
115 "get_ipython().run_line_magic('pinfo', 'c')\n"]
116 )
116 )
117
117
118 HELP_CONTINUED_LINE = ("""\
118 HELP_CONTINUED_LINE = ("""\
119 a = \\
119 a = \\
120 zip?
120 zip?
121 """.splitlines(keepends=True), (1, 0),
121 """.splitlines(keepends=True), (1, 0),
122 [r"get_ipython().set_next_input('a = \\\nzip');get_ipython().run_line_magic('pinfo', 'zip')" + "\n"]
122 [r"get_ipython().set_next_input('a = \\\nzip');get_ipython().run_line_magic('pinfo', 'zip')" + "\n"]
123 )
123 )
124
124
125 HELP_MULTILINE = ("""\
125 HELP_MULTILINE = ("""\
126 (a,
126 (a,
127 b) = zip?
127 b) = zip?
128 """.splitlines(keepends=True), (1, 0),
128 """.splitlines(keepends=True), (1, 0),
129 [r"get_ipython().set_next_input('(a,\nb) = zip');get_ipython().run_line_magic('pinfo', 'zip')" + "\n"]
129 [r"get_ipython().set_next_input('(a,\nb) = zip');get_ipython().run_line_magic('pinfo', 'zip')" + "\n"]
130 )
130 )
131
131
132 HELP_UNICODE = (
132 HELP_UNICODE = (
133 ["π.foo?\n"], (1, 0),
133 ["π.foo?\n"], (1, 0),
134 ["get_ipython().run_line_magic('pinfo', 'π.foo')\n"]
134 ["get_ipython().run_line_magic('pinfo', 'π.foo')\n"]
135 )
135 )
136
136
137
137
138 def null_cleanup_transformer(lines):
138 def null_cleanup_transformer(lines):
139 """
139 """
140 A cleanup transform that returns an empty list.
140 A cleanup transform that returns an empty list.
141 """
141 """
142 return []
142 return []
143
143
144 def check_make_token_by_line_never_ends_empty():
144 def check_make_token_by_line_never_ends_empty():
145 """
145 """
146 Check that not sequence of single or double characters ends up leading to en empty list of tokens
146 Check that not sequence of single or double characters ends up leading to en empty list of tokens
147 """
147 """
148 from string import printable
148 from string import printable
149 for c in printable:
149 for c in printable:
150 nt.assert_not_equal(make_tokens_by_line(c)[-1], [])
150 nt.assert_not_equal(make_tokens_by_line(c)[-1], [])
151 for k in printable:
151 for k in printable:
152 nt.assert_not_equal(make_tokens_by_line(c+k)[-1], [])
152 nt.assert_not_equal(make_tokens_by_line(c+k)[-1], [])
153
153
154 def check_find(transformer, case, match=True):
154 def check_find(transformer, case, match=True):
155 sample, expected_start, _ = case
155 sample, expected_start, _ = case
156 tbl = make_tokens_by_line(sample)
156 tbl = make_tokens_by_line(sample)
157 res = transformer.find(tbl)
157 res = transformer.find(tbl)
158 if match:
158 if match:
159 # start_line is stored 0-indexed, expected values are 1-indexed
159 # start_line is stored 0-indexed, expected values are 1-indexed
160 nt.assert_equal((res.start_line+1, res.start_col), expected_start)
160 nt.assert_equal((res.start_line+1, res.start_col), expected_start)
161 return res
161 return res
162 else:
162 else:
163 nt.assert_is(res, None)
163 nt.assert_is(res, None)
164
164
165 def check_transform(transformer_cls, case):
165 def check_transform(transformer_cls, case):
166 lines, start, expected = case
166 lines, start, expected = case
167 transformer = transformer_cls(start)
167 transformer = transformer_cls(start)
168 nt.assert_equal(transformer.transform(lines), expected)
168 nt.assert_equal(transformer.transform(lines), expected)
169
169
170 def test_continued_line():
170 def test_continued_line():
171 lines = MULTILINE_MAGIC_ASSIGN[0]
171 lines = MULTILINE_MAGIC_ASSIGN[0]
172 nt.assert_equal(ipt2.find_end_of_continued_line(lines, 1), 2)
172 nt.assert_equal(ipt2.find_end_of_continued_line(lines, 1), 2)
173
173
174 nt.assert_equal(ipt2.assemble_continued_line(lines, (1, 5), 2), "foo bar")
174 nt.assert_equal(ipt2.assemble_continued_line(lines, (1, 5), 2), "foo bar")
175
175
176 def test_find_assign_magic():
176 def test_find_assign_magic():
177 check_find(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
177 check_find(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
178 check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN, match=False)
178 check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN, match=False)
179 check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT, match=False)
179 check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT, match=False)
180
180
181 def test_transform_assign_magic():
181 def test_transform_assign_magic():
182 check_transform(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
182 check_transform(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
183
183
184 def test_find_assign_system():
184 def test_find_assign_system():
185 check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
185 check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
186 check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
186 check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
187 check_find(ipt2.SystemAssign, (["a = !ls\n"], (1, 5), None))
187 check_find(ipt2.SystemAssign, (["a = !ls\n"], (1, 5), None))
188 check_find(ipt2.SystemAssign, (["a=!ls\n"], (1, 2), None))
188 check_find(ipt2.SystemAssign, (["a=!ls\n"], (1, 2), None))
189 check_find(ipt2.SystemAssign, MULTILINE_MAGIC_ASSIGN, match=False)
189 check_find(ipt2.SystemAssign, MULTILINE_MAGIC_ASSIGN, match=False)
190
190
191 def test_transform_assign_system():
191 def test_transform_assign_system():
192 check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
192 check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
193 check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
193 check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
194
194
195 def test_find_magic_escape():
195 def test_find_magic_escape():
196 check_find(ipt2.EscapedCommand, MULTILINE_MAGIC)
196 check_find(ipt2.EscapedCommand, MULTILINE_MAGIC)
197 check_find(ipt2.EscapedCommand, INDENTED_MAGIC)
197 check_find(ipt2.EscapedCommand, INDENTED_MAGIC)
198 check_find(ipt2.EscapedCommand, MULTILINE_MAGIC_ASSIGN, match=False)
198 check_find(ipt2.EscapedCommand, MULTILINE_MAGIC_ASSIGN, match=False)
199
199
200 def test_transform_magic_escape():
200 def test_transform_magic_escape():
201 check_transform(ipt2.EscapedCommand, MULTILINE_MAGIC)
201 check_transform(ipt2.EscapedCommand, MULTILINE_MAGIC)
202 check_transform(ipt2.EscapedCommand, INDENTED_MAGIC)
202 check_transform(ipt2.EscapedCommand, INDENTED_MAGIC)
203 check_transform(ipt2.EscapedCommand, CRLF_MAGIC)
203 check_transform(ipt2.EscapedCommand, CRLF_MAGIC)
204
204
205 def test_find_autocalls():
205 def test_find_autocalls():
206 for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
206 for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
207 print("Testing %r" % case[0])
207 print("Testing %r" % case[0])
208 check_find(ipt2.EscapedCommand, case)
208 check_find(ipt2.EscapedCommand, case)
209
209
210 def test_transform_autocall():
210 def test_transform_autocall():
211 for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
211 for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
212 print("Testing %r" % case[0])
212 print("Testing %r" % case[0])
213 check_transform(ipt2.EscapedCommand, case)
213 check_transform(ipt2.EscapedCommand, case)
214
214
215 def test_find_help():
215 def test_find_help():
216 for case in [SIMPLE_HELP, DETAILED_HELP, MAGIC_HELP, HELP_IN_EXPR]:
216 for case in [SIMPLE_HELP, DETAILED_HELP, MAGIC_HELP, HELP_IN_EXPR]:
217 check_find(ipt2.HelpEnd, case)
217 check_find(ipt2.HelpEnd, case)
218
218
219 tf = check_find(ipt2.HelpEnd, HELP_CONTINUED_LINE)
219 tf = check_find(ipt2.HelpEnd, HELP_CONTINUED_LINE)
220 nt.assert_equal(tf.q_line, 1)
220 nt.assert_equal(tf.q_line, 1)
221 nt.assert_equal(tf.q_col, 3)
221 nt.assert_equal(tf.q_col, 3)
222
222
223 tf = check_find(ipt2.HelpEnd, HELP_MULTILINE)
223 tf = check_find(ipt2.HelpEnd, HELP_MULTILINE)
224 nt.assert_equal(tf.q_line, 1)
224 nt.assert_equal(tf.q_line, 1)
225 nt.assert_equal(tf.q_col, 8)
225 nt.assert_equal(tf.q_col, 8)
226
226
227 # ? in a comment does not trigger help
227 # ? in a comment does not trigger help
228 check_find(ipt2.HelpEnd, (["foo # bar?\n"], None, None), match=False)
228 check_find(ipt2.HelpEnd, (["foo # bar?\n"], None, None), match=False)
229 # Nor in a string
229 # Nor in a string
230 check_find(ipt2.HelpEnd, (["foo = '''bar?\n"], None, None), match=False)
230 check_find(ipt2.HelpEnd, (["foo = '''bar?\n"], None, None), match=False)
231
231
232 def test_transform_help():
232 def test_transform_help():
233 tf = ipt2.HelpEnd((1, 0), (1, 9))
233 tf = ipt2.HelpEnd((1, 0), (1, 9))
234 nt.assert_equal(tf.transform(HELP_IN_EXPR[0]), HELP_IN_EXPR[2])
234 nt.assert_equal(tf.transform(HELP_IN_EXPR[0]), HELP_IN_EXPR[2])
235
235
236 tf = ipt2.HelpEnd((1, 0), (2, 3))
236 tf = ipt2.HelpEnd((1, 0), (2, 3))
237 nt.assert_equal(tf.transform(HELP_CONTINUED_LINE[0]), HELP_CONTINUED_LINE[2])
237 nt.assert_equal(tf.transform(HELP_CONTINUED_LINE[0]), HELP_CONTINUED_LINE[2])
238
238
239 tf = ipt2.HelpEnd((1, 0), (2, 8))
239 tf = ipt2.HelpEnd((1, 0), (2, 8))
240 nt.assert_equal(tf.transform(HELP_MULTILINE[0]), HELP_MULTILINE[2])
240 nt.assert_equal(tf.transform(HELP_MULTILINE[0]), HELP_MULTILINE[2])
241
241
242 tf = ipt2.HelpEnd((1, 0), (1, 0))
242 tf = ipt2.HelpEnd((1, 0), (1, 0))
243 nt.assert_equal(tf.transform(HELP_UNICODE[0]), HELP_UNICODE[2])
243 nt.assert_equal(tf.transform(HELP_UNICODE[0]), HELP_UNICODE[2])
244
244
245 def test_find_assign_op_dedent():
245 def test_find_assign_op_dedent():
246 """
246 """
247 be careful that empty token like dedent are not counted as parens
247 be careful that empty token like dedent are not counted as parens
248 """
248 """
249 class Tk:
249 class Tk:
250 def __init__(self, s):
250 def __init__(self, s):
251 self.string = s
251 self.string = s
252
252
253 nt.assert_equal(_find_assign_op([Tk(s) for s in ('','a','=','b')]), 2)
253 nt.assert_equal(_find_assign_op([Tk(s) for s in ('','a','=','b')]), 2)
254 nt.assert_equal(_find_assign_op([Tk(s) for s in ('','(', 'a','=','b', ')', '=' ,'5')]), 6)
254 nt.assert_equal(_find_assign_op([Tk(s) for s in ('','(', 'a','=','b', ')', '=' ,'5')]), 6)
255
255
256 def test_check_complete():
256 def test_check_complete():
257 cc = ipt2.TransformerManager().check_complete
257 cc = ipt2.TransformerManager().check_complete
258 nt.assert_equal(cc("a = 1"), ("complete", None))
258 nt.assert_equal(cc("a = 1"), ("complete", None))
259 nt.assert_equal(cc("for a in range(5):"), ("incomplete", 4))
259 nt.assert_equal(cc("for a in range(5):"), ("incomplete", 4))
260 nt.assert_equal(cc("for a in range(5):\n if a > 0:"), ("incomplete", 8))
260 nt.assert_equal(cc("for a in range(5):\n if a > 0:"), ("incomplete", 8))
261 nt.assert_equal(cc("raise = 2"), ("invalid", None))
261 nt.assert_equal(cc("raise = 2"), ("invalid", None))
262 nt.assert_equal(cc("a = [1,\n2,"), ("incomplete", 0))
262 nt.assert_equal(cc("a = [1,\n2,"), ("incomplete", 0))
263 nt.assert_equal(cc("(\n))"), ("incomplete", 0))
263 nt.assert_equal(cc("(\n))"), ("incomplete", 0))
264 nt.assert_equal(cc("\\\r\n"), ("incomplete", 0))
264 nt.assert_equal(cc("\\\r\n"), ("incomplete", 0))
265 nt.assert_equal(cc("a = '''\n hi"), ("incomplete", 3))
265 nt.assert_equal(cc("a = '''\n hi"), ("incomplete", 3))
266 nt.assert_equal(cc("def a():\n x=1\n global x"), ("invalid", None))
266 nt.assert_equal(cc("def a():\n x=1\n global x"), ("invalid", None))
267 nt.assert_equal(cc("a \\ "), ("invalid", None)) # Nothing allowed after backslash
267 nt.assert_equal(cc("a \\ "), ("invalid", None)) # Nothing allowed after backslash
268 nt.assert_equal(cc("1\\\n+2"), ("complete", None))
268 nt.assert_equal(cc("1\\\n+2"), ("complete", None))
269 nt.assert_equal(cc("exit"), ("complete", None))
269 nt.assert_equal(cc("exit"), ("complete", None))
270
270
271 example = dedent("""
271 example = dedent("""
272 if True:
272 if True:
273 a=1""" )
273 a=1""" )
274
274
275 nt.assert_equal(cc(example), ('incomplete', 4))
275 nt.assert_equal(cc(example), ('incomplete', 4))
276 nt.assert_equal(cc(example+'\n'), ('complete', None))
276 nt.assert_equal(cc(example+'\n'), ('complete', None))
277 nt.assert_equal(cc(example+'\n '), ('complete', None))
277 nt.assert_equal(cc(example+'\n '), ('complete', None))
278
278
279 # no need to loop on all the letters/numbers.
279 # no need to loop on all the letters/numbers.
280 short = '12abAB'+string.printable[62:]
280 short = '12abAB'+string.printable[62:]
281 for c in short:
281 for c in short:
282 # test does not raise:
282 # test does not raise:
283 cc(c)
283 cc(c)
284 for k in short:
284 for k in short:
285 cc(c+k)
285 cc(c+k)
286
286
287 nt.assert_equal(cc("def f():\n x=0\n \\\n "), ('incomplete', 2))
287 nt.assert_equal(cc("def f():\n x=0\n \\\n "), ('incomplete', 2))
288
288
289 def test_check_complete_II():
289 def test_check_complete_II():
290 """
290 """
291 Test that multiple line strings are properly handled.
291 Test that multiple line strings are properly handled.
292
292
293 Separate test function for convenience
293 Separate test function for convenience
294
294
295 """
295 """
296 cc = ipt2.TransformerManager().check_complete
296 cc = ipt2.TransformerManager().check_complete
297 nt.assert_equal(cc('''def foo():\n """'''), ('incomplete', 4))
297 nt.assert_equal(cc('''def foo():\n """'''), ('incomplete', 4))
298
298
299
299
300 def test_check_complete_invalidates_sunken_brackets():
300 def test_check_complete_invalidates_sunken_brackets():
301 """
301 """
302 Test that a single line with more closing brackets than the opening ones is
302 Test that a single line with more closing brackets than the opening ones is
303 interpretted as invalid
303 interpreted as invalid
304 """
304 """
305 cc = ipt2.TransformerManager().check_complete
305 cc = ipt2.TransformerManager().check_complete
306 nt.assert_equal(cc(")"), ("invalid", None))
306 nt.assert_equal(cc(")"), ("invalid", None))
307 nt.assert_equal(cc("]"), ("invalid", None))
307 nt.assert_equal(cc("]"), ("invalid", None))
308 nt.assert_equal(cc("}"), ("invalid", None))
308 nt.assert_equal(cc("}"), ("invalid", None))
309 nt.assert_equal(cc(")("), ("invalid", None))
309 nt.assert_equal(cc(")("), ("invalid", None))
310 nt.assert_equal(cc("]["), ("invalid", None))
310 nt.assert_equal(cc("]["), ("invalid", None))
311 nt.assert_equal(cc("}{"), ("invalid", None))
311 nt.assert_equal(cc("}{"), ("invalid", None))
312 nt.assert_equal(cc("]()("), ("invalid", None))
312 nt.assert_equal(cc("]()("), ("invalid", None))
313 nt.assert_equal(cc("())("), ("invalid", None))
313 nt.assert_equal(cc("())("), ("invalid", None))
314 nt.assert_equal(cc(")[]("), ("invalid", None))
314 nt.assert_equal(cc(")[]("), ("invalid", None))
315 nt.assert_equal(cc("()]("), ("invalid", None))
315 nt.assert_equal(cc("()]("), ("invalid", None))
316
316
317
317
318 def test_null_cleanup_transformer():
318 def test_null_cleanup_transformer():
319 manager = ipt2.TransformerManager()
319 manager = ipt2.TransformerManager()
320 manager.cleanup_transforms.insert(0, null_cleanup_transformer)
320 manager.cleanup_transforms.insert(0, null_cleanup_transformer)
321 assert manager.transform_cell("") == ""
321 assert manager.transform_cell("") == ""
322
322
323
323
324
324
325
325
326 def test_side_effects_I():
326 def test_side_effects_I():
327 count = 0
327 count = 0
328 def counter(lines):
328 def counter(lines):
329 nonlocal count
329 nonlocal count
330 count += 1
330 count += 1
331 return lines
331 return lines
332
332
333 counter.has_side_effects = True
333 counter.has_side_effects = True
334
334
335 manager = ipt2.TransformerManager()
335 manager = ipt2.TransformerManager()
336 manager.cleanup_transforms.insert(0, counter)
336 manager.cleanup_transforms.insert(0, counter)
337 assert manager.check_complete("a=1\n") == ('complete', None)
337 assert manager.check_complete("a=1\n") == ('complete', None)
338 assert count == 0
338 assert count == 0
339
339
340
340
341
341
342
342
343 def test_side_effects_II():
343 def test_side_effects_II():
344 count = 0
344 count = 0
345 def counter(lines):
345 def counter(lines):
346 nonlocal count
346 nonlocal count
347 count += 1
347 count += 1
348 return lines
348 return lines
349
349
350 counter.has_side_effects = True
350 counter.has_side_effects = True
351
351
352 manager = ipt2.TransformerManager()
352 manager = ipt2.TransformerManager()
353 manager.line_transforms.insert(0, counter)
353 manager.line_transforms.insert(0, counter)
354 assert manager.check_complete("b=1\n") == ('complete', None)
354 assert manager.check_complete("b=1\n") == ('complete', None)
355 assert count == 0
355 assert count == 0
@@ -1,601 +1,601 b''
1 # encoding: utf-8
1 # encoding: utf-8
2 """Tests for code execution (%run and related), which is particularly tricky.
2 """Tests for code execution (%run and related), which is particularly tricky.
3
3
4 Because of how %run manages namespaces, and the fact that we are trying here to
4 Because of how %run manages namespaces, and the fact that we are trying here to
5 verify subtle object deletion and reference counting issues, the %run tests
5 verify subtle object deletion and reference counting issues, the %run tests
6 will be kept in this separate file. This makes it easier to aggregate in one
6 will be kept in this separate file. This makes it easier to aggregate in one
7 place the tricks needed to handle it; most other magics are much easier to test
7 place the tricks needed to handle it; most other magics are much easier to test
8 and we do so in a common test_magic file.
8 and we do so in a common test_magic file.
9
9
10 Note that any test using `run -i` should make sure to do a `reset` afterwards,
10 Note that any test using `run -i` should make sure to do a `reset` afterwards,
11 as otherwise it may influence later tests.
11 as otherwise it may influence later tests.
12 """
12 """
13
13
14 # Copyright (c) IPython Development Team.
14 # Copyright (c) IPython Development Team.
15 # Distributed under the terms of the Modified BSD License.
15 # Distributed under the terms of the Modified BSD License.
16
16
17
17
18
18
19 import functools
19 import functools
20 import os
20 import os
21 from os.path import join as pjoin
21 from os.path import join as pjoin
22 import random
22 import random
23 import string
23 import string
24 import sys
24 import sys
25 import textwrap
25 import textwrap
26 import unittest
26 import unittest
27 from unittest.mock import patch
27 from unittest.mock import patch
28
28
29 import nose.tools as nt
29 import nose.tools as nt
30 from nose import SkipTest
30 from nose import SkipTest
31
31
32 from IPython.testing import decorators as dec
32 from IPython.testing import decorators as dec
33 from IPython.testing import tools as tt
33 from IPython.testing import tools as tt
34 from IPython.utils.io import capture_output
34 from IPython.utils.io import capture_output
35 from IPython.utils.tempdir import TemporaryDirectory
35 from IPython.utils.tempdir import TemporaryDirectory
36 from IPython.core import debugger
36 from IPython.core import debugger
37
37
38 def doctest_refbug():
38 def doctest_refbug():
39 """Very nasty problem with references held by multiple runs of a script.
39 """Very nasty problem with references held by multiple runs of a script.
40 See: https://github.com/ipython/ipython/issues/141
40 See: https://github.com/ipython/ipython/issues/141
41
41
42 In [1]: _ip.clear_main_mod_cache()
42 In [1]: _ip.clear_main_mod_cache()
43 # random
43 # random
44
44
45 In [2]: %run refbug
45 In [2]: %run refbug
46
46
47 In [3]: call_f()
47 In [3]: call_f()
48 lowercased: hello
48 lowercased: hello
49
49
50 In [4]: %run refbug
50 In [4]: %run refbug
51
51
52 In [5]: call_f()
52 In [5]: call_f()
53 lowercased: hello
53 lowercased: hello
54 lowercased: hello
54 lowercased: hello
55 """
55 """
56
56
57
57
58 def doctest_run_builtins():
58 def doctest_run_builtins():
59 r"""Check that %run doesn't damage __builtins__.
59 r"""Check that %run doesn't damage __builtins__.
60
60
61 In [1]: import tempfile
61 In [1]: import tempfile
62
62
63 In [2]: bid1 = id(__builtins__)
63 In [2]: bid1 = id(__builtins__)
64
64
65 In [3]: fname = tempfile.mkstemp('.py')[1]
65 In [3]: fname = tempfile.mkstemp('.py')[1]
66
66
67 In [3]: f = open(fname,'w')
67 In [3]: f = open(fname,'w')
68
68
69 In [4]: dummy= f.write('pass\n')
69 In [4]: dummy= f.write('pass\n')
70
70
71 In [5]: f.flush()
71 In [5]: f.flush()
72
72
73 In [6]: t1 = type(__builtins__)
73 In [6]: t1 = type(__builtins__)
74
74
75 In [7]: %run $fname
75 In [7]: %run $fname
76
76
77 In [7]: f.close()
77 In [7]: f.close()
78
78
79 In [8]: bid2 = id(__builtins__)
79 In [8]: bid2 = id(__builtins__)
80
80
81 In [9]: t2 = type(__builtins__)
81 In [9]: t2 = type(__builtins__)
82
82
83 In [10]: t1 == t2
83 In [10]: t1 == t2
84 Out[10]: True
84 Out[10]: True
85
85
86 In [10]: bid1 == bid2
86 In [10]: bid1 == bid2
87 Out[10]: True
87 Out[10]: True
88
88
89 In [12]: try:
89 In [12]: try:
90 ....: os.unlink(fname)
90 ....: os.unlink(fname)
91 ....: except:
91 ....: except:
92 ....: pass
92 ....: pass
93 ....:
93 ....:
94 """
94 """
95
95
96
96
97 def doctest_run_option_parser():
97 def doctest_run_option_parser():
98 r"""Test option parser in %run.
98 r"""Test option parser in %run.
99
99
100 In [1]: %run print_argv.py
100 In [1]: %run print_argv.py
101 []
101 []
102
102
103 In [2]: %run print_argv.py print*.py
103 In [2]: %run print_argv.py print*.py
104 ['print_argv.py']
104 ['print_argv.py']
105
105
106 In [3]: %run -G print_argv.py print*.py
106 In [3]: %run -G print_argv.py print*.py
107 ['print*.py']
107 ['print*.py']
108
108
109 """
109 """
110
110
111
111
112 @dec.skip_win32
112 @dec.skip_win32
113 def doctest_run_option_parser_for_posix():
113 def doctest_run_option_parser_for_posix():
114 r"""Test option parser in %run (Linux/OSX specific).
114 r"""Test option parser in %run (Linux/OSX specific).
115
115
116 You need double quote to escape glob in POSIX systems:
116 You need double quote to escape glob in POSIX systems:
117
117
118 In [1]: %run print_argv.py print\\*.py
118 In [1]: %run print_argv.py print\\*.py
119 ['print*.py']
119 ['print*.py']
120
120
121 You can't use quote to escape glob in POSIX systems:
121 You can't use quote to escape glob in POSIX systems:
122
122
123 In [2]: %run print_argv.py 'print*.py'
123 In [2]: %run print_argv.py 'print*.py'
124 ['print_argv.py']
124 ['print_argv.py']
125
125
126 """
126 """
127
127
128
128
129 @dec.skip_if_not_win32
129 @dec.skip_if_not_win32
130 def doctest_run_option_parser_for_windows():
130 def doctest_run_option_parser_for_windows():
131 r"""Test option parser in %run (Windows specific).
131 r"""Test option parser in %run (Windows specific).
132
132
133 In Windows, you can't escape ``*` `by backslash:
133 In Windows, you can't escape ``*` `by backslash:
134
134
135 In [1]: %run print_argv.py print\\*.py
135 In [1]: %run print_argv.py print\\*.py
136 ['print\\*.py']
136 ['print\\*.py']
137
137
138 You can use quote to escape glob:
138 You can use quote to escape glob:
139
139
140 In [2]: %run print_argv.py 'print*.py'
140 In [2]: %run print_argv.py 'print*.py'
141 ['print*.py']
141 ['print*.py']
142
142
143 """
143 """
144
144
145
145
146 def doctest_reset_del():
146 def doctest_reset_del():
147 """Test that resetting doesn't cause errors in __del__ methods.
147 """Test that resetting doesn't cause errors in __del__ methods.
148
148
149 In [2]: class A(object):
149 In [2]: class A(object):
150 ...: def __del__(self):
150 ...: def __del__(self):
151 ...: print(str("Hi"))
151 ...: print(str("Hi"))
152 ...:
152 ...:
153
153
154 In [3]: a = A()
154 In [3]: a = A()
155
155
156 In [4]: get_ipython().reset()
156 In [4]: get_ipython().reset()
157 Hi
157 Hi
158
158
159 In [5]: 1+1
159 In [5]: 1+1
160 Out[5]: 2
160 Out[5]: 2
161 """
161 """
162
162
163 # For some tests, it will be handy to organize them in a class with a common
163 # For some tests, it will be handy to organize them in a class with a common
164 # setup that makes a temp file
164 # setup that makes a temp file
165
165
166 class TestMagicRunPass(tt.TempFileMixin):
166 class TestMagicRunPass(tt.TempFileMixin):
167
167
168 def setUp(self):
168 def setUp(self):
169 content = "a = [1,2,3]\nb = 1"
169 content = "a = [1,2,3]\nb = 1"
170 self.mktmp(content)
170 self.mktmp(content)
171
171
172 def run_tmpfile(self):
172 def run_tmpfile(self):
173 _ip = get_ipython()
173 _ip = get_ipython()
174 # This fails on Windows if self.tmpfile.name has spaces or "~" in it.
174 # This fails on Windows if self.tmpfile.name has spaces or "~" in it.
175 # See below and ticket https://bugs.launchpad.net/bugs/366353
175 # See below and ticket https://bugs.launchpad.net/bugs/366353
176 _ip.magic('run %s' % self.fname)
176 _ip.magic('run %s' % self.fname)
177
177
178 def run_tmpfile_p(self):
178 def run_tmpfile_p(self):
179 _ip = get_ipython()
179 _ip = get_ipython()
180 # This fails on Windows if self.tmpfile.name has spaces or "~" in it.
180 # This fails on Windows if self.tmpfile.name has spaces or "~" in it.
181 # See below and ticket https://bugs.launchpad.net/bugs/366353
181 # See below and ticket https://bugs.launchpad.net/bugs/366353
182 _ip.magic('run -p %s' % self.fname)
182 _ip.magic('run -p %s' % self.fname)
183
183
184 def test_builtins_id(self):
184 def test_builtins_id(self):
185 """Check that %run doesn't damage __builtins__ """
185 """Check that %run doesn't damage __builtins__ """
186 _ip = get_ipython()
186 _ip = get_ipython()
187 # Test that the id of __builtins__ is not modified by %run
187 # Test that the id of __builtins__ is not modified by %run
188 bid1 = id(_ip.user_ns['__builtins__'])
188 bid1 = id(_ip.user_ns['__builtins__'])
189 self.run_tmpfile()
189 self.run_tmpfile()
190 bid2 = id(_ip.user_ns['__builtins__'])
190 bid2 = id(_ip.user_ns['__builtins__'])
191 nt.assert_equal(bid1, bid2)
191 nt.assert_equal(bid1, bid2)
192
192
193 def test_builtins_type(self):
193 def test_builtins_type(self):
194 """Check that the type of __builtins__ doesn't change with %run.
194 """Check that the type of __builtins__ doesn't change with %run.
195
195
196 However, the above could pass if __builtins__ was already modified to
196 However, the above could pass if __builtins__ was already modified to
197 be a dict (it should be a module) by a previous use of %run. So we
197 be a dict (it should be a module) by a previous use of %run. So we
198 also check explicitly that it really is a module:
198 also check explicitly that it really is a module:
199 """
199 """
200 _ip = get_ipython()
200 _ip = get_ipython()
201 self.run_tmpfile()
201 self.run_tmpfile()
202 nt.assert_equal(type(_ip.user_ns['__builtins__']),type(sys))
202 nt.assert_equal(type(_ip.user_ns['__builtins__']),type(sys))
203
203
204 def test_run_profile( self ):
204 def test_run_profile( self ):
205 """Test that the option -p, which invokes the profiler, do not
205 """Test that the option -p, which invokes the profiler, do not
206 crash by invoking execfile"""
206 crash by invoking execfile"""
207 self.run_tmpfile_p()
207 self.run_tmpfile_p()
208
208
209 def test_run_debug_twice(self):
209 def test_run_debug_twice(self):
210 # https://github.com/ipython/ipython/issues/10028
210 # https://github.com/ipython/ipython/issues/10028
211 _ip = get_ipython()
211 _ip = get_ipython()
212 with tt.fake_input(['c']):
212 with tt.fake_input(['c']):
213 _ip.magic('run -d %s' % self.fname)
213 _ip.magic('run -d %s' % self.fname)
214 with tt.fake_input(['c']):
214 with tt.fake_input(['c']):
215 _ip.magic('run -d %s' % self.fname)
215 _ip.magic('run -d %s' % self.fname)
216
216
217 def test_run_debug_twice_with_breakpoint(self):
217 def test_run_debug_twice_with_breakpoint(self):
218 """Make a valid python temp file."""
218 """Make a valid python temp file."""
219 _ip = get_ipython()
219 _ip = get_ipython()
220 with tt.fake_input(['b 2', 'c', 'c']):
220 with tt.fake_input(['b 2', 'c', 'c']):
221 _ip.magic('run -d %s' % self.fname)
221 _ip.magic('run -d %s' % self.fname)
222
222
223 with tt.fake_input(['c']):
223 with tt.fake_input(['c']):
224 with tt.AssertNotPrints('KeyError'):
224 with tt.AssertNotPrints('KeyError'):
225 _ip.magic('run -d %s' % self.fname)
225 _ip.magic('run -d %s' % self.fname)
226
226
227
227
228 class TestMagicRunSimple(tt.TempFileMixin):
228 class TestMagicRunSimple(tt.TempFileMixin):
229
229
230 def test_simpledef(self):
230 def test_simpledef(self):
231 """Test that simple class definitions work."""
231 """Test that simple class definitions work."""
232 src = ("class foo: pass\n"
232 src = ("class foo: pass\n"
233 "def f(): return foo()")
233 "def f(): return foo()")
234 self.mktmp(src)
234 self.mktmp(src)
235 _ip.magic('run %s' % self.fname)
235 _ip.magic('run %s' % self.fname)
236 _ip.run_cell('t = isinstance(f(), foo)')
236 _ip.run_cell('t = isinstance(f(), foo)')
237 nt.assert_true(_ip.user_ns['t'])
237 nt.assert_true(_ip.user_ns['t'])
238
238
239 def test_obj_del(self):
239 def test_obj_del(self):
240 """Test that object's __del__ methods are called on exit."""
240 """Test that object's __del__ methods are called on exit."""
241 if sys.platform == 'win32':
241 if sys.platform == 'win32':
242 try:
242 try:
243 import win32api
243 import win32api
244 except ImportError as e:
244 except ImportError as e:
245 raise SkipTest("Test requires pywin32") from e
245 raise SkipTest("Test requires pywin32") from e
246 src = ("class A(object):\n"
246 src = ("class A(object):\n"
247 " def __del__(self):\n"
247 " def __del__(self):\n"
248 " print('object A deleted')\n"
248 " print('object A deleted')\n"
249 "a = A()\n")
249 "a = A()\n")
250 self.mktmp(src)
250 self.mktmp(src)
251 err = None
251 err = None
252 tt.ipexec_validate(self.fname, 'object A deleted', err)
252 tt.ipexec_validate(self.fname, 'object A deleted', err)
253
253
254 def test_aggressive_namespace_cleanup(self):
254 def test_aggressive_namespace_cleanup(self):
255 """Test that namespace cleanup is not too aggressive GH-238
255 """Test that namespace cleanup is not too aggressive GH-238
256
256
257 Returning from another run magic deletes the namespace"""
257 Returning from another run magic deletes the namespace"""
258 # see ticket https://github.com/ipython/ipython/issues/238
258 # see ticket https://github.com/ipython/ipython/issues/238
259
259
260 with tt.TempFileMixin() as empty:
260 with tt.TempFileMixin() as empty:
261 empty.mktmp('')
261 empty.mktmp('')
262 # On Windows, the filename will have \users in it, so we need to use the
262 # On Windows, the filename will have \users in it, so we need to use the
263 # repr so that the \u becomes \\u.
263 # repr so that the \u becomes \\u.
264 src = ("ip = get_ipython()\n"
264 src = ("ip = get_ipython()\n"
265 "for i in range(5):\n"
265 "for i in range(5):\n"
266 " try:\n"
266 " try:\n"
267 " ip.magic(%r)\n"
267 " ip.magic(%r)\n"
268 " except NameError as e:\n"
268 " except NameError as e:\n"
269 " print(i)\n"
269 " print(i)\n"
270 " break\n" % ('run ' + empty.fname))
270 " break\n" % ('run ' + empty.fname))
271 self.mktmp(src)
271 self.mktmp(src)
272 _ip.magic('run %s' % self.fname)
272 _ip.magic('run %s' % self.fname)
273 _ip.run_cell('ip == get_ipython()')
273 _ip.run_cell('ip == get_ipython()')
274 nt.assert_equal(_ip.user_ns['i'], 4)
274 nt.assert_equal(_ip.user_ns['i'], 4)
275
275
276 def test_run_second(self):
276 def test_run_second(self):
277 """Test that running a second file doesn't clobber the first, gh-3547
277 """Test that running a second file doesn't clobber the first, gh-3547
278 """
278 """
279 self.mktmp("avar = 1\n"
279 self.mktmp("avar = 1\n"
280 "def afunc():\n"
280 "def afunc():\n"
281 " return avar\n")
281 " return avar\n")
282
282
283 with tt.TempFileMixin() as empty:
283 with tt.TempFileMixin() as empty:
284 empty.mktmp("")
284 empty.mktmp("")
285
285
286 _ip.magic('run %s' % self.fname)
286 _ip.magic('run %s' % self.fname)
287 _ip.magic('run %s' % empty.fname)
287 _ip.magic('run %s' % empty.fname)
288 nt.assert_equal(_ip.user_ns['afunc'](), 1)
288 nt.assert_equal(_ip.user_ns['afunc'](), 1)
289
289
290 @dec.skip_win32
290 @dec.skip_win32
291 def test_tclass(self):
291 def test_tclass(self):
292 mydir = os.path.dirname(__file__)
292 mydir = os.path.dirname(__file__)
293 tc = os.path.join(mydir, 'tclass')
293 tc = os.path.join(mydir, 'tclass')
294 src = ("%%run '%s' C-first\n"
294 src = ("%%run '%s' C-first\n"
295 "%%run '%s' C-second\n"
295 "%%run '%s' C-second\n"
296 "%%run '%s' C-third\n") % (tc, tc, tc)
296 "%%run '%s' C-third\n") % (tc, tc, tc)
297 self.mktmp(src, '.ipy')
297 self.mktmp(src, '.ipy')
298 out = """\
298 out = """\
299 ARGV 1-: ['C-first']
299 ARGV 1-: ['C-first']
300 ARGV 1-: ['C-second']
300 ARGV 1-: ['C-second']
301 tclass.py: deleting object: C-first
301 tclass.py: deleting object: C-first
302 ARGV 1-: ['C-third']
302 ARGV 1-: ['C-third']
303 tclass.py: deleting object: C-second
303 tclass.py: deleting object: C-second
304 tclass.py: deleting object: C-third
304 tclass.py: deleting object: C-third
305 """
305 """
306 err = None
306 err = None
307 tt.ipexec_validate(self.fname, out, err)
307 tt.ipexec_validate(self.fname, out, err)
308
308
309 def test_run_i_after_reset(self):
309 def test_run_i_after_reset(self):
310 """Check that %run -i still works after %reset (gh-693)"""
310 """Check that %run -i still works after %reset (gh-693)"""
311 src = "yy = zz\n"
311 src = "yy = zz\n"
312 self.mktmp(src)
312 self.mktmp(src)
313 _ip.run_cell("zz = 23")
313 _ip.run_cell("zz = 23")
314 try:
314 try:
315 _ip.magic('run -i %s' % self.fname)
315 _ip.magic('run -i %s' % self.fname)
316 nt.assert_equal(_ip.user_ns['yy'], 23)
316 nt.assert_equal(_ip.user_ns['yy'], 23)
317 finally:
317 finally:
318 _ip.magic('reset -f')
318 _ip.magic('reset -f')
319
319
320 _ip.run_cell("zz = 23")
320 _ip.run_cell("zz = 23")
321 try:
321 try:
322 _ip.magic('run -i %s' % self.fname)
322 _ip.magic('run -i %s' % self.fname)
323 nt.assert_equal(_ip.user_ns['yy'], 23)
323 nt.assert_equal(_ip.user_ns['yy'], 23)
324 finally:
324 finally:
325 _ip.magic('reset -f')
325 _ip.magic('reset -f')
326
326
327 def test_unicode(self):
327 def test_unicode(self):
328 """Check that files in odd encodings are accepted."""
328 """Check that files in odd encodings are accepted."""
329 mydir = os.path.dirname(__file__)
329 mydir = os.path.dirname(__file__)
330 na = os.path.join(mydir, 'nonascii.py')
330 na = os.path.join(mydir, 'nonascii.py')
331 _ip.magic('run "%s"' % na)
331 _ip.magic('run "%s"' % na)
332 nt.assert_equal(_ip.user_ns['u'], u'Ўт№Ф')
332 nt.assert_equal(_ip.user_ns['u'], u'Ўт№Ф')
333
333
334 def test_run_py_file_attribute(self):
334 def test_run_py_file_attribute(self):
335 """Test handling of `__file__` attribute in `%run <file>.py`."""
335 """Test handling of `__file__` attribute in `%run <file>.py`."""
336 src = "t = __file__\n"
336 src = "t = __file__\n"
337 self.mktmp(src)
337 self.mktmp(src)
338 _missing = object()
338 _missing = object()
339 file1 = _ip.user_ns.get('__file__', _missing)
339 file1 = _ip.user_ns.get('__file__', _missing)
340 _ip.magic('run %s' % self.fname)
340 _ip.magic('run %s' % self.fname)
341 file2 = _ip.user_ns.get('__file__', _missing)
341 file2 = _ip.user_ns.get('__file__', _missing)
342
342
343 # Check that __file__ was equal to the filename in the script's
343 # Check that __file__ was equal to the filename in the script's
344 # namespace.
344 # namespace.
345 nt.assert_equal(_ip.user_ns['t'], self.fname)
345 nt.assert_equal(_ip.user_ns['t'], self.fname)
346
346
347 # Check that __file__ was not leaked back into user_ns.
347 # Check that __file__ was not leaked back into user_ns.
348 nt.assert_equal(file1, file2)
348 nt.assert_equal(file1, file2)
349
349
350 def test_run_ipy_file_attribute(self):
350 def test_run_ipy_file_attribute(self):
351 """Test handling of `__file__` attribute in `%run <file.ipy>`."""
351 """Test handling of `__file__` attribute in `%run <file.ipy>`."""
352 src = "t = __file__\n"
352 src = "t = __file__\n"
353 self.mktmp(src, ext='.ipy')
353 self.mktmp(src, ext='.ipy')
354 _missing = object()
354 _missing = object()
355 file1 = _ip.user_ns.get('__file__', _missing)
355 file1 = _ip.user_ns.get('__file__', _missing)
356 _ip.magic('run %s' % self.fname)
356 _ip.magic('run %s' % self.fname)
357 file2 = _ip.user_ns.get('__file__', _missing)
357 file2 = _ip.user_ns.get('__file__', _missing)
358
358
359 # Check that __file__ was equal to the filename in the script's
359 # Check that __file__ was equal to the filename in the script's
360 # namespace.
360 # namespace.
361 nt.assert_equal(_ip.user_ns['t'], self.fname)
361 nt.assert_equal(_ip.user_ns['t'], self.fname)
362
362
363 # Check that __file__ was not leaked back into user_ns.
363 # Check that __file__ was not leaked back into user_ns.
364 nt.assert_equal(file1, file2)
364 nt.assert_equal(file1, file2)
365
365
366 def test_run_formatting(self):
366 def test_run_formatting(self):
367 """ Test that %run -t -N<N> does not raise a TypeError for N > 1."""
367 """ Test that %run -t -N<N> does not raise a TypeError for N > 1."""
368 src = "pass"
368 src = "pass"
369 self.mktmp(src)
369 self.mktmp(src)
370 _ip.magic('run -t -N 1 %s' % self.fname)
370 _ip.magic('run -t -N 1 %s' % self.fname)
371 _ip.magic('run -t -N 10 %s' % self.fname)
371 _ip.magic('run -t -N 10 %s' % self.fname)
372
372
373 def test_ignore_sys_exit(self):
373 def test_ignore_sys_exit(self):
374 """Test the -e option to ignore sys.exit()"""
374 """Test the -e option to ignore sys.exit()"""
375 src = "import sys; sys.exit(1)"
375 src = "import sys; sys.exit(1)"
376 self.mktmp(src)
376 self.mktmp(src)
377 with tt.AssertPrints('SystemExit'):
377 with tt.AssertPrints('SystemExit'):
378 _ip.magic('run %s' % self.fname)
378 _ip.magic('run %s' % self.fname)
379
379
380 with tt.AssertNotPrints('SystemExit'):
380 with tt.AssertNotPrints('SystemExit'):
381 _ip.magic('run -e %s' % self.fname)
381 _ip.magic('run -e %s' % self.fname)
382
382
383 def test_run_nb(self):
383 def test_run_nb(self):
384 """Test %run notebook.ipynb"""
384 """Test %run notebook.ipynb"""
385 from nbformat import v4, writes
385 from nbformat import v4, writes
386 nb = v4.new_notebook(
386 nb = v4.new_notebook(
387 cells=[
387 cells=[
388 v4.new_markdown_cell("The Ultimate Question of Everything"),
388 v4.new_markdown_cell("The Ultimate Question of Everything"),
389 v4.new_code_cell("answer=42")
389 v4.new_code_cell("answer=42")
390 ]
390 ]
391 )
391 )
392 src = writes(nb, version=4)
392 src = writes(nb, version=4)
393 self.mktmp(src, ext='.ipynb')
393 self.mktmp(src, ext='.ipynb')
394
394
395 _ip.magic("run %s" % self.fname)
395 _ip.magic("run %s" % self.fname)
396
396
397 nt.assert_equal(_ip.user_ns['answer'], 42)
397 nt.assert_equal(_ip.user_ns['answer'], 42)
398
398
399 def test_run_nb_error(self):
399 def test_run_nb_error(self):
400 """Test %run notebook.ipynb error"""
400 """Test %run notebook.ipynb error"""
401 from nbformat import v4, writes
401 from nbformat import v4, writes
402 # %run when a file name isn't provided
402 # %run when a file name isn't provided
403 nt.assert_raises(Exception, _ip.magic, "run")
403 nt.assert_raises(Exception, _ip.magic, "run")
404
404
405 # %run when a file doesn't exist
405 # %run when a file doesn't exist
406 nt.assert_raises(Exception, _ip.magic, "run foobar.ipynb")
406 nt.assert_raises(Exception, _ip.magic, "run foobar.ipynb")
407
407
408 # %run on a notebook with an error
408 # %run on a notebook with an error
409 nb = v4.new_notebook(
409 nb = v4.new_notebook(
410 cells=[
410 cells=[
411 v4.new_code_cell("0/0")
411 v4.new_code_cell("0/0")
412 ]
412 ]
413 )
413 )
414 src = writes(nb, version=4)
414 src = writes(nb, version=4)
415 self.mktmp(src, ext='.ipynb')
415 self.mktmp(src, ext='.ipynb')
416 nt.assert_raises(Exception, _ip.magic, "run %s" % self.fname)
416 nt.assert_raises(Exception, _ip.magic, "run %s" % self.fname)
417
417
418 def test_file_options(self):
418 def test_file_options(self):
419 src = ('import sys\n'
419 src = ('import sys\n'
420 'a = " ".join(sys.argv[1:])\n')
420 'a = " ".join(sys.argv[1:])\n')
421 self.mktmp(src)
421 self.mktmp(src)
422 test_opts = '-x 3 --verbose'
422 test_opts = '-x 3 --verbose'
423 _ip.run_line_magic("run", '{0} {1}'.format(self.fname, test_opts))
423 _ip.run_line_magic("run", '{0} {1}'.format(self.fname, test_opts))
424 nt.assert_equal(_ip.user_ns['a'], test_opts)
424 nt.assert_equal(_ip.user_ns['a'], test_opts)
425
425
426
426
427 class TestMagicRunWithPackage(unittest.TestCase):
427 class TestMagicRunWithPackage(unittest.TestCase):
428
428
429 def writefile(self, name, content):
429 def writefile(self, name, content):
430 path = os.path.join(self.tempdir.name, name)
430 path = os.path.join(self.tempdir.name, name)
431 d = os.path.dirname(path)
431 d = os.path.dirname(path)
432 if not os.path.isdir(d):
432 if not os.path.isdir(d):
433 os.makedirs(d)
433 os.makedirs(d)
434 with open(path, 'w') as f:
434 with open(path, 'w') as f:
435 f.write(textwrap.dedent(content))
435 f.write(textwrap.dedent(content))
436
436
437 def setUp(self):
437 def setUp(self):
438 self.package = package = 'tmp{0}'.format(''.join([random.choice(string.ascii_letters) for i in range(10)]))
438 self.package = package = 'tmp{0}'.format(''.join([random.choice(string.ascii_letters) for i in range(10)]))
439 """Temporary (probably) valid python package name."""
439 """Temporary (probably) valid python package name."""
440
440
441 self.value = int(random.random() * 10000)
441 self.value = int(random.random() * 10000)
442
442
443 self.tempdir = TemporaryDirectory()
443 self.tempdir = TemporaryDirectory()
444 self.__orig_cwd = os.getcwd()
444 self.__orig_cwd = os.getcwd()
445 sys.path.insert(0, self.tempdir.name)
445 sys.path.insert(0, self.tempdir.name)
446
446
447 self.writefile(os.path.join(package, '__init__.py'), '')
447 self.writefile(os.path.join(package, '__init__.py'), '')
448 self.writefile(os.path.join(package, 'sub.py'), """
448 self.writefile(os.path.join(package, 'sub.py'), """
449 x = {0!r}
449 x = {0!r}
450 """.format(self.value))
450 """.format(self.value))
451 self.writefile(os.path.join(package, 'relative.py'), """
451 self.writefile(os.path.join(package, 'relative.py'), """
452 from .sub import x
452 from .sub import x
453 """)
453 """)
454 self.writefile(os.path.join(package, 'absolute.py'), """
454 self.writefile(os.path.join(package, 'absolute.py'), """
455 from {0}.sub import x
455 from {0}.sub import x
456 """.format(package))
456 """.format(package))
457 self.writefile(os.path.join(package, 'args.py'), """
457 self.writefile(os.path.join(package, 'args.py'), """
458 import sys
458 import sys
459 a = " ".join(sys.argv[1:])
459 a = " ".join(sys.argv[1:])
460 """.format(package))
460 """.format(package))
461
461
462 def tearDown(self):
462 def tearDown(self):
463 os.chdir(self.__orig_cwd)
463 os.chdir(self.__orig_cwd)
464 sys.path[:] = [p for p in sys.path if p != self.tempdir.name]
464 sys.path[:] = [p for p in sys.path if p != self.tempdir.name]
465 self.tempdir.cleanup()
465 self.tempdir.cleanup()
466
466
467 def check_run_submodule(self, submodule, opts=''):
467 def check_run_submodule(self, submodule, opts=''):
468 _ip.user_ns.pop('x', None)
468 _ip.user_ns.pop('x', None)
469 _ip.magic('run {2} -m {0}.{1}'.format(self.package, submodule, opts))
469 _ip.magic('run {2} -m {0}.{1}'.format(self.package, submodule, opts))
470 self.assertEqual(_ip.user_ns['x'], self.value,
470 self.assertEqual(_ip.user_ns['x'], self.value,
471 'Variable `x` is not loaded from module `{0}`.'
471 'Variable `x` is not loaded from module `{0}`.'
472 .format(submodule))
472 .format(submodule))
473
473
474 def test_run_submodule_with_absolute_import(self):
474 def test_run_submodule_with_absolute_import(self):
475 self.check_run_submodule('absolute')
475 self.check_run_submodule('absolute')
476
476
477 def test_run_submodule_with_relative_import(self):
477 def test_run_submodule_with_relative_import(self):
478 """Run submodule that has a relative import statement (#2727)."""
478 """Run submodule that has a relative import statement (#2727)."""
479 self.check_run_submodule('relative')
479 self.check_run_submodule('relative')
480
480
481 def test_prun_submodule_with_absolute_import(self):
481 def test_prun_submodule_with_absolute_import(self):
482 self.check_run_submodule('absolute', '-p')
482 self.check_run_submodule('absolute', '-p')
483
483
484 def test_prun_submodule_with_relative_import(self):
484 def test_prun_submodule_with_relative_import(self):
485 self.check_run_submodule('relative', '-p')
485 self.check_run_submodule('relative', '-p')
486
486
487 def with_fake_debugger(func):
487 def with_fake_debugger(func):
488 @functools.wraps(func)
488 @functools.wraps(func)
489 def wrapper(*args, **kwds):
489 def wrapper(*args, **kwds):
490 with patch.object(debugger.Pdb, 'run', staticmethod(eval)):
490 with patch.object(debugger.Pdb, 'run', staticmethod(eval)):
491 return func(*args, **kwds)
491 return func(*args, **kwds)
492 return wrapper
492 return wrapper
493
493
494 @with_fake_debugger
494 @with_fake_debugger
495 def test_debug_run_submodule_with_absolute_import(self):
495 def test_debug_run_submodule_with_absolute_import(self):
496 self.check_run_submodule('absolute', '-d')
496 self.check_run_submodule('absolute', '-d')
497
497
498 @with_fake_debugger
498 @with_fake_debugger
499 def test_debug_run_submodule_with_relative_import(self):
499 def test_debug_run_submodule_with_relative_import(self):
500 self.check_run_submodule('relative', '-d')
500 self.check_run_submodule('relative', '-d')
501
501
502 def test_module_options(self):
502 def test_module_options(self):
503 _ip.user_ns.pop('a', None)
503 _ip.user_ns.pop('a', None)
504 test_opts = '-x abc -m test'
504 test_opts = '-x abc -m test'
505 _ip.run_line_magic('run', '-m {0}.args {1}'.format(self.package, test_opts))
505 _ip.run_line_magic('run', '-m {0}.args {1}'.format(self.package, test_opts))
506 nt.assert_equal(_ip.user_ns['a'], test_opts)
506 nt.assert_equal(_ip.user_ns['a'], test_opts)
507
507
508 def test_module_options_with_separator(self):
508 def test_module_options_with_separator(self):
509 _ip.user_ns.pop('a', None)
509 _ip.user_ns.pop('a', None)
510 test_opts = '-x abc -m test'
510 test_opts = '-x abc -m test'
511 _ip.run_line_magic('run', '-m {0}.args -- {1}'.format(self.package, test_opts))
511 _ip.run_line_magic('run', '-m {0}.args -- {1}'.format(self.package, test_opts))
512 nt.assert_equal(_ip.user_ns['a'], test_opts)
512 nt.assert_equal(_ip.user_ns['a'], test_opts)
513
513
514 def test_run__name__():
514 def test_run__name__():
515 with TemporaryDirectory() as td:
515 with TemporaryDirectory() as td:
516 path = pjoin(td, 'foo.py')
516 path = pjoin(td, 'foo.py')
517 with open(path, 'w') as f:
517 with open(path, 'w') as f:
518 f.write("q = __name__")
518 f.write("q = __name__")
519
519
520 _ip.user_ns.pop('q', None)
520 _ip.user_ns.pop('q', None)
521 _ip.magic('run {}'.format(path))
521 _ip.magic('run {}'.format(path))
522 nt.assert_equal(_ip.user_ns.pop('q'), '__main__')
522 nt.assert_equal(_ip.user_ns.pop('q'), '__main__')
523
523
524 _ip.magic('run -n {}'.format(path))
524 _ip.magic('run -n {}'.format(path))
525 nt.assert_equal(_ip.user_ns.pop('q'), 'foo')
525 nt.assert_equal(_ip.user_ns.pop('q'), 'foo')
526
526
527 try:
527 try:
528 _ip.magic('run -i -n {}'.format(path))
528 _ip.magic('run -i -n {}'.format(path))
529 nt.assert_equal(_ip.user_ns.pop('q'), 'foo')
529 nt.assert_equal(_ip.user_ns.pop('q'), 'foo')
530 finally:
530 finally:
531 _ip.magic('reset -f')
531 _ip.magic('reset -f')
532
532
533
533
534 def test_run_tb():
534 def test_run_tb():
535 """Test traceback offset in %run"""
535 """Test traceback offset in %run"""
536 with TemporaryDirectory() as td:
536 with TemporaryDirectory() as td:
537 path = pjoin(td, 'foo.py')
537 path = pjoin(td, 'foo.py')
538 with open(path, 'w') as f:
538 with open(path, 'w') as f:
539 f.write('\n'.join([
539 f.write('\n'.join([
540 "def foo():",
540 "def foo():",
541 " return bar()",
541 " return bar()",
542 "def bar():",
542 "def bar():",
543 " raise RuntimeError('hello!')",
543 " raise RuntimeError('hello!')",
544 "foo()",
544 "foo()",
545 ]))
545 ]))
546 with capture_output() as io:
546 with capture_output() as io:
547 _ip.magic('run {}'.format(path))
547 _ip.magic('run {}'.format(path))
548 out = io.stdout
548 out = io.stdout
549 nt.assert_not_in("execfile", out)
549 nt.assert_not_in("execfile", out)
550 nt.assert_in("RuntimeError", out)
550 nt.assert_in("RuntimeError", out)
551 nt.assert_equal(out.count("---->"), 3)
551 nt.assert_equal(out.count("---->"), 3)
552 del ip.user_ns['bar']
552 del ip.user_ns['bar']
553 del ip.user_ns['foo']
553 del ip.user_ns['foo']
554
554
555
555
556 def test_multiprocessing_run():
556 def test_multiprocessing_run():
557 """Set we can run mutiprocesgin without messing up up main namespace
557 """Set we can run mutiprocesgin without messing up up main namespace
558
558
559 Note that import `nose.tools as nt` mdify the value s
559 Note that import `nose.tools as nt` mdify the value s
560 sys.module['__mp_main__'] so wee need to temporarily set it to None to test
560 sys.module['__mp_main__'] so we need to temporarily set it to None to test
561 the issue.
561 the issue.
562 """
562 """
563 with TemporaryDirectory() as td:
563 with TemporaryDirectory() as td:
564 mpm = sys.modules.get('__mp_main__')
564 mpm = sys.modules.get('__mp_main__')
565 assert mpm is not None
565 assert mpm is not None
566 sys.modules['__mp_main__'] = None
566 sys.modules['__mp_main__'] = None
567 try:
567 try:
568 path = pjoin(td, 'test.py')
568 path = pjoin(td, 'test.py')
569 with open(path, 'w') as f:
569 with open(path, 'w') as f:
570 f.write("import multiprocessing\nprint('hoy')")
570 f.write("import multiprocessing\nprint('hoy')")
571 with capture_output() as io:
571 with capture_output() as io:
572 _ip.run_line_magic('run', path)
572 _ip.run_line_magic('run', path)
573 _ip.run_cell("i_m_undefined")
573 _ip.run_cell("i_m_undefined")
574 out = io.stdout
574 out = io.stdout
575 nt.assert_in("hoy", out)
575 nt.assert_in("hoy", out)
576 nt.assert_not_in("AttributeError", out)
576 nt.assert_not_in("AttributeError", out)
577 nt.assert_in("NameError", out)
577 nt.assert_in("NameError", out)
578 nt.assert_equal(out.count("---->"), 1)
578 nt.assert_equal(out.count("---->"), 1)
579 except:
579 except:
580 raise
580 raise
581 finally:
581 finally:
582 sys.modules['__mp_main__'] = mpm
582 sys.modules['__mp_main__'] = mpm
583
583
584 @dec.knownfailureif(sys.platform == 'win32', "writes to io.stdout aren't captured on Windows")
584 @dec.knownfailureif(sys.platform == 'win32', "writes to io.stdout aren't captured on Windows")
585 def test_script_tb():
585 def test_script_tb():
586 """Test traceback offset in `ipython script.py`"""
586 """Test traceback offset in `ipython script.py`"""
587 with TemporaryDirectory() as td:
587 with TemporaryDirectory() as td:
588 path = pjoin(td, 'foo.py')
588 path = pjoin(td, 'foo.py')
589 with open(path, 'w') as f:
589 with open(path, 'w') as f:
590 f.write('\n'.join([
590 f.write('\n'.join([
591 "def foo():",
591 "def foo():",
592 " return bar()",
592 " return bar()",
593 "def bar():",
593 "def bar():",
594 " raise RuntimeError('hello!')",
594 " raise RuntimeError('hello!')",
595 "foo()",
595 "foo()",
596 ]))
596 ]))
597 out, err = tt.ipexec(path)
597 out, err = tt.ipexec(path)
598 nt.assert_not_in("execfile", out)
598 nt.assert_not_in("execfile", out)
599 nt.assert_in("RuntimeError", out)
599 nt.assert_in("RuntimeError", out)
600 nt.assert_equal(out.count("---->"), 3)
600 nt.assert_equal(out.count("---->"), 3)
601
601
@@ -1,555 +1,555 b''
1 """Tests for autoreload extension.
1 """Tests for autoreload extension.
2 """
2 """
3 # -----------------------------------------------------------------------------
3 # -----------------------------------------------------------------------------
4 # Copyright (c) 2012 IPython Development Team.
4 # Copyright (c) 2012 IPython Development Team.
5 #
5 #
6 # Distributed under the terms of the Modified BSD License.
6 # Distributed under the terms of the Modified BSD License.
7 #
7 #
8 # The full license is in the file COPYING.txt, distributed with this software.
8 # The full license is in the file COPYING.txt, distributed with this software.
9 # -----------------------------------------------------------------------------
9 # -----------------------------------------------------------------------------
10
10
11 # -----------------------------------------------------------------------------
11 # -----------------------------------------------------------------------------
12 # Imports
12 # Imports
13 # -----------------------------------------------------------------------------
13 # -----------------------------------------------------------------------------
14
14
15 import os
15 import os
16 import sys
16 import sys
17 import tempfile
17 import tempfile
18 import textwrap
18 import textwrap
19 import shutil
19 import shutil
20 import random
20 import random
21 import time
21 import time
22 from io import StringIO
22 from io import StringIO
23
23
24 import nose.tools as nt
24 import nose.tools as nt
25 import IPython.testing.tools as tt
25 import IPython.testing.tools as tt
26
26
27 from unittest import TestCase
27 from unittest import TestCase
28
28
29 from IPython.extensions.autoreload import AutoreloadMagics
29 from IPython.extensions.autoreload import AutoreloadMagics
30 from IPython.core.events import EventManager, pre_run_cell
30 from IPython.core.events import EventManager, pre_run_cell
31
31
32 # -----------------------------------------------------------------------------
32 # -----------------------------------------------------------------------------
33 # Test fixture
33 # Test fixture
34 # -----------------------------------------------------------------------------
34 # -----------------------------------------------------------------------------
35
35
36 noop = lambda *a, **kw: None
36 noop = lambda *a, **kw: None
37
37
38
38
39 class FakeShell:
39 class FakeShell:
40 def __init__(self):
40 def __init__(self):
41 self.ns = {}
41 self.ns = {}
42 self.user_ns = self.ns
42 self.user_ns = self.ns
43 self.user_ns_hidden = {}
43 self.user_ns_hidden = {}
44 self.events = EventManager(self, {"pre_run_cell", pre_run_cell})
44 self.events = EventManager(self, {"pre_run_cell", pre_run_cell})
45 self.auto_magics = AutoreloadMagics(shell=self)
45 self.auto_magics = AutoreloadMagics(shell=self)
46 self.events.register("pre_run_cell", self.auto_magics.pre_run_cell)
46 self.events.register("pre_run_cell", self.auto_magics.pre_run_cell)
47
47
48 register_magics = set_hook = noop
48 register_magics = set_hook = noop
49
49
50 def run_code(self, code):
50 def run_code(self, code):
51 self.events.trigger("pre_run_cell")
51 self.events.trigger("pre_run_cell")
52 exec(code, self.user_ns)
52 exec(code, self.user_ns)
53 self.auto_magics.post_execute_hook()
53 self.auto_magics.post_execute_hook()
54
54
55 def push(self, items):
55 def push(self, items):
56 self.ns.update(items)
56 self.ns.update(items)
57
57
58 def magic_autoreload(self, parameter):
58 def magic_autoreload(self, parameter):
59 self.auto_magics.autoreload(parameter)
59 self.auto_magics.autoreload(parameter)
60
60
61 def magic_aimport(self, parameter, stream=None):
61 def magic_aimport(self, parameter, stream=None):
62 self.auto_magics.aimport(parameter, stream=stream)
62 self.auto_magics.aimport(parameter, stream=stream)
63 self.auto_magics.post_execute_hook()
63 self.auto_magics.post_execute_hook()
64
64
65
65
66 class Fixture(TestCase):
66 class Fixture(TestCase):
67 """Fixture for creating test module files"""
67 """Fixture for creating test module files"""
68
68
69 test_dir = None
69 test_dir = None
70 old_sys_path = None
70 old_sys_path = None
71 filename_chars = "abcdefghijklmopqrstuvwxyz0123456789"
71 filename_chars = "abcdefghijklmopqrstuvwxyz0123456789"
72
72
73 def setUp(self):
73 def setUp(self):
74 self.test_dir = tempfile.mkdtemp()
74 self.test_dir = tempfile.mkdtemp()
75 self.old_sys_path = list(sys.path)
75 self.old_sys_path = list(sys.path)
76 sys.path.insert(0, self.test_dir)
76 sys.path.insert(0, self.test_dir)
77 self.shell = FakeShell()
77 self.shell = FakeShell()
78
78
79 def tearDown(self):
79 def tearDown(self):
80 shutil.rmtree(self.test_dir)
80 shutil.rmtree(self.test_dir)
81 sys.path = self.old_sys_path
81 sys.path = self.old_sys_path
82
82
83 self.test_dir = None
83 self.test_dir = None
84 self.old_sys_path = None
84 self.old_sys_path = None
85 self.shell = None
85 self.shell = None
86
86
87 def get_module(self):
87 def get_module(self):
88 module_name = "tmpmod_" + "".join(random.sample(self.filename_chars, 20))
88 module_name = "tmpmod_" + "".join(random.sample(self.filename_chars, 20))
89 if module_name in sys.modules:
89 if module_name in sys.modules:
90 del sys.modules[module_name]
90 del sys.modules[module_name]
91 file_name = os.path.join(self.test_dir, module_name + ".py")
91 file_name = os.path.join(self.test_dir, module_name + ".py")
92 return module_name, file_name
92 return module_name, file_name
93
93
94 def write_file(self, filename, content):
94 def write_file(self, filename, content):
95 """
95 """
96 Write a file, and force a timestamp difference of at least one second
96 Write a file, and force a timestamp difference of at least one second
97
97
98 Notes
98 Notes
99 -----
99 -----
100 Python's .pyc files record the timestamp of their compilation
100 Python's .pyc files record the timestamp of their compilation
101 with a time resolution of one second.
101 with a time resolution of one second.
102
102
103 Therefore, we need to force a timestamp difference between .py
103 Therefore, we need to force a timestamp difference between .py
104 and .pyc, without having the .py file be timestamped in the
104 and .pyc, without having the .py file be timestamped in the
105 future, and without changing the timestamp of the .pyc file
105 future, and without changing the timestamp of the .pyc file
106 (because that is stored in the file). The only reliable way
106 (because that is stored in the file). The only reliable way
107 to achieve this seems to be to sleep.
107 to achieve this seems to be to sleep.
108 """
108 """
109 content = textwrap.dedent(content)
109 content = textwrap.dedent(content)
110 # Sleep one second + eps
110 # Sleep one second + eps
111 time.sleep(1.05)
111 time.sleep(1.05)
112
112
113 # Write
113 # Write
114 with open(filename, "w") as f:
114 with open(filename, "w") as f:
115 f.write(content)
115 f.write(content)
116
116
117 def new_module(self, code):
117 def new_module(self, code):
118 code = textwrap.dedent(code)
118 code = textwrap.dedent(code)
119 mod_name, mod_fn = self.get_module()
119 mod_name, mod_fn = self.get_module()
120 with open(mod_fn, "w") as f:
120 with open(mod_fn, "w") as f:
121 f.write(code)
121 f.write(code)
122 return mod_name, mod_fn
122 return mod_name, mod_fn
123
123
124
124
125 # -----------------------------------------------------------------------------
125 # -----------------------------------------------------------------------------
126 # Test automatic reloading
126 # Test automatic reloading
127 # -----------------------------------------------------------------------------
127 # -----------------------------------------------------------------------------
128
128
129
129
130 def pickle_get_current_class(obj):
130 def pickle_get_current_class(obj):
131 """
131 """
132 Original issue comes from pickle; hence the name.
132 Original issue comes from pickle; hence the name.
133 """
133 """
134 name = obj.__class__.__name__
134 name = obj.__class__.__name__
135 module_name = getattr(obj, "__module__", None)
135 module_name = getattr(obj, "__module__", None)
136 obj2 = sys.modules[module_name]
136 obj2 = sys.modules[module_name]
137 for subpath in name.split("."):
137 for subpath in name.split("."):
138 obj2 = getattr(obj2, subpath)
138 obj2 = getattr(obj2, subpath)
139 return obj2
139 return obj2
140
140
141
141
142 class TestAutoreload(Fixture):
142 class TestAutoreload(Fixture):
143 def test_reload_enums(self):
143 def test_reload_enums(self):
144 mod_name, mod_fn = self.new_module(
144 mod_name, mod_fn = self.new_module(
145 textwrap.dedent(
145 textwrap.dedent(
146 """
146 """
147 from enum import Enum
147 from enum import Enum
148 class MyEnum(Enum):
148 class MyEnum(Enum):
149 A = 'A'
149 A = 'A'
150 B = 'B'
150 B = 'B'
151 """
151 """
152 )
152 )
153 )
153 )
154 self.shell.magic_autoreload("2")
154 self.shell.magic_autoreload("2")
155 self.shell.magic_aimport(mod_name)
155 self.shell.magic_aimport(mod_name)
156 self.write_file(
156 self.write_file(
157 mod_fn,
157 mod_fn,
158 textwrap.dedent(
158 textwrap.dedent(
159 """
159 """
160 from enum import Enum
160 from enum import Enum
161 class MyEnum(Enum):
161 class MyEnum(Enum):
162 A = 'A'
162 A = 'A'
163 B = 'B'
163 B = 'B'
164 C = 'C'
164 C = 'C'
165 """
165 """
166 ),
166 ),
167 )
167 )
168 with tt.AssertNotPrints(
168 with tt.AssertNotPrints(
169 ("[autoreload of %s failed:" % mod_name), channel="stderr"
169 ("[autoreload of %s failed:" % mod_name), channel="stderr"
170 ):
170 ):
171 self.shell.run_code("pass") # trigger another reload
171 self.shell.run_code("pass") # trigger another reload
172
172
173 def test_reload_class_type(self):
173 def test_reload_class_type(self):
174 self.shell.magic_autoreload("2")
174 self.shell.magic_autoreload("2")
175 mod_name, mod_fn = self.new_module(
175 mod_name, mod_fn = self.new_module(
176 """
176 """
177 class Test():
177 class Test():
178 def meth(self):
178 def meth(self):
179 return "old"
179 return "old"
180 """
180 """
181 )
181 )
182 assert "test" not in self.shell.ns
182 assert "test" not in self.shell.ns
183 assert "result" not in self.shell.ns
183 assert "result" not in self.shell.ns
184
184
185 self.shell.run_code("from %s import Test" % mod_name)
185 self.shell.run_code("from %s import Test" % mod_name)
186 self.shell.run_code("test = Test()")
186 self.shell.run_code("test = Test()")
187
187
188 self.write_file(
188 self.write_file(
189 mod_fn,
189 mod_fn,
190 """
190 """
191 class Test():
191 class Test():
192 def meth(self):
192 def meth(self):
193 return "new"
193 return "new"
194 """,
194 """,
195 )
195 )
196
196
197 test_object = self.shell.ns["test"]
197 test_object = self.shell.ns["test"]
198
198
199 # important to trigger autoreload logic !
199 # important to trigger autoreload logic !
200 self.shell.run_code("pass")
200 self.shell.run_code("pass")
201
201
202 test_class = pickle_get_current_class(test_object)
202 test_class = pickle_get_current_class(test_object)
203 assert isinstance(test_object, test_class)
203 assert isinstance(test_object, test_class)
204
204
205 # extra check.
205 # extra check.
206 self.shell.run_code("import pickle")
206 self.shell.run_code("import pickle")
207 self.shell.run_code("p = pickle.dumps(test)")
207 self.shell.run_code("p = pickle.dumps(test)")
208
208
209 def test_reload_class_attributes(self):
209 def test_reload_class_attributes(self):
210 self.shell.magic_autoreload("2")
210 self.shell.magic_autoreload("2")
211 mod_name, mod_fn = self.new_module(
211 mod_name, mod_fn = self.new_module(
212 textwrap.dedent(
212 textwrap.dedent(
213 """
213 """
214 class MyClass:
214 class MyClass:
215
215
216 def __init__(self, a=10):
216 def __init__(self, a=10):
217 self.a = a
217 self.a = a
218 self.b = 22
218 self.b = 22
219 # self.toto = 33
219 # self.toto = 33
220
220
221 def square(self):
221 def square(self):
222 print('compute square')
222 print('compute square')
223 return self.a*self.a
223 return self.a*self.a
224 """
224 """
225 )
225 )
226 )
226 )
227 self.shell.run_code("from %s import MyClass" % mod_name)
227 self.shell.run_code("from %s import MyClass" % mod_name)
228 self.shell.run_code("first = MyClass(5)")
228 self.shell.run_code("first = MyClass(5)")
229 self.shell.run_code("first.square()")
229 self.shell.run_code("first.square()")
230 with nt.assert_raises(AttributeError):
230 with nt.assert_raises(AttributeError):
231 self.shell.run_code("first.cube()")
231 self.shell.run_code("first.cube()")
232 with nt.assert_raises(AttributeError):
232 with nt.assert_raises(AttributeError):
233 self.shell.run_code("first.power(5)")
233 self.shell.run_code("first.power(5)")
234 self.shell.run_code("first.b")
234 self.shell.run_code("first.b")
235 with nt.assert_raises(AttributeError):
235 with nt.assert_raises(AttributeError):
236 self.shell.run_code("first.toto")
236 self.shell.run_code("first.toto")
237
237
238 # remove square, add power
238 # remove square, add power
239
239
240 self.write_file(
240 self.write_file(
241 mod_fn,
241 mod_fn,
242 textwrap.dedent(
242 textwrap.dedent(
243 """
243 """
244 class MyClass:
244 class MyClass:
245
245
246 def __init__(self, a=10):
246 def __init__(self, a=10):
247 self.a = a
247 self.a = a
248 self.b = 11
248 self.b = 11
249
249
250 def power(self, p):
250 def power(self, p):
251 print('compute power '+str(p))
251 print('compute power '+str(p))
252 return self.a**p
252 return self.a**p
253 """
253 """
254 ),
254 ),
255 )
255 )
256
256
257 self.shell.run_code("second = MyClass(5)")
257 self.shell.run_code("second = MyClass(5)")
258
258
259 for object_name in {"first", "second"}:
259 for object_name in {"first", "second"}:
260 self.shell.run_code(f"{object_name}.power(5)")
260 self.shell.run_code(f"{object_name}.power(5)")
261 with nt.assert_raises(AttributeError):
261 with nt.assert_raises(AttributeError):
262 self.shell.run_code(f"{object_name}.cube()")
262 self.shell.run_code(f"{object_name}.cube()")
263 with nt.assert_raises(AttributeError):
263 with nt.assert_raises(AttributeError):
264 self.shell.run_code(f"{object_name}.square()")
264 self.shell.run_code(f"{object_name}.square()")
265 self.shell.run_code(f"{object_name}.b")
265 self.shell.run_code(f"{object_name}.b")
266 self.shell.run_code(f"{object_name}.a")
266 self.shell.run_code(f"{object_name}.a")
267 with nt.assert_raises(AttributeError):
267 with nt.assert_raises(AttributeError):
268 self.shell.run_code(f"{object_name}.toto")
268 self.shell.run_code(f"{object_name}.toto")
269
269
270 def test_autoload_newly_added_objects(self):
270 def test_autoload_newly_added_objects(self):
271 self.shell.magic_autoreload("3")
271 self.shell.magic_autoreload("3")
272 mod_code = """
272 mod_code = """
273 def func1(): pass
273 def func1(): pass
274 """
274 """
275 mod_name, mod_fn = self.new_module(textwrap.dedent(mod_code))
275 mod_name, mod_fn = self.new_module(textwrap.dedent(mod_code))
276 self.shell.run_code(f"from {mod_name} import *")
276 self.shell.run_code(f"from {mod_name} import *")
277 self.shell.run_code("func1()")
277 self.shell.run_code("func1()")
278 with nt.assert_raises(NameError):
278 with nt.assert_raises(NameError):
279 self.shell.run_code("func2()")
279 self.shell.run_code("func2()")
280 with nt.assert_raises(NameError):
280 with nt.assert_raises(NameError):
281 self.shell.run_code("t = Test()")
281 self.shell.run_code("t = Test()")
282 with nt.assert_raises(NameError):
282 with nt.assert_raises(NameError):
283 self.shell.run_code("number")
283 self.shell.run_code("number")
284
284
285 # ----------- TEST NEW OBJ LOADED --------------------------
285 # ----------- TEST NEW OBJ LOADED --------------------------
286
286
287 new_code = """
287 new_code = """
288 def func1(): pass
288 def func1(): pass
289 def func2(): pass
289 def func2(): pass
290 class Test: pass
290 class Test: pass
291 number = 0
291 number = 0
292 from enum import Enum
292 from enum import Enum
293 class TestEnum(Enum):
293 class TestEnum(Enum):
294 A = 'a'
294 A = 'a'
295 """
295 """
296 self.write_file(mod_fn, textwrap.dedent(new_code))
296 self.write_file(mod_fn, textwrap.dedent(new_code))
297
297
298 # test function now exists in shell's namespace namespace
298 # test function now exists in shell's namespace namespace
299 self.shell.run_code("func2()")
299 self.shell.run_code("func2()")
300 # test function now exists in module's dict
300 # test function now exists in module's dict
301 self.shell.run_code(f"import sys; sys.modules['{mod_name}'].func2()")
301 self.shell.run_code(f"import sys; sys.modules['{mod_name}'].func2()")
302 # test class now exists
302 # test class now exists
303 self.shell.run_code("t = Test()")
303 self.shell.run_code("t = Test()")
304 # test global built-in var now exists
304 # test global built-in var now exists
305 self.shell.run_code("number")
305 self.shell.run_code("number")
306 # test the enumerations gets loaded succesfully
306 # test the enumerations gets loaded successfully
307 self.shell.run_code("TestEnum.A")
307 self.shell.run_code("TestEnum.A")
308
308
309 # ----------- TEST NEW OBJ CAN BE CHANGED --------------------
309 # ----------- TEST NEW OBJ CAN BE CHANGED --------------------
310
310
311 new_code = """
311 new_code = """
312 def func1(): return 'changed'
312 def func1(): return 'changed'
313 def func2(): return 'changed'
313 def func2(): return 'changed'
314 class Test:
314 class Test:
315 def new_func(self):
315 def new_func(self):
316 return 'changed'
316 return 'changed'
317 number = 1
317 number = 1
318 from enum import Enum
318 from enum import Enum
319 class TestEnum(Enum):
319 class TestEnum(Enum):
320 A = 'a'
320 A = 'a'
321 B = 'added'
321 B = 'added'
322 """
322 """
323 self.write_file(mod_fn, textwrap.dedent(new_code))
323 self.write_file(mod_fn, textwrap.dedent(new_code))
324 self.shell.run_code("assert func1() == 'changed'")
324 self.shell.run_code("assert func1() == 'changed'")
325 self.shell.run_code("assert func2() == 'changed'")
325 self.shell.run_code("assert func2() == 'changed'")
326 self.shell.run_code("t = Test(); assert t.new_func() == 'changed'")
326 self.shell.run_code("t = Test(); assert t.new_func() == 'changed'")
327 self.shell.run_code("assert number == 1")
327 self.shell.run_code("assert number == 1")
328 self.shell.run_code("assert TestEnum.B.value == 'added'")
328 self.shell.run_code("assert TestEnum.B.value == 'added'")
329
329
330 # ----------- TEST IMPORT FROM MODULE --------------------------
330 # ----------- TEST IMPORT FROM MODULE --------------------------
331
331
332 new_mod_code = """
332 new_mod_code = """
333 from enum import Enum
333 from enum import Enum
334 class Ext(Enum):
334 class Ext(Enum):
335 A = 'ext'
335 A = 'ext'
336 def ext_func():
336 def ext_func():
337 return 'ext'
337 return 'ext'
338 class ExtTest:
338 class ExtTest:
339 def meth(self):
339 def meth(self):
340 return 'ext'
340 return 'ext'
341 ext_int = 2
341 ext_int = 2
342 """
342 """
343 new_mod_name, new_mod_fn = self.new_module(textwrap.dedent(new_mod_code))
343 new_mod_name, new_mod_fn = self.new_module(textwrap.dedent(new_mod_code))
344 current_mod_code = f"""
344 current_mod_code = f"""
345 from {new_mod_name} import *
345 from {new_mod_name} import *
346 """
346 """
347 self.write_file(mod_fn, textwrap.dedent(current_mod_code))
347 self.write_file(mod_fn, textwrap.dedent(current_mod_code))
348 self.shell.run_code("assert Ext.A.value == 'ext'")
348 self.shell.run_code("assert Ext.A.value == 'ext'")
349 self.shell.run_code("assert ext_func() == 'ext'")
349 self.shell.run_code("assert ext_func() == 'ext'")
350 self.shell.run_code("t = ExtTest(); assert t.meth() == 'ext'")
350 self.shell.run_code("t = ExtTest(); assert t.meth() == 'ext'")
351 self.shell.run_code("assert ext_int == 2")
351 self.shell.run_code("assert ext_int == 2")
352
352
353 def _check_smoketest(self, use_aimport=True):
353 def _check_smoketest(self, use_aimport=True):
354 """
354 """
355 Functional test for the automatic reloader using either
355 Functional test for the automatic reloader using either
356 '%autoreload 1' or '%autoreload 2'
356 '%autoreload 1' or '%autoreload 2'
357 """
357 """
358
358
359 mod_name, mod_fn = self.new_module(
359 mod_name, mod_fn = self.new_module(
360 """
360 """
361 x = 9
361 x = 9
362
362
363 z = 123 # this item will be deleted
363 z = 123 # this item will be deleted
364
364
365 def foo(y):
365 def foo(y):
366 return y + 3
366 return y + 3
367
367
368 class Baz(object):
368 class Baz(object):
369 def __init__(self, x):
369 def __init__(self, x):
370 self.x = x
370 self.x = x
371 def bar(self, y):
371 def bar(self, y):
372 return self.x + y
372 return self.x + y
373 @property
373 @property
374 def quux(self):
374 def quux(self):
375 return 42
375 return 42
376 def zzz(self):
376 def zzz(self):
377 '''This method will be deleted below'''
377 '''This method will be deleted below'''
378 return 99
378 return 99
379
379
380 class Bar: # old-style class: weakref doesn't work for it on Python < 2.7
380 class Bar: # old-style class: weakref doesn't work for it on Python < 2.7
381 def foo(self):
381 def foo(self):
382 return 1
382 return 1
383 """
383 """
384 )
384 )
385
385
386 #
386 #
387 # Import module, and mark for reloading
387 # Import module, and mark for reloading
388 #
388 #
389 if use_aimport:
389 if use_aimport:
390 self.shell.magic_autoreload("1")
390 self.shell.magic_autoreload("1")
391 self.shell.magic_aimport(mod_name)
391 self.shell.magic_aimport(mod_name)
392 stream = StringIO()
392 stream = StringIO()
393 self.shell.magic_aimport("", stream=stream)
393 self.shell.magic_aimport("", stream=stream)
394 nt.assert_in(("Modules to reload:\n%s" % mod_name), stream.getvalue())
394 nt.assert_in(("Modules to reload:\n%s" % mod_name), stream.getvalue())
395
395
396 with nt.assert_raises(ImportError):
396 with nt.assert_raises(ImportError):
397 self.shell.magic_aimport("tmpmod_as318989e89ds")
397 self.shell.magic_aimport("tmpmod_as318989e89ds")
398 else:
398 else:
399 self.shell.magic_autoreload("2")
399 self.shell.magic_autoreload("2")
400 self.shell.run_code("import %s" % mod_name)
400 self.shell.run_code("import %s" % mod_name)
401 stream = StringIO()
401 stream = StringIO()
402 self.shell.magic_aimport("", stream=stream)
402 self.shell.magic_aimport("", stream=stream)
403 nt.assert_true(
403 nt.assert_true(
404 "Modules to reload:\nall-except-skipped" in stream.getvalue()
404 "Modules to reload:\nall-except-skipped" in stream.getvalue()
405 )
405 )
406 nt.assert_in(mod_name, self.shell.ns)
406 nt.assert_in(mod_name, self.shell.ns)
407
407
408 mod = sys.modules[mod_name]
408 mod = sys.modules[mod_name]
409
409
410 #
410 #
411 # Test module contents
411 # Test module contents
412 #
412 #
413 old_foo = mod.foo
413 old_foo = mod.foo
414 old_obj = mod.Baz(9)
414 old_obj = mod.Baz(9)
415 old_obj2 = mod.Bar()
415 old_obj2 = mod.Bar()
416
416
417 def check_module_contents():
417 def check_module_contents():
418 nt.assert_equal(mod.x, 9)
418 nt.assert_equal(mod.x, 9)
419 nt.assert_equal(mod.z, 123)
419 nt.assert_equal(mod.z, 123)
420
420
421 nt.assert_equal(old_foo(0), 3)
421 nt.assert_equal(old_foo(0), 3)
422 nt.assert_equal(mod.foo(0), 3)
422 nt.assert_equal(mod.foo(0), 3)
423
423
424 obj = mod.Baz(9)
424 obj = mod.Baz(9)
425 nt.assert_equal(old_obj.bar(1), 10)
425 nt.assert_equal(old_obj.bar(1), 10)
426 nt.assert_equal(obj.bar(1), 10)
426 nt.assert_equal(obj.bar(1), 10)
427 nt.assert_equal(obj.quux, 42)
427 nt.assert_equal(obj.quux, 42)
428 nt.assert_equal(obj.zzz(), 99)
428 nt.assert_equal(obj.zzz(), 99)
429
429
430 obj2 = mod.Bar()
430 obj2 = mod.Bar()
431 nt.assert_equal(old_obj2.foo(), 1)
431 nt.assert_equal(old_obj2.foo(), 1)
432 nt.assert_equal(obj2.foo(), 1)
432 nt.assert_equal(obj2.foo(), 1)
433
433
434 check_module_contents()
434 check_module_contents()
435
435
436 #
436 #
437 # Simulate a failed reload: no reload should occur and exactly
437 # Simulate a failed reload: no reload should occur and exactly
438 # one error message should be printed
438 # one error message should be printed
439 #
439 #
440 self.write_file(
440 self.write_file(
441 mod_fn,
441 mod_fn,
442 """
442 """
443 a syntax error
443 a syntax error
444 """,
444 """,
445 )
445 )
446
446
447 with tt.AssertPrints(
447 with tt.AssertPrints(
448 ("[autoreload of %s failed:" % mod_name), channel="stderr"
448 ("[autoreload of %s failed:" % mod_name), channel="stderr"
449 ):
449 ):
450 self.shell.run_code("pass") # trigger reload
450 self.shell.run_code("pass") # trigger reload
451 with tt.AssertNotPrints(
451 with tt.AssertNotPrints(
452 ("[autoreload of %s failed:" % mod_name), channel="stderr"
452 ("[autoreload of %s failed:" % mod_name), channel="stderr"
453 ):
453 ):
454 self.shell.run_code("pass") # trigger another reload
454 self.shell.run_code("pass") # trigger another reload
455 check_module_contents()
455 check_module_contents()
456
456
457 #
457 #
458 # Rewrite module (this time reload should succeed)
458 # Rewrite module (this time reload should succeed)
459 #
459 #
460 self.write_file(
460 self.write_file(
461 mod_fn,
461 mod_fn,
462 """
462 """
463 x = 10
463 x = 10
464
464
465 def foo(y):
465 def foo(y):
466 return y + 4
466 return y + 4
467
467
468 class Baz(object):
468 class Baz(object):
469 def __init__(self, x):
469 def __init__(self, x):
470 self.x = x
470 self.x = x
471 def bar(self, y):
471 def bar(self, y):
472 return self.x + y + 1
472 return self.x + y + 1
473 @property
473 @property
474 def quux(self):
474 def quux(self):
475 return 43
475 return 43
476
476
477 class Bar: # old-style class
477 class Bar: # old-style class
478 def foo(self):
478 def foo(self):
479 return 2
479 return 2
480 """,
480 """,
481 )
481 )
482
482
483 def check_module_contents():
483 def check_module_contents():
484 nt.assert_equal(mod.x, 10)
484 nt.assert_equal(mod.x, 10)
485 nt.assert_false(hasattr(mod, "z"))
485 nt.assert_false(hasattr(mod, "z"))
486
486
487 nt.assert_equal(old_foo(0), 4) # superreload magic!
487 nt.assert_equal(old_foo(0), 4) # superreload magic!
488 nt.assert_equal(mod.foo(0), 4)
488 nt.assert_equal(mod.foo(0), 4)
489
489
490 obj = mod.Baz(9)
490 obj = mod.Baz(9)
491 nt.assert_equal(old_obj.bar(1), 11) # superreload magic!
491 nt.assert_equal(old_obj.bar(1), 11) # superreload magic!
492 nt.assert_equal(obj.bar(1), 11)
492 nt.assert_equal(obj.bar(1), 11)
493
493
494 nt.assert_equal(old_obj.quux, 43)
494 nt.assert_equal(old_obj.quux, 43)
495 nt.assert_equal(obj.quux, 43)
495 nt.assert_equal(obj.quux, 43)
496
496
497 nt.assert_false(hasattr(old_obj, "zzz"))
497 nt.assert_false(hasattr(old_obj, "zzz"))
498 nt.assert_false(hasattr(obj, "zzz"))
498 nt.assert_false(hasattr(obj, "zzz"))
499
499
500 obj2 = mod.Bar()
500 obj2 = mod.Bar()
501 nt.assert_equal(old_obj2.foo(), 2)
501 nt.assert_equal(old_obj2.foo(), 2)
502 nt.assert_equal(obj2.foo(), 2)
502 nt.assert_equal(obj2.foo(), 2)
503
503
504 self.shell.run_code("pass") # trigger reload
504 self.shell.run_code("pass") # trigger reload
505 check_module_contents()
505 check_module_contents()
506
506
507 #
507 #
508 # Another failure case: deleted file (shouldn't reload)
508 # Another failure case: deleted file (shouldn't reload)
509 #
509 #
510 os.unlink(mod_fn)
510 os.unlink(mod_fn)
511
511
512 self.shell.run_code("pass") # trigger reload
512 self.shell.run_code("pass") # trigger reload
513 check_module_contents()
513 check_module_contents()
514
514
515 #
515 #
516 # Disable autoreload and rewrite module: no reload should occur
516 # Disable autoreload and rewrite module: no reload should occur
517 #
517 #
518 if use_aimport:
518 if use_aimport:
519 self.shell.magic_aimport("-" + mod_name)
519 self.shell.magic_aimport("-" + mod_name)
520 stream = StringIO()
520 stream = StringIO()
521 self.shell.magic_aimport("", stream=stream)
521 self.shell.magic_aimport("", stream=stream)
522 nt.assert_true(("Modules to skip:\n%s" % mod_name) in stream.getvalue())
522 nt.assert_true(("Modules to skip:\n%s" % mod_name) in stream.getvalue())
523
523
524 # This should succeed, although no such module exists
524 # This should succeed, although no such module exists
525 self.shell.magic_aimport("-tmpmod_as318989e89ds")
525 self.shell.magic_aimport("-tmpmod_as318989e89ds")
526 else:
526 else:
527 self.shell.magic_autoreload("0")
527 self.shell.magic_autoreload("0")
528
528
529 self.write_file(
529 self.write_file(
530 mod_fn,
530 mod_fn,
531 """
531 """
532 x = -99
532 x = -99
533 """,
533 """,
534 )
534 )
535
535
536 self.shell.run_code("pass") # trigger reload
536 self.shell.run_code("pass") # trigger reload
537 self.shell.run_code("pass")
537 self.shell.run_code("pass")
538 check_module_contents()
538 check_module_contents()
539
539
540 #
540 #
541 # Re-enable autoreload: reload should now occur
541 # Re-enable autoreload: reload should now occur
542 #
542 #
543 if use_aimport:
543 if use_aimport:
544 self.shell.magic_aimport(mod_name)
544 self.shell.magic_aimport(mod_name)
545 else:
545 else:
546 self.shell.magic_autoreload("")
546 self.shell.magic_autoreload("")
547
547
548 self.shell.run_code("pass") # trigger reload
548 self.shell.run_code("pass") # trigger reload
549 nt.assert_equal(mod.x, -99)
549 nt.assert_equal(mod.x, -99)
550
550
551 def test_smoketest_aimport(self):
551 def test_smoketest_aimport(self):
552 self._check_smoketest(use_aimport=True)
552 self._check_smoketest(use_aimport=True)
553
553
554 def test_smoketest_autoreload(self):
554 def test_smoketest_autoreload(self):
555 self._check_smoketest(use_aimport=False)
555 self._check_smoketest(use_aimport=False)
@@ -1,401 +1,401 b''
1 """
1 """
2 This module contains factory functions that attempt
2 This module contains factory functions that attempt
3 to return Qt submodules from the various python Qt bindings.
3 to return Qt submodules from the various python Qt bindings.
4
4
5 It also protects against double-importing Qt with different
5 It also protects against double-importing Qt with different
6 bindings, which is unstable and likely to crash
6 bindings, which is unstable and likely to crash
7
7
8 This is used primarily by qt and qt_for_kernel, and shouldn't
8 This is used primarily by qt and qt_for_kernel, and shouldn't
9 be accessed directly from the outside
9 be accessed directly from the outside
10 """
10 """
11 import sys
11 import sys
12 import types
12 import types
13 from functools import partial, lru_cache
13 from functools import partial, lru_cache
14 import operator
14 import operator
15
15
16 from IPython.utils.version import check_version
16 from IPython.utils.version import check_version
17
17
18 # ### Available APIs.
18 # ### Available APIs.
19 # Qt6
19 # Qt6
20 QT_API_PYQT6 = "pyqt6"
20 QT_API_PYQT6 = "pyqt6"
21 QT_API_PYSIDE6 = "pyside6"
21 QT_API_PYSIDE6 = "pyside6"
22
22
23 # Qt5
23 # Qt5
24 QT_API_PYQT5 = 'pyqt5'
24 QT_API_PYQT5 = 'pyqt5'
25 QT_API_PYSIDE2 = 'pyside2'
25 QT_API_PYSIDE2 = 'pyside2'
26
26
27 # Qt4
27 # Qt4
28 QT_API_PYQT = "pyqt" # Force version 2
28 QT_API_PYQT = "pyqt" # Force version 2
29 QT_API_PYQTv1 = "pyqtv1" # Force version 2
29 QT_API_PYQTv1 = "pyqtv1" # Force version 2
30 QT_API_PYSIDE = "pyside"
30 QT_API_PYSIDE = "pyside"
31
31
32 QT_API_PYQT_DEFAULT = "pyqtdefault" # use system default for version 1 vs. 2
32 QT_API_PYQT_DEFAULT = "pyqtdefault" # use system default for version 1 vs. 2
33
33
34 api_to_module = {
34 api_to_module = {
35 # Qt6
35 # Qt6
36 QT_API_PYQT6: "PyQt6",
36 QT_API_PYQT6: "PyQt6",
37 QT_API_PYSIDE6: "PySide6",
37 QT_API_PYSIDE6: "PySide6",
38 # Qt5
38 # Qt5
39 QT_API_PYQT5: "PyQt5",
39 QT_API_PYQT5: "PyQt5",
40 QT_API_PYSIDE2: "PySide2",
40 QT_API_PYSIDE2: "PySide2",
41 # Qt4
41 # Qt4
42 QT_API_PYSIDE: "PySide",
42 QT_API_PYSIDE: "PySide",
43 QT_API_PYQT: "PyQt4",
43 QT_API_PYQT: "PyQt4",
44 QT_API_PYQTv1: "PyQt4",
44 QT_API_PYQTv1: "PyQt4",
45 # default
45 # default
46 QT_API_PYQT_DEFAULT: "PyQt6",
46 QT_API_PYQT_DEFAULT: "PyQt6",
47 }
47 }
48
48
49
49
50 class ImportDenier(object):
50 class ImportDenier(object):
51 """Import Hook that will guard against bad Qt imports
51 """Import Hook that will guard against bad Qt imports
52 once IPython commits to a specific binding
52 once IPython commits to a specific binding
53 """
53 """
54
54
55 def __init__(self):
55 def __init__(self):
56 self.__forbidden = set()
56 self.__forbidden = set()
57
57
58 def forbid(self, module_name):
58 def forbid(self, module_name):
59 sys.modules.pop(module_name, None)
59 sys.modules.pop(module_name, None)
60 self.__forbidden.add(module_name)
60 self.__forbidden.add(module_name)
61
61
62 def find_module(self, fullname, path=None):
62 def find_module(self, fullname, path=None):
63 if path:
63 if path:
64 return
64 return
65 if fullname in self.__forbidden:
65 if fullname in self.__forbidden:
66 return self
66 return self
67
67
68 def load_module(self, fullname):
68 def load_module(self, fullname):
69 raise ImportError("""
69 raise ImportError("""
70 Importing %s disabled by IPython, which has
70 Importing %s disabled by IPython, which has
71 already imported an Incompatible QT Binding: %s
71 already imported an Incompatible QT Binding: %s
72 """ % (fullname, loaded_api()))
72 """ % (fullname, loaded_api()))
73
73
74
74
75 ID = ImportDenier()
75 ID = ImportDenier()
76 sys.meta_path.insert(0, ID)
76 sys.meta_path.insert(0, ID)
77
77
78
78
79 def commit_api(api):
79 def commit_api(api):
80 """Commit to a particular API, and trigger ImportErrors on subsequent
80 """Commit to a particular API, and trigger ImportErrors on subsequent
81 dangerous imports"""
81 dangerous imports"""
82 modules = set(api_to_module.values())
82 modules = set(api_to_module.values())
83
83
84 modules.remove(api_to_module[api])
84 modules.remove(api_to_module[api])
85 for mod in modules:
85 for mod in modules:
86 ID.forbid(mod)
86 ID.forbid(mod)
87
87
88
88
89 def loaded_api():
89 def loaded_api():
90 """Return which API is loaded, if any
90 """Return which API is loaded, if any
91
91
92 If this returns anything besides None,
92 If this returns anything besides None,
93 importing any other Qt binding is unsafe.
93 importing any other Qt binding is unsafe.
94
94
95 Returns
95 Returns
96 -------
96 -------
97 None, 'pyside6', 'pyqt6', 'pyside2', 'pyside', 'pyqt', 'pyqt5', 'pyqtv1'
97 None, 'pyside6', 'pyqt6', 'pyside2', 'pyside', 'pyqt', 'pyqt5', 'pyqtv1'
98 """
98 """
99 if sys.modules.get("PyQt6.QtCore"):
99 if sys.modules.get("PyQt6.QtCore"):
100 return QT_API_PYQT6
100 return QT_API_PYQT6
101 elif sys.modules.get("PySide6.QtCore"):
101 elif sys.modules.get("PySide6.QtCore"):
102 return QT_API_PYSIDE6
102 return QT_API_PYSIDE6
103 elif sys.modules.get("PyQt5.QtCore"):
103 elif sys.modules.get("PyQt5.QtCore"):
104 return QT_API_PYQT5
104 return QT_API_PYQT5
105 elif sys.modules.get("PySide2.QtCore"):
105 elif sys.modules.get("PySide2.QtCore"):
106 return QT_API_PYSIDE2
106 return QT_API_PYSIDE2
107 elif sys.modules.get("PyQt4.QtCore"):
107 elif sys.modules.get("PyQt4.QtCore"):
108 if qtapi_version() == 2:
108 if qtapi_version() == 2:
109 return QT_API_PYQT
109 return QT_API_PYQT
110 else:
110 else:
111 return QT_API_PYQTv1
111 return QT_API_PYQTv1
112 elif sys.modules.get("PySide.QtCore"):
112 elif sys.modules.get("PySide.QtCore"):
113 return QT_API_PYSIDE
113 return QT_API_PYSIDE
114
114
115 return None
115 return None
116
116
117
117
118 def has_binding(api):
118 def has_binding(api):
119 """Safely check for PyQt4/5, PySide or PySide2, without importing submodules
119 """Safely check for PyQt4/5, PySide or PySide2, without importing submodules
120
120
121 Parameters
121 Parameters
122 ----------
122 ----------
123 api : str [ 'pyqtv1' | 'pyqt' | 'pyqt5' | 'pyside' | 'pyside2' | 'pyqtdefault']
123 api : str [ 'pyqtv1' | 'pyqt' | 'pyqt5' | 'pyside' | 'pyside2' | 'pyqtdefault']
124 Which module to check for
124 Which module to check for
125
125
126 Returns
126 Returns
127 -------
127 -------
128 True if the relevant module appears to be importable
128 True if the relevant module appears to be importable
129 """
129 """
130 module_name = api_to_module[api]
130 module_name = api_to_module[api]
131 from importlib.util import find_spec
131 from importlib.util import find_spec
132
132
133 required = ['QtCore', 'QtGui', 'QtSvg']
133 required = ['QtCore', 'QtGui', 'QtSvg']
134 if api in (QT_API_PYQT5, QT_API_PYSIDE2, QT_API_PYQT6, QT_API_PYSIDE6):
134 if api in (QT_API_PYQT5, QT_API_PYSIDE2, QT_API_PYQT6, QT_API_PYSIDE6):
135 # QT5 requires QtWidgets too
135 # QT5 requires QtWidgets too
136 required.append('QtWidgets')
136 required.append('QtWidgets')
137
137
138 for submod in required:
138 for submod in required:
139 try:
139 try:
140 spec = find_spec('%s.%s' % (module_name, submod))
140 spec = find_spec('%s.%s' % (module_name, submod))
141 except ImportError:
141 except ImportError:
142 # Package (e.g. PyQt5) not found
142 # Package (e.g. PyQt5) not found
143 return False
143 return False
144 else:
144 else:
145 if spec is None:
145 if spec is None:
146 # Submodule (e.g. PyQt5.QtCore) not found
146 # Submodule (e.g. PyQt5.QtCore) not found
147 return False
147 return False
148
148
149 if api == QT_API_PYSIDE:
149 if api == QT_API_PYSIDE:
150 # We can also safely check PySide version
150 # We can also safely check PySide version
151 import PySide
151 import PySide
152 return check_version(PySide.__version__, '1.0.3')
152 return check_version(PySide.__version__, '1.0.3')
153
153
154 return True
154 return True
155
155
156
156
157 def qtapi_version():
157 def qtapi_version():
158 """Return which QString API has been set, if any
158 """Return which QString API has been set, if any
159
159
160 Returns
160 Returns
161 -------
161 -------
162 The QString API version (1 or 2), or None if not set
162 The QString API version (1 or 2), or None if not set
163 """
163 """
164 try:
164 try:
165 import sip
165 import sip
166 except ImportError:
166 except ImportError:
167 # as of PyQt5 5.11, sip is no longer available as a top-level
167 # as of PyQt5 5.11, sip is no longer available as a top-level
168 # module and needs to be imported from the PyQt5 namespace
168 # module and needs to be imported from the PyQt5 namespace
169 try:
169 try:
170 from PyQt5 import sip
170 from PyQt5 import sip
171 except ImportError:
171 except ImportError:
172 return
172 return
173 try:
173 try:
174 return sip.getapi('QString')
174 return sip.getapi('QString')
175 except ValueError:
175 except ValueError:
176 return
176 return
177
177
178
178
179 def can_import(api):
179 def can_import(api):
180 """Safely query whether an API is importable, without importing it"""
180 """Safely query whether an API is importable, without importing it"""
181 if not has_binding(api):
181 if not has_binding(api):
182 return False
182 return False
183
183
184 current = loaded_api()
184 current = loaded_api()
185 if api == QT_API_PYQT_DEFAULT:
185 if api == QT_API_PYQT_DEFAULT:
186 return current in [QT_API_PYQT6, None]
186 return current in [QT_API_PYQT6, None]
187 else:
187 else:
188 return current in [api, None]
188 return current in [api, None]
189
189
190
190
191 def import_pyqt4(version=2):
191 def import_pyqt4(version=2):
192 """
192 """
193 Import PyQt4
193 Import PyQt4
194
194
195 Parameters
195 Parameters
196 ----------
196 ----------
197 version : 1, 2, or None
197 version : 1, 2, or None
198 Which QString/QVariant API to use. Set to None to use the system
198 Which QString/QVariant API to use. Set to None to use the system
199 default
199 default
200
200
201 ImportErrors rasied within this function are non-recoverable
201 ImportErrors raised within this function are non-recoverable
202 """
202 """
203 # The new-style string API (version=2) automatically
203 # The new-style string API (version=2) automatically
204 # converts QStrings to Unicode Python strings. Also, automatically unpacks
204 # converts QStrings to Unicode Python strings. Also, automatically unpacks
205 # QVariants to their underlying objects.
205 # QVariants to their underlying objects.
206 import sip
206 import sip
207
207
208 if version is not None:
208 if version is not None:
209 sip.setapi('QString', version)
209 sip.setapi('QString', version)
210 sip.setapi('QVariant', version)
210 sip.setapi('QVariant', version)
211
211
212 from PyQt4 import QtGui, QtCore, QtSvg
212 from PyQt4 import QtGui, QtCore, QtSvg
213
213
214 if not check_version(QtCore.PYQT_VERSION_STR, '4.7'):
214 if not check_version(QtCore.PYQT_VERSION_STR, '4.7'):
215 raise ImportError("IPython requires PyQt4 >= 4.7, found %s" %
215 raise ImportError("IPython requires PyQt4 >= 4.7, found %s" %
216 QtCore.PYQT_VERSION_STR)
216 QtCore.PYQT_VERSION_STR)
217
217
218 # Alias PyQt-specific functions for PySide compatibility.
218 # Alias PyQt-specific functions for PySide compatibility.
219 QtCore.Signal = QtCore.pyqtSignal
219 QtCore.Signal = QtCore.pyqtSignal
220 QtCore.Slot = QtCore.pyqtSlot
220 QtCore.Slot = QtCore.pyqtSlot
221
221
222 # query for the API version (in case version == None)
222 # query for the API version (in case version == None)
223 version = sip.getapi('QString')
223 version = sip.getapi('QString')
224 api = QT_API_PYQTv1 if version == 1 else QT_API_PYQT
224 api = QT_API_PYQTv1 if version == 1 else QT_API_PYQT
225 return QtCore, QtGui, QtSvg, api
225 return QtCore, QtGui, QtSvg, api
226
226
227
227
228 def import_pyqt5():
228 def import_pyqt5():
229 """
229 """
230 Import PyQt5
230 Import PyQt5
231
231
232 ImportErrors rasied within this function are non-recoverable
232 ImportErrors raised within this function are non-recoverable
233 """
233 """
234
234
235 from PyQt5 import QtCore, QtSvg, QtWidgets, QtGui
235 from PyQt5 import QtCore, QtSvg, QtWidgets, QtGui
236
236
237 # Alias PyQt-specific functions for PySide compatibility.
237 # Alias PyQt-specific functions for PySide compatibility.
238 QtCore.Signal = QtCore.pyqtSignal
238 QtCore.Signal = QtCore.pyqtSignal
239 QtCore.Slot = QtCore.pyqtSlot
239 QtCore.Slot = QtCore.pyqtSlot
240
240
241 # Join QtGui and QtWidgets for Qt4 compatibility.
241 # Join QtGui and QtWidgets for Qt4 compatibility.
242 QtGuiCompat = types.ModuleType('QtGuiCompat')
242 QtGuiCompat = types.ModuleType('QtGuiCompat')
243 QtGuiCompat.__dict__.update(QtGui.__dict__)
243 QtGuiCompat.__dict__.update(QtGui.__dict__)
244 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
244 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
245
245
246 api = QT_API_PYQT5
246 api = QT_API_PYQT5
247 return QtCore, QtGuiCompat, QtSvg, api
247 return QtCore, QtGuiCompat, QtSvg, api
248
248
249
249
250 def import_pyqt6():
250 def import_pyqt6():
251 """
251 """
252 Import PyQt6
252 Import PyQt6
253
253
254 ImportErrors rasied within this function are non-recoverable
254 ImportErrors raised within this function are non-recoverable
255 """
255 """
256
256
257 from PyQt6 import QtCore, QtSvg, QtWidgets, QtGui
257 from PyQt6 import QtCore, QtSvg, QtWidgets, QtGui
258
258
259 # Alias PyQt-specific functions for PySide compatibility.
259 # Alias PyQt-specific functions for PySide compatibility.
260 QtCore.Signal = QtCore.pyqtSignal
260 QtCore.Signal = QtCore.pyqtSignal
261 QtCore.Slot = QtCore.pyqtSlot
261 QtCore.Slot = QtCore.pyqtSlot
262
262
263 # Join QtGui and QtWidgets for Qt4 compatibility.
263 # Join QtGui and QtWidgets for Qt4 compatibility.
264 QtGuiCompat = types.ModuleType("QtGuiCompat")
264 QtGuiCompat = types.ModuleType("QtGuiCompat")
265 QtGuiCompat.__dict__.update(QtGui.__dict__)
265 QtGuiCompat.__dict__.update(QtGui.__dict__)
266 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
266 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
267
267
268 api = QT_API_PYQT6
268 api = QT_API_PYQT6
269 return QtCore, QtGuiCompat, QtSvg, api
269 return QtCore, QtGuiCompat, QtSvg, api
270
270
271
271
272 def import_pyside():
272 def import_pyside():
273 """
273 """
274 Import PySide
274 Import PySide
275
275
276 ImportErrors raised within this function are non-recoverable
276 ImportErrors raised within this function are non-recoverable
277 """
277 """
278 from PySide import QtGui, QtCore, QtSvg
278 from PySide import QtGui, QtCore, QtSvg
279 return QtCore, QtGui, QtSvg, QT_API_PYSIDE
279 return QtCore, QtGui, QtSvg, QT_API_PYSIDE
280
280
281 def import_pyside2():
281 def import_pyside2():
282 """
282 """
283 Import PySide2
283 Import PySide2
284
284
285 ImportErrors raised within this function are non-recoverable
285 ImportErrors raised within this function are non-recoverable
286 """
286 """
287 from PySide2 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
287 from PySide2 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
288
288
289 # Join QtGui and QtWidgets for Qt4 compatibility.
289 # Join QtGui and QtWidgets for Qt4 compatibility.
290 QtGuiCompat = types.ModuleType('QtGuiCompat')
290 QtGuiCompat = types.ModuleType('QtGuiCompat')
291 QtGuiCompat.__dict__.update(QtGui.__dict__)
291 QtGuiCompat.__dict__.update(QtGui.__dict__)
292 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
292 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
293 QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
293 QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
294
294
295 return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE2
295 return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE2
296
296
297
297
298 def import_pyside6():
298 def import_pyside6():
299 """
299 """
300 Import PySide6
300 Import PySide6
301
301
302 ImportErrors raised within this function are non-recoverable
302 ImportErrors raised within this function are non-recoverable
303 """
303 """
304 from PySide6 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
304 from PySide6 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
305
305
306 # Join QtGui and QtWidgets for Qt4 compatibility.
306 # Join QtGui and QtWidgets for Qt4 compatibility.
307 QtGuiCompat = types.ModuleType("QtGuiCompat")
307 QtGuiCompat = types.ModuleType("QtGuiCompat")
308 QtGuiCompat.__dict__.update(QtGui.__dict__)
308 QtGuiCompat.__dict__.update(QtGui.__dict__)
309 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
309 QtGuiCompat.__dict__.update(QtWidgets.__dict__)
310 QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
310 QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
311
311
312 return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE6
312 return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE6
313
313
314
314
315 def load_qt(api_options):
315 def load_qt(api_options):
316 """
316 """
317 Attempt to import Qt, given a preference list
317 Attempt to import Qt, given a preference list
318 of permissible bindings
318 of permissible bindings
319
319
320 It is safe to call this function multiple times.
320 It is safe to call this function multiple times.
321
321
322 Parameters
322 Parameters
323 ----------
323 ----------
324 api_options: List of strings
324 api_options: List of strings
325 The order of APIs to try. Valid items are 'pyside', 'pyside2',
325 The order of APIs to try. Valid items are 'pyside', 'pyside2',
326 'pyqt', 'pyqt5', 'pyqtv1' and 'pyqtdefault'
326 'pyqt', 'pyqt5', 'pyqtv1' and 'pyqtdefault'
327
327
328 Returns
328 Returns
329 -------
329 -------
330
330
331 A tuple of QtCore, QtGui, QtSvg, QT_API
331 A tuple of QtCore, QtGui, QtSvg, QT_API
332 The first three are the Qt modules. The last is the
332 The first three are the Qt modules. The last is the
333 string indicating which module was loaded.
333 string indicating which module was loaded.
334
334
335 Raises
335 Raises
336 ------
336 ------
337 ImportError, if it isn't possible to import any requested
337 ImportError, if it isn't possible to import any requested
338 bindings (either because they aren't installed, or because
338 bindings (either because they aren't installed, or because
339 an incompatible library has already been installed)
339 an incompatible library has already been installed)
340 """
340 """
341 loaders = {
341 loaders = {
342 # Qt6
342 # Qt6
343 QT_API_PYQT6: import_pyqt6,
343 QT_API_PYQT6: import_pyqt6,
344 QT_API_PYSIDE6: import_pyside6,
344 QT_API_PYSIDE6: import_pyside6,
345 # Qt5
345 # Qt5
346 QT_API_PYQT5: import_pyqt5,
346 QT_API_PYQT5: import_pyqt5,
347 QT_API_PYSIDE2: import_pyside2,
347 QT_API_PYSIDE2: import_pyside2,
348 # Qt4
348 # Qt4
349 QT_API_PYSIDE: import_pyside,
349 QT_API_PYSIDE: import_pyside,
350 QT_API_PYQT: import_pyqt4,
350 QT_API_PYQT: import_pyqt4,
351 QT_API_PYQTv1: partial(import_pyqt4, version=1),
351 QT_API_PYQTv1: partial(import_pyqt4, version=1),
352 # default
352 # default
353 QT_API_PYQT_DEFAULT: import_pyqt6,
353 QT_API_PYQT_DEFAULT: import_pyqt6,
354 }
354 }
355
355
356 for api in api_options:
356 for api in api_options:
357
357
358 if api not in loaders:
358 if api not in loaders:
359 raise RuntimeError(
359 raise RuntimeError(
360 "Invalid Qt API %r, valid values are: %s" %
360 "Invalid Qt API %r, valid values are: %s" %
361 (api, ", ".join(["%r" % k for k in loaders.keys()])))
361 (api, ", ".join(["%r" % k for k in loaders.keys()])))
362
362
363 if not can_import(api):
363 if not can_import(api):
364 continue
364 continue
365
365
366 #cannot safely recover from an ImportError during this
366 #cannot safely recover from an ImportError during this
367 result = loaders[api]()
367 result = loaders[api]()
368 api = result[-1] # changed if api = QT_API_PYQT_DEFAULT
368 api = result[-1] # changed if api = QT_API_PYQT_DEFAULT
369 commit_api(api)
369 commit_api(api)
370 return result
370 return result
371 else:
371 else:
372 raise ImportError("""
372 raise ImportError("""
373 Could not load requested Qt binding. Please ensure that
373 Could not load requested Qt binding. Please ensure that
374 PyQt4 >= 4.7, PyQt5, PySide >= 1.0.3 or PySide2 is available,
374 PyQt4 >= 4.7, PyQt5, PySide >= 1.0.3 or PySide2 is available,
375 and only one is imported per session.
375 and only one is imported per session.
376
376
377 Currently-imported Qt library: %r
377 Currently-imported Qt library: %r
378 PyQt4 available (requires QtCore, QtGui, QtSvg): %s
378 PyQt4 available (requires QtCore, QtGui, QtSvg): %s
379 PyQt5 available (requires QtCore, QtGui, QtSvg, QtWidgets): %s
379 PyQt5 available (requires QtCore, QtGui, QtSvg, QtWidgets): %s
380 PySide >= 1.0.3 installed: %s
380 PySide >= 1.0.3 installed: %s
381 PySide2 installed: %s
381 PySide2 installed: %s
382 Tried to load: %r
382 Tried to load: %r
383 """ % (loaded_api(),
383 """ % (loaded_api(),
384 has_binding(QT_API_PYQT),
384 has_binding(QT_API_PYQT),
385 has_binding(QT_API_PYQT5),
385 has_binding(QT_API_PYQT5),
386 has_binding(QT_API_PYSIDE),
386 has_binding(QT_API_PYSIDE),
387 has_binding(QT_API_PYSIDE2),
387 has_binding(QT_API_PYSIDE2),
388 api_options))
388 api_options))
389
389
390
390
391 def enum_factory(QT_API, QtCore):
391 def enum_factory(QT_API, QtCore):
392 """Construct an enum helper to account for PyQt5 <-> PyQt6 changes."""
392 """Construct an enum helper to account for PyQt5 <-> PyQt6 changes."""
393
393
394 @lru_cache(None)
394 @lru_cache(None)
395 def _enum(name):
395 def _enum(name):
396 # foo.bar.Enum.Entry (PyQt6) <=> foo.bar.Entry (non-PyQt6).
396 # foo.bar.Enum.Entry (PyQt6) <=> foo.bar.Entry (non-PyQt6).
397 return operator.attrgetter(
397 return operator.attrgetter(
398 name if QT_API == QT_API_PYQT6 else name.rpartition(".")[0]
398 name if QT_API == QT_API_PYQT6 else name.rpartition(".")[0]
399 )(sys.modules[QtCore.__package__])
399 )(sys.modules[QtCore.__package__])
400
400
401 return _enum
401 return _enum
@@ -1,491 +1,491 b''
1 # -*- coding: utf-8 -*-
1 # -*- coding: utf-8 -*-
2 """Manage background (threaded) jobs conveniently from an interactive shell.
2 """Manage background (threaded) jobs conveniently from an interactive shell.
3
3
4 This module provides a BackgroundJobManager class. This is the main class
4 This module provides a BackgroundJobManager class. This is the main class
5 meant for public usage, it implements an object which can create and manage
5 meant for public usage, it implements an object which can create and manage
6 new background jobs.
6 new background jobs.
7
7
8 It also provides the actual job classes managed by these BackgroundJobManager
8 It also provides the actual job classes managed by these BackgroundJobManager
9 objects, see their docstrings below.
9 objects, see their docstrings below.
10
10
11
11
12 This system was inspired by discussions with B. Granger and the
12 This system was inspired by discussions with B. Granger and the
13 BackgroundCommand class described in the book Python Scripting for
13 BackgroundCommand class described in the book Python Scripting for
14 Computational Science, by H. P. Langtangen:
14 Computational Science, by H. P. Langtangen:
15
15
16 http://folk.uio.no/hpl/scripting
16 http://folk.uio.no/hpl/scripting
17
17
18 (although ultimately no code from this text was used, as IPython's system is a
18 (although ultimately no code from this text was used, as IPython's system is a
19 separate implementation).
19 separate implementation).
20
20
21 An example notebook is provided in our documentation illustrating interactive
21 An example notebook is provided in our documentation illustrating interactive
22 use of the system.
22 use of the system.
23 """
23 """
24
24
25 #*****************************************************************************
25 #*****************************************************************************
26 # Copyright (C) 2005-2006 Fernando Perez <fperez@colorado.edu>
26 # Copyright (C) 2005-2006 Fernando Perez <fperez@colorado.edu>
27 #
27 #
28 # Distributed under the terms of the BSD License. The full license is in
28 # Distributed under the terms of the BSD License. The full license is in
29 # the file COPYING, distributed as part of this software.
29 # the file COPYING, distributed as part of this software.
30 #*****************************************************************************
30 #*****************************************************************************
31
31
32 # Code begins
32 # Code begins
33 import sys
33 import sys
34 import threading
34 import threading
35
35
36 from IPython import get_ipython
36 from IPython import get_ipython
37 from IPython.core.ultratb import AutoFormattedTB
37 from IPython.core.ultratb import AutoFormattedTB
38 from logging import error, debug
38 from logging import error, debug
39
39
40
40
41 class BackgroundJobManager(object):
41 class BackgroundJobManager(object):
42 """Class to manage a pool of backgrounded threaded jobs.
42 """Class to manage a pool of backgrounded threaded jobs.
43
43
44 Below, we assume that 'jobs' is a BackgroundJobManager instance.
44 Below, we assume that 'jobs' is a BackgroundJobManager instance.
45
45
46 Usage summary (see the method docstrings for details):
46 Usage summary (see the method docstrings for details):
47
47
48 jobs.new(...) -> start a new job
48 jobs.new(...) -> start a new job
49
49
50 jobs() or jobs.status() -> print status summary of all jobs
50 jobs() or jobs.status() -> print status summary of all jobs
51
51
52 jobs[N] -> returns job number N.
52 jobs[N] -> returns job number N.
53
53
54 foo = jobs[N].result -> assign to variable foo the result of job N
54 foo = jobs[N].result -> assign to variable foo the result of job N
55
55
56 jobs[N].traceback() -> print the traceback of dead job N
56 jobs[N].traceback() -> print the traceback of dead job N
57
57
58 jobs.remove(N) -> remove (finished) job N
58 jobs.remove(N) -> remove (finished) job N
59
59
60 jobs.flush() -> remove all finished jobs
60 jobs.flush() -> remove all finished jobs
61
61
62 As a convenience feature, BackgroundJobManager instances provide the
62 As a convenience feature, BackgroundJobManager instances provide the
63 utility result and traceback methods which retrieve the corresponding
63 utility result and traceback methods which retrieve the corresponding
64 information from the jobs list:
64 information from the jobs list:
65
65
66 jobs.result(N) <--> jobs[N].result
66 jobs.result(N) <--> jobs[N].result
67 jobs.traceback(N) <--> jobs[N].traceback()
67 jobs.traceback(N) <--> jobs[N].traceback()
68
68
69 While this appears minor, it allows you to use tab completion
69 While this appears minor, it allows you to use tab completion
70 interactively on the job manager instance.
70 interactively on the job manager instance.
71 """
71 """
72
72
73 def __init__(self):
73 def __init__(self):
74 # Lists for job management, accessed via a property to ensure they're
74 # Lists for job management, accessed via a property to ensure they're
75 # up to date.x
75 # up to date.x
76 self._running = []
76 self._running = []
77 self._completed = []
77 self._completed = []
78 self._dead = []
78 self._dead = []
79 # A dict of all jobs, so users can easily access any of them
79 # A dict of all jobs, so users can easily access any of them
80 self.all = {}
80 self.all = {}
81 # For reporting
81 # For reporting
82 self._comp_report = []
82 self._comp_report = []
83 self._dead_report = []
83 self._dead_report = []
84 # Store status codes locally for fast lookups
84 # Store status codes locally for fast lookups
85 self._s_created = BackgroundJobBase.stat_created_c
85 self._s_created = BackgroundJobBase.stat_created_c
86 self._s_running = BackgroundJobBase.stat_running_c
86 self._s_running = BackgroundJobBase.stat_running_c
87 self._s_completed = BackgroundJobBase.stat_completed_c
87 self._s_completed = BackgroundJobBase.stat_completed_c
88 self._s_dead = BackgroundJobBase.stat_dead_c
88 self._s_dead = BackgroundJobBase.stat_dead_c
89 self._current_job_id = 0
89 self._current_job_id = 0
90
90
91 @property
91 @property
92 def running(self):
92 def running(self):
93 self._update_status()
93 self._update_status()
94 return self._running
94 return self._running
95
95
96 @property
96 @property
97 def dead(self):
97 def dead(self):
98 self._update_status()
98 self._update_status()
99 return self._dead
99 return self._dead
100
100
101 @property
101 @property
102 def completed(self):
102 def completed(self):
103 self._update_status()
103 self._update_status()
104 return self._completed
104 return self._completed
105
105
106 def new(self, func_or_exp, *args, **kwargs):
106 def new(self, func_or_exp, *args, **kwargs):
107 """Add a new background job and start it in a separate thread.
107 """Add a new background job and start it in a separate thread.
108
108
109 There are two types of jobs which can be created:
109 There are two types of jobs which can be created:
110
110
111 1. Jobs based on expressions which can be passed to an eval() call.
111 1. Jobs based on expressions which can be passed to an eval() call.
112 The expression must be given as a string. For example:
112 The expression must be given as a string. For example:
113
113
114 job_manager.new('myfunc(x,y,z=1)'[,glob[,loc]])
114 job_manager.new('myfunc(x,y,z=1)'[,glob[,loc]])
115
115
116 The given expression is passed to eval(), along with the optional
116 The given expression is passed to eval(), along with the optional
117 global/local dicts provided. If no dicts are given, they are
117 global/local dicts provided. If no dicts are given, they are
118 extracted automatically from the caller's frame.
118 extracted automatically from the caller's frame.
119
119
120 A Python statement is NOT a valid eval() expression. Basically, you
120 A Python statement is NOT a valid eval() expression. Basically, you
121 can only use as an eval() argument something which can go on the right
121 can only use as an eval() argument something which can go on the right
122 of an '=' sign and be assigned to a variable.
122 of an '=' sign and be assigned to a variable.
123
123
124 For example,"print 'hello'" is not valid, but '2+3' is.
124 For example,"print 'hello'" is not valid, but '2+3' is.
125
125
126 2. Jobs given a function object, optionally passing additional
126 2. Jobs given a function object, optionally passing additional
127 positional arguments:
127 positional arguments:
128
128
129 job_manager.new(myfunc, x, y)
129 job_manager.new(myfunc, x, y)
130
130
131 The function is called with the given arguments.
131 The function is called with the given arguments.
132
132
133 If you need to pass keyword arguments to your function, you must
133 If you need to pass keyword arguments to your function, you must
134 supply them as a dict named kw:
134 supply them as a dict named kw:
135
135
136 job_manager.new(myfunc, x, y, kw=dict(z=1))
136 job_manager.new(myfunc, x, y, kw=dict(z=1))
137
137
138 The reason for this assymmetry is that the new() method needs to
138 The reason for this asymmetry is that the new() method needs to
139 maintain access to its own keywords, and this prevents name collisions
139 maintain access to its own keywords, and this prevents name collisions
140 between arguments to new() and arguments to your own functions.
140 between arguments to new() and arguments to your own functions.
141
141
142 In both cases, the result is stored in the job.result field of the
142 In both cases, the result is stored in the job.result field of the
143 background job object.
143 background job object.
144
144
145 You can set `daemon` attribute of the thread by giving the keyword
145 You can set `daemon` attribute of the thread by giving the keyword
146 argument `daemon`.
146 argument `daemon`.
147
147
148 Notes and caveats:
148 Notes and caveats:
149
149
150 1. All threads running share the same standard output. Thus, if your
150 1. All threads running share the same standard output. Thus, if your
151 background jobs generate output, it will come out on top of whatever
151 background jobs generate output, it will come out on top of whatever
152 you are currently writing. For this reason, background jobs are best
152 you are currently writing. For this reason, background jobs are best
153 used with silent functions which simply return their output.
153 used with silent functions which simply return their output.
154
154
155 2. Threads also all work within the same global namespace, and this
155 2. Threads also all work within the same global namespace, and this
156 system does not lock interactive variables. So if you send job to the
156 system does not lock interactive variables. So if you send job to the
157 background which operates on a mutable object for a long time, and
157 background which operates on a mutable object for a long time, and
158 start modifying that same mutable object interactively (or in another
158 start modifying that same mutable object interactively (or in another
159 backgrounded job), all sorts of bizarre behaviour will occur.
159 backgrounded job), all sorts of bizarre behaviour will occur.
160
160
161 3. If a background job is spending a lot of time inside a C extension
161 3. If a background job is spending a lot of time inside a C extension
162 module which does not release the Python Global Interpreter Lock
162 module which does not release the Python Global Interpreter Lock
163 (GIL), this will block the IPython prompt. This is simply because the
163 (GIL), this will block the IPython prompt. This is simply because the
164 Python interpreter can only switch between threads at Python
164 Python interpreter can only switch between threads at Python
165 bytecodes. While the execution is inside C code, the interpreter must
165 bytecodes. While the execution is inside C code, the interpreter must
166 simply wait unless the extension module releases the GIL.
166 simply wait unless the extension module releases the GIL.
167
167
168 4. There is no way, due to limitations in the Python threads library,
168 4. There is no way, due to limitations in the Python threads library,
169 to kill a thread once it has started."""
169 to kill a thread once it has started."""
170
170
171 if callable(func_or_exp):
171 if callable(func_or_exp):
172 kw = kwargs.get('kw',{})
172 kw = kwargs.get('kw',{})
173 job = BackgroundJobFunc(func_or_exp,*args,**kw)
173 job = BackgroundJobFunc(func_or_exp,*args,**kw)
174 elif isinstance(func_or_exp, str):
174 elif isinstance(func_or_exp, str):
175 if not args:
175 if not args:
176 frame = sys._getframe(1)
176 frame = sys._getframe(1)
177 glob, loc = frame.f_globals, frame.f_locals
177 glob, loc = frame.f_globals, frame.f_locals
178 elif len(args)==1:
178 elif len(args)==1:
179 glob = loc = args[0]
179 glob = loc = args[0]
180 elif len(args)==2:
180 elif len(args)==2:
181 glob,loc = args
181 glob,loc = args
182 else:
182 else:
183 raise ValueError(
183 raise ValueError(
184 'Expression jobs take at most 2 args (globals,locals)')
184 'Expression jobs take at most 2 args (globals,locals)')
185 job = BackgroundJobExpr(func_or_exp, glob, loc)
185 job = BackgroundJobExpr(func_or_exp, glob, loc)
186 else:
186 else:
187 raise TypeError('invalid args for new job')
187 raise TypeError('invalid args for new job')
188
188
189 if kwargs.get('daemon', False):
189 if kwargs.get('daemon', False):
190 job.daemon = True
190 job.daemon = True
191 job.num = self._current_job_id
191 job.num = self._current_job_id
192 self._current_job_id += 1
192 self._current_job_id += 1
193 self.running.append(job)
193 self.running.append(job)
194 self.all[job.num] = job
194 self.all[job.num] = job
195 debug('Starting job # %s in a separate thread.' % job.num)
195 debug('Starting job # %s in a separate thread.' % job.num)
196 job.start()
196 job.start()
197 return job
197 return job
198
198
199 def __getitem__(self, job_key):
199 def __getitem__(self, job_key):
200 num = job_key if isinstance(job_key, int) else job_key.num
200 num = job_key if isinstance(job_key, int) else job_key.num
201 return self.all[num]
201 return self.all[num]
202
202
203 def __call__(self):
203 def __call__(self):
204 """An alias to self.status(),
204 """An alias to self.status(),
205
205
206 This allows you to simply call a job manager instance much like the
206 This allows you to simply call a job manager instance much like the
207 Unix `jobs` shell command."""
207 Unix `jobs` shell command."""
208
208
209 return self.status()
209 return self.status()
210
210
211 def _update_status(self):
211 def _update_status(self):
212 """Update the status of the job lists.
212 """Update the status of the job lists.
213
213
214 This method moves finished jobs to one of two lists:
214 This method moves finished jobs to one of two lists:
215 - self.completed: jobs which completed successfully
215 - self.completed: jobs which completed successfully
216 - self.dead: jobs which finished but died.
216 - self.dead: jobs which finished but died.
217
217
218 It also copies those jobs to corresponding _report lists. These lists
218 It also copies those jobs to corresponding _report lists. These lists
219 are used to report jobs completed/dead since the last update, and are
219 are used to report jobs completed/dead since the last update, and are
220 then cleared by the reporting function after each call."""
220 then cleared by the reporting function after each call."""
221
221
222 # Status codes
222 # Status codes
223 srun, scomp, sdead = self._s_running, self._s_completed, self._s_dead
223 srun, scomp, sdead = self._s_running, self._s_completed, self._s_dead
224 # State lists, use the actual lists b/c the public names are properties
224 # State lists, use the actual lists b/c the public names are properties
225 # that call this very function on access
225 # that call this very function on access
226 running, completed, dead = self._running, self._completed, self._dead
226 running, completed, dead = self._running, self._completed, self._dead
227
227
228 # Now, update all state lists
228 # Now, update all state lists
229 for num, job in enumerate(running):
229 for num, job in enumerate(running):
230 stat = job.stat_code
230 stat = job.stat_code
231 if stat == srun:
231 if stat == srun:
232 continue
232 continue
233 elif stat == scomp:
233 elif stat == scomp:
234 completed.append(job)
234 completed.append(job)
235 self._comp_report.append(job)
235 self._comp_report.append(job)
236 running[num] = False
236 running[num] = False
237 elif stat == sdead:
237 elif stat == sdead:
238 dead.append(job)
238 dead.append(job)
239 self._dead_report.append(job)
239 self._dead_report.append(job)
240 running[num] = False
240 running[num] = False
241 # Remove dead/completed jobs from running list
241 # Remove dead/completed jobs from running list
242 running[:] = filter(None, running)
242 running[:] = filter(None, running)
243
243
244 def _group_report(self,group,name):
244 def _group_report(self,group,name):
245 """Report summary for a given job group.
245 """Report summary for a given job group.
246
246
247 Return True if the group had any elements."""
247 Return True if the group had any elements."""
248
248
249 if group:
249 if group:
250 print('%s jobs:' % name)
250 print('%s jobs:' % name)
251 for job in group:
251 for job in group:
252 print('%s : %s' % (job.num,job))
252 print('%s : %s' % (job.num,job))
253 print()
253 print()
254 return True
254 return True
255
255
256 def _group_flush(self,group,name):
256 def _group_flush(self,group,name):
257 """Flush a given job group
257 """Flush a given job group
258
258
259 Return True if the group had any elements."""
259 Return True if the group had any elements."""
260
260
261 njobs = len(group)
261 njobs = len(group)
262 if njobs:
262 if njobs:
263 plural = {1:''}.setdefault(njobs,'s')
263 plural = {1:''}.setdefault(njobs,'s')
264 print('Flushing %s %s job%s.' % (njobs,name,plural))
264 print('Flushing %s %s job%s.' % (njobs,name,plural))
265 group[:] = []
265 group[:] = []
266 return True
266 return True
267
267
268 def _status_new(self):
268 def _status_new(self):
269 """Print the status of newly finished jobs.
269 """Print the status of newly finished jobs.
270
270
271 Return True if any new jobs are reported.
271 Return True if any new jobs are reported.
272
272
273 This call resets its own state every time, so it only reports jobs
273 This call resets its own state every time, so it only reports jobs
274 which have finished since the last time it was called."""
274 which have finished since the last time it was called."""
275
275
276 self._update_status()
276 self._update_status()
277 new_comp = self._group_report(self._comp_report, 'Completed')
277 new_comp = self._group_report(self._comp_report, 'Completed')
278 new_dead = self._group_report(self._dead_report,
278 new_dead = self._group_report(self._dead_report,
279 'Dead, call jobs.traceback() for details')
279 'Dead, call jobs.traceback() for details')
280 self._comp_report[:] = []
280 self._comp_report[:] = []
281 self._dead_report[:] = []
281 self._dead_report[:] = []
282 return new_comp or new_dead
282 return new_comp or new_dead
283
283
284 def status(self,verbose=0):
284 def status(self,verbose=0):
285 """Print a status of all jobs currently being managed."""
285 """Print a status of all jobs currently being managed."""
286
286
287 self._update_status()
287 self._update_status()
288 self._group_report(self.running,'Running')
288 self._group_report(self.running,'Running')
289 self._group_report(self.completed,'Completed')
289 self._group_report(self.completed,'Completed')
290 self._group_report(self.dead,'Dead')
290 self._group_report(self.dead,'Dead')
291 # Also flush the report queues
291 # Also flush the report queues
292 self._comp_report[:] = []
292 self._comp_report[:] = []
293 self._dead_report[:] = []
293 self._dead_report[:] = []
294
294
295 def remove(self,num):
295 def remove(self,num):
296 """Remove a finished (completed or dead) job."""
296 """Remove a finished (completed or dead) job."""
297
297
298 try:
298 try:
299 job = self.all[num]
299 job = self.all[num]
300 except KeyError:
300 except KeyError:
301 error('Job #%s not found' % num)
301 error('Job #%s not found' % num)
302 else:
302 else:
303 stat_code = job.stat_code
303 stat_code = job.stat_code
304 if stat_code == self._s_running:
304 if stat_code == self._s_running:
305 error('Job #%s is still running, it can not be removed.' % num)
305 error('Job #%s is still running, it can not be removed.' % num)
306 return
306 return
307 elif stat_code == self._s_completed:
307 elif stat_code == self._s_completed:
308 self.completed.remove(job)
308 self.completed.remove(job)
309 elif stat_code == self._s_dead:
309 elif stat_code == self._s_dead:
310 self.dead.remove(job)
310 self.dead.remove(job)
311
311
312 def flush(self):
312 def flush(self):
313 """Flush all finished jobs (completed and dead) from lists.
313 """Flush all finished jobs (completed and dead) from lists.
314
314
315 Running jobs are never flushed.
315 Running jobs are never flushed.
316
316
317 It first calls _status_new(), to update info. If any jobs have
317 It first calls _status_new(), to update info. If any jobs have
318 completed since the last _status_new() call, the flush operation
318 completed since the last _status_new() call, the flush operation
319 aborts."""
319 aborts."""
320
320
321 # Remove the finished jobs from the master dict
321 # Remove the finished jobs from the master dict
322 alljobs = self.all
322 alljobs = self.all
323 for job in self.completed+self.dead:
323 for job in self.completed+self.dead:
324 del(alljobs[job.num])
324 del(alljobs[job.num])
325
325
326 # Now flush these lists completely
326 # Now flush these lists completely
327 fl_comp = self._group_flush(self.completed, 'Completed')
327 fl_comp = self._group_flush(self.completed, 'Completed')
328 fl_dead = self._group_flush(self.dead, 'Dead')
328 fl_dead = self._group_flush(self.dead, 'Dead')
329 if not (fl_comp or fl_dead):
329 if not (fl_comp or fl_dead):
330 print('No jobs to flush.')
330 print('No jobs to flush.')
331
331
332 def result(self,num):
332 def result(self,num):
333 """result(N) -> return the result of job N."""
333 """result(N) -> return the result of job N."""
334 try:
334 try:
335 return self.all[num].result
335 return self.all[num].result
336 except KeyError:
336 except KeyError:
337 error('Job #%s not found' % num)
337 error('Job #%s not found' % num)
338
338
339 def _traceback(self, job):
339 def _traceback(self, job):
340 num = job if isinstance(job, int) else job.num
340 num = job if isinstance(job, int) else job.num
341 try:
341 try:
342 self.all[num].traceback()
342 self.all[num].traceback()
343 except KeyError:
343 except KeyError:
344 error('Job #%s not found' % num)
344 error('Job #%s not found' % num)
345
345
346 def traceback(self, job=None):
346 def traceback(self, job=None):
347 if job is None:
347 if job is None:
348 self._update_status()
348 self._update_status()
349 for deadjob in self.dead:
349 for deadjob in self.dead:
350 print("Traceback for: %r" % deadjob)
350 print("Traceback for: %r" % deadjob)
351 self._traceback(deadjob)
351 self._traceback(deadjob)
352 print()
352 print()
353 else:
353 else:
354 self._traceback(job)
354 self._traceback(job)
355
355
356
356
357 class BackgroundJobBase(threading.Thread):
357 class BackgroundJobBase(threading.Thread):
358 """Base class to build BackgroundJob classes.
358 """Base class to build BackgroundJob classes.
359
359
360 The derived classes must implement:
360 The derived classes must implement:
361
361
362 - Their own __init__, since the one here raises NotImplementedError. The
362 - Their own __init__, since the one here raises NotImplementedError. The
363 derived constructor must call self._init() at the end, to provide common
363 derived constructor must call self._init() at the end, to provide common
364 initialization.
364 initialization.
365
365
366 - A strform attribute used in calls to __str__.
366 - A strform attribute used in calls to __str__.
367
367
368 - A call() method, which will make the actual execution call and must
368 - A call() method, which will make the actual execution call and must
369 return a value to be held in the 'result' field of the job object.
369 return a value to be held in the 'result' field of the job object.
370 """
370 """
371
371
372 # Class constants for status, in string and as numerical codes (when
372 # Class constants for status, in string and as numerical codes (when
373 # updating jobs lists, we don't want to do string comparisons). This will
373 # updating jobs lists, we don't want to do string comparisons). This will
374 # be done at every user prompt, so it has to be as fast as possible
374 # be done at every user prompt, so it has to be as fast as possible
375 stat_created = 'Created'; stat_created_c = 0
375 stat_created = 'Created'; stat_created_c = 0
376 stat_running = 'Running'; stat_running_c = 1
376 stat_running = 'Running'; stat_running_c = 1
377 stat_completed = 'Completed'; stat_completed_c = 2
377 stat_completed = 'Completed'; stat_completed_c = 2
378 stat_dead = 'Dead (Exception), call jobs.traceback() for details'
378 stat_dead = 'Dead (Exception), call jobs.traceback() for details'
379 stat_dead_c = -1
379 stat_dead_c = -1
380
380
381 def __init__(self):
381 def __init__(self):
382 """Must be implemented in subclasses.
382 """Must be implemented in subclasses.
383
383
384 Subclasses must call :meth:`_init` for standard initialisation.
384 Subclasses must call :meth:`_init` for standard initialisation.
385 """
385 """
386 raise NotImplementedError("This class can not be instantiated directly.")
386 raise NotImplementedError("This class can not be instantiated directly.")
387
387
388 def _init(self):
388 def _init(self):
389 """Common initialization for all BackgroundJob objects"""
389 """Common initialization for all BackgroundJob objects"""
390
390
391 for attr in ['call','strform']:
391 for attr in ['call','strform']:
392 assert hasattr(self,attr), "Missing attribute <%s>" % attr
392 assert hasattr(self,attr), "Missing attribute <%s>" % attr
393
393
394 # The num tag can be set by an external job manager
394 # The num tag can be set by an external job manager
395 self.num = None
395 self.num = None
396
396
397 self.status = BackgroundJobBase.stat_created
397 self.status = BackgroundJobBase.stat_created
398 self.stat_code = BackgroundJobBase.stat_created_c
398 self.stat_code = BackgroundJobBase.stat_created_c
399 self.finished = False
399 self.finished = False
400 self.result = '<BackgroundJob has not completed>'
400 self.result = '<BackgroundJob has not completed>'
401
401
402 # reuse the ipython traceback handler if we can get to it, otherwise
402 # reuse the ipython traceback handler if we can get to it, otherwise
403 # make a new one
403 # make a new one
404 try:
404 try:
405 make_tb = get_ipython().InteractiveTB.text
405 make_tb = get_ipython().InteractiveTB.text
406 except:
406 except:
407 make_tb = AutoFormattedTB(mode = 'Context',
407 make_tb = AutoFormattedTB(mode = 'Context',
408 color_scheme='NoColor',
408 color_scheme='NoColor',
409 tb_offset = 1).text
409 tb_offset = 1).text
410 # Note that the actual API for text() requires the three args to be
410 # Note that the actual API for text() requires the three args to be
411 # passed in, so we wrap it in a simple lambda.
411 # passed in, so we wrap it in a simple lambda.
412 self._make_tb = lambda : make_tb(None, None, None)
412 self._make_tb = lambda : make_tb(None, None, None)
413
413
414 # Hold a formatted traceback if one is generated.
414 # Hold a formatted traceback if one is generated.
415 self._tb = None
415 self._tb = None
416
416
417 threading.Thread.__init__(self)
417 threading.Thread.__init__(self)
418
418
419 def __str__(self):
419 def __str__(self):
420 return self.strform
420 return self.strform
421
421
422 def __repr__(self):
422 def __repr__(self):
423 return '<BackgroundJob #%d: %s>' % (self.num, self.strform)
423 return '<BackgroundJob #%d: %s>' % (self.num, self.strform)
424
424
425 def traceback(self):
425 def traceback(self):
426 print(self._tb)
426 print(self._tb)
427
427
428 def run(self):
428 def run(self):
429 try:
429 try:
430 self.status = BackgroundJobBase.stat_running
430 self.status = BackgroundJobBase.stat_running
431 self.stat_code = BackgroundJobBase.stat_running_c
431 self.stat_code = BackgroundJobBase.stat_running_c
432 self.result = self.call()
432 self.result = self.call()
433 except:
433 except:
434 self.status = BackgroundJobBase.stat_dead
434 self.status = BackgroundJobBase.stat_dead
435 self.stat_code = BackgroundJobBase.stat_dead_c
435 self.stat_code = BackgroundJobBase.stat_dead_c
436 self.finished = None
436 self.finished = None
437 self.result = ('<BackgroundJob died, call jobs.traceback() for details>')
437 self.result = ('<BackgroundJob died, call jobs.traceback() for details>')
438 self._tb = self._make_tb()
438 self._tb = self._make_tb()
439 else:
439 else:
440 self.status = BackgroundJobBase.stat_completed
440 self.status = BackgroundJobBase.stat_completed
441 self.stat_code = BackgroundJobBase.stat_completed_c
441 self.stat_code = BackgroundJobBase.stat_completed_c
442 self.finished = True
442 self.finished = True
443
443
444
444
445 class BackgroundJobExpr(BackgroundJobBase):
445 class BackgroundJobExpr(BackgroundJobBase):
446 """Evaluate an expression as a background job (uses a separate thread)."""
446 """Evaluate an expression as a background job (uses a separate thread)."""
447
447
448 def __init__(self, expression, glob=None, loc=None):
448 def __init__(self, expression, glob=None, loc=None):
449 """Create a new job from a string which can be fed to eval().
449 """Create a new job from a string which can be fed to eval().
450
450
451 global/locals dicts can be provided, which will be passed to the eval
451 global/locals dicts can be provided, which will be passed to the eval
452 call."""
452 call."""
453
453
454 # fail immediately if the given expression can't be compiled
454 # fail immediately if the given expression can't be compiled
455 self.code = compile(expression,'<BackgroundJob compilation>','eval')
455 self.code = compile(expression,'<BackgroundJob compilation>','eval')
456
456
457 glob = {} if glob is None else glob
457 glob = {} if glob is None else glob
458 loc = {} if loc is None else loc
458 loc = {} if loc is None else loc
459 self.expression = self.strform = expression
459 self.expression = self.strform = expression
460 self.glob = glob
460 self.glob = glob
461 self.loc = loc
461 self.loc = loc
462 self._init()
462 self._init()
463
463
464 def call(self):
464 def call(self):
465 return eval(self.code,self.glob,self.loc)
465 return eval(self.code,self.glob,self.loc)
466
466
467
467
468 class BackgroundJobFunc(BackgroundJobBase):
468 class BackgroundJobFunc(BackgroundJobBase):
469 """Run a function call as a background job (uses a separate thread)."""
469 """Run a function call as a background job (uses a separate thread)."""
470
470
471 def __init__(self, func, *args, **kwargs):
471 def __init__(self, func, *args, **kwargs):
472 """Create a new job from a callable object.
472 """Create a new job from a callable object.
473
473
474 Any positional arguments and keyword args given to this constructor
474 Any positional arguments and keyword args given to this constructor
475 after the initial callable are passed directly to it."""
475 after the initial callable are passed directly to it."""
476
476
477 if not callable(func):
477 if not callable(func):
478 raise TypeError(
478 raise TypeError(
479 'first argument to BackgroundJobFunc must be callable')
479 'first argument to BackgroundJobFunc must be callable')
480
480
481 self.func = func
481 self.func = func
482 self.args = args
482 self.args = args
483 self.kwargs = kwargs
483 self.kwargs = kwargs
484 # The string form will only include the function passed, because
484 # The string form will only include the function passed, because
485 # generating string representations of the arguments is a potentially
485 # generating string representations of the arguments is a potentially
486 # _very_ expensive operation (e.g. with large arrays).
486 # _very_ expensive operation (e.g. with large arrays).
487 self.strform = str(func)
487 self.strform = str(func)
488 self._init()
488 self._init()
489
489
490 def call(self):
490 def call(self):
491 return self.func(*self.args, **self.kwargs)
491 return self.func(*self.args, **self.kwargs)
@@ -1,672 +1,672 b''
1 """Module for interactive demos using IPython.
1 """Module for interactive demos using IPython.
2
2
3 This module implements a few classes for running Python scripts interactively
3 This module implements a few classes for running Python scripts interactively
4 in IPython for demonstrations. With very simple markup (a few tags in
4 in IPython for demonstrations. With very simple markup (a few tags in
5 comments), you can control points where the script stops executing and returns
5 comments), you can control points where the script stops executing and returns
6 control to IPython.
6 control to IPython.
7
7
8
8
9 Provided classes
9 Provided classes
10 ----------------
10 ----------------
11
11
12 The classes are (see their docstrings for further details):
12 The classes are (see their docstrings for further details):
13
13
14 - Demo: pure python demos
14 - Demo: pure python demos
15
15
16 - IPythonDemo: demos with input to be processed by IPython as if it had been
16 - IPythonDemo: demos with input to be processed by IPython as if it had been
17 typed interactively (so magics work, as well as any other special syntax you
17 typed interactively (so magics work, as well as any other special syntax you
18 may have added via input prefilters).
18 may have added via input prefilters).
19
19
20 - LineDemo: single-line version of the Demo class. These demos are executed
20 - LineDemo: single-line version of the Demo class. These demos are executed
21 one line at a time, and require no markup.
21 one line at a time, and require no markup.
22
22
23 - IPythonLineDemo: IPython version of the LineDemo class (the demo is
23 - IPythonLineDemo: IPython version of the LineDemo class (the demo is
24 executed a line at a time, but processed via IPython).
24 executed a line at a time, but processed via IPython).
25
25
26 - ClearMixin: mixin to make Demo classes with less visual clutter. It
26 - ClearMixin: mixin to make Demo classes with less visual clutter. It
27 declares an empty marquee and a pre_cmd that clears the screen before each
27 declares an empty marquee and a pre_cmd that clears the screen before each
28 block (see Subclassing below).
28 block (see Subclassing below).
29
29
30 - ClearDemo, ClearIPDemo: mixin-enabled versions of the Demo and IPythonDemo
30 - ClearDemo, ClearIPDemo: mixin-enabled versions of the Demo and IPythonDemo
31 classes.
31 classes.
32
32
33 Inheritance diagram:
33 Inheritance diagram:
34
34
35 .. inheritance-diagram:: IPython.lib.demo
35 .. inheritance-diagram:: IPython.lib.demo
36 :parts: 3
36 :parts: 3
37
37
38 Subclassing
38 Subclassing
39 -----------
39 -----------
40
40
41 The classes here all include a few methods meant to make customization by
41 The classes here all include a few methods meant to make customization by
42 subclassing more convenient. Their docstrings below have some more details:
42 subclassing more convenient. Their docstrings below have some more details:
43
43
44 - highlight(): format every block and optionally highlight comments and
44 - highlight(): format every block and optionally highlight comments and
45 docstring content.
45 docstring content.
46
46
47 - marquee(): generates a marquee to provide visible on-screen markers at each
47 - marquee(): generates a marquee to provide visible on-screen markers at each
48 block start and end.
48 block start and end.
49
49
50 - pre_cmd(): run right before the execution of each block.
50 - pre_cmd(): run right before the execution of each block.
51
51
52 - post_cmd(): run right after the execution of each block. If the block
52 - post_cmd(): run right after the execution of each block. If the block
53 raises an exception, this is NOT called.
53 raises an exception, this is NOT called.
54
54
55
55
56 Operation
56 Operation
57 ---------
57 ---------
58
58
59 The file is run in its own empty namespace (though you can pass it a string of
59 The file is run in its own empty namespace (though you can pass it a string of
60 arguments as if in a command line environment, and it will see those as
60 arguments as if in a command line environment, and it will see those as
61 sys.argv). But at each stop, the global IPython namespace is updated with the
61 sys.argv). But at each stop, the global IPython namespace is updated with the
62 current internal demo namespace, so you can work interactively with the data
62 current internal demo namespace, so you can work interactively with the data
63 accumulated so far.
63 accumulated so far.
64
64
65 By default, each block of code is printed (with syntax highlighting) before
65 By default, each block of code is printed (with syntax highlighting) before
66 executing it and you have to confirm execution. This is intended to show the
66 executing it and you have to confirm execution. This is intended to show the
67 code to an audience first so you can discuss it, and only proceed with
67 code to an audience first so you can discuss it, and only proceed with
68 execution once you agree. There are a few tags which allow you to modify this
68 execution once you agree. There are a few tags which allow you to modify this
69 behavior.
69 behavior.
70
70
71 The supported tags are:
71 The supported tags are:
72
72
73 # <demo> stop
73 # <demo> stop
74
74
75 Defines block boundaries, the points where IPython stops execution of the
75 Defines block boundaries, the points where IPython stops execution of the
76 file and returns to the interactive prompt.
76 file and returns to the interactive prompt.
77
77
78 You can optionally mark the stop tag with extra dashes before and after the
78 You can optionally mark the stop tag with extra dashes before and after the
79 word 'stop', to help visually distinguish the blocks in a text editor:
79 word 'stop', to help visually distinguish the blocks in a text editor:
80
80
81 # <demo> --- stop ---
81 # <demo> --- stop ---
82
82
83
83
84 # <demo> silent
84 # <demo> silent
85
85
86 Make a block execute silently (and hence automatically). Typically used in
86 Make a block execute silently (and hence automatically). Typically used in
87 cases where you have some boilerplate or initialization code which you need
87 cases where you have some boilerplate or initialization code which you need
88 executed but do not want to be seen in the demo.
88 executed but do not want to be seen in the demo.
89
89
90 # <demo> auto
90 # <demo> auto
91
91
92 Make a block execute automatically, but still being printed. Useful for
92 Make a block execute automatically, but still being printed. Useful for
93 simple code which does not warrant discussion, since it avoids the extra
93 simple code which does not warrant discussion, since it avoids the extra
94 manual confirmation.
94 manual confirmation.
95
95
96 # <demo> auto_all
96 # <demo> auto_all
97
97
98 This tag can _only_ be in the first block, and if given it overrides the
98 This tag can _only_ be in the first block, and if given it overrides the
99 individual auto tags to make the whole demo fully automatic (no block asks
99 individual auto tags to make the whole demo fully automatic (no block asks
100 for confirmation). It can also be given at creation time (or the attribute
100 for confirmation). It can also be given at creation time (or the attribute
101 set later) to override what's in the file.
101 set later) to override what's in the file.
102
102
103 While _any_ python file can be run as a Demo instance, if there are no stop
103 While _any_ python file can be run as a Demo instance, if there are no stop
104 tags the whole file will run in a single block (no different that calling
104 tags the whole file will run in a single block (no different that calling
105 first %pycat and then %run). The minimal markup to make this useful is to
105 first %pycat and then %run). The minimal markup to make this useful is to
106 place a set of stop tags; the other tags are only there to let you fine-tune
106 place a set of stop tags; the other tags are only there to let you fine-tune
107 the execution.
107 the execution.
108
108
109 This is probably best explained with the simple example file below. You can
109 This is probably best explained with the simple example file below. You can
110 copy this into a file named ex_demo.py, and try running it via::
110 copy this into a file named ex_demo.py, and try running it via::
111
111
112 from IPython.lib.demo import Demo
112 from IPython.lib.demo import Demo
113 d = Demo('ex_demo.py')
113 d = Demo('ex_demo.py')
114 d()
114 d()
115
115
116 Each time you call the demo object, it runs the next block. The demo object
116 Each time you call the demo object, it runs the next block. The demo object
117 has a few useful methods for navigation, like again(), edit(), jump(), seek()
117 has a few useful methods for navigation, like again(), edit(), jump(), seek()
118 and back(). It can be reset for a new run via reset() or reloaded from disk
118 and back(). It can be reset for a new run via reset() or reloaded from disk
119 (in case you've edited the source) via reload(). See their docstrings below.
119 (in case you've edited the source) via reload(). See their docstrings below.
120
120
121 Note: To make this simpler to explore, a file called "demo-exercizer.py" has
121 Note: To make this simpler to explore, a file called "demo-exercizer.py" has
122 been added to the "docs/examples/core" directory. Just cd to this directory in
122 been added to the "docs/examples/core" directory. Just cd to this directory in
123 an IPython session, and type::
123 an IPython session, and type::
124
124
125 %run demo-exercizer.py
125 %run demo-exercizer.py
126
126
127 and then follow the directions.
127 and then follow the directions.
128
128
129 Example
129 Example
130 -------
130 -------
131
131
132 The following is a very simple example of a valid demo file.
132 The following is a very simple example of a valid demo file.
133
133
134 ::
134 ::
135
135
136 #################### EXAMPLE DEMO <ex_demo.py> ###############################
136 #################### EXAMPLE DEMO <ex_demo.py> ###############################
137 '''A simple interactive demo to illustrate the use of IPython's Demo class.'''
137 '''A simple interactive demo to illustrate the use of IPython's Demo class.'''
138
138
139 print 'Hello, welcome to an interactive IPython demo.'
139 print 'Hello, welcome to an interactive IPython demo.'
140
140
141 # The mark below defines a block boundary, which is a point where IPython will
141 # The mark below defines a block boundary, which is a point where IPython will
142 # stop execution and return to the interactive prompt. The dashes are actually
142 # stop execution and return to the interactive prompt. The dashes are actually
143 # optional and used only as a visual aid to clearly separate blocks while
143 # optional and used only as a visual aid to clearly separate blocks while
144 # editing the demo code.
144 # editing the demo code.
145 # <demo> stop
145 # <demo> stop
146
146
147 x = 1
147 x = 1
148 y = 2
148 y = 2
149
149
150 # <demo> stop
150 # <demo> stop
151
151
152 # the mark below makes this block as silent
152 # the mark below makes this block as silent
153 # <demo> silent
153 # <demo> silent
154
154
155 print 'This is a silent block, which gets executed but not printed.'
155 print 'This is a silent block, which gets executed but not printed.'
156
156
157 # <demo> stop
157 # <demo> stop
158 # <demo> auto
158 # <demo> auto
159 print 'This is an automatic block.'
159 print 'This is an automatic block.'
160 print 'It is executed without asking for confirmation, but printed.'
160 print 'It is executed without asking for confirmation, but printed.'
161 z = x+y
161 z = x+y
162
162
163 print 'z=',x
163 print 'z=',x
164
164
165 # <demo> stop
165 # <demo> stop
166 # This is just another normal block.
166 # This is just another normal block.
167 print 'z is now:', z
167 print 'z is now:', z
168
168
169 print 'bye!'
169 print 'bye!'
170 ################### END EXAMPLE DEMO <ex_demo.py> ############################
170 ################### END EXAMPLE DEMO <ex_demo.py> ############################
171 """
171 """
172
172
173
173
174 #*****************************************************************************
174 #*****************************************************************************
175 # Copyright (C) 2005-2006 Fernando Perez. <Fernando.Perez@colorado.edu>
175 # Copyright (C) 2005-2006 Fernando Perez. <Fernando.Perez@colorado.edu>
176 #
176 #
177 # Distributed under the terms of the BSD License. The full license is in
177 # Distributed under the terms of the BSD License. The full license is in
178 # the file COPYING, distributed as part of this software.
178 # the file COPYING, distributed as part of this software.
179 #
179 #
180 #*****************************************************************************
180 #*****************************************************************************
181
181
182 import os
182 import os
183 import re
183 import re
184 import shlex
184 import shlex
185 import sys
185 import sys
186 import pygments
186 import pygments
187 from pathlib import Path
187 from pathlib import Path
188
188
189 from IPython.utils.text import marquee
189 from IPython.utils.text import marquee
190 from IPython.utils import openpy
190 from IPython.utils import openpy
191 from IPython.utils import py3compat
191 from IPython.utils import py3compat
192 __all__ = ['Demo','IPythonDemo','LineDemo','IPythonLineDemo','DemoError']
192 __all__ = ['Demo','IPythonDemo','LineDemo','IPythonLineDemo','DemoError']
193
193
194 class DemoError(Exception): pass
194 class DemoError(Exception): pass
195
195
196 def re_mark(mark):
196 def re_mark(mark):
197 return re.compile(r'^\s*#\s+<demo>\s+%s\s*$' % mark,re.MULTILINE)
197 return re.compile(r'^\s*#\s+<demo>\s+%s\s*$' % mark,re.MULTILINE)
198
198
199 class Demo(object):
199 class Demo(object):
200
200
201 re_stop = re_mark(r'-*\s?stop\s?-*')
201 re_stop = re_mark(r'-*\s?stop\s?-*')
202 re_silent = re_mark('silent')
202 re_silent = re_mark('silent')
203 re_auto = re_mark('auto')
203 re_auto = re_mark('auto')
204 re_auto_all = re_mark('auto_all')
204 re_auto_all = re_mark('auto_all')
205
205
206 def __init__(self,src,title='',arg_str='',auto_all=None, format_rst=False,
206 def __init__(self,src,title='',arg_str='',auto_all=None, format_rst=False,
207 formatter='terminal', style='default'):
207 formatter='terminal', style='default'):
208 """Make a new demo object. To run the demo, simply call the object.
208 """Make a new demo object. To run the demo, simply call the object.
209
209
210 See the module docstring for full details and an example (you can use
210 See the module docstring for full details and an example (you can use
211 IPython.Demo? in IPython to see it).
211 IPython.Demo? in IPython to see it).
212
212
213 Inputs:
213 Inputs:
214
214
215 - src is either a file, or file-like object, or a
215 - src is either a file, or file-like object, or a
216 string that can be resolved to a filename.
216 string that can be resolved to a filename.
217
217
218 Optional inputs:
218 Optional inputs:
219
219
220 - title: a string to use as the demo name. Of most use when the demo
220 - title: a string to use as the demo name. Of most use when the demo
221 you are making comes from an object that has no filename, or if you
221 you are making comes from an object that has no filename, or if you
222 want an alternate denotation distinct from the filename.
222 want an alternate denotation distinct from the filename.
223
223
224 - arg_str(''): a string of arguments, internally converted to a list
224 - arg_str(''): a string of arguments, internally converted to a list
225 just like sys.argv, so the demo script can see a similar
225 just like sys.argv, so the demo script can see a similar
226 environment.
226 environment.
227
227
228 - auto_all(None): global flag to run all blocks automatically without
228 - auto_all(None): global flag to run all blocks automatically without
229 confirmation. This attribute overrides the block-level tags and
229 confirmation. This attribute overrides the block-level tags and
230 applies to the whole demo. It is an attribute of the object, and
230 applies to the whole demo. It is an attribute of the object, and
231 can be changed at runtime simply by reassigning it to a boolean
231 can be changed at runtime simply by reassigning it to a boolean
232 value.
232 value.
233
233
234 - format_rst(False): a bool to enable comments and doc strings
234 - format_rst(False): a bool to enable comments and doc strings
235 formatting with pygments rst lexer
235 formatting with pygments rst lexer
236
236
237 - formatter('terminal'): a string of pygments formatter name to be
237 - formatter('terminal'): a string of pygments formatter name to be
238 used. Useful values for terminals: terminal, terminal256,
238 used. Useful values for terminals: terminal, terminal256,
239 terminal16m
239 terminal16m
240
240
241 - style('default'): a string of pygments style name to be used.
241 - style('default'): a string of pygments style name to be used.
242 """
242 """
243 if hasattr(src, "read"):
243 if hasattr(src, "read"):
244 # It seems to be a file or a file-like object
244 # It seems to be a file or a file-like object
245 self.fname = "from a file-like object"
245 self.fname = "from a file-like object"
246 if title == '':
246 if title == '':
247 self.title = "from a file-like object"
247 self.title = "from a file-like object"
248 else:
248 else:
249 self.title = title
249 self.title = title
250 else:
250 else:
251 # Assume it's a string or something that can be converted to one
251 # Assume it's a string or something that can be converted to one
252 self.fname = src
252 self.fname = src
253 if title == '':
253 if title == '':
254 (filepath, filename) = os.path.split(src)
254 (filepath, filename) = os.path.split(src)
255 self.title = filename
255 self.title = filename
256 else:
256 else:
257 self.title = title
257 self.title = title
258 self.sys_argv = [src] + shlex.split(arg_str)
258 self.sys_argv = [src] + shlex.split(arg_str)
259 self.auto_all = auto_all
259 self.auto_all = auto_all
260 self.src = src
260 self.src = src
261
261
262 try:
262 try:
263 ip = get_ipython() # this is in builtins whenever IPython is running
263 ip = get_ipython() # this is in builtins whenever IPython is running
264 self.inside_ipython = True
264 self.inside_ipython = True
265 except NameError:
265 except NameError:
266 self.inside_ipython = False
266 self.inside_ipython = False
267
267
268 if self.inside_ipython:
268 if self.inside_ipython:
269 # get a few things from ipython. While it's a bit ugly design-wise,
269 # get a few things from ipython. While it's a bit ugly design-wise,
270 # it ensures that things like color scheme and the like are always in
270 # it ensures that things like color scheme and the like are always in
271 # sync with the ipython mode being used. This class is only meant to
271 # sync with the ipython mode being used. This class is only meant to
272 # be used inside ipython anyways, so it's OK.
272 # be used inside ipython anyways, so it's OK.
273 self.ip_ns = ip.user_ns
273 self.ip_ns = ip.user_ns
274 self.ip_colorize = ip.pycolorize
274 self.ip_colorize = ip.pycolorize
275 self.ip_showtb = ip.showtraceback
275 self.ip_showtb = ip.showtraceback
276 self.ip_run_cell = ip.run_cell
276 self.ip_run_cell = ip.run_cell
277 self.shell = ip
277 self.shell = ip
278
278
279 self.formatter = pygments.formatters.get_formatter_by_name(formatter,
279 self.formatter = pygments.formatters.get_formatter_by_name(formatter,
280 style=style)
280 style=style)
281 self.python_lexer = pygments.lexers.get_lexer_by_name("py3")
281 self.python_lexer = pygments.lexers.get_lexer_by_name("py3")
282 self.format_rst = format_rst
282 self.format_rst = format_rst
283 if format_rst:
283 if format_rst:
284 self.rst_lexer = pygments.lexers.get_lexer_by_name("rst")
284 self.rst_lexer = pygments.lexers.get_lexer_by_name("rst")
285
285
286 # load user data and initialize data structures
286 # load user data and initialize data structures
287 self.reload()
287 self.reload()
288
288
289 def fload(self):
289 def fload(self):
290 """Load file object."""
290 """Load file object."""
291 # read data and parse into blocks
291 # read data and parse into blocks
292 if hasattr(self, 'fobj') and self.fobj is not None:
292 if hasattr(self, 'fobj') and self.fobj is not None:
293 self.fobj.close()
293 self.fobj.close()
294 if hasattr(self.src, "read"):
294 if hasattr(self.src, "read"):
295 # It seems to be a file or a file-like object
295 # It seems to be a file or a file-like object
296 self.fobj = self.src
296 self.fobj = self.src
297 else:
297 else:
298 # Assume it's a string or something that can be converted to one
298 # Assume it's a string or something that can be converted to one
299 self.fobj = openpy.open(self.fname)
299 self.fobj = openpy.open(self.fname)
300
300
301 def reload(self):
301 def reload(self):
302 """Reload source from disk and initialize state."""
302 """Reload source from disk and initialize state."""
303 self.fload()
303 self.fload()
304
304
305 self.src = "".join(openpy.strip_encoding_cookie(self.fobj))
305 self.src = "".join(openpy.strip_encoding_cookie(self.fobj))
306 src_b = [b.strip() for b in self.re_stop.split(self.src) if b]
306 src_b = [b.strip() for b in self.re_stop.split(self.src) if b]
307 self._silent = [bool(self.re_silent.findall(b)) for b in src_b]
307 self._silent = [bool(self.re_silent.findall(b)) for b in src_b]
308 self._auto = [bool(self.re_auto.findall(b)) for b in src_b]
308 self._auto = [bool(self.re_auto.findall(b)) for b in src_b]
309
309
310 # if auto_all is not given (def. None), we read it from the file
310 # if auto_all is not given (def. None), we read it from the file
311 if self.auto_all is None:
311 if self.auto_all is None:
312 self.auto_all = bool(self.re_auto_all.findall(src_b[0]))
312 self.auto_all = bool(self.re_auto_all.findall(src_b[0]))
313 else:
313 else:
314 self.auto_all = bool(self.auto_all)
314 self.auto_all = bool(self.auto_all)
315
315
316 # Clean the sources from all markup so it doesn't get displayed when
316 # Clean the sources from all markup so it doesn't get displayed when
317 # running the demo
317 # running the demo
318 src_blocks = []
318 src_blocks = []
319 auto_strip = lambda s: self.re_auto.sub('',s)
319 auto_strip = lambda s: self.re_auto.sub('',s)
320 for i,b in enumerate(src_b):
320 for i,b in enumerate(src_b):
321 if self._auto[i]:
321 if self._auto[i]:
322 src_blocks.append(auto_strip(b))
322 src_blocks.append(auto_strip(b))
323 else:
323 else:
324 src_blocks.append(b)
324 src_blocks.append(b)
325 # remove the auto_all marker
325 # remove the auto_all marker
326 src_blocks[0] = self.re_auto_all.sub('',src_blocks[0])
326 src_blocks[0] = self.re_auto_all.sub('',src_blocks[0])
327
327
328 self.nblocks = len(src_blocks)
328 self.nblocks = len(src_blocks)
329 self.src_blocks = src_blocks
329 self.src_blocks = src_blocks
330
330
331 # also build syntax-highlighted source
331 # also build syntax-highlighted source
332 self.src_blocks_colored = list(map(self.highlight,self.src_blocks))
332 self.src_blocks_colored = list(map(self.highlight,self.src_blocks))
333
333
334 # ensure clean namespace and seek offset
334 # ensure clean namespace and seek offset
335 self.reset()
335 self.reset()
336
336
337 def reset(self):
337 def reset(self):
338 """Reset the namespace and seek pointer to restart the demo"""
338 """Reset the namespace and seek pointer to restart the demo"""
339 self.user_ns = {}
339 self.user_ns = {}
340 self.finished = False
340 self.finished = False
341 self.block_index = 0
341 self.block_index = 0
342
342
343 def _validate_index(self,index):
343 def _validate_index(self,index):
344 if index<0 or index>=self.nblocks:
344 if index<0 or index>=self.nblocks:
345 raise ValueError('invalid block index %s' % index)
345 raise ValueError('invalid block index %s' % index)
346
346
347 def _get_index(self,index):
347 def _get_index(self,index):
348 """Get the current block index, validating and checking status.
348 """Get the current block index, validating and checking status.
349
349
350 Returns None if the demo is finished"""
350 Returns None if the demo is finished"""
351
351
352 if index is None:
352 if index is None:
353 if self.finished:
353 if self.finished:
354 print('Demo finished. Use <demo_name>.reset() if you want to rerun it.')
354 print('Demo finished. Use <demo_name>.reset() if you want to rerun it.')
355 return None
355 return None
356 index = self.block_index
356 index = self.block_index
357 else:
357 else:
358 self._validate_index(index)
358 self._validate_index(index)
359 return index
359 return index
360
360
361 def seek(self,index):
361 def seek(self,index):
362 """Move the current seek pointer to the given block.
362 """Move the current seek pointer to the given block.
363
363
364 You can use negative indices to seek from the end, with identical
364 You can use negative indices to seek from the end, with identical
365 semantics to those of Python lists."""
365 semantics to those of Python lists."""
366 if index<0:
366 if index<0:
367 index = self.nblocks + index
367 index = self.nblocks + index
368 self._validate_index(index)
368 self._validate_index(index)
369 self.block_index = index
369 self.block_index = index
370 self.finished = False
370 self.finished = False
371
371
372 def back(self,num=1):
372 def back(self,num=1):
373 """Move the seek pointer back num blocks (default is 1)."""
373 """Move the seek pointer back num blocks (default is 1)."""
374 self.seek(self.block_index-num)
374 self.seek(self.block_index-num)
375
375
376 def jump(self,num=1):
376 def jump(self,num=1):
377 """Jump a given number of blocks relative to the current one.
377 """Jump a given number of blocks relative to the current one.
378
378
379 The offset can be positive or negative, defaults to 1."""
379 The offset can be positive or negative, defaults to 1."""
380 self.seek(self.block_index+num)
380 self.seek(self.block_index+num)
381
381
382 def again(self):
382 def again(self):
383 """Move the seek pointer back one block and re-execute."""
383 """Move the seek pointer back one block and re-execute."""
384 self.back(1)
384 self.back(1)
385 self()
385 self()
386
386
387 def edit(self,index=None):
387 def edit(self,index=None):
388 """Edit a block.
388 """Edit a block.
389
389
390 If no number is given, use the last block executed.
390 If no number is given, use the last block executed.
391
391
392 This edits the in-memory copy of the demo, it does NOT modify the
392 This edits the in-memory copy of the demo, it does NOT modify the
393 original source file. If you want to do that, simply open the file in
393 original source file. If you want to do that, simply open the file in
394 an editor and use reload() when you make changes to the file. This
394 an editor and use reload() when you make changes to the file. This
395 method is meant to let you change a block during a demonstration for
395 method is meant to let you change a block during a demonstration for
396 explanatory purposes, without damaging your original script."""
396 explanatory purposes, without damaging your original script."""
397
397
398 index = self._get_index(index)
398 index = self._get_index(index)
399 if index is None:
399 if index is None:
400 return
400 return
401 # decrease the index by one (unless we're at the very beginning), so
401 # decrease the index by one (unless we're at the very beginning), so
402 # that the default demo.edit() call opens up the sblock we've last run
402 # that the default demo.edit() call opens up the sblock we've last run
403 if index>0:
403 if index>0:
404 index -= 1
404 index -= 1
405
405
406 filename = self.shell.mktempfile(self.src_blocks[index])
406 filename = self.shell.mktempfile(self.src_blocks[index])
407 self.shell.hooks.editor(filename, 1)
407 self.shell.hooks.editor(filename, 1)
408 with open(Path(filename), "r") as f:
408 with open(Path(filename), "r") as f:
409 new_block = f.read()
409 new_block = f.read()
410 # update the source and colored block
410 # update the source and colored block
411 self.src_blocks[index] = new_block
411 self.src_blocks[index] = new_block
412 self.src_blocks_colored[index] = self.highlight(new_block)
412 self.src_blocks_colored[index] = self.highlight(new_block)
413 self.block_index = index
413 self.block_index = index
414 # call to run with the newly edited index
414 # call to run with the newly edited index
415 self()
415 self()
416
416
417 def show(self,index=None):
417 def show(self,index=None):
418 """Show a single block on screen"""
418 """Show a single block on screen"""
419
419
420 index = self._get_index(index)
420 index = self._get_index(index)
421 if index is None:
421 if index is None:
422 return
422 return
423
423
424 print(self.marquee('<%s> block # %s (%s remaining)' %
424 print(self.marquee('<%s> block # %s (%s remaining)' %
425 (self.title,index,self.nblocks-index-1)))
425 (self.title,index,self.nblocks-index-1)))
426 print(self.src_blocks_colored[index])
426 print(self.src_blocks_colored[index])
427 sys.stdout.flush()
427 sys.stdout.flush()
428
428
429 def show_all(self):
429 def show_all(self):
430 """Show entire demo on screen, block by block"""
430 """Show entire demo on screen, block by block"""
431
431
432 fname = self.title
432 fname = self.title
433 title = self.title
433 title = self.title
434 nblocks = self.nblocks
434 nblocks = self.nblocks
435 silent = self._silent
435 silent = self._silent
436 marquee = self.marquee
436 marquee = self.marquee
437 for index,block in enumerate(self.src_blocks_colored):
437 for index,block in enumerate(self.src_blocks_colored):
438 if silent[index]:
438 if silent[index]:
439 print(marquee('<%s> SILENT block # %s (%s remaining)' %
439 print(marquee('<%s> SILENT block # %s (%s remaining)' %
440 (title,index,nblocks-index-1)))
440 (title,index,nblocks-index-1)))
441 else:
441 else:
442 print(marquee('<%s> block # %s (%s remaining)' %
442 print(marquee('<%s> block # %s (%s remaining)' %
443 (title,index,nblocks-index-1)))
443 (title,index,nblocks-index-1)))
444 print(block, end=' ')
444 print(block, end=' ')
445 sys.stdout.flush()
445 sys.stdout.flush()
446
446
447 def run_cell(self,source):
447 def run_cell(self,source):
448 """Execute a string with one or more lines of code"""
448 """Execute a string with one or more lines of code"""
449
449
450 exec(source, self.user_ns)
450 exec(source, self.user_ns)
451
451
452 def __call__(self,index=None):
452 def __call__(self,index=None):
453 """run a block of the demo.
453 """run a block of the demo.
454
454
455 If index is given, it should be an integer >=1 and <= nblocks. This
455 If index is given, it should be an integer >=1 and <= nblocks. This
456 means that the calling convention is one off from typical Python
456 means that the calling convention is one off from typical Python
457 lists. The reason for the inconsistency is that the demo always
457 lists. The reason for the inconsistency is that the demo always
458 prints 'Block n/N, and N is the total, so it would be very odd to use
458 prints 'Block n/N, and N is the total, so it would be very odd to use
459 zero-indexing here."""
459 zero-indexing here."""
460
460
461 index = self._get_index(index)
461 index = self._get_index(index)
462 if index is None:
462 if index is None:
463 return
463 return
464 try:
464 try:
465 marquee = self.marquee
465 marquee = self.marquee
466 next_block = self.src_blocks[index]
466 next_block = self.src_blocks[index]
467 self.block_index += 1
467 self.block_index += 1
468 if self._silent[index]:
468 if self._silent[index]:
469 print(marquee('Executing silent block # %s (%s remaining)' %
469 print(marquee('Executing silent block # %s (%s remaining)' %
470 (index,self.nblocks-index-1)))
470 (index,self.nblocks-index-1)))
471 else:
471 else:
472 self.pre_cmd()
472 self.pre_cmd()
473 self.show(index)
473 self.show(index)
474 if self.auto_all or self._auto[index]:
474 if self.auto_all or self._auto[index]:
475 print(marquee('output:'))
475 print(marquee('output:'))
476 else:
476 else:
477 print(marquee('Press <q> to quit, <Enter> to execute...'), end=' ')
477 print(marquee('Press <q> to quit, <Enter> to execute...'), end=' ')
478 ans = py3compat.input().strip()
478 ans = py3compat.input().strip()
479 if ans:
479 if ans:
480 print(marquee('Block NOT executed'))
480 print(marquee('Block NOT executed'))
481 return
481 return
482 try:
482 try:
483 save_argv = sys.argv
483 save_argv = sys.argv
484 sys.argv = self.sys_argv
484 sys.argv = self.sys_argv
485 self.run_cell(next_block)
485 self.run_cell(next_block)
486 self.post_cmd()
486 self.post_cmd()
487 finally:
487 finally:
488 sys.argv = save_argv
488 sys.argv = save_argv
489
489
490 except:
490 except:
491 if self.inside_ipython:
491 if self.inside_ipython:
492 self.ip_showtb(filename=self.fname)
492 self.ip_showtb(filename=self.fname)
493 else:
493 else:
494 if self.inside_ipython:
494 if self.inside_ipython:
495 self.ip_ns.update(self.user_ns)
495 self.ip_ns.update(self.user_ns)
496
496
497 if self.block_index == self.nblocks:
497 if self.block_index == self.nblocks:
498 mq1 = self.marquee('END OF DEMO')
498 mq1 = self.marquee('END OF DEMO')
499 if mq1:
499 if mq1:
500 # avoid spurious print if empty marquees are used
500 # avoid spurious print if empty marquees are used
501 print()
501 print()
502 print(mq1)
502 print(mq1)
503 print(self.marquee('Use <demo_name>.reset() if you want to rerun it.'))
503 print(self.marquee('Use <demo_name>.reset() if you want to rerun it.'))
504 self.finished = True
504 self.finished = True
505
505
506 # These methods are meant to be overridden by subclasses who may wish to
506 # These methods are meant to be overridden by subclasses who may wish to
507 # customize the behavior of of their demos.
507 # customize the behavior of of their demos.
508 def marquee(self,txt='',width=78,mark='*'):
508 def marquee(self,txt='',width=78,mark='*'):
509 """Return the input string centered in a 'marquee'."""
509 """Return the input string centered in a 'marquee'."""
510 return marquee(txt,width,mark)
510 return marquee(txt,width,mark)
511
511
512 def pre_cmd(self):
512 def pre_cmd(self):
513 """Method called before executing each block."""
513 """Method called before executing each block."""
514 pass
514 pass
515
515
516 def post_cmd(self):
516 def post_cmd(self):
517 """Method called after executing each block."""
517 """Method called after executing each block."""
518 pass
518 pass
519
519
520 def highlight(self, block):
520 def highlight(self, block):
521 """Method called on each block to highlight it content"""
521 """Method called on each block to highlight it content"""
522 tokens = pygments.lex(block, self.python_lexer)
522 tokens = pygments.lex(block, self.python_lexer)
523 if self.format_rst:
523 if self.format_rst:
524 from pygments.token import Token
524 from pygments.token import Token
525 toks = []
525 toks = []
526 for token in tokens:
526 for token in tokens:
527 if token[0] == Token.String.Doc and len(token[1]) > 6:
527 if token[0] == Token.String.Doc and len(token[1]) > 6:
528 toks += pygments.lex(token[1][:3], self.python_lexer)
528 toks += pygments.lex(token[1][:3], self.python_lexer)
529 # parse doc string content by rst lexer
529 # parse doc string content by rst lexer
530 toks += pygments.lex(token[1][3:-3], self.rst_lexer)
530 toks += pygments.lex(token[1][3:-3], self.rst_lexer)
531 toks += pygments.lex(token[1][-3:], self.python_lexer)
531 toks += pygments.lex(token[1][-3:], self.python_lexer)
532 elif token[0] == Token.Comment.Single:
532 elif token[0] == Token.Comment.Single:
533 toks.append((Token.Comment.Single, token[1][0]))
533 toks.append((Token.Comment.Single, token[1][0]))
534 # parse comment content by rst lexer
534 # parse comment content by rst lexer
535 # remove the extrat newline added by rst lexer
535 # remove the extra newline added by rst lexer
536 toks += list(pygments.lex(token[1][1:], self.rst_lexer))[:-1]
536 toks += list(pygments.lex(token[1][1:], self.rst_lexer))[:-1]
537 else:
537 else:
538 toks.append(token)
538 toks.append(token)
539 tokens = toks
539 tokens = toks
540 return pygments.format(tokens, self.formatter)
540 return pygments.format(tokens, self.formatter)
541
541
542
542
543 class IPythonDemo(Demo):
543 class IPythonDemo(Demo):
544 """Class for interactive demos with IPython's input processing applied.
544 """Class for interactive demos with IPython's input processing applied.
545
545
546 This subclasses Demo, but instead of executing each block by the Python
546 This subclasses Demo, but instead of executing each block by the Python
547 interpreter (via exec), it actually calls IPython on it, so that any input
547 interpreter (via exec), it actually calls IPython on it, so that any input
548 filters which may be in place are applied to the input block.
548 filters which may be in place are applied to the input block.
549
549
550 If you have an interactive environment which exposes special input
550 If you have an interactive environment which exposes special input
551 processing, you can use this class instead to write demo scripts which
551 processing, you can use this class instead to write demo scripts which
552 operate exactly as if you had typed them interactively. The default Demo
552 operate exactly as if you had typed them interactively. The default Demo
553 class requires the input to be valid, pure Python code.
553 class requires the input to be valid, pure Python code.
554 """
554 """
555
555
556 def run_cell(self,source):
556 def run_cell(self,source):
557 """Execute a string with one or more lines of code"""
557 """Execute a string with one or more lines of code"""
558
558
559 self.shell.run_cell(source)
559 self.shell.run_cell(source)
560
560
561 class LineDemo(Demo):
561 class LineDemo(Demo):
562 """Demo where each line is executed as a separate block.
562 """Demo where each line is executed as a separate block.
563
563
564 The input script should be valid Python code.
564 The input script should be valid Python code.
565
565
566 This class doesn't require any markup at all, and it's meant for simple
566 This class doesn't require any markup at all, and it's meant for simple
567 scripts (with no nesting or any kind of indentation) which consist of
567 scripts (with no nesting or any kind of indentation) which consist of
568 multiple lines of input to be executed, one at a time, as if they had been
568 multiple lines of input to be executed, one at a time, as if they had been
569 typed in the interactive prompt.
569 typed in the interactive prompt.
570
570
571 Note: the input can not have *any* indentation, which means that only
571 Note: the input can not have *any* indentation, which means that only
572 single-lines of input are accepted, not even function definitions are
572 single-lines of input are accepted, not even function definitions are
573 valid."""
573 valid."""
574
574
575 def reload(self):
575 def reload(self):
576 """Reload source from disk and initialize state."""
576 """Reload source from disk and initialize state."""
577 # read data and parse into blocks
577 # read data and parse into blocks
578 self.fload()
578 self.fload()
579 lines = self.fobj.readlines()
579 lines = self.fobj.readlines()
580 src_b = [l for l in lines if l.strip()]
580 src_b = [l for l in lines if l.strip()]
581 nblocks = len(src_b)
581 nblocks = len(src_b)
582 self.src = ''.join(lines)
582 self.src = ''.join(lines)
583 self._silent = [False]*nblocks
583 self._silent = [False]*nblocks
584 self._auto = [True]*nblocks
584 self._auto = [True]*nblocks
585 self.auto_all = True
585 self.auto_all = True
586 self.nblocks = nblocks
586 self.nblocks = nblocks
587 self.src_blocks = src_b
587 self.src_blocks = src_b
588
588
589 # also build syntax-highlighted source
589 # also build syntax-highlighted source
590 self.src_blocks_colored = list(map(self.highlight,self.src_blocks))
590 self.src_blocks_colored = list(map(self.highlight,self.src_blocks))
591
591
592 # ensure clean namespace and seek offset
592 # ensure clean namespace and seek offset
593 self.reset()
593 self.reset()
594
594
595
595
596 class IPythonLineDemo(IPythonDemo,LineDemo):
596 class IPythonLineDemo(IPythonDemo,LineDemo):
597 """Variant of the LineDemo class whose input is processed by IPython."""
597 """Variant of the LineDemo class whose input is processed by IPython."""
598 pass
598 pass
599
599
600
600
601 class ClearMixin(object):
601 class ClearMixin(object):
602 """Use this mixin to make Demo classes with less visual clutter.
602 """Use this mixin to make Demo classes with less visual clutter.
603
603
604 Demos using this mixin will clear the screen before every block and use
604 Demos using this mixin will clear the screen before every block and use
605 blank marquees.
605 blank marquees.
606
606
607 Note that in order for the methods defined here to actually override those
607 Note that in order for the methods defined here to actually override those
608 of the classes it's mixed with, it must go /first/ in the inheritance
608 of the classes it's mixed with, it must go /first/ in the inheritance
609 tree. For example:
609 tree. For example:
610
610
611 class ClearIPDemo(ClearMixin,IPythonDemo): pass
611 class ClearIPDemo(ClearMixin,IPythonDemo): pass
612
612
613 will provide an IPythonDemo class with the mixin's features.
613 will provide an IPythonDemo class with the mixin's features.
614 """
614 """
615
615
616 def marquee(self,txt='',width=78,mark='*'):
616 def marquee(self,txt='',width=78,mark='*'):
617 """Blank marquee that returns '' no matter what the input."""
617 """Blank marquee that returns '' no matter what the input."""
618 return ''
618 return ''
619
619
620 def pre_cmd(self):
620 def pre_cmd(self):
621 """Method called before executing each block.
621 """Method called before executing each block.
622
622
623 This one simply clears the screen."""
623 This one simply clears the screen."""
624 from IPython.utils.terminal import _term_clear
624 from IPython.utils.terminal import _term_clear
625 _term_clear()
625 _term_clear()
626
626
627 class ClearDemo(ClearMixin,Demo):
627 class ClearDemo(ClearMixin,Demo):
628 pass
628 pass
629
629
630
630
631 class ClearIPDemo(ClearMixin,IPythonDemo):
631 class ClearIPDemo(ClearMixin,IPythonDemo):
632 pass
632 pass
633
633
634
634
635 def slide(file_path, noclear=False, format_rst=True, formatter="terminal",
635 def slide(file_path, noclear=False, format_rst=True, formatter="terminal",
636 style="native", auto_all=False, delimiter='...'):
636 style="native", auto_all=False, delimiter='...'):
637 if noclear:
637 if noclear:
638 demo_class = Demo
638 demo_class = Demo
639 else:
639 else:
640 demo_class = ClearDemo
640 demo_class = ClearDemo
641 demo = demo_class(file_path, format_rst=format_rst, formatter=formatter,
641 demo = demo_class(file_path, format_rst=format_rst, formatter=formatter,
642 style=style, auto_all=auto_all)
642 style=style, auto_all=auto_all)
643 while not demo.finished:
643 while not demo.finished:
644 demo()
644 demo()
645 try:
645 try:
646 py3compat.input('\n' + delimiter)
646 py3compat.input('\n' + delimiter)
647 except KeyboardInterrupt:
647 except KeyboardInterrupt:
648 exit(1)
648 exit(1)
649
649
650 if __name__ == '__main__':
650 if __name__ == '__main__':
651 import argparse
651 import argparse
652 parser = argparse.ArgumentParser(description='Run python demos')
652 parser = argparse.ArgumentParser(description='Run python demos')
653 parser.add_argument('--noclear', '-C', action='store_true',
653 parser.add_argument('--noclear', '-C', action='store_true',
654 help='Do not clear terminal on each slide')
654 help='Do not clear terminal on each slide')
655 parser.add_argument('--rst', '-r', action='store_true',
655 parser.add_argument('--rst', '-r', action='store_true',
656 help='Highlight comments and dostrings as rst')
656 help='Highlight comments and dostrings as rst')
657 parser.add_argument('--formatter', '-f', default='terminal',
657 parser.add_argument('--formatter', '-f', default='terminal',
658 help='pygments formatter name could be: terminal, '
658 help='pygments formatter name could be: terminal, '
659 'terminal256, terminal16m')
659 'terminal256, terminal16m')
660 parser.add_argument('--style', '-s', default='default',
660 parser.add_argument('--style', '-s', default='default',
661 help='pygments style name')
661 help='pygments style name')
662 parser.add_argument('--auto', '-a', action='store_true',
662 parser.add_argument('--auto', '-a', action='store_true',
663 help='Run all blocks automatically without'
663 help='Run all blocks automatically without'
664 'confirmation')
664 'confirmation')
665 parser.add_argument('--delimiter', '-d', default='...',
665 parser.add_argument('--delimiter', '-d', default='...',
666 help='slides delimiter added after each slide run')
666 help='slides delimiter added after each slide run')
667 parser.add_argument('file', nargs=1,
667 parser.add_argument('file', nargs=1,
668 help='python demo file')
668 help='python demo file')
669 args = parser.parse_args()
669 args = parser.parse_args()
670 slide(args.file[0], noclear=args.noclear, format_rst=args.rst,
670 slide(args.file[0], noclear=args.noclear, format_rst=args.rst,
671 formatter=args.formatter, style=args.style, auto_all=args.auto,
671 formatter=args.formatter, style=args.style, auto_all=args.auto,
672 delimiter=args.delimiter)
672 delimiter=args.delimiter)
@@ -1,399 +1,399 b''
1 # -*- coding: utf-8 -*-
1 # -*- coding: utf-8 -*-
2 """Decorators for labeling test objects.
2 """Decorators for labeling test objects.
3
3
4 Decorators that merely return a modified version of the original function
4 Decorators that merely return a modified version of the original function
5 object are straightforward. Decorators that return a new function object need
5 object are straightforward. Decorators that return a new function object need
6 to use nose.tools.make_decorator(original_function)(decorator) in returning the
6 to use nose.tools.make_decorator(original_function)(decorator) in returning the
7 decorator, in order to preserve metadata such as function name, setup and
7 decorator, in order to preserve metadata such as function name, setup and
8 teardown functions and so on - see nose.tools for more information.
8 teardown functions and so on - see nose.tools for more information.
9
9
10 This module provides a set of useful decorators meant to be ready to use in
10 This module provides a set of useful decorators meant to be ready to use in
11 your own tests. See the bottom of the file for the ready-made ones, and if you
11 your own tests. See the bottom of the file for the ready-made ones, and if you
12 find yourself writing a new one that may be of generic use, add it here.
12 find yourself writing a new one that may be of generic use, add it here.
13
13
14 Included decorators:
14 Included decorators:
15
15
16
16
17 Lightweight testing that remains unittest-compatible.
17 Lightweight testing that remains unittest-compatible.
18
18
19 - An @as_unittest decorator can be used to tag any normal parameter-less
19 - An @as_unittest decorator can be used to tag any normal parameter-less
20 function as a unittest TestCase. Then, both nose and normal unittest will
20 function as a unittest TestCase. Then, both nose and normal unittest will
21 recognize it as such. This will make it easier to migrate away from Nose if
21 recognize it as such. This will make it easier to migrate away from Nose if
22 we ever need/want to while maintaining very lightweight tests.
22 we ever need/want to while maintaining very lightweight tests.
23
23
24 NOTE: This file contains IPython-specific decorators. Using the machinery in
24 NOTE: This file contains IPython-specific decorators. Using the machinery in
25 IPython.external.decorators, we import either numpy.testing.decorators if numpy is
25 IPython.external.decorators, we import either numpy.testing.decorators if numpy is
26 available, OR use equivalent code in IPython.external._decorators, which
26 available, OR use equivalent code in IPython.external._decorators, which
27 we've copied verbatim from numpy.
27 we've copied verbatim from numpy.
28
28
29 """
29 """
30
30
31 # Copyright (c) IPython Development Team.
31 # Copyright (c) IPython Development Team.
32 # Distributed under the terms of the Modified BSD License.
32 # Distributed under the terms of the Modified BSD License.
33
33
34 import os
34 import os
35 import shutil
35 import shutil
36 import sys
36 import sys
37 import tempfile
37 import tempfile
38 import unittest
38 import unittest
39 import warnings
39 import warnings
40 from importlib import import_module
40 from importlib import import_module
41
41
42 from decorator import decorator
42 from decorator import decorator
43
43
44 # Expose the unittest-driven decorators
44 # Expose the unittest-driven decorators
45 from .ipunittest import ipdoctest, ipdocstring
45 from .ipunittest import ipdoctest, ipdocstring
46
46
47 # Grab the numpy-specific decorators which we keep in a file that we
47 # Grab the numpy-specific decorators which we keep in a file that we
48 # occasionally update from upstream: decorators.py is a copy of
48 # occasionally update from upstream: decorators.py is a copy of
49 # numpy.testing.decorators, we expose all of it here.
49 # numpy.testing.decorators, we expose all of it here.
50 from IPython.external.decorators import knownfailureif
50 from IPython.external.decorators import knownfailureif
51
51
52 #-----------------------------------------------------------------------------
52 #-----------------------------------------------------------------------------
53 # Classes and functions
53 # Classes and functions
54 #-----------------------------------------------------------------------------
54 #-----------------------------------------------------------------------------
55
55
56 # Simple example of the basic idea
56 # Simple example of the basic idea
57 def as_unittest(func):
57 def as_unittest(func):
58 """Decorator to make a simple function into a normal test via unittest."""
58 """Decorator to make a simple function into a normal test via unittest."""
59 class Tester(unittest.TestCase):
59 class Tester(unittest.TestCase):
60 def test(self):
60 def test(self):
61 func()
61 func()
62
62
63 Tester.__name__ = func.__name__
63 Tester.__name__ = func.__name__
64
64
65 return Tester
65 return Tester
66
66
67 # Utility functions
67 # Utility functions
68
68
69 def apply_wrapper(wrapper, func):
69 def apply_wrapper(wrapper, func):
70 """Apply a wrapper to a function for decoration.
70 """Apply a wrapper to a function for decoration.
71
71
72 This mixes Michele Simionato's decorator tool with nose's make_decorator,
72 This mixes Michele Simionato's decorator tool with nose's make_decorator,
73 to apply a wrapper in a decorator so that all nose attributes, as well as
73 to apply a wrapper in a decorator so that all nose attributes, as well as
74 function signature and other properties, survive the decoration cleanly.
74 function signature and other properties, survive the decoration cleanly.
75 This will ensure that wrapped functions can still be well introspected via
75 This will ensure that wrapped functions can still be well introspected via
76 IPython, for example.
76 IPython, for example.
77 """
77 """
78 warnings.warn("The function `apply_wrapper` is deprecated since IPython 4.0",
78 warnings.warn("The function `apply_wrapper` is deprecated since IPython 4.0",
79 DeprecationWarning, stacklevel=2)
79 DeprecationWarning, stacklevel=2)
80 import nose.tools
80 import nose.tools
81
81
82 return decorator(wrapper,nose.tools.make_decorator(func)(wrapper))
82 return decorator(wrapper,nose.tools.make_decorator(func)(wrapper))
83
83
84
84
85 def make_label_dec(label, ds=None):
85 def make_label_dec(label, ds=None):
86 """Factory function to create a decorator that applies one or more labels.
86 """Factory function to create a decorator that applies one or more labels.
87
87
88 Parameters
88 Parameters
89 ----------
89 ----------
90 label : string or sequence
90 label : string or sequence
91 One or more labels that will be applied by the decorator to the functions
91 One or more labels that will be applied by the decorator to the functions
92 it decorates. Labels are attributes of the decorated function with their
92 it decorates. Labels are attributes of the decorated function with their
93 value set to True.
93 value set to True.
94
94
95 ds : string
95 ds : string
96 An optional docstring for the resulting decorator. If not given, a
96 An optional docstring for the resulting decorator. If not given, a
97 default docstring is auto-generated.
97 default docstring is auto-generated.
98
98
99 Returns
99 Returns
100 -------
100 -------
101 A decorator.
101 A decorator.
102
102
103 Examples
103 Examples
104 --------
104 --------
105
105
106 A simple labeling decorator:
106 A simple labeling decorator:
107
107
108 >>> slow = make_label_dec('slow')
108 >>> slow = make_label_dec('slow')
109 >>> slow.__doc__
109 >>> slow.__doc__
110 "Labels a test as 'slow'."
110 "Labels a test as 'slow'."
111
111
112 And one that uses multiple labels and a custom docstring:
112 And one that uses multiple labels and a custom docstring:
113
113
114 >>> rare = make_label_dec(['slow','hard'],
114 >>> rare = make_label_dec(['slow','hard'],
115 ... "Mix labels 'slow' and 'hard' for rare tests.")
115 ... "Mix labels 'slow' and 'hard' for rare tests.")
116 >>> rare.__doc__
116 >>> rare.__doc__
117 "Mix labels 'slow' and 'hard' for rare tests."
117 "Mix labels 'slow' and 'hard' for rare tests."
118
118
119 Now, let's test using this one:
119 Now, let's test using this one:
120 >>> @rare
120 >>> @rare
121 ... def f(): pass
121 ... def f(): pass
122 ...
122 ...
123 >>>
123 >>>
124 >>> f.slow
124 >>> f.slow
125 True
125 True
126 >>> f.hard
126 >>> f.hard
127 True
127 True
128 """
128 """
129
129
130 warnings.warn("The function `make_label_dec` is deprecated since IPython 4.0",
130 warnings.warn("The function `make_label_dec` is deprecated since IPython 4.0",
131 DeprecationWarning, stacklevel=2)
131 DeprecationWarning, stacklevel=2)
132 if isinstance(label, str):
132 if isinstance(label, str):
133 labels = [label]
133 labels = [label]
134 else:
134 else:
135 labels = label
135 labels = label
136
136
137 # Validate that the given label(s) are OK for use in setattr() by doing a
137 # Validate that the given label(s) are OK for use in setattr() by doing a
138 # dry run on a dummy function.
138 # dry run on a dummy function.
139 tmp = lambda : None
139 tmp = lambda : None
140 for label in labels:
140 for label in labels:
141 setattr(tmp,label,True)
141 setattr(tmp,label,True)
142
142
143 # This is the actual decorator we'll return
143 # This is the actual decorator we'll return
144 def decor(f):
144 def decor(f):
145 for label in labels:
145 for label in labels:
146 setattr(f,label,True)
146 setattr(f,label,True)
147 return f
147 return f
148
148
149 # Apply the user's docstring, or autogenerate a basic one
149 # Apply the user's docstring, or autogenerate a basic one
150 if ds is None:
150 if ds is None:
151 ds = "Labels a test as %r." % label
151 ds = "Labels a test as %r." % label
152 decor.__doc__ = ds
152 decor.__doc__ = ds
153
153
154 return decor
154 return decor
155
155
156
156
157 def skip_iptest_but_not_pytest(f):
157 def skip_iptest_but_not_pytest(f):
158 """
158 """
159 Warnign this will make the test invisible to iptest.
159 Warning this will make the test invisible to iptest.
160 """
160 """
161 import os
161 import os
162
162
163 if os.environ.get("IPTEST_WORKING_DIR", None) is not None:
163 if os.environ.get("IPTEST_WORKING_DIR", None) is not None:
164 f.__test__ = False
164 f.__test__ = False
165 return f
165 return f
166
166
167
167
168 # Inspired by numpy's skipif, but uses the full apply_wrapper utility to
168 # Inspired by numpy's skipif, but uses the full apply_wrapper utility to
169 # preserve function metadata better and allows the skip condition to be a
169 # preserve function metadata better and allows the skip condition to be a
170 # callable.
170 # callable.
171 def skipif(skip_condition, msg=None):
171 def skipif(skip_condition, msg=None):
172 ''' Make function raise SkipTest exception if skip_condition is true
172 ''' Make function raise SkipTest exception if skip_condition is true
173
173
174 Parameters
174 Parameters
175 ----------
175 ----------
176
176
177 skip_condition : bool or callable
177 skip_condition : bool or callable
178 Flag to determine whether to skip test. If the condition is a
178 Flag to determine whether to skip test. If the condition is a
179 callable, it is used at runtime to dynamically make the decision. This
179 callable, it is used at runtime to dynamically make the decision. This
180 is useful for tests that may require costly imports, to delay the cost
180 is useful for tests that may require costly imports, to delay the cost
181 until the test suite is actually executed.
181 until the test suite is actually executed.
182 msg : string
182 msg : string
183 Message to give on raising a SkipTest exception.
183 Message to give on raising a SkipTest exception.
184
184
185 Returns
185 Returns
186 -------
186 -------
187 decorator : function
187 decorator : function
188 Decorator, which, when applied to a function, causes SkipTest
188 Decorator, which, when applied to a function, causes SkipTest
189 to be raised when the skip_condition was True, and the function
189 to be raised when the skip_condition was True, and the function
190 to be called normally otherwise.
190 to be called normally otherwise.
191
191
192 Notes
192 Notes
193 -----
193 -----
194 You will see from the code that we had to further decorate the
194 You will see from the code that we had to further decorate the
195 decorator with the nose.tools.make_decorator function in order to
195 decorator with the nose.tools.make_decorator function in order to
196 transmit function name, and various other metadata.
196 transmit function name, and various other metadata.
197 '''
197 '''
198
198
199 def skip_decorator(f):
199 def skip_decorator(f):
200 # Local import to avoid a hard nose dependency and only incur the
200 # Local import to avoid a hard nose dependency and only incur the
201 # import time overhead at actual test-time.
201 # import time overhead at actual test-time.
202 import nose
202 import nose
203
203
204 # Allow for both boolean or callable skip conditions.
204 # Allow for both boolean or callable skip conditions.
205 if callable(skip_condition):
205 if callable(skip_condition):
206 skip_val = skip_condition
206 skip_val = skip_condition
207 else:
207 else:
208 skip_val = lambda : skip_condition
208 skip_val = lambda : skip_condition
209
209
210 def get_msg(func,msg=None):
210 def get_msg(func,msg=None):
211 """Skip message with information about function being skipped."""
211 """Skip message with information about function being skipped."""
212 if msg is None: out = 'Test skipped due to test condition.'
212 if msg is None: out = 'Test skipped due to test condition.'
213 else: out = msg
213 else: out = msg
214 return "Skipping test: %s. %s" % (func.__name__,out)
214 return "Skipping test: %s. %s" % (func.__name__,out)
215
215
216 # We need to define *two* skippers because Python doesn't allow both
216 # We need to define *two* skippers because Python doesn't allow both
217 # return with value and yield inside the same function.
217 # return with value and yield inside the same function.
218 def skipper_func(*args, **kwargs):
218 def skipper_func(*args, **kwargs):
219 """Skipper for normal test functions."""
219 """Skipper for normal test functions."""
220 if skip_val():
220 if skip_val():
221 raise nose.SkipTest(get_msg(f,msg))
221 raise nose.SkipTest(get_msg(f,msg))
222 else:
222 else:
223 return f(*args, **kwargs)
223 return f(*args, **kwargs)
224
224
225 def skipper_gen(*args, **kwargs):
225 def skipper_gen(*args, **kwargs):
226 """Skipper for test generators."""
226 """Skipper for test generators."""
227 if skip_val():
227 if skip_val():
228 raise nose.SkipTest(get_msg(f,msg))
228 raise nose.SkipTest(get_msg(f,msg))
229 else:
229 else:
230 for x in f(*args, **kwargs):
230 for x in f(*args, **kwargs):
231 yield x
231 yield x
232
232
233 # Choose the right skipper to use when building the actual generator.
233 # Choose the right skipper to use when building the actual generator.
234 if nose.util.isgenerator(f):
234 if nose.util.isgenerator(f):
235 skipper = skipper_gen
235 skipper = skipper_gen
236 else:
236 else:
237 skipper = skipper_func
237 skipper = skipper_func
238
238
239 return nose.tools.make_decorator(f)(skipper)
239 return nose.tools.make_decorator(f)(skipper)
240
240
241 return skip_decorator
241 return skip_decorator
242
242
243 # A version with the condition set to true, common case just to attach a message
243 # A version with the condition set to true, common case just to attach a message
244 # to a skip decorator
244 # to a skip decorator
245 def skip(msg=None):
245 def skip(msg=None):
246 """Decorator factory - mark a test function for skipping from test suite.
246 """Decorator factory - mark a test function for skipping from test suite.
247
247
248 Parameters
248 Parameters
249 ----------
249 ----------
250 msg : string
250 msg : string
251 Optional message to be added.
251 Optional message to be added.
252
252
253 Returns
253 Returns
254 -------
254 -------
255 decorator : function
255 decorator : function
256 Decorator, which, when applied to a function, causes SkipTest
256 Decorator, which, when applied to a function, causes SkipTest
257 to be raised, with the optional message added.
257 to be raised, with the optional message added.
258 """
258 """
259 if msg and not isinstance(msg, str):
259 if msg and not isinstance(msg, str):
260 raise ValueError('invalid object passed to `@skip` decorator, did you '
260 raise ValueError('invalid object passed to `@skip` decorator, did you '
261 'meant `@skip()` with brackets ?')
261 'meant `@skip()` with brackets ?')
262 return skipif(True, msg)
262 return skipif(True, msg)
263
263
264
264
265 def onlyif(condition, msg):
265 def onlyif(condition, msg):
266 """The reverse from skipif, see skipif for details."""
266 """The reverse from skipif, see skipif for details."""
267
267
268 if callable(condition):
268 if callable(condition):
269 skip_condition = lambda : not condition()
269 skip_condition = lambda : not condition()
270 else:
270 else:
271 skip_condition = lambda : not condition
271 skip_condition = lambda : not condition
272
272
273 return skipif(skip_condition, msg)
273 return skipif(skip_condition, msg)
274
274
275 #-----------------------------------------------------------------------------
275 #-----------------------------------------------------------------------------
276 # Utility functions for decorators
276 # Utility functions for decorators
277 def module_not_available(module):
277 def module_not_available(module):
278 """Can module be imported? Returns true if module does NOT import.
278 """Can module be imported? Returns true if module does NOT import.
279
279
280 This is used to make a decorator to skip tests that require module to be
280 This is used to make a decorator to skip tests that require module to be
281 available, but delay the 'import numpy' to test execution time.
281 available, but delay the 'import numpy' to test execution time.
282 """
282 """
283 try:
283 try:
284 mod = import_module(module)
284 mod = import_module(module)
285 mod_not_avail = False
285 mod_not_avail = False
286 except ImportError:
286 except ImportError:
287 mod_not_avail = True
287 mod_not_avail = True
288
288
289 return mod_not_avail
289 return mod_not_avail
290
290
291
291
292 def decorated_dummy(dec, name):
292 def decorated_dummy(dec, name):
293 """Return a dummy function decorated with dec, with the given name.
293 """Return a dummy function decorated with dec, with the given name.
294
294
295 Examples
295 Examples
296 --------
296 --------
297 import IPython.testing.decorators as dec
297 import IPython.testing.decorators as dec
298 setup = dec.decorated_dummy(dec.skip_if_no_x11, __name__)
298 setup = dec.decorated_dummy(dec.skip_if_no_x11, __name__)
299 """
299 """
300 warnings.warn("The function `decorated_dummy` is deprecated since IPython 4.0",
300 warnings.warn("The function `decorated_dummy` is deprecated since IPython 4.0",
301 DeprecationWarning, stacklevel=2)
301 DeprecationWarning, stacklevel=2)
302 dummy = lambda: None
302 dummy = lambda: None
303 dummy.__name__ = name
303 dummy.__name__ = name
304 return dec(dummy)
304 return dec(dummy)
305
305
306 #-----------------------------------------------------------------------------
306 #-----------------------------------------------------------------------------
307 # Decorators for public use
307 # Decorators for public use
308
308
309 # Decorators to skip certain tests on specific platforms.
309 # Decorators to skip certain tests on specific platforms.
310 skip_win32 = skipif(sys.platform == 'win32',
310 skip_win32 = skipif(sys.platform == 'win32',
311 "This test does not run under Windows")
311 "This test does not run under Windows")
312 skip_linux = skipif(sys.platform.startswith('linux'),
312 skip_linux = skipif(sys.platform.startswith('linux'),
313 "This test does not run under Linux")
313 "This test does not run under Linux")
314 skip_osx = skipif(sys.platform == 'darwin',"This test does not run under OS X")
314 skip_osx = skipif(sys.platform == 'darwin',"This test does not run under OS X")
315
315
316
316
317 # Decorators to skip tests if not on specific platforms.
317 # Decorators to skip tests if not on specific platforms.
318 skip_if_not_win32 = skipif(sys.platform != 'win32',
318 skip_if_not_win32 = skipif(sys.platform != 'win32',
319 "This test only runs under Windows")
319 "This test only runs under Windows")
320 skip_if_not_linux = skipif(not sys.platform.startswith('linux'),
320 skip_if_not_linux = skipif(not sys.platform.startswith('linux'),
321 "This test only runs under Linux")
321 "This test only runs under Linux")
322 skip_if_not_osx = skipif(sys.platform != 'darwin',
322 skip_if_not_osx = skipif(sys.platform != 'darwin',
323 "This test only runs under OSX")
323 "This test only runs under OSX")
324
324
325
325
326 _x11_skip_cond = (sys.platform not in ('darwin', 'win32') and
326 _x11_skip_cond = (sys.platform not in ('darwin', 'win32') and
327 os.environ.get('DISPLAY', '') == '')
327 os.environ.get('DISPLAY', '') == '')
328 _x11_skip_msg = "Skipped under *nix when X11/XOrg not available"
328 _x11_skip_msg = "Skipped under *nix when X11/XOrg not available"
329
329
330 skip_if_no_x11 = skipif(_x11_skip_cond, _x11_skip_msg)
330 skip_if_no_x11 = skipif(_x11_skip_cond, _x11_skip_msg)
331
331
332
332
333 # Decorators to skip certain tests on specific platform/python combinations
333 # Decorators to skip certain tests on specific platform/python combinations
334 skip_win32_py38 = skipif(sys.version_info > (3,8) and os.name == 'nt')
334 skip_win32_py38 = skipif(sys.version_info > (3,8) and os.name == 'nt')
335
335
336
336
337 # not a decorator itself, returns a dummy function to be used as setup
337 # not a decorator itself, returns a dummy function to be used as setup
338 def skip_file_no_x11(name):
338 def skip_file_no_x11(name):
339 warnings.warn("The function `skip_file_no_x11` is deprecated since IPython 4.0",
339 warnings.warn("The function `skip_file_no_x11` is deprecated since IPython 4.0",
340 DeprecationWarning, stacklevel=2)
340 DeprecationWarning, stacklevel=2)
341 return decorated_dummy(skip_if_no_x11, name) if _x11_skip_cond else None
341 return decorated_dummy(skip_if_no_x11, name) if _x11_skip_cond else None
342
342
343 # Other skip decorators
343 # Other skip decorators
344
344
345 # generic skip without module
345 # generic skip without module
346 skip_without = lambda mod: skipif(module_not_available(mod), "This test requires %s" % mod)
346 skip_without = lambda mod: skipif(module_not_available(mod), "This test requires %s" % mod)
347
347
348 skipif_not_numpy = skip_without('numpy')
348 skipif_not_numpy = skip_without('numpy')
349
349
350 skipif_not_matplotlib = skip_without('matplotlib')
350 skipif_not_matplotlib = skip_without('matplotlib')
351
351
352 skipif_not_sympy = skip_without('sympy')
352 skipif_not_sympy = skip_without('sympy')
353
353
354 skip_known_failure = knownfailureif(True,'This test is known to fail')
354 skip_known_failure = knownfailureif(True,'This test is known to fail')
355
355
356 # A null 'decorator', useful to make more readable code that needs to pick
356 # A null 'decorator', useful to make more readable code that needs to pick
357 # between different decorators based on OS or other conditions
357 # between different decorators based on OS or other conditions
358 null_deco = lambda f: f
358 null_deco = lambda f: f
359
359
360 # Some tests only run where we can use unicode paths. Note that we can't just
360 # Some tests only run where we can use unicode paths. Note that we can't just
361 # check os.path.supports_unicode_filenames, which is always False on Linux.
361 # check os.path.supports_unicode_filenames, which is always False on Linux.
362 try:
362 try:
363 f = tempfile.NamedTemporaryFile(prefix=u"tmp€")
363 f = tempfile.NamedTemporaryFile(prefix=u"tmp€")
364 except UnicodeEncodeError:
364 except UnicodeEncodeError:
365 unicode_paths = False
365 unicode_paths = False
366 else:
366 else:
367 unicode_paths = True
367 unicode_paths = True
368 f.close()
368 f.close()
369
369
370 onlyif_unicode_paths = onlyif(unicode_paths, ("This test is only applicable "
370 onlyif_unicode_paths = onlyif(unicode_paths, ("This test is only applicable "
371 "where we can use unicode in filenames."))
371 "where we can use unicode in filenames."))
372
372
373
373
374 def onlyif_cmds_exist(*commands):
374 def onlyif_cmds_exist(*commands):
375 """
375 """
376 Decorator to skip test when at least one of `commands` is not found.
376 Decorator to skip test when at least one of `commands` is not found.
377 """
377 """
378 for cmd in commands:
378 for cmd in commands:
379 reason = "This test runs only if command '{cmd}' is installed"
379 reason = "This test runs only if command '{cmd}' is installed"
380 if not shutil.which(cmd):
380 if not shutil.which(cmd):
381 if os.environ.get("IPTEST_WORKING_DIR", None) is not None:
381 if os.environ.get("IPTEST_WORKING_DIR", None) is not None:
382 return skip(reason)
382 return skip(reason)
383 else:
383 else:
384 import pytest
384 import pytest
385
385
386 return pytest.mark.skip(reason=reason)
386 return pytest.mark.skip(reason=reason)
387 return null_deco
387 return null_deco
388
388
389 def onlyif_any_cmd_exists(*commands):
389 def onlyif_any_cmd_exists(*commands):
390 """
390 """
391 Decorator to skip test unless at least one of `commands` is found.
391 Decorator to skip test unless at least one of `commands` is found.
392 """
392 """
393 warnings.warn("The function `onlyif_any_cmd_exists` is deprecated since IPython 4.0",
393 warnings.warn("The function `onlyif_any_cmd_exists` is deprecated since IPython 4.0",
394 DeprecationWarning, stacklevel=2)
394 DeprecationWarning, stacklevel=2)
395 for cmd in commands:
395 for cmd in commands:
396 if shutil.which(cmd):
396 if shutil.which(cmd):
397 return null_deco
397 return null_deco
398 return skip("This test runs only if one of the commands {0} "
398 return skip("This test runs only if one of the commands {0} "
399 "is installed".format(commands))
399 "is installed".format(commands))
@@ -1,440 +1,440 b''
1 # encoding: utf-8
1 # encoding: utf-8
2 """
2 """
3 Utilities for path handling.
3 Utilities for path handling.
4 """
4 """
5
5
6 # Copyright (c) IPython Development Team.
6 # Copyright (c) IPython Development Team.
7 # Distributed under the terms of the Modified BSD License.
7 # Distributed under the terms of the Modified BSD License.
8
8
9 import os
9 import os
10 import sys
10 import sys
11 import errno
11 import errno
12 import shutil
12 import shutil
13 import random
13 import random
14 import glob
14 import glob
15 from warnings import warn
15 from warnings import warn
16
16
17 from IPython.utils.process import system
17 from IPython.utils.process import system
18 from IPython.utils.decorators import undoc
18 from IPython.utils.decorators import undoc
19
19
20 #-----------------------------------------------------------------------------
20 #-----------------------------------------------------------------------------
21 # Code
21 # Code
22 #-----------------------------------------------------------------------------
22 #-----------------------------------------------------------------------------
23 fs_encoding = sys.getfilesystemencoding()
23 fs_encoding = sys.getfilesystemencoding()
24
24
25 def _writable_dir(path):
25 def _writable_dir(path):
26 """Whether `path` is a directory, to which the user has write access."""
26 """Whether `path` is a directory, to which the user has write access."""
27 return os.path.isdir(path) and os.access(path, os.W_OK)
27 return os.path.isdir(path) and os.access(path, os.W_OK)
28
28
29 if sys.platform == 'win32':
29 if sys.platform == 'win32':
30 def _get_long_path_name(path):
30 def _get_long_path_name(path):
31 """Get a long path name (expand ~) on Windows using ctypes.
31 """Get a long path name (expand ~) on Windows using ctypes.
32
32
33 Examples
33 Examples
34 --------
34 --------
35
35
36 >>> get_long_path_name('c:\\docume~1')
36 >>> get_long_path_name('c:\\docume~1')
37 'c:\\\\Documents and Settings'
37 'c:\\\\Documents and Settings'
38
38
39 """
39 """
40 try:
40 try:
41 import ctypes
41 import ctypes
42 except ImportError as e:
42 except ImportError as e:
43 raise ImportError('you need to have ctypes installed for this to work') from e
43 raise ImportError('you need to have ctypes installed for this to work') from e
44 _GetLongPathName = ctypes.windll.kernel32.GetLongPathNameW
44 _GetLongPathName = ctypes.windll.kernel32.GetLongPathNameW
45 _GetLongPathName.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p,
45 _GetLongPathName.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p,
46 ctypes.c_uint ]
46 ctypes.c_uint ]
47
47
48 buf = ctypes.create_unicode_buffer(260)
48 buf = ctypes.create_unicode_buffer(260)
49 rv = _GetLongPathName(path, buf, 260)
49 rv = _GetLongPathName(path, buf, 260)
50 if rv == 0 or rv > 260:
50 if rv == 0 or rv > 260:
51 return path
51 return path
52 else:
52 else:
53 return buf.value
53 return buf.value
54 else:
54 else:
55 def _get_long_path_name(path):
55 def _get_long_path_name(path):
56 """Dummy no-op."""
56 """Dummy no-op."""
57 return path
57 return path
58
58
59
59
60
60
61 def get_long_path_name(path):
61 def get_long_path_name(path):
62 """Expand a path into its long form.
62 """Expand a path into its long form.
63
63
64 On Windows this expands any ~ in the paths. On other platforms, it is
64 On Windows this expands any ~ in the paths. On other platforms, it is
65 a null operation.
65 a null operation.
66 """
66 """
67 return _get_long_path_name(path)
67 return _get_long_path_name(path)
68
68
69
69
70 def unquote_filename(name, win32=(sys.platform=='win32')):
70 def unquote_filename(name, win32=(sys.platform=='win32')):
71 """ On Windows, remove leading and trailing quotes from filenames.
71 """ On Windows, remove leading and trailing quotes from filenames.
72
72
73 This function has been deprecated and should not be used any more:
73 This function has been deprecated and should not be used any more:
74 unquoting is now taken care of by :func:`IPython.utils.process.arg_split`.
74 unquoting is now taken care of by :func:`IPython.utils.process.arg_split`.
75 """
75 """
76 warn("'unquote_filename' is deprecated since IPython 5.0 and should not "
76 warn("'unquote_filename' is deprecated since IPython 5.0 and should not "
77 "be used anymore", DeprecationWarning, stacklevel=2)
77 "be used anymore", DeprecationWarning, stacklevel=2)
78 if win32:
78 if win32:
79 if name.startswith(("'", '"')) and name.endswith(("'", '"')):
79 if name.startswith(("'", '"')) and name.endswith(("'", '"')):
80 name = name[1:-1]
80 name = name[1:-1]
81 return name
81 return name
82
82
83
83
84 def compress_user(path):
84 def compress_user(path):
85 """Reverse of :func:`os.path.expanduser`
85 """Reverse of :func:`os.path.expanduser`
86 """
86 """
87 home = os.path.expanduser('~')
87 home = os.path.expanduser('~')
88 if path.startswith(home):
88 if path.startswith(home):
89 path = "~" + path[len(home):]
89 path = "~" + path[len(home):]
90 return path
90 return path
91
91
92 def get_py_filename(name, force_win32=None):
92 def get_py_filename(name, force_win32=None):
93 """Return a valid python filename in the current directory.
93 """Return a valid python filename in the current directory.
94
94
95 If the given name is not a file, it adds '.py' and searches again.
95 If the given name is not a file, it adds '.py' and searches again.
96 Raises IOError with an informative message if the file isn't found.
96 Raises IOError with an informative message if the file isn't found.
97 """
97 """
98
98
99 name = os.path.expanduser(name)
99 name = os.path.expanduser(name)
100 if force_win32 is not None:
100 if force_win32 is not None:
101 warn("The 'force_win32' argument to 'get_py_filename' is deprecated "
101 warn("The 'force_win32' argument to 'get_py_filename' is deprecated "
102 "since IPython 5.0 and should not be used anymore",
102 "since IPython 5.0 and should not be used anymore",
103 DeprecationWarning, stacklevel=2)
103 DeprecationWarning, stacklevel=2)
104 if not os.path.isfile(name) and not name.endswith('.py'):
104 if not os.path.isfile(name) and not name.endswith('.py'):
105 name += '.py'
105 name += '.py'
106 if os.path.isfile(name):
106 if os.path.isfile(name):
107 return name
107 return name
108 else:
108 else:
109 raise IOError('File `%r` not found.' % name)
109 raise IOError('File `%r` not found.' % name)
110
110
111
111
112 def filefind(filename: str, path_dirs=None) -> str:
112 def filefind(filename: str, path_dirs=None) -> str:
113 """Find a file by looking through a sequence of paths.
113 """Find a file by looking through a sequence of paths.
114
114
115 This iterates through a sequence of paths looking for a file and returns
115 This iterates through a sequence of paths looking for a file and returns
116 the full, absolute path of the first occurrence of the file. If no set of
116 the full, absolute path of the first occurrence of the file. If no set of
117 path dirs is given, the filename is tested as is, after running through
117 path dirs is given, the filename is tested as is, after running through
118 :func:`expandvars` and :func:`expanduser`. Thus a simple call::
118 :func:`expandvars` and :func:`expanduser`. Thus a simple call::
119
119
120 filefind('myfile.txt')
120 filefind('myfile.txt')
121
121
122 will find the file in the current working dir, but::
122 will find the file in the current working dir, but::
123
123
124 filefind('~/myfile.txt')
124 filefind('~/myfile.txt')
125
125
126 Will find the file in the users home directory. This function does not
126 Will find the file in the users home directory. This function does not
127 automatically try any paths, such as the cwd or the user's home directory.
127 automatically try any paths, such as the cwd or the user's home directory.
128
128
129 Parameters
129 Parameters
130 ----------
130 ----------
131 filename : str
131 filename : str
132 The filename to look for.
132 The filename to look for.
133 path_dirs : str, None or sequence of str
133 path_dirs : str, None or sequence of str
134 The sequence of paths to look for the file in. If None, the filename
134 The sequence of paths to look for the file in. If None, the filename
135 need to be absolute or be in the cwd. If a string, the string is
135 need to be absolute or be in the cwd. If a string, the string is
136 put into a sequence and the searched. If a sequence, walk through
136 put into a sequence and the searched. If a sequence, walk through
137 each element and join with ``filename``, calling :func:`expandvars`
137 each element and join with ``filename``, calling :func:`expandvars`
138 and :func:`expanduser` before testing for existence.
138 and :func:`expanduser` before testing for existence.
139
139
140 Returns
140 Returns
141 -------
141 -------
142 path : str
142 path : str
143 returns absolute path to file.
143 returns absolute path to file.
144
144
145 Raises
145 Raises
146 ------
146 ------
147 IOError
147 IOError
148 """
148 """
149
149
150 # If paths are quoted, abspath gets confused, strip them...
150 # If paths are quoted, abspath gets confused, strip them...
151 filename = filename.strip('"').strip("'")
151 filename = filename.strip('"').strip("'")
152 # If the input is an absolute path, just check it exists
152 # If the input is an absolute path, just check it exists
153 if os.path.isabs(filename) and os.path.isfile(filename):
153 if os.path.isabs(filename) and os.path.isfile(filename):
154 return filename
154 return filename
155
155
156 if path_dirs is None:
156 if path_dirs is None:
157 path_dirs = ("",)
157 path_dirs = ("",)
158 elif isinstance(path_dirs, str):
158 elif isinstance(path_dirs, str):
159 path_dirs = (path_dirs,)
159 path_dirs = (path_dirs,)
160
160
161 for path in path_dirs:
161 for path in path_dirs:
162 if path == '.': path = os.getcwd()
162 if path == '.': path = os.getcwd()
163 testname = expand_path(os.path.join(path, filename))
163 testname = expand_path(os.path.join(path, filename))
164 if os.path.isfile(testname):
164 if os.path.isfile(testname):
165 return os.path.abspath(testname)
165 return os.path.abspath(testname)
166
166
167 raise IOError("File %r does not exist in any of the search paths: %r" %
167 raise IOError("File %r does not exist in any of the search paths: %r" %
168 (filename, path_dirs) )
168 (filename, path_dirs) )
169
169
170
170
171 class HomeDirError(Exception):
171 class HomeDirError(Exception):
172 pass
172 pass
173
173
174
174
175 def get_home_dir(require_writable=False) -> str:
175 def get_home_dir(require_writable=False) -> str:
176 """Return the 'home' directory, as a unicode string.
176 """Return the 'home' directory, as a unicode string.
177
177
178 Uses os.path.expanduser('~'), and checks for writability.
178 Uses os.path.expanduser('~'), and checks for writability.
179
179
180 See stdlib docs for how this is determined.
180 See stdlib docs for how this is determined.
181 For Python <3.8, $HOME is first priority on *ALL* platforms.
181 For Python <3.8, $HOME is first priority on *ALL* platforms.
182 For Python >=3.8 on Windows, %HOME% is no longer considered.
182 For Python >=3.8 on Windows, %HOME% is no longer considered.
183
183
184 Parameters
184 Parameters
185 ----------
185 ----------
186 require_writable : bool [default: False]
186 require_writable : bool [default: False]
187 if True:
187 if True:
188 guarantees the return value is a writable directory, otherwise
188 guarantees the return value is a writable directory, otherwise
189 raises HomeDirError
189 raises HomeDirError
190 if False:
190 if False:
191 The path is resolved, but it is not guaranteed to exist or be writable.
191 The path is resolved, but it is not guaranteed to exist or be writable.
192 """
192 """
193
193
194 homedir = os.path.expanduser('~')
194 homedir = os.path.expanduser('~')
195 # Next line will make things work even when /home/ is a symlink to
195 # Next line will make things work even when /home/ is a symlink to
196 # /usr/home as it is on FreeBSD, for example
196 # /usr/home as it is on FreeBSD, for example
197 homedir = os.path.realpath(homedir)
197 homedir = os.path.realpath(homedir)
198
198
199 if not _writable_dir(homedir) and os.name == 'nt':
199 if not _writable_dir(homedir) and os.name == 'nt':
200 # expanduser failed, use the registry to get the 'My Documents' folder.
200 # expanduser failed, use the registry to get the 'My Documents' folder.
201 try:
201 try:
202 import winreg as wreg
202 import winreg as wreg
203 with wreg.OpenKey(
203 with wreg.OpenKey(
204 wreg.HKEY_CURRENT_USER,
204 wreg.HKEY_CURRENT_USER,
205 r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders"
205 r"Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders"
206 ) as key:
206 ) as key:
207 homedir = wreg.QueryValueEx(key,'Personal')[0]
207 homedir = wreg.QueryValueEx(key,'Personal')[0]
208 except:
208 except:
209 pass
209 pass
210
210
211 if (not require_writable) or _writable_dir(homedir):
211 if (not require_writable) or _writable_dir(homedir):
212 assert isinstance(homedir, str), "Homedir shoudl be unicode not bytes"
212 assert isinstance(homedir, str), "Homedir should be unicode not bytes"
213 return homedir
213 return homedir
214 else:
214 else:
215 raise HomeDirError('%s is not a writable dir, '
215 raise HomeDirError('%s is not a writable dir, '
216 'set $HOME environment variable to override' % homedir)
216 'set $HOME environment variable to override' % homedir)
217
217
218 def get_xdg_dir():
218 def get_xdg_dir():
219 """Return the XDG_CONFIG_HOME, if it is defined and exists, else None.
219 """Return the XDG_CONFIG_HOME, if it is defined and exists, else None.
220
220
221 This is only for non-OS X posix (Linux,Unix,etc.) systems.
221 This is only for non-OS X posix (Linux,Unix,etc.) systems.
222 """
222 """
223
223
224 env = os.environ
224 env = os.environ
225
225
226 if os.name == 'posix' and sys.platform != 'darwin':
226 if os.name == 'posix' and sys.platform != 'darwin':
227 # Linux, Unix, AIX, etc.
227 # Linux, Unix, AIX, etc.
228 # use ~/.config if empty OR not set
228 # use ~/.config if empty OR not set
229 xdg = env.get("XDG_CONFIG_HOME", None) or os.path.join(get_home_dir(), '.config')
229 xdg = env.get("XDG_CONFIG_HOME", None) or os.path.join(get_home_dir(), '.config')
230 if xdg and _writable_dir(xdg):
230 if xdg and _writable_dir(xdg):
231 assert isinstance(xdg, str)
231 assert isinstance(xdg, str)
232 return xdg
232 return xdg
233
233
234 return None
234 return None
235
235
236
236
237 def get_xdg_cache_dir():
237 def get_xdg_cache_dir():
238 """Return the XDG_CACHE_HOME, if it is defined and exists, else None.
238 """Return the XDG_CACHE_HOME, if it is defined and exists, else None.
239
239
240 This is only for non-OS X posix (Linux,Unix,etc.) systems.
240 This is only for non-OS X posix (Linux,Unix,etc.) systems.
241 """
241 """
242
242
243 env = os.environ
243 env = os.environ
244
244
245 if os.name == 'posix' and sys.platform != 'darwin':
245 if os.name == 'posix' and sys.platform != 'darwin':
246 # Linux, Unix, AIX, etc.
246 # Linux, Unix, AIX, etc.
247 # use ~/.cache if empty OR not set
247 # use ~/.cache if empty OR not set
248 xdg = env.get("XDG_CACHE_HOME", None) or os.path.join(get_home_dir(), '.cache')
248 xdg = env.get("XDG_CACHE_HOME", None) or os.path.join(get_home_dir(), '.cache')
249 if xdg and _writable_dir(xdg):
249 if xdg and _writable_dir(xdg):
250 assert isinstance(xdg, str)
250 assert isinstance(xdg, str)
251 return xdg
251 return xdg
252
252
253 return None
253 return None
254
254
255
255
256 @undoc
256 @undoc
257 def get_ipython_dir():
257 def get_ipython_dir():
258 warn("get_ipython_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
258 warn("get_ipython_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
259 from IPython.paths import get_ipython_dir
259 from IPython.paths import get_ipython_dir
260 return get_ipython_dir()
260 return get_ipython_dir()
261
261
262 @undoc
262 @undoc
263 def get_ipython_cache_dir():
263 def get_ipython_cache_dir():
264 warn("get_ipython_cache_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
264 warn("get_ipython_cache_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
265 from IPython.paths import get_ipython_cache_dir
265 from IPython.paths import get_ipython_cache_dir
266 return get_ipython_cache_dir()
266 return get_ipython_cache_dir()
267
267
268 @undoc
268 @undoc
269 def get_ipython_package_dir():
269 def get_ipython_package_dir():
270 warn("get_ipython_package_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
270 warn("get_ipython_package_dir has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
271 from IPython.paths import get_ipython_package_dir
271 from IPython.paths import get_ipython_package_dir
272 return get_ipython_package_dir()
272 return get_ipython_package_dir()
273
273
274 @undoc
274 @undoc
275 def get_ipython_module_path(module_str):
275 def get_ipython_module_path(module_str):
276 warn("get_ipython_module_path has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
276 warn("get_ipython_module_path has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
277 from IPython.paths import get_ipython_module_path
277 from IPython.paths import get_ipython_module_path
278 return get_ipython_module_path(module_str)
278 return get_ipython_module_path(module_str)
279
279
280 @undoc
280 @undoc
281 def locate_profile(profile='default'):
281 def locate_profile(profile='default'):
282 warn("locate_profile has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
282 warn("locate_profile has moved to the IPython.paths module since IPython 4.0.", DeprecationWarning, stacklevel=2)
283 from IPython.paths import locate_profile
283 from IPython.paths import locate_profile
284 return locate_profile(profile=profile)
284 return locate_profile(profile=profile)
285
285
286 def expand_path(s):
286 def expand_path(s):
287 """Expand $VARS and ~names in a string, like a shell
287 """Expand $VARS and ~names in a string, like a shell
288
288
289 :Examples:
289 :Examples:
290
290
291 In [2]: os.environ['FOO']='test'
291 In [2]: os.environ['FOO']='test'
292
292
293 In [3]: expand_path('variable FOO is $FOO')
293 In [3]: expand_path('variable FOO is $FOO')
294 Out[3]: 'variable FOO is test'
294 Out[3]: 'variable FOO is test'
295 """
295 """
296 # This is a pretty subtle hack. When expand user is given a UNC path
296 # This is a pretty subtle hack. When expand user is given a UNC path
297 # on Windows (\\server\share$\%username%), os.path.expandvars, removes
297 # on Windows (\\server\share$\%username%), os.path.expandvars, removes
298 # the $ to get (\\server\share\%username%). I think it considered $
298 # the $ to get (\\server\share\%username%). I think it considered $
299 # alone an empty var. But, we need the $ to remains there (it indicates
299 # alone an empty var. But, we need the $ to remains there (it indicates
300 # a hidden share).
300 # a hidden share).
301 if os.name=='nt':
301 if os.name=='nt':
302 s = s.replace('$\\', 'IPYTHON_TEMP')
302 s = s.replace('$\\', 'IPYTHON_TEMP')
303 s = os.path.expandvars(os.path.expanduser(s))
303 s = os.path.expandvars(os.path.expanduser(s))
304 if os.name=='nt':
304 if os.name=='nt':
305 s = s.replace('IPYTHON_TEMP', '$\\')
305 s = s.replace('IPYTHON_TEMP', '$\\')
306 return s
306 return s
307
307
308
308
309 def unescape_glob(string):
309 def unescape_glob(string):
310 """Unescape glob pattern in `string`."""
310 """Unescape glob pattern in `string`."""
311 def unescape(s):
311 def unescape(s):
312 for pattern in '*[]!?':
312 for pattern in '*[]!?':
313 s = s.replace(r'\{0}'.format(pattern), pattern)
313 s = s.replace(r'\{0}'.format(pattern), pattern)
314 return s
314 return s
315 return '\\'.join(map(unescape, string.split('\\\\')))
315 return '\\'.join(map(unescape, string.split('\\\\')))
316
316
317
317
318 def shellglob(args):
318 def shellglob(args):
319 """
319 """
320 Do glob expansion for each element in `args` and return a flattened list.
320 Do glob expansion for each element in `args` and return a flattened list.
321
321
322 Unmatched glob pattern will remain as-is in the returned list.
322 Unmatched glob pattern will remain as-is in the returned list.
323
323
324 """
324 """
325 expanded = []
325 expanded = []
326 # Do not unescape backslash in Windows as it is interpreted as
326 # Do not unescape backslash in Windows as it is interpreted as
327 # path separator:
327 # path separator:
328 unescape = unescape_glob if sys.platform != 'win32' else lambda x: x
328 unescape = unescape_glob if sys.platform != 'win32' else lambda x: x
329 for a in args:
329 for a in args:
330 expanded.extend(glob.glob(a) or [unescape(a)])
330 expanded.extend(glob.glob(a) or [unescape(a)])
331 return expanded
331 return expanded
332
332
333
333
334 def target_outdated(target,deps):
334 def target_outdated(target,deps):
335 """Determine whether a target is out of date.
335 """Determine whether a target is out of date.
336
336
337 target_outdated(target,deps) -> 1/0
337 target_outdated(target,deps) -> 1/0
338
338
339 deps: list of filenames which MUST exist.
339 deps: list of filenames which MUST exist.
340 target: single filename which may or may not exist.
340 target: single filename which may or may not exist.
341
341
342 If target doesn't exist or is older than any file listed in deps, return
342 If target doesn't exist or is older than any file listed in deps, return
343 true, otherwise return false.
343 true, otherwise return false.
344 """
344 """
345 try:
345 try:
346 target_time = os.path.getmtime(target)
346 target_time = os.path.getmtime(target)
347 except os.error:
347 except os.error:
348 return 1
348 return 1
349 for dep in deps:
349 for dep in deps:
350 dep_time = os.path.getmtime(dep)
350 dep_time = os.path.getmtime(dep)
351 if dep_time > target_time:
351 if dep_time > target_time:
352 #print "For target",target,"Dep failed:",dep # dbg
352 #print "For target",target,"Dep failed:",dep # dbg
353 #print "times (dep,tar):",dep_time,target_time # dbg
353 #print "times (dep,tar):",dep_time,target_time # dbg
354 return 1
354 return 1
355 return 0
355 return 0
356
356
357
357
358 def target_update(target,deps,cmd):
358 def target_update(target,deps,cmd):
359 """Update a target with a given command given a list of dependencies.
359 """Update a target with a given command given a list of dependencies.
360
360
361 target_update(target,deps,cmd) -> runs cmd if target is outdated.
361 target_update(target,deps,cmd) -> runs cmd if target is outdated.
362
362
363 This is just a wrapper around target_outdated() which calls the given
363 This is just a wrapper around target_outdated() which calls the given
364 command if target is outdated."""
364 command if target is outdated."""
365
365
366 if target_outdated(target,deps):
366 if target_outdated(target,deps):
367 system(cmd)
367 system(cmd)
368
368
369
369
370 ENOLINK = 1998
370 ENOLINK = 1998
371
371
372 def link(src, dst):
372 def link(src, dst):
373 """Hard links ``src`` to ``dst``, returning 0 or errno.
373 """Hard links ``src`` to ``dst``, returning 0 or errno.
374
374
375 Note that the special errno ``ENOLINK`` will be returned if ``os.link`` isn't
375 Note that the special errno ``ENOLINK`` will be returned if ``os.link`` isn't
376 supported by the operating system.
376 supported by the operating system.
377 """
377 """
378
378
379 if not hasattr(os, "link"):
379 if not hasattr(os, "link"):
380 return ENOLINK
380 return ENOLINK
381 link_errno = 0
381 link_errno = 0
382 try:
382 try:
383 os.link(src, dst)
383 os.link(src, dst)
384 except OSError as e:
384 except OSError as e:
385 link_errno = e.errno
385 link_errno = e.errno
386 return link_errno
386 return link_errno
387
387
388
388
389 def link_or_copy(src, dst):
389 def link_or_copy(src, dst):
390 """Attempts to hardlink ``src`` to ``dst``, copying if the link fails.
390 """Attempts to hardlink ``src`` to ``dst``, copying if the link fails.
391
391
392 Attempts to maintain the semantics of ``shutil.copy``.
392 Attempts to maintain the semantics of ``shutil.copy``.
393
393
394 Because ``os.link`` does not overwrite files, a unique temporary file
394 Because ``os.link`` does not overwrite files, a unique temporary file
395 will be used if the target already exists, then that file will be moved
395 will be used if the target already exists, then that file will be moved
396 into place.
396 into place.
397 """
397 """
398
398
399 if os.path.isdir(dst):
399 if os.path.isdir(dst):
400 dst = os.path.join(dst, os.path.basename(src))
400 dst = os.path.join(dst, os.path.basename(src))
401
401
402 link_errno = link(src, dst)
402 link_errno = link(src, dst)
403 if link_errno == errno.EEXIST:
403 if link_errno == errno.EEXIST:
404 if os.stat(src).st_ino == os.stat(dst).st_ino:
404 if os.stat(src).st_ino == os.stat(dst).st_ino:
405 # dst is already a hard link to the correct file, so we don't need
405 # dst is already a hard link to the correct file, so we don't need
406 # to do anything else. If we try to link and rename the file
406 # to do anything else. If we try to link and rename the file
407 # anyway, we get duplicate files - see http://bugs.python.org/issue21876
407 # anyway, we get duplicate files - see http://bugs.python.org/issue21876
408 return
408 return
409
409
410 new_dst = dst + "-temp-%04X" %(random.randint(1, 16**4), )
410 new_dst = dst + "-temp-%04X" %(random.randint(1, 16**4), )
411 try:
411 try:
412 link_or_copy(src, new_dst)
412 link_or_copy(src, new_dst)
413 except:
413 except:
414 try:
414 try:
415 os.remove(new_dst)
415 os.remove(new_dst)
416 except OSError:
416 except OSError:
417 pass
417 pass
418 raise
418 raise
419 os.rename(new_dst, dst)
419 os.rename(new_dst, dst)
420 elif link_errno != 0:
420 elif link_errno != 0:
421 # Either link isn't supported, or the filesystem doesn't support
421 # Either link isn't supported, or the filesystem doesn't support
422 # linking, or 'src' and 'dst' are on different filesystems.
422 # linking, or 'src' and 'dst' are on different filesystems.
423 shutil.copy(src, dst)
423 shutil.copy(src, dst)
424
424
425 def ensure_dir_exists(path, mode=0o755):
425 def ensure_dir_exists(path, mode=0o755):
426 """ensure that a directory exists
426 """ensure that a directory exists
427
427
428 If it doesn't exist, try to create it and protect against a race condition
428 If it doesn't exist, try to create it and protect against a race condition
429 if another process is doing the same.
429 if another process is doing the same.
430
430
431 The default permissions are 755, which differ from os.makedirs default of 777.
431 The default permissions are 755, which differ from os.makedirs default of 777.
432 """
432 """
433 if not os.path.exists(path):
433 if not os.path.exists(path):
434 try:
434 try:
435 os.makedirs(path, mode=mode)
435 os.makedirs(path, mode=mode)
436 except OSError as e:
436 except OSError as e:
437 if e.errno != errno.EEXIST:
437 if e.errno != errno.EEXIST:
438 raise
438 raise
439 elif not os.path.isdir(path):
439 elif not os.path.isdir(path):
440 raise IOError("%r exists but is not a directory" % path)
440 raise IOError("%r exists but is not a directory" % path)
@@ -1,39 +1,39 b''
1 #!/usr/bin/env python
1 #!/usr/bin/env python
2 """Simple GTK example to manually test event loop integration.
2 """Simple GTK example to manually test event loop integration.
3
3
4 This is meant to run tests manually in ipython as:
4 This is meant to run tests manually in ipython as:
5
5
6 In [5]: %gui gtk
6 In [5]: %gui gtk
7
7
8 In [6]: %run gui-gtk.py
8 In [6]: %run gui-gtk.py
9 """
9 """
10
10
11 import pygtk
11 import pygtk
12 pygtk.require('2.0')
12 pygtk.require('2.0')
13 import gtk
13 import gtk
14
14
15
15
16 def hello_world(wigdet, data=None):
16 def hello_world(widget, data=None):
17 print("Hello World")
17 print("Hello World")
18
18
19 def delete_event(widget, event, data=None):
19 def delete_event(widget, event, data=None):
20 return False
20 return False
21
21
22 def destroy(widget, data=None):
22 def destroy(widget, data=None):
23 gtk.main_quit()
23 gtk.main_quit()
24
24
25 window = gtk.Window(gtk.WINDOW_TOPLEVEL)
25 window = gtk.Window(gtk.WINDOW_TOPLEVEL)
26 window.connect("delete_event", delete_event)
26 window.connect("delete_event", delete_event)
27 window.connect("destroy", destroy)
27 window.connect("destroy", destroy)
28 button = gtk.Button("Hello World")
28 button = gtk.Button("Hello World")
29 button.connect("clicked", hello_world, None)
29 button.connect("clicked", hello_world, None)
30
30
31 window.add(button)
31 window.add(button)
32 button.show()
32 button.show()
33 window.show()
33 window.show()
34
34
35 try:
35 try:
36 from IPython.lib.inputhook import enable_gui
36 from IPython.lib.inputhook import enable_gui
37 enable_gui('gtk')
37 enable_gui('gtk')
38 except ImportError:
38 except ImportError:
39 gtk.main()
39 gtk.main()
@@ -1,37 +1,37 b''
1 #!/usr/bin/env python
1 #!/usr/bin/env python
2 """Simple Gtk example to manually test event loop integration.
2 """Simple Gtk example to manually test event loop integration.
3
3
4 This is meant to run tests manually in ipython as:
4 This is meant to run tests manually in ipython as:
5
5
6 In [1]: %gui gtk3
6 In [1]: %gui gtk3
7
7
8 In [2]: %run gui-gtk3.py
8 In [2]: %run gui-gtk3.py
9 """
9 """
10
10
11 from gi.repository import Gtk
11 from gi.repository import Gtk
12
12
13
13
14 def hello_world(wigdet, data=None):
14 def hello_world(widget, data=None):
15 print("Hello World")
15 print("Hello World")
16
16
17 def delete_event(widget, event, data=None):
17 def delete_event(widget, event, data=None):
18 return False
18 return False
19
19
20 def destroy(widget, data=None):
20 def destroy(widget, data=None):
21 Gtk.main_quit()
21 Gtk.main_quit()
22
22
23 window = Gtk.Window(type=Gtk.WindowType.TOPLEVEL)
23 window = Gtk.Window(type=Gtk.WindowType.TOPLEVEL)
24 window.connect("delete_event", delete_event)
24 window.connect("delete_event", delete_event)
25 window.connect("destroy", destroy)
25 window.connect("destroy", destroy)
26 button = Gtk.Button(label="Hello World")
26 button = Gtk.Button(label="Hello World")
27 button.connect("clicked", hello_world, None)
27 button.connect("clicked", hello_world, None)
28
28
29 window.add(button)
29 window.add(button)
30 button.show()
30 button.show()
31 window.show()
31 window.show()
32
32
33 try:
33 try:
34 from IPython.lib.inputhook import enable_gui
34 from IPython.lib.inputhook import enable_gui
35 enable_gui('gtk3')
35 enable_gui('gtk3')
36 except ImportError:
36 except ImportError:
37 Gtk.main()
37 Gtk.main()
@@ -1,37 +1,37 b''
1 #!/usr/bin/env python
1 #!/usr/bin/env python
2 """Simple Gtk example to manually test event loop integration.
2 """Simple Gtk example to manually test event loop integration.
3
3
4 This is meant to run tests manually in ipython as:
4 This is meant to run tests manually in ipython as:
5
5
6 In [1]: %gui gtk4
6 In [1]: %gui gtk4
7
7
8 In [2]: %run gui-gtk4.py
8 In [2]: %run gui-gtk4.py
9 """
9 """
10
10
11 import gi
11 import gi
12
12
13 gi.require_version("Gtk", "4.0")
13 gi.require_version("Gtk", "4.0")
14 from gi.repository import Gtk, GLib # noqa
14 from gi.repository import Gtk, GLib # noqa
15
15
16
16
17 def hello_world(wigdet, data=None):
17 def hello_world(widget, data=None):
18 print("Hello World")
18 print("Hello World")
19
19
20
20
21 def close_request_cb(widget, data=None):
21 def close_request_cb(widget, data=None):
22 global running
22 global running
23 running = False
23 running = False
24
24
25
25
26 running = True
26 running = True
27 window = Gtk.Window()
27 window = Gtk.Window()
28 window.connect("close-request", close_request_cb)
28 window.connect("close-request", close_request_cb)
29 button = Gtk.Button(label="Hello World")
29 button = Gtk.Button(label="Hello World")
30 button.connect("clicked", hello_world, None)
30 button.connect("clicked", hello_world, None)
31
31
32 window.set_child(button)
32 window.set_child(button)
33 window.show()
33 window.show()
34
34
35 context = GLib.MainContext.default()
35 context = GLib.MainContext.default()
36 while running:
36 while running:
37 context.iteration(True)
37 context.iteration(True)
@@ -1,239 +1,239 b''
1 # Simple tool to help for release
1 # Simple tool to help for release
2 # when releasing with bash, simple source it to get asked questions.
2 # when releasing with bash, simple source it to get asked questions.
3
3
4 # misc check before starting
4 # misc check before starting
5
5
6 python -c 'import keyring'
6 python -c 'import keyring'
7 python -c 'import twine'
7 python -c 'import twine'
8 python -c 'import sphinx'
8 python -c 'import sphinx'
9 python -c 'import sphinx_rtd_theme'
9 python -c 'import sphinx_rtd_theme'
10 python -c 'import nose'
10 python -c 'import nose'
11
11
12
12
13 BLACK=$(tput setaf 1)
13 BLACK=$(tput setaf 1)
14 RED=$(tput setaf 1)
14 RED=$(tput setaf 1)
15 GREEN=$(tput setaf 2)
15 GREEN=$(tput setaf 2)
16 YELLOW=$(tput setaf 3)
16 YELLOW=$(tput setaf 3)
17 BLUE=$(tput setaf 4)
17 BLUE=$(tput setaf 4)
18 MAGENTA=$(tput setaf 5)
18 MAGENTA=$(tput setaf 5)
19 CYAN=$(tput setaf 6)
19 CYAN=$(tput setaf 6)
20 WHITE=$(tput setaf 7)
20 WHITE=$(tput setaf 7)
21 NOR=$(tput sgr0)
21 NOR=$(tput sgr0)
22
22
23
23
24 echo "Will use $EDITOR to edit files when necessary"
24 echo "Will use $EDITOR to edit files when necessary"
25 echo -n "PREV_RELEASE (X.y.z) [$PREV_RELEASE]: "
25 echo -n "PREV_RELEASE (X.y.z) [$PREV_RELEASE]: "
26 read input
26 read input
27 PREV_RELEASE=${input:-$PREV_RELEASE}
27 PREV_RELEASE=${input:-$PREV_RELEASE}
28 echo -n "MILESTONE (X.y) [$MILESTONE]: "
28 echo -n "MILESTONE (X.y) [$MILESTONE]: "
29 read input
29 read input
30 MILESTONE=${input:-$MILESTONE}
30 MILESTONE=${input:-$MILESTONE}
31 echo -n "VERSION (X.y.z) [$VERSION]:"
31 echo -n "VERSION (X.y.z) [$VERSION]:"
32 read input
32 read input
33 VERSION=${input:-$VERSION}
33 VERSION=${input:-$VERSION}
34 echo -n "BRANCH (master|X.y) [$BRANCH]:"
34 echo -n "BRANCH (master|X.y) [$BRANCH]:"
35 read input
35 read input
36 BRANCH=${input:-$BRANCH}
36 BRANCH=${input:-$BRANCH}
37
37
38 ask_section(){
38 ask_section(){
39 echo
39 echo
40 echo $BLUE"$1"$NOR
40 echo $BLUE"$1"$NOR
41 echo -n $GREEN"Press Enter to continue, S to skip: "$NOR
41 echo -n $GREEN"Press Enter to continue, S to skip: "$NOR
42 read -n1 value
42 read -n1 value
43 echo
43 echo
44 if [ -z $value ] || [ $value = 'y' ] ; then
44 if [ -z $value ] || [ $value = 'y' ] ; then
45 return 0
45 return 0
46 fi
46 fi
47 return 1
47 return 1
48 }
48 }
49
49
50
50
51 maybe_edit(){
51 maybe_edit(){
52 echo
52 echo
53 echo $BLUE"$1"$NOR
53 echo $BLUE"$1"$NOR
54 echo -n $GREEN"Press e to Edit $1, any other keys to skip: "$NOR
54 echo -n $GREEN"Press e to Edit $1, any other keys to skip: "$NOR
55 read -n1 value
55 read -n1 value
56 echo
56 echo
57 if [ $value = 'e' ] ; then
57 if [ $value = 'e' ] ; then
58 $EDITOR $1
58 $EDITOR $1
59 fi
59 fi
60 }
60 }
61
61
62
62
63
63
64 echo
64 echo
65 if ask_section "Updating what's new with informations from docs/source/whatsnew/pr"
65 if ask_section "Updating what's new with information from docs/source/whatsnew/pr"
66 then
66 then
67 python tools/update_whatsnew.py
67 python tools/update_whatsnew.py
68
68
69 echo
69 echo
70 echo $BLUE"please move the contents of "docs/source/whatsnew/development.rst" to version-X.rst"$NOR
70 echo $BLUE"please move the contents of "docs/source/whatsnew/development.rst" to version-X.rst"$NOR
71 echo $GREEN"Press enter to continue"$NOR
71 echo $GREEN"Press enter to continue"$NOR
72 read
72 read
73 fi
73 fi
74
74
75 if ask_section "Gen Stats, and authors"
75 if ask_section "Gen Stats, and authors"
76 then
76 then
77
77
78 echo
78 echo
79 echo $BLUE"here are all the authors that contributed to this release:"$NOR
79 echo $BLUE"here are all the authors that contributed to this release:"$NOR
80 git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
80 git log --format="%aN <%aE>" $PREV_RELEASE... | sort -u -f
81
81
82 echo
82 echo
83 echo $BLUE"If you see any duplicates cancel (Ctrl-C), then edit .mailmap."
83 echo $BLUE"If you see any duplicates cancel (Ctrl-C), then edit .mailmap."
84 echo $GREEN"Press enter to continue:"$NOR
84 echo $GREEN"Press enter to continue:"$NOR
85 read
85 read
86
86
87 echo $BLUE"generating stats"$NOR
87 echo $BLUE"generating stats"$NOR
88 python tools/github_stats.py --milestone $MILESTONE > stats.rst
88 python tools/github_stats.py --milestone $MILESTONE > stats.rst
89
89
90 echo $BLUE"stats.rst files generated."$NOR
90 echo $BLUE"stats.rst files generated."$NOR
91 echo $GREEN"Please merge it with the right file (github-stats-X.rst) and commit."$NOR
91 echo $GREEN"Please merge it with the right file (github-stats-X.rst) and commit."$NOR
92 echo $GREEN"press enter to continue."$NOR
92 echo $GREEN"press enter to continue."$NOR
93 read
93 read
94
94
95 fi
95 fi
96
96
97 if ask_section "Generate API difference (using frapuccino)"
97 if ask_section "Generate API difference (using frapuccino)"
98 then
98 then
99 echo $BLUE"Checking out $PREV_RELEASE"$NOR
99 echo $BLUE"Checking out $PREV_RELEASE"$NOR
100 git checkout $PREV_RELEASE
100 git checkout $PREV_RELEASE
101 echo $BLUE"Saving API to file $PREV_RELEASE"$NOR
101 echo $BLUE"Saving API to file $PREV_RELEASE"$NOR
102 frappuccino IPython --save IPython-$PREV_RELEASE.json
102 frappuccino IPython --save IPython-$PREV_RELEASE.json
103 echo $BLUE"comming back to $BRANCH"$NOR
103 echo $BLUE"coming back to $BRANCH"$NOR
104 git checkout $BRANCH
104 git checkout $BRANCH
105 echo $BLUE"comparing ..."$NOR
105 echo $BLUE"comparing ..."$NOR
106 frappuccino IPython --compare IPython-$PREV_RELEASE.json
106 frappuccino IPython --compare IPython-$PREV_RELEASE.json
107 echo $GREEN"Use the above guideline to write an API changelog ..."$NOR
107 echo $GREEN"Use the above guideline to write an API changelog ..."$NOR
108 echo $GREEN"Press any keys to continue"$NOR
108 echo $GREEN"Press any keys to continue"$NOR
109 read
109 read
110 fi
110 fi
111
111
112 echo "Cleaning repository"
112 echo "Cleaning repository"
113 git clean -xfdi
113 git clean -xfdi
114
114
115 echo $GREEN"please update version number in ${RED}IPython/core/release.py${NOR} , Do not commit yet – we'll do it later."$NOR
115 echo $GREEN"please update version number in ${RED}IPython/core/release.py${NOR} , Do not commit yet – we'll do it later."$NOR
116 echo $GREEN"I tried ${RED}sed -i bkp -e '/Uncomment/s/^# //g' IPython/core/release.py${NOR}"
116 echo $GREEN"I tried ${RED}sed -i bkp -e '/Uncomment/s/^# //g' IPython/core/release.py${NOR}"
117 sed -i bkp -e '/Uncomment/s/^# //g' IPython/core/release.py
117 sed -i bkp -e '/Uncomment/s/^# //g' IPython/core/release.py
118 rm IPython/core/release.pybkp
118 rm IPython/core/release.pybkp
119 git diff
119 git diff
120 maybe_edit IPython/core/release.py
120 maybe_edit IPython/core/release.py
121
121
122 echo $GREEN"Press enter to continue"$NOR
122 echo $GREEN"Press enter to continue"$NOR
123 read
123 read
124
124
125 if ask_section "Build the documentation ?"
125 if ask_section "Build the documentation ?"
126 then
126 then
127 make html -C docs
127 make html -C docs
128 echo
128 echo
129 echo $GREEN"Check the docs, press enter to continue"$NOR
129 echo $GREEN"Check the docs, press enter to continue"$NOR
130 read
130 read
131
131
132 fi
132 fi
133
133
134 if ask_section "Should we commit, tag, push... etc ? "
134 if ask_section "Should we commit, tag, push... etc ? "
135 then
135 then
136 echo
136 echo
137 echo $BLUE"Let's commit : git commit -am \"release $VERSION\" -S"
137 echo $BLUE"Let's commit : git commit -am \"release $VERSION\" -S"
138 echo $GREEN"Press enter to commit"$NOR
138 echo $GREEN"Press enter to commit"$NOR
139 read
139 read
140 git commit -am "release $VERSION" -S
140 git commit -am "release $VERSION" -S
141
141
142 echo
142 echo
143 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
143 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
144 echo $GREEN"Make sure you can push"$NOR
144 echo $GREEN"Make sure you can push"$NOR
145 echo $GREEN"Press enter to continue"$NOR
145 echo $GREEN"Press enter to continue"$NOR
146 read
146 read
147 git push origin $BRANCH
147 git push origin $BRANCH
148
148
149 echo
149 echo
150 echo "Let's tag : git tag -am \"release $VERSION\" \"$VERSION\" -s"
150 echo "Let's tag : git tag -am \"release $VERSION\" \"$VERSION\" -s"
151 echo $GREEN"Press enter to tag commit"$NOR
151 echo $GREEN"Press enter to tag commit"$NOR
152 read
152 read
153 git tag -am "release $VERSION" "$VERSION" -s
153 git tag -am "release $VERSION" "$VERSION" -s
154
154
155 echo
155 echo
156 echo $BLUE"And push the tag: git push origin \$VERSION ?"$NOR
156 echo $BLUE"And push the tag: git push origin \$VERSION ?"$NOR
157 echo $GREEN"Press enter to continue"$NOR
157 echo $GREEN"Press enter to continue"$NOR
158 read
158 read
159 git push origin $VERSION
159 git push origin $VERSION
160
160
161
161
162 echo $GREEN"please update version number and back to .dev in ${RED}IPython/core/release.py"
162 echo $GREEN"please update version number and back to .dev in ${RED}IPython/core/release.py"
163 echo $GREEN"I tried ${RED}sed -i bkp -e '/Uncomment/s/^/# /g' IPython/core/release.py${NOR}"
163 echo $GREEN"I tried ${RED}sed -i bkp -e '/Uncomment/s/^/# /g' IPython/core/release.py${NOR}"
164 sed -i bkp -e '/Uncomment/s/^/# /g' IPython/core/release.py
164 sed -i bkp -e '/Uncomment/s/^/# /g' IPython/core/release.py
165 rm IPython/core/release.pybkp
165 rm IPython/core/release.pybkp
166 git diff
166 git diff
167 echo $GREEN"Please bump ${RED}the minor version number${NOR}"
167 echo $GREEN"Please bump ${RED}the minor version number${NOR}"
168 maybe_edit IPython/core/release.py
168 maybe_edit IPython/core/release.py
169 echo ${BLUE}"Do not commit yet – we'll do it later."$NOR
169 echo ${BLUE}"Do not commit yet – we'll do it later."$NOR
170
170
171
171
172 echo $GREEN"Press enter to continue"$NOR
172 echo $GREEN"Press enter to continue"$NOR
173 read
173 read
174
174
175 echo
175 echo
176 echo "Let's commit : "$BLUE"git commit -am \"back to dev\""$NOR
176 echo "Let's commit : "$BLUE"git commit -am \"back to dev\""$NOR
177 echo $GREEN"Press enter to commit"$NOR
177 echo $GREEN"Press enter to commit"$NOR
178 read
178 read
179 git commit -am "back to dev"
179 git commit -am "back to dev"
180
180
181 echo
181 echo
182 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
182 echo $BLUE"git push origin \$BRANCH ($BRANCH)?"$NOR
183 echo $GREEN"Press enter to continue"$NOR
183 echo $GREEN"Press enter to continue"$NOR
184 read
184 read
185 git push origin $BRANCH
185 git push origin $BRANCH
186
186
187
187
188 echo
188 echo
189 echo $BLUE"let's : git checkout $VERSION"$NOR
189 echo $BLUE"let's : git checkout $VERSION"$NOR
190 echo $GREEN"Press enter to continue"$NOR
190 echo $GREEN"Press enter to continue"$NOR
191 read
191 read
192 git checkout $VERSION
192 git checkout $VERSION
193 fi
193 fi
194
194
195 if ask_section "Should we build and release ?"
195 if ask_section "Should we build and release ?"
196 then
196 then
197
197
198 echo $BLUE"going to set SOURCE_DATE_EPOCH"$NOR
198 echo $BLUE"going to set SOURCE_DATE_EPOCH"$NOR
199 echo $BLUE'export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)'$NOR
199 echo $BLUE'export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)'$NOR
200 echo $GREEN"Press enter to continue"$NOR
200 echo $GREEN"Press enter to continue"$NOR
201 read
201 read
202
202
203 export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
203 export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
204
204
205 echo $BLUE"SOURCE_DATE_EPOCH set to $SOURCE_DATE_EPOCH"$NOR
205 echo $BLUE"SOURCE_DATE_EPOCH set to $SOURCE_DATE_EPOCH"$NOR
206 echo $GREEN"Press enter to continue"$NOR
206 echo $GREEN"Press enter to continue"$NOR
207 read
207 read
208
208
209
209
210
210
211 echo
211 echo
212 echo $BLUE"Attempting to build package..."$NOR
212 echo $BLUE"Attempting to build package..."$NOR
213
213
214 tools/release
214 tools/release
215
215
216
216
217 echo $RED'$ shasum -a 256 dist/*'
217 echo $RED'$ shasum -a 256 dist/*'
218 shasum -a 256 dist/*
218 shasum -a 256 dist/*
219 echo $NOR
219 echo $NOR
220
220
221 echo $BLUE"We are going to rebuild, node the hash above, and compare them to the rebuild"$NOR
221 echo $BLUE"We are going to rebuild, node the hash above, and compare them to the rebuild"$NOR
222 echo $GREEN"Press enter to continue"$NOR
222 echo $GREEN"Press enter to continue"$NOR
223 read
223 read
224
224
225 echo
225 echo
226 echo $BLUE"Attempting to build package..."$NOR
226 echo $BLUE"Attempting to build package..."$NOR
227
227
228 tools/release
228 tools/release
229
229
230 echo $RED"Check the shasum for SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
230 echo $RED"Check the shasum for SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
231 echo $RED'$ shasum -a 256 dist/*'
231 echo $RED'$ shasum -a 256 dist/*'
232 shasum -a 256 dist/*
232 shasum -a 256 dist/*
233 echo $NOR
233 echo $NOR
234
234
235 if ask_section "upload packages ?"
235 if ask_section "upload packages ?"
236 then
236 then
237 tools/release upload
237 tools/release upload
238 fi
238 fi
239 fi
239 fi
@@ -1,68 +1,68 b''
1 """
1 """
2 Un-targz and retargz a targz file to ensure reproducible build.
2 Un-targz and retargz a targz file to ensure reproducible build.
3
3
4 usage:
4 usage:
5
5
6 $ export SOURCE_DATE_EPOCH=$(date +%s)
6 $ export SOURCE_DATE_EPOCH=$(date +%s)
7 ...
7 ...
8 $ python retar.py <tarfile.gz>
8 $ python retar.py <tarfile.gz>
9
9
10 The process of creating an sdist can be non-reproducible:
10 The process of creating an sdist can be non-reproducible:
11 - directory created during the process get a mtime of the creation date;
11 - directory created during the process get a mtime of the creation date;
12 - gziping files embed the timestamp of fo zip creation.
12 - gziping files embed the timestamp of zip creation.
13
13
14 This will untar-retar; ensuring that all mtime > SOURCE_DATE_EPOCH will be set
14 This will untar-retar; ensuring that all mtime > SOURCE_DATE_EPOCH will be set
15 equal to SOURCE_DATE_EPOCH.
15 equal to SOURCE_DATE_EPOCH.
16
16
17 """
17 """
18
18
19 import tarfile
19 import tarfile
20 import sys
20 import sys
21 import os
21 import os
22 import gzip
22 import gzip
23 import io
23 import io
24
24
25 from pathlib import Path
25 from pathlib import Path
26
26
27 if len(sys.argv) > 2:
27 if len(sys.argv) > 2:
28 raise ValueError("Too many arguments")
28 raise ValueError("Too many arguments")
29
29
30
30
31 timestamp = int(os.environ["SOURCE_DATE_EPOCH"])
31 timestamp = int(os.environ["SOURCE_DATE_EPOCH"])
32
32
33 path = Path(sys.argv[1])
33 path = Path(sys.argv[1])
34 old_buf = io.BytesIO()
34 old_buf = io.BytesIO()
35 with open(path, "rb") as f:
35 with open(path, "rb") as f:
36 old_buf.write(f.read())
36 old_buf.write(f.read())
37 old_buf.seek(0)
37 old_buf.seek(0)
38 old = tarfile.open(fileobj=old_buf, mode="r:gz")
38 old = tarfile.open(fileobj=old_buf, mode="r:gz")
39
39
40 buf = io.BytesIO()
40 buf = io.BytesIO()
41 new = tarfile.open(fileobj=buf, mode="w", format=tarfile.GNU_FORMAT)
41 new = tarfile.open(fileobj=buf, mode="w", format=tarfile.GNU_FORMAT)
42 for i, m in enumerate(old):
42 for i, m in enumerate(old):
43 data = None
43 data = None
44 # mutation does not work, copy
44 # mutation does not work, copy
45 if m.name.endswith('.DS_Store'):
45 if m.name.endswith('.DS_Store'):
46 continue
46 continue
47 m2 = tarfile.TarInfo(m.name)
47 m2 = tarfile.TarInfo(m.name)
48 m2.mtime = min(timestamp, m.mtime)
48 m2.mtime = min(timestamp, m.mtime)
49 m2.size = m.size
49 m2.size = m.size
50 m2.type = m.type
50 m2.type = m.type
51 m2.linkname = m.linkname
51 m2.linkname = m.linkname
52 m2.mode = m.mode
52 m2.mode = m.mode
53 if m.isdir():
53 if m.isdir():
54 new.addfile(m2)
54 new.addfile(m2)
55 else:
55 else:
56 data = old.extractfile(m)
56 data = old.extractfile(m)
57 new.addfile(m2, data)
57 new.addfile(m2, data)
58 new.close()
58 new.close()
59 old.close()
59 old.close()
60
60
61 buf.seek(0)
61 buf.seek(0)
62 with open(path, "wb") as f:
62 with open(path, "wb") as f:
63 with gzip.GzipFile('', "wb", fileobj=f, mtime=timestamp) as gzf:
63 with gzip.GzipFile('', "wb", fileobj=f, mtime=timestamp) as gzf:
64 gzf.write(buf.read())
64 gzf.write(buf.read())
65
65
66 # checks the archive is valid.
66 # checks the archive is valid.
67 archive = tarfile.open(path)
67 archive = tarfile.open(path)
68 names = archive.getnames()
68 names = archive.getnames()
1 NO CONTENT: modified file
NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
General Comments 0
You need to be logged in to leave comments. Login now