reformat

IPython/core/completer.py

@@ -1,3389 +1,3389
 """Completion for IPython.
 
 This module started as fork of the rlcompleter module in the Python standard
 library.  The original enhancements made to rlcompleter have been sent
 upstream and were accepted as of Python 2.3,
 
 This module now support a wide variety of completion mechanism both available
 for normal classic Python code, as well as completer for IPython specific
 Syntax like magics.
 
 Latex and Unicode completion
 ============================
 
 IPython and compatible frontends not only can complete your code, but can help
 you to input a wide range of characters. In particular we allow you to insert
 a unicode character using the tab completion mechanism.
 
 Forward latex/unicode completion
 --------------------------------
 
 Forward completion allows you to easily type a unicode character using its latex
 name, or unicode long description. To do so type a backslash follow by the
 relevant name and press tab:
 
 
 Using latex completion:
 
 .. code::
 
     \\alpha<tab>
     α
 
 or using unicode completion:
 
 
 .. code::
 
     \\GREEK SMALL LETTER ALPHA<tab>
     α
 
 
 Only valid Python identifiers will complete. Combining characters (like arrow or
 dots) are also available, unlike latex they need to be put after the their
 counterpart that is to say, ``F\\\\vec<tab>`` is correct, not ``\\\\vec<tab>F``.
 
 Some browsers are known to display combining characters incorrectly.
 
 Backward latex completion
 -------------------------
 
 It is sometime challenging to know how to type a character, if you are using
 IPython, or any compatible frontend you can prepend backslash to the character
 and press :kbd:`Tab` to expand it to its latex form.
 
 .. code::
 
     \\α<tab>
     \\alpha
 
 
 Both forward and backward completions can be deactivated by setting the
 :std:configtrait:`Completer.backslash_combining_completions` option to
 ``False``.
 
 
 Experimental
 ============
 
 Starting with IPython 6.0, this module can make use of the Jedi library to
 generate completions both using static analysis of the code, and dynamically
 inspecting multiple namespaces. Jedi is an autocompletion and static analysis
 for Python. The APIs attached to this new mechanism is unstable and will
 raise unless use in an :any:`provisionalcompleter` context manager.
 
 You will find that the following are experimental:
 
     - :any:`provisionalcompleter`
     - :any:`IPCompleter.completions`
     - :any:`Completion`
     - :any:`rectify_completions`
 
 .. note::
 
     better name for :any:`rectify_completions` ?
 
 We welcome any feedback on these new API, and we also encourage you to try this
 module in debug mode (start IPython with ``--Completer.debug=True``) in order
 to have extra logging information if :any:`jedi` is crashing, or if current
 IPython completer pending deprecations are returning results not yet handled
 by :any:`jedi`
 
 Using Jedi for tab completion allow snippets like the following to work without
 having to execute any code:
 
    >>> myvar = ['hello', 42]
    ... myvar[1].bi<tab>
 
 Tab completion will be able to infer that ``myvar[1]`` is a real number without
 executing almost any code unlike the deprecated :any:`IPCompleter.greedy`
 option.
 
 Be sure to update :any:`jedi` to the latest stable version or to try the
 current development version to get better completions.
 
 Matchers
 ========
 
 All completions routines are implemented using unified *Matchers* API.
 The matchers API is provisional and subject to change without notice.
 
 The built-in matchers include:
 
 - :any:`IPCompleter.dict_key_matcher`:  dictionary key completions,
 - :any:`IPCompleter.magic_matcher`: completions for magics,
 - :any:`IPCompleter.unicode_name_matcher`,
   :any:`IPCompleter.fwd_unicode_matcher`
   and :any:`IPCompleter.latex_name_matcher`: see `Forward latex/unicode completion`_,
 - :any:`back_unicode_name_matcher` and :any:`back_latex_name_matcher`: see `Backward latex completion`_,
 - :any:`IPCompleter.file_matcher`: paths to files and directories,
 - :any:`IPCompleter.python_func_kw_matcher` - function keywords,
 - :any:`IPCompleter.python_matches` - globals and attributes (v1 API),
 - ``IPCompleter.jedi_matcher`` - static analysis with Jedi,
 - :any:`IPCompleter.custom_completer_matcher` - pluggable completer with a default
   implementation in :any:`InteractiveShell` which uses IPython hooks system
   (`complete_command`) with string dispatch (including regular expressions).
   Differently to other matchers, ``custom_completer_matcher`` will not suppress
   Jedi results to match behaviour in earlier IPython versions.
 
 Custom matchers can be added by appending to ``IPCompleter.custom_matchers`` list.
 
 Matcher API
 -----------
 
 Simplifying some details, the ``Matcher`` interface can described as
 
 .. code-block::
 
     MatcherAPIv1 = Callable[[str], list[str]]
     MatcherAPIv2 = Callable[[CompletionContext], SimpleMatcherResult]
 
     Matcher = MatcherAPIv1 | MatcherAPIv2
 
 The ``MatcherAPIv1`` reflects the matcher API as available prior to IPython 8.6.0
 and remains supported as a simplest way for generating completions. This is also
 currently the only API supported by the IPython hooks system `complete_command`.
 
 To distinguish between matcher versions ``matcher_api_version`` attribute is used.
 More precisely, the API allows to omit ``matcher_api_version`` for v1 Matchers,
 and requires a literal ``2`` for v2 Matchers.
 
 Once the API stabilises future versions may relax the requirement for specifying
 ``matcher_api_version`` by switching to :any:`functools.singledispatch`, therefore
 please do not rely on the presence of ``matcher_api_version`` for any purposes.
 
 Suppression of competing matchers
 ---------------------------------
 
 By default results from all matchers are combined, in the order determined by
 their priority. Matchers can request to suppress results from subsequent
 matchers by setting ``suppress`` to ``True`` in the ``MatcherResult``.
 
 When multiple matchers simultaneously request suppression, the results from of
 the matcher with higher priority will be returned.
 
 Sometimes it is desirable to suppress most but not all other matchers;
 this can be achieved by adding a set of identifiers of matchers which
 should not be suppressed to ``MatcherResult`` under ``do_not_suppress`` key.
 
 The suppression behaviour can is user-configurable via
 :std:configtrait:`IPCompleter.suppress_competing_matchers`.
 """
 
 
 # Copyright (c) IPython Development Team.
 # Distributed under the terms of the Modified BSD License.
 #
 # Some of this code originated from rlcompleter in the Python standard library
 # Copyright (C) 2001 Python Software Foundation, www.python.org
 
 from __future__ import annotations
 import builtins as builtin_mod
 import enum
 import glob
 import inspect
 import itertools
 import keyword
 import os
 import re
 import string
 import sys
 import tokenize
 import time
 import unicodedata
 import uuid
 import warnings
 from ast import literal_eval
 from collections import defaultdict
 from contextlib import contextmanager
 from dataclasses import dataclass
 from functools import cached_property, partial
 from types import SimpleNamespace
 from typing import (
     Iterable,
     Iterator,
     List,
     Tuple,
     Union,
     Any,
     Sequence,
     Dict,
     Optional,
     TYPE_CHECKING,
     Set,
     Sized,
     TypeVar,
     Literal,
 )
 
 from IPython.core.guarded_eval import guarded_eval, EvaluationContext
 from IPython.core.error import TryNext
 from IPython.core.inputtransformer2 import ESC_MAGIC
 from IPython.core.latex_symbols import latex_symbols, reverse_latex_symbol
 from IPython.core.oinspect import InspectColors
 from IPython.testing.skipdoctest import skip_doctest
 from IPython.utils import generics
 from IPython.utils.decorators import sphinx_options
 from IPython.utils.dir2 import dir2, get_real_method
 from IPython.utils.docs import GENERATING_DOCUMENTATION
 from IPython.utils.path import ensure_dir_exists
 from IPython.utils.process import arg_split
 from traitlets import (
     Bool,
     Enum,
     Int,
     List as ListTrait,
     Unicode,
     Dict as DictTrait,
     Union as UnionTrait,
     observe,
 )
 from traitlets.config.configurable import Configurable
 
 import __main__
 
 # skip module docstests
 __skip_doctest__ = True
 
 
 try:
     import jedi
     jedi.settings.case_insensitive_completion = False
     import jedi.api.helpers
     import jedi.api.classes
     JEDI_INSTALLED = True
 except ImportError:
     JEDI_INSTALLED = False
 
 
 if TYPE_CHECKING or GENERATING_DOCUMENTATION and sys.version_info >= (3, 11):
     from typing import cast
     from typing_extensions import TypedDict, NotRequired, Protocol, TypeAlias, TypeGuard
 else:
     from typing import Generic
 
     def cast(type_, obj):
         """Workaround for `TypeError: MatcherAPIv2() takes no arguments`"""
         return obj
 
     # do not require on runtime
     NotRequired = Tuple  # requires Python >=3.11
     TypedDict = Dict  # by extension of `NotRequired` requires 3.11 too
     Protocol = object  # requires Python >=3.8
     TypeAlias = Any  # requires Python >=3.10
     TypeGuard = Generic  # requires Python >=3.10
 if GENERATING_DOCUMENTATION:
     from typing import TypedDict
 
 # -----------------------------------------------------------------------------
 # Globals
 #-----------------------------------------------------------------------------
 
 # ranges where we have most of the valid unicode names. We could be more finer
 # grained but is it worth it for performance  While unicode have character in the
 # range 0, 0x110000, we seem to have name for about 10% of those. (131808 as I
 # write this). With below range we cover them all, with a density of ~67%
 # biggest next gap we consider only adds up about 1% density and there are 600
 # gaps that would need hard coding.
 _UNICODE_RANGES = [(32, 0x323B0), (0xE0001, 0xE01F0)]
 
 # Public API
 __all__ = ["Completer", "IPCompleter"]
 
 if sys.platform == 'win32':
     PROTECTABLES = ' '
 else:
     PROTECTABLES = ' ()[]{}?=\\|;:\'#*"^&'
 
 # Protect against returning an enormous number of completions which the frontend
 # may have trouble processing.
 MATCHES_LIMIT = 500
 
 # Completion type reported when no type can be inferred.
 _UNKNOWN_TYPE = "<unknown>"
 
 # sentinel value to signal lack of a match
 not_found = object()
 
 class ProvisionalCompleterWarning(FutureWarning):
     """
     Exception raise by an experimental feature in this module.
 
     Wrap code in :any:`provisionalcompleter` context manager if you
     are certain you want to use an unstable feature.
     """
     pass
 
 warnings.filterwarnings('error', category=ProvisionalCompleterWarning)
 
 
 @skip_doctest
 @contextmanager
 def provisionalcompleter(action='ignore'):
     """
     This context manager has to be used in any place where unstable completer
     behavior and API may be called.
 
     >>> with provisionalcompleter():
     ...     completer.do_experimental_things() # works
 
     >>> completer.do_experimental_things() # raises.
 
     .. note::
 
         Unstable
 
         By using this context manager you agree that the API in use may change
         without warning, and that you won't complain if they do so.
 
         You also understand that, if the API is not to your liking, you should report
         a bug to explain your use case upstream.
 
         We'll be happy to get your feedback, feature requests, and improvements on
         any of the unstable APIs!
     """
     with warnings.catch_warnings():
         warnings.filterwarnings(action, category=ProvisionalCompleterWarning)
         yield
 
 
 def has_open_quotes(s):
     """Return whether a string has open quotes.
 
     This simply counts whether the number of quote characters of either type in
     the string is odd.
 
     Returns
     -------
     If there is an open quote, the quote character is returned.  Else, return
     False.
     """
     # We check " first, then ', so complex cases with nested quotes will get
     # the " to take precedence.
     if s.count('"') % 2:
         return '"'
     elif s.count("'") % 2:
         return "'"
     else:
         return False
 
 
 def protect_filename(s, protectables=PROTECTABLES):
     """Escape a string to protect certain characters."""
     if set(s) & set(protectables):
         if sys.platform == "win32":
             return '"' + s + '"'
         else:
             return "".join(("\\" + c if c in protectables else c) for c in s)
     else:
         return s
 
 
 def expand_user(path:str) -> Tuple[str, bool, str]:
     """Expand ``~``-style usernames in strings.
 
     This is similar to :func:`os.path.expanduser`, but it computes and returns
     extra information that will be useful if the input was being used in
     computing completions, and you wish to return the completions with the
     original '~' instead of its expanded value.
 
     Parameters
     ----------
     path : str
         String to be expanded.  If no ~ is present, the output is the same as the
         input.
 
     Returns
     -------
     newpath : str
         Result of ~ expansion in the input path.
     tilde_expand : bool
         Whether any expansion was performed or not.
     tilde_val : str
         The value that ~ was replaced with.
     """
     # Default values
     tilde_expand = False
     tilde_val = ''
     newpath = path
 
     if path.startswith('~'):
         tilde_expand = True
         rest = len(path)-1
         newpath = os.path.expanduser(path)
         if rest:
             tilde_val = newpath[:-rest]
         else:
             tilde_val = newpath
 
     return newpath, tilde_expand, tilde_val
 
 
 def compress_user(path:str, tilde_expand:bool, tilde_val:str) -> str:
     """Does the opposite of expand_user, with its outputs.
     """
     if tilde_expand:
         return path.replace(tilde_val, '~')
     else:
         return path
 
 
 def completions_sorting_key(word):
     """key for sorting completions
 
     This does several things:
 
     - Demote any completions starting with underscores to the end
     - Insert any %magic and %%cellmagic completions in the alphabetical order
       by their name
     """
     prio1, prio2 = 0, 0
 
     if word.startswith('__'):
         prio1 = 2
     elif word.startswith('_'):
         prio1 = 1
 
     if word.endswith('='):
         prio1 = -1
 
     if word.startswith('%%'):
         # If there's another % in there, this is something else, so leave it alone
         if not "%" in word[2:]:
             word = word[2:]
             prio2 = 2
     elif word.startswith('%'):
         if not "%" in word[1:]:
             word = word[1:]
             prio2 = 1
 
     return prio1, word, prio2
 
 
 class _FakeJediCompletion:
     """
     This is a workaround to communicate to the UI that Jedi has crashed and to
     report a bug. Will be used only id :any:`IPCompleter.debug` is set to true.
 
     Added in IPython 6.0 so should likely be removed for 7.0
 
     """
 
     def __init__(self, name):
 
         self.name = name
         self.complete = name
         self.type = 'crashed'
         self.name_with_symbols = name
         self.signature = ""
         self._origin = "fake"
         self.text = "crashed"
 
     def __repr__(self):
         return '<Fake completion object jedi has crashed>'
 
 
 _JediCompletionLike = Union["jedi.api.Completion", _FakeJediCompletion]
 
 
 class Completion:
     """
     Completion object used and returned by IPython completers.
 
     .. warning::
 
         Unstable
 
         This function is unstable, API may change without warning.
         It will also raise unless use in proper context manager.
 
     This act as a middle ground :any:`Completion` object between the
     :any:`jedi.api.classes.Completion` object and the Prompt Toolkit completion
     object. While Jedi need a lot of information about evaluator and how the
     code should be ran/inspected, PromptToolkit (and other frontend) mostly
     need user facing information.
 
     - Which range should be replaced replaced by what.
     - Some metadata (like completion type), or meta information to displayed to
       the use user.
 
     For debugging purpose we can also store the origin of the completion (``jedi``,
     ``IPython.python_matches``, ``IPython.magics_matches``...).
     """
 
     __slots__ = ['start', 'end', 'text', 'type', 'signature', '_origin']
 
     def __init__(
         self,
         start: int,
         end: int,
         text: str,
         *,
         type: Optional[str] = None,
         _origin="",
         signature="",
     ) -> None:
         warnings.warn(
             "``Completion`` is a provisional API (as of IPython 6.0). "
             "It may change without warnings. "
             "Use in corresponding context manager.",
             category=ProvisionalCompleterWarning,
             stacklevel=2,
         )
 
         self.start = start
         self.end = end
         self.text = text
         self.type = type
         self.signature = signature
         self._origin = _origin
 
     def __repr__(self):
         return '<Completion start=%s end=%s text=%r type=%r, signature=%r,>' % \
                 (self.start, self.end, self.text, self.type or '?', self.signature or '?')
 
     def __eq__(self, other) -> bool:
         """
         Equality and hash do not hash the type (as some completer may not be
         able to infer the type), but are use to (partially) de-duplicate
         completion.
 
         Completely de-duplicating completion is a bit tricker that just
         comparing as it depends on surrounding text, which Completions are not
         aware of.
         """
         return self.start == other.start and \
             self.end == other.end and \
             self.text == other.text
 
     def __hash__(self):
         return hash((self.start, self.end, self.text))
 
 
 class SimpleCompletion:
     """Completion item to be included in the dictionary returned by new-style Matcher (API v2).
 
     .. warning::
 
         Provisional
 
         This class is used to describe the currently supported attributes of
         simple completion items, and any additional implementation details
         should not be relied on. Additional attributes may be included in
         future versions, and meaning of text disambiguated from the current
         dual meaning of "text to insert" and "text to used as a label".
     """
 
     __slots__ = ["text", "type"]
 
     def __init__(self, text: str, *, type: Optional[str] = None):
         self.text = text
         self.type = type
 
     def __repr__(self):
         return f"<SimpleCompletion text={self.text!r} type={self.type!r}>"
 
 
 class _MatcherResultBase(TypedDict):
     """Definition of dictionary to be returned by new-style Matcher (API v2)."""
 
     #: Suffix of the provided ``CompletionContext.token``, if not given defaults to full token.
     matched_fragment: NotRequired[str]
 
     #: Whether to suppress results from all other matchers (True), some
     #: matchers (set of identifiers) or none (False); default is False.
     suppress: NotRequired[Union[bool, Set[str]]]
 
     #: Identifiers of matchers which should NOT be suppressed when this matcher
     #: requests to suppress all other matchers; defaults to an empty set.
     do_not_suppress: NotRequired[Set[str]]
 
     #: Are completions already ordered and should be left as-is? default is False.
     ordered: NotRequired[bool]
 
 
 @sphinx_options(show_inherited_members=True, exclude_inherited_from=["dict"])
 class SimpleMatcherResult(_MatcherResultBase, TypedDict):
     """Result of new-style completion matcher."""
 
     # note: TypedDict is added again to the inheritance chain
     # in order to get __orig_bases__ for documentation
 
     #: List of candidate completions
     completions: Sequence[SimpleCompletion] | Iterator[SimpleCompletion]
 
 
 class _JediMatcherResult(_MatcherResultBase):
     """Matching result returned by Jedi (will be processed differently)"""
 
     #: list of candidate completions
     completions: Iterator[_JediCompletionLike]
 
 
 AnyMatcherCompletion = Union[_JediCompletionLike, SimpleCompletion]
 AnyCompletion = TypeVar("AnyCompletion", AnyMatcherCompletion, Completion)
 
 
 @dataclass
 class CompletionContext:
     """Completion context provided as an argument to matchers in the Matcher API v2."""
 
     # rationale: many legacy matchers relied on completer state (`self.text_until_cursor`)
     # which was not explicitly visible as an argument of the matcher, making any refactor
     # prone to errors; by explicitly passing `cursor_position` we can decouple the matchers
     # from the completer, and make substituting them in sub-classes easier.
 
     #: Relevant fragment of code directly preceding the cursor.
     #: The extraction of token is implemented via splitter heuristic
     #: (following readline behaviour for legacy reasons), which is user configurable
     #: (by switching the greedy mode).
     token: str
 
     #: The full available content of the editor or buffer
     full_text: str
 
     #: Cursor position in the line (the same for ``full_text`` and ``text``).
     cursor_position: int
 
     #: Cursor line in ``full_text``.
     cursor_line: int
 
     #: The maximum number of completions that will be used downstream.
     #: Matchers can use this information to abort early.
     #: The built-in Jedi matcher is currently excepted from this limit.
     # If not given, return all possible completions.
     limit: Optional[int]
 
     @cached_property
     def text_until_cursor(self) -> str:
         return self.line_with_cursor[: self.cursor_position]
 
     @cached_property
     def line_with_cursor(self) -> str:
         return self.full_text.split("\n")[self.cursor_line]
 
 
 #: Matcher results for API v2.
 MatcherResult = Union[SimpleMatcherResult, _JediMatcherResult]
 
 
 class _MatcherAPIv1Base(Protocol):
     def __call__(self, text: str) -> List[str]:
         """Call signature."""
         ...
 
     #: Used to construct the default matcher identifier
     __qualname__: str
 
 
 class _MatcherAPIv1Total(_MatcherAPIv1Base, Protocol):
     #: API version
     matcher_api_version: Optional[Literal[1]]
 
     def __call__(self, text: str) -> List[str]:
         """Call signature."""
         ...
 
 
 #: Protocol describing Matcher API v1.
 MatcherAPIv1: TypeAlias = Union[_MatcherAPIv1Base, _MatcherAPIv1Total]
 
 
 class MatcherAPIv2(Protocol):
     """Protocol describing Matcher API v2."""
 
     #: API version
     matcher_api_version: Literal[2] = 2
 
     def __call__(self, context: CompletionContext) -> MatcherResult:
         """Call signature."""
         ...
 
     #: Used to construct the default matcher identifier
     __qualname__: str
 
 
 Matcher: TypeAlias = Union[MatcherAPIv1, MatcherAPIv2]
 
 
 def _is_matcher_v1(matcher: Matcher) -> TypeGuard[MatcherAPIv1]:
     api_version = _get_matcher_api_version(matcher)
     return api_version == 1
 
 
 def _is_matcher_v2(matcher: Matcher) -> TypeGuard[MatcherAPIv2]:
     api_version = _get_matcher_api_version(matcher)
     return api_version == 2
 
 
 def _is_sizable(value: Any) -> TypeGuard[Sized]:
     """Determines whether objects is sizable"""
     return hasattr(value, "__len__")
 
 
 def _is_iterator(value: Any) -> TypeGuard[Iterator]:
     """Determines whether objects is sizable"""
     return hasattr(value, "__next__")
 
 
 def has_any_completions(result: MatcherResult) -> bool:
     """Check if any result includes any completions."""
     completions = result["completions"]
     if _is_sizable(completions):
         return len(completions) != 0
     if _is_iterator(completions):
         try:
             old_iterator = completions
             first = next(old_iterator)
             result["completions"] = cast(
                 Iterator[SimpleCompletion],
                 itertools.chain([first], old_iterator),
             )
             return True
         except StopIteration:
             return False
     raise ValueError(
         "Completions returned by matcher need to be an Iterator or a Sizable"
     )
 
 
 def completion_matcher(
     *,
     priority: Optional[float] = None,
     identifier: Optional[str] = None,
     api_version: int = 1,
 ):
     """Adds attributes describing the matcher.
 
     Parameters
     ----------
     priority : Optional[float]
         The priority of the matcher, determines the order of execution of matchers.
         Higher priority means that the matcher will be executed first. Defaults to 0.
     identifier : Optional[str]
         identifier of the matcher allowing users to modify the behaviour via traitlets,
         and also used to for debugging (will be passed as ``origin`` with the completions).
 
         Defaults to matcher function's ``__qualname__`` (for example,
         ``IPCompleter.file_matcher`` for the built-in matched defined
         as a ``file_matcher`` method of the ``IPCompleter`` class).
     api_version: Optional[int]
         version of the Matcher API used by this matcher.
         Currently supported values are 1 and 2.
         Defaults to 1.
     """
 
     def wrapper(func: Matcher):
         func.matcher_priority = priority or 0  # type: ignore
         func.matcher_identifier = identifier or func.__qualname__  # type: ignore
         func.matcher_api_version = api_version  # type: ignore
         if TYPE_CHECKING:
             if api_version == 1:
                 func = cast(MatcherAPIv1, func)
             elif api_version == 2:
                 func = cast(MatcherAPIv2, func)
         return func
 
     return wrapper
 
 
 def _get_matcher_priority(matcher: Matcher):
     return getattr(matcher, "matcher_priority", 0)
 
 
 def _get_matcher_id(matcher: Matcher):
     return getattr(matcher, "matcher_identifier", matcher.__qualname__)
 
 
 def _get_matcher_api_version(matcher):
     return getattr(matcher, "matcher_api_version", 1)
 
 
 context_matcher = partial(completion_matcher, api_version=2)
 
 
 _IC = Iterable[Completion]
 
 
 def _deduplicate_completions(text: str, completions: _IC)-> _IC:
     """
     Deduplicate a set of completions.
 
     .. warning::
 
         Unstable
 
         This function is unstable, API may change without warning.
 
     Parameters
     ----------
     text : str
         text that should be completed.
     completions : Iterator[Completion]
         iterator over the completions to deduplicate
 
     Yields
     ------
     `Completions` objects
     Completions coming from multiple sources, may be different but end up having
     the same effect when applied to ``text``. If this is the case, this will
     consider completions as equal and only emit the first encountered.
     Not folded in `completions()` yet for debugging purpose, and to detect when
     the IPython completer does return things that Jedi does not, but should be
     at some point.
     """
     completions = list(completions)
     if not completions:
         return
 
     new_start = min(c.start for c in completions)
     new_end = max(c.end for c in completions)
 
     seen = set()
     for c in completions:
         new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
         if new_text not in seen:
             yield c
             seen.add(new_text)
 
 
 def rectify_completions(text: str, completions: _IC, *, _debug: bool = False) -> _IC:
     """
     Rectify a set of completions to all have the same ``start`` and ``end``
 
     .. warning::
 
         Unstable
 
         This function is unstable, API may change without warning.
         It will also raise unless use in proper context manager.
 
     Parameters
     ----------
     text : str
         text that should be completed.
     completions : Iterator[Completion]
         iterator over the completions to rectify
     _debug : bool
         Log failed completion
 
     Notes
     -----
     :any:`jedi.api.classes.Completion` s returned by Jedi may not have the same start and end, though
     the Jupyter Protocol requires them to behave like so. This will readjust
     the completion to have the same ``start`` and ``end`` by padding both
     extremities with surrounding text.
 
     During stabilisation should support a ``_debug`` option to log which
     completion are return by the IPython completer and not found in Jedi in
     order to make upstream bug report.
     """
     warnings.warn("`rectify_completions` is a provisional API (as of IPython 6.0). "
                  "It may change without warnings. "
                  "Use in corresponding context manager.",
                   category=ProvisionalCompleterWarning, stacklevel=2)
 
     completions = list(completions)
     if not completions:
         return
     starts = (c.start for c in completions)
     ends = (c.end for c in completions)
 
     new_start = min(starts)
     new_end = max(ends)
 
     seen_jedi = set()
     seen_python_matches = set()
     for c in completions:
         new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
         if c._origin == 'jedi':
             seen_jedi.add(new_text)
         elif c._origin == "IPCompleter.python_matcher":
             seen_python_matches.add(new_text)
         yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin, signature=c.signature)
     diff = seen_python_matches.difference(seen_jedi)
     if diff and _debug:
         print('IPython.python matches have extras:', diff)
 
 
 if sys.platform == 'win32':
     DELIMS = ' \t\n`!@#$^&*()=+[{]}|;\'",<>?'
 else:
     DELIMS = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
 
 GREEDY_DELIMS = ' =\r\n'
 
 
 class CompletionSplitter(object):
     """An object to split an input line in a manner similar to readline.
 
     By having our own implementation, we can expose readline-like completion in
     a uniform manner to all frontends.  This object only needs to be given the
     line of text to be split and the cursor position on said line, and it
     returns the 'word' to be completed on at the cursor after splitting the
     entire line.
 
     What characters are used as splitting delimiters can be controlled by
     setting the ``delims`` attribute (this is a property that internally
     automatically builds the necessary regular expression)"""
 
     # Private interface
 
     # A string of delimiter characters.  The default value makes sense for
     # IPython's most typical usage patterns.
     _delims = DELIMS
 
     # The expression (a normal string) to be compiled into a regular expression
     # for actual splitting.  We store it as an attribute mostly for ease of
     # debugging, since this type of code can be so tricky to debug.
     _delim_expr = None
 
     # The regular expression that does the actual splitting
     _delim_re = None
 
     def __init__(self, delims=None):
         delims = CompletionSplitter._delims if delims is None else delims
         self.delims = delims
 
     @property
     def delims(self):
         """Return the string of delimiter characters."""
         return self._delims
 
     @delims.setter
     def delims(self, delims):
         """Set the delimiters for line splitting."""
         expr = '[' + ''.join('\\'+ c for c in delims) + ']'
         self._delim_re = re.compile(expr)
         self._delims = delims
         self._delim_expr = expr
 
     def split_line(self, line, cursor_pos=None):
         """Split a line of text with a cursor at the given position.
         """
         l = line if cursor_pos is None else line[:cursor_pos]
         return self._delim_re.split(l)[-1]
 
 
 
 class Completer(Configurable):
 
     greedy = Bool(
         False,
         help="""Activate greedy completion.
 
         .. deprecated:: 8.8
             Use :std:configtrait:`Completer.evaluation` and :std:configtrait:`Completer.auto_close_dict_keys` instead.
 
         When enabled in IPython 8.8 or newer, changes configuration as follows:
 
         - ``Completer.evaluation = 'unsafe'``
         - ``Completer.auto_close_dict_keys = True``
         """,
     ).tag(config=True)
 
     evaluation = Enum(
         ("forbidden", "minimal", "limited", "unsafe", "dangerous"),
         default_value="limited",
         help="""Policy for code evaluation under completion.
 
         Successive options allow to enable more eager evaluation for better
         completion suggestions, including for nested dictionaries, nested lists,
         or even results of function calls.
         Setting ``unsafe`` or higher can lead to evaluation of arbitrary user
         code on :kbd:`Tab` with potentially unwanted or dangerous side effects.
 
         Allowed values are:
 
         - ``forbidden``: no evaluation of code is permitted,
         - ``minimal``: evaluation of literals and access to built-in namespace;
           no item/attribute evaluationm no access to locals/globals,
           no evaluation of any operations or comparisons.
         - ``limited``: access to all namespaces, evaluation of hard-coded methods
           (for example: :any:`dict.keys`, :any:`object.__getattr__`,
           :any:`object.__getitem__`) on allow-listed objects (for example:
           :any:`dict`, :any:`list`, :any:`tuple`, ``pandas.Series``),
         - ``unsafe``: evaluation of all methods and function calls but not of
           syntax with side-effects like `del x`,
         - ``dangerous``: completely arbitrary evaluation.
         """,
     ).tag(config=True)
 
     use_jedi = Bool(default_value=JEDI_INSTALLED,
                     help="Experimental: Use Jedi to generate autocompletions. "
                     "Default to True if jedi is installed.").tag(config=True)
 
     jedi_compute_type_timeout = Int(default_value=400,
         help="""Experimental: restrict time (in milliseconds) during which Jedi can compute types.
         Set to 0 to stop computing types. Non-zero value lower than 100ms may hurt
         performance by preventing jedi to build its cache.
         """).tag(config=True)
 
     debug = Bool(default_value=False,
                  help='Enable debug for the Completer. Mostly print extra '
                       'information for experimental jedi integration.')\
                       .tag(config=True)
 
     backslash_combining_completions = Bool(True,
         help="Enable unicode completions, e.g. \\alpha<tab> . "
              "Includes completion of latex commands, unicode names, and expanding "
              "unicode characters back to latex commands.").tag(config=True)
 
     auto_close_dict_keys = Bool(
         False,
         help="""
         Enable auto-closing dictionary keys.
 
         When enabled string keys will be suffixed with a final quote
         (matching the opening quote), tuple keys will also receive a
         separating comma if needed, and keys which are final will
         receive a closing bracket (``]``).
         """,
     ).tag(config=True)
 
     def __init__(self, namespace=None, global_namespace=None, **kwargs):
         """Create a new completer for the command line.
 
         Completer(namespace=ns, global_namespace=ns2) -> completer instance.
 
         If unspecified, the default namespace where completions are performed
         is __main__ (technically, __main__.__dict__). Namespaces should be
         given as dictionaries.
 
         An optional second namespace can be given.  This allows the completer
         to handle cases where both the local and global scopes need to be
         distinguished.
         """
 
         # Don't bind to namespace quite yet, but flag whether the user wants a
         # specific namespace or to use __main__.__dict__. This will allow us
         # to bind to __main__.__dict__ at completion time, not now.
         if namespace is None:
             self.use_main_ns = True
         else:
             self.use_main_ns = False
             self.namespace = namespace
 
         # The global namespace, if given, can be bound directly
         if global_namespace is None:
             self.global_namespace = {}
         else:
             self.global_namespace = global_namespace
 
         self.custom_matchers = []
 
         super(Completer, self).__init__(**kwargs)
 
     def complete(self, text, state):
         """Return the next possible completion for 'text'.
 
         This is called successively with state == 0, 1, 2, ... until it
         returns None.  The completion should begin with 'text'.
 
         """
         if self.use_main_ns:
             self.namespace = __main__.__dict__
 
         if state == 0:
             if "." in text:
                 self.matches = self.attr_matches(text)
             else:
                 self.matches = self.global_matches(text)
         try:
             return self.matches[state]
         except IndexError:
             return None
 
     def global_matches(self, text):
         """Compute matches when text is a simple name.
 
         Return a list of all keywords, built-in functions and names currently
         defined in self.namespace or self.global_namespace that match.
 
         """
         matches = []
         match_append = matches.append
         n = len(text)
         for lst in [
             keyword.kwlist,
             builtin_mod.__dict__.keys(),
             list(self.namespace.keys()),
             list(self.global_namespace.keys()),
         ]:
             for word in lst:
                 if word[:n] == text and word != "__builtins__":
                     match_append(word)
 
         snake_case_re = re.compile(r"[^_]+(_[^_]+)+?\Z")
         for lst in [list(self.namespace.keys()), list(self.global_namespace.keys())]:
             shortened = {
                 "_".join([sub[0] for sub in word.split("_")]): word
                 for word in lst
                 if snake_case_re.match(word)
             }
             for word in shortened.keys():
                 if word[:n] == text and word != "__builtins__":
                     match_append(shortened[word])
         return matches
 
     def attr_matches(self, text):
         """Compute matches when text contains a dot.
 
         Assuming the text is of the form NAME.NAME....[NAME], and is
         evaluatable in self.namespace or self.global_namespace, it will be
         evaluated and its attributes (as revealed by dir()) are used as
         possible completions.  (For class instances, class members are
         also considered.)
 
         WARNING: this can still invoke arbitrary C code, if an object
         with a __getattr__ hook is evaluated.
 
         """
         return self._attr_matches(text)[0]
 
     def _attr_matches(self, text, include_prefix=True) -> Tuple[Sequence[str], str]:
         m2 = re.match(r"(.+)\.(\w*)$", self.line_buffer)
         if not m2:
             return [], ""
         expr, attr = m2.group(1, 2)
 
         obj = self._evaluate_expr(expr)
 
         if obj is not_found:
             return [], ""
 
         if self.limit_to__all__ and hasattr(obj, '__all__'):
             words = get__all__entries(obj)
         else:
             words = dir2(obj)
 
         try:
             words = generics.complete_object(obj, words)
         except TryNext:
             pass
         except AssertionError:
             raise
         except Exception:
             # Silence errors from completion function
             pass
         # Build match list to return
         n = len(attr)
 
         # Note: ideally we would just return words here and the prefix
         # reconciliator would know that we intend to append to rather than
         # replace the input text; this requires refactoring to return range
         # which ought to be replaced (as does jedi).
         if include_prefix:
             tokens = _parse_tokens(expr)
             rev_tokens = reversed(tokens)
             skip_over = {tokenize.ENDMARKER, tokenize.NEWLINE}
             name_turn = True
 
             parts = []
             for token in rev_tokens:
                 if token.type in skip_over:
                     continue
                 if token.type == tokenize.NAME and name_turn:
                     parts.append(token.string)
                     name_turn = False
                 elif (
                     token.type == tokenize.OP and token.string == "." and not name_turn
                 ):
                     parts.append(token.string)
                     name_turn = True
                 else:
                     # short-circuit if not empty nor name token
                     break
 
             prefix_after_space = "".join(reversed(parts))
         else:
             prefix_after_space = ""
 
         return (
             ["%s.%s" % (prefix_after_space, w) for w in words if w[:n] == attr],
             "." + attr,
         )
 
     def _evaluate_expr(self, expr):
         obj = not_found
         done = False
         while not done and expr:
             try:
                 obj = guarded_eval(
                     expr,
                     EvaluationContext(
                         globals=self.global_namespace,
                         locals=self.namespace,
                         evaluation=self.evaluation,
                     ),
                 )
                 done = True
             except Exception as e:
                 if self.debug:
                     print("Evaluation exception", e)
                 # trim the expression to remove any invalid prefix
                 # e.g. user starts `(d[`, so we get `expr = '(d'`,
                 # where parenthesis is not closed.
                 # TODO: make this faster by reusing parts of the computation?
                 expr = expr[1:]
         return obj
 
 def get__all__entries(obj):
     """returns the strings in the __all__ attribute"""
     try:
         words = getattr(obj, '__all__')
     except:
         return []
 
     return [w for w in words if isinstance(w, str)]
 
 
 class _DictKeyState(enum.Flag):
     """Represent state of the key match in context of other possible matches.
 
     - given `d1 = {'a': 1}` completion on `d1['<tab>` will yield `{'a': END_OF_ITEM}` as there is no tuple.
     - given `d2 = {('a', 'b'): 1}`: `d2['a', '<tab>` will yield `{'b': END_OF_TUPLE}` as there is no tuple members to add beyond `'b'`.
     - given `d3 = {('a', 'b'): 1}`: `d3['<tab>` will yield `{'a': IN_TUPLE}` as `'a'` can be added.
     - given `d4 = {'a': 1, ('a', 'b'): 2}`: `d4['<tab>` will yield `{'a': END_OF_ITEM & END_OF_TUPLE}`
     """
 
     BASELINE = 0
     END_OF_ITEM = enum.auto()
     END_OF_TUPLE = enum.auto()
     IN_TUPLE = enum.auto()
 
 
 def _parse_tokens(c):
     """Parse tokens even if there is an error."""
     tokens = []
     token_generator = tokenize.generate_tokens(iter(c.splitlines()).__next__)
     while True:
         try:
             tokens.append(next(token_generator))
         except tokenize.TokenError:
             return tokens
         except StopIteration:
             return tokens
 
 
 def _match_number_in_dict_key_prefix(prefix: str) -> Union[str, None]:
     """Match any valid Python numeric literal in a prefix of dictionary keys.
 
     References:
     - https://docs.python.org/3/reference/lexical_analysis.html#numeric-literals
     - https://docs.python.org/3/library/tokenize.html
     """
     if prefix[-1].isspace():
         # if user typed a space we do not have anything to complete
         # even if there was a valid number token before
         return None
     tokens = _parse_tokens(prefix)
     rev_tokens = reversed(tokens)
     skip_over = {tokenize.ENDMARKER, tokenize.NEWLINE}
     number = None
     for token in rev_tokens:
         if token.type in skip_over:
             continue
         if number is None:
             if token.type == tokenize.NUMBER:
                 number = token.string
                 continue
             else:
                 # we did not match a number
                 return None
         if token.type == tokenize.OP:
             if token.string == ",":
                 break
             if token.string in {"+", "-"}:
                 number = token.string + number
         else:
             return None
     return number
 
 
 _INT_FORMATS = {
     "0b": bin,
     "0o": oct,
     "0x": hex,
 }
 
 
 def match_dict_keys(
     keys: List[Union[str, bytes, Tuple[Union[str, bytes], ...]]],
     prefix: str,
     delims: str,
     extra_prefix: Optional[Tuple[Union[str, bytes], ...]] = None,
 ) -> Tuple[str, int, Dict[str, _DictKeyState]]:
     """Used by dict_key_matches, matching the prefix to a list of keys
 
     Parameters
     ----------
     keys
         list of keys in dictionary currently being completed.
     prefix
         Part of the text already typed by the user. E.g. `mydict[b'fo`
     delims
         String of delimiters to consider when finding the current key.
     extra_prefix : optional
         Part of the text already typed in multi-key index cases. E.g. for
         `mydict['foo', "bar", 'b`, this would be `('foo', 'bar')`.
 
     Returns
     -------
     A tuple of three elements: ``quote``, ``token_start``, ``matched``, with
     ``quote`` being the quote that need to be used to close current string.
     ``token_start`` the position where the replacement should start occurring,
     ``matches`` a dictionary of replacement/completion keys on keys and values
         indicating whether the state.
     """
     prefix_tuple = extra_prefix if extra_prefix else ()
 
     prefix_tuple_size = sum(
         [
             # for pandas, do not count slices as taking space
             not isinstance(k, slice)
             for k in prefix_tuple
         ]
     )
     text_serializable_types = (str, bytes, int, float, slice)
 
     def filter_prefix_tuple(key):
         # Reject too short keys
         if len(key) <= prefix_tuple_size:
             return False
         # Reject keys which cannot be serialised to text
         for k in key:
             if not isinstance(k, text_serializable_types):
                 return False
         # Reject keys that do not match the prefix
         for k, pt in zip(key, prefix_tuple):
             if k != pt and not isinstance(pt, slice):
                 return False
         # All checks passed!
         return True
 
-    filtered_key_is_final: Dict[
-        Union[str, bytes, int, float], _DictKeyState
-    ] = defaultdict(lambda: _DictKeyState.BASELINE)
+    filtered_key_is_final: Dict[Union[str, bytes, int, float], _DictKeyState] = (
+        defaultdict(lambda: _DictKeyState.BASELINE)
+    )
 
     for k in keys:
         # If at least one of the matches is not final, mark as undetermined.
         # This can happen with `d = {111: 'b', (111, 222): 'a'}` where
         # `111` appears final on first match but is not final on the second.
 
         if isinstance(k, tuple):
             if filter_prefix_tuple(k):
                 key_fragment = k[prefix_tuple_size]
                 filtered_key_is_final[key_fragment] |= (
                     _DictKeyState.END_OF_TUPLE
                     if len(k) == prefix_tuple_size + 1
                     else _DictKeyState.IN_TUPLE
                 )
         elif prefix_tuple_size > 0:
             # we are completing a tuple but this key is not a tuple,
             # so we should ignore it
             pass
         else:
             if isinstance(k, text_serializable_types):
                 filtered_key_is_final[k] |= _DictKeyState.END_OF_ITEM
 
     filtered_keys = filtered_key_is_final.keys()
 
     if not prefix:
         return "", 0, {repr(k): v for k, v in filtered_key_is_final.items()}
 
     quote_match = re.search("(?:\"|')", prefix)
     is_user_prefix_numeric = False
 
     if quote_match:
         quote = quote_match.group()
         valid_prefix = prefix + quote
         try:
             prefix_str = literal_eval(valid_prefix)
         except Exception:
             return "", 0, {}
     else:
         # If it does not look like a string, let's assume
         # we are dealing with a number or variable.
         number_match = _match_number_in_dict_key_prefix(prefix)
 
         # We do not want the key matcher to suggest variable names so we yield:
         if number_match is None:
             # The alternative would be to assume that user forgort the quote
             # and if the substring matches, suggest adding it at the start.
             return "", 0, {}
 
         prefix_str = number_match
         is_user_prefix_numeric = True
         quote = ""
 
     pattern = '[^' + ''.join('\\' + c for c in delims) + ']*$'
     token_match = re.search(pattern, prefix, re.UNICODE)
     assert token_match is not None # silence mypy
     token_start = token_match.start()
     token_prefix = token_match.group()
 
     matched: Dict[str, _DictKeyState] = {}
 
     str_key: Union[str, bytes]
 
     for key in filtered_keys:
         if isinstance(key, (int, float)):
             # User typed a number but this key is not a number.
             if not is_user_prefix_numeric:
                 continue
             str_key = str(key)
             if isinstance(key, int):
                 int_base = prefix_str[:2].lower()
                 # if user typed integer using binary/oct/hex notation:
                 if int_base in _INT_FORMATS:
                     int_format = _INT_FORMATS[int_base]
                     str_key = int_format(key)
         else:
             # User typed a string but this key is a number.
             if is_user_prefix_numeric:
                 continue
             str_key = key
         try:
             if not str_key.startswith(prefix_str):
                 continue
         except (AttributeError, TypeError, UnicodeError) as e:
             # Python 3+ TypeError on b'a'.startswith('a') or vice-versa
             continue
 
         # reformat remainder of key to begin with prefix
         rem = str_key[len(prefix_str) :]
         # force repr wrapped in '
         rem_repr = repr(rem + '"') if isinstance(rem, str) else repr(rem + b'"')
         rem_repr = rem_repr[1 + rem_repr.index("'"):-2]
         if quote == '"':
             # The entered prefix is quoted with ",
             # but the match is quoted with '.
             # A contained " hence needs escaping for comparison:
             rem_repr = rem_repr.replace('"', '\\"')
 
         # then reinsert prefix from start of token
         match = "%s%s" % (token_prefix, rem_repr)
 
         matched[match] = filtered_key_is_final[key]
     return quote, token_start, matched
 
 
 def cursor_to_position(text:str, line:int, column:int)->int:
     """
     Convert the (line,column) position of the cursor in text to an offset in a
     string.
 
     Parameters
     ----------
     text : str
         The text in which to calculate the cursor offset
     line : int
         Line of the cursor; 0-indexed
     column : int
         Column of the cursor 0-indexed
 
     Returns
     -------
     Position of the cursor in ``text``, 0-indexed.
 
     See Also
     --------
     position_to_cursor : reciprocal of this function
 
     """
     lines = text.split('\n')
     assert line <= len(lines), '{} <= {}'.format(str(line), str(len(lines)))
 
     return sum(len(l) + 1 for l in lines[:line]) + column
 
 def position_to_cursor(text:str, offset:int)->Tuple[int, int]:
     """
     Convert the position of the cursor in text (0 indexed) to a line
     number(0-indexed) and a column number (0-indexed) pair
 
     Position should be a valid position in ``text``.
 
     Parameters
     ----------
     text : str
         The text in which to calculate the cursor offset
     offset : int
         Position of the cursor in ``text``, 0-indexed.
 
     Returns
     -------
     (line, column) : (int, int)
         Line of the cursor; 0-indexed, column of the cursor 0-indexed
 
     See Also
     --------
     cursor_to_position : reciprocal of this function
 
     """
 
     assert 0 <= offset <= len(text) , "0 <= %s <= %s" % (offset , len(text))
 
     before = text[:offset]
     blines = before.split('\n')  # ! splitnes trim trailing \n
     line = before.count('\n')
     col = len(blines[-1])
     return line, col
 
 
 def _safe_isinstance(obj, module, class_name, *attrs):
     """Checks if obj is an instance of module.class_name if loaded
     """
     if module in sys.modules:
         m = sys.modules[module]
         for attr in [class_name, *attrs]:
             m = getattr(m, attr)
         return isinstance(obj, m)
 
 
 @context_matcher()
 def back_unicode_name_matcher(context: CompletionContext):
     """Match Unicode characters back to Unicode name
 
     Same as :any:`back_unicode_name_matches`, but adopted to new Matcher API.
     """
     fragment, matches = back_unicode_name_matches(context.text_until_cursor)
     return _convert_matcher_v1_result_to_v2(
         matches, type="unicode", fragment=fragment, suppress_if_matches=True
     )
 
 
 def back_unicode_name_matches(text: str) -> Tuple[str, Sequence[str]]:
     """Match Unicode characters back to Unicode name
 
     This does  ``☃`` -> ``\\snowman``
 
     Note that snowman is not a valid python3 combining character but will be expanded.
     Though it will not recombine back to the snowman character by the completion machinery.
 
     This will not either back-complete standard sequences like \\n, \\b ...
 
     .. deprecated:: 8.6
         You can use :meth:`back_unicode_name_matcher` instead.
 
     Returns
     =======
 
     Return a tuple with two elements:
 
     - The Unicode character that was matched (preceded with a backslash), or
         empty string,
     - a sequence (of 1), name for the match Unicode character, preceded by
         backslash, or empty if no match.
     """
     if len(text)<2:
         return '', ()
     maybe_slash = text[-2]
     if maybe_slash != '\\':
         return '', ()
 
     char = text[-1]
     # no expand on quote for completion in strings.
     # nor backcomplete standard ascii keys
     if char in string.ascii_letters or char in ('"',"'"):
         return '', ()
     try :
         unic = unicodedata.name(char)
         return '\\'+char,('\\'+unic,)
     except KeyError:
         pass
     return '', ()
 
 
 @context_matcher()
 def back_latex_name_matcher(context: CompletionContext):
     """Match latex characters back to unicode name
 
     Same as :any:`back_latex_name_matches`, but adopted to new Matcher API.
     """
     fragment, matches = back_latex_name_matches(context.text_until_cursor)
     return _convert_matcher_v1_result_to_v2(
         matches, type="latex", fragment=fragment, suppress_if_matches=True
     )
 
 
 def back_latex_name_matches(text: str) -> Tuple[str, Sequence[str]]:
     """Match latex characters back to unicode name
 
     This does ``\\ℵ`` -> ``\\aleph``
 
     .. deprecated:: 8.6
         You can use :meth:`back_latex_name_matcher` instead.
     """
     if len(text)<2:
         return '', ()
     maybe_slash = text[-2]
     if maybe_slash != '\\':
         return '', ()
 
 
     char = text[-1]
     # no expand on quote for completion in strings.
     # nor backcomplete standard ascii keys
     if char in string.ascii_letters or char in ('"',"'"):
         return '', ()
     try :
         latex = reverse_latex_symbol[char]
         # '\\' replace the \ as well
         return '\\'+char,[latex]
     except KeyError:
         pass
     return '', ()
 
 
 def _formatparamchildren(parameter) -> str:
     """
     Get parameter name and value from Jedi Private API
 
     Jedi does not expose a simple way to get `param=value` from its API.
 
     Parameters
     ----------
     parameter
         Jedi's function `Param`
 
     Returns
     -------
     A string like 'a', 'b=1', '*args', '**kwargs'
 
     """
     description = parameter.description
     if not description.startswith('param '):
         raise ValueError('Jedi function parameter description have change format.'
                          'Expected "param ...", found %r".' % description)
     return description[6:]
 
 def _make_signature(completion)-> str:
     """
     Make the signature from a jedi completion
 
     Parameters
     ----------
     completion : jedi.Completion
         object does not complete a function type
 
     Returns
     -------
     a string consisting of the function signature, with the parenthesis but
     without the function name. example:
     `(a, *args, b=1, **kwargs)`
 
     """
 
     # it looks like this might work on jedi 0.17
     if hasattr(completion, 'get_signatures'):
         signatures = completion.get_signatures()
         if not signatures:
             return  '(?)'
 
         c0 = completion.get_signatures()[0]
         return '('+c0.to_string().split('(', maxsplit=1)[1]
 
     return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for signature in completion.get_signatures()
                                           for p in signature.defined_names()) if f])
 
 
 _CompleteResult = Dict[str, MatcherResult]
 
 
 DICT_MATCHER_REGEX = re.compile(
     r"""(?x)
 (  # match dict-referring - or any get item object - expression
     .+
 )
 \[   # open bracket
 \s*  # and optional whitespace
 # Capture any number of serializable objects (e.g. "a", "b", 'c')
 # and slices
 ((?:(?:
     (?: # closed string
         [uUbB]?  # string prefix (r not handled)
         (?:
             '(?:[^']|(?<!\\)\\')*'
         |
             "(?:[^"]|(?<!\\)\\")*"
         )
     )
     |
         # capture integers and slices
         (?:[-+]?\d+)?(?::(?:[-+]?\d+)?){0,2}
     |
         # integer in bin/hex/oct notation
         0[bBxXoO]_?(?:\w|\d)+
     )
     \s*,\s*
 )*)
 ((?:
     (?: # unclosed string
         [uUbB]?  # string prefix (r not handled)
         (?:
             '(?:[^']|(?<!\\)\\')*
             |
             "(?:[^"]|(?<!\\)\\")*
         )
     )
     |
         # unfinished integer
         (?:[-+]?\d+)
     |
         # integer in bin/hex/oct notation
         0[bBxXoO]_?(?:\w|\d)+
     )
 )?
 $
 """
 )
 
 
 def _convert_matcher_v1_result_to_v2(
     matches: Sequence[str],
     type: str,
     fragment: Optional[str] = None,
     suppress_if_matches: bool = False,
 ) -> SimpleMatcherResult:
     """Utility to help with transition"""
     result = {
         "completions": [SimpleCompletion(text=match, type=type) for match in matches],
         "suppress": (True if matches else False) if suppress_if_matches else False,
     }
     if fragment is not None:
         result["matched_fragment"] = fragment
     return cast(SimpleMatcherResult, result)
 
 
 class IPCompleter(Completer):
     """Extension of the completer class with IPython-specific features"""
 
     @observe('greedy')
     def _greedy_changed(self, change):
         """update the splitter and readline delims when greedy is changed"""
         if change["new"]:
             self.evaluation = "unsafe"
             self.auto_close_dict_keys = True
             self.splitter.delims = GREEDY_DELIMS
         else:
             self.evaluation = "limited"
             self.auto_close_dict_keys = False
             self.splitter.delims = DELIMS
 
     dict_keys_only = Bool(
         False,
         help="""
         Whether to show dict key matches only.
 
         (disables all matchers except for `IPCompleter.dict_key_matcher`).
         """,
     )
 
     suppress_competing_matchers = UnionTrait(
         [Bool(allow_none=True), DictTrait(Bool(None, allow_none=True))],
         default_value=None,
         help="""
         Whether to suppress completions from other *Matchers*.
 
         When set to ``None`` (default) the matchers will attempt to auto-detect
         whether suppression of other matchers is desirable. For example, at
         the beginning of a line followed by `%` we expect a magic completion
         to be the only applicable option, and after ``my_dict['`` we usually
         expect a completion with an existing dictionary key.
 
         If you want to disable this heuristic and see completions from all matchers,
         set ``IPCompleter.suppress_competing_matchers = False``.
         To disable the heuristic for specific matchers provide a dictionary mapping:
         ``IPCompleter.suppress_competing_matchers = {'IPCompleter.dict_key_matcher': False}``.
 
         Set ``IPCompleter.suppress_competing_matchers = True`` to limit
         completions to the set of matchers with the highest priority;
         this is equivalent to ``IPCompleter.merge_completions`` and
         can be beneficial for performance, but will sometimes omit relevant
         candidates from matchers further down the priority list.
         """,
     ).tag(config=True)
 
     merge_completions = Bool(
         True,
         help="""Whether to merge completion results into a single list
 
         If False, only the completion results from the first non-empty
         completer will be returned.
 
         As of version 8.6.0, setting the value to ``False`` is an alias for:
         ``IPCompleter.suppress_competing_matchers = True.``.
         """,
     ).tag(config=True)
 
     disable_matchers = ListTrait(
         Unicode(),
         help="""List of matchers to disable.
 
         The list should contain matcher identifiers (see :any:`completion_matcher`).
         """,
     ).tag(config=True)
 
     omit__names = Enum(
         (0, 1, 2),
         default_value=2,
         help="""Instruct the completer to omit private method names
 
         Specifically, when completing on ``object.<tab>``.
 
         When 2 [default]: all names that start with '_' will be excluded.
 
         When 1: all 'magic' names (``__foo__``) will be excluded.
 
         When 0: nothing will be excluded.
         """
     ).tag(config=True)
     limit_to__all__ = Bool(False,
         help="""
         DEPRECATED as of version 5.0.
 
         Instruct the completer to use __all__ for the completion
 
         Specifically, when completing on ``object.<tab>``.
 
         When True: only those names in obj.__all__ will be included.
 
         When False [default]: the __all__ attribute is ignored
         """,
     ).tag(config=True)
 
     profile_completions = Bool(
         default_value=False,
         help="If True, emit profiling data for completion subsystem using cProfile."
     ).tag(config=True)
 
     profiler_output_dir = Unicode(
         default_value=".completion_profiles",
         help="Template for path at which to output profile data for completions."
     ).tag(config=True)
 
     @observe('limit_to__all__')
     def _limit_to_all_changed(self, change):
         warnings.warn('`IPython.core.IPCompleter.limit_to__all__` configuration '
             'value has been deprecated since IPython 5.0, will be made to have '
             'no effects and then removed in future version of IPython.',
             UserWarning)
 
     def __init__(
         self, shell=None, namespace=None, global_namespace=None, config=None, **kwargs
     ):
         """IPCompleter() -> completer
 
         Return a completer object.
 
         Parameters
         ----------
         shell
             a pointer to the ipython shell itself.  This is needed
             because this completer knows about magic functions, and those can
             only be accessed via the ipython instance.
         namespace : dict, optional
             an optional dict where completions are performed.
         global_namespace : dict, optional
             secondary optional dict for completions, to
             handle cases (such as IPython embedded inside functions) where
             both Python scopes are visible.
         config : Config
             traitlet's config object
         **kwargs
             passed to super class unmodified.
         """
 
         self.magic_escape = ESC_MAGIC
         self.splitter = CompletionSplitter()
 
         # _greedy_changed() depends on splitter and readline being defined:
         super().__init__(
             namespace=namespace,
             global_namespace=global_namespace,
             config=config,
             **kwargs,
         )
 
         # List where completion matches will be stored
         self.matches = []
         self.shell = shell
         # Regexp to split filenames with spaces in them
         self.space_name_re = re.compile(r'([^\\] )')
         # Hold a local ref. to glob.glob for speed
         self.glob = glob.glob
 
         # Determine if we are running on 'dumb' terminals, like (X)Emacs
         # buffers, to avoid completion problems.
         term = os.environ.get('TERM','xterm')
         self.dumb_terminal = term in ['dumb','emacs']
 
         # Special handling of backslashes needed in win32 platforms
         if sys.platform == "win32":
             self.clean_glob = self._clean_glob_win32
         else:
             self.clean_glob = self._clean_glob
 
         #regexp to parse docstring for function signature
         self.docstring_sig_re = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
         self.docstring_kwd_re = re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
         #use this if positional argument name is also needed
         #= re.compile(r'[\s|\[]*(\w+)(?:\s*=?\s*.*)')
 
         self.magic_arg_matchers = [
             self.magic_config_matcher,
             self.magic_color_matcher,
         ]
 
         # This is set externally by InteractiveShell
         self.custom_completers = None
 
         # This is a list of names of unicode characters that can be completed
         # into their corresponding unicode value. The list is large, so we
         # lazily initialize it on first use. Consuming code should access this
         # attribute through the `@unicode_names` property.
         self._unicode_names = None
 
         self._backslash_combining_matchers = [
             self.latex_name_matcher,
             self.unicode_name_matcher,
             back_latex_name_matcher,
             back_unicode_name_matcher,
             self.fwd_unicode_matcher,
         ]
 
         if not self.backslash_combining_completions:
             for matcher in self._backslash_combining_matchers:
                 self.disable_matchers.append(_get_matcher_id(matcher))
 
         if not self.merge_completions:
             self.suppress_competing_matchers = True
 
     @property
     def matchers(self) -> List[Matcher]:
         """All active matcher routines for completion"""
         if self.dict_keys_only:
             return [self.dict_key_matcher]
 
         if self.use_jedi:
             return [
                 *self.custom_matchers,
                 *self._backslash_combining_matchers,
                 *self.magic_arg_matchers,
                 self.custom_completer_matcher,
                 self.magic_matcher,
                 self._jedi_matcher,
                 self.dict_key_matcher,
                 self.file_matcher,
             ]
         else:
             return [
                 *self.custom_matchers,
                 *self._backslash_combining_matchers,
                 *self.magic_arg_matchers,
                 self.custom_completer_matcher,
                 self.dict_key_matcher,
                 self.magic_matcher,
                 self.python_matcher,
                 self.file_matcher,
                 self.python_func_kw_matcher,
             ]
 
     def all_completions(self, text:str) -> List[str]:
         """
         Wrapper around the completion methods for the benefit of emacs.
         """
         prefix = text.rpartition('.')[0]
         with provisionalcompleter():
             return ['.'.join([prefix, c.text]) if prefix and self.use_jedi else c.text
                     for c in self.completions(text, len(text))]
 
         return self.complete(text)[1]
 
     def _clean_glob(self, text:str):
         return self.glob("%s*" % text)
 
     def _clean_glob_win32(self, text:str):
         return [f.replace("\\","/")
                 for f in self.glob("%s*" % text)]
 
     @context_matcher()
     def file_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Same as :any:`file_matches`, but adopted to new Matcher API."""
         matches = self.file_matches(context.token)
         # TODO: add a heuristic for suppressing (e.g. if it has OS-specific delimiter,
         #  starts with `/home/`, `C:\`, etc)
         return _convert_matcher_v1_result_to_v2(matches, type="path")
 
     def file_matches(self, text: str) -> List[str]:
         """Match filenames, expanding ~USER type strings.
 
         Most of the seemingly convoluted logic in this completer is an
         attempt to handle filenames with spaces in them.  And yet it's not
         quite perfect, because Python's readline doesn't expose all of the
         GNU readline details needed for this to be done correctly.
 
         For a filename with a space in it, the printed completions will be
         only the parts after what's already been typed (instead of the
         full completions, as is normally done).  I don't think with the
         current (as of Python 2.3) Python readline it's possible to do
         better.
 
         .. deprecated:: 8.6
             You can use :meth:`file_matcher` instead.
         """
 
         # chars that require escaping with backslash - i.e. chars
         # that readline treats incorrectly as delimiters, but we
         # don't want to treat as delimiters in filename matching
         # when escaped with backslash
         if text.startswith('!'):
             text = text[1:]
             text_prefix = u'!'
         else:
             text_prefix = u''
 
         text_until_cursor = self.text_until_cursor
         # track strings with open quotes
         open_quotes = has_open_quotes(text_until_cursor)
 
         if '(' in text_until_cursor or '[' in text_until_cursor:
             lsplit = text
         else:
             try:
                 # arg_split ~ shlex.split, but with unicode bugs fixed by us
                 lsplit = arg_split(text_until_cursor)[-1]
             except ValueError:
                 # typically an unmatched ", or backslash without escaped char.
                 if open_quotes:
                     lsplit = text_until_cursor.split(open_quotes)[-1]
                 else:
                     return []
             except IndexError:
                 # tab pressed on empty line
                 lsplit = ""
 
         if not open_quotes and lsplit != protect_filename(lsplit):
             # if protectables are found, do matching on the whole escaped name
             has_protectables = True
             text0,text = text,lsplit
         else:
             has_protectables = False
             text = os.path.expanduser(text)
 
         if text == "":
             return [text_prefix + protect_filename(f) for f in self.glob("*")]
 
         # Compute the matches from the filesystem
         if sys.platform == 'win32':
             m0 = self.clean_glob(text)
         else:
             m0 = self.clean_glob(text.replace('\\', ''))
 
         if has_protectables:
             # If we had protectables, we need to revert our changes to the
             # beginning of filename so that we don't double-write the part
             # of the filename we have so far
             len_lsplit = len(lsplit)
             matches = [text_prefix + text0 +
                        protect_filename(f[len_lsplit:]) for f in m0]
         else:
             if open_quotes:
                 # if we have a string with an open quote, we don't need to
                 # protect the names beyond the quote (and we _shouldn't_, as
                 # it would cause bugs when the filesystem call is made).
                 matches = m0 if sys.platform == "win32" else\
                     [protect_filename(f, open_quotes) for f in m0]
             else:
                 matches = [text_prefix +
                            protect_filename(f) for f in m0]
 
         # Mark directories in input list by appending '/' to their names.
         return [x+'/' if os.path.isdir(x) else x for x in matches]
 
     @context_matcher()
     def magic_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match magics."""
         text = context.token
         matches = self.magic_matches(text)
         result = _convert_matcher_v1_result_to_v2(matches, type="magic")
         is_magic_prefix = len(text) > 0 and text[0] == "%"
         result["suppress"] = is_magic_prefix and bool(result["completions"])
         return result
 
     def magic_matches(self, text: str):
         """Match magics.
 
         .. deprecated:: 8.6
             You can use :meth:`magic_matcher` instead.
         """
         # Get all shell magics now rather than statically, so magics loaded at
         # runtime show up too.
         lsm = self.shell.magics_manager.lsmagic()
         line_magics = lsm['line']
         cell_magics = lsm['cell']
         pre = self.magic_escape
         pre2 = pre+pre
 
         explicit_magic = text.startswith(pre)
 
         # Completion logic:
         # - user gives %%: only do cell magics
         # - user gives %: do both line and cell magics
         # - no prefix: do both
         # In other words, line magics are skipped if the user gives %% explicitly
         #
         # We also exclude magics that match any currently visible names:
         # https://github.com/ipython/ipython/issues/4877, unless the user has
         # typed a %:
         # https://github.com/ipython/ipython/issues/10754
         bare_text = text.lstrip(pre)
         global_matches = self.global_matches(bare_text)
         if not explicit_magic:
             def matches(magic):
                 """
                 Filter magics, in particular remove magics that match
                 a name present in global namespace.
                 """
                 return ( magic.startswith(bare_text) and
                          magic not in global_matches )
         else:
             def matches(magic):
                 return magic.startswith(bare_text)
 
         comp = [ pre2+m for m in cell_magics if matches(m)]
         if not text.startswith(pre2):
             comp += [ pre+m for m in line_magics if matches(m)]
 
         return comp
 
     @context_matcher()
     def magic_config_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match class names and attributes for %config magic."""
         # NOTE: uses `line_buffer` equivalent for compatibility
         matches = self.magic_config_matches(context.line_with_cursor)
         return _convert_matcher_v1_result_to_v2(matches, type="param")
 
     def magic_config_matches(self, text: str) -> List[str]:
         """Match class names and attributes for %config magic.
 
         .. deprecated:: 8.6
             You can use :meth:`magic_config_matcher` instead.
         """
         texts = text.strip().split()
 
         if len(texts) > 0 and (texts[0] == 'config' or texts[0] == '%config'):
             # get all configuration classes
             classes = sorted(set([ c for c in self.shell.configurables
                                    if c.__class__.class_traits(config=True)
                                    ]), key=lambda x: x.__class__.__name__)
             classnames = [ c.__class__.__name__ for c in classes ]
 
             # return all classnames if config or %config is given
             if len(texts) == 1:
                 return classnames
 
             # match classname
             classname_texts = texts[1].split('.')
             classname = classname_texts[0]
             classname_matches = [ c for c in classnames
                                   if c.startswith(classname) ]
 
             # return matched classes or the matched class with attributes
             if texts[1].find('.') < 0:
                 return classname_matches
             elif len(classname_matches) == 1 and \
                             classname_matches[0] == classname:
                 cls = classes[classnames.index(classname)].__class__
                 help = cls.class_get_help()
                 # strip leading '--' from cl-args:
                 help = re.sub(re.compile(r'^--', re.MULTILINE), '', help)
                 return [ attr.split('=')[0]
                          for attr in help.strip().splitlines()
                          if attr.startswith(texts[1]) ]
         return []
 
     @context_matcher()
     def magic_color_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match color schemes for %colors magic."""
         # NOTE: uses `line_buffer` equivalent for compatibility
         matches = self.magic_color_matches(context.line_with_cursor)
         return _convert_matcher_v1_result_to_v2(matches, type="param")
 
     def magic_color_matches(self, text: str) -> List[str]:
         """Match color schemes for %colors magic.
 
         .. deprecated:: 8.6
             You can use :meth:`magic_color_matcher` instead.
         """
         texts = text.split()
         if text.endswith(' '):
             # .split() strips off the trailing whitespace. Add '' back
             # so that: '%colors ' -> ['%colors', '']
             texts.append('')
 
         if len(texts) == 2 and (texts[0] == 'colors' or texts[0] == '%colors'):
             prefix = texts[1]
             return [ color for color in InspectColors.keys()
                      if color.startswith(prefix) ]
         return []
 
     @context_matcher(identifier="IPCompleter.jedi_matcher")
     def _jedi_matcher(self, context: CompletionContext) -> _JediMatcherResult:
         matches = self._jedi_matches(
             cursor_column=context.cursor_position,
             cursor_line=context.cursor_line,
             text=context.full_text,
         )
         return {
             "completions": matches,
             # static analysis should not suppress other matchers
             "suppress": False,
         }
 
     def _jedi_matches(
         self, cursor_column: int, cursor_line: int, text: str
     ) -> Iterator[_JediCompletionLike]:
         """
         Return a list of :any:`jedi.api.Completion`\\s object from a ``text`` and
         cursor position.
 
         Parameters
         ----------
         cursor_column : int
             column position of the cursor in ``text``, 0-indexed.
         cursor_line : int
             line position of the cursor in ``text``, 0-indexed
         text : str
             text to complete
 
         Notes
         -----
         If ``IPCompleter.debug`` is ``True`` may return a :any:`_FakeJediCompletion`
         object containing a string with the Jedi debug information attached.
 
         .. deprecated:: 8.6
             You can use :meth:`_jedi_matcher` instead.
         """
         namespaces = [self.namespace]
         if self.global_namespace is not None:
             namespaces.append(self.global_namespace)
 
         completion_filter = lambda x:x
         offset = cursor_to_position(text, cursor_line, cursor_column)
         # filter output if we are completing for object members
         if offset:
             pre = text[offset-1]
             if pre == '.':
                 if self.omit__names == 2:
                     completion_filter = lambda c:not c.name.startswith('_')
                 elif self.omit__names == 1:
                     completion_filter = lambda c:not (c.name.startswith('__') and c.name.endswith('__'))
                 elif self.omit__names == 0:
                     completion_filter = lambda x:x
                 else:
                     raise ValueError("Don't understand self.omit__names == {}".format(self.omit__names))
 
         interpreter = jedi.Interpreter(text[:offset], namespaces)
         try_jedi = True
 
         try:
             # find the first token in the current tree -- if it is a ' or " then we are in a string
             completing_string = False
             try:
                 first_child = next(c for c in interpreter._get_module().tree_node.children if hasattr(c, 'value'))
             except StopIteration:
                 pass
             else:
                 # note the value may be ', ", or it may also be ''' or """, or
                 # in some cases, """what/you/typed..., but all of these are
                 # strings.
                 completing_string = len(first_child.value) > 0 and first_child.value[0] in {"'", '"'}
 
             # if we are in a string jedi is likely not the right candidate for
             # now. Skip it.
             try_jedi = not completing_string
         except Exception as e:
             # many of things can go wrong, we are using private API just don't crash.
             if self.debug:
                 print("Error detecting if completing a non-finished string :", e, '|')
 
         if not try_jedi:
             return iter([])
         try:
             return filter(completion_filter, interpreter.complete(column=cursor_column, line=cursor_line + 1))
         except Exception as e:
             if self.debug:
                 return iter(
                     [
                         _FakeJediCompletion(
                             'Oops Jedi has crashed, please report a bug with the following:\n"""\n%s\ns"""'
                             % (e)
                         )
                     ]
                 )
             else:
                 return iter([])
 
     @context_matcher()
     def python_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match attributes or global python names"""
         text = context.line_with_cursor
         if "." in text:
             try:
                 matches, fragment = self._attr_matches(text, include_prefix=False)
                 if text.endswith(".") and self.omit__names:
                     if self.omit__names == 1:
                         # true if txt is _not_ a __ name, false otherwise:
                         no__name = lambda txt: re.match(r".*\.__.*?__", txt) is None
                     else:
                         # true if txt is _not_ a _ name, false otherwise:
                         no__name = (
                             lambda txt: re.match(r"\._.*?", txt[txt.rindex(".") :])
                             is None
                         )
                     matches = filter(no__name, matches)
                 return _convert_matcher_v1_result_to_v2(
                     matches, type="attribute", fragment=fragment
                 )
             except NameError:
                 # catches <undefined attributes>.<tab>
                 matches = []
                 return _convert_matcher_v1_result_to_v2(matches, type="attribute")
         else:
             matches = self.global_matches(context.token)
             # TODO: maybe distinguish between functions, modules and just "variables"
             return _convert_matcher_v1_result_to_v2(matches, type="variable")
 
     @completion_matcher(api_version=1)
     def python_matches(self, text: str) -> Iterable[str]:
         """Match attributes or global python names.
 
         .. deprecated:: 8.27
             You can use :meth:`python_matcher` instead."""
         if "." in text:
             try:
                 matches = self.attr_matches(text)
                 if text.endswith('.') and self.omit__names:
                     if self.omit__names == 1:
                         # true if txt is _not_ a __ name, false otherwise:
                         no__name = (lambda txt:
                                     re.match(r'.*\.__.*?__',txt) is None)
                     else:
                         # true if txt is _not_ a _ name, false otherwise:
                         no__name = (lambda txt:
                                     re.match(r'\._.*?',txt[txt.rindex('.'):]) is None)
                     matches = filter(no__name, matches)
             except NameError:
                 # catches <undefined attributes>.<tab>
                 matches = []
         else:
             matches = self.global_matches(text)
         return matches
 
     def _default_arguments_from_docstring(self, doc):
         """Parse the first line of docstring for call signature.
 
         Docstring should be of the form 'min(iterable[, key=func])\n'.
         It can also parse cython docstring of the form
         'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)'.
         """
         if doc is None:
             return []
 
         #care only the firstline
         line = doc.lstrip().splitlines()[0]
 
         #p = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
         #'min(iterable[, key=func])\n' -> 'iterable[, key=func]'
         sig = self.docstring_sig_re.search(line)
         if sig is None:
             return []
         # iterable[, key=func]' -> ['iterable[' ,' key=func]']
         sig = sig.groups()[0].split(',')
         ret = []
         for s in sig:
             #re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
             ret += self.docstring_kwd_re.findall(s)
         return ret
 
     def _default_arguments(self, obj):
         """Return the list of default arguments of obj if it is callable,
         or empty list otherwise."""
         call_obj = obj
         ret = []
         if inspect.isbuiltin(obj):
             pass
         elif not (inspect.isfunction(obj) or inspect.ismethod(obj)):
             if inspect.isclass(obj):
                 #for cython embedsignature=True the constructor docstring
                 #belongs to the object itself not __init__
                 ret += self._default_arguments_from_docstring(
                             getattr(obj, '__doc__', ''))
                 # for classes, check for __init__,__new__
                 call_obj = (getattr(obj, '__init__', None) or
                        getattr(obj, '__new__', None))
             # for all others, check if they are __call__able
             elif hasattr(obj, '__call__'):
                 call_obj = obj.__call__
         ret += self._default_arguments_from_docstring(
                  getattr(call_obj, '__doc__', ''))
 
         _keeps = (inspect.Parameter.KEYWORD_ONLY,
                   inspect.Parameter.POSITIONAL_OR_KEYWORD)
 
         try:
             sig = inspect.signature(obj)
             ret.extend(k for k, v in sig.parameters.items() if
                        v.kind in _keeps)
         except ValueError:
             pass
 
         return list(set(ret))
 
     @context_matcher()
     def python_func_kw_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match named parameters (kwargs) of the last open function."""
         matches = self.python_func_kw_matches(context.token)
         return _convert_matcher_v1_result_to_v2(matches, type="param")
 
     def python_func_kw_matches(self, text):
         """Match named parameters (kwargs) of the last open function.
 
         .. deprecated:: 8.6
             You can use :meth:`python_func_kw_matcher` instead.
         """
 
         if "." in text: # a parameter cannot be dotted
             return []
         try: regexp = self.__funcParamsRegex
         except AttributeError:
             regexp = self.__funcParamsRegex = re.compile(r'''
                 '.*?(?<!\\)' |    # single quoted strings or
                 ".*?(?<!\\)" |    # double quoted strings or
                 \w+          |    # identifier
                 \S                # other characters
                 ''', re.VERBOSE | re.DOTALL)
         # 1. find the nearest identifier that comes before an unclosed
         # parenthesis before the cursor
         # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
         tokens = regexp.findall(self.text_until_cursor)
         iterTokens = reversed(tokens); openPar = 0
 
         for token in iterTokens:
             if token == ')':
                 openPar -= 1
             elif token == '(':
                 openPar += 1
                 if openPar > 0:
                     # found the last unclosed parenthesis
                     break
         else:
             return []
         # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
         ids = []
         isId = re.compile(r'\w+$').match
 
         while True:
             try:
                 ids.append(next(iterTokens))
                 if not isId(ids[-1]):
                     ids.pop(); break
                 if not next(iterTokens) == '.':
                     break
             except StopIteration:
                 break
 
         # Find all named arguments already assigned to, as to avoid suggesting
         # them again
         usedNamedArgs = set()
         par_level = -1
         for token, next_token in zip(tokens, tokens[1:]):
             if token == '(':
                 par_level += 1
             elif token == ')':
                 par_level -= 1
 
             if par_level != 0:
                 continue
 
             if next_token != '=':
                 continue
 
             usedNamedArgs.add(token)
 
         argMatches = []
         try:
             callableObj = '.'.join(ids[::-1])
             namedArgs = self._default_arguments(eval(callableObj,
                                                     self.namespace))
 
             # Remove used named arguments from the list, no need to show twice
             for namedArg in set(namedArgs) - usedNamedArgs:
                 if namedArg.startswith(text):
                     argMatches.append("%s=" %namedArg)
         except:
             pass
 
         return argMatches
 
     @staticmethod
     def _get_keys(obj: Any) -> List[Any]:
         # Objects can define their own completions by defining an
         # _ipy_key_completions_() method.
         method = get_real_method(obj, '_ipython_key_completions_')
         if method is not None:
             return method()
 
         # Special case some common in-memory dict-like types
         if isinstance(obj, dict) or _safe_isinstance(obj, "pandas", "DataFrame"):
             try:
                 return list(obj.keys())
             except Exception:
                 return []
         elif _safe_isinstance(obj, "pandas", "core", "indexing", "_LocIndexer"):
             try:
                 return list(obj.obj.keys())
             except Exception:
                 return []
         elif _safe_isinstance(obj, 'numpy', 'ndarray') or\
              _safe_isinstance(obj, 'numpy', 'void'):
             return obj.dtype.names or []
         return []
 
     @context_matcher()
     def dict_key_matcher(self, context: CompletionContext) -> SimpleMatcherResult:
         """Match string keys in a dictionary, after e.g. ``foo[``."""
         matches = self.dict_key_matches(context.token)
         return _convert_matcher_v1_result_to_v2(
             matches, type="dict key", suppress_if_matches=True
         )
 
     def dict_key_matches(self, text: str) -> List[str]:
         """Match string keys in a dictionary, after e.g. ``foo[``.
 
         .. deprecated:: 8.6
             You can use :meth:`dict_key_matcher` instead.
         """
 
         # Short-circuit on closed dictionary (regular expression would
         # not match anyway, but would take quite a while).
         if self.text_until_cursor.strip().endswith("]"):
             return []
 
         match = DICT_MATCHER_REGEX.search(self.text_until_cursor)
 
         if match is None:
             return []
 
         expr, prior_tuple_keys, key_prefix = match.groups()
 
         obj = self._evaluate_expr(expr)
 
         if obj is not_found:
             return []
 
         keys = self._get_keys(obj)
         if not keys:
             return keys
 
         tuple_prefix = guarded_eval(
             prior_tuple_keys,
             EvaluationContext(
                 globals=self.global_namespace,
                 locals=self.namespace,
                 evaluation=self.evaluation,  # type: ignore
                 in_subscript=True,
             ),
         )
 
         closing_quote, token_offset, matches = match_dict_keys(
             keys, key_prefix, self.splitter.delims, extra_prefix=tuple_prefix
         )
         if not matches:
             return []
 
         # get the cursor position of
         # - the text being completed
         # - the start of the key text
         # - the start of the completion
         text_start = len(self.text_until_cursor) - len(text)
         if key_prefix:
             key_start = match.start(3)
             completion_start = key_start + token_offset
         else:
             key_start = completion_start = match.end()
 
         # grab the leading prefix, to make sure all completions start with `text`
         if text_start > key_start:
             leading = ''
         else:
             leading = text[text_start:completion_start]
 
         # append closing quote and bracket as appropriate
         # this is *not* appropriate if the opening quote or bracket is outside
         # the text given to this method, e.g. `d["""a\nt
         can_close_quote = False
         can_close_bracket = False
 
         continuation = self.line_buffer[len(self.text_until_cursor) :].strip()
 
         if continuation.startswith(closing_quote):
             # do not close if already closed, e.g. `d['a<tab>'`
             continuation = continuation[len(closing_quote) :]
         else:
             can_close_quote = True
 
         continuation = continuation.strip()
 
         # e.g. `pandas.DataFrame` has different tuple indexer behaviour,
         # handling it is out of scope, so let's avoid appending suffixes.
         has_known_tuple_handling = isinstance(obj, dict)
 
         can_close_bracket = (
             not continuation.startswith("]") and self.auto_close_dict_keys
         )
         can_close_tuple_item = (
             not continuation.startswith(",")
             and has_known_tuple_handling
             and self.auto_close_dict_keys
         )
         can_close_quote = can_close_quote and self.auto_close_dict_keys
 
         # fast path if closing quote should be appended but not suffix is allowed
         if not can_close_quote and not can_close_bracket and closing_quote:
             return [leading + k for k in matches]
 
         results = []
 
         end_of_tuple_or_item = _DictKeyState.END_OF_TUPLE | _DictKeyState.END_OF_ITEM
 
         for k, state_flag in matches.items():
             result = leading + k
             if can_close_quote and closing_quote:
                 result += closing_quote
 
             if state_flag == end_of_tuple_or_item:
                 # We do not know which suffix to add,
                 # e.g. both tuple item and string
                 # match this item.
                 pass
 
             if state_flag in end_of_tuple_or_item and can_close_bracket:
                 result += "]"
             if state_flag == _DictKeyState.IN_TUPLE and can_close_tuple_item:
                 result += ", "
             results.append(result)
         return results
 
     @context_matcher()
     def unicode_name_matcher(self, context: CompletionContext):
         """Same as :any:`unicode_name_matches`, but adopted to new Matcher API."""
         fragment, matches = self.unicode_name_matches(context.text_until_cursor)
         return _convert_matcher_v1_result_to_v2(
             matches, type="unicode", fragment=fragment, suppress_if_matches=True
         )
 
     @staticmethod
     def unicode_name_matches(text: str) -> Tuple[str, List[str]]:
         """Match Latex-like syntax for unicode characters base
         on the name of the character.
 
         This does  ``\\GREEK SMALL LETTER ETA`` -> ``η``
 
         Works only on valid python 3 identifier, or on combining characters that
         will combine to form a valid identifier.
         """
         slashpos = text.rfind('\\')
         if slashpos > -1:
             s = text[slashpos+1:]
             try :
                 unic = unicodedata.lookup(s)
                 # allow combining chars
                 if ('a'+unic).isidentifier():
                     return '\\'+s,[unic]
             except KeyError:
                 pass
         return '', []
 
     @context_matcher()
     def latex_name_matcher(self, context: CompletionContext):
         """Match Latex syntax for unicode characters.
 
         This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
         """
         fragment, matches = self.latex_matches(context.text_until_cursor)
         return _convert_matcher_v1_result_to_v2(
             matches, type="latex", fragment=fragment, suppress_if_matches=True
         )
 
     def latex_matches(self, text: str) -> Tuple[str, Sequence[str]]:
         """Match Latex syntax for unicode characters.
 
         This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
 
         .. deprecated:: 8.6
             You can use :meth:`latex_name_matcher` instead.
         """
         slashpos = text.rfind('\\')
         if slashpos > -1:
             s = text[slashpos:]
             if s in latex_symbols:
                 # Try to complete a full latex symbol to unicode
                 # \\alpha -> α
                 return s, [latex_symbols[s]]
             else:
                 # If a user has partially typed a latex symbol, give them
                 # a full list of options \al -> [\aleph, \alpha]
                 matches = [k for k in latex_symbols if k.startswith(s)]
                 if matches:
                     return s, matches
         return '', ()
 
     @context_matcher()
     def custom_completer_matcher(self, context):
         """Dispatch custom completer.
 
         If a match is found, suppresses all other matchers except for Jedi.
         """
         matches = self.dispatch_custom_completer(context.token) or []
         result = _convert_matcher_v1_result_to_v2(
             matches, type=_UNKNOWN_TYPE, suppress_if_matches=True
         )
         result["ordered"] = True
         result["do_not_suppress"] = {_get_matcher_id(self._jedi_matcher)}
         return result
 
     def dispatch_custom_completer(self, text):
         """
         .. deprecated:: 8.6
             You can use :meth:`custom_completer_matcher` instead.
         """
         if not self.custom_completers:
             return
 
         line = self.line_buffer
         if not line.strip():
             return None
 
         # Create a little structure to pass all the relevant information about
         # the current completion to any custom completer.
         event = SimpleNamespace()
         event.line = line
         event.symbol = text
         cmd = line.split(None,1)[0]
         event.command = cmd
         event.text_until_cursor = self.text_until_cursor
 
         # for foo etc, try also to find completer for %foo
         if not cmd.startswith(self.magic_escape):
             try_magic = self.custom_completers.s_matches(
                 self.magic_escape + cmd)
         else:
             try_magic = []
 
         for c in itertools.chain(self.custom_completers.s_matches(cmd),
                  try_magic,
                  self.custom_completers.flat_matches(self.text_until_cursor)):
             try:
                 res = c(event)
                 if res:
                     # first, try case sensitive match
                     withcase = [r for r in res if r.startswith(text)]
                     if withcase:
                         return withcase
                     # if none, then case insensitive ones are ok too
                     text_low = text.lower()
                     return [r for r in res if r.lower().startswith(text_low)]
             except TryNext:
                 pass
             except KeyboardInterrupt:
                 """
                 If custom completer take too long,
                 let keyboard interrupt abort and return nothing.
                 """
                 break
 
         return None
 
     def completions(self, text: str, offset: int)->Iterator[Completion]:
         """
         Returns an iterator over the possible completions
 
         .. warning::
 
             Unstable
 
             This function is unstable, API may change without warning.
             It will also raise unless use in proper context manager.
 
         Parameters
         ----------
         text : str
             Full text of the current input, multi line string.
         offset : int
             Integer representing the position of the cursor in ``text``. Offset
             is 0-based indexed.
 
         Yields
         ------
         Completion
 
         Notes
         -----
         The cursor on a text can either be seen as being "in between"
         characters or "On" a character depending on the interface visible to
         the user. For consistency the cursor being on "in between" characters X
         and Y is equivalent to the cursor being "on" character Y, that is to say
         the character the cursor is on is considered as being after the cursor.
 
         Combining characters may span more that one position in the
         text.
 
         .. note::
 
             If ``IPCompleter.debug`` is :any:`True` will yield a ``--jedi/ipython--``
             fake Completion token to distinguish completion returned by Jedi
             and usual IPython completion.
 
         .. note::
 
             Completions are not completely deduplicated yet. If identical
             completions are coming from different sources this function does not
             ensure that each completion object will only be present once.
         """
         warnings.warn("_complete is a provisional API (as of IPython 6.0). "
                       "It may change without warnings. "
                       "Use in corresponding context manager.",
                       category=ProvisionalCompleterWarning, stacklevel=2)
 
         seen = set()
         profiler:Optional[cProfile.Profile]
         try:
             if self.profile_completions:
                 import cProfile
                 profiler = cProfile.Profile()
                 profiler.enable()
             else:
                 profiler = None
 
             for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
                 if c and (c in seen):
                     continue
                 yield c
                 seen.add(c)
         except KeyboardInterrupt:
             """if completions take too long and users send keyboard interrupt,
             do not crash and return ASAP. """
             pass
         finally:
             if profiler is not None:
                 profiler.disable()
                 ensure_dir_exists(self.profiler_output_dir)
                 output_path = os.path.join(self.profiler_output_dir, str(uuid.uuid4()))
                 print("Writing profiler output to", output_path)
                 profiler.dump_stats(output_path)
 
     def _completions(self, full_text: str, offset: int, *, _timeout) -> Iterator[Completion]:
         """
         Core completion module.Same signature as :any:`completions`, with the
         extra `timeout` parameter (in seconds).
 
         Computing jedi's completion ``.type`` can be quite expensive (it is a
         lazy property) and can require some warm-up, more warm up than just
         computing the ``name`` of a completion. The warm-up can be :
 
             - Long warm-up the first time a module is encountered after
             install/update: actually build parse/inference tree.
 
             - first time the module is encountered in a session: load tree from
             disk.
 
         We don't want to block completions for tens of seconds so we give the
         completer a "budget" of ``_timeout`` seconds per invocation to compute
         completions types, the completions that have not yet been computed will
         be marked as "unknown" an will have a chance to be computed next round
         are things get cached.
 
         Keep in mind that Jedi is not the only thing treating the completion so
         keep the timeout short-ish as if we take more than 0.3 second we still
         have lots of processing to do.
 
         """
         deadline = time.monotonic() + _timeout
 
         before = full_text[:offset]
         cursor_line, cursor_column = position_to_cursor(full_text, offset)
 
         jedi_matcher_id = _get_matcher_id(self._jedi_matcher)
 
         def is_non_jedi_result(
             result: MatcherResult, identifier: str
         ) -> TypeGuard[SimpleMatcherResult]:
             return identifier != jedi_matcher_id
 
         results = self._complete(
             full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column
         )
 
         non_jedi_results: Dict[str, SimpleMatcherResult] = {
             identifier: result
             for identifier, result in results.items()
             if is_non_jedi_result(result, identifier)
         }
 
         jedi_matches = (
             cast(_JediMatcherResult, results[jedi_matcher_id])["completions"]
             if jedi_matcher_id in results
             else ()
         )
 
         iter_jm = iter(jedi_matches)
         if _timeout:
             for jm in iter_jm:
                 try:
                     type_ = jm.type
                 except Exception:
                     if self.debug:
                         print("Error in Jedi getting type of ", jm)
                     type_ = None
                 delta = len(jm.name_with_symbols) - len(jm.complete)
                 if type_ == 'function':
                     signature = _make_signature(jm)
                 else:
                     signature = ''
                 yield Completion(start=offset - delta,
                                  end=offset,
                                  text=jm.name_with_symbols,
                                  type=type_,
                                  signature=signature,
                                  _origin='jedi')
 
                 if time.monotonic() > deadline:
                     break
 
         for jm in iter_jm:
             delta = len(jm.name_with_symbols) - len(jm.complete)
             yield Completion(
                 start=offset - delta,
                 end=offset,
                 text=jm.name_with_symbols,
                 type=_UNKNOWN_TYPE,  # don't compute type for speed
                 _origin="jedi",
                 signature="",
             )
 
         # TODO:
         # Suppress this, right now just for debug.
         if jedi_matches and non_jedi_results and self.debug:
             some_start_offset = before.rfind(
                 next(iter(non_jedi_results.values()))["matched_fragment"]
             )
             yield Completion(
                 start=some_start_offset,
                 end=offset,
                 text="--jedi/ipython--",
                 _origin="debug",
                 type="none",
                 signature="",
             )
 
         ordered: List[Completion] = []
         sortable: List[Completion] = []
 
         for origin, result in non_jedi_results.items():
             matched_text = result["matched_fragment"]
             start_offset = before.rfind(matched_text)
             is_ordered = result.get("ordered", False)
             container = ordered if is_ordered else sortable
 
             # I'm unsure if this is always true, so let's assert and see if it
             # crash
             assert before.endswith(matched_text)
 
             for simple_completion in result["completions"]:
                 completion = Completion(
                     start=start_offset,
                     end=offset,
                     text=simple_completion.text,
                     _origin=origin,
                     signature="",
                     type=simple_completion.type or _UNKNOWN_TYPE,
                 )
                 container.append(completion)
 
         yield from list(self._deduplicate(ordered + self._sort(sortable)))[
             :MATCHES_LIMIT
         ]
 
     def complete(self, text=None, line_buffer=None, cursor_pos=None) -> Tuple[str, Sequence[str]]:
         """Find completions for the given text and line context.
 
         Note that both the text and the line_buffer are optional, but at least
         one of them must be given.
 
         Parameters
         ----------
         text : string, optional
             Text to perform the completion on.  If not given, the line buffer
             is split using the instance's CompletionSplitter object.
         line_buffer : string, optional
             If not given, the completer attempts to obtain the current line
             buffer via readline.  This keyword allows clients which are
             requesting for text completions in non-readline contexts to inform
             the completer of the entire text.
         cursor_pos : int, optional
             Index of the cursor in the full line buffer.  Should be provided by
             remote frontends where kernel has no access to frontend state.
 
         Returns
         -------
         Tuple of two items:
         text : str
             Text that was actually used in the completion.
         matches : list
             A list of completion matches.
 
         Notes
         -----
             This API is likely to be deprecated and replaced by
             :any:`IPCompleter.completions` in the future.
 
         """
         warnings.warn('`Completer.complete` is pending deprecation since '
                 'IPython 6.0 and will be replaced by `Completer.completions`.',
                       PendingDeprecationWarning)
         # potential todo, FOLD the 3rd throw away argument of _complete
         # into the first 2 one.
         # TODO: Q: does the above refer to jedi completions (i.e. 0-indexed?)
         # TODO: should we deprecate now, or does it stay?
 
         results = self._complete(
             line_buffer=line_buffer, cursor_pos=cursor_pos, text=text, cursor_line=0
         )
 
         jedi_matcher_id = _get_matcher_id(self._jedi_matcher)
 
         return self._arrange_and_extract(
             results,
             # TODO: can we confirm that excluding Jedi here was a deliberate choice in previous version?
             skip_matchers={jedi_matcher_id},
             # this API does not support different start/end positions (fragments of token).
             abort_if_offset_changes=True,
         )
 
     def _arrange_and_extract(
         self,
         results: Dict[str, MatcherResult],
         skip_matchers: Set[str],
         abort_if_offset_changes: bool,
     ):
         sortable: List[AnyMatcherCompletion] = []
         ordered: List[AnyMatcherCompletion] = []
         most_recent_fragment = None
         for identifier, result in results.items():
             if identifier in skip_matchers:
                 continue
             if not result["completions"]:
                 continue
             if not most_recent_fragment:
                 most_recent_fragment = result["matched_fragment"]
             if (
                 abort_if_offset_changes
                 and result["matched_fragment"] != most_recent_fragment
             ):
                 break
             if result.get("ordered", False):
                 ordered.extend(result["completions"])
             else:
                 sortable.extend(result["completions"])
 
         if not most_recent_fragment:
             most_recent_fragment = ""  # to satisfy typechecker (and just in case)
 
         return most_recent_fragment, [
             m.text for m in self._deduplicate(ordered + self._sort(sortable))
         ]
 
     def _complete(self, *, cursor_line, cursor_pos, line_buffer=None, text=None,
                   full_text=None) -> _CompleteResult:
         """
         Like complete but can also returns raw jedi completions as well as the
         origin of the completion text. This could (and should) be made much
         cleaner but that will be simpler once we drop the old (and stateful)
         :any:`complete` API.
 
         With current provisional API, cursor_pos act both (depending on the
         caller) as the offset in the ``text`` or ``line_buffer``, or as the
         ``column`` when passing multiline strings this could/should be renamed
         but would add extra noise.
 
         Parameters
         ----------
         cursor_line
             Index of the line the cursor is on. 0 indexed.
         cursor_pos
             Position of the cursor in the current line/line_buffer/text. 0
             indexed.
         line_buffer : optional, str
             The current line the cursor is in, this is mostly due to legacy
             reason that readline could only give a us the single current line.
             Prefer `full_text`.
         text : str
             The current "token" the cursor is in, mostly also for historical
             reasons. as the completer would trigger only after the current line
             was parsed.
         full_text : str
             Full text of the current cell.
 
         Returns
         -------
         An ordered dictionary where keys are identifiers of completion
         matchers and values are ``MatcherResult``s.
         """
 
         # if the cursor position isn't given, the only sane assumption we can
         # make is that it's at the end of the line (the common case)
         if cursor_pos is None:
             cursor_pos = len(line_buffer) if text is None else len(text)
 
         if self.use_main_ns:
             self.namespace = __main__.__dict__
 
         # if text is either None or an empty string, rely on the line buffer
         if (not line_buffer) and full_text:
             line_buffer = full_text.split('\n')[cursor_line]
         if not text:  # issue #11508: check line_buffer before calling split_line
             text = (
                 self.splitter.split_line(line_buffer, cursor_pos) if line_buffer else ""
             )
 
         # If no line buffer is given, assume the input text is all there was
         if line_buffer is None:
             line_buffer = text
 
         # deprecated - do not use `line_buffer` in new code.
         self.line_buffer = line_buffer
         self.text_until_cursor = self.line_buffer[:cursor_pos]
 
         if not full_text:
             full_text = line_buffer
 
         context = CompletionContext(
             full_text=full_text,
             cursor_position=cursor_pos,
             cursor_line=cursor_line,
             token=text,
             limit=MATCHES_LIMIT,
         )
 
         # Start with a clean slate of completions
         results: Dict[str, MatcherResult] = {}
 
         jedi_matcher_id = _get_matcher_id(self._jedi_matcher)
 
         suppressed_matchers: Set[str] = set()
 
         matchers = {
             _get_matcher_id(matcher): matcher
             for matcher in sorted(
                 self.matchers, key=_get_matcher_priority, reverse=True
             )
         }
 
         for matcher_id, matcher in matchers.items():
             matcher_id = _get_matcher_id(matcher)
 
             if matcher_id in self.disable_matchers:
                 continue
 
             if matcher_id in results:
                 warnings.warn(f"Duplicate matcher ID: {matcher_id}.")
 
             if matcher_id in suppressed_matchers:
                 continue
 
             result: MatcherResult
             try:
                 if _is_matcher_v1(matcher):
                     result = _convert_matcher_v1_result_to_v2(
                         matcher(text), type=_UNKNOWN_TYPE
                     )
                 elif _is_matcher_v2(matcher):
                     result = matcher(context)
                 else:
                     api_version = _get_matcher_api_version(matcher)
                     raise ValueError(f"Unsupported API version {api_version}")
             except:
                 # Show the ugly traceback if the matcher causes an
                 # exception, but do NOT crash the kernel!
                 sys.excepthook(*sys.exc_info())
                 continue
 
             # set default value for matched fragment if suffix was not selected.
             result["matched_fragment"] = result.get("matched_fragment", context.token)
 
             if not suppressed_matchers:
                 suppression_recommended: Union[bool, Set[str]] = result.get(
                     "suppress", False
                 )
 
                 suppression_config = (
                     self.suppress_competing_matchers.get(matcher_id, None)
                     if isinstance(self.suppress_competing_matchers, dict)
                     else self.suppress_competing_matchers
                 )
                 should_suppress = (
                     (suppression_config is True)
                     or (suppression_recommended and (suppression_config is not False))
                 ) and has_any_completions(result)
 
                 if should_suppress:
                     suppression_exceptions: Set[str] = result.get(
                         "do_not_suppress", set()
                     )
                     if isinstance(suppression_recommended, Iterable):
                         to_suppress = set(suppression_recommended)
                     else:
                         to_suppress = set(matchers)
                     suppressed_matchers = to_suppress - suppression_exceptions
 
                     new_results = {}
                     for previous_matcher_id, previous_result in results.items():
                         if previous_matcher_id not in suppressed_matchers:
                             new_results[previous_matcher_id] = previous_result
                     results = new_results
 
             results[matcher_id] = result
 
         _, matches = self._arrange_and_extract(
             results,
             # TODO Jedi completions non included in legacy stateful API; was this deliberate or omission?
             #  if it was omission, we can remove the filtering step, otherwise remove this comment.
             skip_matchers={jedi_matcher_id},
             abort_if_offset_changes=False,
         )
 
         # populate legacy stateful API
         self.matches = matches
 
         return results
 
     @staticmethod
     def _deduplicate(
         matches: Sequence[AnyCompletion],
     ) -> Iterable[AnyCompletion]:
         filtered_matches: Dict[str, AnyCompletion] = {}
         for match in matches:
             text = match.text
             if (
                 text not in filtered_matches
                 or filtered_matches[text].type == _UNKNOWN_TYPE
             ):
                 filtered_matches[text] = match
 
         return filtered_matches.values()
 
     @staticmethod
     def _sort(matches: Sequence[AnyCompletion]):
         return sorted(matches, key=lambda x: completions_sorting_key(x.text))
 
     @context_matcher()
     def fwd_unicode_matcher(self, context: CompletionContext):
         """Same as :any:`fwd_unicode_match`, but adopted to new Matcher API."""
         # TODO: use `context.limit` to terminate early once we matched the maximum
         #  number that will be used downstream; can be added as an optional to
         #  `fwd_unicode_match(text: str, limit: int = None)` or we could re-implement here.
         fragment, matches = self.fwd_unicode_match(context.text_until_cursor)
         return _convert_matcher_v1_result_to_v2(
             matches, type="unicode", fragment=fragment, suppress_if_matches=True
         )
 
     def fwd_unicode_match(self, text: str) -> Tuple[str, Sequence[str]]:
         """
         Forward match a string starting with a backslash with a list of
         potential Unicode completions.
 
         Will compute list of Unicode character names on first call and cache it.
 
         .. deprecated:: 8.6
             You can use :meth:`fwd_unicode_matcher` instead.
 
         Returns
         -------
         At tuple with:
             - matched text (empty if no matches)
             - list of potential completions, empty tuple  otherwise)
         """
         # TODO: self.unicode_names is here a list we traverse each time with ~100k elements.
         # We could do a faster match using a Trie.
 
         # Using pygtrie the following seem to work:
 
         #     s = PrefixSet()
 
         #     for c in range(0,0x10FFFF + 1):
         #         try:
         #             s.add(unicodedata.name(chr(c)))
         #         except ValueError:
         #             pass
         #     [''.join(k) for k in s.iter(prefix)]
 
         # But need to be timed and adds an extra dependency.
 
         slashpos = text.rfind('\\')
         # if text starts with slash
         if slashpos > -1:
             # PERF: It's important that we don't access self._unicode_names
             # until we're inside this if-block. _unicode_names is lazily
             # initialized, and it takes a user-noticeable amount of time to
             # initialize it, so we don't want to initialize it unless we're
             # actually going to use it.
             s = text[slashpos + 1 :]
             sup = s.upper()
             candidates = [x for x in self.unicode_names if x.startswith(sup)]
             if candidates:
                 return s, candidates
             candidates = [x for x in self.unicode_names if sup in x]
             if candidates:
                 return s, candidates
             splitsup = sup.split(" ")
             candidates = [
                 x for x in self.unicode_names if all(u in x for u in splitsup)
             ]
             if candidates:
                 return s, candidates
 
             return "", ()
 
         # if text does not start with slash
         else:
             return '', ()
 
     @property
     def unicode_names(self) -> List[str]:
         """List of names of unicode code points that can be completed.
 
         The list is lazily initialized on first access.
         """
         if self._unicode_names is None:
             names = []
             for c in range(0,0x10FFFF + 1):
                 try:
                     names.append(unicodedata.name(chr(c)))
                 except ValueError:
                     pass
             self._unicode_names = _unicode_name_compute(_UNICODE_RANGES)
 
         return self._unicode_names
 
 def _unicode_name_compute(ranges:List[Tuple[int,int]]) -> List[str]:
     names = []
     for start,stop in ranges:
         for c in range(start, stop) :
             try:
                 names.append(unicodedata.name(chr(c)))
             except ValueError:
                 pass
     return names

IPython/core/guarded_eval.py

@@ -1,898 +1,895
 from inspect import isclass, signature, Signature
 from typing import (
     Annotated,
     AnyStr,
     Callable,
     Dict,
     Literal,
     NamedTuple,
     NewType,
     Optional,
     Protocol,
     Set,
     Sequence,
     Tuple,
     Type,
     TypeGuard,
     Union,
     get_args,
     get_origin,
     is_typeddict,
 )
 import ast
 import builtins
 import collections
 import operator
 import sys
 from functools import cached_property
 from dataclasses import dataclass, field
 from types import MethodDescriptorType, ModuleType
 
 from IPython.utils.decorators import undoc
 
 
 if sys.version_info < (3, 11):
     from typing_extensions import Self, LiteralString
 else:
     from typing import Self, LiteralString
 
 if sys.version_info < (3, 12):
     from typing_extensions import TypeAliasType
 else:
     from typing import TypeAliasType
 
 
 @undoc
 class HasGetItem(Protocol):
-    def __getitem__(self, key) -> None:
-        ...
+    def __getitem__(self, key) -> None: ...
 
 
 @undoc
 class InstancesHaveGetItem(Protocol):
-    def __call__(self, *args, **kwargs) -> HasGetItem:
-        ...
+    def __call__(self, *args, **kwargs) -> HasGetItem: ...
 
 
 @undoc
 class HasGetAttr(Protocol):
-    def __getattr__(self, key) -> None:
-        ...
+    def __getattr__(self, key) -> None: ...
 
 
 @undoc
 class DoesNotHaveGetAttr(Protocol):
     pass
 
 
 # By default `__getattr__` is not explicitly implemented on most objects
 MayHaveGetattr = Union[HasGetAttr, DoesNotHaveGetAttr]
 
 
 def _unbind_method(func: Callable) -> Union[Callable, None]:
     """Get unbound method for given bound method.
 
     Returns None if cannot get unbound method, or method is already unbound.
     """
     owner = getattr(func, "__self__", None)
     owner_class = type(owner)
     name = getattr(func, "__name__", None)
     instance_dict_overrides = getattr(owner, "__dict__", None)
     if (
         owner is not None
         and name
         and (
             not instance_dict_overrides
             or (instance_dict_overrides and name not in instance_dict_overrides)
         )
     ):
         return getattr(owner_class, name)
     return None
 
 
 @undoc
 @dataclass
 class EvaluationPolicy:
     """Definition of evaluation policy."""
 
     allow_locals_access: bool = False
     allow_globals_access: bool = False
     allow_item_access: bool = False
     allow_attr_access: bool = False
     allow_builtins_access: bool = False
     allow_all_operations: bool = False
     allow_any_calls: bool = False
     allowed_calls: Set[Callable] = field(default_factory=set)
 
     def can_get_item(self, value, item):
         return self.allow_item_access
 
     def can_get_attr(self, value, attr):
         return self.allow_attr_access
 
     def can_operate(self, dunders: Tuple[str, ...], a, b=None):
         if self.allow_all_operations:
             return True
 
     def can_call(self, func):
         if self.allow_any_calls:
             return True
 
         if func in self.allowed_calls:
             return True
 
         owner_method = _unbind_method(func)
 
         if owner_method and owner_method in self.allowed_calls:
             return True
 
 
 def _get_external(module_name: str, access_path: Sequence[str]):
     """Get value from external module given a dotted access path.
 
     Raises:
     * `KeyError` if module is removed not found, and
     * `AttributeError` if access path does not match an exported object
     """
     member_type = sys.modules[module_name]
     for attr in access_path:
         member_type = getattr(member_type, attr)
     return member_type
 
 
 def _has_original_dunder_external(
     value,
     module_name: str,
     access_path: Sequence[str],
     method_name: str,
 ):
     if module_name not in sys.modules:
         # LBYLB as it is faster
         return False
     try:
         member_type = _get_external(module_name, access_path)
         value_type = type(value)
         if type(value) == member_type:
             return True
         if method_name == "__getattribute__":
             # we have to short-circuit here due to an unresolved issue in
             # `isinstance` implementation: https://bugs.python.org/issue32683
             return False
         if isinstance(value, member_type):
             method = getattr(value_type, method_name, None)
             member_method = getattr(member_type, method_name, None)
             if member_method == method:
                 return True
     except (AttributeError, KeyError):
         return False
 
 
 def _has_original_dunder(
     value, allowed_types, allowed_methods, allowed_external, method_name
 ):
     # note: Python ignores `__getattr__`/`__getitem__` on instances,
     # we only need to check at class level
     value_type = type(value)
 
     # strict type check passes → no need to check method
     if value_type in allowed_types:
         return True
 
     method = getattr(value_type, method_name, None)
 
     if method is None:
         return None
 
     if method in allowed_methods:
         return True
 
     for module_name, *access_path in allowed_external:
         if _has_original_dunder_external(value, module_name, access_path, method_name):
             return True
 
     return False
 
 
 @undoc
 @dataclass
 class SelectivePolicy(EvaluationPolicy):
     allowed_getitem: Set[InstancesHaveGetItem] = field(default_factory=set)
     allowed_getitem_external: Set[Tuple[str, ...]] = field(default_factory=set)
 
     allowed_getattr: Set[MayHaveGetattr] = field(default_factory=set)
     allowed_getattr_external: Set[Tuple[str, ...]] = field(default_factory=set)
 
     allowed_operations: Set = field(default_factory=set)
     allowed_operations_external: Set[Tuple[str, ...]] = field(default_factory=set)
 
     _operation_methods_cache: Dict[str, Set[Callable]] = field(
         default_factory=dict, init=False
     )
 
     def can_get_attr(self, value, attr):
         has_original_attribute = _has_original_dunder(
             value,
             allowed_types=self.allowed_getattr,
             allowed_methods=self._getattribute_methods,
             allowed_external=self.allowed_getattr_external,
             method_name="__getattribute__",
         )
         has_original_attr = _has_original_dunder(
             value,
             allowed_types=self.allowed_getattr,
             allowed_methods=self._getattr_methods,
             allowed_external=self.allowed_getattr_external,
             method_name="__getattr__",
         )
 
         accept = False
 
         # Many objects do not have `__getattr__`, this is fine.
         if has_original_attr is None and has_original_attribute:
             accept = True
         else:
             # Accept objects without modifications to `__getattr__` and `__getattribute__`
             accept = has_original_attr and has_original_attribute
 
         if accept:
             # We still need to check for overridden properties.
 
             value_class = type(value)
             if not hasattr(value_class, attr):
                 return True
 
             class_attr_val = getattr(value_class, attr)
             is_property = isinstance(class_attr_val, property)
 
             if not is_property:
                 return True
 
             # Properties in allowed types are ok (although we do not include any
             # properties in our default allow list currently).
             if type(value) in self.allowed_getattr:
                 return True  # pragma: no cover
 
             # Properties in subclasses of allowed types may be ok if not changed
             for module_name, *access_path in self.allowed_getattr_external:
                 try:
                     external_class = _get_external(module_name, access_path)
                     external_class_attr_val = getattr(external_class, attr)
                 except (KeyError, AttributeError):
                     return False  # pragma: no cover
                 return class_attr_val == external_class_attr_val
 
         return False
 
     def can_get_item(self, value, item):
         """Allow accessing `__getiitem__` of allow-listed instances unless it was not modified."""
         return _has_original_dunder(
             value,
             allowed_types=self.allowed_getitem,
             allowed_methods=self._getitem_methods,
             allowed_external=self.allowed_getitem_external,
             method_name="__getitem__",
         )
 
     def can_operate(self, dunders: Tuple[str, ...], a, b=None):
         objects = [a]
         if b is not None:
             objects.append(b)
         return all(
             [
                 _has_original_dunder(
                     obj,
                     allowed_types=self.allowed_operations,
                     allowed_methods=self._operator_dunder_methods(dunder),
                     allowed_external=self.allowed_operations_external,
                     method_name=dunder,
                 )
                 for dunder in dunders
                 for obj in objects
             ]
         )
 
     def _operator_dunder_methods(self, dunder: str) -> Set[Callable]:
         if dunder not in self._operation_methods_cache:
             self._operation_methods_cache[dunder] = self._safe_get_methods(
                 self.allowed_operations, dunder
             )
         return self._operation_methods_cache[dunder]
 
     @cached_property
     def _getitem_methods(self) -> Set[Callable]:
         return self._safe_get_methods(self.allowed_getitem, "__getitem__")
 
     @cached_property
     def _getattr_methods(self) -> Set[Callable]:
         return self._safe_get_methods(self.allowed_getattr, "__getattr__")
 
     @cached_property
     def _getattribute_methods(self) -> Set[Callable]:
         return self._safe_get_methods(self.allowed_getattr, "__getattribute__")
 
     def _safe_get_methods(self, classes, name) -> Set[Callable]:
         return {
             method
             for class_ in classes
             for method in [getattr(class_, name, None)]
             if method
         }
 
 
 class _DummyNamedTuple(NamedTuple):
     """Used internally to retrieve methods of named tuple instance."""
 
 
 class EvaluationContext(NamedTuple):
     #: Local namespace
     locals: dict
     #: Global namespace
     globals: dict
     #: Evaluation policy identifier
-    evaluation: Literal[
-        "forbidden", "minimal", "limited", "unsafe", "dangerous"
-    ] = "forbidden"
+    evaluation: Literal["forbidden", "minimal", "limited", "unsafe", "dangerous"] = (
+        "forbidden"
+    )
     #: Whether the evaluation of code takes place inside of a subscript.
     #: Useful for evaluating ``:-1, 'col'`` in ``df[:-1, 'col']``.
     in_subscript: bool = False
 
 
 class _IdentitySubscript:
     """Returns the key itself when item is requested via subscript."""
 
     def __getitem__(self, key):
         return key
 
 
 IDENTITY_SUBSCRIPT = _IdentitySubscript()
 SUBSCRIPT_MARKER = "__SUBSCRIPT_SENTINEL__"
 UNKNOWN_SIGNATURE = Signature()
 NOT_EVALUATED = object()
 
 
 class GuardRejection(Exception):
     """Exception raised when guard rejects evaluation attempt."""
 
     pass
 
 
 def guarded_eval(code: str, context: EvaluationContext):
     """Evaluate provided code in the evaluation context.
 
     If evaluation policy given by context is set to ``forbidden``
     no evaluation will be performed; if it is set to ``dangerous``
     standard :func:`eval` will be used; finally, for any other,
     policy :func:`eval_node` will be called on parsed AST.
     """
     locals_ = context.locals
 
     if context.evaluation == "forbidden":
         raise GuardRejection("Forbidden mode")
 
     # note: not using `ast.literal_eval` as it does not implement
     # getitem at all, for example it fails on simple `[0][1]`
 
     if context.in_subscript:
         # syntactic sugar for ellipsis (:) is only available in subscripts
         # so we need to trick the ast parser into thinking that we have
         # a subscript, but we need to be able to later recognise that we did
         # it so we can ignore the actual __getitem__ operation
         if not code:
             return tuple()
         locals_ = locals_.copy()
         locals_[SUBSCRIPT_MARKER] = IDENTITY_SUBSCRIPT
         code = SUBSCRIPT_MARKER + "[" + code + "]"
         context = EvaluationContext(**{**context._asdict(), **{"locals": locals_}})
 
     if context.evaluation == "dangerous":
         return eval(code, context.globals, context.locals)
 
     expression = ast.parse(code, mode="eval")
 
     return eval_node(expression, context)
 
 
 BINARY_OP_DUNDERS: Dict[Type[ast.operator], Tuple[str]] = {
     ast.Add: ("__add__",),
     ast.Sub: ("__sub__",),
     ast.Mult: ("__mul__",),
     ast.Div: ("__truediv__",),
     ast.FloorDiv: ("__floordiv__",),
     ast.Mod: ("__mod__",),
     ast.Pow: ("__pow__",),
     ast.LShift: ("__lshift__",),
     ast.RShift: ("__rshift__",),
     ast.BitOr: ("__or__",),
     ast.BitXor: ("__xor__",),
     ast.BitAnd: ("__and__",),
     ast.MatMult: ("__matmul__",),
 }
 
 COMP_OP_DUNDERS: Dict[Type[ast.cmpop], Tuple[str, ...]] = {
     ast.Eq: ("__eq__",),
     ast.NotEq: ("__ne__", "__eq__"),
     ast.Lt: ("__lt__", "__gt__"),
     ast.LtE: ("__le__", "__ge__"),
     ast.Gt: ("__gt__", "__lt__"),
     ast.GtE: ("__ge__", "__le__"),
     ast.In: ("__contains__",),
     # Note: ast.Is, ast.IsNot, ast.NotIn are handled specially
 }
 
 UNARY_OP_DUNDERS: Dict[Type[ast.unaryop], Tuple[str, ...]] = {
     ast.USub: ("__neg__",),
     ast.UAdd: ("__pos__",),
     # we have to check both __inv__ and __invert__!
     ast.Invert: ("__invert__", "__inv__"),
     ast.Not: ("__not__",),
 }
 
 
 class ImpersonatingDuck:
     """A dummy class used to create objects of other classes without calling their ``__init__``"""
 
     # no-op: override __class__ to impersonate
 
 
 class _Duck:
     """A dummy class used to create objects pretending to have given attributes"""
 
     def __init__(self, attributes: Optional[dict] = None, items: Optional[dict] = None):
         self.attributes = attributes or {}
         self.items = items or {}
 
     def __getattr__(self, attr: str):
         return self.attributes[attr]
 
     def __hasattr__(self, attr: str):
         return attr in self.attributes
 
     def __dir__(self):
         return [*dir(super), *self.attributes]
 
     def __getitem__(self, key: str):
         return self.items[key]
 
     def __hasitem__(self, key: str):
         return self.items[key]
 
     def _ipython_key_completions_(self):
         return self.items.keys()
 
 
 def _find_dunder(node_op, dunders) -> Union[Tuple[str, ...], None]:
     dunder = None
     for op, candidate_dunder in dunders.items():
         if isinstance(node_op, op):
             dunder = candidate_dunder
     return dunder
 
 
 def eval_node(node: Union[ast.AST, None], context: EvaluationContext):
     """Evaluate AST node in provided context.
 
     Applies evaluation restrictions defined in the context. Currently does not support evaluation of functions with keyword arguments.
 
     Does not evaluate actions that always have side effects:
 
     - class definitions (``class sth: ...``)
     - function definitions (``def sth: ...``)
     - variable assignments (``x = 1``)
     - augmented assignments (``x += 1``)
     - deletions (``del x``)
 
     Does not evaluate operations which do not return values:
 
     - assertions (``assert x``)
     - pass (``pass``)
     - imports (``import x``)
     - control flow:
 
         - conditionals (``if x:``) except for ternary IfExp (``a if x else b``)
         - loops (``for`` and ``while``)
         - exception handling
 
     The purpose of this function is to guard against unwanted side-effects;
     it does not give guarantees on protection from malicious code execution.
     """
     policy = EVALUATION_POLICIES[context.evaluation]
     if node is None:
         return None
     if isinstance(node, ast.Expression):
         return eval_node(node.body, context)
     if isinstance(node, ast.BinOp):
         left = eval_node(node.left, context)
         right = eval_node(node.right, context)
         dunders = _find_dunder(node.op, BINARY_OP_DUNDERS)
         if dunders:
             if policy.can_operate(dunders, left, right):
                 return getattr(left, dunders[0])(right)
             else:
                 raise GuardRejection(
                     f"Operation (`{dunders}`) for",
                     type(left),
                     f"not allowed in {context.evaluation} mode",
                 )
     if isinstance(node, ast.Compare):
         left = eval_node(node.left, context)
         all_true = True
         negate = False
         for op, right in zip(node.ops, node.comparators):
             right = eval_node(right, context)
             dunder = None
             dunders = _find_dunder(op, COMP_OP_DUNDERS)
             if not dunders:
                 if isinstance(op, ast.NotIn):
                     dunders = COMP_OP_DUNDERS[ast.In]
                     negate = True
                 if isinstance(op, ast.Is):
                     dunder = "is_"
                 if isinstance(op, ast.IsNot):
                     dunder = "is_"
                     negate = True
             if not dunder and dunders:
                 dunder = dunders[0]
             if dunder:
                 a, b = (right, left) if dunder == "__contains__" else (left, right)
                 if dunder == "is_" or dunders and policy.can_operate(dunders, a, b):
                     result = getattr(operator, dunder)(a, b)
                     if negate:
                         result = not result
                     if not result:
                         all_true = False
                     left = right
                 else:
                     raise GuardRejection(
                         f"Comparison (`{dunder}`) for",
                         type(left),
                         f"not allowed in {context.evaluation} mode",
                     )
             else:
                 raise ValueError(
                     f"Comparison `{dunder}` not supported"
                 )  # pragma: no cover
         return all_true
     if isinstance(node, ast.Constant):
         return node.value
     if isinstance(node, ast.Tuple):
         return tuple(eval_node(e, context) for e in node.elts)
     if isinstance(node, ast.List):
         return [eval_node(e, context) for e in node.elts]
     if isinstance(node, ast.Set):
         return {eval_node(e, context) for e in node.elts}
     if isinstance(node, ast.Dict):
         return dict(
             zip(
                 [eval_node(k, context) for k in node.keys],
                 [eval_node(v, context) for v in node.values],
             )
         )
     if isinstance(node, ast.Slice):
         return slice(
             eval_node(node.lower, context),
             eval_node(node.upper, context),
             eval_node(node.step, context),
         )
     if isinstance(node, ast.UnaryOp):
         value = eval_node(node.operand, context)
         dunders = _find_dunder(node.op, UNARY_OP_DUNDERS)
         if dunders:
             if policy.can_operate(dunders, value):
                 return getattr(value, dunders[0])()
             else:
                 raise GuardRejection(
                     f"Operation (`{dunders}`) for",
                     type(value),
                     f"not allowed in {context.evaluation} mode",
                 )
     if isinstance(node, ast.Subscript):
         value = eval_node(node.value, context)
         slice_ = eval_node(node.slice, context)
         if policy.can_get_item(value, slice_):
             return value[slice_]
         raise GuardRejection(
             "Subscript access (`__getitem__`) for",
             type(value),  # not joined to avoid calling `repr`
             f" not allowed in {context.evaluation} mode",
         )
     if isinstance(node, ast.Name):
         return _eval_node_name(node.id, context)
     if isinstance(node, ast.Attribute):
         value = eval_node(node.value, context)
         if policy.can_get_attr(value, node.attr):
             return getattr(value, node.attr)
         raise GuardRejection(
             "Attribute access (`__getattr__`) for",
             type(value),  # not joined to avoid calling `repr`
             f"not allowed in {context.evaluation} mode",
         )
     if isinstance(node, ast.IfExp):
         test = eval_node(node.test, context)
         if test:
             return eval_node(node.body, context)
         else:
             return eval_node(node.orelse, context)
     if isinstance(node, ast.Call):
         func = eval_node(node.func, context)
         if policy.can_call(func) and not node.keywords:
             args = [eval_node(arg, context) for arg in node.args]
             return func(*args)
         if isclass(func):
             # this code path gets entered when calling class e.g. `MyClass()`
             # or `my_instance.__class__()` - in both cases `func` is `MyClass`.
             # Should return `MyClass` if `__new__` is not overridden,
             # otherwise whatever `__new__` return type is.
             overridden_return_type = _eval_return_type(func.__new__, node, context)
             if overridden_return_type is not NOT_EVALUATED:
                 return overridden_return_type
             return _create_duck_for_heap_type(func)
         else:
             return_type = _eval_return_type(func, node, context)
             if return_type is not NOT_EVALUATED:
                 return return_type
         raise GuardRejection(
             "Call for",
             func,  # not joined to avoid calling `repr`
             f"not allowed in {context.evaluation} mode",
         )
     raise ValueError("Unhandled node", ast.dump(node))
 
 
 def _eval_return_type(func: Callable, node: ast.Call, context: EvaluationContext):
     """Evaluate return type of a given callable function.
 
     Returns the built-in type, a duck or NOT_EVALUATED sentinel.
     """
     try:
         sig = signature(func)
     except ValueError:
         sig = UNKNOWN_SIGNATURE
     # if annotation was not stringized, or it was stringized
     # but resolved by signature call we know the return type
     not_empty = sig.return_annotation is not Signature.empty
     if not_empty:
         return _resolve_annotation(sig.return_annotation, sig, func, node, context)
     return NOT_EVALUATED
 
 
 def _resolve_annotation(
     annotation,
     sig: Signature,
     func: Callable,
     node: ast.Call,
     context: EvaluationContext,
 ):
     """Resolve annotation created by user with `typing` module and custom objects."""
     annotation = (
         _eval_node_name(annotation, context)
         if isinstance(annotation, str)
         else annotation
     )
     origin = get_origin(annotation)
     if annotation is Self and hasattr(func, "__self__"):
         return func.__self__
     elif origin is Literal:
         type_args = get_args(annotation)
         if len(type_args) == 1:
             return type_args[0]
     elif annotation is LiteralString:
         return ""
     elif annotation is AnyStr:
         index = None
         for i, (key, value) in enumerate(sig.parameters.items()):
             if value.annotation is AnyStr:
                 index = i
                 break
         if index is not None and index < len(node.args):
             return eval_node(node.args[index], context)
     elif origin is TypeGuard:
         return bool()
     elif origin is Union:
         attributes = [
             attr
             for type_arg in get_args(annotation)
             for attr in dir(_resolve_annotation(type_arg, sig, func, node, context))
         ]
         return _Duck(attributes=dict.fromkeys(attributes))
     elif is_typeddict(annotation):
         return _Duck(
             attributes=dict.fromkeys(dir(dict())),
             items={
                 k: _resolve_annotation(v, sig, func, node, context)
                 for k, v in annotation.__annotations__.items()
             },
         )
     elif hasattr(annotation, "_is_protocol"):
         return _Duck(attributes=dict.fromkeys(dir(annotation)))
     elif origin is Annotated:
         type_arg = get_args(annotation)[0]
         return _resolve_annotation(type_arg, sig, func, node, context)
     elif isinstance(annotation, NewType):
         return _eval_or_create_duck(annotation.__supertype__, node, context)
     elif isinstance(annotation, TypeAliasType):
         return _eval_or_create_duck(annotation.__value__, node, context)
     else:
         return _eval_or_create_duck(annotation, node, context)
 
 
 def _eval_node_name(node_id: str, context: EvaluationContext):
     policy = EVALUATION_POLICIES[context.evaluation]
     if policy.allow_locals_access and node_id in context.locals:
         return context.locals[node_id]
     if policy.allow_globals_access and node_id in context.globals:
         return context.globals[node_id]
     if policy.allow_builtins_access and hasattr(builtins, node_id):
         # note: do not use __builtins__, it is implementation detail of cPython
         return getattr(builtins, node_id)
     if not policy.allow_globals_access and not policy.allow_locals_access:
         raise GuardRejection(
             f"Namespace access not allowed in {context.evaluation} mode"
         )
     else:
         raise NameError(f"{node_id} not found in locals, globals, nor builtins")
 
 
 def _eval_or_create_duck(duck_type, node: ast.Call, context: EvaluationContext):
     policy = EVALUATION_POLICIES[context.evaluation]
     # if allow-listed builtin is on type annotation, instantiate it
     if policy.can_call(duck_type) and not node.keywords:
         args = [eval_node(arg, context) for arg in node.args]
         return duck_type(*args)
     # if custom class is in type annotation, mock it
     return _create_duck_for_heap_type(duck_type)
 
 
 def _create_duck_for_heap_type(duck_type):
     """Create an imitation of an object of a given type (a duck).
 
     Returns the duck or NOT_EVALUATED sentinel if duck could not be created.
     """
     duck = ImpersonatingDuck()
     try:
         # this only works for heap types, not builtins
         duck.__class__ = duck_type
         return duck
     except TypeError:
         pass
     return NOT_EVALUATED
 
 
 SUPPORTED_EXTERNAL_GETITEM = {
     ("pandas", "core", "indexing", "_iLocIndexer"),
     ("pandas", "core", "indexing", "_LocIndexer"),
     ("pandas", "DataFrame"),
     ("pandas", "Series"),
     ("numpy", "ndarray"),
     ("numpy", "void"),
 }
 
 
 BUILTIN_GETITEM: Set[InstancesHaveGetItem] = {
     dict,
     str,  # type: ignore[arg-type]
     bytes,  # type: ignore[arg-type]
     list,
     tuple,
     collections.defaultdict,
     collections.deque,
     collections.OrderedDict,
     collections.ChainMap,
     collections.UserDict,
     collections.UserList,
     collections.UserString,  # type: ignore[arg-type]
     _DummyNamedTuple,
     _IdentitySubscript,
 }
 
 
 def _list_methods(cls, source=None):
     """For use on immutable objects or with methods returning a copy"""
     return [getattr(cls, k) for k in (source if source else dir(cls))]
 
 
 dict_non_mutating_methods = ("copy", "keys", "values", "items")
 list_non_mutating_methods = ("copy", "index", "count")
 set_non_mutating_methods = set(dir(set)) & set(dir(frozenset))
 
 
 dict_keys: Type[collections.abc.KeysView] = type({}.keys())
 
 NUMERICS = {int, float, complex}
 
 ALLOWED_CALLS = {
     bytes,
     *_list_methods(bytes),
     dict,
     *_list_methods(dict, dict_non_mutating_methods),
     dict_keys.isdisjoint,
     list,
     *_list_methods(list, list_non_mutating_methods),
     set,
     *_list_methods(set, set_non_mutating_methods),
     frozenset,
     *_list_methods(frozenset),
     range,
     str,
     *_list_methods(str),
     tuple,
     *_list_methods(tuple),
     *NUMERICS,
     *[method for numeric_cls in NUMERICS for method in _list_methods(numeric_cls)],
     collections.deque,
     *_list_methods(collections.deque, list_non_mutating_methods),
     collections.defaultdict,
     *_list_methods(collections.defaultdict, dict_non_mutating_methods),
     collections.OrderedDict,
     *_list_methods(collections.OrderedDict, dict_non_mutating_methods),
     collections.UserDict,
     *_list_methods(collections.UserDict, dict_non_mutating_methods),
     collections.UserList,
     *_list_methods(collections.UserList, list_non_mutating_methods),
     collections.UserString,
     *_list_methods(collections.UserString, dir(str)),
     collections.Counter,
     *_list_methods(collections.Counter, dict_non_mutating_methods),
     collections.Counter.elements,
     collections.Counter.most_common,
 }
 
 BUILTIN_GETATTR: Set[MayHaveGetattr] = {
     *BUILTIN_GETITEM,
     set,
     frozenset,
     object,
     type,  # `type` handles a lot of generic cases, e.g. numbers as in `int.real`.
     *NUMERICS,
     dict_keys,
     MethodDescriptorType,
     ModuleType,
 }
 
 
 BUILTIN_OPERATIONS = {*BUILTIN_GETATTR}
 
 EVALUATION_POLICIES = {
     "minimal": EvaluationPolicy(
         allow_builtins_access=True,
         allow_locals_access=False,
         allow_globals_access=False,
         allow_item_access=False,
         allow_attr_access=False,
         allowed_calls=set(),
         allow_any_calls=False,
         allow_all_operations=False,
     ),
     "limited": SelectivePolicy(
         allowed_getitem=BUILTIN_GETITEM,
         allowed_getitem_external=SUPPORTED_EXTERNAL_GETITEM,
         allowed_getattr=BUILTIN_GETATTR,
         allowed_getattr_external={
             # pandas Series/Frame implements custom `__getattr__`
             ("pandas", "DataFrame"),
             ("pandas", "Series"),
         },
         allowed_operations=BUILTIN_OPERATIONS,
         allow_builtins_access=True,
         allow_locals_access=True,
         allow_globals_access=True,
         allowed_calls=ALLOWED_CALLS,
     ),
     "unsafe": EvaluationPolicy(
         allow_builtins_access=True,
         allow_locals_access=True,
         allow_globals_access=True,
         allow_attr_access=True,
         allow_item_access=True,
         allow_any_calls=True,
         allow_all_operations=True,
     ),
 }
 
 
 __all__ = [
     "guarded_eval",
     "eval_node",
     "GuardRejection",
     "EvaluationContext",
     "_unbind_method",
 ]

IPython/core/inputsplitter.py

@@ -1,798 +1,799
 """DEPRECATED: Input handling and transformation machinery.
 
 This module was deprecated in IPython 7.0, in favour of inputtransformer2.
 
 The first class in this module, :class:`InputSplitter`, is designed to tell when
 input from a line-oriented frontend is complete and should be executed, and when
 the user should be prompted for another line of code instead. The name 'input
 splitter' is largely for historical reasons.
 
 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
 with full support for the extended IPython syntax (magics, system calls, etc).
 The code to actually do these transformations is in :mod:`IPython.core.inputtransformer`.
 :class:`IPythonInputSplitter` feeds the raw code to the transformers in order
 and stores the results.
 
 For more details, see the class docstrings below.
 """
+
 from __future__ import annotations
 
 from warnings import warn
 
 warn('IPython.core.inputsplitter is deprecated since IPython 7 in favor of `IPython.core.inputtransformer2`',
      DeprecationWarning)
 
 # Copyright (c) IPython Development Team.
 # Distributed under the terms of the Modified BSD License.
 import ast
 import codeop
 import io
 import re
 import sys
 import tokenize
 import warnings
 
 from typing import List, Tuple, Union, Optional, TYPE_CHECKING
 from types import CodeType
 
 from IPython.core.inputtransformer import (leading_indent,
                                            classic_prompt,
                                            ipy_prompt,
                                            cellmagic,
                                            assemble_logical_lines,
                                            help_end,
                                            escaped_commands,
                                            assign_from_magic,
                                            assign_from_system,
                                            assemble_python_lines,
                                            )
 from IPython.utils import tokenutil
 
 # These are available in this module for backwards compatibility.
 from IPython.core.inputtransformer import (ESC_SHELL, ESC_SH_CAP, ESC_HELP,
                                         ESC_HELP2, ESC_MAGIC, ESC_MAGIC2,
                                         ESC_QUOTE, ESC_QUOTE2, ESC_PAREN, ESC_SEQUENCES)
 
 if TYPE_CHECKING:
     from typing_extensions import Self
 #-----------------------------------------------------------------------------
 # Utilities
 #-----------------------------------------------------------------------------
 
 # FIXME: These are general-purpose utilities that later can be moved to the
 # general ward.  Kept here for now because we're being very strict about test
 # coverage with this code, and this lets us ensure that we keep 100% coverage
 # while developing.
 
 # compiled regexps for autoindent management
 dedent_re = re.compile('|'.join([
     r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
     r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
     r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
     r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
     r'^\s+pass\s*$', # pass (optionally followed by trailing spaces)
     r'^\s+break\s*$', # break (optionally followed by trailing spaces)
     r'^\s+continue\s*$', # continue (optionally followed by trailing spaces)
 ]))
 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
 
 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
 # before pure comments
 comment_line_re = re.compile(r'^\s*\#')
 
 
 def num_ini_spaces(s):
     """Return the number of initial spaces in a string.
 
     Note that tabs are counted as a single space.  For now, we do *not* support
     mixing of tabs and spaces in the user's input.
 
     Parameters
     ----------
     s : string
 
     Returns
     -------
     n : int
     """
     warnings.warn(
         "`num_ini_spaces` is Pending Deprecation since IPython 8.17."
         "It is considered for removal in in future version. "
         "Please open an issue if you believe it should be kept.",
         stacklevel=2,
         category=PendingDeprecationWarning,
     )
     ini_spaces = ini_spaces_re.match(s)
     if ini_spaces:
         return ini_spaces.end()
     else:
         return 0
 
 # Fake token types for partial_tokenize:
 INCOMPLETE_STRING = tokenize.N_TOKENS
 IN_MULTILINE_STATEMENT = tokenize.N_TOKENS + 1
 
 # The 2 classes below have the same API as TokenInfo, but don't try to look up
 # a token type name that they won't find.
 class IncompleteString:
     type = exact_type = INCOMPLETE_STRING
     def __init__(self, s, start, end, line):
         self.s = s
         self.start = start
         self.end = end
         self.line = line
 
 class InMultilineStatement:
     type = exact_type = IN_MULTILINE_STATEMENT
     def __init__(self, pos, line):
         self.s = ''
         self.start = self.end = pos
         self.line = line
 
 def partial_tokens(s):
     """Iterate over tokens from a possibly-incomplete string of code.
 
     This adds two special token types: INCOMPLETE_STRING and
     IN_MULTILINE_STATEMENT. These can only occur as the last token yielded, and
     represent the two main ways for code to be incomplete.
     """
     readline = io.StringIO(s).readline
     token = tokenize.TokenInfo(tokenize.NEWLINE, '', (1, 0), (1, 0), '')
     try:
         for token in tokenutil.generate_tokens_catch_errors(readline):
             yield token
     except tokenize.TokenError as e:
         # catch EOF error
         lines = s.splitlines(keepends=True)
         end = len(lines), len(lines[-1])
         if 'multi-line string' in e.args[0]:
             l, c = start = token.end
             s = lines[l-1][c:] + ''.join(lines[l:])
             yield IncompleteString(s, start, end, lines[-1])
         elif 'multi-line statement' in e.args[0]:
             yield InMultilineStatement(end, lines[-1])
         else:
             raise
 
 def find_next_indent(code) -> int:
     """Find the number of spaces for the next line of indentation"""
     tokens = list(partial_tokens(code))
     if tokens[-1].type == tokenize.ENDMARKER:
         tokens.pop()
     if not tokens:
         return 0
 
     while tokens[-1].type in {
         tokenize.DEDENT,
         tokenize.NEWLINE,
         tokenize.COMMENT,
         tokenize.ERRORTOKEN,
     }:
         tokens.pop()
 
     # Starting in Python 3.12, the tokenize module adds implicit newlines at the end
     # of input. We need to remove those if we're in a multiline statement
     if tokens[-1].type == IN_MULTILINE_STATEMENT:
         while tokens[-2].type in {tokenize.NL}:
             tokens.pop(-2)
 
 
     if tokens[-1].type == INCOMPLETE_STRING:
         # Inside a multiline string
         return 0
 
     # Find the indents used before
     prev_indents = [0]
     def _add_indent(n):
         if n != prev_indents[-1]:
             prev_indents.append(n)
 
     tokiter = iter(tokens)
     for tok in tokiter:
         if tok.type in {tokenize.INDENT, tokenize.DEDENT}:
             _add_indent(tok.end[1])
         elif (tok.type == tokenize.NL):
             try:
                 _add_indent(next(tokiter).start[1])
             except StopIteration:
                 break
 
     last_indent = prev_indents.pop()
 
     # If we've just opened a multiline statement (e.g. 'a = ['), indent more
     if tokens[-1].type == IN_MULTILINE_STATEMENT:
         if tokens[-2].exact_type in {tokenize.LPAR, tokenize.LSQB, tokenize.LBRACE}:
             return last_indent + 4
         return last_indent
 
     if tokens[-1].exact_type == tokenize.COLON:
         # Line ends with colon - indent
         return last_indent + 4
 
     if last_indent:
         # Examine the last line for dedent cues - statements like return or
         # raise which normally end a block of code.
         last_line_starts = 0
         for i, tok in enumerate(tokens):
             if tok.type == tokenize.NEWLINE:
                 last_line_starts = i + 1
 
         last_line_tokens = tokens[last_line_starts:]
         names = [t.string for t in last_line_tokens if t.type == tokenize.NAME]
         if names and names[0] in {'raise', 'return', 'pass', 'break', 'continue'}:
             # Find the most recent indentation less than the current level
             for indent in reversed(prev_indents):
                 if indent < last_indent:
                     return indent
 
     return last_indent
 
 
 def last_blank(src):
     """Determine if the input source ends in a blank.
 
     A blank is either a newline or a line consisting of whitespace.
 
     Parameters
     ----------
     src : string
         A single or multiline string.
     """
     if not src: return False
     ll  = src.splitlines()[-1]
     return (ll == '') or ll.isspace()
 
 
 last_two_blanks_re = re.compile(r'\n\s*\n\s*$', re.MULTILINE)
 last_two_blanks_re2 = re.compile(r'.+\n\s*\n\s+$', re.MULTILINE)
 
 def last_two_blanks(src):
     """Determine if the input source ends in two blanks.
 
     A blank is either a newline or a line consisting of whitespace.
 
     Parameters
     ----------
     src : string
         A single or multiline string.
     """
     if not src: return False
     # The logic here is tricky: I couldn't get a regexp to work and pass all
     # the tests, so I took a different approach: split the source by lines,
     # grab the last two and prepend '###\n' as a stand-in for whatever was in
     # the body before the last two lines.  Then, with that structure, it's
     # possible to analyze with two regexps.  Not the most elegant solution, but
     # it works.  If anyone tries to change this logic, make sure to validate
     # the whole test suite first!
     new_src = '\n'.join(['###\n'] + src.splitlines()[-2:])
     return (bool(last_two_blanks_re.match(new_src)) or
             bool(last_two_blanks_re2.match(new_src)) )
 
 
 def remove_comments(src):
     """Remove all comments from input source.
 
     Note: comments are NOT recognized inside of strings!
 
     Parameters
     ----------
     src : string
         A single or multiline input string.
 
     Returns
     -------
     String with all Python comments removed.
     """
 
     return re.sub('#.*', '', src)
 
 
 def get_input_encoding():
     """Return the default standard input encoding.
 
     If sys.stdin has no encoding, 'ascii' is returned."""
     # There are strange environments for which sys.stdin.encoding is None. We
     # ensure that a valid encoding is returned.
     encoding = getattr(sys.stdin, 'encoding', None)
     if encoding is None:
         encoding = 'ascii'
     return encoding
 
 #-----------------------------------------------------------------------------
 # Classes and functions for normal Python syntax handling
 #-----------------------------------------------------------------------------
 
 class InputSplitter(object):
     r"""An object that can accumulate lines of Python source before execution.
 
     This object is designed to be fed python source line-by-line, using
     :meth:`push`. It will return on each push whether the currently pushed
     code could be executed already. In addition, it provides a method called
     :meth:`push_accepts_more` that can be used to query whether more input
     can be pushed into a single interactive block.
 
     This is a simple example of how an interactive terminal-based client can use
     this tool::
 
         isp = InputSplitter()
         while isp.push_accepts_more():
             indent = ' '*isp.indent_spaces
             prompt = '>>> ' + indent
             line = indent + raw_input(prompt)
             isp.push(line)
         print('Input source was:\n', isp.source_reset())
     """
     # A cache for storing the current indentation
     # The first value stores the most recently processed source input
     # The second value is the number of spaces for the current indentation 
     # If self.source matches the first value, the second value is a valid
     # current indentation. Otherwise, the cache is invalid and the indentation
     # must be recalculated.
     _indent_spaces_cache: Union[Tuple[None, None], Tuple[str, int]] = None, None
     # String, indicating the default input encoding.  It is computed by default
     # at initialization time via get_input_encoding(), but it can be reset by a
     # client with specific knowledge of the encoding.
     encoding = ''
     # String where the current full source input is stored, properly encoded.
     # Reading this attribute is the normal way of querying the currently pushed
     # source code, that has been properly encoded.
     source: str = ""
     # Code object corresponding to the current source.  It is automatically
     # synced to the source, so it can be queried at any time to obtain the code
     # object; it will be None if the source doesn't compile to valid Python.
     code: Optional[CodeType] = None
 
     # Private attributes
 
     # List with lines of input accumulated so far
     _buffer: List[str]
     # Command compiler
     _compile: codeop.CommandCompiler
     # Boolean indicating whether the current block is complete
     _is_complete: Optional[bool] = None
     # Boolean indicating whether the current block has an unrecoverable syntax error
     _is_invalid: bool = False
 
     def __init__(self) -> None:
         """Create a new InputSplitter instance."""
         self._buffer = []
         self._compile = codeop.CommandCompiler()
         self.encoding = get_input_encoding()
 
     def reset(self):
         """Reset the input buffer and associated state."""
         self._buffer[:] = []
         self.source = ''
         self.code = None
         self._is_complete = False
         self._is_invalid = False
 
     def source_reset(self):
         """Return the input source and perform a full reset.
         """
         out = self.source
         self.reset()
         return out
 
     def check_complete(self, source):
         """Return whether a block of code is ready to execute, or should be continued
 
         This is a non-stateful API, and will reset the state of this InputSplitter.
 
         Parameters
         ----------
         source : string
             Python input code, which can be multiline.
 
         Returns
         -------
         status : str
             One of 'complete', 'incomplete', or 'invalid' if source is not a
             prefix of valid code.
         indent_spaces : int or None
             The number of spaces by which to indent the next line of code. If
             status is not 'incomplete', this is None.
         """
         self.reset()
         try:
             self.push(source)
         except SyntaxError:
             # Transformers in IPythonInputSplitter can raise SyntaxError,
             # which push() will not catch.
             return 'invalid', None
         else:
             if self._is_invalid:
                 return 'invalid', None
             elif self.push_accepts_more():
                 return 'incomplete', self.get_indent_spaces()
             else:
                 return 'complete', None
         finally:
             self.reset()
 
     def push(self, lines:str) -> bool:
         """Push one or more lines of input.
 
         This stores the given lines and returns a status code indicating
         whether the code forms a complete Python block or not.
 
         Any exceptions generated in compilation are swallowed, but if an
         exception was produced, the method returns True.
 
         Parameters
         ----------
         lines : string
             One or more lines of Python input.
 
         Returns
         -------
         is_complete : boolean
             True if the current input source (the result of the current input
             plus prior inputs) forms a complete Python execution block.  Note that
             this value is also stored as a private attribute (``_is_complete``), so it
             can be queried at any time.
         """
         assert isinstance(lines, str)
         self._store(lines)
         source = self.source
 
         # Before calling _compile(), reset the code object to None so that if an
         # exception is raised in compilation, we don't mislead by having
         # inconsistent code/source attributes.
         self.code, self._is_complete = None, None
         self._is_invalid = False
 
         # Honor termination lines properly
         if source.endswith('\\\n'):
             return False
 
         try:
             with warnings.catch_warnings():
                 warnings.simplefilter('error', SyntaxWarning)
                 self.code = self._compile(source, symbol="exec")
         # Invalid syntax can produce any of a number of different errors from
         # inside the compiler, so we have to catch them all.  Syntax errors
         # immediately produce a 'ready' block, so the invalid Python can be
         # sent to the kernel for evaluation with possible ipython
         # special-syntax conversion.
         except (SyntaxError, OverflowError, ValueError, TypeError,
                 MemoryError, SyntaxWarning):
             self._is_complete = True
             self._is_invalid = True
         else:
             # Compilation didn't produce any exceptions (though it may not have
             # given a complete code object)
             self._is_complete = self.code is not None
 
         return self._is_complete
 
     def push_accepts_more(self):
         """Return whether a block of interactive input can accept more input.
 
         This method is meant to be used by line-oriented frontends, who need to
         guess whether a block is complete or not based solely on prior and
         current input lines.  The InputSplitter considers it has a complete
         interactive block and will not accept more input when either:
 
         * A SyntaxError is raised
 
         * The code is complete and consists of a single line or a single
           non-compound statement
 
         * The code is complete and has a blank line at the end
 
         If the current input produces a syntax error, this method immediately
         returns False but does *not* raise the syntax error exception, as
         typically clients will want to send invalid syntax to an execution
         backend which might convert the invalid syntax into valid Python via
         one of the dynamic IPython mechanisms.
         """
 
         # With incomplete input, unconditionally accept more
         # A syntax error also sets _is_complete to True - see push()
         if not self._is_complete:
             #print("Not complete")  # debug
             return True
         
         # The user can make any (complete) input execute by leaving a blank line
         last_line = self.source.splitlines()[-1]
         if (not last_line) or last_line.isspace():
             #print("Blank line")  # debug
             return False
         
         # If there's just a single line or AST node, and we're flush left, as is
         # the case after a simple statement such as 'a=1', we want to execute it
         # straight away.
         if self.get_indent_spaces() == 0:
             if len(self.source.splitlines()) <= 1:
                 return False
             
             try:
                 code_ast = ast.parse("".join(self._buffer))
             except Exception:
                 #print("Can't parse AST")  # debug
                 return False
             else:
                 if len(code_ast.body) == 1 and \
                                     not hasattr(code_ast.body[0], 'body'):
                     #print("Simple statement")  # debug
                     return False
 
         # General fallback - accept more code
         return True
 
     def get_indent_spaces(self) -> int:
         sourcefor, n = self._indent_spaces_cache
         if sourcefor == self.source:
             assert n is not None
             return n
 
         # self.source always has a trailing newline
         n = find_next_indent(self.source[:-1])
         self._indent_spaces_cache = (self.source, n)
         return n
 
     # Backwards compatibility. I think all code that used .indent_spaces was
     # inside IPython, but we can leave this here until IPython 7 in case any
     # other modules are using it. -TK, November 2017
     indent_spaces = property(get_indent_spaces)
 
     def _store(self, lines, buffer=None, store='source'):
         """Store one or more lines of input.
 
         If input lines are not newline-terminated, a newline is automatically
         appended."""
 
         if buffer is None:
             buffer = self._buffer
 
         if lines.endswith('\n'):
             buffer.append(lines)
         else:
             buffer.append(lines+'\n')
         setattr(self, store, self._set_source(buffer))
 
     def _set_source(self, buffer):
         return u''.join(buffer)
 
 
 class IPythonInputSplitter(InputSplitter):
     """An input splitter that recognizes all of IPython's special syntax."""
 
     # String with raw, untransformed input.
     source_raw = ''
     
     # Flag to track when a transformer has stored input that it hasn't given
     # back yet.
     transformer_accumulating = False
     
     # Flag to track when assemble_python_lines has stored input that it hasn't
     # given back yet.
     within_python_line = False
 
     # Private attributes
 
     # List with lines of raw input accumulated so far.
     _buffer_raw: List[str]
 
     def __init__(self, line_input_checker=True, physical_line_transforms=None,
                     logical_line_transforms=None, python_line_transforms=None):
         super(IPythonInputSplitter, self).__init__()
         self._buffer_raw = []
         self._validate = True
         
         if physical_line_transforms is not None:
             self.physical_line_transforms = physical_line_transforms
         else:
             self.physical_line_transforms = [
                                              leading_indent(),
                                              classic_prompt(),
                                              ipy_prompt(),
                                              cellmagic(end_on_blank_line=line_input_checker),
                                             ]
         
         self.assemble_logical_lines = assemble_logical_lines()
         if logical_line_transforms is not None:
             self.logical_line_transforms = logical_line_transforms
         else:
             self.logical_line_transforms = [
                                             help_end(),
                                             escaped_commands(),
                                             assign_from_magic(),
                                             assign_from_system(),
                                            ]
         
         self.assemble_python_lines = assemble_python_lines()
         if python_line_transforms is not None:
             self.python_line_transforms = python_line_transforms
         else:
             # We don't use any of these at present
             self.python_line_transforms = []
     
     @property
     def transforms(self):
         "Quick access to all transformers."
         return self.physical_line_transforms + \
             [self.assemble_logical_lines] + self.logical_line_transforms + \
             [self.assemble_python_lines]  + self.python_line_transforms
     
     @property
     def transforms_in_use(self):
         """Transformers, excluding logical line transformers if we're in a
         Python line."""
         t = self.physical_line_transforms[:]
         if not self.within_python_line:
             t += [self.assemble_logical_lines] + self.logical_line_transforms
         return t + [self.assemble_python_lines] + self.python_line_transforms
 
     def reset(self):
         """Reset the input buffer and associated state."""
         super(IPythonInputSplitter, self).reset()
         self._buffer_raw[:] = []
         self.source_raw = ''
         self.transformer_accumulating = False
         self.within_python_line = False
 
         for t in self.transforms:
             try:
                 t.reset()
             except SyntaxError:
                 # Nothing that calls reset() expects to handle transformer
                 # errors
                 pass
 
     def flush_transformers(self: Self):
         def _flush(transform, outs: List[str]):
             """yield transformed lines
 
             always strings, never None
 
             transform: the current transform
             outs: an iterable of previously transformed inputs.
                  Each may be multiline, which will be passed
                  one line at a time to transform.
             """
             for out in outs:
                 for line in out.splitlines():
                     # push one line at a time
                     tmp = transform.push(line)
                     if tmp is not None:
                         yield tmp
             
             # reset the transform
             tmp = transform.reset()
             if tmp is not None:
                 yield tmp
 
         out: List[str] = []
         for t in self.transforms_in_use:
             out = _flush(t, out)
         
         out = list(out)
         if out:
             self._store('\n'.join(out))
 
     def raw_reset(self):
         """Return raw input only and perform a full reset.
         """
         out = self.source_raw
         self.reset()
         return out
     
     def source_reset(self):
         try:
             self.flush_transformers()
             return self.source
         finally:
             self.reset()
 
     def push_accepts_more(self):
         if self.transformer_accumulating:
             return True
         else:
             return super(IPythonInputSplitter, self).push_accepts_more()
 
     def transform_cell(self, cell):
         """Process and translate a cell of input.
         """
         self.reset()
         try:
             self.push(cell)
             self.flush_transformers()
             return self.source
         finally:
             self.reset()
 
     def push(self, lines:str) -> bool:
         """Push one or more lines of IPython input.
 
         This stores the given lines and returns a status code indicating
         whether the code forms a complete Python block or not, after processing
         all input lines for special IPython syntax.
 
         Any exceptions generated in compilation are swallowed, but if an
         exception was produced, the method returns True.
 
         Parameters
         ----------
         lines : string
             One or more lines of Python input.
 
         Returns
         -------
         is_complete : boolean
             True if the current input source (the result of the current input
             plus prior inputs) forms a complete Python execution block.  Note that
             this value is also stored as a private attribute (_is_complete), so it
             can be queried at any time.
         """
         assert isinstance(lines, str)
         # We must ensure all input is pure unicode
         # ''.splitlines() --> [], but we need to push the empty line to transformers
         lines_list = lines.splitlines()
         if not lines_list:
             lines_list = ['']
 
         # Store raw source before applying any transformations to it.  Note
         # that this must be done *after* the reset() call that would otherwise
         # flush the buffer.
         self._store(lines, self._buffer_raw, 'source_raw')
 
         transformed_lines_list = []
         for line in lines_list:
             transformed = self._transform_line(line)
             if transformed is not None:
                 transformed_lines_list.append(transformed)
 
         if transformed_lines_list:
             transformed_lines = '\n'.join(transformed_lines_list)
             return super(IPythonInputSplitter, self).push(transformed_lines)
         else:
             # Got nothing back from transformers - they must be waiting for
             # more input.
             return False
 
     def _transform_line(self, line):
         """Push a line of input code through the various transformers.
 
         Returns any output from the transformers, or None if a transformer
         is accumulating lines.
 
         Sets self.transformer_accumulating as a side effect.
         """
         def _accumulating(dbg):
             #print(dbg)
             self.transformer_accumulating = True
             return None
 
         for transformer in self.physical_line_transforms:
             line = transformer.push(line)
             if line is None:
                 return _accumulating(transformer)
 
         if not self.within_python_line:
             line = self.assemble_logical_lines.push(line)
             if line is None:
                 return _accumulating('acc logical line')
 
             for transformer in self.logical_line_transforms:
                 line = transformer.push(line)
                 if line is None:
                     return _accumulating(transformer)
 
         line = self.assemble_python_lines.push(line)
         if line is None:
             self.within_python_line = True
             return _accumulating('acc python line')
         else:
             self.within_python_line = False
 
         for transformer in self.python_line_transforms:
             line = transformer.push(line)
             if line is None:
                 return _accumulating(transformer)
 
         #print("transformers clear") #debug
         self.transformer_accumulating = False
         return line
 

IPython/core/tests/test_formatters.py

@@ -1,546 +1,545
 """Tests for the Formatters."""
 
 from math import pi
 
 try:
     import numpy
 except:
     numpy = None
 import pytest
 
 from IPython import get_ipython
 from traitlets.config import Config
 from IPython.core.formatters import (
     PlainTextFormatter, HTMLFormatter, PDFFormatter, _mod_name_key,
     DisplayFormatter, JSONFormatter,
 )
 from IPython.utils.io import capture_output
 
 class A(object):
     def __repr__(self):
         return 'A()'
 
 class B(A):
     def __repr__(self):
         return 'B()'
 
 class C:
     pass
 
 class BadRepr(object):
     def __repr__(self):
         raise ValueError("bad repr")
 
 class BadPretty(object):
     _repr_pretty_ = None
 
 class GoodPretty(object):
     def _repr_pretty_(self, pp, cycle):
         pp.text('foo')
 
     def __repr__(self):
         return 'GoodPretty()'
 
 def foo_printer(obj, pp, cycle):
     pp.text('foo')
 
 def test_pretty():
     f = PlainTextFormatter()
     f.for_type(A, foo_printer)
     assert f(A()) == "foo"
     assert f(B()) == "B()"
     assert f(GoodPretty()) == "foo"
     # Just don't raise an exception for the following:
     f(BadPretty())
 
     f.pprint = False
     assert f(A()) == "A()"
     assert f(B()) == "B()"
     assert f(GoodPretty()) == "GoodPretty()"
 
 
 def test_deferred():
     f = PlainTextFormatter()
 
 def test_precision():
     """test various values for float_precision."""
     f = PlainTextFormatter()
     assert f(pi) == repr(pi)
     f.float_precision = 0
     if numpy:
         po = numpy.get_printoptions()
         assert po["precision"] == 0
     assert f(pi) == "3"
     f.float_precision = 2
     if numpy:
         po = numpy.get_printoptions()
         assert po["precision"] == 2
     assert f(pi) == "3.14"
     f.float_precision = "%g"
     if numpy:
         po = numpy.get_printoptions()
         assert po["precision"] == 2
     assert f(pi) == "3.14159"
     f.float_precision = "%e"
     assert f(pi) == "3.141593e+00"
     f.float_precision = ""
     if numpy:
         po = numpy.get_printoptions()
         assert po["precision"] == 8
     assert f(pi) == repr(pi)
 
 
 def test_bad_precision():
     """test various invalid values for float_precision."""
     f = PlainTextFormatter()
     def set_fp(p):
         f.float_precision = p
 
     pytest.raises(ValueError, set_fp, "%")
     pytest.raises(ValueError, set_fp, "%.3f%i")
     pytest.raises(ValueError, set_fp, "foo")
     pytest.raises(ValueError, set_fp, -1)
 
 def test_for_type():
     f = PlainTextFormatter()
 
     # initial return, None
     assert f.for_type(C, foo_printer) is None
     # no func queries
     assert f.for_type(C) is foo_printer
     # shouldn't change anything
     assert f.for_type(C) is foo_printer
     # None should do the same
     assert f.for_type(C, None) is foo_printer
     assert f.for_type(C, None) is foo_printer
 
 def test_for_type_string():
     f = PlainTextFormatter()
 
     type_str = '%s.%s' % (C.__module__, 'C')
 
     # initial return, None
     assert f.for_type(type_str, foo_printer) is None
     # no func queries
     assert f.for_type(type_str) is foo_printer
     assert _mod_name_key(C) in f.deferred_printers
     assert f.for_type(C) is foo_printer
     assert _mod_name_key(C) not in f.deferred_printers
     assert C in f.type_printers
 
 def test_for_type_by_name():
     f = PlainTextFormatter()
 
     mod = C.__module__
 
     # initial return, None
     assert f.for_type_by_name(mod, "C", foo_printer) is None
     # no func queries
     assert f.for_type_by_name(mod, "C") is foo_printer
     # shouldn't change anything
     assert f.for_type_by_name(mod, "C") is foo_printer
     # None should do the same
     assert f.for_type_by_name(mod, "C", None) is foo_printer
     assert f.for_type_by_name(mod, "C", None) is foo_printer
 
 
 def test_lookup():
     f = PlainTextFormatter()
 
     f.for_type(C, foo_printer)
     assert f.lookup(C()) is foo_printer
     with pytest.raises(KeyError):
         f.lookup(A())
 
 def test_lookup_string():
     f = PlainTextFormatter()
     type_str = '%s.%s' % (C.__module__, 'C')
 
     f.for_type(type_str, foo_printer)
     assert f.lookup(C()) is foo_printer
     # should move from deferred to imported dict
     assert _mod_name_key(C) not in f.deferred_printers
     assert C in f.type_printers
 
 def test_lookup_by_type():
     f = PlainTextFormatter()
     f.for_type(C, foo_printer)
     assert f.lookup_by_type(C) is foo_printer
     with pytest.raises(KeyError):
         f.lookup_by_type(A)
 
 def test_lookup_by_type_string():
     f = PlainTextFormatter()
     type_str = '%s.%s' % (C.__module__, 'C')
     f.for_type(type_str, foo_printer)
 
     # verify insertion
     assert _mod_name_key(C) in f.deferred_printers
     assert C not in f.type_printers
 
     assert f.lookup_by_type(type_str) is foo_printer
     # lookup by string doesn't cause import
     assert _mod_name_key(C) in f.deferred_printers
     assert C not in f.type_printers
 
     assert f.lookup_by_type(C) is foo_printer
     # should move from deferred to imported dict
     assert _mod_name_key(C) not in f.deferred_printers
     assert C in f.type_printers
 
 def test_in_formatter():
     f = PlainTextFormatter()
     f.for_type(C, foo_printer)
     type_str = '%s.%s' % (C.__module__, 'C')
     assert C in f
     assert type_str in f
 
 def test_string_in_formatter():
     f = PlainTextFormatter()
     type_str = '%s.%s' % (C.__module__, 'C')
     f.for_type(type_str, foo_printer)
     assert type_str in f
     assert C in f
 
 def test_pop():
     f = PlainTextFormatter()
     f.for_type(C, foo_printer)
     assert f.lookup_by_type(C) is foo_printer
     assert f.pop(C, None) is foo_printer
     f.for_type(C, foo_printer)
     assert f.pop(C) is foo_printer
     with pytest.raises(KeyError):
         f.lookup_by_type(C)
     with pytest.raises(KeyError):
         f.pop(C)
     with pytest.raises(KeyError):
         f.pop(A)
     assert f.pop(A, None) is None
 
 def test_pop_string():
     f = PlainTextFormatter()
     type_str = '%s.%s' % (C.__module__, 'C')
 
     with pytest.raises(KeyError):
         f.pop(type_str)
 
     f.for_type(type_str, foo_printer)
     f.pop(type_str)
     with pytest.raises(KeyError):
         f.lookup_by_type(C)
     with pytest.raises(KeyError):
         f.pop(type_str)
 
     f.for_type(C, foo_printer)
     assert f.pop(type_str, None) is foo_printer
     with pytest.raises(KeyError):
         f.lookup_by_type(C)
     with pytest.raises(KeyError):
         f.pop(type_str)
     assert f.pop(type_str, None) is None
 
 
 def test_error_method():
     f = HTMLFormatter()
     class BadHTML(object):
         def _repr_html_(self):
             raise ValueError("Bad HTML")
     bad = BadHTML()
     with capture_output() as captured:
         result = f(bad)
     assert result is None
     assert "Traceback" in captured.stdout
     assert "Bad HTML" in captured.stdout
     assert "_repr_html_" in captured.stdout
 
 def test_nowarn_notimplemented():
     f = HTMLFormatter()
     class HTMLNotImplemented(object):
         def _repr_html_(self):
             raise NotImplementedError
     h = HTMLNotImplemented()
     with capture_output() as captured:
         result = f(h)
     assert result is None
     assert "" == captured.stderr
     assert "" == captured.stdout
 
 
 def test_warn_error_for_type():
     f = HTMLFormatter()
     f.for_type(int, lambda i: name_error)
     with capture_output() as captured:
         result = f(5)
     assert result is None
     assert "Traceback" in captured.stdout
     assert "NameError" in captured.stdout
     assert "name_error" in captured.stdout
 
 def test_error_pretty_method():
     f = PlainTextFormatter()
     class BadPretty(object):
         def _repr_pretty_(self):
             return "hello"
     bad = BadPretty()
     with capture_output() as captured:
         result = f(bad)
     assert result is None
     assert "Traceback" in captured.stdout
     assert "_repr_pretty_" in captured.stdout
     assert "given" in captured.stdout
     assert "argument" in captured.stdout
 
 
 def test_bad_repr_traceback():
     f = PlainTextFormatter()
     bad = BadRepr()
     with capture_output() as captured:
         result = f(bad)
     # catches error, returns None
     assert result is None
     assert "Traceback" in captured.stdout
     assert "__repr__" in captured.stdout
     assert "ValueError" in captured.stdout
 
 
 class MakePDF(object):
     def _repr_pdf_(self):
         return 'PDF'
 
 def test_pdf_formatter():
     pdf = MakePDF()
     f = PDFFormatter()
     assert f(pdf) == "PDF"
 
 
 def test_print_method_bound():
     f = HTMLFormatter()
     class MyHTML(object):
         def _repr_html_(self):
             return "hello"
     with capture_output() as captured:
         result = f(MyHTML)
     assert result is None
     assert "FormatterWarning" not in captured.stderr
 
     with capture_output() as captured:
         result = f(MyHTML())
     assert result == "hello"
     assert captured.stderr == ""
 
 
 def test_print_method_weird():
 
     class TextMagicHat(object):
         def __getattr__(self, key):
             return key
 
     f = HTMLFormatter()
 
     text_hat = TextMagicHat()
     assert text_hat._repr_html_ == "_repr_html_"
     with capture_output() as captured:
         result = f(text_hat)
 
     assert result is None
     assert "FormatterWarning" not in captured.stderr
 
     class CallableMagicHat(object):
         def __getattr__(self, key):
             return lambda : key
 
     call_hat = CallableMagicHat()
     with capture_output() as captured:
         result = f(call_hat)
 
     assert result is None
 
     class BadReprArgs(object):
         def _repr_html_(self, extra, args):
             return "html"
 
     bad = BadReprArgs()
     with capture_output() as captured:
         result = f(bad)
 
     assert result is None
     assert "FormatterWarning" not in captured.stderr
 
 
 def test_format_config():
     """config objects don't pretend to support fancy reprs with lazy attrs"""
     f = HTMLFormatter()
     cfg = Config()
     with capture_output() as captured:
         result = f(cfg)
     assert result is None
     assert captured.stderr == ""
 
     with capture_output() as captured:
         result = f(Config)
     assert result is None
     assert captured.stderr == ""
 
 
 def test_pretty_max_seq_length():
     f = PlainTextFormatter(max_seq_length=1)
     lis = list(range(3))
     text = f(lis)
     assert text == "[0, ...]"
     f.max_seq_length = 0
     text = f(lis)
     assert text == "[0, 1, 2]"
     text = f(list(range(1024)))
     lines = text.splitlines()
     assert len(lines) == 1024
 
 
 def test_ipython_display_formatter():
     """Objects with _ipython_display_ defined bypass other formatters"""
     f = get_ipython().display_formatter
     catcher = []
     class SelfDisplaying(object):
         def _ipython_display_(self):
             catcher.append(self)
 
     class NotSelfDisplaying(object):
         def __repr__(self):
             return "NotSelfDisplaying"
 
         def _ipython_display_(self):
             raise NotImplementedError
 
     save_enabled = f.ipython_display_formatter.enabled
     f.ipython_display_formatter.enabled = True
 
     yes = SelfDisplaying()
     no = NotSelfDisplaying()
 
     d, md = f.format(no)
     assert d == {"text/plain": repr(no)}
     assert md == {}
     assert catcher == []
 
     d, md = f.format(yes)
     assert d == {}
     assert md == {}
     assert catcher == [yes]
 
     f.ipython_display_formatter.enabled = save_enabled
 
 
 def test_repr_mime():
     class HasReprMime(object):
         def _repr_mimebundle_(self, include=None, exclude=None):
             return {
                 'application/json+test.v2': {
                     'x': 'y'
                 },
                 'plain/text' : '<HasReprMime>',
                 'image/png' : 'i-overwrite'
             }
 
         def _repr_png_(self):
             return 'should-be-overwritten'
         def _repr_html_(self):
             return '<b>hi!</b>'
 
     f = get_ipython().display_formatter
     html_f = f.formatters['text/html']
     save_enabled = html_f.enabled
     html_f.enabled = True
     obj = HasReprMime()
     d, md = f.format(obj)
     html_f.enabled = save_enabled
 
     assert sorted(d) == [
         "application/json+test.v2",
         "image/png",
         "plain/text",
         "text/html",
         "text/plain",
     ]
     assert md == {}
 
     d, md = f.format(obj, include={"image/png"})
     assert list(d.keys()) == [
         "image/png"
     ], "Include should filter out even things from repr_mimebundle"
 
     assert d["image/png"] == "i-overwrite", "_repr_mimebundle_ take precedence"
 
 
 def test_pass_correct_include_exclude():
     class Tester(object):
 
         def __init__(self, include=None, exclude=None):
             self.include = include
             self.exclude = exclude
 
         def _repr_mimebundle_(self, include, exclude, **kwargs):
             if include and (include != self.include):
                 raise ValueError('include got modified: display() may be broken.')
             if exclude and (exclude != self.exclude):
                 raise ValueError('exclude got modified: display() may be broken.')
 
             return None
 
     include = {'a', 'b', 'c'}
     exclude = {'c', 'e' , 'f'}
 
     f = get_ipython().display_formatter
     f.format(Tester(include=include, exclude=exclude), include=include, exclude=exclude)
     f.format(Tester(exclude=exclude), exclude=exclude)
     f.format(Tester(include=include), include=include)
 
 
 def test_repr_mime_meta():
     class HasReprMimeMeta(object):
         def _repr_mimebundle_(self, include=None, exclude=None):
             data = {
                 'image/png': 'base64-image-data',
             }
             metadata = {
                 'image/png': {
                     'width': 5,
                     'height': 10,
                 }
             }
             return (data, metadata)
 
     f = get_ipython().display_formatter
     obj = HasReprMimeMeta()
     d, md = f.format(obj)
     assert sorted(d) == ["image/png", "text/plain"]
     assert md == {
         "image/png": {
             "width": 5,
             "height": 10,
         }
     }
 
 
 def test_repr_mime_failure():
     class BadReprMime(object):
         def _repr_mimebundle_(self, include=None, exclude=None):
             raise RuntimeError
 
     f = get_ipython().display_formatter
     obj = BadReprMime()
     d, md = f.format(obj)
     assert "text/plain" in d
 
 
 def test_custom_repr_namedtuple_partialmethod():
     from functools import partialmethod
     from typing import NamedTuple
 
-    class Foo(NamedTuple):
-        ...
+    class Foo(NamedTuple): ...
 
     Foo.__repr__ = partialmethod(lambda obj: "Hello World")
     foo = Foo()
 
     f = PlainTextFormatter()
     assert f.pprint
     assert f(foo) == "Hello World"

IPython/core/tests/test_inputtransformer2.py

@@ -1,447 +1,448
 """Tests for the token-based transformers in IPython.core.inputtransformer2
 
 Line-based transformers are the simpler ones; token-based transformers are
 more complex. See test_inputtransformer2_line for tests for line-based
 transformations.
 """
+
 import platform
 import string
 import sys
 from textwrap import dedent
 
 import pytest
 
 from IPython.core import inputtransformer2 as ipt2
 from IPython.core.inputtransformer2 import _find_assign_op, make_tokens_by_line
 
 MULTILINE_MAGIC = (
     """\
 a = f()
 %foo \\
 bar
 g()
 """.splitlines(
         keepends=True
     ),
     (2, 0),
     """\
 a = f()
 get_ipython().run_line_magic('foo', ' bar')
 g()
 """.splitlines(
         keepends=True
     ),
 )
 
 INDENTED_MAGIC = (
     """\
 for a in range(5):
     %ls
 """.splitlines(
         keepends=True
     ),
     (2, 4),
     """\
 for a in range(5):
     get_ipython().run_line_magic('ls', '')
 """.splitlines(
         keepends=True
     ),
 )
 
 CRLF_MAGIC = (
     ["a = f()\n", "%ls\r\n", "g()\n"],
     (2, 0),
     ["a = f()\n", "get_ipython().run_line_magic('ls', '')\n", "g()\n"],
 )
 
 MULTILINE_MAGIC_ASSIGN = (
     """\
 a = f()
 b = %foo \\
   bar
 g()
 """.splitlines(
         keepends=True
     ),
     (2, 4),
     """\
 a = f()
 b = get_ipython().run_line_magic('foo', '   bar')
 g()
 """.splitlines(
         keepends=True
     ),
 )
 
 MULTILINE_SYSTEM_ASSIGN = ("""\
 a = f()
 b = !foo \\
   bar
 g()
 """.splitlines(keepends=True), (2, 4), """\
 a = f()
 b = get_ipython().getoutput('foo    bar')
 g()
 """.splitlines(keepends=True))
 
 #####
 
 MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT = (
     """\
 def test():
   for i in range(1):
     print(i)
   res =! ls
 """.splitlines(
         keepends=True
     ),
     (4, 7),
     """\
 def test():
   for i in range(1):
     print(i)
   res =get_ipython().getoutput(\' ls\')
 """.splitlines(
         keepends=True
     ),
 )
 
 ######
 
 AUTOCALL_QUOTE = ([",f 1 2 3\n"], (1, 0), ['f("1", "2", "3")\n'])
 
 AUTOCALL_QUOTE2 = ([";f 1 2 3\n"], (1, 0), ['f("1 2 3")\n'])
 
 AUTOCALL_PAREN = (["/f 1 2 3\n"], (1, 0), ["f(1, 2, 3)\n"])
 
 SIMPLE_HELP = (["foo?\n"], (1, 0), ["get_ipython().run_line_magic('pinfo', 'foo')\n"])
 
 DETAILED_HELP = (
     ["foo??\n"],
     (1, 0),
     ["get_ipython().run_line_magic('pinfo2', 'foo')\n"],
 )
 
 MAGIC_HELP = (["%foo?\n"], (1, 0), ["get_ipython().run_line_magic('pinfo', '%foo')\n"])
 
 HELP_IN_EXPR = (
     ["a = b + c?\n"],
     (1, 0),
     ["get_ipython().run_line_magic('pinfo', 'c')\n"],
 )
 
 HELP_CONTINUED_LINE = (
     """\
 a = \\
 zip?
 """.splitlines(
         keepends=True
     ),
     (1, 0),
     [r"get_ipython().run_line_magic('pinfo', 'zip')" + "\n"],
 )
 
 HELP_MULTILINE = (
     """\
 (a,
 b) = zip?
 """.splitlines(
         keepends=True
     ),
     (1, 0),
     [r"get_ipython().run_line_magic('pinfo', 'zip')" + "\n"],
 )
 
 HELP_UNICODE = (
     ["π.foo?\n"],
     (1, 0),
     ["get_ipython().run_line_magic('pinfo', 'π.foo')\n"],
 )
 
 
 def null_cleanup_transformer(lines):
     """
     A cleanup transform that returns an empty list.
     """
     return []
 
 
 def test_check_make_token_by_line_never_ends_empty():
     """
     Check that not sequence of single or double characters ends up leading to en empty list of tokens
     """
     from string import printable
 
     for c in printable:
         assert make_tokens_by_line(c)[-1] != []
         for k in printable:
             assert make_tokens_by_line(c + k)[-1] != []
 
 
 def check_find(transformer, case, match=True):
     sample, expected_start, _ = case
     tbl = make_tokens_by_line(sample)
     res = transformer.find(tbl)
     if match:
         # start_line is stored 0-indexed, expected values are 1-indexed
         assert (res.start_line + 1, res.start_col) == expected_start
         return res
     else:
         assert res is None
 
 
 def check_transform(transformer_cls, case):
     lines, start, expected = case
     transformer = transformer_cls(start)
     assert transformer.transform(lines) == expected
 
 
 def test_continued_line():
     lines = MULTILINE_MAGIC_ASSIGN[0]
     assert ipt2.find_end_of_continued_line(lines, 1) == 2
 
     assert ipt2.assemble_continued_line(lines, (1, 5), 2) == "foo    bar"
 
 
 def test_find_assign_magic():
     check_find(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
     check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN, match=False)
     check_find(ipt2.MagicAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT, match=False)
 
 
 def test_transform_assign_magic():
     check_transform(ipt2.MagicAssign, MULTILINE_MAGIC_ASSIGN)
 
 
 def test_find_assign_system():
     check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
     check_find(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
     check_find(ipt2.SystemAssign, (["a =  !ls\n"], (1, 5), None))
     check_find(ipt2.SystemAssign, (["a=!ls\n"], (1, 2), None))
     check_find(ipt2.SystemAssign, MULTILINE_MAGIC_ASSIGN, match=False)
 
 
 def test_transform_assign_system():
     check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN)
     check_transform(ipt2.SystemAssign, MULTILINE_SYSTEM_ASSIGN_AFTER_DEDENT)
 
 
 def test_find_magic_escape():
     check_find(ipt2.EscapedCommand, MULTILINE_MAGIC)
     check_find(ipt2.EscapedCommand, INDENTED_MAGIC)
     check_find(ipt2.EscapedCommand, MULTILINE_MAGIC_ASSIGN, match=False)
 
 
 def test_transform_magic_escape():
     check_transform(ipt2.EscapedCommand, MULTILINE_MAGIC)
     check_transform(ipt2.EscapedCommand, INDENTED_MAGIC)
     check_transform(ipt2.EscapedCommand, CRLF_MAGIC)
 
 
 def test_find_autocalls():
     for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
         print("Testing %r" % case[0])
         check_find(ipt2.EscapedCommand, case)
 
 
 def test_transform_autocall():
     for case in [AUTOCALL_QUOTE, AUTOCALL_QUOTE2, AUTOCALL_PAREN]:
         print("Testing %r" % case[0])
         check_transform(ipt2.EscapedCommand, case)
 
 
 def test_find_help():
     for case in [SIMPLE_HELP, DETAILED_HELP, MAGIC_HELP, HELP_IN_EXPR]:
         check_find(ipt2.HelpEnd, case)
 
     tf = check_find(ipt2.HelpEnd, HELP_CONTINUED_LINE)
     assert tf.q_line == 1
     assert tf.q_col == 3
 
     tf = check_find(ipt2.HelpEnd, HELP_MULTILINE)
     assert tf.q_line == 1
     assert tf.q_col == 8
 
     # ? in a comment does not trigger help
     check_find(ipt2.HelpEnd, (["foo # bar?\n"], None, None), match=False)
     # Nor in a string
     check_find(ipt2.HelpEnd, (["foo = '''bar?\n"], None, None), match=False)
 
 
 def test_transform_help():
     tf = ipt2.HelpEnd((1, 0), (1, 9))
     assert tf.transform(HELP_IN_EXPR[0]) == HELP_IN_EXPR[2]
 
     tf = ipt2.HelpEnd((1, 0), (2, 3))
     assert tf.transform(HELP_CONTINUED_LINE[0]) == HELP_CONTINUED_LINE[2]
 
     tf = ipt2.HelpEnd((1, 0), (2, 8))
     assert tf.transform(HELP_MULTILINE[0]) == HELP_MULTILINE[2]
 
     tf = ipt2.HelpEnd((1, 0), (1, 0))
     assert tf.transform(HELP_UNICODE[0]) == HELP_UNICODE[2]
 
 
 def test_find_assign_op_dedent():
     """
     be careful that empty token like dedent are not counted as parens
     """
 
     class Tk:
         def __init__(self, s):
             self.string = s
 
     assert _find_assign_op([Tk(s) for s in ("", "a", "=", "b")]) == 2
     assert (
         _find_assign_op([Tk(s) for s in ("", "(", "a", "=", "b", ")", "=", "5")]) == 6
     )
 
 
 extra_closing_paren_param = (
     pytest.param("(\n))", "invalid", None)
     if sys.version_info >= (3, 12)
     else pytest.param("(\n))", "incomplete", 0)
 )
 examples = [
     pytest.param("a = 1", "complete", None),
     pytest.param("for a in range(5):", "incomplete", 4),
     pytest.param("for a in range(5):\n    if a > 0:", "incomplete", 8),
     pytest.param("raise = 2", "invalid", None),
     pytest.param("a = [1,\n2,", "incomplete", 0),
     extra_closing_paren_param,
     pytest.param("\\\r\n", "incomplete", 0),
     pytest.param("a = '''\n   hi", "incomplete", 3),
     pytest.param("def a():\n x=1\n global x", "invalid", None),
     pytest.param(
         "a \\ ",
         "invalid",
         None,
         marks=pytest.mark.xfail(
             reason="Bug in python 3.9.8 – bpo 45738",
             condition=sys.version_info in [(3, 11, 0, "alpha", 2)],
             raises=SystemError,
             strict=True,
         ),
     ),  # Nothing allowed after backslash,
     pytest.param("1\\\n+2", "complete", None),
 ]
 
 
 @pytest.mark.parametrize("code, expected, number", examples)
 def test_check_complete_param(code, expected, number):
     cc = ipt2.TransformerManager().check_complete
     assert cc(code) == (expected, number)
 
 
 @pytest.mark.xfail(platform.python_implementation() == "PyPy", reason="fail on pypy")
 @pytest.mark.xfail(
     reason="Bug in python 3.9.8 – bpo 45738",
     condition=sys.version_info in [(3, 11, 0, "alpha", 2)],
     raises=SystemError,
     strict=True,
 )
 def test_check_complete():
     cc = ipt2.TransformerManager().check_complete
 
     example = dedent(
         """
         if True:
             a=1"""
     )
 
     assert cc(example) == ("incomplete", 4)
     assert cc(example + "\n") == ("complete", None)
     assert cc(example + "\n    ") == ("complete", None)
 
     # no need to loop on all the letters/numbers.
     short = "12abAB" + string.printable[62:]
     for c in short:
         # test does not raise:
         cc(c)
         for k in short:
             cc(c + k)
 
     assert cc("def f():\n  x=0\n  \\\n  ") == ("incomplete", 2)
 
 
 @pytest.mark.xfail(platform.python_implementation() == "PyPy", reason="fail on pypy")
 @pytest.mark.parametrize(
     "value, expected",
     [
         ('''def foo():\n    """''', ("incomplete", 4)),
         ("""async with example:\n    pass""", ("incomplete", 4)),
         ("""async with example:\n    pass\n    """, ("complete", None)),
     ],
 )
 def test_check_complete_II(value, expected):
     """
     Test that multiple line strings are properly handled.
 
     Separate test function for convenience
 
     """
     cc = ipt2.TransformerManager().check_complete
     assert cc(value) == expected
 
 
 @pytest.mark.parametrize(
     "value, expected",
     [
         (")", ("invalid", None)),
         ("]", ("invalid", None)),
         ("}", ("invalid", None)),
         (")(", ("invalid", None)),
         ("][", ("invalid", None)),
         ("}{", ("invalid", None)),
         ("]()(", ("invalid", None)),
         ("())(", ("invalid", None)),
         (")[](", ("invalid", None)),
         ("()](", ("invalid", None)),
     ],
 )
 def test_check_complete_invalidates_sunken_brackets(value, expected):
     """
     Test that a single line with more closing brackets than the opening ones is
     interpreted as invalid
     """
     cc = ipt2.TransformerManager().check_complete
     assert cc(value) == expected
 
 
 def test_null_cleanup_transformer():
     manager = ipt2.TransformerManager()
     manager.cleanup_transforms.insert(0, null_cleanup_transformer)
     assert manager.transform_cell("") == ""
 
 
 def test_side_effects_I():
     count = 0
 
     def counter(lines):
         nonlocal count
         count += 1
         return lines
 
     counter.has_side_effects = True
 
     manager = ipt2.TransformerManager()
     manager.cleanup_transforms.insert(0, counter)
     assert manager.check_complete("a=1\n") == ("complete", None)
     assert count == 0
 
 
 def test_side_effects_II():
     count = 0
 
     def counter(lines):
         nonlocal count
         count += 1
         return lines
 
     counter.has_side_effects = True
 
     manager = ipt2.TransformerManager()
     manager.line_transforms.insert(0, counter)
     assert manager.check_complete("b=1\n") == ("complete", None)
     assert count == 0

IPython/core/tests/test_paths.py

@@ -1,200 +1,202
 import errno
 import os
 import shutil
 import tempfile
 import warnings
 from unittest.mock import patch
 
 from tempfile import TemporaryDirectory
 from testpath import assert_isdir, assert_isfile, modified_env
 
 from IPython import paths
 from IPython.testing.decorators import skip_win32
 
 TMP_TEST_DIR = os.path.realpath(tempfile.mkdtemp())
 HOME_TEST_DIR = os.path.join(TMP_TEST_DIR, "home_test_dir")
 XDG_TEST_DIR = os.path.join(HOME_TEST_DIR, "xdg_test_dir")
 XDG_CACHE_DIR = os.path.join(HOME_TEST_DIR, "xdg_cache_dir")
 IP_TEST_DIR = os.path.join(HOME_TEST_DIR,'.ipython')
 
 def setup_module():
     """Setup testenvironment for the module:
 
             - Adds dummy home dir tree
     """
     # Do not mask exceptions here.  In particular, catching WindowsError is a
     # problem because that exception is only defined on Windows...
     os.makedirs(IP_TEST_DIR)
     os.makedirs(os.path.join(XDG_TEST_DIR, 'ipython'))
     os.makedirs(os.path.join(XDG_CACHE_DIR, 'ipython'))
 
 
 def teardown_module():
     """Teardown testenvironment for the module:
 
             - Remove dummy home dir tree
     """
     # Note: we remove the parent test dir, which is the root of all test
     # subdirs we may have created.  Use shutil instead of os.removedirs, so
     # that non-empty directories are all recursively removed.
     shutil.rmtree(TMP_TEST_DIR)
 
 def patch_get_home_dir(dirpath):
     return patch.object(paths, 'get_home_dir', return_value=dirpath)
 
 
 def test_get_ipython_dir_1():
     """test_get_ipython_dir_1, Testcase to see if we can call get_ipython_dir without Exceptions."""
     env_ipdir = os.path.join("someplace", ".ipython")
     with patch.object(paths, '_writable_dir', return_value=True), \
             modified_env({'IPYTHONDIR': env_ipdir}):
         ipdir = paths.get_ipython_dir()
 
     assert ipdir == env_ipdir
 
 def test_get_ipython_dir_2():
     """test_get_ipython_dir_2, Testcase to see if we can call get_ipython_dir without Exceptions."""
     with patch_get_home_dir('someplace'), \
             patch.object(paths, 'get_xdg_dir', return_value=None), \
             patch.object(paths, '_writable_dir', return_value=True), \
             patch('os.name', "posix"), \
             modified_env({'IPYTHON_DIR': None,
                           'IPYTHONDIR': None,
                           'XDG_CONFIG_HOME': None
                          }):
         ipdir = paths.get_ipython_dir()
 
     assert ipdir == os.path.join("someplace", ".ipython")
 
 def test_get_ipython_dir_3():
     """test_get_ipython_dir_3, use XDG if defined and exists, and .ipython doesn't exist."""
     tmphome = TemporaryDirectory()
     try:
         with patch_get_home_dir(tmphome.name), \
                 patch('os.name', 'posix'), \
                 modified_env({
                     'IPYTHON_DIR': None,
                     'IPYTHONDIR': None,
                     'XDG_CONFIG_HOME': XDG_TEST_DIR,
                 }), warnings.catch_warnings(record=True) as w:
             ipdir = paths.get_ipython_dir()
 
         assert ipdir == os.path.join(tmphome.name, XDG_TEST_DIR, "ipython")
         assert len(w) == 0
     finally:
         tmphome.cleanup()
 
 def test_get_ipython_dir_4():
     """test_get_ipython_dir_4, warn if XDG and home both exist."""
     with patch_get_home_dir(HOME_TEST_DIR), \
             patch('os.name', 'posix'):
         try:
             os.mkdir(os.path.join(XDG_TEST_DIR, 'ipython'))
         except OSError as e:
             if e.errno != errno.EEXIST:
                 raise
 
 
         with modified_env({
             'IPYTHON_DIR': None,
             'IPYTHONDIR': None,
             'XDG_CONFIG_HOME': XDG_TEST_DIR,
         }), warnings.catch_warnings(record=True) as w:
             ipdir = paths.get_ipython_dir()
 
         assert len(w) == 1
         assert "Ignoring" in str(w[0])
 
 
 def test_get_ipython_dir_5():
     """test_get_ipython_dir_5, use .ipython if exists and XDG defined, but doesn't exist."""
     with patch_get_home_dir(HOME_TEST_DIR), \
             patch('os.name', 'posix'):
         try:
             os.rmdir(os.path.join(XDG_TEST_DIR, 'ipython'))
         except OSError as e:
             if e.errno != errno.ENOENT:
                 raise
 
         with modified_env({
             'IPYTHON_DIR': None,
             'IPYTHONDIR': None,
             'XDG_CONFIG_HOME': XDG_TEST_DIR,
         }):
             ipdir = paths.get_ipython_dir()
 
         assert ipdir == IP_TEST_DIR
 
 def test_get_ipython_dir_6():
     """test_get_ipython_dir_6, use home over XDG if defined and neither exist."""
     xdg = os.path.join(HOME_TEST_DIR, 'somexdg')
     os.mkdir(xdg)
     shutil.rmtree(os.path.join(HOME_TEST_DIR, '.ipython'))
     print(paths._writable_dir)
     with patch_get_home_dir(HOME_TEST_DIR), \
             patch.object(paths, 'get_xdg_dir', return_value=xdg), \
             patch('os.name', 'posix'), \
             modified_env({
                 'IPYTHON_DIR': None,
                 'IPYTHONDIR': None,
                 'XDG_CONFIG_HOME': None,
             }), warnings.catch_warnings(record=True) as w:
         ipdir = paths.get_ipython_dir()
 
     assert ipdir == os.path.join(HOME_TEST_DIR, ".ipython")
     assert len(w) == 0
 
 def test_get_ipython_dir_7():
     """test_get_ipython_dir_7, test home directory expansion on IPYTHONDIR"""
     home_dir = os.path.normpath(os.path.expanduser('~'))
     with modified_env({'IPYTHONDIR': os.path.join('~', 'somewhere')}), \
             patch.object(paths, '_writable_dir', return_value=True):
         ipdir = paths.get_ipython_dir()
     assert ipdir == os.path.join(home_dir, "somewhere")
 
 
 @skip_win32
 def test_get_ipython_dir_8():
     """test_get_ipython_dir_8, test / home directory"""
     if not os.access("/", os.W_OK):
         # test only when HOME directory actually writable
         return
 
-    with patch.object(paths, "_writable_dir", lambda path: bool(path)), patch.object(
-        paths, "get_xdg_dir", return_value=None
-    ), modified_env(
+    with (
+        patch.object(paths, "_writable_dir", lambda path: bool(path)),
+        patch.object(paths, "get_xdg_dir", return_value=None),
+        modified_env(
         {
             "IPYTHON_DIR": None,
             "IPYTHONDIR": None,
             "HOME": "/",
         }
+        ),
     ):
         assert paths.get_ipython_dir() == "/.ipython"
 
 
 def test_get_ipython_cache_dir():
     with modified_env({'HOME': HOME_TEST_DIR}):
         if os.name == "posix":
             # test default
             os.makedirs(os.path.join(HOME_TEST_DIR, ".cache"))
             with modified_env({'XDG_CACHE_HOME': None}):
                 ipdir = paths.get_ipython_cache_dir()
             assert os.path.join(HOME_TEST_DIR, ".cache", "ipython") == ipdir
             assert_isdir(ipdir)
 
             # test env override
             with modified_env({"XDG_CACHE_HOME": XDG_CACHE_DIR}):
                 ipdir = paths.get_ipython_cache_dir()
             assert_isdir(ipdir)
             assert ipdir == os.path.join(XDG_CACHE_DIR, "ipython")
         else:
             assert paths.get_ipython_cache_dir() == paths.get_ipython_dir()
 
 def test_get_ipython_package_dir():
     ipdir = paths.get_ipython_package_dir()
     assert_isdir(ipdir)
 
 
 def test_get_ipython_module_path():
     ipapp_path = paths.get_ipython_module_path('IPython.terminal.ipapp')
     assert_isfile(ipapp_path)

IPython/external/qt_loaders.py

@@ -1,422 +1,423
 """
 This module contains factory functions that attempt
 to return Qt submodules from the various python Qt bindings.
 
 It also protects against double-importing Qt with different
 bindings, which is unstable and likely to crash
 
 This is used primarily by qt and qt_for_kernel, and shouldn't
 be accessed directly from the outside
 """
+
 import importlib.abc
 import sys
 import os
 import types
 from functools import partial, lru_cache
 import operator
 
 # ### Available APIs.
 # Qt6
 QT_API_PYQT6 = "pyqt6"
 QT_API_PYSIDE6 = "pyside6"
 
 # Qt5
 QT_API_PYQT5 = 'pyqt5'
 QT_API_PYSIDE2 = 'pyside2'
 
 # Qt4
 # NOTE: Here for legacy matplotlib compatibility, but not really supported on the IPython side.
 QT_API_PYQT = "pyqt"  # Force version 2
 QT_API_PYQTv1 = "pyqtv1"  # Force version 2
 QT_API_PYSIDE = "pyside"
 
 QT_API_PYQT_DEFAULT = "pyqtdefault"  # use system default for version 1 vs. 2
 
 api_to_module = {
     # Qt6
     QT_API_PYQT6: "PyQt6",
     QT_API_PYSIDE6: "PySide6",
     # Qt5
     QT_API_PYQT5: "PyQt5",
     QT_API_PYSIDE2: "PySide2",
     # Qt4
     QT_API_PYSIDE: "PySide",
     QT_API_PYQT: "PyQt4",
     QT_API_PYQTv1: "PyQt4",
     # default
     QT_API_PYQT_DEFAULT: "PyQt6",
 }
 
 
 class ImportDenier(importlib.abc.MetaPathFinder):
     """Import Hook that will guard against bad Qt imports
     once IPython commits to a specific binding
     """
 
     def __init__(self):
         self.__forbidden = set()
 
     def forbid(self, module_name):
         sys.modules.pop(module_name, None)
         self.__forbidden.add(module_name)
 
     def find_spec(self, fullname, path, target=None):
         if path:
             return
         if fullname in self.__forbidden:
             raise ImportError(
                 """
     Importing %s disabled by IPython, which has
     already imported an Incompatible QT Binding: %s
     """
                 % (fullname, loaded_api())
             )
 
 
 ID = ImportDenier()
 sys.meta_path.insert(0, ID)
 
 
 def commit_api(api):
     """Commit to a particular API, and trigger ImportErrors on subsequent
     dangerous imports"""
     modules = set(api_to_module.values())
 
     modules.remove(api_to_module[api])
     for mod in modules:
         ID.forbid(mod)
 
 
 def loaded_api():
     """Return which API is loaded, if any
 
     If this returns anything besides None,
     importing any other Qt binding is unsafe.
 
     Returns
     -------
     None, 'pyside6', 'pyqt6', 'pyside2', 'pyside', 'pyqt', 'pyqt5', 'pyqtv1'
     """
     if sys.modules.get("PyQt6.QtCore"):
         return QT_API_PYQT6
     elif sys.modules.get("PySide6.QtCore"):
         return QT_API_PYSIDE6
     elif sys.modules.get("PyQt5.QtCore"):
         return QT_API_PYQT5
     elif sys.modules.get("PySide2.QtCore"):
         return QT_API_PYSIDE2
     elif sys.modules.get("PyQt4.QtCore"):
         if qtapi_version() == 2:
             return QT_API_PYQT
         else:
             return QT_API_PYQTv1
     elif sys.modules.get("PySide.QtCore"):
         return QT_API_PYSIDE
 
     return None
 
 
 def has_binding(api):
     """Safely check for PyQt4/5, PySide or PySide2, without importing submodules
 
     Parameters
     ----------
     api : str [ 'pyqtv1' | 'pyqt' | 'pyqt5' | 'pyside' | 'pyside2' | 'pyqtdefault']
         Which module to check for
 
     Returns
     -------
     True if the relevant module appears to be importable
     """
     module_name = api_to_module[api]
     from importlib.util import find_spec
 
     required = ['QtCore', 'QtGui', 'QtSvg']
     if api in (QT_API_PYQT5, QT_API_PYSIDE2, QT_API_PYQT6, QT_API_PYSIDE6):
         # QT5 requires QtWidgets too
         required.append('QtWidgets')
 
     for submod in required:
         try:
             spec = find_spec('%s.%s' % (module_name, submod))
         except ImportError:
             # Package (e.g. PyQt5) not found
             return False
         else:
             if spec is None:
                 # Submodule (e.g. PyQt5.QtCore) not found
                 return False
 
     if api == QT_API_PYSIDE:
         # We can also safely check PySide version
         import PySide
 
         return PySide.__version_info__ >= (1, 0, 3)
 
     return True
 
 
 def qtapi_version():
     """Return which QString API has been set, if any
 
     Returns
     -------
     The QString API version (1 or 2), or None if not set
     """
     try:
         import sip
     except ImportError:
         # as of PyQt5 5.11, sip is no longer available as a top-level
         # module and needs to be imported from the PyQt5 namespace
         try:
             from PyQt5 import sip
         except ImportError:
             return
     try:
         return sip.getapi('QString')
     except ValueError:
         return
 
 
 def can_import(api):
     """Safely query whether an API is importable, without importing it"""
     if not has_binding(api):
         return False
 
     current = loaded_api()
     if api == QT_API_PYQT_DEFAULT:
         return current in [QT_API_PYQT6, None]
     else:
         return current in [api, None]
 
 
 def import_pyqt4(version=2):
     """
     Import PyQt4
 
     Parameters
     ----------
     version : 1, 2, or None
         Which QString/QVariant API to use. Set to None to use the system
         default
     ImportErrors raised within this function are non-recoverable
     """
     # The new-style string API (version=2) automatically
     # converts QStrings to Unicode Python strings. Also, automatically unpacks
     # QVariants to their underlying objects.
     import sip
 
     if version is not None:
         sip.setapi('QString', version)
         sip.setapi('QVariant', version)
 
     from PyQt4 import QtGui, QtCore, QtSvg
 
     if QtCore.PYQT_VERSION < 0x040700:
         raise ImportError("IPython requires PyQt4 >= 4.7, found %s" %
                           QtCore.PYQT_VERSION_STR)
 
     # Alias PyQt-specific functions for PySide compatibility.
     QtCore.Signal = QtCore.pyqtSignal
     QtCore.Slot = QtCore.pyqtSlot
 
     # query for the API version (in case version == None)
     version = sip.getapi('QString')
     api = QT_API_PYQTv1 if version == 1 else QT_API_PYQT
     return QtCore, QtGui, QtSvg, api
 
 
 def import_pyqt5():
     """
     Import PyQt5
 
     ImportErrors raised within this function are non-recoverable
     """
 
     from PyQt5 import QtCore, QtSvg, QtWidgets, QtGui
 
     # Alias PyQt-specific functions for PySide compatibility.
     QtCore.Signal = QtCore.pyqtSignal
     QtCore.Slot = QtCore.pyqtSlot
 
     # Join QtGui and QtWidgets for Qt4 compatibility.
     QtGuiCompat = types.ModuleType('QtGuiCompat')
     QtGuiCompat.__dict__.update(QtGui.__dict__)
     QtGuiCompat.__dict__.update(QtWidgets.__dict__)
 
     api = QT_API_PYQT5
     return QtCore, QtGuiCompat, QtSvg, api
 
 
 def import_pyqt6():
     """
     Import PyQt6
 
     ImportErrors raised within this function are non-recoverable
     """
 
     from PyQt6 import QtCore, QtSvg, QtWidgets, QtGui
 
     # Alias PyQt-specific functions for PySide compatibility.
     QtCore.Signal = QtCore.pyqtSignal
     QtCore.Slot = QtCore.pyqtSlot
 
     # Join QtGui and QtWidgets for Qt4 compatibility.
     QtGuiCompat = types.ModuleType("QtGuiCompat")
     QtGuiCompat.__dict__.update(QtGui.__dict__)
     QtGuiCompat.__dict__.update(QtWidgets.__dict__)
 
     api = QT_API_PYQT6
     return QtCore, QtGuiCompat, QtSvg, api
 
 
 def import_pyside():
     """
     Import PySide
 
     ImportErrors raised within this function are non-recoverable
     """
     from PySide import QtGui, QtCore, QtSvg
     return QtCore, QtGui, QtSvg, QT_API_PYSIDE
 
 def import_pyside2():
     """
     Import PySide2
 
     ImportErrors raised within this function are non-recoverable
     """
     from PySide2 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
 
     # Join QtGui and QtWidgets for Qt4 compatibility.
     QtGuiCompat = types.ModuleType('QtGuiCompat')
     QtGuiCompat.__dict__.update(QtGui.__dict__)
     QtGuiCompat.__dict__.update(QtWidgets.__dict__)
     QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
 
     return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE2
 
 
 def import_pyside6():
     """
     Import PySide6
 
     ImportErrors raised within this function are non-recoverable
     """
 
     def get_attrs(module):
         return {
             name: getattr(module, name)
             for name in dir(module)
             if not name.startswith("_")
         }
 
     from PySide6 import QtGui, QtCore, QtSvg, QtWidgets, QtPrintSupport
 
     # Join QtGui and QtWidgets for Qt4 compatibility.
     QtGuiCompat = types.ModuleType("QtGuiCompat")
     QtGuiCompat.__dict__.update(QtGui.__dict__)
     if QtCore.__version_info__ < (6, 7):
         QtGuiCompat.__dict__.update(QtWidgets.__dict__)
         QtGuiCompat.__dict__.update(QtPrintSupport.__dict__)
     else:
         QtGuiCompat.__dict__.update(get_attrs(QtWidgets))
         QtGuiCompat.__dict__.update(get_attrs(QtPrintSupport))
 
     return QtCore, QtGuiCompat, QtSvg, QT_API_PYSIDE6
 
 
 def load_qt(api_options):
     """
     Attempt to import Qt, given a preference list
     of permissible bindings
 
     It is safe to call this function multiple times.
 
     Parameters
     ----------
     api_options : List of strings
         The order of APIs to try. Valid items are 'pyside', 'pyside2',
         'pyqt', 'pyqt5', 'pyqtv1' and 'pyqtdefault'
 
     Returns
     -------
     A tuple of QtCore, QtGui, QtSvg, QT_API
     The first three are the Qt modules. The last is the
     string indicating which module was loaded.
 
     Raises
     ------
     ImportError, if it isn't possible to import any requested
     bindings (either because they aren't installed, or because
     an incompatible library has already been installed)
     """
     loaders = {
         # Qt6
         QT_API_PYQT6: import_pyqt6,
         QT_API_PYSIDE6: import_pyside6,
         # Qt5
         QT_API_PYQT5: import_pyqt5,
         QT_API_PYSIDE2: import_pyside2,
         # Qt4
         QT_API_PYSIDE: import_pyside,
         QT_API_PYQT: import_pyqt4,
         QT_API_PYQTv1: partial(import_pyqt4, version=1),
         # default
         QT_API_PYQT_DEFAULT: import_pyqt6,
     }
 
     for api in api_options:
 
         if api not in loaders:
             raise RuntimeError(
                 "Invalid Qt API %r, valid values are: %s" %
                 (api, ", ".join(["%r" % k for k in loaders.keys()])))
 
         if not can_import(api):
             continue
 
         #cannot safely recover from an ImportError during this
         result = loaders[api]()
         api = result[-1]  # changed if api = QT_API_PYQT_DEFAULT
         commit_api(api)
         return result
     else:
         # Clear the environment variable since it doesn't work.
         if "QT_API" in os.environ:
             del os.environ["QT_API"]
 
         raise ImportError(
             """
     Could not load requested Qt binding. Please ensure that
     PyQt4 >= 4.7, PyQt5, PyQt6, PySide >= 1.0.3, PySide2, or
     PySide6 is available, and only one is imported per session.
 
     Currently-imported Qt library:                              %r
     PyQt5 available (requires QtCore, QtGui, QtSvg, QtWidgets): %s
     PyQt6 available (requires QtCore, QtGui, QtSvg, QtWidgets): %s
     PySide2 installed:                                          %s
     PySide6 installed:                                          %s
     Tried to load:                                              %r
     """
             % (
                 loaded_api(),
                 has_binding(QT_API_PYQT5),
                 has_binding(QT_API_PYQT6),
                 has_binding(QT_API_PYSIDE2),
                 has_binding(QT_API_PYSIDE6),
                 api_options,
             )
         )
 
 
 def enum_factory(QT_API, QtCore):
     """Construct an enum helper to account for PyQt5 <-> PyQt6 changes."""
 
     @lru_cache(None)
     def _enum(name):
         # foo.bar.Enum.Entry (PyQt6) <=> foo.bar.Entry (non-PyQt6).
         return operator.attrgetter(
             name if QT_API == QT_API_PYQT6 else name.rpartition(".")[0]
         )(sys.modules[QtCore.__package__])
 
     return _enum

IPython/lib/clipboard.py

@@ -1,101 +1,102
 """ Utilities for accessing the platform's clipboard.
 """
+
 import os
 import subprocess
 
 from IPython.core.error import TryNext
 import IPython.utils.py3compat as py3compat
 
 
 class ClipboardEmpty(ValueError):
     pass
 
 
 def win32_clipboard_get():
     """ Get the current clipboard's text on Windows.
 
     Requires Mark Hammond's pywin32 extensions.
     """
     try:
         import win32clipboard
     except ImportError as e:
         raise TryNext("Getting text from the clipboard requires the pywin32 "
                       "extensions: http://sourceforge.net/projects/pywin32/") from e
     win32clipboard.OpenClipboard()
     try:
         text = win32clipboard.GetClipboardData(win32clipboard.CF_UNICODETEXT)
     except (TypeError, win32clipboard.error):
         try:
             text = win32clipboard.GetClipboardData(win32clipboard.CF_TEXT)
             text = py3compat.cast_unicode(text, py3compat.DEFAULT_ENCODING)
         except (TypeError, win32clipboard.error) as e:
             raise ClipboardEmpty from e
     finally:
         win32clipboard.CloseClipboard()
     return text
 
 
 def osx_clipboard_get() -> str:
     """ Get the clipboard's text on OS X.
     """
     p = subprocess.Popen(['pbpaste', '-Prefer', 'ascii'],
         stdout=subprocess.PIPE)
     bytes_, stderr = p.communicate()
     # Text comes in with old Mac \r line endings. Change them to \n.
     bytes_ = bytes_.replace(b'\r', b'\n')
     text = py3compat.decode(bytes_)
     return text
 
 
 def tkinter_clipboard_get():
     """ Get the clipboard's text using Tkinter.
 
     This is the default on systems that are not Windows or OS X. It may
     interfere with other UI toolkits and should be replaced with an
     implementation that uses that toolkit.
     """
     try:
         from tkinter import Tk, TclError 
     except ImportError as e:
         raise TryNext("Getting text from the clipboard on this platform requires tkinter.") from e
         
     root = Tk()
     root.withdraw()
     try:
         text = root.clipboard_get()
     except TclError as e:
         raise ClipboardEmpty from e
     finally:
         root.destroy()
     text = py3compat.cast_unicode(text, py3compat.DEFAULT_ENCODING)
     return text
 
 
 def wayland_clipboard_get():
     """Get the clipboard's text under Wayland using wl-paste command.
 
     This requires Wayland and wl-clipboard installed and running.
     """
     if os.environ.get("XDG_SESSION_TYPE") != "wayland":
         raise TryNext("wayland is not detected")
 
     try:
         with subprocess.Popen(["wl-paste"], stdout=subprocess.PIPE) as p:
             raw, err = p.communicate()
             if p.wait():
                 raise TryNext(err)
     except FileNotFoundError as e:
         raise TryNext(
             "Getting text from the clipboard under Wayland requires the wl-clipboard "
             "extension: https://github.com/bugaevc/wl-clipboard"
         ) from e
 
     if not raw:
         raise ClipboardEmpty
 
     try:
         text = py3compat.decode(raw)
     except UnicodeDecodeError as e:
         raise ClipboardEmpty from e
 
     return text

IPython/sphinxext/ipython_directive.py

@@ -1,1277 +1,1278
 # -*- coding: utf-8 -*-
 """
 Sphinx directive to support embedded IPython code.
 
 IPython provides an extension for `Sphinx <http://www.sphinx-doc.org/>`_ to
 highlight and run code.
 
 This directive allows pasting of entire interactive IPython sessions, prompts
 and all, and their code will actually get re-executed at doc build time, with
 all prompts renumbered sequentially. It also allows you to input code as a pure
 python input by giving the argument python to the directive. The output looks
 like an interactive ipython section.
 
 Here is an example of how the IPython directive can
 **run** python code, at build time.
 
 .. ipython::
 
    In [1]: 1+1
 
    In [1]: import datetime
       ...: datetime.date.fromisoformat('2022-02-22')
 
 It supports IPython construct that plain
 Python does not understand (like magics):
 
 .. ipython::
 
    In [0]: import time
 
    In [0]: %pdoc time.sleep
 
 This will also support top-level async when using IPython 7.0+
 
 .. ipython::
 
    In [2]: import asyncio
       ...: print('before')
       ...: await asyncio.sleep(1)
       ...: print('after')
 
 
 The namespace will persist across multiple code chucks, Let's define a variable:
 
 .. ipython::
 
    In [0]: who = "World"
 
 And now say hello:
 
 .. ipython::
 
    In [0]: print('Hello,', who)
 
 If the current section raises an exception, you can add the ``:okexcept:`` flag
 to the current block, otherwise the build will fail.
 
 .. ipython::
    :okexcept:
 
    In [1]: 1/0
 
 IPython Sphinx directive module
 ===============================
 
 To enable this directive, simply list it in your Sphinx ``conf.py`` file
 (making sure the directory where you placed it is visible to sphinx, as is
 needed for all Sphinx directives). For example, to enable syntax highlighting
 and the IPython directive::
 
     extensions = ['IPython.sphinxext.ipython_console_highlighting',
                   'IPython.sphinxext.ipython_directive']
 
 The IPython directive outputs code-blocks with the language 'ipython'. So
 if you do not have the syntax highlighting extension enabled as well, then
 all rendered code-blocks will be uncolored. By default this directive assumes
 that your prompts are unchanged IPython ones, but this can be customized.
 The configurable options that can be placed in conf.py are:
 
 ipython_savefig_dir:
     The directory in which to save the figures. This is relative to the
     Sphinx source directory. The default is `html_static_path`.
 ipython_rgxin:
     The compiled regular expression to denote the start of IPython input
     lines. The default is ``re.compile('In \\[(\\d+)\\]:\\s?(.*)\\s*')``. You
     shouldn't need to change this.
 ipython_warning_is_error: [default to True]
     Fail the build if something unexpected happen, for example if a block raise
     an exception but does not have the `:okexcept:` flag. The exact behavior of
     what is considered strict, may change between the sphinx directive version.
 ipython_rgxout:
     The compiled regular expression to denote the start of IPython output
     lines. The default is ``re.compile('Out\\[(\\d+)\\]:\\s?(.*)\\s*')``. You
     shouldn't need to change this.
 ipython_promptin:
     The string to represent the IPython input prompt in the generated ReST.
     The default is ``'In [%d]:'``. This expects that the line numbers are used
     in the prompt.
 ipython_promptout:
     The string to represent the IPython prompt in the generated ReST. The
     default is ``'Out [%d]:'``. This expects that the line numbers are used
     in the prompt.
 ipython_mplbackend:
     The string which specifies if the embedded Sphinx shell should import
     Matplotlib and set the backend. The value specifies a backend that is
     passed to `matplotlib.use()` before any lines in `ipython_execlines` are
     executed. If not specified in conf.py, then the default value of 'agg' is
     used. To use the IPython directive without matplotlib as a dependency, set
     the value to `None`. It may end up that matplotlib is still imported
     if the user specifies so in `ipython_execlines` or makes use of the
     @savefig pseudo decorator.
 ipython_execlines:
     A list of strings to be exec'd in the embedded Sphinx shell. Typical
     usage is to make certain packages always available. Set this to an empty
     list if you wish to have no imports always available. If specified in
     ``conf.py`` as `None`, then it has the effect of making no imports available.
     If omitted from conf.py altogether, then the default value of
     ['import numpy as np', 'import matplotlib.pyplot as plt'] is used.
 ipython_holdcount
     When the @suppress pseudo-decorator is used, the execution count can be
     incremented or not. The default behavior is to hold the execution count,
     corresponding to a value of `True`. Set this to `False` to increment
     the execution count after each suppressed command.
 
 As an example, to use the IPython directive when `matplotlib` is not available,
 one sets the backend to `None`::
 
     ipython_mplbackend = None
 
 An example usage of the directive is:
 
 .. code-block:: rst
 
     .. ipython::
 
         In [1]: x = 1
 
         In [2]: y = x**2
 
         In [3]: print(y)
 
 See http://matplotlib.org/sampledoc/ipython_directive.html for additional
 documentation.
 
 Pseudo-Decorators
 =================
 
 Note: Only one decorator is supported per input. If more than one decorator
 is specified, then only the last one is used.
 
 In addition to the Pseudo-Decorators/options described at the above link,
 several enhancements have been made. The directive will emit a message to the
 console at build-time if code-execution resulted in an exception or warning.
 You can suppress these on a per-block basis by specifying the :okexcept:
 or :okwarning: options:
 
 .. code-block:: rst
 
     .. ipython::
         :okexcept:
         :okwarning:
 
         In [1]: 1/0
         In [2]: # raise warning.
 
 To Do
 =====
 
 - Turn the ad-hoc test() function into a real test suite.
 - Break up ipython-specific functionality from matplotlib stuff into better
   separated code.
 
 """
 
 # Authors
 # =======
 #
 # - John D Hunter: original author.
 # - Fernando Perez: refactoring, documentation, cleanups, port to 0.11.
 # - VáclavŠmilauer <eudoxos-AT-arcig.cz>: Prompt generalizations.
 # - Skipper Seabold, refactoring, cleanups, pure python addition
 
 #-----------------------------------------------------------------------------
 # Imports
 #-----------------------------------------------------------------------------
 
 # Stdlib
 import atexit
 import errno
 import os
 import pathlib
 import re
 import sys
 import tempfile
 import ast
 import warnings
 import shutil
 from io import StringIO
 from typing import Any, Dict, Set
 
 # Third-party
 from docutils.parsers.rst import directives
 from docutils.parsers.rst import Directive
 from sphinx.util import logging
 
 # Our own
 from traitlets.config import Config
 from IPython import InteractiveShell
 from IPython.core.profiledir import ProfileDir
 
 use_matplotlib = False
 try:
     import matplotlib
     use_matplotlib = True
 except Exception:
     pass
 
 #-----------------------------------------------------------------------------
 # Globals
 #-----------------------------------------------------------------------------
 # for tokenizing blocks
 COMMENT, INPUT, OUTPUT =  range(3)
 
 PSEUDO_DECORATORS = ["suppress", "verbatim", "savefig", "doctest"]
 
 #-----------------------------------------------------------------------------
 # Functions and class declarations
 #-----------------------------------------------------------------------------
 
 def block_parser(part, rgxin, rgxout, fmtin, fmtout):
     """
     part is a string of ipython text, comprised of at most one
     input, one output, comments, and blank lines.  The block parser
     parses the text into a list of::
 
       blocks = [ (TOKEN0, data0), (TOKEN1, data1), ...]
 
     where TOKEN is one of [COMMENT | INPUT | OUTPUT ] and
     data is, depending on the type of token::
 
       COMMENT : the comment string
 
       INPUT: the (DECORATOR, INPUT_LINE, REST) where
          DECORATOR: the input decorator (or None)
          INPUT_LINE: the input as string (possibly multi-line)
          REST : any stdout generated by the input line (not OUTPUT)
 
       OUTPUT: the output string, possibly multi-line
 
     """
     block = []
     lines = part.split('\n')
     N = len(lines)
     i = 0
     decorator = None
     while 1:
 
         if i==N:
             # nothing left to parse -- the last line
             break
 
         line = lines[i]
         i += 1
         line_stripped = line.strip()
         if line_stripped.startswith('#'):
             block.append((COMMENT, line))
             continue
 
         if any(
             line_stripped.startswith("@" + pseudo_decorator)
             for pseudo_decorator in PSEUDO_DECORATORS
         ):
             if decorator:
                 raise RuntimeError(
                     "Applying multiple pseudo-decorators on one line is not supported"
                 )
             else:
                 decorator = line_stripped
                 continue
 
         # does this look like an input line?
         matchin = rgxin.match(line)
         if matchin:
             lineno, inputline = int(matchin.group(1)), matchin.group(2)
 
             # the ....: continuation string
             continuation = '   %s:'%''.join(['.']*(len(str(lineno))+2))
             Nc = len(continuation)
             # input lines can continue on for more than one line, if
             # we have a '\' line continuation char or a function call
             # echo line 'print'.  The input line can only be
             # terminated by the end of the block or an output line, so
             # we parse out the rest of the input line if it is
             # multiline as well as any echo text
 
             rest = []
             while i<N:
 
                 # look ahead; if the next line is blank, or a comment, or
                 # an output line, we're done
 
                 nextline = lines[i]
                 matchout = rgxout.match(nextline)
                 # print("nextline=%s, continuation=%s, starts=%s"%(nextline, continuation, nextline.startswith(continuation)))
                 if matchout or nextline.startswith('#'):
                     break
                 elif nextline.startswith(continuation):
                     # The default ipython_rgx* treat the space following the colon as optional.
                     # However, If the space is there we must consume it or code
                     # employing the cython_magic extension will fail to execute.
                     #
                     # This works with the default ipython_rgx* patterns,
                     # If you modify them, YMMV.
                     nextline = nextline[Nc:]
                     if nextline and nextline[0] == ' ':
                         nextline = nextline[1:]
 
                     inputline += '\n' +  nextline
                 else:
                     rest.append(nextline)
                 i+= 1
 
             block.append((INPUT, (decorator, inputline, '\n'.join(rest))))
             continue
 
         # if it looks like an output line grab all the text to the end
         # of the block
         matchout = rgxout.match(line)
         if matchout:
             lineno, output = int(matchout.group(1)), matchout.group(2)
             if i<N-1:
                 output = '\n'.join([output] + lines[i:])
 
             block.append((OUTPUT, output))
             break
 
     return block
 
 
 class EmbeddedSphinxShell(object):
     """An embedded IPython instance to run inside Sphinx"""
 
     def __init__(self, exec_lines=None):
 
         self.cout = StringIO()
 
         if exec_lines is None:
             exec_lines = []
 
         # Create config object for IPython
         config = Config()
         config.HistoryManager.hist_file = ':memory:'
         config.InteractiveShell.autocall = False
         config.InteractiveShell.autoindent = False
         config.InteractiveShell.colors = 'NoColor'
 
         # create a profile so instance history isn't saved
         tmp_profile_dir = tempfile.mkdtemp(prefix='profile_')
         profname = 'auto_profile_sphinx_build'
         pdir = os.path.join(tmp_profile_dir,profname)
         profile = ProfileDir.create_profile_dir(pdir)
 
         # Create and initialize global ipython, but don't start its mainloop.
         # This will persist across different EmbeddedSphinxShell instances.
         IP = InteractiveShell.instance(config=config, profile_dir=profile)
         atexit.register(self.cleanup)
 
         # Store a few parts of IPython we'll need.
         self.IP = IP
         self.user_ns = self.IP.user_ns
         self.user_global_ns = self.IP.user_global_ns
 
         self.input = ''
         self.output = ''
         self.tmp_profile_dir = tmp_profile_dir
 
         self.is_verbatim = False
         self.is_doctest = False
         self.is_suppress = False
 
         # Optionally, provide more detailed information to shell.
         # this is assigned by the SetUp method of IPythonDirective
         # to point at itself.
         #
         # So, you can access handy things at self.directive.state
         self.directive = None
 
         # on the first call to the savefig decorator, we'll import
         # pyplot as plt so we can make a call to the plt.gcf().savefig
         self._pyplot_imported = False
 
         # Prepopulate the namespace.
         for line in exec_lines:
             self.process_input_line(line, store_history=False)
 
     def cleanup(self):
         shutil.rmtree(self.tmp_profile_dir, ignore_errors=True)
 
     def clear_cout(self):
         self.cout.seek(0)
         self.cout.truncate(0)
 
     def process_input_line(self, line, store_history):
         return self.process_input_lines([line], store_history=store_history)
 
     def process_input_lines(self, lines, store_history=True):
         """process the input, capturing stdout"""
         stdout = sys.stdout
         source_raw = '\n'.join(lines)
         try:
             sys.stdout = self.cout
             self.IP.run_cell(source_raw, store_history=store_history)
         finally:
             sys.stdout = stdout
 
     def process_image(self, decorator):
         """
         # build out an image directive like
         # .. image:: somefile.png
         #    :width 4in
         #
         # from an input like
         # savefig somefile.png width=4in
         """
         savefig_dir = self.savefig_dir
         source_dir = self.source_dir
         saveargs = decorator.split(' ')
         filename = saveargs[1]
         # insert relative path to image file in source
         # as absolute path for Sphinx
         # sphinx expects a posix path, even on Windows
         path = pathlib.Path(savefig_dir, filename)
         outfile = '/' + path.relative_to(source_dir).as_posix()
 
         imagerows = ['.. image:: %s' % outfile]
 
         for kwarg in saveargs[2:]:
             arg, val = kwarg.split('=')
             arg = arg.strip()
             val = val.strip()
             imagerows.append('   :%s: %s'%(arg, val))
 
         image_file = os.path.basename(outfile) # only return file name
         image_directive = '\n'.join(imagerows)
         return image_file, image_directive
 
     # Callbacks for each type of token
     def process_input(self, data, input_prompt, lineno):
         """
         Process data block for INPUT token.
 
         """
         decorator, input, rest = data
         image_file = None
         image_directive = None
 
         is_verbatim = decorator=='@verbatim' or self.is_verbatim
         is_doctest = (decorator is not None and \
                      decorator.startswith('@doctest')) or self.is_doctest
         is_suppress = decorator=='@suppress' or self.is_suppress
         is_okexcept = decorator=='@okexcept' or self.is_okexcept
         is_okwarning = decorator=='@okwarning' or self.is_okwarning
         is_savefig = decorator is not None and \
                      decorator.startswith('@savefig')
 
         input_lines = input.split('\n')
         if len(input_lines) > 1:
             if input_lines[-1] != "":
                 input_lines.append('') # make sure there's a blank line
                                        # so splitter buffer gets reset
 
         continuation = '   %s:'%''.join(['.']*(len(str(lineno))+2))
 
         if is_savefig:
             image_file, image_directive = self.process_image(decorator)
 
         ret = []
         is_semicolon = False
 
         # Hold the execution count, if requested to do so.
         if is_suppress and self.hold_count:
             store_history = False
         else:
             store_history = True
 
         # Note: catch_warnings is not thread safe
         with warnings.catch_warnings(record=True) as ws:
             if input_lines[0].endswith(';'):
                 is_semicolon = True
             #for i, line in enumerate(input_lines):
 
             # process the first input line
             if is_verbatim:
                 self.process_input_lines([''])
                 self.IP.execution_count += 1 # increment it anyway
             else:
                 # only submit the line in non-verbatim mode
                 self.process_input_lines(input_lines, store_history=store_history)
 
         if not is_suppress:
             for i, line in enumerate(input_lines):
                 if i == 0:
                     formatted_line = '%s %s'%(input_prompt, line)
                 else:
                     formatted_line = '%s %s'%(continuation, line)
                 ret.append(formatted_line)
 
         if not is_suppress and len(rest.strip()) and is_verbatim:
             # The "rest" is the standard output of the input. This needs to be
             # added when in verbatim mode. If there is no "rest", then we don't
             # add it, as the new line will be added by the processed output.
             ret.append(rest)
 
         # Fetch the processed output. (This is not the submitted output.)
         self.cout.seek(0)
         processed_output = self.cout.read()
         if not is_suppress and not is_semicolon:
             #
             # In IPythonDirective.run, the elements of `ret` are eventually
             # combined such that '' entries correspond to newlines. So if
             # `processed_output` is equal to '', then the adding it to `ret`
             # ensures that there is a blank line between consecutive inputs
             # that have no outputs, as in:
             #
             #    In [1]: x = 4
             #
             #    In [2]: x = 5
             #
             # When there is processed output, it has a '\n' at the tail end. So
             # adding the output to `ret` will provide the necessary spacing
             # between consecutive input/output blocks, as in:
             #
             #   In [1]: x
             #   Out[1]: 5
             #
             #   In [2]: x
             #   Out[2]: 5
             #
             # When there is stdout from the input, it also has a '\n' at the
             # tail end, and so this ensures proper spacing as well. E.g.:
             #
             #   In [1]: print(x)
             #   5
             #
             #   In [2]: x = 5
             #
             # When in verbatim mode, `processed_output` is empty (because
             # nothing was passed to IP. Sometimes the submitted code block has
             # an Out[] portion and sometimes it does not. When it does not, we
             # need to ensure proper spacing, so we have to add '' to `ret`.
             # However, if there is an Out[] in the submitted code, then we do
             # not want to add a newline as `process_output` has stuff to add.
             # The difficulty is that `process_input` doesn't know if
             # `process_output` will be called---so it doesn't know if there is
             # Out[] in the code block. The requires that we include a hack in
             # `process_block`. See the comments there.
             #
             ret.append(processed_output)
         elif is_semicolon:
             # Make sure there is a newline after the semicolon.
             ret.append('')
 
         # context information
         filename = "Unknown"
         lineno = 0
         if self.directive.state:
             filename = self.directive.state.document.current_source
             lineno = self.directive.state.document.current_line
 
         # Use sphinx logger for warnings
         logger = logging.getLogger(__name__)
 
         # output any exceptions raised during execution to stdout
         # unless :okexcept: has been specified.
         if not is_okexcept and (
             ("Traceback" in processed_output) or ("SyntaxError" in processed_output)
         ):
             s = "\n>>>" + ("-" * 73) + "\n"
             s += "Exception in %s at block ending on line %s\n" % (filename, lineno)
             s += "Specify :okexcept: as an option in the ipython:: block to suppress this message\n"
             s += processed_output + "\n"
             s += "<<<" + ("-" * 73)
             logger.warning(s)
             if self.warning_is_error:
                 raise RuntimeError(
                     "Unexpected exception in `{}` line {}".format(filename, lineno)
                 )
 
         # output any warning raised during execution to stdout
         # unless :okwarning: has been specified.
         if not is_okwarning:
             for w in ws:
                 s = "\n>>>" + ("-" * 73) + "\n"
                 s += "Warning in %s at block ending on line %s\n" % (filename, lineno)
                 s += "Specify :okwarning: as an option in the ipython:: block to suppress this message\n"
                 s += ("-" * 76) + "\n"
                 s += warnings.formatwarning(
                     w.message, w.category, w.filename, w.lineno, w.line
                 )
                 s += "<<<" + ("-" * 73)
                 logger.warning(s)
                 if self.warning_is_error:
                     raise RuntimeError(
                         "Unexpected warning in `{}` line {}".format(filename, lineno)
                     )
 
         self.clear_cout()
         return (ret, input_lines, processed_output,
                 is_doctest, decorator, image_file, image_directive)
 
 
     def process_output(self, data, output_prompt, input_lines, output,
                        is_doctest, decorator, image_file):
         """
         Process data block for OUTPUT token.
 
         """
         # Recall: `data` is the submitted output, and `output` is the processed
         # output from `input_lines`.
 
         TAB = ' ' * 4
 
         if is_doctest and output is not None:
 
             found = output # This is the processed output
             found = found.strip()
             submitted = data.strip()
 
             if self.directive is None:
                 source = 'Unavailable'
                 content = 'Unavailable'
             else:
                 source = self.directive.state.document.current_source
                 content = self.directive.content
                 # Add tabs and join into a single string.
                 content = '\n'.join([TAB + line for line in content])
 
             # Make sure the output contains the output prompt.
             ind = found.find(output_prompt)
             if ind < 0:
                 e = ('output does not contain output prompt\n\n'
                      'Document source: {0}\n\n'
                      'Raw content: \n{1}\n\n'
                      'Input line(s):\n{TAB}{2}\n\n'
                      'Output line(s):\n{TAB}{3}\n\n')
                 e = e.format(source, content, '\n'.join(input_lines),
                              repr(found), TAB=TAB)
                 raise RuntimeError(e)
             found = found[len(output_prompt):].strip()
 
             # Handle the actual doctest comparison.
             if decorator.strip() == '@doctest':
                 # Standard doctest
                 if found != submitted:
                     e = ('doctest failure\n\n'
                          'Document source: {0}\n\n'
                          'Raw content: \n{1}\n\n'
                          'On input line(s):\n{TAB}{2}\n\n'
                          'we found output:\n{TAB}{3}\n\n'
                          'instead of the expected:\n{TAB}{4}\n\n')
                     e = e.format(source, content, '\n'.join(input_lines),
                                  repr(found), repr(submitted), TAB=TAB)
                     raise RuntimeError(e)
             else:
                 self.custom_doctest(decorator, input_lines, found, submitted)
 
         # When in verbatim mode, this holds additional submitted output
         # to be written in the final Sphinx output.
         # https://github.com/ipython/ipython/issues/5776
         out_data = []
 
         is_verbatim = decorator=='@verbatim' or self.is_verbatim
         if is_verbatim and data.strip():
             # Note that `ret` in `process_block` has '' as its last element if
             # the code block was in verbatim mode. So if there is no submitted
             # output, then we will have proper spacing only if we do not add
             # an additional '' to `out_data`. This is why we condition on
             # `and data.strip()`.
 
             # The submitted output has no output prompt. If we want the
             # prompt and the code to appear, we need to join them now
             # instead of adding them separately---as this would create an
             # undesired newline. How we do this ultimately depends on the
             # format of the output regex. I'll do what works for the default
             # prompt for now, and we might have to adjust if it doesn't work
             # in other cases. Finally, the submitted output does not have
             # a trailing newline, so we must add it manually.
             out_data.append("{0} {1}\n".format(output_prompt, data))
 
         return out_data
 
     def process_comment(self, data):
         """Process data fPblock for COMMENT token."""
         if not self.is_suppress:
             return [data]
 
     def save_image(self, image_file):
         """
         Saves the image file to disk.
         """
         self.ensure_pyplot()
         command = 'plt.gcf().savefig("%s")'%image_file
         # print('SAVEFIG', command)  # dbg
         self.process_input_line('bookmark ipy_thisdir', store_history=False)
         self.process_input_line('cd -b ipy_savedir', store_history=False)
         self.process_input_line(command, store_history=False)
         self.process_input_line('cd -b ipy_thisdir', store_history=False)
         self.process_input_line('bookmark -d ipy_thisdir', store_history=False)
         self.clear_cout()
 
     def process_block(self, block):
         """
         process block from the block_parser and return a list of processed lines
         """
         ret = []
         output = None
         input_lines = None
         lineno = self.IP.execution_count
 
         input_prompt = self.promptin % lineno
         output_prompt = self.promptout % lineno
         image_file = None
         image_directive = None
 
         found_input = False
         for token, data in block:
             if token == COMMENT:
                 out_data = self.process_comment(data)
             elif token == INPUT:
                 found_input = True
                 (out_data, input_lines, output, is_doctest,
                  decorator, image_file, image_directive) = \
                           self.process_input(data, input_prompt, lineno)
             elif token == OUTPUT:
                 if not found_input:
 
                     TAB = ' ' * 4
                     linenumber = 0
                     source = 'Unavailable'
                     content = 'Unavailable'
                     if self.directive:
                         linenumber = self.directive.state.document.current_line
                         source = self.directive.state.document.current_source
                         content = self.directive.content
                         # Add tabs and join into a single string.
                         content = '\n'.join([TAB + line for line in content])
 
                     e = ('\n\nInvalid block: Block contains an output prompt '
                          'without an input prompt.\n\n'
                          'Document source: {0}\n\n'
                          'Content begins at line {1}: \n\n{2}\n\n'
                          'Problematic block within content: \n\n{TAB}{3}\n\n')
                     e = e.format(source, linenumber, content, block, TAB=TAB)
 
                     # Write, rather than include in exception, since Sphinx
                     # will truncate tracebacks.
                     sys.stdout.write(e)
                     raise RuntimeError('An invalid block was detected.')
                 out_data = \
                     self.process_output(data, output_prompt, input_lines,
                                         output, is_doctest, decorator,
                                         image_file)
                 if out_data:
                     # Then there was user submitted output in verbatim mode.
                     # We need to remove the last element of `ret` that was
                     # added in `process_input`, as it is '' and would introduce
                     # an undesirable newline.
                     assert(ret[-1] == '')
                     del ret[-1]
 
             if out_data:
                 ret.extend(out_data)
 
         # save the image files
         if image_file is not None:
             self.save_image(image_file)
 
         return ret, image_directive
 
     def ensure_pyplot(self):
         """
         Ensures that pyplot has been imported into the embedded IPython shell.
 
         Also, makes sure to set the backend appropriately if not set already.
 
         """
         # We are here if the @figure pseudo decorator was used. Thus, it's
         # possible that we could be here even if python_mplbackend were set to
         # `None`. That's also strange and perhaps worthy of raising an
         # exception, but for now, we just set the backend to 'agg'.
 
         if not self._pyplot_imported:
             if 'matplotlib.backends' not in sys.modules:
                 # Then ipython_matplotlib was set to None but there was a
                 # call to the @figure decorator (and ipython_execlines did
                 # not set a backend).
                 #raise Exception("No backend was set, but @figure was used!")
                 import matplotlib
                 matplotlib.use('agg')
 
             # Always import pyplot into embedded shell.
             self.process_input_line('import matplotlib.pyplot as plt',
                                     store_history=False)
             self._pyplot_imported = True
 
     def process_pure_python(self, content):
         """
         content is a list of strings. it is unedited directive content
 
         This runs it line by line in the InteractiveShell, prepends
         prompts as needed capturing stderr and stdout, then returns
         the content as a list as if it were ipython code
         """
         output = []
         savefig = False # keep up with this to clear figure
         multiline = False # to handle line continuation
         multiline_start = None
         fmtin = self.promptin
 
         ct = 0
 
         for lineno, line in enumerate(content):
 
             line_stripped = line.strip()
             if not len(line):
                 output.append(line)
                 continue
 
             # handle pseudo-decorators, whilst ensuring real python decorators are treated as input
             if any(
                 line_stripped.startswith("@" + pseudo_decorator)
                 for pseudo_decorator in PSEUDO_DECORATORS
             ):
                 output.extend([line])
                 if 'savefig' in line:
                     savefig = True # and need to clear figure
                 continue
 
             # handle comments
             if line_stripped.startswith('#'):
                 output.extend([line])
                 continue
 
             # deal with lines checking for multiline
             continuation  = u'   %s:'% ''.join(['.']*(len(str(ct))+2))
             if not multiline:
                 modified = u"%s %s" % (fmtin % ct, line_stripped)
                 output.append(modified)
                 ct += 1
                 try:
                     ast.parse(line_stripped)
                     output.append(u'')
                 except Exception: # on a multiline
                     multiline = True
                     multiline_start = lineno
             else: # still on a multiline
                 modified = u'%s %s' % (continuation, line)
                 output.append(modified)
 
                 # if the next line is indented, it should be part of multiline
                 if len(content) > lineno + 1:
                     nextline = content[lineno + 1]
                     if len(nextline) - len(nextline.lstrip()) > 3:
                         continue
                 try:
                     mod = ast.parse(
                             '\n'.join(content[multiline_start:lineno+1]))
                     if isinstance(mod.body[0], ast.FunctionDef):
                         # check to see if we have the whole function
                         for element in mod.body[0].body:
                             if isinstance(element, ast.Return):
                                 multiline = False
                     else:
                         output.append(u'')
                         multiline = False
                 except Exception:
                     pass
 
             if savefig: # clear figure if plotted
                 self.ensure_pyplot()
                 self.process_input_line('plt.clf()', store_history=False)
                 self.clear_cout()
                 savefig = False
 
         return output
 
     def custom_doctest(self, decorator, input_lines, found, submitted):
         """
         Perform a specialized doctest.
 
         """
         from .custom_doctests import doctests
 
         args = decorator.split()
         doctest_type = args[1]
         if doctest_type in doctests:
             doctests[doctest_type](self, args, input_lines, found, submitted)
         else:
             e = "Invalid option to @doctest: {0}".format(doctest_type)
             raise Exception(e)
 
 
 class IPythonDirective(Directive):
 
     has_content: bool = True
     required_arguments: int = 0
     optional_arguments: int = 4 # python, suppress, verbatim, doctest
     final_argumuent_whitespace: bool = True
-    option_spec: Dict[str, Any] = { 'python': directives.unchanged,
-                    'suppress' : directives.flag,
-                    'verbatim' : directives.flag,
-                    'doctest' : directives.flag,
-                    'okexcept': directives.flag,
-                    'okwarning': directives.flag
+    option_spec: Dict[str, Any] = {
+        "python": directives.unchanged,
+        "suppress": directives.flag,
+        "verbatim": directives.flag,
+        "doctest": directives.flag,
+        "okexcept": directives.flag,
+        "okwarning": directives.flag,
                   }
 
     shell = None
 
     seen_docs: Set = set()
 
     def get_config_options(self):
         # contains sphinx configuration variables
         config = self.state.document.settings.env.config
 
         # get config variables to set figure output directory
         savefig_dir = config.ipython_savefig_dir
         source_dir = self.state.document.settings.env.srcdir
         savefig_dir = os.path.join(source_dir, savefig_dir)
 
         # get regex and prompt stuff
         rgxin      = config.ipython_rgxin
         rgxout     = config.ipython_rgxout
         warning_is_error= config.ipython_warning_is_error
         promptin   = config.ipython_promptin
         promptout  = config.ipython_promptout
         mplbackend = config.ipython_mplbackend
         exec_lines = config.ipython_execlines
         hold_count = config.ipython_holdcount
 
         return (savefig_dir, source_dir, rgxin, rgxout,
                 promptin, promptout, mplbackend, exec_lines, hold_count, warning_is_error)
 
     def setup(self):
         # Get configuration values.
         (savefig_dir, source_dir, rgxin, rgxout, promptin, promptout,
          mplbackend, exec_lines, hold_count, warning_is_error) = self.get_config_options()
 
         try:
             os.makedirs(savefig_dir)
         except OSError as e:
             if e.errno != errno.EEXIST:
                 raise
 
         if self.shell is None:
             # We will be here many times.  However, when the
             # EmbeddedSphinxShell is created, its interactive shell member
             # is the same for each instance.
 
             if mplbackend and 'matplotlib.backends' not in sys.modules and use_matplotlib:
                 import matplotlib
                 matplotlib.use(mplbackend)
 
             # Must be called after (potentially) importing matplotlib and
             # setting its backend since exec_lines might import pylab.
             self.shell = EmbeddedSphinxShell(exec_lines)
 
             # Store IPython directive to enable better error messages
             self.shell.directive = self
 
         # reset the execution count if we haven't processed this doc
         #NOTE: this may be borked if there are multiple seen_doc tmp files
         #check time stamp?
         if not self.state.document.current_source in self.seen_docs:
             self.shell.IP.history_manager.reset()
             self.shell.IP.execution_count = 1
             self.seen_docs.add(self.state.document.current_source)
 
         # and attach to shell so we don't have to pass them around
         self.shell.rgxin = rgxin
         self.shell.rgxout = rgxout
         self.shell.promptin = promptin
         self.shell.promptout = promptout
         self.shell.savefig_dir = savefig_dir
         self.shell.source_dir = source_dir
         self.shell.hold_count = hold_count
         self.shell.warning_is_error = warning_is_error
 
         # setup bookmark for saving figures directory
         self.shell.process_input_line(
             'bookmark ipy_savedir "%s"' % savefig_dir, store_history=False
         )
         self.shell.clear_cout()
 
         return rgxin, rgxout, promptin, promptout
 
     def teardown(self):
         # delete last bookmark
         self.shell.process_input_line('bookmark -d ipy_savedir',
                                       store_history=False)
         self.shell.clear_cout()
 
     def run(self):
         debug = False
 
         #TODO, any reason block_parser can't be a method of embeddable shell
         # then we wouldn't have to carry these around
         rgxin, rgxout, promptin, promptout = self.setup()
 
         options = self.options
         self.shell.is_suppress = 'suppress' in options
         self.shell.is_doctest = 'doctest' in options
         self.shell.is_verbatim = 'verbatim' in options
         self.shell.is_okexcept = 'okexcept' in options
         self.shell.is_okwarning = 'okwarning' in options
 
         # handle pure python code
         if 'python' in self.arguments:
             content = self.content
             self.content = self.shell.process_pure_python(content)
 
         # parts consists of all text within the ipython-block.
         # Each part is an input/output block.
         parts = '\n'.join(self.content).split('\n\n')
 
         lines = ['.. code-block:: ipython', '']
         figures = []
 
         # Use sphinx logger for warnings
         logger = logging.getLogger(__name__)
 
         for part in parts:
             block = block_parser(part, rgxin, rgxout, promptin, promptout)
             if len(block):
                 rows, figure = self.shell.process_block(block)
                 for row in rows:
                     lines.extend(['   {0}'.format(line)
                                   for line in row.split('\n')])
 
                 if figure is not None:
                     figures.append(figure)
             else:
                 message = 'Code input with no code at {}, line {}'\
                             .format(
                                 self.state.document.current_source,
                                 self.state.document.current_line)
                 if self.shell.warning_is_error:
                     raise RuntimeError(message)
                 else:
                     logger.warning(message)
 
         for figure in figures:
             lines.append('')
             lines.extend(figure.split('\n'))
             lines.append('')
 
         if len(lines) > 2:
             if debug:
                 print('\n'.join(lines))
             else:
                 # This has to do with input, not output. But if we comment
                 # these lines out, then no IPython code will appear in the
                 # final output.
                 self.state_machine.insert_input(
                     lines, self.state_machine.input_lines.source(0))
 
         # cleanup
         self.teardown()
 
         return []
 
 # Enable as a proper Sphinx directive
 def setup(app):
     setup.app = app
 
     app.add_directive('ipython', IPythonDirective)
     app.add_config_value('ipython_savefig_dir', 'savefig', 'env')
     app.add_config_value('ipython_warning_is_error', True, 'env')
     app.add_config_value('ipython_rgxin',
                          re.compile(r'In \[(\d+)\]:\s?(.*)\s*'), 'env')
     app.add_config_value('ipython_rgxout',
                          re.compile(r'Out\[(\d+)\]:\s?(.*)\s*'), 'env')
     app.add_config_value('ipython_promptin', 'In [%d]:', 'env')
     app.add_config_value('ipython_promptout', 'Out[%d]:', 'env')
 
     # We could just let matplotlib pick whatever is specified as the default
     # backend in the matplotlibrc file, but this would cause issues if the
     # backend didn't work in headless environments. For this reason, 'agg'
     # is a good default backend choice.
     app.add_config_value('ipython_mplbackend', 'agg', 'env')
 
     # If the user sets this config value to `None`, then EmbeddedSphinxShell's
     # __init__ method will treat it as [].
     execlines = ['import numpy as np']
     if use_matplotlib:
         execlines.append('import matplotlib.pyplot as plt')
     app.add_config_value('ipython_execlines', execlines, 'env')
 
     app.add_config_value('ipython_holdcount', True, 'env')
 
     metadata = {'parallel_read_safe': True, 'parallel_write_safe': True}
     return metadata
 
 # Simple smoke test, needs to be converted to a proper automatic test.
 def test():
 
     examples = [
         r"""
 In [9]: pwd
 Out[9]: '/home/jdhunter/py4science/book'
 
 In [10]: cd bookdata/
 /home/jdhunter/py4science/book/bookdata
 
 In [2]: from pylab import *
 
 In [2]: ion()
 
 In [3]: im = imread('stinkbug.png')
 
 @savefig mystinkbug.png width=4in
 In [4]: imshow(im)
 Out[4]: <matplotlib.image.AxesImage object at 0x39ea850>
 
 """,
         r"""
 
 In [1]: x = 'hello world'
 
 # string methods can be
 # used to alter the string
 @doctest
 In [2]: x.upper()
 Out[2]: 'HELLO WORLD'
 
 @verbatim
 In [3]: x.st<TAB>
 x.startswith  x.strip
 """,
     r"""
 
 In [130]: url = 'http://ichart.finance.yahoo.com/table.csv?s=CROX\
    .....: &d=9&e=22&f=2009&g=d&a=1&br=8&c=2006&ignore=.csv'
 
 In [131]: print url.split('&')
 ['http://ichart.finance.yahoo.com/table.csv?s=CROX', 'd=9', 'e=22', 'f=2009', 'g=d', 'a=1', 'b=8', 'c=2006', 'ignore=.csv']
 
 In [60]: import urllib
 
 """,
     r"""\
 
 In [133]: import numpy.random
 
 @suppress
 In [134]: numpy.random.seed(2358)
 
 @doctest
 In [135]: numpy.random.rand(10,2)
 Out[135]:
 array([[ 0.64524308,  0.59943846],
        [ 0.47102322,  0.8715456 ],
        [ 0.29370834,  0.74776844],
        [ 0.99539577,  0.1313423 ],
        [ 0.16250302,  0.21103583],
        [ 0.81626524,  0.1312433 ],
        [ 0.67338089,  0.72302393],
        [ 0.7566368 ,  0.07033696],
        [ 0.22591016,  0.77731835],
        [ 0.0072729 ,  0.34273127]])
 
 """,
 
     r"""
 In [106]: print x
 jdh
 
 In [109]: for i in range(10):
    .....:     print i
    .....:
    .....:
 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
 """,
 
         r"""
 
 In [144]: from pylab import *
 
 In [145]: ion()
 
 # use a semicolon to suppress the output
 @savefig test_hist.png width=4in
 In [151]: hist(np.random.randn(10000), 100);
 
 
 @savefig test_plot.png width=4in
 In [151]: plot(np.random.randn(10000), 'o');
    """,
 
         r"""
 # use a semicolon to suppress the output
 In [151]: plt.clf()
 
 @savefig plot_simple.png width=4in
 In [151]: plot([1,2,3])
 
 @savefig hist_simple.png width=4in
 In [151]: hist(np.random.randn(10000), 100);
 
 """,
      r"""
 # update the current fig
 In [151]: ylabel('number')
 
 In [152]: title('normal distribution')
 
 
 @savefig hist_with_text.png
 In [153]: grid(True)
 
 @doctest float
 In [154]: 0.1 + 0.2
 Out[154]: 0.3
 
 @doctest float
 In [155]: np.arange(16).reshape(4,4)
 Out[155]:
 array([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11],
        [12, 13, 14, 15]])
 
 In [1]: x = np.arange(16, dtype=float).reshape(4,4)
 
 In [2]: x[0,0] = np.inf
 
 In [3]: x[0,1] = np.nan
 
 @doctest float
 In [4]: x
 Out[4]:
 array([[ inf,  nan,   2.,   3.],
        [  4.,   5.,   6.,   7.],
        [  8.,   9.,  10.,  11.],
        [ 12.,  13.,  14.,  15.]])
 
 
         """,
         ]
     # skip local-file depending first example:
     examples = examples[1:]
 
     #ipython_directive.DEBUG = True  # dbg
     #options = dict(suppress=True)  # dbg
     options = {}
     for example in examples:
         content = example.split('\n')
         IPythonDirective('debug', arguments=None, options=options,
                           content=content, lineno=0,
                           content_offset=None, block_text=None,
                           state=None, state_machine=None,
                           )
 
 # Run test suite as a script
 if __name__=='__main__':
     if not os.path.isdir('_static'):
         os.mkdir('_static')
     test()
     print('All OK? Check figures in _static/')

IPython/terminal/interactiveshell.py

@@ -1,1021 +1,1023
 """IPython terminal interface using prompt_toolkit"""
 
 import os
 import sys
 import inspect
 from warnings import warn
 from typing import Union as UnionType, Optional
 
 from IPython.core.async_helpers import get_asyncio_loop
 from IPython.core.interactiveshell import InteractiveShell, InteractiveShellABC
 from IPython.utils.py3compat import input
 from IPython.utils.terminal import toggle_set_term_title, set_term_title, restore_term_title
 from IPython.utils.process import abbrev_cwd
 from traitlets import (
     Bool,
     Unicode,
     Dict,
     Integer,
     List,
     observe,
     Instance,
     Type,
     default,
     Enum,
     Union,
     Any,
     validate,
     Float,
 )
 
 from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
 from prompt_toolkit.enums import DEFAULT_BUFFER, EditingMode
 from prompt_toolkit.filters import HasFocus, Condition, IsDone
 from prompt_toolkit.formatted_text import PygmentsTokens
 from prompt_toolkit.history import History
 from prompt_toolkit.layout.processors import ConditionalProcessor, HighlightMatchingBracketProcessor
 from prompt_toolkit.output import ColorDepth
 from prompt_toolkit.patch_stdout import patch_stdout
 from prompt_toolkit.shortcuts import PromptSession, CompleteStyle, print_formatted_text
 from prompt_toolkit.styles import DynamicStyle, merge_styles
 from prompt_toolkit.styles.pygments import style_from_pygments_cls, style_from_pygments_dict
 from prompt_toolkit import __version__ as ptk_version
 
 from pygments.styles import get_style_by_name
 from pygments.style import Style
 from pygments.token import Token
 
 from .debugger import TerminalPdb, Pdb
 from .magics import TerminalMagics
 from .pt_inputhooks import get_inputhook_name_and_func
 from .prompts import Prompts, ClassicPrompts, RichPromptDisplayHook
 from .ptutils import IPythonPTCompleter, IPythonPTLexer
 from .shortcuts import (
     KEY_BINDINGS,
     create_ipython_shortcuts,
     create_identifier,
     RuntimeBinding,
     add_binding,
 )
 from .shortcuts.filters import KEYBINDING_FILTERS, filter_from_string
 from .shortcuts.auto_suggest import (
     NavigableAutoSuggestFromHistory,
     AppendAutoSuggestionInAnyLine,
 )
 
 PTK3 = ptk_version.startswith('3.')
 
 
 class _NoStyle(Style):
     pass
 
 
 _style_overrides_light_bg = {
             Token.Prompt: '#ansibrightblue',
             Token.PromptNum: '#ansiblue bold',
             Token.OutPrompt: '#ansibrightred',
             Token.OutPromptNum: '#ansired bold',
 }
 
 _style_overrides_linux = {
             Token.Prompt: '#ansibrightgreen',
             Token.PromptNum: '#ansigreen bold',
             Token.OutPrompt: '#ansibrightred',
             Token.OutPromptNum: '#ansired bold',
 }
 
 
 def _backward_compat_continuation_prompt_tokens(method, width: int, *, lineno: int):
     """
     Sagemath use custom prompt and we broke them in 8.19.
     """
     sig = inspect.signature(method)
     if "lineno" in inspect.signature(method).parameters or any(
         [p.kind == p.VAR_KEYWORD for p in sig.parameters.values()]
     ):
         return method(width, lineno=lineno)
     else:
         return method(width)
 
 
 def get_default_editor():
     try:
         return os.environ['EDITOR']
     except KeyError:
         pass
     except UnicodeError:
         warn("$EDITOR environment variable is not pure ASCII. Using platform "
              "default editor.")
 
     if os.name == 'posix':
         return 'vi'  # the only one guaranteed to be there!
     else:
         return "notepad"  # same in Windows!
 
 
 # conservatively check for tty
 # overridden streams can result in things like:
 # - sys.stdin = None
 # - no isatty method
 for _name in ('stdin', 'stdout', 'stderr'):
     _stream = getattr(sys, _name)
     try:
         if not _stream or not hasattr(_stream, "isatty") or not _stream.isatty():
             _is_tty = False
             break
     except ValueError:
         # stream is closed
         _is_tty = False
         break
 else:
     _is_tty = True
 
 
 _use_simple_prompt = ('IPY_TEST_SIMPLE_PROMPT' in os.environ) or (not _is_tty)
 
 def black_reformat_handler(text_before_cursor):
     """
     We do not need to protect against error,
     this is taken care at a higher level where any reformat error is ignored.
     Indeed we may call reformatting on incomplete code.
     """
     import black
 
     formatted_text = black.format_str(text_before_cursor, mode=black.FileMode())
     if not text_before_cursor.endswith("\n") and formatted_text.endswith("\n"):
         formatted_text = formatted_text[:-1]
     return formatted_text
 
 
 def yapf_reformat_handler(text_before_cursor):
     from yapf.yapflib import file_resources
     from yapf.yapflib import yapf_api
 
     style_config = file_resources.GetDefaultStyleForDir(os.getcwd())
     formatted_text, was_formatted = yapf_api.FormatCode(
         text_before_cursor, style_config=style_config
     )
     if was_formatted:
         if not text_before_cursor.endswith("\n") and formatted_text.endswith("\n"):
             formatted_text = formatted_text[:-1]
         return formatted_text
     else:
         return text_before_cursor
 
 
 class PtkHistoryAdapter(History):
     """
     Prompt toolkit has it's own way of handling history, Where it assumes it can
     Push/pull from history.
 
     """
 
     def __init__(self, shell):
         super().__init__()
         self.shell = shell
         self._refresh()
 
     def append_string(self, string):
         # we rely on sql for that.
         self._loaded = False
         self._refresh()
 
     def _refresh(self):
         if not self._loaded:
             self._loaded_strings = list(self.load_history_strings())
 
     def load_history_strings(self):
         last_cell = ""
         res = []
         for __, ___, cell in self.shell.history_manager.get_tail(
             self.shell.history_load_length, include_latest=True
         ):
             # Ignore blank lines and consecutive duplicates
             cell = cell.rstrip()
             if cell and (cell != last_cell):
                 res.append(cell)
                 last_cell = cell
         yield from res[::-1]
 
     def store_string(self, string: str) -> None:
         pass
 
 class TerminalInteractiveShell(InteractiveShell):
     mime_renderers = Dict().tag(config=True)
 
     space_for_menu = Integer(6, help='Number of line at the bottom of the screen '
                                      'to reserve for the tab completion menu, '
                                      'search history, ...etc, the height of '
                                      'these menus will at most this value. '
                                      'Increase it is you prefer long and skinny '
                                      'menus, decrease for short and wide.'
                             ).tag(config=True)
 
     pt_app: UnionType[PromptSession, None] = None
     auto_suggest: UnionType[
         AutoSuggestFromHistory, NavigableAutoSuggestFromHistory, None
     ] = None
     debugger_history = None
 
     debugger_history_file = Unicode(
         "~/.pdbhistory", help="File in which to store and read history"
     ).tag(config=True)
 
     simple_prompt = Bool(_use_simple_prompt,
         help="""Use `raw_input` for the REPL, without completion and prompt colors.
 
             Useful when controlling IPython as a subprocess, and piping
             STDIN/OUT/ERR. Known usage are: IPython's own testing machinery,
             and emacs' inferior-python subprocess (assuming you have set
             `python-shell-interpreter` to "ipython") available through the
             built-in `M-x run-python` and third party packages such as elpy.
 
             This mode default to `True` if the `IPY_TEST_SIMPLE_PROMPT`
             environment variable is set, or the current terminal is not a tty.
             Thus the Default value reported in --help-all, or config will often
             be incorrectly reported.
             """,
     ).tag(config=True)
 
     @property
     def debugger_cls(self):
         return Pdb if self.simple_prompt else TerminalPdb
 
     confirm_exit = Bool(True,
         help="""
         Set to confirm when you try to exit IPython with an EOF (Control-D
         in Unix, Control-Z/Enter in Windows). By typing 'exit' or 'quit',
         you can force a direct exit without any confirmation.""",
     ).tag(config=True)
 
     editing_mode = Unicode('emacs',
         help="Shortcut style to use at the prompt. 'vi' or 'emacs'.",
     ).tag(config=True)
 
     emacs_bindings_in_vi_insert_mode = Bool(
         True,
         help="Add shortcuts from 'emacs' insert mode to 'vi' insert mode.",
     ).tag(config=True)
 
     modal_cursor = Bool(
         True,
         help="""
        Cursor shape changes depending on vi mode: beam in vi insert mode,
        block in nav mode, underscore in replace mode.""",
     ).tag(config=True)
 
     ttimeoutlen = Float(
         0.01,
         help="""The time in milliseconds that is waited for a key code
        to complete.""",
     ).tag(config=True)
 
     timeoutlen = Float(
         0.5,
         help="""The time in milliseconds that is waited for a mapped key
        sequence to complete.""",
     ).tag(config=True)
 
     autoformatter = Unicode(
         None,
         help="Autoformatter to reformat Terminal code. Can be `'black'`, `'yapf'` or `None`",
         allow_none=True
     ).tag(config=True)
 
     auto_match = Bool(
         False,
         help="""
         Automatically add/delete closing bracket or quote when opening bracket or quote is entered/deleted.
         Brackets: (), [], {}
         Quotes: '', \"\"
         """,
     ).tag(config=True)
 
     mouse_support = Bool(False,
         help="Enable mouse support in the prompt\n(Note: prevents selecting text with the mouse)"
     ).tag(config=True)
 
     # We don't load the list of styles for the help string, because loading
     # Pygments plugins takes time and can cause unexpected errors.
     highlighting_style = Union([Unicode('legacy'), Type(klass=Style)],
         help="""The name or class of a Pygments style to use for syntax
         highlighting. To see available styles, run `pygmentize -L styles`."""
     ).tag(config=True)
 
     @validate('editing_mode')
     def _validate_editing_mode(self, proposal):
         if proposal['value'].lower() == 'vim':
             proposal['value']= 'vi'
         elif proposal['value'].lower() == 'default':
             proposal['value']= 'emacs'
 
         if hasattr(EditingMode, proposal['value'].upper()):
             return proposal['value'].lower()
 
         return self.editing_mode
 
     @observe('editing_mode')
     def _editing_mode(self, change):
         if self.pt_app:
             self.pt_app.editing_mode = getattr(EditingMode, change.new.upper())
 
     def _set_formatter(self, formatter):
         if formatter is None:
             self.reformat_handler = lambda x:x
         elif formatter == 'black':
             self.reformat_handler = black_reformat_handler
         elif formatter == "yapf":
             self.reformat_handler = yapf_reformat_handler
         else:
             raise ValueError
 
     @observe("autoformatter")
     def _autoformatter_changed(self, change):
         formatter = change.new
         self._set_formatter(formatter)
 
     @observe('highlighting_style')
     @observe('colors')
     def _highlighting_style_changed(self, change):
         self.refresh_style()
 
     def refresh_style(self):
         self._style = self._make_style_from_name_or_cls(self.highlighting_style)
 
     highlighting_style_overrides = Dict(
         help="Override highlighting format for specific tokens"
     ).tag(config=True)
 
     true_color = Bool(False,
         help="""Use 24bit colors instead of 256 colors in prompt highlighting.
         If your terminal supports true color, the following command should
         print ``TRUECOLOR`` in orange::
 
             printf \"\\x1b[38;2;255;100;0mTRUECOLOR\\x1b[0m\\n\"
         """,
     ).tag(config=True)
 
     editor = Unicode(get_default_editor(),
         help="Set the editor used by IPython (default to $EDITOR/vi/notepad)."
     ).tag(config=True)
 
     prompts_class = Type(Prompts, help='Class used to generate Prompt token for prompt_toolkit').tag(config=True)
 
     prompts = Instance(Prompts)
 
     @default('prompts')
     def _prompts_default(self):
         return self.prompts_class(self)
 
 #    @observe('prompts')
 #    def _(self, change):
 #        self._update_layout()
 
     @default('displayhook_class')
     def _displayhook_class_default(self):
         return RichPromptDisplayHook
 
     term_title = Bool(True,
         help="Automatically set the terminal title"
     ).tag(config=True)
 
     term_title_format = Unicode("IPython: {cwd}",
         help="Customize the terminal title format.  This is a python format string. " +
              "Available substitutions are: {cwd}."
     ).tag(config=True)
 
     display_completions = Enum(('column', 'multicolumn','readlinelike'),
         help= ( "Options for displaying tab completions, 'column', 'multicolumn', and "
                 "'readlinelike'. These options are for `prompt_toolkit`, see "
                 "`prompt_toolkit` documentation for more information."
                 ),
         default_value='multicolumn').tag(config=True)
 
     highlight_matching_brackets = Bool(True,
         help="Highlight matching brackets.",
     ).tag(config=True)
 
     extra_open_editor_shortcuts = Bool(False,
         help="Enable vi (v) or Emacs (C-X C-E) shortcuts to open an external editor. "
              "This is in addition to the F2 binding, which is always enabled."
     ).tag(config=True)
 
     handle_return = Any(None,
         help="Provide an alternative handler to be called when the user presses "
              "Return. This is an advanced option intended for debugging, which "
              "may be changed or removed in later releases."
     ).tag(config=True)
 
     enable_history_search = Bool(True,
         help="Allows to enable/disable the prompt toolkit history search"
     ).tag(config=True)
 
     autosuggestions_provider = Unicode(
         "NavigableAutoSuggestFromHistory",
         help="Specifies from which source automatic suggestions are provided. "
         "Can be set to ``'NavigableAutoSuggestFromHistory'`` (:kbd:`up` and "
         ":kbd:`down` swap suggestions), ``'AutoSuggestFromHistory'``, "
         " or ``None`` to disable automatic suggestions. "
         "Default is `'NavigableAutoSuggestFromHistory`'.",
         allow_none=True,
     ).tag(config=True)
 
     def _set_autosuggestions(self, provider):
         # disconnect old handler
         if self.auto_suggest and isinstance(
             self.auto_suggest, NavigableAutoSuggestFromHistory
         ):
             self.auto_suggest.disconnect()
         if provider is None:
             self.auto_suggest = None
         elif provider == "AutoSuggestFromHistory":
             self.auto_suggest = AutoSuggestFromHistory()
         elif provider == "NavigableAutoSuggestFromHistory":
             self.auto_suggest = NavigableAutoSuggestFromHistory()
         else:
             raise ValueError("No valid provider.")
         if self.pt_app:
             self.pt_app.auto_suggest = self.auto_suggest
 
     @observe("autosuggestions_provider")
     def _autosuggestions_provider_changed(self, change):
         provider = change.new
         self._set_autosuggestions(provider)
 
     shortcuts = List(
         trait=Dict(
             key_trait=Enum(
                 [
                     "command",
                     "match_keys",
                     "match_filter",
                     "new_keys",
                     "new_filter",
                     "create",
                 ]
             ),
             per_key_traits={
                 "command": Unicode(),
                 "match_keys": List(Unicode()),
                 "match_filter": Unicode(),
                 "new_keys": List(Unicode()),
                 "new_filter": Unicode(),
                 "create": Bool(False),
             },
         ),
         help="""Add, disable or modifying shortcuts.
 
         Each entry on the list should be a dictionary with ``command`` key
         identifying the target function executed by the shortcut and at least
         one of the following:
 
         - ``match_keys``: list of keys used to match an existing shortcut,
         - ``match_filter``: shortcut filter used to match an existing shortcut,
         - ``new_keys``: list of keys to set,
         - ``new_filter``: a new shortcut filter to set
 
         The filters have to be composed of pre-defined verbs and joined by one
         of the following conjunctions: ``&`` (and), ``|`` (or), ``~`` (not).
         The pre-defined verbs are:
 
         {}
 
 
         To disable a shortcut set ``new_keys`` to an empty list.
         To add a shortcut add key ``create`` with value ``True``.
 
         When modifying/disabling shortcuts, ``match_keys``/``match_filter`` can
         be omitted if the provided specification uniquely identifies a shortcut
         to be modified/disabled. When modifying a shortcut ``new_filter`` or
         ``new_keys`` can be omitted which will result in reuse of the existing
         filter/keys.
 
         Only shortcuts defined in IPython (and not default prompt-toolkit
         shortcuts) can be modified or disabled. The full list of shortcuts,
         command identifiers and filters is available under
         :ref:`terminal-shortcuts-list`.
         """.format(
             "\n        ".join([f"- `{k}`" for k in KEYBINDING_FILTERS])
         ),
     ).tag(config=True)
 
     @observe("shortcuts")
     def _shortcuts_changed(self, change):
         if self.pt_app:
             self.pt_app.key_bindings = self._merge_shortcuts(user_shortcuts=change.new)
 
     def _merge_shortcuts(self, user_shortcuts):
         # rebuild the bindings list from scratch
         key_bindings = create_ipython_shortcuts(self)
 
         # for now we only allow adding shortcuts for commands which are already
         # registered; this is a security precaution.
         known_commands = {
             create_identifier(binding.command): binding.command
             for binding in KEY_BINDINGS
         }
         shortcuts_to_skip = []
         shortcuts_to_add = []
 
         for shortcut in user_shortcuts:
             command_id = shortcut["command"]
             if command_id not in known_commands:
                 allowed_commands = "\n - ".join(known_commands)
                 raise ValueError(
                     f"{command_id} is not a known shortcut command."
                     f" Allowed commands are: \n - {allowed_commands}"
                 )
             old_keys = shortcut.get("match_keys", None)
             old_filter = (
                 filter_from_string(shortcut["match_filter"])
                 if "match_filter" in shortcut
                 else None
             )
             matching = [
                 binding
                 for binding in KEY_BINDINGS
                 if (
                     (old_filter is None or binding.filter == old_filter)
                     and (old_keys is None or [k for k in binding.keys] == old_keys)
                     and create_identifier(binding.command) == command_id
                 )
             ]
 
             new_keys = shortcut.get("new_keys", None)
             new_filter = shortcut.get("new_filter", None)
 
             command = known_commands[command_id]
 
             creating_new = shortcut.get("create", False)
             modifying_existing = not creating_new and (
                 new_keys is not None or new_filter
             )
 
             if creating_new and new_keys == []:
                 raise ValueError("Cannot add a shortcut without keys")
 
             if modifying_existing:
                 specification = {
                     key: shortcut[key]
                     for key in ["command", "filter"]
                     if key in shortcut
                 }
                 if len(matching) == 0:
                     raise ValueError(
                         f"No shortcuts matching {specification} found in {KEY_BINDINGS}"
                     )
                 elif len(matching) > 1:
                     raise ValueError(
                         f"Multiple shortcuts matching {specification} found,"
                         f" please add keys/filter to select one of: {matching}"
                     )
 
                 matched = matching[0]
                 old_filter = matched.filter
                 old_keys = list(matched.keys)
                 shortcuts_to_skip.append(
                     RuntimeBinding(
                         command,
                         keys=old_keys,
                         filter=old_filter,
                     )
                 )
 
             if new_keys != []:
                 shortcuts_to_add.append(
                     RuntimeBinding(
                         command,
                         keys=new_keys or old_keys,
-                        filter=filter_from_string(new_filter)
+                        filter=(
+                            filter_from_string(new_filter)
                         if new_filter is not None
                         else (
                             old_filter
                             if old_filter is not None
                             else filter_from_string("always")
+                            )
                         ),
                     )
                 )
 
         # rebuild the bindings list from scratch
         key_bindings = create_ipython_shortcuts(self, skip=shortcuts_to_skip)
         for binding in shortcuts_to_add:
             add_binding(key_bindings, binding)
 
         return key_bindings
 
     prompt_includes_vi_mode = Bool(True,
         help="Display the current vi mode (when using vi editing mode)."
     ).tag(config=True)
 
     prompt_line_number_format = Unicode(
         "",
         help="The format for line numbering, will be passed `line` (int, 1 based)"
         " the current line number and `rel_line` the relative line number."
         " for example to display both you can use the following template string :"
         " c.TerminalInteractiveShell.prompt_line_number_format='{line: 4d}/{rel_line:+03d} | '"
         " This will display the current line number, with leading space and a width of at least 4"
         " character, as well as the relative line number 0 padded and always with a + or - sign."
         " Note that when using Emacs mode the prompt of the first line may not update.",
     ).tag(config=True)
 
     @observe('term_title')
     def init_term_title(self, change=None):
         # Enable or disable the terminal title.
         if self.term_title and _is_tty:
             toggle_set_term_title(True)
             set_term_title(self.term_title_format.format(cwd=abbrev_cwd()))
         else:
             toggle_set_term_title(False)
 
     def restore_term_title(self):
         if self.term_title and _is_tty:
             restore_term_title()
 
     def init_display_formatter(self):
         super(TerminalInteractiveShell, self).init_display_formatter()
         # terminal only supports plain text
         self.display_formatter.active_types = ["text/plain"]
 
     def init_prompt_toolkit_cli(self):
         if self.simple_prompt:
             # Fall back to plain non-interactive output for tests.
             # This is very limited.
             def prompt():
                 prompt_text = "".join(x[1] for x in self.prompts.in_prompt_tokens())
                 lines = [input(prompt_text)]
                 prompt_continuation = "".join(x[1] for x in self.prompts.continuation_prompt_tokens())
                 while self.check_complete('\n'.join(lines))[0] == 'incomplete':
                     lines.append( input(prompt_continuation) )
                 return '\n'.join(lines)
             self.prompt_for_code = prompt
             return
 
         # Set up keyboard shortcuts
         key_bindings = self._merge_shortcuts(user_shortcuts=self.shortcuts)
 
         # Pre-populate history from IPython's history database
         history = PtkHistoryAdapter(self)
 
         self._style = self._make_style_from_name_or_cls(self.highlighting_style)
         self.style = DynamicStyle(lambda: self._style)
 
         editing_mode = getattr(EditingMode, self.editing_mode.upper())
 
         self._use_asyncio_inputhook = False
         self.pt_app = PromptSession(
             auto_suggest=self.auto_suggest,
             editing_mode=editing_mode,
             key_bindings=key_bindings,
             history=history,
             completer=IPythonPTCompleter(shell=self),
             enable_history_search=self.enable_history_search,
             style=self.style,
             include_default_pygments_style=False,
             mouse_support=self.mouse_support,
             enable_open_in_editor=self.extra_open_editor_shortcuts,
             color_depth=self.color_depth,
             tempfile_suffix=".py",
             **self._extra_prompt_options(),
         )
         if isinstance(self.auto_suggest, NavigableAutoSuggestFromHistory):
             self.auto_suggest.connect(self.pt_app)
 
     def _make_style_from_name_or_cls(self, name_or_cls):
         """
         Small wrapper that make an IPython compatible style from a style name
 
         We need that to add style for prompt ... etc.
         """
         style_overrides = {}
         if name_or_cls == 'legacy':
             legacy = self.colors.lower()
             if legacy == 'linux':
                 style_cls = get_style_by_name('monokai')
                 style_overrides = _style_overrides_linux
             elif legacy == 'lightbg':
                 style_overrides = _style_overrides_light_bg
                 style_cls = get_style_by_name('pastie')
             elif legacy == 'neutral':
                 # The default theme needs to be visible on both a dark background
                 # and a light background, because we can't tell what the terminal
                 # looks like. These tweaks to the default theme help with that.
                 style_cls = get_style_by_name('default')
                 style_overrides.update({
                     Token.Number: '#ansigreen',
                     Token.Operator: 'noinherit',
                     Token.String: '#ansiyellow',
                     Token.Name.Function: '#ansiblue',
                     Token.Name.Class: 'bold #ansiblue',
                     Token.Name.Namespace: 'bold #ansiblue',
                     Token.Name.Variable.Magic: '#ansiblue',
                     Token.Prompt: '#ansigreen',
                     Token.PromptNum: '#ansibrightgreen bold',
                     Token.OutPrompt: '#ansired',
                     Token.OutPromptNum: '#ansibrightred bold',
                 })
 
                 # Hack: Due to limited color support on the Windows console
                 # the prompt colors will be wrong without this
                 if os.name == 'nt':
                     style_overrides.update({
                         Token.Prompt: '#ansidarkgreen',
                         Token.PromptNum: '#ansigreen bold',
                         Token.OutPrompt: '#ansidarkred',
                         Token.OutPromptNum: '#ansired bold',
                     })
             elif legacy =='nocolor':
                 style_cls=_NoStyle
                 style_overrides = {}
             else :
                 raise ValueError('Got unknown colors: ', legacy)
         else :
             if isinstance(name_or_cls, str):
                 style_cls = get_style_by_name(name_or_cls)
             else:
                 style_cls = name_or_cls
             style_overrides = {
                 Token.Prompt: '#ansigreen',
                 Token.PromptNum: '#ansibrightgreen bold',
                 Token.OutPrompt: '#ansired',
                 Token.OutPromptNum: '#ansibrightred bold',
             }
         style_overrides.update(self.highlighting_style_overrides)
         style = merge_styles([
             style_from_pygments_cls(style_cls),
             style_from_pygments_dict(style_overrides),
         ])
 
         return style
 
     @property
     def pt_complete_style(self):
         return {
             'multicolumn': CompleteStyle.MULTI_COLUMN,
             'column': CompleteStyle.COLUMN,
             'readlinelike': CompleteStyle.READLINE_LIKE,
         }[self.display_completions]
 
     @property
     def color_depth(self):
         return (ColorDepth.TRUE_COLOR if self.true_color else None)
 
     def _extra_prompt_options(self):
         """
         Return the current layout option for the current Terminal InteractiveShell
         """
         def get_message():
             return PygmentsTokens(self.prompts.in_prompt_tokens())
 
         if self.editing_mode == "emacs" and self.prompt_line_number_format == "":
             # with emacs mode the prompt is (usually) static, so we call only
             # the function once. With VI mode it can toggle between [ins] and
             # [nor] so we can't precompute.
             # here I'm going to favor the default keybinding which almost
             # everybody uses to decrease CPU usage.
             # if we have issues with users with custom Prompts we can see how to
             # work around this.
             get_message = get_message()
 
         options = {
             "complete_in_thread": False,
             "lexer": IPythonPTLexer(),
             "reserve_space_for_menu": self.space_for_menu,
             "message": get_message,
             "prompt_continuation": (
                 lambda width, lineno, is_soft_wrap: PygmentsTokens(
                     _backward_compat_continuation_prompt_tokens(
                         self.prompts.continuation_prompt_tokens, width, lineno=lineno
                     )
                 )
             ),
             "multiline": True,
             "complete_style": self.pt_complete_style,
             "input_processors": [
                 # Highlight matching brackets, but only when this setting is
                 # enabled, and only when the DEFAULT_BUFFER has the focus.
                 ConditionalProcessor(
                     processor=HighlightMatchingBracketProcessor(chars="[](){}"),
                     filter=HasFocus(DEFAULT_BUFFER)
                     & ~IsDone()
                     & Condition(lambda: self.highlight_matching_brackets),
                 ),
                 # Show auto-suggestion in lines other than the last line.
                 ConditionalProcessor(
                     processor=AppendAutoSuggestionInAnyLine(),
                     filter=HasFocus(DEFAULT_BUFFER)
                     & ~IsDone()
                     & Condition(
                         lambda: isinstance(
                             self.auto_suggest, NavigableAutoSuggestFromHistory
                         )
                     ),
                 ),
             ],
         }
         if not PTK3:
             options['inputhook'] = self.inputhook
 
         return options
 
     def prompt_for_code(self):
         if self.rl_next_input:
             default = self.rl_next_input
             self.rl_next_input = None
         else:
             default = ''
 
         # In order to make sure that asyncio code written in the
         # interactive shell doesn't interfere with the prompt, we run the
         # prompt in a different event loop.
         # If we don't do this, people could spawn coroutine with a
         # while/true inside which will freeze the prompt.
 
         with patch_stdout(raw=True):
             if self._use_asyncio_inputhook:
                 # When we integrate the asyncio event loop, run the UI in the
                 # same event loop as the rest of the code. don't use an actual
                 # input hook. (Asyncio is not made for nesting event loops.)
                 asyncio_loop = get_asyncio_loop()
                 text = asyncio_loop.run_until_complete(
                     self.pt_app.prompt_async(
                         default=default, **self._extra_prompt_options()
                     )
                 )
             else:
                 text = self.pt_app.prompt(
                     default=default,
                     inputhook=self._inputhook,
                     **self._extra_prompt_options(),
                 )
 
         return text
 
     def enable_win_unicode_console(self):
         # Since IPython 7.10 doesn't support python < 3.6 and PEP 528, Python uses the unicode APIs for the Windows
         # console by default, so WUC shouldn't be needed.
         warn("`enable_win_unicode_console` is deprecated since IPython 7.10, does not do anything and will be removed in the future",
              DeprecationWarning,
              stacklevel=2)
 
     def init_io(self):
         if sys.platform not in {'win32', 'cli'}:
             return
 
         import colorama
         colorama.init()
 
     def init_magics(self):
         super(TerminalInteractiveShell, self).init_magics()
         self.register_magics(TerminalMagics)
 
     def init_alias(self):
         # The parent class defines aliases that can be safely used with any
         # frontend.
         super(TerminalInteractiveShell, self).init_alias()
 
         # Now define aliases that only make sense on the terminal, because they
         # need direct access to the console in a way that we can't emulate in
         # GUI or web frontend
         if os.name == 'posix':
             for cmd in ('clear', 'more', 'less', 'man'):
                 self.alias_manager.soft_define_alias(cmd, cmd)
 
     def __init__(self, *args, **kwargs) -> None:
         super(TerminalInteractiveShell, self).__init__(*args, **kwargs)
         self._set_autosuggestions(self.autosuggestions_provider)
         self.init_prompt_toolkit_cli()
         self.init_term_title()
         self.keep_running = True
         self._set_formatter(self.autoformatter)
 
     def ask_exit(self):
         self.keep_running = False
 
     rl_next_input = None
 
     def interact(self):
         self.keep_running = True
         while self.keep_running:
             print(self.separate_in, end='')
 
             try:
                 code = self.prompt_for_code()
             except EOFError:
                 if (not self.confirm_exit) \
                         or self.ask_yes_no('Do you really want to exit ([y]/n)?','y','n'):
                     self.ask_exit()
 
             else:
                 if code:
                     self.run_cell(code, store_history=True)
 
     def mainloop(self):
         # An extra layer of protection in case someone mashing Ctrl-C breaks
         # out of our internal code.
         while True:
             try:
                 self.interact()
                 break
             except KeyboardInterrupt as e:
                 print("\n%s escaped interact()\n" % type(e).__name__)
             finally:
                 # An interrupt during the eventloop will mess up the
                 # internal state of the prompt_toolkit library.
                 # Stopping the eventloop fixes this, see
                 # https://github.com/ipython/ipython/pull/9867
                 if hasattr(self, '_eventloop'):
                     self._eventloop.stop()
 
                 self.restore_term_title()
 
         # try to call some at-exit operation optimistically as some things can't
         # be done during interpreter shutdown. this is technically inaccurate as
         # this make mainlool not re-callable, but that should be a rare if not
         # in existent use case.
 
         self._atexit_once()
 
     _inputhook = None
     def inputhook(self, context):
         if self._inputhook is not None:
             self._inputhook(context)
 
     active_eventloop: Optional[str] = None
 
     def enable_gui(self, gui: Optional[str] = None) -> None:
         if gui:
             from ..core.pylabtools import _convert_gui_from_matplotlib
 
             gui = _convert_gui_from_matplotlib(gui)
 
         if self.simple_prompt is True and gui is not None:
             print(
                 f'Cannot install event loop hook for "{gui}" when running with `--simple-prompt`.'
             )
             print(
                 "NOTE: Tk is supported natively; use Tk apps and Tk backends with `--simple-prompt`."
             )
             return
 
         if self._inputhook is None and gui is None:
             print("No event loop hook running.")
             return
 
         if self._inputhook is not None and gui is not None:
             newev, newinhook = get_inputhook_name_and_func(gui)
             if self._inputhook == newinhook:
                 # same inputhook, do nothing
                 self.log.info(
                     f"Shell is already running the {self.active_eventloop} eventloop. Doing nothing"
                 )
                 return
             self.log.warning(
                 f"Shell is already running a different gui event loop for {self.active_eventloop}. "
                 "Call with no arguments to disable the current loop."
             )
             return
         if self._inputhook is not None and gui is None:
             self.active_eventloop = self._inputhook = None
 
         if gui and (gui not in {None, "webagg"}):
             # This hook runs with each cycle of the `prompt_toolkit`'s event loop.
             self.active_eventloop, self._inputhook = get_inputhook_name_and_func(gui)
         else:
             self.active_eventloop = self._inputhook = None
 
         self._use_asyncio_inputhook = gui == "asyncio"
 
     # Run !system commands directly, not through pipes, so terminal programs
     # work correctly.
     system = InteractiveShell.system_raw
 
     def auto_rewrite_input(self, cmd):
         """Overridden from the parent class to use fancy rewriting prompt"""
         if not self.show_rewritten_input:
             return
 
         tokens = self.prompts.rewrite_prompt_tokens()
         if self.pt_app:
             print_formatted_text(PygmentsTokens(tokens), end='',
                                  style=self.pt_app.app.style)
             print(cmd)
         else:
             prompt = ''.join(s for t, s in tokens)
             print(prompt, cmd, sep='')
 
     _prompts_before = None
     def switch_doctest_mode(self, mode):
         """Switch prompts to classic for %doctest_mode"""
         if mode:
             self._prompts_before = self.prompts
             self.prompts = ClassicPrompts(self)
         elif self._prompts_before:
             self.prompts = self._prompts_before
             self._prompts_before = None
 #        self._update_layout()
 
 
 InteractiveShellABC.register(TerminalInteractiveShell)
 
 if __name__ == '__main__':
     TerminalInteractiveShell.instance().interact()

IPython/terminal/shortcuts/auto_match.py

@@ -1,104 +1,105
 """
 Utilities function for keybinding with prompt toolkit.
 
 This will be bound to specific key press and filter modes,
 like whether we are in edit mode, and whether the completer is open.
 """
+
 import re
 from prompt_toolkit.key_binding import KeyPressEvent
 
 
 def parenthesis(event: KeyPressEvent):
     """Auto-close parenthesis"""
     event.current_buffer.insert_text("()")
     event.current_buffer.cursor_left()
 
 
 def brackets(event: KeyPressEvent):
     """Auto-close brackets"""
     event.current_buffer.insert_text("[]")
     event.current_buffer.cursor_left()
 
 
 def braces(event: KeyPressEvent):
     """Auto-close braces"""
     event.current_buffer.insert_text("{}")
     event.current_buffer.cursor_left()
 
 
 def double_quote(event: KeyPressEvent):
     """Auto-close double quotes"""
     event.current_buffer.insert_text('""')
     event.current_buffer.cursor_left()
 
 
 def single_quote(event: KeyPressEvent):
     """Auto-close single quotes"""
     event.current_buffer.insert_text("''")
     event.current_buffer.cursor_left()
 
 
 def docstring_double_quotes(event: KeyPressEvent):
     """Auto-close docstring (double quotes)"""
     event.current_buffer.insert_text('""""')
     event.current_buffer.cursor_left(3)
 
 
 def docstring_single_quotes(event: KeyPressEvent):
     """Auto-close docstring (single quotes)"""
     event.current_buffer.insert_text("''''")
     event.current_buffer.cursor_left(3)
 
 
 def raw_string_parenthesis(event: KeyPressEvent):
     """Auto-close parenthesis in raw strings"""
     matches = re.match(
         r".*(r|R)[\"'](-*)",
         event.current_buffer.document.current_line_before_cursor,
     )
     dashes = matches.group(2) if matches else ""
     event.current_buffer.insert_text("()" + dashes)
     event.current_buffer.cursor_left(len(dashes) + 1)
 
 
 def raw_string_bracket(event: KeyPressEvent):
     """Auto-close bracker in raw strings"""
     matches = re.match(
         r".*(r|R)[\"'](-*)",
         event.current_buffer.document.current_line_before_cursor,
     )
     dashes = matches.group(2) if matches else ""
     event.current_buffer.insert_text("[]" + dashes)
     event.current_buffer.cursor_left(len(dashes) + 1)
 
 
 def raw_string_braces(event: KeyPressEvent):
     """Auto-close braces in raw strings"""
     matches = re.match(
         r".*(r|R)[\"'](-*)",
         event.current_buffer.document.current_line_before_cursor,
     )
     dashes = matches.group(2) if matches else ""
     event.current_buffer.insert_text("{}" + dashes)
     event.current_buffer.cursor_left(len(dashes) + 1)
 
 
 def skip_over(event: KeyPressEvent):
     """Skip over automatically added parenthesis/quote.
 
     (rather than adding another parenthesis/quote)"""
     event.current_buffer.cursor_right()
 
 
 def delete_pair(event: KeyPressEvent):
     """Delete auto-closed parenthesis"""
     event.current_buffer.delete()
     event.current_buffer.delete_before_cursor()
 
 
 auto_match_parens = {"(": parenthesis, "[": brackets, "{": braces}
 auto_match_parens_raw_string = {
     "(": raw_string_parenthesis,
     "[": raw_string_bracket,
     "{": raw_string_braces,
 }