Show signature with Jedi.

When completion are function, jedi is capable of giving us the function
signature. This adds a signature field to the (still unstable) completer
API and make use of it in the IPython terminal UI.

It is not (yet) exposed by the ipykernel.

Additionally add typing to a couple of locations.

docs/source/whatsnew/pr/jedi-signature.rst

@@ -0,0 +1,4 b''
+Terminal IPython will now show the signature of the function while completing.
+Only the currently highlighted function will show its signature on the line
+below the completer by default. The functionality is recent so might be
+limited, we welcome bug report and enhancement request on it.

IPython/core/completer.py

@@ -1,1953 +1,2017 b''
-# encoding: utf-8
 """Completion for IPython.
 
 This module started as fork of the rlcompleter module in the Python standard
 library.  The original enhancements made to rlcompleter have been sent
 upstream and were accepted as of Python 2.3,
 
 This module now support a wide variety of completion mechanism both available
 for normal classic Python code, as well as completer for IPython specific
 Syntax like magics.
 
 Latex and Unicode completion
 ============================
 
 IPython and compatible frontends not only can complete your code, but can help
 you to input a wide range of characters. In particular we allow you to insert
 a unicode character using the tab completion mechanism.
 
 Forward latex/unicode completion
 --------------------------------
 
 Forward completion allows you to easily type a unicode character using its latex
 name, or unicode long description. To do so type a backslash follow by the
 relevant name and press tab:
 
 
 Using latex completion:
 
 .. code::
 
     \\alpha<tab>
     α
 
 or using unicode completion:
 
 
 .. code::
 
     \\greek small letter alpha<tab>
     α
 
 
 Only valid Python identifiers will complete. Combining characters (like arrow or
 dots) are also available, unlike latex they need to be put after the their
 counterpart that is to say, `F\\\\vec<tab>` is correct, not `\\\\vec<tab>F`.
 
 Some browsers are known to display combining characters incorrectly.
 
 Backward latex completion
 -------------------------
 
 It is sometime challenging to know how to type a character, if you are using
 IPython, or any compatible frontend you can prepend backslash to the character
 and press `<tab>` to expand it to its latex form.
 
 .. code::
 
     \\α<tab>
     \\alpha
 
 
 Both forward and backward completions can be deactivated by setting the
 ``Completer.backslash_combining_completions`` option to ``False``.
 
 
 Experimental
 ============
 
 Starting with IPython 6.0, this module can make use of the Jedi library to
 generate completions both using static analysis of the code, and dynamically
 inspecting multiple namespaces. The APIs attached to this new mechanism is
 unstable and will raise unless use in an :any:`provisionalcompleter` context
 manager.
 
 You will find that the following are experimental:
 
     - :any:`provisionalcompleter`
     - :any:`IPCompleter.completions`
     - :any:`Completion`
     - :any:`rectify_completions`
 
 .. note::
 
     better name for :any:`rectify_completions` ?
 
 We welcome any feedback on these new API, and we also encourage you to try this
 module in debug mode (start IPython with ``--Completer.debug=True``) in order
 to have extra logging information is :any:`jedi` is crashing, or if current
 IPython completer pending deprecations are returning results not yet handled
-by :any:`jedi`.
+by :any:`jedi`
 
 Using Jedi for tab completion allow snippets like the following to work without
 having to execute any code:
 
    >>> myvar = ['hello', 42]
    ... myvar[1].bi<tab>
 
 Tab completion will be able to infer that ``myvar[1]`` is a real number without
 executing any code unlike the previously available ``IPCompleter.greedy``
 option.
 
 Be sure to update :any:`jedi` to the latest stable version or to try the
 current development version to get better completions.
 """
 
-# skip module docstests
-skip_doctest = True
 
 # Copyright (c) IPython Development Team.
 # Distributed under the terms of the Modified BSD License.
 #
 # Some of this code originated from rlcompleter in the Python standard library
 # Copyright (C) 2001 Python Software Foundation, www.python.org
 
 
 import __main__
 import builtins as builtin_mod
 import glob
 import time
 import inspect
 import itertools
 import keyword
 import os
 import re
 import sys
 import unicodedata
 import string
 import warnings
 
 from contextlib import contextmanager
 from importlib import import_module
 from typing import Iterator, List, Tuple, Iterable, Union
 from types import SimpleNamespace
 
 from traitlets.config.configurable import Configurable
 from IPython.core.error import TryNext
 from IPython.core.inputsplitter import ESC_MAGIC
 from IPython.core.latex_symbols import latex_symbols, reverse_latex_symbol
 from IPython.core.oinspect import InspectColors
 from IPython.utils import generics
 from IPython.utils.dir2 import dir2, get_real_method
 from IPython.utils.process import arg_split
 from traitlets import Bool, Enum, observe, Int
 
+# skip module docstests
+skip_doctest = True
+
 try:
     import jedi
     import jedi.api.helpers
+    import jedi.api.classes
     JEDI_INSTALLED = True
 except ImportError:
     JEDI_INSTALLED = False
 #-----------------------------------------------------------------------------
 # Globals
 #-----------------------------------------------------------------------------
 
 # Public API
 __all__ = ['Completer','IPCompleter']
 
 if sys.platform == 'win32':
     PROTECTABLES = ' '
 else:
     PROTECTABLES = ' ()[]{}?=\\|;:\'#*"^&'
 
 
 _deprecation_readline_sentinel = object()
 
 
 class ProvisionalCompleterWarning(FutureWarning):
     """
     Exception raise by an experimental feature in this module.
 
     Wrap code in :any:`provisionalcompleter` context manager if you
     are certain you want to use an unstable feature.
     """
     pass
 
 warnings.filterwarnings('error', category=ProvisionalCompleterWarning)
 
 @contextmanager
 def provisionalcompleter(action='ignore'):
     """
 
 
     This contest manager has to be used in any place where unstable completer
     behavior and API may be called.
 
     >>> with provisionalcompleter():
     ...     completer.do_experimetal_things() # works
 
     >>> completer.do_experimental_things() # raises.
 
     .. note:: Unstable
 
         By using this context manager you agree that the API in use may change
         without warning, and that you won't complain if they do so.
 
         You also understand that if the API is not to you liking you should report
         a bug to explain your use case upstream and improve the API and will loose
         credibility if you complain after the API is make stable.
 
         We'll be happy to get your feedback , feature request and improvement on
         any of the unstable APIs !
     """
     with warnings.catch_warnings():
         warnings.filterwarnings(action, category=ProvisionalCompleterWarning)
         yield
 
 
 def has_open_quotes(s):
     """Return whether a string has open quotes.
 
     This simply counts whether the number of quote characters of either type in
     the string is odd.
 
     Returns
     -------
     If there is an open quote, the quote character is returned.  Else, return
     False.
     """
     # We check " first, then ', so complex cases with nested quotes will get
     # the " to take precedence.
     if s.count('"') % 2:
         return '"'
     elif s.count("'") % 2:
         return "'"
     else:
         return False
 
 
 def protect_filename(s, protectables=PROTECTABLES):
     """Escape a string to protect certain characters."""
     if set(s) & set(protectables):
         if sys.platform == "win32":
             return '"' + s + '"'
         else:
             return "".join(("\\" + c if c in protectables else c) for c in s)
     else:
         return s
 
 
-def expand_user(path):
+def expand_user(path:str) -> Tuple[str, bool, str]:
     """Expand ``~``-style usernames in strings.
 
     This is similar to :func:`os.path.expanduser`, but it computes and returns
     extra information that will be useful if the input was being used in
     computing completions, and you wish to return the completions with the
     original '~' instead of its expanded value.
 
     Parameters
     ----------
     path : str
       String to be expanded.  If no ~ is present, the output is the same as the
       input.
 
     Returns
     -------
     newpath : str
       Result of ~ expansion in the input path.
     tilde_expand : bool
       Whether any expansion was performed or not.
     tilde_val : str
       The value that ~ was replaced with.
     """
     # Default values
     tilde_expand = False
     tilde_val = ''
     newpath = path
 
     if path.startswith('~'):
         tilde_expand = True
         rest = len(path)-1
         newpath = os.path.expanduser(path)
         if rest:
             tilde_val = newpath[:-rest]
         else:
             tilde_val = newpath
 
     return newpath, tilde_expand, tilde_val
 
 
-def compress_user(path, tilde_expand, tilde_val):
+def compress_user(path:str, tilde_expand:bool, tilde_val:str) -> str:
     """Does the opposite of expand_user, with its outputs.
     """
     if tilde_expand:
         return path.replace(tilde_val, '~')
     else:
         return path
 
 
 def completions_sorting_key(word):
     """key for sorting completions
 
     This does several things:
 
     - Lowercase all completions, so they are sorted alphabetically with
       upper and lower case words mingled
     - Demote any completions starting with underscores to the end
     - Insert any %magic and %%cellmagic completions in the alphabetical order
       by their name
     """
     # Case insensitive sort
     word = word.lower()
 
     prio1, prio2 = 0, 0
 
     if word.startswith('__'):
         prio1 = 2
     elif word.startswith('_'):
         prio1 = 1
 
     if word.endswith('='):
         prio1 = -1
 
     if word.startswith('%%'):
         # If there's another % in there, this is something else, so leave it alone
         if not "%" in word[2:]:
             word = word[2:]
             prio2 = 2
     elif word.startswith('%'):
         if not "%" in word[1:]:
             word = word[1:]
             prio2 = 1
 
     return prio1, word, prio2
 
 
 class _FakeJediCompletion:
     """
     This is a workaround to communicate to the UI that Jedi has crashed and to
     report a bug. Will be used only id :any:`IPCompleter.debug` is set to true.
 
     Added in IPython 6.0 so should likely be removed for 7.0
 
     """
 
     def __init__(self, name):
 
         self.name = name
         self.complete = name
         self.type = 'crashed'
         self.name_with_symbols = name
+        self.signature = ''
+        self._origin = 'fake'
 
     def __repr__(self):
         return '<Fake completion object jedi has crashed>'
 
 
 class Completion:
     """
     Completion object used and return by IPython completers.
 
     .. warning:: Unstable
         
         This function is unstable, API may change without warning.
         It will also raise unless use in proper context manager.
 
     This act as a middle ground :any:`Completion` object between the
     :any:`jedi.api.classes.Completion` object and the Prompt Toolkit completion
     object. While Jedi need a lot of information about evaluator and how the
     code should be ran/inspected, PromptToolkit (and other frontend) mostly
     need user facing information.
 
     - Which range should be replaced replaced by what.
     - Some metadata (like completion type), or meta informations to displayed to
       the use user.
 
     For debugging purpose we can also store the origin of the completion (``jedi``,
     ``IPython.python_matches``, ``IPython.magics_matches``...).
     """
 
-    def __init__(self, start: int, end: int, text: str, *, type: str=None, _origin='') -> None:
+    __slots__ = ['start', 'end', 'text', 'type', 'signature', '_origin']
+
+    def __init__(self, start: int, end: int, text: str, *, type: str=None, _origin='', signature='') -> None:
         warnings.warn("``Completion`` is a provisional API (as of IPython 6.0). "
                       "It may change without warnings. "
                       "Use in corresponding context manager.",
                       category=ProvisionalCompleterWarning, stacklevel=2)
 
         self.start = start
         self.end = end
         self.text = text
         self.type = type
+        self.signature = signature
         self._origin = _origin
 
     def __repr__(self):
-        return '<Completion start=%s end=%s text=%r type=%r>' % (self.start, self.end, self.text, self.type or '?')
+        return '<Completion start=%s end=%s text=%r type=%r, signature=%r,>' % \
+                (self.start, self.end, self.text, self.type or '?', self.signature or '?')
 
     def __eq__(self, other)->Bool:
         """
         Equality and hash do not hash the type (as some completer may not be
         able to infer the type), but are use to (partially) de-duplicate
         completion.
 
         Completely de-duplicating completion is a bit tricker that just
         comparing as it depends on surrounding text, which Completions are not
         aware of.
         """
         return self.start == other.start and \
             self.end == other.end and \
             self.text == other.text
 
     def __hash__(self):
         return hash((self.start, self.end, self.text))
 
 
 _IC = Iterable[Completion]
 
 
 def _deduplicate_completions(text: str, completions: _IC)-> _IC:
     """
     Deduplicate a set of completions.
 
     .. warning:: Unstable
 
         This function is unstable, API may change without warning.
 
     Parameters
     ----------
     text: str
         text that should be completed.
     completions: Iterator[Completion]
         iterator over the completions to deduplicate
 
+    Yields
+    ------
+    `Completions` objects
+
 
     Completions coming from multiple sources, may be different but end up having
     the same effect when applied to ``text``. If this is the case, this will
     consider completions as equal and only emit the first encountered.
 
     Not folded in `completions()` yet for debugging purpose, and to detect when
     the IPython completer does return things that Jedi does not, but should be
     at some point.
     """
     completions = list(completions)
     if not completions:
         return
 
     new_start = min(c.start for c in completions)
     new_end = max(c.end for c in completions)
 
     seen = set()
     for c in completions:
         new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
         if new_text not in seen:
             yield c
             seen.add(new_text)
 
 
 def rectify_completions(text: str, completions: _IC, *, _debug=False)->_IC:
     """
     Rectify a set of completions to all have the same ``start`` and ``end``
 
     .. warning:: Unstable
 
         This function is unstable, API may change without warning.
         It will also raise unless use in proper context manager.
 
     Parameters
     ----------
     text: str
         text that should be completed.
     completions: Iterator[Completion]
         iterator over the completions to rectify
 
 
     :any:`jedi.api.classes.Completion` s returned by Jedi may not have the same start and end, though
     the Jupyter Protocol requires them to behave like so. This will readjust
     the completion to have the same ``start`` and ``end`` by padding both
     extremities with surrounding text.
 
     During stabilisation should support a ``_debug`` option to log which
     completion are return by the IPython completer and not found in Jedi in
     order to make upstream bug report.
     """
     warnings.warn("`rectify_completions` is a provisional API (as of IPython 6.0). "
                  "It may change without warnings. "
                  "Use in corresponding context manager.",
                   category=ProvisionalCompleterWarning, stacklevel=2)
 
     completions = list(completions)
     if not completions:
         return
     starts = (c.start for c in completions)
     ends = (c.end for c in completions)
 
     new_start = min(starts)
     new_end = max(ends)
 
     seen_jedi = set()
     seen_python_matches = set()
     for c in completions:
         new_text = text[new_start:c.start] + c.text + text[c.end:new_end]
         if c._origin == 'jedi':
             seen_jedi.add(new_text)
         elif c._origin == 'IPCompleter.python_matches':
             seen_python_matches.add(new_text)
-        yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin)
+        yield Completion(new_start, new_end, new_text, type=c.type, _origin=c._origin, signature=c.signature)
     diff = seen_python_matches.difference(seen_jedi)
     if diff and _debug:
         print('IPython.python matches have extras:', diff)
 
 
 if sys.platform == 'win32':
     DELIMS = ' \t\n`!@#$^&*()=+[{]}|;\'",<>?'
 else:
     DELIMS = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
 
 GREEDY_DELIMS = ' =\r\n'
 
 
 class CompletionSplitter(object):
     """An object to split an input line in a manner similar to readline.
 
     By having our own implementation, we can expose readline-like completion in
     a uniform manner to all frontends.  This object only needs to be given the
     line of text to be split and the cursor position on said line, and it
     returns the 'word' to be completed on at the cursor after splitting the
     entire line.
 
     What characters are used as splitting delimiters can be controlled by
     setting the ``delims`` attribute (this is a property that internally
     automatically builds the necessary regular expression)"""
 
     # Private interface
 
     # A string of delimiter characters.  The default value makes sense for
     # IPython's most typical usage patterns.
     _delims = DELIMS
 
     # The expression (a normal string) to be compiled into a regular expression
     # for actual splitting.  We store it as an attribute mostly for ease of
     # debugging, since this type of code can be so tricky to debug.
     _delim_expr = None
 
     # The regular expression that does the actual splitting
     _delim_re = None
 
     def __init__(self, delims=None):
         delims = CompletionSplitter._delims if delims is None else delims
         self.delims = delims
 
     @property
     def delims(self):
         """Return the string of delimiter characters."""
         return self._delims
 
     @delims.setter
     def delims(self, delims):
         """Set the delimiters for line splitting."""
         expr = '[' + ''.join('\\'+ c for c in delims) + ']'
         self._delim_re = re.compile(expr)
         self._delims = delims
         self._delim_expr = expr
 
     def split_line(self, line, cursor_pos=None):
         """Split a line of text with a cursor at the given position.
         """
         l = line if cursor_pos is None else line[:cursor_pos]
         return self._delim_re.split(l)[-1]
 
 
 
 class Completer(Configurable):
 
     greedy = Bool(False,
         help="""Activate greedy completion
         PENDING DEPRECTION. this is now mostly taken care of with Jedi.
 
         This will enable completion on elements of lists, results of function calls, etc.,
         but can be unsafe because the code is actually evaluated on TAB.
         """
     ).tag(config=True)
 
     use_jedi = Bool(default_value=JEDI_INSTALLED,
                     help="Experimental: Use Jedi to generate autocompletions. "
                     "Default to True if jedi is installed").tag(config=True)
 
     jedi_compute_type_timeout = Int(default_value=400,
         help="""Experimental: restrict time (in milliseconds) during which Jedi can compute types.
         Set to 0 to stop computing types. Non-zero value lower than 100ms may hurt
         performance by preventing jedi to build its cache.
         """).tag(config=True)
 
     debug = Bool(default_value=False,
                  help='Enable debug for the Completer. Mostly print extra '
                       'information for experimental jedi integration.')\
                       .tag(config=True)
 
     backslash_combining_completions = Bool(True, 
         help="Enable unicode completions, e.g. \\alpha<tab> . "
              "Includes completion of latex commands, unicode names, and expanding "
              "unicode characters back to latex commands.").tag(config=True)
 
 
 
     def __init__(self, namespace=None, global_namespace=None, **kwargs):
         """Create a new completer for the command line.
 
         Completer(namespace=ns, global_namespace=ns2) -> completer instance.
 
         If unspecified, the default namespace where completions are performed
         is __main__ (technically, __main__.__dict__). Namespaces should be
         given as dictionaries.
 
         An optional second namespace can be given.  This allows the completer
         to handle cases where both the local and global scopes need to be
         distinguished.
         """
 
         # Don't bind to namespace quite yet, but flag whether the user wants a
         # specific namespace or to use __main__.__dict__. This will allow us
         # to bind to __main__.__dict__ at completion time, not now.
         if namespace is None:
             self.use_main_ns = True
         else:
             self.use_main_ns = False
             self.namespace = namespace
 
         # The global namespace, if given, can be bound directly
         if global_namespace is None:
             self.global_namespace = {}
         else:
             self.global_namespace = global_namespace
 
         super(Completer, self).__init__(**kwargs)
 
     def complete(self, text, state):
         """Return the next possible completion for 'text'.
 
         This is called successively with state == 0, 1, 2, ... until it
         returns None.  The completion should begin with 'text'.
 
         """
         if self.use_main_ns:
             self.namespace = __main__.__dict__
 
         if state == 0:
             if "." in text:
                 self.matches = self.attr_matches(text)
             else:
                 self.matches = self.global_matches(text)
         try:
             return self.matches[state]
         except IndexError:
             return None
 
     def global_matches(self, text):
         """Compute matches when text is a simple name.
 
         Return a list of all keywords, built-in functions and names currently
         defined in self.namespace or self.global_namespace that match.
 
         """
         matches = []
         match_append = matches.append
         n = len(text)
         for lst in [keyword.kwlist,
                     builtin_mod.__dict__.keys(),
                     self.namespace.keys(),
                     self.global_namespace.keys()]:
             for word in lst:
                 if word[:n] == text and word != "__builtins__":
                     match_append(word)
 
         snake_case_re = re.compile(r"[^_]+(_[^_]+)+?\Z")
         for lst in [self.namespace.keys(),
                     self.global_namespace.keys()]:
             shortened = {"_".join([sub[0] for sub in word.split('_')]) : word
                          for word in lst if snake_case_re.match(word)}
             for word in shortened.keys():
                 if word[:n] == text and word != "__builtins__":
                     match_append(shortened[word])
         return matches
 
     def attr_matches(self, text):
         """Compute matches when text contains a dot.
 
         Assuming the text is of the form NAME.NAME....[NAME], and is
         evaluatable in self.namespace or self.global_namespace, it will be
         evaluated and its attributes (as revealed by dir()) are used as
         possible completions.  (For class instances, class members are are
         also considered.)
 
         WARNING: this can still invoke arbitrary C code, if an object
         with a __getattr__ hook is evaluated.
 
         """
 
         # Another option, seems to work great. Catches things like ''.<tab>
         m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
     
         if m:
             expr, attr = m.group(1, 3)
         elif self.greedy:
             m2 = re.match(r"(.+)\.(\w*)$", self.line_buffer)
             if not m2:
                 return []
             expr, attr = m2.group(1,2)
         else:
             return []
     
         try:
             obj = eval(expr, self.namespace)
         except:
             try:
                 obj = eval(expr, self.global_namespace)
             except:
                 return []
 
         if self.limit_to__all__ and hasattr(obj, '__all__'):
             words = get__all__entries(obj)
         else:
             words = dir2(obj)
 
         try:
             words = generics.complete_object(obj, words)
         except TryNext:
             pass
         except AssertionError:
             raise
         except Exception:
             # Silence errors from completion function
             #raise # dbg
             pass
         # Build match list to return
         n = len(attr)
         return [u"%s.%s" % (expr, w) for w in words if w[:n] == attr ]
 
 
 def get__all__entries(obj):
     """returns the strings in the __all__ attribute"""
     try:
         words = getattr(obj, '__all__')
     except:
         return []
  
     return [w for w in words if isinstance(w, str)]
 
 
 def match_dict_keys(keys: List[str], prefix: str, delims: str):
     """Used by dict_key_matches, matching the prefix to a list of keys
 
     Parameters
     ==========
     keys:
         list of keys in dictionary currently being completed.
     prefix:
         Part of the text already typed by the user. e.g. `mydict[b'fo`
     delims:
         String of delimiters to consider when finding the current key.
 
     Returns
     =======
 
     A tuple of three elements: ``quote``, ``token_start``, ``matched``, with
     ``quote`` being the quote that need to be used to close current string.
     ``token_start`` the position where the replacement should start occurring,
     ``matches`` a list of replacement/completion
 
     """
     if not prefix:
         return None, 0, [repr(k) for k in keys
                       if isinstance(k, (str, bytes))]
     quote_match = re.search('["\']', prefix)
     quote = quote_match.group()
     try:
         prefix_str = eval(prefix + quote, {})
     except Exception:
         return None, 0, []
 
     pattern = '[^' + ''.join('\\' + c for c in delims) + ']*$'
     token_match = re.search(pattern, prefix, re.UNICODE)
     token_start = token_match.start()
     token_prefix = token_match.group()
 
     matched = []
     for key in keys:
         try:
             if not key.startswith(prefix_str):
                 continue
         except (AttributeError, TypeError, UnicodeError):
             # Python 3+ TypeError on b'a'.startswith('a') or vice-versa
             continue
 
         # reformat remainder of key to begin with prefix
         rem = key[len(prefix_str):]
         # force repr wrapped in '
         rem_repr = repr(rem + '"') if isinstance(rem, str) else repr(rem + b'"')
         if rem_repr.startswith('u') and prefix[0] not in 'uU':
             # Found key is unicode, but prefix is Py2 string.
             # Therefore attempt to interpret key as string.
             try:
                 rem_repr = repr(rem.encode('ascii') + '"')
             except UnicodeEncodeError:
                 continue
 
         rem_repr = rem_repr[1 + rem_repr.index("'"):-2]
         if quote == '"':
             # The entered prefix is quoted with ",
             # but the match is quoted with '.
             # A contained " hence needs escaping for comparison:
             rem_repr = rem_repr.replace('"', '\\"')
 
         # then reinsert prefix from start of token
         matched.append('%s%s' % (token_prefix, rem_repr))
     return quote, token_start, matched
 
 
 def cursor_to_position(text:str, line:int, column:int)->int:
     """
 
     Convert the (line,column) position of the cursor in text to an offset in a
     string.
 
     Parameters
     ----------
 
     text : str
         The text in which to calculate the cursor offset
     line : int
         Line of the cursor; 0-indexed
     column : int
         Column of the cursor 0-indexed
 
     Return
     ------
         Position of the cursor in ``text``, 0-indexed.
 
     See Also
     --------
     position_to_cursor: reciprocal of this function
 
     """
     lines = text.split('\n')
     assert line <= len(lines), '{} <= {}'.format(str(line), str(len(lines)))
 
     return sum(len(l) + 1 for l in lines[:line]) + column
 
 def position_to_cursor(text:str, offset:int)->Tuple[int, int]:
     """
     Convert the position of the cursor in text (0 indexed) to a line
     number(0-indexed) and a column number (0-indexed) pair
 
     Position should be a valid position in ``text``.
 
     Parameters
     ----------
 
     text : str
         The text in which to calculate the cursor offset
     offset : int
         Position of the cursor in ``text``, 0-indexed.
 
     Return
     ------
     (line, column) : (int, int)
         Line of the cursor; 0-indexed, column of the cursor 0-indexed
 
 
     See Also
     --------
     cursor_to_position : reciprocal of this function
 
 
     """
 
     assert 0 < offset <= len(text) , "0 < %s <= %s" % (offset , len(text))
 
     before = text[:offset]
     blines = before.split('\n')  # ! splitnes trim trailing \n
     line = before.count('\n')
     col = len(blines[-1])
     return line, col
 
 
 def _safe_isinstance(obj, module, class_name):
     """Checks if obj is an instance of module.class_name if loaded
     """
     return (module in sys.modules and
             isinstance(obj, getattr(import_module(module), class_name)))
 
 
 def back_unicode_name_matches(text):
     u"""Match unicode characters back to unicode name
     
     This does  ``☃`` -> ``\\snowman``
 
     Note that snowman is not a valid python3 combining character but will be expanded.
     Though it will not recombine back to the snowman character by the completion machinery.
 
     This will not either back-complete standard sequences like \\n, \\b ...
     
     Used on Python 3 only.
     """
     if len(text)<2:
         return u'', ()
     maybe_slash = text[-2]
     if maybe_slash != '\\':
         return u'', ()
 
     char = text[-1]
     # no expand on quote for completion in strings.
     # nor backcomplete standard ascii keys
     if char in string.ascii_letters or char in ['"',"'"]:
         return u'', ()
     try :
         unic = unicodedata.name(char)
         return '\\'+char,['\\'+unic]
     except KeyError:
         pass
     return u'', ()
 
 def back_latex_name_matches(text:str):
     """Match latex characters back to unicode name
     
     This does ``\\ℵ`` -> ``\\aleph``
 
     Used on Python 3 only.
     """
     if len(text)<2:
         return u'', ()
     maybe_slash = text[-2]
     if maybe_slash != '\\':
         return u'', ()
 
 
     char = text[-1]
     # no expand on quote for completion in strings.
     # nor backcomplete standard ascii keys
     if char in string.ascii_letters or char in ['"',"'"]:
         return u'', ()
     try :
         latex = reverse_latex_symbol[char]
         # '\\' replace the \ as well
         return '\\'+char,[latex]
     except KeyError:
         pass
     return u'', ()
 
 
+def _formatparamchildren(parameter) -> str:
+    """
+    Get parameter name and value from Jedi Private API
+
+    Jedi does not expose a simple way to get `param=value` from its API.
+
+    Prameter
+    ========
+
+    parameter:
+        Jedi's function `Param`
+
+    Returns
+    =======
+
+    A string like 'a', 'b=1', '*args', '**kwargs'
+
+
+    """
+    description = parameter.description
+    if not description.startswith('param '):
+        raise ValueError('Jedi function parameter description have change format.'
+                         'Expected "param ...", found %r".' % description)
+    return description[6:]
+
+def _make_signature(completion)-> str:
+    """
+    Make the signature from a jedi completion
+
+    Parameter
+    =========
+
+    completion: jedi.Completion
+        object does not complete a function type
+
+    Returns
+    =======
+
+    a string consisting of the function signature, with the parenthesis but
+    without the function name. example:
+    `(a, *args, b=1, **kwargs)`
+
+    """
+
+    return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for p in completion.params) if f])
+
 class IPCompleter(Completer):
     """Extension of the completer class with IPython-specific features"""
     
     @observe('greedy')
     def _greedy_changed(self, change):
         """update the splitter and readline delims when greedy is changed"""
         if change['new']:
             self.splitter.delims = GREEDY_DELIMS
         else:
             self.splitter.delims = DELIMS
     
     merge_completions = Bool(True,
         help="""Whether to merge completion results into a single list
         
         If False, only the completion results from the first non-empty
         completer will be returned.
         """
     ).tag(config=True)
     omit__names = Enum((0,1,2), default_value=2,
         help="""Instruct the completer to omit private method names
         
         Specifically, when completing on ``object.<tab>``.
         
         When 2 [default]: all names that start with '_' will be excluded.
         
         When 1: all 'magic' names (``__foo__``) will be excluded.
         
         When 0: nothing will be excluded.
         """
     ).tag(config=True)
     limit_to__all__ = Bool(False,
         help="""
         DEPRECATED as of version 5.0.
         
         Instruct the completer to use __all__ for the completion
         
         Specifically, when completing on ``object.<tab>``.
         
         When True: only those names in obj.__all__ will be included.
         
         When False [default]: the __all__ attribute is ignored
         """,
     ).tag(config=True)
 
     @observe('limit_to__all__')
     def _limit_to_all_changed(self, change):
         warnings.warn('`IPython.core.IPCompleter.limit_to__all__` configuration '
             'value has been deprecated since IPython 5.0, will be made to have '
             'no effects and then removed in future version of IPython.',
             UserWarning)
 
     def __init__(self, shell=None, namespace=None, global_namespace=None,
                  use_readline=_deprecation_readline_sentinel, config=None, **kwargs):
         """IPCompleter() -> completer
 
         Return a completer object.
 
         Parameters
         ----------
 
         shell
             a pointer to the ipython shell itself.  This is needed
             because this completer knows about magic functions, and those can
             only be accessed via the ipython instance.
 
         namespace : dict, optional
             an optional dict where completions are performed.
 
         global_namespace : dict, optional
             secondary optional dict for completions, to
             handle cases (such as IPython embedded inside functions) where
             both Python scopes are visible.
        
         use_readline : bool, optional
             DEPRECATED, ignored since IPython 6.0, will have no effects
         """
 
         self.magic_escape = ESC_MAGIC
         self.splitter = CompletionSplitter()
 
         if use_readline is not _deprecation_readline_sentinel:
             warnings.warn('The `use_readline` parameter is deprecated and ignored since IPython 6.0.',
                           DeprecationWarning, stacklevel=2)
 
         # _greedy_changed() depends on splitter and readline being defined:
         Completer.__init__(self, namespace=namespace, global_namespace=global_namespace,
                             config=config, **kwargs)
 
         # List where completion matches will be stored
         self.matches = []
         self.shell = shell
         # Regexp to split filenames with spaces in them
         self.space_name_re = re.compile(r'([^\\] )')
         # Hold a local ref. to glob.glob for speed
         self.glob = glob.glob
 
         # Determine if we are running on 'dumb' terminals, like (X)Emacs
         # buffers, to avoid completion problems.
         term = os.environ.get('TERM','xterm')
         self.dumb_terminal = term in ['dumb','emacs']
 
         # Special handling of backslashes needed in win32 platforms
         if sys.platform == "win32":
             self.clean_glob = self._clean_glob_win32
         else:
             self.clean_glob = self._clean_glob
 
         #regexp to parse docstring for function signature
         self.docstring_sig_re = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
         self.docstring_kwd_re = re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
         #use this if positional argument name is also needed
         #= re.compile(r'[\s|\[]*(\w+)(?:\s*=?\s*.*)')
 
         # All active matcher routines for completion
         self.matchers = [
                          self.python_matches,
                          self.file_matches,
                          self.magic_matches,
                          self.python_func_kw_matches,
                          self.dict_key_matches,
                          ]
         self.magic_arg_matchers = [
             self.magic_config_matches,
             self.magic_color_matches,
         ]
 
         # This is set externally by InteractiveShell
         self.custom_completers = None
 
     def all_completions(self, text):
         """
         Wrapper around the complete method for the benefit of emacs.
         """
         return self.complete(text)[1]
 
     def _clean_glob(self, text):
         return self.glob("%s*" % text)
 
     def _clean_glob_win32(self,text):
         return [f.replace("\\","/")
                 for f in self.glob("%s*" % text)]
 
     def file_matches(self, text):
         """Match filenames, expanding ~USER type strings.
 
         Most of the seemingly convoluted logic in this completer is an
         attempt to handle filenames with spaces in them.  And yet it's not
         quite perfect, because Python's readline doesn't expose all of the
         GNU readline details needed for this to be done correctly.
 
         For a filename with a space in it, the printed completions will be
         only the parts after what's already been typed (instead of the
         full completions, as is normally done).  I don't think with the
         current (as of Python 2.3) Python readline it's possible to do
         better."""
 
         # chars that require escaping with backslash - i.e. chars
         # that readline treats incorrectly as delimiters, but we
         # don't want to treat as delimiters in filename matching
         # when escaped with backslash
         if text.startswith('!'):
             text = text[1:]
             text_prefix = u'!'
         else:
             text_prefix = u''
 
         text_until_cursor = self.text_until_cursor
         # track strings with open quotes
         open_quotes = has_open_quotes(text_until_cursor)
 
         if '(' in text_until_cursor or '[' in text_until_cursor:
             lsplit = text
         else:
             try:
                 # arg_split ~ shlex.split, but with unicode bugs fixed by us
                 lsplit = arg_split(text_until_cursor)[-1]
             except ValueError:
                 # typically an unmatched ", or backslash without escaped char.
                 if open_quotes:
                     lsplit = text_until_cursor.split(open_quotes)[-1]
                 else:
                     return []
             except IndexError:
                 # tab pressed on empty line
                 lsplit = ""
 
         if not open_quotes and lsplit != protect_filename(lsplit):
             # if protectables are found, do matching on the whole escaped name
             has_protectables = True
             text0,text = text,lsplit
         else:
             has_protectables = False
             text = os.path.expanduser(text)
 
         if text == "":
             return [text_prefix + protect_filename(f) for f in self.glob("*")]
 
         # Compute the matches from the filesystem
         if sys.platform == 'win32':
             m0 = self.clean_glob(text)
         else:
             m0 = self.clean_glob(text.replace('\\', ''))
 
         if has_protectables:
             # If we had protectables, we need to revert our changes to the
             # beginning of filename so that we don't double-write the part
             # of the filename we have so far
             len_lsplit = len(lsplit)
             matches = [text_prefix + text0 +
                        protect_filename(f[len_lsplit:]) for f in m0]
         else:
             if open_quotes:
                 # if we have a string with an open quote, we don't need to
                 # protect the names beyond the quote (and we _shouldn't_, as
                 # it would cause bugs when the filesystem call is made).
                 matches = m0 if sys.platform == "win32" else\
                     [protect_filename(f, open_quotes) for f in m0]
             else:
                 matches = [text_prefix +
                            protect_filename(f) for f in m0]
 
         # Mark directories in input list by appending '/' to their names.
         return [x+'/' if os.path.isdir(x) else x for x in matches]
 
     def magic_matches(self, text):
         """Match magics"""
         # Get all shell magics now rather than statically, so magics loaded at
         # runtime show up too.
         lsm = self.shell.magics_manager.lsmagic()
         line_magics = lsm['line']
         cell_magics = lsm['cell']
         pre = self.magic_escape
         pre2 = pre+pre
         
         # Completion logic:
         # - user gives %%: only do cell magics
         # - user gives %: do both line and cell magics
         # - no prefix: do both
         # In other words, line magics are skipped if the user gives %% explicitly
         #
         # We also exclude magics that match any currently visible names:
         # https://github.com/ipython/ipython/issues/4877
         bare_text = text.lstrip(pre)
         global_matches = self.global_matches(bare_text)
         matches = lambda magic: magic.startswith(bare_text) \
                                 and magic not in global_matches
         comp = [ pre2+m for m in cell_magics if matches(m)]
         if not text.startswith(pre2):
             comp += [ pre+m for m in line_magics if matches(m)]
 
         return comp
 
     def magic_config_matches(self, text:str) -> List[str]:
         """ Match class names and attributes for %config magic """
         texts = text.strip().split()
 
         if len(texts) > 0 and (texts[0] == 'config' or texts[0] == '%config'):
             # get all configuration classes
             classes = sorted(set([ c for c in self.shell.configurables
                                    if c.__class__.class_traits(config=True)
                                    ]), key=lambda x: x.__class__.__name__)
             classnames = [ c.__class__.__name__ for c in classes ]
 
             # return all classnames if config or %config is given
             if len(texts) == 1:
                 return classnames
 
             # match classname
             classname_texts = texts[1].split('.')
             classname = classname_texts[0]
             classname_matches = [ c for c in classnames
                                   if c.startswith(classname) ]
 
             # return matched classes or the matched class with attributes
             if texts[1].find('.') < 0:
                 return classname_matches
             elif len(classname_matches) == 1 and \
                             classname_matches[0] == classname:
                 cls = classes[classnames.index(classname)].__class__
                 help = cls.class_get_help()
                 # strip leading '--' from cl-args:
                 help = re.sub(re.compile(r'^--', re.MULTILINE), '', help)
                 return [ attr.split('=')[0]
                          for attr in help.strip().splitlines()
                          if attr.startswith(texts[1]) ]
         return []
 
     def magic_color_matches(self, text:str) -> List[str] :
         """ Match color schemes for %colors magic"""
         texts = text.strip().split()
 
         if len(texts) > 0 and (texts[0] == 'colors' or texts[0] == '%colors'):
             prefix = texts[1] if len(texts) > 1 else ''
             return [ color for color in InspectColors.keys()
                      if color.startswith(prefix) ]
         return []
 
     def _jedi_matches(self, cursor_column:int, cursor_line:int, text:str):
         """
 
         Return a list of :any:`jedi.api.Completions` object from a ``text`` and
         cursor position.
 
         Parameters
         ----------
         cursor_column : int
             column position of the cursor in ``text``, 0-indexed.
         cursor_line : int
             line position of the cursor in ``text``, 0-indexed
         text : str
             text to complete
 
         Debugging
         ---------
 
         If ``IPCompleter.debug`` is ``True`` may return a :any:`_FakeJediCompletion`
         object containing a string with the Jedi debug information attached.
         """
         namespaces = [self.namespace]
         if self.global_namespace is not None:
             namespaces.append(self.global_namespace)
 
         completion_filter = lambda x:x
         # cursor_pos is an it, jedi wants line and column
         offset = cursor_to_position(text, cursor_line, cursor_column)
         # filter output if we are completing for object members
         if offset:
             pre = text[offset-1]
             if pre == '.':
                 if self.omit__names == 2:
                     completion_filter = lambda c:not c.name.startswith('_')
                 elif self.omit__names == 1:
                     completion_filter = lambda c:not (c.name.startswith('__') and c.name.endswith('__'))
                 elif self.omit__names == 0:
                     completion_filter = lambda x:x
                 else:
                     raise ValueError("Don't understand self.omit__names == {}".format(self.omit__names))
 
         interpreter = jedi.Interpreter(
             text, namespaces, column=cursor_column, line=cursor_line + 1)
 
         try_jedi = False
 
         try:
             # should we check the type of the node is Error ?
             from jedi.parser.tree import ErrorLeaf
             next_to_last_tree = interpreter._get_module().tree_node.children[-2]
             completing_string = False
             if isinstance(next_to_last_tree, ErrorLeaf):
                 completing_string = interpreter._get_module().tree_node.children[-2].value[0] in {'"', "'"}
             # if we are in a string jedi is likely not the right candidate for
             # now. Skip it.
             try_jedi = not completing_string
         except Exception as e:
             # many of things can go wrong, we are using private API just don't crash.
             if self.debug:
                 print("Error detecting if completing a non-finished string :", e, '|')
 
         if not try_jedi:
             return []
         try:
             return filter(completion_filter, interpreter.completions())
         except Exception as e:
             if self.debug:
                 return [_FakeJediCompletion('Oops Jedi has crashed, please report a bug with the following:\n"""\n%s\ns"""' % (e))]
             else:
                 return []
 
     def python_matches(self, text):
         """Match attributes or global python names"""
         if "." in text:
             try:
                 matches = self.attr_matches(text)
                 if text.endswith('.') and self.omit__names:
                     if self.omit__names == 1:
                         # true if txt is _not_ a __ name, false otherwise:
                         no__name = (lambda txt:
                                     re.match(r'.*\.__.*?__',txt) is None)
                     else:
                         # true if txt is _not_ a _ name, false otherwise:
                         no__name = (lambda txt:
                                     re.match(r'\._.*?',txt[txt.rindex('.'):]) is None)
                     matches = filter(no__name, matches)
             except NameError:
                 # catches <undefined attributes>.<tab>
                 matches = []
         else:
             matches = self.global_matches(text)
         return matches
 
     def _default_arguments_from_docstring(self, doc):
         """Parse the first line of docstring for call signature.
 
         Docstring should be of the form 'min(iterable[, key=func])\n'.
         It can also parse cython docstring of the form
         'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)'.
         """
         if doc is None:
             return []
 
         #care only the firstline
         line = doc.lstrip().splitlines()[0]
 
         #p = re.compile(r'^[\w|\s.]+\(([^)]*)\).*')
         #'min(iterable[, key=func])\n' -> 'iterable[, key=func]'
         sig = self.docstring_sig_re.search(line)
         if sig is None:
             return []
         # iterable[, key=func]' -> ['iterable[' ,' key=func]']
         sig = sig.groups()[0].split(',')
         ret = []
         for s in sig:
             #re.compile(r'[\s|\[]*(\w+)(?:\s*=\s*.*)')
             ret += self.docstring_kwd_re.findall(s)
         return ret
 
     def _default_arguments(self, obj):
         """Return the list of default arguments of obj if it is callable,
         or empty list otherwise."""
         call_obj = obj
         ret = []
         if inspect.isbuiltin(obj):
             pass
         elif not (inspect.isfunction(obj) or inspect.ismethod(obj)):
             if inspect.isclass(obj):
                 #for cython embededsignature=True the constructor docstring
                 #belongs to the object itself not __init__
                 ret += self._default_arguments_from_docstring(
                             getattr(obj, '__doc__', ''))
                 # for classes, check for __init__,__new__
                 call_obj = (getattr(obj, '__init__', None) or
                        getattr(obj, '__new__', None))
             # for all others, check if they are __call__able
             elif hasattr(obj, '__call__'):
                 call_obj = obj.__call__
         ret += self._default_arguments_from_docstring(
                  getattr(call_obj, '__doc__', ''))
 
         _keeps = (inspect.Parameter.KEYWORD_ONLY,
                   inspect.Parameter.POSITIONAL_OR_KEYWORD)
 
         try:
             sig = inspect.signature(call_obj)
             ret.extend(k for k, v in sig.parameters.items() if
                        v.kind in _keeps)
         except ValueError:
             pass
 
         return list(set(ret))
 
     def python_func_kw_matches(self,text):
         """Match named parameters (kwargs) of the last open function"""
 
         if "." in text: # a parameter cannot be dotted
             return []
         try: regexp = self.__funcParamsRegex
         except AttributeError:
             regexp = self.__funcParamsRegex = re.compile(r'''
                 '.*?(?<!\\)' |    # single quoted strings or
                 ".*?(?<!\\)" |    # double quoted strings or
                 \w+          |    # identifier
                 \S                # other characters
                 ''', re.VERBOSE | re.DOTALL)
         # 1. find the nearest identifier that comes before an unclosed
         # parenthesis before the cursor
         # e.g. for "foo (1+bar(x), pa<cursor>,a=1)", the candidate is "foo"
         tokens = regexp.findall(self.text_until_cursor)
         iterTokens = reversed(tokens); openPar = 0
 
         for token in iterTokens:
             if token == ')':
                 openPar -= 1
             elif token == '(':
                 openPar += 1
                 if openPar > 0:
                     # found the last unclosed parenthesis
                     break
         else:
             return []
         # 2. Concatenate dotted names ("foo.bar" for "foo.bar(x, pa" )
         ids = []
         isId = re.compile(r'\w+$').match
 
         while True:
             try:
                 ids.append(next(iterTokens))
                 if not isId(ids[-1]):
                     ids.pop(); break
                 if not next(iterTokens) == '.':
                     break
             except StopIteration:
                 break
 
         # Find all named arguments already assigned to, as to avoid suggesting
         # them again
         usedNamedArgs = set()
         par_level = -1
         for token, next_token in zip(tokens, tokens[1:]):
             if token == '(':
                 par_level += 1
             elif token == ')':
                 par_level -= 1
 
             if par_level != 0:
                 continue
 
             if next_token != '=':
                 continue
 
             usedNamedArgs.add(token)
 
         # lookup the candidate callable matches either using global_matches
         # or attr_matches for dotted names
         if len(ids) == 1:
             callableMatches = self.global_matches(ids[0])
         else:
             callableMatches = self.attr_matches('.'.join(ids[::-1]))
         argMatches = []
         for callableMatch in callableMatches:
             try:
                 namedArgs = self._default_arguments(eval(callableMatch,
                                                         self.namespace))
             except:
                 continue
 
             # Remove used named arguments from the list, no need to show twice
             for namedArg in set(namedArgs) - usedNamedArgs:
                 if namedArg.startswith(text):
                     argMatches.append(u"%s=" %namedArg)
         return argMatches
 
     def dict_key_matches(self, text):
         "Match string keys in a dictionary, after e.g. 'foo[' "
         def get_keys(obj):
             # Objects can define their own completions by defining an
             # _ipy_key_completions_() method.
             method = get_real_method(obj, '_ipython_key_completions_')
             if method is not None:
                 return method()
 
             # Special case some common in-memory dict-like types
             if isinstance(obj, dict) or\
                _safe_isinstance(obj, 'pandas', 'DataFrame'):
                 try:
                     return list(obj.keys())
                 except Exception:
                     return []
             elif _safe_isinstance(obj, 'numpy', 'ndarray') or\
                  _safe_isinstance(obj, 'numpy', 'void'):
                 return obj.dtype.names or []
             return []
 
         try:
             regexps = self.__dict_key_regexps
         except AttributeError:
             dict_key_re_fmt = r'''(?x)
             (  # match dict-referring expression wrt greedy setting
                 %s
             )
             \[   # open bracket
             \s*  # and optional whitespace
             ([uUbB]?  # string prefix (r not handled)
                 (?:   # unclosed string
                     '(?:[^']|(?<!\\)\\')*
                 |
                     "(?:[^"]|(?<!\\)\\")*
                 )
             )?
             $
             '''
             regexps = self.__dict_key_regexps = {
                 False: re.compile(dict_key_re_fmt % '''
                                   # identifiers separated by .
                                   (?!\d)\w+
                                   (?:\.(?!\d)\w+)*
                                   '''),
                 True: re.compile(dict_key_re_fmt % '''
                                  .+
                                  ''')
             }
 
         match = regexps[self.greedy].search(self.text_until_cursor)
         if match is None:
             return []
 
         expr, prefix = match.groups()
         try:
             obj = eval(expr, self.namespace)
         except Exception:
             try:
                 obj = eval(expr, self.global_namespace)
             except Exception:
                 return []
 
         keys = get_keys(obj)
         if not keys:
             return keys
         closing_quote, token_offset, matches = match_dict_keys(keys, prefix, self.splitter.delims)
         if not matches:
             return matches
         
         # get the cursor position of
         # - the text being completed
         # - the start of the key text
         # - the start of the completion
         text_start = len(self.text_until_cursor) - len(text)
         if prefix:
             key_start = match.start(2)
             completion_start = key_start + token_offset
         else:
             key_start = completion_start = match.end()
         
         # grab the leading prefix, to make sure all completions start with `text`
         if text_start > key_start:
             leading = ''
         else:
             leading = text[text_start:completion_start]
         
         # the index of the `[` character
         bracket_idx = match.end(1)
 
         # append closing quote and bracket as appropriate
         # this is *not* appropriate if the opening quote or bracket is outside
         # the text given to this method
         suf = ''
         continuation = self.line_buffer[len(self.text_until_cursor):]
         if key_start > text_start and closing_quote:
             # quotes were opened inside text, maybe close them
             if continuation.startswith(closing_quote):
                 continuation = continuation[len(closing_quote):]
             else:
                 suf += closing_quote
         if bracket_idx > text_start:
             # brackets were opened inside text, maybe close them
             if not continuation.startswith(']'):
                 suf += ']'
         
         return [leading + k + suf for k in matches]
 
     def unicode_name_matches(self, text):
         u"""Match Latex-like syntax for unicode characters base
         on the name of the character.
         
         This does  ``\\GREEK SMALL LETTER ETA`` -> ``η``
 
         Works only on valid python 3 identifier, or on combining characters that
         will combine to form a valid identifier.
         
         Used on Python 3 only.
         """
         slashpos = text.rfind('\\')
         if slashpos > -1:
             s = text[slashpos+1:]
             try :
                 unic = unicodedata.lookup(s)
                 # allow combining chars
                 if ('a'+unic).isidentifier():
                     return '\\'+s,[unic]
             except KeyError:
                 pass
         return u'', []
 
 
     def latex_matches(self, text):
         u"""Match Latex syntax for unicode characters.
         
         This does both ``\\alp`` -> ``\\alpha`` and ``\\alpha`` -> ``α``
 
         Used on Python 3 only.
         """
         slashpos = text.rfind('\\')
         if slashpos > -1:
             s = text[slashpos:]
             if s in latex_symbols:
                 # Try to complete a full latex symbol to unicode
                 # \\alpha -> α
                 return s, [latex_symbols[s]]
             else:
                 # If a user has partially typed a latex symbol, give them
                 # a full list of options \al -> [\aleph, \alpha]
                 matches = [k for k in latex_symbols if k.startswith(s)]
                 return s, matches
         return u'', []
 
     def dispatch_custom_completer(self, text):
         if not self.custom_completers:
             return
 
         line = self.line_buffer
         if not line.strip():
             return None
 
         # Create a little structure to pass all the relevant information about
         # the current completion to any custom completer.
         event = SimpleNamespace()
         event.line = line
         event.symbol = text
         cmd = line.split(None,1)[0]
         event.command = cmd
         event.text_until_cursor = self.text_until_cursor
 
         # for foo etc, try also to find completer for %foo
         if not cmd.startswith(self.magic_escape):
             try_magic = self.custom_completers.s_matches(
                 self.magic_escape + cmd)
         else:
             try_magic = []
 
         for c in itertools.chain(self.custom_completers.s_matches(cmd),
                  try_magic,
                  self.custom_completers.flat_matches(self.text_until_cursor)):
             try:
                 res = c(event)
                 if res:
                     # first, try case sensitive match
                     withcase = [r for r in res if r.startswith(text)]
                     if withcase:
                         return withcase
                     # if none, then case insensitive ones are ok too
                     text_low = text.lower()
                     return [r for r in res if r.lower().startswith(text_low)]
             except TryNext:
                 pass
 
         return None
 
     def completions(self, text: str, offset: int)->Iterator[Completion]:
         """
         Returns an iterator over the possible completions
 
         .. warning:: Unstable
             
             This function is unstable, API may change without warning.
             It will also raise unless use in proper context manager.
 
         Parameters
         ----------
         
         text:str
             Full text of the current input, multi line string.
         offset:int
             Integer representing the position of the cursor in ``text``. Offset
             is 0-based indexed.
 
         Yields
         ------
             :any:`Completion` object
 
 
         The cursor on a text can either be seen as being "in between"
         characters or "On" a character depending on the interface visible to
         the user. For consistency the cursor being on "in between" characters X
         and Y is equivalent to the cursor being "on" character Y, that is to say
         the character the cursor is on is considered as being after the cursor.
 
         Combining characters may span more that one position in the
         text.
 
 
         .. note::
 
             If ``IPCompleter.debug`` is :any:`True` will yield a ``--jedi/ipython--``
             fake Completion token to distinguish completion returned by Jedi
             and usual IPython completion.
 
         .. note::
             
             Completions are not completely deduplicated yet. If identical
             completions are coming from different sources this function does not
             ensure that each completion object will only be present once.
         """
         warnings.warn("_complete is a provisional API (as of IPython 6.0). "
                       "It may change without warnings. "
                       "Use in corresponding context manager.",
                       category=ProvisionalCompleterWarning, stacklevel=2)
 
         seen = set()
         for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
             if c and (c in seen):
                 continue
             yield c
             seen.add(c)
 
     def _completions(self, full_text: str, offset: int, *, _timeout)->Iterator[Completion]:
         """
         Core completion module.Same signature as :any:`completions`, with the
         extra `timeout` parameter (in seconds).
 
 
         Computing jedi's completion ``.type`` can be quite expensive (it is a
         lazy property) and can require some warm-up, more warm up than just
         computing the ``name`` of a completion. The warm-up can be :
 
             - Long warm-up the first time a module is encountered after
             install/update: actually build parse/inference tree.
 
             - first time the module is encountered in a session: load tree from
             disk.
 
         We don't want to block completions for tens of seconds so we give the
         completer a "budget" of ``_timeout`` seconds per invocation to compute
         completions types, the completions that have not yet been computed will
         be marked as "unknown" an will have a chance to be computed next round
         are things get cached.
 
         Keep in mind that Jedi is not the only thing treating the completion so
         keep the timeout short-ish as if we take more than 0.3 second we still
         have lots of processing to do.
 
         """
         deadline = time.monotonic() + _timeout
 
 
         before = full_text[:offset]
         cursor_line, cursor_column = position_to_cursor(full_text, offset)
 
         matched_text, matches, matches_origin, jedi_matches = self._complete(
             full_text=full_text, cursor_line=cursor_line, cursor_pos=cursor_column)
 
         iter_jm = iter(jedi_matches)
         if _timeout:
             for jm in iter_jm:
                 try:
                     type_ = jm.type
                 except Exception:
                     if self.debug:
                         print("Error in Jedi getting type of ", jm)
                     type_ = None
                 delta = len(jm.name_with_symbols) - len(jm.complete)
+                if type_ == 'function':
+                    signature = _make_signature(jm)
+                else:
+                    signature = ''
                 yield Completion(start=offset - delta,
                                  end=offset,
                                  text=jm.name_with_symbols,
                                  type=type_,
+                                 signature=signature,
                                  _origin='jedi')
 
                 if time.monotonic() > deadline:
                     break
 
         for jm in iter_jm:
             delta = len(jm.name_with_symbols) - len(jm.complete)
             yield Completion(start=offset - delta,
                              end=offset,
                              text=jm.name_with_symbols,
                              type='<unknown>',  # don't compute type for speed
-                             _origin='jedi')
+                             _origin='jedi',
+                             signature='')
 
 
         start_offset = before.rfind(matched_text)
 
         # TODO:
         # Supress this, right now just for debug.
         if jedi_matches and matches and self.debug:
-            yield Completion(start=start_offset, end=offset, text='--jedi/ipython--', _origin='debug')
+            yield Completion(start=start_offset, end=offset, text='--jedi/ipython--',
+                             _origin='debug', type='none', signature='')
 
         # I'm unsure if this is always true, so let's assert and see if it
         # crash
         assert before.endswith(matched_text)
         for m, t in zip(matches, matches_origin):
-            yield Completion(start=start_offset, end=offset, text=m, _origin=t)
+            yield Completion(start=start_offset, end=offset, text=m, _origin=t, signature='', type='<unknown>')
 
 
     def complete(self, text=None, line_buffer=None, cursor_pos=None):
         """Find completions for the given text and line context.
 
         Note that both the text and the line_buffer are optional, but at least
         one of them must be given.
 
         Parameters
         ----------
           text : string, optional
             Text to perform the completion on.  If not given, the line buffer
             is split using the instance's CompletionSplitter object.
 
           line_buffer : string, optional
             If not given, the completer attempts to obtain the current line
             buffer via readline.  This keyword allows clients which are
             requesting for text completions in non-readline contexts to inform
             the completer of the entire text.
 
           cursor_pos : int, optional
             Index of the cursor in the full line buffer.  Should be provided by
             remote frontends where kernel has no access to frontend state.
 
         Returns
         -------
         text : str
           Text that was actually used in the completion.
 
         matches : list
           A list of completion matches.
 
 
         .. note::
 
             This API is likely to be deprecated and replaced by
             :any:`IPCompleter.completions` in the future.
 
 
         """
         warnings.warn('`Completer.complete` is pending deprecation since '
                 'IPython 6.0 and will be replaced by `Completer.completions`.',
                       PendingDeprecationWarning)
         # potential todo, FOLD the 3rd throw away argument of _complete
         # into the first 2 one.
         return self._complete(line_buffer=line_buffer, cursor_pos=cursor_pos, text=text, cursor_line=0)[:2]
 
     def _complete(self, *, cursor_line, cursor_pos, line_buffer=None, text=None,
                   full_text=None, return_jedi_results=True) -> Tuple[str, List[str], List[str], Iterable[_FakeJediCompletion]]:
         """
 
         Like complete but can also returns raw jedi completions as well as the
         origin of the completion text. This could (and should) be made much
         cleaner but that will be simpler once we drop the old (and stateful)
         :any:`complete` API.
 
 
         With current provisional API, cursor_pos act both (depending on the
         caller) as the offset in the ``text`` or ``line_buffer``, or as the
         ``column`` when passing multiline strings this could/should be renamed
         but would add extra noise.
         """
 
         # if the cursor position isn't given, the only sane assumption we can
         # make is that it's at the end of the line (the common case)
         if cursor_pos is None:
             cursor_pos = len(line_buffer) if text is None else len(text)
 
         if self.use_main_ns:
             self.namespace = __main__.__dict__
 
         # if text is either None or an empty string, rely on the line buffer
         if (not line_buffer) and full_text:
             line_buffer = full_text.split('\n')[cursor_line]
         if not text:
             text = self.splitter.split_line(line_buffer, cursor_pos)
 
         if self.backslash_combining_completions:
             # allow deactivation of these on windows.
             base_text = text if not line_buffer else line_buffer[:cursor_pos]
             latex_text, latex_matches = self.latex_matches(base_text)
             if latex_matches:
                 return latex_text, latex_matches, ['latex_matches']*len(latex_matches), ()
             name_text = ''
             name_matches = []
             for meth in (self.unicode_name_matches, back_latex_name_matches, back_unicode_name_matches):
                 name_text, name_matches = meth(base_text)
                 if name_text:
                     return name_text, name_matches, [meth.__qualname__]*len(name_matches), ()
         
 
         # If no line buffer is given, assume the input text is all there was
         if line_buffer is None:
             line_buffer = text
 
         self.line_buffer = line_buffer
         self.text_until_cursor = self.line_buffer[:cursor_pos]
 
         # Do magic arg matches
         for matcher in self.magic_arg_matchers:
             matches = [(m, matcher.__qualname__) for m in matcher(line_buffer)]
             if matches:
                 matches2 = [m[0] for m in matches]
                 origins = [m[1] for m in matches]
                 return text, matches2, origins, ()
 
         # Start with a clean slate of completions
         matches = []
         custom_res = self.dispatch_custom_completer(text)
         # FIXME: we should extend our api to return a dict with completions for
         # different types of objects.  The rlcomplete() method could then
         # simply collapse the dict into a list for readline, but we'd have
         # richer completion semantics in other evironments.
         completions = ()
         if self.use_jedi and return_jedi_results:
             if not full_text:
                 full_text = line_buffer
             completions = self._jedi_matches(
                 cursor_pos, cursor_line, full_text)
         if custom_res is not None:
             # did custom completers produce something?
             matches = [(m, 'custom') for m in custom_res]
         else:
             # Extend the list of completions with the results of each
             # matcher, so we return results to the user from all
             # namespaces.
             if self.merge_completions:
                 matches = []
                 for matcher in self.matchers:
                     try:
                         matches.extend([(m, matcher.__qualname__)
                                         for m in matcher(text)])
                     except:
                         # Show the ugly traceback if the matcher causes an
                         # exception, but do NOT crash the kernel!
                         sys.excepthook(*sys.exc_info())
             else:
                 for matcher in self.matchers:
                     matches = [(m, matcher.__qualname__)
                                for m in matcher(text)]
                     if matches:
                         break
         seen = set()
         filtered_matches = set()
         for m in matches:
             t, c = m
             if t not in seen:
                 filtered_matches.add(m)
                 seen.add(t)
 
         _filtered_matches = sorted(
             set(filtered_matches), key=lambda x: completions_sorting_key(x[0]))
 
         _matches = [m[0] for m in _filtered_matches]
         origins = [m[1] for m in _filtered_matches]
 
         self.matches = _matches
 
         return text, _matches, origins, completions

IPython/core/completerlib.py

@@ -1,352 +1,354 b''
 # encoding: utf-8
 """Implementations for various useful completers.
 
 These are all loaded by default by IPython.
 """
 #-----------------------------------------------------------------------------
 #  Copyright (C) 2010-2011 The IPython Development Team.
 #
 #  Distributed under the terms of the BSD License.
 #
 #  The full license is in the file COPYING.txt, distributed with this software.
 #-----------------------------------------------------------------------------
 
 #-----------------------------------------------------------------------------
 # Imports
 #-----------------------------------------------------------------------------
 
 # Stdlib imports
 import glob
 import inspect
 import os
 import re
 import sys
 from importlib import import_module
 from importlib.machinery import all_suffixes
 
 
 # Third-party imports
 from time import time
 from zipimport import zipimporter
 
 # Our own imports
 from IPython.core.completer import expand_user, compress_user
 from IPython.core.error import TryNext
 from IPython.utils._process_common import arg_split
 
 # FIXME: this should be pulled in with the right call via the component system
 from IPython import get_ipython
 
+from typing import List
+
 #-----------------------------------------------------------------------------
 # Globals and constants
 #-----------------------------------------------------------------------------
 _suffixes = all_suffixes()
 
 # Time in seconds after which the rootmodules will be stored permanently in the
 # ipython ip.db database (kept in the user's .ipython dir).
 TIMEOUT_STORAGE = 2
 
 # Time in seconds after which we give up
 TIMEOUT_GIVEUP = 20
 
 # Regular expression for the python import statement
 import_re = re.compile(r'(?P<name>[a-zA-Z_][a-zA-Z0-9_]*?)'
                        r'(?P<package>[/\\]__init__)?'
                        r'(?P<suffix>%s)$' %
                        r'|'.join(re.escape(s) for s in _suffixes))
 
 # RE for the ipython %run command (python + ipython scripts)
 magic_run_re = re.compile(r'.*(\.ipy|\.ipynb|\.py[w]?)$')
 
 #-----------------------------------------------------------------------------
 # Local utilities
 #-----------------------------------------------------------------------------
 
 def module_list(path):
     """
     Return the list containing the names of the modules available in the given
     folder.
     """
     # sys.path has the cwd as an empty string, but isdir/listdir need it as '.'
     if path == '':
         path = '.'
 
     # A few local constants to be used in loops below
     pjoin = os.path.join
 
     if os.path.isdir(path):
         # Build a list of all files in the directory and all files
         # in its subdirectories. For performance reasons, do not
         # recurse more than one level into subdirectories.
         files = []
         for root, dirs, nondirs in os.walk(path, followlinks=True):
             subdir = root[len(path)+1:]
             if subdir:
                 files.extend(pjoin(subdir, f) for f in nondirs)
                 dirs[:] = [] # Do not recurse into additional subdirectories.
             else:
                 files.extend(nondirs)
 
     else:
         try:
             files = list(zipimporter(path)._files.keys())
         except:
             files = []
 
     # Build a list of modules which match the import_re regex.
     modules = []
     for f in files:
         m = import_re.match(f)
         if m:
             modules.append(m.group('name'))
     return list(set(modules))
 
 
 def get_root_modules():
     """
     Returns a list containing the names of all the modules available in the
     folders of the pythonpath.
 
     ip.db['rootmodules_cache'] maps sys.path entries to list of modules.
     """
     ip = get_ipython()
     if ip is None:
         # No global shell instance to store cached list of modules.
         # Don't try to scan for modules every time.
         return list(sys.builtin_module_names)
 
     rootmodules_cache = ip.db.get('rootmodules_cache', {})
     rootmodules = list(sys.builtin_module_names)
     start_time = time()
     store = False
     for path in sys.path:
         try:
             modules = rootmodules_cache[path]
         except KeyError:
             modules = module_list(path)
             try:
                 modules.remove('__init__')
             except ValueError:
                 pass
             if path not in ('', '.'): # cwd modules should not be cached
                 rootmodules_cache[path] = modules
             if time() - start_time > TIMEOUT_STORAGE and not store:
                 store = True
                 print("\nCaching the list of root modules, please wait!")
                 print("(This will only be done once - type '%rehashx' to "
                       "reset cache!)\n")
                 sys.stdout.flush()
             if time() - start_time > TIMEOUT_GIVEUP:
                 print("This is taking too long, we give up.\n")
                 return []
         rootmodules.extend(modules)
     if store:
         ip.db['rootmodules_cache'] = rootmodules_cache
     rootmodules = list(set(rootmodules))
     return rootmodules
 
 
 def is_importable(module, attr, only_modules):
     if only_modules:
         return inspect.ismodule(getattr(module, attr))
     else:
         return not(attr[:2] == '__' and attr[-2:] == '__')
 
 
-def try_import(mod: str, only_modules=False):
+def try_import(mod: str, only_modules=False) -> List[str]:
     """
     Try to import given module and return list of potential completions.
     """
     mod = mod.rstrip('.')
     try:
         m = import_module(mod)
     except:
         return []
 
     m_is_init = hasattr(m, '__file__') and '__init__' in m.__file__
 
     completions = []
     if (not hasattr(m, '__file__')) or (not only_modules) or m_is_init:
         completions.extend( [attr for attr in dir(m) if
                              is_importable(m, attr, only_modules)])
 
     completions.extend(getattr(m, '__all__', []))
     if m_is_init:
         completions.extend(module_list(os.path.dirname(m.__file__)))
-    completions = {c for c in completions if isinstance(c, str)}
-    completions.discard('__init__')
-    return list(completions)
+    completions_set = {c for c in completions if isinstance(c, str)}
+    completions_set.discard('__init__')
+    return list(completions_set)
 
 
 #-----------------------------------------------------------------------------
 # Completion-related functions.
 #-----------------------------------------------------------------------------
 
 def quick_completer(cmd, completions):
     """ Easily create a trivial completer for a command.
 
     Takes either a list of completions, or all completions in string (that will
     be split on whitespace).
 
     Example::
 
         [d:\ipython]|1> import ipy_completers
         [d:\ipython]|2> ipy_completers.quick_completer('foo', ['bar','baz'])
         [d:\ipython]|3> foo b<TAB>
         bar baz
         [d:\ipython]|3> foo ba
     """
 
     if isinstance(completions, str):
         completions = completions.split()
 
     def do_complete(self, event):
         return completions
 
     get_ipython().set_hook('complete_command',do_complete, str_key = cmd)
 
 def module_completion(line):
     """
     Returns a list containing the completion possibilities for an import line.
 
     The line looks like this :
     'import xml.d'
     'from xml.dom import'
     """
 
     words = line.split(' ')
     nwords = len(words)
 
     # from whatever <tab> -> 'import '
     if nwords == 3 and words[0] == 'from':
         return ['import ']
 
     # 'from xy<tab>' or 'import xy<tab>'
     if nwords < 3 and (words[0] in {'%aimport', 'import', 'from'}) :
         if nwords == 1:
             return get_root_modules()
         mod = words[1].split('.')
         if len(mod) < 2:
             return get_root_modules()
         completion_list = try_import('.'.join(mod[:-1]), True)
         return ['.'.join(mod[:-1] + [el]) for el in completion_list]
 
     # 'from xyz import abc<tab>'
     if nwords >= 3 and words[0] == 'from':
         mod = words[1]
         return try_import(mod)
 
 #-----------------------------------------------------------------------------
 # Completers
 #-----------------------------------------------------------------------------
 # These all have the func(self, event) signature to be used as custom
 # completers
 
 def module_completer(self,event):
     """Give completions after user has typed 'import ...' or 'from ...'"""
 
     # This works in all versions of python.  While 2.5 has
     # pkgutil.walk_packages(), that particular routine is fairly dangerous,
     # since it imports *EVERYTHING* on sys.path.  That is: a) very slow b) full
     # of possibly problematic side effects.
     # This search the folders in the sys.path for available modules.
 
     return module_completion(event.line)
 
 # FIXME: there's a lot of logic common to the run, cd and builtin file
 # completers, that is currently reimplemented in each.
 
 def magic_run_completer(self, event):
     """Complete files that end in .py or .ipy or .ipynb for the %run command.
     """
     comps = arg_split(event.line, strict=False)
     # relpath should be the current token that we need to complete.
     if (len(comps) > 1) and (not event.line.endswith(' ')):
         relpath = comps[-1].strip("'\"")
     else:
         relpath = ''
 
     #print("\nev=", event)  # dbg
     #print("rp=", relpath)  # dbg
     #print('comps=', comps)  # dbg
 
     lglob = glob.glob
     isdir = os.path.isdir
     relpath, tilde_expand, tilde_val = expand_user(relpath)
 
     # Find if the user has already typed the first filename, after which we
     # should complete on all files, since after the first one other files may
     # be arguments to the input script.
 
     if any(magic_run_re.match(c) for c in comps):
         matches =  [f.replace('\\','/') + ('/' if isdir(f) else '')
                             for f in lglob(relpath+'*')]
     else:
         dirs = [f.replace('\\','/') + "/" for f in lglob(relpath+'*') if isdir(f)]
         pys =  [f.replace('\\','/')
                 for f in lglob(relpath+'*.py') + lglob(relpath+'*.ipy') +
                 lglob(relpath+'*.ipynb') + lglob(relpath + '*.pyw')]
 
         matches = dirs + pys
 
     #print('run comp:', dirs+pys) # dbg
     return [compress_user(p, tilde_expand, tilde_val) for p in matches]
 
 
 def cd_completer(self, event):
     """Completer function for cd, which only returns directories."""
     ip = get_ipython()
     relpath = event.symbol
 
     #print(event) # dbg
     if event.line.endswith('-b') or ' -b ' in event.line:
         # return only bookmark completions
         bkms = self.db.get('bookmarks', None)
         if bkms:
             return bkms.keys()
         else:
             return []
 
     if event.symbol == '-':
         width_dh = str(len(str(len(ip.user_ns['_dh']) + 1)))
         # jump in directory history by number
         fmt = '-%0' + width_dh +'d [%s]'
         ents = [ fmt % (i,s) for i,s in enumerate(ip.user_ns['_dh'])]
         if len(ents) > 1:
             return ents
         return []
 
     if event.symbol.startswith('--'):
         return ["--" + os.path.basename(d) for d in ip.user_ns['_dh']]
 
     # Expand ~ in path and normalize directory separators.
     relpath, tilde_expand, tilde_val = expand_user(relpath)
     relpath = relpath.replace('\\','/')
 
     found = []
     for d in [f.replace('\\','/') + '/' for f in glob.glob(relpath+'*')
               if os.path.isdir(f)]:
         if ' ' in d:
             # we don't want to deal with any of that, complex code
             # for this is elsewhere
             raise TryNext
 
         found.append(d)
 
     if not found:
         if os.path.isdir(relpath):
             return [compress_user(relpath, tilde_expand, tilde_val)]
 
         # if no completions so far, try bookmarks
         bks = self.db.get('bookmarks',{})
         bkmatches = [s for s in bks if s.startswith(event.symbol)]
         if bkmatches:
             return bkmatches
 
         raise TryNext
 
     return [compress_user(p, tilde_expand, tilde_val) for p in found]
 
 def reset_completer(self, event):
     "A completer for %reset magic"
     return '-f -s in out array dhist'.split()

IPython/core/tests/test_completer.py

@@ -1,949 +1,961 b''
 # encoding: utf-8
 """Tests for the IPython tab-completion machinery."""
 
 # Copyright (c) IPython Development Team.
 # Distributed under the terms of the Modified BSD License.
 
 import os
 import sys
 import textwrap
 import unittest
 
 from contextlib import contextmanager
 
 import nose.tools as nt
 
 from traitlets.config.loader import Config
 from IPython import get_ipython
 from IPython.core import completer
 from IPython.external.decorators import knownfailureif
 from IPython.utils.tempdir import TemporaryDirectory, TemporaryWorkingDirectory
 from IPython.utils.generics import complete_object
 from IPython.testing import decorators as dec
 
 from IPython.core.completer import (
     Completion, provisionalcompleter, match_dict_keys, _deduplicate_completions)
 from nose.tools import assert_in, assert_not_in
 
 #-----------------------------------------------------------------------------
 # Test functions
 #-----------------------------------------------------------------------------
 
 @contextmanager
 def greedy_completion():
     ip = get_ipython()
     greedy_original = ip.Completer.greedy
     try:
         ip.Completer.greedy = True
         yield
     finally:
         ip.Completer.greedy = greedy_original
 
 def test_protect_filename():
     if sys.platform == 'win32':
         pairs = [('abc','abc'),
                  (' abc','" abc"'),
                  ('a bc','"a bc"'),
                  ('a  bc','"a  bc"'),
                  ('  bc','"  bc"'),
                  ]
     else:
         pairs = [('abc','abc'),
                  (' abc',r'\ abc'),
                  ('a bc',r'a\ bc'),
                  ('a  bc',r'a\ \ bc'),
                  ('  bc',r'\ \ bc'),
                  # On posix, we also protect parens and other special characters.
                  ('a(bc',r'a\(bc'),
                  ('a)bc',r'a\)bc'),
                  ('a( )bc',r'a\(\ \)bc'),
                  ('a[1]bc', r'a\[1\]bc'),
                  ('a{1}bc', r'a\{1\}bc'),
                  ('a#bc', r'a\#bc'),
                  ('a?bc', r'a\?bc'),
                  ('a=bc', r'a\=bc'),
                  ('a\\bc', r'a\\bc'),
                  ('a|bc', r'a\|bc'),
                  ('a;bc', r'a\;bc'),
                  ('a:bc', r'a\:bc'),
                  ("a'bc", r"a\'bc"),
                  ('a*bc', r'a\*bc'),
                  ('a"bc', r'a\"bc'),
                  ('a^bc', r'a\^bc'),
                  ('a&bc', r'a\&bc'),
                  ]
     # run the actual tests
     for s1, s2 in pairs:
         s1p = completer.protect_filename(s1)
         nt.assert_equal(s1p, s2)
 
 
 def check_line_split(splitter, test_specs):
     for part1, part2, split in test_specs:
         cursor_pos = len(part1)
         line = part1+part2
         out = splitter.split_line(line, cursor_pos)
         nt.assert_equal(out, split)
 
 
 def test_line_split():
     """Basic line splitter test with default specs."""
     sp = completer.CompletionSplitter()
     # The format of the test specs is: part1, part2, expected answer.  Parts 1
     # and 2 are joined into the 'line' sent to the splitter, as if the cursor
     # was at the end of part1.  So an empty part2 represents someone hitting
     # tab at the end of the line, the most common case.
     t = [('run some/scrip', '', 'some/scrip'),
          ('run scripts/er', 'ror.py foo', 'scripts/er'),
          ('echo $HOM', '', 'HOM'),
          ('print sys.pa', '', 'sys.pa'),
          ('print(sys.pa', '', 'sys.pa'),
          ("execfile('scripts/er", '', 'scripts/er'),
          ('a[x.', '', 'x.'),
          ('a[x.', 'y', 'x.'),
          ('cd "some_file/', '', 'some_file/'),
          ]
     check_line_split(sp, t)
     # Ensure splitting works OK with unicode by re-running the tests with
     # all inputs turned into unicode
     check_line_split(sp, [ map(str, p) for p in t] )
 
 
 def test_custom_completion_error():
     """Test that errors from custom attribute completers are silenced."""
     ip = get_ipython()
     class A(object): pass
     ip.user_ns['a'] = A()
     
     @complete_object.when_type(A)
     def complete_A(a, existing_completions):
         raise TypeError("this should be silenced")
     
     ip.complete("a.")
 
 
 def test_unicode_completions():
     ip = get_ipython()
     # Some strings that trigger different types of completion.  Check them both
     # in str and unicode forms
     s = ['ru', '%ru', 'cd /', 'floa', 'float(x)/']
     for t in s + list(map(str, s)):
         # We don't need to check exact completion values (they may change
         # depending on the state of the namespace, but at least no exceptions
         # should be thrown and the return value should be a pair of text, list
         # values.
         text, matches = ip.complete(t)
         nt.assert_true(isinstance(text, str))
         nt.assert_true(isinstance(matches, list))
 
 def test_latex_completions():
     from IPython.core.latex_symbols import latex_symbols
     import random
     ip = get_ipython()
     # Test some random unicode symbols
     keys = random.sample(latex_symbols.keys(), 10)
     for k in keys:
         text, matches = ip.complete(k)
         nt.assert_equal(len(matches),1)
         nt.assert_equal(text, k)
         nt.assert_equal(matches[0], latex_symbols[k])
     # Test a more complex line
     text, matches = ip.complete(u'print(\\alpha')
     nt.assert_equal(text, u'\\alpha')
     nt.assert_equal(matches[0], latex_symbols['\\alpha'])
     # Test multiple matching latex symbols
     text, matches = ip.complete(u'\\al')
     nt.assert_in('\\alpha', matches)
     nt.assert_in('\\aleph', matches)
 
 
 
 
 def test_back_latex_completion():
     ip = get_ipython()
 
     # do not return more than 1 matches fro \beta, only the latex one.
     name, matches = ip.complete('\\β')
     nt.assert_equal(len(matches), 1)
     nt.assert_equal(matches[0], '\\beta')
 
 def test_back_unicode_completion():
     ip = get_ipython()
     
     name, matches = ip.complete('\\Ⅴ')
     nt.assert_equal(len(matches), 1)
     nt.assert_equal(matches[0], '\\ROMAN NUMERAL FIVE')
 
 
 def test_forward_unicode_completion():
     ip = get_ipython()
     
     name, matches = ip.complete('\\ROMAN NUMERAL FIVE')
     nt.assert_equal(len(matches), 1)
     nt.assert_equal(matches[0], 'Ⅴ')
 
 @dec.knownfailureif(sys.platform == 'win32', 'Fails if there is a C:\\j... path')
 def test_no_ascii_back_completion():
     ip = get_ipython()
     with TemporaryWorkingDirectory():  # Avoid any filename completions
         # single ascii letter that don't have yet completions
         for letter in 'jJ' :
             name, matches = ip.complete('\\'+letter)
             nt.assert_equal(matches, [])
 
 
 
 
 class CompletionSplitterTestCase(unittest.TestCase):
     def setUp(self):
         self.sp = completer.CompletionSplitter()
 
     def test_delim_setting(self):
         self.sp.delims = ' '
         nt.assert_equal(self.sp.delims, ' ')
         nt.assert_equal(self.sp._delim_expr, '[\ ]')
 
     def test_spaces(self):
         """Test with only spaces as split chars."""
         self.sp.delims = ' '
         t = [('foo', '', 'foo'),
              ('run foo', '', 'foo'),
              ('run foo', 'bar', 'foo'),
              ]
         check_line_split(self.sp, t)
 
 
 def test_has_open_quotes1():
     for s in ["'", "'''", "'hi' '"]:
         nt.assert_equal(completer.has_open_quotes(s), "'")
 
 
 def test_has_open_quotes2():
     for s in ['"', '"""', '"hi" "']:
         nt.assert_equal(completer.has_open_quotes(s), '"')
 
 
 def test_has_open_quotes3():
     for s in ["''", "''' '''", "'hi' 'ipython'"]:
         nt.assert_false(completer.has_open_quotes(s))
 
 
 def test_has_open_quotes4():
     for s in ['""', '""" """', '"hi" "ipython"']:
         nt.assert_false(completer.has_open_quotes(s))
 
 
 @knownfailureif(sys.platform == 'win32', "abspath completions fail on Windows")
 def test_abspath_file_completions():
     ip = get_ipython()
     with TemporaryDirectory() as tmpdir:
         prefix = os.path.join(tmpdir, 'foo')
         suffixes = ['1', '2']
         names = [prefix+s for s in suffixes]
         for n in names:
             open(n, 'w').close()
 
         # Check simple completion
         c = ip.complete(prefix)[1]
         nt.assert_equal(c, names)
 
         # Now check with a function call
         cmd = 'a = f("%s' % prefix
         c = ip.complete(prefix, cmd)[1]
         comp = [prefix+s for s in suffixes]
         nt.assert_equal(c, comp)
 
 
 def test_local_file_completions():
     ip = get_ipython()
     with TemporaryWorkingDirectory():
         prefix = './foo'
         suffixes = ['1', '2']
         names = [prefix+s for s in suffixes]
         for n in names:
             open(n, 'w').close()
 
         # Check simple completion
         c = ip.complete(prefix)[1]
         nt.assert_equal(c, names)
 
         # Now check with a function call
         cmd = 'a = f("%s' % prefix
         c = ip.complete(prefix, cmd)[1]
         comp = set(prefix+s for s in suffixes)
         nt.assert_true(comp.issubset(set(c)))
 
 
 def test_quoted_file_completions():
     ip = get_ipython()
     with TemporaryWorkingDirectory():
         name = "foo'bar"
         open(name, 'w').close()
 
         # Don't escape Windows
         escaped = name if sys.platform == "win32" else "foo\\'bar"
 
         # Single quote matches embedded single quote
         text = "open('foo"
         c = ip.Completer._complete(cursor_line=0,
                                    cursor_pos=len(text),
                                    full_text=text)[1]
         nt.assert_equal(c, [escaped])
 
         # Double quote requires no escape
         text = 'open("foo'
         c = ip.Completer._complete(cursor_line=0,
                                    cursor_pos=len(text),
                                    full_text=text)[1]
         nt.assert_equal(c, [name])
 
         # No quote requires an escape
         text = '%ls foo'
         c = ip.Completer._complete(cursor_line=0,
                                    cursor_pos=len(text),
                                    full_text=text)[1]
         nt.assert_equal(c, [escaped])
 
 
 def test_jedi():
     """
     A couple of issue we had with Jedi
     """
     ip = get_ipython()
 
     def _test_complete(reason, s, comp, start=None, end=None):
         l = len(s)
         start = start if start is not None else l
         end = end if end is not None else l
         with provisionalcompleter():
             completions = set(ip.Completer.completions(s, l))
             assert_in(Completion(start, end, comp), completions, reason)
 
     def _test_not_complete(reason, s, comp):
         l = len(s)
         with provisionalcompleter():
             completions = set(ip.Completer.completions(s, l))
             assert_not_in(Completion(l, l, comp), completions, reason)
 
     import jedi
     jedi_version = tuple(int(i) for i in jedi.__version__.split('.')[:3])
     if jedi_version > (0, 10):
         yield _test_complete, 'jedi >0.9 should complete and not crash', 'a=1;a.', 'real'
     yield _test_complete, 'can infer first argument', 'a=(1,"foo");a[0].', 'real'
     yield _test_complete, 'can infer second argument', 'a=(1,"foo");a[1].', 'capitalize'
     yield _test_complete, 'cover duplicate completions', 'im', 'import', 0, 2
 
     yield _test_not_complete, 'does not mix types', 'a=(1,"foo");a[0].', 'capitalize'
 
+def test_completion_have_signature():
+    """
+    Lets make sure jedi is capable of pulling out the signature of the function we are completing.
+    """
+    ip = get_ipython()
+    with provisionalcompleter():
+        completions = ip.Completer.completions('ope', 3)
+        c = next(completions)  # should be `open`
+    assert 'file' in c.signature, "Signature of function was not found by completer"
+    assert 'encoding' in c.signature, "Signature of function was not found by completer"
+
+
 def test_deduplicate_completions():
     """
     Test that completions are correctly deduplicated (even if ranges are not the same)
     """
     ip = get_ipython()
     ip.ex(textwrap.dedent('''
     class Z:
         zoo = 1
     '''))
     with provisionalcompleter():
         l = list(_deduplicate_completions('Z.z', ip.Completer.completions('Z.z', 3)))
 
     assert len(l) == 1, 'Completions (Z.z<tab>) correctly deduplicate: %s ' % l
     assert l[0].text == 'zoo'  # and not `it.accumulate`
 
 
 def test_greedy_completions():
     """
     Test the capability of the Greedy completer. 
 
     Most of the test here do not really show off the greedy completer, for proof
     each of the text bellow now pass with Jedi. The greedy completer is capable of more. 
 
     See the :any:`test_dict_key_completion_contexts`
 
     """
     ip = get_ipython()
     ip.ex('a=list(range(5))')
     _,c = ip.complete('.',line='a[0].')
     nt.assert_false('.real' in c,
                     "Shouldn't have completed on a[0]: %s"%c)
     with greedy_completion(), provisionalcompleter():
         def _(line, cursor_pos, expect, message, completion):
             _,c = ip.complete('.', line=line, cursor_pos=cursor_pos)
             with provisionalcompleter():
                 completions = ip.Completer.completions(line, cursor_pos)
             nt.assert_in(expect, c, message%c)
             nt.assert_in(completion, completions)
 
         yield _, 'a[0].', 5, 'a[0].real', "Should have completed on a[0].: %s", Completion(5,5, 'real')
         yield _, 'a[0].r', 6, 'a[0].real', "Should have completed on a[0].r: %s", Completion(5,6, 'real')
 
         if sys.version_info > (3, 4):
             yield _, 'a[0].from_', 10, 'a[0].from_bytes', "Should have completed on a[0].from_: %s", Completion(5, 10, 'from_bytes')
 
 
 def test_omit__names():
     # also happens to test IPCompleter as a configurable
     ip = get_ipython()
     ip._hidden_attr = 1
     ip._x = {}
     c = ip.Completer
     ip.ex('ip=get_ipython()')
     cfg = Config()
     cfg.IPCompleter.omit__names = 0
     c.update_config(cfg)
     with provisionalcompleter():
         s,matches = c.complete('ip.')
         completions = set(c.completions('ip.', 3))
 
         nt.assert_in('ip.__str__', matches)
         nt.assert_in(Completion(3, 3, '__str__'), completions)
         
         nt.assert_in('ip._hidden_attr', matches)
         nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
 
 
     cfg = Config()
     cfg.IPCompleter.omit__names = 1
     c.update_config(cfg)
     with provisionalcompleter():
         s,matches = c.complete('ip.')
         completions = set(c.completions('ip.', 3))
 
         nt.assert_not_in('ip.__str__', matches)
         nt.assert_not_in(Completion(3,3,'__str__'), completions)
 
         # nt.assert_in('ip._hidden_attr', matches)
         nt.assert_in(Completion(3,3, "_hidden_attr"), completions)
 
     cfg = Config()
     cfg.IPCompleter.omit__names = 2
     c.update_config(cfg)
     with provisionalcompleter():
         s,matches = c.complete('ip.')
         completions = set(c.completions('ip.', 3))
 
         nt.assert_not_in('ip.__str__', matches)
         nt.assert_not_in(Completion(3,3,'__str__'), completions)
 
         nt.assert_not_in('ip._hidden_attr', matches)
         nt.assert_not_in(Completion(3,3, "_hidden_attr"), completions)
 
     with provisionalcompleter():
         s,matches = c.complete('ip._x.')
         completions = set(c.completions('ip._x.', 6))
 
         nt.assert_in('ip._x.keys', matches)
         nt.assert_in(Completion(6,6, "keys"), completions)
 
     del ip._hidden_attr
     del ip._x
 
 
 def test_limit_to__all__False_ok():
     """
     Limit to all is deprecated, once we remove it this test can go away. 
     """
     ip = get_ipython()
     c = ip.Completer
     ip.ex('class D: x=24')
     ip.ex('d=D()')
     cfg = Config()
     cfg.IPCompleter.limit_to__all__ = False
     c.update_config(cfg)
     s, matches = c.complete('d.')
     nt.assert_in('d.x', matches)
 
 
 def test_get__all__entries_ok():
     class A(object):
         __all__ = ['x', 1]
     words = completer.get__all__entries(A())
     nt.assert_equal(words, ['x'])
 
 
 def test_get__all__entries_no__all__ok():
     class A(object):
         pass
     words = completer.get__all__entries(A())
     nt.assert_equal(words, [])
 
 
 def test_func_kw_completions():
     ip = get_ipython()
     c = ip.Completer
     ip.ex('def myfunc(a=1,b=2): return a+b')
     s, matches = c.complete(None, 'myfunc(1,b')
     nt.assert_in('b=', matches)
     # Simulate completing with cursor right after b (pos==10):
     s, matches = c.complete(None, 'myfunc(1,b)', 10)
     nt.assert_in('b=', matches)
     s, matches = c.complete(None, 'myfunc(a="escaped\\")string",b')
     nt.assert_in('b=', matches)
     #builtin function
     s, matches = c.complete(None, 'min(k, k')
     nt.assert_in('key=', matches)
 
 
 def test_default_arguments_from_docstring():
     ip = get_ipython()
     c = ip.Completer
     kwd = c._default_arguments_from_docstring(
         'min(iterable[, key=func]) -> value')
     nt.assert_equal(kwd, ['key'])
     #with cython type etc
     kwd = c._default_arguments_from_docstring(
         'Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n')
     nt.assert_equal(kwd, ['ncall', 'resume', 'nsplit'])
     #white spaces
     kwd = c._default_arguments_from_docstring(
         '\n Minuit.migrad(self, int ncall=10000, resume=True, int nsplit=1)\n')
     nt.assert_equal(kwd, ['ncall', 'resume', 'nsplit'])
 
 def test_line_magics():
     ip = get_ipython()
     c = ip.Completer
     s, matches = c.complete(None, 'lsmag')
     nt.assert_in('%lsmagic', matches)
     s, matches = c.complete(None, '%lsmag')
     nt.assert_in('%lsmagic', matches)
 
 
 def test_cell_magics():
     from IPython.core.magic import register_cell_magic
 
     @register_cell_magic
     def _foo_cellm(line, cell):
         pass
     
     ip = get_ipython()
     c = ip.Completer
 
     s, matches = c.complete(None, '_foo_ce')
     nt.assert_in('%%_foo_cellm', matches)
     s, matches = c.complete(None, '%%_foo_ce')
     nt.assert_in('%%_foo_cellm', matches)
 
 
 def test_line_cell_magics():
     from IPython.core.magic import register_line_cell_magic
 
     @register_line_cell_magic
     def _bar_cellm(line, cell):
         pass
     
     ip = get_ipython()
     c = ip.Completer
 
     # The policy here is trickier, see comments in completion code.  The
     # returned values depend on whether the user passes %% or not explicitly,
     # and this will show a difference if the same name is both a line and cell
     # magic.
     s, matches = c.complete(None, '_bar_ce')
     nt.assert_in('%_bar_cellm', matches)
     nt.assert_in('%%_bar_cellm', matches)
     s, matches = c.complete(None, '%_bar_ce')
     nt.assert_in('%_bar_cellm', matches)
     nt.assert_in('%%_bar_cellm', matches)
     s, matches = c.complete(None, '%%_bar_ce')
     nt.assert_not_in('%_bar_cellm', matches)
     nt.assert_in('%%_bar_cellm', matches)
 
 
 def test_magic_completion_order():
     ip = get_ipython()
     c = ip.Completer
 
     # Test ordering of line and cell magics.
     text, matches = c.complete("timeit")
     nt.assert_equal(matches, ["%timeit", "%%timeit"])
 
 
 def test_magic_completion_shadowing():
     ip = get_ipython()
     c = ip.Completer
 
     # Before importing matplotlib, %matplotlib magic should be the only option.
     text, matches = c.complete("mat")
     nt.assert_equal(matches, ["%matplotlib"])
 
     # The newly introduced name should shadow the magic.
     ip.run_cell("matplotlib = 1")
     text, matches = c.complete("mat")
     nt.assert_equal(matches, ["matplotlib"])
 
     # After removing matplotlib from namespace, the magic should again be
     # the only option.
     del ip.user_ns["matplotlib"]
     text, matches = c.complete("mat")
     nt.assert_equal(matches, ["%matplotlib"])
 
 
 def test_magic_config():
     ip = get_ipython()
     c = ip.Completer
 
     s, matches = c.complete(None, 'conf')
     nt.assert_in('%config', matches)
     s, matches = c.complete(None, 'conf')
     nt.assert_not_in('AliasManager', matches)
     s, matches = c.complete(None, 'config ')
     nt.assert_in('AliasManager', matches)
     s, matches = c.complete(None, '%config ')
     nt.assert_in('AliasManager', matches)
     s, matches = c.complete(None, 'config Ali')
     nt.assert_list_equal(['AliasManager'], matches)
     s, matches = c.complete(None, '%config Ali')
     nt.assert_list_equal(['AliasManager'], matches)
     s, matches = c.complete(None, 'config AliasManager')
     nt.assert_list_equal(['AliasManager'], matches)
     s, matches = c.complete(None, '%config AliasManager')
     nt.assert_list_equal(['AliasManager'], matches)
     s, matches = c.complete(None, 'config AliasManager.')
     nt.assert_in('AliasManager.default_aliases', matches)
     s, matches = c.complete(None, '%config AliasManager.')
     nt.assert_in('AliasManager.default_aliases', matches)
     s, matches = c.complete(None, 'config AliasManager.de')
     nt.assert_list_equal(['AliasManager.default_aliases'], matches)
     s, matches = c.complete(None, 'config AliasManager.de')
     nt.assert_list_equal(['AliasManager.default_aliases'], matches)
 
 
 def test_magic_color():
     ip = get_ipython()
     c = ip.Completer
 
     s, matches = c.complete(None, 'colo')
     nt.assert_in('%colors', matches)
     s, matches = c.complete(None, 'colo')
     nt.assert_not_in('NoColor', matches)
     s, matches = c.complete(None, 'colors ')
     nt.assert_in('NoColor', matches)
     s, matches = c.complete(None, '%colors ')
     nt.assert_in('NoColor', matches)
     s, matches = c.complete(None, 'colors NoCo')
     nt.assert_list_equal(['NoColor'], matches)
     s, matches = c.complete(None, '%colors NoCo')
     nt.assert_list_equal(['NoColor'], matches)
 
 
 def test_match_dict_keys():
     """
     Test that match_dict_keys works on a couple of use case does return what
     expected, and does not crash
     """
     delims = ' \t\n`!@#$^&*()=+[{]}\\|;:\'",<>?'
 
 
     keys = ['foo', b'far']
     assert match_dict_keys(keys, "b'", delims=delims)  == ("'", 2 ,['far'])
     assert match_dict_keys(keys, "b'f", delims=delims) == ("'", 2 ,['far'])
     assert match_dict_keys(keys, 'b"', delims=delims)  == ('"', 2 ,['far'])
     assert match_dict_keys(keys, 'b"f', delims=delims) == ('"', 2 ,['far'])
 
     assert match_dict_keys(keys, "'", delims=delims)  == ("'", 1 ,['foo'])
     assert match_dict_keys(keys, "'f", delims=delims) == ("'", 1 ,['foo'])
     assert match_dict_keys(keys, '"', delims=delims)  == ('"', 1 ,['foo'])
     assert match_dict_keys(keys, '"f', delims=delims) == ('"', 1 ,['foo'])
     
     match_dict_keys
 
 
 def test_dict_key_completion_string():
     """Test dictionary key completion for string keys"""
     ip = get_ipython()
     complete = ip.Completer.complete
 
     ip.user_ns['d'] = {'abc': None}
 
     # check completion at different stages
     _, matches = complete(line_buffer="d[")
     nt.assert_in("'abc'", matches)
     nt.assert_not_in("'abc']", matches)
 
     _, matches = complete(line_buffer="d['")
     nt.assert_in("abc", matches)
     nt.assert_not_in("abc']", matches)
 
     _, matches = complete(line_buffer="d['a")
     nt.assert_in("abc", matches)
     nt.assert_not_in("abc']", matches)
 
     # check use of different quoting
     _, matches = complete(line_buffer="d[\"")
     nt.assert_in("abc", matches)
     nt.assert_not_in('abc\"]', matches)
 
     _, matches = complete(line_buffer="d[\"a")
     nt.assert_in("abc", matches)
     nt.assert_not_in('abc\"]', matches)
 
     # check sensitivity to following context
     _, matches = complete(line_buffer="d[]", cursor_pos=2)
     nt.assert_in("'abc'", matches)
 
     _, matches = complete(line_buffer="d['']", cursor_pos=3)
     nt.assert_in("abc", matches)
     nt.assert_not_in("abc'", matches)
     nt.assert_not_in("abc']", matches)
 
     # check multiple solutions are correctly returned and that noise is not
     ip.user_ns['d'] = {'abc': None, 'abd': None, 'bad': None, object(): None,
                        5: None}
 
     _, matches = complete(line_buffer="d['a")
     nt.assert_in("abc", matches)
     nt.assert_in("abd", matches)
     nt.assert_not_in("bad", matches)
     assert not any(m.endswith((']', '"', "'")) for m in matches), matches
 
     # check escaping and whitespace
     ip.user_ns['d'] = {'a\nb': None, 'a\'b': None, 'a"b': None, 'a word': None}
     _, matches = complete(line_buffer="d['a")
     nt.assert_in("a\\nb", matches)
     nt.assert_in("a\\'b", matches)
     nt.assert_in("a\"b", matches)
     nt.assert_in("a word", matches)
     assert not any(m.endswith((']', '"', "'")) for m in matches), matches
 
     # - can complete on non-initial word of the string
     _, matches = complete(line_buffer="d['a w")
     nt.assert_in("word", matches)
 
     # - understands quote escaping
     _, matches = complete(line_buffer="d['a\\'")
     nt.assert_in("b", matches)
 
     # - default quoting should work like repr
     _, matches = complete(line_buffer="d[")
     nt.assert_in("\"a'b\"", matches)
 
     # - when opening quote with ", possible to match with unescaped apostrophe
     _, matches = complete(line_buffer="d[\"a'")
     nt.assert_in("b", matches)
 
     # need to not split at delims that readline won't split at
     if '-' not in ip.Completer.splitter.delims:
         ip.user_ns['d'] = {'before-after': None}
         _, matches = complete(line_buffer="d['before-af")
         nt.assert_in('before-after', matches)
 
 def test_dict_key_completion_contexts():
     """Test expression contexts in which dict key completion occurs"""
     ip = get_ipython()
     complete = ip.Completer.complete
     d = {'abc': None}
     ip.user_ns['d'] = d
 
     class C:
         data = d
     ip.user_ns['C'] = C
     ip.user_ns['get'] = lambda: d
 
     def assert_no_completion(**kwargs):
         _, matches = complete(**kwargs)
         nt.assert_not_in('abc', matches)
         nt.assert_not_in('abc\'', matches)
         nt.assert_not_in('abc\']', matches)
         nt.assert_not_in('\'abc\'', matches)
         nt.assert_not_in('\'abc\']', matches)
 
     def assert_completion(**kwargs):
         _, matches = complete(**kwargs)
         nt.assert_in("'abc'", matches)
         nt.assert_not_in("'abc']", matches)
 
     # no completion after string closed, even if reopened
     assert_no_completion(line_buffer="d['a'")
     assert_no_completion(line_buffer="d[\"a\"")
     assert_no_completion(line_buffer="d['a' + ")
     assert_no_completion(line_buffer="d['a' + '")
 
     # completion in non-trivial expressions
     assert_completion(line_buffer="+ d[")
     assert_completion(line_buffer="(d[")
     assert_completion(line_buffer="C.data[")
 
     # greedy flag
     def assert_completion(**kwargs):
         _, matches = complete(**kwargs)
         nt.assert_in("get()['abc']", matches)
     
     assert_no_completion(line_buffer="get()[")
     with greedy_completion():
         assert_completion(line_buffer="get()[")
         assert_completion(line_buffer="get()['")
         assert_completion(line_buffer="get()['a")
         assert_completion(line_buffer="get()['ab")
         assert_completion(line_buffer="get()['abc")
 
 
 
 def test_dict_key_completion_bytes():
     """Test handling of bytes in dict key completion"""
     ip = get_ipython()
     complete = ip.Completer.complete
 
     ip.user_ns['d'] = {'abc': None, b'abd': None}
 
     _, matches = complete(line_buffer="d[")
     nt.assert_in("'abc'", matches)
     nt.assert_in("b'abd'", matches)
 
     if False:  # not currently implemented
         _, matches = complete(line_buffer="d[b")
         nt.assert_in("b'abd'", matches)
         nt.assert_not_in("b'abc'", matches)
 
         _, matches = complete(line_buffer="d[b'")
         nt.assert_in("abd", matches)
         nt.assert_not_in("abc", matches)
 
         _, matches = complete(line_buffer="d[B'")
         nt.assert_in("abd", matches)
         nt.assert_not_in("abc", matches)
 
         _, matches = complete(line_buffer="d['")
         nt.assert_in("abc", matches)
         nt.assert_not_in("abd", matches)
 
 
 def test_dict_key_completion_unicode_py3():
     """Test handling of unicode in dict key completion"""
     ip = get_ipython()
     complete = ip.Completer.complete
 
     ip.user_ns['d'] = {u'a\u05d0': None}
 
     # query using escape
     if sys.platform != 'win32':
         # Known failure on Windows
         _, matches = complete(line_buffer="d['a\\u05d0")
         nt.assert_in("u05d0", matches)  # tokenized after \\
 
     # query using character
     _, matches = complete(line_buffer="d['a\u05d0")
     nt.assert_in(u"a\u05d0", matches)
     
     with greedy_completion():
         # query using escape
         _, matches = complete(line_buffer="d['a\\u05d0")
         nt.assert_in("d['a\\u05d0']", matches)  # tokenized after \\
 
         # query using character
         _, matches = complete(line_buffer="d['a\u05d0")
         nt.assert_in(u"d['a\u05d0']", matches)
         
 
 
 @dec.skip_without('numpy')
 def test_struct_array_key_completion():
     """Test dict key completion applies to numpy struct arrays"""
     import numpy
     ip = get_ipython()
     complete = ip.Completer.complete
     ip.user_ns['d'] = numpy.array([], dtype=[('hello', 'f'), ('world', 'f')])
     _, matches = complete(line_buffer="d['")
     nt.assert_in("hello", matches)
     nt.assert_in("world", matches)
     # complete on the numpy struct itself
     dt = numpy.dtype([('my_head', [('my_dt', '>u4'), ('my_df', '>u4')]),
                       ('my_data', '>f4', 5)])
     x = numpy.zeros(2, dtype=dt)
     ip.user_ns['d'] = x[1]
     _, matches = complete(line_buffer="d['")
     nt.assert_in("my_head", matches)
     nt.assert_in("my_data", matches)
     # complete on a nested level
     with greedy_completion():
         ip.user_ns['d'] = numpy.zeros(2, dtype=dt)
         _, matches = complete(line_buffer="d[1]['my_head']['")
         nt.assert_true(any(["my_dt" in m for m in matches]))
         nt.assert_true(any(["my_df" in m for m in matches]))
 
 
 @dec.skip_without('pandas')
 def test_dataframe_key_completion():
     """Test dict key completion applies to pandas DataFrames"""
     import pandas
     ip = get_ipython()
     complete = ip.Completer.complete
     ip.user_ns['d'] = pandas.DataFrame({'hello': [1], 'world': [2]})
     _, matches = complete(line_buffer="d['")
     nt.assert_in("hello", matches)
     nt.assert_in("world", matches)
 
 
 def test_dict_key_completion_invalids():
     """Smoke test cases dict key completion can't handle"""
     ip = get_ipython()
     complete = ip.Completer.complete
 
     ip.user_ns['no_getitem'] = None
     ip.user_ns['no_keys'] = []
     ip.user_ns['cant_call_keys'] = dict
     ip.user_ns['empty'] = {}
     ip.user_ns['d'] = {'abc': 5}
 
     _, matches = complete(line_buffer="no_getitem['")
     _, matches = complete(line_buffer="no_keys['")
     _, matches = complete(line_buffer="cant_call_keys['")
     _, matches = complete(line_buffer="empty['")
     _, matches = complete(line_buffer="name_error['")
     _, matches = complete(line_buffer="d['\\")  # incomplete escape
 
 class KeyCompletable(object):
     def __init__(self, things=()):
         self.things = things
 
     def _ipython_key_completions_(self):
         return list(self.things)
 
 def test_object_key_completion():
     ip = get_ipython()
     ip.user_ns['key_completable'] = KeyCompletable(['qwerty', 'qwick'])
 
     _, matches = ip.Completer.complete(line_buffer="key_completable['qw")
     nt.assert_in('qwerty', matches)
     nt.assert_in('qwick', matches)
 
 
 def test_tryimport():
     """
     Test that try-import don't crash on trailing dot, and import modules before
     """
     from IPython.core.completerlib import try_import
     assert(try_import("IPython."))
 
 
 def test_aimport_module_completer():
     ip = get_ipython()
     _, matches = ip.complete('i', '%aimport i')
     nt.assert_in('io', matches)
     nt.assert_not_in('int', matches)
 
 def test_nested_import_module_completer():
     ip = get_ipython()
     _, matches = ip.complete(None, 'import IPython.co', 17)
     nt.assert_in('IPython.core', matches)
     nt.assert_not_in('import IPython.core', matches)
     nt.assert_not_in('IPython.display', matches)
 
 def test_import_module_completer():
     ip = get_ipython()
     _, matches = ip.complete('i', 'import i')
     nt.assert_in('io', matches)
     nt.assert_not_in('int', matches)
 
 def test_from_module_completer():
     ip = get_ipython()
     _, matches = ip.complete('B', 'from io import B', 16)
     nt.assert_in('BytesIO', matches)
     nt.assert_not_in('BaseException', matches)
 
 def test_snake_case_completion():
     ip = get_ipython()
     ip.user_ns['some_three'] = 3
     ip.user_ns['some_four'] = 4
     _, matches = ip.complete("s_", "print(s_f")
     nt.assert_in('some_three', matches)
-    nt.assert_in('some_four', matches)
  No newline at end of file
+    nt.assert_in('some_four', matches)

IPython/terminal/ptutils.py

@@ -1,160 +1,160 b''
 """prompt-toolkit utilities
 
 Everything in this module is a private API,
 not to be used outside IPython.
 """
 
 # Copyright (c) IPython Development Team.
 # Distributed under the terms of the Modified BSD License.
 
 import unicodedata
 from wcwidth import wcwidth
 
 from IPython.core.completer import (
     provisionalcompleter, cursor_to_position,
     _deduplicate_completions)
 from prompt_toolkit.completion import Completer, Completion
 from prompt_toolkit.layout.lexers import Lexer
 from prompt_toolkit.layout.lexers import PygmentsLexer
 
 import pygments.lexers as pygments_lexers
 
 _completion_sentinel = object()
 
 def _elide(string, *, min_elide=30):
     """
     If a string is long enough, and has at least 2 dots,
     replace the middle part with ellipses.
 
     If three consecutive dots, or two consecutive dots are encountered these are
     replaced by the equivalents HORIZONTAL ELLIPSIS or TWO DOT LEADER unicode
     equivalents
     """
     string = string.replace('...','\N{HORIZONTAL ELLIPSIS}')
     string = string.replace('..','\N{TWO DOT LEADER}')
     if len(string) < min_elide:
         return string
 
     parts = string.split('.')
 
     if len(parts) <= 3:
         return string
 
     return '{}.{}\N{HORIZONTAL ELLIPSIS}{}.{}'.format(parts[0], parts[1][0], parts[-2][-1], parts[-1])
 
 
 def _adjust_completion_text_based_on_context(text, body, offset):
     if text.endswith('=') and len(body) > offset and body[offset] is '=':
         return text[:-1]
     else:
         return text
 
 
 class IPythonPTCompleter(Completer):
     """Adaptor to provide IPython completions to prompt_toolkit"""
     def __init__(self, ipy_completer=None, shell=None, patch_stdout=None):
         if shell is None and ipy_completer is None:
             raise TypeError("Please pass shell=an InteractiveShell instance.")
         self._ipy_completer = ipy_completer
         self.shell = shell
         if patch_stdout is None:
             raise TypeError("Please pass patch_stdout")
         self.patch_stdout = patch_stdout
 
     @property
     def ipy_completer(self):
         if self._ipy_completer:
             return self._ipy_completer
         else:
             return self.shell.Completer
 
     def get_completions(self, document, complete_event):
         if not document.current_line.strip():
             return
         # Some bits of our completion system may print stuff (e.g. if a module
         # is imported). This context manager ensures that doesn't interfere with
         # the prompt.
 
         with self.patch_stdout(), provisionalcompleter():
             body = document.text
             cursor_row = document.cursor_position_row
             cursor_col = document.cursor_position_col
             cursor_position = document.cursor_position
             offset = cursor_to_position(body, cursor_row, cursor_col)
             yield from self._get_completions(body, offset, cursor_position, self.ipy_completer)
 
     @staticmethod
     def _get_completions(body, offset, cursor_position, ipyc):
         """
         Private equivalent of get_completions() use only for unit_testing.
         """
         debug = getattr(ipyc, 'debug', False)
         completions = _deduplicate_completions(
             body, ipyc.completions(body, offset))
         for c in completions:
             if not c.text:
                 # Guard against completion machinery giving us an empty string.
                 continue
             text = unicodedata.normalize('NFC', c.text)
             # When the first character of the completion has a zero length,
             # then it's probably a decomposed unicode character. E.g. caused by
             # the "\dot" completion. Try to compose again with the previous
             # character.
             if wcwidth(text[0]) == 0:
                 if cursor_position + c.start > 0:
                     char_before = body[c.start - 1]
                     fixed_text = unicodedata.normalize(
                         'NFC', char_before + text)
 
                     # Yield the modified completion instead, if this worked.
                     if wcwidth(text[0:1]) == 1:
                         yield Completion(fixed_text, start_position=c.start - offset - 1)
                         continue
 
             # TODO: Use Jedi to determine meta_text
             # (Jedi currently has a bug that results in incorrect information.)
             # meta_text = ''
             # yield Completion(m, start_position=start_pos,
             #                  display_meta=meta_text)
             display_text = c.text
 
             adjusted_text = _adjust_completion_text_based_on_context(c.text, body, offset)
             if c.type == 'function':
-                display_text = display_text + '()'
-
-            yield Completion(adjusted_text, start_position=c.start - offset, display=_elide(display_text), display_meta=c.type)
+                yield Completion(adjusted_text, start_position=c.start - offset, display=_elide(display_text+'()'), display_meta=c.type+c.signature)
+            else:
+                yield Completion(adjusted_text, start_position=c.start - offset, display=_elide(display_text), display_meta=c.type)
 
 class IPythonPTLexer(Lexer):
     """
     Wrapper around PythonLexer and BashLexer.
     """
     def __init__(self):
         l = pygments_lexers
         self.python_lexer = PygmentsLexer(l.Python3Lexer)
         self.shell_lexer = PygmentsLexer(l.BashLexer)
 
         self.magic_lexers = {
             'HTML': PygmentsLexer(l.HtmlLexer),
             'html': PygmentsLexer(l.HtmlLexer),
             'javascript': PygmentsLexer(l.JavascriptLexer),
             'js': PygmentsLexer(l.JavascriptLexer),
             'perl': PygmentsLexer(l.PerlLexer),
             'ruby': PygmentsLexer(l.RubyLexer),
             'latex': PygmentsLexer(l.TexLexer),
         }
 
     def lex_document(self, cli, document):
         text = document.text.lstrip()
 
         lexer = self.python_lexer
 
         if text.startswith('!') or text.startswith('%%bash'):
             lexer = self.shell_lexer
 
         elif text.startswith('%%'):
             for magic, l in self.magic_lexers.items():
                 if text.startswith('%%' + magic):
                     lexer = l
                     break
 
         return lexer.lex_document(cli, document)

docs/source/interactive/tutorial.rst

@@ -1,280 +1,282 b''
 .. _tutorial:
 
 ======================
 Introducing IPython
 ======================
 
 You don't need to know anything beyond Python to start using IPython – just type
 commands as you would at the standard Python prompt. But IPython can do much
 more than the standard prompt. Some key features are described here. For more
 information, check the :ref:`tips page <tips>`, or look at examples in the
 `IPython cookbook <https://github.com/ipython/ipython/wiki/Cookbook%3A-Index>`_.
 
 If you haven't done that yet see `how to install ipython <install>`_ .
 
 If you've never used Python before, you might want to look at `the official
 tutorial <http://docs.python.org/tutorial/>`_ or an alternative, `Dive into
 Python <http://diveintopython.net/toc/index.html>`_.
 
 Start IPython by issuing the ``ipython`` command from your shell, you should be
 greeted by the following::
 
     Python 3.6.0
     Type 'copyright', 'credits' or 'license' for more information
     IPython 6.0.0.dev -- An enhanced Interactive Python. Type '?' for help.
 
     In [1]:
 
 
 Unlike the Python REPL, you will see that the input prompt is ``In [N]:``
 instead of ``>>>``. The number ``N`` in the prompt will be used later in this
 tutorial but should usually not impact the computation.
 
 You should be able to type single line expressions and press enter to evaluate
 them. If an expression is incomplete, IPython will automatically detect this and
 add a new line when you press :kbd:`Enter` instead of executing right away.
 
 Feel free to explore multi-line text input. Unlike many other REPLs, with
 IPython you can use the up and down arrow keys when editing multi-line
 code blocks.
 
 Here is an example of a longer interaction with the IPython REPL,
 which we often refer to as an IPython *session* ::
 
     In [1]: print('Hello IPython')
     Hello IPython
 
     In [2]: 21 * 2
     Out[2]: 42
 
     In [3]: def say_hello(name):
        ...:     print('Hello {name}'.format(name=name))
        ...:
 
 We won't get into details right now, but you may notice a few differences to
 the standard Python REPL. First, your code should be syntax-highlighted as you
 type. Second, you will see that some results will have an ``Out[N]:`` prompt,
 while some other do not. We'll come to this later.
 
 Depending on the exact command you are typing you might realize that sometimes
 :kbd:`Enter` will add a new line, and sometimes it will execute the current
 statement. IPython tries to guess what you are doing, so most of the time you
 should not have to care. Though if by any chance IPython does not the right
 thing you can force execution of the current code block by pressing in sequence
 :kbd:`Esc` and :kbd:`Enter`. You can also force the insertion of a new line at
 the position of the cursor by using :kbd:`Ctrl-o`.
 
 The four most helpful commands
 ==============================
 
 The four most helpful commands, as well as their brief description, is shown
 to you in a banner, every time you start IPython:
 
 ==========    =========================================================
 command       description
 ==========    =========================================================
 ?             Introduction and overview of IPython's features.
 %quickref     Quick reference.
 help          Python's own help system.
 object?       Details about 'object', use 'object??' for extra details.
 ==========    =========================================================
 
 Tab completion
 ==============
 
 Tab completion, especially for attributes, is a convenient way to explore the
 structure of any object you're dealing with. Simply type ``object_name.<TAB>``
 to view the object's attributes. Besides Python objects and keywords, tab
 completion also works on file and directory names.
 
 Starting with IPython 6.0, if ``jedi`` is installed, IPython will try to pull
 completions from Jedi as well. This allows to not only inspect currently
 existing objects, but also to infer completion statically without executing
 code. There is nothing particular need to get this to work, simply use tab
 completion on more complex expressions like the following::
 
     >>> data = ['Number of users', 123_456]
     ... data[0].<tab>
 
 IPython and Jedi will be able to infer that ``data[0]`` is actually a string
 and should show relevant completions like ``upper()``, ``lower()`` and other
 string methods. You can use the :kbd:`Tab` key to cycle through completions,
 and while a completion is highlighted, its type will be shown as well.
+When the type of the completion is a function, the completer will also show the
+signature of the function when highlighted.
 
 Exploring your objects
 ======================
 
 Typing ``object_name?`` will print all sorts of details about any object,
 including docstrings, function definition lines (for call arguments) and
 constructor details for classes. To get specific information on an object, you
 can use the magic commands ``%pdoc``, ``%pdef``, ``%psource`` and ``%pfile``
 
 .. _magics_explained:
 
 Magic functions
 ===============
 
 IPython has a set of predefined 'magic functions' that you can call with a
 command line style syntax.  There are two kinds of magics, line-oriented and
 cell-oriented.  **Line magics** are prefixed with the ``%`` character and work
 much like OS command-line calls: they get as an argument the rest of the line,
 where arguments are passed without parentheses or quotes. **Lines magics** can
 return results and can be used in the right hand side of an assignment.  **Cell
 magics** are prefixed with a double ``%%``, and they are functions that get as
 an argument not only the rest of the line, but also the lines below it in a
 separate argument.
 
 Magics are useful as convenient functions where Python syntax is not the most
 natural one, or when one want to embed invalid python syntax in their work flow. 
 
 The following examples show how to call the built-in :magic:`timeit` magic, both
 in line and cell mode::
 
       In [1]: %timeit range(1000)
       100000 loops, best of 3: 7.76 us per loop
 
       In [2]: %%timeit x = range(10000)
       ...: max(x)
       ...: 
       1000 loops, best of 3: 223 us per loop
 
 The built-in magics include:
 
 - Functions that work with code: :magic:`run`, :magic:`edit`, :magic:`save`,
   :magic:`macro`, :magic:`recall`, etc.
 
 - Functions which affect the shell: :magic:`colors`, :magic:`xmode`,
   :magic:`autoindent`, :magic:`automagic`, etc.
 
 - Other functions such as :magic:`reset`, :magic:`timeit`,
   :cellmagic:`writefile`, :magic:`load`, or :magic:`paste`.
 
 You can always call magics using the ``%`` prefix, and if you're calling a line
 magic on a line by itself, as long as the identifier is not defined in your
 namespace, you can omit even that::
 
     run thescript.py
 
 You can toggle this behavior by running the :magic:`automagic` magic.  Cell
 magics must always have the ``%%`` prefix.
 
 A more detailed explanation of the magic system can be obtained by calling
 ``%magic``, and for more details on any magic function, call ``%somemagic?`` to
 read its docstring. To see all the available magic functions, call
 ``%lsmagic``.
 
 .. seealso::
 
     The :ref:`magic` section of the documentation goes more in depth into how
     the magics works and how to define your own, and :doc:`magics` for a list of
     built-in magics.
 
     `Cell magics`_ example notebook
 
 Running and Editing
 -------------------
 
 The :magic:`run` magic command allows you to run any python script and load all
 of its data directly into the interactive namespace. Since the file is re-read
 from disk each time, changes you make to it are reflected immediately (unlike
 imported modules, which have to be specifically reloaded). IPython also includes
 :ref:`dreload <dreload>`, a recursive reload function.
 
 ``%run`` has special flags for timing the execution of your scripts (-t), or
 for running them under the control of either Python's pdb debugger (-d) or
 profiler (-p).
 
 The :magic:`edit` command gives a reasonable approximation of multi-line editing,
 by invoking your favorite editor on the spot. IPython will execute the
 code you type in there as if it were typed interactively. Note that for
 :magic:`edit` to work, the call to startup your editor has to be a blocking
 call. In a GUI environment, your editor likely will have such an option.
 
 Debugging
 ---------
 
 After an exception occurs, you can call :magic:`debug` to jump into the Python
 debugger (pdb) and examine the problem. Alternatively, if you call :magic:`pdb`,
 IPython will automatically start the debugger on any uncaught exception. You can
 print variables, see code, execute statements and even walk up and down the call
 stack to track down the true source of the problem. This can be an efficient way
 to develop and debug code, in many cases eliminating the need for print
 statements or external debugging tools.
 
 You can also step through a program from the beginning by calling
 ``%run -d theprogram.py``.
 
 History
 =======
 
 IPython stores both the commands you enter, and the results it produces. You
 can easily go through previous commands with the up- and down-arrow keys, or
 access your history in more sophisticated ways.
 
 Input and output history are kept in variables called ``In`` and ``Out``, keyed
 by the prompt numbers, e.g. ``In[4]``. The last three objects in output history
 are also kept in variables named ``_``, ``__`` and ``___``.
 
 You can use the ``%history`` magic function to examine past input and output.
 Input history from previous sessions is saved in a database, and IPython can be
 configured to save output history.
 
 Several other magic functions can use your input history, including ``%edit``, 
 ``%rerun``, ``%recall``, ``%macro``, ``%save`` and ``%pastebin``. You can use a
 standard format to refer to lines::
 
     %pastebin 3 18-20 ~1/1-5
     
 This will take line 3 and lines 18 to 20 from the current session, and lines
 1-5 from the previous session.
 
 System shell commands
 =====================
 
 To run any command at the system shell, simply prefix it with ``!``, e.g.::
 
     !ping www.bbc.co.uk
     
 You can capture the output into a Python list, e.g.: ``files = !ls``. To pass
 the values of Python variables or expressions to system commands, prefix them
 with $: ``!grep -rF $pattern ipython/*``. See :ref:`our shell section
 <system_shell_access>` for more details.
 
 Define your own system aliases
 ------------------------------
 
 It's convenient to have aliases to the system commands you use most often. This
 allows you to work seamlessly from inside IPython with the same commands you are
 used to in your system shell. IPython comes with some pre-defined aliases and a
 complete system for changing directories, both via a stack (see :magic:`pushd`,
 :magic:`popd` and :magic:`dhist`) and via direct :magic:`cd`. The latter keeps a
 history of visited directories and allows you to go to any previously visited
 one.
 
 
 Configuration
 =============
 
 Much of IPython can be tweaked through :doc:`configuration </config/intro>`.
 To get started, use the command ``ipython profile create`` to produce the
 default config files. These will be placed in
 :file:`~/.ipython/profile_default`, and contain comments explaining
 what the various options do.
 
 Profiles allow you to use IPython for different tasks, keeping separate config
 files and history for each one. More details in :ref:`the profiles section
 <profiles>`.
 
 .. _startup_files:
 
 Startup Files
 -------------
 
 If you want some code to be run at the beginning of every IPython session, the
 easiest way is to add Python (.py) or IPython (.ipy) scripts to your
 :file:`profile_default/startup/` directory. Files here will be executed as soon
 as the IPython shell is constructed, before any other code or scripts you have
 specified. The files will be run in order of their names, so you can control the
 ordering with prefixes, like ``10-myimports.py``.
 
 .. include:: ../links.txt