Merge pull request #9677 from takluyver/docs-5-polish

Some docs work to prepare for 5.0

IPython/core/interactiveshell.py

@@ -250,7 +250,7 b' class InteractiveShell(SingletonConfigurable):'
     ).tag(config=True)
     colors = CaselessStrEnum(('Neutral', 'NoColor','LightBG','Linux'),
                              default_value='Neutral',
-        help="Set the color scheme (NoColor, Linux, or LightBG)."
+        help="Set the color scheme (NoColor, Neutral, Linux, or LightBG)."
     ).tag(config=True)
     colors_force = Bool(False, help=
         """

IPython/terminal/pt_inputhooks/__init__.py

@@ -15,6 +15,12 b' backends = ['
     'osx',
 ]
 
+registered = {}
+
+def register(name, inputhook):
+    """Register the function *inputhook* as an event loop integration."""
+    registered[name] = inputhook
+
 class UnknownBackend(KeyError):
     def __init__(self, name):
         self.name = name
@@ -22,9 +28,12 b' class UnknownBackend(KeyError):'
     def __str__(self):
         return ("No event loop integration for {!r}. "
                 "Supported event loops are: {}").format(self.name,
-                                    ', '.join(backends))
+                                    ', '.join(backends + sorted(registered)))
 
 def get_inputhook_func(gui):
+    if gui in registered:
+        return registered[gui]
+
     if gui not in backends:
         raise UnknownBackend(gui)
 

docs/autogen_config.py

@@ -11,7 +11,8 b" generated = join(options, 'config-generated.txt')"
 
 
 def write_doc(name, title, app, preamble=None):
-    with open(generated, 'a') as f:
+    filename = join(options, name+'.rst')
+    with open(filename, 'w') as f:
         f.write(title + '\n')
         f.write(('=' * len(title)) + '\n')
         f.write('\n')
@@ -21,7 +22,7 b' def write_doc(name, title, app, preamble=None):'
 
 
 if __name__ == '__main__':
-    # create empty file
+    # Touch this file for the make target
     with open(generated, 'w'):
         pass
 

docs/source/config/details.rst

@@ -2,89 +2,85 b''
 Specific config details
 =======================
 
+.. _custom_prompts:
+
+Custom Prompts
+==============
+
+.. versionchanged:: 5.0
+
+From IPython 5, prompts are produced as a list of Pygments tokens, which are
+tuples of (token_type, text). You can customise prompts by writing a method
+which generates a list of tokens.
+
+There are four kinds of prompt:
+
+* The **in** prompt is shown before the first line of input
+  (default like ``In [1]:``).
+* The **continuation** prompt is shown before further lines of input
+  (default like ``...:``).
+* The **rewrite** prompt is shown to highlight how special syntax has been
+  interpreted (default like ``----->``).
+* The **out** prompt is shown before the result from evaluating the input
+  (default like ``Out[1]:``).
+
+Custom prompts are supplied together as a class. If you want to customise only
+some of the prompts, inherit from :class:`IPython.terminal.prompts.Prompts`,
+which defines the defaults. The required interface is like this:
+
+.. class:: MyPrompts(shell)
+
+   Prompt style definition. *shell* is a reference to the
+   :class:`~.TerminalInteractiveShell` instance.
+
+   .. method:: in_prompt_tokens(cli=None)
+               continuation_prompt_tokens(self, cli=None, width=None)
+               rewrite_prompt_tokens()
+               out_prompt_tokens()
+
+      Return the respective prompts as lists of ``(token_type, text)`` tuples.
+
+      For continuation prompts, *width* is an integer representing the width of
+      the prompt area in terminal columns.
+
+      *cli*, where used, is the prompt_toolkit ``CommandLineInterface`` instance.
+      This is mainly for compatibility with the API prompt_toolkit expects.
+
+Inside IPython or in a startup script, you can use a custom prompts class
+by setting ``get_ipython().prompts`` to an *instance* of the class.
+In configuration, ``TerminalInteractiveShell.prompts_class`` may be set to
+either the class object, or a string of its full importable name.
+
 .. _termcolour:
 
 Terminal Colors
 ===============
 
-The default IPython configuration has most bells and whistles turned on
-(they're pretty safe). But there's one that may cause problems on some
-systems: the use of color on screen for displaying information. This is
-very useful, since IPython can show prompts and exception tracebacks
-with various colors, display syntax-highlighted source code, and in
-general make it easier to visually parse information.
-
-The following terminals seem to handle the color sequences fine:
-
-    * Linux main text console, KDE Konsole, Gnome Terminal, E-term,
-      rxvt, xterm.
-    * CDE terminal (tested under Solaris). This one boldfaces light colors.
-    * (X)Emacs buffers. See the :ref:`emacs` section for more details on
-      using IPython with (X)Emacs.
-    * A Windows (XP/2k) CygWin shell. Although some users have reported
-      problems; it is not clear whether there is an issue for everyone
-      or only under specific configurations. If you have full color
-      support under cygwin, please post to the IPython mailing list so
-      this issue can be resolved for all users.
-
-These have shown problems:
-
-    * Windows command prompt in WinXP/2k logged into a Linux machine via
-      telnet or ssh.
-    * Windows native command prompt in WinXP/2k, without Gary Bishop's
-      extensions. Once Gary's readline library is installed, the normal
-      WinXP/2k command prompt works perfectly.
-
-IPython uses colors for various groups of things that may be
-controlled by different configuration options: prompts, tracebacks, "as
-you type" in the terminal, and the object introspection system which
-passes large sets of data through a pager. There are various way to
-change the colors.
-
-We can distinguish the coloration into 2 main categories:
-
-- The one that affects only the terminal client.
-- The ones that also affect clients connected through the Jupyter
-  protocol.
-
-Traceback, debugger, and pager are highlighted kernel-side so they fall
-into the second category. For historical reasons they are often
-governed by a ``colors`` attribute or configuration option that can
-take one of 3 case insensitive values: ``NoColors``, ``Linux`` and
-``LightBG``.
-
-Colors that affect only the terminal client are governed mainly by
-``TerminalInteractiveShell.highlight_style`` taking the name of a
-``Pygments`` style.
-
-As of IPython 5.0 the color configuration works as follows:
-
-  - by default, ``TerminalInteractiveShell.highlight_style`` is set to
-    ``legacy`` which **trys to** emulate the colors of IPython pre 5.0
-    and respect the ``.color`` configuration option.
-    The emulation is an approximation of the current version of Pygments
-    (2.1) and only supports extended ANSI escape sequence, hence the
-    theme cannot adapt to your terminal custom mapping if you have
-    one.
-
-    The last extra difference being that the "as you type" coloration
-    is present using the theme "default" if `color` is `LightBG`, and
-    using the theme "monokai" if `Linux`.
-
-  - if ``TerminalInteractiveShell.highlight_style`` is set to any other
-    themes, this theme is used for "as you type" highlighting. The
-    prompt highlighting is then governed by
-    ``--TerminalInteractiveShell.highlighting_style_overrides``
-
-As a summary, by default IPython 5.0 should mostly behave unchanged
-from IPython 4.x and before. Use
-``TerminalInteractiveShell.highlight_style`` and
-``--TerminalInteractiveShell.highlighting_style_overrides`` for extra
-flexibility.
-
-With default configuration `--colors=[nocolors|linux|ightbg]` as well
-as the `%colors` magic should behave identically as before.
+.. versionchanged:: 5.0
+
+There are two main configuration options controlling colours.
+
+``InteractiveShell.colors`` sets the colour of tracebacks and object info (the
+output from e.g. ``zip?``). It may also affect other things if the option below
+is set to ``'legacy'``. It has four case-insensitive values:
+``'nocolor', 'neutral', 'linux', 'lightbg'``. The default is *neutral*, which
+should be legible on either dark or light terminal backgrounds. *linux* is
+optimised for dark backgrounds and *lightbg* for light ones.
+
+``TerminalInteractiveShell.highlight_style`` determines prompt colours and syntax
+highlighting. It takes the name of a Pygments style as a string, or the special
+value ``'legacy'`` to pick a style in accordance with ``InteractiveShell.colors``.
 
+You can see the Pygments styles available on your system by running::
+
+    import pygments
+    list(pygments.styles.get_all_styles())
+
+Additionally, ``TerminalInteractiveShell.highlight_style_overrides`` can override
+specific styles in the highlighting. It should be a dictionary mapping Pygments
+token types to strings defining the style. See `Pygments' documentation
+<http://pygments.org/docs/styles/#creating-own-styles>`__ for the language used
+to define styles.
 
 Colors in the pager
 -------------------
@@ -98,9 +94,6 b' To configure your default pager to allow these:'
    properly interpret control sequences, which is how color
    information is given to your terminal.
 
-
-
-
 .. _editors:
 
 Editor configuration

docs/source/config/eventloops.rst

@@ -13,47 +13,39 b' so different steps are needed to integrate with each.'
 Event loops in the terminal
 ---------------------------
 
-In the terminal, IPython uses a blocking Python function to wait for user input.
-However, the Python C API provides a hook, :c:func:`PyOS_InputHook`, which is
-called frequently while waiting for input. This can be set to a function which
-briefly runs the event loop and then returns.
-
-IPython provides Python level wrappers for setting and resetting this hook. To
-use them, subclass :class:`IPython.lib.inputhook.InputHookBase`, and define
-an ``enable(app=None)`` method, which initialises the event loop and calls
-``self.manager.set_inputhook(f)`` with a function which will briefly run the
-event loop before exiting. Decorate the class with a call to
-:func:`IPython.lib.inputhook.register`::
-
-    from IPython.lib.inputhook import register, InputHookBase
-
-    @register('clutter')
-    class ClutterInputHook(InputHookBase):
-        def enable(self, app=None):
-            self.manager.set_inputhook(inputhook_clutter)
-
-You can also optionally define a ``disable()`` method, taking no arguments, if
-there are extra steps needed to clean up. IPython will take care of resetting
-the hook, whether or not you provide a disable method.
-
-The simplest way to define the hook function is just to run one iteration of the
-event loop, or to run until no events are pending. Most event loops provide some
-mechanism to do one of these things. However, the GUI may lag slightly,
-because the hook is only called every 0.1 seconds. Alternatively, the hook can
-keep running the event loop until there is input ready on stdin. IPython
-provides a function to facilitate this:
-
-.. currentmodule:: IPython.lib.inputhook
-
-.. function:: stdin_ready()
-
-   Returns True if there is something ready to read on stdin.
-   
-   If this is the case, the hook function should return immediately.
-   
-   This is implemented for Windows and POSIX systems - on other platforms, it
-   always returns True, so that the hook always gives Python a chance to check
-   for input.
+.. versionchanged:: 5.0
+
+   There is a new API for event loop integration using prompt_toolkit.
+
+In the terminal, IPython uses prompt_toolkit to prompt the user for input.
+prompt_toolkit provides hooks to integrate with an external event loop.
+
+To integrate an event loop, define a function which runs the GUI event loop
+until there is input waiting for prompt_toolkit to process. There are two ways
+to detect this condition::
+
+    # Polling for input.
+    def inputhook(context):
+        while not context.input_is_ready():
+            # Replace this with the appropriate call for the event loop:
+            iterate_loop_once()
+
+    # Using a file descriptor to notify the event loop to stop.
+    def inputhook2(context):
+        fd = context.fileno()
+        # Replace the functions below with those for the event loop.
+        add_file_reader(fd, callback=stop_the_loop)
+        run_the_loop()
+
+Once you have defined this function, register it with IPython:
+
+.. currentmodule:: IPython.terminal.pt_inputhooks
+
+.. function:: register(name, inputhook)
+
+   Register the function *inputhook* as the event loop integration for the
+   GUI *name*. If ``name='foo'``, then the user can enable this integration
+   by running ``%gui foo``.
 
 
 Event loops in the kernel

docs/source/config/options/index.rst

@@ -6,4 +6,7 b' Any of the options listed here can be set in config files, at the'
 command line, or from inside IPython. See :ref:`setting_config` for
 details.
 
-.. include:: config-generated.txt
+.. toctree::
+
+   terminal
+   kernel

docs/source/whatsnew/version5.rst

@@ -5,71 +5,72 b''
 IPython 5.0
 ===========
 
-Released June, 2016
+Released July, 2016
 
-IPython 5.0 now uses ``prompt-toolkit`` for the command line interface, thus
-allowing real multi-line editing and syntactic coloration as you type.
+New terminal interface
+----------------------
 
+IPython 5 features a major upgrade to the terminal interface, bringing live
+syntax highlighting as you type, proper multiline editing and multiline paste,
+and tab completions that don't clutter up your history.
 
-When using IPython as a subprocess, like for emacs inferior-shell, IPython can
-be started with a ``--simple-prompt`` flag, which will bypass the prompt_toolkit
-input layer. In this mode, prompt color and many other features are
-disabled.
+.. image:: ../_images/ptshell_features.png
+    :alt: New terminal interface features
+    :align: center
+    :target: ../_images/ptshell_features.png
 
-Backwards incompatible changes
-------------------------------
-
-
-The `install_ext magic` function which was deprecated since 4.0 has now been deleted.
-You can still distribute and install extensions as packages on PyPI.
-
-Update IPython event triggering to ensure callback registration and
-unregistration will only affect the set of callbacks the *next* time that event is
-triggered. See :ghissue:`9447` and :ghpull:`9453`.
-
-This is a change to the existing semantics, wherein one callback registering a
-second callback when triggered for an event would previously be invoked for
-that same event.
-
-Integration with pydb has been removed since pydb development has been stopped
-since 2012, and pydb is not installable from PyPI.
-
-
-
-Replacement of readline in TerminalInteractiveShell and PDB
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+These features are provided by the Python library `prompt_toolkit
+<http://python-prompt-toolkit.readthedocs.io/en/stable/>`__, which replaces
+``readline`` throughout our terminal interface.
 
-IPython 5.0 now uses ``prompt_toolkit``. The
-``IPython.terminal.interactiveshell.TerminalInteractiveShell`` now uses
-``prompt_toolkit``. It is an almost complete rewrite, so many settings have
-thus changed or disappeared. The class keep the same name to avoid breaking
-user configuration for the options with names that are unchanged.
+Relying on this pure-Python, cross platform module also makes it simpler to
+install IPython. We have removed dependencies on ``pyreadline`` for Windows and
+``gnureadline`` for Mac.
 
-The usage of ``prompt_toolkit`` is accompanied by a complete removal of all
-code, using ``readline``. A particular effect of not using `readline` anymore
-is that `.inputrc` settings are note effective anymore. Options having similar
-effects have likely been replaced by a configuration option on IPython itself
-(e.g: vi input mode).
-
-The `PromptManager` class have been removed, and the prompt machinery simplified.
-See `TerminalInteractiveShell.prompts` configurable for how to setup your prompts.
-
-.. note::
-
-    During development and beta cycle, ``TerminalInteractiveShell`` was
-    temporarly moved to ``IPython.terminal.ptshell``.
-
-
-Most of the above remarks also affect `IPython.core.debugger.Pdb`, the `%debug`
-and `%pdb` magic which do not use readline anymore either.
+Backwards incompatible changes
+------------------------------
 
-The color handling has been slightly changed and is now exposed,
-in particular the colors of prompts and as you type
-highlighting can be affected by :
-``TerminalInteractiveShell.highlight_style``. With default
-configuration the ``--colors`` flag and ``%colors`` magic behavior
-should be mostly unchanged. See the `colors <termcolour>`_ section of
-our documentation
+- The ``%install_ext`` magic function, deprecated since 4.0, has now been deleted.
+  You can distribute and install extensions as packages on PyPI.
+- Callbacks registered while an event is being handled will now only be called
+  for subsequent events; previously they could be called for the current event.
+  Similarly, callbacks removed while handling an event *will* always get that
+  event. See :ghissue:`9447` and :ghpull:`9453`.
+- Integration with pydb has been removed since pydb development has been stopped
+  since 2012, and pydb is not installable from PyPI.
+- The ``autoedit_syntax`` option has apparently been broken for many years.
+  It has been removed.
+
+New terminal interface
+~~~~~~~~~~~~~~~~~~~~~~
+
+The overhaul of the terminal interface will probably cause a range of minor
+issues for existing users.
+This is inevitable for such a significant change, and we've done our best to
+minimise these issues.
+Some changes that we're aware of, with suggestions on how to handle them:
+
+IPython no longer uses readline configuration (``~/.inputrc``). We hope that
+the functionality you want (e.g. vi input mode) will be available by configuring
+IPython directly (see :doc:`/config/options/terminal`).
+If something's missing, please file an issue.
+
+The ``PromptManager`` class has been removed, and the prompt machinery simplified.
+See :ref:`custom_prompts` to customise prompts with the new machinery.
+
+:mod:`IPython.core.debugger` now provides a plainer interface.
+:mod:`IPython.terminal.debugger` contains the terminal debugger using
+prompt_toolkit.
+
+There are new options to configure the colours used in syntax highlighting.
+We have tried to integrate them with our classic  ``--colors`` option and
+``%colors`` magic, but there's a mismatch in possibilities, so some configurations
+may produce unexpected results. See :ref:`termcolour` for more information.
+
+The new interface is not compatible with Emacs 'inferior-shell' feature. To
+continue using this, add the ``--simple-prompt`` flag to the command Emacs
+runs. This flag disables most IPython features, relying on Emacs to provide
+things like tab completion.
 
 Provisional Changes
 -------------------
@@ -116,12 +117,6 b' widgets... As stated above this is nightly experimental feature with a lot of'
 it.
 
 
-Removed Feature
----------------
-
-- ``TerminalInteractiveShell.autoedit_syntax`` Has been broken for many years now
-  apparently. It has been removed.
-
 
 Deprecated Features
 -------------------