Misc

• Fix an issue where %recall would both succeeded and print an error message it failed. :ghpull:`12952`
• Drop support for NumPy 1.16 – practically has no effect beyond indicating in package metadata that we do not support it. :ghpull:`12937`

Debugger improvements

The debugger (and %debug magic) have been improved and can skip or hide frames originating from files that are not writable to the user, as these are less likely to be the source of errors, or be part of system files this can be a useful addition when debugging long errors.

In addition to the global skip_hidden True|False command, the debugger has gained finer grained control of predicates as to whether to a frame should be considered hidden. So far 3 predicates are available :

• tbhide: frames containing the local variable __tracebackhide__ set to True.
• readonly: frames originating from readonly files, set to False.
• ipython_internal: frames that are likely to be from IPython internal code, set to True.

You can toggle individual predicates during a session with

ipdb> skip_predicates readonly True

Read-only files will now be considered hidden frames.

You can call skip_predicates without arguments to see the states of current predicates:

ipdb> skip_predicates
current predicates:
    tbhide : True
    readonly : False
    ipython_internal : True

If all predicates are set to False, skip_hidden will practically have no effect. We attempt to warn you when all predicates are False.

Note that the readonly predicate may increase disk access as we check for file access permission for all frames on many command invocation, but is usually cached by operating systems. Let us know if you encounter any issues.

As the IPython debugger does not use the traitlets infrastructure for configuration, by editing your .pdbrc files and appending commands you would like to be executed just before entering the interactive prompt. For example:

# file : ~/.pdbrc
skip_predicates readonly True
skip_predicates tbhide False

Will hide read only frames by default and show frames marked with __tracebackhide__.

Thanks

Many thanks to all the contributors to this release you can find all individual contributions to this milestone on github.

Thanks as well to the D. E. Shaw group for sponsoring work on IPython and related libraries, in particular above mentioned improvements to the debugger.

Thanks

Many thanks to all the contributors to this release you can find all individual contributions to this milestone on github. In particular MrMino for responding to almost all new issues, and triaging many of the old ones, as well as takluyver, minrk, willingc for reacting quikly when we ran out of CI Hours.

Thanks as well to organisations, QuantStack (martinRenou and SylvainCorlay) for extracting matplotlib inline backend into its own package, and the D. E. Shaw group for sponsoring work on IPython and related libraries.

Thanks

We have a new team member that you should see more often on the IPython repository, Błażej Michalik (@MrMino) have been doing regular contributions to IPython, and spent time replying to many issues and guiding new users to the codebase; they now have triage permissions to the IPython repository and we'll work toward giving them more permission in the future.

Many thanks to all the contributors to this release you can find all individual contributions to this milestone on github.

Thanks as well to organisations, QuantStack for working on debugger compatibility for Xeus_python, and the D. E. Shaw group for sponsoring work on IPython and related libraries.

New "context" command in ipdb

It is now possible to change the number of lines shown in the backtrace information in ipdb using "context" command. :ghpull:`12826`

(thanks @MrMino, there are other improvement from them on master).

Other notable changes in IPython 7.21

• Fix some issues on new osx-arm64 :ghpull:`12804`, :ghpull:`12807`.
• Compatibility with Xeus-Python for debugger protocol, :ghpull:`12809`
• Misc docs fixes for compatibility and uniformity with Numpydoc. :ghpull:`12824`

Thanks

Many thanks to all the contributors to this release you can find all individual contribution to this milestone on github.

API Changes

Change of API and exposed objects automatically detected using frappuccino

The following items are new since 7.16.0:

+ IPython.core.interactiveshell.InteractiveShell.get_local_scope(self, stack_depth)

The following signatures differ since 7.16.0:

- IPython.core.interactiveshell.InteractiveShell.run_cell_async(self, raw_cell, store_history=False, silent=False, shell_futures=True)
+ IPython.core.interactiveshell.InteractiveShell.run_cell_async(self, raw_cell, store_history=False, silent=False, shell_futures=True, *, transformed_cell=None, preprocessing_exc_tuple=None)

- IPython.core.interactiveshell.InteractiveShell.should_run_async(self, raw_cell)
+ IPython.core.interactiveshell.InteractiveShell.should_run_async(self, raw_cell, *, transformed_cell=None, preprocessing_exc_tuple=None)

- IPython.terminal.debugger.TerminalPdb.pt_init(self)
+ IPython.terminal.debugger.TerminalPdb.pt_init(self, pt_session_options=None)

This method was added:

+ IPython.core.interactiveshell.InteractiveShell.get_local_scope

Which is now also present on subclasses:

+ IPython.terminal.embed.InteractiveShellEmbed.get_local_scope
+ IPython.terminal.interactiveshell.TerminalInteractiveShell.get_local_scope

Misc Noticeable changes:

• Exceptions are now (re)raised when running notebooks via the :magic:`%run`, helping to catch issues in workflows and pipelines. :ghpull:`12301`
• Fix inputhook for qt 5.15.0 :ghpull:`12355`
• Fix wx inputhook :ghpull:`12375`
• Add handling for malformed pathext env var (Windows) :ghpull:`12367`
• use $SHELL in system_piped :ghpull:`12360` for uniform behavior with ipykernel.

Reproducible Build

IPython 7.15 reproducible build did not work, so we try again this month :ghpull:`12358`.

API Changes

Change of API and exposed objects automatically detected using frappuccino (still in beta):

The following items are new and mostly related to understanding __tracebackbide__:

+ IPython.core.debugger.Pdb.do_down(self, arg)
+ IPython.core.debugger.Pdb.do_skip_hidden(self, arg)
+ IPython.core.debugger.Pdb.do_up(self, arg)
+ IPython.core.debugger.Pdb.hidden_frames(self, stack)
+ IPython.core.debugger.Pdb.stop_here(self, frame)

The following items have been removed:

- IPython.core.debugger.Pdb.new_do_down
- IPython.core.debugger.Pdb.new_do_up

Those were implementation details.

Misc Noticeable changes:

• Long completion name have better elision in terminal :ghpull:`12284`
• I've started to test on Python 3.9 :ghpull:`12307` and fix some errors.
• Hi DPI scaling of figures when using qt eventloop :ghpull:`12314`
• Document the ability to have systemwide configuration for IPython. :ghpull:`12328`
• Fix issues with input autoformatting :ghpull:`12336`
• IPython.core.debugger.Pdb is now interruptible (:ghpull:`12168`, in 7.14 but forgotten in release notes)
• Video HTML attributes (:ghpull:`12212`, in 7.14 but forgotten in release notes)

Reproducible Build

Starting with IPython 7.15, I am attempting to provide reproducible builds, that is to say you should be able from the source tree to generate an sdist and wheel that are identical byte for byte with the publish version on PyPI.

I've only tested on a couple of machines so far and the process is relatively straightforward, so this mean that IPython not only have a deterministic build process, but also I have either removed, or put under control all effects of the build environments on the final artifact. I encourage you to attempt the build process on your machine as documented in :ref:`core_developer_guide` and let me know if you do not obtain an identical artifact.

While reproducible builds is critical to check that the supply chain of (open source) software has not been compromised, it can also help to speedup many of the build processes in large environment (conda, apt...) by allowing better caching of intermediate build steps.

Learn more on https://reproducible-builds.org/. Reflections on trusting trust is also one of the cornerstone and recommended reads on this subject.

NEP29: Last version to support Python 3.6

IPython 7.15 will be the Last IPython version to officially support Python 3.6, as stated by NumPy Enhancement Proposal 29. Starting with next minor version of IPython I may stop testing on Python 3.6 and may stop publishing release artifacts that install on Python 3.6

Highlighted features

Highlighted features are not new, but seem to not be widely known, this section will help you discover in more narrative form what you can do with IPython.

Important changes:

• Fix compatibility with Sphinx 3+ :ghpull:`12235`
• Remove deprecated matplotlib parameter usage, compatibility with matplotlib 3.3+ :122250

Misc Changes

• set .py extension when editing current buffer in vi/emacs. :ghpull:`12167`
• support for unicode identifiers in ?/?? :ghpull:`12208`
• add extra options to the Video Rich objects :ghpull:`12212`
• add pretty-printing to SimpleNamespace :ghpull:`12230`

IPython.core.debugger.Pdb is now interruptible

A KeyboardInterrupt will now interrupt IPython's extended debugger, in order to make Jupyter able to interrupt it. (:ghpull:`12168`)

Video HTML attributes

Add an option to IPython.display.Video to change the attributes of the HTML display of the video (:ghpull:`12212`)

Pending deprecated imports

Many object present in IPython.core.display are there for internal use only, and should already been imported from IPython.display by users and external libraries. Trying to import those from IPython.core.display is still possible but will trigger a deprecation warning in later versions of IPython and will become errors in the future.

This will simplify compatibility with other Python kernels (like Xeus-Python), and simplify code base.

Stop Support for Python 3.5 – Adopt NEP 29

IPython has decided to follow the informational NEP 29 which layout a clear policy as to which version of (C)Python and NumPy are supported.

We thus dropped support for Python 3.5, and cleaned up a number of code path that were Python-version dependant. If you are on 3.5 or earlier pip should automatically give you the latest compatible version of IPython so you do not need to pin to a given version.

Support for Prompt Toolkit 3.0

Prompt Toolkit 3.0 was release a week before IPython 7.10 and introduces a few breaking changes. We believe IPython 7.10 should be compatible with both Prompt Toolkit 2.x and 3.x, though it has not been extensively tested with 3.x so please report any issues.

Prompt Rendering Performance improvements

Pull Request :ghpull:`11933` introduced an optimisation in the prompt rendering logic that should decrease the resource usage of IPython when using the _default_ configuration but could potentially introduce a regression of functionalities if you are using a custom prompt.

We know assume if you haven't changed the default keybindings that the prompt will not change during the duration of your input – which is for example not true when using vi insert mode that switches between [ins] and [nor] for the current mode.

If you are experiencing any issue let us know.

Code autoformatting

The IPython terminal can now auto format your code just before entering a new line or executing a command. To do so use the --TerminalInteractiveShell.autoformatter option and set it to 'black'; if black is installed IPython will use black to format your code when possible.

IPython cannot always properly format your code; in particular it will auto formatting with black will only work if:

• Your code does not contains magics or special python syntax.
• There is no code after your cursor.

The Black API is also still in motion; so this may not work with all versions of black.

It should be possible to register custom formatter, though the API is till in flux.

Arbitrary Mimetypes Handing in Terminal (Aka inline images in terminal)

When using IPython terminal it is now possible to register function to handle arbitrary mimetypes. While rendering non-text based representation was possible in many jupyter frontend; it was not possible in terminal IPython, as usually terminal are limited to displaying text. As many terminal these days provide escape sequences to display non-text; bringing this loved feature to IPython CLI made a lot of sens. This functionality will not only allow inline images; but allow opening of external program; for example mplayer to "display" sound files.

So far only the hooks necessary for this are in place, but no default mime renderers added; so inline images will only be available via extensions. We will progressively enable these features by default in the next few releases, and contribution is welcomed.

We welcome any feedback on the API. See :ref:`shell_mimerenderer` for more informations.

This is originally based on work form in :ghpull:`10610` from @stephanh42 started over two years ago, and still a lot need to be done.

MISC

• Completions can define their own ordering :ghpull:`11855`
• Enable Plotting in the same cell than the one that import matplotlib :ghpull:`11916`
• Allow to store and restore multiple variables at once :ghpull:`11930`

You can see all pull-requests for this release.

API Changes

Change of API and exposed objects automatically detected using frappuccino (still in beta):

The following items are new in IPython 7.10:

+ IPython.terminal.shortcuts.reformat_text_before_cursor(buffer, document, shell)
+ IPython.terminal.interactiveshell.PTK3
+ IPython.terminal.interactiveshell.black_reformat_handler(text_before_cursor)
+ IPython.terminal.prompts.RichPromptDisplayHook.write_format_data(self, format_dict, md_dict='None')

The following items have been removed in 7.10:

- IPython.lib.pretty.DICT_IS_ORDERED

The following signatures differ between versions:

- IPython.extensions.storemagic.restore_aliases(ip)
+ IPython.extensions.storemagic.restore_aliases(ip, alias='None')

Special Thanks

• @stephanh42 who started the work on inline images in terminal 2 years ago
• @augustogoulart who spent a lot of time triaging issues and responding to users.
• @con-f-use who is my (@Carreau) first sponsor on GitHub, as a reminder if you like IPython, Jupyter and many other library of the SciPy stack you can donate to numfocus.org non profit

Expose Pdb API

Expose the built-in pdb.Pdb API. Pdb constructor arguments are generically exposed, regardless of python version. Newly exposed arguments:

• skip - Python 3.1+
• nosiginnt - Python 3.2+
• readrc - Python 3.6+

Try it out:

from IPython.terminal.debugger import TerminalPdb
pdb = TerminalPdb(skip=["skipthismodule"])

See :ghpull:`11840`

Prepare migration to pytest (instead of nose) for testing

Most of the work between 7.5 and 7.6 was to prepare the migration from our testing framework to pytest. Most of the test suite should now work by simply issuing pytest from the root of the repository.

The migration to pytest is just at its beginning. Many of our test still rely on IPython-specific plugins for nose using pytest (doctest using IPython syntax is one example of this where test appear as "passing", while no code has been ran). Many test also need to be updated like yield-test to be properly parametrized tests.

Migration to pytest allowed me to discover a number of issues in our test suite; which was hiding a number of subtle issues – or not actually running some of the tests in our test suite – I have thus corrected many of those; like improperly closed resources; or used of deprecated features. I also made use of the pytest --durations=... to find some of our slowest test and speed them up (our test suite can now be up to 10% faster). Pytest as also a variety of plugins and flags which will make the code quality of IPython and the testing experience better.

Misc

We skipped the release of 7.6 at the end of May, but will attempt to get back on schedule. We are starting to think about making introducing backward incompatible change and start the 8.0 series.

Special Thanks to Gabriel (@gpotter2 on GitHub), who among other took care many of the remaining task for 7.4 and 7.5, like updating the website.

Unicode name completions

Previously, we provided completion for a unicode name with its relative symbol. With this, now IPython provides complete suggestions to unicode name symbols.

As on the PR, if user types \LAT<tab>, IPython provides a list of possible completions. In this case, it would be something like:

'LATIN CAPITAL LETTER A',
'LATIN CAPITAL LETTER B',
'LATIN CAPITAL LETTER C',
'LATIN CAPITAL LETTER D',
....

This help to type unicode character that do not have short latex aliases, and have long unicode names. for example Ͱ, \GREEK CAPITAL LETTER HETA.

This feature was contributed by Luciana Marques :ghpull:`11583`.

Make audio normalization optional

Added 'normalize' argument to IPython.display.Audio. This argument applies when audio data is given as an array of samples. The default of normalize=True preserves prior behavior of normalizing the audio to the maximum possible range. Setting to False disables normalization.

Miscellaneous

• Fix improper acceptation of return outside of functions. :ghpull:`11641`.
• Fixed PyQt 5.11 backwards incompatibility causing sip import failure. :ghpull:`11613`.
• Fix Bug where type? would crash IPython. :ghpull:`1608`.
• Allow to apply @needs_local_scope to cell magics for convenience. :ghpull:`11542`.

Added ability to show subclasses when using pinfo and other utilities

When using ?/?? on a class, IPython will now list the first 10 subclasses.

Special Thanks to Chris Mentzel of the Moore Foundation for this feature. Chris is one of the people who played a critical role in IPython/Jupyter getting funding.

We are grateful for all the help Chris has given us over the years, and we're now proud to have code contributed by Chris in IPython.

OSMagics.cd_force_quiet configuration option

You can set this option to force the %cd magic to behave as if -q was passed:

In [1]: cd /
/

In [2]: %config OSMagics.cd_force_quiet = True

In [3]: cd /tmp

In [4]:

See :ghpull:`11491`

In vi editing mode, whether the prompt includes the current vi mode can now be configured

Set the TerminalInteractiveShell.prompt_includes_vi_mode to a boolean value (default: True) to control this feature. See :ghpull:`11492`

Unicode Completion update

In IPython 7.1 the Unicode completion map has been updated and synchronized with the Julia language.

Added and removed character characters:

\jmath (ȷ), \\underleftrightarrow (U+034D, combining) have been added, while \\textasciicaron have been removed

Some sequences have seen their prefix removed:

• 6 characters \text...<tab> should now be inputed with \...<tab> directly,
• 45 characters \Elz...<tab> should now be inputed with \...<tab> directly,
• 65 characters \B...<tab> should now be inputed with \...<tab> directly,
• 450 characters \m...<tab> should now be inputed with \...<tab> directly,

Some sequences have seen their prefix shortened:

• 5 characters \mitBbb...<tab> should now be inputed with \bbi...<tab> directly,
• 52 characters \mit...<tab> should now be inputed with \i...<tab> directly,
• 216 characters \mbfit...<tab> should now be inputed with \bi...<tab> directly,
• 222 characters \mbf...<tab> should now be inputed with \b...<tab> directly,

A couple of characters had their sequence simplified:

• ð, type \dh<tab>, instead of \eth<tab>
• ħ, type \hbar<tab>, instead of \Elzxh<tab>
• ɸ, type \ltphi<tab>, instead of \textphi<tab>
• ϴ, type \varTheta<tab>, instead of \textTheta<tab>
• ℇ, type \eulermascheroni<tab>, instead of \Eulerconst<tab>
• ℎ, type \planck<tab>, instead of \Planckconst<tab>
• U+0336 (COMBINING LONG STROKE OVERLAY), type \strike<tab>, instead of \Elzbar<tab>.

A couple of sequences have been updated:

• \varepsilon now gives ɛ (GREEK SMALL LETTER EPSILON) instead of ε (GREEK LUNATE EPSILON SYMBOL),
• \underbar now gives U+0331 (COMBINING MACRON BELOW) instead of U+0332 (COMBINING LOW LINE).

Prompt Toolkit 2.0

IPython 7.0+ now uses prompt_toolkit 2.0. If you still need to use an earlier prompt_toolkit version, you may need to pin IPython to <7.0.

Autowait: Asynchronous REPL

Staring with IPython 7.0 on Python 3.6+, IPython can automatically await top level code. You should not need to access an event loop or runner yourself. To learn more, read the :ref:`autoawait` section of our docs, see :ghpull:`11265`, or try the following code:

Python 3.6.0
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import aiohttp
   ...: result = aiohttp.get('https://api.github.com')

In [2]: response = await result
<pause for a few 100s ms>

In [3]: await response.json()
Out[3]:
{'authorizations_url': 'https://api.github.com/authorizations',
 'code_search_url': 'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}',
...
}

Integration is by default with asyncio, but other libraries can be configured -- like curio or trio -- to improve concurrency in the REPL:

In [1]: %autoawait trio

In [2]: import trio

In [3]: async def child(i):
   ...:     print("   child %s goes to sleep"%i)
   ...:     await trio.sleep(2)
   ...:     print("   child %s wakes up"%i)

In [4]: print('parent start')
   ...: async with trio.open_nursery() as n:
   ...:     for i in range(3):
   ...:         n.spawn(child, i)
   ...: print('parent end')
parent start
   child 2 goes to sleep
   child 0 goes to sleep
   child 1 goes to sleep
   <about 2 seconds pause>
   child 2 wakes up
   child 1 wakes up
   child 0 wakes up
parent end

See :ref:`autoawait` for more information.

Asynchronous code in a Notebook interface or any other frontend using the Jupyter Protocol will require further updates to the IPykernel package.

Autoreload Improvement

The magic :magic:`%autoreload 2 <autoreload>` now captures new methods added to classes. Earlier, only methods existing as of the initial import were being tracked and updated.

This new feature helps dual environment development - Jupyter+IDE - where the code gradually moves from notebook cells to package files as it gets structured.

Example: An instance of the class MyClass will be able to access the method cube() after it is uncommented and the file file1.py is saved on disk.

# notebook

from mymodule import MyClass
first = MyClass(5)

# mymodule/file1.py

class MyClass:

    def __init__(self, a=10):
        self.a = a

    def square(self):
        print('compute square')
        return self.a*self.a

    # def cube(self):
    #     print('compute cube')
    #     return self.a*self.a*self.a

Misc

The autoindent feature that was deprecated in 5.x was re-enabled and un-deprecated in :ghpull:`11257`

Make :magic:`%run -n -i ... <run>` work correctly. Earlier, if :magic:`%run` was passed both arguments, -n would be silently ignored. See :ghpull:`10308`

The :cellmagic:`%%script` (as well as :cellmagic:`%%bash`, :cellmagic:`%%ruby`... ) cell magics now raise by default if the return code of the given code is non-zero (thus halting execution of further cells in a notebook). The behavior can be disable by passing the --no-raise-error flag.

Deprecations

A couple of unused functions and methods have been deprecated and will be removed in future versions:

• IPython.utils.io.raw_print_err
• IPython.utils.io.raw_print

Backwards incompatible changes

• The API for transforming input before it is parsed as Python code has been completely redesigned: any custom input transformations will need to be rewritten. See :doc:`/config/inputtransforms` for details of the new API.