autoawait.rst
210 lines
| 7.4 KiB
| text/x-rst
|
RstLexer
Matthias Bussonnier
|
r24466 | .. _autoawait: | ||
Matthias Bussonnier
|
r24463 | |||
Asynchronous in REPL: Autoawait | ||||
=============================== | ||||
Matthias Bussonnier
|
r24475 | Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the | ||
ability to run asynchronous code from the REPL. Constructs which are | ||||
Matthias Bussonnier
|
r24463 | :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython. | ||
Matthias Bussonnier
|
r24475 | The example given here are for terminal IPython, running async code in a | ||
notebook interface or any other frontend using the Jupyter protocol will need to | ||||
use a newer version of IPykernel. The details of how async code runs in | ||||
IPykernel will differ between IPython, IPykernel and their versions. | ||||
Matthias Bussonnier
|
r24480 | When a supported library is used, IPython will automatically `await` Futures and | ||
Coroutines in the REPL. This will happen if an :ref:`await <await>` (or any | ||||
other async constructs like async-with, async-for) is use at top level scope, or | ||||
if any structure valid only in `async def | ||||
Matthias Bussonnier
|
r24463 | <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function | ||
context are present. For example, the following being a syntax error in the | ||||
Python REPL:: | ||||
Python 3.6.0 | ||||
[GCC 4.2.1] | ||||
Type "help", "copyright", "credits" or "license" for more information. | ||||
>>> import aiohttp | ||||
>>> result = aiohttp.get('https://api.github.com') | ||||
>>> response = await result | ||||
File "<stdin>", line 1 | ||||
response = await result | ||||
^ | ||||
SyntaxError: invalid syntax | ||||
Should behave as expected in the IPython REPL:: | ||||
Python 3.6.0 | ||||
Type 'copyright', 'credits' or 'license' for more information | ||||
Matthias Bussonnier
|
r24475 | IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help. | ||
Matthias Bussonnier
|
r24463 | |||
In [1]: import aiohttp | ||||
...: result = aiohttp.get('https://api.github.com') | ||||
In [2]: response = await result | ||||
<pause for a few 100s ms> | ||||
In [3]: await response.json() | ||||
Out[3]: | ||||
{'authorizations_url': 'https://api.github.com/authorizations', | ||||
'code_search_url': 'https://api.github.com/search/code?q={query}...', | ||||
... | ||||
} | ||||
You can use the ``c.InteractiveShell.autoawait`` configuration option and set it | ||||
to :any:`False` to deactivate automatic wrapping of asynchronous code. You can also | ||||
use the :magic:`%autoawait` magic to toggle the behavior at runtime:: | ||||
In [1]: %autoawait False | ||||
In [2]: %autoawait | ||||
IPython autoawait is `Off`, and set to use `IPython.core.interactiveshell._asyncio_runner` | ||||
By default IPython will assume integration with Python's provided | ||||
:mod:`asyncio`, but integration with other libraries is provided. In particular | ||||
Matthias Bussonnier
|
r24475 | we provide experimental integration with the ``curio`` and ``trio`` library, the | ||
later one being necessary if you require the ability to do nested call of | ||||
IPython's ``embed()`` functionality. | ||||
Matthias Bussonnier
|
r24463 | |||
You can switch current integration by using the | ||||
``c.InteractiveShell.loop_runner`` option or the ``autoawait <name | ||||
integration>`` magic. | ||||
For example:: | ||||
In [1]: %autoawait trio | ||||
In [2]: import trio | ||||
In [3]: async def child(i): | ||||
...: print(" child %s goes to sleep"%i) | ||||
...: await trio.sleep(2) | ||||
...: print(" child %s wakes up"%i) | ||||
In [4]: print('parent start') | ||||
...: async with trio.open_nursery() as n: | ||||
...: for i in range(5): | ||||
...: n.spawn(child, i) | ||||
...: print('parent end') | ||||
parent start | ||||
child 2 goes to sleep | ||||
child 0 goes to sleep | ||||
child 3 goes to sleep | ||||
child 1 goes to sleep | ||||
child 4 goes to sleep | ||||
<about 2 seconds pause> | ||||
child 2 wakes up | ||||
child 1 wakes up | ||||
child 0 wakes up | ||||
child 3 wakes up | ||||
child 4 wakes up | ||||
parent end | ||||
In the above example, ``async with`` at top level scope is a syntax error in | ||||
Python. | ||||
Using this mode can have unexpected consequences if used in interaction with | ||||
other features of IPython and various registered extensions. In particular if you | ||||
are a direct or indirect user of the AST transformers, these may not apply to | ||||
your code. | ||||
Matthias Bussonnier
|
r24475 | When Using command line IPython, the default loop (or runner) does not process | ||
in the background, so top level asynchronous code must finish for the REPL to | ||||
allow you to enter more code. As with usual Python semantic, the awaitables are | ||||
started only when awaited for the first time. That is to say, in first example, | ||||
no network request is done between ``In[1]`` and ``In[2]``. | ||||
Matthias Bussonnier
|
r24463 | |||
Matthias Bussonnier
|
r24480 | Effects on IPython.embed() | ||
========================== | ||||
Matthias Bussonnier
|
r24481 | IPython core being asynchronous, the use of ``IPython.embed()`` will now require | ||
a loop to run. In order to allow ``IPython.embed()`` to be nested, as most event | ||||
loops can't be nested, ``IPython.embed()`` default to a pseudo-synchronous mode, | ||||
where async code is not allowed. This mode is available in classical IPython | ||||
using ``%autoawait sync`` | ||||
This affect the ability to nest ``IPython.embed()`` which may | ||||
Matthias Bussonnier
|
r24480 | require you to install alternate IO libraries like ``curio`` and ``trio`` | ||
Matthias Bussonnier
|
r24463 | Internals | ||
========= | ||||
Matthias Bussonnier
|
r24475 | As running asynchronous code is not supported in interactive REPL (as of Python | ||
3.7) we have to rely to a number of complex workaround to allow this to happen. | ||||
It is interesting to understand how this works in order to comprehend potential | ||||
Matthias Bussonnier
|
r24463 | bugs, or provide a custom runner. | ||
Among the many approaches that are at our disposition, we find only one that | ||||
Matthias Bussonnier
|
r24475 | suited out need. Under the hood we use the code object from a async-def function | ||
and run it in global namespace after modifying it to not create a new | ||||
``locals()`` scope:: | ||||
Matthias Bussonnier
|
r24463 | |||
async def inner_async(): | ||||
locals().update(**global_namespace) | ||||
# | ||||
# here is user code | ||||
# | ||||
return last_user_statement | ||||
codeobj = modify(inner_async.__code__) | ||||
coroutine = eval(codeobj, user_ns) | ||||
display(loop_runner(coroutine)) | ||||
The first thing you'll notice is that unlike classical ``exec``, there is only | ||||
Matthias Bussonnier
|
r24475 | one namespace. Second, user code runs in a function scope, and not a module | ||
Matthias Bussonnier
|
r24463 | scope. | ||
On top of the above there are significant modification to the AST of | ||||
``function``, and ``loop_runner`` can be arbitrary complex. So there is a | ||||
significant overhead to this kind of code. | ||||
By default the generated coroutine function will be consumed by Asyncio's | ||||
``loop_runner = asyncio.get_evenloop().run_until_complete()`` method. It is | ||||
though possible to provide your own. | ||||
A loop runner is a *synchronous* function responsible from running a coroutine | ||||
object. | ||||
The runner is responsible from ensuring that ``coroutine`` run to completion, | ||||
and should return the result of executing the coroutine. Let's write a | ||||
runner for ``trio`` that print a message when used as an exercise, ``trio`` is | ||||
special as it usually prefer to run a function object and make a coroutine by | ||||
itself, we can get around this limitation by wrapping it in an async-def without | ||||
parameters and passing this value to ``trio``:: | ||||
In [1]: import trio | ||||
...: from types import CoroutineType | ||||
...: | ||||
...: def trio_runner(coro:CoroutineType): | ||||
...: print('running asynchronous code') | ||||
...: async def corowrap(coro): | ||||
...: return await coro | ||||
...: return trio.run(corowrap, coro) | ||||
We can set it up by passing it to ``%autoawait``:: | ||||
In [2]: %autoawait trio_runner | ||||
In [3]: async def async_hello(name): | ||||
...: await trio.sleep(1) | ||||
...: print(f'Hello {name} world !') | ||||
...: await trio.sleep(1) | ||||
In [4]: await async_hello('async') | ||||
running asynchronous code | ||||
Hello async world ! | ||||
Asynchronous programming in python (and in particular in the REPL) is still a | ||||
Matthias Bussonnier
|
r24475 | relatively young subject. We expect some code to not behave as you expect, so | ||
feel free to contribute improvements to this codebase and give us feedback. | ||||