##// END OF EJS Templates
Merge pull request #13682 from graingert/patch-1...
Blazej Michalik -
r27665:2d7bfdeb merge
parent child Browse files
Show More
@@ -1,319 +1,319 b''
1 .. _autoawait:
1 .. _autoawait:
2
2
3 Asynchronous in REPL: Autoawait
3 Asynchronous in REPL: Autoawait
4 ===============================
4 ===============================
5
5
6 .. note::
6 .. note::
7
7
8 This feature is experimental and behavior can change between python and
8 This feature is experimental and behavior can change between python and
9 IPython version without prior deprecation.
9 IPython version without prior deprecation.
10
10
11 Starting with IPython 7.0, and when using Python 3.6 and above, IPython offer the
11 Starting with IPython 7.0, and when using Python 3.6 and above, IPython offer the
12 ability to run asynchronous code from the REPL. Constructs which are
12 ability to run asynchronous code from the REPL. Constructs which are
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14
14
15 The examples given here are for terminal IPython, running async code in a
15 The examples given here are for terminal IPython, running async code in a
16 notebook interface or any other frontend using the Jupyter protocol needs
16 notebook interface or any other frontend using the Jupyter protocol needs
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 will differ between IPython, IPykernel and their versions.
18 will differ between IPython, IPykernel and their versions.
19
19
20 When a supported library is used, IPython will automatically allow Futures and
20 When a supported library is used, IPython will automatically allow Futures and
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 <await>` (or any other async constructs like async-with, async-for) is used at
22 <await>` (or any other async constructs like async-with, async-for) is used at
23 top level scope, or if any structure valid only in `async def
23 top level scope, or if any structure valid only in `async def
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 context are present. For example, the following being a syntax error in the
25 context are present. For example, the following being a syntax error in the
26 Python REPL::
26 Python REPL::
27
27
28 Python 3.6.0
28 Python 3.6.0
29 [GCC 4.2.1]
29 [GCC 4.2.1]
30 Type "help", "copyright", "credits" or "license" for more information.
30 Type "help", "copyright", "credits" or "license" for more information.
31 >>> import aiohttp
31 >>> import aiohttp
32 >>> session = aiohttp.ClientSession()
32 >>> session = aiohttp.ClientSession()
33 >>> result = session.get('https://api.github.com')
33 >>> result = session.get('https://api.github.com')
34 >>> response = await result
34 >>> response = await result
35 File "<stdin>", line 1
35 File "<stdin>", line 1
36 response = await result
36 response = await result
37 ^
37 ^
38 SyntaxError: invalid syntax
38 SyntaxError: invalid syntax
39
39
40 Should behave as expected in the IPython REPL::
40 Should behave as expected in the IPython REPL::
41
41
42 Python 3.6.0
42 Python 3.6.0
43 Type 'copyright', 'credits' or 'license' for more information
43 Type 'copyright', 'credits' or 'license' for more information
44 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
45
45
46 In [1]: import aiohttp
46 In [1]: import aiohttp
47 ...: session = aiohttp.ClientSession()
47 ...: session = aiohttp.ClientSession()
48 ...: result = session.get('https://api.github.com')
48 ...: result = session.get('https://api.github.com')
49
49
50 In [2]: response = await result
50 In [2]: response = await result
51 <pause for a few 100s ms>
51 <pause for a few 100s ms>
52
52
53 In [3]: await response.json()
53 In [3]: await response.json()
54 Out[3]:
54 Out[3]:
55 {'authorizations_url': 'https://api.github.com/authorizations',
55 {'authorizations_url': 'https://api.github.com/authorizations',
56 'code_search_url': 'https://api.github.com/search/code?q={query}...',
56 'code_search_url': 'https://api.github.com/search/code?q={query}...',
57 ...
57 ...
58 }
58 }
59
59
60
60
61 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
61 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
62 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
62 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
63 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
63 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
64
64
65 In [1]: %autoawait False
65 In [1]: %autoawait False
66
66
67 In [2]: %autoawait
67 In [2]: %autoawait
68 IPython autoawait is `Off`, and set to use `asyncio`
68 IPython autoawait is `Off`, and set to use `asyncio`
69
69
70
70
71
71
72 By default IPython will assume integration with Python's provided
72 By default IPython will assume integration with Python's provided
73 :mod:`asyncio`, but integration with other libraries is provided. In particular
73 :mod:`asyncio`, but integration with other libraries is provided. In particular
74 we provide experimental integration with the ``curio`` and ``trio`` library.
74 we provide experimental integration with the ``curio`` and ``trio`` library.
75
75
76 You can switch the current integration by using the
76 You can switch the current integration by using the
77 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
77 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
78 integration>`` magic.
78 integration>`` magic.
79
79
80 For example::
80 For example::
81
81
82 In [1]: %autoawait trio
82 In [1]: %autoawait trio
83
83
84 In [2]: import trio
84 In [2]: import trio
85
85
86 In [3]: async def child(i):
86 In [3]: async def child(i):
87 ...: print(" child %s goes to sleep"%i)
87 ...: print(" child %s goes to sleep"%i)
88 ...: await trio.sleep(2)
88 ...: await trio.sleep(2)
89 ...: print(" child %s wakes up"%i)
89 ...: print(" child %s wakes up"%i)
90
90
91 In [4]: print('parent start')
91 In [4]: print('parent start')
92 ...: async with trio.open_nursery() as n:
92 ...: async with trio.open_nursery() as n:
93 ...: for i in range(5):
93 ...: for i in range(5):
94 ...: n.spawn(child, i)
94 ...: n.spawn(child, i)
95 ...: print('parent end')
95 ...: print('parent end')
96 parent start
96 parent start
97 child 2 goes to sleep
97 child 2 goes to sleep
98 child 0 goes to sleep
98 child 0 goes to sleep
99 child 3 goes to sleep
99 child 3 goes to sleep
100 child 1 goes to sleep
100 child 1 goes to sleep
101 child 4 goes to sleep
101 child 4 goes to sleep
102 <about 2 seconds pause>
102 <about 2 seconds pause>
103 child 2 wakes up
103 child 2 wakes up
104 child 1 wakes up
104 child 1 wakes up
105 child 0 wakes up
105 child 0 wakes up
106 child 3 wakes up
106 child 3 wakes up
107 child 4 wakes up
107 child 4 wakes up
108 parent end
108 parent end
109
109
110
110
111 In the above example, ``async with`` at top level scope is a syntax error in
111 In the above example, ``async with`` at top level scope is a syntax error in
112 Python.
112 Python.
113
113
114 Using this mode can have unexpected consequences if used in interaction with
114 Using this mode can have unexpected consequences if used in interaction with
115 other features of IPython and various registered extensions. In particular if
115 other features of IPython and various registered extensions. In particular if
116 you are a direct or indirect user of the AST transformers, these may not apply
116 you are a direct or indirect user of the AST transformers, these may not apply
117 to your code.
117 to your code.
118
118
119 When using command line IPython, the default loop (or runner) does not process
119 When using command line IPython, the default loop (or runner) does not process
120 in the background, so top level asynchronous code must finish for the REPL to
120 in the background, so top level asynchronous code must finish for the REPL to
121 allow you to enter more code. As with usual Python semantics, the awaitables are
121 allow you to enter more code. As with usual Python semantics, the awaitables are
122 started only when awaited for the first time. That is to say, in first example,
122 started only when awaited for the first time. That is to say, in first example,
123 no network request is done between ``In[1]`` and ``In[2]``.
123 no network request is done between ``In[1]`` and ``In[2]``.
124
124
125
125
126 Effects on IPython.embed()
126 Effects on IPython.embed()
127 --------------------------
127 --------------------------
128
128
129 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
129 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
130 a loop to run. By default IPython will use a fake coroutine runner which should
130 a loop to run. By default IPython will use a fake coroutine runner which should
131 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
131 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
132 :magic:`%autoawait` feature when using IPython embed.
132 :magic:`%autoawait` feature when using IPython embed.
133
133
134 You can set a coroutine runner explicitly for ``embed()`` if you want to run
134 You can set a coroutine runner explicitly for ``embed()`` if you want to run
135 asynchronous code, though the exact behavior is undefined.
135 asynchronous code, though the exact behavior is undefined.
136
136
137 Effects on Magics
137 Effects on Magics
138 -----------------
138 -----------------
139
139
140 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
140 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
141 yet been updated to work with asynchronous code and will raise syntax errors
141 yet been updated to work with asynchronous code and will raise syntax errors
142 when trying to use top-level ``await``. We welcome any contribution to help fix
142 when trying to use top-level ``await``. We welcome any contribution to help fix
143 those, and extra cases we haven't caught yet. We hope for better support in Core
143 those, and extra cases we haven't caught yet. We hope for better support in Core
144 Python for top-level Async code.
144 Python for top-level Async code.
145
145
146 Internals
146 Internals
147 ---------
147 ---------
148
148
149 As running asynchronous code is not supported in interactive REPL (as of Python
149 As running asynchronous code is not supported in interactive REPL (as of Python
150 3.7) we have to rely to a number of complex workarounds and heuristics to allow
150 3.7) we have to rely to a number of complex workarounds and heuristics to allow
151 this to happen. It is interesting to understand how this works in order to
151 this to happen. It is interesting to understand how this works in order to
152 comprehend potential bugs, or provide a custom runner.
152 comprehend potential bugs, or provide a custom runner.
153
153
154 Among the many approaches that are at our disposition, we find only one that
154 Among the many approaches that are at our disposition, we find only one that
155 suited out need. Under the hood we use the code object from a async-def function
155 suited out need. Under the hood we use the code object from a async-def function
156 and run it in global namespace after modifying it to not create a new
156 and run it in global namespace after modifying it to not create a new
157 ``locals()`` scope::
157 ``locals()`` scope::
158
158
159 async def inner_async():
159 async def inner_async():
160 locals().update(**global_namespace)
160 locals().update(**global_namespace)
161 #
161 #
162 # here is user code
162 # here is user code
163 #
163 #
164 return last_user_statement
164 return last_user_statement
165 codeobj = modify(inner_async.__code__)
165 codeobj = modify(inner_async.__code__)
166 coroutine = eval(codeobj, user_ns)
166 coroutine = eval(codeobj, user_ns)
167 display(loop_runner(coroutine))
167 display(loop_runner(coroutine))
168
168
169
169
170
170
171 The first thing you'll notice is that unlike classical ``exec``, there is only
171 The first thing you'll notice is that unlike classical ``exec``, there is only
172 one namespace. Second, user code runs in a function scope, and not a module
172 one namespace. Second, user code runs in a function scope, and not a module
173 scope.
173 scope.
174
174
175 On top of the above there are significant modification to the AST of
175 On top of the above there are significant modification to the AST of
176 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
176 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
177 significant overhead to this kind of code.
177 significant overhead to this kind of code.
178
178
179 By default the generated coroutine function will be consumed by Asyncio's
179 By default the generated coroutine function will be consumed by Asyncio's
180 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
180 ``loop_runner = asyncio.get_event_loop().run_until_complete()`` method if
181 ``async`` mode is deemed necessary, otherwise the coroutine will just be
181 ``async`` mode is deemed necessary, otherwise the coroutine will just be
182 exhausted in a simple runner. It is possible, though, to change the default
182 exhausted in a simple runner. It is possible, though, to change the default
183 runner.
183 runner.
184
184
185 A loop runner is a *synchronous* function responsible from running a coroutine
185 A loop runner is a *synchronous* function responsible from running a coroutine
186 object.
186 object.
187
187
188 The runner is responsible for ensuring that ``coroutine`` runs to completion,
188 The runner is responsible for ensuring that ``coroutine`` runs to completion,
189 and it should return the result of executing the coroutine. Let's write a
189 and it should return the result of executing the coroutine. Let's write a
190 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
190 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
191 special as it usually prefers to run a function object and make a coroutine by
191 special as it usually prefers to run a function object and make a coroutine by
192 itself, we can get around this limitation by wrapping it in an async-def without
192 itself, we can get around this limitation by wrapping it in an async-def without
193 parameters and passing this value to ``trio``::
193 parameters and passing this value to ``trio``::
194
194
195
195
196 In [1]: import trio
196 In [1]: import trio
197 ...: from types import CoroutineType
197 ...: from types import CoroutineType
198 ...:
198 ...:
199 ...: def trio_runner(coro:CoroutineType):
199 ...: def trio_runner(coro:CoroutineType):
200 ...: print('running asynchronous code')
200 ...: print('running asynchronous code')
201 ...: async def corowrap(coro):
201 ...: async def corowrap(coro):
202 ...: return await coro
202 ...: return await coro
203 ...: return trio.run(corowrap, coro)
203 ...: return trio.run(corowrap, coro)
204
204
205 We can set it up by passing it to ``%autoawait``::
205 We can set it up by passing it to ``%autoawait``::
206
206
207 In [2]: %autoawait trio_runner
207 In [2]: %autoawait trio_runner
208
208
209 In [3]: async def async_hello(name):
209 In [3]: async def async_hello(name):
210 ...: await trio.sleep(1)
210 ...: await trio.sleep(1)
211 ...: print(f'Hello {name} world !')
211 ...: print(f'Hello {name} world !')
212 ...: await trio.sleep(1)
212 ...: await trio.sleep(1)
213
213
214 In [4]: await async_hello('async')
214 In [4]: await async_hello('async')
215 running asynchronous code
215 running asynchronous code
216 Hello async world !
216 Hello async world !
217
217
218
218
219 Asynchronous programming in python (and in particular in the REPL) is still a
219 Asynchronous programming in python (and in particular in the REPL) is still a
220 relatively young subject. We expect some code to not behave as you expect, so
220 relatively young subject. We expect some code to not behave as you expect, so
221 feel free to contribute improvements to this codebase and give us feedback.
221 feel free to contribute improvements to this codebase and give us feedback.
222
222
223 We invite you to thoroughly test this feature and report any unexpected behavior
223 We invite you to thoroughly test this feature and report any unexpected behavior
224 as well as propose any improvement.
224 as well as propose any improvement.
225
225
226 Using Autoawait in a notebook (IPykernel)
226 Using Autoawait in a notebook (IPykernel)
227 -----------------------------------------
227 -----------------------------------------
228
228
229 Update ipykernel to version 5.0 or greater::
229 Update ipykernel to version 5.0 or greater::
230
230
231 pip install ipykernel ipython --upgrade
231 pip install ipykernel ipython --upgrade
232 # or
232 # or
233 conda install ipykernel ipython --upgrade
233 conda install ipykernel ipython --upgrade
234
234
235 This should automatically enable :magic:`autoawait` integration. Unlike
235 This should automatically enable :magic:`autoawait` integration. Unlike
236 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
236 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
237 hand will not work, including with magics like :magic:`%run` or other
237 hand will not work, including with magics like :magic:`%run` or other
238 frameworks that create the eventloop themselves. In cases like these you can
238 frameworks that create the eventloop themselves. In cases like these you can
239 try to use projects like `nest_asyncio
239 try to use projects like `nest_asyncio
240 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
240 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
241 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
241 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
242
242
243 Difference between terminal IPython and IPykernel
243 Difference between terminal IPython and IPykernel
244 -------------------------------------------------
244 -------------------------------------------------
245
245
246 The exact asynchronous code running behavior varies between Terminal IPython and
246 The exact asynchronous code running behavior varies between Terminal IPython and
247 IPykernel. The root cause of this behavior is due to IPykernel having a
247 IPykernel. The root cause of this behavior is due to IPykernel having a
248 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
248 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
249 loop for each code block. This can lead to surprising behavior in some cases if
249 loop for each code block. This can lead to surprising behavior in some cases if
250 you are used to manipulating asyncio loop yourself, see for example
250 you are used to manipulating asyncio loop yourself, see for example
251 :ghissue:`11303` for a longer discussion but here are some of the astonishing
251 :ghissue:`11303` for a longer discussion but here are some of the astonishing
252 cases.
252 cases.
253
253
254 This behavior is an implementation detail, and should not be relied upon. It can
254 This behavior is an implementation detail, and should not be relied upon. It can
255 change without warnings in future versions of IPython.
255 change without warnings in future versions of IPython.
256
256
257 In terminal IPython a loop is started for each code blocks only if there is top
257 In terminal IPython a loop is started for each code blocks only if there is top
258 level async code::
258 level async code::
259
259
260 $ ipython
260 $ ipython
261 In [1]: import asyncio
261 In [1]: import asyncio
262 ...: asyncio.get_event_loop()
262 ...: asyncio.get_event_loop()
263 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
263 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
264
264
265 In [2]:
265 In [2]:
266
266
267 In [2]: import asyncio
267 In [2]: import asyncio
268 ...: await asyncio.sleep(0)
268 ...: await asyncio.sleep(0)
269 ...: asyncio.get_event_loop()
269 ...: asyncio.get_event_loop()
270 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
270 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
271
271
272 See that ``running`` is ``True`` only in the case were we ``await sleep()``
272 See that ``running`` is ``True`` only in the case were we ``await sleep()``
273
273
274 In a Notebook, with ipykernel the asyncio eventloop is always running::
274 In a Notebook, with ipykernel the asyncio eventloop is always running::
275
275
276 $ jupyter notebook
276 $ jupyter notebook
277 In [1]: import asyncio
277 In [1]: import asyncio
278 ...: loop1 = asyncio.get_event_loop()
278 ...: loop1 = asyncio.get_event_loop()
279 ...: loop1
279 ...: loop1
280 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
280 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
281
281
282 In [2]: loop2 = asyncio.get_event_loop()
282 In [2]: loop2 = asyncio.get_event_loop()
283 ...: loop2
283 ...: loop2
284 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
284 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
285
285
286 In [3]: loop1 is loop2
286 In [3]: loop1 is loop2
287 Out[3]: True
287 Out[3]: True
288
288
289 In Terminal IPython background tasks are only processed while the foreground
289 In Terminal IPython background tasks are only processed while the foreground
290 task is running, if and only if the foreground task is async::
290 task is running, if and only if the foreground task is async::
291
291
292 $ ipython
292 $ ipython
293 In [1]: import asyncio
293 In [1]: import asyncio
294 ...:
294 ...:
295 ...: async def repeat(msg, n):
295 ...: async def repeat(msg, n):
296 ...: for i in range(n):
296 ...: for i in range(n):
297 ...: print(f"{msg} {i}")
297 ...: print(f"{msg} {i}")
298 ...: await asyncio.sleep(1)
298 ...: await asyncio.sleep(1)
299 ...: return f"{msg} done"
299 ...: return f"{msg} done"
300 ...:
300 ...:
301 ...: asyncio.ensure_future(repeat("background", 10))
301 ...: asyncio.ensure_future(repeat("background", 10))
302 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
302 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
303
303
304 In [2]: await asyncio.sleep(3)
304 In [2]: await asyncio.sleep(3)
305 background 0
305 background 0
306 background 1
306 background 1
307 background 2
307 background 2
308 background 3
308 background 3
309
309
310 In [3]: import time
310 In [3]: import time
311 ...: time.sleep(5)
311 ...: time.sleep(5)
312
312
313 In [4]: await asyncio.sleep(3)
313 In [4]: await asyncio.sleep(3)
314 background 4
314 background 4
315 background 5
315 background 5
316 background 6g
316 background 6g
317
317
318 In a Notebook, QtConsole, or any other frontend using IPykernel, background
318 In a Notebook, QtConsole, or any other frontend using IPykernel, background
319 tasks should behave as expected.
319 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now