##// END OF EJS Templates
minor typo and style tweak
Paul Ivanov -
Show More
@@ -1,317 +1,316 b''
1 .. _autoawait:
1 .. _autoawait:
2
2
3 Asynchronous in REPL: Autoawait
3 Asynchronous in REPL: Autoawait
4 ===============================
4 ===============================
5
5
6 .. note::
6 .. note::
7
7
8 This feature is experimental and behavior can change between python and
8 This feature is experimental and behavior can change between python and
9 IPython version without prior deprecation.
9 IPython version without prior deprecation.
10
10
11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
12 ability to run asynchronous code from the REPL. Constructs which are
12 ability to run asynchronous code from the REPL. Constructs which are
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14
14
15 The examples given here are for terminal IPython, running async code in a
15 The examples given here are for terminal IPython, running async code in a
16 notebook interface or any other frontend using the Jupyter protocol needs
16 notebook interface or any other frontend using the Jupyter protocol needs
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 will differ between IPython, IPykernel and their versions.
18 will differ between IPython, IPykernel and their versions.
19
19
20 When a supported library is used, IPython will automatically allow Futures and
20 When a supported library is used, IPython will automatically allow Futures and
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 <await>` (or any other async constructs like async-with, async-for) is use at
22 <await>` (or any other async constructs like async-with, async-for) is use at
23 top level scope, or if any structure valid only in `async def
23 top level scope, or if any structure valid only in `async def
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 context are present. For example, the following being a syntax error in the
25 context are present. For example, the following being a syntax error in the
26 Python REPL::
26 Python REPL::
27
27
28 Python 3.6.0
28 Python 3.6.0
29 [GCC 4.2.1]
29 [GCC 4.2.1]
30 Type "help", "copyright", "credits" or "license" for more information.
30 Type "help", "copyright", "credits" or "license" for more information.
31 >>> import aiohttp
31 >>> import aiohttp
32 >>> result = aiohttp.get('https://api.github.com')
32 >>> result = aiohttp.get('https://api.github.com')
33 >>> response = await result
33 >>> response = await result
34 File "<stdin>", line 1
34 File "<stdin>", line 1
35 response = await result
35 response = await result
36 ^
36 ^
37 SyntaxError: invalid syntax
37 SyntaxError: invalid syntax
38
38
39 Should behave as expected in the IPython REPL::
39 Should behave as expected in the IPython REPL::
40
40
41 Python 3.6.0
41 Python 3.6.0
42 Type 'copyright', 'credits' or 'license' for more information
42 Type 'copyright', 'credits' or 'license' for more information
43 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
43 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44
44
45 In [1]: import aiohttp
45 In [1]: import aiohttp
46 ...: result = aiohttp.get('https://api.github.com')
46 ...: result = aiohttp.get('https://api.github.com')
47
47
48 In [2]: response = await result
48 In [2]: response = await result
49 <pause for a few 100s ms>
49 <pause for a few 100s ms>
50
50
51 In [3]: await response.json()
51 In [3]: await response.json()
52 Out[3]:
52 Out[3]:
53 {'authorizations_url': 'https://api.github.com/authorizations',
53 {'authorizations_url': 'https://api.github.com/authorizations',
54 'code_search_url': 'https://api.github.com/search/code?q={query}...',
54 'code_search_url': 'https://api.github.com/search/code?q={query}...',
55 ...
55 ...
56 }
56 }
57
57
58
58
59 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
59 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
60 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
60 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
61 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
61 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
62
62
63 In [1]: %autoawait False
63 In [1]: %autoawait False
64
64
65 In [2]: %autoawait
65 In [2]: %autoawait
66 IPython autoawait is `Off`, and set to use `asyncio`
66 IPython autoawait is `Off`, and set to use `asyncio`
67
67
68
68
69
69
70 By default IPython will assume integration with Python's provided
70 By default IPython will assume integration with Python's provided
71 :mod:`asyncio`, but integration with other libraries is provided. In particular
71 :mod:`asyncio`, but integration with other libraries is provided. In particular
72 we provide experimental integration with the ``curio`` and ``trio`` library.
72 we provide experimental integration with the ``curio`` and ``trio`` library.
73
73
74 You can switch current integration by using the
74 You can switch current integration by using the
75 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
75 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
76 integration>`` magic.
76 integration>`` magic.
77
77
78 For example::
78 For example::
79
79
80 In [1]: %autoawait trio
80 In [1]: %autoawait trio
81
81
82 In [2]: import trio
82 In [2]: import trio
83
83
84 In [3]: async def child(i):
84 In [3]: async def child(i):
85 ...: print(" child %s goes to sleep"%i)
85 ...: print(" child %s goes to sleep"%i)
86 ...: await trio.sleep(2)
86 ...: await trio.sleep(2)
87 ...: print(" child %s wakes up"%i)
87 ...: print(" child %s wakes up"%i)
88
88
89 In [4]: print('parent start')
89 In [4]: print('parent start')
90 ...: async with trio.open_nursery() as n:
90 ...: async with trio.open_nursery() as n:
91 ...: for i in range(5):
91 ...: for i in range(5):
92 ...: n.spawn(child, i)
92 ...: n.spawn(child, i)
93 ...: print('parent end')
93 ...: print('parent end')
94 parent start
94 parent start
95 child 2 goes to sleep
95 child 2 goes to sleep
96 child 0 goes to sleep
96 child 0 goes to sleep
97 child 3 goes to sleep
97 child 3 goes to sleep
98 child 1 goes to sleep
98 child 1 goes to sleep
99 child 4 goes to sleep
99 child 4 goes to sleep
100 <about 2 seconds pause>
100 <about 2 seconds pause>
101 child 2 wakes up
101 child 2 wakes up
102 child 1 wakes up
102 child 1 wakes up
103 child 0 wakes up
103 child 0 wakes up
104 child 3 wakes up
104 child 3 wakes up
105 child 4 wakes up
105 child 4 wakes up
106 parent end
106 parent end
107
107
108
108
109 In the above example, ``async with`` at top level scope is a syntax error in
109 In the above example, ``async with`` at top level scope is a syntax error in
110 Python.
110 Python.
111
111
112 Using this mode can have unexpected consequences if used in interaction with
112 Using this mode can have unexpected consequences if used in interaction with
113 other features of IPython and various registered extensions. In particular if
113 other features of IPython and various registered extensions. In particular if
114 you are a direct or indirect user of the AST transformers, these may not apply
114 you are a direct or indirect user of the AST transformers, these may not apply
115 to your code.
115 to your code.
116
116
117 When using command line IPython, the default loop (or runner) does not process
117 When using command line IPython, the default loop (or runner) does not process
118 in the background, so top level asynchronous code must finish for the REPL to
118 in the background, so top level asynchronous code must finish for the REPL to
119 allow you to enter more code. As with usual Python semantic, the awaitables are
119 allow you to enter more code. As with usual Python semantic, the awaitables are
120 started only when awaited for the first time. That is to say, in first example,
120 started only when awaited for the first time. That is to say, in first example,
121 no network request is done between ``In[1]`` and ``In[2]``.
121 no network request is done between ``In[1]`` and ``In[2]``.
122
122
123
123
124 Effects on IPython.embed()
124 Effects on IPython.embed()
125 ==========================
125 ==========================
126
126
127 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
127 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
128 a loop to run. By default IPython will use a fake coroutine runner which should
128 a loop to run. By default IPython will use a fake coroutine runner which should
129 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
129 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
130 ``autoawait`` feature when using IPython embed.
130 ``autoawait`` feature when using IPython embed.
131
131
132 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
132 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
133 asynchronous code, the exact behavior is though undefined.
133 asynchronous code, the exact behavior is though undefined.
134
134
135 Effects on Magics
135 Effects on Magics
136 =================
136 =================
137
137
138 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
138 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
139 yet been updated to work with asynchronous code and will raise syntax errors
139 yet been updated to work with asynchronous code and will raise syntax errors
140 when trying to use top-level ``await``. We welcome any contribution to help fix
140 when trying to use top-level ``await``. We welcome any contribution to help fix
141 those, and extra cases we haven't caught yet. We hope for better support in Cor
141 those, and extra cases we haven't caught yet. We hope for better support in Cor
142 Python for top-level Async code.
142 Python for top-level Async code.
143
143
144 Internals
144 Internals
145 =========
145 =========
146
146
147 As running asynchronous code is not supported in interactive REPL (as of Python
147 As running asynchronous code is not supported in interactive REPL (as of Python
148 3.7) we have to rely to a number of complex workaround and heuristic to allow
148 3.7) we have to rely to a number of complex workaround and heuristic to allow
149 this to happen. It is interesting to understand how this works in order to
149 this to happen. It is interesting to understand how this works in order to
150 comprehend potential bugs, or provide a custom runner.
150 comprehend potential bugs, or provide a custom runner.
151
151
152 Among the many approaches that are at our disposition, we find only one that
152 Among the many approaches that are at our disposition, we find only one that
153 suited out need. Under the hood we use the code object from a async-def function
153 suited out need. Under the hood we use the code object from a async-def function
154 and run it in global namespace after modifying it to not create a new
154 and run it in global namespace after modifying it to not create a new
155 ``locals()`` scope::
155 ``locals()`` scope::
156
156
157 async def inner_async():
157 async def inner_async():
158 locals().update(**global_namespace)
158 locals().update(**global_namespace)
159 #
159 #
160 # here is user code
160 # here is user code
161 #
161 #
162 return last_user_statement
162 return last_user_statement
163 codeobj = modify(inner_async.__code__)
163 codeobj = modify(inner_async.__code__)
164 coroutine = eval(codeobj, user_ns)
164 coroutine = eval(codeobj, user_ns)
165 display(loop_runner(coroutine))
165 display(loop_runner(coroutine))
166
166
167
167
168
168
169 The first thing you'll notice is that unlike classical ``exec``, there is only
169 The first thing you'll notice is that unlike classical ``exec``, there is only
170 one namespace. Second, user code runs in a function scope, and not a module
170 one namespace. Second, user code runs in a function scope, and not a module
171 scope.
171 scope.
172
172
173 On top of the above there are significant modification to the AST of
173 On top of the above there are significant modification to the AST of
174 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
174 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
175 significant overhead to this kind of code.
175 significant overhead to this kind of code.
176
176
177 By default the generated coroutine function will be consumed by Asyncio's
177 By default the generated coroutine function will be consumed by Asyncio's
178 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
178 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
179 ``async`` mode is deemed necessary, otherwise the coroutine will just be
179 ``async`` mode is deemed necessary, otherwise the coroutine will just be
180 exhausted in a simple runner. It is though possible to change the default
180 exhausted in a simple runner. It is though possible to change the default
181 runner.
181 runner.
182
182
183 A loop runner is a *synchronous* function responsible from running a coroutine
183 A loop runner is a *synchronous* function responsible from running a coroutine
184 object.
184 object.
185
185
186 The runner is responsible from ensuring that ``coroutine`` run to completion,
186 The runner is responsible from ensuring that ``coroutine`` run to completion,
187 and should return the result of executing the coroutine. Let's write a
187 and should return the result of executing the coroutine. Let's write a
188 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
188 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
189 special as it usually prefer to run a function object and make a coroutine by
189 special as it usually prefer to run a function object and make a coroutine by
190 itself, we can get around this limitation by wrapping it in an async-def without
190 itself, we can get around this limitation by wrapping it in an async-def without
191 parameters and passing this value to ``trio``::
191 parameters and passing this value to ``trio``::
192
192
193
193
194 In [1]: import trio
194 In [1]: import trio
195 ...: from types import CoroutineType
195 ...: from types import CoroutineType
196 ...:
196 ...:
197 ...: def trio_runner(coro:CoroutineType):
197 ...: def trio_runner(coro:CoroutineType):
198 ...: print('running asynchronous code')
198 ...: print('running asynchronous code')
199 ...: async def corowrap(coro):
199 ...: async def corowrap(coro):
200 ...: return await coro
200 ...: return await coro
201 ...: return trio.run(corowrap, coro)
201 ...: return trio.run(corowrap, coro)
202
202
203 We can set it up by passing it to ``%autoawait``::
203 We can set it up by passing it to ``%autoawait``::
204
204
205 In [2]: %autoawait trio_runner
205 In [2]: %autoawait trio_runner
206
206
207 In [3]: async def async_hello(name):
207 In [3]: async def async_hello(name):
208 ...: await trio.sleep(1)
208 ...: await trio.sleep(1)
209 ...: print(f'Hello {name} world !')
209 ...: print(f'Hello {name} world !')
210 ...: await trio.sleep(1)
210 ...: await trio.sleep(1)
211
211
212 In [4]: await async_hello('async')
212 In [4]: await async_hello('async')
213 running asynchronous code
213 running asynchronous code
214 Hello async world !
214 Hello async world !
215
215
216
216
217 Asynchronous programming in python (and in particular in the REPL) is still a
217 Asynchronous programming in python (and in particular in the REPL) is still a
218 relatively young subject. We expect some code to not behave as you expect, so
218 relatively young subject. We expect some code to not behave as you expect, so
219 feel free to contribute improvements to this codebase and give us feedback.
219 feel free to contribute improvements to this codebase and give us feedback.
220
220
221 We invite you to thoroughly test this feature and report any unexpected behavior
221 We invite you to thoroughly test this feature and report any unexpected behavior
222 as well as propose any improvement.
222 as well as propose any improvement.
223
223
224 Using Autoawait in a notebook (IPykernel)
224 Using Autoawait in a notebook (IPykernel)
225 =========================================
225 =========================================
226
226
227 Update ipykernel to version 5.0 or greater::
227 Update ipykernel to version 5.0 or greater::
228
228
229 pip install ipykernel ipython --upgrade
229 pip install ipykernel ipython --upgrade
230 # or
230 # or
231 conda install ipykernel ipython --upgrade
231 conda install ipykernel ipython --upgrade
232
232
233 This should automatically enable ``autoawait`` integration. Unlike terminal
233 This should automatically enable ``autoawait`` integration. Unlike terminal
234 IPython, all code runs on ``asynio`` eventloop, so creating a loop by hand will
234 IPython, all code runs on ``asyncio`` eventloop, so creating a loop by hand will
235 not work, including with magics like ``%run`` or other frameworks that create
235 not work, including with magics like ``%run`` or other frameworks that create
236 the eventloop themselves. In case like theses you can try to use projects like
236 the eventloop themselves. In cases like these you can try to use projects like
237 `nest_asyncio <https://github.com/erdewit/nest_asyncio>`_ and see discussion
237 `nest_asyncio <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
238 like `this one
239 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
238 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
240
239
241 Difference between terminal IPython and IPykernel
240 Difference between terminal IPython and IPykernel
242 =================================================
241 =================================================
243
242
244 The exact asynchronous code running behavior varies between Terminal IPython and
243 The exact asynchronous code running behavior varies between Terminal IPython and
245 IPykernel. The root cause of this behavior is due to IPykernel having a
244 IPykernel. The root cause of this behavior is due to IPykernel having a
246 _persistent_ ``asyncio`` loop running, while Terminal IPython starts and stops a
245 _persistent_ ``asyncio`` loop running, while Terminal IPython starts and stops a
247 loop for each code block. This can lead to surprising behavior in some case if
246 loop for each code block. This can lead to surprising behavior in some case if
248 you are used to manipulate asyncio loop yourself, see for example
247 you are used to manipulate asyncio loop yourself, see for example
249 :ghissue:`11303` for a longer discussion but here are some of the astonishing
248 :ghissue:`11303` for a longer discussion but here are some of the astonishing
250 cases.
249 cases.
251
250
252 This behavior is an implementation detail, and should not be relied upon. It can
251 This behavior is an implementation detail, and should not be relied upon. It can
253 change without warnings in future versions of IPython.
252 change without warnings in future versions of IPython.
254
253
255 In terminal IPython a loop is started for each code blocks only if there is top
254 In terminal IPython a loop is started for each code blocks only if there is top
256 level async code::
255 level async code::
257
256
258 $ ipython
257 $ ipython
259 In [1]: import asyncio
258 In [1]: import asyncio
260 ...: asyncio.get_event_loop()
259 ...: asyncio.get_event_loop()
261 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
260 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
262
261
263 In [2]:
262 In [2]:
264
263
265 In [2]: import asyncio
264 In [2]: import asyncio
266 ...: await asyncio.sleep(0)
265 ...: await asyncio.sleep(0)
267 ...: asyncio.get_event_loop()
266 ...: asyncio.get_event_loop()
268 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
267 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
269
268
270 See that ``running`` is ``True`` only in the case were we ``await sleep()``
269 See that ``running`` is ``True`` only in the case were we ``await sleep()``
271
270
272 In a Notebook, with ipykernel the asyncio eventloop is always running::
271 In a Notebook, with ipykernel the asyncio eventloop is always running::
273
272
274 $ jupyter notebook
273 $ jupyter notebook
275 In [1]: import asyncio
274 In [1]: import asyncio
276 ...: loop1 = asyncio.get_event_loop()
275 ...: loop1 = asyncio.get_event_loop()
277 ...: loop1
276 ...: loop1
278 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
277 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
279
278
280 In [2]: loop2 = asyncio.get_event_loop()
279 In [2]: loop2 = asyncio.get_event_loop()
281 ...: loop2
280 ...: loop2
282 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
281 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
283
282
284 In [3]: loop1 is loop2
283 In [3]: loop1 is loop2
285 Out[3]: True
284 Out[3]: True
286
285
287 In Terminal IPython background tasks are only processed while the foreground
286 In Terminal IPython background tasks are only processed while the foreground
288 task is running, if and only if the foreground task is async::
287 task is running, if and only if the foreground task is async::
289
288
290 $ ipython
289 $ ipython
291 In [1]: import asyncio
290 In [1]: import asyncio
292 ...:
291 ...:
293 ...: async def repeat(msg, n):
292 ...: async def repeat(msg, n):
294 ...: for i in range(n):
293 ...: for i in range(n):
295 ...: print(f"{msg} {i}")
294 ...: print(f"{msg} {i}")
296 ...: await asyncio.sleep(1)
295 ...: await asyncio.sleep(1)
297 ...: return f"{msg} done"
296 ...: return f"{msg} done"
298 ...:
297 ...:
299 ...: asyncio.ensure_future(repeat("background", 10))
298 ...: asyncio.ensure_future(repeat("background", 10))
300 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
299 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
301
300
302 In [2]: await asyncio.sleep(3)
301 In [2]: await asyncio.sleep(3)
303 background 0
302 background 0
304 background 1
303 background 1
305 background 2
304 background 2
306 background 3
305 background 3
307
306
308 In [3]: import time
307 In [3]: import time
309 ...: time.sleep(5)
308 ...: time.sleep(5)
310
309
311 In [4]: await asyncio.sleep(3)
310 In [4]: await asyncio.sleep(3)
312 background 4
311 background 4
313 background 5
312 background 5
314 background 6
313 background 6g
315
314
316 In a Notebook, QtConsole, or any other frontend using IPykernel, background
315 In a Notebook, QtConsole, or any other frontend using IPykernel, background
317 tasks should behave as expected.
316 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now