##// END OF EJS Templates
one more rephrasing/plural
Matthias Bussonnier -
Show More
@@ -1,317 +1,317 b''
1 1 .. _autoawait:
2 2
3 3 Asynchronous in REPL: Autoawait
4 4 ===============================
5 5
6 6 .. note::
7 7
8 8 This feature is experimental and behavior can change between python and
9 9 IPython version without prior deprecation.
10 10
11 11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
12 12 ability to run asynchronous code from the REPL. Constructs which are
13 13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14 14
15 The example given here are for terminal IPython, running async code in a
15 The examples given here are for terminal IPython, running async code in a
16 16 notebook interface or any other frontend using the Jupyter protocol needs
17 IPykernel version 5.0 or above. The details of how async code runs in
18 IPykernel will differ between IPython, IPykernel and their versions.
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 will differ between IPython, IPykernel and their versions.
19 19
20 20 When a supported library is used, IPython will automatically allow Futures and
21 21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 22 <await>` (or any other async constructs like async-with, async-for) is use at
23 23 top level scope, or if any structure valid only in `async def
24 24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 25 context are present. For example, the following being a syntax error in the
26 26 Python REPL::
27 27
28 28 Python 3.6.0
29 29 [GCC 4.2.1]
30 30 Type "help", "copyright", "credits" or "license" for more information.
31 31 >>> import aiohttp
32 32 >>> result = aiohttp.get('https://api.github.com')
33 33 >>> response = await result
34 34 File "<stdin>", line 1
35 35 response = await result
36 36 ^
37 37 SyntaxError: invalid syntax
38 38
39 39 Should behave as expected in the IPython REPL::
40 40
41 41 Python 3.6.0
42 42 Type 'copyright', 'credits' or 'license' for more information
43 43 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44 44
45 45 In [1]: import aiohttp
46 46 ...: result = aiohttp.get('https://api.github.com')
47 47
48 48 In [2]: response = await result
49 49 <pause for a few 100s ms>
50 50
51 51 In [3]: await response.json()
52 52 Out[3]:
53 53 {'authorizations_url': 'https://api.github.com/authorizations',
54 54 'code_search_url': 'https://api.github.com/search/code?q={query}...',
55 55 ...
56 56 }
57 57
58 58
59 59 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
60 60 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
61 61 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
62 62
63 63 In [1]: %autoawait False
64 64
65 65 In [2]: %autoawait
66 66 IPython autoawait is `Off`, and set to use `asyncio`
67 67
68 68
69 69
70 70 By default IPython will assume integration with Python's provided
71 71 :mod:`asyncio`, but integration with other libraries is provided. In particular
72 72 we provide experimental integration with the ``curio`` and ``trio`` library.
73 73
74 74 You can switch current integration by using the
75 75 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
76 76 integration>`` magic.
77 77
78 78 For example::
79 79
80 80 In [1]: %autoawait trio
81 81
82 82 In [2]: import trio
83 83
84 84 In [3]: async def child(i):
85 85 ...: print(" child %s goes to sleep"%i)
86 86 ...: await trio.sleep(2)
87 87 ...: print(" child %s wakes up"%i)
88 88
89 89 In [4]: print('parent start')
90 90 ...: async with trio.open_nursery() as n:
91 91 ...: for i in range(5):
92 92 ...: n.spawn(child, i)
93 93 ...: print('parent end')
94 94 parent start
95 95 child 2 goes to sleep
96 96 child 0 goes to sleep
97 97 child 3 goes to sleep
98 98 child 1 goes to sleep
99 99 child 4 goes to sleep
100 100 <about 2 seconds pause>
101 101 child 2 wakes up
102 102 child 1 wakes up
103 103 child 0 wakes up
104 104 child 3 wakes up
105 105 child 4 wakes up
106 106 parent end
107 107
108 108
109 109 In the above example, ``async with`` at top level scope is a syntax error in
110 110 Python.
111 111
112 112 Using this mode can have unexpected consequences if used in interaction with
113 113 other features of IPython and various registered extensions. In particular if
114 114 you are a direct or indirect user of the AST transformers, these may not apply
115 115 to your code.
116 116
117 117 When using command line IPython, the default loop (or runner) does not process
118 118 in the background, so top level asynchronous code must finish for the REPL to
119 119 allow you to enter more code. As with usual Python semantic, the awaitables are
120 120 started only when awaited for the first time. That is to say, in first example,
121 121 no network request is done between ``In[1]`` and ``In[2]``.
122 122
123 123
124 124 Effects on IPython.embed()
125 125 ==========================
126 126
127 127 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
128 128 a loop to run. By default IPython will use a fake coroutine runner which should
129 129 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
130 130 ``autoawait`` feature when using IPython embed.
131 131
132 132 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
133 133 asynchronous code, the exact behavior is though undefined.
134 134
135 135 Effects on Magics
136 136 =================
137 137
138 138 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
139 139 yet been updated to work with asynchronous code and will raise syntax errors
140 140 when trying to use top-level ``await``. We welcome any contribution to help fix
141 141 those, and extra cases we haven't caught yet. We hope for better support in Cor
142 142 Python for top-level Async code.
143 143
144 144 Internals
145 145 =========
146 146
147 147 As running asynchronous code is not supported in interactive REPL (as of Python
148 148 3.7) we have to rely to a number of complex workaround and heuristic to allow
149 149 this to happen. It is interesting to understand how this works in order to
150 150 comprehend potential bugs, or provide a custom runner.
151 151
152 152 Among the many approaches that are at our disposition, we find only one that
153 153 suited out need. Under the hood we use the code object from a async-def function
154 154 and run it in global namespace after modifying it to not create a new
155 155 ``locals()`` scope::
156 156
157 157 async def inner_async():
158 158 locals().update(**global_namespace)
159 159 #
160 160 # here is user code
161 161 #
162 162 return last_user_statement
163 163 codeobj = modify(inner_async.__code__)
164 164 coroutine = eval(codeobj, user_ns)
165 165 display(loop_runner(coroutine))
166 166
167 167
168 168
169 169 The first thing you'll notice is that unlike classical ``exec``, there is only
170 170 one namespace. Second, user code runs in a function scope, and not a module
171 171 scope.
172 172
173 173 On top of the above there are significant modification to the AST of
174 174 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
175 175 significant overhead to this kind of code.
176 176
177 177 By default the generated coroutine function will be consumed by Asyncio's
178 178 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
179 179 ``async`` mode is deemed necessary, otherwise the coroutine will just be
180 180 exhausted in a simple runner. It is though possible to change the default
181 181 runner.
182 182
183 183 A loop runner is a *synchronous* function responsible from running a coroutine
184 184 object.
185 185
186 186 The runner is responsible from ensuring that ``coroutine`` run to completion,
187 187 and should return the result of executing the coroutine. Let's write a
188 188 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
189 189 special as it usually prefer to run a function object and make a coroutine by
190 190 itself, we can get around this limitation by wrapping it in an async-def without
191 191 parameters and passing this value to ``trio``::
192 192
193 193
194 194 In [1]: import trio
195 195 ...: from types import CoroutineType
196 196 ...:
197 197 ...: def trio_runner(coro:CoroutineType):
198 198 ...: print('running asynchronous code')
199 199 ...: async def corowrap(coro):
200 200 ...: return await coro
201 201 ...: return trio.run(corowrap, coro)
202 202
203 203 We can set it up by passing it to ``%autoawait``::
204 204
205 205 In [2]: %autoawait trio_runner
206 206
207 207 In [3]: async def async_hello(name):
208 208 ...: await trio.sleep(1)
209 209 ...: print(f'Hello {name} world !')
210 210 ...: await trio.sleep(1)
211 211
212 212 In [4]: await async_hello('async')
213 213 running asynchronous code
214 214 Hello async world !
215 215
216 216
217 217 Asynchronous programming in python (and in particular in the REPL) is still a
218 218 relatively young subject. We expect some code to not behave as you expect, so
219 219 feel free to contribute improvements to this codebase and give us feedback.
220 220
221 221 We invite you to thoroughly test this feature and report any unexpected behavior
222 222 as well as propose any improvement.
223 223
224 224 Using Autoawait in a notebook (IPykernel)
225 225 =========================================
226 226
227 227 Update ipykernel to version 5.0 or greater::
228 228
229 229 pip install ipykernel ipython --upgrade
230 230 # or
231 231 conda install ipykernel ipython --upgrade
232 232
233 233 This should automatically enable ``autoawait`` integration. Unlike terminal
234 234 IPython, all code runs on ``asynio`` eventloop, so creating a loop by hand will
235 235 not work, including with magics like ``%run`` or other frameworks that create
236 236 the eventloop themselves. In case like theses you can try to use projects like
237 237 `nest_asyncio <https://github.com/erdewit/nest_asyncio>`_ and see discussion
238 238 like `this one
239 239 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
240 240
241 241 Difference between terminal IPython and IPykernel
242 242 =================================================
243 243
244 244 The exact asynchronous code running behavior varies between Terminal IPython and
245 245 IPykernel. The root cause of this behavior is due to IPykernel having a
246 246 _persistent_ ``asyncio`` loop running, while Terminal IPython starts and stop a
247 247 loop for each code block. This can lead to surprising behavior in some case if
248 248 you are used to manipulate asyncio loop yourself, see for example
249 249 :ghissue:`11303` for a longer discussion but here are some of the astonishing
250 250 cases.
251 251
252 252 This behavior is an implementation detail, and should not be relied upon. It can
253 253 change without warnings in future versions of IPython.
254 254
255 255 In terminal IPython a loop is started for each code blocks only if there is top
256 256 level async code::
257 257
258 258 $ ipython
259 259 In [1]: import asyncio
260 260 ...: asyncio.get_event_loop()
261 261 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
262 262
263 263 In [2]:
264 264
265 265 In [2]: import asyncio
266 266 ...: await asyncio.sleep(0)
267 267 ...: asyncio.get_event_loop()
268 268 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
269 269
270 270 See that ``running`` is ``True`` only in the case were we ``await sleep()``
271 271
272 272 In a Notebook, with ipykernel the asyncio eventloop is always running::
273 273
274 274 $ jupyter notebook
275 275 In [1]: import asyncio
276 276 ...: loop1 = asyncio.get_event_loop()
277 277 ...: loop1
278 278 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
279 279
280 280 In [2]: loop2 = asyncio.get_event_loop()
281 281 ...: loop2
282 282 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
283 283
284 284 In [3]: loop1 is loop2
285 285 Out[3]: True
286 286
287 287 In Terminal IPython background tasks are only processed while the foreground
288 288 task is running, if and only if the foreground task is async::
289 289
290 290 $ ipython
291 291 In [1]: import asyncio
292 292 ...:
293 293 ...: async def repeat(msg, n):
294 294 ...: for i in range(n):
295 295 ...: print(f"{msg} {i}")
296 296 ...: await asyncio.sleep(1)
297 297 ...: return f"{msg} done"
298 298 ...:
299 299 ...: asyncio.ensure_future(repeat("background", 10))
300 300 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
301 301
302 302 In [2]: await asyncio.sleep(3)
303 303 background 0
304 304 background 1
305 305 background 2
306 306 background 3
307 307
308 308 In [3]: import time
309 309 ...: time.sleep(5)
310 310
311 311 In [4]: await asyncio.sleep(3)
312 312 background 4
313 313 background 5
314 314 background 6
315 315
316 316 In a Notebook, QtConsole, or any other frontend using IPykernel, background
317 317 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now