##// END OF EJS Templates
fix get_event_loop typo
Thomas Grainger -
Show More
@@ -1,319 +1,319 b''
1 1 .. _autoawait:
2 2
3 3 Asynchronous in REPL: Autoawait
4 4 ===============================
5 5
6 6 .. note::
7 7
8 8 This feature is experimental and behavior can change between python and
9 9 IPython version without prior deprecation.
10 10
11 11 Starting with IPython 7.0, and when using Python 3.6 and above, IPython offer the
12 12 ability to run asynchronous code from the REPL. Constructs which are
13 13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14 14
15 15 The examples given here are for terminal IPython, running async code in a
16 16 notebook interface or any other frontend using the Jupyter protocol needs
17 17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 18 will differ between IPython, IPykernel and their versions.
19 19
20 20 When a supported library is used, IPython will automatically allow Futures and
21 21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 22 <await>` (or any other async constructs like async-with, async-for) is used at
23 23 top level scope, or if any structure valid only in `async def
24 24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 25 context are present. For example, the following being a syntax error in the
26 26 Python REPL::
27 27
28 28 Python 3.6.0
29 29 [GCC 4.2.1]
30 30 Type "help", "copyright", "credits" or "license" for more information.
31 31 >>> import aiohttp
32 32 >>> session = aiohttp.ClientSession()
33 33 >>> result = session.get('https://api.github.com')
34 34 >>> response = await result
35 35 File "<stdin>", line 1
36 36 response = await result
37 37 ^
38 38 SyntaxError: invalid syntax
39 39
40 40 Should behave as expected in the IPython REPL::
41 41
42 42 Python 3.6.0
43 43 Type 'copyright', 'credits' or 'license' for more information
44 44 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
45 45
46 46 In [1]: import aiohttp
47 47 ...: session = aiohttp.ClientSession()
48 48 ...: result = session.get('https://api.github.com')
49 49
50 50 In [2]: response = await result
51 51 <pause for a few 100s ms>
52 52
53 53 In [3]: await response.json()
54 54 Out[3]:
55 55 {'authorizations_url': 'https://api.github.com/authorizations',
56 56 'code_search_url': 'https://api.github.com/search/code?q={query}...',
57 57 ...
58 58 }
59 59
60 60
61 61 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
62 62 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
63 63 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
64 64
65 65 In [1]: %autoawait False
66 66
67 67 In [2]: %autoawait
68 68 IPython autoawait is `Off`, and set to use `asyncio`
69 69
70 70
71 71
72 72 By default IPython will assume integration with Python's provided
73 73 :mod:`asyncio`, but integration with other libraries is provided. In particular
74 74 we provide experimental integration with the ``curio`` and ``trio`` library.
75 75
76 76 You can switch the current integration by using the
77 77 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
78 78 integration>`` magic.
79 79
80 80 For example::
81 81
82 82 In [1]: %autoawait trio
83 83
84 84 In [2]: import trio
85 85
86 86 In [3]: async def child(i):
87 87 ...: print(" child %s goes to sleep"%i)
88 88 ...: await trio.sleep(2)
89 89 ...: print(" child %s wakes up"%i)
90 90
91 91 In [4]: print('parent start')
92 92 ...: async with trio.open_nursery() as n:
93 93 ...: for i in range(5):
94 94 ...: n.spawn(child, i)
95 95 ...: print('parent end')
96 96 parent start
97 97 child 2 goes to sleep
98 98 child 0 goes to sleep
99 99 child 3 goes to sleep
100 100 child 1 goes to sleep
101 101 child 4 goes to sleep
102 102 <about 2 seconds pause>
103 103 child 2 wakes up
104 104 child 1 wakes up
105 105 child 0 wakes up
106 106 child 3 wakes up
107 107 child 4 wakes up
108 108 parent end
109 109
110 110
111 111 In the above example, ``async with`` at top level scope is a syntax error in
112 112 Python.
113 113
114 114 Using this mode can have unexpected consequences if used in interaction with
115 115 other features of IPython and various registered extensions. In particular if
116 116 you are a direct or indirect user of the AST transformers, these may not apply
117 117 to your code.
118 118
119 119 When using command line IPython, the default loop (or runner) does not process
120 120 in the background, so top level asynchronous code must finish for the REPL to
121 121 allow you to enter more code. As with usual Python semantics, the awaitables are
122 122 started only when awaited for the first time. That is to say, in first example,
123 123 no network request is done between ``In[1]`` and ``In[2]``.
124 124
125 125
126 126 Effects on IPython.embed()
127 127 --------------------------
128 128
129 129 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
130 130 a loop to run. By default IPython will use a fake coroutine runner which should
131 131 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
132 132 :magic:`%autoawait` feature when using IPython embed.
133 133
134 134 You can set a coroutine runner explicitly for ``embed()`` if you want to run
135 135 asynchronous code, though the exact behavior is undefined.
136 136
137 137 Effects on Magics
138 138 -----------------
139 139
140 140 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
141 141 yet been updated to work with asynchronous code and will raise syntax errors
142 142 when trying to use top-level ``await``. We welcome any contribution to help fix
143 143 those, and extra cases we haven't caught yet. We hope for better support in Core
144 144 Python for top-level Async code.
145 145
146 146 Internals
147 147 ---------
148 148
149 149 As running asynchronous code is not supported in interactive REPL (as of Python
150 150 3.7) we have to rely to a number of complex workarounds and heuristics to allow
151 151 this to happen. It is interesting to understand how this works in order to
152 152 comprehend potential bugs, or provide a custom runner.
153 153
154 154 Among the many approaches that are at our disposition, we find only one that
155 155 suited out need. Under the hood we use the code object from a async-def function
156 156 and run it in global namespace after modifying it to not create a new
157 157 ``locals()`` scope::
158 158
159 159 async def inner_async():
160 160 locals().update(**global_namespace)
161 161 #
162 162 # here is user code
163 163 #
164 164 return last_user_statement
165 165 codeobj = modify(inner_async.__code__)
166 166 coroutine = eval(codeobj, user_ns)
167 167 display(loop_runner(coroutine))
168 168
169 169
170 170
171 171 The first thing you'll notice is that unlike classical ``exec``, there is only
172 172 one namespace. Second, user code runs in a function scope, and not a module
173 173 scope.
174 174
175 175 On top of the above there are significant modification to the AST of
176 176 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
177 177 significant overhead to this kind of code.
178 178
179 179 By default the generated coroutine function will be consumed by Asyncio's
180 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
180 ``loop_runner = asyncio.get_event_loop().run_until_complete()`` method if
181 181 ``async`` mode is deemed necessary, otherwise the coroutine will just be
182 182 exhausted in a simple runner. It is possible, though, to change the default
183 183 runner.
184 184
185 185 A loop runner is a *synchronous* function responsible from running a coroutine
186 186 object.
187 187
188 188 The runner is responsible for ensuring that ``coroutine`` runs to completion,
189 189 and it should return the result of executing the coroutine. Let's write a
190 190 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
191 191 special as it usually prefers to run a function object and make a coroutine by
192 192 itself, we can get around this limitation by wrapping it in an async-def without
193 193 parameters and passing this value to ``trio``::
194 194
195 195
196 196 In [1]: import trio
197 197 ...: from types import CoroutineType
198 198 ...:
199 199 ...: def trio_runner(coro:CoroutineType):
200 200 ...: print('running asynchronous code')
201 201 ...: async def corowrap(coro):
202 202 ...: return await coro
203 203 ...: return trio.run(corowrap, coro)
204 204
205 205 We can set it up by passing it to ``%autoawait``::
206 206
207 207 In [2]: %autoawait trio_runner
208 208
209 209 In [3]: async def async_hello(name):
210 210 ...: await trio.sleep(1)
211 211 ...: print(f'Hello {name} world !')
212 212 ...: await trio.sleep(1)
213 213
214 214 In [4]: await async_hello('async')
215 215 running asynchronous code
216 216 Hello async world !
217 217
218 218
219 219 Asynchronous programming in python (and in particular in the REPL) is still a
220 220 relatively young subject. We expect some code to not behave as you expect, so
221 221 feel free to contribute improvements to this codebase and give us feedback.
222 222
223 223 We invite you to thoroughly test this feature and report any unexpected behavior
224 224 as well as propose any improvement.
225 225
226 226 Using Autoawait in a notebook (IPykernel)
227 227 -----------------------------------------
228 228
229 229 Update ipykernel to version 5.0 or greater::
230 230
231 231 pip install ipykernel ipython --upgrade
232 232 # or
233 233 conda install ipykernel ipython --upgrade
234 234
235 235 This should automatically enable :magic:`autoawait` integration. Unlike
236 236 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
237 237 hand will not work, including with magics like :magic:`%run` or other
238 238 frameworks that create the eventloop themselves. In cases like these you can
239 239 try to use projects like `nest_asyncio
240 240 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
241 241 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
242 242
243 243 Difference between terminal IPython and IPykernel
244 244 -------------------------------------------------
245 245
246 246 The exact asynchronous code running behavior varies between Terminal IPython and
247 247 IPykernel. The root cause of this behavior is due to IPykernel having a
248 248 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
249 249 loop for each code block. This can lead to surprising behavior in some cases if
250 250 you are used to manipulating asyncio loop yourself, see for example
251 251 :ghissue:`11303` for a longer discussion but here are some of the astonishing
252 252 cases.
253 253
254 254 This behavior is an implementation detail, and should not be relied upon. It can
255 255 change without warnings in future versions of IPython.
256 256
257 257 In terminal IPython a loop is started for each code blocks only if there is top
258 258 level async code::
259 259
260 260 $ ipython
261 261 In [1]: import asyncio
262 262 ...: asyncio.get_event_loop()
263 263 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
264 264
265 265 In [2]:
266 266
267 267 In [2]: import asyncio
268 268 ...: await asyncio.sleep(0)
269 269 ...: asyncio.get_event_loop()
270 270 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
271 271
272 272 See that ``running`` is ``True`` only in the case were we ``await sleep()``
273 273
274 274 In a Notebook, with ipykernel the asyncio eventloop is always running::
275 275
276 276 $ jupyter notebook
277 277 In [1]: import asyncio
278 278 ...: loop1 = asyncio.get_event_loop()
279 279 ...: loop1
280 280 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
281 281
282 282 In [2]: loop2 = asyncio.get_event_loop()
283 283 ...: loop2
284 284 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
285 285
286 286 In [3]: loop1 is loop2
287 287 Out[3]: True
288 288
289 289 In Terminal IPython background tasks are only processed while the foreground
290 290 task is running, if and only if the foreground task is async::
291 291
292 292 $ ipython
293 293 In [1]: import asyncio
294 294 ...:
295 295 ...: async def repeat(msg, n):
296 296 ...: for i in range(n):
297 297 ...: print(f"{msg} {i}")
298 298 ...: await asyncio.sleep(1)
299 299 ...: return f"{msg} done"
300 300 ...:
301 301 ...: asyncio.ensure_future(repeat("background", 10))
302 302 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
303 303
304 304 In [2]: await asyncio.sleep(3)
305 305 background 0
306 306 background 1
307 307 background 2
308 308 background 3
309 309
310 310 In [3]: import time
311 311 ...: time.sleep(5)
312 312
313 313 In [4]: await asyncio.sleep(3)
314 314 background 4
315 315 background 5
316 316 background 6g
317 317
318 318 In a Notebook, QtConsole, or any other frontend using IPykernel, background
319 319 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now