##// END OF EJS Templates
docs: update the aiohttp usage in autoawait.rst...
Sebastian Witowski -
Show More
@@ -1,317 +1,319 b''
1 1 .. _autoawait:
2 2
3 3 Asynchronous in REPL: Autoawait
4 4 ===============================
5 5
6 6 .. note::
7 7
8 8 This feature is experimental and behavior can change between python and
9 9 IPython version without prior deprecation.
10 10
11 11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
12 12 ability to run asynchronous code from the REPL. Constructs which are
13 13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14 14
15 15 The examples given here are for terminal IPython, running async code in a
16 16 notebook interface or any other frontend using the Jupyter protocol needs
17 17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 18 will differ between IPython, IPykernel and their versions.
19 19
20 20 When a supported library is used, IPython will automatically allow Futures and
21 21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 22 <await>` (or any other async constructs like async-with, async-for) is use at
23 23 top level scope, or if any structure valid only in `async def
24 24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 25 context are present. For example, the following being a syntax error in the
26 26 Python REPL::
27 27
28 28 Python 3.6.0
29 29 [GCC 4.2.1]
30 30 Type "help", "copyright", "credits" or "license" for more information.
31 31 >>> import aiohttp
32 >>> result = aiohttp.get('https://api.github.com')
32 >>> session = aiohttp.ClientSession()
33 >>> result = session.get('https://api.github.com')
33 34 >>> response = await result
34 35 File "<stdin>", line 1
35 36 response = await result
36 37 ^
37 38 SyntaxError: invalid syntax
38 39
39 40 Should behave as expected in the IPython REPL::
40 41
41 42 Python 3.6.0
42 43 Type 'copyright', 'credits' or 'license' for more information
43 44 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44 45
45 46 In [1]: import aiohttp
46 ...: result = aiohttp.get('https://api.github.com')
47 ...: session = aiohttp.ClientSession()
48 ...: result = session.get('https://api.github.com')
47 49
48 50 In [2]: response = await result
49 51 <pause for a few 100s ms>
50 52
51 53 In [3]: await response.json()
52 54 Out[3]:
53 55 {'authorizations_url': 'https://api.github.com/authorizations',
54 56 'code_search_url': 'https://api.github.com/search/code?q={query}...',
55 57 ...
56 58 }
57 59
58 60
59 61 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
60 62 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
61 63 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
62 64
63 65 In [1]: %autoawait False
64 66
65 67 In [2]: %autoawait
66 68 IPython autoawait is `Off`, and set to use `asyncio`
67 69
68 70
69 71
70 72 By default IPython will assume integration with Python's provided
71 73 :mod:`asyncio`, but integration with other libraries is provided. In particular
72 74 we provide experimental integration with the ``curio`` and ``trio`` library.
73 75
74 76 You can switch current integration by using the
75 77 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
76 78 integration>`` magic.
77 79
78 80 For example::
79 81
80 82 In [1]: %autoawait trio
81 83
82 84 In [2]: import trio
83 85
84 86 In [3]: async def child(i):
85 87 ...: print(" child %s goes to sleep"%i)
86 88 ...: await trio.sleep(2)
87 89 ...: print(" child %s wakes up"%i)
88 90
89 91 In [4]: print('parent start')
90 92 ...: async with trio.open_nursery() as n:
91 93 ...: for i in range(5):
92 94 ...: n.spawn(child, i)
93 95 ...: print('parent end')
94 96 parent start
95 97 child 2 goes to sleep
96 98 child 0 goes to sleep
97 99 child 3 goes to sleep
98 100 child 1 goes to sleep
99 101 child 4 goes to sleep
100 102 <about 2 seconds pause>
101 103 child 2 wakes up
102 104 child 1 wakes up
103 105 child 0 wakes up
104 106 child 3 wakes up
105 107 child 4 wakes up
106 108 parent end
107 109
108 110
109 111 In the above example, ``async with`` at top level scope is a syntax error in
110 112 Python.
111 113
112 114 Using this mode can have unexpected consequences if used in interaction with
113 115 other features of IPython and various registered extensions. In particular if
114 116 you are a direct or indirect user of the AST transformers, these may not apply
115 117 to your code.
116 118
117 119 When using command line IPython, the default loop (or runner) does not process
118 120 in the background, so top level asynchronous code must finish for the REPL to
119 121 allow you to enter more code. As with usual Python semantic, the awaitables are
120 122 started only when awaited for the first time. That is to say, in first example,
121 123 no network request is done between ``In[1]`` and ``In[2]``.
122 124
123 125
124 126 Effects on IPython.embed()
125 127 --------------------------
126 128
127 129 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
128 130 a loop to run. By default IPython will use a fake coroutine runner which should
129 131 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
130 132 :magic:`%autoawait` feature when using IPython embed.
131 133
132 134 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
133 135 asynchronous code, the exact behavior is though undefined.
134 136
135 137 Effects on Magics
136 138 -----------------
137 139
138 140 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
139 141 yet been updated to work with asynchronous code and will raise syntax errors
140 142 when trying to use top-level ``await``. We welcome any contribution to help fix
141 143 those, and extra cases we haven't caught yet. We hope for better support in Cor
142 144 Python for top-level Async code.
143 145
144 146 Internals
145 147 ---------
146 148
147 149 As running asynchronous code is not supported in interactive REPL (as of Python
148 150 3.7) we have to rely to a number of complex workaround and heuristic to allow
149 151 this to happen. It is interesting to understand how this works in order to
150 152 comprehend potential bugs, or provide a custom runner.
151 153
152 154 Among the many approaches that are at our disposition, we find only one that
153 155 suited out need. Under the hood we use the code object from a async-def function
154 156 and run it in global namespace after modifying it to not create a new
155 157 ``locals()`` scope::
156 158
157 159 async def inner_async():
158 160 locals().update(**global_namespace)
159 161 #
160 162 # here is user code
161 163 #
162 164 return last_user_statement
163 165 codeobj = modify(inner_async.__code__)
164 166 coroutine = eval(codeobj, user_ns)
165 167 display(loop_runner(coroutine))
166 168
167 169
168 170
169 171 The first thing you'll notice is that unlike classical ``exec``, there is only
170 172 one namespace. Second, user code runs in a function scope, and not a module
171 173 scope.
172 174
173 175 On top of the above there are significant modification to the AST of
174 176 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
175 177 significant overhead to this kind of code.
176 178
177 179 By default the generated coroutine function will be consumed by Asyncio's
178 180 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
179 181 ``async`` mode is deemed necessary, otherwise the coroutine will just be
180 182 exhausted in a simple runner. It is though possible to change the default
181 183 runner.
182 184
183 185 A loop runner is a *synchronous* function responsible from running a coroutine
184 186 object.
185 187
186 188 The runner is responsible from ensuring that ``coroutine`` run to completion,
187 189 and should return the result of executing the coroutine. Let's write a
188 190 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
189 191 special as it usually prefer to run a function object and make a coroutine by
190 192 itself, we can get around this limitation by wrapping it in an async-def without
191 193 parameters and passing this value to ``trio``::
192 194
193 195
194 196 In [1]: import trio
195 197 ...: from types import CoroutineType
196 198 ...:
197 199 ...: def trio_runner(coro:CoroutineType):
198 200 ...: print('running asynchronous code')
199 201 ...: async def corowrap(coro):
200 202 ...: return await coro
201 203 ...: return trio.run(corowrap, coro)
202 204
203 205 We can set it up by passing it to ``%autoawait``::
204 206
205 207 In [2]: %autoawait trio_runner
206 208
207 209 In [3]: async def async_hello(name):
208 210 ...: await trio.sleep(1)
209 211 ...: print(f'Hello {name} world !')
210 212 ...: await trio.sleep(1)
211 213
212 214 In [4]: await async_hello('async')
213 215 running asynchronous code
214 216 Hello async world !
215 217
216 218
217 219 Asynchronous programming in python (and in particular in the REPL) is still a
218 220 relatively young subject. We expect some code to not behave as you expect, so
219 221 feel free to contribute improvements to this codebase and give us feedback.
220 222
221 223 We invite you to thoroughly test this feature and report any unexpected behavior
222 224 as well as propose any improvement.
223 225
224 226 Using Autoawait in a notebook (IPykernel)
225 227 -----------------------------------------
226 228
227 229 Update ipykernel to version 5.0 or greater::
228 230
229 231 pip install ipykernel ipython --upgrade
230 232 # or
231 233 conda install ipykernel ipython --upgrade
232 234
233 235 This should automatically enable :magic:`autoawait` integration. Unlike
234 236 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
235 237 hand will not work, including with magics like :magic:`%run` or other
236 238 frameworks that create the eventloop themselves. In cases like these you can
237 239 try to use projects like `nest_asyncio
238 240 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
239 241 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
240 242
241 243 Difference between terminal IPython and IPykernel
242 244 -------------------------------------------------
243 245
244 246 The exact asynchronous code running behavior varies between Terminal IPython and
245 247 IPykernel. The root cause of this behavior is due to IPykernel having a
246 248 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
247 249 loop for each code block. This can lead to surprising behavior in some case if
248 250 you are used to manipulate asyncio loop yourself, see for example
249 251 :ghissue:`11303` for a longer discussion but here are some of the astonishing
250 252 cases.
251 253
252 254 This behavior is an implementation detail, and should not be relied upon. It can
253 255 change without warnings in future versions of IPython.
254 256
255 257 In terminal IPython a loop is started for each code blocks only if there is top
256 258 level async code::
257 259
258 260 $ ipython
259 261 In [1]: import asyncio
260 262 ...: asyncio.get_event_loop()
261 263 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
262 264
263 265 In [2]:
264 266
265 267 In [2]: import asyncio
266 268 ...: await asyncio.sleep(0)
267 269 ...: asyncio.get_event_loop()
268 270 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
269 271
270 272 See that ``running`` is ``True`` only in the case were we ``await sleep()``
271 273
272 274 In a Notebook, with ipykernel the asyncio eventloop is always running::
273 275
274 276 $ jupyter notebook
275 277 In [1]: import asyncio
276 278 ...: loop1 = asyncio.get_event_loop()
277 279 ...: loop1
278 280 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
279 281
280 282 In [2]: loop2 = asyncio.get_event_loop()
281 283 ...: loop2
282 284 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
283 285
284 286 In [3]: loop1 is loop2
285 287 Out[3]: True
286 288
287 289 In Terminal IPython background tasks are only processed while the foreground
288 290 task is running, if and only if the foreground task is async::
289 291
290 292 $ ipython
291 293 In [1]: import asyncio
292 294 ...:
293 295 ...: async def repeat(msg, n):
294 296 ...: for i in range(n):
295 297 ...: print(f"{msg} {i}")
296 298 ...: await asyncio.sleep(1)
297 299 ...: return f"{msg} done"
298 300 ...:
299 301 ...: asyncio.ensure_future(repeat("background", 10))
300 302 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
301 303
302 304 In [2]: await asyncio.sleep(3)
303 305 background 0
304 306 background 1
305 307 background 2
306 308 background 3
307 309
308 310 In [3]: import time
309 311 ...: time.sleep(5)
310 312
311 313 In [4]: await asyncio.sleep(3)
312 314 background 4
313 315 background 5
314 316 background 6g
315 317
316 318 In a Notebook, QtConsole, or any other frontend using IPykernel, background
317 319 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now