##// END OF EJS Templates
docs: update the aiohttp usage in autoawait.rst...
Sebastian Witowski -
Show More
@@ -1,317 +1,319 b''
1 .. _autoawait:
1 .. _autoawait:
2
2
3 Asynchronous in REPL: Autoawait
3 Asynchronous in REPL: Autoawait
4 ===============================
4 ===============================
5
5
6 .. note::
6 .. note::
7
7
8 This feature is experimental and behavior can change between python and
8 This feature is experimental and behavior can change between python and
9 IPython version without prior deprecation.
9 IPython version without prior deprecation.
10
10
11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
11 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
12 ability to run asynchronous code from the REPL. Constructs which are
12 ability to run asynchronous code from the REPL. Constructs which are
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
13 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
14
14
15 The examples given here are for terminal IPython, running async code in a
15 The examples given here are for terminal IPython, running async code in a
16 notebook interface or any other frontend using the Jupyter protocol needs
16 notebook interface or any other frontend using the Jupyter protocol needs
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
17 IPykernel version 5.0 or above. The details of how async code runs in IPykernel
18 will differ between IPython, IPykernel and their versions.
18 will differ between IPython, IPykernel and their versions.
19
19
20 When a supported library is used, IPython will automatically allow Futures and
20 When a supported library is used, IPython will automatically allow Futures and
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
21 Coroutines in the REPL to be ``await`` ed. This will happen if an :ref:`await
22 <await>` (or any other async constructs like async-with, async-for) is use at
22 <await>` (or any other async constructs like async-with, async-for) is use at
23 top level scope, or if any structure valid only in `async def
23 top level scope, or if any structure valid only in `async def
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
24 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
25 context are present. For example, the following being a syntax error in the
25 context are present. For example, the following being a syntax error in the
26 Python REPL::
26 Python REPL::
27
27
28 Python 3.6.0
28 Python 3.6.0
29 [GCC 4.2.1]
29 [GCC 4.2.1]
30 Type "help", "copyright", "credits" or "license" for more information.
30 Type "help", "copyright", "credits" or "license" for more information.
31 >>> import aiohttp
31 >>> import aiohttp
32 >>> result = aiohttp.get('https://api.github.com')
32 >>> session = aiohttp.ClientSession()
33 >>> result = session.get('https://api.github.com')
33 >>> response = await result
34 >>> response = await result
34 File "<stdin>", line 1
35 File "<stdin>", line 1
35 response = await result
36 response = await result
36 ^
37 ^
37 SyntaxError: invalid syntax
38 SyntaxError: invalid syntax
38
39
39 Should behave as expected in the IPython REPL::
40 Should behave as expected in the IPython REPL::
40
41
41 Python 3.6.0
42 Python 3.6.0
42 Type 'copyright', 'credits' or 'license' for more information
43 Type 'copyright', 'credits' or 'license' for more information
43 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
44
45
45 In [1]: import aiohttp
46 In [1]: import aiohttp
46 ...: result = aiohttp.get('https://api.github.com')
47 ...: session = aiohttp.ClientSession()
48 ...: result = session.get('https://api.github.com')
47
49
48 In [2]: response = await result
50 In [2]: response = await result
49 <pause for a few 100s ms>
51 <pause for a few 100s ms>
50
52
51 In [3]: await response.json()
53 In [3]: await response.json()
52 Out[3]:
54 Out[3]:
53 {'authorizations_url': 'https://api.github.com/authorizations',
55 {'authorizations_url': 'https://api.github.com/authorizations',
54 'code_search_url': 'https://api.github.com/search/code?q={query}...',
56 'code_search_url': 'https://api.github.com/search/code?q={query}...',
55 ...
57 ...
56 }
58 }
57
59
58
60
59 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
61 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
60 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
62 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can
61 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
63 also use the :magic:`%autoawait` magic to toggle the behavior at runtime::
62
64
63 In [1]: %autoawait False
65 In [1]: %autoawait False
64
66
65 In [2]: %autoawait
67 In [2]: %autoawait
66 IPython autoawait is `Off`, and set to use `asyncio`
68 IPython autoawait is `Off`, and set to use `asyncio`
67
69
68
70
69
71
70 By default IPython will assume integration with Python's provided
72 By default IPython will assume integration with Python's provided
71 :mod:`asyncio`, but integration with other libraries is provided. In particular
73 :mod:`asyncio`, but integration with other libraries is provided. In particular
72 we provide experimental integration with the ``curio`` and ``trio`` library.
74 we provide experimental integration with the ``curio`` and ``trio`` library.
73
75
74 You can switch current integration by using the
76 You can switch current integration by using the
75 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
77 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
76 integration>`` magic.
78 integration>`` magic.
77
79
78 For example::
80 For example::
79
81
80 In [1]: %autoawait trio
82 In [1]: %autoawait trio
81
83
82 In [2]: import trio
84 In [2]: import trio
83
85
84 In [3]: async def child(i):
86 In [3]: async def child(i):
85 ...: print(" child %s goes to sleep"%i)
87 ...: print(" child %s goes to sleep"%i)
86 ...: await trio.sleep(2)
88 ...: await trio.sleep(2)
87 ...: print(" child %s wakes up"%i)
89 ...: print(" child %s wakes up"%i)
88
90
89 In [4]: print('parent start')
91 In [4]: print('parent start')
90 ...: async with trio.open_nursery() as n:
92 ...: async with trio.open_nursery() as n:
91 ...: for i in range(5):
93 ...: for i in range(5):
92 ...: n.spawn(child, i)
94 ...: n.spawn(child, i)
93 ...: print('parent end')
95 ...: print('parent end')
94 parent start
96 parent start
95 child 2 goes to sleep
97 child 2 goes to sleep
96 child 0 goes to sleep
98 child 0 goes to sleep
97 child 3 goes to sleep
99 child 3 goes to sleep
98 child 1 goes to sleep
100 child 1 goes to sleep
99 child 4 goes to sleep
101 child 4 goes to sleep
100 <about 2 seconds pause>
102 <about 2 seconds pause>
101 child 2 wakes up
103 child 2 wakes up
102 child 1 wakes up
104 child 1 wakes up
103 child 0 wakes up
105 child 0 wakes up
104 child 3 wakes up
106 child 3 wakes up
105 child 4 wakes up
107 child 4 wakes up
106 parent end
108 parent end
107
109
108
110
109 In the above example, ``async with`` at top level scope is a syntax error in
111 In the above example, ``async with`` at top level scope is a syntax error in
110 Python.
112 Python.
111
113
112 Using this mode can have unexpected consequences if used in interaction with
114 Using this mode can have unexpected consequences if used in interaction with
113 other features of IPython and various registered extensions. In particular if
115 other features of IPython and various registered extensions. In particular if
114 you are a direct or indirect user of the AST transformers, these may not apply
116 you are a direct or indirect user of the AST transformers, these may not apply
115 to your code.
117 to your code.
116
118
117 When using command line IPython, the default loop (or runner) does not process
119 When using command line IPython, the default loop (or runner) does not process
118 in the background, so top level asynchronous code must finish for the REPL to
120 in the background, so top level asynchronous code must finish for the REPL to
119 allow you to enter more code. As with usual Python semantic, the awaitables are
121 allow you to enter more code. As with usual Python semantic, the awaitables are
120 started only when awaited for the first time. That is to say, in first example,
122 started only when awaited for the first time. That is to say, in first example,
121 no network request is done between ``In[1]`` and ``In[2]``.
123 no network request is done between ``In[1]`` and ``In[2]``.
122
124
123
125
124 Effects on IPython.embed()
126 Effects on IPython.embed()
125 --------------------------
127 --------------------------
126
128
127 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
129 IPython core being asynchronous, the use of ``IPython.embed()`` will now require
128 a loop to run. By default IPython will use a fake coroutine runner which should
130 a loop to run. By default IPython will use a fake coroutine runner which should
129 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
131 allow ``IPython.embed()`` to be nested. Though this will prevent usage of the
130 :magic:`%autoawait` feature when using IPython embed.
132 :magic:`%autoawait` feature when using IPython embed.
131
133
132 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
134 You can set explicitly a coroutine runner for ``embed()`` if you desire to run
133 asynchronous code, the exact behavior is though undefined.
135 asynchronous code, the exact behavior is though undefined.
134
136
135 Effects on Magics
137 Effects on Magics
136 -----------------
138 -----------------
137
139
138 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
140 A couple of magics (``%%timeit``, ``%timeit``, ``%%time``, ``%%prun``) have not
139 yet been updated to work with asynchronous code and will raise syntax errors
141 yet been updated to work with asynchronous code and will raise syntax errors
140 when trying to use top-level ``await``. We welcome any contribution to help fix
142 when trying to use top-level ``await``. We welcome any contribution to help fix
141 those, and extra cases we haven't caught yet. We hope for better support in Cor
143 those, and extra cases we haven't caught yet. We hope for better support in Cor
142 Python for top-level Async code.
144 Python for top-level Async code.
143
145
144 Internals
146 Internals
145 ---------
147 ---------
146
148
147 As running asynchronous code is not supported in interactive REPL (as of Python
149 As running asynchronous code is not supported in interactive REPL (as of Python
148 3.7) we have to rely to a number of complex workaround and heuristic to allow
150 3.7) we have to rely to a number of complex workaround and heuristic to allow
149 this to happen. It is interesting to understand how this works in order to
151 this to happen. It is interesting to understand how this works in order to
150 comprehend potential bugs, or provide a custom runner.
152 comprehend potential bugs, or provide a custom runner.
151
153
152 Among the many approaches that are at our disposition, we find only one that
154 Among the many approaches that are at our disposition, we find only one that
153 suited out need. Under the hood we use the code object from a async-def function
155 suited out need. Under the hood we use the code object from a async-def function
154 and run it in global namespace after modifying it to not create a new
156 and run it in global namespace after modifying it to not create a new
155 ``locals()`` scope::
157 ``locals()`` scope::
156
158
157 async def inner_async():
159 async def inner_async():
158 locals().update(**global_namespace)
160 locals().update(**global_namespace)
159 #
161 #
160 # here is user code
162 # here is user code
161 #
163 #
162 return last_user_statement
164 return last_user_statement
163 codeobj = modify(inner_async.__code__)
165 codeobj = modify(inner_async.__code__)
164 coroutine = eval(codeobj, user_ns)
166 coroutine = eval(codeobj, user_ns)
165 display(loop_runner(coroutine))
167 display(loop_runner(coroutine))
166
168
167
169
168
170
169 The first thing you'll notice is that unlike classical ``exec``, there is only
171 The first thing you'll notice is that unlike classical ``exec``, there is only
170 one namespace. Second, user code runs in a function scope, and not a module
172 one namespace. Second, user code runs in a function scope, and not a module
171 scope.
173 scope.
172
174
173 On top of the above there are significant modification to the AST of
175 On top of the above there are significant modification to the AST of
174 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
176 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
175 significant overhead to this kind of code.
177 significant overhead to this kind of code.
176
178
177 By default the generated coroutine function will be consumed by Asyncio's
179 By default the generated coroutine function will be consumed by Asyncio's
178 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
180 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method if
179 ``async`` mode is deemed necessary, otherwise the coroutine will just be
181 ``async`` mode is deemed necessary, otherwise the coroutine will just be
180 exhausted in a simple runner. It is though possible to change the default
182 exhausted in a simple runner. It is though possible to change the default
181 runner.
183 runner.
182
184
183 A loop runner is a *synchronous* function responsible from running a coroutine
185 A loop runner is a *synchronous* function responsible from running a coroutine
184 object.
186 object.
185
187
186 The runner is responsible from ensuring that ``coroutine`` run to completion,
188 The runner is responsible from ensuring that ``coroutine`` run to completion,
187 and should return the result of executing the coroutine. Let's write a
189 and should return the result of executing the coroutine. Let's write a
188 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
190 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
189 special as it usually prefer to run a function object and make a coroutine by
191 special as it usually prefer to run a function object and make a coroutine by
190 itself, we can get around this limitation by wrapping it in an async-def without
192 itself, we can get around this limitation by wrapping it in an async-def without
191 parameters and passing this value to ``trio``::
193 parameters and passing this value to ``trio``::
192
194
193
195
194 In [1]: import trio
196 In [1]: import trio
195 ...: from types import CoroutineType
197 ...: from types import CoroutineType
196 ...:
198 ...:
197 ...: def trio_runner(coro:CoroutineType):
199 ...: def trio_runner(coro:CoroutineType):
198 ...: print('running asynchronous code')
200 ...: print('running asynchronous code')
199 ...: async def corowrap(coro):
201 ...: async def corowrap(coro):
200 ...: return await coro
202 ...: return await coro
201 ...: return trio.run(corowrap, coro)
203 ...: return trio.run(corowrap, coro)
202
204
203 We can set it up by passing it to ``%autoawait``::
205 We can set it up by passing it to ``%autoawait``::
204
206
205 In [2]: %autoawait trio_runner
207 In [2]: %autoawait trio_runner
206
208
207 In [3]: async def async_hello(name):
209 In [3]: async def async_hello(name):
208 ...: await trio.sleep(1)
210 ...: await trio.sleep(1)
209 ...: print(f'Hello {name} world !')
211 ...: print(f'Hello {name} world !')
210 ...: await trio.sleep(1)
212 ...: await trio.sleep(1)
211
213
212 In [4]: await async_hello('async')
214 In [4]: await async_hello('async')
213 running asynchronous code
215 running asynchronous code
214 Hello async world !
216 Hello async world !
215
217
216
218
217 Asynchronous programming in python (and in particular in the REPL) is still a
219 Asynchronous programming in python (and in particular in the REPL) is still a
218 relatively young subject. We expect some code to not behave as you expect, so
220 relatively young subject. We expect some code to not behave as you expect, so
219 feel free to contribute improvements to this codebase and give us feedback.
221 feel free to contribute improvements to this codebase and give us feedback.
220
222
221 We invite you to thoroughly test this feature and report any unexpected behavior
223 We invite you to thoroughly test this feature and report any unexpected behavior
222 as well as propose any improvement.
224 as well as propose any improvement.
223
225
224 Using Autoawait in a notebook (IPykernel)
226 Using Autoawait in a notebook (IPykernel)
225 -----------------------------------------
227 -----------------------------------------
226
228
227 Update ipykernel to version 5.0 or greater::
229 Update ipykernel to version 5.0 or greater::
228
230
229 pip install ipykernel ipython --upgrade
231 pip install ipykernel ipython --upgrade
230 # or
232 # or
231 conda install ipykernel ipython --upgrade
233 conda install ipykernel ipython --upgrade
232
234
233 This should automatically enable :magic:`autoawait` integration. Unlike
235 This should automatically enable :magic:`autoawait` integration. Unlike
234 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
236 terminal IPython, all code runs on ``asyncio`` eventloop, so creating a loop by
235 hand will not work, including with magics like :magic:`%run` or other
237 hand will not work, including with magics like :magic:`%run` or other
236 frameworks that create the eventloop themselves. In cases like these you can
238 frameworks that create the eventloop themselves. In cases like these you can
237 try to use projects like `nest_asyncio
239 try to use projects like `nest_asyncio
238 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
240 <https://github.com/erdewit/nest_asyncio>`_ and follow `this discussion
239 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
241 <https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811>`_
240
242
241 Difference between terminal IPython and IPykernel
243 Difference between terminal IPython and IPykernel
242 -------------------------------------------------
244 -------------------------------------------------
243
245
244 The exact asynchronous code running behavior varies between Terminal IPython and
246 The exact asynchronous code running behavior varies between Terminal IPython and
245 IPykernel. The root cause of this behavior is due to IPykernel having a
247 IPykernel. The root cause of this behavior is due to IPykernel having a
246 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
248 *persistent* `asyncio` loop running, while Terminal IPython starts and stops a
247 loop for each code block. This can lead to surprising behavior in some case if
249 loop for each code block. This can lead to surprising behavior in some case if
248 you are used to manipulate asyncio loop yourself, see for example
250 you are used to manipulate asyncio loop yourself, see for example
249 :ghissue:`11303` for a longer discussion but here are some of the astonishing
251 :ghissue:`11303` for a longer discussion but here are some of the astonishing
250 cases.
252 cases.
251
253
252 This behavior is an implementation detail, and should not be relied upon. It can
254 This behavior is an implementation detail, and should not be relied upon. It can
253 change without warnings in future versions of IPython.
255 change without warnings in future versions of IPython.
254
256
255 In terminal IPython a loop is started for each code blocks only if there is top
257 In terminal IPython a loop is started for each code blocks only if there is top
256 level async code::
258 level async code::
257
259
258 $ ipython
260 $ ipython
259 In [1]: import asyncio
261 In [1]: import asyncio
260 ...: asyncio.get_event_loop()
262 ...: asyncio.get_event_loop()
261 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
263 Out[1]: <_UnixSelectorEventLoop running=False closed=False debug=False>
262
264
263 In [2]:
265 In [2]:
264
266
265 In [2]: import asyncio
267 In [2]: import asyncio
266 ...: await asyncio.sleep(0)
268 ...: await asyncio.sleep(0)
267 ...: asyncio.get_event_loop()
269 ...: asyncio.get_event_loop()
268 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
270 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
269
271
270 See that ``running`` is ``True`` only in the case were we ``await sleep()``
272 See that ``running`` is ``True`` only in the case were we ``await sleep()``
271
273
272 In a Notebook, with ipykernel the asyncio eventloop is always running::
274 In a Notebook, with ipykernel the asyncio eventloop is always running::
273
275
274 $ jupyter notebook
276 $ jupyter notebook
275 In [1]: import asyncio
277 In [1]: import asyncio
276 ...: loop1 = asyncio.get_event_loop()
278 ...: loop1 = asyncio.get_event_loop()
277 ...: loop1
279 ...: loop1
278 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
280 Out[1]: <_UnixSelectorEventLoop running=True closed=False debug=False>
279
281
280 In [2]: loop2 = asyncio.get_event_loop()
282 In [2]: loop2 = asyncio.get_event_loop()
281 ...: loop2
283 ...: loop2
282 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
284 Out[2]: <_UnixSelectorEventLoop running=True closed=False debug=False>
283
285
284 In [3]: loop1 is loop2
286 In [3]: loop1 is loop2
285 Out[3]: True
287 Out[3]: True
286
288
287 In Terminal IPython background tasks are only processed while the foreground
289 In Terminal IPython background tasks are only processed while the foreground
288 task is running, if and only if the foreground task is async::
290 task is running, if and only if the foreground task is async::
289
291
290 $ ipython
292 $ ipython
291 In [1]: import asyncio
293 In [1]: import asyncio
292 ...:
294 ...:
293 ...: async def repeat(msg, n):
295 ...: async def repeat(msg, n):
294 ...: for i in range(n):
296 ...: for i in range(n):
295 ...: print(f"{msg} {i}")
297 ...: print(f"{msg} {i}")
296 ...: await asyncio.sleep(1)
298 ...: await asyncio.sleep(1)
297 ...: return f"{msg} done"
299 ...: return f"{msg} done"
298 ...:
300 ...:
299 ...: asyncio.ensure_future(repeat("background", 10))
301 ...: asyncio.ensure_future(repeat("background", 10))
300 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
302 Out[1]: <Task pending coro=<repeat() running at <ipython-input-1-02d0ef250fe7>:3>>
301
303
302 In [2]: await asyncio.sleep(3)
304 In [2]: await asyncio.sleep(3)
303 background 0
305 background 0
304 background 1
306 background 1
305 background 2
307 background 2
306 background 3
308 background 3
307
309
308 In [3]: import time
310 In [3]: import time
309 ...: time.sleep(5)
311 ...: time.sleep(5)
310
312
311 In [4]: await asyncio.sleep(3)
313 In [4]: await asyncio.sleep(3)
312 background 4
314 background 4
313 background 5
315 background 5
314 background 6g
316 background 6g
315
317
316 In a Notebook, QtConsole, or any other frontend using IPykernel, background
318 In a Notebook, QtConsole, or any other frontend using IPykernel, background
317 tasks should behave as expected.
319 tasks should behave as expected.
General Comments 0
You need to be logged in to leave comments. Login now