##// END OF EJS Templates
att triio runner at top level now that 3.4 drop + docs
Matthias Bussonnier -
Show More
@@ -1,96 +1,89 b''
1 1 """
2 2 Async helper function that are invalid syntax on Python 3.5 and below.
3 3
4 4 Known limitation and possible improvement.
5 5
6 6 Top level code that contain a return statement (instead of, or in addition to
7 7 await) will be detected as requiring being wrapped in async calls. This should
8 8 be prevented as early return will not work.
9 9 """
10 10
11 11
12 12 import ast
13 13 import sys
14 14 import inspect
15 15 from textwrap import dedent, indent
16 16 from types import CodeType
17 17
18 18
19 19 def _asyncio_runner(coro):
20 20 """
21 21 Handler for asyncio autoawait
22 22 """
23 23 import asyncio
24 24
25 25 return asyncio.get_event_loop().run_until_complete(coro)
26 26
27 27
28 28 def _curio_runner(coroutine):
29 29 """
30 30 handler for curio autoawait
31 31 """
32 32 import curio
33 33
34 34 return curio.run(coroutine)
35 35
36 36
37 if sys.version_info > (3, 5):
38 # nose refuses to avoid this file and async def is invalidsyntax
39 s = dedent(
40 '''
41 def _trio_runner(function):
42 import trio
43 async def loc(coro):
44 """
45 We need the dummy no-op async def to protect from
46 trio's internal. See https://github.com/python-trio/trio/issues/89
47 """
48 return await coro
49 return trio.run(loc, function)
50 '''
51 )
52 exec(s, globals(), locals())
37 def _trio_runner(function):
38 import trio
39 async def loc(coro):
40 """
41 We need the dummy no-op async def to protect from
42 trio's internal. See https://github.com/python-trio/trio/issues/89
43 """
44 return await coro
45 return trio.run(loc, function)
53 46
54 47
55 48 def _asyncify(code: str) -> str:
56 49 """wrap code in async def definition.
57 50
58 51 And setup a bit of context to run it later.
59 52 """
60 53 res = dedent(
61 54 """
62 55 async def __wrapper__():
63 56 try:
64 57 {usercode}
65 58 finally:
66 59 locals()
67 60 """
68 61 ).format(usercode=indent(code, " " * 8)[8:])
69 62 return res
70 63
71 64
72 65 def _should_be_async(cell: str) -> bool:
73 66 """Detect if a block of code need to be wrapped in an `async def`
74 67
75 68 Attempt to parse the block of code, it it compile we're fine.
76 69 Otherwise we wrap if and try to compile.
77 70
78 71 If it works, assume it should be async. Otherwise Return False.
79 72
80 73 Not handled yet: If the block of code has a return statement as the top
81 74 level, it will be seen as async. This is a know limitation.
82 75 """
83 76
84 77 try:
85 78 # we can't limit ourself to ast.parse, as it __accepts__ to parse on
86 79 # 3.7+, but just does not _compile_
87 80 compile(cell, "<>", "exec")
88 81 return False
89 82 except SyntaxError:
90 83 try:
91 84 ast.parse(_asyncify(cell))
92 85 # TODO verify ast has not "top level" return or yield.
93 86 except SyntaxError:
94 87 return False
95 88 return True
96 89 return False
@@ -1,193 +1,203 b''
1 1 .. _autoawait:
2 2
3 3 Asynchronous in REPL: Autoawait
4 4 ===============================
5 5
6 6 Starting with IPython 7.0, and when user Python 3.6 and above, IPython offer the
7 7 ability to run asynchronous code from the REPL. Constructs which are
8 8 :exc:`SyntaxError` s in the Python REPL can be used seamlessly in IPython.
9 9
10 10 The example given here are for terminal IPython, running async code in a
11 11 notebook interface or any other frontend using the Jupyter protocol will need to
12 12 use a newer version of IPykernel. The details of how async code runs in
13 13 IPykernel will differ between IPython, IPykernel and their versions.
14 14
15 When a supported library is used, IPython will automatically `await` Futures
16 and Coroutines in the REPL. This will happen if an :ref:`await <await>` is
17 use at top level scope, or if any structure valid only in `async def
15 When a supported library is used, IPython will automatically `await` Futures and
16 Coroutines in the REPL. This will happen if an :ref:`await <await>` (or any
17 other async constructs like async-with, async-for) is use at top level scope, or
18 if any structure valid only in `async def
18 19 <https://docs.python.org/3/reference/compound_stmts.html#async-def>`_ function
19 20 context are present. For example, the following being a syntax error in the
20 21 Python REPL::
21 22
22 23 Python 3.6.0
23 24 [GCC 4.2.1]
24 25 Type "help", "copyright", "credits" or "license" for more information.
25 26 >>> import aiohttp
26 27 >>> result = aiohttp.get('https://api.github.com')
27 28 >>> response = await result
28 29 File "<stdin>", line 1
29 30 response = await result
30 31 ^
31 32 SyntaxError: invalid syntax
32 33
33 34 Should behave as expected in the IPython REPL::
34 35
35 36 Python 3.6.0
36 37 Type 'copyright', 'credits' or 'license' for more information
37 38 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
38 39
39 40 In [1]: import aiohttp
40 41 ...: result = aiohttp.get('https://api.github.com')
41 42
42 43 In [2]: response = await result
43 44 <pause for a few 100s ms>
44 45
45 46 In [3]: await response.json()
46 47 Out[3]:
47 48 {'authorizations_url': 'https://api.github.com/authorizations',
48 49 'code_search_url': 'https://api.github.com/search/code?q={query}...',
49 50 ...
50 51 }
51 52
52 53
53 54 You can use the ``c.InteractiveShell.autoawait`` configuration option and set it
54 55 to :any:`False` to deactivate automatic wrapping of asynchronous code. You can also
55 56 use the :magic:`%autoawait` magic to toggle the behavior at runtime::
56 57
57 58 In [1]: %autoawait False
58 59
59 60 In [2]: %autoawait
60 61 IPython autoawait is `Off`, and set to use `IPython.core.interactiveshell._asyncio_runner`
61 62
62 63
63 64
64 65 By default IPython will assume integration with Python's provided
65 66 :mod:`asyncio`, but integration with other libraries is provided. In particular
66 67 we provide experimental integration with the ``curio`` and ``trio`` library, the
67 68 later one being necessary if you require the ability to do nested call of
68 69 IPython's ``embed()`` functionality.
69 70
70 71 You can switch current integration by using the
71 72 ``c.InteractiveShell.loop_runner`` option or the ``autoawait <name
72 73 integration>`` magic.
73 74
74 75 For example::
75 76
76 77 In [1]: %autoawait trio
77 78
78 79 In [2]: import trio
79 80
80 81 In [3]: async def child(i):
81 82 ...: print(" child %s goes to sleep"%i)
82 83 ...: await trio.sleep(2)
83 84 ...: print(" child %s wakes up"%i)
84 85
85 86 In [4]: print('parent start')
86 87 ...: async with trio.open_nursery() as n:
87 88 ...: for i in range(5):
88 89 ...: n.spawn(child, i)
89 90 ...: print('parent end')
90 91 parent start
91 92 child 2 goes to sleep
92 93 child 0 goes to sleep
93 94 child 3 goes to sleep
94 95 child 1 goes to sleep
95 96 child 4 goes to sleep
96 97 <about 2 seconds pause>
97 98 child 2 wakes up
98 99 child 1 wakes up
99 100 child 0 wakes up
100 101 child 3 wakes up
101 102 child 4 wakes up
102 103 parent end
103 104
104 105
105 106 In the above example, ``async with`` at top level scope is a syntax error in
106 107 Python.
107 108
108 109 Using this mode can have unexpected consequences if used in interaction with
109 110 other features of IPython and various registered extensions. In particular if you
110 111 are a direct or indirect user of the AST transformers, these may not apply to
111 112 your code.
112 113
113 114 When Using command line IPython, the default loop (or runner) does not process
114 115 in the background, so top level asynchronous code must finish for the REPL to
115 116 allow you to enter more code. As with usual Python semantic, the awaitables are
116 117 started only when awaited for the first time. That is to say, in first example,
117 118 no network request is done between ``In[1]`` and ``In[2]``.
118 119
119 120
121 Effects on IPython.embed()
122 ==========================
123
124 IPython core being synchronous, the use of ``IPython.embed()`` will now require
125 a loop to run. This affect the ability to nest ``IPython.embed()`` which may
126 require you to install alternate IO libraries like ``curio`` and ``trio``
127
128
129
120 130 Internals
121 131 =========
122 132
123 133 As running asynchronous code is not supported in interactive REPL (as of Python
124 134 3.7) we have to rely to a number of complex workaround to allow this to happen.
125 135 It is interesting to understand how this works in order to comprehend potential
126 136 bugs, or provide a custom runner.
127 137
128 138 Among the many approaches that are at our disposition, we find only one that
129 139 suited out need. Under the hood we use the code object from a async-def function
130 140 and run it in global namespace after modifying it to not create a new
131 141 ``locals()`` scope::
132 142
133 143 async def inner_async():
134 144 locals().update(**global_namespace)
135 145 #
136 146 # here is user code
137 147 #
138 148 return last_user_statement
139 149 codeobj = modify(inner_async.__code__)
140 150 coroutine = eval(codeobj, user_ns)
141 151 display(loop_runner(coroutine))
142 152
143 153
144 154
145 155 The first thing you'll notice is that unlike classical ``exec``, there is only
146 156 one namespace. Second, user code runs in a function scope, and not a module
147 157 scope.
148 158
149 159 On top of the above there are significant modification to the AST of
150 160 ``function``, and ``loop_runner`` can be arbitrary complex. So there is a
151 161 significant overhead to this kind of code.
152 162
153 163 By default the generated coroutine function will be consumed by Asyncio's
154 164 ``loop_runner = asyncio.get_evenloop().run_until_complete()`` method. It is
155 165 though possible to provide your own.
156 166
157 167 A loop runner is a *synchronous* function responsible from running a coroutine
158 168 object.
159 169
160 170 The runner is responsible from ensuring that ``coroutine`` run to completion,
161 171 and should return the result of executing the coroutine. Let's write a
162 172 runner for ``trio`` that print a message when used as an exercise, ``trio`` is
163 173 special as it usually prefer to run a function object and make a coroutine by
164 174 itself, we can get around this limitation by wrapping it in an async-def without
165 175 parameters and passing this value to ``trio``::
166 176
167 177
168 178 In [1]: import trio
169 179 ...: from types import CoroutineType
170 180 ...:
171 181 ...: def trio_runner(coro:CoroutineType):
172 182 ...: print('running asynchronous code')
173 183 ...: async def corowrap(coro):
174 184 ...: return await coro
175 185 ...: return trio.run(corowrap, coro)
176 186
177 187 We can set it up by passing it to ``%autoawait``::
178 188
179 189 In [2]: %autoawait trio_runner
180 190
181 191 In [3]: async def async_hello(name):
182 192 ...: await trio.sleep(1)
183 193 ...: print(f'Hello {name} world !')
184 194 ...: await trio.sleep(1)
185 195
186 196 In [4]: await async_hello('async')
187 197 running asynchronous code
188 198 Hello async world !
189 199
190 200
191 201 Asynchronous programming in python (and in particular in the REPL) is still a
192 202 relatively young subject. We expect some code to not behave as you expect, so
193 203 feel free to contribute improvements to this codebase and give us feedback.
@@ -1,69 +1,69 b''
1 1 Autowait: Asynchronous REPL
2 2 ---------------------------
3 3
4 4 Staring with IPython 7.0 and on Python 3.6+, IPython can automatically await
5 5 code at top level, you should not need to access an event loop or runner
6 6 yourself. To know more read the :ref:`autoawait` section of our docs, see
7 7 :ghpull:`11265` or try the following code::
8 8
9 9 Python 3.6.0
10 10 Type 'copyright', 'credits' or 'license' for more information
11 11 IPython 7.0.0 -- An enhanced Interactive Python. Type '?' for help.
12 12
13 13 In [1]: import aiohttp
14 14 ...: result = aiohttp.get('https://api.github.com')
15 15
16 16 In [2]: response = await result
17 17 <pause for a few 100s ms>
18 18
19 19 In [3]: await response.json()
20 20 Out[3]:
21 21 {'authorizations_url': 'https://api.github.com/authorizations',
22 22 'code_search_url': 'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}',
23 23 ...
24 24 }
25 25
26 26
27 27 Integration is by default with `asyncio`, but other libraries can be configured,
28 28 like ``curio`` or ``trio``, to improve concurrency in the REPL::
29 29
30 30 In [1]: %autoawait trio
31 31
32 32 In [2]: import trio
33 33
34 34 In [3]: async def child(i):
35 35 ...: print(" child %s goes to sleep"%i)
36 36 ...: await trio.sleep(2)
37 37 ...: print(" child %s wakes up"%i)
38 38
39 39 In [4]: print('parent start')
40 40 ...: async with trio.open_nursery() as n:
41 41 ...: for i in range(3):
42 42 ...: n.spawn(child, i)
43 43 ...: print('parent end')
44 44 parent start
45 45 child 2 goes to sleep
46 46 child 0 goes to sleep
47 47 child 1 goes to sleep
48 48 <about 2 seconds pause>
49 49 child 2 wakes up
50 50 child 1 wakes up
51 51 child 0 wakes up
52 52 parent end
53 53
54 54 See :ref:`autoawait` for more information.
55 55
56 56
57 57 Asynchronous code in a Notebook interface or any other frontend using the
58 58 Jupyter Protocol will need further updates of the IPykernel package.
59 59
60 60
61 61 Change to Nested Embed
62 62 ----------------------
63 63
64 64 The introduction of the ability to run async code had ripple effect on the
65 65 ability to use nested IPython. You may need to install the ``trio`` library
66 (version 05 at the time of this writing) to
66 (version 0.5 at the time of this writing) to
67 67 have this feature working.
68 68
69 69
General Comments 0
You need to be logged in to leave comments. Login now