##// END OF EJS Templates
Merge pull request #6550 from drevicko/update-parallel-magic-doc...
Min RK -
r18017:26b04310 merge
parent child Browse files
Show More
@@ -1,387 +1,388
1 1 .. _parallel_magics:
2 2
3 3 =======================
4 4 Parallel Magic Commands
5 5 =======================
6 6
7 7 We provide a few IPython magic commands
8 8 that make it a bit more pleasant to execute Python commands on the engines interactively.
9 9 These are mainly shortcuts to :meth:`.DirectView.execute`
10 10 and :meth:`.AsyncResult.display_outputs` methods respectively.
11 11
12 12 These magics will automatically become available when you create a Client:
13 13
14 14 .. sourcecode:: ipython
15 15
16 In [2]: rc = parallel.Client()
16 In [1]: from IPython.parallel import Client
17 In [2]: rc = Client()
17 18
18 19 The initially active View will have attributes ``targets='all', block=True``,
19 20 which is a blocking view of all engines, evaluated at request time
20 21 (adding/removing engines will change where this view's tasks will run).
21 22
22 23 The Magics
23 24 ==========
24 25
25 26 %px
26 27 ---
27 28
28 29 The %px magic executes a single Python command on the engines
29 30 specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
30 31
31 32 .. sourcecode:: ipython
32 33
33 34 # import numpy here and everywhere
34 35 In [25]: with rc[:].sync_imports():
35 36 ....: import numpy
36 37 importing numpy on engine(s)
37 38
38 39 In [27]: %px a = numpy.random.rand(2,2)
39 40 Parallel execution on engines: [0, 1, 2, 3]
40 41
41 42 In [28]: %px numpy.linalg.eigvals(a)
42 43 Parallel execution on engines: [0, 1, 2, 3]
43 44 Out [0:68]: array([ 0.77120707, -0.19448286])
44 45 Out [1:68]: array([ 1.10815921, 0.05110369])
45 46 Out [2:68]: array([ 0.74625527, -0.37475081])
46 47 Out [3:68]: array([ 0.72931905, 0.07159743])
47 48
48 49 In [29]: %px print 'hi'
49 50 Parallel execution on engine(s): all
50 51 [stdout:0] hi
51 52 [stdout:1] hi
52 53 [stdout:2] hi
53 54 [stdout:3] hi
54 55
55 56
56 57 Since engines are IPython as well, you can even run magics remotely:
57 58
58 59 .. sourcecode:: ipython
59 60
60 61 In [28]: %px %pylab inline
61 62 Parallel execution on engine(s): all
62 63 [stdout:0]
63 64 Populating the interactive namespace from numpy and matplotlib
64 65 [stdout:1]
65 66 Populating the interactive namespace from numpy and matplotlib
66 67 [stdout:2]
67 68 Populating the interactive namespace from numpy and matplotlib
68 69 [stdout:3]
69 70 Populating the interactive namespace from numpy and matplotlib
70 71
71 72 And once in pylab mode with the inline backend,
72 73 you can make plots and they will be displayed in your frontend
73 74 if it supports the inline figures (e.g. notebook or qtconsole):
74 75
75 76 .. sourcecode:: ipython
76 77
77 78 In [40]: %px plot(rand(100))
78 79 Parallel execution on engine(s): all
79 80 <plot0>
80 81 <plot1>
81 82 <plot2>
82 83 <plot3>
83 84 Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
84 85 Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
85 86 Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
86 87 Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
87 88
88 89
89 90 %%px Cell Magic
90 91 ---------------
91 92
92 93 %%px can be used as a Cell Magic, which accepts some arguments for controlling
93 94 the execution.
94 95
95 96
96 97 Targets and Blocking
97 98 ********************
98 99
99 100 %%px accepts ``--targets`` for controlling which engines on which to run,
100 101 and ``--[no]block`` for specifying the blocking behavior of this cell,
101 102 independent of the defaults for the View.
102 103
103 104 .. sourcecode:: ipython
104 105
105 106 In [6]: %%px --targets ::2
106 107 ...: print "I am even"
107 108 ...:
108 109 Parallel execution on engine(s): [0, 2]
109 110 [stdout:0] I am even
110 111 [stdout:2] I am even
111 112
112 113 In [7]: %%px --targets 1
113 114 ...: print "I am number 1"
114 115 ...:
115 116 Parallel execution on engine(s): 1
116 117 I am number 1
117 118
118 119 In [8]: %%px
119 120 ...: print "still 'all' by default"
120 121 ...:
121 122 Parallel execution on engine(s): all
122 123 [stdout:0] still 'all' by default
123 124 [stdout:1] still 'all' by default
124 125 [stdout:2] still 'all' by default
125 126 [stdout:3] still 'all' by default
126 127
127 128 In [9]: %%px --noblock
128 129 ...: import time
129 130 ...: time.sleep(1)
130 131 ...: time.time()
131 132 ...:
132 133 Async parallel execution on engine(s): all
133 134 Out[9]: <AsyncResult: execute>
134 135
135 136 In [10]: %pxresult
136 137 Out[0:12]: 1339454561.069116
137 138 Out[1:10]: 1339454561.076752
138 139 Out[2:12]: 1339454561.072837
139 140 Out[3:10]: 1339454561.066665
140 141
141 142
142 143 .. seealso::
143 144
144 145 :ref:`pxconfig` accepts these same arguments for changing the *default*
145 146 values of targets/blocking for the active View.
146 147
147 148
148 149 Output Display
149 150 **************
150 151
151 152
152 153 %%px also accepts a ``--group-outputs`` argument,
153 154 which adjusts how the outputs of multiple engines are presented.
154 155
155 156 .. seealso::
156 157
157 158 :meth:`.AsyncResult.display_outputs` for the grouping options.
158 159
159 160 .. sourcecode:: ipython
160 161
161 162 In [50]: %%px --block --group-outputs=engine
162 163 ....: import numpy as np
163 164 ....: A = np.random.random((2,2))
164 165 ....: ev = numpy.linalg.eigvals(A)
165 166 ....: print ev
166 167 ....: ev.max()
167 168 ....:
168 169 Parallel execution on engine(s): all
169 170 [stdout:0] [ 0.60640442 0.95919621]
170 171 Out [0:73]: 0.9591962130899806
171 172 [stdout:1] [ 0.38501813 1.29430871]
172 173 Out [1:73]: 1.2943087091452372
173 174 [stdout:2] [-0.85925141 0.9387692 ]
174 175 Out [2:73]: 0.93876920456230284
175 176 [stdout:3] [ 0.37998269 1.24218246]
176 177 Out [3:73]: 1.2421824618493817
177 178
178 179
179 180 %pxresult
180 181 ---------
181 182
182 183 If you are using %px in non-blocking mode, you won't get output.
183 184 You can use %pxresult to display the outputs of the latest command,
184 185 just as is done when %px is blocking:
185 186
186 187 .. sourcecode:: ipython
187 188
188 189 In [39]: dv.block = False
189 190
190 191 In [40]: %px print 'hi'
191 192 Async parallel execution on engine(s): all
192 193
193 194 In [41]: %pxresult
194 195 [stdout:0] hi
195 196 [stdout:1] hi
196 197 [stdout:2] hi
197 198 [stdout:3] hi
198 199
199 200 %pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
200 201 It accepts the same output-grouping arguments as %%px, so you can use it to view
201 202 a result in different ways.
202 203
203 204
204 205 %autopx
205 206 -------
206 207
207 208 The %autopx magic switches to a mode where everything you type is executed
208 209 on the engines until you do %autopx again.
209 210
210 211 .. sourcecode:: ipython
211 212
212 213 In [30]: dv.block=True
213 214
214 215 In [31]: %autopx
215 216 %autopx enabled
216 217
217 218 In [32]: max_evals = []
218 219
219 220 In [33]: for i in range(100):
220 221 ....: a = numpy.random.rand(10,10)
221 222 ....: a = a+a.transpose()
222 223 ....: evals = numpy.linalg.eigvals(a)
223 224 ....: max_evals.append(evals[0].real)
224 225 ....:
225 226
226 227 In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
227 228 [stdout:0] Average max eigenvalue is: 10.193101
228 229 [stdout:1] Average max eigenvalue is: 10.064508
229 230 [stdout:2] Average max eigenvalue is: 10.055724
230 231 [stdout:3] Average max eigenvalue is: 10.086876
231 232
232 233 In [35]: %autopx
233 234 Auto Parallel Disabled
234 235
235 236 .. _pxconfig:
236 237
237 238 %pxconfig
238 239 ---------
239 240
240 241 The default targets and blocking behavior for the magics are governed by the :attr:`block`
241 242 and :attr:`targets` attribute of the active View. If you have a handle for the view,
242 243 you can set these attributes directly, but if you don't, you can change them with
243 244 the %pxconfig magic:
244 245
245 246 .. sourcecode:: ipython
246 247
247 248 In [3]: %pxconfig --block
248 249
249 250 In [5]: %px print 'hi'
250 251 Parallel execution on engine(s): all
251 252 [stdout:0] hi
252 253 [stdout:1] hi
253 254 [stdout:2] hi
254 255 [stdout:3] hi
255 256
256 257 In [6]: %pxconfig --targets ::2
257 258
258 259 In [7]: %px print 'hi'
259 260 Parallel execution on engine(s): [0, 2]
260 261 [stdout:0] hi
261 262 [stdout:2] hi
262 263
263 264 In [8]: %pxconfig --noblock
264 265
265 266 In [9]: %px print 'are you there?'
266 267 Async parallel execution on engine(s): [0, 2]
267 268 Out[9]: <AsyncResult: execute>
268 269
269 270 In [10]: %pxresult
270 271 [stdout:0] are you there?
271 272 [stdout:2] are you there?
272 273
273 274
274 275 Multiple Active Views
275 276 =====================
276 277
277 278 The parallel magics are associated with a particular :class:`~.DirectView` object.
278 279 You can change the active view by calling the :meth:`~.DirectView.activate` method
279 280 on any view.
280 281
281 282 .. sourcecode:: ipython
282 283
283 284 In [11]: even = rc[::2]
284 285
285 286 In [12]: even.activate()
286 287
287 288 In [13]: %px print 'hi'
288 289 Async parallel execution on engine(s): [0, 2]
289 290 Out[13]: <AsyncResult: execute>
290 291
291 292 In [14]: even.block = True
292 293
293 294 In [15]: %px print 'hi'
294 295 Parallel execution on engine(s): [0, 2]
295 296 [stdout:0] hi
296 297 [stdout:2] hi
297 298
298 299 When activating a View, you can also specify a *suffix*, so that a whole different
299 300 set of magics are associated with that view, without replacing the existing ones.
300 301
301 302 .. sourcecode:: ipython
302 303
303 304 # restore the original DirecView to the base %px magics
304 305 In [16]: rc.activate()
305 306 Out[16]: <DirectView all>
306 307
307 308 In [17]: even.activate('_even')
308 309
309 310 In [18]: %px print 'hi all'
310 311 Parallel execution on engine(s): all
311 312 [stdout:0] hi all
312 313 [stdout:1] hi all
313 314 [stdout:2] hi all
314 315 [stdout:3] hi all
315 316
316 317 In [19]: %px_even print "We aren't odd!"
317 318 Parallel execution on engine(s): [0, 2]
318 319 [stdout:0] We aren't odd!
319 320 [stdout:2] We aren't odd!
320 321
321 322 This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.
322 323
323 324 For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
324 325 which creates a DirectView with block=True, activates it, and returns the new View.
325 326
326 327 The initial magics registered when you create a client are the result of a call to
327 328 :meth:`rc.activate` with default args.
328 329
329 330
330 331 Engines as Kernels
331 332 ==================
332 333
333 334 Engines are really the same object as the Kernels used elsewhere in IPython,
334 335 with the minor exception that engines connect to a controller, while regular kernels
335 336 bind their sockets, listening for connections from a QtConsole or other frontends.
336 337
337 338 Sometimes for debugging or inspection purposes, you would like a QtConsole connected
338 339 to an engine for more direct interaction. You can do this by first instructing
339 340 the Engine to *also* bind its kernel, to listen for connections:
340 341
341 342 .. sourcecode:: ipython
342 343
343 344 In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
344 345
345 346 Then, if your engines are local, you can start a qtconsole right on the engine(s):
346 347
347 348 .. sourcecode:: ipython
348 349
349 350 In [51]: %px %qtconsole
350 351
351 352 Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
352 353
353 354 Or you can view just the connection info, and work out the right way to connect to the engines,
354 355 depending on where they live and where you are:
355 356
356 357 .. sourcecode:: ipython
357 358
358 359 In [51]: %px %connect_info
359 360 Parallel execution on engine(s): all
360 361 [stdout:0]
361 362 {
362 363 "stdin_port": 60387,
363 364 "ip": "127.0.0.1",
364 365 "hb_port": 50835,
365 366 "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542",
366 367 "shell_port": 55328,
367 368 "iopub_port": 58264
368 369 }
369 370
370 371 Paste the above JSON into a file, and connect with:
371 372 $> ipython <app> --existing <file>
372 373 or, if you are local, you can connect with just:
373 374 $> ipython <app> --existing kernel-60125.json
374 375 or even just:
375 376 $> ipython <app> --existing
376 377 if this is the most recent IPython session you have started.
377 378 [stdout:1]
378 379 {
379 380 "stdin_port": 61869,
380 381 ...
381 382
382 383 .. note::
383 384
384 385 ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
385 386 so you can often skip that first step.
386 387
387 388
General Comments 0
You need to be logged in to leave comments. Login now