##// END OF EJS Templates
added import command...
drevicko -
Show More
@@ -1,387 +1,388
1 .. _parallel_magics:
1 .. _parallel_magics:
2
2
3 =======================
3 =======================
4 Parallel Magic Commands
4 Parallel Magic Commands
5 =======================
5 =======================
6
6
7 We provide a few IPython magic commands
7 We provide a few IPython magic commands
8 that make it a bit more pleasant to execute Python commands on the engines interactively.
8 that make it a bit more pleasant to execute Python commands on the engines interactively.
9 These are mainly shortcuts to :meth:`.DirectView.execute`
9 These are mainly shortcuts to :meth:`.DirectView.execute`
10 and :meth:`.AsyncResult.display_outputs` methods respectively.
10 and :meth:`.AsyncResult.display_outputs` methods respectively.
11
11
12 These magics will automatically become available when you create a Client:
12 These magics will automatically become available when you create a Client:
13
13
14 .. sourcecode:: ipython
14 .. sourcecode:: ipython
15
15
16 In [2]: rc = parallel.Client()
16 In [1]: from IPython.parallel import Client
17 In [2]: rc = Client()
17
18
18 The initially active View will have attributes ``targets='all', block=True``,
19 The initially active View will have attributes ``targets='all', block=True``,
19 which is a blocking view of all engines, evaluated at request time
20 which is a blocking view of all engines, evaluated at request time
20 (adding/removing engines will change where this view's tasks will run).
21 (adding/removing engines will change where this view's tasks will run).
21
22
22 The Magics
23 The Magics
23 ==========
24 ==========
24
25
25 %px
26 %px
26 ---
27 ---
27
28
28 The %px magic executes a single Python command on the engines
29 The %px magic executes a single Python command on the engines
29 specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
30 specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
30
31
31 .. sourcecode:: ipython
32 .. sourcecode:: ipython
32
33
33 # import numpy here and everywhere
34 # import numpy here and everywhere
34 In [25]: with rc[:].sync_imports():
35 In [25]: with rc[:].sync_imports():
35 ....: import numpy
36 ....: import numpy
36 importing numpy on engine(s)
37 importing numpy on engine(s)
37
38
38 In [27]: %px a = numpy.random.rand(2,2)
39 In [27]: %px a = numpy.random.rand(2,2)
39 Parallel execution on engines: [0, 1, 2, 3]
40 Parallel execution on engines: [0, 1, 2, 3]
40
41
41 In [28]: %px numpy.linalg.eigvals(a)
42 In [28]: %px numpy.linalg.eigvals(a)
42 Parallel execution on engines: [0, 1, 2, 3]
43 Parallel execution on engines: [0, 1, 2, 3]
43 Out [0:68]: array([ 0.77120707, -0.19448286])
44 Out [0:68]: array([ 0.77120707, -0.19448286])
44 Out [1:68]: array([ 1.10815921, 0.05110369])
45 Out [1:68]: array([ 1.10815921, 0.05110369])
45 Out [2:68]: array([ 0.74625527, -0.37475081])
46 Out [2:68]: array([ 0.74625527, -0.37475081])
46 Out [3:68]: array([ 0.72931905, 0.07159743])
47 Out [3:68]: array([ 0.72931905, 0.07159743])
47
48
48 In [29]: %px print 'hi'
49 In [29]: %px print 'hi'
49 Parallel execution on engine(s): all
50 Parallel execution on engine(s): all
50 [stdout:0] hi
51 [stdout:0] hi
51 [stdout:1] hi
52 [stdout:1] hi
52 [stdout:2] hi
53 [stdout:2] hi
53 [stdout:3] hi
54 [stdout:3] hi
54
55
55
56
56 Since engines are IPython as well, you can even run magics remotely:
57 Since engines are IPython as well, you can even run magics remotely:
57
58
58 .. sourcecode:: ipython
59 .. sourcecode:: ipython
59
60
60 In [28]: %px %pylab inline
61 In [28]: %px %pylab inline
61 Parallel execution on engine(s): all
62 Parallel execution on engine(s): all
62 [stdout:0]
63 [stdout:0]
63 Populating the interactive namespace from numpy and matplotlib
64 Populating the interactive namespace from numpy and matplotlib
64 [stdout:1]
65 [stdout:1]
65 Populating the interactive namespace from numpy and matplotlib
66 Populating the interactive namespace from numpy and matplotlib
66 [stdout:2]
67 [stdout:2]
67 Populating the interactive namespace from numpy and matplotlib
68 Populating the interactive namespace from numpy and matplotlib
68 [stdout:3]
69 [stdout:3]
69 Populating the interactive namespace from numpy and matplotlib
70 Populating the interactive namespace from numpy and matplotlib
70
71
71 And once in pylab mode with the inline backend,
72 And once in pylab mode with the inline backend,
72 you can make plots and they will be displayed in your frontend
73 you can make plots and they will be displayed in your frontend
73 if it supports the inline figures (e.g. notebook or qtconsole):
74 if it supports the inline figures (e.g. notebook or qtconsole):
74
75
75 .. sourcecode:: ipython
76 .. sourcecode:: ipython
76
77
77 In [40]: %px plot(rand(100))
78 In [40]: %px plot(rand(100))
78 Parallel execution on engine(s): all
79 Parallel execution on engine(s): all
79 <plot0>
80 <plot0>
80 <plot1>
81 <plot1>
81 <plot2>
82 <plot2>
82 <plot3>
83 <plot3>
83 Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
84 Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
84 Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
85 Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
85 Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
86 Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
86 Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
87 Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
87
88
88
89
89 %%px Cell Magic
90 %%px Cell Magic
90 ---------------
91 ---------------
91
92
92 %%px can be used as a Cell Magic, which accepts some arguments for controlling
93 %%px can be used as a Cell Magic, which accepts some arguments for controlling
93 the execution.
94 the execution.
94
95
95
96
96 Targets and Blocking
97 Targets and Blocking
97 ********************
98 ********************
98
99
99 %%px accepts ``--targets`` for controlling which engines on which to run,
100 %%px accepts ``--targets`` for controlling which engines on which to run,
100 and ``--[no]block`` for specifying the blocking behavior of this cell,
101 and ``--[no]block`` for specifying the blocking behavior of this cell,
101 independent of the defaults for the View.
102 independent of the defaults for the View.
102
103
103 .. sourcecode:: ipython
104 .. sourcecode:: ipython
104
105
105 In [6]: %%px --targets ::2
106 In [6]: %%px --targets ::2
106 ...: print "I am even"
107 ...: print "I am even"
107 ...:
108 ...:
108 Parallel execution on engine(s): [0, 2]
109 Parallel execution on engine(s): [0, 2]
109 [stdout:0] I am even
110 [stdout:0] I am even
110 [stdout:2] I am even
111 [stdout:2] I am even
111
112
112 In [7]: %%px --targets 1
113 In [7]: %%px --targets 1
113 ...: print "I am number 1"
114 ...: print "I am number 1"
114 ...:
115 ...:
115 Parallel execution on engine(s): 1
116 Parallel execution on engine(s): 1
116 I am number 1
117 I am number 1
117
118
118 In [8]: %%px
119 In [8]: %%px
119 ...: print "still 'all' by default"
120 ...: print "still 'all' by default"
120 ...:
121 ...:
121 Parallel execution on engine(s): all
122 Parallel execution on engine(s): all
122 [stdout:0] still 'all' by default
123 [stdout:0] still 'all' by default
123 [stdout:1] still 'all' by default
124 [stdout:1] still 'all' by default
124 [stdout:2] still 'all' by default
125 [stdout:2] still 'all' by default
125 [stdout:3] still 'all' by default
126 [stdout:3] still 'all' by default
126
127
127 In [9]: %%px --noblock
128 In [9]: %%px --noblock
128 ...: import time
129 ...: import time
129 ...: time.sleep(1)
130 ...: time.sleep(1)
130 ...: time.time()
131 ...: time.time()
131 ...:
132 ...:
132 Async parallel execution on engine(s): all
133 Async parallel execution on engine(s): all
133 Out[9]: <AsyncResult: execute>
134 Out[9]: <AsyncResult: execute>
134
135
135 In [10]: %pxresult
136 In [10]: %pxresult
136 Out[0:12]: 1339454561.069116
137 Out[0:12]: 1339454561.069116
137 Out[1:10]: 1339454561.076752
138 Out[1:10]: 1339454561.076752
138 Out[2:12]: 1339454561.072837
139 Out[2:12]: 1339454561.072837
139 Out[3:10]: 1339454561.066665
140 Out[3:10]: 1339454561.066665
140
141
141
142
142 .. seealso::
143 .. seealso::
143
144
144 :ref:`pxconfig` accepts these same arguments for changing the *default*
145 :ref:`pxconfig` accepts these same arguments for changing the *default*
145 values of targets/blocking for the active View.
146 values of targets/blocking for the active View.
146
147
147
148
148 Output Display
149 Output Display
149 **************
150 **************
150
151
151
152
152 %%px also accepts a ``--group-outputs`` argument,
153 %%px also accepts a ``--group-outputs`` argument,
153 which adjusts how the outputs of multiple engines are presented.
154 which adjusts how the outputs of multiple engines are presented.
154
155
155 .. seealso::
156 .. seealso::
156
157
157 :meth:`.AsyncResult.display_outputs` for the grouping options.
158 :meth:`.AsyncResult.display_outputs` for the grouping options.
158
159
159 .. sourcecode:: ipython
160 .. sourcecode:: ipython
160
161
161 In [50]: %%px --block --group-outputs=engine
162 In [50]: %%px --block --group-outputs=engine
162 ....: import numpy as np
163 ....: import numpy as np
163 ....: A = np.random.random((2,2))
164 ....: A = np.random.random((2,2))
164 ....: ev = numpy.linalg.eigvals(A)
165 ....: ev = numpy.linalg.eigvals(A)
165 ....: print ev
166 ....: print ev
166 ....: ev.max()
167 ....: ev.max()
167 ....:
168 ....:
168 Parallel execution on engine(s): all
169 Parallel execution on engine(s): all
169 [stdout:0] [ 0.60640442 0.95919621]
170 [stdout:0] [ 0.60640442 0.95919621]
170 Out [0:73]: 0.9591962130899806
171 Out [0:73]: 0.9591962130899806
171 [stdout:1] [ 0.38501813 1.29430871]
172 [stdout:1] [ 0.38501813 1.29430871]
172 Out [1:73]: 1.2943087091452372
173 Out [1:73]: 1.2943087091452372
173 [stdout:2] [-0.85925141 0.9387692 ]
174 [stdout:2] [-0.85925141 0.9387692 ]
174 Out [2:73]: 0.93876920456230284
175 Out [2:73]: 0.93876920456230284
175 [stdout:3] [ 0.37998269 1.24218246]
176 [stdout:3] [ 0.37998269 1.24218246]
176 Out [3:73]: 1.2421824618493817
177 Out [3:73]: 1.2421824618493817
177
178
178
179
179 %pxresult
180 %pxresult
180 ---------
181 ---------
181
182
182 If you are using %px in non-blocking mode, you won't get output.
183 If you are using %px in non-blocking mode, you won't get output.
183 You can use %pxresult to display the outputs of the latest command,
184 You can use %pxresult to display the outputs of the latest command,
184 just as is done when %px is blocking:
185 just as is done when %px is blocking:
185
186
186 .. sourcecode:: ipython
187 .. sourcecode:: ipython
187
188
188 In [39]: dv.block = False
189 In [39]: dv.block = False
189
190
190 In [40]: %px print 'hi'
191 In [40]: %px print 'hi'
191 Async parallel execution on engine(s): all
192 Async parallel execution on engine(s): all
192
193
193 In [41]: %pxresult
194 In [41]: %pxresult
194 [stdout:0] hi
195 [stdout:0] hi
195 [stdout:1] hi
196 [stdout:1] hi
196 [stdout:2] hi
197 [stdout:2] hi
197 [stdout:3] hi
198 [stdout:3] hi
198
199
199 %pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
200 %pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
200 It accepts the same output-grouping arguments as %%px, so you can use it to view
201 It accepts the same output-grouping arguments as %%px, so you can use it to view
201 a result in different ways.
202 a result in different ways.
202
203
203
204
204 %autopx
205 %autopx
205 -------
206 -------
206
207
207 The %autopx magic switches to a mode where everything you type is executed
208 The %autopx magic switches to a mode where everything you type is executed
208 on the engines until you do %autopx again.
209 on the engines until you do %autopx again.
209
210
210 .. sourcecode:: ipython
211 .. sourcecode:: ipython
211
212
212 In [30]: dv.block=True
213 In [30]: dv.block=True
213
214
214 In [31]: %autopx
215 In [31]: %autopx
215 %autopx enabled
216 %autopx enabled
216
217
217 In [32]: max_evals = []
218 In [32]: max_evals = []
218
219
219 In [33]: for i in range(100):
220 In [33]: for i in range(100):
220 ....: a = numpy.random.rand(10,10)
221 ....: a = numpy.random.rand(10,10)
221 ....: a = a+a.transpose()
222 ....: a = a+a.transpose()
222 ....: evals = numpy.linalg.eigvals(a)
223 ....: evals = numpy.linalg.eigvals(a)
223 ....: max_evals.append(evals[0].real)
224 ....: max_evals.append(evals[0].real)
224 ....:
225 ....:
225
226
226 In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
227 In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
227 [stdout:0] Average max eigenvalue is: 10.193101
228 [stdout:0] Average max eigenvalue is: 10.193101
228 [stdout:1] Average max eigenvalue is: 10.064508
229 [stdout:1] Average max eigenvalue is: 10.064508
229 [stdout:2] Average max eigenvalue is: 10.055724
230 [stdout:2] Average max eigenvalue is: 10.055724
230 [stdout:3] Average max eigenvalue is: 10.086876
231 [stdout:3] Average max eigenvalue is: 10.086876
231
232
232 In [35]: %autopx
233 In [35]: %autopx
233 Auto Parallel Disabled
234 Auto Parallel Disabled
234
235
235 .. _pxconfig:
236 .. _pxconfig:
236
237
237 %pxconfig
238 %pxconfig
238 ---------
239 ---------
239
240
240 The default targets and blocking behavior for the magics are governed by the :attr:`block`
241 The default targets and blocking behavior for the magics are governed by the :attr:`block`
241 and :attr:`targets` attribute of the active View. If you have a handle for the view,
242 and :attr:`targets` attribute of the active View. If you have a handle for the view,
242 you can set these attributes directly, but if you don't, you can change them with
243 you can set these attributes directly, but if you don't, you can change them with
243 the %pxconfig magic:
244 the %pxconfig magic:
244
245
245 .. sourcecode:: ipython
246 .. sourcecode:: ipython
246
247
247 In [3]: %pxconfig --block
248 In [3]: %pxconfig --block
248
249
249 In [5]: %px print 'hi'
250 In [5]: %px print 'hi'
250 Parallel execution on engine(s): all
251 Parallel execution on engine(s): all
251 [stdout:0] hi
252 [stdout:0] hi
252 [stdout:1] hi
253 [stdout:1] hi
253 [stdout:2] hi
254 [stdout:2] hi
254 [stdout:3] hi
255 [stdout:3] hi
255
256
256 In [6]: %pxconfig --targets ::2
257 In [6]: %pxconfig --targets ::2
257
258
258 In [7]: %px print 'hi'
259 In [7]: %px print 'hi'
259 Parallel execution on engine(s): [0, 2]
260 Parallel execution on engine(s): [0, 2]
260 [stdout:0] hi
261 [stdout:0] hi
261 [stdout:2] hi
262 [stdout:2] hi
262
263
263 In [8]: %pxconfig --noblock
264 In [8]: %pxconfig --noblock
264
265
265 In [9]: %px print 'are you there?'
266 In [9]: %px print 'are you there?'
266 Async parallel execution on engine(s): [0, 2]
267 Async parallel execution on engine(s): [0, 2]
267 Out[9]: <AsyncResult: execute>
268 Out[9]: <AsyncResult: execute>
268
269
269 In [10]: %pxresult
270 In [10]: %pxresult
270 [stdout:0] are you there?
271 [stdout:0] are you there?
271 [stdout:2] are you there?
272 [stdout:2] are you there?
272
273
273
274
274 Multiple Active Views
275 Multiple Active Views
275 =====================
276 =====================
276
277
277 The parallel magics are associated with a particular :class:`~.DirectView` object.
278 The parallel magics are associated with a particular :class:`~.DirectView` object.
278 You can change the active view by calling the :meth:`~.DirectView.activate` method
279 You can change the active view by calling the :meth:`~.DirectView.activate` method
279 on any view.
280 on any view.
280
281
281 .. sourcecode:: ipython
282 .. sourcecode:: ipython
282
283
283 In [11]: even = rc[::2]
284 In [11]: even = rc[::2]
284
285
285 In [12]: even.activate()
286 In [12]: even.activate()
286
287
287 In [13]: %px print 'hi'
288 In [13]: %px print 'hi'
288 Async parallel execution on engine(s): [0, 2]
289 Async parallel execution on engine(s): [0, 2]
289 Out[13]: <AsyncResult: execute>
290 Out[13]: <AsyncResult: execute>
290
291
291 In [14]: even.block = True
292 In [14]: even.block = True
292
293
293 In [15]: %px print 'hi'
294 In [15]: %px print 'hi'
294 Parallel execution on engine(s): [0, 2]
295 Parallel execution on engine(s): [0, 2]
295 [stdout:0] hi
296 [stdout:0] hi
296 [stdout:2] hi
297 [stdout:2] hi
297
298
298 When activating a View, you can also specify a *suffix*, so that a whole different
299 When activating a View, you can also specify a *suffix*, so that a whole different
299 set of magics are associated with that view, without replacing the existing ones.
300 set of magics are associated with that view, without replacing the existing ones.
300
301
301 .. sourcecode:: ipython
302 .. sourcecode:: ipython
302
303
303 # restore the original DirecView to the base %px magics
304 # restore the original DirecView to the base %px magics
304 In [16]: rc.activate()
305 In [16]: rc.activate()
305 Out[16]: <DirectView all>
306 Out[16]: <DirectView all>
306
307
307 In [17]: even.activate('_even')
308 In [17]: even.activate('_even')
308
309
309 In [18]: %px print 'hi all'
310 In [18]: %px print 'hi all'
310 Parallel execution on engine(s): all
311 Parallel execution on engine(s): all
311 [stdout:0] hi all
312 [stdout:0] hi all
312 [stdout:1] hi all
313 [stdout:1] hi all
313 [stdout:2] hi all
314 [stdout:2] hi all
314 [stdout:3] hi all
315 [stdout:3] hi all
315
316
316 In [19]: %px_even print "We aren't odd!"
317 In [19]: %px_even print "We aren't odd!"
317 Parallel execution on engine(s): [0, 2]
318 Parallel execution on engine(s): [0, 2]
318 [stdout:0] We aren't odd!
319 [stdout:0] We aren't odd!
319 [stdout:2] We aren't odd!
320 [stdout:2] We aren't odd!
320
321
321 This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.
322 This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.
322
323
323 For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
324 For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
324 which creates a DirectView with block=True, activates it, and returns the new View.
325 which creates a DirectView with block=True, activates it, and returns the new View.
325
326
326 The initial magics registered when you create a client are the result of a call to
327 The initial magics registered when you create a client are the result of a call to
327 :meth:`rc.activate` with default args.
328 :meth:`rc.activate` with default args.
328
329
329
330
330 Engines as Kernels
331 Engines as Kernels
331 ==================
332 ==================
332
333
333 Engines are really the same object as the Kernels used elsewhere in IPython,
334 Engines are really the same object as the Kernels used elsewhere in IPython,
334 with the minor exception that engines connect to a controller, while regular kernels
335 with the minor exception that engines connect to a controller, while regular kernels
335 bind their sockets, listening for connections from a QtConsole or other frontends.
336 bind their sockets, listening for connections from a QtConsole or other frontends.
336
337
337 Sometimes for debugging or inspection purposes, you would like a QtConsole connected
338 Sometimes for debugging or inspection purposes, you would like a QtConsole connected
338 to an engine for more direct interaction. You can do this by first instructing
339 to an engine for more direct interaction. You can do this by first instructing
339 the Engine to *also* bind its kernel, to listen for connections:
340 the Engine to *also* bind its kernel, to listen for connections:
340
341
341 .. sourcecode:: ipython
342 .. sourcecode:: ipython
342
343
343 In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
344 In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
344
345
345 Then, if your engines are local, you can start a qtconsole right on the engine(s):
346 Then, if your engines are local, you can start a qtconsole right on the engine(s):
346
347
347 .. sourcecode:: ipython
348 .. sourcecode:: ipython
348
349
349 In [51]: %px %qtconsole
350 In [51]: %px %qtconsole
350
351
351 Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
352 Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
352
353
353 Or you can view just the connection info, and work out the right way to connect to the engines,
354 Or you can view just the connection info, and work out the right way to connect to the engines,
354 depending on where they live and where you are:
355 depending on where they live and where you are:
355
356
356 .. sourcecode:: ipython
357 .. sourcecode:: ipython
357
358
358 In [51]: %px %connect_info
359 In [51]: %px %connect_info
359 Parallel execution on engine(s): all
360 Parallel execution on engine(s): all
360 [stdout:0]
361 [stdout:0]
361 {
362 {
362 "stdin_port": 60387,
363 "stdin_port": 60387,
363 "ip": "127.0.0.1",
364 "ip": "127.0.0.1",
364 "hb_port": 50835,
365 "hb_port": 50835,
365 "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542",
366 "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542",
366 "shell_port": 55328,
367 "shell_port": 55328,
367 "iopub_port": 58264
368 "iopub_port": 58264
368 }
369 }
369
370
370 Paste the above JSON into a file, and connect with:
371 Paste the above JSON into a file, and connect with:
371 $> ipython <app> --existing <file>
372 $> ipython <app> --existing <file>
372 or, if you are local, you can connect with just:
373 or, if you are local, you can connect with just:
373 $> ipython <app> --existing kernel-60125.json
374 $> ipython <app> --existing kernel-60125.json
374 or even just:
375 or even just:
375 $> ipython <app> --existing
376 $> ipython <app> --existing
376 if this is the most recent IPython session you have started.
377 if this is the most recent IPython session you have started.
377 [stdout:1]
378 [stdout:1]
378 {
379 {
379 "stdin_port": 61869,
380 "stdin_port": 61869,
380 ...
381 ...
381
382
382 .. note::
383 .. note::
383
384
384 ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
385 ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
385 so you can often skip that first step.
386 so you can often skip that first step.
386
387
387
388
General Comments 0
You need to be logged in to leave comments. Login now