##// END OF EJS Templates
Add missing import in IPython parallel magics example
Olivier Grisel -
Show More
@@ -1,476 +1,475 b''
1 1 {
2 2 "metadata": {
3 3 "name": "Parallel Magics"
4 4 },
5 5 "nbformat": 3,
6 6 "nbformat_minor": 0,
7 7 "worksheets": [
8 8 {
9 9 "cells": [
10 10 {
11 11 "cell_type": "heading",
12 12 "level": 1,
13 13 "metadata": {},
14 14 "source": [
15 15 "Using Parallel Magics"
16 16 ]
17 17 },
18 18 {
19 19 "cell_type": "markdown",
20 20 "metadata": {},
21 21 "source": [
22 22 "IPython has a few magics for working with your engines.\n",
23 23 "\n",
24 24 "This assumes you have started an IPython cluster, either with the notebook interface,\n",
25 25 "or the `ipcluster/controller/engine` commands."
26 26 ]
27 27 },
28 28 {
29 29 "cell_type": "code",
30 30 "collapsed": false,
31 31 "input": [
32 32 "from IPython import parallel\n",
33 33 "rc = parallel.Client()\n",
34 34 "dv = rc[:]\n",
35 35 "rc.ids"
36 36 ],
37 37 "language": "python",
38 38 "metadata": {},
39 "outputs": [],
40 "prompt_number": 2
39 "outputs": []
41 40 },
42 41 {
43 42 "cell_type": "markdown",
44 43 "metadata": {},
45 44 "source": [
46 45 "Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`. \n",
47 46 "These magics are initially associated with a DirectView always associated with all currently registered engines."
48 47 ]
49 48 },
50 49 {
51 50 "cell_type": "markdown",
52 51 "metadata": {},
53 52 "source": [
54 53 "Now we can execute code remotely with `%px`:"
55 54 ]
56 55 },
57 56 {
58 57 "cell_type": "code",
59 58 "collapsed": false,
60 59 "input": [
61 60 "%px a=5"
62 61 ],
63 62 "language": "python",
64 63 "metadata": {},
65 64 "outputs": []
66 65 },
67 66 {
68 67 "cell_type": "code",
69 68 "collapsed": false,
70 69 "input": [
71 70 "%px print a"
72 71 ],
73 72 "language": "python",
74 73 "metadata": {},
75 74 "outputs": []
76 75 },
77 76 {
78 77 "cell_type": "code",
79 78 "collapsed": false,
80 79 "input": [
81 80 "%px a"
82 81 ],
83 82 "language": "python",
84 83 "metadata": {},
85 84 "outputs": []
86 85 },
87 86 {
88 87 "cell_type": "code",
89 88 "collapsed": false,
90 89 "input": [
91 90 "with dv.sync_imports():\n",
92 91 " import sys"
93 92 ],
94 93 "language": "python",
95 94 "metadata": {},
96 95 "outputs": []
97 96 },
98 97 {
99 98 "cell_type": "code",
100 99 "collapsed": false,
101 100 "input": [
102 101 "%px print >> sys.stderr, \"ERROR\""
103 102 ],
104 103 "language": "python",
105 104 "metadata": {},
106 105 "outputs": []
107 106 },
108 107 {
109 108 "cell_type": "markdown",
110 109 "metadata": {},
111 110 "source": [
112 111 "You don't have to wait for results. The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:"
113 112 ]
114 113 },
115 114 {
116 115 "cell_type": "code",
117 116 "collapsed": false,
118 117 "input": [
119 118 "%pxconfig --noblock"
120 119 ],
121 120 "language": "python",
122 121 "metadata": {},
123 122 "outputs": []
124 123 },
125 124 {
126 125 "cell_type": "code",
127 126 "collapsed": false,
128 127 "input": [
129 128 "%px import time\n",
130 129 "%px time.sleep(5)\n",
131 130 "%px time.time()"
132 131 ],
133 132 "language": "python",
134 133 "metadata": {},
135 134 "outputs": []
136 135 },
137 136 {
138 137 "cell_type": "markdown",
139 138 "metadata": {},
140 139 "source": [
141 140 "But you will notice that this didn't output the result of the last command.\n",
142 141 "For this, we have `%pxresult`, which displays the output of the latest request:"
143 142 ]
144 143 },
145 144 {
146 145 "cell_type": "code",
147 146 "collapsed": false,
148 147 "input": [
149 148 "%pxresult"
150 149 ],
151 150 "language": "python",
152 151 "metadata": {},
153 152 "outputs": []
154 153 },
155 154 {
156 155 "cell_type": "markdown",
157 156 "metadata": {},
158 157 "source": [
159 158 "Remember, an IPython engine is IPython, so you can do magics remotely as well!"
160 159 ]
161 160 },
162 161 {
163 162 "cell_type": "code",
164 163 "collapsed": false,
165 164 "input": [
166 165 "%pxconfig --block\n",
167 166 "%px %pylab inline"
168 167 ],
169 168 "language": "python",
170 169 "metadata": {},
171 170 "outputs": []
172 171 },
173 172 {
174 173 "cell_type": "markdown",
175 174 "metadata": {},
176 175 "source": [
177 176 "`%%px` can also be used as a cell magic, for submitting whole blocks.\n",
178 177 "This one acceps `--block` and `--noblock` flags to specify\n",
179 178 "the blocking behavior, though the default is unchanged.\n"
180 179 ]
181 180 },
182 181 {
183 182 "cell_type": "code",
184 183 "collapsed": false,
185 184 "input": [
186 185 "dv.scatter('id', dv.targets, flatten=True)\n",
187 186 "dv['stride'] = len(dv)"
188 187 ],
189 188 "language": "python",
190 189 "metadata": {},
191 190 "outputs": []
192 191 },
193 192 {
194 193 "cell_type": "code",
195 194 "collapsed": false,
196 195 "input": [
197 196 "%%px --noblock\n",
198 197 "x = linspace(0,pi,1000)\n",
199 198 "for n in range(id,12, stride):\n",
200 199 " print n\n",
201 200 " plt.plot(x,sin(n*x))\n",
202 201 "plt.title(\"Plot %i\" % id)"
203 202 ],
204 203 "language": "python",
205 204 "metadata": {},
206 205 "outputs": []
207 206 },
208 207 {
209 208 "cell_type": "code",
210 209 "collapsed": false,
211 210 "input": [
212 211 "%pxresult"
213 212 ],
214 213 "language": "python",
215 214 "metadata": {},
216 215 "outputs": []
217 216 },
218 217 {
219 218 "cell_type": "markdown",
220 219 "metadata": {},
221 220 "source": [
222 221 "It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:\n",
223 222 "\n",
224 223 "The choices are:\n",
225 224 "\n",
226 225 "* `engine` - all of an engine's output is collected together\n",
227 226 "* `type` - where stdout of each engine is grouped, etc. (the default)\n",
228 227 "* `order` - same as `type`, but individual displaypub outputs are interleaved.\n",
229 228 " That is, it will output the first plot from each engine, then the second from each,\n",
230 229 " etc."
231 230 ]
232 231 },
233 232 {
234 233 "cell_type": "code",
235 234 "collapsed": false,
236 235 "input": [
237 236 "%%px --group-outputs=engine\n",
238 237 "x = linspace(0,pi,1000)\n",
239 238 "for n in range(id+1,12, stride):\n",
240 239 " print n\n",
241 240 " plt.figure()\n",
242 241 " plt.plot(x,sin(n*x))\n",
243 242 " plt.title(\"Plot %i\" % n)"
244 243 ],
245 244 "language": "python",
246 245 "metadata": {},
247 246 "outputs": []
248 247 },
249 248 {
250 249 "cell_type": "markdown",
251 250 "metadata": {},
252 251 "source": [
253 252 "When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.\n",
254 253 "\n",
255 254 "`%pxresult` takes the same output-ordering arguments as `%%px`, \n",
256 255 "so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:"
257 256 ]
258 257 },
259 258 {
260 259 "cell_type": "code",
261 260 "collapsed": false,
262 261 "input": [
263 262 "%pxresult --group-outputs=order"
264 263 ],
265 264 "language": "python",
266 265 "metadata": {},
267 266 "outputs": []
268 267 },
269 268 {
270 269 "cell_type": "heading",
271 270 "level": 2,
272 271 "metadata": {},
273 272 "source": [
274 273 "Single-engine views"
275 274 ]
276 275 },
277 276 {
278 277 "cell_type": "markdown",
279 278 "metadata": {},
280 279 "source": [
281 280 "When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):"
282 281 ]
283 282 },
284 283 {
285 284 "cell_type": "code",
286 285 "collapsed": false,
287 286 "input": [
288 287 "def generate_output():\n",
289 288 " \"\"\"function for testing output\n",
290 289 " \n",
291 290 " publishes two outputs of each type, and returns something\n",
292 291 " \"\"\"\n",
293 292 " \n",
294 293 " import sys,os\n",
295 294 " from IPython.core.display import display, HTML, Math\n",
296 295 " \n",
297 296 " print \"stdout\"\n",
298 297 " print >> sys.stderr, \"stderr\"\n",
299 298 " \n",
300 299 " display(HTML(\"<b>HTML</b>\"))\n",
301 300 " \n",
302 301 " print \"stdout2\"\n",
303 302 " print >> sys.stderr, \"stderr2\"\n",
304 303 " \n",
305 304 " display(Math(r\"\\alpha=\\beta\"))\n",
306 305 " \n",
307 306 " return os.getpid()\n",
308 307 "\n",
309 308 "dv['generate_output'] = generate_output"
310 309 ],
311 310 "language": "python",
312 311 "metadata": {},
313 312 "outputs": []
314 313 },
315 314 {
316 315 "cell_type": "markdown",
317 316 "metadata": {},
318 317 "source": [
319 318 "You can also have more than one set of parallel magics registered at a time.\n",
320 319 "\n",
321 320 "The `View.activate()` method takes a suffix argument, which is added to `'px'`."
322 321 ]
323 322 },
324 323 {
325 324 "cell_type": "code",
326 325 "collapsed": false,
327 326 "input": [
328 327 "e0 = rc[-1]\n",
329 328 "e0.block = True\n",
330 329 "e0.activate('0')"
331 330 ],
332 331 "language": "python",
333 332 "metadata": {},
334 333 "outputs": []
335 334 },
336 335 {
337 336 "cell_type": "code",
338 337 "collapsed": false,
339 338 "input": [
340 339 "%px0 generate_output()"
341 340 ],
342 341 "language": "python",
343 342 "metadata": {},
344 343 "outputs": []
345 344 },
346 345 {
347 346 "cell_type": "code",
348 347 "collapsed": false,
349 348 "input": [
350 349 "%px generate_output()"
351 350 ],
352 351 "language": "python",
353 352 "metadata": {},
354 353 "outputs": []
355 354 },
356 355 {
357 356 "cell_type": "markdown",
358 357 "metadata": {},
359 358 "source": [
360 359 "As mentioned above, we can redisplay those same results with various grouping:"
361 360 ]
362 361 },
363 362 {
364 363 "cell_type": "code",
365 364 "collapsed": false,
366 365 "input": [
367 366 "%pxresult --group-outputs order"
368 367 ],
369 368 "language": "python",
370 369 "metadata": {},
371 370 "outputs": []
372 371 },
373 372 {
374 373 "cell_type": "code",
375 374 "collapsed": false,
376 375 "input": [
377 376 "%pxresult --group-outputs engine"
378 377 ],
379 378 "language": "python",
380 379 "metadata": {},
381 380 "outputs": []
382 381 },
383 382 {
384 383 "cell_type": "heading",
385 384 "level": 2,
386 385 "metadata": {},
387 386 "source": [
388 387 "Parallel Exceptions"
389 388 ]
390 389 },
391 390 {
392 391 "cell_type": "markdown",
393 392 "metadata": {},
394 393 "source": [
395 394 "When you raise exceptions with the parallel exception,\n",
396 395 "the CompositeError raised locally will display your remote traceback."
397 396 ]
398 397 },
399 398 {
400 399 "cell_type": "code",
401 400 "collapsed": false,
402 401 "input": [
403 402 "%%px\n",
404 403 "from numpy.random import random\n",
405 404 "A = random((100,100,'invalid shape'))"
406 405 ],
407 406 "language": "python",
408 407 "metadata": {},
409 408 "outputs": []
410 409 },
411 410 {
412 411 "cell_type": "heading",
413 412 "level": 2,
414 413 "metadata": {},
415 414 "source": [
416 415 "Remote Cell Magics"
417 416 ]
418 417 },
419 418 {
420 419 "cell_type": "markdown",
421 420 "metadata": {},
422 421 "source": [
423 422 "Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic."
424 423 ]
425 424 },
426 425 {
427 426 "cell_type": "code",
428 427 "collapsed": false,
429 428 "input": [
430 429 "%%px\n",
431 430 "%%timeit\n",
432 431 "from numpy.random import random\n",
433 432 "from numpy.linalg import norm\n",
434 433 "A = random((100,100))\n",
435 434 "norm(A, 2) "
436 435 ],
437 436 "language": "python",
438 437 "metadata": {},
439 438 "outputs": []
440 439 },
441 440 {
442 441 "cell_type": "heading",
443 442 "level": 2,
444 443 "metadata": {},
445 444 "source": [
446 445 "Local Execution"
447 446 ]
448 447 },
449 448 {
450 449 "cell_type": "markdown",
451 450 "metadata": {},
452 451 "source": [
453 452 "As of IPython 0.14, you can instruct `%%px` to also execute the cell locally.\n",
454 453 "This is useful for interactive definitions,\n",
455 454 "or if you want to load a data source everywhere,\n",
456 455 "not just on the engines."
457 456 ]
458 457 },
459 458 {
460 459 "cell_type": "code",
461 460 "collapsed": false,
462 461 "input": [
463 462 "%%px --local\n",
463 "import os\n",
464 464 "thispid = os.getpid()\n",
465 465 "print thispid"
466 466 ],
467 467 "language": "python",
468 468 "metadata": {},
469 "outputs": [],
470 "prompt_number": 5
469 "outputs": []
471 470 }
472 471 ],
473 472 "metadata": {}
474 473 }
475 474 ]
476 475 } No newline at end of file
General Comments 0
You need to be logged in to leave comments. Login now