Parallel Magics.ipynb
521 lines
| 10.7 KiB
| text/plain
|
TextLexer
MinRK
|
r7053 | { | ||
Min RK
|
r18669 | "cells": [ | ||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"# Using Parallel Magics" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"IPython has a few magics for working with your engines.\n", | ||||
"\n", | ||||
"This assumes you have started an IPython cluster, either with the notebook interface,\n", | ||||
"or the `ipcluster/controller/engine` commands." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"from IPython import parallel\n", | ||||
"rc = parallel.Client()\n", | ||||
"dv = rc[:]\n", | ||||
"rc.ids" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`. \n", | ||||
"These magics are initially associated with a DirectView always associated with all currently registered engines." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"Now we can execute code remotely with `%px`:" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px a=5" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px print(a)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px a" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"with dv.sync_imports():\n", | ||||
" import sys" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px from __future__ import print_function\n", | ||||
"%px print(\"ERROR\", file=sys.stderr)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"You don't have to wait for results. The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxconfig --noblock" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px import time\n", | ||||
"%px time.sleep(5)\n", | ||||
"%px time.time()" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"But you will notice that this didn't output the result of the last command.\n", | ||||
"For this, we have `%pxresult`, which displays the output of the latest request:" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxresult" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"Remember, an IPython engine is IPython, so you can do magics remotely as well!" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxconfig --block\n", | ||||
"%px %matplotlib inline" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px\n", | ||||
"import numpy as np\n", | ||||
"import matplotlib.pyplot as plt" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"`%%px` can also be used as a cell magic, for submitting whole blocks.\n", | ||||
"This one acceps `--block` and `--noblock` flags to specify\n", | ||||
"the blocking behavior, though the default is unchanged.\n" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"dv.scatter('id', dv.targets, flatten=True)\n", | ||||
"dv['stride'] = len(dv)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px --noblock\n", | ||||
"x = np.linspace(0,np.pi,1000)\n", | ||||
"for n in range(id,12, stride):\n", | ||||
" print(n)\n", | ||||
" plt.plot(x,np.sin(n*x))\n", | ||||
"plt.title(\"Plot %i\" % id)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxresult" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:\n", | ||||
"\n", | ||||
"The choices are:\n", | ||||
"\n", | ||||
"* `engine` - all of an engine's output is collected together\n", | ||||
"* `type` - where stdout of each engine is grouped, etc. (the default)\n", | ||||
"* `order` - same as `type`, but individual displaypub outputs are interleaved.\n", | ||||
" That is, it will output the first plot from each engine, then the second from each,\n", | ||||
" etc." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px --group-outputs=engine\n", | ||||
"x = np.linspace(0,np.pi,1000)\n", | ||||
"for n in range(id+1,12, stride):\n", | ||||
" print(n)\n", | ||||
" plt.figure()\n", | ||||
" plt.plot(x,np.sin(n*x))\n", | ||||
" plt.title(\"Plot %i\" % n)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.\n", | ||||
"\n", | ||||
"`%pxresult` takes the same output-ordering arguments as `%%px`, \n", | ||||
"so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxresult --group-outputs=order" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"## Single-engine views" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"from __future__ import print_function\n", | ||||
"\n", | ||||
"def generate_output():\n", | ||||
" \"\"\"function for testing output\n", | ||||
" \n", | ||||
" publishes two outputs of each type, and returns something\n", | ||||
" \"\"\"\n", | ||||
" \n", | ||||
" import sys,os\n", | ||||
" from IPython.display import display, HTML, Math\n", | ||||
" \n", | ||||
" print(\"stdout\")\n", | ||||
" print(\"stderr\", file=sys.stderr)\n", | ||||
" \n", | ||||
" display(HTML(\"<b>HTML</b>\"))\n", | ||||
" \n", | ||||
" print(\"stdout2\")\n", | ||||
" print(\"stderr2\", file=sys.stderr)\n", | ||||
" \n", | ||||
" display(Math(r\"\\alpha=\\beta\"))\n", | ||||
" \n", | ||||
" return os.getpid()\n", | ||||
"\n", | ||||
"dv['generate_output'] = generate_output" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"You can also have more than one set of parallel magics registered at a time.\n", | ||||
"\n", | ||||
"The `View.activate()` method takes a suffix argument, which is added to `'px'`." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"e0 = rc[-1]\n", | ||||
"e0.block = True\n", | ||||
"e0.activate('0')" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px0 generate_output()" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%px generate_output()" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"As mentioned above, we can redisplay those same results with various grouping:" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxresult --group-outputs order" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%pxresult --group-outputs engine" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"## Parallel Exceptions" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"When you raise exceptions with the parallel exception,\n", | ||||
"the CompositeError raised locally will display your remote traceback." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px\n", | ||||
"from numpy.random import random\n", | ||||
"A = random((100,100,'invalid shape'))" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"## Remote Cell Magics" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px\n", | ||||
"%%timeit\n", | ||||
"from numpy.random import random\n", | ||||
"from numpy.linalg import norm\n", | ||||
"A = random((100,100))\n", | ||||
"norm(A, 2)" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"## Local Execution" | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "markdown", | ||||
"metadata": {}, | ||||
"source": [ | ||||
"As of IPython 1.0, you can instruct `%%px` to also execute the cell locally.\n", | ||||
"This is useful for interactive definitions,\n", | ||||
"or if you want to load a data source everywhere,\n", | ||||
"not just on the engines." | ||||
] | ||||
}, | ||||
{ | ||||
"cell_type": "code", | ||||
"execution_count": null, | ||||
"metadata": { | ||||
"collapsed": false | ||||
}, | ||||
"outputs": [], | ||||
"source": [ | ||||
"%%px --local\n", | ||||
"import os\n", | ||||
"thispid = os.getpid()\n", | ||||
"print(thispid)" | ||||
] | ||||
MinRK
|
r7053 | } | ||
Min RK
|
r18669 | ], | ||
Min RK
|
r20278 | "metadata": { | ||
"kernelspec": { | ||||
"display_name": "Python 3", | ||||
"language": "python", | ||||
"name": "python3" | ||||
}, | ||||
"language_info": { | ||||
"codemirror_mode": { | ||||
"name": "ipython", | ||||
"version": 3 | ||||
}, | ||||
"file_extension": ".py", | ||||
"mimetype": "text/x-python", | ||||
"name": "python", | ||||
"nbconvert_exporter": "python", | ||||
"pygments_lexer": "ipython3", | ||||
"version": "3.4.2" | ||||
} | ||||
}, | ||||
Min RK
|
r18669 | "nbformat": 4, | ||
"nbformat_minor": 0 | ||||
Min RK
|
r20278 | } | ||