Using Parallel Magics¶
IPython has a few magics for working with your engines.
This assumes you have started an IPython cluster, either with the notebook interface,
or the ipcluster/controller/engine commands.
from IPython import parallel
rc = parallel.Client()
dv = rc[:]
dv.block = True
dv
The parallel magics come from the parallelmagics IPython extension.
The magics are set to work with a particular View object,
so to activate them, you call the activate() method on a particular view:
dv.activate()
Now we can execute code remotely with %px:
%px a=5
%px print a
%px a
with dv.sync_imports():
import sys
%px print >> sys.stderr, "ERROR"
You don't have to wait for results:
dv.block = False
%px import time
%px time.sleep(5)
%px time.time()
But you will notice that this didn't output the result of the last command.
For this, we have %result, which displays the output of the latest request:
%result
Remember, an IPython engine is IPython, so you can do magics remotely as well!
dv.block = True
%px %pylab inline
%%px can also be used as a cell magic, for submitting whole blocks.
This one acceps --block and --noblock flags to specify
the blocking behavior, though the default is unchanged.
dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)
%%px --noblock
x = linspace(0,pi,1000)
for n in range(id,12, stride):
print n
plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)
%result
It also lets you choose some amount of the grouping of the outputs with --group-outputs:
The choices are:
engine- all of an engine's output is collected togethertype- where stdout of each engine is grouped, etc. (the default)order- same astype, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.
%%px --group-outputs=engine
x = linspace(0,pi,1000)
for n in range(id,12, stride):
print n
plt.figure()
plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)
When you specify 'order', then individual display outputs (e.g. plots) will be interleaved:
%%px --group-outputs=order
x = linspace(0,pi,1000)
for n in range(id,12, stride):
print n
plt.figure()
plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)
Single-engine views¶
When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):
def generate_output():
"""function for testing output
publishes two outputs of each type, and returns something
"""
import sys,os
from IPython.core.display import display, HTML, Math
print "stdout"
print >> sys.stderr, "stderr"
display(HTML("<b>HTML</b>"))
print "stdout2"
print >> sys.stderr, "stderr2"
display(Math(r"\alpha=\beta"))
return os.getpid()
dv['generate_output'] = generate_output
e0 = rc[-1]
e0.block = True
e0.activate()
%px generate_output()