##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

update parallel magics...
update parallel magics * no longer in extension * can have multiple active sets of magics with optional suffix * add pxconfig * %result only fetches last result, but accepts display-order args * use magic_arguments decorators
MinRK - - Load All Authors

File last commit:

r7250:f2fab60d
r7476:ad8073a2
Show More
Parallel Magics.ipynb
322 lines | 7.1 KiB | text/plain | TextLexer
/ docs / examples / parallel / Parallel Magics.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Using Parallel Magics

IPython has a few magics for working with your engines.

This assumes you have started an IPython cluster, either with the notebook interface, or the ipcluster/controller/engine commands.

In [ ]:
from IPython import parallel
rc = parallel.Client()
dv = rc[:]
dv.block = True
dv

The parallel magics come from the parallelmagics IPython extension. The magics are set to work with a particular View object, so to activate them, you call the activate() method on a particular view:

In [ ]:
dv.activate()

Now we can execute code remotely with %px:

In [ ]:
%px a=5
In [ ]:
%px print a
In [ ]:
%px a
In [ ]:
with dv.sync_imports():
    import sys
In [ ]:
%px print >> sys.stderr, "ERROR"

You don't have to wait for results:

In [ ]:
dv.block = False
In [ ]:
%px import time
%px time.sleep(5)
%px time.time()

But you will notice that this didn't output the result of the last command. For this, we have %result, which displays the output of the latest request:

In [ ]:
%result

Remember, an IPython engine is IPython, so you can do magics remotely as well!

In [ ]:
dv.block = True
%px %pylab inline

%%px can also be used as a cell magic, for submitting whole blocks. This one acceps --block and --noblock flags to specify the blocking behavior, though the default is unchanged.

In [ ]:
dv.scatter('id', dv.targets, flatten=True)
dv['stride'] = len(dv)
In [ ]:
%%px --noblock
x = linspace(0,pi,1000)
for n in range(id,12, stride):
    print n
    plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)
In [ ]:
%result

It also lets you choose some amount of the grouping of the outputs with --group-outputs:

The choices are:

  • engine - all of an engine's output is collected together
  • type - where stdout of each engine is grouped, etc. (the default)
  • order - same as type, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.
In [ ]:
%%px --group-outputs=engine
x = linspace(0,pi,1000)
for n in range(id,12, stride):
    print n
    plt.figure()
    plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)

When you specify 'order', then individual display outputs (e.g. plots) will be interleaved:

In [ ]:
%%px --group-outputs=order
x = linspace(0,pi,1000)
for n in range(id,12, stride):
    print n
    plt.figure()
    plt.plot(x,sin(n*x))
plt.title("Plot %i" % id)

Single-engine views

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

In [ ]:
def generate_output():
    """function for testing output
    
    publishes two outputs of each type, and returns something
    """
    
    import sys,os
    from IPython.core.display import display, HTML, Math
    
    print "stdout"
    print >> sys.stderr, "stderr"
    
    display(HTML("<b>HTML</b>"))
    
    print "stdout2"
    print >> sys.stderr, "stderr2"
    
    display(Math(r"\alpha=\beta"))
    
    return os.getpid()

dv['generate_output'] = generate_output
In [ ]:
e0 = rc[-1]
e0.block = True
e0.activate()
In [ ]:
%px generate_output()