parallel_multiengine.txt
942 lines
| 33.0 KiB
| text/plain
|
TextLexer
|
r3664 | .. _parallel_multiengine: | ||
|
r3586 | |||
|
r3618 | ========================== | ||
IPython's Direct interface | ||||
========================== | ||||
|
r3586 | |||
|
r3618 | The direct, or multiengine, interface represents one possible way of working with a set of | ||
|
r3586 | IPython engines. The basic idea behind the multiengine interface is that the | ||
capabilities of each engine are directly and explicitly exposed to the user. | ||||
Thus, in the multiengine interface, each engine is given an id that is used to | ||||
identify the engine and give it work to do. This interface is very intuitive | ||||
|
r3664 | and is designed with interactive usage in mind, and is the best place for | ||
|
r3586 | new users of IPython to begin. | ||
Starting the IPython controller and engines | ||||
=========================================== | ||||
To follow along with this tutorial, you will need to start the IPython | ||||
controller and four IPython engines. The simplest way of doing this is to use | ||||
|
r3672 | the :command:`ipcluster` command:: | ||
|
r3586 | |||
|
r4608 | $ ipcluster start -n 4 | ||
|
r3591 | |||
|
r3586 | For more detailed information about starting the controller and engines, see | ||
|
r4435 | our :ref:`introduction <parallel_overview>` to using IPython for parallel computing. | ||
|
r3586 | |||
|
r5169 | Creating a ``DirectView`` instance | ||
================================== | ||||
|
r3586 | |||
|
r3666 | The first step is to import the IPython :mod:`IPython.parallel` | ||
|
r3591 | module and then create a :class:`.Client` instance: | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3666 | In [1]: from IPython.parallel import Client | ||
|
r3591 | |||
|
r3666 | In [2]: rc = Client() | ||
|
r3586 | |||
|
r3618 | This form assumes that the default connection information (stored in | ||
|
r6696 | :file:`ipcontroller-client.json` found in :file:`IPYTHONDIR/profile_default/security`) is | ||
|
r3618 | accurate. If the controller was started on a remote machine, you must copy that connection | ||
file to the client machine, or enter its contents as arguments to the Client constructor: | ||||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3618 | # If you have copied the json connector file from the controller: | ||
|
r3666 | In [2]: rc = Client('/path/to/ipcontroller-client.json') | ||
|
r3663 | # or to connect with a specific profile you have set up: | ||
|
r3666 | In [3]: rc = Client(profile='mpi') | ||
|
r3591 | |||
|
r3586 | |||
|
r3635 | To make sure there are engines connected to the controller, users can get a list | ||
|
r3586 | of engine ids: | ||
.. sourcecode:: ipython | ||||
|
r3591 | In [3]: rc.ids | ||
|
r3635 | Out[3]: [0, 1, 2, 3] | ||
|
r3586 | |||
Here we see that there are four engines ready to do work for us. | ||||
|
r3639 | For direct execution, we will make use of a :class:`DirectView` object, which can be | ||
constructed via list-access to the client: | ||||
|
r3663 | .. sourcecode:: ipython | ||
|
r3639 | |||
In [4]: dview = rc[:] # use all engines | ||||
.. seealso:: | ||||
|
r3655 | For more information, see the in-depth explanation of :ref:`Views <parallel_details>`. | ||
|
r3639 | |||
|
r3586 | Quick and easy parallelism | ||
========================== | ||||
In many cases, you simply want to apply a Python function to a sequence of | ||||
|
r3591 | objects, but *in parallel*. The client interface provides a simple way | ||
|
r3639 | of accomplishing this: using the DirectView's :meth:`~DirectView.map` method. | ||
|
r3586 | |||
Parallel map | ||||
------------ | ||||
Python's builtin :func:`map` functions allows a function to be applied to a | ||||
sequence element-by-element. This type of code is typically trivial to | ||||
|
r3594 | parallelize. In fact, since IPython's interface is all about functions anyway, | ||
|
r3635 | you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a | ||
DirectView's :meth:`map` method: | ||||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3591 | In [62]: serial_result = map(lambda x:x**10, range(32)) | ||
|
r3639 | |||
|
r3664 | In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32)) | ||
|
r3586 | |||
|
r3639 | In [67]: serial_result==parallel_result | ||
|
r3591 | Out[67]: True | ||
|
r3586 | |||
.. note:: | ||||
|
r3635 | The :class:`DirectView`'s version of :meth:`map` does | ||
|
r3639 | not do dynamic load balancing. For a load balanced version, use a | ||
|
r3664 | :class:`LoadBalancedView`. | ||
|
r3586 | |||
.. seealso:: | ||||
|
r3591 | |||
|
r3639 | :meth:`map` is implemented via :class:`ParallelFunction`. | ||
|
r3586 | |||
|
r3639 | Remote function decorators | ||
-------------------------- | ||||
|
r3586 | |||
|
r3591 | Remote functions are just like normal functions, but when they are called, | ||
they execute on one or more engines, rather than locally. IPython provides | ||||
|
r3639 | two decorators: | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3664 | In [10]: @dview.remote(block=True) | ||
|
r5169 | ....: def getpid(): | ||
....: import os | ||||
....: return os.getpid() | ||||
....: | ||||
|
r3639 | |||
In [11]: getpid() | ||||
Out[11]: [12345, 12346, 12347, 12348] | ||||
|
r3586 | |||
|
r3664 | The ``@parallel`` decorator creates parallel functions, that break up an element-wise | ||
|
r3639 | operations and distribute them, reconstructing the result. | ||
.. sourcecode:: ipython | ||||
In [12]: import numpy as np | ||||
In [13]: A = np.random.random((64,48)) | ||||
|
r3664 | In [14]: @dview.parallel(block=True) | ||
|
r5169 | ....: def pmul(A,B): | ||
....: return A*B | ||||
|
r3639 | |||
In [15]: C_local = A*A | ||||
|
r3664 | In [16]: C_remote = pmul(A,A) | ||
|
r3639 | |||
In [17]: (C_local == C_remote).all() | ||||
Out[17]: True | ||||
|
r3586 | |||
|
r5634 | Calling a ``@parallel`` function *does not* correspond to map. It is used for splitting | ||
element-wise operations that operate on a sequence or array. For ``map`` behavior, | ||||
parallel functions do have a map method. | ||||
==================== ============================ ============================= | ||||
call pfunc(seq) pfunc.map(seq) | ||||
==================== ============================ ============================= | ||||
# of tasks # of engines (1 per engine) # of engines (1 per engine) | ||||
# of remote calls # of engines (1 per engine) ``len(seq)`` | ||||
argument to remote ``seq[i:j]`` (sub-sequence) ``seq[i]`` (single element) | ||||
==================== ============================ ============================= | ||||
A quick example to illustrate the difference in arguments for the two modes: | ||||
.. sourcecode:: ipython | ||||
In [16]: @dview.parallel(block=True) | ||||
....: def echo(x): | ||||
....: return str(x) | ||||
....: | ||||
In [17]: echo(range(5)) | ||||
Out[17]: ['[0, 1]', '[2]', '[3]', '[4]'] | ||||
In [18]: echo.map(range(5)) | ||||
Out[18]: ['0', '1', '2', '3', '4'] | ||||
|
r3635 | .. seealso:: | ||
|
r5634 | See the :func:`~.remotefunction.parallel` and :func:`~.remotefunction.remote` | ||
decorators for options. | ||||
|
r3586 | |||
|
r3591 | Calling Python functions | ||
======================== | ||||
|
r3586 | |||
The most basic type of operation that can be performed on the engines is to | ||||
|
r3591 | execute Python code or call Python functions. Executing Python code can be | ||
done in blocking or non-blocking mode (non-blocking is default) using the | ||||
|
r3664 | :meth:`.View.execute` method, and calling functions can be done via the | ||
|
r3591 | :meth:`.View.apply` method. | ||
|
r3586 | |||
|
r3594 | apply | ||
----- | ||||
The main method for doing remote execution (in fact, all methods that | ||||
|
r3664 | communicate with the engines are built on top of it), is :meth:`View.apply`. | ||
|
r3594 | |||
|
r3664 | We strive to provide the cleanest interface we can, so `apply` has the following | ||
signature: | ||||
|
r3594 | |||
|
r3664 | .. sourcecode:: python | ||
|
r3642 | |||
|
r3664 | view.apply(f, *args, **kwargs) | ||
|
r3642 | |||
|
r3664 | There are various ways to call functions with IPython, and these flags are set as | ||
attributes of the View. The ``DirectView`` has just two of these flags: | ||||
|
r3642 | |||
|
r3664 | dv.block : bool | ||
whether to wait for the result, or return an :class:`AsyncResult` object | ||||
immediately | ||||
dv.track : bool | ||||
|
r5167 | whether to instruct pyzmq to track when zeromq is done sending the message. | ||
|
r3664 | This is primarily useful for non-copying sends of numpy arrays that you plan to | ||
edit in-place. You need to know when it becomes safe to edit the buffer | ||||
without corrupting the message. | ||||
|
r5169 | dv.targets : int, list of ints | ||
which targets this view is associated with. | ||||
|
r3642 | |||
|
r3664 | Creating a view is simple: index-access on a client creates a :class:`.DirectView`. | ||
|
r3594 | |||
.. sourcecode:: ipython | ||||
In [4]: view = rc[1:3] | ||||
Out[4]: <DirectView [1, 2]> | ||||
In [5]: view.apply<tab> | ||||
|
r3670 | view.apply view.apply_async view.apply_sync | ||
|
r3594 | |||
|
r3664 | For convenience, you can set block temporarily for a single call with the extra sync/async methods. | ||
|
r3594 | |||
|
r3586 | Blocking execution | ||
------------------ | ||||
|
r3591 | In blocking mode, the :class:`.DirectView` object (called ``dview`` in | ||
|
r3586 | these examples) submits the command to the controller, which places the | ||
|
r3591 | command in the engines' queues for execution. The :meth:`apply` call then | ||
|
r3586 | blocks until the engines are done executing the command: | ||
.. sourcecode:: ipython | ||||
|
r3639 | In [2]: dview = rc[:] # A DirectView of all engines | ||
In [3]: dview.block=True | ||||
|
r3591 | In [4]: dview['a'] = 5 | ||
|
r3586 | |||
|
r3591 | In [5]: dview['b'] = 10 | ||
|
r3586 | |||
|
r3664 | In [6]: dview.apply(lambda x: a+b+x, 27) | ||
|
r3655 | Out[6]: [42, 42, 42, 42] | ||
|
r3586 | |||
|
r3664 | You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync` | ||
method: | ||||
|
r3586 | |||
|
r3664 | In [7]: dview.block=False | ||
|
r3586 | |||
|
r3664 | In [8]: dview.apply_sync(lambda x: a+b+x, 27) | ||
Out[8]: [42, 42, 42, 42] | ||||
|
r3639 | |||
|
r3664 | Python commands can be executed as strings on specific engines by using a View's ``execute`` | ||
method: | ||||
|
r3655 | |||
|
r3664 | .. sourcecode:: ipython | ||
|
r3655 | |||
|
r3664 | In [6]: rc[::2].execute('c=a+b') | ||
|
r3586 | |||
|
r3664 | In [7]: rc[1::2].execute('c=a-b') | ||
|
r3586 | |||
|
r3670 | In [8]: dview['c'] # shorthand for dview.pull('c', block=True) | ||
|
r3664 | Out[8]: [15, -5, 15, -5] | ||
|
r3586 | |||
Non-blocking execution | ||||
---------------------- | ||||
|
r3591 | In non-blocking mode, :meth:`apply` submits the command to be executed and | ||
then returns a :class:`AsyncResult` object immediately. The | ||||
:class:`AsyncResult` object gives you a way of getting a result at a later | ||||
|
r3639 | time through its :meth:`get` method. | ||
|
r3586 | |||
|
r6472 | .. seealso:: | ||
|
r3586 | |||
|
r6472 | Docs on the :ref:`AsyncResult <parallel_asyncresult>` object. | ||
|
r3591 | |||
This allows you to quickly submit long running commands without blocking your | ||||
local Python/IPython session: | ||||
.. sourcecode:: ipython | ||||
# define our function | ||||
|
r3624 | In [6]: def wait(t): | ||
|
r5169 | ....: import time | ||
....: tic = time.time() | ||||
....: time.sleep(t) | ||||
....: return time.time()-tic | ||||
|
r3591 | |||
# In non-blocking mode | ||||
|
r3639 | In [7]: ar = dview.apply_async(wait, 2) | ||
|
r3591 | |||
# Now block for the result | ||||
|
r3639 | In [8]: ar.get() | ||
|
r3591 | Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154] | ||
# Again in non-blocking mode | ||||
|
r3639 | In [9]: ar = dview.apply_async(wait, 10) | ||
|
r3591 | |||
# Poll to see if the result is ready | ||||
|
r3639 | In [10]: ar.ready() | ||
|
r3591 | Out[10]: False | ||
# ask for the result, but wait a maximum of 1 second: | ||||
|
r3639 | In [45]: ar.get(1) | ||
|
r3591 | --------------------------------------------------------------------------- | ||
TimeoutError Traceback (most recent call last) | ||||
/home/you/<ipython-input-45-7cd858bbb8e0> in <module>() | ||||
|
r3639 | ----> 1 ar.get(1) | ||
|
r3591 | |||
|
r3666 | /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout) | ||
|
r3591 | 62 raise self._exception | ||
63 else: | ||||
---> 64 raise error.TimeoutError("Result not ready.") | ||||
65 | ||||
66 def ready(self): | ||||
TimeoutError: Result not ready. | ||||
.. Note:: | ||||
Note the import inside the function. This is a common model, to ensure | ||||
|
r3664 | that the appropriate modules are imported where the task is run. You can | ||
also manually import modules into the engine(s) namespace(s) via | ||||
:meth:`view.execute('import numpy')`. | ||||
|
r3591 | |||
Often, it is desirable to wait until a set of :class:`AsyncResult` objects | ||||
|
r3664 | are done. For this, there is a the method :meth:`wait`. This method takes a | ||
|
r3639 | tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History), | ||
and blocks until all of the associated results are ready: | ||||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3664 | In [72]: dview.block=False | ||
|
r3586 | |||
|
r3591 | # A trivial list of AsyncResults objects | ||
|
r3635 | In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)] | ||
|
r3586 | |||
|
r3591 | # Wait until all of them are done | ||
|
r3664 | In [74]: dview.wait(pr_list) | ||
|
r3586 | |||
|
r3624 | # Then, their results are ready using get() or the `.r` attribute | ||
|
r3591 | In [75]: pr_list[0].get() | ||
Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752] | ||||
|
r3586 | |||
|
r3591 | |||
|
r3666 | The ``block`` and ``targets`` keyword arguments and attributes | ||
-------------------------------------------------------------- | ||||
|
r5169 | Most DirectView methods (excluding :meth:`apply`) accept ``block`` and | ||
|
r3666 | ``targets`` as keyword arguments. As we have seen above, these keyword arguments control the | ||
blocking mode and which engines the command is applied to. The :class:`View` class also has | ||||
:attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword | ||||
arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`: | ||||
|
r3586 | |||
* If no keyword argument is provided, the instance attributes are used. | ||||
|
r3635 | * Keyword argument, if provided override the instance attributes for | ||
the duration of a single call. | ||||
|
r3639 | |||
|
r3586 | The following examples demonstrate how to use the instance attributes: | ||
.. sourcecode:: ipython | ||||
|
r3666 | In [16]: dview.targets = [0,2] | ||
|
r3664 | In [17]: dview.block = False | ||
|
r3586 | |||
|
r3664 | In [18]: ar = dview.apply(lambda : 10) | ||
|
r3586 | |||
|
r3635 | In [19]: ar.get() | ||
|
r3666 | Out[19]: [10, 10] | ||
|
r3586 | |||
|
r3666 | In [16]: dview.targets = v.client.ids # all engines (4) | ||
|
r3664 | In [21]: dview.block = True | ||
|
r3586 | |||
|
r3664 | In [22]: dview.apply(lambda : 42) | ||
|
r3635 | Out[22]: [42, 42, 42, 42] | ||
|
r3586 | |||
|
r3664 | The :attr:`block` and :attr:`targets` instance attributes of the | ||
|
r3635 | :class:`.DirectView` also determine the behavior of the parallel magic commands. | ||
|
r3586 | |||
Parallel magic commands | ||||
----------------------- | ||||
We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``) | ||||
|
r7057 | that make it a bit more pleasant to execute Python commands on the engines interactively. | ||
These are simply shortcuts to :meth:`.DirectView.execute` | ||||
and :meth:`.AsyncResult.display_outputs` methods repsectively. | ||||
The ``%px`` magic executes a single Python command on the engines | ||||
specified by the :attr:`targets` attribute of the :class:`DirectView` instance: | ||||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3591 | # Create a DirectView for all targets | ||
In [22]: dv = rc[:] | ||||
# Make this DirectView active for parallel magic commands | ||||
In [23]: dv.activate() | ||||
|
r3586 | |||
|
r3591 | In [24]: dv.block=True | ||
|
r3586 | |||
|
r5169 | # import numpy here and everywhere | ||
In [25]: with dv.sync_imports(): | ||||
....: import numpy | ||||
importing numpy on engine(s) | ||||
|
r3586 | |||
|
r3591 | In [27]: %px a = numpy.random.rand(2,2) | ||
Parallel execution on engines: [0, 1, 2, 3] | ||||
|
r3586 | |||
|
r7057 | In [28]: %px numpy.linalg.eigvals(a) | ||
|
r3591 | Parallel execution on engines: [0, 1, 2, 3] | ||
|
r7059 | [0] Out[68]: array([ 0.77120707, -0.19448286]) | ||
[1] Out[68]: array([ 1.10815921, 0.05110369]) | ||||
[2] Out[68]: array([ 0.74625527, -0.37475081]) | ||||
[3] Out[68]: array([ 0.72931905, 0.07159743]) | ||||
|
r7057 | |||
In [29]: %px print 'hi' | ||||
Parallel execution on engine(s): [0, 1, 2, 3] | ||||
|
r7059 | [stdout:0] hi | ||
[stdout:1] hi | ||||
[stdout:2] hi | ||||
[stdout:3] hi | ||||
|
r7057 | |||
Since engines are IPython as well, you can even run magics remotely: | ||||
.. sourcecode:: ipython | ||||
In [28]: %px %pylab inline | ||||
Parallel execution on engine(s): [0, 1, 2, 3] | ||||
|
r7059 | [stdout:0] | ||
|
r7057 | Welcome to pylab, a matplotlib-based Python environment... | ||
For more information, type 'help(pylab)'. | ||||
|
r7059 | [stdout:1] | ||
|
r7057 | Welcome to pylab, a matplotlib-based Python environment... | ||
For more information, type 'help(pylab)'. | ||||
|
r7059 | [stdout:2] | ||
|
r7057 | Welcome to pylab, a matplotlib-based Python environment... | ||
For more information, type 'help(pylab)'. | ||||
|
r7059 | [stdout:3] | ||
|
r7057 | Welcome to pylab, a matplotlib-based Python environment... | ||
For more information, type 'help(pylab)'. | ||||
And once in pylab mode with the inline backend, | ||||
you can make plots and they will be displayed in your frontend | ||||
if it suports the inline figures (e.g. notebook or qtconsole): | ||||
.. sourcecode:: ipython | ||||
In [40]: %px plot(rand(100)) | ||||
Parallel execution on engine(s): [0, 1, 2, 3] | ||||
<plot0> | ||||
<plot1> | ||||
<plot2> | ||||
<plot3> | ||||
|
r7059 | [0] Out[79]: [<matplotlib.lines.Line2D at 0x10a6286d0>] | ||
[1] Out[79]: [<matplotlib.lines.Line2D at 0x10b9476d0>] | ||||
[2] Out[79]: [<matplotlib.lines.Line2D at 0x110652750>] | ||||
[3] Out[79]: [<matplotlib.lines.Line2D at 0x10c6566d0>] | ||||
|
r3586 | |||
|
r7057 | ``%%px`` Cell Magic | ||
******************* | ||||
`%%px` can also be used as a Cell Magic, which accepts ``--[no]block`` flags, | ||||
and a ``--group-outputs`` argument, which adjust how the outputs of multiple | ||||
engines are presented. | ||||
.. seealso:: | ||||
:meth:`.AsyncResult.display_outputs` for the grouping options. | ||||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r7057 | |||
In [50]: %%px --block --group-outputs=engine | ||||
....: import numpy as np | ||||
....: A = np.random.random((2,2)) | ||||
....: ev = numpy.linalg.eigvals(A) | ||||
....: print ev | ||||
....: ev.max() | ||||
....: | ||||
Parallel execution on engine(s): [0, 1, 2, 3] | ||||
|
r7059 | [stdout:0] [ 0.60640442 0.95919621] | ||
[0] Out[73]: 0.9591962130899806 | ||||
[stdout:1] [ 0.38501813 1.29430871] | ||||
[1] Out[73]: 1.2943087091452372 | ||||
[stdout:2] [-0.85925141 0.9387692 ] | ||||
[2] Out[73]: 0.93876920456230284 | ||||
[stdout:3] [ 0.37998269 1.24218246] | ||||
[3] Out[73]: 1.2421824618493817 | ||||
|
r7057 | |||
``%result`` Magic | ||||
***************** | ||||
If you are using ``%px`` in non-blocking mode, you won't get output. | ||||
You can use ``%result`` to display the outputs of the latest command, | ||||
just as is done when ``%px`` is blocking: | ||||
.. sourcecode:: ipython | ||||
In [39]: dv.block = False | ||||
|
r3639 | |||
|
r7057 | In [40]: %px print 'hi' | ||
Async parallel execution on engine(s): [0, 1, 2, 3] | ||||
|
r3639 | |||
|
r7057 | In [41]: %result | ||
|
r7059 | [stdout:0] hi | ||
[stdout:1] hi | ||||
[stdout:2] hi | ||||
[stdout:3] hi | ||||
|
r7057 | |||
``%result`` simply calls :meth:`.AsyncResult.display_outputs` on the most recent request. | ||||
You can pass integers as indices if you want a result other than the latest, | ||||
e.g. ``%result -2``, or ``%result 0`` for the first. | ||||
``%autopx`` | ||||
*********** | ||||
|
r3586 | |||
The ``%autopx`` magic switches to a mode where everything you type is executed | ||||
|
r7057 | on the engines until you do ``%autopx`` again. | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r7057 | In [30]: dv.block=True | ||
|
r3586 | |||
|
r3591 | In [31]: %autopx | ||
|
r7057 | %autopx enabled | ||
|
r3586 | |||
|
r3591 | In [32]: max_evals = [] | ||
|
r3586 | |||
|
r3591 | In [33]: for i in range(100): | ||
....: a = numpy.random.rand(10,10) | ||||
....: a = a+a.transpose() | ||||
....: evals = numpy.linalg.eigvals(a) | ||||
....: max_evals.append(evals[0].real) | ||||
....: | ||||
|
r3586 | |||
|
r7057 | In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals)) | ||
|
r7059 | [stdout:0] Average max eigenvalue is: 10.193101 | ||
[stdout:1] Average max eigenvalue is: 10.064508 | ||||
[stdout:2] Average max eigenvalue is: 10.055724 | ||||
[stdout:3] Average max eigenvalue is: 10.086876 | ||||
|
r3591 | |||
|
r7057 | In [35]: %autopx | ||
Auto Parallel Disabled | ||||
|
r3586 | |||
Moving Python objects around | ||||
============================ | ||||
|
r3591 | In addition to calling functions and executing code on engines, you can | ||
transfer Python objects to and from your IPython session and the engines. In | ||||
IPython, these operations are called :meth:`push` (sending an object to the | ||||
|
r3664 | engines) and :meth:`pull` (getting an object from the engines). | ||
|
r3586 | |||
Basic push and pull | ||||
------------------- | ||||
Here are some examples of how you use :meth:`push` and :meth:`pull`: | ||||
.. sourcecode:: ipython | ||||
|
r3664 | In [38]: dview.push(dict(a=1.03234,b=3453)) | ||
|
r3597 | Out[38]: [None,None,None,None] | ||
|
r3586 | |||
|
r3664 | In [39]: dview.pull('a') | ||
|
r3597 | Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234] | ||
|
r3586 | |||
|
r3670 | In [40]: dview.pull('b', targets=0) | ||
|
r3591 | Out[40]: 3453 | ||
|
r3586 | |||
|
r3664 | In [41]: dview.pull(('a','b')) | ||
|
r3597 | Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ] | ||
|
r3591 | |||
|
r3664 | In [43]: dview.push(dict(c='speed')) | ||
|
r3597 | Out[43]: [None,None,None,None] | ||
|
r3586 | |||
In non-blocking mode :meth:`push` and :meth:`pull` also return | ||||
|
r3591 | :class:`AsyncResult` objects: | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3664 | In [48]: ar = dview.pull('a', block=False) | ||
|
r3586 | |||
|
r3639 | In [49]: ar.get() | ||
|
r3591 | Out[49]: [1.03234, 1.03234, 1.03234, 1.03234] | ||
|
r3586 | |||
Dictionary interface | ||||
-------------------- | ||||
|
r3642 | Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide | ||
|
r3591 | dictionary-style access by key and methods such as :meth:`get` and | ||
:meth:`update` for convenience. This make the remote namespaces of the engines | ||||
|
r3642 | appear as a local dictionary. Underneath, these methods call :meth:`apply`: | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3635 | In [51]: dview['a']=['foo','bar'] | ||
|
r3586 | |||
|
r3635 | In [52]: dview['a'] | ||
|
r3597 | Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ] | ||
|
r3586 | |||
Scatter and gather | ||||
------------------ | ||||
Sometimes it is useful to partition a sequence and push the partitions to | ||||
different engines. In MPI language, this is know as scatter/gather and we | ||||
follow that terminology. However, it is important to remember that in | ||||
|
r3591 | IPython's :class:`Client` class, :meth:`scatter` is from the | ||
|
r3586 | interactive IPython session to the engines and :meth:`gather` is from the | ||
engines back to the interactive IPython session. For scatter/gather operations | ||||
|
r5169 | between engines, MPI, pyzmq, or some other direct interconnect should be used. | ||
|
r3586 | |||
.. sourcecode:: ipython | ||||
|
r3635 | In [58]: dview.scatter('a',range(16)) | ||
|
r3597 | Out[58]: [None,None,None,None] | ||
|
r3586 | |||
|
r3635 | In [59]: dview['a'] | ||
|
r3597 | Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ] | ||
|
r3586 | |||
|
r3635 | In [60]: dview.gather('a') | ||
|
r3591 | Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] | ||
|
r3586 | |||
Other things to look at | ||||
======================= | ||||
How to do parallel list comprehensions | ||||
-------------------------------------- | ||||
In many cases list comprehensions are nicer than using the map function. While | ||||
we don't have fully parallel list comprehensions, it is simple to get the | ||||
basic effect using :meth:`scatter` and :meth:`gather`: | ||||
.. sourcecode:: ipython | ||||
|
r3635 | In [66]: dview.scatter('x',range(64)) | ||
|
r3586 | |||
|
r3664 | In [67]: %px y = [i**10 for i in x] | ||
|
r3600 | Parallel execution on engines: [0, 1, 2, 3] | ||
|
r3670 | Out[67]: | ||
|
r3586 | |||
|
r3635 | In [68]: y = dview.gather('y') | ||
|
r3586 | |||
|
r3591 | In [69]: print y | ||
[0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...] | ||||
|
r3586 | |||
|
r3670 | Remote imports | ||
-------------- | ||||
Sometimes you will want to import packages both in your interactive session | ||||
and on your remote engines. This can be done with the :class:`ContextManager` | ||||
created by a DirectView's :meth:`sync_imports` method: | ||||
.. sourcecode:: ipython | ||||
In [69]: with dview.sync_imports(): | ||||
|
r5169 | ....: import numpy | ||
|
r3670 | importing numpy on engine(s) | ||
Any imports made inside the block will also be performed on the view's engines. | ||||
sync_imports also takes a `local` boolean flag that defaults to True, which specifies | ||||
whether the local imports should also be performed. However, support for `local=False` | ||||
has not been implemented, so only packages that can be imported locally will work | ||||
this way. | ||||
You can also specify imports via the ``@require`` decorator. This is a decorator | ||||
designed for use in Dependencies, but can be used to handle remote imports as well. | ||||
Modules or module names passed to ``@require`` will be imported before the decorated | ||||
function is called. If they cannot be imported, the decorated function will never | ||||
execution, and will fail with an UnmetDependencyError. | ||||
.. sourcecode:: ipython | ||||
In [69]: from IPython.parallel import require | ||||
|
r4271 | In [70]: @require('re'): | ||
|
r5169 | ....: def findall(pat, x): | ||
....: # re is guaranteed to be available | ||||
....: return re.findall(pat, x) | ||||
|
r3670 | |||
# you can also pass modules themselves, that you already have locally: | ||||
|
r4271 | In [71]: @require(time): | ||
|
r5169 | ....: def wait(t): | ||
....: time.sleep(t) | ||||
....: return t | ||||
|
r3670 | |||
|
r4109 | .. _parallel_exceptions: | ||
|
r3670 | |||
|
r3586 | Parallel exceptions | ||
------------------- | ||||
In the multiengine interface, parallel commands can raise Python exceptions, | ||||
just like serial commands. But, it is a little subtle, because a single | ||||
parallel command can actually raise multiple exceptions (one for each engine | ||||
|
r3642 | the command was run on). To express this idea, we have a | ||
|
r3586 | :exc:`CompositeError` exception class that will be raised in most cases. The | ||
:exc:`CompositeError` class is a special type of exception that wraps one or | ||||
more other types of exceptions. Here is how it works: | ||||
.. sourcecode:: ipython | ||||
|
r3642 | In [76]: dview.block=True | ||
|
r3586 | |||
|
r3642 | In [77]: dview.execute('1/0') | ||
|
r3591 | --------------------------------------------------------------------------- | ||
CompositeError Traceback (most recent call last) | ||||
|
r4109 | /home/user/<ipython-input-10-5d56b303a66c> in <module>() | ||
----> 1 dview.execute('1/0') | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block) | ||||
591 default: self.block | ||||
592 """ | ||||
--> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets) | ||||
594 | ||||
595 def run(self, filename, targets=None, block=None): | ||||
/home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs) | ||||
55 def sync_results(f, self, *args, **kwargs): | ||||
56 """sync relevant results from self.client to our results attribute.""" | ||||
---> 57 ret = f(self, *args, **kwargs) | ||||
58 delta = self.outstanding.difference(self.client.outstanding) | ||||
59 completed = self.outstanding.intersection(delta) | ||||
/home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs) | ||||
44 n_previous = len(self.client.history) | ||||
45 try: | ||||
---> 46 ret = f(self, *args, **kwargs) | ||||
47 finally: | ||||
48 nmsgs = len(self.client.history) - n_previous | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
529 if block: | ||||
530 try: | ||||
--> 531 return ar.get() | ||||
532 except KeyboardInterrupt: | ||||
533 pass | ||||
/path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout) | ||||
101 return self._result | ||||
102 else: | ||||
--> 103 raise self._exception | ||||
104 else: | ||||
105 raise error.TimeoutError("Result not ready.") | ||||
|
r3642 | |||
CompositeError: one or more exceptions from call to method: _execute | ||||
[0:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[1:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[2:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[3:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
|
r3586 | |||
Notice how the error message printed when :exc:`CompositeError` is raised has | ||||
information about the individual exceptions that were raised on each engine. | ||||
If you want, you can even raise one of these original exceptions: | ||||
.. sourcecode:: ipython | ||||
|
r3591 | In [80]: try: | ||
|
r3664 | ....: dview.execute('1/0') | ||
|
r4109 | ....: except parallel.error.CompositeError, e: | ||
|
r3591 | ....: e.raise_exception() | ||
....: | ||||
....: | ||||
--------------------------------------------------------------------------- | ||||
|
r4109 | RemoteError Traceback (most recent call last) | ||
/home/user/<ipython-input-17-8597e7e39858> in <module>() | ||||
2 dview.execute('1/0') | ||||
3 except CompositeError as e: | ||||
----> 4 e.raise_exception() | ||||
/path/to/site-packages/IPython/parallel/error.pyc in raise_exception(self, excid) | ||||
266 raise IndexError("an exception with index %i does not exist"%excid) | ||||
267 else: | ||||
--> 268 raise RemoteError(en, ev, etb, ei) | ||||
269 | ||||
270 | ||||
RemoteError: ZeroDivisionError(integer division or modulo by zero) | ||||
Traceback (most recent call last): | ||||
File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request | ||||
exec code in working,working | ||||
File "<string>", line 1, in <module> | ||||
File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute | ||||
exec code in globals() | ||||
File "<string>", line 1, in <module> | ||||
|
r3591 | ZeroDivisionError: integer division or modulo by zero | ||
|
r3586 | |||
If you are working in IPython, you can simple type ``%debug`` after one of | ||||
these :exc:`CompositeError` exceptions is raised, and inspect the exception | ||||
instance: | ||||
.. sourcecode:: ipython | ||||
|
r3664 | In [81]: dview.execute('1/0') | ||
|
r3591 | --------------------------------------------------------------------------- | ||
CompositeError Traceback (most recent call last) | ||||
|
r4109 | /home/user/<ipython-input-10-5d56b303a66c> in <module>() | ||
----> 1 dview.execute('1/0') | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block) | ||||
591 default: self.block | ||||
592 """ | ||||
--> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets) | ||||
594 | ||||
595 def run(self, filename, targets=None, block=None): | ||||
/home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs) | ||||
55 def sync_results(f, self, *args, **kwargs): | ||||
56 """sync relevant results from self.client to our results attribute.""" | ||||
---> 57 ret = f(self, *args, **kwargs) | ||||
58 delta = self.outstanding.difference(self.client.outstanding) | ||||
59 completed = self.outstanding.intersection(delta) | ||||
/home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs) | ||||
44 n_previous = len(self.client.history) | ||||
45 try: | ||||
---> 46 ret = f(self, *args, **kwargs) | ||||
47 finally: | ||||
48 nmsgs = len(self.client.history) - n_previous | ||||
/path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track) | ||||
529 if block: | ||||
530 try: | ||||
--> 531 return ar.get() | ||||
532 except KeyboardInterrupt: | ||||
533 pass | ||||
/path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout) | ||||
101 return self._result | ||||
102 else: | ||||
--> 103 raise self._exception | ||||
104 else: | ||||
105 raise error.TimeoutError("Result not ready.") | ||||
|
r3642 | |||
CompositeError: one or more exceptions from call to method: _execute | ||||
[0:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[1:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[2:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[3:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
|
r3591 | In [82]: %debug | ||
|
r4109 | > /path/to/site-packages/IPython/parallel/client/asyncresult.py(103)get() | ||
102 else: | ||||
--> 103 raise self._exception | ||||
104 else: | ||||
|
r3642 | |||
|
r4109 | # With the debugger running, self._exception is the exceptions instance. We can tab complete | ||
|
r3591 | # on it and see the extra methods that are available. | ||
|
r4109 | ipdb> self._exception.<tab> | ||
|
r3591 | e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args | ||
e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist | ||||
e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message | ||||
e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks | ||||
e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception | ||||
|
r4109 | ipdb> self._exception.print_tracebacks() | ||
|
r3642 | [0:apply]: | ||
Traceback (most recent call last): | ||||
|
r4109 | File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request | ||
exec code in working,working | ||||
|
r3642 | File "<string>", line 1, in <module> | ||
|
r4109 | File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute | ||
|
r3642 | exec code in globals() | ||
File "<string>", line 1, in <module> | ||||
|
r3591 | ZeroDivisionError: integer division or modulo by zero | ||
|
r3586 | |||
|
r3642 | [1:apply]: | ||
Traceback (most recent call last): | ||||
|
r4109 | File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request | ||
exec code in working,working | ||||
|
r3642 | File "<string>", line 1, in <module> | ||
|
r4109 | File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute | ||
|
r3642 | exec code in globals() | ||
File "<string>", line 1, in <module> | ||||
|
r3591 | ZeroDivisionError: integer division or modulo by zero | ||
|
r3586 | |||
|
r3642 | [2:apply]: | ||
Traceback (most recent call last): | ||||
|
r4109 | File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request | ||
exec code in working,working | ||||
|
r3642 | File "<string>", line 1, in <module> | ||
|
r4109 | File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute | ||
|
r3642 | exec code in globals() | ||
File "<string>", line 1, in <module> | ||||
|
r3591 | ZeroDivisionError: integer division or modulo by zero | ||
|
r3586 | |||
|
r3642 | [3:apply]: | ||
Traceback (most recent call last): | ||||
|
r4109 | File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request | ||
exec code in working,working | ||||
|
r3642 | File "<string>", line 1, in <module> | ||
|
r4109 | File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute | ||
|
r3642 | exec code in globals() | ||
File "<string>", line 1, in <module> | ||||
|
r3591 | ZeroDivisionError: integer division or modulo by zero | ||
|
r3664 | |||
|
r3586 | |||
All of this same error handling magic even works in non-blocking mode: | ||||
.. sourcecode:: ipython | ||||
|
r3664 | In [83]: dview.block=False | ||
|
r3591 | |||
|
r3664 | In [84]: ar = dview.execute('1/0') | ||
|
r3591 | |||
|
r3639 | In [85]: ar.get() | ||
|
r3591 | --------------------------------------------------------------------------- | ||
CompositeError Traceback (most recent call last) | ||||
|
r4109 | /home/user/<ipython-input-21-8531eb3d26fb> in <module>() | ||
|
r3642 | ----> 1 ar.get() | ||
|
r4109 | /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout) | ||
101 return self._result | ||||
102 else: | ||||
--> 103 raise self._exception | ||||
104 else: | ||||
105 raise error.TimeoutError("Result not ready.") | ||||
|
r3642 | |||
CompositeError: one or more exceptions from call to method: _execute | ||||
[0:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[1:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[2:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
[3:apply]: ZeroDivisionError: integer division or modulo by zero | ||||
|
r3586 | |||