##// END OF EJS Templates
use Context.instance() for default in SessionFactory
use Context.instance() for default in SessionFactory

File last commit:

r3990:080d3c71
r3998:80c7e64d
Show More
parallel_multiengine.txt
843 lines | 29.2 KiB | text/plain | TextLexer
/ docs / source / parallel / parallel_multiengine.txt
MinRK
update API after sagedays29...
r3664 .. _parallel_multiengine:
MinRK
clone parallel docs to parallelz
r3586
MinRK
parallelz doc updates, metadata bug fixed.
r3618 ==========================
IPython's Direct interface
==========================
MinRK
clone parallel docs to parallelz
r3586
MinRK
parallelz doc updates, metadata bug fixed.
r3618 The direct, or multiengine, interface represents one possible way of working with a set of
MinRK
clone parallel docs to parallelz
r3586 IPython engines. The basic idea behind the multiengine interface is that the
capabilities of each engine are directly and explicitly exposed to the user.
Thus, in the multiengine interface, each engine is given an id that is used to
identify the engine and give it work to do. This interface is very intuitive
MinRK
update API after sagedays29...
r3664 and is designed with interactive usage in mind, and is the best place for
MinRK
clone parallel docs to parallelz
r3586 new users of IPython to begin.
Starting the IPython controller and engines
===========================================
To follow along with this tutorial, you will need to start the IPython
controller and four IPython engines. The simplest way of doing this is to use
MinRK
rebase IPython.parallel after removal of IPython.kernel...
r3672 the :command:`ipcluster` command::
MinRK
clone parallel docs to parallelz
r3586
MinRK
parallel docs, tests, default config updated to newconfig
r3990 $ ipcluster start n=4
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
clone parallel docs to parallelz
r3586 For more detailed information about starting the controller and engines, see
our :ref:`introduction <ip1par>` to using IPython for parallel computing.
MinRK
initial draft of core zmq.parallel docs
r3591 Creating a ``Client`` instance
==============================
MinRK
clone parallel docs to parallelz
r3586
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 The first step is to import the IPython :mod:`IPython.parallel`
MinRK
initial draft of core zmq.parallel docs
r3591 module and then create a :class:`.Client` instance:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [1]: from IPython.parallel import Client
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [2]: rc = Client()
MinRK
clone parallel docs to parallelz
r3586
MinRK
parallelz doc updates, metadata bug fixed.
r3618 This form assumes that the default connection information (stored in
MinRK
rebase IPython.parallel after removal of IPython.kernel...
r3672 :file:`ipcontroller-client.json` found in :file:`IPYTHON_DIR/cluster_default/security`) is
MinRK
parallelz doc updates, metadata bug fixed.
r3618 accurate. If the controller was started on a remote machine, you must copy that connection
file to the client machine, or enter its contents as arguments to the Client constructor:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
parallelz doc updates, metadata bug fixed.
r3618 # If you have copied the json connector file from the controller:
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [2]: rc = Client('/path/to/ipcontroller-client.json')
MinRK
Doc tweaks and updates
r3663 # or to connect with a specific profile you have set up:
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [3]: rc = Client(profile='mpi')
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 To make sure there are engines connected to the controller, users can get a list
MinRK
clone parallel docs to parallelz
r3586 of engine ids:
.. sourcecode:: ipython
MinRK
initial draft of core zmq.parallel docs
r3591 In [3]: rc.ids
MinRK
API update involving map and load-balancing
r3635 Out[3]: [0, 1, 2, 3]
MinRK
clone parallel docs to parallelz
r3586
Here we see that there are four engines ready to do work for us.
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 For direct execution, we will make use of a :class:`DirectView` object, which can be
constructed via list-access to the client:
MinRK
Doc tweaks and updates
r3663 .. sourcecode:: ipython
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
In [4]: dview = rc[:] # use all engines
.. seealso::
MinRK
reflect revised apply_bound pattern
r3655 For more information, see the in-depth explanation of :ref:`Views <parallel_details>`.
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
MinRK
clone parallel docs to parallelz
r3586 Quick and easy parallelism
==========================
In many cases, you simply want to apply a Python function to a sequence of
MinRK
initial draft of core zmq.parallel docs
r3591 objects, but *in parallel*. The client interface provides a simple way
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 of accomplishing this: using the DirectView's :meth:`~DirectView.map` method.
MinRK
clone parallel docs to parallelz
r3586
Parallel map
------------
Python's builtin :func:`map` functions allows a function to be applied to a
sequence element-by-element. This type of code is typically trivial to
MinRK
docs include 'apply'
r3594 parallelize. In fact, since IPython's interface is all about functions anyway,
MinRK
API update involving map and load-balancing
r3635 you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a
DirectView's :meth:`map` method:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
initial draft of core zmq.parallel docs
r3591 In [62]: serial_result = map(lambda x:x**10, range(32))
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
MinRK
update API after sagedays29...
r3664 In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32))
MinRK
clone parallel docs to parallelz
r3586
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [67]: serial_result==parallel_result
MinRK
initial draft of core zmq.parallel docs
r3591 Out[67]: True
MinRK
clone parallel docs to parallelz
r3586
.. note::
MinRK
API update involving map and load-balancing
r3635 The :class:`DirectView`'s version of :meth:`map` does
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 not do dynamic load balancing. For a load balanced version, use a
MinRK
update API after sagedays29...
r3664 :class:`LoadBalancedView`.
MinRK
clone parallel docs to parallelz
r3586
.. seealso::
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 :meth:`map` is implemented via :class:`ParallelFunction`.
MinRK
clone parallel docs to parallelz
r3586
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 Remote function decorators
--------------------------
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 Remote functions are just like normal functions, but when they are called,
they execute on one or more engines, rather than locally. IPython provides
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 two decorators:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [10]: @dview.remote(block=True)
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 ...: def getpid():
...: import os
...: return os.getpid()
...:
In [11]: getpid()
Out[11]: [12345, 12346, 12347, 12348]
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 The ``@parallel`` decorator creates parallel functions, that break up an element-wise
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 operations and distribute them, reconstructing the result.
.. sourcecode:: ipython
In [12]: import numpy as np
In [13]: A = np.random.random((64,48))
MinRK
update API after sagedays29...
r3664 In [14]: @dview.parallel(block=True)
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 ...: def pmul(A,B):
...: return A*B
In [15]: C_local = A*A
MinRK
update API after sagedays29...
r3664 In [16]: C_remote = pmul(A,A)
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
In [17]: (C_local == C_remote).all()
Out[17]: True
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 .. seealso::
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 See the docstrings for the :func:`parallel` and :func:`remote` decorators for
MinRK
API update involving map and load-balancing
r3635 options.
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 Calling Python functions
========================
MinRK
clone parallel docs to parallelz
r3586
The most basic type of operation that can be performed on the engines is to
MinRK
initial draft of core zmq.parallel docs
r3591 execute Python code or call Python functions. Executing Python code can be
done in blocking or non-blocking mode (non-blocking is default) using the
MinRK
update API after sagedays29...
r3664 :meth:`.View.execute` method, and calling functions can be done via the
MinRK
initial draft of core zmq.parallel docs
r3591 :meth:`.View.apply` method.
MinRK
clone parallel docs to parallelz
r3586
MinRK
docs include 'apply'
r3594 apply
-----
The main method for doing remote execution (in fact, all methods that
MinRK
update API after sagedays29...
r3664 communicate with the engines are built on top of it), is :meth:`View.apply`.
MinRK
docs include 'apply'
r3594
MinRK
update API after sagedays29...
r3664 We strive to provide the cleanest interface we can, so `apply` has the following
signature:
MinRK
docs include 'apply'
r3594
MinRK
update API after sagedays29...
r3664 .. sourcecode:: python
MinRK
eliminate relative imports
r3642
MinRK
update API after sagedays29...
r3664 view.apply(f, *args, **kwargs)
MinRK
eliminate relative imports
r3642
MinRK
update API after sagedays29...
r3664 There are various ways to call functions with IPython, and these flags are set as
attributes of the View. The ``DirectView`` has just two of these flags:
MinRK
eliminate relative imports
r3642
MinRK
update API after sagedays29...
r3664 dv.block : bool
whether to wait for the result, or return an :class:`AsyncResult` object
immediately
dv.track : bool
whether to instruct pyzmq to track when
This is primarily useful for non-copying sends of numpy arrays that you plan to
edit in-place. You need to know when it becomes safe to edit the buffer
without corrupting the message.
MinRK
eliminate relative imports
r3642
MinRK
update API after sagedays29...
r3664 Creating a view is simple: index-access on a client creates a :class:`.DirectView`.
MinRK
docs include 'apply'
r3594
.. sourcecode:: ipython
In [4]: view = rc[1:3]
Out[4]: <DirectView [1, 2]>
In [5]: view.apply<tab>
MinRK
updates to docs and examples
r3670 view.apply view.apply_async view.apply_sync
MinRK
docs include 'apply'
r3594
MinRK
update API after sagedays29...
r3664 For convenience, you can set block temporarily for a single call with the extra sync/async methods.
MinRK
docs include 'apply'
r3594
MinRK
clone parallel docs to parallelz
r3586 Blocking execution
------------------
MinRK
initial draft of core zmq.parallel docs
r3591 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
MinRK
clone parallel docs to parallelz
r3586 these examples) submits the command to the controller, which places the
MinRK
initial draft of core zmq.parallel docs
r3591 command in the engines' queues for execution. The :meth:`apply` call then
MinRK
clone parallel docs to parallelz
r3586 blocks until the engines are done executing the command:
.. sourcecode:: ipython
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [2]: dview = rc[:] # A DirectView of all engines
In [3]: dview.block=True
MinRK
initial draft of core zmq.parallel docs
r3591 In [4]: dview['a'] = 5
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [5]: dview['b'] = 10
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [6]: dview.apply(lambda x: a+b+x, 27)
MinRK
reflect revised apply_bound pattern
r3655 Out[6]: [42, 42, 42, 42]
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync`
method:
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [7]: dview.block=False
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [8]: dview.apply_sync(lambda x: a+b+x, 27)
Out[8]: [42, 42, 42, 42]
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
MinRK
update API after sagedays29...
r3664 Python commands can be executed as strings on specific engines by using a View's ``execute``
method:
MinRK
reflect revised apply_bound pattern
r3655
MinRK
update API after sagedays29...
r3664 .. sourcecode:: ipython
MinRK
reflect revised apply_bound pattern
r3655
MinRK
update API after sagedays29...
r3664 In [6]: rc[::2].execute('c=a+b')
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [7]: rc[1::2].execute('c=a-b')
MinRK
clone parallel docs to parallelz
r3586
MinRK
updates to docs and examples
r3670 In [8]: dview['c'] # shorthand for dview.pull('c', block=True)
MinRK
update API after sagedays29...
r3664 Out[8]: [15, -5, 15, -5]
MinRK
clone parallel docs to parallelz
r3586
Non-blocking execution
----------------------
MinRK
initial draft of core zmq.parallel docs
r3591 In non-blocking mode, :meth:`apply` submits the command to be executed and
then returns a :class:`AsyncResult` object immediately. The
:class:`AsyncResult` object gives you a way of getting a result at a later
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 time through its :meth:`get` method.
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 .. Note::
MinRK
dependency tweaks + dependency/scheduler docs
r3624 The :class:`AsyncResult` object provides a superset of the interface in
MinRK
initial draft of core zmq.parallel docs
r3591 :py:class:`multiprocessing.pool.AsyncResult`. See the
`official Python documentation <http://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_
for more.
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591
This allows you to quickly submit long running commands without blocking your
local Python/IPython session:
.. sourcecode:: ipython
# define our function
MinRK
dependency tweaks + dependency/scheduler docs
r3624 In [6]: def wait(t):
...: import time
...: tic = time.time()
...: time.sleep(t)
...: return time.time()-tic
MinRK
initial draft of core zmq.parallel docs
r3591
# In non-blocking mode
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [7]: ar = dview.apply_async(wait, 2)
MinRK
initial draft of core zmq.parallel docs
r3591
# Now block for the result
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [8]: ar.get()
MinRK
initial draft of core zmq.parallel docs
r3591 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
# Again in non-blocking mode
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [9]: ar = dview.apply_async(wait, 10)
MinRK
initial draft of core zmq.parallel docs
r3591
# Poll to see if the result is ready
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [10]: ar.ready()
MinRK
initial draft of core zmq.parallel docs
r3591 Out[10]: False
# ask for the result, but wait a maximum of 1 second:
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [45]: ar.get(1)
MinRK
initial draft of core zmq.parallel docs
r3591 ---------------------------------------------------------------------------
TimeoutError Traceback (most recent call last)
/home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 ----> 1 ar.get(1)
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
MinRK
initial draft of core zmq.parallel docs
r3591 62 raise self._exception
63 else:
---> 64 raise error.TimeoutError("Result not ready.")
65
66 def ready(self):
TimeoutError: Result not ready.
.. Note::
Note the import inside the function. This is a common model, to ensure
MinRK
update API after sagedays29...
r3664 that the appropriate modules are imported where the task is run. You can
also manually import modules into the engine(s) namespace(s) via
:meth:`view.execute('import numpy')`.
MinRK
initial draft of core zmq.parallel docs
r3591
Often, it is desirable to wait until a set of :class:`AsyncResult` objects
MinRK
update API after sagedays29...
r3664 are done. For this, there is a the method :meth:`wait`. This method takes a
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History),
and blocks until all of the associated results are ready:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [72]: dview.block=False
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 # A trivial list of AsyncResults objects
MinRK
API update involving map and load-balancing
r3635 In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)]
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 # Wait until all of them are done
MinRK
update API after sagedays29...
r3664 In [74]: dview.wait(pr_list)
MinRK
clone parallel docs to parallelz
r3586
MinRK
dependency tweaks + dependency/scheduler docs
r3624 # Then, their results are ready using get() or the `.r` attribute
MinRK
initial draft of core zmq.parallel docs
r3591 In [75]: pr_list[0].get()
Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 The ``block`` and ``targets`` keyword arguments and attributes
--------------------------------------------------------------
Most DirectView methods (excluding :meth:`apply` and :meth:`map`) accept ``block`` and
``targets`` as keyword arguments. As we have seen above, these keyword arguments control the
blocking mode and which engines the command is applied to. The :class:`View` class also has
:attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword
arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`:
MinRK
clone parallel docs to parallelz
r3586
* If no keyword argument is provided, the instance attributes are used.
MinRK
API update involving map and load-balancing
r3635 * Keyword argument, if provided override the instance attributes for
the duration of a single call.
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
MinRK
clone parallel docs to parallelz
r3586 The following examples demonstrate how to use the instance attributes:
.. sourcecode:: ipython
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [16]: dview.targets = [0,2]
MinRK
update API after sagedays29...
r3664 In [17]: dview.block = False
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [18]: ar = dview.apply(lambda : 10)
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 In [19]: ar.get()
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 Out[19]: [10, 10]
MinRK
clone parallel docs to parallelz
r3586
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 In [16]: dview.targets = v.client.ids # all engines (4)
MinRK
update API after sagedays29...
r3664 In [21]: dview.block = True
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [22]: dview.apply(lambda : 42)
MinRK
API update involving map and load-balancing
r3635 Out[22]: [42, 42, 42, 42]
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 The :attr:`block` and :attr:`targets` instance attributes of the
MinRK
API update involving map and load-balancing
r3635 :class:`.DirectView` also determine the behavior of the parallel magic commands.
MinRK
clone parallel docs to parallelz
r3586
Parallel magic commands
-----------------------
MinRK
initial draft of core zmq.parallel docs
r3591 .. warning::
MinRK
update API after sagedays29...
r3664 The magics have not been changed to work with the zeromq system. The
magics do work, but *do not* print stdin/out like they used to in IPython.kernel.
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
clone parallel docs to parallelz
r3586 We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
that make it more pleasant to execute Python commands on the engines
interactively. These are simply shortcuts to :meth:`execute` and
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 :meth:`get_result` of the :class:`DirectView`. The ``%px`` magic executes a single
Python command on the engines specified by the :attr:`targets` attribute of the
:class:`DirectView` instance:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 # load the parallel magic extension:
In [21]: %load_ext parallelmagic
MinRK
initial draft of core zmq.parallel docs
r3591 # Create a DirectView for all targets
In [22]: dv = rc[:]
# Make this DirectView active for parallel magic commands
In [23]: dv.activate()
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [24]: dv.block=True
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [25]: import numpy
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [26]: %px import numpy
Parallel execution on engines: [0, 1, 2, 3]
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [27]: %px a = numpy.random.rand(2,2)
Parallel execution on engines: [0, 1, 2, 3]
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [28]: %px ev = numpy.linalg.eigvals(a)
Parallel execution on engines: [0, 1, 2, 3]
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [28]: dv['ev']
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 Out[28]: [ array([ 1.09522024, -0.09645227]),
MinRK
parallelz updates
r3597 array([ 1.21435496, -0.35546712]),
array([ 0.72180653, 0.07133042]),
array([ 1.46384341e+00, 1.04353244e-04])
]
MinRK
clone parallel docs to parallelz
r3586
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 The ``%result`` magic gets the most recent result, or takes an argument
specifying the index of the result to be requested. It is simply a shortcut to the
MinRK
clone parallel docs to parallelz
r3586 :meth:`get_result` method:
.. sourcecode:: ipython
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
MinRK
reflect revised apply_bound pattern
r3655 In [29]: dv.apply_async(lambda : ev)
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639
In [30]: %result
Out[30]: [ [ 1.28167017 0.14197338],
[-0.14093616 1.27877273],
[-0.37023573 1.06779409],
[ 0.83664764 -0.25602658] ]
MinRK
clone parallel docs to parallelz
r3586
The ``%autopx`` magic switches to a mode where everything you type is executed
on the engines given by the :attr:`targets` attribute:
.. sourcecode:: ipython
MinRK
initial draft of core zmq.parallel docs
r3591 In [30]: dv.block=False
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [31]: %autopx
Auto Parallel Enabled
Type %autopx to disable
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [32]: max_evals = []
MinRK
organize IPython.parallel into subpackages
r3673 <IPython.parallel.AsyncResult object at 0x17b8a70>
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [33]: for i in range(100):
....: a = numpy.random.rand(10,10)
....: a = a+a.transpose()
....: evals = numpy.linalg.eigvals(a)
....: max_evals.append(evals[0].real)
....:
....:
MinRK
organize IPython.parallel into subpackages
r3673 <IPython.parallel.AsyncResult object at 0x17af8f0>
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [34]: %autopx
Auto Parallel Disabled
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [35]: dv.block=True
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [36]: px ans= "Average max eigenvalue is: %f"%(sum(max_evals)/len(max_evals))
Parallel execution on engines: [0, 1, 2, 3]
In [37]: dv['ans']
MinRK
parallelz updates
r3597 Out[37]: [ 'Average max eigenvalue is: 10.1387247332',
'Average max eigenvalue is: 10.2076902286',
'Average max eigenvalue is: 10.1891484655',
'Average max eigenvalue is: 10.1158837784',]
MinRK
clone parallel docs to parallelz
r3586
Moving Python objects around
============================
MinRK
initial draft of core zmq.parallel docs
r3591 In addition to calling functions and executing code on engines, you can
transfer Python objects to and from your IPython session and the engines. In
IPython, these operations are called :meth:`push` (sending an object to the
MinRK
update API after sagedays29...
r3664 engines) and :meth:`pull` (getting an object from the engines).
MinRK
clone parallel docs to parallelz
r3586
Basic push and pull
-------------------
Here are some examples of how you use :meth:`push` and :meth:`pull`:
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [38]: dview.push(dict(a=1.03234,b=3453))
MinRK
parallelz updates
r3597 Out[38]: [None,None,None,None]
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [39]: dview.pull('a')
MinRK
parallelz updates
r3597 Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234]
MinRK
clone parallel docs to parallelz
r3586
MinRK
updates to docs and examples
r3670 In [40]: dview.pull('b', targets=0)
MinRK
initial draft of core zmq.parallel docs
r3591 Out[40]: 3453
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [41]: dview.pull(('a','b'))
MinRK
parallelz updates
r3597 Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ]
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
update API after sagedays29...
r3664 In [43]: dview.push(dict(c='speed'))
MinRK
parallelz updates
r3597 Out[43]: [None,None,None,None]
MinRK
clone parallel docs to parallelz
r3586
In non-blocking mode :meth:`push` and :meth:`pull` also return
MinRK
initial draft of core zmq.parallel docs
r3591 :class:`AsyncResult` objects:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [48]: ar = dview.pull('a', block=False)
MinRK
clone parallel docs to parallelz
r3586
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [49]: ar.get()
MinRK
initial draft of core zmq.parallel docs
r3591 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
MinRK
clone parallel docs to parallelz
r3586
Dictionary interface
--------------------
MinRK
eliminate relative imports
r3642 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
MinRK
initial draft of core zmq.parallel docs
r3591 dictionary-style access by key and methods such as :meth:`get` and
:meth:`update` for convenience. This make the remote namespaces of the engines
MinRK
eliminate relative imports
r3642 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
MinRK
clone parallel docs to parallelz
r3586
.. sourcecode:: ipython
MinRK
API update involving map and load-balancing
r3635 In [51]: dview['a']=['foo','bar']
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 In [52]: dview['a']
MinRK
parallelz updates
r3597 Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
MinRK
clone parallel docs to parallelz
r3586
Scatter and gather
------------------
Sometimes it is useful to partition a sequence and push the partitions to
different engines. In MPI language, this is know as scatter/gather and we
follow that terminology. However, it is important to remember that in
MinRK
initial draft of core zmq.parallel docs
r3591 IPython's :class:`Client` class, :meth:`scatter` is from the
MinRK
clone parallel docs to parallelz
r3586 interactive IPython session to the engines and :meth:`gather` is from the
engines back to the interactive IPython session. For scatter/gather operations
between engines, MPI should be used:
.. sourcecode:: ipython
MinRK
API update involving map and load-balancing
r3635 In [58]: dview.scatter('a',range(16))
MinRK
parallelz updates
r3597 Out[58]: [None,None,None,None]
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 In [59]: dview['a']
MinRK
parallelz updates
r3597 Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 In [60]: dview.gather('a')
MinRK
initial draft of core zmq.parallel docs
r3591 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
MinRK
clone parallel docs to parallelz
r3586
Other things to look at
=======================
How to do parallel list comprehensions
--------------------------------------
In many cases list comprehensions are nicer than using the map function. While
we don't have fully parallel list comprehensions, it is simple to get the
basic effect using :meth:`scatter` and :meth:`gather`:
.. sourcecode:: ipython
MinRK
API update involving map and load-balancing
r3635 In [66]: dview.scatter('x',range(64))
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 In [67]: %px y = [i**10 for i in x]
MinRK
update connection/message docs for newparallel
r3600 Parallel execution on engines: [0, 1, 2, 3]
MinRK
updates to docs and examples
r3670 Out[67]:
MinRK
clone parallel docs to parallelz
r3586
MinRK
API update involving map and load-balancing
r3635 In [68]: y = dview.gather('y')
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 In [69]: print y
[0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
MinRK
clone parallel docs to parallelz
r3586
MinRK
updates to docs and examples
r3670 Remote imports
--------------
Sometimes you will want to import packages both in your interactive session
and on your remote engines. This can be done with the :class:`ContextManager`
created by a DirectView's :meth:`sync_imports` method:
.. sourcecode:: ipython
In [69]: with dview.sync_imports():
...: import numpy
importing numpy on engine(s)
Any imports made inside the block will also be performed on the view's engines.
sync_imports also takes a `local` boolean flag that defaults to True, which specifies
whether the local imports should also be performed. However, support for `local=False`
has not been implemented, so only packages that can be imported locally will work
this way.
You can also specify imports via the ``@require`` decorator. This is a decorator
designed for use in Dependencies, but can be used to handle remote imports as well.
Modules or module names passed to ``@require`` will be imported before the decorated
function is called. If they cannot be imported, the decorated function will never
execution, and will fail with an UnmetDependencyError.
.. sourcecode:: ipython
In [69]: from IPython.parallel import require
In [70]: @requre('re'):
...: def findall(pat, x):
...: # re is guaranteed to be available
...: return re.findall(pat, x)
# you can also pass modules themselves, that you already have locally:
In [71]: @requre(time):
...: def wait(t):
...: time.sleep(t)
...: return t
MinRK
clone parallel docs to parallelz
r3586 Parallel exceptions
-------------------
In the multiengine interface, parallel commands can raise Python exceptions,
just like serial commands. But, it is a little subtle, because a single
parallel command can actually raise multiple exceptions (one for each engine
MinRK
eliminate relative imports
r3642 the command was run on). To express this idea, we have a
MinRK
clone parallel docs to parallelz
r3586 :exc:`CompositeError` exception class that will be raised in most cases. The
:exc:`CompositeError` class is a special type of exception that wraps one or
more other types of exceptions. Here is how it works:
.. sourcecode:: ipython
MinRK
eliminate relative imports
r3642 In [76]: dview.block=True
MinRK
clone parallel docs to parallelz
r3586
MinRK
eliminate relative imports
r3642 In [77]: dview.execute('1/0')
MinRK
initial draft of core zmq.parallel docs
r3591 ---------------------------------------------------------------------------
CompositeError Traceback (most recent call last)
MinRK
update API after sagedays29...
r3664 /home/you/<ipython-input-10-15c2c22dec39> in <module>()
----> 1 dview.execute('1/0', block=True)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in execute(self, code, block)
MinRK
update API after sagedays29...
r3664 460 default: self.block
461 """
--> 462 return self.apply_with_flags(util._execute, args=(code,), block=block)
463
464 def run(self, filename, block=None):
/home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in sync_results(f, self, *args, **kwargs)
MinRK
update API after sagedays29...
r3664 46 def sync_results(f, self, *args, **kwargs):
47 """sync relevant results from self.client to our results attribute."""
---> 48 ret = f(self, *args, **kwargs)
49 delta = self.outstanding.difference(self.client.outstanding)
50 completed = self.outstanding.intersection(delta)
/home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in save_ids(f, self, *args, **kwargs)
MinRK
update API after sagedays29...
r3664 35 n_previous = len(self.client.history)
36 try:
---> 37 ret = f(self, *args, **kwargs)
38 finally:
39 nmsgs = len(self.client.history) - n_previous
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
update API after sagedays29...
r3664 398 if block:
399 try:
--> 400 return ar.get()
401 except KeyboardInterrupt:
402 pass
MinRK
eliminate relative imports
r3642
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
MinRK
update API after sagedays29...
r3664 87 return self._result
88 else:
---> 89 raise self._exception
90 else:
91 raise error.TimeoutError("Result not ready.")
MinRK
eliminate relative imports
r3642
CompositeError: one or more exceptions from call to method: _execute
[0:apply]: ZeroDivisionError: integer division or modulo by zero
[1:apply]: ZeroDivisionError: integer division or modulo by zero
[2:apply]: ZeroDivisionError: integer division or modulo by zero
[3:apply]: ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664
MinRK
clone parallel docs to parallelz
r3586 Notice how the error message printed when :exc:`CompositeError` is raised has
information about the individual exceptions that were raised on each engine.
If you want, you can even raise one of these original exceptions:
.. sourcecode:: ipython
MinRK
initial draft of core zmq.parallel docs
r3591 In [80]: try:
MinRK
update API after sagedays29...
r3664 ....: dview.execute('1/0')
MinRK
initial draft of core zmq.parallel docs
r3591 ....: except client.CompositeError, e:
....: e.raise_exception()
....:
....:
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 /ipython1-client-r3021/ipython1/kernel/error.pyc in raise_exception(self, excid)
156 raise IndexError("an exception with index %i does not exist"%excid)
157 else:
--> 158 raise et, ev, etb
159
160 def collect_exceptions(rlist, method):
MinRK
clone parallel docs to parallelz
r3586
MinRK
initial draft of core zmq.parallel docs
r3591 ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586
If you are working in IPython, you can simple type ``%debug`` after one of
these :exc:`CompositeError` exceptions is raised, and inspect the exception
instance:
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [81]: dview.execute('1/0')
MinRK
initial draft of core zmq.parallel docs
r3591 ---------------------------------------------------------------------------
CompositeError Traceback (most recent call last)
MinRK
update API after sagedays29...
r3664 /home/you/<ipython-input-10-15c2c22dec39> in <module>()
----> 1 dview.execute('1/0', block=True)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in execute(self, code, block)
MinRK
update API after sagedays29...
r3664 460 default: self.block
461 """
--> 462 return self.apply_with_flags(util._execute, args=(code,), block=block)
463
464 def run(self, filename, block=None):
/home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in sync_results(f, self, *args, **kwargs)
MinRK
update API after sagedays29...
r3664 46 def sync_results(f, self, *args, **kwargs):
47 """sync relevant results from self.client to our results attribute."""
---> 48 ret = f(self, *args, **kwargs)
49 delta = self.outstanding.difference(self.client.outstanding)
50 completed = self.outstanding.intersection(delta)
/home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in save_ids(f, self, *args, **kwargs)
MinRK
update API after sagedays29...
r3664 35 n_previous = len(self.client.history)
36 try:
---> 37 ret = f(self, *args, **kwargs)
38 finally:
39 nmsgs = len(self.client.history) - n_previous
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/view.py in apply_with_flags(self, f, args, kwargs, block, track)
MinRK
update API after sagedays29...
r3664 398 if block:
399 try:
--> 400 return ar.get()
401 except KeyboardInterrupt:
402 pass
MinRK
eliminate relative imports
r3642
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
MinRK
update API after sagedays29...
r3664 87 return self._result
88 else:
---> 89 raise self._exception
90 else:
91 raise error.TimeoutError("Result not ready.")
MinRK
eliminate relative imports
r3642
CompositeError: one or more exceptions from call to method: _execute
[0:apply]: ZeroDivisionError: integer division or modulo by zero
[1:apply]: ZeroDivisionError: integer division or modulo by zero
[2:apply]: ZeroDivisionError: integer division or modulo by zero
[3:apply]: ZeroDivisionError: integer division or modulo by zero
MinRK
initial draft of core zmq.parallel docs
r3591 In [82]: %debug
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 > /path/to/site-packages/IPython/parallel/asyncresult.py(80)get()
MinRK
eliminate relative imports
r3642 79 else:
---> 80 raise self._exception
81 else:
MinRK
initial draft of core zmq.parallel docs
r3591
# With the debugger running, e is the exceptions instance. We can tab complete
# on it and see the extra methods that are available.
ipdb> e.
e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args
e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist
e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message
e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks
e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception
ipdb> e.print_tracebacks()
MinRK
eliminate relative imports
r3642 [0:apply]:
Traceback (most recent call last):
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
MinRK
eliminate relative imports
r3642 exec code in working, working
File "<string>", line 1, in <module>
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
MinRK
eliminate relative imports
r3642 exec code in globals()
File "<string>", line 1, in <module>
MinRK
initial draft of core zmq.parallel docs
r3591 ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586
MinRK
eliminate relative imports
r3642 [1:apply]:
Traceback (most recent call last):
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
MinRK
eliminate relative imports
r3642 exec code in working, working
File "<string>", line 1, in <module>
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
MinRK
eliminate relative imports
r3642 exec code in globals()
File "<string>", line 1, in <module>
MinRK
initial draft of core zmq.parallel docs
r3591 ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586
MinRK
eliminate relative imports
r3642 [2:apply]:
Traceback (most recent call last):
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
MinRK
eliminate relative imports
r3642 exec code in working, working
File "<string>", line 1, in <module>
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
MinRK
eliminate relative imports
r3642 exec code in globals()
File "<string>", line 1, in <module>
MinRK
initial draft of core zmq.parallel docs
r3591 ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586
MinRK
eliminate relative imports
r3642 [3:apply]:
Traceback (most recent call last):
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
MinRK
eliminate relative imports
r3642 exec code in working, working
File "<string>", line 1, in <module>
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
MinRK
eliminate relative imports
r3642 exec code in globals()
File "<string>", line 1, in <module>
MinRK
initial draft of core zmq.parallel docs
r3591 ZeroDivisionError: integer division or modulo by zero
MinRK
eliminate relative imports
r3642
MinRK
clone parallel docs to parallelz
r3586
MinRK
update API after sagedays29...
r3664 .. note::
TODO: The above tracebacks are not up to date
MinRK
clone parallel docs to parallelz
r3586
All of this same error handling magic even works in non-blocking mode:
.. sourcecode:: ipython
MinRK
update API after sagedays29...
r3664 In [83]: dview.block=False
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
update API after sagedays29...
r3664 In [84]: ar = dview.execute('1/0')
MinRK
initial draft of core zmq.parallel docs
r3591
MinRK
split get_results into get_result/result_status, add AsyncHubResult
r3639 In [85]: ar.get()
MinRK
initial draft of core zmq.parallel docs
r3591 ---------------------------------------------------------------------------
CompositeError Traceback (most recent call last)
MinRK
eliminate relative imports
r3642 /Users/minrk/<ipython-input-3-8531eb3d26fb> in <module>()
----> 1 ar.get()
MinRK
move IPython.zmq.parallel to IPython.parallel
r3666 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
MinRK
eliminate relative imports
r3642 78 return self._result
79 else:
---> 80 raise self._exception
81 else:
82 raise error.TimeoutError("Result not ready.")
CompositeError: one or more exceptions from call to method: _execute
[0:apply]: ZeroDivisionError: integer division or modulo by zero
[1:apply]: ZeroDivisionError: integer division or modulo by zero
[2:apply]: ZeroDivisionError: integer division or modulo by zero
[3:apply]: ZeroDivisionError: integer division or modulo by zero
MinRK
clone parallel docs to parallelz
r3586