parallel_task.txt
121 lines
| 4.1 KiB
| text/plain
|
TextLexer
|
r1256 | .. _paralleltask: | ||
|
r1678 | ========================== | ||
The IPython task interface | ||||
========================== | ||||
|
r1256 | |||
|
r2197 | The task interface to the controller presents the engines as a fault tolerant, | ||
dynamic load-balanced system or workers. Unlike the multiengine interface, in | ||||
the task interface, the user have no direct access to individual engines. In | ||||
some ways, this interface is simpler, but in other ways it is more powerful. | ||||
Best of all the user can use both of these interfaces running at the same time | ||||
to take advantage or both of their strengths. When the user can break up the | ||||
user's work into segments that do not depend on previous execution, the task | ||||
interface is ideal. But it also has more power and flexibility, allowing the | ||||
user to guide the distribution of jobs, without having to assign tasks to | ||||
engines explicitly. | ||||
|
r1256 | |||
Starting the IPython controller and engines | ||||
=========================================== | ||||
|
r1678 | To follow along with this tutorial, you will need to start the IPython | ||
controller and four IPython engines. The simplest way of doing this is to use | ||||
the :command:`ipcluster` command:: | ||||
|
r1256 | |||
|
r1788 | $ ipcluster local -n 4 | ||
|
r1256 | |||
|
r1678 | For more detailed information about starting the controller and engines, see | ||
our :ref:`introduction <ip1par>` to using IPython for parallel computing. | ||||
Creating a ``TaskClient`` instance | ||||
========================================= | ||||
The first step is to import the IPython :mod:`IPython.kernel.client` module | ||||
|
r1788 | and then create a :class:`TaskClient` instance: | ||
.. sourcecode:: ipython | ||||
|
r1678 | |||
In [1]: from IPython.kernel import client | ||||
In [2]: tc = client.TaskClient() | ||||
This form assumes that the :file:`ipcontroller-tc.furl` is in the | ||||
|
r3351 | :file:`$IPYTHON_DIR/cluster_<profile>/security` directory on the client's host. If not, the | ||
|
r1788 | location of the FURL file must be given as an argument to the | ||
constructor: | ||||
.. sourcecode:: ipython | ||||
|
r1678 | |||
|
r1788 | In [2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl') | ||
|
r1678 | |||
Quick and easy parallelism | ||||
========================== | ||||
|
r2197 | In many cases, you simply want to apply a Python function to a sequence of | ||
objects, but *in parallel*. Like the multiengine interface, the task interface | ||||
provides two simple ways of accomplishing this: a parallel version of | ||||
:func:`map` and ``@parallel`` function decorator. However, the verions in the | ||||
task interface have one important difference: they are dynamically load | ||||
balanced. Thus, if the execution time per item varies significantly, you | ||||
should use the versions in the task interface. | ||||
|
r1678 | |||
Parallel map | ||||
------------ | ||||
|
r2197 | The parallel :meth:`map` in the task interface is similar to that in the | ||
multiengine interface: | ||||
|
r1788 | |||
.. sourcecode:: ipython | ||||
|
r1678 | |||
In [63]: serial_result = map(lambda x:x**10, range(32)) | ||||
In [64]: parallel_result = tc.map(lambda x:x**10, range(32)) | ||||
In [65]: serial_result==parallel_result | ||||
Out[65]: True | ||||
Parallel function decorator | ||||
--------------------------- | ||||
|
r2197 | Parallel functions are just like normal function, but they can be called on | ||
sequences and *in parallel*. The multiengine interface provides a decorator | ||||
that turns any Python function into a parallel function: | ||||
|
r1788 | |||
.. sourcecode:: ipython | ||||
|
r1256 | |||
|
r1678 | In [10]: @tc.parallel() | ||
....: def f(x): | ||||
....: return 10.0*x**4 | ||||
....: | ||||
|
r1256 | |||
|
r1678 | In [11]: f(range(32)) # this is done in parallel | ||
Out[11]: | ||||
[0.0,10.0,160.0,...] | ||||
|
r1256 | |||
|
r1678 | More details | ||
============ | ||||
|
r1256 | |||
|
r2197 | The :class:`TaskClient` has many more powerful features that allow quite a bit | ||
of flexibility in how tasks are defined and run. The next places to look are | ||||
in the following classes: | ||||
|
r1256 | |||
|
r1678 | * :class:`IPython.kernel.client.TaskClient` | ||
* :class:`IPython.kernel.client.StringTask` | ||||
* :class:`IPython.kernel.client.MapTask` | ||||
|
r1256 | |||
|
r1678 | The following is an overview of how to use these classes together: | ||
|
r1256 | |||
|
r1678 | 1. Create a :class:`TaskClient`. | ||
2. Create one or more instances of :class:`StringTask` or :class:`MapTask` | ||||
to define your tasks. | ||||
3. Submit your tasks to using the :meth:`run` method of your | ||||
:class:`TaskClient` instance. | ||||
4. Use :meth:`TaskClient.get_task_result` to get the results of the | ||||
tasks. | ||||
|
r1256 | |||
|
r2197 | We are in the process of developing more detailed information about the task | ||
interface. For now, the docstrings of the :class:`TaskClient`, | ||||
:class:`StringTask` and :class:`MapTask` classes should be consulted. | ||||
|
r1256 | |||
|
r2275 | |||