##// END OF EJS Templates
initial draft of core zmq.parallel docs
MinRK -
Show More
@@ -350,8 +350,38 b' and the built-in Python on OS X comes with wxPython preinstalled. For Windows,'
350 a binary installer is available on the `wxPython website
350 a binary installer is available on the `wxPython website
351 <http://www.wxpython.org/>`_.
351 <http://www.wxpython.org/>`_.
352
352
353 Dependencies for IPython.zmq (new parallel)
354 ===========================================
355
356 pyzmq
357 -----
358
359 IPython 0.11 introduced some new functionality, including a two-process
360 execution model using ZeroMQ for communication [ZeroMQ]_. The Python bindings
361 to ZeroMQ are found in the pyzmq project, which is easy_install-able once you
362 have ZeroMQ installed. :mod:`IPython.kernel` is also in the process of being
363 replaced by :mod:`IPython.zmq.parallel`, which uses ZeroMQ for all
364 communication.
365
366 Dependencies for ipython-qtconsole (new GUI)
367 ============================================
368
369 PyQt
370 ----
371
372 Also with 0.11, a new GUI was added using the work in :mod:`IPython.zmq`,
373 which can be launched with ``ipython-qtconsole``. The GUI is built on PyQt ,
374 which can be installed from the
375 `PyQt website <http://www.riverbankcomputing.co.uk/>`_.
376
377 pygments
378 --------
379
380 The syntax-highlighting in ``ipython-qtconsole`` is done with the pygments project, which is easy_install-able.
381
353 .. [Twisted] Twisted matrix. http://twistedmatrix.org
382 .. [Twisted] Twisted matrix. http://twistedmatrix.org
354 .. [ZopeInterface] http://pypi.python.org/pypi/zope.interface
383 .. [ZopeInterface] http://pypi.python.org/pypi/zope.interface
355 .. [Foolscap] Foolscap network protocol. http://foolscap.lothar.com/trac
384 .. [Foolscap] Foolscap network protocol. http://foolscap.lothar.com/trac
356 .. [pyOpenSSL] pyOpenSSL. http://pyopenssl.sourceforge.net
385 .. [pyOpenSSL] pyOpenSSL. http://pyopenssl.sourceforge.net
386 .. [ZeroMQ] ZeroMQ. http://www.zeromq.org
357
387
@@ -50,17 +50,20 b' the ``I`` in IPython. The following are some example usage cases for IPython:'
50 Architecture overview
50 Architecture overview
51 =====================
51 =====================
52
52
53 The IPython architecture consists of three components:
53 The IPython architecture consists of four components:
54
54
55 * The IPython engine.
55 * The IPython engine.
56 * The IPython controller.
56 * The IPython controller.
57 * Various controller clients.
57 * The IPython scheduler.
58 * The controller client.
58
59
59 These components live in the :mod:`IPython.kernel` package and are
60 These components live in the :mod:`IPython.zmq.parallel` package and are
60 installed with IPython. They do, however, have additional dependencies
61 installed with IPython. They do, however, have additional dependencies
61 that must be installed. For more information, see our
62 that must be installed. For more information, see our
62 :ref:`installation documentation <install_index>`.
63 :ref:`installation documentation <install_index>`.
63
64
65 .. TODO: include zmq in install_index
66
64 IPython engine
67 IPython engine
65 ---------------
68 ---------------
66
69
@@ -78,92 +81,66 b' IPython controller'
78 ------------------
81 ------------------
79
82
80 The IPython controller provides an interface for working with a set of
83 The IPython controller provides an interface for working with a set of
81 engines. At an general level, the controller is a process to which
84 engines. At an general level, the controller is a collection of processes to
82 IPython engines can connect. For each connected engine, the controller
85 which IPython engines can connect. For each connected engine, the controller
83 manages a queue. All actions that can be performed on the engine go
86 manages two queues. All actions that can be performed on the engine go through
84 through this queue. While the engines themselves block when user code is
87 this queue. While the engines themselves block when user code is run, the
85 run, the controller hides that from the user to provide a fully
88 controller hides that from the user to provide a fully asynchronous interface
86 asynchronous interface to a set of engines.
89 to a set of engines.
87
88 .. note::
89
90 Because the controller listens on a network port for engines to
91 connect to it, it must be started *before* any engines are started.
92
90
93 The controller also provides a single point of contact for users who wish to
91 The controller also provides a single point of contact for users who wish to
94 utilize the engines connected to the controller. There are different ways of
92 utilize the engines connected to the controller. There are different ways of
95 working with a controller. In IPython these ways correspond to different
93 working with a controller. In IPython, all of these models are implemented via
96 interfaces that the controller is adapted to. Currently we have two default
94 the client's :meth:`.Client.apply` method, with various arguments, or
97 interfaces to the controller:
95 constructing :class:`.View` objects to represent subsets of engines. The two
96 primary models for interacting with engines are:
98
97
99 * The MultiEngine interface, which provides the simplest possible way of
98 * A MUX interface, where engines are addressed explicitly.
100 working with engines interactively.
99 * A Task interface, where the Scheduler is trusted with assigning work to
101 * The Task interface, which presents the engines as a load balanced
100 appropriate engines.
102 task farming system.
103
101
104 Advanced users can easily add new custom interfaces to enable other
102 Advanced users can readily extend the View models to enable other
105 styles of parallelism.
103 styles of parallelism.
106
104
107 .. note::
105 .. note::
108
106
109 A single controller and set of engines can be accessed
107 A single controller and set of engines can be used with multiple models
110 through multiple interfaces simultaneously. This opens the
108 simultaneously. This opens the door for lots of interesting things.
111 door for lots of interesting things.
112
109
113 Controller clients
110 Controller client
114 ------------------
111 -----------------
115
112
116 For each controller interface, there is a corresponding client. These
113 There is one primary object, the :class:`~.parallel.client.Client`, for connecting to a controller. For each model, there is a corresponding view. These views allow users to interact with a set of engines through the
117 clients allow users to interact with a set of engines through the
114 interface. Here are the two default views:
118 interface. Here are the two default clients:
119
115
120 * The :class:`MultiEngineClient` class.
116 * The :class:`DirectView` class for explicit addressing.
121 * The :class:`TaskClient` class.
117 * The :class:`LoadBalancedView` class for destination-agnostic scheduling.
122
118
123 Security
119 Security
124 --------
120 --------
125
121
126 By default (as long as `pyOpenSSL` is installed) all network connections
122 IPython uses ZeroMQ for networking, which has provided many advantages, but
127 between the controller and engines and the controller and clients are secure.
123 one of the setbacks is its utter lack of security [ZeroMQ]_. By default, no IPython
128 What does this mean? First of all, all of the connections will be encrypted
124 connections are secured, but open ports only listen on localhost. The only
129 using SSL. Second, the connections are authenticated. We handle authentication
125 source of security for IPython is via ssh-tunnel. IPython supports both shell
130 in a capability based security model [Capability]_. In this model, a
126 (`openssh`) and `paramiko` based tunnels for connections.
131 "capability (known in some systems as a key) is a communicable, unforgeable
132 token of authority". Put simply, a capability is like a key to your house. If
133 you have the key to your house, you can get in. If not, you can't.
134
127
135 In our architecture, the controller is the only process that listens on
128 In our architecture, the controller is the only process that listens on
136 network ports, and is thus responsible to creating these keys. In IPython,
129 network ports, and is thus the main point of vulnerability. The standard model
137 these keys are known as Foolscap URLs, or FURLs, because of the underlying
130 for secure connections is to designate that the controller listen on
138 network protocol we are using. As a user, you don't need to know anything
131 localhost, and use ssh-tunnels on the same machine to connect clients and/or
139 about the details of these FURLs, other than that when the controller starts,
132 engines.
140 it saves a set of FURLs to files named :file:`something.furl`. The default
141 location of these files is the :file:`~./ipython/security` directory.
142
143 To connect and authenticate to the controller an engine or client simply needs
144 to present an appropriate FURL (that was originally created by the controller)
145 to the controller. Thus, the FURL files need to be copied to a location where
146 the clients and engines can find them. Typically, this is the
147 :file:`~./ipython/security` directory on the host where the client/engine is
148 running (which could be a different host than the controller). Once the FURL
149 files are copied over, everything should work fine.
150
151 Currently, there are three FURL files that the controller creates:
152
153 ipcontroller-engine.furl
154 This FURL file is the key that gives an engine the ability to connect
155 to a controller.
156
157 ipcontroller-tc.furl
158 This FURL file is the key that a :class:`TaskClient` must use to
159 connect to the task interface of a controller.
160
161 ipcontroller-mec.furl
162 This FURL file is the key that a :class:`MultiEngineClient` must use
163 to connect to the multiengine interface of a controller.
164
165 More details of how these FURL files are used are given below.
166
133
134 .. warning::
135
136 Even at its most secure, the Controller listens on ports on localhost, and
137 every time you make a tunnel, you open a localhost port on the connecting
138 machine that points to the Controller. If localhost on the Controller's
139 machine, or the machine of any client or engine, is untrusted, then your
140 Controller is insecure. There is no way around this with ZeroMQ.
141
142
143 .. TODO: edit parallelsecurity
167 A detailed description of the security model and its implementation in IPython
144 A detailed description of the security model and its implementation in IPython
168 can be found :ref:`here <parallelsecurity>`.
145 can be found :ref:`here <parallelsecurity>`.
169
146
@@ -173,10 +150,10 b' Getting Started'
173 To use IPython for parallel computing, you need to start one instance of the
150 To use IPython for parallel computing, you need to start one instance of the
174 controller and one or more instances of the engine. Initially, it is best to
151 controller and one or more instances of the engine. Initially, it is best to
175 simply start a controller and engines on a single host using the
152 simply start a controller and engines on a single host using the
176 :command:`ipcluster` command. To start a controller and 4 engines on your
153 :command:`ipclusterz` command. To start a controller and 4 engines on your
177 localhost, just do::
154 localhost, just do::
178
155
179 $ ipcluster local -n 4
156 $ ipclusterz -n 4
180
157
181 More details about starting the IPython controller and engines can be found
158 More details about starting the IPython controller and engines can be found
182 :ref:`here <parallel_process>`
159 :ref:`here <parallel_process>`
@@ -187,51 +164,27 b' everything is working correctly, try the following commands:'
187
164
188 .. sourcecode:: ipython
165 .. sourcecode:: ipython
189
166
190 In [1]: from IPython.kernel import client
167 In [1]: from IPython.zmq.parallel import client
191
168
192 In [2]: mec = client.MultiEngineClient()
169 In [2]: c = client.Client()
193
170
194 In [4]: mec.get_ids()
171 In [4]: c.ids
195 Out[4]: [0, 1, 2, 3]
172 Out[4]: set([0, 1, 2, 3])
196
173
197 In [5]: mec.execute('print "Hello World"')
174 In [5]: c.apply(lambda : "Hello, World", targets='all', block=True)
198 Out[5]:
175 Out[5]: {0: 'Hello, World', 1: 'Hello, World', 2: 'Hello, World', 3:
199 <Results List>
176 'Hello, World'}
200 [0] In [1]: print "Hello World"
201 [0] Out[1]: Hello World
202
203 [1] In [1]: print "Hello World"
204 [1] Out[1]: Hello World
205
177
206 [2] In [1]: print "Hello World"
178 Remember, a client needs to be able to see the Controller. So if the controller is on a different machine, and you have ssh access to that machine, then you would connect to it with::
207 [2] Out[1]: Hello World
208
179
209 [3] In [1]: print "Hello World"
180 .. sourcecode:: ipython
210 [3] Out[1]: Hello World
211
212 Remember, a client also needs to present a FURL file to the controller. How
213 does this happen? When a multiengine client is created with no arguments, the
214 client tries to find the corresponding FURL file in the local
215 :file:`~./ipython/security` directory. If it finds it, you are set. If you
216 have put the FURL file in a different location or it has a different name,
217 create the client like this::
218
219 mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
220
181
221 Same thing hold true of creating a task client::
182 In [2]: c = client.Client(sshserver='mycontroller.example.com')
222
183
223 tc = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
184 Where 'mycontroller.example.com' is the url or IP address of the machine on which the Controller is running.
224
185
225 You are now ready to learn more about the :ref:`MultiEngine
186 You are now ready to learn more about the :ref:`MUX
226 <parallelmultiengine>` and :ref:`Task <paralleltask>` interfaces to the
187 <parallelmultiengine>` and :ref:`Task <paralleltask>` interfaces to the
227 controller.
188 controller.
228
189
229 .. note::
190 .. [ZeroMQ] ZeroMQ. http://www.zeromq.org
230
231 Don't forget that the engine, multiengine client and task client all have
232 *different* furl files. You must move *each* of these around to an
233 appropriate location so that the engines and clients can use them to
234 connect to the controller.
235
236 .. [Capability] Capability-based security, http://en.wikipedia.org/wiki/Capability-based_security
237
This diff has been collapsed as it changes many lines, (974 lines changed) Show them Hide them
@@ -17,41 +17,44 b' Starting the IPython controller and engines'
17
17
18 To follow along with this tutorial, you will need to start the IPython
18 To follow along with this tutorial, you will need to start the IPython
19 controller and four IPython engines. The simplest way of doing this is to use
19 controller and four IPython engines. The simplest way of doing this is to use
20 the :command:`ipcluster` command::
20 the :command:`ipclusterz` command::
21
21
22 $ ipcluster local -n 4
22 $ ipclusterz -n 4
23
23
24 For more detailed information about starting the controller and engines, see
24 For more detailed information about starting the controller and engines, see
25 our :ref:`introduction <ip1par>` to using IPython for parallel computing.
25 our :ref:`introduction <ip1par>` to using IPython for parallel computing.
26
26
27 Creating a ``MultiEngineClient`` instance
27 Creating a ``Client`` instance
28 =========================================
28 ==============================
29
29
30 The first step is to import the IPython :mod:`IPython.kernel.client` module
30 The first step is to import the IPython :mod:`IPython.zmq.parallel.client`
31 and then create a :class:`MultiEngineClient` instance:
31 module and then create a :class:`.Client` instance:
32
32
33 .. sourcecode:: ipython
33 .. sourcecode:: ipython
34
34
35 In [1]: from IPython.kernel import client
35 In [1]: from IPython.zmq.parallel import client
36
36
37 In [2]: mec = client.MultiEngineClient()
37 In [2]: rc = client.Client()
38
38
39 This form assumes that the :file:`ipcontroller-mec.furl` is in the
39 This form assumes that the controller was started on localhost with default
40 :file:`~./ipython/security` directory on the client's host. If not, the
40 configuration. If not, the location of the controller must be given as an
41 location of the FURL file must be given as an argument to the
41 argument to the constructor:
42 constructor:
43
42
44 .. sourcecode:: ipython
43 .. sourcecode:: ipython
45
44
46 In [2]: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
45 # for a visible LAN controller listening on an external port:
46 In [2]: rc = client.Client('tcp://192.168.1.16:10101')
47 # for a remote controller at my.server.com listening on localhost:
48 In [3]: rc = client.Client(sshserver='my.server.com')
49
47
50
48 To make sure there are engines connected to the controller, use can get a list
51 To make sure there are engines connected to the controller, use can get a list
49 of engine ids:
52 of engine ids:
50
53
51 .. sourcecode:: ipython
54 .. sourcecode:: ipython
52
55
53 In [3]: mec.get_ids()
56 In [3]: rc.ids
54 Out[3]: [0, 1, 2, 3]
57 Out[3]: set([0, 1, 2, 3])
55
58
56 Here we see that there are four engines ready to do work for us.
59 Here we see that there are four engines ready to do work for us.
57
60
@@ -59,8 +62,8 b' Quick and easy parallelism'
59 ==========================
62 ==========================
60
63
61 In many cases, you simply want to apply a Python function to a sequence of
64 In many cases, you simply want to apply a Python function to a sequence of
62 objects, but *in parallel*. The multiengine interface provides two simple ways
65 objects, but *in parallel*. The client interface provides a simple way
63 of accomplishing this: a parallel version of :func:`map` and ``@parallel``
66 of accomplishing this: useing the builtin :func:`map` and the ``@remote``
64 function decorator.
67 function decorator.
65
68
66 Parallel map
69 Parallel map
@@ -68,213 +71,240 b' Parallel map'
68
71
69 Python's builtin :func:`map` functions allows a function to be applied to a
72 Python's builtin :func:`map` functions allows a function to be applied to a
70 sequence element-by-element. This type of code is typically trivial to
73 sequence element-by-element. This type of code is typically trivial to
71 parallelize. In fact, the multiengine interface in IPython already has a
74 parallelize. In fact, since IPython's interface is all about functions anyway, you can just use the builtin :func:`map`, or a client's :map: method:
72 parallel version of :meth:`map` that works just like its serial counterpart:
73
75
74 .. sourcecode:: ipython
76 .. sourcecode:: ipython
75
77
76 In [63]: serial_result = map(lambda x:x**10, range(32))
78 In [62]: serial_result = map(lambda x:x**10, range(32))
77
79
78 In [64]: parallel_result = mec.map(lambda x:x**10, range(32))
80 In [66]: parallel_result = rc.map(lambda x: x**10, range(32))
81
82 In [67]: serial_result==parallel_result
83 Out[67]: True
79
84
80 In [65]: serial_result==parallel_result
81 Out[65]: True
82
85
83 .. note::
86 .. note::
84
87
85 The multiengine interface version of :meth:`map` does not do any load
88 The client's own version of :meth:`map` or that of :class:`.DirectView` do
86 balancing. For a load balanced version, see the task interface.
89 not do any load balancing. For a load balanced version, use a
90 :class:`LoadBalancedView`, or a :class:`ParallelFunction` with
91 `targets=None`.
87
92
88 .. seealso::
93 .. seealso::
94
95 :meth:`map` is implemented via :class:`.ParallelFunction`.
89
96
90 The :meth:`map` method has a number of options that can be controlled by
97 Remote function decorator
91 the :meth:`mapper` method. See its docstring for more information.
98 -------------------------
92
93 Parallel function decorator
94 ---------------------------
95
99
96 Parallel functions are just like normal function, but they can be called on
100 Remote functions are just like normal functions, but when they are called,
97 sequences and *in parallel*. The multiengine interface provides a decorator
101 they execute on one or more engines, rather than locally. IPython provides
98 that turns any Python function into a parallel function:
102 some decorators:
99
103
100 .. sourcecode:: ipython
104 .. sourcecode:: ipython
101
105
102 In [10]: @mec.parallel()
106 In [10]: @rc.remote(block=True)
103 ....: def f(x):
107 ....: def f(x):
104 ....: return 10.0*x**4
108 ....: return 10.0*x**4
105 ....:
109 ....:
106
110
107 In [11]: f(range(32)) # this is done in parallel
111 In [11]: map(f, range(32)) # this is done in parallel
108 Out[11]:
112 Out[11]:
109 [0.0,10.0,160.0,...]
113 [0.0,10.0,160.0,...]
110
114
111 See the docstring for the :meth:`parallel` decorator for options.
115 See the docstring for the :func:`parallel` and :func:`remote` decorators for
116 options.
112
117
113 Running Python commands
118 Calling Python functions
114 =======================
119 ========================
115
120
116 The most basic type of operation that can be performed on the engines is to
121 The most basic type of operation that can be performed on the engines is to
117 execute Python code. Executing Python code can be done in blocking or
122 execute Python code or call Python functions. Executing Python code can be
118 non-blocking mode (blocking is default) using the :meth:`execute` method.
123 done in blocking or non-blocking mode (non-blocking is default) using the
124 :meth:`execute` method, and calling functions can be done via the
125 :meth:`.View.apply` method.
119
126
120 Blocking execution
127 Blocking execution
121 ------------------
128 ------------------
122
129
123 In blocking mode, the :class:`MultiEngineClient` object (called ``mec`` in
130 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
124 these examples) submits the command to the controller, which places the
131 these examples) submits the command to the controller, which places the
125 command in the engines' queues for execution. The :meth:`execute` call then
132 command in the engines' queues for execution. The :meth:`apply` call then
126 blocks until the engines are done executing the command:
133 blocks until the engines are done executing the command:
127
134
128 .. sourcecode:: ipython
135 .. sourcecode:: ipython
129
136
130 # The default is to run on all engines
137 In [2]: rc.block=True
131 In [4]: mec.execute('a=5')
138 In [3]: dview = rc[:] # A DirectView of all engines
132 Out[4]:
139 In [4]: dview['a'] = 5
133 <Results List>
140
134 [0] In [1]: a=5
135 [1] In [1]: a=5
136 [2] In [1]: a=5
137 [3] In [1]: a=5
138
139 In [5]: mec.execute('b=10')
140 Out[5]:
141 <Results List>
142 [0] In [2]: b=10
143 [1] In [2]: b=10
144 [2] In [2]: b=10
145 [3] In [2]: b=10
146
141
147 Python commands can be executed on specific engines by calling execute using
142 In [5]: dview['b'] = 10
148 the ``targets`` keyword argument:
149
150 .. sourcecode:: ipython
151
143
152 In [6]: mec.execute('c=a+b',targets=[0,2])
144 In [6]: dview.apply_bound(lambda x: a+b+x, 27)
153 Out[6]:
145 Out[6]: {0: 42, 1: 42, 2: 42, 3: 42}
154 <Results List>
155 [0] In [3]: c=a+b
156 [2] In [3]: c=a+b
157
146
147 Python commands can be executed on specific engines by calling execute using
148 the ``targets`` keyword argument, or creating a :class:`DirectView` instance
149 by index-access to the client:
158
150
159 In [7]: mec.execute('c=a-b',targets=[1,3])
151 .. sourcecode:: ipython
160 Out[7]:
161 <Results List>
162 [1] In [3]: c=a-b
163 [3] In [3]: c=a-b
164
152
153 In [6]: rc.execute('c=a+b',targets=[0,2])
165
154
166 In [8]: mec.execute('print c')
155 In [7]: rc.execute('c=a-b',targets=[1,3])
167 Out[8]:
168 <Results List>
169 [0] In [4]: print c
170 [0] Out[4]: 15
171
156
172 [1] In [4]: print c
157 In [8]: rc[:]['c']
173 [1] Out[4]: -5
158 Out[8]: {0: 15, 1: -5, 2: 15, 3: -5}
174
159
175 [2] In [4]: print c
160 .. note::
176 [2] Out[4]: 15
177
161
178 [3] In [4]: print c
162 Note that every call to ``rc.<meth>(...,targets=x)`` can be made via
179 [3] Out[4]: -5
163 ``rc[<x>].<meth>(...)``, which constructs a View object. The only place
164 where this differs in in :meth:`apply`. The :class:`Client` takes many
165 arguments to apply, so it requires `args` and `kwargs` to be passed as
166 individual arguments. Extended options such as `bound`,`targets`, and
167 `block` are controlled by the attributes of the :class:`View` objects, so
168 they can provide the much more convenient
169 :meth:`View.apply(f,*args,**kwargs)`, which simply calls
170 ``f(*args,**kwargs)`` remotely.
180
171
181 This example also shows one of the most important things about the IPython
172 This example also shows one of the most important things about the IPython
182 engines: they have a persistent user namespaces. The :meth:`execute` method
173 engines: they have a persistent user namespaces. The :meth:`apply` method can
183 returns a Python ``dict`` that contains useful information:
174 be run in either a bound or unbound way. The default for a View is to be
175 unbound, unless called by the :meth:`apply_bound` method:
184
176
185 .. sourcecode:: ipython
177 .. sourcecode:: ipython
186
178
187 In [9]: result_dict = mec.execute('d=10; print d')
179 In [9]: rc[:]['b'] = 5 # assign b to 5 everywhere
180
181 In [10]: v0 = rc[0]
182
183 In [12]: v0.apply_bound(lambda : b)
184 Out[12]: 5
185
186 In [13]: v0.apply(lambda : b)
187 ---------------------------------------------------------------------------
188 RemoteError Traceback (most recent call last)
189 /home/you/<ipython-input-34-21a468eb10f0> in <module>()
190 ----> 1 v0.apply(lambda : b)
191 ...
192 RemoteError: NameError(global name 'b' is not defined)
193 Traceback (most recent call last):
194 File "/Users/minrk/dev/ip/mine/IPython/zmq/parallel/streamkernel.py", line 294, in apply_request
195 exec code in working, working
196 File "<string>", line 1, in <module>
197 File "<ipython-input-34-21a468eb10f0>", line 1, in <lambda>
198 NameError: global name 'b' is not defined
199
200
201 Specifically, `bound=True` specifies that the engine's namespace is to be used
202 for execution, and `bound=False` specifies that the engine's namespace is not
203 to be used (hence, 'b' is undefined during unbound execution, since the
204 function is called in an empty namespace). Unbound execution is often useful
205 for large numbers of atomic tasks, which prevents bloating the engine's
206 memory, while bound execution lets you build on your previous work.
188
207
189 In [10]: for r in result_dict:
190 ....: print r
191 ....:
192 ....:
193 {'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 0, 'stdout': '10\n'}
194 {'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 1, 'stdout': '10\n'}
195 {'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 2, 'stdout': '10\n'}
196 {'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 3, 'stdout': '10\n'}
197
208
198 Non-blocking execution
209 Non-blocking execution
199 ----------------------
210 ----------------------
200
211
201 In non-blocking mode, :meth:`execute` submits the command to be executed and
212 In non-blocking mode, :meth:`apply` submits the command to be executed and
202 then returns a :class:`PendingResult` object immediately. The
213 then returns a :class:`AsyncResult` object immediately. The
203 :class:`PendingResult` object gives you a way of getting a result at a later
214 :class:`AsyncResult` object gives you a way of getting a result at a later
204 time through its :meth:`get_result` method or :attr:`r` attribute. This allows
215 time through its :meth:`get` method.
205 you to quickly submit long running commands without blocking your local
206 Python/IPython session:
207
216
208 .. sourcecode:: ipython
217 .. Note::
218
219 The :class:`AsyncResult` object provides the exact same interface as
220 :py:class:`multiprocessing.pool.AsyncResult`. See the
221 `official Python documentation <http://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_
222 for more.
209
223
210 # In blocking mode
224
211 In [6]: mec.execute('import time')
225 This allows you to quickly submit long running commands without blocking your
212 Out[6]:
226 local Python/IPython session:
213 <Results List>
227
214 [0] In [1]: import time
228 .. sourcecode:: ipython
215 [1] In [1]: import time
229
216 [2] In [1]: import time
230 # define our function
217 [3] In [1]: import time
231 In [35]: def wait(t):
218
232 ....: import time
219 # In non-blocking mode
233 ....: tic = time.time()
220 In [7]: pr = mec.execute('time.sleep(10)',block=False)
234 ....: time.sleep(t)
221
235 ....: return time.time()-tic
222 # Now block for the result
236
223 In [8]: pr.get_result()
237 # In blocking mode
224 Out[8]:
238 In [6]: rc.apply('import time')
225 <Results List>
239
226 [0] In [2]: time.sleep(10)
240 # In non-blocking mode
227 [1] In [2]: time.sleep(10)
241 In [7]: pr = rc[:].apply_async(wait, 2)
228 [2] In [2]: time.sleep(10)
242
229 [3] In [2]: time.sleep(10)
243 # Now block for the result
230
244 In [8]: pr.get()
231 # Again in non-blocking mode
245 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
232 In [9]: pr = mec.execute('time.sleep(10)',block=False)
246
233
247 # Again in non-blocking mode
234 # Poll to see if the result is ready
248 In [9]: pr = rc[:].apply_async(wait, 10)
235 In [10]: pr.get_result(block=False)
249
236
250 # Poll to see if the result is ready
237 # A shorthand for get_result(block=True)
251 In [10]: pr.ready()
238 In [11]: pr.r
252 Out[10]: False
239 Out[11]:
253
240 <Results List>
254 # ask for the result, but wait a maximum of 1 second:
241 [0] In [3]: time.sleep(10)
255 In [45]: pr.get(1)
242 [1] In [3]: time.sleep(10)
256 ---------------------------------------------------------------------------
243 [2] In [3]: time.sleep(10)
257 TimeoutError Traceback (most recent call last)
244 [3] In [3]: time.sleep(10)
258 /home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
245
259 ----> 1 pr.get(1)
246 Often, it is desirable to wait until a set of :class:`PendingResult` objects
260
261 /path/to/site-packages/IPython/zmq/parallel/asyncresult.pyc in get(self, timeout)
262 62 raise self._exception
263 63 else:
264 ---> 64 raise error.TimeoutError("Result not ready.")
265 65
266 66 def ready(self):
267
268 TimeoutError: Result not ready.
269
270 .. Note::
271
272 Note the import inside the function. This is a common model, to ensure
273 that the appropriate modules are imported where the task is run.
274
275 Often, it is desirable to wait until a set of :class:`AsyncResult` objects
247 are done. For this, there is a the method :meth:`barrier`. This method takes a
276 are done. For this, there is a the method :meth:`barrier`. This method takes a
248 tuple of :class:`PendingResult` objects and blocks until all of the associated
277 tuple of :class:`AsyncResult` objects (or `msg_ids`) and blocks until all of the associated
249 results are ready:
278 results are ready:
250
279
251 .. sourcecode:: ipython
280 .. sourcecode:: ipython
252
281
253 In [72]: mec.block=False
282 In [72]: rc.block=False
254
283
255 # A trivial list of PendingResults objects
284 # A trivial list of AsyncResults objects
256 In [73]: pr_list = [mec.execute('time.sleep(3)') for i in range(10)]
285 In [73]: pr_list = [rc[:].apply_async(wait, 3) for i in range(10)]
257
286
258 # Wait until all of them are done
287 # Wait until all of them are done
259 In [74]: mec.barrier(pr_list)
288 In [74]: rc.barrier(pr_list)
260
289
261 # Then, their results are ready using get_result or the r attribute
290 # Then, their results are ready using get_result or the r attribute
262 In [75]: pr_list[0].r
291 In [75]: pr_list[0].get()
263 Out[75]:
292 Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
264 <Results List>
293
265 [0] In [20]: time.sleep(3)
266 [1] In [19]: time.sleep(3)
267 [2] In [20]: time.sleep(3)
268 [3] In [19]: time.sleep(3)
269
294
270
295
271 The ``block`` and ``targets`` keyword arguments and attributes
296 The ``block`` and ``targets`` keyword arguments and attributes
272 --------------------------------------------------------------
297 --------------------------------------------------------------
273
298
274 Most methods in the multiengine interface (like :meth:`execute`) accept
299 .. warning::
300
301 This is different now, I haven't updated this section.
302 -MinRK
303
304 Most methods(like :meth:`apply`) accept
275 ``block`` and ``targets`` as keyword arguments. As we have seen above, these
305 ``block`` and ``targets`` as keyword arguments. As we have seen above, these
276 keyword arguments control the blocking mode and which engines the command is
306 keyword arguments control the blocking mode and which engines the command is
277 applied to. The :class:`MultiEngineClient` class also has :attr:`block` and
307 applied to. The :class:`Client` class also has :attr:`block` and
278 :attr:`targets` attributes that control the default behavior when the keyword
308 :attr:`targets` attributes that control the default behavior when the keyword
279 arguments are not provided. Thus the following logic is used for :attr:`block`
309 arguments are not provided. Thus the following logic is used for :attr:`block`
280 and :attr:`targets`:
310 and :attr:`targets`:
@@ -286,37 +316,37 b' The following examples demonstrate how to use the instance attributes:'
286
316
287 .. sourcecode:: ipython
317 .. sourcecode:: ipython
288
318
289 In [16]: mec.targets = [0,2]
319 In [16]: rc.targets = [0,2]
290
320
291 In [17]: mec.block = False
321 In [17]: rc.block = False
292
322
293 In [18]: pr = mec.execute('a=5')
323 In [18]: pr = rc.execute('a=5')
294
324
295 In [19]: pr.r
325 In [19]: pr.r
296 Out[19]:
326 Out[19]:
297 <Results List>
327 <Results List>
298 [0] In [6]: a=5
328 [0] In [6]: a=5
299 [2] In [6]: a=5
329 [2] In [6]: a=5
300
330
301 # Note targets='all' means all engines
331 # Note targets='all' means all engines
302 In [20]: mec.targets = 'all'
332 In [20]: rc.targets = 'all'
303
333
304 In [21]: mec.block = True
334 In [21]: rc.block = True
305
335
306 In [22]: mec.execute('b=10; print b')
336 In [22]: rc.execute('b=10; print b')
307 Out[22]:
337 Out[22]:
308 <Results List>
338 <Results List>
309 [0] In [7]: b=10; print b
339 [0] In [7]: b=10; print b
310 [0] Out[7]: 10
340 [0] Out[7]: 10
311
341
312 [1] In [6]: b=10; print b
342 [1] In [6]: b=10; print b
313 [1] Out[6]: 10
343 [1] Out[6]: 10
314
344
315 [2] In [7]: b=10; print b
345 [2] In [7]: b=10; print b
316 [2] Out[7]: 10
346 [2] Out[7]: 10
317
347
318 [3] In [6]: b=10; print b
348 [3] In [6]: b=10; print b
319 [3] Out[6]: 10
349 [3] Out[6]: 10
320
350
321 The :attr:`block` and :attr:`targets` instance attributes also determine the
351 The :attr:`block` and :attr:`targets` instance attributes also determine the
322 behavior of the parallel magic commands.
352 behavior of the parallel magic commands.
@@ -325,6 +355,12 b' behavior of the parallel magic commands.'
325 Parallel magic commands
355 Parallel magic commands
326 -----------------------
356 -----------------------
327
357
358 .. warning::
359
360 The magics have not been changed to work with the zeromq system. ``%px``
361 and ``%autopx`` do work, but ``%result`` does not. %px and %autopx *do
362 not* print stdin/out.
363
328 We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
364 We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
329 that make it more pleasant to execute Python commands on the engines
365 that make it more pleasant to execute Python commands on the engines
330 interactively. These are simply shortcuts to :meth:`execute` and
366 interactively. These are simply shortcuts to :meth:`execute` and
@@ -334,48 +370,35 b' engines specified by the :attr:`targets` attribute of the'
334
370
335 .. sourcecode:: ipython
371 .. sourcecode:: ipython
336
372
337 # Make this MultiEngineClient active for parallel magic commands
373 # Create a DirectView for all targets
338 In [23]: mec.activate()
374 In [22]: dv = rc[:]
339
375
340 In [24]: mec.block=True
376 # Make this DirectView active for parallel magic commands
377 In [23]: dv.activate()
341
378
342 In [25]: import numpy
379 In [24]: dv.block=True
343
380
344 In [26]: %px import numpy
381 In [25]: import numpy
345 Executing command on Controller
346 Out[26]:
347 <Results List>
348 [0] In [8]: import numpy
349 [1] In [7]: import numpy
350 [2] In [8]: import numpy
351 [3] In [7]: import numpy
352
382
383 In [26]: %px import numpy
384 Parallel execution on engines: [0, 1, 2, 3]
385 Out[26]:{0: None, 1: None, 2: None, 3: None}
353
386
354 In [27]: %px a = numpy.random.rand(2,2)
387 In [27]: %px a = numpy.random.rand(2,2)
355 Executing command on Controller
388 Parallel execution on engines: [0, 1, 2, 3]
356 Out[27]:
357 <Results List>
358 [0] In [9]: a = numpy.random.rand(2,2)
359 [1] In [8]: a = numpy.random.rand(2,2)
360 [2] In [9]: a = numpy.random.rand(2,2)
361 [3] In [8]: a = numpy.random.rand(2,2)
362
389
390 In [28]: %px ev = numpy.linalg.eigvals(a)
391 Parallel execution on engines: [0, 1, 2, 3]
363
392
364 In [28]: %px print numpy.linalg.eigvals(a)
393 In [28]: dv['ev']
365 Executing command on Controller
394 Out[44]: {0: array([ 1.09522024, -0.09645227]),
366 Out[28]:
395 1: array([ 1.21435496, -0.35546712]),
367 <Results List>
396 2: array([ 0.72180653, 0.07133042]),
368 [0] In [10]: print numpy.linalg.eigvals(a)
397 3: array([ 1.46384341e+00, 1.04353244e-04])}
369 [0] Out[10]: [ 1.28167017 0.14197338]
370
398
371 [1] In [9]: print numpy.linalg.eigvals(a)
399 .. Note::
372 [1] Out[9]: [-0.14093616 1.27877273]
373
400
374 [2] In [10]: print numpy.linalg.eigvals(a)
401 ``%result`` doesn't work
375 [2] Out[10]: [-0.37023573 1.06779409]
376
377 [3] In [9]: print numpy.linalg.eigvals(a)
378 [3] Out[9]: [ 0.83664764 -0.25602658]
379
402
380 The ``%result`` magic gets and prints the stdin/stdout/stderr of the last
403 The ``%result`` magic gets and prints the stdin/stdout/stderr of the last
381 command executed on each engine. It is simply a shortcut to the
404 command executed on each engine. It is simply a shortcut to the
@@ -383,73 +406,72 b' command executed on each engine. It is simply a shortcut to the'
383
406
384 .. sourcecode:: ipython
407 .. sourcecode:: ipython
385
408
386 In [29]: %result
409 In [29]: %result
387 Out[29]:
410 Out[29]:
388 <Results List>
411 <Results List>
389 [0] In [10]: print numpy.linalg.eigvals(a)
412 [0] In [10]: print numpy.linalg.eigvals(a)
390 [0] Out[10]: [ 1.28167017 0.14197338]
413 [0] Out[10]: [ 1.28167017 0.14197338]
391
414
392 [1] In [9]: print numpy.linalg.eigvals(a)
415 [1] In [9]: print numpy.linalg.eigvals(a)
393 [1] Out[9]: [-0.14093616 1.27877273]
416 [1] Out[9]: [-0.14093616 1.27877273]
394
417
395 [2] In [10]: print numpy.linalg.eigvals(a)
418 [2] In [10]: print numpy.linalg.eigvals(a)
396 [2] Out[10]: [-0.37023573 1.06779409]
419 [2] Out[10]: [-0.37023573 1.06779409]
397
420
398 [3] In [9]: print numpy.linalg.eigvals(a)
421 [3] In [9]: print numpy.linalg.eigvals(a)
399 [3] Out[9]: [ 0.83664764 -0.25602658]
422 [3] Out[9]: [ 0.83664764 -0.25602658]
400
423
401 The ``%autopx`` magic switches to a mode where everything you type is executed
424 The ``%autopx`` magic switches to a mode where everything you type is executed
402 on the engines given by the :attr:`targets` attribute:
425 on the engines given by the :attr:`targets` attribute:
403
426
404 .. sourcecode:: ipython
427 .. sourcecode:: ipython
405
428
406 In [30]: mec.block=False
429 In [30]: dv.block=False
407
430
408 In [31]: %autopx
431 In [31]: %autopx
409 Auto Parallel Enabled
432 Auto Parallel Enabled
410 Type %autopx to disable
433 Type %autopx to disable
411
434
412 In [32]: max_evals = []
435 In [32]: max_evals = []
413 <IPython.kernel.multiengineclient.PendingResult object at 0x17b8a70>
436 <IPython.zmq.parallel.asyncresult.AsyncResult object at 0x17b8a70>
414
437
415 In [33]: for i in range(100):
438 In [33]: for i in range(100):
416 ....: a = numpy.random.rand(10,10)
439 ....: a = numpy.random.rand(10,10)
417 ....: a = a+a.transpose()
440 ....: a = a+a.transpose()
418 ....: evals = numpy.linalg.eigvals(a)
441 ....: evals = numpy.linalg.eigvals(a)
419 ....: max_evals.append(evals[0].real)
442 ....: max_evals.append(evals[0].real)
420 ....:
443 ....:
421 ....:
444 ....:
422 <IPython.kernel.multiengineclient.PendingResult object at 0x17af8f0>
445 <IPython.zmq.parallel.asyncresult.AsyncResult object at 0x17af8f0>
423
446
424 In [34]: %autopx
447 In [34]: %autopx
425 Auto Parallel Disabled
448 Auto Parallel Disabled
426
449
427 In [35]: mec.block=True
450 In [35]: dv.block=True
428
451
429 In [36]: px print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
452 In [36]: px ans= "Average max eigenvalue is: %f"%(sum(max_evals)/len(max_evals))
430 Executing command on Controller
453 Parallel execution on engines: [0, 1, 2, 3]
431 Out[36]:
454
432 <Results List>
455 In [37]: dv['ans']
433 [0] In [13]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
456 Out[37]: {0 : 'Average max eigenvalue is: 10.1387247332',
434 [0] Out[13]: Average max eigenvalue is: 10.1387247332
457 1 : 'Average max eigenvalue is: 10.2076902286',
458 2 : 'Average max eigenvalue is: 10.1891484655',
459 3 : 'Average max eigenvalue is: 10.1158837784',}
435
460
436 [1] In [12]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
437 [1] Out[12]: Average max eigenvalue is: 10.2076902286
438
461
439 [2] In [13]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
462 .. Note::
440 [2] Out[13]: Average max eigenvalue is: 10.1891484655
441
463
442 [3] In [12]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
464 Multiline ``%autpx`` gets fouled up by NameErrors, because IPython
443 [3] Out[12]: Average max eigenvalue is: 10.1158837784
465 currently introspects too much.
444
466
445
467
446 Moving Python objects around
468 Moving Python objects around
447 ============================
469 ============================
448
470
449 In addition to executing code on engines, you can transfer Python objects to
471 In addition to calling functions and executing code on engines, you can
450 and from your IPython session and the engines. In IPython, these operations
472 transfer Python objects to and from your IPython session and the engines. In
451 are called :meth:`push` (sending an object to the engines) and :meth:`pull`
473 IPython, these operations are called :meth:`push` (sending an object to the
452 (getting an object from the engines).
474 engines) and :meth:`pull` (getting an object from the engines).
453
475
454 Basic push and pull
476 Basic push and pull
455 -------------------
477 -------------------
@@ -458,112 +480,57 b' Here are some examples of how you use :meth:`push` and :meth:`pull`:'
458
480
459 .. sourcecode:: ipython
481 .. sourcecode:: ipython
460
482
461 In [38]: mec.push(dict(a=1.03234,b=3453))
483 In [38]: rc.push(dict(a=1.03234,b=3453))
462 Out[38]: [None, None, None, None]
484 Out[38]: {0: None, 1: None, 2: None, 3: None}
463
464 In [39]: mec.pull('a')
465 Out[39]: [1.03234, 1.03234, 1.03234, 1.03234]
466
485
467 In [40]: mec.pull('b',targets=0)
486 In [39]: rc.pull('a')
468 Out[40]: [3453]
487 Out[39]: {0: 1.03234, 1: 1.03234, 2: 1.03234, 3: 1.03234}
469
488
470 In [41]: mec.pull(('a','b'))
489 In [40]: rc.pull('b',targets=0)
471 Out[41]: [[1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453]]
490 Out[40]: 3453
472
491
473 In [42]: mec.zip_pull(('a','b'))
492 In [41]: rc.pull(('a','b'))
474 Out[42]: [(1.03234, 1.03234, 1.03234, 1.03234), (3453, 3453, 3453, 3453)]
493 Out[41]: {0: [1.03234, 3453], 1: [1.03234, 3453], 2: [1.03234, 3453], 3:[1.03234, 3453]}
494
495 # zmq client does not have zip_pull
496 In [42]: rc.zip_pull(('a','b'))
497 Out[42]: [(1.03234, 1.03234, 1.03234, 1.03234), (3453, 3453, 3453, 3453)]
475
498
476 In [43]: mec.push(dict(c='speed'))
499 In [43]: rc.push(dict(c='speed'))
477 Out[43]: [None, None, None, None]
500 Out[43]: {0: None, 1: None, 2: None, 3: None}
478
479 In [44]: %px print c
480 Executing command on Controller
481 Out[44]:
482 <Results List>
483 [0] In [14]: print c
484 [0] Out[14]: speed
485
486 [1] In [13]: print c
487 [1] Out[13]: speed
488
489 [2] In [14]: print c
490 [2] Out[14]: speed
491
492 [3] In [13]: print c
493 [3] Out[13]: speed
494
501
495 In non-blocking mode :meth:`push` and :meth:`pull` also return
502 In non-blocking mode :meth:`push` and :meth:`pull` also return
496 :class:`PendingResult` objects:
503 :class:`AsyncResult` objects:
497
504
498 .. sourcecode:: ipython
505 .. sourcecode:: ipython
499
506
500 In [47]: mec.block=False
507 In [47]: rc.block=False
501
502 In [48]: pr = mec.pull('a')
503
504 In [49]: pr.r
505 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
506
507
508 Push and pull for functions
509 ---------------------------
510
511 Functions can also be pushed and pulled using :meth:`push_function` and
512 :meth:`pull_function`:
513
514 .. sourcecode:: ipython
515
508
516 In [52]: mec.block=True
509 In [48]: pr = rc.pull('a')
517
510
518 In [53]: def f(x):
511 In [49]: pr.get()
519 ....: return 2.0*x**4
512 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
520 ....:
521
513
522 In [54]: mec.push_function(dict(f=f))
523 Out[54]: [None, None, None, None]
524
514
525 In [55]: mec.execute('y = f(4.0)')
526 Out[55]:
527 <Results List>
528 [0] In [15]: y = f(4.0)
529 [1] In [14]: y = f(4.0)
530 [2] In [15]: y = f(4.0)
531 [3] In [14]: y = f(4.0)
532
533
534 In [56]: px print y
535 Executing command on Controller
536 Out[56]:
537 <Results List>
538 [0] In [16]: print y
539 [0] Out[16]: 512.0
540
541 [1] In [15]: print y
542 [1] Out[15]: 512.0
543
544 [2] In [16]: print y
545 [2] Out[16]: 512.0
546
547 [3] In [15]: print y
548 [3] Out[15]: 512.0
549
515
550
516
551 Dictionary interface
517 Dictionary interface
552 --------------------
518 --------------------
553
519
554 As a shorthand to :meth:`push` and :meth:`pull`, the
520 Since a namespace is just a :class:`dict`, :class:`DirectView` objects provide
555 :class:`MultiEngineClient` class implements some of the Python dictionary
521 dictionary-style access by key and methods such as :meth:`get` and
556 interface. This make the remote namespaces of the engines appear as a local
522 :meth:`update` for convenience. This make the remote namespaces of the engines
557 dictionary. Underneath, this uses :meth:`push` and :meth:`pull`:
523 appear as a local dictionary. Underneath, this uses :meth:`push` and
524 :meth:`pull`:
558
525
559 .. sourcecode:: ipython
526 .. sourcecode:: ipython
560
527
561 In [50]: mec.block=True
528 In [50]: rc.block=True
562
529
563 In [51]: mec['a']=['foo','bar']
530 In [51]: rc[:]['a']=['foo','bar']
564
531
565 In [52]: mec['a']
532 In [52]: rc[:]['a']
566 Out[52]: [['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar']]
533 Out[52]: {0: ['foo', 'bar'], 1: ['foo', 'bar'], 2: ['foo', 'bar'], 3: ['foo', 'bar']}
567
534
568 Scatter and gather
535 Scatter and gather
569 ------------------
536 ------------------
@@ -571,35 +538,24 b' Scatter and gather'
571 Sometimes it is useful to partition a sequence and push the partitions to
538 Sometimes it is useful to partition a sequence and push the partitions to
572 different engines. In MPI language, this is know as scatter/gather and we
539 different engines. In MPI language, this is know as scatter/gather and we
573 follow that terminology. However, it is important to remember that in
540 follow that terminology. However, it is important to remember that in
574 IPython's :class:`MultiEngineClient` class, :meth:`scatter` is from the
541 IPython's :class:`Client` class, :meth:`scatter` is from the
575 interactive IPython session to the engines and :meth:`gather` is from the
542 interactive IPython session to the engines and :meth:`gather` is from the
576 engines back to the interactive IPython session. For scatter/gather operations
543 engines back to the interactive IPython session. For scatter/gather operations
577 between engines, MPI should be used:
544 between engines, MPI should be used:
578
545
579 .. sourcecode:: ipython
546 .. sourcecode:: ipython
580
547
581 In [58]: mec.scatter('a',range(16))
548 In [58]: rc.scatter('a',range(16))
582 Out[58]: [None, None, None, None]
549 Out[58]: {0: None, 1: None, 2: None, 3: None}
583
550
584 In [59]: px print a
551 In [59]: rc[:]['a']
585 Executing command on Controller
552 Out[59]: {0: [0, 1, 2, 3],
586 Out[59]:
553 1: [4, 5, 6, 7],
587 <Results List>
554 2: [8, 9, 10, 11],
588 [0] In [17]: print a
555 3: [12, 13, 14, 15]}
589 [0] Out[17]: [0, 1, 2, 3]
590
556
591 [1] In [16]: print a
557 In [60]: rc.gather('a')
592 [1] Out[16]: [4, 5, 6, 7]
558 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
593
594 [2] In [17]: print a
595 [2] Out[17]: [8, 9, 10, 11]
596
597 [3] In [16]: print a
598 [3] Out[16]: [12, 13, 14, 15]
599
600
601 In [60]: mec.gather('a')
602 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
603
559
604 Other things to look at
560 Other things to look at
605 =======================
561 =======================
@@ -613,23 +569,17 b' basic effect using :meth:`scatter` and :meth:`gather`:'
613
569
614 .. sourcecode:: ipython
570 .. sourcecode:: ipython
615
571
616 In [66]: mec.scatter('x',range(64))
572 In [66]: rc.scatter('x',range(64))
617 Out[66]: [None, None, None, None]
573 Out[66]: {0: None, 1: None, 2: None, 3: None}
618
619 In [67]: px y = [i**10 for i in x]
620 Executing command on Controller
621 Out[67]:
622 <Results List>
623 [0] In [19]: y = [i**10 for i in x]
624 [1] In [18]: y = [i**10 for i in x]
625 [2] In [19]: y = [i**10 for i in x]
626 [3] In [18]: y = [i**10 for i in x]
627
574
575 In [67]: px y = [i**10 for i in x]
576 Executing command on Controller
577 Out[67]:
628
578
629 In [68]: y = mec.gather('y')
579 In [68]: y = rc.gather('y')
630
580
631 In [69]: print y
581 In [69]: print y
632 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
582 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
633
583
634 Parallel exceptions
584 Parallel exceptions
635 -------------------
585 -------------------
@@ -644,33 +594,33 b' more other types of exceptions. Here is how it works:'
644
594
645 .. sourcecode:: ipython
595 .. sourcecode:: ipython
646
596
647 In [76]: mec.block=True
597 In [76]: rc.block=True
648
598
649 In [77]: mec.execute('1/0')
599 In [77]: rc.execute('1/0')
650 ---------------------------------------------------------------------------
600 ---------------------------------------------------------------------------
651 CompositeError Traceback (most recent call last)
601 CompositeError Traceback (most recent call last)
652
602
653 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
603 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
654
604
655 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
605 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
656 432 targets, block = self._findTargetsAndBlock(targets, block)
606 432 targets, block = self._findTargetsAndBlock(targets, block)
657 433 result = blockingCallFromThread(self.smultiengine.execute, lines,
607 433 result = blockingCallFromThread(self.smultiengine.execute, lines,
658 --> 434 targets=targets, block=block)
608 --> 434 targets=targets, block=block)
659 435 if block:
609 435 if block:
660 436 result = ResultList(result)
610 436 result = ResultList(result)
661
611
662 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
612 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
663 72 result.raiseException()
613 72 result.raiseException()
664 73 except Exception, e:
614 73 except Exception, e:
665 ---> 74 raise e
615 ---> 74 raise e
666 75 return result
616 75 return result
667 76
617 76
668
618
669 CompositeError: one or more exceptions from call to method: execute
619 CompositeError: one or more exceptions from call to method: execute
670 [0:execute]: ZeroDivisionError: integer division or modulo by zero
620 [0:execute]: ZeroDivisionError: integer division or modulo by zero
671 [1:execute]: ZeroDivisionError: integer division or modulo by zero
621 [1:execute]: ZeroDivisionError: integer division or modulo by zero
672 [2:execute]: ZeroDivisionError: integer division or modulo by zero
622 [2:execute]: ZeroDivisionError: integer division or modulo by zero
673 [3:execute]: ZeroDivisionError: integer division or modulo by zero
623 [3:execute]: ZeroDivisionError: integer division or modulo by zero
674
624
675 Notice how the error message printed when :exc:`CompositeError` is raised has
625 Notice how the error message printed when :exc:`CompositeError` is raised has
676 information about the individual exceptions that were raised on each engine.
626 information about the individual exceptions that were raised on each engine.
@@ -678,25 +628,25 b' If you want, you can even raise one of these original exceptions:'
678
628
679 .. sourcecode:: ipython
629 .. sourcecode:: ipython
680
630
681 In [80]: try:
631 In [80]: try:
682 ....: mec.execute('1/0')
632 ....: rc.execute('1/0')
683 ....: except client.CompositeError, e:
633 ....: except client.CompositeError, e:
684 ....: e.raise_exception()
634 ....: e.raise_exception()
685 ....:
635 ....:
686 ....:
636 ....:
687 ---------------------------------------------------------------------------
637 ---------------------------------------------------------------------------
688 ZeroDivisionError Traceback (most recent call last)
638 ZeroDivisionError Traceback (most recent call last)
689
639
690 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
640 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
691
641
692 /ipython1-client-r3021/ipython1/kernel/error.pyc in raise_exception(self, excid)
642 /ipython1-client-r3021/ipython1/kernel/error.pyc in raise_exception(self, excid)
693 156 raise IndexError("an exception with index %i does not exist"%excid)
643 156 raise IndexError("an exception with index %i does not exist"%excid)
694 157 else:
644 157 else:
695 --> 158 raise et, ev, etb
645 --> 158 raise et, ev, etb
696 159
646 159
697 160 def collect_exceptions(rlist, method):
647 160 def collect_exceptions(rlist, method):
698
648
699 ZeroDivisionError: integer division or modulo by zero
649 ZeroDivisionError: integer division or modulo by zero
700
650
701 If you are working in IPython, you can simple type ``%debug`` after one of
651 If you are working in IPython, you can simple type ``%debug`` after one of
702 these :exc:`CompositeError` exceptions is raised, and inspect the exception
652 these :exc:`CompositeError` exceptions is raised, and inspect the exception
@@ -704,80 +654,80 b' instance:'
704
654
705 .. sourcecode:: ipython
655 .. sourcecode:: ipython
706
656
707 In [81]: mec.execute('1/0')
657 In [81]: rc.execute('1/0')
708 ---------------------------------------------------------------------------
658 ---------------------------------------------------------------------------
709 CompositeError Traceback (most recent call last)
659 CompositeError Traceback (most recent call last)
710
660
711 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
661 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
712
662
713 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
663 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
714 432 targets, block = self._findTargetsAndBlock(targets, block)
664 432 targets, block = self._findTargetsAndBlock(targets, block)
715 433 result = blockingCallFromThread(self.smultiengine.execute, lines,
665 433 result = blockingCallFromThread(self.smultiengine.execute, lines,
716 --> 434 targets=targets, block=block)
666 --> 434 targets=targets, block=block)
717 435 if block:
667 435 if block:
718 436 result = ResultList(result)
668 436 result = ResultList(result)
719
669
720 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
670 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
721 72 result.raiseException()
671 72 result.raiseException()
722 73 except Exception, e:
672 73 except Exception, e:
723 ---> 74 raise e
673 ---> 74 raise e
724 75 return result
674 75 return result
725 76
675 76
726
676
727 CompositeError: one or more exceptions from call to method: execute
677 CompositeError: one or more exceptions from call to method: execute
728 [0:execute]: ZeroDivisionError: integer division or modulo by zero
678 [0:execute]: ZeroDivisionError: integer division or modulo by zero
729 [1:execute]: ZeroDivisionError: integer division or modulo by zero
679 [1:execute]: ZeroDivisionError: integer division or modulo by zero
730 [2:execute]: ZeroDivisionError: integer division or modulo by zero
680 [2:execute]: ZeroDivisionError: integer division or modulo by zero
731 [3:execute]: ZeroDivisionError: integer division or modulo by zero
681 [3:execute]: ZeroDivisionError: integer division or modulo by zero
732
682
733 In [82]: %debug
683 In [82]: %debug
734 >
684 >
735
685
736 /ipython1-client-r3021/ipython1/kernel/twistedutil.py(74)blockingCallFromThread()
686 /ipython1-client-r3021/ipython1/kernel/twistedutil.py(74)blockingCallFromThread()
737 73 except Exception, e:
687 73 except Exception, e:
738 ---> 74 raise e
688 ---> 74 raise e
739 75 return result
689 75 return result
740
690
741 # With the debugger running, e is the exceptions instance. We can tab complete
691 # With the debugger running, e is the exceptions instance. We can tab complete
742 # on it and see the extra methods that are available.
692 # on it and see the extra methods that are available.
743 ipdb> e.
693 ipdb> e.
744 e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args
694 e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args
745 e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist
695 e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist
746 e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message
696 e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message
747 e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks
697 e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks
748 e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception
698 e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception
749 ipdb> e.print_tracebacks()
699 ipdb> e.print_tracebacks()
750 [0:execute]:
700 [0:execute]:
751 ---------------------------------------------------------------------------
701 ---------------------------------------------------------------------------
752 ZeroDivisionError Traceback (most recent call last)
702 ZeroDivisionError Traceback (most recent call last)
753
703
754 /ipython1-client-r3021/docs/examples/<string> in <module>()
704 /ipython1-client-r3021/docs/examples/<string> in <module>()
755
705
756 ZeroDivisionError: integer division or modulo by zero
706 ZeroDivisionError: integer division or modulo by zero
757
707
758 [1:execute]:
708 [1:execute]:
759 ---------------------------------------------------------------------------
709 ---------------------------------------------------------------------------
760 ZeroDivisionError Traceback (most recent call last)
710 ZeroDivisionError Traceback (most recent call last)
761
711
762 /ipython1-client-r3021/docs/examples/<string> in <module>()
712 /ipython1-client-r3021/docs/examples/<string> in <module>()
763
713
764 ZeroDivisionError: integer division or modulo by zero
714 ZeroDivisionError: integer division or modulo by zero
765
715
766 [2:execute]:
716 [2:execute]:
767 ---------------------------------------------------------------------------
717 ---------------------------------------------------------------------------
768 ZeroDivisionError Traceback (most recent call last)
718 ZeroDivisionError Traceback (most recent call last)
769
719
770 /ipython1-client-r3021/docs/examples/<string> in <module>()
720 /ipython1-client-r3021/docs/examples/<string> in <module>()
771
721
772 ZeroDivisionError: integer division or modulo by zero
722 ZeroDivisionError: integer division or modulo by zero
773
723
774 [3:execute]:
724 [3:execute]:
775 ---------------------------------------------------------------------------
725 ---------------------------------------------------------------------------
776 ZeroDivisionError Traceback (most recent call last)
726 ZeroDivisionError Traceback (most recent call last)
777
727
778 /ipython1-client-r3021/docs/examples/<string> in <module>()
728 /ipython1-client-r3021/docs/examples/<string> in <module>()
779
729
780 ZeroDivisionError: integer division or modulo by zero
730 ZeroDivisionError: integer division or modulo by zero
781
731
782 .. note::
732 .. note::
783
733
@@ -788,48 +738,48 b' All of this same error handling magic even works in non-blocking mode:'
788
738
789 .. sourcecode:: ipython
739 .. sourcecode:: ipython
790
740
791 In [83]: mec.block=False
741 In [83]: rc.block=False
792
742
793 In [84]: pr = mec.execute('1/0')
743 In [84]: pr = rc.execute('1/0')
794
744
795 In [85]: pr.r
745 In [85]: pr.get()
796 ---------------------------------------------------------------------------
746 ---------------------------------------------------------------------------
797 CompositeError Traceback (most recent call last)
747 CompositeError Traceback (most recent call last)
798
748
799 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
749 /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
800
750
801 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in _get_r(self)
751 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in _get_r(self)
802 170
752 170
803 171 def _get_r(self):
753 171 def _get_r(self):
804 --> 172 return self.get_result(block=True)
754 --> 172 return self.get_result(block=True)
805 173
755 173
806 174 r = property(_get_r)
756 174 r = property(_get_r)
807
757
808 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_result(self, default, block)
758 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_result(self, default, block)
809 131 return self.result
759 131 return self.result
810 132 try:
760 132 try:
811 --> 133 result = self.client.get_pending_deferred(self.result_id, block)
761 --> 133 result = self.client.get_pending_deferred(self.result_id, block)
812 134 except error.ResultNotCompleted:
762 134 except error.ResultNotCompleted:
813 135 return default
763 135 return default
814
764
815 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_pending_deferred(self, deferredID, block)
765 /ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_pending_deferred(self, deferredID, block)
816 385
766 385
817 386 def get_pending_deferred(self, deferredID, block):
767 386 def get_pending_deferred(self, deferredID, block):
818 --> 387 return blockingCallFromThread(self.smultiengine.get_pending_deferred, deferredID, block)
768 --> 387 return blockingCallFromThread(self.smultiengine.get_pending_deferred, deferredID, block)
819 388
769 388
820 389 def barrier(self, pendingResults):
770 389 def barrier(self, pendingResults):
821
771
822 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
772 /ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
823 72 result.raiseException()
773 72 result.raiseException()
824 73 except Exception, e:
774 73 except Exception, e:
825 ---> 74 raise e
775 ---> 74 raise e
826 75 return result
776 75 return result
827 76
777 76
828
778
829 CompositeError: one or more exceptions from call to method: execute
779 CompositeError: one or more exceptions from call to method: execute
830 [0:execute]: ZeroDivisionError: integer division or modulo by zero
780 [0:execute]: ZeroDivisionError: integer division or modulo by zero
831 [1:execute]: ZeroDivisionError: integer division or modulo by zero
781 [1:execute]: ZeroDivisionError: integer division or modulo by zero
832 [2:execute]: ZeroDivisionError: integer division or modulo by zero
782 [2:execute]: ZeroDivisionError: integer division or modulo by zero
833 [3:execute]: ZeroDivisionError: integer division or modulo by zero
783 [3:execute]: ZeroDivisionError: integer division or modulo by zero
834
784
835
785
@@ -5,15 +5,16 b' The IPython task interface'
5 ==========================
5 ==========================
6
6
7 The task interface to the controller presents the engines as a fault tolerant,
7 The task interface to the controller presents the engines as a fault tolerant,
8 dynamic load-balanced system or workers. Unlike the multiengine interface, in
8 dynamic load-balanced system of workers. Unlike the multiengine interface, in
9 the task interface, the user have no direct access to individual engines. In
9 the task interface, the user have no direct access to individual engines. By
10 some ways, this interface is simpler, but in other ways it is more powerful.
10 allowing the IPython scheduler to assign work, this interface is both simpler
11 and more powerful.
11
12
12 Best of all the user can use both of these interfaces running at the same time
13 Best of all the user can use both of these interfaces running at the same time
13 to take advantage or both of their strengths. When the user can break up the
14 to take advantage of their respective strengths. When the user can break up
14 user's work into segments that do not depend on previous execution, the task
15 the user's work into segments that do not depend on previous execution, the
15 interface is ideal. But it also has more power and flexibility, allowing the
16 task interface is ideal. But it also has more power and flexibility, allowing
16 user to guide the distribution of jobs, without having to assign tasks to
17 the user to guide the distribution of jobs, without having to assign tasks to
17 engines explicitly.
18 engines explicitly.
18
19
19 Starting the IPython controller and engines
20 Starting the IPython controller and engines
@@ -21,56 +22,65 b' Starting the IPython controller and engines'
21
22
22 To follow along with this tutorial, you will need to start the IPython
23 To follow along with this tutorial, you will need to start the IPython
23 controller and four IPython engines. The simplest way of doing this is to use
24 controller and four IPython engines. The simplest way of doing this is to use
24 the :command:`ipcluster` command::
25 the :command:`ipclusterz` command::
25
26
26 $ ipcluster local -n 4
27 $ ipclusterz -n 4
27
28
28 For more detailed information about starting the controller and engines, see
29 For more detailed information about starting the controller and engines, see
29 our :ref:`introduction <ip1par>` to using IPython for parallel computing.
30 our :ref:`introduction <ip1par>` to using IPython for parallel computing.
30
31
31 Creating a ``TaskClient`` instance
32 Creating a ``Client`` instance
32 =========================================
33 ==============================
33
34
34 The first step is to import the IPython :mod:`IPython.kernel.client` module
35 The first step is to import the IPython :mod:`IPython.zmq.parallel.client`
35 and then create a :class:`TaskClient` instance:
36 module and then create a :class:`.Client` instance:
36
37
37 .. sourcecode:: ipython
38 .. sourcecode:: ipython
38
39
39 In [1]: from IPython.kernel import client
40 In [1]: from IPython.zmq.parallel import client
40
41
41 In [2]: tc = client.TaskClient()
42 In [2]: rc = client.Client()
43
44 In [3]: lview = rc[None]
45 Out[3]: <LoadBalancedView tcp://127.0.0.1:10101>
46
42
47
43 This form assumes that the :file:`ipcontroller-tc.furl` is in the
48 This form assumes that the controller was started on localhost with default
44 :file:`~./ipython/security` directory on the client's host. If not, the
49 configuration. If not, the location of the controller must be given as an
45 location of the FURL file must be given as an argument to the
50 argument to the constructor:
46 constructor:
47
51
48 .. sourcecode:: ipython
52 .. sourcecode:: ipython
49
53
50 In [2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
54 # for a visible LAN controller listening on an external port:
55 In [2]: rc = client.Client('tcp://192.168.1.16:10101')
56 # for a remote controller at my.server.com listening on localhost:
57 In [3]: rc = client.Client(sshserver='my.server.com')
58
59
51
60
52 Quick and easy parallelism
61 Quick and easy parallelism
53 ==========================
62 ==========================
54
63
55 In many cases, you simply want to apply a Python function to a sequence of
64 In many cases, you simply want to apply a Python function to a sequence of
56 objects, but *in parallel*. Like the multiengine interface, the task interface
65 objects, but *in parallel*. Like the multiengine interface, these can be
57 provides two simple ways of accomplishing this: a parallel version of
66 implemented via the task interface. The exact same tools can perform these
58 :func:`map` and ``@parallel`` function decorator. However, the verions in the
67 actions in load-balanced ways as well as multiplexed ways: a parallel version
59 task interface have one important difference: they are dynamically load
68 of :func:`map` and :func:`@parallel` function decorator. If one specifies the
60 balanced. Thus, if the execution time per item varies significantly, you
69 argument `targets=None`, then they are dynamically load balanced. Thus, if the
61 should use the versions in the task interface.
70 execution time per item varies significantly, you should use the versions in
71 the task interface.
62
72
63 Parallel map
73 Parallel map
64 ------------
74 ------------
65
75
66 The parallel :meth:`map` in the task interface is similar to that in the
76 To load-balance :meth:`map`,simply use a LoadBalancedView, created by asking
67 multiengine interface:
77 for the ``None`` element:
68
78
69 .. sourcecode:: ipython
79 .. sourcecode:: ipython
70
80
71 In [63]: serial_result = map(lambda x:x**10, range(32))
81 In [63]: serial_result = map(lambda x:x**10, range(32))
72
82
73 In [64]: parallel_result = tc.map(lambda x:x**10, range(32))
83 In [64]: parallel_result = tc[None].map(lambda x:x**10, range(32))
74
84
75 In [65]: serial_result==parallel_result
85 In [65]: serial_result==parallel_result
76 Out[65]: True
86 Out[65]: True
@@ -84,38 +94,39 b' that turns any Python function into a parallel function:'
84
94
85 .. sourcecode:: ipython
95 .. sourcecode:: ipython
86
96
87 In [10]: @tc.parallel()
97 In [10]: @lview.parallel()
88 ....: def f(x):
98 ....: def f(x):
89 ....: return 10.0*x**4
99 ....: return 10.0*x**4
90 ....:
100 ....:
91
101
92 In [11]: f(range(32)) # this is done in parallel
102 In [11]: f.map(range(32)) # this is done in parallel
93 Out[11]:
103 Out[11]: [0.0,10.0,160.0,...]
94 [0.0,10.0,160.0,...]
95
104
96 More details
105 More details
97 ============
106 ============
98
107
99 The :class:`TaskClient` has many more powerful features that allow quite a bit
108 The :class:`Client` has many more powerful features that allow quite a bit
100 of flexibility in how tasks are defined and run. The next places to look are
109 of flexibility in how tasks are defined and run. The next places to look are
101 in the following classes:
110 in the following classes:
102
111
103 * :class:`IPython.kernel.client.TaskClient`
112 * :class:`IPython.zmq.parallel.client.Client`
104 * :class:`IPython.kernel.client.StringTask`
113 * :class:`IPython.zmq.parallel.client.AsyncResult`
105 * :class:`IPython.kernel.client.MapTask`
114 * :meth:`IPython.zmq.parallel.client.Client.apply`
115 * :mod:`IPython.zmq.parallel.dependency`
106
116
107 The following is an overview of how to use these classes together:
117 The following is an overview of how to use these classes together:
108
118
109 1. Create a :class:`TaskClient`.
119 1. Create a :class:`Client`.
110 2. Create one or more instances of :class:`StringTask` or :class:`MapTask`
120 2. Define some functions to be run as tasks
111 to define your tasks.
121 3. Submit your tasks to using the :meth:`apply` method of your
112 3. Submit your tasks to using the :meth:`run` method of your
122 :class:`Client` instance, specifying `targets=None`. This signals
113 :class:`TaskClient` instance.
123 the :class:`Client` to entrust the Scheduler with assigning tasks to engines.
114 4. Use :meth:`TaskClient.get_task_result` to get the results of the
124 4. Use :meth:`Client.get_results` to get the results of the
115 tasks.
125 tasks, or use the :meth:`AsyncResult.get` method of the results to wait
126 for and then receive the results.
116
127
117 We are in the process of developing more detailed information about the task
128 We are in the process of developing more detailed information about the task
118 interface. For now, the docstrings of the :class:`TaskClient`,
129 interface. For now, the docstrings of the :meth:`Client.apply`,
119 :class:`StringTask` and :class:`MapTask` classes should be consulted.
130 and :func:`depend` methods should be consulted.
120
131
121
132
General Comments 0
You need to be logged in to leave comments. Login now