##// END OF EJS Templates
document CompositeError.tb_limit
MinRK -
Show More
@@ -1,694 +1,697 b''
1 .. _parallel_multiengine:
1 .. _parallel_multiengine:
2
2
3 ==========================
3 ==========================
4 IPython's Direct interface
4 IPython's Direct interface
5 ==========================
5 ==========================
6
6
7 The direct, or multiengine, interface represents one possible way of working with a set of
7 The direct, or multiengine, interface represents one possible way of working with a set of
8 IPython engines. The basic idea behind the multiengine interface is that the
8 IPython engines. The basic idea behind the multiengine interface is that the
9 capabilities of each engine are directly and explicitly exposed to the user.
9 capabilities of each engine are directly and explicitly exposed to the user.
10 Thus, in the multiengine interface, each engine is given an id that is used to
10 Thus, in the multiengine interface, each engine is given an id that is used to
11 identify the engine and give it work to do. This interface is very intuitive
11 identify the engine and give it work to do. This interface is very intuitive
12 and is designed with interactive usage in mind, and is the best place for
12 and is designed with interactive usage in mind, and is the best place for
13 new users of IPython to begin.
13 new users of IPython to begin.
14
14
15 Starting the IPython controller and engines
15 Starting the IPython controller and engines
16 ===========================================
16 ===========================================
17
17
18 To follow along with this tutorial, you will need to start the IPython
18 To follow along with this tutorial, you will need to start the IPython
19 controller and four IPython engines. The simplest way of doing this is to use
19 controller and four IPython engines. The simplest way of doing this is to use
20 the :command:`ipcluster` command::
20 the :command:`ipcluster` command::
21
21
22 $ ipcluster start -n 4
22 $ ipcluster start -n 4
23
23
24 For more detailed information about starting the controller and engines, see
24 For more detailed information about starting the controller and engines, see
25 our :ref:`introduction <parallel_overview>` to using IPython for parallel computing.
25 our :ref:`introduction <parallel_overview>` to using IPython for parallel computing.
26
26
27 Creating a ``DirectView`` instance
27 Creating a ``DirectView`` instance
28 ==================================
28 ==================================
29
29
30 The first step is to import the IPython :mod:`IPython.parallel`
30 The first step is to import the IPython :mod:`IPython.parallel`
31 module and then create a :class:`.Client` instance:
31 module and then create a :class:`.Client` instance:
32
32
33 .. sourcecode:: ipython
33 .. sourcecode:: ipython
34
34
35 In [1]: from IPython.parallel import Client
35 In [1]: from IPython.parallel import Client
36
36
37 In [2]: rc = Client()
37 In [2]: rc = Client()
38
38
39 This form assumes that the default connection information (stored in
39 This form assumes that the default connection information (stored in
40 :file:`ipcontroller-client.json` found in :file:`IPYTHONDIR/profile_default/security`) is
40 :file:`ipcontroller-client.json` found in :file:`IPYTHONDIR/profile_default/security`) is
41 accurate. If the controller was started on a remote machine, you must copy that connection
41 accurate. If the controller was started on a remote machine, you must copy that connection
42 file to the client machine, or enter its contents as arguments to the Client constructor:
42 file to the client machine, or enter its contents as arguments to the Client constructor:
43
43
44 .. sourcecode:: ipython
44 .. sourcecode:: ipython
45
45
46 # If you have copied the json connector file from the controller:
46 # If you have copied the json connector file from the controller:
47 In [2]: rc = Client('/path/to/ipcontroller-client.json')
47 In [2]: rc = Client('/path/to/ipcontroller-client.json')
48 # or to connect with a specific profile you have set up:
48 # or to connect with a specific profile you have set up:
49 In [3]: rc = Client(profile='mpi')
49 In [3]: rc = Client(profile='mpi')
50
50
51
51
52 To make sure there are engines connected to the controller, users can get a list
52 To make sure there are engines connected to the controller, users can get a list
53 of engine ids:
53 of engine ids:
54
54
55 .. sourcecode:: ipython
55 .. sourcecode:: ipython
56
56
57 In [3]: rc.ids
57 In [3]: rc.ids
58 Out[3]: [0, 1, 2, 3]
58 Out[3]: [0, 1, 2, 3]
59
59
60 Here we see that there are four engines ready to do work for us.
60 Here we see that there are four engines ready to do work for us.
61
61
62 For direct execution, we will make use of a :class:`DirectView` object, which can be
62 For direct execution, we will make use of a :class:`DirectView` object, which can be
63 constructed via list-access to the client:
63 constructed via list-access to the client:
64
64
65 .. sourcecode:: ipython
65 .. sourcecode:: ipython
66
66
67 In [4]: dview = rc[:] # use all engines
67 In [4]: dview = rc[:] # use all engines
68
68
69 .. seealso::
69 .. seealso::
70
70
71 For more information, see the in-depth explanation of :ref:`Views <parallel_details>`.
71 For more information, see the in-depth explanation of :ref:`Views <parallel_details>`.
72
72
73
73
74 Quick and easy parallelism
74 Quick and easy parallelism
75 ==========================
75 ==========================
76
76
77 In many cases, you simply want to apply a Python function to a sequence of
77 In many cases, you simply want to apply a Python function to a sequence of
78 objects, but *in parallel*. The client interface provides a simple way
78 objects, but *in parallel*. The client interface provides a simple way
79 of accomplishing this: using the DirectView's :meth:`~DirectView.map` method.
79 of accomplishing this: using the DirectView's :meth:`~DirectView.map` method.
80
80
81 Parallel map
81 Parallel map
82 ------------
82 ------------
83
83
84 Python's builtin :func:`map` functions allows a function to be applied to a
84 Python's builtin :func:`map` functions allows a function to be applied to a
85 sequence element-by-element. This type of code is typically trivial to
85 sequence element-by-element. This type of code is typically trivial to
86 parallelize. In fact, since IPython's interface is all about functions anyway,
86 parallelize. In fact, since IPython's interface is all about functions anyway,
87 you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a
87 you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a
88 DirectView's :meth:`map` method:
88 DirectView's :meth:`map` method:
89
89
90 .. sourcecode:: ipython
90 .. sourcecode:: ipython
91
91
92 In [62]: serial_result = map(lambda x:x**10, range(32))
92 In [62]: serial_result = map(lambda x:x**10, range(32))
93
93
94 In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32))
94 In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32))
95
95
96 In [67]: serial_result==parallel_result
96 In [67]: serial_result==parallel_result
97 Out[67]: True
97 Out[67]: True
98
98
99
99
100 .. note::
100 .. note::
101
101
102 The :class:`DirectView`'s version of :meth:`map` does
102 The :class:`DirectView`'s version of :meth:`map` does
103 not do dynamic load balancing. For a load balanced version, use a
103 not do dynamic load balancing. For a load balanced version, use a
104 :class:`LoadBalancedView`.
104 :class:`LoadBalancedView`.
105
105
106 .. seealso::
106 .. seealso::
107
107
108 :meth:`map` is implemented via :class:`ParallelFunction`.
108 :meth:`map` is implemented via :class:`ParallelFunction`.
109
109
110 Remote function decorators
110 Remote function decorators
111 --------------------------
111 --------------------------
112
112
113 Remote functions are just like normal functions, but when they are called,
113 Remote functions are just like normal functions, but when they are called,
114 they execute on one or more engines, rather than locally. IPython provides
114 they execute on one or more engines, rather than locally. IPython provides
115 two decorators:
115 two decorators:
116
116
117 .. sourcecode:: ipython
117 .. sourcecode:: ipython
118
118
119 In [10]: @dview.remote(block=True)
119 In [10]: @dview.remote(block=True)
120 ....: def getpid():
120 ....: def getpid():
121 ....: import os
121 ....: import os
122 ....: return os.getpid()
122 ....: return os.getpid()
123 ....:
123 ....:
124
124
125 In [11]: getpid()
125 In [11]: getpid()
126 Out[11]: [12345, 12346, 12347, 12348]
126 Out[11]: [12345, 12346, 12347, 12348]
127
127
128 The ``@parallel`` decorator creates parallel functions, that break up an element-wise
128 The ``@parallel`` decorator creates parallel functions, that break up an element-wise
129 operations and distribute them, reconstructing the result.
129 operations and distribute them, reconstructing the result.
130
130
131 .. sourcecode:: ipython
131 .. sourcecode:: ipython
132
132
133 In [12]: import numpy as np
133 In [12]: import numpy as np
134
134
135 In [13]: A = np.random.random((64,48))
135 In [13]: A = np.random.random((64,48))
136
136
137 In [14]: @dview.parallel(block=True)
137 In [14]: @dview.parallel(block=True)
138 ....: def pmul(A,B):
138 ....: def pmul(A,B):
139 ....: return A*B
139 ....: return A*B
140
140
141 In [15]: C_local = A*A
141 In [15]: C_local = A*A
142
142
143 In [16]: C_remote = pmul(A,A)
143 In [16]: C_remote = pmul(A,A)
144
144
145 In [17]: (C_local == C_remote).all()
145 In [17]: (C_local == C_remote).all()
146 Out[17]: True
146 Out[17]: True
147
147
148 Calling a ``@parallel`` function *does not* correspond to map. It is used for splitting
148 Calling a ``@parallel`` function *does not* correspond to map. It is used for splitting
149 element-wise operations that operate on a sequence or array. For ``map`` behavior,
149 element-wise operations that operate on a sequence or array. For ``map`` behavior,
150 parallel functions do have a map method.
150 parallel functions do have a map method.
151
151
152 ==================== ============================ =============================
152 ==================== ============================ =============================
153 call pfunc(seq) pfunc.map(seq)
153 call pfunc(seq) pfunc.map(seq)
154 ==================== ============================ =============================
154 ==================== ============================ =============================
155 # of tasks # of engines (1 per engine) # of engines (1 per engine)
155 # of tasks # of engines (1 per engine) # of engines (1 per engine)
156 # of remote calls # of engines (1 per engine) ``len(seq)``
156 # of remote calls # of engines (1 per engine) ``len(seq)``
157 argument to remote ``seq[i:j]`` (sub-sequence) ``seq[i]`` (single element)
157 argument to remote ``seq[i:j]`` (sub-sequence) ``seq[i]`` (single element)
158 ==================== ============================ =============================
158 ==================== ============================ =============================
159
159
160 A quick example to illustrate the difference in arguments for the two modes:
160 A quick example to illustrate the difference in arguments for the two modes:
161
161
162 .. sourcecode:: ipython
162 .. sourcecode:: ipython
163
163
164 In [16]: @dview.parallel(block=True)
164 In [16]: @dview.parallel(block=True)
165 ....: def echo(x):
165 ....: def echo(x):
166 ....: return str(x)
166 ....: return str(x)
167 ....:
167 ....:
168
168
169 In [17]: echo(range(5))
169 In [17]: echo(range(5))
170 Out[17]: ['[0, 1]', '[2]', '[3]', '[4]']
170 Out[17]: ['[0, 1]', '[2]', '[3]', '[4]']
171
171
172 In [18]: echo.map(range(5))
172 In [18]: echo.map(range(5))
173 Out[18]: ['0', '1', '2', '3', '4']
173 Out[18]: ['0', '1', '2', '3', '4']
174
174
175
175
176 .. seealso::
176 .. seealso::
177
177
178 See the :func:`~.remotefunction.parallel` and :func:`~.remotefunction.remote`
178 See the :func:`~.remotefunction.parallel` and :func:`~.remotefunction.remote`
179 decorators for options.
179 decorators for options.
180
180
181 Calling Python functions
181 Calling Python functions
182 ========================
182 ========================
183
183
184 The most basic type of operation that can be performed on the engines is to
184 The most basic type of operation that can be performed on the engines is to
185 execute Python code or call Python functions. Executing Python code can be
185 execute Python code or call Python functions. Executing Python code can be
186 done in blocking or non-blocking mode (non-blocking is default) using the
186 done in blocking or non-blocking mode (non-blocking is default) using the
187 :meth:`.View.execute` method, and calling functions can be done via the
187 :meth:`.View.execute` method, and calling functions can be done via the
188 :meth:`.View.apply` method.
188 :meth:`.View.apply` method.
189
189
190 apply
190 apply
191 -----
191 -----
192
192
193 The main method for doing remote execution (in fact, all methods that
193 The main method for doing remote execution (in fact, all methods that
194 communicate with the engines are built on top of it), is :meth:`View.apply`.
194 communicate with the engines are built on top of it), is :meth:`View.apply`.
195
195
196 We strive to provide the cleanest interface we can, so `apply` has the following
196 We strive to provide the cleanest interface we can, so `apply` has the following
197 signature:
197 signature:
198
198
199 .. sourcecode:: python
199 .. sourcecode:: python
200
200
201 view.apply(f, *args, **kwargs)
201 view.apply(f, *args, **kwargs)
202
202
203 There are various ways to call functions with IPython, and these flags are set as
203 There are various ways to call functions with IPython, and these flags are set as
204 attributes of the View. The ``DirectView`` has just two of these flags:
204 attributes of the View. The ``DirectView`` has just two of these flags:
205
205
206 dv.block : bool
206 dv.block : bool
207 whether to wait for the result, or return an :class:`AsyncResult` object
207 whether to wait for the result, or return an :class:`AsyncResult` object
208 immediately
208 immediately
209 dv.track : bool
209 dv.track : bool
210 whether to instruct pyzmq to track when zeromq is done sending the message.
210 whether to instruct pyzmq to track when zeromq is done sending the message.
211 This is primarily useful for non-copying sends of numpy arrays that you plan to
211 This is primarily useful for non-copying sends of numpy arrays that you plan to
212 edit in-place. You need to know when it becomes safe to edit the buffer
212 edit in-place. You need to know when it becomes safe to edit the buffer
213 without corrupting the message.
213 without corrupting the message.
214 dv.targets : int, list of ints
214 dv.targets : int, list of ints
215 which targets this view is associated with.
215 which targets this view is associated with.
216
216
217
217
218 Creating a view is simple: index-access on a client creates a :class:`.DirectView`.
218 Creating a view is simple: index-access on a client creates a :class:`.DirectView`.
219
219
220 .. sourcecode:: ipython
220 .. sourcecode:: ipython
221
221
222 In [4]: view = rc[1:3]
222 In [4]: view = rc[1:3]
223 Out[4]: <DirectView [1, 2]>
223 Out[4]: <DirectView [1, 2]>
224
224
225 In [5]: view.apply<tab>
225 In [5]: view.apply<tab>
226 view.apply view.apply_async view.apply_sync
226 view.apply view.apply_async view.apply_sync
227
227
228 For convenience, you can set block temporarily for a single call with the extra sync/async methods.
228 For convenience, you can set block temporarily for a single call with the extra sync/async methods.
229
229
230 Blocking execution
230 Blocking execution
231 ------------------
231 ------------------
232
232
233 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
233 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
234 these examples) submits the command to the controller, which places the
234 these examples) submits the command to the controller, which places the
235 command in the engines' queues for execution. The :meth:`apply` call then
235 command in the engines' queues for execution. The :meth:`apply` call then
236 blocks until the engines are done executing the command:
236 blocks until the engines are done executing the command:
237
237
238 .. sourcecode:: ipython
238 .. sourcecode:: ipython
239
239
240 In [2]: dview = rc[:] # A DirectView of all engines
240 In [2]: dview = rc[:] # A DirectView of all engines
241 In [3]: dview.block=True
241 In [3]: dview.block=True
242 In [4]: dview['a'] = 5
242 In [4]: dview['a'] = 5
243
243
244 In [5]: dview['b'] = 10
244 In [5]: dview['b'] = 10
245
245
246 In [6]: dview.apply(lambda x: a+b+x, 27)
246 In [6]: dview.apply(lambda x: a+b+x, 27)
247 Out[6]: [42, 42, 42, 42]
247 Out[6]: [42, 42, 42, 42]
248
248
249 You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync`
249 You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync`
250 method:
250 method:
251
251
252 .. sourcecode:: ipython
252 .. sourcecode:: ipython
253
253
254 In [7]: dview.block=False
254 In [7]: dview.block=False
255
255
256 In [8]: dview.apply_sync(lambda x: a+b+x, 27)
256 In [8]: dview.apply_sync(lambda x: a+b+x, 27)
257 Out[8]: [42, 42, 42, 42]
257 Out[8]: [42, 42, 42, 42]
258
258
259 Python commands can be executed as strings on specific engines by using a View's ``execute``
259 Python commands can be executed as strings on specific engines by using a View's ``execute``
260 method:
260 method:
261
261
262 .. sourcecode:: ipython
262 .. sourcecode:: ipython
263
263
264 In [6]: rc[::2].execute('c=a+b')
264 In [6]: rc[::2].execute('c=a+b')
265
265
266 In [7]: rc[1::2].execute('c=a-b')
266 In [7]: rc[1::2].execute('c=a-b')
267
267
268 In [8]: dview['c'] # shorthand for dview.pull('c', block=True)
268 In [8]: dview['c'] # shorthand for dview.pull('c', block=True)
269 Out[8]: [15, -5, 15, -5]
269 Out[8]: [15, -5, 15, -5]
270
270
271
271
272 Non-blocking execution
272 Non-blocking execution
273 ----------------------
273 ----------------------
274
274
275 In non-blocking mode, :meth:`apply` submits the command to be executed and
275 In non-blocking mode, :meth:`apply` submits the command to be executed and
276 then returns a :class:`AsyncResult` object immediately. The
276 then returns a :class:`AsyncResult` object immediately. The
277 :class:`AsyncResult` object gives you a way of getting a result at a later
277 :class:`AsyncResult` object gives you a way of getting a result at a later
278 time through its :meth:`get` method.
278 time through its :meth:`get` method.
279
279
280 .. seealso::
280 .. seealso::
281
281
282 Docs on the :ref:`AsyncResult <parallel_asyncresult>` object.
282 Docs on the :ref:`AsyncResult <parallel_asyncresult>` object.
283
283
284 This allows you to quickly submit long running commands without blocking your
284 This allows you to quickly submit long running commands without blocking your
285 local Python/IPython session:
285 local Python/IPython session:
286
286
287 .. sourcecode:: ipython
287 .. sourcecode:: ipython
288
288
289 # define our function
289 # define our function
290 In [6]: def wait(t):
290 In [6]: def wait(t):
291 ....: import time
291 ....: import time
292 ....: tic = time.time()
292 ....: tic = time.time()
293 ....: time.sleep(t)
293 ....: time.sleep(t)
294 ....: return time.time()-tic
294 ....: return time.time()-tic
295
295
296 # In non-blocking mode
296 # In non-blocking mode
297 In [7]: ar = dview.apply_async(wait, 2)
297 In [7]: ar = dview.apply_async(wait, 2)
298
298
299 # Now block for the result
299 # Now block for the result
300 In [8]: ar.get()
300 In [8]: ar.get()
301 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
301 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
302
302
303 # Again in non-blocking mode
303 # Again in non-blocking mode
304 In [9]: ar = dview.apply_async(wait, 10)
304 In [9]: ar = dview.apply_async(wait, 10)
305
305
306 # Poll to see if the result is ready
306 # Poll to see if the result is ready
307 In [10]: ar.ready()
307 In [10]: ar.ready()
308 Out[10]: False
308 Out[10]: False
309
309
310 # ask for the result, but wait a maximum of 1 second:
310 # ask for the result, but wait a maximum of 1 second:
311 In [45]: ar.get(1)
311 In [45]: ar.get(1)
312 ---------------------------------------------------------------------------
312 ---------------------------------------------------------------------------
313 TimeoutError Traceback (most recent call last)
313 TimeoutError Traceback (most recent call last)
314 /home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
314 /home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
315 ----> 1 ar.get(1)
315 ----> 1 ar.get(1)
316
316
317 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
317 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
318 62 raise self._exception
318 62 raise self._exception
319 63 else:
319 63 else:
320 ---> 64 raise error.TimeoutError("Result not ready.")
320 ---> 64 raise error.TimeoutError("Result not ready.")
321 65
321 65
322 66 def ready(self):
322 66 def ready(self):
323
323
324 TimeoutError: Result not ready.
324 TimeoutError: Result not ready.
325
325
326 .. Note::
326 .. Note::
327
327
328 Note the import inside the function. This is a common model, to ensure
328 Note the import inside the function. This is a common model, to ensure
329 that the appropriate modules are imported where the task is run. You can
329 that the appropriate modules are imported where the task is run. You can
330 also manually import modules into the engine(s) namespace(s) via
330 also manually import modules into the engine(s) namespace(s) via
331 :meth:`view.execute('import numpy')`.
331 :meth:`view.execute('import numpy')`.
332
332
333 Often, it is desirable to wait until a set of :class:`AsyncResult` objects
333 Often, it is desirable to wait until a set of :class:`AsyncResult` objects
334 are done. For this, there is a the method :meth:`wait`. This method takes a
334 are done. For this, there is a the method :meth:`wait`. This method takes a
335 tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History),
335 tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History),
336 and blocks until all of the associated results are ready:
336 and blocks until all of the associated results are ready:
337
337
338 .. sourcecode:: ipython
338 .. sourcecode:: ipython
339
339
340 In [72]: dview.block=False
340 In [72]: dview.block=False
341
341
342 # A trivial list of AsyncResults objects
342 # A trivial list of AsyncResults objects
343 In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)]
343 In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)]
344
344
345 # Wait until all of them are done
345 # Wait until all of them are done
346 In [74]: dview.wait(pr_list)
346 In [74]: dview.wait(pr_list)
347
347
348 # Then, their results are ready using get() or the `.r` attribute
348 # Then, their results are ready using get() or the `.r` attribute
349 In [75]: pr_list[0].get()
349 In [75]: pr_list[0].get()
350 Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
350 Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
351
351
352
352
353
353
354 The ``block`` and ``targets`` keyword arguments and attributes
354 The ``block`` and ``targets`` keyword arguments and attributes
355 --------------------------------------------------------------
355 --------------------------------------------------------------
356
356
357 Most DirectView methods (excluding :meth:`apply`) accept ``block`` and
357 Most DirectView methods (excluding :meth:`apply`) accept ``block`` and
358 ``targets`` as keyword arguments. As we have seen above, these keyword arguments control the
358 ``targets`` as keyword arguments. As we have seen above, these keyword arguments control the
359 blocking mode and which engines the command is applied to. The :class:`View` class also has
359 blocking mode and which engines the command is applied to. The :class:`View` class also has
360 :attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword
360 :attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword
361 arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`:
361 arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`:
362
362
363 * If no keyword argument is provided, the instance attributes are used.
363 * If no keyword argument is provided, the instance attributes are used.
364 * The Keyword arguments, if provided overrides the instance attributes for
364 * The Keyword arguments, if provided overrides the instance attributes for
365 the duration of a single call.
365 the duration of a single call.
366
366
367 The following examples demonstrate how to use the instance attributes:
367 The following examples demonstrate how to use the instance attributes:
368
368
369 .. sourcecode:: ipython
369 .. sourcecode:: ipython
370
370
371 In [16]: dview.targets = [0,2]
371 In [16]: dview.targets = [0,2]
372
372
373 In [17]: dview.block = False
373 In [17]: dview.block = False
374
374
375 In [18]: ar = dview.apply(lambda : 10)
375 In [18]: ar = dview.apply(lambda : 10)
376
376
377 In [19]: ar.get()
377 In [19]: ar.get()
378 Out[19]: [10, 10]
378 Out[19]: [10, 10]
379
379
380 In [20]: dview.targets = v.client.ids # all engines (4)
380 In [20]: dview.targets = v.client.ids # all engines (4)
381
381
382 In [21]: dview.block = True
382 In [21]: dview.block = True
383
383
384 In [22]: dview.apply(lambda : 42)
384 In [22]: dview.apply(lambda : 42)
385 Out[22]: [42, 42, 42, 42]
385 Out[22]: [42, 42, 42, 42]
386
386
387 The :attr:`block` and :attr:`targets` instance attributes of the
387 The :attr:`block` and :attr:`targets` instance attributes of the
388 :class:`.DirectView` also determine the behavior of the parallel magic commands.
388 :class:`.DirectView` also determine the behavior of the parallel magic commands.
389
389
390 .. seealso::
390 .. seealso::
391
391
392 See the documentation of the :ref:`Parallel Magics <parallel_magics>`.
392 See the documentation of the :ref:`Parallel Magics <parallel_magics>`.
393
393
394
394
395 Moving Python objects around
395 Moving Python objects around
396 ============================
396 ============================
397
397
398 In addition to calling functions and executing code on engines, you can
398 In addition to calling functions and executing code on engines, you can
399 transfer Python objects to and from your IPython session and the engines. In
399 transfer Python objects to and from your IPython session and the engines. In
400 IPython, these operations are called :meth:`push` (sending an object to the
400 IPython, these operations are called :meth:`push` (sending an object to the
401 engines) and :meth:`pull` (getting an object from the engines).
401 engines) and :meth:`pull` (getting an object from the engines).
402
402
403 Basic push and pull
403 Basic push and pull
404 -------------------
404 -------------------
405
405
406 Here are some examples of how you use :meth:`push` and :meth:`pull`:
406 Here are some examples of how you use :meth:`push` and :meth:`pull`:
407
407
408 .. sourcecode:: ipython
408 .. sourcecode:: ipython
409
409
410 In [38]: dview.push(dict(a=1.03234,b=3453))
410 In [38]: dview.push(dict(a=1.03234,b=3453))
411 Out[38]: [None,None,None,None]
411 Out[38]: [None,None,None,None]
412
412
413 In [39]: dview.pull('a')
413 In [39]: dview.pull('a')
414 Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234]
414 Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234]
415
415
416 In [40]: dview.pull('b', targets=0)
416 In [40]: dview.pull('b', targets=0)
417 Out[40]: 3453
417 Out[40]: 3453
418
418
419 In [41]: dview.pull(('a','b'))
419 In [41]: dview.pull(('a','b'))
420 Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ]
420 Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ]
421
421
422 In [42]: dview.push(dict(c='speed'))
422 In [42]: dview.push(dict(c='speed'))
423 Out[42]: [None,None,None,None]
423 Out[42]: [None,None,None,None]
424
424
425 In non-blocking mode :meth:`push` and :meth:`pull` also return
425 In non-blocking mode :meth:`push` and :meth:`pull` also return
426 :class:`AsyncResult` objects:
426 :class:`AsyncResult` objects:
427
427
428 .. sourcecode:: ipython
428 .. sourcecode:: ipython
429
429
430 In [48]: ar = dview.pull('a', block=False)
430 In [48]: ar = dview.pull('a', block=False)
431
431
432 In [49]: ar.get()
432 In [49]: ar.get()
433 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
433 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
434
434
435
435
436 Dictionary interface
436 Dictionary interface
437 --------------------
437 --------------------
438
438
439 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
439 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
440 dictionary-style access by key and methods such as :meth:`get` and
440 dictionary-style access by key and methods such as :meth:`get` and
441 :meth:`update` for convenience. This make the remote namespaces of the engines
441 :meth:`update` for convenience. This make the remote namespaces of the engines
442 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
442 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
443
443
444 .. sourcecode:: ipython
444 .. sourcecode:: ipython
445
445
446 In [51]: dview['a']=['foo','bar']
446 In [51]: dview['a']=['foo','bar']
447
447
448 In [52]: dview['a']
448 In [52]: dview['a']
449 Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
449 Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
450
450
451 Scatter and gather
451 Scatter and gather
452 ------------------
452 ------------------
453
453
454 Sometimes it is useful to partition a sequence and push the partitions to
454 Sometimes it is useful to partition a sequence and push the partitions to
455 different engines. In MPI language, this is know as scatter/gather and we
455 different engines. In MPI language, this is know as scatter/gather and we
456 follow that terminology. However, it is important to remember that in
456 follow that terminology. However, it is important to remember that in
457 IPython's :class:`Client` class, :meth:`scatter` is from the
457 IPython's :class:`Client` class, :meth:`scatter` is from the
458 interactive IPython session to the engines and :meth:`gather` is from the
458 interactive IPython session to the engines and :meth:`gather` is from the
459 engines back to the interactive IPython session. For scatter/gather operations
459 engines back to the interactive IPython session. For scatter/gather operations
460 between engines, MPI, pyzmq, or some other direct interconnect should be used.
460 between engines, MPI, pyzmq, or some other direct interconnect should be used.
461
461
462 .. sourcecode:: ipython
462 .. sourcecode:: ipython
463
463
464 In [58]: dview.scatter('a',range(16))
464 In [58]: dview.scatter('a',range(16))
465 Out[58]: [None,None,None,None]
465 Out[58]: [None,None,None,None]
466
466
467 In [59]: dview['a']
467 In [59]: dview['a']
468 Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
468 Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
469
469
470 In [60]: dview.gather('a')
470 In [60]: dview.gather('a')
471 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
471 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
472
472
473 Other things to look at
473 Other things to look at
474 =======================
474 =======================
475
475
476 How to do parallel list comprehensions
476 How to do parallel list comprehensions
477 --------------------------------------
477 --------------------------------------
478
478
479 In many cases list comprehensions are nicer than using the map function. While
479 In many cases list comprehensions are nicer than using the map function. While
480 we don't have fully parallel list comprehensions, it is simple to get the
480 we don't have fully parallel list comprehensions, it is simple to get the
481 basic effect using :meth:`scatter` and :meth:`gather`:
481 basic effect using :meth:`scatter` and :meth:`gather`:
482
482
483 .. sourcecode:: ipython
483 .. sourcecode:: ipython
484
484
485 In [66]: dview.scatter('x',range(64))
485 In [66]: dview.scatter('x',range(64))
486
486
487 In [67]: %px y = [i**10 for i in x]
487 In [67]: %px y = [i**10 for i in x]
488 Parallel execution on engines: [0, 1, 2, 3]
488 Parallel execution on engines: [0, 1, 2, 3]
489
489
490 In [68]: y = dview.gather('y')
490 In [68]: y = dview.gather('y')
491
491
492 In [69]: print y
492 In [69]: print y
493 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
493 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
494
494
495 Remote imports
495 Remote imports
496 --------------
496 --------------
497
497
498 Sometimes you will want to import packages both in your interactive session
498 Sometimes you will want to import packages both in your interactive session
499 and on your remote engines. This can be done with the :class:`ContextManager`
499 and on your remote engines. This can be done with the :class:`ContextManager`
500 created by a DirectView's :meth:`sync_imports` method:
500 created by a DirectView's :meth:`sync_imports` method:
501
501
502 .. sourcecode:: ipython
502 .. sourcecode:: ipython
503
503
504 In [69]: with dview.sync_imports():
504 In [69]: with dview.sync_imports():
505 ....: import numpy
505 ....: import numpy
506 importing numpy on engine(s)
506 importing numpy on engine(s)
507
507
508 Any imports made inside the block will also be performed on the view's engines.
508 Any imports made inside the block will also be performed on the view's engines.
509 sync_imports also takes a `local` boolean flag that defaults to True, which specifies
509 sync_imports also takes a `local` boolean flag that defaults to True, which specifies
510 whether the local imports should also be performed. However, support for `local=False`
510 whether the local imports should also be performed. However, support for `local=False`
511 has not been implemented, so only packages that can be imported locally will work
511 has not been implemented, so only packages that can be imported locally will work
512 this way.
512 this way.
513
513
514 You can also specify imports via the ``@require`` decorator. This is a decorator
514 You can also specify imports via the ``@require`` decorator. This is a decorator
515 designed for use in Dependencies, but can be used to handle remote imports as well.
515 designed for use in Dependencies, but can be used to handle remote imports as well.
516 Modules or module names passed to ``@require`` will be imported before the decorated
516 Modules or module names passed to ``@require`` will be imported before the decorated
517 function is called. If they cannot be imported, the decorated function will never
517 function is called. If they cannot be imported, the decorated function will never
518 execute and will fail with an UnmetDependencyError. Failures of single Engines will
518 execute and will fail with an UnmetDependencyError. Failures of single Engines will
519 be collected and raise a CompositeError, as demonstrated in the next section.
519 be collected and raise a CompositeError, as demonstrated in the next section.
520
520
521 .. sourcecode:: ipython
521 .. sourcecode:: ipython
522
522
523 In [69]: from IPython.parallel import require
523 In [69]: from IPython.parallel import require
524
524
525 In [70]: @require('re'):
525 In [70]: @require('re'):
526 ....: def findall(pat, x):
526 ....: def findall(pat, x):
527 ....: # re is guaranteed to be available
527 ....: # re is guaranteed to be available
528 ....: return re.findall(pat, x)
528 ....: return re.findall(pat, x)
529
529
530 # you can also pass modules themselves, that you already have locally:
530 # you can also pass modules themselves, that you already have locally:
531 In [71]: @require(time):
531 In [71]: @require(time):
532 ....: def wait(t):
532 ....: def wait(t):
533 ....: time.sleep(t)
533 ....: time.sleep(t)
534 ....: return t
534 ....: return t
535
535
536 .. note::
536 .. note::
537
537
538 :func:`sync_imports` does not allow ``import foo as bar`` syntax,
538 :func:`sync_imports` does not allow ``import foo as bar`` syntax,
539 because the assignment represented by the ``as bar`` part is not
539 because the assignment represented by the ``as bar`` part is not
540 available to the import hook.
540 available to the import hook.
541
541
542
542
543 .. _parallel_exceptions:
543 .. _parallel_exceptions:
544
544
545 Parallel exceptions
545 Parallel exceptions
546 -------------------
546 -------------------
547
547
548 In the multiengine interface, parallel commands can raise Python exceptions,
548 In the multiengine interface, parallel commands can raise Python exceptions,
549 just like serial commands. But it is a little subtle, because a single
549 just like serial commands. But it is a little subtle, because a single
550 parallel command can actually raise multiple exceptions (one for each engine
550 parallel command can actually raise multiple exceptions (one for each engine
551 the command was run on). To express this idea, we have a
551 the command was run on). To express this idea, we have a
552 :exc:`CompositeError` exception class that will be raised in most cases. The
552 :exc:`CompositeError` exception class that will be raised in most cases. The
553 :exc:`CompositeError` class is a special type of exception that wraps one or
553 :exc:`CompositeError` class is a special type of exception that wraps one or
554 more other types of exceptions. Here is how it works:
554 more other types of exceptions. Here is how it works:
555
555
556 .. sourcecode:: ipython
556 .. sourcecode:: ipython
557
557
558 In [78]: dview.block = True
558 In [78]: dview.block = True
559
559
560 In [79]: dview.execute("1/0")
560 In [79]: dview.execute("1/0")
561 [0:execute]:
561 [0:execute]:
562 ---------------------------------------------------------------------------
562 ---------------------------------------------------------------------------
563 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
563 ZeroDivisionError Traceback (most recent call last)
564 ----> 1 1/0
564 ----> 1 1/0
565 ZeroDivisionError: integer division or modulo by zero
565 ZeroDivisionError: integer division or modulo by zero
566
566
567 [1:execute]:
567 [1:execute]:
568 ---------------------------------------------------------------------------
568 ---------------------------------------------------------------------------
569 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
569 ZeroDivisionError Traceback (most recent call last)
570 ----> 1 1/0
570 ----> 1 1/0
571 ZeroDivisionError: integer division or modulo by zero
571 ZeroDivisionError: integer division or modulo by zero
572
572
573 [2:execute]:
573 [2:execute]:
574 ---------------------------------------------------------------------------
574 ---------------------------------------------------------------------------
575 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
575 ZeroDivisionError Traceback (most recent call last)
576 ----> 1 1/0
576 ----> 1 1/0
577 ZeroDivisionError: integer division or modulo by zero
577 ZeroDivisionError: integer division or modulo by zero
578
578
579 [3:execute]:
579 [3:execute]:
580 ---------------------------------------------------------------------------
580 ---------------------------------------------------------------------------
581 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
581 ZeroDivisionError Traceback (most recent call last)
582 ----> 1 1/0
582 ----> 1 1/0
583 ZeroDivisionError: integer division or modulo by zero
583 ZeroDivisionError: integer division or modulo by zero
584
584
585 Notice how the error message printed when :exc:`CompositeError` is raised has
585 Notice how the error message printed when :exc:`CompositeError` is raised has
586 information about the individual exceptions that were raised on each engine.
586 information about the individual exceptions that were raised on each engine.
587 If you want, you can even raise one of these original exceptions:
587 If you want, you can even raise one of these original exceptions:
588
588
589 .. sourcecode:: ipython
589 .. sourcecode:: ipython
590
590
591 In [80]: try:
591 In [80]: try:
592 ....: dview.execute('1/0', block=True)
592 ....: dview.execute('1/0', block=True)
593 ....: except parallel.error.CompositeError, e:
593 ....: except parallel.error.CompositeError, e:
594 ....: e.raise_exception()
594 ....: e.raise_exception()
595 ....:
595 ....:
596 ....:
596 ....:
597 ---------------------------------------------------------------------------
597 ---------------------------------------------------------------------------
598 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
598 ZeroDivisionError Traceback (most recent call last)
599 ----> 1 1/0
599 ----> 1 1/0
600 ZeroDivisionError: integer division or modulo by zero
600 ZeroDivisionError: integer division or modulo by zero
601
601
602 If you are working in IPython, you can simple type ``%debug`` after one of
602 If you are working in IPython, you can simple type ``%debug`` after one of
603 these :exc:`CompositeError` exceptions is raised, and inspect the exception
603 these :exc:`CompositeError` exceptions is raised, and inspect the exception
604 instance:
604 instance:
605
605
606 .. sourcecode:: ipython
606 .. sourcecode:: ipython
607
607
608 In [81]: dview.execute('1/0')
608 In [81]: dview.execute('1/0')
609 [0:execute]:
609 [0:execute]:
610 ---------------------------------------------------------------------------
610 ---------------------------------------------------------------------------
611 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
611 ZeroDivisionError Traceback (most recent call last)
612 ----> 1 1/0
612 ----> 1 1/0
613 ZeroDivisionError: integer division or modulo by zero
613 ZeroDivisionError: integer division or modulo by zero
614
614
615 [1:execute]:
615 [1:execute]:
616 ---------------------------------------------------------------------------
616 ---------------------------------------------------------------------------
617 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
617 ZeroDivisionError Traceback (most recent call last)
618 ----> 1 1/0
618 ----> 1 1/0
619 ZeroDivisionError: integer division or modulo by zero
619 ZeroDivisionError: integer division or modulo by zero
620
620
621 [2:execute]:
621 [2:execute]:
622 ---------------------------------------------------------------------------
622 ---------------------------------------------------------------------------
623 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
623 ZeroDivisionError Traceback (most recent call last)
624 ----> 1 1/0
624 ----> 1 1/0
625 ZeroDivisionError: integer division or modulo by zero
625 ZeroDivisionError: integer division or modulo by zero
626
626
627 [3:execute]:
627 [3:execute]:
628 ---------------------------------------------------------------------------
628 ---------------------------------------------------------------------------
629 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
629 ZeroDivisionError Traceback (most recent call last)
630 ----> 1 1/0
630 ----> 1 1/0
631 ZeroDivisionError: integer division or modulo by zero
631 ZeroDivisionError: integer division or modulo by zero
632
632
633 In [82]: %debug
633 In [82]: %debug
634 > /.../site-packages/IPython/parallel/client/asyncresult.py(125)get()
634 > /.../site-packages/IPython/parallel/client/asyncresult.py(125)get()
635 124 else:
635 124 else:
636 --> 125 raise self._exception
636 --> 125 raise self._exception
637 126 else:
637 126 else:
638
638
639 # Here, self._exception is the CompositeError instance:
639 # Here, self._exception is the CompositeError instance:
640
640
641 ipdb> e = self._exception
641 ipdb> e = self._exception
642 ipdb> e
642 ipdb> e
643 CompositeError(4)
643 CompositeError(4)
644
644
645 # we can tab-complete on e to see available methods:
645 # we can tab-complete on e to see available methods:
646 ipdb> e.<TAB>
646 ipdb> e.<TAB>
647 e.args e.message e.traceback
647 e.args e.message e.traceback
648 e.elist e.msg
648 e.elist e.msg
649 e.ename e.print_traceback
649 e.ename e.print_traceback
650 e.engine_info e.raise_exception
650 e.engine_info e.raise_exception
651 e.evalue e.render_traceback
651 e.evalue e.render_traceback
652
652
653 # We can then display the individual tracebacks, if we want:
653 # We can then display the individual tracebacks, if we want:
654 ipdb> e.print_traceback(1)
654 ipdb> e.print_traceback(1)
655 [1:execute]:
655 [1:execute]:
656 ---------------------------------------------------------------------------
656 ---------------------------------------------------------------------------
657 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
657 ZeroDivisionError Traceback (most recent call last)
658 ----> 1 1/0
658 ----> 1 1/0
659 ZeroDivisionError: integer division or modulo by zero
659 ZeroDivisionError: integer division or modulo by zero
660
660
661
661
662 Since you might have 100 engines, you probably don't want to see 100 tracebacks
663 for a simple NameError because of a typo.
664 For this reason, CompositeError truncates the list of exceptions it will print
665 to :attr:`CompositeError.tb_limit` (default is five).
666 You can change this limit to suit your needs with:
667
668 .. sourcecode:: ipython
669
670 In [20]: from IPython.parallel import CompositeError
671 In [21]: CompositeError.tb_limit = 1
672 In [22]: %px a=b
673 [0:execute]:
674 ---------------------------------------------------------------------------
675 NameError Traceback (most recent call last)
676 ----> 1 a=b
677 NameError: name 'b' is not defined
678
679 ... 3 more exceptions ...
680
681
662 All of this same error handling magic even works in non-blocking mode:
682 All of this same error handling magic even works in non-blocking mode:
663
683
664 .. sourcecode:: ipython
684 .. sourcecode:: ipython
665
685
666 In [83]: dview.block=False
686 In [83]: dview.block=False
667
687
668 In [84]: ar = dview.execute('1/0')
688 In [84]: ar = dview.execute('1/0')
669
689
670 In [85]: ar.get()
690 In [85]: ar.get()
671 [0:execute]:
691 [0:execute]:
672 ---------------------------------------------------------------------------
692 ---------------------------------------------------------------------------
673 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
693 ZeroDivisionError Traceback (most recent call last)
674 ----> 1 1/0
675 ZeroDivisionError: integer division or modulo by zero
676
677 [1:execute]:
678 ---------------------------------------------------------------------------
679 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
680 ----> 1 1/0
694 ----> 1 1/0
681 ZeroDivisionError: integer division or modulo by zero
695 ZeroDivisionError: integer division or modulo by zero
682
696
683 [2:execute]:
697 ... 3 more exceptions ...
684 ---------------------------------------------------------------------------
685 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
686 ----> 1 1/0
687 ZeroDivisionError: integer division or modulo by zero
688
689 [3:execute]:
690 ---------------------------------------------------------------------------
691 ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
692 ----> 1 1/0
693 ZeroDivisionError: integer division or modulo by zero
694
General Comments 0
You need to be logged in to leave comments. Login now