##// END OF EJS Templates
fix a few doc typos...
MinRK -
Show More
@@ -1,23 +1,24 b''
1 .. _plugins_overview:
1 .. _plugins_overview:
2
2
3 ===============
3 ===============
4 IPython plugins
4 IPython plugins
5 ===============
5 ===============
6
6
7 IPython has a plugin mechanism that allows users to create new and custom
7 IPython has a plugin mechanism that allows users to create new and custom
8 runtime components for IPython. Plugins are different from extensions:
8 runtime components for IPython. Plugins are different from extensions:
9
9
10 * Extensions are used to load plugins.
10 * Extensions are used to load plugins.
11 * Extensions are a more advanced configuration system that gives you access
11 * Extensions are a more advanced configuration system that gives you access
12 to the running IPython instance.
12 to the running IPython instance.
13 * Plugins add entirely new capabilities to IPython.
13 * Plugins add entirely new capabilities to IPython.
14 * Plugins are traited and configurable.
14 * Plugins are traited and configurable.
15
15
16 At this point, our plugin system is brand new and the documentation is
16 At this point, our plugin system is brand new and the documentation is
17 minimal. If you are interested in creating a new plugin, see the following
17 minimal. If you are interested in creating a new plugin, see the following
18 files:
18 files:
19
19
20 * :file:`IPython/extensions/parallemagic.py`
20 * :file:`IPython/extensions/parallelmagic.py`
21 * :file:`IPython/extensions/pretty.`
21 * :file:`IPython/extensions/autoreload.py`
22 * :file:`IPython/extensions/sympyprinting.py`
22
23
23 As well as our documentation on the configuration system and extensions.
24 As well as our documentation on the configuration system and extensions.
@@ -1,847 +1,847 b''
1 .. _parallel_multiengine:
1 .. _parallel_multiengine:
2
2
3 ==========================
3 ==========================
4 IPython's Direct interface
4 IPython's Direct interface
5 ==========================
5 ==========================
6
6
7 The direct, or multiengine, interface represents one possible way of working with a set of
7 The direct, or multiengine, interface represents one possible way of working with a set of
8 IPython engines. The basic idea behind the multiengine interface is that the
8 IPython engines. The basic idea behind the multiengine interface is that the
9 capabilities of each engine are directly and explicitly exposed to the user.
9 capabilities of each engine are directly and explicitly exposed to the user.
10 Thus, in the multiengine interface, each engine is given an id that is used to
10 Thus, in the multiengine interface, each engine is given an id that is used to
11 identify the engine and give it work to do. This interface is very intuitive
11 identify the engine and give it work to do. This interface is very intuitive
12 and is designed with interactive usage in mind, and is the best place for
12 and is designed with interactive usage in mind, and is the best place for
13 new users of IPython to begin.
13 new users of IPython to begin.
14
14
15 Starting the IPython controller and engines
15 Starting the IPython controller and engines
16 ===========================================
16 ===========================================
17
17
18 To follow along with this tutorial, you will need to start the IPython
18 To follow along with this tutorial, you will need to start the IPython
19 controller and four IPython engines. The simplest way of doing this is to use
19 controller and four IPython engines. The simplest way of doing this is to use
20 the :command:`ipcluster` command::
20 the :command:`ipcluster` command::
21
21
22 $ ipcluster start -n 4
22 $ ipcluster start -n 4
23
23
24 For more detailed information about starting the controller and engines, see
24 For more detailed information about starting the controller and engines, see
25 our :ref:`introduction <parallel_overview>` to using IPython for parallel computing.
25 our :ref:`introduction <parallel_overview>` to using IPython for parallel computing.
26
26
27 Creating a ``Client`` instance
27 Creating a ``Client`` instance
28 ==============================
28 ==============================
29
29
30 The first step is to import the IPython :mod:`IPython.parallel`
30 The first step is to import the IPython :mod:`IPython.parallel`
31 module and then create a :class:`.Client` instance:
31 module and then create a :class:`.Client` instance:
32
32
33 .. sourcecode:: ipython
33 .. sourcecode:: ipython
34
34
35 In [1]: from IPython.parallel import Client
35 In [1]: from IPython.parallel import Client
36
36
37 In [2]: rc = Client()
37 In [2]: rc = Client()
38
38
39 This form assumes that the default connection information (stored in
39 This form assumes that the default connection information (stored in
40 :file:`ipcontroller-client.json` found in :file:`IPYTHON_DIR/profile_default/security`) is
40 :file:`ipcontroller-client.json` found in :file:`IPYTHON_DIR/profile_default/security`) is
41 accurate. If the controller was started on a remote machine, you must copy that connection
41 accurate. If the controller was started on a remote machine, you must copy that connection
42 file to the client machine, or enter its contents as arguments to the Client constructor:
42 file to the client machine, or enter its contents as arguments to the Client constructor:
43
43
44 .. sourcecode:: ipython
44 .. sourcecode:: ipython
45
45
46 # If you have copied the json connector file from the controller:
46 # If you have copied the json connector file from the controller:
47 In [2]: rc = Client('/path/to/ipcontroller-client.json')
47 In [2]: rc = Client('/path/to/ipcontroller-client.json')
48 # or to connect with a specific profile you have set up:
48 # or to connect with a specific profile you have set up:
49 In [3]: rc = Client(profile='mpi')
49 In [3]: rc = Client(profile='mpi')
50
50
51
51
52 To make sure there are engines connected to the controller, users can get a list
52 To make sure there are engines connected to the controller, users can get a list
53 of engine ids:
53 of engine ids:
54
54
55 .. sourcecode:: ipython
55 .. sourcecode:: ipython
56
56
57 In [3]: rc.ids
57 In [3]: rc.ids
58 Out[3]: [0, 1, 2, 3]
58 Out[3]: [0, 1, 2, 3]
59
59
60 Here we see that there are four engines ready to do work for us.
60 Here we see that there are four engines ready to do work for us.
61
61
62 For direct execution, we will make use of a :class:`DirectView` object, which can be
62 For direct execution, we will make use of a :class:`DirectView` object, which can be
63 constructed via list-access to the client:
63 constructed via list-access to the client:
64
64
65 .. sourcecode:: ipython
65 .. sourcecode:: ipython
66
66
67 In [4]: dview = rc[:] # use all engines
67 In [4]: dview = rc[:] # use all engines
68
68
69 .. seealso::
69 .. seealso::
70
70
71 For more information, see the in-depth explanation of :ref:`Views <parallel_details>`.
71 For more information, see the in-depth explanation of :ref:`Views <parallel_details>`.
72
72
73
73
74 Quick and easy parallelism
74 Quick and easy parallelism
75 ==========================
75 ==========================
76
76
77 In many cases, you simply want to apply a Python function to a sequence of
77 In many cases, you simply want to apply a Python function to a sequence of
78 objects, but *in parallel*. The client interface provides a simple way
78 objects, but *in parallel*. The client interface provides a simple way
79 of accomplishing this: using the DirectView's :meth:`~DirectView.map` method.
79 of accomplishing this: using the DirectView's :meth:`~DirectView.map` method.
80
80
81 Parallel map
81 Parallel map
82 ------------
82 ------------
83
83
84 Python's builtin :func:`map` functions allows a function to be applied to a
84 Python's builtin :func:`map` functions allows a function to be applied to a
85 sequence element-by-element. This type of code is typically trivial to
85 sequence element-by-element. This type of code is typically trivial to
86 parallelize. In fact, since IPython's interface is all about functions anyway,
86 parallelize. In fact, since IPython's interface is all about functions anyway,
87 you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a
87 you can just use the builtin :func:`map` with a :class:`RemoteFunction`, or a
88 DirectView's :meth:`map` method:
88 DirectView's :meth:`map` method:
89
89
90 .. sourcecode:: ipython
90 .. sourcecode:: ipython
91
91
92 In [62]: serial_result = map(lambda x:x**10, range(32))
92 In [62]: serial_result = map(lambda x:x**10, range(32))
93
93
94 In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32))
94 In [63]: parallel_result = dview.map_sync(lambda x: x**10, range(32))
95
95
96 In [67]: serial_result==parallel_result
96 In [67]: serial_result==parallel_result
97 Out[67]: True
97 Out[67]: True
98
98
99
99
100 .. note::
100 .. note::
101
101
102 The :class:`DirectView`'s version of :meth:`map` does
102 The :class:`DirectView`'s version of :meth:`map` does
103 not do dynamic load balancing. For a load balanced version, use a
103 not do dynamic load balancing. For a load balanced version, use a
104 :class:`LoadBalancedView`.
104 :class:`LoadBalancedView`.
105
105
106 .. seealso::
106 .. seealso::
107
107
108 :meth:`map` is implemented via :class:`ParallelFunction`.
108 :meth:`map` is implemented via :class:`ParallelFunction`.
109
109
110 Remote function decorators
110 Remote function decorators
111 --------------------------
111 --------------------------
112
112
113 Remote functions are just like normal functions, but when they are called,
113 Remote functions are just like normal functions, but when they are called,
114 they execute on one or more engines, rather than locally. IPython provides
114 they execute on one or more engines, rather than locally. IPython provides
115 two decorators:
115 two decorators:
116
116
117 .. sourcecode:: ipython
117 .. sourcecode:: ipython
118
118
119 In [10]: @dview.remote(block=True)
119 In [10]: @dview.remote(block=True)
120 ...: def getpid():
120 ...: def getpid():
121 ...: import os
121 ...: import os
122 ...: return os.getpid()
122 ...: return os.getpid()
123 ...:
123 ...:
124
124
125 In [11]: getpid()
125 In [11]: getpid()
126 Out[11]: [12345, 12346, 12347, 12348]
126 Out[11]: [12345, 12346, 12347, 12348]
127
127
128 The ``@parallel`` decorator creates parallel functions, that break up an element-wise
128 The ``@parallel`` decorator creates parallel functions, that break up an element-wise
129 operations and distribute them, reconstructing the result.
129 operations and distribute them, reconstructing the result.
130
130
131 .. sourcecode:: ipython
131 .. sourcecode:: ipython
132
132
133 In [12]: import numpy as np
133 In [12]: import numpy as np
134
134
135 In [13]: A = np.random.random((64,48))
135 In [13]: A = np.random.random((64,48))
136
136
137 In [14]: @dview.parallel(block=True)
137 In [14]: @dview.parallel(block=True)
138 ...: def pmul(A,B):
138 ...: def pmul(A,B):
139 ...: return A*B
139 ...: return A*B
140
140
141 In [15]: C_local = A*A
141 In [15]: C_local = A*A
142
142
143 In [16]: C_remote = pmul(A,A)
143 In [16]: C_remote = pmul(A,A)
144
144
145 In [17]: (C_local == C_remote).all()
145 In [17]: (C_local == C_remote).all()
146 Out[17]: True
146 Out[17]: True
147
147
148 .. seealso::
148 .. seealso::
149
149
150 See the docstrings for the :func:`parallel` and :func:`remote` decorators for
150 See the docstrings for the :func:`parallel` and :func:`remote` decorators for
151 options.
151 options.
152
152
153 Calling Python functions
153 Calling Python functions
154 ========================
154 ========================
155
155
156 The most basic type of operation that can be performed on the engines is to
156 The most basic type of operation that can be performed on the engines is to
157 execute Python code or call Python functions. Executing Python code can be
157 execute Python code or call Python functions. Executing Python code can be
158 done in blocking or non-blocking mode (non-blocking is default) using the
158 done in blocking or non-blocking mode (non-blocking is default) using the
159 :meth:`.View.execute` method, and calling functions can be done via the
159 :meth:`.View.execute` method, and calling functions can be done via the
160 :meth:`.View.apply` method.
160 :meth:`.View.apply` method.
161
161
162 apply
162 apply
163 -----
163 -----
164
164
165 The main method for doing remote execution (in fact, all methods that
165 The main method for doing remote execution (in fact, all methods that
166 communicate with the engines are built on top of it), is :meth:`View.apply`.
166 communicate with the engines are built on top of it), is :meth:`View.apply`.
167
167
168 We strive to provide the cleanest interface we can, so `apply` has the following
168 We strive to provide the cleanest interface we can, so `apply` has the following
169 signature:
169 signature:
170
170
171 .. sourcecode:: python
171 .. sourcecode:: python
172
172
173 view.apply(f, *args, **kwargs)
173 view.apply(f, *args, **kwargs)
174
174
175 There are various ways to call functions with IPython, and these flags are set as
175 There are various ways to call functions with IPython, and these flags are set as
176 attributes of the View. The ``DirectView`` has just two of these flags:
176 attributes of the View. The ``DirectView`` has just two of these flags:
177
177
178 dv.block : bool
178 dv.block : bool
179 whether to wait for the result, or return an :class:`AsyncResult` object
179 whether to wait for the result, or return an :class:`AsyncResult` object
180 immediately
180 immediately
181 dv.track : bool
181 dv.track : bool
182 whether to instruct pyzmq to track when
182 whether to instruct pyzmq to track when zeromq is done sending the message.
183 This is primarily useful for non-copying sends of numpy arrays that you plan to
183 This is primarily useful for non-copying sends of numpy arrays that you plan to
184 edit in-place. You need to know when it becomes safe to edit the buffer
184 edit in-place. You need to know when it becomes safe to edit the buffer
185 without corrupting the message.
185 without corrupting the message.
186
186
187
187
188 Creating a view is simple: index-access on a client creates a :class:`.DirectView`.
188 Creating a view is simple: index-access on a client creates a :class:`.DirectView`.
189
189
190 .. sourcecode:: ipython
190 .. sourcecode:: ipython
191
191
192 In [4]: view = rc[1:3]
192 In [4]: view = rc[1:3]
193 Out[4]: <DirectView [1, 2]>
193 Out[4]: <DirectView [1, 2]>
194
194
195 In [5]: view.apply<tab>
195 In [5]: view.apply<tab>
196 view.apply view.apply_async view.apply_sync
196 view.apply view.apply_async view.apply_sync
197
197
198 For convenience, you can set block temporarily for a single call with the extra sync/async methods.
198 For convenience, you can set block temporarily for a single call with the extra sync/async methods.
199
199
200 Blocking execution
200 Blocking execution
201 ------------------
201 ------------------
202
202
203 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
203 In blocking mode, the :class:`.DirectView` object (called ``dview`` in
204 these examples) submits the command to the controller, which places the
204 these examples) submits the command to the controller, which places the
205 command in the engines' queues for execution. The :meth:`apply` call then
205 command in the engines' queues for execution. The :meth:`apply` call then
206 blocks until the engines are done executing the command:
206 blocks until the engines are done executing the command:
207
207
208 .. sourcecode:: ipython
208 .. sourcecode:: ipython
209
209
210 In [2]: dview = rc[:] # A DirectView of all engines
210 In [2]: dview = rc[:] # A DirectView of all engines
211 In [3]: dview.block=True
211 In [3]: dview.block=True
212 In [4]: dview['a'] = 5
212 In [4]: dview['a'] = 5
213
213
214 In [5]: dview['b'] = 10
214 In [5]: dview['b'] = 10
215
215
216 In [6]: dview.apply(lambda x: a+b+x, 27)
216 In [6]: dview.apply(lambda x: a+b+x, 27)
217 Out[6]: [42, 42, 42, 42]
217 Out[6]: [42, 42, 42, 42]
218
218
219 You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync`
219 You can also select blocking execution on a call-by-call basis with the :meth:`apply_sync`
220 method:
220 method:
221
221
222 In [7]: dview.block=False
222 In [7]: dview.block=False
223
223
224 In [8]: dview.apply_sync(lambda x: a+b+x, 27)
224 In [8]: dview.apply_sync(lambda x: a+b+x, 27)
225 Out[8]: [42, 42, 42, 42]
225 Out[8]: [42, 42, 42, 42]
226
226
227 Python commands can be executed as strings on specific engines by using a View's ``execute``
227 Python commands can be executed as strings on specific engines by using a View's ``execute``
228 method:
228 method:
229
229
230 .. sourcecode:: ipython
230 .. sourcecode:: ipython
231
231
232 In [6]: rc[::2].execute('c=a+b')
232 In [6]: rc[::2].execute('c=a+b')
233
233
234 In [7]: rc[1::2].execute('c=a-b')
234 In [7]: rc[1::2].execute('c=a-b')
235
235
236 In [8]: dview['c'] # shorthand for dview.pull('c', block=True)
236 In [8]: dview['c'] # shorthand for dview.pull('c', block=True)
237 Out[8]: [15, -5, 15, -5]
237 Out[8]: [15, -5, 15, -5]
238
238
239
239
240 Non-blocking execution
240 Non-blocking execution
241 ----------------------
241 ----------------------
242
242
243 In non-blocking mode, :meth:`apply` submits the command to be executed and
243 In non-blocking mode, :meth:`apply` submits the command to be executed and
244 then returns a :class:`AsyncResult` object immediately. The
244 then returns a :class:`AsyncResult` object immediately. The
245 :class:`AsyncResult` object gives you a way of getting a result at a later
245 :class:`AsyncResult` object gives you a way of getting a result at a later
246 time through its :meth:`get` method.
246 time through its :meth:`get` method.
247
247
248 .. Note::
248 .. Note::
249
249
250 The :class:`AsyncResult` object provides a superset of the interface in
250 The :class:`AsyncResult` object provides a superset of the interface in
251 :py:class:`multiprocessing.pool.AsyncResult`. See the
251 :py:class:`multiprocessing.pool.AsyncResult`. See the
252 `official Python documentation <http://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_
252 `official Python documentation <http://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_
253 for more.
253 for more.
254
254
255
255
256 This allows you to quickly submit long running commands without blocking your
256 This allows you to quickly submit long running commands without blocking your
257 local Python/IPython session:
257 local Python/IPython session:
258
258
259 .. sourcecode:: ipython
259 .. sourcecode:: ipython
260
260
261 # define our function
261 # define our function
262 In [6]: def wait(t):
262 In [6]: def wait(t):
263 ...: import time
263 ...: import time
264 ...: tic = time.time()
264 ...: tic = time.time()
265 ...: time.sleep(t)
265 ...: time.sleep(t)
266 ...: return time.time()-tic
266 ...: return time.time()-tic
267
267
268 # In non-blocking mode
268 # In non-blocking mode
269 In [7]: ar = dview.apply_async(wait, 2)
269 In [7]: ar = dview.apply_async(wait, 2)
270
270
271 # Now block for the result
271 # Now block for the result
272 In [8]: ar.get()
272 In [8]: ar.get()
273 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
273 Out[8]: [2.0006198883056641, 1.9997570514678955, 1.9996809959411621, 2.0003249645233154]
274
274
275 # Again in non-blocking mode
275 # Again in non-blocking mode
276 In [9]: ar = dview.apply_async(wait, 10)
276 In [9]: ar = dview.apply_async(wait, 10)
277
277
278 # Poll to see if the result is ready
278 # Poll to see if the result is ready
279 In [10]: ar.ready()
279 In [10]: ar.ready()
280 Out[10]: False
280 Out[10]: False
281
281
282 # ask for the result, but wait a maximum of 1 second:
282 # ask for the result, but wait a maximum of 1 second:
283 In [45]: ar.get(1)
283 In [45]: ar.get(1)
284 ---------------------------------------------------------------------------
284 ---------------------------------------------------------------------------
285 TimeoutError Traceback (most recent call last)
285 TimeoutError Traceback (most recent call last)
286 /home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
286 /home/you/<ipython-input-45-7cd858bbb8e0> in <module>()
287 ----> 1 ar.get(1)
287 ----> 1 ar.get(1)
288
288
289 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
289 /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
290 62 raise self._exception
290 62 raise self._exception
291 63 else:
291 63 else:
292 ---> 64 raise error.TimeoutError("Result not ready.")
292 ---> 64 raise error.TimeoutError("Result not ready.")
293 65
293 65
294 66 def ready(self):
294 66 def ready(self):
295
295
296 TimeoutError: Result not ready.
296 TimeoutError: Result not ready.
297
297
298 .. Note::
298 .. Note::
299
299
300 Note the import inside the function. This is a common model, to ensure
300 Note the import inside the function. This is a common model, to ensure
301 that the appropriate modules are imported where the task is run. You can
301 that the appropriate modules are imported where the task is run. You can
302 also manually import modules into the engine(s) namespace(s) via
302 also manually import modules into the engine(s) namespace(s) via
303 :meth:`view.execute('import numpy')`.
303 :meth:`view.execute('import numpy')`.
304
304
305 Often, it is desirable to wait until a set of :class:`AsyncResult` objects
305 Often, it is desirable to wait until a set of :class:`AsyncResult` objects
306 are done. For this, there is a the method :meth:`wait`. This method takes a
306 are done. For this, there is a the method :meth:`wait`. This method takes a
307 tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History),
307 tuple of :class:`AsyncResult` objects (or `msg_ids` or indices to the client's History),
308 and blocks until all of the associated results are ready:
308 and blocks until all of the associated results are ready:
309
309
310 .. sourcecode:: ipython
310 .. sourcecode:: ipython
311
311
312 In [72]: dview.block=False
312 In [72]: dview.block=False
313
313
314 # A trivial list of AsyncResults objects
314 # A trivial list of AsyncResults objects
315 In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)]
315 In [73]: pr_list = [dview.apply_async(wait, 3) for i in range(10)]
316
316
317 # Wait until all of them are done
317 # Wait until all of them are done
318 In [74]: dview.wait(pr_list)
318 In [74]: dview.wait(pr_list)
319
319
320 # Then, their results are ready using get() or the `.r` attribute
320 # Then, their results are ready using get() or the `.r` attribute
321 In [75]: pr_list[0].get()
321 In [75]: pr_list[0].get()
322 Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
322 Out[75]: [2.9982571601867676, 2.9982588291168213, 2.9987530708312988, 2.9990990161895752]
323
323
324
324
325
325
326 The ``block`` and ``targets`` keyword arguments and attributes
326 The ``block`` and ``targets`` keyword arguments and attributes
327 --------------------------------------------------------------
327 --------------------------------------------------------------
328
328
329 Most DirectView methods (excluding :meth:`apply` and :meth:`map`) accept ``block`` and
329 Most DirectView methods (excluding :meth:`apply` and :meth:`map`) accept ``block`` and
330 ``targets`` as keyword arguments. As we have seen above, these keyword arguments control the
330 ``targets`` as keyword arguments. As we have seen above, these keyword arguments control the
331 blocking mode and which engines the command is applied to. The :class:`View` class also has
331 blocking mode and which engines the command is applied to. The :class:`View` class also has
332 :attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword
332 :attr:`block` and :attr:`targets` attributes that control the default behavior when the keyword
333 arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`:
333 arguments are not provided. Thus the following logic is used for :attr:`block` and :attr:`targets`:
334
334
335 * If no keyword argument is provided, the instance attributes are used.
335 * If no keyword argument is provided, the instance attributes are used.
336 * Keyword argument, if provided override the instance attributes for
336 * Keyword argument, if provided override the instance attributes for
337 the duration of a single call.
337 the duration of a single call.
338
338
339 The following examples demonstrate how to use the instance attributes:
339 The following examples demonstrate how to use the instance attributes:
340
340
341 .. sourcecode:: ipython
341 .. sourcecode:: ipython
342
342
343 In [16]: dview.targets = [0,2]
343 In [16]: dview.targets = [0,2]
344
344
345 In [17]: dview.block = False
345 In [17]: dview.block = False
346
346
347 In [18]: ar = dview.apply(lambda : 10)
347 In [18]: ar = dview.apply(lambda : 10)
348
348
349 In [19]: ar.get()
349 In [19]: ar.get()
350 Out[19]: [10, 10]
350 Out[19]: [10, 10]
351
351
352 In [16]: dview.targets = v.client.ids # all engines (4)
352 In [16]: dview.targets = v.client.ids # all engines (4)
353
353
354 In [21]: dview.block = True
354 In [21]: dview.block = True
355
355
356 In [22]: dview.apply(lambda : 42)
356 In [22]: dview.apply(lambda : 42)
357 Out[22]: [42, 42, 42, 42]
357 Out[22]: [42, 42, 42, 42]
358
358
359 The :attr:`block` and :attr:`targets` instance attributes of the
359 The :attr:`block` and :attr:`targets` instance attributes of the
360 :class:`.DirectView` also determine the behavior of the parallel magic commands.
360 :class:`.DirectView` also determine the behavior of the parallel magic commands.
361
361
362 Parallel magic commands
362 Parallel magic commands
363 -----------------------
363 -----------------------
364
364
365 .. warning::
365 .. warning::
366
366
367 The magics have not been changed to work with the zeromq system. The
367 The magics have not been changed to work with the zeromq system. The
368 magics do work, but *do not* print stdin/out like they used to in IPython.kernel.
368 magics do work, but *do not* print stdin/out like they used to in IPython.kernel.
369
369
370 We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
370 We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
371 that make it more pleasant to execute Python commands on the engines
371 that make it more pleasant to execute Python commands on the engines
372 interactively. These are simply shortcuts to :meth:`execute` and
372 interactively. These are simply shortcuts to :meth:`execute` and
373 :meth:`get_result` of the :class:`DirectView`. The ``%px`` magic executes a single
373 :meth:`get_result` of the :class:`DirectView`. The ``%px`` magic executes a single
374 Python command on the engines specified by the :attr:`targets` attribute of the
374 Python command on the engines specified by the :attr:`targets` attribute of the
375 :class:`DirectView` instance:
375 :class:`DirectView` instance:
376
376
377 .. sourcecode:: ipython
377 .. sourcecode:: ipython
378
378
379 # load the parallel magic extension:
379 # load the parallel magic extension:
380 In [21]: %load_ext parallelmagic
380 In [21]: %load_ext parallelmagic
381
381
382 # Create a DirectView for all targets
382 # Create a DirectView for all targets
383 In [22]: dv = rc[:]
383 In [22]: dv = rc[:]
384
384
385 # Make this DirectView active for parallel magic commands
385 # Make this DirectView active for parallel magic commands
386 In [23]: dv.activate()
386 In [23]: dv.activate()
387
387
388 In [24]: dv.block=True
388 In [24]: dv.block=True
389
389
390 In [25]: import numpy
390 In [25]: import numpy
391
391
392 In [26]: %px import numpy
392 In [26]: %px import numpy
393 Parallel execution on engines: [0, 1, 2, 3]
393 Parallel execution on engines: [0, 1, 2, 3]
394
394
395 In [27]: %px a = numpy.random.rand(2,2)
395 In [27]: %px a = numpy.random.rand(2,2)
396 Parallel execution on engines: [0, 1, 2, 3]
396 Parallel execution on engines: [0, 1, 2, 3]
397
397
398 In [28]: %px ev = numpy.linalg.eigvals(a)
398 In [28]: %px ev = numpy.linalg.eigvals(a)
399 Parallel execution on engines: [0, 1, 2, 3]
399 Parallel execution on engines: [0, 1, 2, 3]
400
400
401 In [28]: dv['ev']
401 In [28]: dv['ev']
402 Out[28]: [ array([ 1.09522024, -0.09645227]),
402 Out[28]: [ array([ 1.09522024, -0.09645227]),
403 array([ 1.21435496, -0.35546712]),
403 array([ 1.21435496, -0.35546712]),
404 array([ 0.72180653, 0.07133042]),
404 array([ 0.72180653, 0.07133042]),
405 array([ 1.46384341e+00, 1.04353244e-04])
405 array([ 1.46384341e+00, 1.04353244e-04])
406 ]
406 ]
407
407
408 The ``%result`` magic gets the most recent result, or takes an argument
408 The ``%result`` magic gets the most recent result, or takes an argument
409 specifying the index of the result to be requested. It is simply a shortcut to the
409 specifying the index of the result to be requested. It is simply a shortcut to the
410 :meth:`get_result` method:
410 :meth:`get_result` method:
411
411
412 .. sourcecode:: ipython
412 .. sourcecode:: ipython
413
413
414 In [29]: dv.apply_async(lambda : ev)
414 In [29]: dv.apply_async(lambda : ev)
415
415
416 In [30]: %result
416 In [30]: %result
417 Out[30]: [ [ 1.28167017 0.14197338],
417 Out[30]: [ [ 1.28167017 0.14197338],
418 [-0.14093616 1.27877273],
418 [-0.14093616 1.27877273],
419 [-0.37023573 1.06779409],
419 [-0.37023573 1.06779409],
420 [ 0.83664764 -0.25602658] ]
420 [ 0.83664764 -0.25602658] ]
421
421
422 The ``%autopx`` magic switches to a mode where everything you type is executed
422 The ``%autopx`` magic switches to a mode where everything you type is executed
423 on the engines given by the :attr:`targets` attribute:
423 on the engines given by the :attr:`targets` attribute:
424
424
425 .. sourcecode:: ipython
425 .. sourcecode:: ipython
426
426
427 In [30]: dv.block=False
427 In [30]: dv.block=False
428
428
429 In [31]: %autopx
429 In [31]: %autopx
430 Auto Parallel Enabled
430 Auto Parallel Enabled
431 Type %autopx to disable
431 Type %autopx to disable
432
432
433 In [32]: max_evals = []
433 In [32]: max_evals = []
434 <IPython.parallel.AsyncResult object at 0x17b8a70>
434 <IPython.parallel.AsyncResult object at 0x17b8a70>
435
435
436 In [33]: for i in range(100):
436 In [33]: for i in range(100):
437 ....: a = numpy.random.rand(10,10)
437 ....: a = numpy.random.rand(10,10)
438 ....: a = a+a.transpose()
438 ....: a = a+a.transpose()
439 ....: evals = numpy.linalg.eigvals(a)
439 ....: evals = numpy.linalg.eigvals(a)
440 ....: max_evals.append(evals[0].real)
440 ....: max_evals.append(evals[0].real)
441 ....:
441 ....:
442 ....:
442 ....:
443 <IPython.parallel.AsyncResult object at 0x17af8f0>
443 <IPython.parallel.AsyncResult object at 0x17af8f0>
444
444
445 In [34]: %autopx
445 In [34]: %autopx
446 Auto Parallel Disabled
446 Auto Parallel Disabled
447
447
448 In [35]: dv.block=True
448 In [35]: dv.block=True
449
449
450 In [36]: px ans= "Average max eigenvalue is: %f"%(sum(max_evals)/len(max_evals))
450 In [36]: px ans= "Average max eigenvalue is: %f"%(sum(max_evals)/len(max_evals))
451 Parallel execution on engines: [0, 1, 2, 3]
451 Parallel execution on engines: [0, 1, 2, 3]
452
452
453 In [37]: dv['ans']
453 In [37]: dv['ans']
454 Out[37]: [ 'Average max eigenvalue is: 10.1387247332',
454 Out[37]: [ 'Average max eigenvalue is: 10.1387247332',
455 'Average max eigenvalue is: 10.2076902286',
455 'Average max eigenvalue is: 10.2076902286',
456 'Average max eigenvalue is: 10.1891484655',
456 'Average max eigenvalue is: 10.1891484655',
457 'Average max eigenvalue is: 10.1158837784',]
457 'Average max eigenvalue is: 10.1158837784',]
458
458
459
459
460 Moving Python objects around
460 Moving Python objects around
461 ============================
461 ============================
462
462
463 In addition to calling functions and executing code on engines, you can
463 In addition to calling functions and executing code on engines, you can
464 transfer Python objects to and from your IPython session and the engines. In
464 transfer Python objects to and from your IPython session and the engines. In
465 IPython, these operations are called :meth:`push` (sending an object to the
465 IPython, these operations are called :meth:`push` (sending an object to the
466 engines) and :meth:`pull` (getting an object from the engines).
466 engines) and :meth:`pull` (getting an object from the engines).
467
467
468 Basic push and pull
468 Basic push and pull
469 -------------------
469 -------------------
470
470
471 Here are some examples of how you use :meth:`push` and :meth:`pull`:
471 Here are some examples of how you use :meth:`push` and :meth:`pull`:
472
472
473 .. sourcecode:: ipython
473 .. sourcecode:: ipython
474
474
475 In [38]: dview.push(dict(a=1.03234,b=3453))
475 In [38]: dview.push(dict(a=1.03234,b=3453))
476 Out[38]: [None,None,None,None]
476 Out[38]: [None,None,None,None]
477
477
478 In [39]: dview.pull('a')
478 In [39]: dview.pull('a')
479 Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234]
479 Out[39]: [ 1.03234, 1.03234, 1.03234, 1.03234]
480
480
481 In [40]: dview.pull('b', targets=0)
481 In [40]: dview.pull('b', targets=0)
482 Out[40]: 3453
482 Out[40]: 3453
483
483
484 In [41]: dview.pull(('a','b'))
484 In [41]: dview.pull(('a','b'))
485 Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ]
485 Out[41]: [ [1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453] ]
486
486
487 In [43]: dview.push(dict(c='speed'))
487 In [43]: dview.push(dict(c='speed'))
488 Out[43]: [None,None,None,None]
488 Out[43]: [None,None,None,None]
489
489
490 In non-blocking mode :meth:`push` and :meth:`pull` also return
490 In non-blocking mode :meth:`push` and :meth:`pull` also return
491 :class:`AsyncResult` objects:
491 :class:`AsyncResult` objects:
492
492
493 .. sourcecode:: ipython
493 .. sourcecode:: ipython
494
494
495 In [48]: ar = dview.pull('a', block=False)
495 In [48]: ar = dview.pull('a', block=False)
496
496
497 In [49]: ar.get()
497 In [49]: ar.get()
498 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
498 Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
499
499
500
500
501 Dictionary interface
501 Dictionary interface
502 --------------------
502 --------------------
503
503
504 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
504 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
505 dictionary-style access by key and methods such as :meth:`get` and
505 dictionary-style access by key and methods such as :meth:`get` and
506 :meth:`update` for convenience. This make the remote namespaces of the engines
506 :meth:`update` for convenience. This make the remote namespaces of the engines
507 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
507 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
508
508
509 .. sourcecode:: ipython
509 .. sourcecode:: ipython
510
510
511 In [51]: dview['a']=['foo','bar']
511 In [51]: dview['a']=['foo','bar']
512
512
513 In [52]: dview['a']
513 In [52]: dview['a']
514 Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
514 Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
515
515
516 Scatter and gather
516 Scatter and gather
517 ------------------
517 ------------------
518
518
519 Sometimes it is useful to partition a sequence and push the partitions to
519 Sometimes it is useful to partition a sequence and push the partitions to
520 different engines. In MPI language, this is know as scatter/gather and we
520 different engines. In MPI language, this is know as scatter/gather and we
521 follow that terminology. However, it is important to remember that in
521 follow that terminology. However, it is important to remember that in
522 IPython's :class:`Client` class, :meth:`scatter` is from the
522 IPython's :class:`Client` class, :meth:`scatter` is from the
523 interactive IPython session to the engines and :meth:`gather` is from the
523 interactive IPython session to the engines and :meth:`gather` is from the
524 engines back to the interactive IPython session. For scatter/gather operations
524 engines back to the interactive IPython session. For scatter/gather operations
525 between engines, MPI should be used:
525 between engines, MPI should be used:
526
526
527 .. sourcecode:: ipython
527 .. sourcecode:: ipython
528
528
529 In [58]: dview.scatter('a',range(16))
529 In [58]: dview.scatter('a',range(16))
530 Out[58]: [None,None,None,None]
530 Out[58]: [None,None,None,None]
531
531
532 In [59]: dview['a']
532 In [59]: dview['a']
533 Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
533 Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
534
534
535 In [60]: dview.gather('a')
535 In [60]: dview.gather('a')
536 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
536 Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
537
537
538 Other things to look at
538 Other things to look at
539 =======================
539 =======================
540
540
541 How to do parallel list comprehensions
541 How to do parallel list comprehensions
542 --------------------------------------
542 --------------------------------------
543
543
544 In many cases list comprehensions are nicer than using the map function. While
544 In many cases list comprehensions are nicer than using the map function. While
545 we don't have fully parallel list comprehensions, it is simple to get the
545 we don't have fully parallel list comprehensions, it is simple to get the
546 basic effect using :meth:`scatter` and :meth:`gather`:
546 basic effect using :meth:`scatter` and :meth:`gather`:
547
547
548 .. sourcecode:: ipython
548 .. sourcecode:: ipython
549
549
550 In [66]: dview.scatter('x',range(64))
550 In [66]: dview.scatter('x',range(64))
551
551
552 In [67]: %px y = [i**10 for i in x]
552 In [67]: %px y = [i**10 for i in x]
553 Parallel execution on engines: [0, 1, 2, 3]
553 Parallel execution on engines: [0, 1, 2, 3]
554 Out[67]:
554 Out[67]:
555
555
556 In [68]: y = dview.gather('y')
556 In [68]: y = dview.gather('y')
557
557
558 In [69]: print y
558 In [69]: print y
559 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
559 [0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
560
560
561 Remote imports
561 Remote imports
562 --------------
562 --------------
563
563
564 Sometimes you will want to import packages both in your interactive session
564 Sometimes you will want to import packages both in your interactive session
565 and on your remote engines. This can be done with the :class:`ContextManager`
565 and on your remote engines. This can be done with the :class:`ContextManager`
566 created by a DirectView's :meth:`sync_imports` method:
566 created by a DirectView's :meth:`sync_imports` method:
567
567
568 .. sourcecode:: ipython
568 .. sourcecode:: ipython
569
569
570 In [69]: with dview.sync_imports():
570 In [69]: with dview.sync_imports():
571 ...: import numpy
571 ...: import numpy
572 importing numpy on engine(s)
572 importing numpy on engine(s)
573
573
574 Any imports made inside the block will also be performed on the view's engines.
574 Any imports made inside the block will also be performed on the view's engines.
575 sync_imports also takes a `local` boolean flag that defaults to True, which specifies
575 sync_imports also takes a `local` boolean flag that defaults to True, which specifies
576 whether the local imports should also be performed. However, support for `local=False`
576 whether the local imports should also be performed. However, support for `local=False`
577 has not been implemented, so only packages that can be imported locally will work
577 has not been implemented, so only packages that can be imported locally will work
578 this way.
578 this way.
579
579
580 You can also specify imports via the ``@require`` decorator. This is a decorator
580 You can also specify imports via the ``@require`` decorator. This is a decorator
581 designed for use in Dependencies, but can be used to handle remote imports as well.
581 designed for use in Dependencies, but can be used to handle remote imports as well.
582 Modules or module names passed to ``@require`` will be imported before the decorated
582 Modules or module names passed to ``@require`` will be imported before the decorated
583 function is called. If they cannot be imported, the decorated function will never
583 function is called. If they cannot be imported, the decorated function will never
584 execution, and will fail with an UnmetDependencyError.
584 execution, and will fail with an UnmetDependencyError.
585
585
586 .. sourcecode:: ipython
586 .. sourcecode:: ipython
587
587
588 In [69]: from IPython.parallel import require
588 In [69]: from IPython.parallel import require
589
589
590 In [70]: @require('re'):
590 In [70]: @require('re'):
591 ...: def findall(pat, x):
591 ...: def findall(pat, x):
592 ...: # re is guaranteed to be available
592 ...: # re is guaranteed to be available
593 ...: return re.findall(pat, x)
593 ...: return re.findall(pat, x)
594
594
595 # you can also pass modules themselves, that you already have locally:
595 # you can also pass modules themselves, that you already have locally:
596 In [71]: @require(time):
596 In [71]: @require(time):
597 ...: def wait(t):
597 ...: def wait(t):
598 ...: time.sleep(t)
598 ...: time.sleep(t)
599 ...: return t
599 ...: return t
600
600
601 .. _parallel_exceptions:
601 .. _parallel_exceptions:
602
602
603 Parallel exceptions
603 Parallel exceptions
604 -------------------
604 -------------------
605
605
606 In the multiengine interface, parallel commands can raise Python exceptions,
606 In the multiengine interface, parallel commands can raise Python exceptions,
607 just like serial commands. But, it is a little subtle, because a single
607 just like serial commands. But, it is a little subtle, because a single
608 parallel command can actually raise multiple exceptions (one for each engine
608 parallel command can actually raise multiple exceptions (one for each engine
609 the command was run on). To express this idea, we have a
609 the command was run on). To express this idea, we have a
610 :exc:`CompositeError` exception class that will be raised in most cases. The
610 :exc:`CompositeError` exception class that will be raised in most cases. The
611 :exc:`CompositeError` class is a special type of exception that wraps one or
611 :exc:`CompositeError` class is a special type of exception that wraps one or
612 more other types of exceptions. Here is how it works:
612 more other types of exceptions. Here is how it works:
613
613
614 .. sourcecode:: ipython
614 .. sourcecode:: ipython
615
615
616 In [76]: dview.block=True
616 In [76]: dview.block=True
617
617
618 In [77]: dview.execute('1/0')
618 In [77]: dview.execute('1/0')
619 ---------------------------------------------------------------------------
619 ---------------------------------------------------------------------------
620 CompositeError Traceback (most recent call last)
620 CompositeError Traceback (most recent call last)
621 /home/user/<ipython-input-10-5d56b303a66c> in <module>()
621 /home/user/<ipython-input-10-5d56b303a66c> in <module>()
622 ----> 1 dview.execute('1/0')
622 ----> 1 dview.execute('1/0')
623
623
624 /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
624 /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
625 591 default: self.block
625 591 default: self.block
626 592 """
626 592 """
627 --> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
627 --> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
628 594
628 594
629 595 def run(self, filename, targets=None, block=None):
629 595 def run(self, filename, targets=None, block=None):
630
630
631 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
631 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
632
632
633 /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
633 /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
634 55 def sync_results(f, self, *args, **kwargs):
634 55 def sync_results(f, self, *args, **kwargs):
635 56 """sync relevant results from self.client to our results attribute."""
635 56 """sync relevant results from self.client to our results attribute."""
636 ---> 57 ret = f(self, *args, **kwargs)
636 ---> 57 ret = f(self, *args, **kwargs)
637 58 delta = self.outstanding.difference(self.client.outstanding)
637 58 delta = self.outstanding.difference(self.client.outstanding)
638 59 completed = self.outstanding.intersection(delta)
638 59 completed = self.outstanding.intersection(delta)
639
639
640 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
640 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
641
641
642 /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
642 /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
643 44 n_previous = len(self.client.history)
643 44 n_previous = len(self.client.history)
644 45 try:
644 45 try:
645 ---> 46 ret = f(self, *args, **kwargs)
645 ---> 46 ret = f(self, *args, **kwargs)
646 47 finally:
646 47 finally:
647 48 nmsgs = len(self.client.history) - n_previous
647 48 nmsgs = len(self.client.history) - n_previous
648
648
649 /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
649 /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
650 529 if block:
650 529 if block:
651 530 try:
651 530 try:
652 --> 531 return ar.get()
652 --> 531 return ar.get()
653 532 except KeyboardInterrupt:
653 532 except KeyboardInterrupt:
654 533 pass
654 533 pass
655
655
656 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
656 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
657 101 return self._result
657 101 return self._result
658 102 else:
658 102 else:
659 --> 103 raise self._exception
659 --> 103 raise self._exception
660 104 else:
660 104 else:
661 105 raise error.TimeoutError("Result not ready.")
661 105 raise error.TimeoutError("Result not ready.")
662
662
663 CompositeError: one or more exceptions from call to method: _execute
663 CompositeError: one or more exceptions from call to method: _execute
664 [0:apply]: ZeroDivisionError: integer division or modulo by zero
664 [0:apply]: ZeroDivisionError: integer division or modulo by zero
665 [1:apply]: ZeroDivisionError: integer division or modulo by zero
665 [1:apply]: ZeroDivisionError: integer division or modulo by zero
666 [2:apply]: ZeroDivisionError: integer division or modulo by zero
666 [2:apply]: ZeroDivisionError: integer division or modulo by zero
667 [3:apply]: ZeroDivisionError: integer division or modulo by zero
667 [3:apply]: ZeroDivisionError: integer division or modulo by zero
668
668
669 Notice how the error message printed when :exc:`CompositeError` is raised has
669 Notice how the error message printed when :exc:`CompositeError` is raised has
670 information about the individual exceptions that were raised on each engine.
670 information about the individual exceptions that were raised on each engine.
671 If you want, you can even raise one of these original exceptions:
671 If you want, you can even raise one of these original exceptions:
672
672
673 .. sourcecode:: ipython
673 .. sourcecode:: ipython
674
674
675 In [80]: try:
675 In [80]: try:
676 ....: dview.execute('1/0')
676 ....: dview.execute('1/0')
677 ....: except parallel.error.CompositeError, e:
677 ....: except parallel.error.CompositeError, e:
678 ....: e.raise_exception()
678 ....: e.raise_exception()
679 ....:
679 ....:
680 ....:
680 ....:
681 ---------------------------------------------------------------------------
681 ---------------------------------------------------------------------------
682 RemoteError Traceback (most recent call last)
682 RemoteError Traceback (most recent call last)
683 /home/user/<ipython-input-17-8597e7e39858> in <module>()
683 /home/user/<ipython-input-17-8597e7e39858> in <module>()
684 2 dview.execute('1/0')
684 2 dview.execute('1/0')
685 3 except CompositeError as e:
685 3 except CompositeError as e:
686 ----> 4 e.raise_exception()
686 ----> 4 e.raise_exception()
687
687
688 /path/to/site-packages/IPython/parallel/error.pyc in raise_exception(self, excid)
688 /path/to/site-packages/IPython/parallel/error.pyc in raise_exception(self, excid)
689 266 raise IndexError("an exception with index %i does not exist"%excid)
689 266 raise IndexError("an exception with index %i does not exist"%excid)
690 267 else:
690 267 else:
691 --> 268 raise RemoteError(en, ev, etb, ei)
691 --> 268 raise RemoteError(en, ev, etb, ei)
692 269
692 269
693 270
693 270
694
694
695 RemoteError: ZeroDivisionError(integer division or modulo by zero)
695 RemoteError: ZeroDivisionError(integer division or modulo by zero)
696 Traceback (most recent call last):
696 Traceback (most recent call last):
697 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
697 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
698 exec code in working,working
698 exec code in working,working
699 File "<string>", line 1, in <module>
699 File "<string>", line 1, in <module>
700 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
700 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
701 exec code in globals()
701 exec code in globals()
702 File "<string>", line 1, in <module>
702 File "<string>", line 1, in <module>
703 ZeroDivisionError: integer division or modulo by zero
703 ZeroDivisionError: integer division or modulo by zero
704
704
705 If you are working in IPython, you can simple type ``%debug`` after one of
705 If you are working in IPython, you can simple type ``%debug`` after one of
706 these :exc:`CompositeError` exceptions is raised, and inspect the exception
706 these :exc:`CompositeError` exceptions is raised, and inspect the exception
707 instance:
707 instance:
708
708
709 .. sourcecode:: ipython
709 .. sourcecode:: ipython
710
710
711 In [81]: dview.execute('1/0')
711 In [81]: dview.execute('1/0')
712 ---------------------------------------------------------------------------
712 ---------------------------------------------------------------------------
713 CompositeError Traceback (most recent call last)
713 CompositeError Traceback (most recent call last)
714 /home/user/<ipython-input-10-5d56b303a66c> in <module>()
714 /home/user/<ipython-input-10-5d56b303a66c> in <module>()
715 ----> 1 dview.execute('1/0')
715 ----> 1 dview.execute('1/0')
716
716
717 /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
717 /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
718 591 default: self.block
718 591 default: self.block
719 592 """
719 592 """
720 --> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
720 --> 593 return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
721 594
721 594
722 595 def run(self, filename, targets=None, block=None):
722 595 def run(self, filename, targets=None, block=None):
723
723
724 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
724 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
725
725
726 /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
726 /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
727 55 def sync_results(f, self, *args, **kwargs):
727 55 def sync_results(f, self, *args, **kwargs):
728 56 """sync relevant results from self.client to our results attribute."""
728 56 """sync relevant results from self.client to our results attribute."""
729 ---> 57 ret = f(self, *args, **kwargs)
729 ---> 57 ret = f(self, *args, **kwargs)
730 58 delta = self.outstanding.difference(self.client.outstanding)
730 58 delta = self.outstanding.difference(self.client.outstanding)
731 59 completed = self.outstanding.intersection(delta)
731 59 completed = self.outstanding.intersection(delta)
732
732
733 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
733 /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
734
734
735 /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
735 /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
736 44 n_previous = len(self.client.history)
736 44 n_previous = len(self.client.history)
737 45 try:
737 45 try:
738 ---> 46 ret = f(self, *args, **kwargs)
738 ---> 46 ret = f(self, *args, **kwargs)
739 47 finally:
739 47 finally:
740 48 nmsgs = len(self.client.history) - n_previous
740 48 nmsgs = len(self.client.history) - n_previous
741
741
742 /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
742 /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
743 529 if block:
743 529 if block:
744 530 try:
744 530 try:
745 --> 531 return ar.get()
745 --> 531 return ar.get()
746 532 except KeyboardInterrupt:
746 532 except KeyboardInterrupt:
747 533 pass
747 533 pass
748
748
749 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
749 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
750 101 return self._result
750 101 return self._result
751 102 else:
751 102 else:
752 --> 103 raise self._exception
752 --> 103 raise self._exception
753 104 else:
753 104 else:
754 105 raise error.TimeoutError("Result not ready.")
754 105 raise error.TimeoutError("Result not ready.")
755
755
756 CompositeError: one or more exceptions from call to method: _execute
756 CompositeError: one or more exceptions from call to method: _execute
757 [0:apply]: ZeroDivisionError: integer division or modulo by zero
757 [0:apply]: ZeroDivisionError: integer division or modulo by zero
758 [1:apply]: ZeroDivisionError: integer division or modulo by zero
758 [1:apply]: ZeroDivisionError: integer division or modulo by zero
759 [2:apply]: ZeroDivisionError: integer division or modulo by zero
759 [2:apply]: ZeroDivisionError: integer division or modulo by zero
760 [3:apply]: ZeroDivisionError: integer division or modulo by zero
760 [3:apply]: ZeroDivisionError: integer division or modulo by zero
761
761
762 In [82]: %debug
762 In [82]: %debug
763 > /path/to/site-packages/IPython/parallel/client/asyncresult.py(103)get()
763 > /path/to/site-packages/IPython/parallel/client/asyncresult.py(103)get()
764 102 else:
764 102 else:
765 --> 103 raise self._exception
765 --> 103 raise self._exception
766 104 else:
766 104 else:
767
767
768 # With the debugger running, self._exception is the exceptions instance. We can tab complete
768 # With the debugger running, self._exception is the exceptions instance. We can tab complete
769 # on it and see the extra methods that are available.
769 # on it and see the extra methods that are available.
770 ipdb> self._exception.<tab>
770 ipdb> self._exception.<tab>
771 e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args
771 e.__class__ e.__getitem__ e.__new__ e.__setstate__ e.args
772 e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist
772 e.__delattr__ e.__getslice__ e.__reduce__ e.__str__ e.elist
773 e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message
773 e.__dict__ e.__hash__ e.__reduce_ex__ e.__weakref__ e.message
774 e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks
774 e.__doc__ e.__init__ e.__repr__ e._get_engine_str e.print_tracebacks
775 e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception
775 e.__getattribute__ e.__module__ e.__setattr__ e._get_traceback e.raise_exception
776 ipdb> self._exception.print_tracebacks()
776 ipdb> self._exception.print_tracebacks()
777 [0:apply]:
777 [0:apply]:
778 Traceback (most recent call last):
778 Traceback (most recent call last):
779 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
779 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
780 exec code in working,working
780 exec code in working,working
781 File "<string>", line 1, in <module>
781 File "<string>", line 1, in <module>
782 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
782 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
783 exec code in globals()
783 exec code in globals()
784 File "<string>", line 1, in <module>
784 File "<string>", line 1, in <module>
785 ZeroDivisionError: integer division or modulo by zero
785 ZeroDivisionError: integer division or modulo by zero
786
786
787
787
788 [1:apply]:
788 [1:apply]:
789 Traceback (most recent call last):
789 Traceback (most recent call last):
790 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
790 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
791 exec code in working,working
791 exec code in working,working
792 File "<string>", line 1, in <module>
792 File "<string>", line 1, in <module>
793 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
793 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
794 exec code in globals()
794 exec code in globals()
795 File "<string>", line 1, in <module>
795 File "<string>", line 1, in <module>
796 ZeroDivisionError: integer division or modulo by zero
796 ZeroDivisionError: integer division or modulo by zero
797
797
798
798
799 [2:apply]:
799 [2:apply]:
800 Traceback (most recent call last):
800 Traceback (most recent call last):
801 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
801 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
802 exec code in working,working
802 exec code in working,working
803 File "<string>", line 1, in <module>
803 File "<string>", line 1, in <module>
804 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
804 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
805 exec code in globals()
805 exec code in globals()
806 File "<string>", line 1, in <module>
806 File "<string>", line 1, in <module>
807 ZeroDivisionError: integer division or modulo by zero
807 ZeroDivisionError: integer division or modulo by zero
808
808
809
809
810 [3:apply]:
810 [3:apply]:
811 Traceback (most recent call last):
811 Traceback (most recent call last):
812 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
812 File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
813 exec code in working,working
813 exec code in working,working
814 File "<string>", line 1, in <module>
814 File "<string>", line 1, in <module>
815 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
815 File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
816 exec code in globals()
816 exec code in globals()
817 File "<string>", line 1, in <module>
817 File "<string>", line 1, in <module>
818 ZeroDivisionError: integer division or modulo by zero
818 ZeroDivisionError: integer division or modulo by zero
819
819
820
820
821 All of this same error handling magic even works in non-blocking mode:
821 All of this same error handling magic even works in non-blocking mode:
822
822
823 .. sourcecode:: ipython
823 .. sourcecode:: ipython
824
824
825 In [83]: dview.block=False
825 In [83]: dview.block=False
826
826
827 In [84]: ar = dview.execute('1/0')
827 In [84]: ar = dview.execute('1/0')
828
828
829 In [85]: ar.get()
829 In [85]: ar.get()
830 ---------------------------------------------------------------------------
830 ---------------------------------------------------------------------------
831 CompositeError Traceback (most recent call last)
831 CompositeError Traceback (most recent call last)
832 /home/user/<ipython-input-21-8531eb3d26fb> in <module>()
832 /home/user/<ipython-input-21-8531eb3d26fb> in <module>()
833 ----> 1 ar.get()
833 ----> 1 ar.get()
834
834
835 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
835 /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
836 101 return self._result
836 101 return self._result
837 102 else:
837 102 else:
838 --> 103 raise self._exception
838 --> 103 raise self._exception
839 104 else:
839 104 else:
840 105 raise error.TimeoutError("Result not ready.")
840 105 raise error.TimeoutError("Result not ready.")
841
841
842 CompositeError: one or more exceptions from call to method: _execute
842 CompositeError: one or more exceptions from call to method: _execute
843 [0:apply]: ZeroDivisionError: integer division or modulo by zero
843 [0:apply]: ZeroDivisionError: integer division or modulo by zero
844 [1:apply]: ZeroDivisionError: integer division or modulo by zero
844 [1:apply]: ZeroDivisionError: integer division or modulo by zero
845 [2:apply]: ZeroDivisionError: integer division or modulo by zero
845 [2:apply]: ZeroDivisionError: integer division or modulo by zero
846 [3:apply]: ZeroDivisionError: integer division or modulo by zero
846 [3:apply]: ZeroDivisionError: integer division or modulo by zero
847
847
General Comments 0
You need to be logged in to leave comments. Login now