Show More
@@ -1,390 +1,393 b'' | |||
|
1 | 1 | .. _changes: |
|
2 | 2 | |
|
3 | 3 | ========== |
|
4 | 4 | What's new |
|
5 | 5 | ========== |
|
6 | 6 | |
|
7 | 7 | .. contents:: |
|
8 | 8 | .. |
|
9 | 9 | 1 Release 0.9.1 |
|
10 | 10 | 2 Release 0.9 |
|
11 | 11 | 2.1 New features |
|
12 | 12 | 2.2 Bug fixes |
|
13 | 13 | 2.3 Backwards incompatible changes |
|
14 | 14 | 2.4 Changes merged in from IPython1 |
|
15 | 15 | 2.4.1 New features |
|
16 | 16 | 2.4.2 Bug fixes |
|
17 | 17 | 2.4.3 Backwards incompatible changes |
|
18 | 18 | 3 Release 0.8.4 |
|
19 | 19 | 4 Release 0.8.3 |
|
20 | 20 | 5 Release 0.8.2 |
|
21 | 21 | 6 Older releases |
|
22 | 22 | .. |
|
23 | 23 | |
|
24 | 24 | Release dev |
|
25 | 25 | =========== |
|
26 | 26 | |
|
27 | 27 | New features |
|
28 | 28 | ------------ |
|
29 | 29 | |
|
30 | * The wonderful TextMate editor can now be used with %edit on OS X. Thanks | |
|
31 | to Matt Foster for this patch. | |
|
32 | ||
|
30 | 33 | * Fully refactored :command:`ipcluster` command line program for starting |
|
31 | 34 | IPython clusters. This new version is a complete rewrite and 1) is fully |
|
32 | 35 | cross platform (we now use Twisted's process management), 2) has much |
|
33 | 36 | improved performance, 3) uses subcommands for different types of clusters, |
|
34 | 37 | 4) uses argparse for parsing command line options, 5) has better support |
|
35 | 38 | for starting clusters using :command:`mpirun`, 6) has experimental support |
|
36 | 39 | for starting engines using PBS. However, this new version of ipcluster |
|
37 | 40 | should be considered a technology preview. We plan on changing the API |
|
38 | 41 | in significant ways before it is final. |
|
39 | 42 | |
|
40 | 43 | * The :mod:`argparse` module has been added to :mod:`IPython.external`. |
|
41 | 44 | |
|
42 | 45 | * Fully description of the security model added to the docs. |
|
43 | 46 | |
|
44 | 47 | * cd completer: show bookmarks if no other completions are available. |
|
45 | 48 | |
|
46 | 49 | * sh profile: easy way to give 'title' to prompt: assign to variable |
|
47 | 50 | '_prompt_title'. It looks like this:: |
|
48 | 51 | |
|
49 | 52 | [~]|1> _prompt_title = 'sudo!' |
|
50 | 53 | sudo![~]|2> |
|
51 | 54 | |
|
52 | 55 | * %edit: If you do '%edit pasted_block', pasted_block |
|
53 | 56 | variable gets updated with new data (so repeated |
|
54 | 57 | editing makes sense) |
|
55 | 58 | |
|
56 | 59 | Bug fixes |
|
57 | 60 | --------- |
|
58 | 61 | |
|
59 | 62 | * The ipengine and ipcontroller scripts now handle missing furl files |
|
60 | 63 | more gracefully by giving better error messages. |
|
61 | 64 | |
|
62 | 65 | * %rehashx: Aliases no longer contain dots. python3.0 binary |
|
63 | 66 | will create alias python30. Fixes: |
|
64 | 67 | #259716 "commands with dots in them don't work" |
|
65 | 68 | |
|
66 | 69 | * %cpaste: %cpaste -r repeats the last pasted block. |
|
67 | 70 | The block is assigned to pasted_block even if code |
|
68 | 71 | raises exception. |
|
69 | 72 | |
|
70 | 73 | Backwards incompatible changes |
|
71 | 74 | ------------------------------ |
|
72 | 75 | |
|
73 | 76 | * The controller now has a ``-r`` flag that needs to be used if you want to |
|
74 | 77 | reuse existing furl files. Otherwise they are deleted (the default). |
|
75 | 78 | |
|
76 | 79 | * Remove ipy_leo.py. "easy_install ipython-extension" to get it. |
|
77 | 80 | (done to decouple it from ipython release cycle) |
|
78 | 81 | |
|
79 | 82 | |
|
80 | 83 | |
|
81 | 84 | Release 0.9.1 |
|
82 | 85 | ============= |
|
83 | 86 | |
|
84 | 87 | This release was quickly made to restore compatibility with Python 2.4, which |
|
85 | 88 | version 0.9 accidentally broke. No new features were introduced, other than |
|
86 | 89 | some additional testing support for internal use. |
|
87 | 90 | |
|
88 | 91 | |
|
89 | 92 | Release 0.9 |
|
90 | 93 | =========== |
|
91 | 94 | |
|
92 | 95 | New features |
|
93 | 96 | ------------ |
|
94 | 97 | |
|
95 | 98 | * All furl files and security certificates are now put in a read-only |
|
96 | 99 | directory named ~./ipython/security. |
|
97 | 100 | |
|
98 | 101 | * A single function :func:`get_ipython_dir`, in :mod:`IPython.genutils` that |
|
99 | 102 | determines the user's IPython directory in a robust manner. |
|
100 | 103 | |
|
101 | 104 | * Laurent's WX application has been given a top-level script called |
|
102 | 105 | ipython-wx, and it has received numerous fixes. We expect this code to be |
|
103 | 106 | architecturally better integrated with Gael's WX 'ipython widget' over the |
|
104 | 107 | next few releases. |
|
105 | 108 | |
|
106 | 109 | * The Editor synchronization work by Vivian De Smedt has been merged in. This |
|
107 | 110 | code adds a number of new editor hooks to synchronize with editors under |
|
108 | 111 | Windows. |
|
109 | 112 | |
|
110 | 113 | * A new, still experimental but highly functional, WX shell by Gael Varoquaux. |
|
111 | 114 | This work was sponsored by Enthought, and while it's still very new, it is |
|
112 | 115 | based on a more cleanly organized arhictecture of the various IPython |
|
113 | 116 | components. We will continue to develop this over the next few releases as a |
|
114 | 117 | model for GUI components that use IPython. |
|
115 | 118 | |
|
116 | 119 | * Another GUI frontend, Cocoa based (Cocoa is the OSX native GUI framework), |
|
117 | 120 | authored by Barry Wark. Currently the WX and the Cocoa ones have slightly |
|
118 | 121 | different internal organizations, but the whole team is working on finding |
|
119 | 122 | what the right abstraction points are for a unified codebase. |
|
120 | 123 | |
|
121 | 124 | * As part of the frontend work, Barry Wark also implemented an experimental |
|
122 | 125 | event notification system that various ipython components can use. In the |
|
123 | 126 | next release the implications and use patterns of this system regarding the |
|
124 | 127 | various GUI options will be worked out. |
|
125 | 128 | |
|
126 | 129 | * IPython finally has a full test system, that can test docstrings with |
|
127 | 130 | IPython-specific functionality. There are still a few pieces missing for it |
|
128 | 131 | to be widely accessible to all users (so they can run the test suite at any |
|
129 | 132 | time and report problems), but it now works for the developers. We are |
|
130 | 133 | working hard on continuing to improve it, as this was probably IPython's |
|
131 | 134 | major Achilles heel (the lack of proper test coverage made it effectively |
|
132 | 135 | impossible to do large-scale refactoring). The full test suite can now |
|
133 | 136 | be run using the :command:`iptest` command line program. |
|
134 | 137 | |
|
135 | 138 | * The notion of a task has been completely reworked. An `ITask` interface has |
|
136 | 139 | been created. This interface defines the methods that tasks need to |
|
137 | 140 | implement. These methods are now responsible for things like submitting |
|
138 | 141 | tasks and processing results. There are two basic task types: |
|
139 | 142 | :class:`IPython.kernel.task.StringTask` (this is the old `Task` object, but |
|
140 | 143 | renamed) and the new :class:`IPython.kernel.task.MapTask`, which is based on |
|
141 | 144 | a function. |
|
142 | 145 | |
|
143 | 146 | * A new interface, :class:`IPython.kernel.mapper.IMapper` has been defined to |
|
144 | 147 | standardize the idea of a `map` method. This interface has a single `map` |
|
145 | 148 | method that has the same syntax as the built-in `map`. We have also defined |
|
146 | 149 | a `mapper` factory interface that creates objects that implement |
|
147 | 150 | :class:`IPython.kernel.mapper.IMapper` for different controllers. Both the |
|
148 | 151 | multiengine and task controller now have mapping capabilties. |
|
149 | 152 | |
|
150 | 153 | * The parallel function capabilities have been reworks. The major changes are |
|
151 | 154 | that i) there is now an `@parallel` magic that creates parallel functions, |
|
152 | 155 | ii) the syntax for mulitple variable follows that of `map`, iii) both the |
|
153 | 156 | multiengine and task controller now have a parallel function implementation. |
|
154 | 157 | |
|
155 | 158 | * All of the parallel computing capabilities from `ipython1-dev` have been |
|
156 | 159 | merged into IPython proper. This resulted in the following new subpackages: |
|
157 | 160 | :mod:`IPython.kernel`, :mod:`IPython.kernel.core`, :mod:`IPython.config`, |
|
158 | 161 | :mod:`IPython.tools` and :mod:`IPython.testing`. |
|
159 | 162 | |
|
160 | 163 | * As part of merging in the `ipython1-dev` stuff, the `setup.py` script and |
|
161 | 164 | friends have been completely refactored. Now we are checking for |
|
162 | 165 | dependencies using the approach that matplotlib uses. |
|
163 | 166 | |
|
164 | 167 | * The documentation has been completely reorganized to accept the |
|
165 | 168 | documentation from `ipython1-dev`. |
|
166 | 169 | |
|
167 | 170 | * We have switched to using Foolscap for all of our network protocols in |
|
168 | 171 | :mod:`IPython.kernel`. This gives us secure connections that are both |
|
169 | 172 | encrypted and authenticated. |
|
170 | 173 | |
|
171 | 174 | * We have a brand new `COPYING.txt` files that describes the IPython license |
|
172 | 175 | and copyright. The biggest change is that we are putting "The IPython |
|
173 | 176 | Development Team" as the copyright holder. We give more details about |
|
174 | 177 | exactly what this means in this file. All developer should read this and use |
|
175 | 178 | the new banner in all IPython source code files. |
|
176 | 179 | |
|
177 | 180 | * sh profile: ./foo runs foo as system command, no need to do !./foo anymore |
|
178 | 181 | |
|
179 | 182 | * String lists now support ``sort(field, nums = True)`` method (to easily sort |
|
180 | 183 | system command output). Try it with ``a = !ls -l ; a.sort(1, nums=1)``. |
|
181 | 184 | |
|
182 | 185 | * '%cpaste foo' now assigns the pasted block as string list, instead of string |
|
183 | 186 | |
|
184 | 187 | * The ipcluster script now run by default with no security. This is done |
|
185 | 188 | because the main usage of the script is for starting things on localhost. |
|
186 | 189 | Eventually when ipcluster is able to start things on other hosts, we will put |
|
187 | 190 | security back. |
|
188 | 191 | |
|
189 | 192 | * 'cd --foo' searches directory history for string foo, and jumps to that dir. |
|
190 | 193 | Last part of dir name is checked first. If no matches for that are found, |
|
191 | 194 | look at the whole path. |
|
192 | 195 | |
|
193 | 196 | |
|
194 | 197 | Bug fixes |
|
195 | 198 | --------- |
|
196 | 199 | |
|
197 | 200 | * The Windows installer has been fixed. Now all IPython scripts have ``.bat`` |
|
198 | 201 | versions created. Also, the Start Menu shortcuts have been updated. |
|
199 | 202 | |
|
200 | 203 | * The colors escapes in the multiengine client are now turned off on win32 as |
|
201 | 204 | they don't print correctly. |
|
202 | 205 | |
|
203 | 206 | * The :mod:`IPython.kernel.scripts.ipengine` script was exec'ing |
|
204 | 207 | mpi_import_statement incorrectly, which was leading the engine to crash when |
|
205 | 208 | mpi was enabled. |
|
206 | 209 | |
|
207 | 210 | * A few subpackages had missing ``__init__.py`` files. |
|
208 | 211 | |
|
209 | 212 | * The documentation is only created if Sphinx is found. Previously, the |
|
210 | 213 | ``setup.py`` script would fail if it was missing. |
|
211 | 214 | |
|
212 | 215 | * Greedy ``cd`` completion has been disabled again (it was enabled in 0.8.4) as |
|
213 | 216 | it caused problems on certain platforms. |
|
214 | 217 | |
|
215 | 218 | |
|
216 | 219 | Backwards incompatible changes |
|
217 | 220 | ------------------------------ |
|
218 | 221 | |
|
219 | 222 | * The ``clusterfile`` options of the :command:`ipcluster` command has been |
|
220 | 223 | removed as it was not working and it will be replaced soon by something much |
|
221 | 224 | more robust. |
|
222 | 225 | |
|
223 | 226 | * The :mod:`IPython.kernel` configuration now properly find the user's |
|
224 | 227 | IPython directory. |
|
225 | 228 | |
|
226 | 229 | * In ipapi, the :func:`make_user_ns` function has been replaced with |
|
227 | 230 | :func:`make_user_namespaces`, to support dict subclasses in namespace |
|
228 | 231 | creation. |
|
229 | 232 | |
|
230 | 233 | * :class:`IPython.kernel.client.Task` has been renamed |
|
231 | 234 | :class:`IPython.kernel.client.StringTask` to make way for new task types. |
|
232 | 235 | |
|
233 | 236 | * The keyword argument `style` has been renamed `dist` in `scatter`, `gather` |
|
234 | 237 | and `map`. |
|
235 | 238 | |
|
236 | 239 | * Renamed the values that the rename `dist` keyword argument can have from |
|
237 | 240 | `'basic'` to `'b'`. |
|
238 | 241 | |
|
239 | 242 | * IPython has a larger set of dependencies if you want all of its capabilities. |
|
240 | 243 | See the `setup.py` script for details. |
|
241 | 244 | |
|
242 | 245 | * The constructors for :class:`IPython.kernel.client.MultiEngineClient` and |
|
243 | 246 | :class:`IPython.kernel.client.TaskClient` no longer take the (ip,port) tuple. |
|
244 | 247 | Instead they take the filename of a file that contains the FURL for that |
|
245 | 248 | client. If the FURL file is in your IPYTHONDIR, it will be found automatically |
|
246 | 249 | and the constructor can be left empty. |
|
247 | 250 | |
|
248 | 251 | * The asynchronous clients in :mod:`IPython.kernel.asyncclient` are now created |
|
249 | 252 | using the factory functions :func:`get_multiengine_client` and |
|
250 | 253 | :func:`get_task_client`. These return a `Deferred` to the actual client. |
|
251 | 254 | |
|
252 | 255 | * The command line options to `ipcontroller` and `ipengine` have changed to |
|
253 | 256 | reflect the new Foolscap network protocol and the FURL files. Please see the |
|
254 | 257 | help for these scripts for details. |
|
255 | 258 | |
|
256 | 259 | * The configuration files for the kernel have changed because of the Foolscap |
|
257 | 260 | stuff. If you were using custom config files before, you should delete them |
|
258 | 261 | and regenerate new ones. |
|
259 | 262 | |
|
260 | 263 | Changes merged in from IPython1 |
|
261 | 264 | ------------------------------- |
|
262 | 265 | |
|
263 | 266 | New features |
|
264 | 267 | ............ |
|
265 | 268 | |
|
266 | 269 | * Much improved ``setup.py`` and ``setupegg.py`` scripts. Because Twisted and |
|
267 | 270 | zope.interface are now easy installable, we can declare them as dependencies |
|
268 | 271 | in our setupegg.py script. |
|
269 | 272 | |
|
270 | 273 | * IPython is now compatible with Twisted 2.5.0 and 8.x. |
|
271 | 274 | |
|
272 | 275 | * Added a new example of how to use :mod:`ipython1.kernel.asynclient`. |
|
273 | 276 | |
|
274 | 277 | * Initial draft of a process daemon in :mod:`ipython1.daemon`. This has not |
|
275 | 278 | been merged into IPython and is still in `ipython1-dev`. |
|
276 | 279 | |
|
277 | 280 | * The ``TaskController`` now has methods for getting the queue status. |
|
278 | 281 | |
|
279 | 282 | * The ``TaskResult`` objects not have information about how long the task |
|
280 | 283 | took to run. |
|
281 | 284 | |
|
282 | 285 | * We are attaching additional attributes to exceptions ``(_ipython_*)`` that |
|
283 | 286 | we use to carry additional info around. |
|
284 | 287 | |
|
285 | 288 | * New top-level module :mod:`asyncclient` that has asynchronous versions (that |
|
286 | 289 | return deferreds) of the client classes. This is designed to users who want |
|
287 | 290 | to run their own Twisted reactor. |
|
288 | 291 | |
|
289 | 292 | * All the clients in :mod:`client` are now based on Twisted. This is done by |
|
290 | 293 | running the Twisted reactor in a separate thread and using the |
|
291 | 294 | :func:`blockingCallFromThread` function that is in recent versions of Twisted. |
|
292 | 295 | |
|
293 | 296 | * Functions can now be pushed/pulled to/from engines using |
|
294 | 297 | :meth:`MultiEngineClient.push_function` and |
|
295 | 298 | :meth:`MultiEngineClient.pull_function`. |
|
296 | 299 | |
|
297 | 300 | * Gather/scatter are now implemented in the client to reduce the work load |
|
298 | 301 | of the controller and improve performance. |
|
299 | 302 | |
|
300 | 303 | * Complete rewrite of the IPython docuementation. All of the documentation |
|
301 | 304 | from the IPython website has been moved into docs/source as restructured |
|
302 | 305 | text documents. PDF and HTML documentation are being generated using |
|
303 | 306 | Sphinx. |
|
304 | 307 | |
|
305 | 308 | * New developer oriented documentation: development guidelines and roadmap. |
|
306 | 309 | |
|
307 | 310 | * Traditional ``ChangeLog`` has been changed to a more useful ``changes.txt`` |
|
308 | 311 | file that is organized by release and is meant to provide something more |
|
309 | 312 | relevant for users. |
|
310 | 313 | |
|
311 | 314 | Bug fixes |
|
312 | 315 | ......... |
|
313 | 316 | |
|
314 | 317 | * Created a proper ``MANIFEST.in`` file to create source distributions. |
|
315 | 318 | |
|
316 | 319 | * Fixed a bug in the ``MultiEngine`` interface. Previously, multi-engine |
|
317 | 320 | actions were being collected with a :class:`DeferredList` with |
|
318 | 321 | ``fireononeerrback=1``. This meant that methods were returning |
|
319 | 322 | before all engines had given their results. This was causing extremely odd |
|
320 | 323 | bugs in certain cases. To fix this problem, we have 1) set |
|
321 | 324 | ``fireononeerrback=0`` to make sure all results (or exceptions) are in |
|
322 | 325 | before returning and 2) introduced a :exc:`CompositeError` exception |
|
323 | 326 | that wraps all of the engine exceptions. This is a huge change as it means |
|
324 | 327 | that users will have to catch :exc:`CompositeError` rather than the actual |
|
325 | 328 | exception. |
|
326 | 329 | |
|
327 | 330 | Backwards incompatible changes |
|
328 | 331 | .............................. |
|
329 | 332 | |
|
330 | 333 | * All names have been renamed to conform to the lowercase_with_underscore |
|
331 | 334 | convention. This will require users to change references to all names like |
|
332 | 335 | ``queueStatus`` to ``queue_status``. |
|
333 | 336 | |
|
334 | 337 | * Previously, methods like :meth:`MultiEngineClient.push` and |
|
335 | 338 | :meth:`MultiEngineClient.push` used ``*args`` and ``**kwargs``. This was |
|
336 | 339 | becoming a problem as we weren't able to introduce new keyword arguments into |
|
337 | 340 | the API. Now these methods simple take a dict or sequence. This has also |
|
338 | 341 | allowed us to get rid of the ``*All`` methods like :meth:`pushAll` and |
|
339 | 342 | :meth:`pullAll`. These things are now handled with the ``targets`` keyword |
|
340 | 343 | argument that defaults to ``'all'``. |
|
341 | 344 | |
|
342 | 345 | * The :attr:`MultiEngineClient.magicTargets` has been renamed to |
|
343 | 346 | :attr:`MultiEngineClient.targets`. |
|
344 | 347 | |
|
345 | 348 | * All methods in the MultiEngine interface now accept the optional keyword |
|
346 | 349 | argument ``block``. |
|
347 | 350 | |
|
348 | 351 | * Renamed :class:`RemoteController` to :class:`MultiEngineClient` and |
|
349 | 352 | :class:`TaskController` to :class:`TaskClient`. |
|
350 | 353 | |
|
351 | 354 | * Renamed the top-level module from :mod:`api` to :mod:`client`. |
|
352 | 355 | |
|
353 | 356 | * Most methods in the multiengine interface now raise a :exc:`CompositeError` |
|
354 | 357 | exception that wraps the user's exceptions, rather than just raising the raw |
|
355 | 358 | user's exception. |
|
356 | 359 | |
|
357 | 360 | * Changed the ``setupNS`` and ``resultNames`` in the ``Task`` class to ``push`` |
|
358 | 361 | and ``pull``. |
|
359 | 362 | |
|
360 | 363 | |
|
361 | 364 | Release 0.8.4 |
|
362 | 365 | ============= |
|
363 | 366 | |
|
364 | 367 | This was a quick release to fix an unfortunate bug that slipped into the 0.8.3 |
|
365 | 368 | release. The ``--twisted`` option was disabled, as it turned out to be broken |
|
366 | 369 | across several platforms. |
|
367 | 370 | |
|
368 | 371 | |
|
369 | 372 | Release 0.8.3 |
|
370 | 373 | ============= |
|
371 | 374 | |
|
372 | 375 | * pydb is now disabled by default (due to %run -d problems). You can enable |
|
373 | 376 | it by passing -pydb command line argument to IPython. Note that setting |
|
374 | 377 | it in config file won't work. |
|
375 | 378 | |
|
376 | 379 | |
|
377 | 380 | Release 0.8.2 |
|
378 | 381 | ============= |
|
379 | 382 | |
|
380 | 383 | * %pushd/%popd behave differently; now "pushd /foo" pushes CURRENT directory |
|
381 | 384 | and jumps to /foo. The current behaviour is closer to the documented |
|
382 | 385 | behaviour, and should not trip anyone. |
|
383 | 386 | |
|
384 | 387 | |
|
385 | 388 | Older releases |
|
386 | 389 | ============== |
|
387 | 390 | |
|
388 | 391 | Changes in earlier releases of IPython are described in the older file |
|
389 | 392 | ``ChangeLog``. Please refer to this document for details. |
|
390 | 393 |
@@ -1,78 +1,155 b'' | |||
|
1 | 1 | .. _parallelmpi: |
|
2 | 2 | |
|
3 | 3 | ======================= |
|
4 | 4 | Using MPI with IPython |
|
5 | 5 | ======================= |
|
6 | 6 | |
|
7 | 7 | Often, a parallel algorithm will require moving data between the engines. One way of accomplishing this is by doing a pull and then a push using the multiengine client. However, this will be slow as all the data has to go through the controller to the client and then back through the controller, to its final destination. |
|
8 | 8 | |
|
9 | 9 | A much better way of moving data between engines is to use a message passing library, such as the Message Passing Interface (`MPI`_). IPython's parallel computing architecture has been designed from the ground up to integrate with `MPI`_. This document describe how to use MPI with IPython. |
|
10 | 10 | |
|
11 | 11 | Additional installation requirements |
|
12 | 12 | ==================================== |
|
13 | 13 | |
|
14 | 14 | If you want to use MPI with IPython, you will need to install: |
|
15 | 15 | |
|
16 | 16 | * A standard MPI implementation such as `Open MPI`_ or MPICH. |
|
17 | 17 | * The `mpi4py`_ package. |
|
18 | 18 | |
|
19 | 19 | .. note:: |
|
20 | 20 | |
|
21 | 21 | The `mpi4py`_ package is not a strict requirement. However, you need to |
|
22 | 22 | have *some* way of calling MPI from Python. You also need some way of |
|
23 |
making sure that `MPI_Init` is called when the IPython engines start |
|
|
24 |
There are a number of ways of doing this and a good number of |
|
|
25 |
subtleties. We highly recommend just using `mpi4py`_ as it |
|
|
26 |
most of these problems. If you want to do something |
|
|
27 | and we can help you get started. | |
|
23 | making sure that :func:`MPI_Init` is called when the IPython engines start | |
|
24 | up. There are a number of ways of doing this and a good number of | |
|
25 | associated subtleties. We highly recommend just using `mpi4py`_ as it | |
|
26 | takes care of most of these problems. If you want to do something | |
|
27 | different, let us know and we can help you get started. | |
|
28 | 28 | |
|
29 | 29 | Starting the engines with MPI enabled |
|
30 | 30 | ===================================== |
|
31 | 31 | |
|
32 | 32 | To use code that calls `MPI`_, there are typically two things that `MPI`_ requires. |
|
33 | 33 | |
|
34 | 34 | 1. The process that wants to call `MPI`_ must be started using |
|
35 | 35 | :command:`mpirun` or a batch system (like PBS) that has `MPI`_ support. |
|
36 | 2. Once the process starts, it must call `MPI_Init`. | |
|
36 | 2. Once the process starts, it must call :func:`MPI_Init`. | |
|
37 | 37 | |
|
38 | 38 | There are a couple of ways that you can start the IPython engines and get these things to happen. |
|
39 | 39 | |
|
40 | 40 | Manual starting using :command:`mpirun` |
|
41 | 41 | --------------------------------------- |
|
42 | 42 | |
|
43 | 43 | If you want to start the IPython engines using the :command:`mpirun`, just do:: |
|
44 | 44 | |
|
45 | 45 | $ mpirun -n 4 ipengine --mpi=mpi4py |
|
46 | 46 | |
|
47 | 47 | This requires that you already have a controller running. We also have built |
|
48 | 48 | in support for `PyTrilinos`_, which can be used (assuming `PyTrilinos`_ is |
|
49 | 49 | installed) by starting the engines with:: |
|
50 | 50 | |
|
51 | 51 | mpirun -n 4 ipengine --mpi=pytrilinos |
|
52 | 52 | |
|
53 | 53 | Automatic starting using :command:`mpirun` and :command:`ipcluster` |
|
54 | 54 | ------------------------------------------------------------------- |
|
55 | 55 | |
|
56 | 56 | The easiest approach is to use the `mpirun` mode of :command:`ipcluster`, which will first start a controller and then a set of engines using :command:`mpirun`:: |
|
57 | 57 | |
|
58 | 58 | $ ipcluster mpirun -n 4 |
|
59 | 59 | |
|
60 | 60 | Automatic starting using PBS and :command:`ipcluster` |
|
61 | 61 | ----------------------------------------------------- |
|
62 | 62 | |
|
63 | 63 | The :command:`ipcluster` command also has built-in integration with PBS. For more information on this approach, see our documentation on :ref:`ipcluster <parallel_process>`. |
|
64 | 64 | |
|
65 | 65 | Actually using MPI |
|
66 | 66 | ================== |
|
67 | 67 | |
|
68 | Once the engines are running with `MPI`_ enabled, you are ready to go. You can now call any code that uses MPI in the IPython engines. And all of this can be done interactively. | |
|
68 | Once the engines are running with `MPI`_ enabled, you are ready to go. You can now call any code that uses MPI in the IPython engines. And, all of this can be done interactively. Here we show a simple example that uses `mpi4py`_. | |
|
69 | ||
|
70 | First, lets define a simply function that uses MPI to calculate the sum of a distributed array. Save the following text in a file called :file:`psum.py`:: | |
|
71 | ||
|
72 | from mpi4py import MPI | |
|
73 | import numpy as np | |
|
74 | ||
|
75 | def psum(a): | |
|
76 | s = np.sum(a) | |
|
77 | return MPI.COMM_WORLD.Allreduce(s,MPI.SUM) | |
|
78 | ||
|
79 | Now, start an IPython cluster in the same directory as :file:`psum.py`:: | |
|
80 | ||
|
81 | $ ipcluster mpirun -n 4 | |
|
82 | ||
|
83 | Finally, connect to the cluster and use this function interactively. In this case, we create a random array on each engine and sum up all the random arrays using our :func:`psum` function:: | |
|
84 | ||
|
85 | In [1]: from IPython.kernel import client | |
|
86 | ||
|
87 | In [2]: mec = client.MultiEngineClient() | |
|
88 | ||
|
89 | In [3]: mec.activate() | |
|
90 | ||
|
91 | In [4]: px import numpy as np | |
|
92 | Parallel execution on engines: all | |
|
93 | Out[4]: | |
|
94 | <Results List> | |
|
95 | [0] In [13]: import numpy as np | |
|
96 | [1] In [13]: import numpy as np | |
|
97 | [2] In [13]: import numpy as np | |
|
98 | [3] In [13]: import numpy as np | |
|
99 | ||
|
100 | In [6]: px a = np.random.rand(100) | |
|
101 | Parallel execution on engines: all | |
|
102 | Out[6]: | |
|
103 | <Results List> | |
|
104 | [0] In [15]: a = np.random.rand(100) | |
|
105 | [1] In [15]: a = np.random.rand(100) | |
|
106 | [2] In [15]: a = np.random.rand(100) | |
|
107 | [3] In [15]: a = np.random.rand(100) | |
|
108 | ||
|
109 | In [7]: px from psum import psum | |
|
110 | Parallel execution on engines: all | |
|
111 | Out[7]: | |
|
112 | <Results List> | |
|
113 | [0] In [16]: from psum import psum | |
|
114 | [1] In [16]: from psum import psum | |
|
115 | [2] In [16]: from psum import psum | |
|
116 | [3] In [16]: from psum import psum | |
|
117 | ||
|
118 | In [8]: px s = psum(a) | |
|
119 | Parallel execution on engines: all | |
|
120 | Out[8]: | |
|
121 | <Results List> | |
|
122 | [0] In [17]: s = psum(a) | |
|
123 | [1] In [17]: s = psum(a) | |
|
124 | [2] In [17]: s = psum(a) | |
|
125 | [3] In [17]: s = psum(a) | |
|
126 | ||
|
127 | In [9]: px print s | |
|
128 | Parallel execution on engines: all | |
|
129 | Out[9]: | |
|
130 | <Results List> | |
|
131 | [0] In [18]: print s | |
|
132 | [0] Out[18]: 187.451545803 | |
|
133 | ||
|
134 | [1] In [18]: print s | |
|
135 | [1] Out[18]: 187.451545803 | |
|
136 | ||
|
137 | [2] In [18]: print s | |
|
138 | [2] Out[18]: 187.451545803 | |
|
139 | ||
|
140 | [3] In [18]: print s | |
|
141 | [3] Out[18]: 187.451545803 | |
|
142 | ||
|
143 | Any Python code that makes calls to MPI [MPIref]_ can be used in this manner. | |
|
69 | 144 | |
|
70 | 145 | Complications |
|
71 | 146 | ============= |
|
72 | 147 | |
|
73 | 148 | Talk about how some older MPI implementations are broken and need to have a custom Python mail loop. |
|
74 | 149 | |
|
150 | .. [MPIref] http://www-unix.mcs.anl.gov/mpi/ | |
|
151 | ||
|
75 | 152 | .. _MPI: http://www-unix.mcs.anl.gov/mpi/ |
|
76 | 153 | .. _mpi4py: http://mpi4py.scipy.org/ |
|
77 | 154 | .. _Open MPI: http://www.open-mpi.org/ |
|
78 | 155 | .. _PyTrilinos: http://trilinos.sandia.gov/packages/pytrilinos/ No newline at end of file |
General Comments 0
You need to be logged in to leave comments.
Login now