Show More
@@ -1,1150 +1,1150 b'' | |||
|
1 | 1 | ================= |
|
2 | 2 | IPython reference |
|
3 | 3 | ================= |
|
4 | 4 | |
|
5 | 5 | .. _command_line_options: |
|
6 | 6 | |
|
7 | 7 | Command-line usage |
|
8 | 8 | ================== |
|
9 | 9 | |
|
10 | 10 | You start IPython with the command:: |
|
11 | 11 | |
|
12 | 12 | $ ipython [options] files |
|
13 | 13 | |
|
14 | 14 | .. note:: |
|
15 | 15 | |
|
16 | 16 | For IPython on Python 3, use ``ipython3`` in place of ``ipython``. |
|
17 | 17 | |
|
18 | 18 | If invoked with no options, it executes all the files listed in sequence |
|
19 | 19 | and drops you into the interpreter while still acknowledging any options |
|
20 | 20 | you may have set in your ipython_config.py. This behavior is different from |
|
21 | 21 | standard Python, which when called as python -i will only execute one |
|
22 | 22 | file and ignore your configuration setup. |
|
23 | 23 | |
|
24 | 24 | Please note that some of the configuration options are not available at |
|
25 | 25 | the command line, simply because they are not practical here. Look into |
|
26 | 26 | your configuration files for details on those. There are separate configuration |
|
27 | 27 | files for each profile, and the files look like "ipython_config.py" or |
|
28 | 28 | "ipython_config_<frontendname>.py". Profile directories look like |
|
29 | 29 | "profile_profilename" and are typically installed in the IPYTHONDIR directory. |
|
30 | 30 | For Linux users, this will be $HOME/.config/ipython, and for other users it |
|
31 | 31 | will be $HOME/.ipython. For Windows users, $HOME resolves to C:\\Documents and |
|
32 | 32 | Settings\\YourUserName in most instances. |
|
33 | 33 | |
|
34 | 34 | |
|
35 | 35 | Eventloop integration |
|
36 | 36 | --------------------- |
|
37 | 37 | |
|
38 | 38 | Previously IPython had command line options for controlling GUI event loop |
|
39 | 39 | integration (-gthread, -qthread, -q4thread, -wthread, -pylab). As of IPython |
|
40 | 40 | version 0.11, these have been removed. Please see the new ``%gui`` |
|
41 | 41 | magic command or :ref:`this section <gui_support>` for details on the new |
|
42 | 42 | interface, or specify the gui at the commandline:: |
|
43 | 43 | |
|
44 | 44 | $ ipython --gui=qt |
|
45 | 45 | |
|
46 | 46 | |
|
47 | 47 | Command-line Options |
|
48 | 48 | -------------------- |
|
49 | 49 | |
|
50 | 50 | To see the options IPython accepts, use ``ipython --help`` (and you probably |
|
51 | 51 | should run the output through a pager such as ``ipython --help | less`` for |
|
52 | 52 | more convenient reading). This shows all the options that have a single-word |
|
53 | 53 | alias to control them, but IPython lets you configure all of its objects from |
|
54 | 54 | the command-line by passing the full class name and a corresponding value; type |
|
55 | 55 | ``ipython --help-all`` to see this full list. For example:: |
|
56 | 56 | |
|
57 | 57 | ipython --pylab qt |
|
58 | 58 | |
|
59 | 59 | is equivalent to:: |
|
60 | 60 | |
|
61 | 61 | ipython --TerminalIPythonApp.pylab='qt' |
|
62 | 62 | |
|
63 | 63 | Note that in the second form, you *must* use the equal sign, as the expression |
|
64 | 64 | is evaluated as an actual Python assignment. While in the above example the |
|
65 | 65 | short form is more convenient, only the most common options have a short form, |
|
66 | 66 | while any configurable variable in IPython can be set at the command-line by |
|
67 | 67 | using the long form. This long form is the same syntax used in the |
|
68 | 68 | configuration files, if you want to set these options permanently. |
|
69 | 69 | |
|
70 | 70 | |
|
71 | 71 | Interactive use |
|
72 | 72 | =============== |
|
73 | 73 | |
|
74 | 74 | IPython is meant to work as a drop-in replacement for the standard interactive |
|
75 | 75 | interpreter. As such, any code which is valid python should execute normally |
|
76 | 76 | under IPython (cases where this is not true should be reported as bugs). It |
|
77 | 77 | does, however, offer many features which are not available at a standard python |
|
78 | 78 | prompt. What follows is a list of these. |
|
79 | 79 | |
|
80 | 80 | |
|
81 | 81 | Caution for Windows users |
|
82 | 82 | ------------------------- |
|
83 | 83 | |
|
84 | 84 | Windows, unfortunately, uses the '\\' character as a path separator. This is a |
|
85 | 85 | terrible choice, because '\\' also represents the escape character in most |
|
86 | 86 | modern programming languages, including Python. For this reason, using '/' |
|
87 | 87 | character is recommended if you have problems with ``\``. However, in Windows |
|
88 | 88 | commands '/' flags options, so you can not use it for the root directory. This |
|
89 | 89 | means that paths beginning at the root must be typed in a contrived manner |
|
90 | 90 | like: ``%copy \opt/foo/bar.txt \tmp`` |
|
91 | 91 | |
|
92 | 92 | .. _magic: |
|
93 | 93 | |
|
94 | 94 | Magic command system |
|
95 | 95 | -------------------- |
|
96 | 96 | |
|
97 | 97 | IPython will treat any line whose first character is a % as a special |
|
98 | 98 | call to a 'magic' function. These allow you to control the behavior of |
|
99 | 99 | IPython itself, plus a lot of system-type features. They are all |
|
100 | 100 | prefixed with a % character, but parameters are given without |
|
101 | 101 | parentheses or quotes. |
|
102 | 102 | |
|
103 | 103 | Lines that begin with ``%%`` signal a *cell magic*: they take as arguments not |
|
104 | 104 | only the rest of the current line, but all lines below them as well, in the |
|
105 | 105 | current execution block. Cell magics can in fact make arbitrary modifications |
|
106 | 106 | to the input they receive, which need not even be valid Python code at all. |
|
107 | 107 | They receive the whole block as a single string. |
|
108 | 108 | |
|
109 | 109 | As a line magic example, the ``%cd`` magic works just like the OS command of |
|
110 | 110 | the same name:: |
|
111 | 111 | |
|
112 | 112 | In [8]: %cd |
|
113 | 113 | /home/fperez |
|
114 | 114 | |
|
115 | 115 | The following uses the builtin ``timeit`` in cell mode:: |
|
116 | 116 | |
|
117 | 117 | In [10]: %%timeit x = range(10000) |
|
118 | 118 | ...: min(x) |
|
119 | 119 | ...: max(x) |
|
120 | 120 | ...: |
|
121 | 121 | 1000 loops, best of 3: 438 us per loop |
|
122 | 122 | |
|
123 | 123 | In this case, ``x = range(10000)`` is called as the line argument, and the |
|
124 | 124 | block with ``min(x)`` and ``max(x)`` is called as the cell body. The |
|
125 | 125 | ``timeit`` magic receives both. |
|
126 | 126 | |
|
127 | 127 | If you have 'automagic' enabled (as it by default), you don't need to type in |
|
128 | 128 | the single ``%`` explicitly for line magics; IPython will scan its internal |
|
129 | 129 | list of magic functions and call one if it exists. With automagic on you can |
|
130 | 130 | then just type ``cd mydir`` to go to directory 'mydir':: |
|
131 | 131 | |
|
132 | 132 | In [9]: cd mydir |
|
133 | 133 | /home/fperez/mydir |
|
134 | 134 | |
|
135 | 135 | Note that cell magics *always* require an explicit ``%%`` prefix, automagic |
|
136 | 136 | calling only works for line magics. |
|
137 | 137 | |
|
138 | 138 | The automagic system has the lowest possible precedence in name searches, so |
|
139 | 139 | defining an identifier with the same name as an existing magic function will |
|
140 | 140 | shadow it for automagic use. You can still access the shadowed magic function |
|
141 | 141 | by explicitly using the ``%`` character at the beginning of the line. |
|
142 | 142 | |
|
143 | 143 | An example (with automagic on) should clarify all this: |
|
144 | 144 | |
|
145 | 145 | .. sourcecode:: ipython |
|
146 | 146 | |
|
147 | 147 | In [1]: cd ipython # %cd is called by automagic |
|
148 | 148 | /home/fperez/ipython |
|
149 | 149 | |
|
150 | 150 | In [2]: cd=1 # now cd is just a variable |
|
151 | 151 | |
|
152 | 152 | In [3]: cd .. # and doesn't work as a function anymore |
|
153 | 153 | File "<ipython-input-3-9fedb3aff56c>", line 1 |
|
154 | 154 | cd .. |
|
155 | 155 | ^ |
|
156 | 156 | SyntaxError: invalid syntax |
|
157 | 157 | |
|
158 | 158 | |
|
159 | 159 | In [4]: %cd .. # but %cd always works |
|
160 | 160 | /home/fperez |
|
161 | 161 | |
|
162 | 162 | In [5]: del cd # if you remove the cd variable, automagic works again |
|
163 | 163 | |
|
164 | 164 | In [6]: cd ipython |
|
165 | 165 | |
|
166 | 166 | /home/fperez/ipython |
|
167 | 167 | |
|
168 | 168 | Defining your own magics |
|
169 | 169 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
|
170 | 170 | |
|
171 | 171 | There are two main ways to define your own magic functions: from standalone |
|
172 | 172 | functions and by inheriting from a base class provided by IPython: |
|
173 | 173 | :class:`IPython.core.magic.Magics`. Below we show code you can place in a file |
|
174 | 174 | that you load from your configuration, such as any file in the ``startup`` |
|
175 | 175 | subdirectory of your default IPython profile. |
|
176 | 176 | |
|
177 | 177 | First, let us see the simplest case. The following shows how to create a line |
|
178 | 178 | magic, a cell one and one that works in both modes, using just plain functions: |
|
179 | 179 | |
|
180 | 180 | .. sourcecode:: python |
|
181 | 181 | |
|
182 | 182 | from IPython.core.magic import (register_line_magic, register_cell_magic, |
|
183 | 183 | register_line_cell_magic) |
|
184 | 184 | |
|
185 | 185 | @register_line_magic |
|
186 | 186 | def lmagic(line): |
|
187 | 187 | "my line magic" |
|
188 | 188 | return line |
|
189 | 189 | |
|
190 | 190 | @register_cell_magic |
|
191 | 191 | def cmagic(line, cell): |
|
192 | 192 | "my cell magic" |
|
193 | 193 | return line, cell |
|
194 | 194 | |
|
195 | 195 | @register_line_cell_magic |
|
196 | 196 | def lcmagic(line, cell=None): |
|
197 | 197 | "Magic that works both as %lcmagic and as %%lcmagic" |
|
198 | 198 | if cell is None: |
|
199 | 199 | print "Called as line magic" |
|
200 | 200 | return line |
|
201 | 201 | else: |
|
202 | 202 | print "Called as cell magic" |
|
203 | 203 | return line, cell |
|
204 | 204 | |
|
205 | 205 | # We delete these to avoid name conflicts for automagic to work |
|
206 | 206 | del lmagic, lcmagic |
|
207 | 207 | |
|
208 | 208 | |
|
209 | 209 | You can also create magics of all three kinds by inheriting from the |
|
210 | 210 | :class:`IPython.core.magic.Magics` class. This lets you create magics that can |
|
211 | 211 | potentially hold state in between calls, and that have full access to the main |
|
212 | 212 | IPython object: |
|
213 | 213 | |
|
214 | 214 | .. sourcecode:: python |
|
215 | 215 | |
|
216 | 216 | # This code can be put in any Python module, it does not require IPython |
|
217 | 217 | # itself to be running already. It only creates the magics subclass but |
|
218 | 218 | # doesn't instantiate it yet. |
|
219 | 219 | from IPython.core.magic import (Magics, magics_class, line_magic, |
|
220 | 220 | cell_magic, line_cell_magic) |
|
221 | 221 | |
|
222 | 222 | # The class MUST call this class decorator at creation time |
|
223 | 223 | @magics_class |
|
224 | 224 | class MyMagics(Magics): |
|
225 | 225 | |
|
226 | 226 | @line_magic |
|
227 | 227 | def lmagic(self, line): |
|
228 | 228 | "my line magic" |
|
229 | 229 | print "Full access to the main IPython object:", self.shell |
|
230 | 230 | print "Variables in the user namespace:", self.shell.user_ns.keys() |
|
231 | 231 | return line |
|
232 | 232 | |
|
233 | 233 | @cell_magic |
|
234 | 234 | def cmagic(self, line, cell): |
|
235 | 235 | "my cell magic" |
|
236 | 236 | return line, cell |
|
237 | 237 | |
|
238 | 238 | @line_cell_magic |
|
239 | 239 | def lcmagic(self, line, cell=None): |
|
240 | 240 | "Magic that works both as %lcmagic and as %%lcmagic" |
|
241 | 241 | if cell is None: |
|
242 | 242 | print "Called as line magic" |
|
243 | 243 | return line |
|
244 | 244 | else: |
|
245 | 245 | print "Called as cell magic" |
|
246 | 246 | return line, cell |
|
247 | 247 | |
|
248 | 248 | |
|
249 | 249 | # In order to actually use these magics, you must register them with a |
|
250 | 250 | # running IPython. This code must be placed in a file that is loaded once |
|
251 | 251 | # IPython is up and running: |
|
252 | 252 | ip = get_ipython() |
|
253 | 253 | # You can register the class itself without instantiating it. IPython will |
|
254 | 254 | # call the default constructor on it. |
|
255 | 255 | ip.register_magics(MyMagics) |
|
256 | 256 | |
|
257 | 257 | If you want to create a class with a different constructor that holds |
|
258 | 258 | additional state, then you should always call the parent constructor and |
|
259 | 259 | instantiate the class yourself before registration: |
|
260 | 260 | |
|
261 | 261 | .. sourcecode:: python |
|
262 | 262 | |
|
263 | 263 | @magics_class |
|
264 | 264 | class StatefulMagics(Magics): |
|
265 | 265 | "Magics that hold additional state" |
|
266 | 266 | |
|
267 | 267 | def __init__(self, shell, data): |
|
268 | 268 | # You must call the parent constructor |
|
269 | 269 | super(StatefulMagics, self).__init__(shell) |
|
270 | 270 | self.data = data |
|
271 | 271 | |
|
272 | 272 | # etc... |
|
273 | 273 | |
|
274 | 274 | # This class must then be registered with a manually created instance, |
|
275 | 275 | # since its constructor has different arguments from the default: |
|
276 | 276 | ip = get_ipython() |
|
277 | 277 | magics = StatefulMagics(ip, some_data) |
|
278 | 278 | ip.register_magics(magics) |
|
279 | 279 | |
|
280 | 280 | |
|
281 | 281 | In earlier versions, IPython had an API for the creation of line magics (cell |
|
282 | 282 | magics did not exist at the time) that required you to create functions with a |
|
283 | 283 | method-looking signature and to manually pass both the function and the name. |
|
284 | 284 | While this API is no longer recommended, it remains indefinitely supported for |
|
285 | 285 | backwards compatibility purposes. With the old API, you'd create a magic as |
|
286 | 286 | follows: |
|
287 | 287 | |
|
288 | 288 | .. sourcecode:: python |
|
289 | 289 | |
|
290 | 290 | def func(self, line): |
|
291 | 291 | print "Line magic called with line:", line |
|
292 | 292 | print "IPython object:", self.shell |
|
293 | 293 | |
|
294 | 294 | ip = get_ipython() |
|
295 | 295 | # Declare this function as the magic %mycommand |
|
296 | 296 | ip.define_magic('mycommand', func) |
|
297 | 297 | |
|
298 | 298 | Type ``%magic`` for more information, including a list of all available magic |
|
299 | 299 | functions at any time and their docstrings. You can also type |
|
300 | 300 | ``%magic_function_name?`` (see :ref:`below <dynamic_object_info>` for |
|
301 | 301 | information on the '?' system) to get information about any particular magic |
|
302 | 302 | function you are interested in. |
|
303 | 303 | |
|
304 | 304 | The API documentation for the :mod:`IPython.core.magic` module contains the full |
|
305 | 305 | docstrings of all currently available magic commands. |
|
306 | 306 | |
|
307 | 307 | |
|
308 | 308 | Access to the standard Python help |
|
309 | 309 | ---------------------------------- |
|
310 | 310 | |
|
311 | 311 | Simply type ``help()`` to access Python's standard help system. You can |
|
312 | 312 | also type ``help(object)`` for information about a given object, or |
|
313 | 313 | ``help('keyword')`` for information on a keyword. You may need to configure your |
|
314 | 314 | PYTHONDOCS environment variable for this feature to work correctly. |
|
315 | 315 | |
|
316 | 316 | .. _dynamic_object_info: |
|
317 | 317 | |
|
318 | 318 | Dynamic object information |
|
319 | 319 | -------------------------- |
|
320 | 320 | |
|
321 | 321 | Typing ``?word`` or ``word?`` prints detailed information about an object. If |
|
322 | 322 | certain strings in the object are too long (e.g. function signatures) they get |
|
323 | 323 | snipped in the center for brevity. This system gives access variable types and |
|
324 | 324 | values, docstrings, function prototypes and other useful information. |
|
325 | 325 | |
|
326 | 326 | If the information will not fit in the terminal, it is displayed in a pager |
|
327 | 327 | (``less`` if available, otherwise a basic internal pager). |
|
328 | 328 | |
|
329 | 329 | Typing ``??word`` or ``word??`` gives access to the full information, including |
|
330 | 330 | the source code where possible. Long strings are not snipped. |
|
331 | 331 | |
|
332 | 332 | The following magic functions are particularly useful for gathering |
|
333 | 333 | information about your working environment. You can get more details by |
|
334 | 334 | typing ``%magic`` or querying them individually (``%function_name?``); |
|
335 | 335 | this is just a summary: |
|
336 | 336 | |
|
337 | 337 | * **%pdoc <object>**: Print (or run through a pager if too long) the |
|
338 | 338 | docstring for an object. If the given object is a class, it will |
|
339 | 339 | print both the class and the constructor docstrings. |
|
340 | 340 | * **%pdef <object>**: Print the call signature for any callable |
|
341 | 341 | object. If the object is a class, print the constructor information. |
|
342 | 342 | * **%psource <object>**: Print (or run through a pager if too long) |
|
343 | 343 | the source code for an object. |
|
344 | 344 | * **%pfile <object>**: Show the entire source file where an object was |
|
345 | 345 | defined via a pager, opening it at the line where the object |
|
346 | 346 | definition begins. |
|
347 | 347 | * **%who/%whos**: These functions give information about identifiers |
|
348 | 348 | you have defined interactively (not things you loaded or defined |
|
349 | 349 | in your configuration files). %who just prints a list of |
|
350 | 350 | identifiers and %whos prints a table with some basic details about |
|
351 | 351 | each identifier. |
|
352 | 352 | |
|
353 | 353 | Note that the dynamic object information functions (?/??, ``%pdoc``, |
|
354 | 354 | ``%pfile``, ``%pdef``, ``%psource``) work on object attributes, as well as |
|
355 | 355 | directly on variables. For example, after doing ``import os``, you can use |
|
356 | 356 | ``os.path.abspath??``. |
|
357 | 357 | |
|
358 | 358 | .. _readline: |
|
359 | 359 | |
|
360 | 360 | Readline-based features |
|
361 | 361 | ----------------------- |
|
362 | 362 | |
|
363 | 363 | These features require the GNU readline library, so they won't work if your |
|
364 | 364 | Python installation lacks readline support. We will first describe the default |
|
365 | 365 | behavior IPython uses, and then how to change it to suit your preferences. |
|
366 | 366 | |
|
367 | 367 | |
|
368 | 368 | Command line completion |
|
369 | 369 | +++++++++++++++++++++++ |
|
370 | 370 | |
|
371 | 371 | At any time, hitting TAB will complete any available python commands or |
|
372 | 372 | variable names, and show you a list of the possible completions if |
|
373 | 373 | there's no unambiguous one. It will also complete filenames in the |
|
374 | 374 | current directory if no python names match what you've typed so far. |
|
375 | 375 | |
|
376 | 376 | |
|
377 | 377 | Search command history |
|
378 | 378 | ++++++++++++++++++++++ |
|
379 | 379 | |
|
380 | 380 | IPython provides two ways for searching through previous input and thus |
|
381 | 381 | reduce the need for repetitive typing: |
|
382 | 382 | |
|
383 | 383 | 1. Start typing, and then use Ctrl-p (previous,up) and Ctrl-n |
|
384 | 384 | (next,down) to search through only the history items that match |
|
385 | 385 | what you've typed so far. If you use Ctrl-p/Ctrl-n at a blank |
|
386 | 386 | prompt, they just behave like normal arrow keys. |
|
387 | 387 | 2. Hit Ctrl-r: opens a search prompt. Begin typing and the system |
|
388 | 388 | searches your history for lines that contain what you've typed so |
|
389 | 389 | far, completing as much as it can. |
|
390 | 390 | |
|
391 | 391 | |
|
392 | 392 | Persistent command history across sessions |
|
393 | 393 | ++++++++++++++++++++++++++++++++++++++++++ |
|
394 | 394 | |
|
395 | 395 | IPython will save your input history when it leaves and reload it next |
|
396 | 396 | time you restart it. By default, the history file is named |
|
397 | 397 | $IPYTHONDIR/profile_<name>/history.sqlite. This allows you to keep |
|
398 | 398 | separate histories related to various tasks: commands related to |
|
399 | 399 | numerical work will not be clobbered by a system shell history, for |
|
400 | 400 | example. |
|
401 | 401 | |
|
402 | 402 | |
|
403 | 403 | Autoindent |
|
404 | 404 | ++++++++++ |
|
405 | 405 | |
|
406 | 406 | IPython can recognize lines ending in ':' and indent the next line, |
|
407 | 407 | while also un-indenting automatically after 'raise' or 'return'. |
|
408 | 408 | |
|
409 | 409 | This feature uses the readline library, so it will honor your |
|
410 | 410 | :file:`~/.inputrc` configuration (or whatever file your INPUTRC variable points |
|
411 | 411 | to). Adding the following lines to your :file:`.inputrc` file can make |
|
412 | 412 | indenting/unindenting more convenient (M-i indents, M-u unindents):: |
|
413 | 413 | |
|
414 | 414 | $if Python |
|
415 | 415 | "\M-i": " " |
|
416 | 416 | "\M-u": "\d\d\d\d" |
|
417 | 417 | $endif |
|
418 | 418 | |
|
419 | 419 | Note that there are 4 spaces between the quote marks after "M-i" above. |
|
420 | 420 | |
|
421 | 421 | .. warning:: |
|
422 | 422 | |
|
423 | 423 | Setting the above indents will cause problems with unicode text entry in |
|
424 | 424 | the terminal. |
|
425 | 425 | |
|
426 | 426 | .. warning:: |
|
427 | 427 | |
|
428 | 428 | Autoindent is ON by default, but it can cause problems with the pasting of |
|
429 | 429 | multi-line indented code (the pasted code gets re-indented on each line). A |
|
430 | 430 | magic function %autoindent allows you to toggle it on/off at runtime. You |
|
431 | 431 | can also disable it permanently on in your :file:`ipython_config.py` file |
|
432 | 432 | (set TerminalInteractiveShell.autoindent=False). |
|
433 | 433 | |
|
434 | 434 | If you want to paste multiple lines in the terminal, it is recommended that |
|
435 | 435 | you use ``%paste``. |
|
436 | 436 | |
|
437 | 437 | |
|
438 | 438 | Customizing readline behavior |
|
439 | 439 | +++++++++++++++++++++++++++++ |
|
440 | 440 | |
|
441 | 441 | All these features are based on the GNU readline library, which has an |
|
442 | 442 | extremely customizable interface. Normally, readline is configured via a |
|
443 | 443 | file which defines the behavior of the library; the details of the |
|
444 | 444 | syntax for this can be found in the readline documentation available |
|
445 | 445 | with your system or on the Internet. IPython doesn't read this file (if |
|
446 | 446 | it exists) directly, but it does support passing to readline valid |
|
447 | 447 | options via a simple interface. In brief, you can customize readline by |
|
448 | 448 | setting the following options in your configuration file (note |
|
449 | 449 | that these options can not be specified at the command line): |
|
450 | 450 | |
|
451 | 451 | * **readline_parse_and_bind**: this holds a list of strings to be executed |
|
452 | 452 | via a readline.parse_and_bind() command. The syntax for valid commands |
|
453 | 453 | of this kind can be found by reading the documentation for the GNU |
|
454 | 454 | readline library, as these commands are of the kind which readline |
|
455 | 455 | accepts in its configuration file. |
|
456 | 456 | * **readline_remove_delims**: a string of characters to be removed |
|
457 | 457 | from the default word-delimiters list used by readline, so that |
|
458 | 458 | completions may be performed on strings which contain them. Do not |
|
459 | 459 | change the default value unless you know what you're doing. |
|
460 | 460 | |
|
461 | 461 | You will find the default values in your configuration file. |
|
462 | 462 | |
|
463 | 463 | |
|
464 | 464 | Session logging and restoring |
|
465 | 465 | ----------------------------- |
|
466 | 466 | |
|
467 | 467 | You can log all input from a session either by starting IPython with the |
|
468 | 468 | command line switch ``--logfile=foo.py`` (see :ref:`here <command_line_options>`) |
|
469 | 469 | or by activating the logging at any moment with the magic function %logstart. |
|
470 | 470 | |
|
471 | 471 | Log files can later be reloaded by running them as scripts and IPython |
|
472 | 472 | will attempt to 'replay' the log by executing all the lines in it, thus |
|
473 | 473 | restoring the state of a previous session. This feature is not quite |
|
474 | 474 | perfect, but can still be useful in many cases. |
|
475 | 475 | |
|
476 | 476 | The log files can also be used as a way to have a permanent record of |
|
477 | 477 | any code you wrote while experimenting. Log files are regular text files |
|
478 | 478 | which you can later open in your favorite text editor to extract code or |
|
479 | 479 | to 'clean them up' before using them to replay a session. |
|
480 | 480 | |
|
481 | 481 | The `%logstart` function for activating logging in mid-session is used as |
|
482 | 482 | follows:: |
|
483 | 483 | |
|
484 | 484 | %logstart [log_name [log_mode]] |
|
485 | 485 | |
|
486 | 486 | If no name is given, it defaults to a file named 'ipython_log.py' in your |
|
487 | 487 | current working directory, in 'rotate' mode (see below). |
|
488 | 488 | |
|
489 | 489 | '%logstart name' saves to file 'name' in 'backup' mode. It saves your |
|
490 | 490 | history up to that point and then continues logging. |
|
491 | 491 | |
|
492 | 492 | %logstart takes a second optional parameter: logging mode. This can be |
|
493 | 493 | one of (note that the modes are given unquoted): |
|
494 | 494 | |
|
495 | 495 | * [over:] overwrite existing log_name. |
|
496 | 496 | * [backup:] rename (if exists) to log_name~ and start log_name. |
|
497 | 497 | * [append:] well, that says it. |
|
498 | 498 | * [rotate:] create rotating logs log_name.1~, log_name.2~, etc. |
|
499 | 499 | |
|
500 | 500 | The %logoff and %logon functions allow you to temporarily stop and |
|
501 | 501 | resume logging to a file which had previously been started with |
|
502 | 502 | %logstart. They will fail (with an explanation) if you try to use them |
|
503 | 503 | before logging has been started. |
|
504 | 504 | |
|
505 | 505 | .. _system_shell_access: |
|
506 | 506 | |
|
507 | 507 | System shell access |
|
508 | 508 | ------------------- |
|
509 | 509 | |
|
510 | 510 | Any input line beginning with a ! character is passed verbatim (minus |
|
511 | 511 | the !, of course) to the underlying operating system. For example, |
|
512 | 512 | typing ``!ls`` will run 'ls' in the current directory. |
|
513 | 513 | |
|
514 | 514 | Manual capture of command output |
|
515 | 515 | -------------------------------- |
|
516 | 516 | |
|
517 | 517 | You can assign the result of a system command to a Python variable with the |
|
518 | 518 | syntax ``myfiles = !ls``. This gets machine readable output from stdout |
|
519 | 519 | (e.g. without colours), and splits on newlines. To explicitly get this sort of |
|
520 | 520 | output without assigning to a variable, use two exclamation marks (``!!ls``) or |
|
521 | 521 | the ``%sx`` magic command. |
|
522 | 522 | |
|
523 | 523 | The captured list has some convenience features. ``myfiles.n`` or ``myfiles.s`` |
|
524 | 524 | returns a string delimited by newlines or spaces, respectively. ``myfiles.p`` |
|
525 | 525 | produces `path objects <http://pypi.python.org/pypi/path.py>`_ from the list items. |
|
526 | 526 | See :ref:`string_lists` for details. |
|
527 | 527 | |
|
528 | 528 | IPython also allows you to expand the value of python variables when |
|
529 | 529 | making system calls. Wrap variables or expressions in {braces}:: |
|
530 | 530 | |
|
531 | 531 | In [1]: pyvar = 'Hello world' |
|
532 | 532 | In [2]: !echo "A python variable: {pyvar}" |
|
533 | 533 | A python variable: Hello world |
|
534 | 534 | In [3]: import math |
|
535 | 535 | In [4]: x = 8 |
|
536 | 536 | In [5]: !echo {math.factorial(x)} |
|
537 | 537 | 40320 |
|
538 | 538 | |
|
539 | 539 | For simple cases, you can alternatively prepend $ to a variable name:: |
|
540 | 540 | |
|
541 | 541 | In [6]: !echo $sys.argv |
|
542 | 542 | [/home/fperez/usr/bin/ipython] |
|
543 | 543 | In [7]: !echo "A system variable: $$HOME" # Use $$ for literal $ |
|
544 | 544 | A system variable: /home/fperez |
|
545 | 545 | |
|
546 | 546 | System command aliases |
|
547 | 547 | ---------------------- |
|
548 | 548 | |
|
549 | 549 | The %alias magic function allows you to define magic functions which are in fact |
|
550 | 550 | system shell commands. These aliases can have parameters. |
|
551 | 551 | |
|
552 | 552 | ``%alias alias_name cmd`` defines 'alias_name' as an alias for 'cmd' |
|
553 | 553 | |
|
554 | 554 | Then, typing ``alias_name params`` will execute the system command 'cmd |
|
555 | 555 | params' (from your underlying operating system). |
|
556 | 556 | |
|
557 | 557 | You can also define aliases with parameters using %s specifiers (one per |
|
558 | 558 | parameter). The following example defines the parts function as an |
|
559 | 559 | alias to the command 'echo first %s second %s' where each %s will be |
|
560 | 560 | replaced by a positional parameter to the call to %parts:: |
|
561 | 561 | |
|
562 | 562 | In [1]: %alias parts echo first %s second %s |
|
563 | 563 | In [2]: parts A B |
|
564 | 564 | first A second B |
|
565 | 565 | In [3]: parts A |
|
566 | 566 | ERROR: Alias <parts> requires 2 arguments, 1 given. |
|
567 | 567 | |
|
568 | 568 | If called with no parameters, %alias prints the table of currently |
|
569 | 569 | defined aliases. |
|
570 | 570 | |
|
571 | 571 | The %rehashx magic allows you to load your entire $PATH as |
|
572 | 572 | ipython aliases. See its docstring for further details. |
|
573 | 573 | |
|
574 | 574 | |
|
575 | 575 | .. _dreload: |
|
576 | 576 | |
|
577 | 577 | Recursive reload |
|
578 | 578 | ---------------- |
|
579 | 579 | |
|
580 | 580 | The :mod:`IPython.lib.deepreload` module allows you to recursively reload a |
|
581 | 581 | module: changes made to any of its dependencies will be reloaded without |
|
582 | 582 | having to exit. To start using it, do:: |
|
583 | 583 | |
|
584 | 584 | from IPython.lib.deepreload import reload as dreload |
|
585 | 585 | |
|
586 | 586 | |
|
587 | 587 | Verbose and colored exception traceback printouts |
|
588 | 588 | ------------------------------------------------- |
|
589 | 589 | |
|
590 | 590 | IPython provides the option to see very detailed exception tracebacks, |
|
591 | 591 | which can be especially useful when debugging large programs. You can |
|
592 | 592 | run any Python file with the %run function to benefit from these |
|
593 | 593 | detailed tracebacks. Furthermore, both normal and verbose tracebacks can |
|
594 | 594 | be colored (if your terminal supports it) which makes them much easier |
|
595 | 595 | to parse visually. |
|
596 | 596 | |
|
597 | 597 | See the magic xmode and colors functions for details (just type %magic). |
|
598 | 598 | |
|
599 | 599 | These features are basically a terminal version of Ka-Ping Yee's cgitb |
|
600 | 600 | module, now part of the standard Python library. |
|
601 | 601 | |
|
602 | 602 | |
|
603 | 603 | .. _input_caching: |
|
604 | 604 | |
|
605 | 605 | Input caching system |
|
606 | 606 | -------------------- |
|
607 | 607 | |
|
608 | 608 | IPython offers numbered prompts (In/Out) with input and output caching |
|
609 | 609 | (also referred to as 'input history'). All input is saved and can be |
|
610 | 610 | retrieved as variables (besides the usual arrow key recall), in |
|
611 | 611 | addition to the %rep magic command that brings a history entry |
|
612 | 612 | up for editing on the next command line. |
|
613 | 613 | |
|
614 | 614 | The following GLOBAL variables always exist (so don't overwrite them!): |
|
615 | 615 | |
|
616 | 616 | * _i, _ii, _iii: store previous, next previous and next-next previous inputs. |
|
617 | 617 | * In, _ih : a list of all inputs; _ih[n] is the input from line n. If you |
|
618 | 618 | overwrite In with a variable of your own, you can remake the assignment to the |
|
619 | 619 | internal list with a simple ``In=_ih``. |
|
620 | 620 | |
|
621 | 621 | Additionally, global variables named _i<n> are dynamically created (<n> |
|
622 | 622 | being the prompt counter), so ``_i<n> == _ih[<n>] == In[<n>]``. |
|
623 | 623 | |
|
624 | 624 | For example, what you typed at prompt 14 is available as _i14, _ih[14] |
|
625 | 625 | and In[14]. |
|
626 | 626 | |
|
627 | 627 | This allows you to easily cut and paste multi line interactive prompts |
|
628 | 628 | by printing them out: they print like a clean string, without prompt |
|
629 | 629 | characters. You can also manipulate them like regular variables (they |
|
630 | 630 | are strings), modify or exec them (typing ``exec _i9`` will re-execute the |
|
631 | 631 | contents of input prompt 9. |
|
632 | 632 | |
|
633 | 633 | You can also re-execute multiple lines of input easily by using the |
|
634 | 634 | magic %rerun or %macro functions. The macro system also allows you to re-execute |
|
635 | 635 | previous lines which include magic function calls (which require special |
|
636 | 636 | processing). Type %macro? for more details on the macro system. |
|
637 | 637 | |
|
638 | 638 | A history function %hist allows you to see any part of your input |
|
639 | 639 | history by printing a range of the _i variables. |
|
640 | 640 | |
|
641 | 641 | You can also search ('grep') through your history by typing |
|
642 | 642 | ``%hist -g somestring``. This is handy for searching for URLs, IP addresses, |
|
643 | 643 | etc. You can bring history entries listed by '%hist -g' up for editing |
|
644 | 644 | with the %recall command, or run them immediately with %rerun. |
|
645 | 645 | |
|
646 | 646 | .. _output_caching: |
|
647 | 647 | |
|
648 | 648 | Output caching system |
|
649 | 649 | --------------------- |
|
650 | 650 | |
|
651 | 651 | For output that is returned from actions, a system similar to the input |
|
652 | 652 | cache exists but using _ instead of _i. Only actions that produce a |
|
653 | 653 | result (NOT assignments, for example) are cached. If you are familiar |
|
654 | 654 | with Mathematica, IPython's _ variables behave exactly like |
|
655 | 655 | Mathematica's % variables. |
|
656 | 656 | |
|
657 | 657 | The following GLOBAL variables always exist (so don't overwrite them!): |
|
658 | 658 | |
|
659 | 659 | * [_] (a single underscore) : stores previous output, like Python's |
|
660 | 660 | default interpreter. |
|
661 | 661 | * [__] (two underscores): next previous. |
|
662 | 662 | * [___] (three underscores): next-next previous. |
|
663 | 663 | |
|
664 | 664 | Additionally, global variables named _<n> are dynamically created (<n> |
|
665 | 665 | being the prompt counter), such that the result of output <n> is always |
|
666 | 666 | available as _<n> (don't use the angle brackets, just the number, e.g. |
|
667 | 667 | _21). |
|
668 | 668 | |
|
669 | 669 | These variables are also stored in a global dictionary (not a |
|
670 | 670 | list, since it only has entries for lines which returned a result) |
|
671 | 671 | available under the names _oh and Out (similar to _ih and In). So the |
|
672 | 672 | output from line 12 can be obtained as _12, Out[12] or _oh[12]. If you |
|
673 | 673 | accidentally overwrite the Out variable you can recover it by typing |
|
674 | 674 | 'Out=_oh' at the prompt. |
|
675 | 675 | |
|
676 | 676 | This system obviously can potentially put heavy memory demands on your |
|
677 | 677 | system, since it prevents Python's garbage collector from removing any |
|
678 | 678 | previously computed results. You can control how many results are kept |
|
679 | 679 | in memory with the option (at the command line or in your configuration |
|
680 | 680 | file) cache_size. If you set it to 0, the whole system is completely |
|
681 | 681 | disabled and the prompts revert to the classic '>>>' of normal Python. |
|
682 | 682 | |
|
683 | 683 | |
|
684 | 684 | Directory history |
|
685 | 685 | ----------------- |
|
686 | 686 | |
|
687 | 687 | Your history of visited directories is kept in the global list _dh, and |
|
688 | 688 | the magic %cd command can be used to go to any entry in that list. The |
|
689 | 689 | %dhist command allows you to view this history. Do ``cd -<TAB>`` to |
|
690 | 690 | conveniently view the directory history. |
|
691 | 691 | |
|
692 | 692 | |
|
693 | 693 | Automatic parentheses and quotes |
|
694 | 694 | -------------------------------- |
|
695 | 695 | |
|
696 | 696 | These features were adapted from Nathan Gray's LazyPython. They are |
|
697 | 697 | meant to allow less typing for common situations. |
|
698 | 698 | |
|
699 | 699 | |
|
700 | 700 | Automatic parentheses |
|
701 | 701 | +++++++++++++++++++++ |
|
702 | 702 | |
|
703 | 703 | Callable objects (i.e. functions, methods, etc) can be invoked like this |
|
704 | 704 | (notice the commas between the arguments):: |
|
705 | 705 | |
|
706 | 706 | In [1]: callable_ob arg1, arg2, arg3 |
|
707 | 707 | ------> callable_ob(arg1, arg2, arg3) |
|
708 | 708 | |
|
709 | 709 | You can force automatic parentheses by using '/' as the first character |
|
710 | 710 | of a line. For example:: |
|
711 | 711 | |
|
712 | 712 | In [2]: /globals # becomes 'globals()' |
|
713 | 713 | |
|
714 | 714 | Note that the '/' MUST be the first character on the line! This won't work:: |
|
715 | 715 | |
|
716 | 716 | In [3]: print /globals # syntax error |
|
717 | 717 | |
|
718 | 718 | In most cases the automatic algorithm should work, so you should rarely |
|
719 | 719 | need to explicitly invoke /. One notable exception is if you are trying |
|
720 | 720 | to call a function with a list of tuples as arguments (the parenthesis |
|
721 | 721 | will confuse IPython):: |
|
722 | 722 | |
|
723 | 723 | In [4]: zip (1,2,3),(4,5,6) # won't work |
|
724 | 724 | |
|
725 | 725 | but this will work:: |
|
726 | 726 | |
|
727 | 727 | In [5]: /zip (1,2,3),(4,5,6) |
|
728 | 728 | ------> zip ((1,2,3),(4,5,6)) |
|
729 | 729 | Out[5]: [(1, 4), (2, 5), (3, 6)] |
|
730 | 730 | |
|
731 | 731 | IPython tells you that it has altered your command line by displaying |
|
732 | 732 | the new command line preceded by ->. e.g.:: |
|
733 | 733 | |
|
734 | 734 | In [6]: callable list |
|
735 | 735 | ------> callable(list) |
|
736 | 736 | |
|
737 | 737 | |
|
738 | 738 | Automatic quoting |
|
739 | 739 | +++++++++++++++++ |
|
740 | 740 | |
|
741 | 741 | You can force automatic quoting of a function's arguments by using ',' |
|
742 | 742 | or ';' as the first character of a line. For example:: |
|
743 | 743 | |
|
744 | 744 | In [1]: ,my_function /home/me # becomes my_function("/home/me") |
|
745 | 745 | |
|
746 | 746 | If you use ';' the whole argument is quoted as a single string, while ',' splits |
|
747 | 747 | on whitespace:: |
|
748 | 748 | |
|
749 | 749 | In [2]: ,my_function a b c # becomes my_function("a","b","c") |
|
750 | 750 | |
|
751 | 751 | In [3]: ;my_function a b c # becomes my_function("a b c") |
|
752 | 752 | |
|
753 | 753 | Note that the ',' or ';' MUST be the first character on the line! This |
|
754 | 754 | won't work:: |
|
755 | 755 | |
|
756 | 756 | In [4]: x = ,my_function /home/me # syntax error |
|
757 | 757 | |
|
758 | 758 | IPython as your default Python environment |
|
759 | 759 | ========================================== |
|
760 | 760 | |
|
761 | 761 | Python honors the environment variable PYTHONSTARTUP and will execute at |
|
762 | 762 | startup the file referenced by this variable. If you put the following code at |
|
763 | 763 | the end of that file, then IPython will be your working environment anytime you |
|
764 | 764 | start Python:: |
|
765 | 765 | |
|
766 | 766 | from IPython.frontend.terminal.ipapp import launch_new_instance |
|
767 | 767 | launch_new_instance() |
|
768 | 768 | raise SystemExit |
|
769 | 769 | |
|
770 | 770 | The ``raise SystemExit`` is needed to exit Python when |
|
771 | 771 | it finishes, otherwise you'll be back at the normal Python '>>>' |
|
772 | 772 | prompt. |
|
773 | 773 | |
|
774 | 774 | This is probably useful to developers who manage multiple Python |
|
775 | 775 | versions and don't want to have correspondingly multiple IPython |
|
776 | 776 | versions. Note that in this mode, there is no way to pass IPython any |
|
777 | 777 | command-line options, as those are trapped first by Python itself. |
|
778 | 778 | |
|
779 | 779 | .. _Embedding: |
|
780 | 780 | |
|
781 | 781 | Embedding IPython |
|
782 | 782 | ================= |
|
783 | 783 | |
|
784 | 784 | It is possible to start an IPython instance inside your own Python |
|
785 | 785 | programs. This allows you to evaluate dynamically the state of your |
|
786 | 786 | code, operate with your variables, analyze them, etc. Note however that |
|
787 | 787 | any changes you make to values while in the shell do not propagate back |
|
788 | 788 | to the running code, so it is safe to modify your values because you |
|
789 | 789 | won't break your code in bizarre ways by doing so. |
|
790 | 790 | |
|
791 | 791 | .. note:: |
|
792 | 792 | |
|
793 | 793 | At present, trying to embed IPython from inside IPython causes problems. Run |
|
794 | 794 | the code samples below outside IPython. |
|
795 | 795 | |
|
796 | 796 | This feature allows you to easily have a fully functional python |
|
797 | 797 | environment for doing object introspection anywhere in your code with a |
|
798 | 798 | simple function call. In some cases a simple print statement is enough, |
|
799 | 799 | but if you need to do more detailed analysis of a code fragment this |
|
800 | 800 | feature can be very valuable. |
|
801 | 801 | |
|
802 | 802 | It can also be useful in scientific computing situations where it is |
|
803 | 803 | common to need to do some automatic, computationally intensive part and |
|
804 | 804 | then stop to look at data, plots, etc. |
|
805 | 805 | Opening an IPython instance will give you full access to your data and |
|
806 | 806 | functions, and you can resume program execution once you are done with |
|
807 | 807 | the interactive part (perhaps to stop again later, as many times as |
|
808 | 808 | needed). |
|
809 | 809 | |
|
810 | 810 | The following code snippet is the bare minimum you need to include in |
|
811 | 811 | your Python programs for this to work (detailed examples follow later):: |
|
812 | 812 | |
|
813 | 813 | from IPython import embed |
|
814 | 814 | |
|
815 | 815 | embed() # this call anywhere in your program will start IPython |
|
816 | 816 | |
|
817 | 817 | .. note:: |
|
818 | 818 | |
|
819 | 819 | As of 0.13, you can embed an IPython *kernel*, for use with qtconsole, |
|
820 | 820 | etc. via ``IPython.embed_kernel()`` instead of ``IPython.embed()``. |
|
821 | 821 | It should function just the same as regular embed, but you connect |
|
822 | 822 | an external frontend rather than IPython starting up in the local |
|
823 | 823 | terminal. |
|
824 | 824 | |
|
825 | 825 | You can run embedded instances even in code which is itself being run at |
|
826 | 826 | the IPython interactive prompt with '%run <filename>'. Since it's easy |
|
827 | 827 | to get lost as to where you are (in your top-level IPython or in your |
|
828 | 828 | embedded one), it's a good idea in such cases to set the in/out prompts |
|
829 | 829 | to something different for the embedded instances. The code examples |
|
830 | 830 | below illustrate this. |
|
831 | 831 | |
|
832 | 832 | You can also have multiple IPython instances in your program and open |
|
833 | 833 | them separately, for example with different options for data |
|
834 | 834 | presentation. If you close and open the same instance multiple times, |
|
835 | 835 | its prompt counters simply continue from each execution to the next. |
|
836 | 836 | |
|
837 | 837 | Please look at the docstrings in the :mod:`~IPython.frontend.terminal.embed` |
|
838 | 838 | module for more details on the use of this system. |
|
839 | 839 | |
|
840 | 840 | The following sample file illustrating how to use the embedding |
|
841 | 841 | functionality is provided in the examples directory as example-embed.py. |
|
842 | 842 | It should be fairly self-explanatory: |
|
843 | 843 | |
|
844 | .. literalinclude:: ../../examples/core/example-embed.py | |
|
844 | .. literalinclude:: ../../../examples/core/example-embed.py | |
|
845 | 845 | :language: python |
|
846 | 846 | |
|
847 | 847 | Once you understand how the system functions, you can use the following |
|
848 | 848 | code fragments in your programs which are ready for cut and paste: |
|
849 | 849 | |
|
850 | .. literalinclude:: ../../examples/core/example-embed-short.py | |
|
850 | .. literalinclude:: ../../../examples/core/example-embed-short.py | |
|
851 | 851 | :language: python |
|
852 | 852 | |
|
853 | 853 | Using the Python debugger (pdb) |
|
854 | 854 | =============================== |
|
855 | 855 | |
|
856 | 856 | Running entire programs via pdb |
|
857 | 857 | ------------------------------- |
|
858 | 858 | |
|
859 | 859 | pdb, the Python debugger, is a powerful interactive debugger which |
|
860 | 860 | allows you to step through code, set breakpoints, watch variables, |
|
861 | 861 | etc. IPython makes it very easy to start any script under the control |
|
862 | 862 | of pdb, regardless of whether you have wrapped it into a 'main()' |
|
863 | 863 | function or not. For this, simply type '%run -d myscript' at an |
|
864 | 864 | IPython prompt. See the %run command's documentation (via '%run?' or |
|
865 | 865 | in Sec. magic_ for more details, including how to control where pdb |
|
866 | 866 | will stop execution first. |
|
867 | 867 | |
|
868 | 868 | For more information on the use of the pdb debugger, read the included |
|
869 | 869 | pdb.doc file (part of the standard Python distribution). On a stock |
|
870 | 870 | Linux system it is located at /usr/lib/python2.3/pdb.doc, but the |
|
871 | 871 | easiest way to read it is by using the help() function of the pdb module |
|
872 | 872 | as follows (in an IPython prompt):: |
|
873 | 873 | |
|
874 | 874 | In [1]: import pdb |
|
875 | 875 | In [2]: pdb.help() |
|
876 | 876 | |
|
877 | 877 | This will load the pdb.doc document in a file viewer for you automatically. |
|
878 | 878 | |
|
879 | 879 | |
|
880 | 880 | Automatic invocation of pdb on exceptions |
|
881 | 881 | ----------------------------------------- |
|
882 | 882 | |
|
883 | 883 | IPython, if started with the ``--pdb`` option (or if the option is set in |
|
884 | 884 | your config file) can call the Python pdb debugger every time your code |
|
885 | 885 | triggers an uncaught exception. This feature |
|
886 | 886 | can also be toggled at any time with the %pdb magic command. This can be |
|
887 | 887 | extremely useful in order to find the origin of subtle bugs, because pdb |
|
888 | 888 | opens up at the point in your code which triggered the exception, and |
|
889 | 889 | while your program is at this point 'dead', all the data is still |
|
890 | 890 | available and you can walk up and down the stack frame and understand |
|
891 | 891 | the origin of the problem. |
|
892 | 892 | |
|
893 | 893 | Furthermore, you can use these debugging facilities both with the |
|
894 | 894 | embedded IPython mode and without IPython at all. For an embedded shell |
|
895 | 895 | (see sec. Embedding_), simply call the constructor with |
|
896 | 896 | ``--pdb`` in the argument string and pdb will automatically be called if an |
|
897 | 897 | uncaught exception is triggered by your code. |
|
898 | 898 | |
|
899 | 899 | For stand-alone use of the feature in your programs which do not use |
|
900 | 900 | IPython at all, put the following lines toward the top of your 'main' |
|
901 | 901 | routine:: |
|
902 | 902 | |
|
903 | 903 | import sys |
|
904 | 904 | from IPython.core import ultratb |
|
905 | 905 | sys.excepthook = ultratb.FormattedTB(mode='Verbose', |
|
906 | 906 | color_scheme='Linux', call_pdb=1) |
|
907 | 907 | |
|
908 | 908 | The mode keyword can be either 'Verbose' or 'Plain', giving either very |
|
909 | 909 | detailed or normal tracebacks respectively. The color_scheme keyword can |
|
910 | 910 | be one of 'NoColor', 'Linux' (default) or 'LightBG'. These are the same |
|
911 | 911 | options which can be set in IPython with ``--colors`` and ``--xmode``. |
|
912 | 912 | |
|
913 | 913 | This will give any of your programs detailed, colored tracebacks with |
|
914 | 914 | automatic invocation of pdb. |
|
915 | 915 | |
|
916 | 916 | |
|
917 | 917 | Extensions for syntax processing |
|
918 | 918 | ================================ |
|
919 | 919 | |
|
920 | 920 | This isn't for the faint of heart, because the potential for breaking |
|
921 | 921 | things is quite high. But it can be a very powerful and useful feature. |
|
922 | 922 | In a nutshell, you can redefine the way IPython processes the user input |
|
923 | 923 | line to accept new, special extensions to the syntax without needing to |
|
924 | 924 | change any of IPython's own code. |
|
925 | 925 | |
|
926 | 926 | In the IPython/extensions directory you will find some examples |
|
927 | 927 | supplied, which we will briefly describe now. These can be used 'as is' |
|
928 | 928 | (and both provide very useful functionality), or you can use them as a |
|
929 | 929 | starting point for writing your own extensions. |
|
930 | 930 | |
|
931 | 931 | .. _pasting_with_prompts: |
|
932 | 932 | |
|
933 | 933 | Pasting of code starting with Python or IPython prompts |
|
934 | 934 | ------------------------------------------------------- |
|
935 | 935 | |
|
936 | 936 | IPython is smart enough to filter out input prompts, be they plain Python ones |
|
937 | 937 | (``>>>`` and ``...``) or IPython ones (``In [N]:`` and `` ...:``). You can |
|
938 | 938 | therefore copy and paste from existing interactive sessions without worry. |
|
939 | 939 | |
|
940 | 940 | The following is a 'screenshot' of how things work, copying an example from the |
|
941 | 941 | standard Python tutorial:: |
|
942 | 942 | |
|
943 | 943 | In [1]: >>> # Fibonacci series: |
|
944 | 944 | |
|
945 | 945 | In [2]: ... # the sum of two elements defines the next |
|
946 | 946 | |
|
947 | 947 | In [3]: ... a, b = 0, 1 |
|
948 | 948 | |
|
949 | 949 | In [4]: >>> while b < 10: |
|
950 | 950 | ...: ... print b |
|
951 | 951 | ...: ... a, b = b, a+b |
|
952 | 952 | ...: |
|
953 | 953 | 1 |
|
954 | 954 | 1 |
|
955 | 955 | 2 |
|
956 | 956 | 3 |
|
957 | 957 | 5 |
|
958 | 958 | 8 |
|
959 | 959 | |
|
960 | 960 | And pasting from IPython sessions works equally well:: |
|
961 | 961 | |
|
962 | 962 | In [1]: In [5]: def f(x): |
|
963 | 963 | ...: ...: "A simple function" |
|
964 | 964 | ...: ...: return x**2 |
|
965 | 965 | ...: ...: |
|
966 | 966 | |
|
967 | 967 | In [2]: f(3) |
|
968 | 968 | Out[2]: 9 |
|
969 | 969 | |
|
970 | 970 | .. _gui_support: |
|
971 | 971 | |
|
972 | 972 | GUI event loop support |
|
973 | 973 | ====================== |
|
974 | 974 | |
|
975 | 975 | .. versionadded:: 0.11 |
|
976 | 976 | The ``%gui`` magic and :mod:`IPython.lib.inputhook`. |
|
977 | 977 | |
|
978 | 978 | IPython has excellent support for working interactively with Graphical User |
|
979 | 979 | Interface (GUI) toolkits, such as wxPython, PyQt4/PySide, PyGTK and Tk. This is |
|
980 | 980 | implemented using Python's builtin ``PyOSInputHook`` hook. This implementation |
|
981 | 981 | is extremely robust compared to our previous thread-based version. The |
|
982 | 982 | advantages of this are: |
|
983 | 983 | |
|
984 | 984 | * GUIs can be enabled and disabled dynamically at runtime. |
|
985 | 985 | * The active GUI can be switched dynamically at runtime. |
|
986 | 986 | * In some cases, multiple GUIs can run simultaneously with no problems. |
|
987 | 987 | * There is a developer API in :mod:`IPython.lib.inputhook` for customizing |
|
988 | 988 | all of these things. |
|
989 | 989 | |
|
990 | 990 | For users, enabling GUI event loop integration is simple. You simple use the |
|
991 | 991 | ``%gui`` magic as follows:: |
|
992 | 992 | |
|
993 | 993 | %gui [GUINAME] |
|
994 | 994 | |
|
995 | 995 | With no arguments, ``%gui`` removes all GUI support. Valid ``GUINAME`` |
|
996 | 996 | arguments are ``wx``, ``qt``, ``gtk`` and ``tk``. |
|
997 | 997 | |
|
998 | 998 | Thus, to use wxPython interactively and create a running :class:`wx.App` |
|
999 | 999 | object, do:: |
|
1000 | 1000 | |
|
1001 | 1001 | %gui wx |
|
1002 | 1002 | |
|
1003 | 1003 | For information on IPython's Matplotlib integration (and the ``pylab`` mode) |
|
1004 | 1004 | see :ref:`this section <matplotlib_support>`. |
|
1005 | 1005 | |
|
1006 | 1006 | For developers that want to use IPython's GUI event loop integration in the |
|
1007 | 1007 | form of a library, these capabilities are exposed in library form in the |
|
1008 | 1008 | :mod:`IPython.lib.inputhook` and :mod:`IPython.lib.guisupport` modules. |
|
1009 | 1009 | Interested developers should see the module docstrings for more information, |
|
1010 | 1010 | but there are a few points that should be mentioned here. |
|
1011 | 1011 | |
|
1012 | 1012 | First, the ``PyOSInputHook`` approach only works in command line settings |
|
1013 | 1013 | where readline is activated. The integration with various eventloops |
|
1014 | 1014 | is handled somewhat differently (and more simply) when using the standalone |
|
1015 | 1015 | kernel, as in the qtconsole and notebook. |
|
1016 | 1016 | |
|
1017 | 1017 | Second, when using the ``PyOSInputHook`` approach, a GUI application should |
|
1018 | 1018 | *not* start its event loop. Instead all of this is handled by the |
|
1019 | 1019 | ``PyOSInputHook``. This means that applications that are meant to be used both |
|
1020 | 1020 | in IPython and as standalone apps need to have special code to detects how the |
|
1021 | 1021 | application is being run. We highly recommend using IPython's support for this. |
|
1022 | 1022 | Since the details vary slightly between toolkits, we point you to the various |
|
1023 | 1023 | examples in our source directory :file:`docs/examples/lib` that demonstrate |
|
1024 | 1024 | these capabilities. |
|
1025 | 1025 | |
|
1026 | 1026 | Third, unlike previous versions of IPython, we no longer "hijack" (replace |
|
1027 | 1027 | them with no-ops) the event loops. This is done to allow applications that |
|
1028 | 1028 | actually need to run the real event loops to do so. This is often needed to |
|
1029 | 1029 | process pending events at critical points. |
|
1030 | 1030 | |
|
1031 | 1031 | Finally, we also have a number of examples in our source directory |
|
1032 | 1032 | :file:`docs/examples/lib` that demonstrate these capabilities. |
|
1033 | 1033 | |
|
1034 | 1034 | PyQt and PySide |
|
1035 | 1035 | --------------- |
|
1036 | 1036 | |
|
1037 | 1037 | .. attempt at explanation of the complete mess that is Qt support |
|
1038 | 1038 | |
|
1039 | 1039 | When you use ``--gui=qt`` or ``--pylab=qt``, IPython can work with either |
|
1040 | 1040 | PyQt4 or PySide. There are three options for configuration here, because |
|
1041 | 1041 | PyQt4 has two APIs for QString and QVariant - v1, which is the default on |
|
1042 | 1042 | Python 2, and the more natural v2, which is the only API supported by PySide. |
|
1043 | 1043 | v2 is also the default for PyQt4 on Python 3. IPython's code for the QtConsole |
|
1044 | 1044 | uses v2, but you can still use any interface in your code, since the |
|
1045 | 1045 | Qt frontend is in a different process. |
|
1046 | 1046 | |
|
1047 | 1047 | The default will be to import PyQt4 without configuration of the APIs, thus |
|
1048 | 1048 | matching what most applications would expect. It will fall back of PySide if |
|
1049 | 1049 | PyQt4 is unavailable. |
|
1050 | 1050 | |
|
1051 | 1051 | If specified, IPython will respect the environment variable ``QT_API`` used |
|
1052 | 1052 | by ETS. ETS 4.0 also works with both PyQt4 and PySide, but it requires |
|
1053 | 1053 | PyQt4 to use its v2 API. So if ``QT_API=pyside`` PySide will be used, |
|
1054 | 1054 | and if ``QT_API=pyqt`` then PyQt4 will be used *with the v2 API* for |
|
1055 | 1055 | QString and QVariant, so ETS codes like MayaVi will also work with IPython. |
|
1056 | 1056 | |
|
1057 | 1057 | If you launch IPython in pylab mode with ``ipython --pylab=qt``, then IPython |
|
1058 | 1058 | will ask matplotlib which Qt library to use (only if QT_API is *not set*), via |
|
1059 | 1059 | the 'backend.qt4' rcParam. If matplotlib is version 1.0.1 or older, then |
|
1060 | 1060 | IPython will always use PyQt4 without setting the v2 APIs, since neither v2 |
|
1061 | 1061 | PyQt nor PySide work. |
|
1062 | 1062 | |
|
1063 | 1063 | .. warning:: |
|
1064 | 1064 | |
|
1065 | 1065 | Note that this means for ETS 4 to work with PyQt4, ``QT_API`` *must* be set |
|
1066 | 1066 | to work with IPython's qt integration, because otherwise PyQt4 will be |
|
1067 | 1067 | loaded in an incompatible mode. |
|
1068 | 1068 | |
|
1069 | 1069 | It also means that you must *not* have ``QT_API`` set if you want to |
|
1070 | 1070 | use ``--gui=qt`` with code that requires PyQt4 API v1. |
|
1071 | 1071 | |
|
1072 | 1072 | |
|
1073 | 1073 | .. _matplotlib_support: |
|
1074 | 1074 | |
|
1075 | 1075 | Plotting with matplotlib |
|
1076 | 1076 | ======================== |
|
1077 | 1077 | |
|
1078 | 1078 | `Matplotlib`_ provides high quality 2D and 3D plotting for Python. Matplotlib |
|
1079 | 1079 | can produce plots on screen using a variety of GUI toolkits, including Tk, |
|
1080 | 1080 | PyGTK, PyQt4 and wxPython. It also provides a number of commands useful for |
|
1081 | 1081 | scientific computing, all with a syntax compatible with that of the popular |
|
1082 | 1082 | Matlab program. |
|
1083 | 1083 | |
|
1084 | 1084 | To start IPython with matplotlib support, use the ``--pylab`` switch. If no |
|
1085 | 1085 | arguments are given, IPython will automatically detect your choice of |
|
1086 | 1086 | matplotlib backend. You can also request a specific backend with ``--pylab |
|
1087 | 1087 | backend``, where ``backend`` must be one of: 'tk', 'qt', 'wx', 'gtk', 'osx'. |
|
1088 | 1088 | In the web notebook and Qt console, 'inline' is also a valid backend value, |
|
1089 | 1089 | which produces static figures inlined inside the application window instead of |
|
1090 | 1090 | matplotlib's interactive figures that live in separate windows. |
|
1091 | 1091 | |
|
1092 | 1092 | .. _Matplotlib: http://matplotlib.sourceforge.net |
|
1093 | 1093 | |
|
1094 | 1094 | .. _interactive_demos: |
|
1095 | 1095 | |
|
1096 | 1096 | Interactive demos with IPython |
|
1097 | 1097 | ============================== |
|
1098 | 1098 | |
|
1099 | 1099 | IPython ships with a basic system for running scripts interactively in |
|
1100 | 1100 | sections, useful when presenting code to audiences. A few tags embedded |
|
1101 | 1101 | in comments (so that the script remains valid Python code) divide a file |
|
1102 | 1102 | into separate blocks, and the demo can be run one block at a time, with |
|
1103 | 1103 | IPython printing (with syntax highlighting) the block before executing |
|
1104 | 1104 | it, and returning to the interactive prompt after each block. The |
|
1105 | 1105 | interactive namespace is updated after each block is run with the |
|
1106 | 1106 | contents of the demo's namespace. |
|
1107 | 1107 | |
|
1108 | 1108 | This allows you to show a piece of code, run it and then execute |
|
1109 | 1109 | interactively commands based on the variables just created. Once you |
|
1110 | 1110 | want to continue, you simply execute the next block of the demo. The |
|
1111 | 1111 | following listing shows the markup necessary for dividing a script into |
|
1112 | 1112 | sections for execution as a demo: |
|
1113 | 1113 | |
|
1114 | .. literalinclude:: ../../examples/lib/example-demo.py | |
|
1114 | .. literalinclude:: ../../../examples/lib/example-demo.py | |
|
1115 | 1115 | :language: python |
|
1116 | 1116 | |
|
1117 | 1117 | In order to run a file as a demo, you must first make a Demo object out |
|
1118 | 1118 | of it. If the file is named myscript.py, the following code will make a |
|
1119 | 1119 | demo:: |
|
1120 | 1120 | |
|
1121 | 1121 | from IPython.lib.demo import Demo |
|
1122 | 1122 | |
|
1123 | 1123 | mydemo = Demo('myscript.py') |
|
1124 | 1124 | |
|
1125 | 1125 | This creates the mydemo object, whose blocks you run one at a time by |
|
1126 | 1126 | simply calling the object with no arguments. If you have autocall active |
|
1127 | 1127 | in IPython (the default), all you need to do is type:: |
|
1128 | 1128 | |
|
1129 | 1129 | mydemo |
|
1130 | 1130 | |
|
1131 | 1131 | and IPython will call it, executing each block. Demo objects can be |
|
1132 | 1132 | restarted, you can move forward or back skipping blocks, re-execute the |
|
1133 | 1133 | last block, etc. Simply use the Tab key on a demo object to see its |
|
1134 | 1134 | methods, and call '?' on them to see their docstrings for more usage |
|
1135 | 1135 | details. In addition, the demo module itself contains a comprehensive |
|
1136 | 1136 | docstring, which you can access via:: |
|
1137 | 1137 | |
|
1138 | 1138 | from IPython.lib import demo |
|
1139 | 1139 | |
|
1140 | 1140 | demo? |
|
1141 | 1141 | |
|
1142 | 1142 | Limitations: It is important to note that these demos are limited to |
|
1143 | 1143 | fairly simple uses. In particular, you cannot break up sections within |
|
1144 | 1144 | indented code (loops, if statements, function definitions, etc.) |
|
1145 | 1145 | Supporting something like this would basically require tracking the |
|
1146 | 1146 | internal execution state of the Python interpreter, so only top-level |
|
1147 | 1147 | divisions are allowed. If you want to be able to open an IPython |
|
1148 | 1148 | instance at an arbitrary point in a program, you can use IPython's |
|
1149 | 1149 | embedding facilities, see :func:`IPython.embed` for details. |
|
1150 | 1150 |
@@ -1,150 +1,150 b'' | |||
|
1 | 1 | .. _parallel_asyncresult: |
|
2 | 2 | |
|
3 | 3 | ====================== |
|
4 | 4 | The AsyncResult object |
|
5 | 5 | ====================== |
|
6 | 6 | |
|
7 | 7 | In non-blocking mode, :meth:`apply` submits the command to be executed and |
|
8 | 8 | then returns a :class:`~.AsyncResult` object immediately. The |
|
9 | 9 | AsyncResult object gives you a way of getting a result at a later |
|
10 | 10 | time through its :meth:`get` method, but it also collects metadata |
|
11 | 11 | on execution. |
|
12 | 12 | |
|
13 | 13 | |
|
14 | 14 | Beyond multiprocessing's AsyncResult |
|
15 | 15 | ==================================== |
|
16 | 16 | |
|
17 | 17 | .. Note:: |
|
18 | 18 | |
|
19 | 19 | The :class:`~.AsyncResult` object provides a superset of the interface in |
|
20 | 20 | :py:class:`multiprocessing.pool.AsyncResult`. See the |
|
21 | 21 | `official Python documentation <http://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_ |
|
22 | 22 | for more on the basics of this interface. |
|
23 | 23 | |
|
24 | 24 | Our AsyncResult objects add a number of convenient features for working with |
|
25 | 25 | parallel results, beyond what is provided by the original AsyncResult. |
|
26 | 26 | |
|
27 | 27 | |
|
28 | 28 | get_dict |
|
29 | 29 | -------- |
|
30 | 30 | |
|
31 | 31 | First, is :meth:`.AsyncResult.get_dict`, which pulls results as a dictionary |
|
32 | 32 | keyed by engine_id, rather than a flat list. This is useful for quickly |
|
33 | 33 | coordinating or distributing information about all of the engines. |
|
34 | 34 | |
|
35 | 35 | As an example, here is a quick call that gives every engine a dict showing |
|
36 | 36 | the PID of every other engine: |
|
37 | 37 | |
|
38 | 38 | .. sourcecode:: ipython |
|
39 | 39 | |
|
40 | 40 | In [10]: ar = rc[:].apply_async(os.getpid) |
|
41 | 41 | In [11]: pids = ar.get_dict() |
|
42 | 42 | In [12]: rc[:]['pid_map'] = pids |
|
43 | 43 | |
|
44 | 44 | This trick is particularly useful when setting up inter-engine communication, |
|
45 | 45 | as in IPython's :file:`examples/parallel/interengine` examples. |
|
46 | 46 | |
|
47 | 47 | |
|
48 | 48 | Metadata |
|
49 | 49 | ======== |
|
50 | 50 | |
|
51 | 51 | IPython.parallel tracks some metadata about the tasks, which is stored |
|
52 | 52 | in the :attr:`.Client.metadata` dict. The AsyncResult object gives you an |
|
53 | 53 | interface for this information as well, including timestamps stdout/err, |
|
54 | 54 | and engine IDs. |
|
55 | 55 | |
|
56 | 56 | |
|
57 | 57 | Timing |
|
58 | 58 | ------ |
|
59 | 59 | |
|
60 | 60 | IPython tracks various timestamps as :py:class:`.datetime` objects, |
|
61 | 61 | and the AsyncResult object has a few properties that turn these into useful |
|
62 | 62 | times (in seconds as floats). |
|
63 | 63 | |
|
64 | 64 | For use while the tasks are still pending: |
|
65 | 65 | |
|
66 | 66 | * :attr:`ar.elapsed` is just the elapsed seconds since submission, for use |
|
67 | 67 | before the AsyncResult is complete. |
|
68 | 68 | * :attr:`ar.progress` is the number of tasks that have completed. Fractional progress |
|
69 | 69 | would be:: |
|
70 | 70 | |
|
71 | 71 | 1.0 * ar.progress / len(ar) |
|
72 | 72 | |
|
73 | 73 | * :meth:`AsyncResult.wait_interactive` will wait for the result to finish, but |
|
74 | 74 | print out status updates on progress and elapsed time while it waits. |
|
75 | 75 | |
|
76 | 76 | For use after the tasks are done: |
|
77 | 77 | |
|
78 | 78 | * :attr:`ar.serial_time` is the sum of the computation time of all of the tasks |
|
79 | 79 | done in parallel. |
|
80 | 80 | * :attr:`ar.wall_time` is the time between the first task submitted and last result |
|
81 | 81 | received. This is the actual cost of computation, including IPython overhead. |
|
82 | 82 | |
|
83 | 83 | |
|
84 | 84 | .. note:: |
|
85 | 85 | |
|
86 | 86 | wall_time is only precise if the Client is waiting for results when |
|
87 | 87 | the task finished, because the `received` timestamp is made when the result is |
|
88 | 88 | unpacked by the Client, triggered by the :meth:`~Client.spin` call. If you |
|
89 | 89 | are doing work in the Client, and not waiting/spinning, then `received` might |
|
90 | 90 | be artificially high. |
|
91 | 91 | |
|
92 | 92 | An often interesting metric is the time it actually cost to do the work in parallel |
|
93 | 93 | relative to the serial computation, and this can be given simply with |
|
94 | 94 | |
|
95 | 95 | .. sourcecode:: python |
|
96 | 96 | |
|
97 | 97 | speedup = ar.serial_time / ar.wall_time |
|
98 | 98 | |
|
99 | 99 | |
|
100 | 100 | Map results are iterable! |
|
101 | 101 | ========================= |
|
102 | 102 | |
|
103 | 103 | When an AsyncResult object has multiple results (e.g. the :class:`~AsyncMapResult` |
|
104 | 104 | object), you can actually iterate through results themselves, and act on them as they arrive: |
|
105 | 105 | |
|
106 | .. literalinclude:: ../../examples/parallel/itermapresult.py | |
|
106 | .. literalinclude:: ../../../examples/parallel/itermapresult.py | |
|
107 | 107 | :language: python |
|
108 | 108 | :lines: 20-67 |
|
109 | 109 | |
|
110 | 110 | That is to say, if you treat an AsyncMapResult as if it were a list of your actual |
|
111 | 111 | results, it should behave as you would expect, with the only difference being |
|
112 | 112 | that you can start iterating through the results before they have even been computed. |
|
113 | 113 | |
|
114 | 114 | This lets you do a dumb version of map/reduce with the builtin Python functions, |
|
115 | 115 | and the only difference between doing this locally and doing it remotely in parallel |
|
116 | 116 | is using the asynchronous view.map instead of the builtin map. |
|
117 | 117 | |
|
118 | 118 | |
|
119 | 119 | Here is a simple one-line RMS (root-mean-square) implemented with Python's builtin map/reduce. |
|
120 | 120 | |
|
121 | 121 | .. sourcecode:: ipython |
|
122 | 122 | |
|
123 | 123 | In [38]: X = np.linspace(0,100) |
|
124 | 124 | |
|
125 | 125 | In [39]: from math import sqrt |
|
126 | 126 | |
|
127 | 127 | In [40]: add = lambda a,b: a+b |
|
128 | 128 | |
|
129 | 129 | In [41]: sq = lambda x: x*x |
|
130 | 130 | |
|
131 | 131 | In [42]: sqrt(reduce(add, map(sq, X)) / len(X)) |
|
132 | 132 | Out[42]: 58.028845747399714 |
|
133 | 133 | |
|
134 | 134 | In [43]: sqrt(reduce(add, view.map(sq, X)) / len(X)) |
|
135 | 135 | Out[43]: 58.028845747399714 |
|
136 | 136 | |
|
137 | 137 | To break that down: |
|
138 | 138 | |
|
139 | 139 | 1. ``map(sq, X)`` Compute the square of each element in the list (locally, or in parallel) |
|
140 | 140 | 2. ``reduce(add, sqX) / len(X)`` compute the mean by summing over the list (or AsyncMapResult) |
|
141 | 141 | and dividing by the size |
|
142 | 142 | 3. take the square root of the resulting number |
|
143 | 143 | |
|
144 | 144 | .. seealso:: |
|
145 | 145 | |
|
146 | 146 | When AsyncResult or the AsyncMapResult don't provide what you need (for instance, |
|
147 | 147 | handling individual results as they arrive, but with metadata), you can always |
|
148 | 148 | just split the original result's ``msg_ids`` attribute, and handle them as you like. |
|
149 | 149 | |
|
150 | 150 | For an example of this, see :file:`docs/examples/parallel/customresult.py` |
@@ -1,177 +1,177 b'' | |||
|
1 | 1 | .. _dag_dependencies: |
|
2 | 2 | |
|
3 | 3 | ================ |
|
4 | 4 | DAG Dependencies |
|
5 | 5 | ================ |
|
6 | 6 | |
|
7 | 7 | Often, parallel workflow is described in terms of a `Directed Acyclic Graph |
|
8 | 8 | <http://en.wikipedia.org/wiki/Directed_acyclic_graph>`_ or DAG. A popular library |
|
9 | 9 | for working with Graphs is NetworkX_. Here, we will walk through a demo mapping |
|
10 | 10 | a nx DAG to task dependencies. |
|
11 | 11 | |
|
12 | 12 | The full script that runs this demo can be found in |
|
13 | 13 | :file:`docs/examples/parallel/dagdeps.py`. |
|
14 | 14 | |
|
15 | 15 | Why are DAGs good for task dependencies? |
|
16 | 16 | ---------------------------------------- |
|
17 | 17 | |
|
18 | 18 | The 'G' in DAG is 'Graph'. A Graph is a collection of **nodes** and **edges** that connect |
|
19 | 19 | the nodes. For our purposes, each node would be a task, and each edge would be a |
|
20 | 20 | dependency. The 'D' in DAG stands for 'Directed'. This means that each edge has a |
|
21 | 21 | direction associated with it. So we can interpret the edge (a,b) as meaning that b depends |
|
22 | 22 | on a, whereas the edge (b,a) would mean a depends on b. The 'A' is 'Acyclic', meaning that |
|
23 | 23 | there must not be any closed loops in the graph. This is important for dependencies, |
|
24 | 24 | because if a loop were closed, then a task could ultimately depend on itself, and never be |
|
25 | 25 | able to run. If your workflow can be described as a DAG, then it is impossible for your |
|
26 | 26 | dependencies to cause a deadlock. |
|
27 | 27 | |
|
28 | 28 | A Sample DAG |
|
29 | 29 | ------------ |
|
30 | 30 | |
|
31 | 31 | Here, we have a very simple 5-node DAG: |
|
32 | 32 | |
|
33 | 33 | .. figure:: figs/simpledag.* |
|
34 | 34 | :width: 600px |
|
35 | 35 | |
|
36 | 36 | With NetworkX, an arrow is just a fattened bit on the edge. Here, we can see that task 0 |
|
37 | 37 | depends on nothing, and can run immediately. 1 and 2 depend on 0; 3 depends on |
|
38 | 38 | 1 and 2; and 4 depends only on 1. |
|
39 | 39 | |
|
40 | 40 | A possible sequence of events for this workflow: |
|
41 | 41 | |
|
42 | 42 | 0. Task 0 can run right away |
|
43 | 43 | 1. 0 finishes, so 1,2 can start |
|
44 | 44 | 2. 1 finishes, 3 is still waiting on 2, but 4 can start right away |
|
45 | 45 | 3. 2 finishes, and 3 can finally start |
|
46 | 46 | |
|
47 | 47 | |
|
48 | 48 | Further, taking failures into account, assuming all dependencies are run with the default |
|
49 | 49 | `success=True,failure=False`, the following cases would occur for each node's failure: |
|
50 | 50 | |
|
51 | 51 | 0. fails: all other tasks fail as Impossible |
|
52 | 52 | 1. 2 can still succeed, but 3,4 are unreachable |
|
53 | 53 | 2. 3 becomes unreachable, but 4 is unaffected |
|
54 | 54 | 3. and 4. are terminal, and can have no effect on other nodes |
|
55 | 55 | |
|
56 | 56 | The code to generate the simple DAG: |
|
57 | 57 | |
|
58 | 58 | .. sourcecode:: python |
|
59 | 59 | |
|
60 | 60 | import networkx as nx |
|
61 | 61 | |
|
62 | 62 | G = nx.DiGraph() |
|
63 | 63 | |
|
64 | 64 | # add 5 nodes, labeled 0-4: |
|
65 | 65 | map(G.add_node, range(5)) |
|
66 | 66 | # 1,2 depend on 0: |
|
67 | 67 | G.add_edge(0,1) |
|
68 | 68 | G.add_edge(0,2) |
|
69 | 69 | # 3 depends on 1,2 |
|
70 | 70 | G.add_edge(1,3) |
|
71 | 71 | G.add_edge(2,3) |
|
72 | 72 | # 4 depends on 1 |
|
73 | 73 | G.add_edge(1,4) |
|
74 | 74 | |
|
75 | 75 | # now draw the graph: |
|
76 | 76 | pos = { 0 : (0,0), 1 : (1,1), 2 : (-1,1), |
|
77 | 77 | 3 : (0,2), 4 : (2,2)} |
|
78 | 78 | nx.draw(G, pos, edge_color='r') |
|
79 | 79 | |
|
80 | 80 | |
|
81 | 81 | For demonstration purposes, we have a function that generates a random DAG with a given |
|
82 | 82 | number of nodes and edges. |
|
83 | 83 | |
|
84 | .. literalinclude:: ../../examples/parallel/dagdeps.py | |
|
84 | .. literalinclude:: ../../../examples/parallel/dagdeps.py | |
|
85 | 85 | :language: python |
|
86 | 86 | :lines: 20-36 |
|
87 | 87 | |
|
88 | 88 | So first, we start with a graph of 32 nodes, with 128 edges: |
|
89 | 89 | |
|
90 | 90 | .. sourcecode:: ipython |
|
91 | 91 | |
|
92 | 92 | In [2]: G = random_dag(32,128) |
|
93 | 93 | |
|
94 | 94 | Now, we need to build our dict of jobs corresponding to the nodes on the graph: |
|
95 | 95 | |
|
96 | 96 | .. sourcecode:: ipython |
|
97 | 97 | |
|
98 | 98 | In [3]: jobs = {} |
|
99 | 99 | |
|
100 | 100 | # in reality, each job would presumably be different |
|
101 | 101 | # randomwait is just a function that sleeps for a random interval |
|
102 | 102 | In [4]: for node in G: |
|
103 | 103 | ...: jobs[node] = randomwait |
|
104 | 104 | |
|
105 | 105 | Once we have a dict of jobs matching the nodes on the graph, we can start submitting jobs, |
|
106 | 106 | and linking up the dependencies. Since we don't know a job's msg_id until it is submitted, |
|
107 | 107 | which is necessary for building dependencies, it is critical that we don't submit any jobs |
|
108 | 108 | before other jobs it may depend on. Fortunately, NetworkX provides a |
|
109 | 109 | :meth:`topological_sort` method which ensures exactly this. It presents an iterable, that |
|
110 | 110 | guarantees that when you arrive at a node, you have already visited all the nodes it |
|
111 | 111 | on which it depends: |
|
112 | 112 | |
|
113 | 113 | .. sourcecode:: ipython |
|
114 | 114 | |
|
115 | 115 | In [5]: rc = Client() |
|
116 | 116 | In [5]: view = rc.load_balanced_view() |
|
117 | 117 | |
|
118 | 118 | In [6]: results = {} |
|
119 | 119 | |
|
120 | 120 | In [7]: for node in G.topological_sort(): |
|
121 | 121 | ...: # get list of AsyncResult objects from nodes |
|
122 | 122 | ...: # leading into this one as dependencies |
|
123 | 123 | ...: deps = [ results[n] for n in G.predecessors(node) ] |
|
124 | 124 | ...: # submit and store AsyncResult object |
|
125 | 125 | ...: with view.temp_flags(after=deps, block=False): |
|
126 | 126 | ...: results[node] = view.apply_with_flags(jobs[node]) |
|
127 | 127 | |
|
128 | 128 | |
|
129 | 129 | Now that we have submitted all the jobs, we can wait for the results: |
|
130 | 130 | |
|
131 | 131 | .. sourcecode:: ipython |
|
132 | 132 | |
|
133 | 133 | In [8]: view.wait(results.values()) |
|
134 | 134 | |
|
135 | 135 | Now, at least we know that all the jobs ran and did not fail (``r.get()`` would have |
|
136 | 136 | raised an error if a task failed). But we don't know that the ordering was properly |
|
137 | 137 | respected. For this, we can use the :attr:`metadata` attribute of each AsyncResult. |
|
138 | 138 | |
|
139 | 139 | These objects store a variety of metadata about each task, including various timestamps. |
|
140 | 140 | We can validate that the dependencies were respected by checking that each task was |
|
141 | 141 | started after all of its predecessors were completed: |
|
142 | 142 | |
|
143 | .. literalinclude:: ../../examples/parallel/dagdeps.py | |
|
143 | .. literalinclude:: ../../../examples/parallel/dagdeps.py | |
|
144 | 144 | :language: python |
|
145 | 145 | :lines: 64-70 |
|
146 | 146 | |
|
147 | 147 | We can also validate the graph visually. By drawing the graph with each node's x-position |
|
148 | 148 | as its start time, all arrows must be pointing to the right if dependencies were respected. |
|
149 | 149 | For spreading, the y-position will be the runtime of the task, so long tasks |
|
150 | 150 | will be at the top, and quick, small tasks will be at the bottom. |
|
151 | 151 | |
|
152 | 152 | .. sourcecode:: ipython |
|
153 | 153 | |
|
154 | 154 | In [10]: from matplotlib.dates import date2num |
|
155 | 155 | |
|
156 | 156 | In [11]: from matplotlib.cm import gist_rainbow |
|
157 | 157 | |
|
158 | 158 | In [12]: pos = {}; colors = {} |
|
159 | 159 | |
|
160 | 160 | In [12]: for node in G: |
|
161 | 161 | ....: md = results[node].metadata |
|
162 | 162 | ....: start = date2num(md.started) |
|
163 | 163 | ....: runtime = date2num(md.completed) - start |
|
164 | 164 | ....: pos[node] = (start, runtime) |
|
165 | 165 | ....: colors[node] = md.engine_id |
|
166 | 166 | |
|
167 | 167 | In [13]: nx.draw(G, pos, node_list=colors.keys(), node_color=colors.values(), |
|
168 | 168 | ....: cmap=gist_rainbow) |
|
169 | 169 | |
|
170 | 170 | .. figure:: figs/dagdeps.* |
|
171 | 171 | :width: 600px |
|
172 | 172 | |
|
173 | 173 | Time started on x, runtime on y, and color-coded by engine-id (in this case there |
|
174 | 174 | were four engines). Edges denote dependencies. |
|
175 | 175 | |
|
176 | 176 | |
|
177 | 177 | .. _NetworkX: http://networkx.lanl.gov/ |
@@ -1,277 +1,205 b'' | |||
|
1 | 1 | .. _parallel_examples: |
|
2 | 2 | |
|
3 | 3 | ================= |
|
4 | 4 | Parallel examples |
|
5 | 5 | ================= |
|
6 | 6 | |
|
7 | 7 | In this section we describe two more involved examples of using an IPython |
|
8 | 8 | cluster to perform a parallel computation. In these examples, we will be using |
|
9 | 9 | IPython's "pylab" mode, which enables interactive plotting using the |
|
10 | 10 | Matplotlib package. IPython can be started in this mode by typing:: |
|
11 | 11 | |
|
12 | 12 | ipython --pylab |
|
13 | 13 | |
|
14 | 14 | at the system command line. |
|
15 | 15 | |
|
16 | 16 | 150 million digits of pi |
|
17 | 17 | ======================== |
|
18 | 18 | |
|
19 | 19 | In this example we would like to study the distribution of digits in the |
|
20 | 20 | number pi (in base 10). While it is not known if pi is a normal number (a |
|
21 | 21 | number is normal in base 10 if 0-9 occur with equal likelihood) numerical |
|
22 | 22 | investigations suggest that it is. We will begin with a serial calculation on |
|
23 | 23 | 10,000 digits of pi and then perform a parallel calculation involving 150 |
|
24 | 24 | million digits. |
|
25 | 25 | |
|
26 | 26 | In both the serial and parallel calculation we will be using functions defined |
|
27 | 27 | in the :file:`pidigits.py` file, which is available in the |
|
28 | 28 | :file:`docs/examples/parallel` directory of the IPython source distribution. |
|
29 | 29 | These functions provide basic facilities for working with the digits of pi and |
|
30 | 30 | can be loaded into IPython by putting :file:`pidigits.py` in your current |
|
31 | 31 | working directory and then doing: |
|
32 | 32 | |
|
33 | 33 | .. sourcecode:: ipython |
|
34 | 34 | |
|
35 | 35 | In [1]: run pidigits.py |
|
36 | 36 | |
|
37 | 37 | Serial calculation |
|
38 | 38 | ------------------ |
|
39 | 39 | |
|
40 | 40 | For the serial calculation, we will use `SymPy <http://www.sympy.org>`_ to |
|
41 | 41 | calculate 10,000 digits of pi and then look at the frequencies of the digits |
|
42 | 42 | 0-9. Out of 10,000 digits, we expect each digit to occur 1,000 times. While |
|
43 | 43 | SymPy is capable of calculating many more digits of pi, our purpose here is to |
|
44 | 44 | set the stage for the much larger parallel calculation. |
|
45 | 45 | |
|
46 | 46 | In this example, we use two functions from :file:`pidigits.py`: |
|
47 | 47 | :func:`one_digit_freqs` (which calculates how many times each digit occurs) |
|
48 | 48 | and :func:`plot_one_digit_freqs` (which uses Matplotlib to plot the result). |
|
49 | 49 | Here is an interactive IPython session that uses these functions with |
|
50 | 50 | SymPy: |
|
51 | 51 | |
|
52 | 52 | .. sourcecode:: ipython |
|
53 | 53 | |
|
54 | 54 | In [7]: import sympy |
|
55 | 55 | |
|
56 | 56 | In [8]: pi = sympy.pi.evalf(40) |
|
57 | 57 | |
|
58 | 58 | In [9]: pi |
|
59 | 59 | Out[9]: 3.141592653589793238462643383279502884197 |
|
60 | 60 | |
|
61 | 61 | In [10]: pi = sympy.pi.evalf(10000) |
|
62 | 62 | |
|
63 | 63 | In [11]: digits = (d for d in str(pi)[2:]) # create a sequence of digits |
|
64 | 64 | |
|
65 | 65 | In [12]: run pidigits.py # load one_digit_freqs/plot_one_digit_freqs |
|
66 | 66 | |
|
67 | 67 | In [13]: freqs = one_digit_freqs(digits) |
|
68 | 68 | |
|
69 | 69 | In [14]: plot_one_digit_freqs(freqs) |
|
70 | 70 | Out[14]: [<matplotlib.lines.Line2D object at 0x18a55290>] |
|
71 | 71 | |
|
72 | 72 | The resulting plot of the single digit counts shows that each digit occurs |
|
73 | 73 | approximately 1,000 times, but that with only 10,000 digits the |
|
74 | 74 | statistical fluctuations are still rather large: |
|
75 | 75 | |
|
76 | 76 | .. image:: figs/single_digits.* |
|
77 | 77 | |
|
78 | 78 | It is clear that to reduce the relative fluctuations in the counts, we need |
|
79 | 79 | to look at many more digits of pi. That brings us to the parallel calculation. |
|
80 | 80 | |
|
81 | 81 | Parallel calculation |
|
82 | 82 | -------------------- |
|
83 | 83 | |
|
84 | 84 | Calculating many digits of pi is a challenging computational problem in itself. |
|
85 | 85 | Because we want to focus on the distribution of digits in this example, we |
|
86 | 86 | will use pre-computed digit of pi from the website of Professor Yasumasa |
|
87 | 87 | Kanada at the University of Tokyo (http://www.super-computing.org). These |
|
88 | 88 | digits come in a set of text files (ftp://pi.super-computing.org/.2/pi200m/) |
|
89 | 89 | that each have 10 million digits of pi. |
|
90 | 90 | |
|
91 | 91 | For the parallel calculation, we have copied these files to the local hard |
|
92 | 92 | drives of the compute nodes. A total of 15 of these files will be used, for a |
|
93 | 93 | total of 150 million digits of pi. To make things a little more interesting we |
|
94 | 94 | will calculate the frequencies of all 2 digits sequences (00-99) and then plot |
|
95 | 95 | the result using a 2D matrix in Matplotlib. |
|
96 | 96 | |
|
97 | 97 | The overall idea of the calculation is simple: each IPython engine will |
|
98 | 98 | compute the two digit counts for the digits in a single file. Then in a final |
|
99 | 99 | step the counts from each engine will be added up. To perform this |
|
100 | 100 | calculation, we will need two top-level functions from :file:`pidigits.py`: |
|
101 | 101 | |
|
102 | .. literalinclude:: ../../examples/parallel/pi/pidigits.py | |
|
102 | .. literalinclude:: ../../../examples/parallel/pi/pidigits.py | |
|
103 | 103 | :language: python |
|
104 | 104 | :lines: 47-62 |
|
105 | 105 | |
|
106 | 106 | We will also use the :func:`plot_two_digit_freqs` function to plot the |
|
107 | 107 | results. The code to run this calculation in parallel is contained in |
|
108 | 108 | :file:`docs/examples/parallel/parallelpi.py`. This code can be run in parallel |
|
109 | 109 | using IPython by following these steps: |
|
110 | 110 | |
|
111 | 111 | 1. Use :command:`ipcluster` to start 15 engines. We used 16 cores of an SGE linux |
|
112 | 112 | cluster (1 controller + 15 engines). |
|
113 | 113 | 2. With the file :file:`parallelpi.py` in your current working directory, open |
|
114 | 114 | up IPython in pylab mode and type ``run parallelpi.py``. This will download |
|
115 | 115 | the pi files via ftp the first time you run it, if they are not |
|
116 | 116 | present in the Engines' working directory. |
|
117 | 117 | |
|
118 | 118 | When run on our 16 cores, we observe a speedup of 14.2x. This is slightly |
|
119 | 119 | less than linear scaling (16x) because the controller is also running on one of |
|
120 | 120 | the cores. |
|
121 | 121 | |
|
122 | 122 | To emphasize the interactive nature of IPython, we now show how the |
|
123 | 123 | calculation can also be run by simply typing the commands from |
|
124 | 124 | :file:`parallelpi.py` interactively into IPython: |
|
125 | 125 | |
|
126 | 126 | .. sourcecode:: ipython |
|
127 | 127 | |
|
128 | 128 | In [1]: from IPython.parallel import Client |
|
129 | 129 | |
|
130 | 130 | # The Client allows us to use the engines interactively. |
|
131 | 131 | # We simply pass Client the name of the cluster profile we |
|
132 | 132 | # are using. |
|
133 | 133 | In [2]: c = Client(profile='mycluster') |
|
134 | 134 | In [3]: v = c[:] |
|
135 | 135 | |
|
136 | 136 | In [3]: c.ids |
|
137 | 137 | Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] |
|
138 | 138 | |
|
139 | 139 | In [4]: run pidigits.py |
|
140 | 140 | |
|
141 | 141 | In [5]: filestring = 'pi200m.ascii.%(i)02dof20' |
|
142 | 142 | |
|
143 | 143 | # Create the list of files to process. |
|
144 | 144 | In [6]: files = [filestring % {'i':i} for i in range(1,16)] |
|
145 | 145 | |
|
146 | 146 | In [7]: files |
|
147 | 147 | Out[7]: |
|
148 | 148 | ['pi200m.ascii.01of20', |
|
149 | 149 | 'pi200m.ascii.02of20', |
|
150 | 150 | 'pi200m.ascii.03of20', |
|
151 | 151 | 'pi200m.ascii.04of20', |
|
152 | 152 | 'pi200m.ascii.05of20', |
|
153 | 153 | 'pi200m.ascii.06of20', |
|
154 | 154 | 'pi200m.ascii.07of20', |
|
155 | 155 | 'pi200m.ascii.08of20', |
|
156 | 156 | 'pi200m.ascii.09of20', |
|
157 | 157 | 'pi200m.ascii.10of20', |
|
158 | 158 | 'pi200m.ascii.11of20', |
|
159 | 159 | 'pi200m.ascii.12of20', |
|
160 | 160 | 'pi200m.ascii.13of20', |
|
161 | 161 | 'pi200m.ascii.14of20', |
|
162 | 162 | 'pi200m.ascii.15of20'] |
|
163 | 163 | |
|
164 | 164 | # download the data files if they don't already exist: |
|
165 | 165 | In [8]: v.map(fetch_pi_file, files) |
|
166 | 166 | |
|
167 | 167 | # This is the parallel calculation using the Client.map method |
|
168 | 168 | # which applies compute_two_digit_freqs to each file in files in parallel. |
|
169 | 169 | In [9]: freqs_all = v.map(compute_two_digit_freqs, files) |
|
170 | 170 | |
|
171 | 171 | # Add up the frequencies from each engine. |
|
172 | 172 | In [10]: freqs = reduce_freqs(freqs_all) |
|
173 | 173 | |
|
174 | 174 | In [11]: plot_two_digit_freqs(freqs) |
|
175 | 175 | Out[11]: <matplotlib.image.AxesImage object at 0x18beb110> |
|
176 | 176 | |
|
177 | 177 | In [12]: plt.title('2 digit counts of 150m digits of pi') |
|
178 | 178 | Out[12]: <matplotlib.text.Text object at 0x18d1f9b0> |
|
179 | 179 | |
|
180 | 180 | The resulting plot generated by Matplotlib is shown below. The colors indicate |
|
181 | 181 | which two digit sequences are more (red) or less (blue) likely to occur in the |
|
182 | 182 | first 150 million digits of pi. We clearly see that the sequence "41" is |
|
183 | 183 | most likely and that "06" and "07" are least likely. Further analysis would |
|
184 | 184 | show that the relative size of the statistical fluctuations have decreased |
|
185 | 185 | compared to the 10,000 digit calculation. |
|
186 | 186 | |
|
187 | 187 | .. image:: figs/two_digit_counts.* |
|
188 | 188 | |
|
189 | ||
|
190 | Parallel options pricing | |
|
191 | ======================== | |
|
192 | ||
|
193 | An option is a financial contract that gives the buyer of the contract the | |
|
194 | right to buy (a "call") or sell (a "put") a secondary asset (a stock for | |
|
195 | example) at a particular date in the future (the expiration date) for a | |
|
196 | pre-agreed upon price (the strike price). For this right, the buyer pays the | |
|
197 | seller a premium (the option price). There are a wide variety of flavors of | |
|
198 | options (American, European, Asian, etc.) that are useful for different | |
|
199 | purposes: hedging against risk, speculation, etc. | |
|
200 | ||
|
201 | Much of modern finance is driven by the need to price these contracts | |
|
202 | accurately based on what is known about the properties (such as volatility) of | |
|
203 | the underlying asset. One method of pricing options is to use a Monte Carlo | |
|
204 | simulation of the underlying asset price. In this example we use this approach | |
|
205 | to price both European and Asian (path dependent) options for various strike | |
|
206 | prices and volatilities. | |
|
207 | ||
|
208 | The code for this example can be found in the :file:`docs/examples/parallel/options` | |
|
209 | directory of the IPython source. The function :func:`price_options` in | |
|
210 | :file:`mckernel.py` implements the basic Monte Carlo pricing algorithm using | |
|
211 | the NumPy package and is shown here: | |
|
212 | ||
|
213 | .. literalinclude:: ../../examples/parallel/options/mckernel.py | |
|
214 | :language: python | |
|
215 | ||
|
216 | To run this code in parallel, we will use IPython's :class:`LoadBalancedView` class, | |
|
217 | which distributes work to the engines using dynamic load balancing. This | |
|
218 | view is a wrapper of the :class:`Client` class shown in | |
|
219 | the previous example. The parallel calculation using :class:`LoadBalancedView` can | |
|
220 | be found in the file :file:`mcpricer.py`. The code in this file creates a | |
|
221 | :class:`LoadBalancedView` instance and then submits a set of tasks using | |
|
222 | :meth:`LoadBalancedView.apply` that calculate the option prices for different | |
|
223 | volatilities and strike prices. The results are then plotted as a 2D contour | |
|
224 | plot using Matplotlib. | |
|
225 | ||
|
226 | .. literalinclude:: ../../examples/parallel/options/mcpricer.py | |
|
227 | :language: python | |
|
228 | ||
|
229 | To use this code, start an IPython cluster using :command:`ipcluster`, open | |
|
230 | IPython in the pylab mode with the file :file:`mckernel.py` in your current | |
|
231 | working directory and then type: | |
|
232 | ||
|
233 | .. sourcecode:: ipython | |
|
234 | ||
|
235 | In [7]: run mcpricer.py | |
|
236 | ||
|
237 | Submitted tasks: 30 | |
|
238 | ||
|
239 | Once all the tasks have finished, the results can be plotted using the | |
|
240 | :func:`plot_options` function. Here we make contour plots of the Asian | |
|
241 | call and Asian put options as function of the volatility and strike price: | |
|
242 | ||
|
243 | .. sourcecode:: ipython | |
|
244 | ||
|
245 | In [8]: plot_options(sigma_vals, strike_vals, prices['acall']) | |
|
246 | ||
|
247 | In [9]: plt.figure() | |
|
248 | Out[9]: <matplotlib.figure.Figure object at 0x18c178d0> | |
|
249 | ||
|
250 | In [10]: plot_options(sigma_vals, strike_vals, prices['aput']) | |
|
251 | ||
|
252 | These results are shown in the two figures below. On our 15 engines, the | |
|
253 | entire calculation (15 strike prices, 15 volatilities, 100,000 paths for each) | |
|
254 | took 37 seconds in parallel, giving a speedup of 14.1x, which is comparable | |
|
255 | to the speedup observed in our previous example. | |
|
256 | ||
|
257 | .. image:: figs/asian_call.* | |
|
258 | ||
|
259 | .. image:: figs/asian_put.* | |
|
260 | ||
|
261 | 189 | Conclusion |
|
262 | 190 | ========== |
|
263 | 191 | |
|
264 | 192 | To conclude these examples, we summarize the key features of IPython's |
|
265 | 193 | parallel architecture that have been demonstrated: |
|
266 | 194 | |
|
267 | 195 | * Serial code can be parallelized often with only a few extra lines of code. |
|
268 | 196 | We have used the :class:`DirectView` and :class:`LoadBalancedView` classes |
|
269 | 197 | for this purpose. |
|
270 | 198 | * The resulting parallel code can be run without ever leaving the IPython's |
|
271 | 199 | interactive shell. |
|
272 | 200 | * Any data computed in parallel can be explored interactively through |
|
273 | 201 | visualization or further numerical calculations. |
|
274 | 202 | * We have run these examples on a cluster running RHEL 5 and Sun GridEngine. |
|
275 | 203 | IPython's built in support for SGE (and other batch systems) makes it easy |
|
276 | 204 | to get started with IPython's parallel capabilities. |
|
277 | 205 |
General Comments 0
You need to be logged in to leave comments.
Login now