diff --git a/docs/source/parallel/index.txt b/docs/source/parallel/index.txt index 9bfbc76..be8b957 100644 --- a/docs/source/parallel/index.txt +++ b/docs/source/parallel/index.txt @@ -10,6 +10,7 @@ Using IPython for parallel computing parallel_intro.txt parallel_process.txt parallel_multiengine.txt + magics.txt parallel_task.txt asyncresult.txt parallel_mpi.txt diff --git a/docs/source/parallel/magics.txt b/docs/source/parallel/magics.txt new file mode 100644 index 0000000..d9db175 --- /dev/null +++ b/docs/source/parallel/magics.txt @@ -0,0 +1,389 @@ +.. _parallel_magics: + +======================= +Parallel Magic Commands +======================= + +We provide a few IPython magic commands +that make it a bit more pleasant to execute Python commands on the engines interactively. +These are mainly shortcuts to :meth:`.DirectView.execute` +and :meth:`.AsyncResult.display_outputs` methods repsectively. + +These magics will automatically become available when you create a Client: + +.. sourcecode:: ipython + + In [2]: rc = parallel.Client() + +The initially active View will have attributes ``targets='all', block=True``, +which is a blocking view of all engines, evaluated at request time +(adding/removing engines will change where this view's tasks will run). + +The Magics +========== + +%px +--- + +The %px magic executes a single Python command on the engines +specified by the :attr:`targets` attribute of the :class:`DirectView` instance: + +.. sourcecode:: ipython + + # import numpy here and everywhere + In [25]: with rc[:].sync_imports(): + ....: import numpy + importing numpy on engine(s) + + In [27]: %px a = numpy.random.rand(2,2) + Parallel execution on engines: [0, 1, 2, 3] + + In [28]: %px numpy.linalg.eigvals(a) + Parallel execution on engines: [0, 1, 2, 3] + Out [0:68]: array([ 0.77120707, -0.19448286]) + Out [1:68]: array([ 1.10815921, 0.05110369]) + Out [2:68]: array([ 0.74625527, -0.37475081]) + Out [3:68]: array([ 0.72931905, 0.07159743]) + + In [29]: %px print 'hi' + Parallel execution on engine(s): all + [stdout:0] hi + [stdout:1] hi + [stdout:2] hi + [stdout:3] hi + + +Since engines are IPython as well, you can even run magics remotely: + +.. sourcecode:: ipython + + In [28]: %px %pylab inline + Parallel execution on engine(s): all + [stdout:0] + Welcome to pylab, a matplotlib-based Python environment... + For more information, type 'help(pylab)'. + [stdout:1] + Welcome to pylab, a matplotlib-based Python environment... + For more information, type 'help(pylab)'. + [stdout:2] + Welcome to pylab, a matplotlib-based Python environment... + For more information, type 'help(pylab)'. + [stdout:3] + Welcome to pylab, a matplotlib-based Python environment... + For more information, type 'help(pylab)'. + +And once in pylab mode with the inline backend, +you can make plots and they will be displayed in your frontend +if it suports the inline figures (e.g. notebook or qtconsole): + +.. sourcecode:: ipython + + In [40]: %px plot(rand(100)) + Parallel execution on engine(s): all + + + + + Out[0:79]: [] + Out[1:79]: [] + Out[2:79]: [] + Out[3:79]: [] + + +%%px Cell Magic +--------------- + +%%px can be used as a Cell Magic, which accepts some arguments for controlling +the execution. + + +Targets and Blocking +******************** + +%%px accepts ``--targets`` for controlling which engines on which to run, +and ``--[no]block`` for specifying the blocking behavior of this cell, +independent of the defaults for the View. + +.. sourcecode:: ipython + + In [6]: %%px --targets ::2 + ...: print "I am even" + ...: + Parallel execution on engine(s): [0, 2] + [stdout:0] I am even + [stdout:2] I am even + + In [7]: %%px --targets 1 + ...: print "I am number 1" + ...: + Parallel execution on engine(s): 1 + I am number 1 + + In [8]: %%px + ...: print "still 'all' by default" + ...: + Parallel execution on engine(s): all + [stdout:0] still 'all' by default + [stdout:1] still 'all' by default + [stdout:2] still 'all' by default + [stdout:3] still 'all' by default + + In [9]: %%px --noblock + ...: import time + ...: time.sleep(1) + ...: time.time() + ...: + Async parallel execution on engine(s): all + Out[9]: + + In [10]: %pxresult + Out[0:12]: 1339454561.069116 + Out[1:10]: 1339454561.076752 + Out[2:12]: 1339454561.072837 + Out[3:10]: 1339454561.066665 + + +.. seealso:: + + :ref:`%pxconfig` accepts these same arguments for changing the *default* + values of targets/blocking for the active View. + + +Output Display +************** + + +%%px also accepts a ``--group-outputs`` argument, +which adjusts how the outputs of multiple engines are presented. + +.. seealso:: + + :meth:`.AsyncResult.display_outputs` for the grouping options. + +.. sourcecode:: ipython + + In [50]: %%px --block --group-outputs=engine + ....: import numpy as np + ....: A = np.random.random((2,2)) + ....: ev = numpy.linalg.eigvals(A) + ....: print ev + ....: ev.max() + ....: + Parallel execution on engine(s): all + [stdout:0] [ 0.60640442 0.95919621] + Out [0:73]: 0.9591962130899806 + [stdout:1] [ 0.38501813 1.29430871] + Out [1:73]: 1.2943087091452372 + [stdout:2] [-0.85925141 0.9387692 ] + Out [2:73]: 0.93876920456230284 + [stdout:3] [ 0.37998269 1.24218246] + Out [3:73]: 1.2421824618493817 + + +%pxresult +--------- + +If you are using %px in non-blocking mode, you won't get output. +You can use %pxresult to display the outputs of the latest command, +just as is done when %px is blocking: + +.. sourcecode:: ipython + + In [39]: dv.block = False + + In [40]: %px print 'hi' + Async parallel execution on engine(s): all + + In [41]: %pxresult + [stdout:0] hi + [stdout:1] hi + [stdout:2] hi + [stdout:3] hi + +%pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request. +It accepts the same output-grouping arguments as %%px, so you can use it to view +a result in different ways. + + +%autopx +------- + +The %autopx magic switches to a mode where everything you type is executed +on the engines until you do %autopx again. + +.. sourcecode:: ipython + + In [30]: dv.block=True + + In [31]: %autopx + %autopx enabled + + In [32]: max_evals = [] + + In [33]: for i in range(100): + ....: a = numpy.random.rand(10,10) + ....: a = a+a.transpose() + ....: evals = numpy.linalg.eigvals(a) + ....: max_evals.append(evals[0].real) + ....: + + In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals)) + [stdout:0] Average max eigenvalue is: 10.193101 + [stdout:1] Average max eigenvalue is: 10.064508 + [stdout:2] Average max eigenvalue is: 10.055724 + [stdout:3] Average max eigenvalue is: 10.086876 + + In [35]: %autopx + Auto Parallel Disabled + +%pxconfig +--------- + +The default targets and blocking behavior for the magics are governed by the :attr:`block` +and :attr:`targets` attribute of the active View. If you have a handle for the view, +you can set these attributes directly, but if you don't, you can change them with +the %pxconfig magic: + +.. sourcecode:: ipython + + In [3]: %pxconfig --block + + In [5]: %px print 'hi' + Parallel execution on engine(s): all + [stdout:0] hi + [stdout:1] hi + [stdout:2] hi + [stdout:3] hi + + In [6]: %pxconfig --targets ::2 + + In [7]: %px print 'hi' + Parallel execution on engine(s): [0, 2] + [stdout:0] hi + [stdout:2] hi + + In [8]: %pxconfig --noblock + + In [9]: %px print 'are you there?' + Async parallel execution on engine(s): [0, 2] + Out[9]: + + In [10]: %pxresult + [stdout:0] are you there? + [stdout:2] are you there? + + +Multiple Active Views +===================== + +The parallel magics are associated with a particular :class:`~.DirectView` object. +You can change the active view by calling the :meth:`~.DirectView.activate` method +on any view. + +.. sourcecode:: ipython + + In [11]: even = rc[::2] + + In [12]: even.activate() + + In [13]: %px print 'hi' + Async parallel execution on engine(s): [0, 2] + Out[13]: + + In [14]: even.block = True + + In [15]: %px print 'hi' + Parallel execution on engine(s): [0, 2] + [stdout:0] hi + [stdout:2] hi + +When activating a View, you can also specify a *suffix*, so that a whole different +set of magics are associated with that view, without replacing the existing ones. + +.. sourcecode:: ipython + + # restore the original DirecView to the base %px magics + In [16]: rc.activate() + Out[16]: + + In [17]: even.activate('_even') + + In [18]: %px print 'hi all' + Parallel execution on engine(s): all + [stdout:0] hi all + [stdout:1] hi all + [stdout:2] hi all + [stdout:3] hi all + + In [19]: %px_even print "We aren't odd!" + Parallel execution on engine(s): [0, 2] + [stdout:0] We aren't odd! + [stdout:2] We aren't odd! + +This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc. + +For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well, +which creates a DirectView with block=True, activates it, and returns the new View. + +The initial magics registered when you create a client are the result of a call to +:meth:`rc.activate` with default args. + + +Engines as Kernels +================== + +Engines are really the same object as the Kernels used elsewhere in IPython, +with the minor exception that engines connect to a controller, while regular kernels +bind their sockets, listening for connections from a QtConsole or other frontends. + +Sometimes for debugging or inspection purposes, you would like a QtConsole connected +to an engine for more direct interaction. You can do this by first instructing +the Engine to *also* bind its kernel, to listen for connections: + +.. sourcecode:: ipython + + In [50]: %px from IPython.parallel import bind_kernel; bind_kernel() + +Then, if your engines are local, you can start a qtconsole right on the engine(s): + +.. sourcecode:: ipython + + In [51]: %px %qtconsole + +Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles! + +Or you can view just the connection info, and work out the right way to connect to the engines, +depending on where they live and where you are: + +.. sourcecode:: ipython + + In [51]: %px %connect_info + Parallel execution on engine(s): all + [stdout:0] + { + "stdin_port": 60387, + "ip": "127.0.0.1", + "hb_port": 50835, + "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", + "shell_port": 55328, + "iopub_port": 58264 + } + + Paste the above JSON into a file, and connect with: + $> ipython --existing + or, if you are local, you can connect with just: + $> ipython --existing kernel-60125.json + or even just: + $> ipython --existing + if this is the most recent IPython session you have started. + [stdout:1] + { + "stdin_port": 61869, + ... + +.. note:: + + ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already, + so you can often skip that first step. + + diff --git a/docs/source/parallel/parallel_multiengine.txt b/docs/source/parallel/parallel_multiengine.txt index 7f10458..ef25fe1 100644 --- a/docs/source/parallel/parallel_multiengine.txt +++ b/docs/source/parallel/parallel_multiengine.txt @@ -387,229 +387,9 @@ The following examples demonstrate how to use the instance attributes: The :attr:`block` and :attr:`targets` instance attributes of the :class:`.DirectView` also determine the behavior of the parallel magic commands. -Parallel magic commands ------------------------ - -We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``) -that make it a bit more pleasant to execute Python commands on the engines interactively. -These are simply shortcuts to :meth:`.DirectView.execute` -and :meth:`.AsyncResult.display_outputs` methods repsectively. -The ``%px`` magic executes a single Python command on the engines -specified by the :attr:`targets` attribute of the :class:`DirectView` instance: - -.. sourcecode:: ipython - - # Create a DirectView for all targets - In [22]: dv = rc[:] - - # Make this DirectView active for parallel magic commands - In [23]: dv.activate() - - In [24]: dv.block=True - - # import numpy here and everywhere - In [25]: with dv.sync_imports(): - ....: import numpy - importing numpy on engine(s) - - In [27]: %px a = numpy.random.rand(2,2) - Parallel execution on engines: [0, 1, 2, 3] - - In [28]: %px numpy.linalg.eigvals(a) - Parallel execution on engines: [0, 1, 2, 3] - [0] Out[68]: array([ 0.77120707, -0.19448286]) - [1] Out[68]: array([ 1.10815921, 0.05110369]) - [2] Out[68]: array([ 0.74625527, -0.37475081]) - [3] Out[68]: array([ 0.72931905, 0.07159743]) - - In [29]: %px print 'hi' - Parallel execution on engine(s): [0, 1, 2, 3] - [stdout:0] hi - [stdout:1] hi - [stdout:2] hi - [stdout:3] hi - - -Since engines are IPython as well, you can even run magics remotely: - -.. sourcecode:: ipython - - In [28]: %px %pylab inline - Parallel execution on engine(s): [0, 1, 2, 3] - [stdout:0] - Welcome to pylab, a matplotlib-based Python environment... - For more information, type 'help(pylab)'. - [stdout:1] - Welcome to pylab, a matplotlib-based Python environment... - For more information, type 'help(pylab)'. - [stdout:2] - Welcome to pylab, a matplotlib-based Python environment... - For more information, type 'help(pylab)'. - [stdout:3] - Welcome to pylab, a matplotlib-based Python environment... - For more information, type 'help(pylab)'. - -And once in pylab mode with the inline backend, -you can make plots and they will be displayed in your frontend -if it suports the inline figures (e.g. notebook or qtconsole): - -.. sourcecode:: ipython - - In [40]: %px plot(rand(100)) - Parallel execution on engine(s): [0, 1, 2, 3] - - - - - [0] Out[79]: [] - [1] Out[79]: [] - [2] Out[79]: [] - [3] Out[79]: [] - - -``%%px`` Cell Magic -******************* - -`%%px` can also be used as a Cell Magic, which accepts ``--[no]block`` flags, -and a ``--group-outputs`` argument, which adjust how the outputs of multiple -engines are presented. - .. seealso:: - :meth:`.AsyncResult.display_outputs` for the grouping options. - -.. sourcecode:: ipython - - In [50]: %%px --block --group-outputs=engine - ....: import numpy as np - ....: A = np.random.random((2,2)) - ....: ev = numpy.linalg.eigvals(A) - ....: print ev - ....: ev.max() - ....: - Parallel execution on engine(s): [0, 1, 2, 3] - [stdout:0] [ 0.60640442 0.95919621] - [0] Out[73]: 0.9591962130899806 - [stdout:1] [ 0.38501813 1.29430871] - [1] Out[73]: 1.2943087091452372 - [stdout:2] [-0.85925141 0.9387692 ] - [2] Out[73]: 0.93876920456230284 - [stdout:3] [ 0.37998269 1.24218246] - [3] Out[73]: 1.2421824618493817 - -``%result`` Magic -***************** - -If you are using ``%px`` in non-blocking mode, you won't get output. -You can use ``%result`` to display the outputs of the latest command, -just as is done when ``%px`` is blocking: - -.. sourcecode:: ipython - - In [39]: dv.block = False - - In [40]: %px print 'hi' - Async parallel execution on engine(s): [0, 1, 2, 3] - - In [41]: %result - [stdout:0] hi - [stdout:1] hi - [stdout:2] hi - [stdout:3] hi - -``%result`` simply calls :meth:`.AsyncResult.display_outputs` on the most recent request. -You can pass integers as indices if you want a result other than the latest, -e.g. ``%result -2``, or ``%result 0`` for the first. - - -``%autopx`` -*********** - -The ``%autopx`` magic switches to a mode where everything you type is executed -on the engines until you do ``%autopx`` again. - -.. sourcecode:: ipython - - In [30]: dv.block=True - - In [31]: %autopx - %autopx enabled - - In [32]: max_evals = [] - - In [33]: for i in range(100): - ....: a = numpy.random.rand(10,10) - ....: a = a+a.transpose() - ....: evals = numpy.linalg.eigvals(a) - ....: max_evals.append(evals[0].real) - ....: - - In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals)) - [stdout:0] Average max eigenvalue is: 10.193101 - [stdout:1] Average max eigenvalue is: 10.064508 - [stdout:2] Average max eigenvalue is: 10.055724 - [stdout:3] Average max eigenvalue is: 10.086876 - - In [35]: %autopx - Auto Parallel Disabled - - -Engines as Kernels -****************** - -Engines are really the same object as the Kernels used elsewhere in IPython, -with the minor exception that engines connect to a controller, while regular kernels -bind their sockets, listening for connections from a QtConsole or other frontends. - -Sometimes for debugging or inspection purposes, you would like a QtConsole connected -to an engine for more direct interaction. You can do this by first instructing -the Engine to *also* bind its kernel, to listen for connections: - -.. sourcecode:: ipython - - In [50]: %px from IPython.parallel import bind_kernel; bind_kernel() - -Then, if your engines are local, you can start a qtconsole right on the engine(s): - -.. sourcecode:: ipython - - In [51]: %px %qtconsole - -Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles! - -Or you can view just the connection info, and work out the right way to connect to the engines, -depending on where they live and where you are: - -.. sourcecode:: ipython - - In [51]: %px %connect_info - Parallel execution on engine(s): [0, 1, 2, 3] - [stdout:0] - { - "stdin_port": 60387, - "ip": "127.0.0.1", - "hb_port": 50835, - "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", - "shell_port": 55328, - "iopub_port": 58264 - } - - Paste the above JSON into a file, and connect with: - $> ipython --existing - or, if you are local, you can connect with just: - $> ipython --existing kernel-60125.json - or even just: - $> ipython --existing - if this is the most recent IPython session you have started. - [stdout:1] - { - "stdin_port": 61869, - ... - -.. note:: - - ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already, - so you can often skip that first step. + See the documentation of the :ref:`Parallel Magics `. Moving Python objects around