parallel magics get their own doc file

docs/source/parallel/magics.txt

@@ -0,0 +1,389 b''
+.. _parallel_magics:
+
+=======================
+Parallel Magic Commands
+=======================
+
+We provide a few IPython magic commands
+that make it a bit more pleasant to execute Python commands on the engines interactively.
+These are mainly shortcuts to :meth:`.DirectView.execute`
+and :meth:`.AsyncResult.display_outputs` methods repsectively.
+
+These magics will automatically become available when you create a Client:
+
+.. sourcecode:: ipython
+
+    In [2]: rc = parallel.Client()
+
+The initially active View will have attributes ``targets='all', block=True``,
+which is a blocking view of all engines, evaluated at request time
+(adding/removing engines will change where this view's tasks will run).
+
+The Magics
+==========
+
+%px
+---
+
+The %px magic executes a single Python command on the engines
+specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
+
+.. sourcecode:: ipython
+
+    # import numpy here and everywhere
+    In [25]: with rc[:].sync_imports():
+       ....:    import numpy
+    importing numpy on engine(s)
+
+    In [27]: %px a = numpy.random.rand(2,2)
+    Parallel execution on engines: [0, 1, 2, 3]
+
+    In [28]: %px numpy.linalg.eigvals(a)
+    Parallel execution on engines: [0, 1, 2, 3]
+    Out [0:68]: array([ 0.77120707, -0.19448286])
+    Out [1:68]: array([ 1.10815921,  0.05110369])
+    Out [2:68]: array([ 0.74625527, -0.37475081])
+    Out [3:68]: array([ 0.72931905,  0.07159743])
+    
+    In [29]: %px print 'hi'
+    Parallel execution on engine(s): all
+    [stdout:0] hi
+    [stdout:1] hi
+    [stdout:2] hi
+    [stdout:3] hi
+
+
+Since engines are IPython as well, you can even run magics remotely:
+
+.. sourcecode:: ipython
+
+    In [28]: %px %pylab inline
+    Parallel execution on engine(s): all
+    [stdout:0] 
+    Welcome to pylab, a matplotlib-based Python environment...
+    For more information, type 'help(pylab)'.
+    [stdout:1] 
+    Welcome to pylab, a matplotlib-based Python environment...
+    For more information, type 'help(pylab)'.
+    [stdout:2] 
+    Welcome to pylab, a matplotlib-based Python environment...
+    For more information, type 'help(pylab)'.
+    [stdout:3] 
+    Welcome to pylab, a matplotlib-based Python environment...
+    For more information, type 'help(pylab)'.
+
+And once in pylab mode with the inline backend,
+you can make plots and they will be displayed in your frontend
+if it suports the inline figures (e.g. notebook or qtconsole):
+
+.. sourcecode:: ipython
+
+    In [40]: %px plot(rand(100))
+    Parallel execution on engine(s): all
+    <plot0>
+    <plot1>
+    <plot2>
+    <plot3>
+    Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
+    Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
+    Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
+    Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
+
+
+%%px Cell Magic
+---------------
+
+%%px can be used as a Cell Magic, which accepts some arguments for controlling
+the execution.
+
+
+Targets and Blocking
+********************
+
+%%px accepts ``--targets`` for controlling which engines on which to run,
+and ``--[no]block`` for specifying the blocking behavior of this cell,
+independent of the defaults for the View.
+
+.. sourcecode:: ipython
+
+    In [6]: %%px --targets ::2
+       ...: print "I am even"
+       ...: 
+    Parallel execution on engine(s): [0, 2]
+    [stdout:0] I am even
+    [stdout:2] I am even
+
+    In [7]: %%px --targets 1
+       ...: print "I am number 1"
+       ...: 
+    Parallel execution on engine(s): 1
+    I am number 1
+
+    In [8]: %%px
+       ...: print "still 'all' by default"
+       ...: 
+    Parallel execution on engine(s): all
+    [stdout:0] still 'all' by default
+    [stdout:1] still 'all' by default
+    [stdout:2] still 'all' by default
+    [stdout:3] still 'all' by default
+
+    In [9]: %%px --noblock
+       ...: import time
+       ...: time.sleep(1)
+       ...: time.time()
+       ...: 
+    Async parallel execution on engine(s): all
+    Out[9]: <AsyncResult: execute>
+
+    In [10]: %pxresult
+    Out[0:12]: 1339454561.069116
+    Out[1:10]: 1339454561.076752
+    Out[2:12]: 1339454561.072837
+    Out[3:10]: 1339454561.066665
+
+
+.. seealso::
+
+    :ref:`%pxconfig` accepts these same arguments for changing the *default*
+    values of targets/blocking for the active View.
+
+
+Output Display
+**************
+
+
+%%px also accepts a ``--group-outputs`` argument,
+which adjusts how the outputs of multiple engines are presented.
+
+.. seealso::
+
+    :meth:`.AsyncResult.display_outputs` for the grouping options.
+
+.. sourcecode:: ipython
+
+    In [50]: %%px --block --group-outputs=engine
+       ....: import numpy as np
+       ....: A = np.random.random((2,2))
+       ....: ev = numpy.linalg.eigvals(A)
+       ....: print ev
+       ....: ev.max()
+       ....:
+    Parallel execution on engine(s): all
+    [stdout:0] [ 0.60640442  0.95919621]
+    Out [0:73]: 0.9591962130899806
+    [stdout:1] [ 0.38501813  1.29430871]
+    Out [1:73]: 1.2943087091452372
+    [stdout:2] [-0.85925141  0.9387692 ]
+    Out [2:73]: 0.93876920456230284
+    [stdout:3] [ 0.37998269  1.24218246]
+    Out [3:73]: 1.2421824618493817
+
+
+%pxresult
+---------
+
+If you are using %px in non-blocking mode, you won't get output.
+You can use %pxresult to display the outputs of the latest command,
+just as is done when %px is blocking:
+
+.. sourcecode:: ipython
+
+    In [39]: dv.block = False
+    
+    In [40]: %px print 'hi'
+    Async parallel execution on engine(s): all
+    
+    In [41]: %pxresult
+    [stdout:0] hi
+    [stdout:1] hi
+    [stdout:2] hi
+    [stdout:3] hi
+
+%pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
+It accepts the same output-grouping arguments as %%px, so you can use it to view
+a result in different ways.
+
+
+%autopx
+-------
+
+The %autopx magic switches to a mode where everything you type is executed
+on the engines until you do %autopx again.
+
+.. sourcecode:: ipython
+
+    In [30]: dv.block=True
+
+    In [31]: %autopx
+    %autopx enabled
+
+    In [32]: max_evals = []
+
+    In [33]: for i in range(100):
+       ....:     a = numpy.random.rand(10,10)
+       ....:     a = a+a.transpose()
+       ....:     evals = numpy.linalg.eigvals(a)
+       ....:     max_evals.append(evals[0].real)
+       ....:
+
+    In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
+    [stdout:0] Average max eigenvalue is: 10.193101
+    [stdout:1] Average max eigenvalue is: 10.064508
+    [stdout:2] Average max eigenvalue is: 10.055724
+    [stdout:3] Average max eigenvalue is: 10.086876
+
+    In [35]: %autopx
+    Auto Parallel Disabled
+
+%pxconfig
+---------
+
+The default targets and blocking behavior for the magics are governed by the :attr:`block`
+and :attr:`targets` attribute of the active View.  If you have a handle for the view,
+you can set these attributes directly, but if you don't, you can change them with
+the %pxconfig magic:
+
+.. sourcecode:: ipython
+
+    In [3]: %pxconfig --block
+
+    In [5]: %px print 'hi'
+    Parallel execution on engine(s): all
+    [stdout:0] hi
+    [stdout:1] hi
+    [stdout:2] hi
+    [stdout:3] hi
+
+    In [6]: %pxconfig --targets ::2
+
+    In [7]: %px print 'hi'
+    Parallel execution on engine(s): [0, 2]
+    [stdout:0] hi
+    [stdout:2] hi
+
+    In [8]: %pxconfig --noblock
+
+    In [9]: %px print 'are you there?'
+    Async parallel execution on engine(s): [0, 2]
+    Out[9]: <AsyncResult: execute>
+
+    In [10]: %pxresult
+    [stdout:0] are you there?
+    [stdout:2] are you there?
+
+
+Multiple Active Views
+=====================
+
+The parallel magics are associated with a particular :class:`~.DirectView` object.
+You can change the active view by calling the :meth:`~.DirectView.activate` method
+on any view.
+
+.. sourcecode:: ipython
+
+    In [11]: even = rc[::2]
+
+    In [12]: even.activate()
+
+    In [13]: %px print 'hi'
+    Async parallel execution on engine(s): [0, 2]
+    Out[13]: <AsyncResult: execute>
+
+    In [14]: even.block = True
+
+    In [15]: %px print 'hi'
+    Parallel execution on engine(s): [0, 2]
+    [stdout:0] hi
+    [stdout:2] hi
+
+When activating a View, you can also specify a *suffix*, so that a whole different
+set of magics are associated with that view, without replacing the existing ones.
+
+.. sourcecode:: ipython
+
+    # restore the original DirecView to the base %px magics
+    In [16]: rc.activate()
+    Out[16]: <DirectView all>
+
+    In [17]: even.activate('_even')
+
+    In [18]: %px print 'hi all'
+    Parallel execution on engine(s): all
+    [stdout:0] hi all
+    [stdout:1] hi all
+    [stdout:2] hi all
+    [stdout:3] hi all
+
+    In [19]: %px_even print "We aren't odd!"
+    Parallel execution on engine(s): [0, 2]
+    [stdout:0] We aren't odd!
+    [stdout:2] We aren't odd!
+
+This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.
+
+For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
+which creates a DirectView with block=True, activates it, and returns the new View.
+
+The initial magics registered when you create a client are the result of a call to
+:meth:`rc.activate` with default args.
+
+
+Engines as Kernels
+==================
+
+Engines are really the same object as the Kernels used elsewhere in IPython,
+with the minor exception that engines connect to a controller, while regular kernels
+bind their sockets, listening for connections from a QtConsole or other frontends.
+
+Sometimes for debugging or inspection purposes, you would like a QtConsole connected
+to an engine for more direct interaction.  You can do this by first instructing
+the Engine to *also* bind its kernel, to listen for connections:
+
+.. sourcecode:: ipython
+
+    In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
+
+Then, if your engines are local, you can start a qtconsole right on the engine(s):
+
+.. sourcecode:: ipython
+
+    In [51]: %px %qtconsole
+
+Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
+
+Or you can view just the connection info, and work out the right way to connect to the engines,
+depending on where they live and where you are:
+
+.. sourcecode:: ipython
+
+    In [51]: %px %connect_info
+    Parallel execution on engine(s): all
+    [stdout:0] 
+    {
+      "stdin_port": 60387, 
+      "ip": "127.0.0.1", 
+      "hb_port": 50835, 
+      "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", 
+      "shell_port": 55328, 
+      "iopub_port": 58264
+    }
+
+    Paste the above JSON into a file, and connect with:
+        $> ipython <app> --existing <file>
+    or, if you are local, you can connect with just:
+        $> ipython <app> --existing kernel-60125.json 
+    or even just:
+        $> ipython <app> --existing 
+    if this is the most recent IPython session you have started.
+    [stdout:1] 
+    {
+      "stdin_port": 61869,
+    ...
+
+.. note::
+
+    ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
+    so you can often skip that first step.
+
+

docs/source/parallel/index.txt

@@ -10,6 +10,7 b' Using IPython for parallel computing'
    parallel_intro.txt
    parallel_process.txt
    parallel_multiengine.txt
+   magics.txt
    parallel_task.txt
    asyncresult.txt
    parallel_mpi.txt

docs/source/parallel/parallel_multiengine.txt

@@ -387,229 +387,9 b' The following examples demonstrate how to use the instance attributes:'
 The :attr:`block` and :attr:`targets` instance attributes of the
 :class:`.DirectView` also determine the behavior of the parallel magic commands.
 
-Parallel magic commands
------------------------
-
-We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
-that make it a bit more pleasant to execute Python commands on the engines interactively.
-These are simply shortcuts to :meth:`.DirectView.execute`
-and :meth:`.AsyncResult.display_outputs` methods repsectively.
-The ``%px`` magic executes a single Python command on the engines
-specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
-
-.. sourcecode:: ipython
-
-    # Create a DirectView for all targets
-    In [22]: dv = rc[:]
-    
-    # Make this DirectView active for parallel magic commands
-    In [23]: dv.activate()
-
-    In [24]: dv.block=True
-
-    # import numpy here and everywhere
-    In [25]: with dv.sync_imports():
-       ....:    import numpy
-    importing numpy on engine(s)
-
-    In [27]: %px a = numpy.random.rand(2,2)
-    Parallel execution on engines: [0, 1, 2, 3]
-
-    In [28]: %px numpy.linalg.eigvals(a)
-    Parallel execution on engines: [0, 1, 2, 3]
-    [0] Out[68]: array([ 0.77120707, -0.19448286])
-    [1] Out[68]: array([ 1.10815921,  0.05110369])
-    [2] Out[68]: array([ 0.74625527, -0.37475081])
-    [3] Out[68]: array([ 0.72931905,  0.07159743])
-    
-    In [29]: %px print 'hi'
-    Parallel execution on engine(s): [0, 1, 2, 3]
-    [stdout:0] hi
-    [stdout:1] hi
-    [stdout:2] hi
-    [stdout:3] hi
-
-
-Since engines are IPython as well, you can even run magics remotely:
-
-.. sourcecode:: ipython
-
-    In [28]: %px %pylab inline
-    Parallel execution on engine(s): [0, 1, 2, 3]
-    [stdout:0] 
-    Welcome to pylab, a matplotlib-based Python environment...
-    For more information, type 'help(pylab)'.
-    [stdout:1] 
-    Welcome to pylab, a matplotlib-based Python environment...
-    For more information, type 'help(pylab)'.
-    [stdout:2] 
-    Welcome to pylab, a matplotlib-based Python environment...
-    For more information, type 'help(pylab)'.
-    [stdout:3] 
-    Welcome to pylab, a matplotlib-based Python environment...
-    For more information, type 'help(pylab)'.
-
-And once in pylab mode with the inline backend,
-you can make plots and they will be displayed in your frontend
-if it suports the inline figures (e.g. notebook or qtconsole):
-
-.. sourcecode:: ipython
-
-    In [40]: %px plot(rand(100))
-    Parallel execution on engine(s): [0, 1, 2, 3]
-    <plot0>
-    <plot1>
-    <plot2>
-    <plot3>
-    [0] Out[79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
-    [1] Out[79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
-    [2] Out[79]: [<matplotlib.lines.Line2D at 0x110652750>]
-    [3] Out[79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
-
-
-``%%px`` Cell Magic
-*******************
-
-`%%px` can also be used as a Cell Magic, which accepts ``--[no]block`` flags,
-and a ``--group-outputs`` argument, which adjust how the outputs of multiple
-engines are presented.
-
 .. seealso::
 
-    :meth:`.AsyncResult.display_outputs` for the grouping options.
-
-.. sourcecode:: ipython
-
-    In [50]: %%px --block --group-outputs=engine
-       ....: import numpy as np
-       ....: A = np.random.random((2,2))
-       ....: ev = numpy.linalg.eigvals(A)
-       ....: print ev
-       ....: ev.max()
-       ....:
-    Parallel execution on engine(s): [0, 1, 2, 3]
-    [stdout:0] [ 0.60640442  0.95919621]
-    [0] Out[73]: 0.9591962130899806
-    [stdout:1] [ 0.38501813  1.29430871]
-    [1] Out[73]: 1.2943087091452372
-    [stdout:2] [-0.85925141  0.9387692 ]
-    [2] Out[73]: 0.93876920456230284
-    [stdout:3] [ 0.37998269  1.24218246]
-    [3] Out[73]: 1.2421824618493817
-
-``%result`` Magic
-*****************
-
-If you are using ``%px`` in non-blocking mode, you won't get output.
-You can use ``%result`` to display the outputs of the latest command,
-just as is done when ``%px`` is blocking:
-
-.. sourcecode:: ipython
-
-    In [39]: dv.block = False
-    
-    In [40]: %px print 'hi'
-    Async parallel execution on engine(s): [0, 1, 2, 3]
-    
-    In [41]: %result
-    [stdout:0] hi
-    [stdout:1] hi
-    [stdout:2] hi
-    [stdout:3] hi
-
-``%result`` simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
-You can pass integers as indices if you want a result other than the latest,
-e.g. ``%result -2``, or ``%result 0`` for the first.
-
-
-``%autopx``
-***********
-
-The ``%autopx`` magic switches to a mode where everything you type is executed
-on the engines until you do ``%autopx`` again.
-
-.. sourcecode:: ipython
-
-    In [30]: dv.block=True
-
-    In [31]: %autopx
-    %autopx enabled
-
-    In [32]: max_evals = []
-
-    In [33]: for i in range(100):
-       ....:     a = numpy.random.rand(10,10)
-       ....:     a = a+a.transpose()
-       ....:     evals = numpy.linalg.eigvals(a)
-       ....:     max_evals.append(evals[0].real)
-       ....:
-
-    In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
-    [stdout:0] Average max eigenvalue is: 10.193101
-    [stdout:1] Average max eigenvalue is: 10.064508
-    [stdout:2] Average max eigenvalue is: 10.055724
-    [stdout:3] Average max eigenvalue is: 10.086876
-
-    In [35]: %autopx
-    Auto Parallel Disabled
-
-
-Engines as Kernels
-******************
-
-Engines are really the same object as the Kernels used elsewhere in IPython,
-with the minor exception that engines connect to a controller, while regular kernels
-bind their sockets, listening for connections from a QtConsole or other frontends.
-
-Sometimes for debugging or inspection purposes, you would like a QtConsole connected
-to an engine for more direct interaction.  You can do this by first instructing
-the Engine to *also* bind its kernel, to listen for connections:
-
-.. sourcecode:: ipython
-
-    In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
-
-Then, if your engines are local, you can start a qtconsole right on the engine(s):
-
-.. sourcecode:: ipython
-
-    In [51]: %px %qtconsole
-
-Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
-
-Or you can view just the connection info, and work out the right way to connect to the engines,
-depending on where they live and where you are:
-
-.. sourcecode:: ipython
-
-    In [51]: %px %connect_info
-    Parallel execution on engine(s): [0, 1, 2, 3]
-    [stdout:0] 
-    {
-      "stdin_port": 60387, 
-      "ip": "127.0.0.1", 
-      "hb_port": 50835, 
-      "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", 
-      "shell_port": 55328, 
-      "iopub_port": 58264
-    }
-
-    Paste the above JSON into a file, and connect with:
-        $> ipython <app> --existing <file>
-    or, if you are local, you can connect with just:
-        $> ipython <app> --existing kernel-60125.json 
-    or even just:
-        $> ipython <app> --existing 
-    if this is the most recent IPython session you have started.
-    [stdout:1] 
-    {
-      "stdin_port": 61869,
-    ...
-
-.. note::
-
-    ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
-    so you can often skip that first step.
+    See the documentation of the :ref:`Parallel Magics <parallel_magics>`.
 
 
 Moving Python objects around