Fix up some problems with parallel docs

docs/source/parallel/magics.rst

@@ -1,385 +1,387
 .. _parallel_magics:
 
 =======================
 Parallel Magic Commands
 =======================
 
 We provide a few IPython magic commands
 that make it a bit more pleasant to execute Python commands on the engines interactively.
 These are mainly shortcuts to :meth:`.DirectView.execute`
 and :meth:`.AsyncResult.display_outputs` methods repsectively.
 
 These magics will automatically become available when you create a Client:
 
 .. sourcecode:: ipython
 
     In [2]: rc = parallel.Client()
 
 The initially active View will have attributes ``targets='all', block=True``,
 which is a blocking view of all engines, evaluated at request time
 (adding/removing engines will change where this view's tasks will run).
 
 The Magics
 ==========
 
 %px
 ---
 
 The %px magic executes a single Python command on the engines
 specified by the :attr:`targets` attribute of the :class:`DirectView` instance:
 
 .. sourcecode:: ipython
 
     # import numpy here and everywhere
     In [25]: with rc[:].sync_imports():
        ....:    import numpy
     importing numpy on engine(s)
 
     In [27]: %px a = numpy.random.rand(2,2)
     Parallel execution on engines: [0, 1, 2, 3]
 
     In [28]: %px numpy.linalg.eigvals(a)
     Parallel execution on engines: [0, 1, 2, 3]
     Out [0:68]: array([ 0.77120707, -0.19448286])
     Out [1:68]: array([ 1.10815921,  0.05110369])
     Out [2:68]: array([ 0.74625527, -0.37475081])
     Out [3:68]: array([ 0.72931905,  0.07159743])
     
     In [29]: %px print 'hi'
     Parallel execution on engine(s): all
     [stdout:0] hi
     [stdout:1] hi
     [stdout:2] hi
     [stdout:3] hi
 
 
 Since engines are IPython as well, you can even run magics remotely:
 
 .. sourcecode:: ipython
 
     In [28]: %px %pylab inline
     Parallel execution on engine(s): all
     [stdout:0] 
     Populating the interactive namespace from numpy and matplotlib
     [stdout:1] 
     Populating the interactive namespace from numpy and matplotlib
     [stdout:2] 
     Populating the interactive namespace from numpy and matplotlib
     [stdout:3] 
     Populating the interactive namespace from numpy and matplotlib
 
 And once in pylab mode with the inline backend,
 you can make plots and they will be displayed in your frontend
 if it suports the inline figures (e.g. notebook or qtconsole):
 
 .. sourcecode:: ipython
 
     In [40]: %px plot(rand(100))
     Parallel execution on engine(s): all
     <plot0>
     <plot1>
     <plot2>
     <plot3>
     Out[0:79]: [<matplotlib.lines.Line2D at 0x10a6286d0>]
     Out[1:79]: [<matplotlib.lines.Line2D at 0x10b9476d0>]
     Out[2:79]: [<matplotlib.lines.Line2D at 0x110652750>]
     Out[3:79]: [<matplotlib.lines.Line2D at 0x10c6566d0>]
 
 
 %%px Cell Magic
 ---------------
 
 %%px can be used as a Cell Magic, which accepts some arguments for controlling
 the execution.
 
 
 Targets and Blocking
 ********************
 
 %%px accepts ``--targets`` for controlling which engines on which to run,
 and ``--[no]block`` for specifying the blocking behavior of this cell,
 independent of the defaults for the View.
 
 .. sourcecode:: ipython
 
     In [6]: %%px --targets ::2
        ...: print "I am even"
        ...: 
     Parallel execution on engine(s): [0, 2]
     [stdout:0] I am even
     [stdout:2] I am even
 
     In [7]: %%px --targets 1
        ...: print "I am number 1"
        ...: 
     Parallel execution on engine(s): 1
     I am number 1
 
     In [8]: %%px
        ...: print "still 'all' by default"
        ...: 
     Parallel execution on engine(s): all
     [stdout:0] still 'all' by default
     [stdout:1] still 'all' by default
     [stdout:2] still 'all' by default
     [stdout:3] still 'all' by default
 
     In [9]: %%px --noblock
        ...: import time
        ...: time.sleep(1)
        ...: time.time()
        ...: 
     Async parallel execution on engine(s): all
     Out[9]: <AsyncResult: execute>
 
     In [10]: %pxresult
     Out[0:12]: 1339454561.069116
     Out[1:10]: 1339454561.076752
     Out[2:12]: 1339454561.072837
     Out[3:10]: 1339454561.066665
 
 
 .. seealso::
 
-    :ref:`%pxconfig` accepts these same arguments for changing the *default*
+    :ref:`pxconfig` accepts these same arguments for changing the *default*
     values of targets/blocking for the active View.
 
 
 Output Display
 **************
 
 
 %%px also accepts a ``--group-outputs`` argument,
 which adjusts how the outputs of multiple engines are presented.
 
 .. seealso::
 
     :meth:`.AsyncResult.display_outputs` for the grouping options.
 
 .. sourcecode:: ipython
 
     In [50]: %%px --block --group-outputs=engine
        ....: import numpy as np
        ....: A = np.random.random((2,2))
        ....: ev = numpy.linalg.eigvals(A)
        ....: print ev
        ....: ev.max()
        ....:
     Parallel execution on engine(s): all
     [stdout:0] [ 0.60640442  0.95919621]
     Out [0:73]: 0.9591962130899806
     [stdout:1] [ 0.38501813  1.29430871]
     Out [1:73]: 1.2943087091452372
     [stdout:2] [-0.85925141  0.9387692 ]
     Out [2:73]: 0.93876920456230284
     [stdout:3] [ 0.37998269  1.24218246]
     Out [3:73]: 1.2421824618493817
 
 
 %pxresult
 ---------
 
 If you are using %px in non-blocking mode, you won't get output.
 You can use %pxresult to display the outputs of the latest command,
 just as is done when %px is blocking:
 
 .. sourcecode:: ipython
 
     In [39]: dv.block = False
     
     In [40]: %px print 'hi'
     Async parallel execution on engine(s): all
     
     In [41]: %pxresult
     [stdout:0] hi
     [stdout:1] hi
     [stdout:2] hi
     [stdout:3] hi
 
 %pxresult simply calls :meth:`.AsyncResult.display_outputs` on the most recent request.
 It accepts the same output-grouping arguments as %%px, so you can use it to view
 a result in different ways.
 
 
 %autopx
 -------
 
 The %autopx magic switches to a mode where everything you type is executed
 on the engines until you do %autopx again.
 
 .. sourcecode:: ipython
 
     In [30]: dv.block=True
 
     In [31]: %autopx
     %autopx enabled
 
     In [32]: max_evals = []
 
     In [33]: for i in range(100):
        ....:     a = numpy.random.rand(10,10)
        ....:     a = a+a.transpose()
        ....:     evals = numpy.linalg.eigvals(a)
        ....:     max_evals.append(evals[0].real)
        ....:
 
     In [34]: print "Average max eigenvalue is: %f" % (sum(max_evals)/len(max_evals))
     [stdout:0] Average max eigenvalue is: 10.193101
     [stdout:1] Average max eigenvalue is: 10.064508
     [stdout:2] Average max eigenvalue is: 10.055724
     [stdout:3] Average max eigenvalue is: 10.086876
 
     In [35]: %autopx
     Auto Parallel Disabled
 
+.. _pxconfig:
+
 %pxconfig
 ---------
 
 The default targets and blocking behavior for the magics are governed by the :attr:`block`
 and :attr:`targets` attribute of the active View.  If you have a handle for the view,
 you can set these attributes directly, but if you don't, you can change them with
 the %pxconfig magic:
 
 .. sourcecode:: ipython
 
     In [3]: %pxconfig --block
 
     In [5]: %px print 'hi'
     Parallel execution on engine(s): all
     [stdout:0] hi
     [stdout:1] hi
     [stdout:2] hi
     [stdout:3] hi
 
     In [6]: %pxconfig --targets ::2
 
     In [7]: %px print 'hi'
     Parallel execution on engine(s): [0, 2]
     [stdout:0] hi
     [stdout:2] hi
 
     In [8]: %pxconfig --noblock
 
     In [9]: %px print 'are you there?'
     Async parallel execution on engine(s): [0, 2]
     Out[9]: <AsyncResult: execute>
 
     In [10]: %pxresult
     [stdout:0] are you there?
     [stdout:2] are you there?
 
 
 Multiple Active Views
 =====================
 
 The parallel magics are associated with a particular :class:`~.DirectView` object.
 You can change the active view by calling the :meth:`~.DirectView.activate` method
 on any view.
 
 .. sourcecode:: ipython
 
     In [11]: even = rc[::2]
 
     In [12]: even.activate()
 
     In [13]: %px print 'hi'
     Async parallel execution on engine(s): [0, 2]
     Out[13]: <AsyncResult: execute>
 
     In [14]: even.block = True
 
     In [15]: %px print 'hi'
     Parallel execution on engine(s): [0, 2]
     [stdout:0] hi
     [stdout:2] hi
 
 When activating a View, you can also specify a *suffix*, so that a whole different
 set of magics are associated with that view, without replacing the existing ones.
 
 .. sourcecode:: ipython
 
     # restore the original DirecView to the base %px magics
     In [16]: rc.activate()
     Out[16]: <DirectView all>
 
     In [17]: even.activate('_even')
 
     In [18]: %px print 'hi all'
     Parallel execution on engine(s): all
     [stdout:0] hi all
     [stdout:1] hi all
     [stdout:2] hi all
     [stdout:3] hi all
 
     In [19]: %px_even print "We aren't odd!"
     Parallel execution on engine(s): [0, 2]
     [stdout:0] We aren't odd!
     [stdout:2] We aren't odd!
 
 This suffix is applied to the end of all magics, e.g. %autopx_even, %pxresult_even, etc.
 
 For convenience, the :class:`~.Client` has a :meth:`~.Client.activate` method as well,
 which creates a DirectView with block=True, activates it, and returns the new View.
 
 The initial magics registered when you create a client are the result of a call to
 :meth:`rc.activate` with default args.
 
 
 Engines as Kernels
 ==================
 
 Engines are really the same object as the Kernels used elsewhere in IPython,
 with the minor exception that engines connect to a controller, while regular kernels
 bind their sockets, listening for connections from a QtConsole or other frontends.
 
 Sometimes for debugging or inspection purposes, you would like a QtConsole connected
 to an engine for more direct interaction.  You can do this by first instructing
 the Engine to *also* bind its kernel, to listen for connections:
 
 .. sourcecode:: ipython
 
     In [50]: %px from IPython.parallel import bind_kernel; bind_kernel()
 
 Then, if your engines are local, you can start a qtconsole right on the engine(s):
 
 .. sourcecode:: ipython
 
     In [51]: %px %qtconsole
 
 Careful with this one, because if your view is of 16 engines it will start 16 QtConsoles!
 
 Or you can view just the connection info, and work out the right way to connect to the engines,
 depending on where they live and where you are:
 
 .. sourcecode:: ipython
 
     In [51]: %px %connect_info
     Parallel execution on engine(s): all
     [stdout:0] 
     {
       "stdin_port": 60387, 
       "ip": "127.0.0.1", 
       "hb_port": 50835, 
       "key": "eee2dd69-7dd3-4340-bf3e-7e2e22a62542", 
       "shell_port": 55328, 
       "iopub_port": 58264
     }
 
     Paste the above JSON into a file, and connect with:
         $> ipython <app> --existing <file>
     or, if you are local, you can connect with just:
         $> ipython <app> --existing kernel-60125.json 
     or even just:
         $> ipython <app> --existing 
     if this is the most recent IPython session you have started.
     [stdout:1] 
     {
       "stdin_port": 61869,
     ...
 
 .. note::
 
     ``%qtconsole`` will call :func:`bind_kernel` on an engine if it hasn't been done already,
     so you can often skip that first step.
 
 

docs/source/parallel/parallel_details.rst

@@ -1,652 +1,656
 .. _parallel_details:
 
 ==========================================
 Details of Parallel Computing with IPython
 ==========================================
 
 .. note::
 
     There are still many sections to fill out in this doc
 
 
 Caveats
 =======
 
 First, some caveats about the detailed workings of parallel computing with 0MQ and IPython.
 
 Non-copying sends and numpy arrays
 ----------------------------------
 
 When numpy arrays are passed as arguments to apply or via data-movement methods, they are not
 copied. This means that you must be careful if you are sending an array that you intend to work
 on. PyZMQ does allow you to track when a message has been sent so you can know when it is safe
 to edit the buffer, but IPython only allows for this.
 
 It is also important to note that the non-copying receive of a message is *read-only*. That
 means that if you intend to work in-place on an array that you have sent or received, you must
 copy it. This is true for both numpy arrays sent to engines and numpy arrays retrieved as
 results.
 
 The following will fail:
 
 .. sourcecode:: ipython
 
     In [3]: A = numpy.zeros(2)
     
     In [4]: def setter(a):
        ...:   a[0]=1
        ...:   return a
 
     In [5]: rc[0].apply_sync(setter, A)
     ---------------------------------------------------------------------------
     RuntimeError                              Traceback (most recent call last)<string> in <module>()
     <ipython-input-12-c3e7afeb3075> in setter(a)
     RuntimeError: array is not writeable
 
 If you do need to edit the array in-place, just remember to copy the array if it's read-only.
 The :attr:`ndarray.flags.writeable` flag will tell you if you can write to an array.
 
 .. sourcecode:: ipython
 
     In [3]: A = numpy.zeros(2)
     
     In [4]: def setter(a):
        ...:     """only copy read-only arrays"""
        ...:     if not a.flags.writeable:
        ...:         a=a.copy()
        ...:     a[0]=1
        ...:     return a
 
     In [5]: rc[0].apply_sync(setter, A)
     Out[5]: array([ 1.,  0.])
     
     # note that results will also be read-only:
     In [6]: _.flags.writeable
     Out[6]: False
 
 If you want to safely edit an array in-place after *sending* it, you must use the `track=True`
 flag. IPython always performs non-copying sends of arrays, which return immediately. You must
 instruct IPython track those messages *at send time* in order to know for sure that the send has
 completed. AsyncResults have a :attr:`sent` property, and :meth:`wait_on_send` method for
 checking and waiting for 0MQ to finish with a buffer.
 
 .. sourcecode:: ipython
 
     In [5]: A = numpy.random.random((1024,1024))
     
     In [6]: view.track=True
     
     In [7]: ar = view.apply_async(lambda x: 2*x, A)
     
     In [8]: ar.sent
     Out[8]: False
     
     In [9]: ar.wait_on_send() # blocks until sent is True
 
 
 What is sendable?
 -----------------
 
 If IPython doesn't know what to do with an object, it will pickle it. There is a short list of
 objects that are not pickled: ``buffers``, ``str/bytes`` objects, and ``numpy``
 arrays. These are handled specially by IPython in order to prevent the copying of data. Sending
 bytes or numpy arrays will result in exactly zero in-memory copies of your data (unless the data
 is very small).
 
 If you have an object that provides a Python buffer interface, then you can always send that
 buffer without copying - and reconstruct the object on the other side in your own code. It is
 possible that the object reconstruction will become extensible, so you can add your own
 non-copying types, but this does not yet exist.
 
 Closures
 ********
 
 Just about anything in Python is pickleable. The one notable exception is objects (generally
 functions) with *closures*. Closures can be a complicated topic, but the basic principal is that
 functions that refer to variables in their parent scope have closures.
 
 An example of a function that uses a closure:
 
 .. sourcecode:: python
 
     def f(a):
         def inner():
             # inner will have a closure
             return a
         return inner
     
     f1 = f(1)
     f2 = f(2)
     f1() # returns 1
     f2() # returns 2
 
 ``f1`` and ``f2`` will have closures referring to the scope in which `inner` was defined,
 because they use the variable 'a'. As a result, you would not be able to send ``f1`` or ``f2``
 with IPython. Note that you *would* be able to send `f`. This is only true for interactively
 defined functions (as are often used in decorators), and only when there are variables used
 inside the inner function, that are defined in the outer function. If the names are *not* in the
 outer function, then there will not be a closure, and the generated function will look in
 ``globals()`` for the name:
 
 .. sourcecode:: python
 
     def g(b):
         # note that `b` is not referenced in inner's scope
         def inner():
             # this inner will *not* have a closure
             return a
         return inner
     g1 = g(1)
     g2 = g(2)
     g1() # raises NameError on 'a'
     a=5
     g2() # returns 5
 
 `g1` and `g2` *will* be sendable with IPython, and will treat the engine's namespace as
 globals().  The :meth:`pull` method is implemented based on this principle.  If we did not
 provide pull, you could implement it yourself with `apply`, by simply returning objects out
 of the global namespace:
 
 .. sourcecode:: ipython
 
     In [10]: view.apply(lambda : a)
     
     # is equivalent to
     In [11]: view.pull('a')
 
 Running Code
 ============
 
 There are two principal units of execution in Python: strings of Python code (e.g. 'a=5'),
 and Python functions.  IPython is designed around the use of functions via the core
 Client method, called `apply`.
 
 Apply
 -----
 
 The principal method of remote execution is :meth:`apply`, of
 :class:`~IPython.parallel.client.view.View` objects. The Client provides the full execution and
 communication API for engines via its low-level :meth:`send_apply_message` method, which is used
 by all higher level methods of its Views.
 
 f : function
     The fuction to be called remotely
 args : tuple/list
     The positional arguments passed to `f`
 kwargs : dict
     The keyword arguments passed to `f`
 
 flags for all views:
 
 block : bool (default: view.block)
     Whether to wait for the result, or return immediately.
+
     False:
         returns AsyncResult
     True:
-        returns actual result(s) of f(*args, **kwargs)
+        returns actual result(s) of ``f(*args, **kwargs)``
+
         if multiple targets:
             list of results, matching `targets`
+
 track : bool [default view.track]
     whether to track non-copying sends.
 
 targets : int,list of ints, 'all', None [default view.targets]
     Specify the destination of the job.
+
     if 'all' or None:
         Run on all active engines
     if list:
         Run on each specified engine
     if int:
         Run on single engine
 
-Note that LoadBalancedView uses targets to restrict possible destinations.  LoadBalanced calls
-will always execute in just one location.
+Note that :class:`LoadBalancedView` uses targets to restrict possible destinations.
+LoadBalanced calls will always execute in just one location.
 
 flags only in LoadBalancedViews:
 
 after : Dependency or collection of msg_ids
     Only for load-balanced execution (targets=None)
     Specify a list of msg_ids as a time-based dependency.
     This job will only be run *after* the dependencies
     have been met.
     
 follow : Dependency or collection of msg_ids
     Only for load-balanced execution (targets=None)
     Specify a list of msg_ids as a location-based dependency.
     This job will only be run on an engine where this dependency
     is met.
 
 timeout : float/int or None
     Only for load-balanced execution (targets=None)
     Specify an amount of time (in seconds) for the scheduler to
     wait for dependencies to be met before failing with a
     DependencyTimeout.
 
 execute and run
 ---------------
 
 For executing strings of Python code, :class:`DirectView` 's also provide an :meth:`execute` and
 a :meth:`run` method, which rather than take functions and arguments, take simple strings.
 `execute` simply takes a string of Python code to execute, and sends it to the Engine(s). `run`
 is the same as `execute`, but for a *file*, rather than a string. It is simply a wrapper that
 does something very similar to ``execute(open(f).read())``.
 
 .. note::
 
     TODO: Examples for execute and run
 
 Views
 =====
 
 The principal extension of the :class:`~parallel.Client` is the :class:`~parallel.View`
 class. The client is typically a singleton for connecting to a cluster, and presents a
 low-level interface to the Hub and Engines. Most real usage will involve creating one or more
 :class:`~parallel.View` objects for working with engines in various ways.
 
 
 DirectView
 ----------
 
 The :class:`.DirectView` is the class for the IPython :ref:`Multiplexing Interface
 <parallel_multiengine>`.
 
 Creating a DirectView
 *********************
 
 DirectViews can be created in two ways, by index access to a client, or by a client's
 :meth:`view` method.  Index access to a Client works in a few ways.  First, you can create
 DirectViews to single engines simply by accessing the client by engine id:
 
 .. sourcecode:: ipython
 
     In [2]: rc[0]
     Out[2]: <DirectView 0>
 
 You can also create a DirectView with a list of engines:
 
 .. sourcecode:: ipython
 
     In [2]: rc[0,1,2]
     Out[2]: <DirectView [0,1,2]>
 
 Other methods for accessing elements, such as slicing and negative indexing, work by passing
 the index directly to the client's :attr:`ids` list, so:
 
 .. sourcecode:: ipython
 
     # negative index
     In [2]: rc[-1]
     Out[2]: <DirectView 3>
     
     # or slicing:
     In [3]: rc[::2]
     Out[3]: <DirectView [0,2]>
 
 are always the same as:
 
 .. sourcecode:: ipython
 
     In [2]: rc[rc.ids[-1]]
     Out[2]: <DirectView 3>
     
     In [3]: rc[rc.ids[::2]]
     Out[3]: <DirectView [0,2]>
 
 Also note that the slice is evaluated at the time of construction of the DirectView, so the 
 targets will not change over time if engines are added/removed from the cluster.
 
 Execution via DirectView
 ************************
 
 The DirectView is the simplest way to work with one or more engines directly (hence the name).
 
 For instance, to get the process ID of all your engines:
 
 .. sourcecode:: ipython
 
     In [5]: import os
     
     In [6]: dview.apply_sync(os.getpid)
     Out[6]: [1354, 1356, 1358, 1360]
 
 Or to see the hostname of the machine they are on:
 
 .. sourcecode:: ipython
 
     In [5]: import socket
     
     In [6]: dview.apply_sync(socket.gethostname)
     Out[6]: ['tesla', 'tesla', 'edison', 'edison', 'edison']
 
 .. note::
 
     TODO: expand on direct execution
 
 Data movement via DirectView
 ****************************
 
 Since a Python namespace is just a :class:`dict`, :class:`DirectView` objects provide
 dictionary-style access by key and methods such as :meth:`get` and
 :meth:`update` for convenience. This make the remote namespaces of the engines
 appear as a local dictionary. Underneath, these methods call :meth:`apply`:
 
 .. sourcecode:: ipython
 
     In [51]: dview['a']=['foo','bar']
 
     In [52]: dview['a']
     Out[52]: [ ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'] ]
 
 Scatter and gather
 ------------------
 
 Sometimes it is useful to partition a sequence and push the partitions to
 different engines. In MPI language, this is know as scatter/gather and we
 follow that terminology. However, it is important to remember that in
 IPython's :class:`Client` class, :meth:`scatter` is from the
 interactive IPython session to the engines and :meth:`gather` is from the
 engines back to the interactive IPython session. For scatter/gather operations
 between engines, MPI should be used:
 
 .. sourcecode:: ipython
 
     In [58]: dview.scatter('a',range(16))
     Out[58]: [None,None,None,None]
 
     In [59]: dview['a']
     Out[59]: [ [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15] ]
 
     In [60]: dview.gather('a')
     Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
 Push and pull
 -------------
 
 :meth:`~IPython.parallel.client.view.DirectView.push`
 
 :meth:`~IPython.parallel.client.view.DirectView.pull`
 
 .. note::
 
     TODO: write this section
 
 
 LoadBalancedView
 ----------------
 
 The :class:`~.LoadBalancedView` is the class for load-balanced execution via the task scheduler.
 These views always run tasks on exactly one engine, but let the scheduler determine where that
 should be, allowing load-balancing of tasks. The LoadBalancedView does allow you to specify
 restrictions on where and when tasks can execute, for more complicated load-balanced workflows.
 
 Data Movement
 =============
 
 Since the :class:`~.LoadBalancedView` does not know where execution will take place, explicit
 data movement methods like push/pull and scatter/gather do not make sense, and are not provided.
 
 Results
 =======
 
 AsyncResults
 ------------
 
 Our primary representation of the results of remote execution is the :class:`~.AsyncResult`
 object, based on the object of the same name in the built-in :mod:`multiprocessing.pool`
 module. Our version provides a superset of that interface.
 
 The basic principle of the AsyncResult is the encapsulation of one or more results not yet completed.  Execution methods (including data movement, such as push/pull) will all return
 AsyncResults when `block=False`.
 
 The mp.pool.AsyncResult interface
 ---------------------------------
 
 The basic interface of the AsyncResult is exactly that of the AsyncResult in :mod:`multiprocessing.pool`, and consists of four methods:
 
 .. AsyncResult spec directly from docs.python.org
 
 .. class:: AsyncResult
 
    The stdlib AsyncResult spec
    
    .. method:: wait([timeout])
 
       Wait until the result is available or until *timeout* seconds pass. This
       method always returns ``None``.
 
    .. method:: ready()
 
       Return whether the call has completed.
 
    .. method:: successful()
 
       Return whether the call completed without raising an exception.  Will
       raise :exc:`AssertionError` if the result is not ready.
 
    .. method:: get([timeout])
 
       Return the result when it arrives.  If *timeout* is not ``None`` and the
       result does not arrive within *timeout* seconds then
       :exc:`TimeoutError` is raised.  If the remote call raised
       an exception then that exception will be reraised as a :exc:`RemoteError`
       by :meth:`get`.
 
 
 While an AsyncResult is not done, you can check on it with its :meth:`ready` method, which will
 return whether the AR is done. You can also wait on an AsyncResult with its :meth:`wait` method.
 This method blocks until the result arrives. If you don't want to wait forever, you can pass a
 timeout (in seconds) as an argument to :meth:`wait`. :meth:`wait` will *always return None*, and
 should never raise an error.
 
 :meth:`ready` and :meth:`wait` are insensitive to the success or failure of the call. After a
 result is done, :meth:`successful` will tell you whether the call completed without raising an
 exception.
 
 If you actually want the result of the call, you can use :meth:`get`. Initially, :meth:`get`
 behaves just like :meth:`wait`, in that it will block until the result is ready, or until a
 timeout is met. However, unlike :meth:`wait`, :meth:`get` will raise a :exc:`TimeoutError` if
 the timeout is reached and the result is still not ready. If the result arrives before the
 timeout is reached, then :meth:`get` will return the result itself if no exception was raised,
 and will raise an exception if there was.
 
 Here is where we start to expand on the multiprocessing interface. Rather than raising the
 original exception, a RemoteError will be raised, encapsulating the remote exception with some
 metadata. If the AsyncResult represents multiple calls (e.g. any time `targets` is plural), then
 a CompositeError, a subclass of RemoteError, will be raised.
 
 .. seealso::
 
     For more information on remote exceptions, see :ref:`the section in the Direct Interface
     <parallel_exceptions>`.
 
 Extended interface
 ******************
 
 
 Other extensions of the AsyncResult interface include convenience wrappers for :meth:`get`.
 AsyncResults have a property, :attr:`result`, with the short alias :attr:`r`, which simply call
 :meth:`get`. Since our object is designed for representing *parallel* results, it is expected
 that many calls (any of those submitted via DirectView) will map results to engine IDs. We
 provide a :meth:`get_dict`, which is also a wrapper on :meth:`get`, which returns a dictionary
 of the individual results, keyed by engine ID.
 
 You can also prevent a submitted job from actually executing, via the AsyncResult's
 :meth:`abort` method. This will instruct engines to not execute the job when it arrives.
 
 The larger extension of the AsyncResult API is the :attr:`metadata` attribute.  The metadata
 is a dictionary (with attribute access) that contains, logically enough, metadata about the
 execution.
 
 Metadata keys:
 
 timestamps
 
 submitted
     When the task left the Client
 started
     When the task started execution on the engine
 completed
     When execution finished on the engine
 received
     When the result arrived on the Client
     
     note that it is not known when the result arrived in 0MQ on the client, only when it
     arrived in Python via :meth:`Client.spin`, so in interactive use, this may not be
     strictly informative.
 
 Information about the engine
 
 engine_id
     The integer id
 engine_uuid
     The UUID of the engine
 
 output of the call
 
 pyerr
     Python exception, if there was one
 pyout
     Python output
 stderr
     stderr stream
 stdout
     stdout (e.g. print) stream
 
 And some extended information
 
 status
     either 'ok' or 'error'
 msg_id
     The UUID of the message
 after
     For tasks: the time-based msg_id dependencies
 follow
     For tasks: the location-based msg_id dependencies
 
 While in most cases, the Clients that submitted a request will be the ones using the results,
 other Clients can also request results directly from the Hub. This is done via the Client's
 :meth:`get_result` method. This method will *always* return an AsyncResult object. If the call
 was not submitted by the client, then it will be a subclass, called :class:`AsyncHubResult`.
 These behave in the same way as an AsyncResult, but if the result is not ready, waiting on an
 AsyncHubResult polls the Hub, which is much more expensive than the passive polling used
 in regular AsyncResults.
 
 
 The Client keeps track of all results
 history, results, metadata
 
 Querying the Hub
 ================
 
 The Hub sees all traffic that may pass through the schedulers between engines and clients.
 It does this so that it can track state, allowing multiple clients to retrieve results of
 computations submitted by their peers, as well as persisting the state to a database.
 
 queue_status
 
     You can check the status of the queues of the engines with this command.
 
 result_status
 
     check on results
 
 purge_results
 
     forget results (conserve resources)
 
 Controlling the Engines
 =======================
 
 There are a few actions you can do with Engines that do not involve execution.  These
 messages are sent via the Control socket, and bypass any long queues of waiting execution
 jobs
 
 abort
 
     Sometimes you may want to prevent a job you have submitted from actually running. The method
     for this is :meth:`abort`. It takes a container of msg_ids, and instructs the Engines to not
     run the jobs if they arrive. The jobs will then fail with an AbortedTask error.
 
 clear
 
     You may want to purge the Engine(s) namespace of any data you have left in it.  After
     running `clear`, there will be no names in the Engine's namespace
 
 shutdown
 
     You can also instruct engines (and the Controller) to terminate from a Client.  This 
     can be useful when a job is finished, since you can shutdown all the processes with a
     single command.
 
 Synchronization
 ===============
 
 Since the Client is a synchronous object, events do not automatically trigger in your
 interactive session - you must poll the 0MQ sockets for incoming messages.  Note that 
 this polling *does not* actually make any network requests.  It simply performs a `select`
 operation, to check if messages are already in local memory, waiting to be handled.
 
 The method that handles incoming messages is :meth:`spin`. This method flushes any waiting
 messages on the various incoming sockets, and updates the state of the Client.
 
 If you need to wait for particular results to finish, you can use the :meth:`wait` method,
 which will call :meth:`spin` until the messages are no longer outstanding. Anything that
 represents a collection of messages, such as a list of msg_ids or one or more AsyncResult
 objects, can be passed as argument to wait. A timeout can be specified, which will prevent
 the call from blocking for more than a specified time, but the default behavior is to wait
 forever.
 
 The client also has an ``outstanding`` attribute - a ``set`` of msg_ids that are awaiting
 replies. This is the default if wait is called with no arguments - i.e. wait on *all*
 outstanding messages.
 
 
 .. note::
 
     TODO wait example
 
 Map
 ===
 
 Many parallel computing problems can be expressed as a ``map``, or running a single program with
 a variety of different inputs. Python has a built-in :py:func:`map`, which does exactly this,
 and many parallel execution tools in Python, such as the built-in
 :py:class:`multiprocessing.Pool` object provide implementations of `map`. All View objects
 provide a :meth:`map` method as well, but the load-balanced and direct implementations differ.
 
 Views' map methods can be called on any number of sequences, but they can also take the `block`
 and `bound` keyword arguments, just like :meth:`~client.apply`, but *only as keywords*.
 
 .. sourcecode:: python
 
     dview.map(*sequences, block=None)
 
 
 * iter, map_async, reduce
 
 Decorators and RemoteFunctions
 ==============================
 
 .. note::
 
     TODO: write this section
 
 :func:`~IPython.parallel.client.remotefunction.@parallel`
 
 :func:`~IPython.parallel.client.remotefunction.@remote`
 
 :class:`~IPython.parallel.client.remotefunction.RemoteFunction`
 
 :class:`~IPython.parallel.client.remotefunction.ParallelFunction`
 
 Dependencies
 ============
 
 .. note::
 
     TODO: write this section
 
 :func:`~IPython.parallel.controller.dependency.@depend`
 
 :func:`~IPython.parallel.controller.dependency.@require`
 
 :class:`~IPython.parallel.controller.dependency.Dependency`

docs/source/parallel/parallel_transition.rst

@@ -1,245 +1,245
 .. _parallel_transition:
 
 =====================================================
 Transitioning from IPython.kernel to IPython.parallel
 =====================================================
 
 
 We have rewritten our parallel computing tools to use 0MQ_ and Tornado_.  The redesign
 has resulted in dramatically improved performance, as well as (we think), an improved
 interface for executing code remotely.  This doc is to help users of IPython.kernel
 transition their codes to the new code.
 
 .. _0MQ: http://zeromq.org
 .. _Tornado: https://github.com/facebook/tornado
 
 
 Processes
 =========
 
 The process model for the new parallel code is very similar to that of IPython.kernel. There is
 still a Controller, Engines, and Clients. However, the the Controller is now split into multiple
 processes, and can even be split across multiple machines. There does remain a single
 ipcontroller script for starting all of the controller processes.
 
 
 .. note::
 
     TODO: fill this out after config system is updated
 
 
 .. seealso::
 
     Detailed :ref:`Parallel Process <parallel_process>` doc for configuring and launching
     IPython processes.
 
 Creating a Client
 =================
 
 Creating a client with default settings has not changed much, though the extended options have.
 One significant change is that there are no longer multiple Client classes to represent the
 various execution models. There is just one low-level Client object for connecting to the
 cluster, and View objects are created from that Client that provide the different interfaces for
 execution.
 
 
 To create a new client, and set up the default direct and load-balanced objects:
 
 .. sourcecode:: ipython
 
     # old
     In [1]: from IPython.kernel import client as kclient
     
     In [2]: mec = kclient.MultiEngineClient()
 
     In [3]: tc = kclient.TaskClient()
     
     # new 
     In [1]: from IPython.parallel import Client
     
     In [2]: rc = Client()
 
     In [3]: dview = rc[:]
     
     In [4]: lbview = rc.load_balanced_view()
 
 Apply
 =====
 
 The main change to the API is the addition of the :meth:`apply` to the View objects. This is a
 method that takes `view.apply(f,*args,**kwargs)`, and calls `f(*args, **kwargs)` remotely on one
 or more engines, returning the result. This means that the natural unit of remote execution
 is no longer a string of Python code, but rather a Python function.
 
 * non-copying sends (track)
 * remote References
 
 The flags for execution have also changed.  Previously, there was only `block` denoting whether
 to wait for results.  This remains, but due to the addition of fully non-copying sends of 
 arrays and buffers, there is also a `track` flag, which instructs PyZMQ to produce a :class:`MessageTracker` that will let you know when it is safe again to edit arrays in-place.
 
 The result of a non-blocking call to `apply` is now an AsyncResult_ object, described below.
 
 MultiEngine to DirectView
 =========================
 
 The multiplexing interface previously provided by the MultiEngineClient is now provided by the
 DirectView. Once you have a Client connected, you can create a DirectView with index-access
 to the client (``view = client[1:5]``). The core methods for
 communicating with engines remain: `execute`, `run`, `push`, `pull`, `scatter`, `gather`. These
 methods all behave in much the same way as they did on a MultiEngineClient.
 
 
 .. sourcecode:: ipython
 
     # old
     In [2]: mec.execute('a=5', targets=[0,1,2])
     
     # new
     In [2]: view.execute('a=5', targets=[0,1,2])
     # or
     In [2]: rc[0,1,2].execute('a=5')
     
 
 This extends to any method that communicates with the engines. 
 
 Requests of the Hub (queue status, etc.) are no-longer asynchronous, and do not take a `block`
 argument.
 
 
 * :meth:`get_ids` is now the property :attr:`ids`, which is passively updated by the Hub (no
   need for network requests for an up-to-date list).
 * :meth:`barrier` has been renamed to :meth:`wait`, and now takes an optional timeout. :meth:`flush` is removed, as it is redundant with :meth:`wait`
 * :meth:`zip_pull` has been removed
 * :meth:`keys` has been removed, but is easily implemented as::
 
     dview.apply(lambda : globals().keys())
 
 * :meth:`push_function` and :meth:`push_serialized` are removed, as :meth:`push` handles 
   functions without issue.
  
 .. seealso::
 
     :ref:`Our Direct Interface doc <parallel_multiengine>` for a simple tutorial with the 
     DirectView.
 
 
 The other major difference is the use of :meth:`apply`. When remote work is simply functions,
 the natural return value is the actual Python objects. It is no longer the recommended pattern
 to use stdout as your results, due to stream decoupling and the asynchronous nature of how the
 stdout streams are handled in the new system.
 
 Task to LoadBalancedView
 ========================
 
 Load-Balancing has changed more than Multiplexing.  This is because there is no longer a notion
 of a StringTask or a MapTask, there are simply Python functions to call.  Tasks are now
 simpler, because they are no longer composites of push/execute/pull/clear calls, they are
 a single function that takes arguments, and returns objects.
 
 The load-balanced interface is provided by the :class:`LoadBalancedView` class, created by the client:
 
 .. sourcecode:: ipython
 
     In [10]: lbview = rc.load_balanced_view()
     
     # load-balancing can also be restricted to a subset of engines:
     In [10]: lbview = rc.load_balanced_view([1,2,3])
 
 A simple task would consist of sending some data, calling a function on that data, plus some
 data that was resident on the engine already, and then pulling back some results.  This can
 all be done with a single function.
 
 
 Let's say you want to compute the dot product of two matrices, one of which resides on the
 engine, and another resides on the client.  You might construct a task that looks like this:
 
 .. sourcecode:: ipython
 
     In [10]: st = kclient.StringTask("""
                 import numpy
                 C=numpy.dot(A,B)
                 """,
                 push=dict(B=B),
                 pull='C'
                 )
     
     In [11]: tid = tc.run(st)
     
     In [12]: tr = tc.get_task_result(tid)
     
     In [13]: C = tc['C']
 
 In the new code, this is simpler:
 
 .. sourcecode:: ipython
 
     In [10]: import numpy
     
     In [11]: from IPython.parallel import Reference
     
     In [12]: ar = lbview.apply(numpy.dot, Reference('A'), B)
     
     In [13]: C = ar.get()
 
 Note the use of ``Reference`` This is a convenient representation of an object that exists
 in the engine's namespace, so you can pass remote objects as arguments to your task functions.
 
 Also note that in the kernel model, after the task is run, 'A', 'B', and 'C' are all defined on
 the engine. In order to deal with this, there is also a `clear_after` flag for Tasks to prevent
 pollution of the namespace, and bloating of engine memory. This is not necessary with the new
 code, because only those objects explicitly pushed (or set via `globals()`) will be resident on
 the engine beyond the duration of the task.
 
 .. seealso::
 
     Dependencies also work very differently than in IPython.kernel.  See our :ref:`doc on Dependencies<parallel_dependencies>` for details.
 
 .. seealso::
 
     :ref:`Our Task Interface doc <parallel_task>` for a simple tutorial with the 
     LoadBalancedView.
 
 
 PendingResults to AsyncResults
 ------------------------------
 
 With the departure from Twisted, we no longer have the :class:`Deferred` class for representing
 unfinished results. For this, we have an AsyncResult object, based on the object of the same
 name in the built-in :mod:`multiprocessing.pool` module. Our version provides a superset of that
 interface.
 
 However, unlike in IPython.kernel, we do not have PendingDeferred, PendingResult, or TaskResult
 objects. Simply this one object, the AsyncResult. Every asynchronous (`block=False`) call
 returns one.
 
 The basic methods of an AsyncResult are:
 
 .. sourcecode:: python
 
     AsyncResult.wait([timeout]): # wait for the result to arrive
     AsyncResult.get([timeout]): # wait for the result to arrive, and then return it
     AsyncResult.metadata: # dict of extra information about execution.
 
 There are still some things that behave the same as IPython.kernel:
 
 .. sourcecode:: ipython
 
     # old
     In [5]: pr = mec.pull('a', targets=[0,1], block=False)
     In [6]: pr.r
     Out[6]: [5, 5]
 
     # new
     In [5]: ar = dview.pull('a', targets=[0,1], block=False)
     In [6]: ar.r
     Out[6]: [5, 5]
 
 The ``.r`` or ``.result`` property simply calls :meth:`get`, waiting for and returning the
 result.
 
 .. seealso::
 
-    :ref:`AsyncResult details <AsyncResult>`
+    :doc:`AsyncResult details <asyncresult>`