add note about specifying IP for external engines in parallel_process doc

* also cleanup parallel_details, which still has several unwritten sections
* and rebuild outdated exception output in multiengine doc

docs/source/parallel/parallel_details.txt

@@ -6,7 +6,7 Details of Parallel Computing with IPython
 
 .. note::
 
-    There are still many sections to fill out
+    There are still many sections to fill out in this doc
 
 
 Caveats
@@ -70,9 +70,11 The :attr:`ndarray.flags.writeable` flag will tell you if you can write to an ar
     In [6]: _.flags.writeable
     Out[6]: False
 
-If you want to safely edit an array in-place after *sending* it, you must use the `track=True` flag.  IPython always performs non-copying sends of arrays, which return immediately.  You
-must instruct IPython track those messages *at send time* in order to know for sure that the send has completed.  AsyncResults have a :attr:`sent` property, and :meth:`wait_on_send` method
-for checking and waiting for 0MQ to finish with a buffer.
+If you want to safely edit an array in-place after *sending* it, you must use the `track=True`
+flag. IPython always performs non-copying sends of arrays, which return immediately. You must
+instruct IPython track those messages *at send time* in order to know for sure that the send has
+completed. AsyncResults have a :attr:`sent` property, and :meth:`wait_on_send` method for
+checking and waiting for 0MQ to finish with a buffer.
 
 .. sourcecode:: ipython
 
@@ -124,12 +126,12 An example of a function that uses a closure:
     f1() # returns 1
     f2() # returns 2
 
-f1 and f2 will have closures referring to the scope in which `inner` was defined, because they
-use the variable 'a'. As a result, you would not be able to send ``f1`` or ``f2`` with IPython.
-Note that you *would* be able to send `f`. This is only true for interactively defined
-functions (as are often used in decorators), and only when there are variables used inside the
-inner function, that are defined in the outer function. If the names are *not* in the outer
-function, then there will not be a closure, and the generated function will look in
+``f1`` and ``f2`` will have closures referring to the scope in which `inner` was defined,
+because they use the variable 'a'. As a result, you would not be able to send ``f1`` or ``f2``
+with IPython. Note that you *would* be able to send `f`. This is only true for interactively
+defined functions (as are often used in decorators), and only when there are variables used
+inside the inner function, that are defined in the outer function. If the names are *not* in the
+outer function, then there will not be a closure, and the generated function will look in
 ``globals()`` for the name:
 
 .. sourcecode:: python
@@ -168,9 +170,10 Client method, called `apply`.
 Apply
 -----
 
-The principal method of remote execution is :meth:`apply`, of View objects. The Client provides
-the full execution and communication API for engines via its low-level
-:meth:`send_apply_message` method.
+The principal method of remote execution is :meth:`apply`, of
+:class:`~IPython.parallel.client.view.View` objects. The Client provides the full execution and
+communication API for engines via its low-level :meth:`send_apply_message` method, which is used
+by all higher level methods of its Views.
 
 f : function
     The fuction to be called remotely
@@ -227,21 +230,23 timeout : float/int or None
 execute and run
 ---------------
 
-For executing strings of Python code, :class:`DirectView`s also provide an :meth:`execute` and a
-:meth:`run` method, which rather than take functions and arguments, take simple strings.
+For executing strings of Python code, :class:`DirectView` 's also provide an :meth:`execute` and
+a :meth:`run` method, which rather than take functions and arguments, take simple strings.
 `execute` simply takes a string of Python code to execute, and sends it to the Engine(s). `run`
 is the same as `execute`, but for a *file*, rather than a string. It is simply a wrapper that
 does something very similar to ``execute(open(f).read())``.
 
 .. note::
 
-    TODO: Example
+    TODO: Examples for execute and run
 
 Views
 =====
 
-The principal extension of the :class:`~parallel.Client` is the
-:class:`~parallel.View` class. The client
+The principal extension of the :class:`~parallel.Client` is the :class:`~parallel.View`
+class. The client is typically a singleton for connecting to a cluster, and presents a
+low-level interface to the Hub and Engines. Most real usage will involve creating one or more
+:class:`~parallel.View` objects for working with engines in various ways.
 
 
 DirectView
@@ -300,6 +305,27 Execution via DirectView
 
 The DirectView is the simplest way to work with one or more engines directly (hence the name).
 
+For instance, to get the process ID of all your engines:
+
+.. sourcecode:: ipython
+
+    In [5]: import os
+    
+    In [6]: dview.apply_sync(os.getpid)
+    Out[6]: [1354, 1356, 1358, 1360]
+
+Or to see the hostname of the machine they are on:
+
+.. sourcecode:: ipython
+
+    In [5]: import socket
+    
+    In [6]: dview.apply_sync(socket.gethostname)
+    Out[6]: ['tesla', 'tesla', 'edison', 'edison', 'edison']
+
+.. note::
+
+    TODO: expand on direct execution
 
 Data movement via DirectView
 ****************************
@@ -341,24 +367,28 between engines, MPI should be used:
 Push and pull
 -------------
 
-push
+:meth:`~IPython.parallel.client.view.DirectView.push`
 
-pull
+:meth:`~IPython.parallel.client.view.DirectView.pull`
 
+.. note::
 
+    TODO: write this section
 
 
-    
 LoadBalancedView
 ----------------
 
-The :class:`.LoadBalancedView` 
-
+The :class:`~.LoadBalancedView` is the class for load-balanced execution via the task scheduler.
+These views always run tasks on exactly one engine, but let the scheduler determine where that
+should be, allowing load-balancing of tasks. The LoadBalancedView does allow you to specify
+restrictions on where and when tasks can execute, for more complicated load-balanced workflows.
 
 Data Movement
 =============
 
-Reference
+Since the :class:`~.LoadBalancedView` does not know where execution will take place, explicit
+data movement methods like push/pull and scatter/gather do not make sense, and are not provided.
 
 Results
 =======
@@ -366,9 +396,9 Results
 AsyncResults
 ------------
 
-Our primary representation is the AsyncResult object, based on the object of the same name in
-the built-in :mod:`multiprocessing.pool` module. Our version provides a superset of that
-interface.
+Our primary representation of the results of remote execution is the :class:`~.AsyncResult`
+object, based on the object of the same name in the built-in :mod:`multiprocessing.pool`
+module. Our version provides a superset of that interface.
 
 The basic principle of the AsyncResult is the encapsulation of one or more results not yet completed.  Execution methods (including data movement, such as push/pull) will all return
 AsyncResults when `block=False`.
@@ -432,7 +462,7 a CompositeError, a subclass of RemoteError, will be raised.
 .. seealso::
 
     For more information on remote exceptions, see :ref:`the section in the Direct Interface
-    <Parallel_exceptions>`.
+    <parallel_exceptions>`.
 
 Extended interface
 ******************
@@ -445,7 +475,8 that many calls (any of those submitted via DirectView) will map results to engi
 provide a :meth:`get_dict`, which is also a wrapper on :meth:`get`, which returns a dictionary
 of the individual results, keyed by engine ID.
 
-You can also prevent a submitted job from actually executing, via the AsyncResult's :meth:`abort` method.  This will instruct engines to not execute the job when it arrives.
+You can also prevent a submitted job from actually executing, via the AsyncResult's
+:meth:`abort` method. This will instruct engines to not execute the job when it arrives.
 
 The larger extension of the AsyncResult API is the :attr:`metadata` attribute.  The metadata
 is a dictionary (with attribute access) that contains, logically enough, metadata about the
@@ -570,11 +601,9 objects, can be passed as argument to wait. A timeout can be specified, which wi
 the call from blocking for more than a specified time, but the default behavior is to wait
 forever.
 
-
-
-The client also has an `outstanding` attribute - a ``set`` of msg_ids that are awaiting replies.
-This is the default if wait is called with no arguments - i.e. wait on *all* outstanding
-messages.
+The client also has an ``outstanding`` attribute - a ``set`` of msg_ids that are awaiting
+replies. This is the default if wait is called with no arguments - i.e. wait on *all*
+outstanding messages.
 
 
 .. note::
@@ -584,11 +613,11 messages.
 Map
 ===
 
-Many parallel computing problems can be expressed as a `map`, or running a single program with a
-variety of different inputs. Python has a built-in :py-func:`map`, which does exactly this, and
-many parallel execution tools in Python, such as the built-in :py-class:`multiprocessing.Pool`
-object provide implementations of `map`. All View objects provide a :meth:`map` method as well,
-but the load-balanced and direct implementations differ.
+Many parallel computing problems can be expressed as a ``map``, or running a single program with
+a variety of different inputs. Python has a built-in :py:func:`map`, which does exactly this,
+and many parallel execution tools in Python, such as the built-in
+:py:class:`multiprocessing.Pool` object provide implementations of `map`. All View objects
+provide a :meth:`map` method as well, but the load-balanced and direct implementations differ.
 
 Views' map methods can be called on any number of sequences, but they can also take the `block`
 and `bound` keyword arguments, just like :meth:`~client.apply`, but *only as keywords*.
@@ -603,19 +632,27 and `bound` keyword arguments, just like :meth:`~client.apply`, but *only as key
 Decorators and RemoteFunctions
 ==============================
 
-@parallel
+.. note::
 
-@remote
+    TODO: write this section
 
-RemoteFunction
+:func:`~IPython.parallel.client.remotefunction.@parallel`
 
-ParallelFunction
+:func:`~IPython.parallel.client.remotefunction.@remote`
+
+:class:`~IPython.parallel.client.remotefunction.RemoteFunction`
+
+:class:`~IPython.parallel.client.remotefunction.ParallelFunction`
 
 Dependencies
 ============
 
-@depend
+.. note::
+
+    TODO: write this section
+
+:func:`~IPython.parallel.controller.dependency.@depend`
 
-@require
+:func:`~IPython.parallel.controller.dependency.@require`
 
-Dependency
+:class:`~IPython.parallel.controller.dependency.Dependency`

docs/source/parallel/parallel_multiengine.txt

@@ -598,6 +598,7 execution, and will fail with an UnmetDependencyError.
         ...:     time.sleep(t)
         ...:     return t
 
+.. _parallel_exceptions:
 
 Parallel exceptions
 -------------------
@@ -617,47 +618,47 more other types of exceptions. Here is how it works:
     In [77]: dview.execute('1/0')
     ---------------------------------------------------------------------------
     CompositeError                            Traceback (most recent call last)
-    /home/you/<ipython-input-10-15c2c22dec39> in <module>()
-    ----> 1 dview.execute('1/0', block=True)
-
-    /path/to/site-packages/IPython/parallel/view.py in execute(self, code, block)
-        460                 default: self.block
-        461         """
-    --> 462         return self.apply_with_flags(util._execute, args=(code,), block=block)
-        463 
-        464     def run(self, filename, block=None):
-
-    /home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
-
-    /path/to/site-packages/IPython/parallel/view.py in sync_results(f, self, *args, **kwargs)
-         46 def sync_results(f, self, *args, **kwargs):
-         47     """sync relevant results from self.client to our results attribute."""
-    ---> 48     ret = f(self, *args, **kwargs)
-         49     delta = self.outstanding.difference(self.client.outstanding)
-         50     completed = self.outstanding.intersection(delta)
-
-    /home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
-
-    /path/to/site-packages/IPython/parallel/view.py in save_ids(f, self, *args, **kwargs)
-         35     n_previous = len(self.client.history)
-         36     try:
-    ---> 37         ret = f(self, *args, **kwargs)
-         38     finally:
-         39         nmsgs = len(self.client.history) - n_previous
-
-    /path/to/site-packages/IPython/parallel/view.py in apply_with_flags(self, f, args, kwargs, block, track)
-        398         if block:
-        399             try:
-    --> 400                 return ar.get()
-        401             except KeyboardInterrupt:
-        402                 pass
-
-    /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
-         87                 return self._result
-         88             else:
-    ---> 89                 raise self._exception
-         90         else:
-         91             raise error.TimeoutError("Result not ready.")
+    /home/user/<ipython-input-10-5d56b303a66c> in <module>()
+    ----> 1 dview.execute('1/0')
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
+        591                 default: self.block
+        592         """
+    --> 593         return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
+        594 
+        595     def run(self, filename, targets=None, block=None):
+
+    /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
+         55 def sync_results(f, self, *args, **kwargs):
+         56     """sync relevant results from self.client to our results attribute."""
+    ---> 57     ret = f(self, *args, **kwargs)
+         58     delta = self.outstanding.difference(self.client.outstanding)
+         59     completed = self.outstanding.intersection(delta)
+
+    /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
+         44     n_previous = len(self.client.history)
+         45     try:
+    ---> 46         ret = f(self, *args, **kwargs)
+         47     finally:
+         48         nmsgs = len(self.client.history) - n_previous
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
+        529         if block:
+        530             try:
+    --> 531                 return ar.get()
+        532             except KeyboardInterrupt:
+        533                 pass
+
+    /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
+        101                 return self._result
+        102             else:
+    --> 103                 raise self._exception
+        104         else:
+        105             raise error.TimeoutError("Result not ready.")
 
     CompositeError: one or more exceptions from call to method: _execute
     [0:apply]: ZeroDivisionError: integer division or modulo by zero
@@ -665,7 +666,6 more other types of exceptions. Here is how it works:
     [2:apply]: ZeroDivisionError: integer division or modulo by zero
     [3:apply]: ZeroDivisionError: integer division or modulo by zero
 
-
 Notice how the error message printed when :exc:`CompositeError` is raised has
 information about the individual exceptions that were raised on each engine.
 If you want, you can even raise one of these original exceptions:
@@ -674,22 +674,32 If you want, you can even raise one of these original exceptions:
 
     In [80]: try:
        ....:     dview.execute('1/0')
-       ....: except client.CompositeError, e:
+       ....: except parallel.error.CompositeError, e:
        ....:     e.raise_exception()
        ....:     
        ....:     
     ---------------------------------------------------------------------------
-    ZeroDivisionError                         Traceback (most recent call last)
-
-    /ipython1-client-r3021/docs/examples/<ipython console> in <module>()
-
-    /ipython1-client-r3021/ipython1/kernel/error.pyc in raise_exception(self, excid)
-        156             raise IndexError("an exception with index %i does not exist"%excid)
-        157         else:
-    --> 158             raise et, ev, etb
-        159 
-        160 def collect_exceptions(rlist, method):
-
+    RemoteError                               Traceback (most recent call last)
+    /home/user/<ipython-input-17-8597e7e39858> in <module>()
+          2     dview.execute('1/0')
+          3 except CompositeError as e:
+    ----> 4     e.raise_exception()
+
+    /path/to/site-packages/IPython/parallel/error.pyc in raise_exception(self, excid)
+        266             raise IndexError("an exception with index %i does not exist"%excid)
+        267         else:
+    --> 268             raise RemoteError(en, ev, etb, ei)
+        269 
+        270 
+
+    RemoteError: ZeroDivisionError(integer division or modulo by zero)
+    Traceback (most recent call last):
+      File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
+        exec code in working,working
+      File "<string>", line 1, in <module>
+      File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
+        exec code in globals()
+      File "<string>", line 1, in <module>
     ZeroDivisionError: integer division or modulo by zero
 
 If you are working in IPython, you can simple type ``%debug`` after one of
@@ -701,47 +711,47 instance:
     In [81]: dview.execute('1/0')
     ---------------------------------------------------------------------------
     CompositeError                            Traceback (most recent call last)
-    /home/you/<ipython-input-10-15c2c22dec39> in <module>()
-    ----> 1 dview.execute('1/0', block=True)
-
-    /path/to/site-packages/IPython/parallel/view.py in execute(self, code, block)
-        460                 default: self.block
-        461         """
-    --> 462         return self.apply_with_flags(util._execute, args=(code,), block=block)
-        463 
-        464     def run(self, filename, block=None):
-
-    /home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
-
-    /path/to/site-packages/IPython/parallel/view.py in sync_results(f, self, *args, **kwargs)
-         46 def sync_results(f, self, *args, **kwargs):
-         47     """sync relevant results from self.client to our results attribute."""
-    ---> 48     ret = f(self, *args, **kwargs)
-         49     delta = self.outstanding.difference(self.client.outstanding)
-         50     completed = self.outstanding.intersection(delta)
-
-    /home/you/<string> in apply_with_flags(self, f, args, kwargs, block, track)
-
-    /path/to/site-packages/IPython/parallel/view.py in save_ids(f, self, *args, **kwargs)
-         35     n_previous = len(self.client.history)
-         36     try:
-    ---> 37         ret = f(self, *args, **kwargs)
-         38     finally:
-         39         nmsgs = len(self.client.history) - n_previous
-
-    /path/to/site-packages/IPython/parallel/view.py in apply_with_flags(self, f, args, kwargs, block, track)
-        398         if block:
-        399             try:
-    --> 400                 return ar.get()
-        401             except KeyboardInterrupt:
-        402                 pass
-
-    /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
-         87                 return self._result
-         88             else:
-    ---> 89                 raise self._exception
-         90         else:
-         91             raise error.TimeoutError("Result not ready.")
+    /home/user/<ipython-input-10-5d56b303a66c> in <module>()
+    ----> 1 dview.execute('1/0')
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in execute(self, code, targets, block)
+        591                 default: self.block
+        592         """
+    --> 593         return self._really_apply(util._execute, args=(code,), block=block, targets=targets)
+        594 
+        595     def run(self, filename, targets=None, block=None):
+
+    /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in sync_results(f, self, *args, **kwargs)
+         55 def sync_results(f, self, *args, **kwargs):
+         56     """sync relevant results from self.client to our results attribute."""
+    ---> 57     ret = f(self, *args, **kwargs)
+         58     delta = self.outstanding.difference(self.client.outstanding)
+         59     completed = self.outstanding.intersection(delta)
+
+    /home/user/<string> in _really_apply(self, f, args, kwargs, targets, block, track)
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in save_ids(f, self, *args, **kwargs)
+         44     n_previous = len(self.client.history)
+         45     try:
+    ---> 46         ret = f(self, *args, **kwargs)
+         47     finally:
+         48         nmsgs = len(self.client.history) - n_previous
+
+    /path/to/site-packages/IPython/parallel/client/view.pyc in _really_apply(self, f, args, kwargs, targets, block, track)
+        529         if block:
+        530             try:
+    --> 531                 return ar.get()
+        532             except KeyboardInterrupt:
+        533                 pass
+
+    /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
+        101                 return self._result
+        102             else:
+    --> 103                 raise self._exception
+        104         else:
+        105             raise error.TimeoutError("Result not ready.")
 
     CompositeError: one or more exceptions from call to method: _execute
     [0:apply]: ZeroDivisionError: integer division or modulo by zero
@@ -750,27 +760,26 instance:
     [3:apply]: ZeroDivisionError: integer division or modulo by zero
     
     In [82]: %debug
-    > /path/to/site-packages/IPython/parallel/asyncresult.py(80)get()
-         79             else:
-    ---> 80                 raise self._exception
-         81         else:
+    > /path/to/site-packages/IPython/parallel/client/asyncresult.py(103)get()
+        102             else:
+    --> 103                 raise self._exception
+        104         else:
     
-    
-    # With the debugger running, e is the exceptions instance.  We can tab complete
+    # With the debugger running, self._exception is the exceptions instance.  We can tab complete
     # on it and see the extra methods that are available.
-    ipdb> e.
+    ipdb> self._exception.<tab>
     e.__class__         e.__getitem__       e.__new__           e.__setstate__      e.args
     e.__delattr__       e.__getslice__      e.__reduce__        e.__str__           e.elist
     e.__dict__          e.__hash__          e.__reduce_ex__     e.__weakref__       e.message
     e.__doc__           e.__init__          e.__repr__          e._get_engine_str   e.print_tracebacks
     e.__getattribute__  e.__module__        e.__setattr__       e._get_traceback    e.raise_exception
-    ipdb> e.print_tracebacks()
+    ipdb> self._exception.print_tracebacks()
     [0:apply]: 
     Traceback (most recent call last):
-      File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
-        exec code in working, working
+      File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
+        exec code in working,working
       File "<string>", line 1, in <module>
-      File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
+      File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
         exec code in globals()
       File "<string>", line 1, in <module>
     ZeroDivisionError: integer division or modulo by zero
@@ -778,10 +787,10 instance:
 
     [1:apply]: 
     Traceback (most recent call last):
-      File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
-        exec code in working, working
+      File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
+        exec code in working,working
       File "<string>", line 1, in <module>
-      File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
+      File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
         exec code in globals()
       File "<string>", line 1, in <module>
     ZeroDivisionError: integer division or modulo by zero
@@ -789,10 +798,10 instance:
 
     [2:apply]: 
     Traceback (most recent call last):
-      File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
-        exec code in working, working
+      File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
+        exec code in working,working
       File "<string>", line 1, in <module>
-      File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
+      File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
         exec code in globals()
       File "<string>", line 1, in <module>
     ZeroDivisionError: integer division or modulo by zero
@@ -800,18 +809,13 instance:
 
     [3:apply]: 
     Traceback (most recent call last):
-      File "/path/to/site-packages/IPython/parallel/streamkernel.py", line 332, in apply_request
-        exec code in working, working
+      File "/path/to/site-packages/IPython/parallel/engine/streamkernel.py", line 330, in apply_request
+        exec code in working,working
       File "<string>", line 1, in <module>
-      File "/path/to/site-packages/IPython/parallel/client.py", line 69, in _execute
+      File "/path/to/site-packages/IPython/parallel/util.py", line 354, in _execute
         exec code in globals()
       File "<string>", line 1, in <module>
     ZeroDivisionError: integer division or modulo by zero
-    
-
-.. note::
-
-    TODO: The above tracebacks are not up to date
 
 
 All of this same error handling magic even works in non-blocking mode:
@@ -825,15 +829,15 All of this same error handling magic even works in non-blocking mode:
     In [85]: ar.get()
     ---------------------------------------------------------------------------
     CompositeError                            Traceback (most recent call last)
-    /Users/minrk/<ipython-input-3-8531eb3d26fb> in <module>()
+    /home/user/<ipython-input-21-8531eb3d26fb> in <module>()
     ----> 1 ar.get()
 
-    /path/to/site-packages/IPython/parallel/asyncresult.pyc in get(self, timeout)
-         78                 return self._result
-         79             else:
-    ---> 80                 raise self._exception
-         81         else:
-         82             raise error.TimeoutError("Result not ready.")
+    /path/to/site-packages/IPython/parallel/client/asyncresult.pyc in get(self, timeout)
+        101                 return self._result
+        102             else:
+    --> 103                 raise self._exception
+        104         else:
+        105             raise error.TimeoutError("Result not ready.")
 
     CompositeError: one or more exceptions from call to method: _execute
     [0:apply]: ZeroDivisionError: integer division or modulo by zero

docs/source/parallel/parallel_process.txt

@@ -27,11 +27,40 engines using the various methods, we outline some of the general issues that
 come up when starting the controller and engines. These things come up no
 matter which method you use to start your IPython cluster.
 
+If you are running engines on multiple machines, you will likely need to instruct the
+controller to listen for connections on an external interface. This can be done by specifying
+the ``ip`` argument on the command-line, or the ``HubFactory.ip`` configurable in
+:file:`ipcontroller_config.py`.
+
+If your machines are on a trusted network, you can safely instruct the controller to listen
+on all public interfaces with::
+
+    $> ipcontroller ip=*
+
+Or you can set the same behavior as the default by adding the following line to your :file:`ipcontroller_config.py`:
+
+.. sourcecode:: python
+
+    c.HubFactory.ip = '*'
+
+.. note::
+
+    Due to the lack of security in ZeroMQ, the controller will only listen for connections on
+    localhost by default. If you see Timeout errors on engines or clients, then the first
+    thing you should check is the ip address the controller is listening on, and make sure
+    that it is visible from the timing out machine.
+    
+.. seealso::
+
+    Our `notes <parallel_security>`_ on security in the new parallel computing code.
+
 Let's say that you want to start the controller on ``host0`` and engines on
 hosts ``host1``-``hostn``. The following steps are then required:
 
 1. Start the controller on ``host0`` by running :command:`ipcontroller` on
-   ``host0``.
+   ``host0``.  The controller must be instructed to listen on an interface visible
+   to the engine machines, via the ``ip`` command-line argument or ``HubFactory.ip``
+   in :file:`ipcontroller_config.py`.
 2. Move the JSON file (:file:`ipcontroller-engine.json`) created by the 
    controller from ``host0`` to hosts ``host1``-``hostn``.
 3. Start the engines on hosts ``host1``-``hostn`` by running 
@@ -108,7 +137,7 The configuration files are loaded with commented-out settings and explanations,
 which should cover most of the available possibilities.
 
 Using various batch systems with :command:`ipcluster`
-------------------------------------------------------
+-----------------------------------------------------
 
 :command:`ipcluster` has a notion of Launchers that can start controllers
 and engines with various remote execution schemes.  Currently supported
@@ -345,7 +374,7 The controller's remote location and configuration can be specified:
     # note that remotely launched ipcontroller will not get the contents of 
     # the local ipcontroller_config.py unless it resides on the *remote host*
     # in the location specified by the `profile_dir` argument.
-    # c.SSHControllerLauncher.program_args = ['--reuse', 'ip=0.0.0.0', 'profile_dir=/path/to/cd']
+    # c.SSHControllerLauncher.program_args = ['--reuse', 'ip=*', 'profile_dir=/path/to/cd']
 
 .. note::
 
@@ -414,7 +443,7 controller and engines from IPython.
     
     The order of the above operations may be important.  You *must*
     start the controller before the engines, unless you are reusing connection
-    information (via `--reuse`), in which case ordering is not important.
+    information (via ``--reuse``), in which case ordering is not important.
 
 .. note::
 
@@ -429,7 +458,9 Starting the controller and engines on different hosts
 When the controller and engines are running on different hosts, things are
 slightly more complicated, but the underlying ideas are the same:
 
-1. Start the controller on a host using :command:`ipcontroller`.
+1. Start the controller on a host using :command:`ipcontroller`. The controller must be
+   instructed to listen on an interface visible to the engine machines, via the ``ip``
+   command-line argument or ``HubFactory.ip`` in :file:`ipcontroller_config.py`.
 2. Copy :file:`ipcontroller-engine.json` from :file:`~/.ipython/profile_<name>/security` on
    the controller's host to the host where the engines will run.
 3. Use :command:`ipengine` on the engine's hosts to start the engines.
@@ -507,8 +538,7 To instruct the controller to listen on a specific interface, you can set the
 
     c.HubFactory.ip = '*'
 
-
-When connecting to a Controller that is listening on loopback, or behind a firewall, it may
+When connecting to a Controller that is listening on loopback or behind a firewall, it may
 be necessary to specify an SSH server to use for tunnels, and the external IP of the
 Controller. If you specified that the HubFactory listen on loopback, or all interfaces,
 then IPython will try to guess the external IP. If you are on a system with VM network