Updated the multiengine and task interface documentation.

The multiengine interface documentation is in very good condition.

The task interface documentation is in OK condition.  The main problem
is that the task documentation had fallen very out of date.  Rather
than trying to update all of it, I moved the old task documentation
to a file (docs/source/parallel/parallel_task_old.txt) and started
a new shorter task documentation that is fully up to date.

This should get user's started with the new stuff.  But, we do need
to write more complete task interface docs.

docs/source/parallel/parallel_task_old.txt

@@ -0,0 +1,240 b''
+.. _paralleltask:
+
+==========================
+The IPython task interface
+==========================
+
+.. contents::
+
+The ``Task`` interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the ``MultiEngine`` interface, in the ``Task`` interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful. Best of all the user can use both of these interfaces at the same time to take advantage or both of their strengths.  When the user can break up the user's work into segments that do not depend on previous execution, the ``Task`` interface is ideal.  But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign Tasks to engines explicitly.
+
+Starting the IPython controller and engines
+===========================================
+
+To follow along with this tutorial, the user will need to start the IPython
+controller and four IPython engines. The simplest way of doing this is to
+use the ``ipcluster`` command::
+
+	$ ipcluster -n 4
+	
+For more detailed information about starting the controller and engines, see our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+
+The magic here is that this single controller and set of engines is running both the MultiEngine and ``Task`` interfaces simultaneously.
+
+QuickStart Task Farming
+=======================
+
+First, a quick example of how to start running the most basic Tasks.
+The first step is to import the IPython ``client`` module and then create a ``TaskClient`` instance::
+    
+    In [1]: from IPython.kernel import client
+    
+    In [2]: tc = client.TaskClient()
+
+Then the user wrap the commands the user want to run in Tasks::
+
+    In [3]: tasklist = []
+    In [4]: for n in range(1000):
+        ...     tasklist.append(client.Task("a = %i"%n, pull="a"))
+
+The first argument of the ``Task`` constructor is a string, the command to be executed.  The most important optional keyword argument is ``pull``, which can be a string or list of strings, and it specifies the variable names to be saved as results of the ``Task``.
+
+Next, the user need to submit the Tasks to the ``TaskController`` with the ``TaskClient``::
+
+    In [5]: taskids = [ tc.run(t) for t in tasklist ]
+
+This will give the user a list of the TaskIDs used by the controller to keep track of the Tasks and their results.  Now at some point the user are going to want to get those results back.  The ``barrier`` method allows the user to wait for the Tasks to finish running::
+
+    In [6]: tc.barrier(taskids)
+
+This command will block until all the Tasks in ``taskids`` have finished.  Now, the user probably want to look at the user's results::
+
+    In [7]: task_results = [ tc.get_task_result(taskid) for taskid in taskids ]
+
+Now the user have a list of ``TaskResult`` objects, which have the actual result as a dictionary, but also keep track of some useful metadata about the ``Task``::
+
+    In [8]: tr = ``Task``_results[73]
+    
+    In [9]: tr
+    Out[9]: ``TaskResult``[ID:73]:{'a':73}
+    
+    In [10]: tr.engineid
+    Out[10]: 1
+    
+    In [11]: tr.submitted, tr.completed, tr.duration
+    Out[11]: ("2008/03/08 03:41:42", "2008/03/08 03:41:44", 2.12345)
+
+The actual results are stored in a dictionary, ``tr.results``, and a namespace object ``tr.ns`` which accesses the result keys by attribute::
+
+    In [12]: tr.results['a']
+    Out[12]: 73
+    
+    In [13]: tr.ns.a
+    Out[13]: 73
+
+That should cover the basics of running simple Tasks.  There are several more powerful things the user can do with Tasks covered later.  The most useful probably being using a ``MutiEngineClient`` interface to initialize all the engines with the import dependencies necessary to run the user's Tasks.
+
+There are many options for running and managing Tasks. The best way to learn further about the ``Task`` interface is to study the examples in ``docs/examples``. If the user do so and learn a lots about this interface, we encourage the user to expand this documentation about the ``Task`` system.
+
+Overview of the Task System
+===========================
+
+The user's view of the ``Task`` system has three basic objects:  The ``TaskClient``, the ``Task``, and the ``TaskResult``.  The names of these three objects well indicate their role.
+
+The ``TaskClient`` is the user's ``Task`` farming connection to the IPython cluster.  Unlike the ``MultiEngineClient``, the ``TaskControler`` handles all the scheduling and distribution of work, so the ``TaskClient`` has no notion of engines, it just submits Tasks and requests their results.  The Tasks are described as ``Task`` objects, and their results are wrapped in ``TaskResult`` objects.  Thus, there are very few necessary methods for the user to manage.
+
+Inside the task system is a Scheduler object, which assigns tasks to workers.  The default scheduler is a simple FIFO queue.  Subclassing the Scheduler should be easy, just implementing your own priority system.
+
+The TaskClient
+==============
+
+The ``TaskClient`` is the object the user use to connect to the ``Controller`` that is managing the user's Tasks.  It is the analog of the ``MultiEngineClient`` for the standard IPython multiplexing interface.  As with all client interfaces, the first step is to import the IPython Client Module::
+
+    In [1]: from IPython.kernel import client
+
+Just as with the ``MultiEngineClient``, the user create the ``TaskClient`` with a tuple, containing the ip-address and port of the ``Controller``.  the ``client`` module conveniently has the default address of the ``Task`` interface of the controller.  Creating a default ``TaskClient`` object would be done with this::
+
+    In [2]: tc = client.TaskClient(client.default_task_address)
+
+or, if the user want to specify a non default location of the ``Controller``, the user can specify explicitly::
+
+    In [3]: tc = client.TaskClient(("192.168.1.1", 10113))
+
+As discussed earlier, the ``TaskClient`` only has a few basic methods.
+
+ * ``tc.run(task)``
+    ``run`` is the method by which the user submits Tasks.  It takes exactly one argument, a ``Task`` object.  All the advanced control of ``Task`` behavior is handled by properties of the ``Task`` object, rather than the submission command, so they will be discussed later in the `Task`_ section.  ``run`` returns an integer, the ``Task``ID by which the ``Task`` and its results can be tracked and retrieved::
+    
+        In [4]: ``Task``ID = tc.run(``Task``)
+    
+ * ``tc.get_task_result(taskid, block=``False``)``
+    ``get_task_result`` is the method by which results are retrieved.  It takes a single integer argument, the ``Task``ID`` of the result the user wish to retrieve.  ``get_task_result`` also takes a keyword argument ``block``.  ``block`` specifies whether the user actually want to wait for the result.  If ``block`` is false, as it is by default, ``get_task_result`` will return immediately.  If the ``Task`` has completed, it will return the ``TaskResult`` object for that ``Task``.  But if the ``Task`` has not completed, it will return ``None``.  If the user specify ``block=``True``, then ``get_task_result`` will wait for the ``Task`` to complete, and always return the ``TaskResult`` for the requested ``Task``.
+ * ``tc.barrier(taskid(s))``
+    ``barrier`` is a synchronization method.  It takes exactly one argument, a ``Task``ID or list of taskIDs.  ``barrier`` will block until all the specified Tasks have completed.  In practice, a barrier is often called between the ``Task`` submission section of the code and the result gathering section::
+        
+        In [5]: taskIDs = [ tc.run(``Task``) for ``Task`` in myTasks ]
+        
+        In [6]: tc.get_task_result(taskIDs[-1]) is None
+        Out[6]: ``True``
+        
+        In [7]: tc.barrier(``Task``ID)
+        
+        In [8]: results = [ tc.get_task_result(tid) for tid in taskIDs ]
+        
+ * ``tc.queue_status(verbose=``False``)``
+    ``queue_status`` is a method for querying the state of the ``TaskControler``.  ``queue_status`` returns a dict of the form::
+        
+        {'scheduled': Tasks that have been submitted but yet run
+         'pending'  : Tasks that are currently running
+         'succeeded': Tasks that have completed successfully
+         'failed'   : Tasks that have finished with a failure
+        }
+    
+    if @verbose is not specified (or is ``False``), then the values of the dict are integers - the number of Tasks in each state.  if @verbose is ``True``, then each element in the dict is a list of the taskIDs in that state::
+    
+        In [8]: tc.queue_status()
+        Out[8]: {'scheduled': 4,
+                'pending'  : 2,
+                'succeeded': 5,
+                'failed'   : 1
+                }
+        
+        In [9]: tc.queue_status(verbose=True)
+        Out[9]: {'scheduled': [8,9,10,11],
+                'pending'  : [6,7],
+                'succeeded': [0,1,2,4,5],
+                'failed'   : [3]
+                }
+        
+ * ``tc.abort(taskid)``
+    ``abort`` allows the user to abort Tasks that have already been submitted.  ``abort`` will always return immediately.  If the ``Task`` has completed, ``abort`` will raise an ``IndexError ``Task`` Already Completed``.  An obvious case for ``abort`` would be where the user submits a long-running ``Task`` with a number of retries (see ``Task``_ section for how to specify retries) in an interactive session, but realizes there has been a typo.  The user can then abort the ``Task``, preventing certain failures from cluttering up the queue.  It can also be used for parallel search-type problems, where only one ``Task`` will give the solution, so once the user find the solution, the user would want to abort all remaining Tasks to prevent wasted work.
+ * ``tc.spin()``
+    ``spin`` simply triggers the scheduler in the ``TaskControler``.  Under most normal circumstances, this will do nothing.  The primary known usage case involves the ``Task`` dependency (see `Dependencies`_).  The dependency is a function of an Engine's ``properties``, but changing the ``properties`` via the ``MutliEngineClient`` does not trigger a reschedule event.  The main example case for this requires the following event sequence:
+     * ``engine`` is available, ``Task`` is submitted, but ``engine`` does not have ``Task``'s dependencies.
+     * ``engine`` gets necessary dependencies while no new Tasks are submitted or completed.
+     * now ``engine`` can run ``Task``, but a ``Task`` event is required for the ``TaskControler`` to try scheduling ``Task`` again.
+     
+    ``spin`` is just an empty ping method to ensure that the Controller has scheduled all available Tasks, and should not be needed under most normal circumstances.
+
+That covers the ``TaskClient``, a simple interface to the cluster.  With this, the user can submit jobs (and abort if necessary), request their results, synchronize on arbitrary subsets of jobs.
+
+.. _task: The Task Object
+
+The Task Object
+===============
+
+The ``Task`` is the basic object for describing a job.  It can be used in a very simple manner, where the user just specifies a command string to be executed as the ``Task``.  The usage of this first argument is exactly the same as the ``execute`` method of the ``MultiEngine`` (in fact, ``execute`` is called to run the code)::
+
+    In [1]: t = client.Task("a = str(id)")
+    
+This ``Task`` would run, and store the string representation of the ``id`` element in ``a`` in each worker's namespace, but it is fairly useless because the user does not know anything about the state of the ``worker`` on which it ran at the time of retrieving results.  It is important that each ``Task`` not expect the state of the ``worker`` to persist after the ``Task`` is completed.  
+There are many different situations for using ``Task`` Farming, and the ``Task`` object has many attributes for use in customizing the ``Task`` behavior.  All of a ``Task``'s attributes may be specified in the constructor, through keyword arguments, or after ``Task`` construction through attribute assignment.
+
+Data Attributes
+***************
+It is likely that the user may want to move data around before or after executing the ``Task``.  We provide methods of sending data to initialize the worker's namespace, and specifying what data to bring back as the ``Task``'s results.
+
+ * pull = []
+    The obvious case is as above, where ``t`` would execute and store the result of ``myfunc`` in ``a``, it is likely that the user would want to bring ``a`` back to their namespace.  This is done through the ``pull`` attribute.  ``pull`` can be a string or list of strings, and it specifies the names of variables to be retrieved.  The ``TaskResult`` object retrieved by ``get_task_result`` will have a dictionary of keys and values, and the ``Task``'s ``pull`` attribute determines what goes into it::
+        
+        In [2]: t = client.Task("a = str(id)", pull = "a")
+        
+        In [3]: t = client.Task("a = str(id)", pull = ["a", "id"])
+        
+ * push = {}
+    A user might also want to initialize some data into the namespace before the code part of the ``Task`` is run.  Enter ``push``.  ``push`` is a dictionary of key/value pairs to be loaded from the user's namespace into the worker's immediately before execution::
+
+        In [4]: t = client.Task("a = f(submitted)", push=dict(submitted=time.time()), pull="a")
+
+push and pull result directly in calling an ``engine``'s ``push`` and ``pull`` methods before and after ``Task`` execution respectively, and thus their api is the same.
+
+Namespace Cleaning
+******************
+When a user is running a large number of Tasks, it is likely that the namespace of the worker's could become cluttered.  Some Tasks might be sensitive to clutter, while others might be known to cause namespace pollution.  For these reasons, Tasks have two boolean attributes for cleaning up the namespace.
+
+ * ``clear_after``
+    if clear_after is specified ``True``, the worker on which the ``Task`` was run will be reset (via ``engine.reset``) upon completion of the ``Task``.  This can be useful for both Tasks that produce clutter or Tasks whose intermediate data one might wish to be kept private::
+    
+        In [5]: t = client.Task("a = range(1e10)", pull = "a",clear_after=True)
+        
+    
+ * ``clear_before``
+    as one might guess, clear_before is identical to ``clear_after``, but it takes place before the ``Task`` is run.  This ensures that the ``Task`` runs on a fresh worker::
+
+        In [6]: t = client.Task("a = globals()", pull = "a",clear_before=True)
+
+Of course, a user can both at the same time, ensuring that all workers are clear except when they are currently running a job.  Both of these default to ``False``.
+
+Fault Tolerance
+***************
+It is possible that Tasks might fail, and there are a variety of reasons this could happen.  One might be that the worker it was running on disconnected, and there was nothing wrong with the ``Task`` itself.  With the fault tolerance attributes of the ``Task``, the user can specify how many times to resubmit the ``Task``, and what to do if it never succeeds.
+
+ * ``retries``
+    ``retries`` is an integer, specifying the number of times a ``Task`` is to be retried.  It defaults to zero.  It is often a good idea for this number to be 1 or 2, to protect the ``Task`` from disconnecting engines, but not a large number.  If a ``Task`` is failing 100 times, there is probably something wrong with the ``Task``.  The canonical bad example:
+    
+        In [7]: t = client.Task("os.kill(os.getpid(), 9)", retries=99)
+        
+    This would actually take down 100 workers.
+    
+ * ``recovery_task``
+    ``recovery_task`` is another ``Task`` object, to be run in the event of the original ``Task`` still failing after running out of retries.  Since ``recovery_task`` is another ``Task`` object, it can have its own ``recovery_task``.  The chain of Tasks is limitless, except loops are not allowed (that would be bad!).
+
+Dependencies
+************
+Dependencies are the most powerful part of the ``Task`` farming system, because it allows the user to do some classification of the workers, and guide the ``Task`` distribution without meddling with the controller directly.  It makes use of two objects - the ``Task``'s ``depend`` attribute, and the engine's ``properties``.  See the `MultiEngine`_ reference for how to use engine properties.  The engine properties api exists for extending IPython, allowing conditional execution and new controllers that make decisions based on properties of its engines.  Currently the ``Task`` dependency is the only internal use of the properties api.
+
+.. _MultiEngine: ./parallel_multiengine
+
+The ``depend`` attribute of a ``Task`` must be a function of exactly one argument, the worker's properties dictionary, and it should return ``True`` if the ``Task`` should be allowed to run on the worker and ``False`` if not.  The usage in the controller is fault tolerant, so exceptions raised by ``Task.depend`` will be ignored and functionally equivalent to always returning ``False``.  Tasks`` with invalid ``depend`` functions will never be assigned to a worker::
+
+    In [8]: def dep(properties):
+        ...     return properties["RAM"] > 2**32 # have at least 4GB
+    In [9]: t = client.Task("a = bigfunc()", depend=dep)
+    
+It is important to note that assignment of values to the properties dict is done entirely by the user, either locally (in the engine) using the EngineAPI, or remotely, through the ``MultiEngineClient``'s get/set_properties methods.
+
+
+
+
+
+    

docs/source/parallel/parallel_intro.txt

@@ -1,8 +1,8 b''
 .. _ip1par:
 
-======================================
-Using IPython for parallel computing
-======================================
+============================
+Overview and getting started
+============================
 
 .. contents::
 

docs/source/parallel/parallel_multiengine.txt

@@ -1,57 +1,115 b''
 .. _parallelmultiengine:
 
-=================================
-IPython's MultiEngine interface
-=================================
+===============================
+IPython's multiengine interface
+===============================
 
 .. contents::
 
-The MultiEngine interface represents one possible way of working with a
-set of IPython engines. The basic idea behind the MultiEngine interface is
-that the capabilities of each engine are explicitly exposed to the user.
-Thus, in the MultiEngine interface, each engine is given an id that is
-used to identify the engine and give it work to do. This interface is very
-intuitive and is designed with interactive usage in mind, and is thus the
-best place for new users of IPython to begin.
+The multiengine interface represents one possible way of working with a set of
+IPython engines. The basic idea behind the multiengine interface is that the
+capabilities of each engine are directly and explicitly exposed to the user.
+Thus, in the multiengine interface, each engine is given an id that is used to
+identify the engine and give it work to do. This interface is very intuitive
+and is designed with interactive usage in mind, and is thus the best place for
+new users of IPython to begin.
 
 Starting the IPython controller and engines
 ===========================================
 
 To follow along with this tutorial, you will need to start the IPython
-controller and four IPython engines. The simplest way of doing this is to
-use the ``ipcluster`` command::
+controller and four IPython engines. The simplest way of doing this is to use
+the :command:`ipcluster` command::
 
 	$ ipcluster -n 4
 	
-For more detailed information about starting the controller and engines, see our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+For more detailed information about starting the controller and engines, see
+our :ref:`introduction <ip1par>` to using IPython for parallel computing.
 
 Creating a ``MultiEngineClient`` instance
 =========================================
 
-The first step is to import the IPython ``client`` module and then create a ``MultiEngineClient`` instance::
+The first step is to import the IPython :mod:`IPython.kernel.client` module
+and then create a :class:`MultiEngineClient` instance::
 
 	In [1]: from IPython.kernel import client
 	
 	In [2]: mec = client.MultiEngineClient()
 
-To make sure there are engines connected to the controller, use can get a list of engine ids::
+This form assumes that the :file:`ipcontroller-mec.furl` is in the
+:file:`~./ipython/security` directory on the client's host. If not, the
+location of the ``.furl`` file must be given as an argument to the
+constructor::
+
+    In[2]: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
+
+To make sure there are engines connected to the controller, use can get a list
+of engine ids::
 
 	In [3]: mec.get_ids()
 	Out[3]: [0, 1, 2, 3]
 
 Here we see that there are four engines ready to do work for us.
 
+Quick and easy parallelism
+==========================
+
+In many cases, you simply want to apply a Python function to a sequence of objects, but *in parallel*.  The multiengine interface provides two simple ways of accomplishing this:  a parallel version of :func:`map` and ``@parallel`` function decorator.
+
+Parallel map
+------------
+
+Python's builtin :func:`map` functions allows a function to be applied to a
+sequence element-by-element. This type of code is typically trivial to
+parallelize. In fact, the multiengine interface in IPython already has a
+parallel version of :meth:`map` that works just like its serial counterpart::
+
+	In [63]: serial_result = map(lambda x:x**10, range(32))
+
+	In [64]: parallel_result = mec.map(lambda x:x**10, range(32))
+
+	In [65]: serial_result==parallel_result
+	Out[65]: True
+
+.. note::
+
+    The multiengine interface version of :meth:`map` does not do any load
+    balancing.  For a load balanced version, see the task interface.
+
+.. seealso::
+
+    The :meth:`map` method has a number of options that can be controlled by
+    the :meth:`mapper` method.  See its docstring for more information.
+
+Parallel function decorator
+---------------------------
+
+Parallel functions are just like normal function, but they can be called on sequences and *in parallel*.  The multiengine interface provides a decorator that turns any Python function into a parallel function::
+
+    In [10]: @mec.parallel()
+       ....: def f(x):
+       ....:     return 10.0*x**4
+       ....: 
+
+    In [11]: f(range(32))    # this is done in parallel
+    Out[11]: 
+    [0.0,10.0,160.0,...]
+
+See the docstring for the :meth:`parallel` decorator for options.
+
 Running Python commands
 =======================
 
-The most basic type of operation that can be performed on the engines is to execute Python code. Executing Python code can be done in blocking or non-blocking mode (blocking is default) using the ``execute`` method.
+The most basic type of operation that can be performed on the engines is to
+execute Python code. Executing Python code can be done in blocking or
+non-blocking mode (blocking is default) using the :meth:`execute` method.
 
 Blocking execution
 ------------------
 
-In blocking mode, the ``MultiEngineClient`` object (called ``mec`` in
+In blocking mode, the :class:`MultiEngineClient` object (called ``mec`` in
 these examples) submits the command to the controller, which places the
-command in the engines' queues for execution. The ``execute`` call then
+command in the engines' queues for execution. The :meth:`execute` call then
 blocks until the engines are done executing the command::
 
 	# The default is to run on all engines
@@ -71,7 +129,8 b' blocks until the engines are done executing the command::'
 	[2] In [2]: b=10
 	[3] In [2]: b=10
 
-Python commands can be executed on specific engines by calling execute using the ``targets`` keyword argument::
+Python commands can be executed on specific engines by calling execute using
+the ``targets`` keyword argument::
 
 	In [6]: mec.execute('c=a+b',targets=[0,2])
 	Out[6]: 
@@ -102,7 +161,9 b' Python commands can be executed on specific engines by calling execute using the'
 	[3] In [4]: print c
 	[3] Out[4]: -5
 
-This example also shows one of the most important things about the IPython engines: they have a persistent user namespaces.  The ``execute`` method returns a Python ``dict`` that contains useful information::
+This example also shows one of the most important things about the IPython
+engines: they have a persistent user namespaces. The :meth:`execute` method
+returns a Python ``dict`` that contains useful information::
 
 	In [9]: result_dict = mec.execute('d=10; print d')
 
@@ -118,10 +179,12 b' This example also shows one of the most important things about the IPython engin'
 Non-blocking execution
 ----------------------
 
-In non-blocking mode, ``execute`` submits the command to be executed and then returns a
-``PendingResult`` object immediately. The ``PendingResult`` object gives you a way of getting a
-result at a later time through its ``get_result`` method or ``r`` attribute. This allows you to
-quickly submit long running commands without blocking your local Python/IPython session::
+In non-blocking mode, :meth:`execute` submits the command to be executed and
+then returns a :class:`PendingResult` object immediately. The
+:class:`PendingResult` object gives you a way of getting a result at a later
+time through its :meth:`get_result` method or :attr:`r` attribute. This allows
+you to quickly submit long running commands without blocking your local
+Python/IPython session::
 
 	# In blocking mode
 	In [6]: mec.execute('import time')
@@ -159,7 +222,10 b' quickly submit long running commands without blocking your local Python/IPython '
 	[2] In [3]: time.sleep(10)
 	[3] In [3]: time.sleep(10)
 
-Often, it is desirable to wait until a set of ``PendingResult`` objects are done.  For this, there is a the method ``barrier``.  This method takes a tuple of ``PendingResult`` objects and blocks until all of the associated results are ready::
+Often, it is desirable to wait until a set of :class:`PendingResult` objects
+are done. For this, there is a the method :meth:`barrier`. This method takes a
+tuple of :class:`PendingResult` objects and blocks until all of the associated
+results are ready::
 
 	In [72]: mec.block=False
 
@@ -182,14 +248,16 b' Often, it is desirable to wait until a set of ``PendingResult`` objects are done'
 The ``block`` and ``targets`` keyword arguments and attributes
 --------------------------------------------------------------
 
-Most commands in the multiengine interface (like ``execute``) accept ``block`` and ``targets``
-as keyword arguments. As we have seen above, these keyword arguments control the blocking mode
-and which engines the command is applied to. The ``MultiEngineClient`` class also has ``block``
-and ``targets`` attributes that control the default behavior when the keyword arguments are not
-provided. Thus the following logic is used for ``block`` and ``targets``:
+Most methods in the multiengine interface (like :meth:`execute`) accept
+``block`` and ``targets`` as keyword arguments. As we have seen above, these
+keyword arguments control the blocking mode and which engines the command is
+applied to. The :class:`MultiEngineClient` class also has :attr:`block` and
+:attr:`targets` attributes that control the default behavior when the keyword
+arguments are not provided. Thus the following logic is used for :attr:`block`
+and :attr:`targets`:
 
-	* If no keyword argument is provided, the instance attributes are used.
-	* Keyword argument, if provided override the instance attributes.
+* If no keyword argument is provided, the instance attributes are used.
+* Keyword argument, if provided override the instance attributes.
 
 The following examples demonstrate how to use the instance attributes::
 
@@ -225,14 +293,19 b' The following examples demonstrate how to use the instance attributes::'
 	[3] In [6]: b=10; print b
 	[3] Out[6]: 10
 
-The ``block`` and ``targets`` instance attributes also determine the behavior of the parallel
-magic commands...
+The :attr:`block` and :attr:`targets` instance attributes also determine the
+behavior of the parallel magic commands.
 
 
 Parallel magic commands
 -----------------------
 
-We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``) that make it more pleasant to execute Python commands on the engines interactively. These are simply shortcuts to ``execute`` and ``get_result``. The ``%px`` magic executes a single Python command on the engines specified by the `magicTargets``targets` attribute of the ``MultiEngineClient`` instance (by default this is 'all')::
+We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
+that make it more pleasant to execute Python commands on the engines
+interactively. These are simply shortcuts to :meth:`execute` and
+:meth:`get_result`. The ``%px`` magic executes a single Python command on the
+engines specified by the :attr:`targets` attribute of the
+:class:`MultiEngineClient` instance (by default this is ``'all'``)::
 
 	# Make this MultiEngineClient active for parallel magic commands
 	In [23]: mec.activate()
@@ -277,7 +350,9 b' We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``) t'
 	[3] In [9]: print numpy.linalg.eigvals(a)
 	[3] Out[9]: [ 0.83664764 -0.25602658]
 
-The ``%result`` magic gets and prints the stdin/stdout/stderr of the last command executed on each engine.  It is simply a shortcut to the ``get_result`` method::
+The ``%result`` magic gets and prints the stdin/stdout/stderr of the last
+command executed on each engine. It is simply a shortcut to the
+:meth:`get_result` method::
 
 	In [29]: %result
 	Out[29]:
@@ -294,7 +369,8 b' The ``%result`` magic gets and prints the stdin/stdout/stderr of the last comman'
 	[3] In [9]: print numpy.linalg.eigvals(a)
 	[3] Out[9]: [ 0.83664764 -0.25602658]
 
-The ``%autopx`` magic switches to a mode where everything you type is executed on the engines given by the ``targets`` attribute::
+The ``%autopx`` magic switches to a mode where everything you type is executed
+on the engines given by the :attr:`targets` attribute::
 
 	In [30]: mec.block=False
 
@@ -335,51 +411,19 b' The ``%autopx`` magic switches to a mode where everything you type is executed o'
 	[3] In [12]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
 	[3] Out[12]: Average max eigenvalue is:  10.1158837784
 
-Using the ``with`` statement of Python 2.5
-------------------------------------------
 
-Python 2.5 introduced the ``with`` statement.  The ``MultiEngineClient`` can be used with the ``with`` statement to execute a block of code on the engines indicated by the ``targets`` attribute::
+Moving Python objects around
+============================
 
-	In [3]: with mec:
-	   ...:     client.remote()    # Required so the following code is not run locally
-	   ...:     a = 10
-	   ...:     b = 30
-	   ...:     c = a+b
-	   ...:     
-	   ...:     
-
-	In [4]: mec.get_result()
-	Out[4]: 
-	<Results List>
-	[0] In [1]: a = 10
-	b = 30
-	c = a+b
-
-	[1] In [1]: a = 10
-	b = 30
-	c = a+b
-
-	[2] In [1]: a = 10
-	b = 30
-	c = a+b
-
-	[3] In [1]: a = 10
-	b = 30
-	c = a+b
-
-This is basically another way of calling execute, but one with allows you to avoid writing code in strings.  When used in this way, the attributes ``targets`` and ``block`` are used to control how the code is executed.  For now, if you run code in non-blocking mode you won't have access to the ``PendingResult``.
-
-Moving Python object around
-===========================
-
-In addition to executing code on engines, you can transfer Python objects to and from your
-IPython session and the engines. In IPython, these operations are called ``push`` (sending an
-object to the engines) and ``pull`` (getting an object from the engines).
+In addition to executing code on engines, you can transfer Python objects to
+and from your IPython session and the engines. In IPython, these operations
+are called :meth:`push` (sending an object to the engines) and :meth:`pull`
+(getting an object from the engines).
 
 Basic push and pull
 -------------------
 
-Here are some examples of how you use ``push`` and ``pull``::
+Here are some examples of how you use :meth:`push` and :meth:`pull`::
 
 	In [38]: mec.push(dict(a=1.03234,b=3453))
 	Out[38]: [None, None, None, None]
@@ -415,7 +459,8 b' Here are some examples of how you use ``push`` and ``pull``::'
 	[3] In [13]: print c
 	[3] Out[13]: speed
 
-In non-blocking mode ``push`` and ``pull`` also return ``PendingResult`` objects::
+In non-blocking mode :meth:`push` and :meth:`pull` also return
+:class:`PendingResult` objects::
 
 	In [47]: mec.block=False
 
@@ -428,7 +473,11 b' In non-blocking mode ``push`` and ``pull`` also return ``PendingResult`` objects'
 Push and pull for functions
 ---------------------------
 
-Functions can also be pushed and pulled using ``push_function`` and ``pull_function``::
+Functions can also be pushed and pulled using :meth:`push_function` and
+:meth:`pull_function`::
+
+
+    In [52]: mec.block=True
 
 	In [53]: def f(x):
 	   ....:     return 2.0*x**4
@@ -466,7 +515,10 b' Functions can also be pushed and pulled using ``push_function`` and ``pull_funct'
 Dictionary interface
 --------------------
 
-As a shorthand to ``push`` and ``pull``, the ``MultiEngineClient`` class implements some of the Python dictionary interface. This make the remote namespaces of the engines appear as a local dictionary. Underneath, this uses ``push`` and ``pull``::
+As a shorthand to :meth:`push` and :meth:`pull`, the
+:class:`MultiEngineClient` class implements some of the Python dictionary
+interface. This make the remote namespaces of the engines appear as a local
+dictionary. Underneath, this uses :meth:`push` and :meth:`pull`::
 
 	In [50]: mec.block=True
 
@@ -478,11 +530,13 b' As a shorthand to ``push`` and ``pull``, the ``MultiEngineClient`` class impleme'
 Scatter and gather
 ------------------
 
-Sometimes it is useful to partition a sequence and push the partitions to different engines. In
-MPI language, this is know as scatter/gather and we follow that terminology. However, it is
-important to remember that in IPython ``scatter`` is from the interactive IPython session to
-the engines and ``gather`` is from the engines back to the interactive IPython session. For
-scatter/gather operations between engines, MPI should be used::
+Sometimes it is useful to partition a sequence and push the partitions to
+different engines. In MPI language, this is know as scatter/gather and we
+follow that terminology. However, it is important to remember that in
+IPython's :class:`MultiEngineClient` class, :meth:`scatter` is from the
+interactive IPython session to the engines and :meth:`gather` is from the
+engines back to the interactive IPython session. For scatter/gather operations
+between engines, MPI should be used::
 
 	In [58]: mec.scatter('a',range(16))
 	Out[58]: [None, None, None, None]
@@ -510,24 +564,12 b' scatter/gather operations between engines, MPI should be used::'
 Other things to look at
 =======================
 
-Parallel map
-------------
-
-Python's builtin ``map`` functions allows a function to be applied to a sequence element-by-element.  This type of code is typically trivial to parallelize.  In fact, the MultiEngine interface in IPython already has a parallel version of ``map`` that works just like its serial counterpart::
-
-	In [63]: serial_result = map(lambda x:x**10, range(32))
-
-	In [64]: parallel_result = mec.map(lambda x:x**10, range(32))
-
-	In [65]: serial_result==parallel_result
-	Out[65]: True
-
-As you would expect, the parallel version of ``map`` is also influenced by the ``block`` and ``targets`` keyword arguments and attributes.
-
 How to do parallel list comprehensions
 --------------------------------------
 
-In many cases list comprehensions are nicer than using the map function.  While we don't have fully parallel list comprehensions, it is simple to get the basic effect using ``scatter`` and ``gather``::
+In many cases list comprehensions are nicer than using the map function. While
+we don't have fully parallel list comprehensions, it is simple to get the
+basic effect using :meth:`scatter` and :meth:`gather`::
 
 	In [66]: mec.scatter('x',range(64))
 	Out[66]: [None, None, None, None]
@@ -547,10 +589,16 b' In many cases list comprehensions are nicer than using the map function. While '
 	In [69]: print y
 	[0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
 
-Parallel Exceptions
+Parallel exceptions
 -------------------
 
-In the MultiEngine interface, parallel commands can raise Python exceptions, just like serial commands.  But, it is a little subtle, because a single parallel command can actually raise multiple exceptions (one for each engine the command was run on).  To express this idea, the MultiEngine interface has a ``CompositeError`` exception class that will be raised in most cases.  The ``CompositeError`` class is a special type of exception that wraps one or more other types of exceptions.  Here is how it works::
+In the multiengine interface, parallel commands can raise Python exceptions,
+just like serial commands. But, it is a little subtle, because a single
+parallel command can actually raise multiple exceptions (one for each engine
+the command was run on). To express this idea, the MultiEngine interface has a
+:exc:`CompositeError` exception class that will be raised in most cases. The
+:exc:`CompositeError` class is a special type of exception that wraps one or
+more other types of exceptions. Here is how it works::
 
 	In [76]: mec.block=True
 
@@ -580,7 +628,7 b' In the MultiEngine interface, parallel commands can raise Python exceptions, jus'
 	[2:execute]: ZeroDivisionError: integer division or modulo by zero
 	[3:execute]: ZeroDivisionError: integer division or modulo by zero
 
-Notice how the error message printed when ``CompositeError`` is raised has information about the individual exceptions that were raised on each engine.  If you want, you can even raise one of these original exceptions::
+Notice how the error message printed when :exc:`CompositeError` is raised has information about the individual exceptions that were raised on each engine.  If you want, you can even raise one of these original exceptions::
 
 	In [80]: try:
 	   ....:     mec.execute('1/0')
@@ -602,7 +650,9 b' Notice how the error message printed when ``CompositeError`` is raised has infor'
 
 	ZeroDivisionError: integer division or modulo by zero
 
-If you are working in IPython, you can simple type ``%debug`` after one of these ``CompositeError`` is raised, and inspect the exception instance::
+If you are working in IPython, you can simple type ``%debug`` after one of
+these :exc:`CompositeError` exceptions is raised, and inspect the exception
+instance::
 
 	In [81]: mec.execute('1/0')
 	---------------------------------------------------------------------------
@@ -679,6 +729,11 b' If you are working in IPython, you can simple type ``%debug`` after one of these'
 
 	ZeroDivisionError: integer division or modulo by zero
 
+.. note::
+
+    The above example appears to be broken right now because of a change in 
+    how we are using Twisted.
+
 All of this same error handling magic even works in non-blocking mode::
 
 	In [83]: mec.block=False

docs/source/parallel/parallel_task.txt

@@ -1,240 +1,93 b''
 .. _paralleltask:
 
-=================================
-The IPython Task interface
-=================================
+==========================
+The IPython task interface
+==========================
 
 .. contents::
 
-The ``Task`` interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the ``MultiEngine`` interface, in the ``Task`` interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful. Best of all the user can use both of these interfaces at the same time to take advantage or both of their strengths.  When the user can break up the user's work into segments that do not depend on previous execution, the ``Task`` interface is ideal.  But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign Tasks to engines explicitly.
+The task interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the multiengine interface, in the task interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful.
+
+Best of all the user can use both of these interfaces running at the same time to take advantage or both of their strengths.  When the user can break up the user's work into segments that do not depend on previous execution, the task interface is ideal.  But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign tasks to engines explicitly.
 
 Starting the IPython controller and engines
 ===========================================
 
-To follow along with this tutorial, the user will need to start the IPython
-controller and four IPython engines. The simplest way of doing this is to
-use the ``ipcluster`` command::
+To follow along with this tutorial, you will need to start the IPython
+controller and four IPython engines. The simplest way of doing this is to use
+the :command:`ipcluster` command::
 
 	$ ipcluster -n 4
 	
-For more detailed information about starting the controller and engines, see our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+For more detailed information about starting the controller and engines, see
+our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+
+Creating a ``TaskClient`` instance
+=========================================
+
+The first step is to import the IPython :mod:`IPython.kernel.client` module
+and then create a :class:`TaskClient` instance::
+
+	In [1]: from IPython.kernel import client
+	
+	In [2]: tc = client.TaskClient()
+
+This form assumes that the :file:`ipcontroller-tc.furl` is in the
+:file:`~./ipython/security` directory on the client's host. If not, the
+location of the ``.furl`` file must be given as an argument to the
+constructor::
+
+    In[2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
+
+Quick and easy parallelism
+==========================
+
+In many cases, you simply want to apply a Python function to a sequence of objects, but *in parallel*.  Like the multiengine interface, the task interface provides two simple ways of accomplishing this:  a parallel version of :func:`map` and ``@parallel`` function decorator.  However, the verions in the task interface have one important difference:  they are dynamically load balanced.  Thus, if the execution time per item varies significantly, you should use the versions in the task interface.
+
+Parallel map
+------------
+
+The parallel :meth:`map` in the task interface is similar to that in the multiengine interface::
+
+	In [63]: serial_result = map(lambda x:x**10, range(32))
+
+	In [64]: parallel_result = tc.map(lambda x:x**10, range(32))
+
+	In [65]: serial_result==parallel_result
+	Out[65]: True
+
+Parallel function decorator
+---------------------------
+
+Parallel functions are just like normal function, but they can be called on sequences and *in parallel*.  The multiengine interface provides a decorator that turns any Python function into a parallel function::
 
-The magic here is that this single controller and set of engines is running both the MultiEngine and ``Task`` interfaces simultaneously.
+    In [10]: @tc.parallel()
+       ....: def f(x):
+       ....:     return 10.0*x**4
+       ....: 
 
-QuickStart Task Farming
-=======================
+    In [11]: f(range(32))    # this is done in parallel
+    Out[11]: 
+    [0.0,10.0,160.0,...]
 
-First, a quick example of how to start running the most basic Tasks.
-The first step is to import the IPython ``client`` module and then create a ``TaskClient`` instance::
-    
-    In [1]: from IPython.kernel import client
-    
-    In [2]: tc = client.TaskClient()
+More details
+============
 
-Then the user wrap the commands the user want to run in Tasks::
+The :class:`TaskClient` has many more powerful features that allow quite a bit of flexibility in how tasks are defined and run.  The next places to look are in the following classes:
 
-    In [3]: tasklist = []
-    In [4]: for n in range(1000):
-        ...     tasklist.append(client.Task("a = %i"%n, pull="a"))
+* :class:`IPython.kernel.client.TaskClient`
+* :class:`IPython.kernel.client.StringTask`
+* :class:`IPython.kernel.client.MapTask`
 
-The first argument of the ``Task`` constructor is a string, the command to be executed.  The most important optional keyword argument is ``pull``, which can be a string or list of strings, and it specifies the variable names to be saved as results of the ``Task``.
+The following is an overview of how to use these classes together:
 
-Next, the user need to submit the Tasks to the ``TaskController`` with the ``TaskClient``::
+1. Create a :class:`TaskClient`.
+2. Create one or more instances of :class:`StringTask` or :class:`MapTask`
+   to define your tasks.
+3. Submit your tasks to using the :meth:`run` method of your
+   :class:`TaskClient` instance.
+4. Use :meth:`TaskClient.get_task_result` to get the results of the
+   tasks.
 
-    In [5]: taskids = [ tc.run(t) for t in tasklist ]
+We are in the process of developing more detailed information about the task interface.  For now, the docstrings of the :class:`TaskClient`, :class:`StringTask` and :class:`MapTask` classes should be consulted.
 
-This will give the user a list of the TaskIDs used by the controller to keep track of the Tasks and their results.  Now at some point the user are going to want to get those results back.  The ``barrier`` method allows the user to wait for the Tasks to finish running::
-
-    In [6]: tc.barrier(taskids)
-
-This command will block until all the Tasks in ``taskids`` have finished.  Now, the user probably want to look at the user's results::
-
-    In [7]: task_results = [ tc.get_task_result(taskid) for taskid in taskids ]
-
-Now the user have a list of ``TaskResult`` objects, which have the actual result as a dictionary, but also keep track of some useful metadata about the ``Task``::
-
-    In [8]: tr = ``Task``_results[73]
-    
-    In [9]: tr
-    Out[9]: ``TaskResult``[ID:73]:{'a':73}
-    
-    In [10]: tr.engineid
-    Out[10]: 1
-    
-    In [11]: tr.submitted, tr.completed, tr.duration
-    Out[11]: ("2008/03/08 03:41:42", "2008/03/08 03:41:44", 2.12345)
-
-The actual results are stored in a dictionary, ``tr.results``, and a namespace object ``tr.ns`` which accesses the result keys by attribute::
-
-    In [12]: tr.results['a']
-    Out[12]: 73
-    
-    In [13]: tr.ns.a
-    Out[13]: 73
-
-That should cover the basics of running simple Tasks.  There are several more powerful things the user can do with Tasks covered later.  The most useful probably being using a ``MutiEngineClient`` interface to initialize all the engines with the import dependencies necessary to run the user's Tasks.
-
-There are many options for running and managing Tasks. The best way to learn further about the ``Task`` interface is to study the examples in ``docs/examples``. If the user do so and learn a lots about this interface, we encourage the user to expand this documentation about the ``Task`` system.
-
-Overview of the Task System
-===========================
-
-The user's view of the ``Task`` system has three basic objects:  The ``TaskClient``, the ``Task``, and the ``TaskResult``.  The names of these three objects well indicate their role.
-
-The ``TaskClient`` is the user's ``Task`` farming connection to the IPython cluster.  Unlike the ``MultiEngineClient``, the ``TaskControler`` handles all the scheduling and distribution of work, so the ``TaskClient`` has no notion of engines, it just submits Tasks and requests their results.  The Tasks are described as ``Task`` objects, and their results are wrapped in ``TaskResult`` objects.  Thus, there are very few necessary methods for the user to manage.
-
-Inside the task system is a Scheduler object, which assigns tasks to workers.  The default scheduler is a simple FIFO queue.  Subclassing the Scheduler should be easy, just implementing your own priority system.
-
-The TaskClient
-==============
-
-The ``TaskClient`` is the object the user use to connect to the ``Controller`` that is managing the user's Tasks.  It is the analog of the ``MultiEngineClient`` for the standard IPython multiplexing interface.  As with all client interfaces, the first step is to import the IPython Client Module::
-
-    In [1]: from IPython.kernel import client
-
-Just as with the ``MultiEngineClient``, the user create the ``TaskClient`` with a tuple, containing the ip-address and port of the ``Controller``.  the ``client`` module conveniently has the default address of the ``Task`` interface of the controller.  Creating a default ``TaskClient`` object would be done with this::
-
-    In [2]: tc = client.TaskClient(client.default_task_address)
-
-or, if the user want to specify a non default location of the ``Controller``, the user can specify explicitly::
-
-    In [3]: tc = client.TaskClient(("192.168.1.1", 10113))
-
-As discussed earlier, the ``TaskClient`` only has a few basic methods.
-
- * ``tc.run(task)``
-    ``run`` is the method by which the user submits Tasks.  It takes exactly one argument, a ``Task`` object.  All the advanced control of ``Task`` behavior is handled by properties of the ``Task`` object, rather than the submission command, so they will be discussed later in the `Task`_ section.  ``run`` returns an integer, the ``Task``ID by which the ``Task`` and its results can be tracked and retrieved::
-    
-        In [4]: ``Task``ID = tc.run(``Task``)
-    
- * ``tc.get_task_result(taskid, block=``False``)``
-    ``get_task_result`` is the method by which results are retrieved.  It takes a single integer argument, the ``Task``ID`` of the result the user wish to retrieve.  ``get_task_result`` also takes a keyword argument ``block``.  ``block`` specifies whether the user actually want to wait for the result.  If ``block`` is false, as it is by default, ``get_task_result`` will return immediately.  If the ``Task`` has completed, it will return the ``TaskResult`` object for that ``Task``.  But if the ``Task`` has not completed, it will return ``None``.  If the user specify ``block=``True``, then ``get_task_result`` will wait for the ``Task`` to complete, and always return the ``TaskResult`` for the requested ``Task``.
- * ``tc.barrier(taskid(s))``
-    ``barrier`` is a synchronization method.  It takes exactly one argument, a ``Task``ID or list of taskIDs.  ``barrier`` will block until all the specified Tasks have completed.  In practice, a barrier is often called between the ``Task`` submission section of the code and the result gathering section::
-        
-        In [5]: taskIDs = [ tc.run(``Task``) for ``Task`` in myTasks ]
-        
-        In [6]: tc.get_task_result(taskIDs[-1]) is None
-        Out[6]: ``True``
-        
-        In [7]: tc.barrier(``Task``ID)
-        
-        In [8]: results = [ tc.get_task_result(tid) for tid in taskIDs ]
-        
- * ``tc.queue_status(verbose=``False``)``
-    ``queue_status`` is a method for querying the state of the ``TaskControler``.  ``queue_status`` returns a dict of the form::
-        
-        {'scheduled': Tasks that have been submitted but yet run
-         'pending'  : Tasks that are currently running
-         'succeeded': Tasks that have completed successfully
-         'failed'   : Tasks that have finished with a failure
-        }
-    
-    if @verbose is not specified (or is ``False``), then the values of the dict are integers - the number of Tasks in each state.  if @verbose is ``True``, then each element in the dict is a list of the taskIDs in that state::
-    
-        In [8]: tc.queue_status()
-        Out[8]: {'scheduled': 4,
-                'pending'  : 2,
-                'succeeded': 5,
-                'failed'   : 1
-                }
-        
-        In [9]: tc.queue_status(verbose=True)
-        Out[9]: {'scheduled': [8,9,10,11],
-                'pending'  : [6,7],
-                'succeeded': [0,1,2,4,5],
-                'failed'   : [3]
-                }
-        
- * ``tc.abort(taskid)``
-    ``abort`` allows the user to abort Tasks that have already been submitted.  ``abort`` will always return immediately.  If the ``Task`` has completed, ``abort`` will raise an ``IndexError ``Task`` Already Completed``.  An obvious case for ``abort`` would be where the user submits a long-running ``Task`` with a number of retries (see ``Task``_ section for how to specify retries) in an interactive session, but realizes there has been a typo.  The user can then abort the ``Task``, preventing certain failures from cluttering up the queue.  It can also be used for parallel search-type problems, where only one ``Task`` will give the solution, so once the user find the solution, the user would want to abort all remaining Tasks to prevent wasted work.
- * ``tc.spin()``
-    ``spin`` simply triggers the scheduler in the ``TaskControler``.  Under most normal circumstances, this will do nothing.  The primary known usage case involves the ``Task`` dependency (see `Dependencies`_).  The dependency is a function of an Engine's ``properties``, but changing the ``properties`` via the ``MutliEngineClient`` does not trigger a reschedule event.  The main example case for this requires the following event sequence:
-     * ``engine`` is available, ``Task`` is submitted, but ``engine`` does not have ``Task``'s dependencies.
-     * ``engine`` gets necessary dependencies while no new Tasks are submitted or completed.
-     * now ``engine`` can run ``Task``, but a ``Task`` event is required for the ``TaskControler`` to try scheduling ``Task`` again.
-     
-    ``spin`` is just an empty ping method to ensure that the Controller has scheduled all available Tasks, and should not be needed under most normal circumstances.
-
-That covers the ``TaskClient``, a simple interface to the cluster.  With this, the user can submit jobs (and abort if necessary), request their results, synchronize on arbitrary subsets of jobs.
-
-.. _task: The Task Object
-
-The Task Object
-===============
-
-The ``Task`` is the basic object for describing a job.  It can be used in a very simple manner, where the user just specifies a command string to be executed as the ``Task``.  The usage of this first argument is exactly the same as the ``execute`` method of the ``MultiEngine`` (in fact, ``execute`` is called to run the code)::
-
-    In [1]: t = client.Task("a = str(id)")
-    
-This ``Task`` would run, and store the string representation of the ``id`` element in ``a`` in each worker's namespace, but it is fairly useless because the user does not know anything about the state of the ``worker`` on which it ran at the time of retrieving results.  It is important that each ``Task`` not expect the state of the ``worker`` to persist after the ``Task`` is completed.  
-There are many different situations for using ``Task`` Farming, and the ``Task`` object has many attributes for use in customizing the ``Task`` behavior.  All of a ``Task``'s attributes may be specified in the constructor, through keyword arguments, or after ``Task`` construction through attribute assignment.
-
-Data Attributes
-***************
-It is likely that the user may want to move data around before or after executing the ``Task``.  We provide methods of sending data to initialize the worker's namespace, and specifying what data to bring back as the ``Task``'s results.
-
- * pull = []
-    The obvious case is as above, where ``t`` would execute and store the result of ``myfunc`` in ``a``, it is likely that the user would want to bring ``a`` back to their namespace.  This is done through the ``pull`` attribute.  ``pull`` can be a string or list of strings, and it specifies the names of variables to be retrieved.  The ``TaskResult`` object retrieved by ``get_task_result`` will have a dictionary of keys and values, and the ``Task``'s ``pull`` attribute determines what goes into it::
-        
-        In [2]: t = client.Task("a = str(id)", pull = "a")
-        
-        In [3]: t = client.Task("a = str(id)", pull = ["a", "id"])
-        
- * push = {}
-    A user might also want to initialize some data into the namespace before the code part of the ``Task`` is run.  Enter ``push``.  ``push`` is a dictionary of key/value pairs to be loaded from the user's namespace into the worker's immediately before execution::
-
-        In [4]: t = client.Task("a = f(submitted)", push=dict(submitted=time.time()), pull="a")
-
-push and pull result directly in calling an ``engine``'s ``push`` and ``pull`` methods before and after ``Task`` execution respectively, and thus their api is the same.
-
-Namespace Cleaning
-******************
-When a user is running a large number of Tasks, it is likely that the namespace of the worker's could become cluttered.  Some Tasks might be sensitive to clutter, while others might be known to cause namespace pollution.  For these reasons, Tasks have two boolean attributes for cleaning up the namespace.
-
- * ``clear_after``
-    if clear_after is specified ``True``, the worker on which the ``Task`` was run will be reset (via ``engine.reset``) upon completion of the ``Task``.  This can be useful for both Tasks that produce clutter or Tasks whose intermediate data one might wish to be kept private::
-    
-        In [5]: t = client.Task("a = range(1e10)", pull = "a",clear_after=True)
-        
-    
- * ``clear_before``
-    as one might guess, clear_before is identical to ``clear_after``, but it takes place before the ``Task`` is run.  This ensures that the ``Task`` runs on a fresh worker::
-
-        In [6]: t = client.Task("a = globals()", pull = "a",clear_before=True)
-
-Of course, a user can both at the same time, ensuring that all workers are clear except when they are currently running a job.  Both of these default to ``False``.
-
-Fault Tolerance
-***************
-It is possible that Tasks might fail, and there are a variety of reasons this could happen.  One might be that the worker it was running on disconnected, and there was nothing wrong with the ``Task`` itself.  With the fault tolerance attributes of the ``Task``, the user can specify how many times to resubmit the ``Task``, and what to do if it never succeeds.
-
- * ``retries``
-    ``retries`` is an integer, specifying the number of times a ``Task`` is to be retried.  It defaults to zero.  It is often a good idea for this number to be 1 or 2, to protect the ``Task`` from disconnecting engines, but not a large number.  If a ``Task`` is failing 100 times, there is probably something wrong with the ``Task``.  The canonical bad example:
-    
-        In [7]: t = client.Task("os.kill(os.getpid(), 9)", retries=99)
-        
-    This would actually take down 100 workers.
-    
- * ``recovery_task``
-    ``recovery_task`` is another ``Task`` object, to be run in the event of the original ``Task`` still failing after running out of retries.  Since ``recovery_task`` is another ``Task`` object, it can have its own ``recovery_task``.  The chain of Tasks is limitless, except loops are not allowed (that would be bad!).
-
-Dependencies
-************
-Dependencies are the most powerful part of the ``Task`` farming system, because it allows the user to do some classification of the workers, and guide the ``Task`` distribution without meddling with the controller directly.  It makes use of two objects - the ``Task``'s ``depend`` attribute, and the engine's ``properties``.  See the `MultiEngine`_ reference for how to use engine properties.  The engine properties api exists for extending IPython, allowing conditional execution and new controllers that make decisions based on properties of its engines.  Currently the ``Task`` dependency is the only internal use of the properties api.
-
-.. _MultiEngine: ./parallel_multiengine
-
-The ``depend`` attribute of a ``Task`` must be a function of exactly one argument, the worker's properties dictionary, and it should return ``True`` if the ``Task`` should be allowed to run on the worker and ``False`` if not.  The usage in the controller is fault tolerant, so exceptions raised by ``Task.depend`` will be ignored and functionally equivalent to always returning ``False``.  Tasks`` with invalid ``depend`` functions will never be assigned to a worker::
-
-    In [8]: def dep(properties):
-        ...     return properties["RAM"] > 2**32 # have at least 4GB
-    In [9]: t = client.Task("a = bigfunc()", depend=dep)
-    
-It is important to note that assignment of values to the properties dict is done entirely by the user, either locally (in the engine) using the EngineAPI, or remotely, through the ``MultiEngineClient``'s get/set_properties methods.
-
-
-
-
-
-