Updated the docs for using MPI with IPython.

docs/source/parallel/index.txt

@@ -1,15 +1,16
 .. _parallel_index:
 
 ====================================
 Using IPython for parallel computing
 ====================================
 
 .. toctree::
    :maxdepth: 2
 
    parallel_intro.txt
+   parallel_process.txt
    parallel_multiengine.txt
    parallel_task.txt
    parallel_mpi.txt
    parallel_security.txt
 

docs/source/parallel/parallel_intro.txt

@@ -1,330 +1,204
 .. _ip1par:
 
 ============================
 Overview and getting started
 ============================
 
 .. contents::
 
 Introduction
 ============
 
 This file gives an overview of IPython's sophisticated and
 powerful architecture for parallel and distributed computing. This
 architecture abstracts out parallelism in a very general way, which
 enables IPython to support many different styles of parallelism
 including:
 
 * Single program, multiple data (SPMD) parallelism.
 * Multiple program, multiple data (MPMD) parallelism.
 * Message passing using ``MPI``.
 * Task farming.
 * Data parallel.
 * Combinations of these approaches.
 * Custom user defined approaches.
 
 Most importantly, IPython enables all types of parallel applications to
 be developed, executed, debugged and monitored *interactively*. Hence,
 the ``I`` in IPython.  The following are some example usage cases for IPython:
 
 * Quickly parallelize algorithms that are embarrassingly parallel
   using a number of simple approaches.  Many simple things can be
   parallelized interactively in one or two lines of code.
 
 * Steer traditional MPI applications on a supercomputer from an
   IPython session on your laptop.
 
 * Analyze and visualize large datasets (that could be remote and/or
   distributed) interactively using IPython and tools like
   matplotlib/TVTK.
 
 * Develop, test and debug new parallel algorithms 
   (that may use MPI) interactively.
 
 * Tie together multiple MPI jobs running on different systems into
   one giant distributed and parallel system.
 
 * Start a parallel job on your cluster and then have a remote 	
   collaborator connect to it and pull back data into their 
   local IPython session for plotting and analysis.
 
 * Run a set of tasks on a set of CPUs using dynamic load balancing.
 
 Architecture overview
 =====================
 
 The IPython architecture consists of three components:
 
 * The IPython engine.
 * The IPython controller.
 * Various controller clients.
 
 These components live in the :mod:`IPython.kernel` package and are
 installed with IPython.  They do, however, have additional dependencies
 that must be installed.  For more information, see our
 :ref:`installation documentation <install_index>`.
 
 IPython engine
 ---------------
 
 The IPython engine is a Python instance that takes Python commands over a
 network connection. Eventually, the IPython engine will be a full IPython
 interpreter, but for now, it is a regular Python interpreter. The engine
 can also handle incoming and outgoing Python objects sent over a network
 connection.  When multiple engines are started, parallel and distributed
 computing becomes possible. An important feature of an IPython engine is
 that it blocks while user code is being executed. Read on for how the
 IPython controller solves this problem to expose a clean asynchronous API
 to the user.
 
 IPython controller
 ------------------
 
 The IPython controller provides an interface for working with a set of
 engines. At an general level, the controller is a process to which
 IPython engines can connect. For each connected engine, the controller
 manages a queue. All actions that can be performed on the engine go
 through this queue. While the engines themselves block when user code is
 run, the controller hides that from the user to provide a fully
 asynchronous interface to a set of engines. 
 
 .. note::
 
     Because the controller listens on a network port for engines to 
     connect to it, it must be started *before* any engines are started.  
 
 The controller also provides a single point of contact for users who wish
 to utilize the engines connected to the controller. There are different
 ways of working with a controller. In IPython these ways correspond to different interfaces that the controller is adapted to.  Currently we have two default interfaces to the controller: 
 
 * The MultiEngine interface, which provides the simplest possible way of working
   with engines interactively.
 * The Task interface, which provides presents the engines as a load balanced
   task farming system.
 
 Advanced users can easily add new custom interfaces to enable other
 styles of parallelism. 
 
 .. note:: 
 
 	A single controller and set of engines can be accessed 
 	through multiple interfaces simultaneously.  This opens the
 	door for lots of interesting things.  
 
 Controller clients
 ------------------
 
 For each controller interface, there is a corresponding client. These
 clients allow users to interact with a set of engines through the
 interface.  Here are the two default clients:
 
 * The :class:`MultiEngineClient` class.
 * The :class:`TaskClient` class.
 
 Security
 --------
 
 By default (as long as `pyOpenSSL` is installed) all network connections between the controller and engines and the controller and clients are secure.  What does this mean?  First of all, all of the connections will be encrypted using SSL.  Second, the connections are authenticated.  We handle authentication in a `capabilities`__ based security model.  In this model,  a "capability (known in some systems as a key) is a communicable, unforgeable token of authority".  Put simply, a capability is like a key to your house.  If you have the key to your house, you can get in.  If not, you can't.  
 
 .. __: http://en.wikipedia.org/wiki/Capability-based_security
 
 In our architecture, the controller is the only process that listens on network ports, and is thus responsible to creating these keys.  In IPython, these keys are known as Foolscap URLs, or FURLs, because of the underlying network protocol we are using.  As a user, you don't need to know anything about the details of these FURLs, other than that when the controller starts, it saves a set of FURLs to files named :file:`something.furl`.  The default location of these files is the :file:`~./ipython/security` directory.
 
 To connect and authenticate to the controller an engine or client simply needs to present an appropriate furl (that was originally created by the controller) to the controller.  Thus, the .furl files need to be copied to a location where the clients and engines can find them.  Typically, this is the :file:`~./ipython/security` directory on the host where the client/engine is running (which could be a different host than the controller).  Once the .furl files are copied over, everything should work fine.
 
 Currently, there are three .furl files that the controller creates:
 
 ipcontroller-engine.furl
     This ``.furl`` file is the key that gives an engine the ability to connect
     to a controller.
 
 ipcontroller-tc.furl
     This ``.furl`` file is the key that a :class:`TaskClient` must use to
     connect to the task interface of a controller.
     
 ipcontroller-mec.furl
     This ``.furl`` file is the key that a :class:`MultiEngineClient` must use to
     connect to the multiengine interface of a controller.
 
 More details of how these ``.furl`` files are used are given below.
 
 A detailed description of the security model and its implementation in IPython
 can be found :ref:`here <parallelsecurity>`.
 
 Getting Started
 ===============
 
 To use IPython for parallel computing, you need to start one instance of
-the controller and one or more instances of the engine. The controller
-and each engine can run on different machines or on the same machine.
-Because of this, there are many different possibilities for setting up
-the IP addresses and ports used by the various processes.
-
-Starting the controller and engine on your local machine
---------------------------------------------------------
-
-This is the simplest configuration that can be used and is useful for
-testing the system and on machines that have multiple cores and/or
-multple CPUs. The easiest way of getting started is to use the :command:`ipcluster`
-command::
+the controller and one or more instances of the engine. Initially, it is best to simply start a controller and engines on a single host using the :command:`ipcluster` command.  To start a controller and 4 engines on you localhost, just do::
 
 	$ ipcluster -n 4
 
-This will start an IPython controller and then 4 engines that connect to
-the controller. Lastly, the script will print out the Python commands
-that you can use to connect to the controller. It is that easy.
-
-.. warning::
-
-    The :command:`ipcluster` does not currently work on Windows.  We are
-    working on it though.
-
-Underneath the hood, the controller creates ``.furl`` files in the
-:file:`~./ipython/security` directory.  Because the engines are on the
-same host, they automatically find the needed :file:`ipcontroller-engine.furl`
-there and use it to connect to the controller.
-
-The :command:`ipcluster` script uses two other top-level
-scripts that you can also use yourself. These scripts are
-:command:`ipcontroller`, which starts the controller and :command:`ipengine` which
-starts one engine. To use these scripts to start things on your local
-machine, do the following.
-
-First start the controller::
-
-	$ ipcontroller
-	
-Next, start however many instances of the engine you want using (repeatedly) the command::
-
-	$ ipengine
-
-The engines should start and automatically connect to the controller using the ``.furl`` files in :file:`~./ipython/security`. You are now ready to use the controller and engines from IPython.
-
-.. warning:: 
-	
-	The order of the above operations is very important.  You *must*
- 	start the controller before the engines, since the engines connect 
-	to the controller as they get started.
-
-.. note::
-
-    On some platforms (OS X), to put the controller and engine into the background
-    you may need to give these commands in the form ``(ipcontroller &)`` 
-    and ``(ipengine &)`` (with the parentheses) for them to work properly. 
-
-
-Starting the controller and engines on different hosts
-------------------------------------------------------
-
-When the controller and engines are running on different hosts, things are
-slightly more complicated, but the underlying ideas are the same:
-
-1. Start the controller on a host using :command:`ipcontroler`.
-2. Copy :file:`ipcontroller-engine.furl` from :file:`~./ipython/security` on the controller's host to the host where the engines will run.
-3. Use :command:`ipengine` on the engine's hosts to start the engines.
-
-The only thing you have to be careful of is to tell :command:`ipengine` where the :file:`ipcontroller-engine.furl` file is located.  There are two ways you can do this:
-
-* Put :file:`ipcontroller-engine.furl` in the :file:`~./ipython/security` directory
-  on the engine's host, where it will be found automatically.
-* Call :command:`ipengine` with the ``--furl-file=full_path_to_the_file`` flag.
-
-The ``--furl-file`` flag works like this::
-
-    $ ipengine --furl-file=/path/to/my/ipcontroller-engine.furl
-
-.. note::
-    
-    If the controller's and engine's hosts all have a shared file system  
-    (:file:`~./ipython/security` is the same on all of them), then things
-    will just work!
-
-Make .furl files persistent
----------------------------
-
-At fist glance it may seem that that managing the ``.furl`` files is a bit annoying.  Going back to the house and key analogy, copying the ``.furl`` around each time you start the controller is like having to make a new key everytime you want to unlock the door and enter your house.  As with your house, you want to be able to create the key (or ``.furl`` file) once, and then simply use it at any point in the future.
-
-This is possible.  The only thing you have to do is decide what ports the controller will listen on for the engines and clients.  This is done as follows::
-
-    $ ipcontroller --client-port=10101 --engine-port=10102
-
-Then, just copy the furl files over the first time and you are set.  You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to use the *same* ports.
-
-.. note::
-
-    You may ask the question: what ports does the controller listen on if you
-    don't tell is to use specific ones?  The default is to use high random port
-    numbers.  We do this for two reasons: i) to increase security through obcurity
-    and ii) to multiple controllers on a given host to start and automatically
-    use different ports.
-
-Starting engines using ``mpirun``
----------------------------------
-
-The IPython engines can be started using ``mpirun``/``mpiexec``, even if
-the engines don't call ``MPI_Init()`` or use the MPI API in any way. This is
-supported on modern MPI implementations like `Open MPI`_.. This provides
-an really nice way of starting a bunch of engine. On a system with MPI
-installed you can do::
-
-	mpirun -n 4 ipengine
-
-to start 4 engine on a cluster.  This works even if you don't have any
-Python-MPI bindings installed.
-
-.. _Open MPI: http://www.open-mpi.org/
-
-More details on using MPI with IPython can be found :ref:`here <parallelmpi>`.
-
-Log files
----------
-
-All of the components of IPython have log files associated with them.
-These log files can be extremely useful in debugging problems with
-IPython and can be found in the directory ``~/.ipython/log``.  Sending
-the log files to us will often help us to debug any problems.
-
-Next Steps
-==========
+More details about starting the IPython controller and engines can be found :ref:`here <parallel_process>`
 
 Once you have started the IPython controller and one or more engines, you
 are ready to use the engines to do something useful. To make sure
 everything is working correctly, try the following commands::
 
 	In [1]: from IPython.kernel import client
 	
 	In [2]: mec = client.MultiEngineClient()
 	
 	In [4]: mec.get_ids()
 	Out[4]: [0, 1, 2, 3]
 	
 	In [5]: mec.execute('print "Hello World"')
 	Out[5]: 
 	<Results List>
 	[0] In [1]: print "Hello World"
 	[0] Out[1]: Hello World
 
 	[1] In [1]: print "Hello World"
 	[1] Out[1]: Hello World
 
 	[2] In [1]: print "Hello World"
 	[2] Out[1]: Hello World
 
 	[3] In [1]: print "Hello World"
 	[3] Out[1]: Hello World
 
 Remember, a client also needs to present a ``.furl`` file to the controller.  How does this happen?  When a multiengine client is created with no arguments, the client tries to find the corresponding ``.furl`` file in the local :file:`~./ipython/security` directory.  If it finds it, you are set.  If you have put the ``.furl`` file in a different location or it has a different name, create the client like this::
 
     mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
 
 Same thing hold true of creating a task client::
 
     tc = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
 
 You are now ready to learn more about the :ref:`MultiEngine <parallelmultiengine>` and :ref:`Task <paralleltask>` interfaces to the controller.
 
 .. note:: 
 
     Don't forget that the engine, multiengine client and task client all have
     *different* furl files.  You must move *each* of these around to an appropriate
     location so that the engines and clients can use them to connect to the controller.

docs/source/parallel/parallel_mpi.txt

@@ -1,22 +1,78
 .. _parallelmpi:
 
 =======================
 Using MPI with IPython
 =======================
 
-The simplest way of getting started with MPI is to install an MPI implementation
-(we recommend `Open MPI`_) and `mpi4py`_ and then start the engines using the
-``mpirun`` command::
+Often, a parallel algorithm will require moving data between the engines.  One way of accomplishing this is by doing a pull and then a push using the multiengine client.  However, this will be slow as all the data has to go through the controller to the client and then back through the controller, to its final destination.
 
-	mpirun -n 4 ipengine --mpi=mpi4py
+A much better way of moving data between engines is to use a message passing library, such as the Message Passing Interface (`MPI`_).  IPython's parallel computing architecture has been designed from the ground up to integrate with `MPI`_.  This document describe how to use MPI with IPython.
 	
-This will automatically import `mpi4py`_ and make sure that `MPI_Init` is called
-at the right time. We also have built in support for `PyTrilinos`_, which can be
-used (assuming `PyTrilinos`_ is installed) by starting the engines with::
+Additional installation requirements
+====================================
+
+If you want to use MPI with IPython, you will need to install:
+
+* A standard MPI implementation such as `Open MPI`_ or MPICH.
+* The `mpi4py`_ package.
+
+.. note::
+
+    The `mpi4py`_ package is not a strict requirement. However, you need to
+    have *some* way of calling MPI from Python. You also need some way of
+    making sure that `MPI_Init` is called when the IPython engines start up.
+    There are a number of ways of doing this and a good number of associated
+    subtleties. We highly recommend just using `mpi4py`_ as it takes care of
+    most of these problems. If you want to do something different, let us know
+    and we can help you get started.
+
+Starting the engines with MPI enabled
+=====================================
+
+To use code that calls `MPI`_, there are typically two things that `MPI`_ requires.
+
+1. The process that wants to call `MPI`_ must be started using
+   :command:`mpirun` or a batch system (like PBS) that has `MPI`_ support.
+2. Once the process starts, it must call `MPI_Init`.
+
+There are a couple of ways that you can start the IPython engines and get these things to happen.
+
+Manual starting using :command:`mpirun`
+---------------------------------------
+
+If you want to start the IPython engines using the :command:`mpirun`, just do::
+
+    $ mpirun -n 4 ipengine --mpi=mpi4py
+
+This requires that you already have a controller running. We also have built
+in support for `PyTrilinos`_, which can be used (assuming `PyTrilinos`_ is
+installed) by starting the engines with::
 
 	mpirun -n 4 ipengine --mpi=pytrilinos
 
+Automatic starting using :command:`mpirun` and :command:`ipcluster`
+-------------------------------------------------------------------
+
+The easiest approach is to use the `mpirun` mode of :command:`ipcluster`, which will first start a controller and then a set of engines using :command:`mpirun`::
+
+    $ ipcluster mpirun -n 4
+
+Automatic starting using PBS and :command:`ipcluster`
+-----------------------------------------------------
+
+The :command:`ipcluster` command also has built-in integration with PBS. For more information on this approach, see our documentation on :ref:`ipcluster <parallel_process>`.
+
+Actually using MPI
+==================
+
+Once the engines are running with `MPI`_ enabled, you are ready to go.  You can now call any code that uses MPI in the IPython engines.  And all of this can be done interactively.
+
+Complications
+=============
+
+Talk about how some older MPI implementations are broken and need to have a custom Python mail loop.
+
 .. _MPI: http://www-unix.mcs.anl.gov/mpi/
 .. _mpi4py: http://mpi4py.scipy.org/
 .. _Open MPI: http://www.open-mpi.org/
 .. _PyTrilinos: http://trilinos.sandia.gov/packages/pytrilinos/
  No newline at end of file