clone parallel docs to parallelz

docs/source/parallelz/index.txt

@@ -0,0 +1,19 b''
+.. _parallelz_index:
+
+==========================================
+Using IPython for parallel computing (ZMQ)
+==========================================
+
+.. toctree::
+   :maxdepth: 2
+
+   parallel_intro.txt
+   parallel_process.txt
+   parallel_multiengine.txt
+   parallel_task.txt
+   parallel_mpi.txt
+   parallel_security.txt
+   parallel_winhpc.txt
+   parallel_demos.txt
+
+

docs/source/parallelz/parallel_demos.txt

@@ -0,0 +1,282 b''
+=================
+Parallel examples
+=================
+
+In this section we describe two more involved examples of using an IPython
+cluster to perform a parallel computation. In these examples, we will be using
+IPython's "pylab" mode, which enables interactive plotting using the
+Matplotlib package. IPython can be started in this mode by typing::
+
+    ipython -p pylab
+
+at the system command line. If this prints an error message, you will 
+need to install the default profiles from within IPython by doing,
+
+.. sourcecode:: ipython
+
+    In [1]: %install_profiles
+
+and then restarting IPython.
+
+150 million digits of pi
+========================
+
+In this example we would like to study the distribution of digits in the
+number pi (in base 10). While it is not known if pi is a normal number (a
+number is normal in base 10 if 0-9 occur with equal likelihood) numerical
+investigations suggest that it is. We will begin with a serial calculation on
+10,000 digits of pi and then perform a parallel calculation involving 150
+million digits.
+
+In both the serial and parallel calculation we will be using functions defined
+in the :file:`pidigits.py` file, which is available in the
+:file:`docs/examples/kernel` directory of the IPython source distribution.
+These functions provide basic facilities for working with the digits of pi and
+can be loaded into IPython by putting :file:`pidigits.py` in your current
+working directory and then doing:
+
+.. sourcecode:: ipython
+
+    In [1]: run pidigits.py
+
+Serial calculation
+------------------
+
+For the serial calculation, we will use SymPy (http://www.sympy.org) to
+calculate 10,000 digits of pi and then look at the frequencies of the digits
+0-9. Out of 10,000 digits, we expect each digit to occur 1,000 times. While
+SymPy is capable of calculating many more digits of pi, our purpose here is to
+set the stage for the much larger parallel calculation.
+
+In this example, we use two functions from :file:`pidigits.py`: 
+:func:`one_digit_freqs` (which calculates how many times each digit occurs)
+and :func:`plot_one_digit_freqs` (which uses Matplotlib to plot the result).
+Here is an interactive IPython session that uses these functions with
+SymPy:
+
+.. sourcecode:: ipython
+
+    In [7]: import sympy
+
+    In [8]: pi = sympy.pi.evalf(40)
+
+    In [9]: pi
+    Out[9]: 3.141592653589793238462643383279502884197
+
+    In [10]: pi = sympy.pi.evalf(10000)
+
+    In [11]: digits = (d for d in str(pi)[2:])  # create a sequence of digits
+
+    In [12]: run pidigits.py  # load one_digit_freqs/plot_one_digit_freqs
+
+    In [13]: freqs = one_digit_freqs(digits)
+
+    In [14]: plot_one_digit_freqs(freqs)
+    Out[14]: [<matplotlib.lines.Line2D object at 0x18a55290>]
+
+The resulting plot of the single digit counts shows that each digit occurs
+approximately 1,000 times, but that with only 10,000 digits the
+statistical fluctuations are still rather large:
+
+.. image:: single_digits.*
+
+It is clear that to reduce the relative fluctuations in the counts, we need
+to look at many more digits of pi. That brings us to the parallel calculation.
+
+Parallel calculation
+--------------------
+
+Calculating many digits of pi is a challenging computational problem in itself. 
+Because we want to focus on the distribution of digits in this example, we
+will use pre-computed digit of pi from the website of Professor Yasumasa
+Kanada at the University of Tokoyo (http://www.super-computing.org). These
+digits come in a set of text files (ftp://pi.super-computing.org/.2/pi200m/)
+that each have 10 million digits of pi.
+
+For the parallel calculation, we have copied these files to the local hard
+drives of the compute nodes. A total of 15 of these files will be used, for a
+total of 150 million digits of pi. To make things a little more interesting we
+will calculate the frequencies of all 2 digits sequences (00-99) and then plot
+the result using a 2D matrix in Matplotlib.
+
+The overall idea of the calculation is simple: each IPython engine will
+compute the two digit counts for the digits in a single file. Then in a final
+step the counts from each engine will be added up. To perform this
+calculation, we will need two top-level functions from :file:`pidigits.py`:
+
+.. literalinclude:: ../../examples/kernel/pidigits.py
+   :language: python
+   :lines: 34-49
+
+We will also use the :func:`plot_two_digit_freqs` function to plot the
+results. The code to run this calculation in parallel is contained in
+:file:`docs/examples/kernel/parallelpi.py`. This code can be run in parallel
+using IPython by following these steps:
+
+1. Copy the text files with the digits of pi 
+   (ftp://pi.super-computing.org/.2/pi200m/) to the working directory of the
+   engines on the compute nodes.
+2. Use :command:`ipcluster` to start 15 engines. We used an 8 core (2 quad 
+   core CPUs) cluster with hyperthreading enabled which makes the 8 cores 
+   looks like 16 (1 controller + 15 engines) in the OS. However, the maximum
+   speedup we can observe is still only 8x.
+3. With the file :file:`parallelpi.py` in your current working directory, open
+   up IPython in pylab mode and type ``run parallelpi.py``.
+
+When run on our 8 core cluster, we observe a speedup of 7.7x. This is slightly
+less than linear scaling (8x) because the controller is also running on one of
+the cores.
+
+To emphasize the interactive nature of IPython, we now show how the
+calculation can also be run by simply typing the commands from
+:file:`parallelpi.py` interactively into IPython:
+
+.. sourcecode:: ipython
+
+    In [1]: from IPython.kernel import client
+    2009-11-19 11:32:38-0800 [-] Log opened.
+
+    # The MultiEngineClient allows us to use the engines interactively.
+    # We simply pass MultiEngineClient the name of the cluster profile we
+    # are using.
+    In [2]: mec = client.MultiEngineClient(profile='mycluster')
+    2009-11-19 11:32:44-0800 [-] Connecting [0]
+    2009-11-19 11:32:44-0800 [Negotiation,client] Connected: ./ipcontroller-mec.furl
+
+    In [3]: mec.get_ids()
+    Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
+
+    In [4]: run pidigits.py
+
+    In [5]: filestring = 'pi200m-ascii-%(i)02dof20.txt'
+
+    # Create the list of files to process.
+    In [6]: files = [filestring % {'i':i} for i in range(1,16)]
+
+    In [7]: files
+    Out[7]: 
+    ['pi200m-ascii-01of20.txt',
+     'pi200m-ascii-02of20.txt',
+     'pi200m-ascii-03of20.txt',
+     'pi200m-ascii-04of20.txt',
+     'pi200m-ascii-05of20.txt',
+     'pi200m-ascii-06of20.txt',
+     'pi200m-ascii-07of20.txt',
+     'pi200m-ascii-08of20.txt',
+     'pi200m-ascii-09of20.txt',
+     'pi200m-ascii-10of20.txt',
+     'pi200m-ascii-11of20.txt',
+     'pi200m-ascii-12of20.txt',
+     'pi200m-ascii-13of20.txt',
+     'pi200m-ascii-14of20.txt',
+     'pi200m-ascii-15of20.txt']
+
+    # This is the parallel calculation using the MultiEngineClient.map method 
+    # which applies compute_two_digit_freqs to each file in files in parallel.
+    In [8]: freqs_all = mec.map(compute_two_digit_freqs, files)
+
+    # Add up the frequencies from each engine.
+    In [8]: freqs = reduce_freqs(freqs_all)
+
+    In [9]: plot_two_digit_freqs(freqs)
+    Out[9]: <matplotlib.image.AxesImage object at 0x18beb110>
+
+    In [10]: plt.title('2 digit counts of 150m digits of pi')
+    Out[10]: <matplotlib.text.Text object at 0x18d1f9b0>
+
+The resulting plot generated by Matplotlib is shown below. The colors indicate
+which two digit sequences are more (red) or less (blue) likely to occur in the
+first 150 million digits of pi. We clearly see that the sequence "41" is 
+most likely and that "06" and "07" are least likely. Further analysis would
+show that the relative size of the statistical fluctuations have decreased
+compared to the 10,000 digit calculation.
+
+.. image:: two_digit_counts.*
+
+
+Parallel options pricing
+========================
+
+An option is a financial contract that gives the buyer of the contract the
+right to buy (a "call") or sell (a "put") a secondary asset (a stock for
+example) at a particular date in the future (the expiration date) for a
+pre-agreed upon price (the strike price). For this right, the buyer pays the
+seller a premium (the option price). There are a wide variety of flavors of
+options (American, European, Asian, etc.) that are useful for different
+purposes: hedging against risk, speculation, etc.
+
+Much of modern finance is driven by the need to price these contracts
+accurately based on what is known about the properties (such as volatility) of
+the underlying asset. One method of pricing options is to use a Monte Carlo
+simulation of the underlying asset price. In this example we use this approach
+to price both European and Asian (path dependent) options for various strike
+prices and volatilities.
+
+The code for this example can be found in the :file:`docs/examples/kernel`
+directory of the IPython source. The function :func:`price_options` in
+:file:`mcpricer.py` implements the basic Monte Carlo pricing algorithm using
+the NumPy package and is shown here:
+
+.. literalinclude:: ../../examples/kernel/mcpricer.py
+   :language: python
+
+To run this code in parallel, we will use IPython's :class:`TaskClient` class,
+which distributes work to the engines using dynamic load balancing. This
+client can be used along side the :class:`MultiEngineClient` class shown in
+the previous example. The parallel calculation using :class:`TaskClient` can
+be found in the file :file:`mcpricer.py`. The code in this file creates a
+:class:`TaskClient` instance and then submits a set of tasks using
+:meth:`TaskClient.run` that calculate the option prices for different
+volatilities and strike prices. The results are then plotted as a 2D contour
+plot using Matplotlib.
+
+.. literalinclude:: ../../examples/kernel/mcdriver.py
+   :language: python
+
+To use this code, start an IPython cluster using :command:`ipcluster`, open
+IPython in the pylab mode with the file :file:`mcdriver.py` in your current
+working directory and then type:
+
+.. sourcecode:: ipython
+
+    In [7]: run mcdriver.py
+    Submitted tasks:  [0, 1, 2, ...]
+
+Once all the tasks have finished, the results can be plotted using the
+:func:`plot_options` function. Here we make contour plots of the Asian
+call and Asian put options as function of the volatility and strike price:
+
+.. sourcecode:: ipython
+
+    In [8]: plot_options(sigma_vals, K_vals, prices['acall'])
+
+    In [9]: plt.figure()
+    Out[9]: <matplotlib.figure.Figure object at 0x18c178d0>
+
+    In [10]: plot_options(sigma_vals, K_vals, prices['aput'])
+
+These results are shown in the two figures below. On a 8 core cluster the
+entire calculation (10 strike prices, 10 volatilities, 100,000 paths for each)
+took 30 seconds in parallel, giving a speedup of 7.7x, which is comparable
+to the speedup observed in our previous example.
+
+.. image:: asian_call.*
+
+.. image:: asian_put.*
+
+Conclusion
+==========
+
+To conclude these examples, we summarize the key features of IPython's
+parallel architecture that have been demonstrated:
+
+* Serial code can be parallelized often with only a few extra lines of code.
+  We have used the :class:`MultiEngineClient` and :class:`TaskClient` classes
+  for this purpose.
+* The resulting parallel code can be run without ever leaving the IPython's
+  interactive shell.
+* Any data computed in parallel can be explored interactively through 
+  visualization or further numerical calculations.
+* We have run these examples on a cluster running Windows HPC Server 2008.
+  IPython's built in support for the Windows HPC job scheduler makes it
+  easy to get started with IPython's parallel capabilities.

docs/source/parallelz/parallel_intro.txt

@@ -0,0 +1,237 b''
+.. _ip1par:
+
+============================
+Overview and getting started
+============================
+
+Introduction
+============
+
+This section gives an overview of IPython's sophisticated and powerful
+architecture for parallel and distributed computing. This architecture
+abstracts out parallelism in a very general way, which enables IPython to
+support many different styles of parallelism including:
+
+* Single program, multiple data (SPMD) parallelism.
+* Multiple program, multiple data (MPMD) parallelism.
+* Message passing using MPI.
+* Task farming.
+* Data parallel.
+* Combinations of these approaches.
+* Custom user defined approaches.
+
+Most importantly, IPython enables all types of parallel applications to
+be developed, executed, debugged and monitored *interactively*. Hence,
+the ``I`` in IPython.  The following are some example usage cases for IPython:
+
+* Quickly parallelize algorithms that are embarrassingly parallel
+  using a number of simple approaches.  Many simple things can be
+  parallelized interactively in one or two lines of code.
+
+* Steer traditional MPI applications on a supercomputer from an
+  IPython session on your laptop.
+
+* Analyze and visualize large datasets (that could be remote and/or
+  distributed) interactively using IPython and tools like
+  matplotlib/TVTK.
+
+* Develop, test and debug new parallel algorithms 
+  (that may use MPI) interactively.
+
+* Tie together multiple MPI jobs running on different systems into
+  one giant distributed and parallel system.
+
+* Start a parallel job on your cluster and then have a remote 	
+  collaborator connect to it and pull back data into their 
+  local IPython session for plotting and analysis.
+
+* Run a set of tasks on a set of CPUs using dynamic load balancing.
+
+Architecture overview
+=====================
+
+The IPython architecture consists of three components:
+
+* The IPython engine.
+* The IPython controller.
+* Various controller clients.
+
+These components live in the :mod:`IPython.kernel` package and are
+installed with IPython.  They do, however, have additional dependencies
+that must be installed.  For more information, see our
+:ref:`installation documentation <install_index>`.
+
+IPython engine
+---------------
+
+The IPython engine is a Python instance that takes Python commands over a
+network connection. Eventually, the IPython engine will be a full IPython
+interpreter, but for now, it is a regular Python interpreter. The engine
+can also handle incoming and outgoing Python objects sent over a network
+connection.  When multiple engines are started, parallel and distributed
+computing becomes possible. An important feature of an IPython engine is
+that it blocks while user code is being executed. Read on for how the
+IPython controller solves this problem to expose a clean asynchronous API
+to the user.
+
+IPython controller
+------------------
+
+The IPython controller provides an interface for working with a set of
+engines. At an general level, the controller is a process to which
+IPython engines can connect. For each connected engine, the controller
+manages a queue. All actions that can be performed on the engine go
+through this queue. While the engines themselves block when user code is
+run, the controller hides that from the user to provide a fully
+asynchronous interface to a set of engines. 
+
+.. note::
+
+    Because the controller listens on a network port for engines to 
+    connect to it, it must be started *before* any engines are started.  
+
+The controller also provides a single point of contact for users who wish to
+utilize the engines connected to the controller. There are different ways of
+working with a controller. In IPython these ways correspond to different
+interfaces that the controller is adapted to. Currently we have two default
+interfaces to the controller:
+
+* The MultiEngine interface, which provides the simplest possible way of
+  working with engines interactively.
+* The Task interface, which presents the engines as a load balanced
+  task farming system.
+
+Advanced users can easily add new custom interfaces to enable other
+styles of parallelism. 
+
+.. note:: 
+
+	A single controller and set of engines can be accessed 
+	through multiple interfaces simultaneously.  This opens the
+	door for lots of interesting things.  
+
+Controller clients
+------------------
+
+For each controller interface, there is a corresponding client. These
+clients allow users to interact with a set of engines through the
+interface.  Here are the two default clients:
+
+* The :class:`MultiEngineClient` class.
+* The :class:`TaskClient` class.
+
+Security
+--------
+
+By default (as long as `pyOpenSSL` is installed) all network connections
+between the controller and engines and the controller and clients are secure.
+What does this mean? First of all, all of the connections will be encrypted
+using SSL. Second, the connections are authenticated. We handle authentication
+in a capability based security model [Capability]_. In this model, a
+"capability (known in some systems as a key) is a communicable, unforgeable
+token of authority". Put simply, a capability is like a key to your house. If
+you have the key to your house, you can get in. If not, you can't.
+
+In our architecture, the controller is the only process that listens on
+network ports, and is thus responsible to creating these keys. In IPython,
+these keys are known as Foolscap URLs, or FURLs, because of the underlying
+network protocol we are using. As a user, you don't need to know anything
+about the details of these FURLs, other than that when the controller starts,
+it saves a set of FURLs to files named :file:`something.furl`. The default
+location of these files is the :file:`~./ipython/security` directory.
+
+To connect and authenticate to the controller an engine or client simply needs
+to present an appropriate FURL (that was originally created by the controller)
+to the controller. Thus, the FURL files need to be copied to a location where
+the clients and engines can find them. Typically, this is the
+:file:`~./ipython/security` directory on the host where the client/engine is
+running (which could be a different host than the controller). Once the FURL
+files are copied over, everything should work fine.
+
+Currently, there are three FURL files that the controller creates:
+
+ipcontroller-engine.furl
+    This FURL file is the key that gives an engine the ability to connect
+    to a controller.
+
+ipcontroller-tc.furl
+    This FURL file is the key that a :class:`TaskClient` must use to
+    connect to the task interface of a controller.
+    
+ipcontroller-mec.furl
+    This FURL file is the key that a :class:`MultiEngineClient` must use
+    to connect to the multiengine interface of a controller.
+
+More details of how these FURL files are used are given below.
+
+A detailed description of the security model and its implementation in IPython
+can be found :ref:`here <parallelsecurity>`.
+
+Getting Started
+===============
+
+To use IPython for parallel computing, you need to start one instance of the
+controller and one or more instances of the engine. Initially, it is best to
+simply start a controller and engines on a single host using the
+:command:`ipcluster` command. To start a controller and 4 engines on your
+localhost, just do::
+
+    $ ipcluster local -n 4
+
+More details about starting the IPython controller and engines can be found
+:ref:`here <parallel_process>`
+
+Once you have started the IPython controller and one or more engines, you
+are ready to use the engines to do something useful. To make sure
+everything is working correctly, try the following commands:
+
+.. sourcecode:: ipython
+
+	In [1]: from IPython.kernel import client
+	
+	In [2]: mec = client.MultiEngineClient()
+	
+	In [4]: mec.get_ids()
+	Out[4]: [0, 1, 2, 3]
+	
+	In [5]: mec.execute('print "Hello World"')
+	Out[5]: 
+	<Results List>
+	[0] In [1]: print "Hello World"
+	[0] Out[1]: Hello World
+
+	[1] In [1]: print "Hello World"
+	[1] Out[1]: Hello World
+
+	[2] In [1]: print "Hello World"
+	[2] Out[1]: Hello World
+
+	[3] In [1]: print "Hello World"
+	[3] Out[1]: Hello World
+
+Remember, a client also needs to present a FURL file to the controller. How
+does this happen? When a multiengine client is created with no arguments, the
+client tries to find the corresponding FURL file in the local
+:file:`~./ipython/security` directory. If it finds it, you are set. If you
+have put the FURL file in a different location or it has a different name,
+create the client like this::
+
+    mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
+
+Same thing hold true of creating a task client::
+
+    tc = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
+
+You are now ready to learn more about the :ref:`MultiEngine
+<parallelmultiengine>` and :ref:`Task <paralleltask>` interfaces to the
+controller.
+
+.. note:: 
+
+    Don't forget that the engine, multiengine client and task client all have
+    *different* furl files. You must move *each* of these around to an
+    appropriate location so that the engines and clients can use them to
+    connect to the controller.
+
+.. [Capability] Capability-based security, http://en.wikipedia.org/wiki/Capability-based_security
+

docs/source/parallelz/parallel_mpi.txt

@@ -0,0 +1,182 b''
+.. _parallelmpi:
+
+=======================
+Using MPI with IPython
+=======================
+
+Often, a parallel algorithm will require moving data between the engines. One
+way of accomplishing this is by doing a pull and then a push using the
+multiengine client. However, this will be slow as all the data has to go
+through the controller to the client and then back through the controller, to
+its final destination.
+
+A much better way of moving data between engines is to use a message passing
+library, such as the Message Passing Interface (MPI) [MPI]_. IPython's
+parallel computing architecture has been designed from the ground up to
+integrate with MPI. This document describes how to use MPI with IPython.
+
+Additional installation requirements
+====================================
+
+If you want to use MPI with IPython, you will need to install:
+
+* A standard MPI implementation such as OpenMPI [OpenMPI]_ or MPICH.
+* The mpi4py [mpi4py]_ package.
+
+.. note::
+
+    The mpi4py package is not a strict requirement. However, you need to
+    have *some* way of calling MPI from Python. You also need some way of
+    making sure that :func:`MPI_Init` is called when the IPython engines start
+    up. There are a number of ways of doing this and a good number of
+    associated subtleties. We highly recommend just using mpi4py as it
+    takes care of most of these problems. If you want to do something
+    different, let us know and we can help you get started.
+
+Starting the engines with MPI enabled
+=====================================
+
+To use code that calls MPI, there are typically two things that MPI requires.
+
+1. The process that wants to call MPI must be started using
+   :command:`mpiexec` or a batch system (like PBS) that has MPI support.
+2. Once the process starts, it must call :func:`MPI_Init`.
+
+There are a couple of ways that you can start the IPython engines and get
+these things to happen.
+
+Automatic starting using :command:`mpiexec` and :command:`ipcluster`
+--------------------------------------------------------------------
+
+The easiest approach is to use the `mpiexec` mode of :command:`ipcluster`,
+which will first start a controller and then a set of engines using
+:command:`mpiexec`::
+
+    $ ipcluster mpiexec -n 4
+
+This approach is best as interrupting :command:`ipcluster` will automatically
+stop and clean up the controller and engines.
+
+Manual starting using :command:`mpiexec`
+----------------------------------------
+
+If you want to start the IPython engines using the :command:`mpiexec`, just
+do::
+
+    $ mpiexec -n 4 ipengine --mpi=mpi4py
+
+This requires that you already have a controller running and that the FURL
+files for the engines are in place. We also have built in support for
+PyTrilinos [PyTrilinos]_, which can be used (assuming is installed) by
+starting the engines with::
+
+    	mpiexec -n 4 ipengine --mpi=pytrilinos
+
+Automatic starting using PBS and :command:`ipcluster`
+-----------------------------------------------------
+
+The :command:`ipcluster` command also has built-in integration with PBS. For
+more information on this approach, see our documentation on :ref:`ipcluster
+<parallel_process>`.
+
+Actually using MPI
+==================
+
+Once the engines are running with MPI enabled, you are ready to go. You can
+now call any code that uses MPI in the IPython engines. And, all of this can
+be done interactively. Here we show a simple example that uses mpi4py
+[mpi4py]_ version 1.1.0 or later.
+
+First, lets define a simply function that uses MPI to calculate the sum of a
+distributed array. Save the following text in a file called :file:`psum.py`:
+
+.. sourcecode:: python
+
+    from mpi4py import MPI
+    import numpy as np
+
+    def psum(a):
+        s = np.sum(a)
+        rcvBuf = np.array(0.0,'d')
+        MPI.COMM_WORLD.Allreduce([s, MPI.DOUBLE],
+            [rcvBuf, MPI.DOUBLE],
+            op=MPI.SUM)
+        return rcvBuf
+
+Now, start an IPython cluster in the same directory as :file:`psum.py`::
+
+    $ ipcluster mpiexec -n 4
+
+Finally, connect to the cluster and use this function interactively. In this
+case, we create a random array on each engine and sum up all the random arrays
+using our :func:`psum` function:
+
+.. sourcecode:: ipython
+
+    In [1]: from IPython.kernel import client
+    
+    In [2]: mec = client.MultiEngineClient()
+    
+    In [3]: mec.activate()
+    
+    In [4]: px import numpy as np
+    Parallel execution on engines: all
+    Out[4]: 
+    <Results List>
+    [0] In [13]: import numpy as np
+    [1] In [13]: import numpy as np
+    [2] In [13]: import numpy as np
+    [3] In [13]: import numpy as np
+    
+    In [6]: px a = np.random.rand(100)
+    Parallel execution on engines: all
+    Out[6]: 
+    <Results List>
+    [0] In [15]: a = np.random.rand(100)
+    [1] In [15]: a = np.random.rand(100)
+    [2] In [15]: a = np.random.rand(100)
+    [3] In [15]: a = np.random.rand(100)
+    
+    In [7]: px from psum import psum
+    Parallel execution on engines: all
+    Out[7]: 
+    <Results List>
+    [0] In [16]: from psum import psum
+    [1] In [16]: from psum import psum
+    [2] In [16]: from psum import psum
+    [3] In [16]: from psum import psum
+    
+    In [8]: px s = psum(a)
+    Parallel execution on engines: all
+    Out[8]: 
+    <Results List>
+    [0] In [17]: s = psum(a)
+    [1] In [17]: s = psum(a)
+    [2] In [17]: s = psum(a)
+    [3] In [17]: s = psum(a)
+    
+    In [9]: px print s
+    Parallel execution on engines: all
+    Out[9]: 
+    <Results List>
+    [0] In [18]: print s
+    [0] Out[18]: 187.451545803
+    
+    [1] In [18]: print s
+    [1] Out[18]: 187.451545803
+    
+    [2] In [18]: print s
+    [2] Out[18]: 187.451545803
+
+    [3] In [18]: print s
+    [3] Out[18]: 187.451545803
+
+Any Python code that makes calls to MPI can be used in this manner, including
+compiled C, C++ and Fortran libraries that have been exposed to Python.
+
+.. [MPI] Message Passing Interface.  http://www-unix.mcs.anl.gov/mpi/
+.. [mpi4py] MPI for Python. mpi4py: http://mpi4py.scipy.org/
+.. [OpenMPI] Open MPI. http://www.open-mpi.org/
+.. [PyTrilinos] PyTrilinos. http://trilinos.sandia.gov/packages/pytrilinos/
+
+

docs/source/parallelz/parallel_multiengine.txt

@@ -0,0 +1,835 b''
+.. _parallelmultiengine:
+
+===============================
+IPython's multiengine interface
+===============================
+
+The multiengine interface represents one possible way of working with a set of
+IPython engines. The basic idea behind the multiengine interface is that the
+capabilities of each engine are directly and explicitly exposed to the user.
+Thus, in the multiengine interface, each engine is given an id that is used to
+identify the engine and give it work to do. This interface is very intuitive
+and is designed with interactive usage in mind, and is thus the best place for
+new users of IPython to begin.
+
+Starting the IPython controller and engines
+===========================================
+
+To follow along with this tutorial, you will need to start the IPython
+controller and four IPython engines. The simplest way of doing this is to use
+the :command:`ipcluster` command::
+
+	$ ipcluster local -n 4
+	
+For more detailed information about starting the controller and engines, see
+our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+
+Creating a ``MultiEngineClient`` instance
+=========================================
+
+The first step is to import the IPython :mod:`IPython.kernel.client` module
+and then create a :class:`MultiEngineClient` instance:
+
+.. sourcecode:: ipython
+
+	In [1]: from IPython.kernel import client
+	
+	In [2]: mec = client.MultiEngineClient()
+
+This form assumes that the :file:`ipcontroller-mec.furl` is in the
+:file:`~./ipython/security` directory on the client's host. If not, the
+location of the FURL file must be given as an argument to the
+constructor:
+
+.. sourcecode:: ipython
+
+    In [2]: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
+
+To make sure there are engines connected to the controller, use can get a list
+of engine ids:
+
+.. sourcecode:: ipython
+
+	In [3]: mec.get_ids()
+	Out[3]: [0, 1, 2, 3]
+
+Here we see that there are four engines ready to do work for us.
+
+Quick and easy parallelism
+==========================
+
+In many cases, you simply want to apply a Python function to a sequence of
+objects, but *in parallel*. The multiengine interface provides two simple ways
+of accomplishing this: a parallel version of :func:`map` and ``@parallel``
+function decorator.
+
+Parallel map
+------------
+
+Python's builtin :func:`map` functions allows a function to be applied to a
+sequence element-by-element. This type of code is typically trivial to
+parallelize. In fact, the multiengine interface in IPython already has a
+parallel version of :meth:`map` that works just like its serial counterpart:
+
+.. sourcecode:: ipython
+
+	In [63]: serial_result = map(lambda x:x**10, range(32))
+
+	In [64]: parallel_result = mec.map(lambda x:x**10, range(32))
+
+	In [65]: serial_result==parallel_result
+	Out[65]: True
+
+.. note::
+
+    The multiengine interface version of :meth:`map` does not do any load
+    balancing.  For a load balanced version, see the task interface.
+
+.. seealso::
+
+    The :meth:`map` method has a number of options that can be controlled by
+    the :meth:`mapper` method.  See its docstring for more information.
+
+Parallel function decorator
+---------------------------
+
+Parallel functions are just like normal function, but they can be called on
+sequences and *in parallel*. The multiengine interface provides a decorator
+that turns any Python function into a parallel function:
+
+.. sourcecode:: ipython
+
+    In [10]: @mec.parallel()
+       ....: def f(x):
+       ....:     return 10.0*x**4
+       ....: 
+
+    In [11]: f(range(32))    # this is done in parallel
+    Out[11]: 
+    [0.0,10.0,160.0,...]
+
+See the docstring for the :meth:`parallel` decorator for options.
+
+Running Python commands
+=======================
+
+The most basic type of operation that can be performed on the engines is to
+execute Python code. Executing Python code can be done in blocking or
+non-blocking mode (blocking is default) using the :meth:`execute` method.
+
+Blocking execution
+------------------
+
+In blocking mode, the :class:`MultiEngineClient` object (called ``mec`` in
+these examples) submits the command to the controller, which places the
+command in the engines' queues for execution. The :meth:`execute` call then
+blocks until the engines are done executing the command:
+
+.. sourcecode:: ipython
+
+	# The default is to run on all engines
+	In [4]: mec.execute('a=5')
+	Out[4]: 
+	<Results List>
+	[0] In [1]: a=5
+	[1] In [1]: a=5
+	[2] In [1]: a=5
+	[3] In [1]: a=5
+
+	In [5]: mec.execute('b=10')
+	Out[5]: 
+	<Results List>
+	[0] In [2]: b=10
+	[1] In [2]: b=10
+	[2] In [2]: b=10
+	[3] In [2]: b=10
+
+Python commands can be executed on specific engines by calling execute using
+the ``targets`` keyword argument:
+
+.. sourcecode:: ipython
+
+	In [6]: mec.execute('c=a+b',targets=[0,2])
+	Out[6]: 
+	<Results List>
+	[0] In [3]: c=a+b
+	[2] In [3]: c=a+b
+
+
+	In [7]: mec.execute('c=a-b',targets=[1,3])
+	Out[7]: 
+	<Results List>
+	[1] In [3]: c=a-b
+	[3] In [3]: c=a-b
+
+
+	In [8]: mec.execute('print c')
+	Out[8]: 
+	<Results List>
+	[0] In [4]: print c
+	[0] Out[4]: 15
+
+	[1] In [4]: print c
+	[1] Out[4]: -5
+
+	[2] In [4]: print c
+	[2] Out[4]: 15
+
+	[3] In [4]: print c
+	[3] Out[4]: -5
+
+This example also shows one of the most important things about the IPython
+engines: they have a persistent user namespaces. The :meth:`execute` method
+returns a Python ``dict`` that contains useful information:
+
+.. sourcecode:: ipython
+
+	In [9]: result_dict = mec.execute('d=10; print d')
+
+	In [10]: for r in result_dict:
+	   ....:     print r
+	   ....:     
+	   ....:     
+	{'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 0, 'stdout': '10\n'}
+	{'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 1, 'stdout': '10\n'}
+	{'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 2, 'stdout': '10\n'}
+	{'input': {'translated': 'd=10; print d', 'raw': 'd=10; print d'}, 'number': 5, 'id': 3, 'stdout': '10\n'}
+
+Non-blocking execution
+----------------------
+
+In non-blocking mode, :meth:`execute` submits the command to be executed and
+then returns a :class:`PendingResult` object immediately. The
+:class:`PendingResult` object gives you a way of getting a result at a later
+time through its :meth:`get_result` method or :attr:`r` attribute. This allows
+you to quickly submit long running commands without blocking your local
+Python/IPython session:
+
+.. sourcecode:: ipython
+
+	# In blocking mode
+	In [6]: mec.execute('import time')
+	Out[6]:
+	<Results List>
+	[0] In [1]: import time
+	[1] In [1]: import time
+	[2] In [1]: import time
+	[3] In [1]: import time
+
+	# In non-blocking mode
+	In [7]: pr = mec.execute('time.sleep(10)',block=False)
+
+	# Now block for the result
+	In [8]: pr.get_result()
+	Out[8]:
+	<Results List>
+	[0] In [2]: time.sleep(10)
+	[1] In [2]: time.sleep(10)
+	[2] In [2]: time.sleep(10)
+	[3] In [2]: time.sleep(10)
+
+	# Again in non-blocking mode
+	In [9]: pr = mec.execute('time.sleep(10)',block=False)
+
+	# Poll to see if the result is ready
+	In [10]: pr.get_result(block=False)
+
+	# A shorthand for get_result(block=True)
+	In [11]: pr.r
+	Out[11]:
+	<Results List>
+	[0] In [3]: time.sleep(10)
+	[1] In [3]: time.sleep(10)
+	[2] In [3]: time.sleep(10)
+	[3] In [3]: time.sleep(10)
+
+Often, it is desirable to wait until a set of :class:`PendingResult` objects
+are done. For this, there is a the method :meth:`barrier`. This method takes a
+tuple of :class:`PendingResult` objects and blocks until all of the associated
+results are ready:
+
+.. sourcecode:: ipython
+
+	In [72]: mec.block=False
+
+	# A trivial list of PendingResults objects
+	In [73]: pr_list = [mec.execute('time.sleep(3)') for i in range(10)]
+
+	# Wait until all of them are done
+	In [74]: mec.barrier(pr_list)
+
+	# Then, their results are ready using get_result or the r attribute
+	In [75]: pr_list[0].r
+	Out[75]: 
+	<Results List>
+	[0] In [20]: time.sleep(3)
+	[1] In [19]: time.sleep(3)
+	[2] In [20]: time.sleep(3)
+	[3] In [19]: time.sleep(3)
+
+
+The ``block`` and ``targets`` keyword arguments and attributes
+--------------------------------------------------------------
+
+Most methods in the multiengine interface (like :meth:`execute`) accept
+``block`` and ``targets`` as keyword arguments. As we have seen above, these
+keyword arguments control the blocking mode and which engines the command is
+applied to. The :class:`MultiEngineClient` class also has :attr:`block` and
+:attr:`targets` attributes that control the default behavior when the keyword
+arguments are not provided. Thus the following logic is used for :attr:`block`
+and :attr:`targets`:
+
+* If no keyword argument is provided, the instance attributes are used.
+* Keyword argument, if provided override the instance attributes.
+
+The following examples demonstrate how to use the instance attributes:
+
+.. sourcecode:: ipython
+
+	In [16]: mec.targets = [0,2]
+
+	In [17]: mec.block = False
+
+	In [18]: pr = mec.execute('a=5')
+
+	In [19]: pr.r
+	Out[19]: 
+	<Results List>
+	[0] In [6]: a=5
+	[2] In [6]: a=5
+
+	# Note targets='all' means all engines
+	In [20]: mec.targets = 'all'
+
+	In [21]: mec.block = True
+
+	In [22]: mec.execute('b=10; print b')
+	Out[22]: 
+	<Results List>
+	[0] In [7]: b=10; print b
+	[0] Out[7]: 10
+
+	[1] In [6]: b=10; print b
+	[1] Out[6]: 10
+
+	[2] In [7]: b=10; print b
+	[2] Out[7]: 10
+
+	[3] In [6]: b=10; print b
+	[3] Out[6]: 10
+
+The :attr:`block` and :attr:`targets` instance attributes also determine the
+behavior of the parallel magic commands.
+
+
+Parallel magic commands
+-----------------------
+
+We provide a few IPython magic commands (``%px``, ``%autopx`` and ``%result``)
+that make it more pleasant to execute Python commands on the engines
+interactively. These are simply shortcuts to :meth:`execute` and
+:meth:`get_result`. The ``%px`` magic executes a single Python command on the
+engines specified by the :attr:`targets` attribute of the
+:class:`MultiEngineClient` instance (by default this is ``'all'``):
+
+.. sourcecode:: ipython
+
+	# Make this MultiEngineClient active for parallel magic commands
+	In [23]: mec.activate()
+
+	In [24]: mec.block=True
+
+	In [25]: import numpy
+
+	In [26]: %px import numpy
+	Executing command on Controller
+	Out[26]:
+	<Results List>
+	[0] In [8]: import numpy
+	[1] In [7]: import numpy
+	[2] In [8]: import numpy
+	[3] In [7]: import numpy
+
+
+	In [27]: %px a = numpy.random.rand(2,2)
+	Executing command on Controller
+	Out[27]:
+	<Results List>
+	[0] In [9]: a = numpy.random.rand(2,2)
+	[1] In [8]: a = numpy.random.rand(2,2)
+	[2] In [9]: a = numpy.random.rand(2,2)
+	[3] In [8]: a = numpy.random.rand(2,2)
+
+
+	In [28]: %px print numpy.linalg.eigvals(a)
+	Executing command on Controller
+	Out[28]:
+	<Results List>
+	[0] In [10]: print numpy.linalg.eigvals(a)
+	[0] Out[10]: [ 1.28167017  0.14197338]
+
+	[1] In [9]: print numpy.linalg.eigvals(a)
+	[1] Out[9]: [-0.14093616  1.27877273]
+
+	[2] In [10]: print numpy.linalg.eigvals(a)
+	[2] Out[10]: [-0.37023573  1.06779409]
+
+	[3] In [9]: print numpy.linalg.eigvals(a)
+	[3] Out[9]: [ 0.83664764 -0.25602658]
+
+The ``%result`` magic gets and prints the stdin/stdout/stderr of the last
+command executed on each engine. It is simply a shortcut to the
+:meth:`get_result` method:
+
+.. sourcecode:: ipython
+
+	In [29]: %result
+	Out[29]:
+	<Results List>
+	[0] In [10]: print numpy.linalg.eigvals(a)
+	[0] Out[10]: [ 1.28167017  0.14197338]
+
+	[1] In [9]: print numpy.linalg.eigvals(a)
+	[1] Out[9]: [-0.14093616  1.27877273]
+
+	[2] In [10]: print numpy.linalg.eigvals(a)
+	[2] Out[10]: [-0.37023573  1.06779409]
+
+	[3] In [9]: print numpy.linalg.eigvals(a)
+	[3] Out[9]: [ 0.83664764 -0.25602658]
+
+The ``%autopx`` magic switches to a mode where everything you type is executed
+on the engines given by the :attr:`targets` attribute:
+
+.. sourcecode:: ipython
+
+	In [30]: mec.block=False
+
+	In [31]: %autopx
+	Auto Parallel Enabled
+	Type %autopx to disable
+
+	In [32]: max_evals = []
+	<IPython.kernel.multiengineclient.PendingResult object at 0x17b8a70>
+
+	In [33]: for i in range(100):
+	   ....:     a = numpy.random.rand(10,10)
+	   ....:     a = a+a.transpose()
+	   ....:     evals = numpy.linalg.eigvals(a)
+	   ....:     max_evals.append(evals[0].real)
+	   ....:
+	   ....:
+	<IPython.kernel.multiengineclient.PendingResult object at 0x17af8f0>
+
+	In [34]: %autopx
+	Auto Parallel Disabled
+
+	In [35]: mec.block=True
+
+	In [36]: px print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
+	Executing command on Controller
+	Out[36]:
+	<Results List>
+	[0] In [13]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
+	[0] Out[13]: Average max eigenvalue is:  10.1387247332
+
+	[1] In [12]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
+	[1] Out[12]: Average max eigenvalue is:  10.2076902286
+
+	[2] In [13]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
+	[2] Out[13]: Average max eigenvalue is:  10.1891484655
+
+	[3] In [12]: print "Average max eigenvalue is: ", sum(max_evals)/len(max_evals)
+	[3] Out[12]: Average max eigenvalue is:  10.1158837784
+
+
+Moving Python objects around
+============================
+
+In addition to executing code on engines, you can transfer Python objects to
+and from your IPython session and the engines. In IPython, these operations
+are called :meth:`push` (sending an object to the engines) and :meth:`pull`
+(getting an object from the engines).
+
+Basic push and pull
+-------------------
+
+Here are some examples of how you use :meth:`push` and :meth:`pull`:
+
+.. sourcecode:: ipython
+
+	In [38]: mec.push(dict(a=1.03234,b=3453))
+	Out[38]: [None, None, None, None]
+
+	In [39]: mec.pull('a')
+	Out[39]: [1.03234, 1.03234, 1.03234, 1.03234]
+
+	In [40]: mec.pull('b',targets=0)
+	Out[40]: [3453]
+
+	In [41]: mec.pull(('a','b'))
+	Out[41]: [[1.03234, 3453], [1.03234, 3453], [1.03234, 3453], [1.03234, 3453]]
+
+	In [42]: mec.zip_pull(('a','b'))
+	Out[42]: [(1.03234, 1.03234, 1.03234, 1.03234), (3453, 3453, 3453, 3453)]
+
+	In [43]: mec.push(dict(c='speed'))
+	Out[43]: [None, None, None, None]
+
+	In [44]: %px print c
+	Executing command on Controller
+	Out[44]: 
+	<Results List>
+	[0] In [14]: print c
+	[0] Out[14]: speed
+
+	[1] In [13]: print c
+	[1] Out[13]: speed
+
+	[2] In [14]: print c
+	[2] Out[14]: speed
+
+	[3] In [13]: print c
+	[3] Out[13]: speed
+
+In non-blocking mode :meth:`push` and :meth:`pull` also return
+:class:`PendingResult` objects:
+
+.. sourcecode:: ipython
+
+	In [47]: mec.block=False
+
+	In [48]: pr = mec.pull('a')
+
+	In [49]: pr.r
+	Out[49]: [1.03234, 1.03234, 1.03234, 1.03234]
+
+
+Push and pull for functions
+---------------------------
+
+Functions can also be pushed and pulled using :meth:`push_function` and
+:meth:`pull_function`:
+
+.. sourcecode:: ipython
+
+    In [52]: mec.block=True
+
+	In [53]: def f(x):
+	   ....:     return 2.0*x**4
+	   ....: 
+
+	In [54]: mec.push_function(dict(f=f))
+	Out[54]: [None, None, None, None]
+
+	In [55]: mec.execute('y = f(4.0)')
+	Out[55]: 
+	<Results List>
+	[0] In [15]: y = f(4.0)
+	[1] In [14]: y = f(4.0)
+	[2] In [15]: y = f(4.0)
+	[3] In [14]: y = f(4.0)
+
+
+	In [56]: px print y
+	Executing command on Controller
+	Out[56]: 
+	<Results List>
+	[0] In [16]: print y
+	[0] Out[16]: 512.0
+
+	[1] In [15]: print y
+	[1] Out[15]: 512.0
+
+	[2] In [16]: print y
+	[2] Out[16]: 512.0
+
+	[3] In [15]: print y
+	[3] Out[15]: 512.0
+
+
+Dictionary interface
+--------------------
+
+As a shorthand to :meth:`push` and :meth:`pull`, the
+:class:`MultiEngineClient` class implements some of the Python dictionary
+interface. This make the remote namespaces of the engines appear as a local
+dictionary. Underneath, this uses :meth:`push` and :meth:`pull`:
+
+.. sourcecode:: ipython
+
+	In [50]: mec.block=True
+
+	In [51]: mec['a']=['foo','bar']
+
+	In [52]: mec['a']
+	Out[52]: [['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar'], ['foo', 'bar']]
+
+Scatter and gather
+------------------
+
+Sometimes it is useful to partition a sequence and push the partitions to
+different engines. In MPI language, this is know as scatter/gather and we
+follow that terminology. However, it is important to remember that in
+IPython's :class:`MultiEngineClient` class, :meth:`scatter` is from the
+interactive IPython session to the engines and :meth:`gather` is from the
+engines back to the interactive IPython session. For scatter/gather operations
+between engines, MPI should be used:
+
+.. sourcecode:: ipython
+
+	In [58]: mec.scatter('a',range(16))
+	Out[58]: [None, None, None, None]
+
+	In [59]: px print a
+	Executing command on Controller
+	Out[59]: 
+	<Results List>
+	[0] In [17]: print a
+	[0] Out[17]: [0, 1, 2, 3]
+
+	[1] In [16]: print a
+	[1] Out[16]: [4, 5, 6, 7]
+
+	[2] In [17]: print a
+	[2] Out[17]: [8, 9, 10, 11]
+
+	[3] In [16]: print a
+	[3] Out[16]: [12, 13, 14, 15]
+
+
+	In [60]: mec.gather('a')
+	Out[60]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+
+Other things to look at
+=======================
+
+How to do parallel list comprehensions
+--------------------------------------
+
+In many cases list comprehensions are nicer than using the map function. While
+we don't have fully parallel list comprehensions, it is simple to get the
+basic effect using :meth:`scatter` and :meth:`gather`:
+
+.. sourcecode:: ipython
+
+	In [66]: mec.scatter('x',range(64))
+	Out[66]: [None, None, None, None]
+
+	In [67]: px y = [i**10 for i in x]
+	Executing command on Controller
+	Out[67]: 
+	<Results List>
+	[0] In [19]: y = [i**10 for i in x]
+	[1] In [18]: y = [i**10 for i in x]
+	[2] In [19]: y = [i**10 for i in x]
+	[3] In [18]: y = [i**10 for i in x]
+
+
+	In [68]: y = mec.gather('y')
+
+	In [69]: print y
+	[0, 1, 1024, 59049, 1048576, 9765625, 60466176, 282475249, 1073741824,...]
+
+Parallel exceptions
+-------------------
+
+In the multiengine interface, parallel commands can raise Python exceptions,
+just like serial commands. But, it is a little subtle, because a single
+parallel command can actually raise multiple exceptions (one for each engine
+the command was run on). To express this idea, the MultiEngine interface has a
+:exc:`CompositeError` exception class that will be raised in most cases. The
+:exc:`CompositeError` class is a special type of exception that wraps one or
+more other types of exceptions. Here is how it works:
+
+.. sourcecode:: ipython
+
+	In [76]: mec.block=True
+
+	In [77]: mec.execute('1/0')
+	---------------------------------------------------------------------------
+	CompositeError                            Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<ipython console> in <module>()
+
+	/ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
+	    432         targets, block = self._findTargetsAndBlock(targets, block)
+	    433         result = blockingCallFromThread(self.smultiengine.execute, lines,
+	--> 434             targets=targets, block=block)
+	    435         if block:
+	    436             result = ResultList(result)
+
+	/ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
+	     72             result.raiseException()
+	     73         except Exception, e:
+	---> 74             raise e
+	     75     return result
+	     76 
+
+	CompositeError: one or more exceptions from call to method: execute
+	[0:execute]: ZeroDivisionError: integer division or modulo by zero
+	[1:execute]: ZeroDivisionError: integer division or modulo by zero
+	[2:execute]: ZeroDivisionError: integer division or modulo by zero
+	[3:execute]: ZeroDivisionError: integer division or modulo by zero
+
+Notice how the error message printed when :exc:`CompositeError` is raised has
+information about the individual exceptions that were raised on each engine.
+If you want, you can even raise one of these original exceptions:
+
+.. sourcecode:: ipython
+
+	In [80]: try:
+	   ....:     mec.execute('1/0')
+	   ....: except client.CompositeError, e:
+	   ....:     e.raise_exception()
+	   ....:     
+	   ....:     
+	---------------------------------------------------------------------------
+	ZeroDivisionError                         Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<ipython console> in <module>()
+
+	/ipython1-client-r3021/ipython1/kernel/error.pyc in raise_exception(self, excid)
+	    156             raise IndexError("an exception with index %i does not exist"%excid)
+	    157         else:
+	--> 158             raise et, ev, etb
+	    159 
+	    160 def collect_exceptions(rlist, method):
+
+	ZeroDivisionError: integer division or modulo by zero
+
+If you are working in IPython, you can simple type ``%debug`` after one of
+these :exc:`CompositeError` exceptions is raised, and inspect the exception
+instance:
+
+.. sourcecode:: ipython
+
+	In [81]: mec.execute('1/0')
+	---------------------------------------------------------------------------
+	CompositeError                            Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<ipython console> in <module>()
+
+	/ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in execute(self, lines, targets, block)
+	    432         targets, block = self._findTargetsAndBlock(targets, block)
+	    433         result = blockingCallFromThread(self.smultiengine.execute, lines,
+	--> 434             targets=targets, block=block)
+	    435         if block:
+	    436             result = ResultList(result)
+
+	/ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
+	     72             result.raiseException()
+	     73         except Exception, e:
+	---> 74             raise e
+	     75     return result
+	     76 
+
+	CompositeError: one or more exceptions from call to method: execute
+	[0:execute]: ZeroDivisionError: integer division or modulo by zero
+	[1:execute]: ZeroDivisionError: integer division or modulo by zero
+	[2:execute]: ZeroDivisionError: integer division or modulo by zero
+	[3:execute]: ZeroDivisionError: integer division or modulo by zero
+
+	In [82]: %debug
+	> 
+	
+	/ipython1-client-r3021/ipython1/kernel/twistedutil.py(74)blockingCallFromThread()
+	     73         except Exception, e:
+	---> 74             raise e
+	     75     return result
+
+	# With the debugger running, e is the exceptions instance.  We can tab complete
+	# on it and see the extra methods that are available.
+	ipdb> e.
+	e.__class__         e.__getitem__       e.__new__           e.__setstate__      e.args
+	e.__delattr__       e.__getslice__      e.__reduce__        e.__str__           e.elist
+	e.__dict__          e.__hash__          e.__reduce_ex__     e.__weakref__       e.message
+	e.__doc__           e.__init__          e.__repr__          e._get_engine_str   e.print_tracebacks
+	e.__getattribute__  e.__module__        e.__setattr__       e._get_traceback    e.raise_exception
+	ipdb> e.print_tracebacks()
+	[0:execute]: 
+	---------------------------------------------------------------------------
+	ZeroDivisionError                         Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<string> in <module>()
+
+	ZeroDivisionError: integer division or modulo by zero
+
+	[1:execute]: 
+	---------------------------------------------------------------------------
+	ZeroDivisionError                         Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<string> in <module>()
+
+	ZeroDivisionError: integer division or modulo by zero
+
+	[2:execute]: 
+	---------------------------------------------------------------------------
+	ZeroDivisionError                         Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<string> in <module>()
+
+	ZeroDivisionError: integer division or modulo by zero
+
+	[3:execute]: 
+	---------------------------------------------------------------------------
+	ZeroDivisionError                         Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<string> in <module>()
+
+	ZeroDivisionError: integer division or modulo by zero
+
+.. note::
+
+    The above example appears to be broken right now because of a change in 
+    how we are using Twisted.
+
+All of this same error handling magic even works in non-blocking mode:
+
+.. sourcecode:: ipython
+
+	In [83]: mec.block=False
+
+	In [84]: pr = mec.execute('1/0')
+
+	In [85]: pr.r
+	---------------------------------------------------------------------------
+	CompositeError                            Traceback (most recent call last)
+
+	/ipython1-client-r3021/docs/examples/<ipython console> in <module>()
+
+	/ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in _get_r(self)
+	    170 
+	    171     def _get_r(self):
+	--> 172         return self.get_result(block=True)
+	    173 
+	    174     r = property(_get_r)
+
+	/ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_result(self, default, block)
+	    131                 return self.result
+	    132         try:
+	--> 133             result = self.client.get_pending_deferred(self.result_id, block)
+	    134         except error.ResultNotCompleted:
+	    135             return default
+
+	/ipython1-client-r3021/ipython1/kernel/multiengineclient.pyc in get_pending_deferred(self, deferredID, block)
+	    385 
+	    386     def get_pending_deferred(self, deferredID, block):
+	--> 387         return blockingCallFromThread(self.smultiengine.get_pending_deferred, deferredID, block)
+	    388 
+	    389     def barrier(self, pendingResults):
+
+	/ipython1-client-r3021/ipython1/kernel/twistedutil.pyc in blockingCallFromThread(f, *a, **kw)
+	     72             result.raiseException()
+	     73         except Exception, e:
+	---> 74             raise e
+	     75     return result
+	     76 
+
+	CompositeError: one or more exceptions from call to method: execute
+	[0:execute]: ZeroDivisionError: integer division or modulo by zero
+	[1:execute]: ZeroDivisionError: integer division or modulo by zero
+	[2:execute]: ZeroDivisionError: integer division or modulo by zero
+	[3:execute]: ZeroDivisionError: integer division or modulo by zero
+
+

docs/source/parallelz/parallel_process.txt

@@ -0,0 +1,389 b''
+.. _parallel_process:
+
+===========================================
+Starting the IPython controller and engines
+===========================================
+
+To use IPython for parallel computing, you need to start one instance of
+the controller and one or more instances of the engine. The controller
+and each engine can run on different machines or on the same machine.
+Because of this, there are many different possibilities.
+
+Broadly speaking, there are two ways of going about starting a controller and engines:
+
+* In an automated manner using the :command:`ipcluster` command.
+* In a more manual way using the :command:`ipcontroller` and 
+  :command:`ipengine` commands.
+
+This document describes both of these methods. We recommend that new users
+start with the :command:`ipcluster` command as it simplifies many common usage
+cases.
+
+General considerations
+======================
+
+Before delving into the details about how you can start a controller and
+engines using the various methods, we outline some of the general issues that
+come up when starting the controller and engines. These things come up no
+matter which method you use to start your IPython cluster.
+
+Let's say that you want to start the controller on ``host0`` and engines on
+hosts ``host1``-``hostn``. The following steps are then required:
+
+1. Start the controller on ``host0`` by running :command:`ipcontroller` on
+   ``host0``.
+2. Move the FURL file (:file:`ipcontroller-engine.furl`) created by the 
+   controller from ``host0`` to hosts ``host1``-``hostn``.
+3. Start the engines on hosts ``host1``-``hostn`` by running 
+   :command:`ipengine`.  This command has to be told where the FURL file 
+   (:file:`ipcontroller-engine.furl`) is located.
+
+At this point, the controller and engines will be connected. By default, the
+FURL files created by the controller are put into the
+:file:`~/.ipython/security` directory. If the engines share a filesystem with
+the controller, step 2 can be skipped as the engines will automatically look
+at that location.
+
+The final step required required to actually use the running controller from a
+client is to move the FURL files :file:`ipcontroller-mec.furl` and
+:file:`ipcontroller-tc.furl` from ``host0`` to the host where the clients will
+be run. If these file are put into the :file:`~/.ipython/security` directory
+of the client's host, they will be found automatically. Otherwise, the full
+path to them has to be passed to the client's constructor.
+
+Using :command:`ipcluster`
+==========================
+
+The :command:`ipcluster` command provides a simple way of starting a
+controller and engines in the following situations:
+
+1. When the controller and engines are all run on localhost. This is useful
+   for testing or running on a multicore computer.
+2. When engines are started using the :command:`mpirun` command that comes 
+   with most MPI [MPI]_ implementations
+3. When engines are started using the PBS [PBS]_ batch system.
+4. When the controller is started on localhost and the engines are started on 
+   remote nodes using :command:`ssh`.
+
+.. note::
+
+    It is also possible for advanced users to add support to
+    :command:`ipcluster` for starting controllers and engines using other
+    methods (like Sun's Grid Engine for example).
+
+.. note::
+
+    Currently :command:`ipcluster` requires that the
+    :file:`~/.ipython/security` directory live on a shared filesystem that is
+    seen by both the controller and engines. If you don't have a shared file
+    system you will need to use :command:`ipcontroller` and
+    :command:`ipengine` directly.  This constraint can be relaxed if you are 
+    using the :command:`ssh` method to start the cluster.
+
+Underneath the hood, :command:`ipcluster` just uses :command:`ipcontroller`
+and :command:`ipengine` to perform the steps described above.
+
+Using :command:`ipcluster` in local mode
+----------------------------------------
+
+To start one controller and 4 engines on localhost, just do::
+
+    $ ipcluster local -n 4
+
+To see other command line options for the local mode, do::
+
+    $ ipcluster local -h
+
+Using :command:`ipcluster` in mpiexec/mpirun mode
+-------------------------------------------------
+
+The mpiexec/mpirun mode is useful if you:
+
+1. Have MPI installed.
+2. Your systems are configured to use the :command:`mpiexec` or
+   :command:`mpirun` commands to start MPI processes.
+
+.. note::
+
+    The preferred command to use is :command:`mpiexec`.  However, we also 
+    support :command:`mpirun` for backwards compatibility.  The underlying
+    logic used is exactly the same, the only difference being the name of the 
+    command line program that is called.
+
+If these are satisfied, you can start an IPython cluster using::
+
+    $ ipcluster mpiexec -n 4
+
+This does the following:
+
+1. Starts the IPython controller on current host.
+2. Uses :command:`mpiexec` to start 4 engines.
+
+On newer MPI implementations (such as OpenMPI), this will work even if you
+don't make any calls to MPI or call :func:`MPI_Init`. However, older MPI
+implementations actually require each process to call :func:`MPI_Init` upon
+starting. The easiest way of having this done is to install the mpi4py
+[mpi4py]_ package and then call ipcluster with the ``--mpi`` option::
+
+    $ ipcluster mpiexec -n 4 --mpi=mpi4py
+
+Unfortunately, even this won't work for some MPI implementations. If you are
+having problems with this, you will likely have to use a custom Python
+executable that itself calls :func:`MPI_Init` at the appropriate time.
+Fortunately, mpi4py comes with such a custom Python executable that is easy to
+install and use. However, this custom Python executable approach will not work
+with :command:`ipcluster` currently.
+
+Additional command line options for this mode can be found by doing::
+
+    $ ipcluster mpiexec -h
+
+More details on using MPI with IPython can be found :ref:`here <parallelmpi>`.
+
+
+Using :command:`ipcluster` in PBS mode
+--------------------------------------
+
+The PBS mode uses the Portable Batch System [PBS]_ to start the engines. To
+use this mode, you first need to create a PBS script template that will be
+used to start the engines. Here is a sample PBS script template:
+
+.. sourcecode:: bash
+
+    #PBS -N ipython
+    #PBS -j oe
+    #PBS -l walltime=00:10:00
+    #PBS -l nodes=${n/4}:ppn=4
+    #PBS -q parallel
+
+    cd $$PBS_O_WORKDIR
+    export PATH=$$HOME/usr/local/bin
+    export PYTHONPATH=$$HOME/usr/local/lib/python2.4/site-packages
+    /usr/local/bin/mpiexec -n ${n} ipengine --logfile=$$PBS_O_WORKDIR/ipengine
+
+There are a few important points about this template:
+
+1. This template will be rendered at runtime using IPython's :mod:`Itpl` 
+   template engine.
+
+2. Instead of putting in the actual number of engines, use the notation
+   ``${n}`` to indicate the number of engines to be started. You can also uses
+   expressions like ``${n/4}`` in the template to indicate the number of 
+   nodes.
+
+3. Because ``$`` is a special character used by the template engine, you must  
+   escape any ``$`` by using ``$$``.  This is important when referring to 
+   environment variables in the template.
+
+4. Any options to :command:`ipengine` should be given in the batch script 
+   template.
+
+5. Depending on the configuration of you system, you may have to set 
+   environment variables in the script template.
+
+Once you have created such a script, save it with a name like
+:file:`pbs.template`. Now you are ready to start your job::
+
+    $ ipcluster pbs -n 128 --pbs-script=pbs.template
+
+Additional command line options for this mode can be found by doing::
+
+    $ ipcluster pbs -h
+
+Using :command:`ipcluster` in SSH mode
+--------------------------------------
+
+The SSH mode uses :command:`ssh` to execute :command:`ipengine` on remote
+nodes and the :command:`ipcontroller` on localhost.
+
+When using using this mode it highly recommended that you have set up SSH keys
+and are using ssh-agent [SSH]_ for password-less logins.
+
+To use this mode you need a python file describing the cluster, here is an
+example of such a "clusterfile":
+
+.. sourcecode:: python
+    
+    send_furl = True
+    engines = { 'host1.example.com' : 2,
+                'host2.example.com' : 5,
+                'host3.example.com' : 1,
+                'host4.example.com' : 8 }
+
+Since this is a regular python file usual python syntax applies. Things to
+note:
+
+* The `engines` dict, where the keys is the host we want to run engines on and
+  the value is the number of engines to run on that host.
+* send_furl can either be `True` or `False`, if `True` it will copy over the
+  furl needed for :command:`ipengine` to each host.
+
+The ``--clusterfile`` command line option lets you specify the file to use for
+the cluster definition. Once you have your cluster file and you can
+:command:`ssh` into the remote hosts with out an password you are ready to
+start your cluster like so:
+
+.. sourcecode:: bash 
+
+   $ ipcluster ssh --clusterfile /path/to/my/clusterfile.py
+
+
+Two helper shell scripts are used to start and stop :command:`ipengine` on
+remote hosts:
+
+* sshx.sh 
+* engine_killer.sh
+
+Defaults for both of these are contained in the source code for
+:command:`ipcluster`. The default scripts are written to a local file in a
+tmep directory and then copied to a temp directory on the remote host and
+executed from there. On most Unix, Linux and OS X systems this is /tmp.
+
+The default sshx.sh is the following:
+
+.. sourcecode:: bash
+
+   #!/bin/sh
+   "$@" &> /dev/null &
+   echo $!
+
+If you want to use a custom sshx.sh script you need to use the ``--sshx``
+option and specify the file to use. Using a custom sshx.sh file could be
+helpful when you need to setup the environment on the remote host before
+executing :command:`ipengine`.
+
+For a detailed options list:
+
+.. sourcecode:: bash
+
+   $ ipcluster ssh -h
+
+Current limitations of the SSH mode of :command:`ipcluster` are:
+
+* Untested on Windows. Would require a working :command:`ssh` on Windows.
+  Also, we are using shell scripts to setup and execute commands on remote
+  hosts.
+* :command:`ipcontroller` is started on localhost, with no option to start it
+  on a remote node.
+
+Using the :command:`ipcontroller` and :command:`ipengine` commands
+==================================================================
+
+It is also possible to use the :command:`ipcontroller` and :command:`ipengine`
+commands to start your controller and engines. This approach gives you full
+control over all aspects of the startup process.
+
+Starting the controller and engine on your local machine
+--------------------------------------------------------
+
+To use :command:`ipcontroller` and :command:`ipengine` to start things on your
+local machine, do the following.
+
+First start the controller::
+
+	$ ipcontroller
+	
+Next, start however many instances of the engine you want using (repeatedly)
+the command::
+
+	$ ipengine
+
+The engines should start and automatically connect to the controller using the
+FURL files in :file:`~./ipython/security`. You are now ready to use the
+controller and engines from IPython.
+
+.. warning:: 
+	
+	The order of the above operations is very important.  You *must*
+ 	start the controller before the engines, since the engines connect 
+	to the controller as they get started.
+
+.. note::
+
+    On some platforms (OS X), to put the controller and engine into the
+    background you may need to give these commands in the form ``(ipcontroller
+    &)`` and ``(ipengine &)`` (with the parentheses) for them to work
+    properly.
+
+Starting the controller and engines on different hosts
+------------------------------------------------------
+
+When the controller and engines are running on different hosts, things are
+slightly more complicated, but the underlying ideas are the same:
+
+1. Start the controller on a host using :command:`ipcontroller`.
+2. Copy :file:`ipcontroller-engine.furl` from :file:`~./ipython/security` on
+   the controller's host to the host where the engines will run.
+3. Use :command:`ipengine` on the engine's hosts to start the engines.
+
+The only thing you have to be careful of is to tell :command:`ipengine` where
+the :file:`ipcontroller-engine.furl` file is located. There are two ways you
+can do this:
+
+* Put :file:`ipcontroller-engine.furl` in the :file:`~./ipython/security`
+  directory on the engine's host, where it will be found automatically.
+* Call :command:`ipengine` with the ``--furl-file=full_path_to_the_file``
+  flag.
+
+The ``--furl-file`` flag works like this::
+
+    $ ipengine --furl-file=/path/to/my/ipcontroller-engine.furl
+
+.. note::
+    
+    If the controller's and engine's hosts all have a shared file system  
+    (:file:`~./ipython/security` is the same on all of them), then things
+    will just work!
+
+Make FURL files persistent
+---------------------------
+
+At fist glance it may seem that that managing the FURL files is a bit
+annoying. Going back to the house and key analogy, copying the FURL around
+each time you start the controller is like having to make a new key every time
+you want to unlock the door and enter your house. As with your house, you want
+to be able to create the key (or FURL file) once, and then simply use it at
+any point in the future.
+
+This is possible, but before you do this, you **must** remove any old FURL
+files in the :file:`~/.ipython/security` directory.
+
+.. warning::
+
+    You **must** remove old FURL files before using persistent FURL files.
+
+Then, The only thing you have to do is decide what ports the controller will
+listen on for the engines and clients. This is done as follows::
+
+    $ ipcontroller -r --client-port=10101 --engine-port=10102
+
+These options also work with all of the various modes of
+:command:`ipcluster`::
+
+    $ ipcluster local -n 2 -r --client-port=10101 --engine-port=10102
+
+Then, just copy the furl files over the first time and you are set. You can
+start and stop the controller and engines any many times as you want in the
+future, just make sure to tell the controller to use the *same* ports.
+
+.. note::
+
+    You may ask the question: what ports does the controller listen on if you
+    don't tell is to use specific ones? The default is to use high random port
+    numbers. We do this for two reasons: i) to increase security through
+    obscurity and ii) to multiple controllers on a given host to start and
+    automatically use different ports.
+
+Log files
+---------
+
+All of the components of IPython have log files associated with them.
+These log files can be extremely useful in debugging problems with
+IPython and can be found in the directory :file:`~/.ipython/log`.  Sending
+the log files to us will often help us to debug any problems.
+
+
+.. [PBS] Portable Batch System.  http://www.openpbs.org/
+.. [SSH] SSH-Agent http://en.wikipedia.org/wiki/Ssh-agent
+
+

docs/source/parallelz/parallel_security.txt

@@ -0,0 +1,366 b''
+.. _parallelsecurity:
+
+===========================
+Security details of IPython
+===========================
+
+IPython's :mod:`IPython.kernel` package exposes the full power of the Python
+interpreter over a TCP/IP network for the purposes of parallel computing. This
+feature brings up the important question of IPython's security model. This
+document gives details about this model and how it is implemented in IPython's
+architecture.
+
+Processs and network topology
+=============================
+
+To enable parallel computing, IPython has a number of different processes that
+run. These processes are discussed at length in the IPython documentation and
+are summarized here:
+
+* The IPython *engine*.  This process is a full blown Python
+  interpreter in which user code is executed.  Multiple
+  engines are started to make parallel computing possible.
+* The IPython *controller*.  This process manages a set of 
+  engines, maintaining a queue for each and presenting
+  an asynchronous interface to the set of engines.
+* The IPython *client*.  This process is typically an
+  interactive Python process that is used to coordinate the
+  engines to get a parallel computation done.
+
+Collectively, these three processes are called the IPython *kernel*.
+
+These three processes communicate over TCP/IP connections with a well defined
+topology. The IPython controller is the only process that listens on TCP/IP
+sockets. Upon starting, an engine connects to a controller and registers
+itself with the controller. These engine/controller TCP/IP connections persist
+for the lifetime of each engine.
+
+The IPython client also connects to the controller using one or more TCP/IP
+connections. These connections persist for the lifetime of the client only.
+
+A given IPython controller and set of engines typically has a relatively short
+lifetime. Typically this lifetime corresponds to the duration of a single
+parallel simulation performed by a single user. Finally, the controller,
+engines and client processes typically execute with the permissions of that
+same user. More specifically, the controller and engines are *not* executed as
+root or with any other superuser permissions.
+
+Application logic
+=================
+
+When running the IPython kernel to perform a parallel computation, a user
+utilizes the IPython client to send Python commands and data through the
+IPython controller to the IPython engines, where those commands are executed
+and the data processed. The design of IPython ensures that the client is the
+only access point for the capabilities of the engines. That is, the only way
+of addressing the engines is through a client.
+
+A user can utilize the client to instruct the IPython engines to execute
+arbitrary Python commands. These Python commands can include calls to the
+system shell, access the filesystem, etc., as required by the user's
+application code. From this perspective, when a user runs an IPython engine on
+a host, that engine has the same capabilities and permissions as the user
+themselves (as if they were logged onto the engine's host with a terminal).
+
+Secure network connections
+==========================
+
+Overview
+--------
+
+All TCP/IP connections between the client and controller as well as the
+engines and controller are fully encrypted and authenticated. This section
+describes the details of the encryption and authentication approached used
+within IPython.
+
+IPython uses the Foolscap network protocol [Foolscap]_ for all communications
+between processes. Thus, the details of IPython's security model are directly
+related to those of Foolscap. Thus, much of the following discussion is
+actually just a discussion of the security that is built in to Foolscap.
+
+Encryption
+----------
+
+For encryption purposes, IPython and Foolscap use the well known Secure Socket
+Layer (SSL) protocol [RFC5246]_. We use the implementation of this protocol
+provided by the OpenSSL project through the pyOpenSSL [pyOpenSSL]_ Python
+bindings to OpenSSL.
+
+Authentication
+--------------
+
+IPython clients and engines must also authenticate themselves with the
+controller. This is handled in a capabilities based security model
+[Capability]_. In this model, the controller creates a strong cryptographic
+key or token that represents each set of capability that the controller
+offers. Any party who has this key and presents it to the controller has full
+access to the corresponding capabilities of the controller. This model is
+analogous to using a physical key to gain access to physical items
+(capabilities) behind a locked door.
+
+For a capabilities based authentication system to prevent unauthorized access,
+two things must be ensured:
+
+* The keys must be cryptographically strong.  Otherwise attackers could gain 
+  access by a simple brute force key guessing attack.
+* The actual keys must be distributed only to authorized parties.
+
+The keys in Foolscap are called Foolscap URL's or FURLs. The following section
+gives details about how these FURLs are created in Foolscap. The IPython
+controller creates a number of FURLs for different purposes:
+
+* One FURL that grants IPython engines access to the controller.  Also
+  implicit in this access is permission to execute code sent by an
+  authenticated IPython client.
+* Two or more FURLs that grant IPython clients access to the controller.
+  Implicit in this access is permission to give the controller's engine code 
+  to execute.
+
+Upon starting, the controller creates these different FURLS and writes them
+files in the user-read-only directory :file:`$HOME/.ipython/security`. Thus,
+only the user who starts the controller has access to the FURLs.
+
+For an IPython client or engine to authenticate with a controller, it must
+present the appropriate FURL to the controller upon connecting. If the
+FURL matches what the controller expects for a given capability, access is
+granted. If not, access is denied. The exchange of FURLs is done after
+encrypted communications channels have been established to prevent attackers
+from capturing them.
+
+.. note::
+
+    The FURL is similar to an unsigned private key in SSH.
+
+Details of the Foolscap handshake
+---------------------------------
+
+In this section we detail the precise security handshake that takes place at
+the beginning of any network connection in IPython. For the purposes of this
+discussion, the SERVER is the IPython controller process and the CLIENT is the
+IPython engine or client process.
+
+Upon starting, all IPython processes do the following:
+
+1. Create a public key x509 certificate (ISO/IEC 9594).
+2. Create a hash of the contents of the certificate using the SHA-1 algorithm.
+   The base-32 encoded version of this hash is saved by the process as its 
+   process id (actually in Foolscap, this is the Tub id, but here refer to
+   it as the process id).
+
+Upon starting, the IPython controller also does the following:
+
+1. Save the x509 certificate to disk in a secure location. The CLIENT
+   certificate is never saved to disk.
+2. Create a FURL for each capability that the controller has.  There are
+   separate capabilities the controller offers for clients and engines.  The
+   FURL is created using:  a) the process id of the SERVER, b) the IP 
+   address and port the SERVER is listening on and c) a 160 bit,    
+   cryptographically secure string that represents the capability (the
+   "capability id").
+3. The FURLs are saved to disk in a secure location on the SERVER's host.
+
+For a CLIENT to be able to connect to the SERVER and access a capability of
+that SERVER, the CLIENT must have knowledge of the FURL for that SERVER's
+capability. This typically requires that the file containing the FURL be
+moved from the SERVER's host to the CLIENT's host. This is done by the end
+user who started the SERVER and wishes to have a CLIENT connect to the SERVER.
+
+When a CLIENT connects to the SERVER, the following handshake protocol takes
+place:
+
+1. The CLIENT tells the SERVER what process (or Tub) id it expects the SERVER
+   to have.
+2. If the SERVER has that process id, it notifies the CLIENT that it will now 
+   enter encrypted mode.  If the SERVER has a different id, the SERVER aborts.
+3. Both CLIENT and SERVER initiate the SSL handshake protocol.
+4. Both CLIENT and SERVER request the certificate of their peer and verify
+   that certificate.  If this succeeds, all further communications are 
+   encrypted.
+5. Both CLIENT and SERVER send a hello block containing connection parameters
+   and their process id.
+6. The CLIENT and SERVER check that their peer's stated process id matches the
+   hash of the x509 certificate the peer presented.  If not, the connection is
+   aborted.
+7. The CLIENT verifies that the SERVER's stated id matches the id of the
+   SERVER the CLIENT is intending to connect to.  If not, the connection is 
+   aborted.
+8. The CLIENT and SERVER elect a master who decides on the final connection
+   parameters.
+
+The public/private key pair associated with each process's x509 certificate
+are completely hidden from this handshake protocol. There are however, used
+internally by OpenSSL as part of the SSL handshake protocol. Each process
+keeps their own private key hidden and sends its peer only the public key
+(embedded in the certificate).
+
+Finally, when the CLIENT requests access to a particular SERVER capability,
+the following happens:
+
+1. The CLIENT asks the SERVER for access to a capability by presenting that 
+   capabilities id.
+2. If the SERVER has a capability with that id, access is granted. If not,
+   access is not granted.
+3. Once access has been gained, the CLIENT can use the capability.
+
+Specific security vulnerabilities
+=================================
+
+There are a number of potential security vulnerabilities present in IPython's
+architecture. In this section we discuss those vulnerabilities and detail how
+the security architecture described above prevents them from being exploited.
+
+Unauthorized clients
+--------------------
+
+The IPython client can instruct the IPython engines to execute arbitrary
+Python code with the permissions of the user who started the engines. If an
+attacker were able to connect their own hostile IPython client to the IPython
+controller, they could instruct the engines to execute code.
+
+This attack is prevented by the capabilities based client authentication
+performed after the encrypted channel has been established. The relevant
+authentication information is encoded into the FURL that clients must
+present to gain access to the IPython controller. By limiting the distribution
+of those FURLs, a user can grant access to only authorized persons.
+
+It is highly unlikely that a client FURL could be guessed by an attacker
+in a brute force guessing attack. A given instance of the IPython controller
+only runs for a relatively short amount of time (on the order of hours). Thus
+an attacker would have only a limited amount of time to test a search space of
+size 2**320. Furthermore, even if a controller were to run for a longer amount
+of time, this search space is quite large (larger for instance than that of
+typical username/password pair).
+
+Unauthorized engines
+--------------------
+
+If an attacker were able to connect a hostile engine to a user's controller,
+the user might unknowingly send sensitive code or data to the hostile engine.
+This attacker's engine would then have full access to that code and data.
+
+This type of attack is prevented in the same way as the unauthorized client
+attack, through the usage of the capabilities based authentication scheme.
+
+Unauthorized controllers
+------------------------
+
+It is also possible that an attacker could try to convince a user's IPython
+client or engine to connect to a hostile IPython controller. That controller
+would then have full access to the code and data sent between the IPython
+client and the IPython engines.
+
+Again, this attack is prevented through the FURLs, which ensure that a
+client or engine connects to the correct controller. It is also important to
+note that the FURLs also encode the IP address and port that the
+controller is listening on, so there is little chance of mistakenly connecting
+to a controller running on a different IP address and port.
+
+When starting an engine or client, a user must specify which FURL to use
+for that connection. Thus, in order to introduce a hostile controller, the
+attacker must convince the user to use the FURLs associated with the
+hostile controller. As long as a user is diligent in only using FURLs from
+trusted sources, this attack is not possible.
+
+Other security measures
+=======================
+
+A number of other measures are taken to further limit the security risks
+involved in running the IPython kernel.
+
+First, by default, the IPython controller listens on random port numbers.
+While this can be overridden by the user, in the default configuration, an
+attacker would have to do a port scan to even find a controller to attack.
+When coupled with the relatively short running time of a typical controller
+(on the order of hours), an attacker would have to work extremely hard and
+extremely *fast* to even find a running controller to attack.
+
+Second, much of the time, especially when run on supercomputers or clusters,
+the controller is running behind a firewall. Thus, for engines or client to
+connect to the controller:
+
+* The different processes have to all be behind the firewall.
+
+or:
+
+* The user has to use SSH port forwarding to tunnel the
+  connections through the firewall.
+   
+In either case, an attacker is presented with addition barriers that prevent
+attacking or even probing the system.
+
+Summary
+=======
+
+IPython's architecture has been carefully designed with security in mind. The
+capabilities based authentication model, in conjunction with the encrypted
+TCP/IP channels, address the core potential vulnerabilities in the system,
+while still enabling user's to use the system in open networks.
+
+Other questions
+===============
+
+About keys
+----------
+
+Can you clarify the roles of the certificate and its keys versus the FURL,
+which is also called a key?
+
+The certificate created by IPython processes is a standard public key x509
+certificate, that is used by the SSL handshake protocol to setup encrypted
+channel between the controller and the IPython engine or client. This public
+and private key associated with this certificate are used only by the SSL
+handshake protocol in setting up this encrypted channel.
+
+The FURL serves a completely different and independent purpose from the
+key pair associated with the certificate. When we refer to a FURL as a
+key, we are using the word "key" in the capabilities based security model
+sense. This has nothing to do with "key" in the public/private key sense used
+in the SSL protocol.
+
+With that said the FURL is used as an cryptographic key, to grant
+IPython engines and clients access to particular capabilities that the
+controller offers.
+
+Self signed certificates
+------------------------
+
+Is the controller creating a self-signed certificate? Is this created for per
+instance/session, one-time-setup or each-time the controller is started?
+
+The Foolscap network protocol, which handles the SSL protocol details, creates
+a self-signed x509 certificate using OpenSSL for each IPython process. The
+lifetime of the certificate is handled differently for the IPython controller
+and the engines/client.
+
+For the IPython engines and client, the certificate is only held in memory for
+the lifetime of its process. It is never written to disk.
+
+For the controller, the certificate can be created anew each time the
+controller starts or it can be created once and reused each time the
+controller starts. If at any point, the certificate is deleted, a new one is
+created the next time the controller starts.
+
+SSL private key
+---------------
+
+How the private key (associated with the certificate) is distributed?
+
+In the usual implementation of the SSL protocol, the private key is never
+distributed. We follow this standard always.
+
+SSL versus Foolscap authentication
+----------------------------------
+
+Many SSL connections only perform one sided authentication (the server to the
+client). How is the client authentication in IPython's system related to SSL
+authentication?
+
+We perform a two way SSL handshake in which both parties request and verify
+the certificate of their peer. This mutual authentication is handled by the
+SSL handshake and is separate and independent from the additional
+authentication steps that the CLIENT and SERVER perform after an encrypted
+channel is established.
+
+.. [RFC5246] <http://tools.ietf.org/html/rfc5246>
+
+

docs/source/parallelz/parallel_task.txt

@@ -0,0 +1,121 b''
+.. _paralleltask:
+
+==========================
+The IPython task interface
+==========================
+
+The task interface to the controller presents the engines as a fault tolerant,
+dynamic load-balanced system or workers. Unlike the multiengine interface, in
+the task interface, the user have no direct access to individual engines. In
+some ways, this interface is simpler, but in other ways it is more powerful.
+
+Best of all the user can use both of these interfaces running at the same time
+to take advantage or both of their strengths. When the user can break up the
+user's work into segments that do not depend on previous execution, the task
+interface is ideal. But it also has more power and flexibility, allowing the
+user to guide the distribution of jobs, without having to assign tasks to
+engines explicitly.
+
+Starting the IPython controller and engines
+===========================================
+
+To follow along with this tutorial, you will need to start the IPython
+controller and four IPython engines. The simplest way of doing this is to use
+the :command:`ipcluster` command::
+
+	$ ipcluster local -n 4
+	
+For more detailed information about starting the controller and engines, see
+our :ref:`introduction <ip1par>` to using IPython for parallel computing.
+
+Creating a ``TaskClient`` instance
+=========================================
+
+The first step is to import the IPython :mod:`IPython.kernel.client` module
+and then create a :class:`TaskClient` instance:
+
+.. sourcecode:: ipython
+
+	In [1]: from IPython.kernel import client
+	
+	In [2]: tc = client.TaskClient()
+
+This form assumes that the :file:`ipcontroller-tc.furl` is in the
+:file:`~./ipython/security` directory on the client's host. If not, the
+location of the FURL file must be given as an argument to the
+constructor:
+
+.. sourcecode:: ipython
+
+    In [2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
+
+Quick and easy parallelism
+==========================
+
+In many cases, you simply want to apply a Python function to a sequence of
+objects, but *in parallel*. Like the multiengine interface, the task interface
+provides two simple ways of accomplishing this: a parallel version of
+:func:`map` and ``@parallel`` function decorator. However, the verions in the
+task interface have one important difference: they are dynamically load
+balanced. Thus, if the execution time per item varies significantly, you
+should use the versions in the task interface.
+
+Parallel map
+------------
+
+The parallel :meth:`map` in the task interface is similar to that in the
+multiengine interface:
+
+.. sourcecode:: ipython
+
+	In [63]: serial_result = map(lambda x:x**10, range(32))
+
+	In [64]: parallel_result = tc.map(lambda x:x**10, range(32))
+
+	In [65]: serial_result==parallel_result
+	Out[65]: True
+
+Parallel function decorator
+---------------------------
+
+Parallel functions are just like normal function, but they can be called on
+sequences and *in parallel*. The multiengine interface provides a decorator
+that turns any Python function into a parallel function:
+
+.. sourcecode:: ipython
+
+    In [10]: @tc.parallel()
+       ....: def f(x):
+       ....:     return 10.0*x**4
+       ....: 
+
+    In [11]: f(range(32))    # this is done in parallel
+    Out[11]: 
+    [0.0,10.0,160.0,...]
+
+More details
+============
+
+The :class:`TaskClient` has many more powerful features that allow quite a bit
+of flexibility in how tasks are defined and run. The next places to look are
+in the following classes:
+
+* :class:`IPython.kernel.client.TaskClient`
+* :class:`IPython.kernel.client.StringTask`
+* :class:`IPython.kernel.client.MapTask`
+
+The following is an overview of how to use these classes together:
+
+1. Create a :class:`TaskClient`.
+2. Create one or more instances of :class:`StringTask` or :class:`MapTask`
+   to define your tasks.
+3. Submit your tasks to using the :meth:`run` method of your
+   :class:`TaskClient` instance.
+4. Use :meth:`TaskClient.get_task_result` to get the results of the
+   tasks.
+
+We are in the process of developing more detailed information about the task
+interface. For now, the docstrings of the :class:`TaskClient`,
+:class:`StringTask` and :class:`MapTask` classes should be consulted.
+
+

docs/source/parallelz/parallel_winhpc.txt

@@ -0,0 +1,333 b''
+============================================
+Getting started with Windows HPC Server 2008
+============================================
+
+Introduction
+============
+
+The Python programming language is an increasingly popular language for
+numerical computing. This is due to a unique combination of factors. First,
+Python is a high-level and *interactive* language that is well matched to
+interactive numerical work. Second, it is easy (often times trivial) to
+integrate legacy C/C++/Fortran code into Python. Third, a large number of
+high-quality open source projects provide all the needed building blocks for
+numerical computing: numerical arrays (NumPy), algorithms (SciPy), 2D/3D
+Visualization (Matplotlib, Mayavi, Chaco), Symbolic Mathematics (Sage, Sympy)
+and others.
+
+The IPython project is a core part of this open-source toolchain and is
+focused on creating a comprehensive environment for interactive and
+exploratory computing in the Python programming language. It enables all of
+the above tools to be used interactively and consists of two main components:
+
+* An enhanced interactive Python shell with support for interactive plotting
+  and visualization.
+* An architecture for interactive parallel computing.
+
+With these components, it is possible to perform all aspects of a parallel
+computation interactively. This type of workflow is particularly relevant in
+scientific and numerical computing where algorithms, code and data are
+continually evolving as the user/developer explores a problem. The broad
+treads in computing (commodity clusters, multicore, cloud computing, etc.)
+make these capabilities of IPython particularly relevant.
+
+While IPython is a cross platform tool, it has particularly strong support for
+Windows based compute clusters running Windows HPC Server 2008. This document
+describes how to get started with IPython on Windows HPC Server 2008. The
+content and emphasis here is practical: installing IPython, configuring
+IPython to use the Windows job scheduler and running example parallel programs
+interactively. A more complete description of IPython's parallel computing
+capabilities can be found in IPython's online documentation
+(http://ipython.scipy.org/moin/Documentation).
+
+Setting up your Windows cluster
+===============================
+
+This document assumes that you already have a cluster running Windows
+HPC Server 2008. Here is a broad overview of what is involved with setting up
+such a cluster:
+
+1. Install Windows Server 2008 on the head and compute nodes in the cluster.
+2. Setup the network configuration on each host. Each host should have a 
+   static IP address.
+3. On the head node, activate the "Active Directory Domain Services" role 
+   and make the head node the domain controller.
+4. Join the compute nodes to the newly created Active Directory (AD) domain.
+5. Setup user accounts in the domain with shared home directories.
+6. Install the HPC Pack 2008 on the head node to create a cluster.
+7. Install the HPC Pack 2008 on the compute nodes.
+
+More details about installing and configuring Windows HPC Server 2008 can be
+found on the Windows HPC Home Page (http://www.microsoft.com/hpc). Regardless
+of what steps you follow to set up your cluster, the remainder of this
+document will assume that:
+
+* There are domain users that can log on to the AD domain and submit jobs
+  to the cluster scheduler.
+* These domain users have shared home directories. While shared home
+  directories are not required to use IPython, they make it much easier to
+  use IPython.
+
+Installation of IPython and its dependencies
+============================================
+
+IPython and all of its dependencies are freely available and open source.
+These packages provide a powerful and cost-effective approach to numerical and
+scientific computing on Windows. The following dependencies are needed to run
+IPython on Windows:
+
+* Python 2.5 or 2.6 (http://www.python.org)
+* pywin32 (http://sourceforge.net/projects/pywin32/)
+* PyReadline (https://launchpad.net/pyreadline)
+* zope.interface and Twisted (http://twistedmatrix.com)
+* Foolcap (http://foolscap.lothar.com/trac)
+* pyOpenSSL (https://launchpad.net/pyopenssl)
+* IPython (http://ipython.scipy.org)
+
+In addition, the following dependencies are needed to run the demos described
+in this document.
+
+* NumPy and SciPy (http://www.scipy.org)
+* wxPython (http://www.wxpython.org)
+* Matplotlib (http://matplotlib.sourceforge.net/)
+
+The easiest way of obtaining these dependencies is through the Enthought
+Python Distribution (EPD) (http://www.enthought.com/products/epd.php). EPD is
+produced by Enthought, Inc. and contains all of these packages and others in a
+single installer and is available free for academic users. While it is also
+possible to download and install each package individually, this is a tedious
+process. Thus, we highly recommend using EPD to install these packages on
+Windows.
+
+Regardless of how you install the dependencies, here are the steps you will
+need to follow:
+
+1. Install all of the packages listed above, either individually or using EPD
+   on the head node, compute nodes and user workstations.
+
+2. Make sure that :file:`C:\\Python25` and :file:`C:\\Python25\\Scripts` are
+   in the system :envvar:`%PATH%` variable on each node.
+
+3. Install the latest development version of IPython. This can be done by
+   downloading the the development version from the IPython website
+   (http://ipython.scipy.org) and following the installation instructions.
+
+Further details about installing IPython or its dependencies can be found in
+the online IPython documentation (http://ipython.scipy.org/moin/Documentation)
+Once you are finished with the installation, you can try IPython out by
+opening a Windows Command Prompt and typing ``ipython``. This will
+start IPython's interactive shell and you should see something like the
+following screenshot:
+
+.. image:: ipython_shell.*
+
+Starting an IPython cluster
+===========================
+
+To use IPython's parallel computing capabilities, you will need to start an
+IPython cluster. An IPython cluster consists of one controller and multiple
+engines:
+
+IPython controller
+    The IPython controller manages the engines and acts as a gateway between 
+    the engines and the client, which runs in the user's interactive IPython
+    session. The controller is started using the :command:`ipcontroller`
+    command.
+
+IPython engine
+    IPython engines run a user's Python code in parallel on the compute nodes. 
+    Engines are starting using the :command:`ipengine` command.
+
+Once these processes are started, a user can run Python code interactively and
+in parallel on the engines from within the IPython shell using an appropriate
+client. This includes the ability to interact with, plot and visualize data
+from the engines.
+
+IPython has a command line program called :command:`ipcluster` that automates
+all aspects of starting the controller and engines on the compute nodes.
+:command:`ipcluster` has full support for the Windows HPC job scheduler,
+meaning that :command:`ipcluster` can use this job scheduler to start the
+controller and engines. In our experience, the Windows HPC job scheduler is
+particularly well suited for interactive applications, such as IPython. Once
+:command:`ipcluster` is configured properly, a user can start an IPython
+cluster from their local workstation almost instantly, without having to log
+on to the head node (as is typically required by Unix based job schedulers).
+This enables a user to move seamlessly between serial and parallel
+computations.
+
+In this section we show how to use :command:`ipcluster` to start an IPython
+cluster using the Windows HPC Server 2008 job scheduler. To make sure that
+:command:`ipcluster` is installed and working properly, you should first try
+to start an IPython cluster on your local host. To do this, open a Windows
+Command Prompt and type the following command::
+
+    ipcluster start -n 2
+
+You should see a number of messages printed to the screen, ending with 
+"IPython cluster: started". The result should look something like the following
+screenshot:
+
+.. image:: ipcluster_start.*
+
+At this point, the controller and two engines are running on your local host.
+This configuration is useful for testing and for situations where you want to
+take advantage of multiple cores on your local computer. 
+
+Now that we have confirmed that :command:`ipcluster` is working properly, we
+describe how to configure and run an IPython cluster on an actual compute
+cluster running Windows HPC Server 2008. Here is an outline of the needed
+steps:
+
+1. Create a cluster profile using: ``ipcluster create -p mycluster``
+
+2. Edit configuration files in the directory :file:`.ipython\\cluster_mycluster`
+
+3. Start the cluster using: ``ipcluser start -p mycluster -n 32``
+
+Creating a cluster profile
+--------------------------
+
+In most cases, you will have to create a cluster profile to use IPython on a
+cluster. A cluster profile is a name (like "mycluster") that is associated
+with a particular cluster configuration. The profile name is used by
+:command:`ipcluster` when working with the cluster.
+
+Associated with each cluster profile is a cluster directory. This cluster
+directory is a specially named directory (typically located in the
+:file:`.ipython` subdirectory of your home directory) that contains the
+configuration files for a particular cluster profile, as well as log files and
+security keys. The naming convention for cluster directories is:
+:file:`cluster_<profile name>`. Thus, the cluster directory for a profile named
+"foo" would be :file:`.ipython\\cluster_foo`.
+
+To create a new cluster profile (named "mycluster") and the associated cluster
+directory, type the following command at the Windows Command Prompt::
+
+    ipcluster create -p mycluster
+
+The output of this command is shown in the screenshot below. Notice how
+:command:`ipcluster` prints out the location of the newly created cluster
+directory.
+
+.. image:: ipcluster_create.*
+
+Configuring a cluster profile
+-----------------------------
+
+Next, you will need to configure the newly created cluster profile by editing
+the following configuration files in the cluster directory:
+
+* :file:`ipcluster_config.py`
+* :file:`ipcontroller_config.py`
+* :file:`ipengine_config.py`
+
+When :command:`ipcluster` is run, these configuration files are used to 
+determine how the engines and controller will be started. In most cases,
+you will only have to set a few of the attributes in these files.
+
+To configure :command:`ipcluster` to use the Windows HPC job scheduler, you
+will need to edit the following attributes in the file
+:file:`ipcluster_config.py`::
+
+    # Set these at the top of the file to tell ipcluster to use the
+    # Windows HPC job scheduler.
+    c.Global.controller_launcher = \
+        'IPython.kernel.launcher.WindowsHPCControllerLauncher'
+    c.Global.engine_launcher = \
+        'IPython.kernel.launcher.WindowsHPCEngineSetLauncher'
+
+    # Set these to the host name of the scheduler (head node) of your cluster.
+    c.WindowsHPCControllerLauncher.scheduler = 'HEADNODE'
+    c.WindowsHPCEngineSetLauncher.scheduler = 'HEADNODE'
+
+There are a number of other configuration attributes that can be set, but
+in most cases these will be sufficient to get you started.
+
+.. warning::
+    If any of your configuration attributes involve specifying the location
+    of shared directories or files, you must make sure that you use UNC paths
+    like :file:`\\\\host\\share`. It is also important that you specify
+    these paths using raw Python strings: ``r'\\host\share'`` to make sure
+    that the backslashes are properly escaped.
+
+Starting the cluster profile
+----------------------------
+
+Once a cluster profile has been configured, starting an IPython cluster using
+the profile is simple::
+
+    ipcluster start -p mycluster -n 32
+
+The ``-n`` option tells :command:`ipcluster` how many engines to start (in
+this case 32). Stopping the cluster is as simple as typing Control-C.
+
+Using the HPC Job Manager
+-------------------------
+
+When ``ipcluster start`` is run the first time, :command:`ipcluster` creates
+two XML job description files in the cluster directory:
+
+* :file:`ipcontroller_job.xml`
+* :file:`ipengineset_job.xml`
+
+Once these files have been created, they can be imported into the HPC Job
+Manager application. Then, the controller and engines for that profile can be
+started using the HPC Job Manager directly, without using :command:`ipcluster`.
+However, anytime the cluster profile is re-configured, ``ipcluster start``
+must be run again to regenerate the XML job description files. The
+following screenshot shows what the HPC Job Manager interface looks like
+with a running IPython cluster.
+
+.. image:: hpc_job_manager.*
+
+Performing a simple interactive parallel computation
+====================================================
+
+Once you have started your IPython cluster, you can start to use it. To do
+this, open up a new Windows Command Prompt and start up IPython's interactive
+shell by typing::
+
+    ipython
+
+Then you can create a :class:`MultiEngineClient` instance for your profile and
+use the resulting instance to do a simple interactive parallel computation. In
+the code and screenshot that follows, we take a simple Python function and
+apply it to each element of an array of integers in parallel using the
+:meth:`MultiEngineClient.map` method:
+
+.. sourcecode:: ipython
+
+    In [1]: from IPython.kernel.client import *
+
+    In [2]: mec = MultiEngineClient(profile='mycluster')
+
+    In [3]: mec.get_ids()
+    Out[3]: [0, 1, 2, 3, 4, 5, 67, 8, 9, 10, 11, 12, 13, 14]
+
+    In [4]: def f(x):
+       ...:     return x**10
+
+    In [5]: mec.map(f, range(15))  # f is applied in parallel
+    Out[5]:
+    [0,
+     1,
+     1024,
+     59049,
+     1048576,
+     9765625,
+     60466176,
+     282475249,
+     1073741824,
+     3486784401L,
+     10000000000L,
+     25937424601L,
+     61917364224L,
+     137858491849L,
+     289254654976L]
+
+The :meth:`map` method has the same signature as Python's builtin :func:`map`
+function, but runs the calculation in parallel. More involved examples of using
+:class:`MultiEngineClient` are provided in the examples that follow.
+
+.. image:: mec_simple.*
+

docs/source/parallelz/winhpc_index.txt

@@ -0,0 +1,14 b''
+========================================
+Using IPython on Windows HPC Server 2008
+========================================
+
+
+Contents
+========
+
+.. toctree::
+   :maxdepth: 1
+
+   parallel_winhpc.txt
+   parallel_demos.txt
+

docs/source/index.txt

@@ -20,6 +20,7 b' Contents'
    install/index.txt
    interactive/index.txt
    parallel/index.txt
+   parallelz/index.txt
    config/index.txt
    development/index.txt
    api/index.txt