update some parallel docs

* add ssh ipcontroller/ipengine example (closes  #1017)
* generated new performance numbers with SGE cluster
* fix some remaining stale references
* removed some screenshots, in favor of editable text

closes  #980

docs/source/parallel/parallel_demos.txt

@@ -2,10 +2,6 b''
 Parallel examples
 =================
 
-.. note::
-
-    Performance numbers from ``IPython.kernel``, not new ``IPython.parallel``.
-
 In this section we describe two more involved examples of using an IPython
 cluster to perform a parallel computation. In these examples, we will be using
 IPython's "pylab" mode, which enables interactive plotting using the
@@ -110,17 +106,15 b' results. The code to run this calculation in parallel is contained in'
 :file:`docs/examples/parallel/parallelpi.py`. This code can be run in parallel
 using IPython by following these steps:
 
-1. Use :command:`ipcluster` to start 15 engines. We used an 8 core (2 quad 
-   core CPUs) cluster with hyperthreading enabled which makes the 8 cores 
-   looks like 16 (1 controller + 15 engines) in the OS. However, the maximum
-   speedup we can observe is still only 8x.
+1. Use :command:`ipcluster` to start 15 engines. We used 16 cores of an SGE linux
+   cluster (1 controller + 15 engines).
 2. With the file :file:`parallelpi.py` in your current working directory, open
    up IPython in pylab mode and type ``run parallelpi.py``.  This will download
    the pi files via ftp the first time you run it, if they are not
    present in the Engines' working directory.
 
-When run on our 8 core cluster, we observe a speedup of 7.7x. This is slightly
-less than linear scaling (8x) because the controller is also running on one of
+When run on our 16 cores, we observe a speedup of 14.2x. This is slightly
+less than linear scaling (16x) because the controller is also running on one of
 the cores.
 
 To emphasize the interactive nature of IPython, we now show how the
@@ -135,7 +129,7 b' calculation can also be run by simply typing the commands from'
     # We simply pass Client the name of the cluster profile we
     # are using.
     In [2]: c = Client(profile='mycluster')
-    In [3]: view = c.load_balanced_view()
+    In [3]: v = c[:]
 
     In [3]: c.ids
     Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
@@ -209,12 +203,12 b' simulation of the underlying asset price. In this example we use this approach'
 to price both European and Asian (path dependent) options for various strike
 prices and volatilities.
 
-The code for this example can be found in the :file:`docs/examples/parallel`
+The code for this example can be found in the :file:`docs/examples/parallel/options`
 directory of the IPython source. The function :func:`price_options` in
-:file:`mcpricer.py` implements the basic Monte Carlo pricing algorithm using
+:file:`mckernel.py` implements the basic Monte Carlo pricing algorithm using
 the NumPy package and is shown here:
 
-.. literalinclude:: ../../examples/parallel/options/mcpricer.py
+.. literalinclude:: ../../examples/parallel/options/mckernel.py
    :language: python
 
 To run this code in parallel, we will use IPython's :class:`LoadBalancedView` class,
@@ -227,7 +221,7 b' be found in the file :file:`mcpricer.py`. The code in this file creates a'
 volatilities and strike prices. The results are then plotted as a 2D contour
 plot using Matplotlib.
 
-.. literalinclude:: ../../examples/parallel/options/mckernel.py
+.. literalinclude:: ../../examples/parallel/options/mcpricer.py
    :language: python
 
 To use this code, start an IPython cluster using :command:`ipcluster`, open
@@ -236,8 +230,9 b' working directory and then type:'
 
 .. sourcecode:: ipython
 
-    In [7]: run mckernel.py
-    Submitted tasks:  [0, 1, 2, ...]
+    In [7]: run mcpricer.py
+    
+    Submitted tasks:  30
 
 Once all the tasks have finished, the results can be plotted using the
 :func:`plot_options` function. Here we make contour plots of the Asian
@@ -245,16 +240,16 b' call and Asian put options as function of the volatility and strike price:'
 
 .. sourcecode:: ipython
 
-    In [8]: plot_options(sigma_vals, K_vals, prices['acall'])
+    In [8]: plot_options(sigma_vals, strike_vals, prices['acall'])
 
     In [9]: plt.figure()
     Out[9]: <matplotlib.figure.Figure object at 0x18c178d0>
 
-    In [10]: plot_options(sigma_vals, K_vals, prices['aput'])
+    In [10]: plot_options(sigma_vals, strike_vals, prices['aput'])
 
-These results are shown in the two figures below. On a 8 core cluster the
-entire calculation (10 strike prices, 10 volatilities, 100,000 paths for each)
-took 30 seconds in parallel, giving a speedup of 7.7x, which is comparable
+These results are shown in the two figures below. On our 15 engines, the
+entire calculation (15 strike prices, 15 volatilities, 100,000 paths for each)
+took 37 seconds in parallel, giving a speedup of 14.1x, which is comparable
 to the speedup observed in our previous example.
 
 .. image:: figs/asian_call.*
@@ -274,11 +269,7 b' parallel architecture that have been demonstrated:'
   interactive shell.
 * Any data computed in parallel can be explored interactively through 
   visualization or further numerical calculations.
-* We have run these examples on a cluster running Windows HPC Server 2008.
-  IPython's built in support for the Windows HPC job scheduler makes it
-  easy to get started with IPython's parallel capabilities.
-
-.. note::
+* We have run these examples on a cluster running RHEL 5 and Sun GridEngine.
+  IPython's built in support for SGE (and other batch systems) makes it easy
+  to get started with IPython's parallel capabilities.
 
-    The new parallel code has never been run on Windows HPC Server, so the last
-    conclusion is untested.

docs/source/parallel/parallel_process.txt

@@ -181,7 +181,7 b' Assuming that the default MPI config is sufficient.'
     have not yet supported (such as Condor)
 
 Using :command:`ipcluster` in mpiexec/mpirun mode
---------------------------------------------------
+-------------------------------------------------
 
 
 The mpiexec/mpirun mode is useful if you:
@@ -243,7 +243,7 b' More details on using MPI with IPython can be found :ref:`here <parallelmpi>`.'
 
 
 Using :command:`ipcluster` in PBS mode
----------------------------------------
+--------------------------------------
 
 The PBS mode uses the Portable Batch System (PBS) to start the engines. 
 
@@ -364,7 +364,7 b' Additional configuration options can be found in the PBS section of :file:`ipclu'
 
 
 Using :command:`ipcluster` in SSH mode
----------------------------------------
+--------------------------------------
 
 
 The SSH mode uses :command:`ssh` to execute :command:`ipengine` on remote
@@ -401,7 +401,7 b" The controller's remote location and configuration can be specified:"
     # note that remotely launched ipcontroller will not get the contents of 
     # the local ipcontroller_config.py unless it resides on the *remote host*
     # in the location specified by the `profile-dir` argument.
-    # c.SSHControllerLauncher.program_args = ['--reuse', '--ip=*', '--profile-dir=/path/to/cd']
+    # c.SSHControllerLauncher.controller_args = ['--reuse', '--ip=*', '--profile-dir=/path/to/cd']
 
 .. note::
 
@@ -438,10 +438,11 b' Current limitations of the SSH mode of :command:`ipcluster` are:'
   Also, we are using shell scripts to setup and execute commands on remote
   hosts.
 * No file movement - This is a regression from 0.10, which moved connection files
-  around with scp. This will be improved, but not before 0.11 release.
+  around with scp. This will be improved, Pull Requests are welcome.
+
 
 Using the :command:`ipcontroller` and :command:`ipengine` commands
-====================================================================
+==================================================================
 
 It is also possible to use the :command:`ipcontroller` and :command:`ipengine`
 commands to start your controller and engines. This approach gives you full
@@ -487,7 +488,15 b' slightly more complicated, but the underlying ideas are the same:'
 
 1. Start the controller on a host using :command:`ipcontroller`. The controller must be
    instructed to listen on an interface visible to the engine machines, via the ``ip``
-   command-line argument or ``HubFactory.ip`` in :file:`ipcontroller_config.py`.
+   command-line argument or ``HubFactory.ip`` in :file:`ipcontroller_config.py`::
+   
+        $ ipcontroller --ip=192.168.1.16
+   
+   .. sourcecode:: python
+   
+        # in ipcontroller_config.py
+        HubFactory.ip = '192.168.1.16'
+
 2. Copy :file:`ipcontroller-engine.json` from :file:`~/.ipython/profile_<name>/security` on
    the controller's host to the host where the engines will run.
 3. Use :command:`ipengine` on the engine's hosts to start the engines.
@@ -553,6 +562,62 b' loopback, and ssh tunnels will be used to connect engines to the controller::'
 Or if you want to start many engines on each node, the command `ipcluster engines --n=4`
 without any configuration is equivalent to running ipengine 4 times.
 
+An example using ipcontroller/engine with ssh
+---------------------------------------------
+
+No configuration files are necessary to use ipcontroller/engine in an SSH environment
+without a shared filesystem. You simply need to make sure that the controller is listening
+on an interface visible to the engines, and move the connection file from the controller to
+the engines.
+
+1. start the controller, listening on an ip-address visible to the engine machines::
+
+    [controller.host] $ ipcontroller --ip=192.168.1.16
+    
+    [IPControllerApp] Using existing profile dir: u'/Users/me/.ipython/profile_default'
+    [IPControllerApp] Hub listening on tcp://192.168.1.16:63320 for registration.
+    [IPControllerApp] Hub using DB backend: 'IPython.parallel.controller.dictdb.DictDB'
+    [IPControllerApp] hub::created hub
+    [IPControllerApp] writing connection info to /Users/me/.ipython/profile_default/security/ipcontroller-client.json
+    [IPControllerApp] writing connection info to /Users/me/.ipython/profile_default/security/ipcontroller-engine.json
+    [IPControllerApp] task::using Python leastload Task scheduler
+    [IPControllerApp] Heartmonitor started
+    [IPControllerApp] Creating pid file: /Users/me/.ipython/profile_default/pid/ipcontroller.pid
+    Scheduler started [leastload]
+
+2. on each engine, fetch the connection file with scp::
+
+    [engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ./
+
+   .. note::
+
+        The log output of ipcontroller above shows you where the json files were written.
+        They will be in :file:`~/.ipython` (or :file:`~/.config/ipython`) under
+        :file:`profile_default/security/ipcontroller-engine.json`
+
+3. start the engines, using the connection file::
+
+    [engine.host.n] $ ipengine --file=./ipcontroller-engine.json
+
+A couple of notes:
+
+* You can avoid having to fetch the connection file every time by adding ``--reuse`` flag
+  to ipcontroller, which instructs the controller to read the previous connection file for
+  connection info, rather than generate a new one with randomized ports.
+
+* In step 2, if you fetch the connection file directly into the security dir of a profile,
+  then you need not specify its path directly, only the profile (assumes the path exists,
+  otherwise you must create it first)::
+
+    [engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ~/.ipython/profile_ssh/security/
+    [engine.host.n] $ ipengine --profile=ssh
+
+  Of course, if you fetch the file into the default profile, no arguments must be passed to
+  ipengine at all.
+
+* Note that ipengine *did not* specify the ip argument. In general, it is unlikely for any
+  connection information to be specified at the command-line to ipengine, as all of this
+  information should be contained in the connection file written by ipcontroller.
 
 Make JSON files persistent
 --------------------------

docs/source/parallel/parallel_winhpc.txt

@@ -2,10 +2,6 b''
 Getting started with Windows HPC Server 2008
 ============================================
 
-.. note::
-
-    Not adapted to zmq yet
-
 Introduction
 ============
 
@@ -118,9 +114,23 b' the online IPython documentation (http://ipython.org/documentation.html)'
 Once you are finished with the installation, you can try IPython out by
 opening a Windows Command Prompt and typing ``ipython``. This will
 start IPython's interactive shell and you should see something like the
-following screenshot:
+following::
+
+    Microsoft Windows [Version 6.0.6001]
+    Copyright (c) 2006 Microsoft Corporation.  All rights reserved.
+
+    Z:\>ipython
+    Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]
+    Type "copyright", "credits" or "license" for more information.
+
+    IPython 0.12.dev -- An enhanced Interactive Python.
+    ?         -> Introduction and overview of IPython's features.
+    %quickref -> Quick reference.
+    help      -> Python's own help system.
+    object?   -> Details about 'object', use 'object??' for extra details.
+
+    In [1]:
 
-.. image:: figs/ipython_shell.*
 
 Starting an IPython cluster
 ===========================
@@ -162,13 +172,24 b' cluster using the Windows HPC Server 2008 job scheduler. To make sure that'
 to start an IPython cluster on your local host. To do this, open a Windows
 Command Prompt and type the following command::
 
-    ipcluster start n=2
+    ipcluster start -n 2
 
-You should see a number of messages printed to the screen, ending with 
-"IPython cluster: started". The result should look something like the following
-screenshot:
+You should see a number of messages printed to the screen.
+The result should look something like this::
 
-.. image:: figs/ipcluster_start.*
+    Microsoft Windows [Version 6.1.7600]
+    Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
+
+    Z:\>ipcluster start --profile=mycluster
+    [IPClusterStart] Using existing profile dir: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster'
+    [IPClusterStart] Starting ipcluster with [daemon=False]
+    [IPClusterStart] Creating pid file: \\blue\domainusers$\bgranger\.ipython\profile_mycluster\pid\ipcluster.pid
+    [IPClusterStart] Writing job description file: \\blue\domainusers$\bgranger\.ipython\profile_mycluster\ipcontroller_job.xml
+    [IPClusterStart] Starting Win HPC Job: job submit /jobfile:\\blue\domainusers$\bgranger\.ipython\profile_mycluster\ipcontroller_job.xml /scheduler:HEADNODE
+    [IPClusterStart] Starting 15 engines
+    [IPClusterStart] Writing job description file: \\blue\domainusers$\bgranger\.ipython\profile_mycluster\ipcontroller_job.xml
+    [IPClusterStart] Starting Win HPC Job: job submit /jobfile:\\blue\domainusers$\bgranger\.ipython\profile_mycluster\ipengineset_job.xml /scheduler:HEADNODE
+    
 
 At this point, the controller and two engines are running on your local host.
 This configuration is useful for testing and for situations where you want to
@@ -179,11 +200,11 b' describe how to configure and run an IPython cluster on an actual compute'
 cluster running Windows HPC Server 2008. Here is an outline of the needed
 steps:
 
-1. Create a cluster profile using: ``ipython profile create --parallel profile=mycluster``
+1. Create a cluster profile using: ``ipython profile create mycluster --parallel``
 
 2. Edit configuration files in the directory :file:`.ipython\\cluster_mycluster`
 
-3. Start the cluster using: ``ipcluser start profile=mycluster n=32``
+3. Start the cluster using: ``ipcluster start --profile=mycluster -n 32``
 
 Creating a cluster profile
 --------------------------
@@ -207,10 +228,17 b' directory, type the following command at the Windows Command Prompt::'
     ipython profile create --parallel --profile=mycluster
 
 The output of this command is shown in the screenshot below. Notice how
-:command:`ipcluster` prints out the location of the newly created cluster
-directory.
+:command:`ipcluster` prints out the location of the newly created profile
+directory::
+
+    Z:\>ipython profile create mycluster --parallel
+    [ProfileCreate] Generating default config file: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster\\ipython_config.py'
+    [ProfileCreate] Generating default config file: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster\\ipcontroller_config.py'
+    [ProfileCreate] Generating default config file: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster\\ipengine_config.py'
+    [ProfileCreate] Generating default config file: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster\\ipcluster_config.py'
+    [ProfileCreate] Generating default config file: u'\\\\blue\\domainusers$\\bgranger\\.ipython\\profile_mycluster\\iplogger_config.py'
 
-.. image:: figs/ipcluster_create.*
+    Z:\>
 
 Configuring a cluster profile
 -----------------------------
@@ -245,7 +273,7 b' in most cases these will be sufficient to get you started.'
 .. warning::
     If any of your configuration attributes involve specifying the location
     of shared directories or files, you must make sure that you use UNC paths
-    like :file:`\\\\host\\share`. It is also important that you specify
+    like :file:`\\\\host\\share`. It is helpful to specify
     these paths using raw Python strings: ``r'\\host\share'`` to make sure
     that the backslashes are properly escaped.
 
@@ -262,7 +290,7 b' this case 32). Stopping the cluster is as simple as typing Control-C.'
 
 Using the HPC Job Manager
 -------------------------
-
+føø
 When ``ipcluster start`` is run the first time, :command:`ipcluster` creates
 two XML job description files in the cluster directory:
 
@@ -288,26 +316,28 b' shell by typing::'
 
     ipython
 
-Then you can create a :class:`MultiEngineClient` instance for your profile and
+Then you can create a :class:`DirectView` instance for your profile and
 use the resulting instance to do a simple interactive parallel computation. In
 the code and screenshot that follows, we take a simple Python function and
 apply it to each element of an array of integers in parallel using the
-:meth:`MultiEngineClient.map` method:
+:meth:`DirectView.map` method:
 
 .. sourcecode:: ipython
 
     In [1]: from IPython.parallel import *
 
-    In [2]: c = MultiEngineClient(profile='mycluster')
+    In [2]: c = Client(profile='mycluster')
 
-    In [3]: mec.get_ids()
-    Out[3]: [0, 1, 2, 3, 4, 5, 67, 8, 9, 10, 11, 12, 13, 14]
+    In [3]: view = c[:]
 
-    In [4]: def f(x):
+    In [4]: c.ids
+    Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
+
+    In [5]: def f(x):
        ...:     return x**10
 
-    In [5]: mec.map(f, range(15))  # f is applied in parallel
-    Out[5]:
+    In [6]: view.map(f, range(15))  # f is applied in parallel
+    Out[6]:
     [0,
      1,
      1024,
@@ -326,7 +356,5 b' apply it to each element of an array of integers in parallel using the'
 
 The :meth:`map` method has the same signature as Python's builtin :func:`map`
 function, but runs the calculation in parallel. More involved examples of using
-:class:`MultiEngineClient` are provided in the examples that follow.
-
-.. image:: figs/mec_simple.*
+:class:`DirectView` are provided in the examples that follow.