Initial draft of Windows HPC documentation.

docs/source/parallel/parallel_demos.txt

@@ -0,0 +1,126 b''
+=================
+Parallel examples
+=================
+
+In this section we describe a few more involved examples of using an IPython
+cluster to perform a parallel computation.
+
+150 million digits of pi
+========================
+
+In this example we would like to study the distribution of digits in the
+number pi. More specifically, we are going to study how often each 2
+digits sequence occurs in the first 150 million digits of pi. If the digits
+0-9 occur with equal probability, we expect that each two digits sequence
+(00, 01, ..., 99) will occur 1% of the time.
+
+This examples uses precomputed digits of pi from the website of Professor
+Yasumasa Kanada at the University of Tokoyo (http://www.super-computing.org).
+These digits come in a set of ``.txt`` files
+(ftp://pi.super-computing.org/.2/pi200m/) that each have 10 million digits of
+pi. In the parallel computation, we will use the :meth:`MultiEngineClient.map`
+method to have each engine compute the desired statistics on a subset of these
+files. Before I started the parallel computation, I copied the data files
+to the compute nodes so the engine have fast access to them.
+
+Here are the Python functions for counting the frequencies of each two digit
+sequence in serial::
+
+    def compute_two_digit_freqs(filename):
+        """
+        Compute the two digit frequencies from a single file.
+        """
+        d = txt_file_to_digits(filename)
+        freqs = two_digit_freqs(d)
+        return freqs
+    
+    def txt_file_to_digits(filename, the_type=str):
+        """
+        Yield the digits of pi read from a .txt file.
+        """
+        with open(filename, 'r') as f:
+            for line in f.readlines():
+                for c in line:
+                    if c != '\n' and c!= ' ':
+                        yield the_type(c)
+
+    def two_digit_freqs(digits, normalize=False):
+        """
+        Consume digits of pi and compute 2 digits freq. counts.
+        """
+        freqs = np.zeros(100, dtype='i4')
+        last = digits.next()
+        this = digits.next()
+        for d in digits:
+            index = int(last + this)
+            freqs[index] += 1
+            last = this
+            this = d
+        if normalize:
+            freqs = freqs/freqs.sum()
+        return freqs
+
+These functions are defined in the file :file:`pidigits.py`. To perform the
+calculation in parallel, we use an additional file: :file:`parallelpi.py`::
+
+    from IPython.kernel import client
+    from matplotlib import pyplot as plt
+    import numpy as np
+    from pidigits import *
+    from timeit import default_timer as clock
+
+    # Files with digits of pi (10m digits each)
+    filestring = 'pi200m-ascii-%(i)02dof20.txt'
+    files = [filestring % {'i':i} for i in range(1,16)]
+
+    # A function for reducing the frequencies calculated
+    # by different engines.
+    def reduce_freqs(freqlist):
+        allfreqs = np.zeros_like(freqlist[0])
+        for f in freqlist:
+            allfreqs += f
+        return allfreqs
+
+    # Connect to the IPython cluster
+    mec = client.MultiEngineClient(profile='mycluster')
+    mec.run('pidigits.py')
+
+    # Run 10m digits on 1 engine
+    mapper = mec.mapper(targets=0)
+    t1 = clock()
+    freqs10m = mapper.map(compute_two_digit_freqs, files[:1])[0]
+    t2 = clock()
+    digits_per_second1 = 10.0e6/(t2-t1)
+    print "Digits per second (1 core, 10m digits):   ", digits_per_second1
+
+    # Run 150m digits on 15 engines (8 cores)
+    t1 = clock()
+    freqs_all = mec.map(compute_two_digit_freqs, files[:len(mec)])
+    freqs150m = reduce_freqs(freqs_all)
+    t2 = clock()
+    digits_per_second8 = 150.0e6/(t2-t1)
+    print "Digits per second (8 cores, 150m digits): ", digits_per_second8
+
+    print "Speedup: ", digits_per_second8/digits_per_second1
+
+    plot_two_digit_freqs(freqs150m)
+    plt.title("2 digit sequences in 150m digits of pi")
+
+To run this code on an IPython cluster:
+
+1. Start an IPython cluster with 15 engines: ``ipcluster start -p mycluster -n 15``
+2. Open IPython's interactive shell using the pylab profile
+   ``ipython -p pylab`` and type ``run parallelpi.py``.
+
+At this point, the parallel calculation will begin. On a small an 8 core
+cluster, we observe a speedup of 7.7x. The resulting plot of the two digit
+sequences is shown in the following screenshot.
+
+.. image:: parallel_pi.*
+
+
+Parallel option pricing
+=======================
+
+The example will be added at a later point.
+

docs/source/parallel/parallel_winhpc.txt

@@ -0,0 +1,282 b''
+========================================
+Getting started
+========================================
+
+Introduction
+============
+
+IPython is an open source project focused on interactive and exploratory
+computing in the Python programming language. It consists of two
+main componenents:
+
+* An enhanced interactive Python shell with support for interactive plotting
+  and visualization.
+* An architecture for interactive parallel computing.
+
+With these components, it is possible to perform all aspects of a parallel
+computation interactively. This document describes how to get started with
+IPython on Window HPC Server 2008. A more complete desription of IPython's
+parallel computing capabilities can be found in IPython's online documentation
+(http://ipython.scipy.org/moin/Documentation).
+
+Setting up your Windows cluster
+===============================
+
+This document assumes that you already have a cluster running Windows
+HPC Server 2008. Here is a broad overview of what is involved with setting up
+such a cluster:
+
+1. Install Windows Server 2008 on the head and compute nodes in the cluster.
+2. Setup the network configuration on each host. Each host should have a 
+   static IP address.
+3. On the head node, activate the "Active Directory Domain Services" role 
+   and make the head node the domain controller.
+4. Join the compute nodes to the newly created Active Directory (AD) domain.
+5. Setup user accounts in the domain with shared home directories.
+6. Install the HPC Pack 2008 on the head node to create a cluster.
+7. Install the HPC Pack 2008 on the compute nodes.
+
+More details about installing and configuring Windows HPC Server 2008 can be
+found on the Windows HPC Home Page (http://www.microsoft.com/hpc). Regardless
+of what steps you go through to set up your cluster, the remainder of this
+document will assume that:
+
+* There are domain users that can log on to the AD domain and submit jobs
+  to the cluster scheduler.
+* These domain users have shared home directories. While shared home
+  directories are not required to use IPython, they make it much easier to
+  use IPython.
+
+Installation of IPython and its dependencies
+============================================
+
+IPython and all of its dependencies are freely available and open source.
+These packages provide a powerful and cost-effective approach to numerical and
+scientific computing on Windows. The following dependencies are needed to run
+IPython on Windows:
+
+* Python 2.5 or 2.6 (http://www.python.org)
+* pywin32 (http://sourceforge.net/projects/pywin32/)
+* PyReadline (https://launchpad.net/pyreadline)
+* zope.interface and Twisted (http://twistedmatrix.com)
+* Foolcap (http://foolscap.lothar.com/trac)
+* pyOpenSSL (https://launchpad.net/pyopenssl)
+* IPython (http://ipython.scipy.org)
+
+In addition, the following dependencies are needed to run the demos
+described in this document.
+
+* NumPy and SciPy (http://www.scipy.org)
+* wxPython (http://www.wxpython.org)
+* Matplotlib (http://matplotlib.sourceforge.net/)
+
+The easiest way of obtaining these dependencies is through the Enthought
+Python Distribution (EPD) (http://www.enthought.com/products/epd.php). EPD is
+produced by Enthought, Inc. and contains all of these packages and others in a
+single installer and is available free for academic users. While it is also
+possible to download and install each package individually, this is a tedious
+process. Thus, we highly recommend using EPD to install these packages on
+Windows.
+
+Regardless of how you install the dependencies, here are the steps you will
+need to follow:
+
+1. Install all of the packages listed above, either individually or using EPD
+   on the head node, compute nodes and user workstations.
+
+2. Make sure that :file:`C:\\Python25` and :file:`C:\\Python25\\Scripts` are
+   in the system :envvar:`%PATH%` variable on each node.
+
+3. Install the latest development version of IPython. This can be done by
+   downloading the the development version from the IPython website
+   (http://ipython.scipy.org) and following the installation instructions.
+
+Further details about installing IPython or its dependencies can be found in
+the online IPython documentation (http://ipython.scipy.org/moin/Documentation)
+Once you are finished with the installation, you can try IPython out by
+opening a Windows Command Prompt and typing :command:`ipython`. This will
+start IPython's interactive shell and you should see something like the
+following screenshot:
+
+.. image:: ipython_shell.*
+
+Starting an IPython cluster
+===========================
+
+To use IPython's parallel computing capabilities, you will need to start an
+IPython cluster. An IPython cluster consists of one controller and multiple
+engines:
+
+IPython controller
+    The IPython controller manages the engines and acts as a gateway between 
+    the engines and the client, which runs in the user's interactive IPython
+    session. The controller is started using the :command:`ipcontroller`
+    command.
+
+IPython engine
+    IPython engines run your Python code in parallel on the compute nodes. 
+    Engines are starting using the :command:`ipengine` command.
+
+Once these processes are started, a user can run Python code interactively and
+in parallel on the engines from within the IPython shell. This includes the
+ability to interact with, plot and visualize data from the engines.
+
+IPython has a command line program called :command:`ipcluster` that handles
+all aspects of starting the controller and engines on the compute nodes.
+:command:`ipcluster` has full support for the Windows HPC job scheduler,
+meaning that :command:`ipcluster` can use this job scheduler to start the
+controller and engines. In our experience, the Windows HPC job scheduler is
+particularly well suited for interactive applications, such as IPython. Once
+:command:`ipcluster` is configured properly, a user can start an IPython
+cluster from their local workstation almost instantly, without having to log
+on to the head node (as is typically required by Unix based job schedulers).
+This enables a user to move seamlessly between serial and parallel
+computations.
+
+In this section we show how to use :command:`ipcluster` to start an IPython
+cluster using the Windows HPC Server 2008 job scheduler. To make sure that
+:command:`ipcluster` is installed and working properly, you should first try
+to start an IPython cluster on your local host. To do this, open a Windows
+Command Prompt and type the following command::
+
+    ipcluster start -n 2
+
+You should see a number of messages printed to the screen, ending with 
+"IPython cluster: started". A screenshot of this follows.
+
+
+.. image:: ipcluster_start.*
+
+At this point, the controller and two engines are running on your local host.
+This configuration is useful for testing and for situations where you
+have multiple cores on your local computer. 
+
+Now that we have confirmed that :command:`ipcluster` is working properly, we
+describe how to configure and run an IPython cluster on an actual cluster
+running Windows HPC Server 2008. Here is an outline of the needed steps:
+
+1. Create a cluster profile:  ``ipcluster create -p mycluster``
+
+2. Edit confguration files in :file:`.ipython\\cluster_mycluster`.
+
+3. Start the cluster:  ``ipcluser start -p mycluster -n 32``
+
+Creating a cluster profile
+--------------------------
+
+In most cases, you will have to create and configure a cluster profile to use
+IPython on a cluster. A cluster profile is a specially named directory
+(typically located in the :file:`.ipython` subdirectory of your home
+directory) that contains the configuration files for a particular IPython
+cluster, as well as log files and security keys. The naming convention
+for cluster directories is: "cluster_<profile name>". Thus, the cluster
+directory for a profile named "foo" would be :file:`.ipython\\cluster_foo`.
+
+To create a new cluster profile (named "mycluster"), type the following
+command at the Windows Command Prompt::
+
+    ipcluster create -p mycluster
+
+The output of this command is shown in the screenshot below. Notice how
+:command:`ipcluster` prints out the location of the newly created cluster
+directory.
+
+
+.. image:: ipcluster_create.*
+
+
+Configuring a cluster profile
+-----------------------------
+
+Next, you will need to configure the newly created cluster profile by editing
+the following configuration files in the cluster directory:
+
+* :file:`ipcluster_config.py`
+* :file:`ipcontroller_config.py`
+* :file:`ipengine_config.py`
+
+When :command:`ipcluster` is run, these configuration files are used to 
+determine how the engines and controller will be started. In most cases,
+you will only have to set a few of the attributes in these files.
+
+To configure :command:`ipcluster` to use the Windows HPC job scheduler, you
+will need to edit the following attributes in the file
+:file:`ipcluster_config.py`::
+
+    # Set these at the top of the file to tell ipcluster to use the
+    # Windows HPC job scheduler.
+    c.Global.controller_launcher = \
+        'IPython.kernel.launcher.WindowsHPCControllerLauncher'
+    c.Global.engine_launcher = \
+        'IPython.kernel.launcher.WindowsHPCEngineSetLauncher'
+
+    # Set these to the host name of the scheduler (head node) of your cluster.
+    c.WindowsHPCControllerLauncher.scheduler = 'HEADNODE'
+    c.WindowsHPCEngineSetLauncher.scheduler = 'HEADNODE'
+
+There are a number of other configuration attributes that can be set, but
+in most cases these will be sufficient to get you started.
+
+.. warning::
+    If any of your configuration attributes involve specifying the location
+    of shared directories or files, you must make sure that you use UNC paths
+    like :file:`\\\\host\\share`. It is also important that you specify
+    these paths using raw Python strings: ``r'\\host\share'``.
+
+Starting the cluster profile
+----------------------------
+
+Once a cluster profile has been configured, starting an IPython cluster using
+the profile is simple:
+
+    ipcluster start -p mycluster -n 32
+
+The ``-n 32`` option tells :command:`ipcluster` how many engines to start.
+Stopping the cluster is as simple as typing Control-C. 
+
+Using the HPC Job Manager
+-------------------------
+
+When ``ipcluster start`` is run the first time, :command:`ipcluster` creates
+two XML job description files in the cluster directory:
+
+* :file:`ipcontroller_job.xml`
+* :file:`ipengineset_job.xml`
+
+Once these files have been created, they can be imported into the HPC Job
+Manager application. Then, the controller and engines for that profile can be
+started using the HPC Job Manager directly, without using :command:`ipcluster`.
+However, anytime the cluster profile is re-configured, ``ipcluster start``
+has to be run again to regenerate the XML job description files. The
+following screenshot shows what the HPC Job Manager interface looks like
+with a running IPython cluster.
+
+
+.. image:: hpc_job_manager.*
+
+Performing a simple interactive parallel computation
+====================================================
+
+Once you have started your IPython cluster, you can start to use it. To do
+this, start up IPython's interactive shell by typing::
+
+    ipython
+
+at the Windows Command Prompt. Then you can create a :class:`MultiEngineClient`
+instance for your profile and use the resulting instance to
+have the cluster do a simple interactive parallel computation. In the 
+screenshot that follows, we take a simple Python function::
+
+    def f(x): return x**10
+
+and apply it to each element of an array of integers in
+parallel using the :meth:`MultiEngineClient.map` method::
+
+    mec.map(f, range(15))
+
+The :meth:`map` method has the same signature as Python's builtin :func:`map`
+function, but runs the calculation in parallel. More involved examples of using
+:class:`MultiEngineClient` are provided in the examples that follow.
+
+.. image:: mec_simple.*
+

docs/source/parallel/winhpc_index.txt

@@ -0,0 +1,14 b''
+========================================
+Using IPython on Windows HPC Server 2008
+========================================
+
+
+Contents
+========
+
+.. toctree::
+   :maxdepth: 1
+
+   parallel_winhpc.txt
+   parallel_demos.txt
+

docs/examples/kernel/mcdriver.py

@@ -3,37 +3,36 b''
 """Run a Monte-Carlo options pricer in parallel."""
 
 from IPython.kernel import client
-import numpy as N
-from mcpricer import MCOptionPricer
+import numpy as np
+from mcpricer import price_options
 
 
-tc = client.TaskClient()
-rc = client.MultiEngineClient()
+tc = client.TaskClient(profile='default')
+mec = client.MultiEngineClient(profile='default')
 
-# Initialize the common code on the engines
-rc.run('mcpricer.py')
 
-# Push the variables that won't change 
-#(stock print, interest rate, days and MC paths)
-rc.push(dict(S=100.0, r=0.05, days=260, paths=10000))
+# Initialize the common code on the engines
+mec.run('mcpricer.py')
 
-task_string = """\
-op = MCOptionPricer(S,K,sigma,r,days,paths)
-op.run()
-vp, ap, vc, ac = op.vanilla_put, op.asian_put, op.vanilla_call, op.asian_call
-"""
+# Define the function that will do the calculation
+def my_prices(K, sigma):
+    S = 100.0
+    r = 0.05
+    days = 260
+    paths = 10000
+    return price_options(S, K, sigma, r, days, paths)
 
 # Create arrays of strike prices and volatilities
-K_vals = N.linspace(90.0,100.0,5)
-sigma_vals = N.linspace(0.0, 0.2,5)
+nK = 5
+nsigma = 5
+K_vals = np.linspace(90.0, 100.0, nK)
+sigma_vals = np.linspace(0.0, 0.2, nsigma)
 
 # Submit tasks
 taskids = []
 for K in K_vals:
     for sigma in sigma_vals:
-        t = client.StringTask(task_string, 
-            push=dict(sigma=sigma,K=K),
-            pull=('vp','ap','vc','ac','sigma','K'))
+        t = client.MapTask(my_prices, args=(K, sigma))
         taskids.append(tc.run(t))
 
 print "Submitted tasks: ", taskids
@@ -45,27 +44,21 b' tc.barrier(taskids)'
 results = [tc.get_task_result(tid) for tid in taskids]
 
 # Assemble the result
-vc = N.empty(K_vals.shape[0]*sigma_vals.shape[0],dtype='float64')
-vp = N.empty(K_vals.shape[0]*sigma_vals.shape[0],dtype='float64')
-ac = N.empty(K_vals.shape[0]*sigma_vals.shape[0],dtype='float64')
-ap = N.empty(K_vals.shape[0]*sigma_vals.shape[0],dtype='float64')
-for i, tr in enumerate(results):
-    ns = tr.ns
-    vc[i] = ns.vc
-    vp[i] = ns.vp
-    ac[i] = ns.ac
-    ap[i] = ns.ap
-vc.shape = (K_vals.shape[0],sigma_vals.shape[0])
-vp.shape = (K_vals.shape[0],sigma_vals.shape[0])
-ac.shape = (K_vals.shape[0],sigma_vals.shape[0])
-ap.shape = (K_vals.shape[0],sigma_vals.shape[0])
+prices = np.empty(nK*nsigma,
+    dtype=[('vcall',float),('vput',float),('acall',float),('aput',float)]
+)
+for i, price_tuple in enumerate(results):
+    prices[i] = price_tuple
+prices.shape = (nK, nsigma)
 
 
 def plot_options(K_vals, sigma_vals, prices):
-    """Make a contour plot of the option prices."""
-    import pylab
-    pylab.contourf(sigma_vals, K_vals, prices)
-    pylab.colorbar()
-    pylab.title("Option Price")
-    pylab.xlabel("Volatility")
-    pylab.ylabel("Strike Price")
+    """
+    Make a contour plot of the option prices.
+    """
+    from matplotlib import pyplot as plt
+    plt.contourf(sigma_vals, K_vals, prices)
+    plt.colorbar()
+    plt.title("Option Price")
+    plt.xlabel("Volatility")
+    plt.ylabel("Strike Price")

docs/examples/kernel/mcpricer.py

@@ -1,43 +1,33 b''
-import numpy as N
+import numpy as np
 from math import *
 
-class MCOptionPricer(object):
-    def __init__(self, S=100.0, K=100.0, sigma=0.25, r=0.05, days=260, paths=10000):
-        self.S = S
-        self.K = K
-        self.sigma = sigma
-        self.r = r
-        self.days = days
-        self.paths = paths
-        self.h = 1.0/self.days
-        self.const1 = exp((self.r-0.5*self.sigma**2)*self.h)
-        self.const2 = self.sigma*sqrt(self.h)
         
-    def run(self):
-        stock_price = self.S*N.ones(self.paths, dtype='float64')
-        stock_price_sum = N.zeros(self.paths, dtype='float64')
-        for j in range(self.days):
-            growth_factor = self.const1*N.exp(self.const2*N.random.standard_normal(self.paths))
+def price_options(S=100.0, K=100.0, sigma=0.25, r=0.05, days=260, paths=10000):
+    """
+    Price vanilla and asian options using a Monte Carlo method.
+    """
+    h = 1.0/days
+    const1 = exp((r-0.5*sigma**2)*h)
+    const2 = sigma*sqrt(h)
+    stock_price = S*np.ones(paths, dtype='float64')
+    stock_price_sum = np.zeros(paths, dtype='float64')
+    for j in range(days):
+        growth_factor = const1*np.exp(const2*np.random.standard_normal(paths))
             stock_price = stock_price*growth_factor
             stock_price_sum = stock_price_sum + stock_price
-        stock_price_avg = stock_price_sum/self.days
-        zeros = N.zeros(self.paths, dtype='float64')
-        r_factor = exp(-self.r*self.h*self.days)
-        self.vanilla_put = r_factor*N.mean(N.maximum(zeros,self.K-stock_price))
-        self.asian_put = r_factor*N.mean(N.maximum(zeros,self.K-stock_price_avg))
-        self.vanilla_call = r_factor*N.mean(N.maximum(zeros,stock_price-self.K))
-        self.asian_call = r_factor*N.mean(N.maximum(zeros,stock_price_avg-self.K))
-
-
-def main():
-    op = MCOptionPricer()
-    op.run()
-    print "Vanilla Put Price = ", op.vanilla_put
-    print "Asian Put Price = ", op.asian_put
-    print "Vanilla Call Price = ", op.vanilla_call
-    print "Asian Call Price = ", op.asian_call
+    stock_price_avg = stock_price_sum/days
+    zeros = np.zeros(paths, dtype='float64')
+    r_factor = exp(-r*h*days)
+    vanilla_put = r_factor*np.mean(np.maximum(zeros, K-stock_price))
+    asian_put = r_factor*np.mean(np.maximum(zeros, K-stock_price_avg))
+    vanilla_call = r_factor*np.mean(np.maximum(zeros, stock_price-K))
+    asian_call = r_factor*np.mean(np.maximum(zeros, stock_price_avg-K))
+    return (vanilla_call, vanilla_put, asian_call, asian_put)
 
 
 if __name__ == '__main__':
-    main()
-
+    (vc, vp, ac, ap) = price_options()
+    print "Vanilla Put Price = ", vp
+    print "Asian Put Price = ", ap
+    print "Vanilla Call Price = ", vc
+    print "Asian Call Price = ", ac

docs/source/conf.py

@@ -162,9 +162,12 b" latex_font_size = '11pt'"
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, document class [howto/manual]).
 
-latex_documents = [ ('index', 'ipython.tex', 'IPython Documentation',
-                     ur"""The IPython Development Team""",
-                     'manual', True),
+latex_documents = [
+    ('index', 'ipython.tex', 'IPython Documentation',
+     ur"""The IPython Development Team""", 'manual', True),
+    ('parallel/winhpc_index', 'winhpc_whitepaper.tex',
+     'Using IPython on Windows HPC Server 2008',
+     ur"Brian E. Granger", 'manual', True)
                     ]
 
 # The name of an image file (relative to this directory) to place at the top of