diff --git a/IPython/kernel/scripts/ipcluster.py b/IPython/kernel/scripts/ipcluster.py index 0984597..02f8060 100644 --- a/IPython/kernel/scripts/ipcluster.py +++ b/IPython/kernel/scripts/ipcluster.py @@ -358,7 +358,8 @@ def main_mpirun(args): raw_args.append('ipengine') raw_args.append('-l') raw_args.append(pjoin(args.logdir,'ipengine%s-' % cont_pid)) - raw_args.append('--mpi=%s' % args.mpi) + if args.mpi: + raw_args.append('--mpi=%s' % args.mpi) eset = ProcessLauncher(raw_args) def shutdown(signum, frame): log.msg('Stopping local cluster') @@ -454,8 +455,7 @@ def get_args(): parser_mpirun.add_argument( "--mpi", type=str, - dest="mpi", - default='mpi4py', + dest="mpi", # Don't put a default here to allow no MPI support help="how to call MPI_Init (default=mpi4py)" ) parser_mpirun.set_defaults(func=main_mpirun) diff --git a/docs/source/config/index.txt b/docs/source/config/index.txt index b8263b1..1214811 100644 --- a/docs/source/config/index.txt +++ b/docs/source/config/index.txt @@ -3,7 +3,7 @@ Configuration and customization =============================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 initial_config.txt customization.txt diff --git a/docs/source/development/roadmap.txt b/docs/source/development/roadmap.txt index 09cf686..cc570b8 100644 --- a/docs/source/development/roadmap.txt +++ b/docs/source/development/roadmap.txt @@ -4,8 +4,6 @@ Development roadmap =================== -.. contents:: - IPython is an ambitious project that is still under heavy development. However, we want IPython to become useful to as many people as possible, as quickly as possible. To help us accomplish this, we are laying out a roadmap of where we are headed and what needs to happen to get there. Hopefully, this will help the IPython developers figure out the best things to work on for each upcoming release. Speaking of releases, we are going to begin releasing a new version of IPython every four weeks. We are hoping that a regular release schedule, along with a clear roadmap of where we are headed will propel the project forward. diff --git a/docs/source/index.txt b/docs/source/index.txt index bde7c08..688e167 100644 --- a/docs/source/index.txt +++ b/docs/source/index.txt @@ -24,9 +24,7 @@ IPython Documentation license_and_copyright.txt credits.txt - .. htmlonly:: - - * :ref:`genindex` - * :ref:`modindex` - * :ref:`search` + * :ref:`genindex` + * :ref:`modindex` + * :ref:`search` diff --git a/docs/source/install/install.txt b/docs/source/install/install.txt index dfad03e..41c5050 100644 --- a/docs/source/install/install.txt +++ b/docs/source/install/install.txt @@ -119,7 +119,13 @@ Most users on OS X will want to get the full :mod:`readline` module. To get a w If needed, the readline egg can be build and installed from source (see the wiki page at http://ipython.scipy.org/moin/InstallationOSXLeopard). -On Windows, you will need the PyReadline module. PyReadline is a separate, Windows only implementation of readline that uses native Windows calls through :mod:`ctypes`. The easiest way of installing PyReadline is you use the binary installer available `here `_. The :mod:`ctypes` module, which comes with Python 2.5 and greater, is required by PyReadline. It is available for Python 2.4 at http://python.net/crew/theller/ctypes. +On Windows, you will need the PyReadline module. PyReadline is a separate, +Windows only implementation of readline that uses native Windows calls through +:mod:`ctypes`. The easiest way of installing PyReadline is you use the binary +installer available `here `_. The +:mod:`ctypes` module, which comes with Python 2.5 and greater, is required by +PyReadline. It is available for Python 2.4 at +http://python.net/crew/theller/ctypes. nose ---- @@ -164,7 +170,9 @@ On a Unix style platform (including OS X), if you want to use :mod:`setuptools`, zope.interface and Twisted -------------------------- -On Unix style platforms (including OS X), the simplest way of getting the these is to use :command:`easy_install`:: +Twisted [Twisted]_ and zope.interface [ZopeInterface]_ are used for networking related things. On Unix +style platforms (including OS X), the simplest way of getting the these is to +use :command:`easy_install`:: $ easy_install zope.interface $ easy_install Twisted @@ -176,7 +184,7 @@ Windows is a bit different. For zope.interface and Twisted, simply get the late Foolscap -------- -Foolscap uses Twisted to provide a very nice secure RPC protocol that we use to implement our parallel computing features. +Foolscap [Foolscap]_ uses Twisted to provide a very nice secure RPC protocol that we use to implement our parallel computing features. On all platforms a simple:: @@ -187,7 +195,7 @@ should work. You can also download the source tarballs from the `Foolscap websi pyOpenSSL --------- -IPython requires an older version of pyOpenSSL (0.6 rather than the current 0.7). There are a couple of options for getting this: +IPython requires an older version of pyOpenSSL [pyOpenSSL]_ (0.6 rather than the current 0.7). There are a couple of options for getting this: 1. Most Linux distributions have packages for pyOpenSSL. 2. The built-in Python 2.5 on OS X 10.5 already has it installed. @@ -201,4 +209,9 @@ Dependencies for IPython.frontend (the IPython GUI) wxPython -------- -Starting with IPython 0.9, IPython has a new IPython.frontend package that has a nice wxPython based IPython GUI. As you would expect, this GUI requires wxPython. Most Linux distributions have wxPython packages available and the built-in Python on OS X comes with wxPython preinstalled. For Windows, a binary installer is available on the `wxPython website `_. \ No newline at end of file +Starting with IPython 0.9, IPython has a new IPython.frontend package that has a nice wxPython based IPython GUI. As you would expect, this GUI requires wxPython. Most Linux distributions have wxPython packages available and the built-in Python on OS X comes with wxPython preinstalled. For Windows, a binary installer is available on the `wxPython website `_. + +.. [Twisted] Twisted matrix. http://twistedmatrix.org +.. [ZopeInterface] http://pypi.python.org/pypi/zope.interface +.. [Foolscap] Foolscap network protocol. http://foolscap.lothar.com/trac +.. [pyOpenSSL] pyOpenSSL. http://pyopenssl.sourceforge.net \ No newline at end of file diff --git a/docs/source/interactive/index.txt b/docs/source/interactive/index.txt index 01111b0..ae45bc5 100644 --- a/docs/source/interactive/index.txt +++ b/docs/source/interactive/index.txt @@ -3,7 +3,7 @@ Using IPython for interactive work ================================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 tutorial.txt reference.txt diff --git a/docs/source/interactive/reference.txt b/docs/source/interactive/reference.txt index ef60af4..f81587f 100644 --- a/docs/source/interactive/reference.txt +++ b/docs/source/interactive/reference.txt @@ -1,13 +1,7 @@ -.. IPython documentation master file, created by sphinx-quickstart.py on Mon Mar 24 17:01:34 2008. - You can adapt this file completely to your liking, but it should at least - contain the root 'toctree' directive. - ================= IPython reference ================= -.. contents:: - .. _command_line_options: Command-line usage diff --git a/docs/source/interactive/tutorial.txt b/docs/source/interactive/tutorial.txt index e7271d5..23a2854 100644 --- a/docs/source/interactive/tutorial.txt +++ b/docs/source/interactive/tutorial.txt @@ -4,8 +4,6 @@ Quick IPython tutorial ====================== -.. contents:: - IPython can be used as an improved replacement for the Python prompt, and for that you don't really need to read any more of this manual. But in this section we'll try to summarize a few tips on how to make the diff --git a/docs/source/license_and_copyright.txt b/docs/source/license_and_copyright.txt index 1c9840e..9c61f1e 100644 --- a/docs/source/license_and_copyright.txt +++ b/docs/source/license_and_copyright.txt @@ -13,39 +13,43 @@ IPython is licensed under the terms of the new or revised BSD license, as follow All rights reserved. - Redistribution and use in source and binary forms, with or without modification, - are permitted provided that the following conditions are met: - - Redistributions of source code must retain the above copyright notice, this list of - conditions and the following disclaimer. - - Redistributions in binary form must reproduce the above copyright notice, this list - of conditions and the following disclaimer in the documentation and/or other - materials provided with the distribution. - - Neither the name of the IPython Development Team nor the names of its contributors - may be used to endorse or promote products derived from this software without - specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY - EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. - IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, - INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT - NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, - WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + Neither the name of the IPython Development Team nor the names of its + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS + IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, + EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, + PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR + PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING + NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. About the IPython Development Team ================================== -Fernando Perez began IPython in 2001 based on code from Janko Hauser -and Nathaniel Gray . Fernando is still the project lead. +Fernando Perez began IPython in 2001 based on code from Janko Hauser + and Nathaniel Gray . Fernando is still +the project lead. -The IPython Development Team is the set of all contributors to the IPython project. -This includes all of the IPython subprojects. Here is a list of the currently active contributors: +The IPython Development Team is the set of all contributors to the IPython +project. This includes all of the IPython subprojects. Here is a list of the +currently active contributors: * Matthieu Brucher * Ondrej Certik @@ -65,23 +69,23 @@ If your name is missing, please add it. Our Copyright Policy ==================== -IPython uses a shared copyright model. Each contributor maintains copyright over -their contributions to IPython. But, it is important to note that these -contributions are typically only changes to the repositories. Thus, the IPython -source code, in its entirety is not the copyright of any single person or -institution. Instead, it is the collective copyright of the entire IPython +IPython uses a shared copyright model. Each contributor maintains copyright +over their contributions to IPython. But, it is important to note that these +contributions are typically only changes to the repositories. Thus, the +IPython source code, in its entirety is not the copyright of any single person +or institution. Instead, it is the collective copyright of the entire IPython Development Team. If individual contributors want to maintain a record of what -changes/contributions they have specific copyright on, they should indicate their -copyright in the commit message of the change, when they commit the change to -one of the IPython repositories. +changes/contributions they have specific copyright on, they should indicate +their copyright in the commit message of the change, when they commit the +change to one of the IPython repositories. Miscellaneous ============= Some files (DPyGetOpt.py, for example) may be licensed under different -conditions. Ultimately each file indicates clearly the conditions under -which its author/authors have decided to publish the code. +conditions. Ultimately each file indicates clearly the conditions under which +its author/authors have decided to publish the code. -Versions of IPython up to and including 0.6.3 were released under the -GNU Lesser General Public License (LGPL), available at +Versions of IPython up to and including 0.6.3 were released under the GNU +Lesser General Public License (LGPL), available at http://www.gnu.org/copyleft/lesser.html. \ No newline at end of file diff --git a/docs/source/parallel/parallel_intro.txt b/docs/source/parallel/parallel_intro.txt index 2945b23..92db52b 100644 --- a/docs/source/parallel/parallel_intro.txt +++ b/docs/source/parallel/parallel_intro.txt @@ -4,20 +4,17 @@ Overview and getting started ============================ -.. contents:: - Introduction ============ -This file gives an overview of IPython's sophisticated and -powerful architecture for parallel and distributed computing. This -architecture abstracts out parallelism in a very general way, which -enables IPython to support many different styles of parallelism -including: +This section gives an overview of IPython's sophisticated and powerful +architecture for parallel and distributed computing. This architecture +abstracts out parallelism in a very general way, which enables IPython to +support many different styles of parallelism including: * Single program, multiple data (SPMD) parallelism. * Multiple program, multiple data (MPMD) parallelism. -* Message passing using ``MPI``. +* Message passing using MPI. * Task farming. * Data parallel. * Combinations of these approaches. @@ -97,8 +94,8 @@ The controller also provides a single point of contact for users who wish to utilize the engines connected to the controller. There are different ways of working with a controller. In IPython these ways correspond to different interfaces that the controller is adapted to. Currently we have two default interfaces to the controller: -* The MultiEngine interface, which provides the simplest possible way of working - with engines interactively. +* The MultiEngine interface, which provides the simplest possible way of + working with engines interactively. * The Task interface, which provides presents the engines as a load balanced task farming system. @@ -124,29 +121,27 @@ interface. Here are the two default clients: Security -------- -By default (as long as `pyOpenSSL` is installed) all network connections between the controller and engines and the controller and clients are secure. What does this mean? First of all, all of the connections will be encrypted using SSL. Second, the connections are authenticated. We handle authentication in a `capabilities`__ based security model. In this model, a "capability (known in some systems as a key) is a communicable, unforgeable token of authority". Put simply, a capability is like a key to your house. If you have the key to your house, you can get in. If not, you can't. - -.. __: http://en.wikipedia.org/wiki/Capability-based_security +By default (as long as `pyOpenSSL` is installed) all network connections between the controller and engines and the controller and clients are secure. What does this mean? First of all, all of the connections will be encrypted using SSL. Second, the connections are authenticated. We handle authentication in a capability based security model [Capability]_. In this model, a "capability (known in some systems as a key) is a communicable, unforgeable token of authority". Put simply, a capability is like a key to your house. If you have the key to your house, you can get in. If not, you can't. In our architecture, the controller is the only process that listens on network ports, and is thus responsible to creating these keys. In IPython, these keys are known as Foolscap URLs, or FURLs, because of the underlying network protocol we are using. As a user, you don't need to know anything about the details of these FURLs, other than that when the controller starts, it saves a set of FURLs to files named :file:`something.furl`. The default location of these files is the :file:`~./ipython/security` directory. -To connect and authenticate to the controller an engine or client simply needs to present an appropriate furl (that was originally created by the controller) to the controller. Thus, the .furl files need to be copied to a location where the clients and engines can find them. Typically, this is the :file:`~./ipython/security` directory on the host where the client/engine is running (which could be a different host than the controller). Once the .furl files are copied over, everything should work fine. +To connect and authenticate to the controller an engine or client simply needs to present an appropriate FURL (that was originally created by the controller) to the controller. Thus, the FURL files need to be copied to a location where the clients and engines can find them. Typically, this is the :file:`~./ipython/security` directory on the host where the client/engine is running (which could be a different host than the controller). Once the FURL files are copied over, everything should work fine. -Currently, there are three .furl files that the controller creates: +Currently, there are three FURL files that the controller creates: ipcontroller-engine.furl - This ``.furl`` file is the key that gives an engine the ability to connect + This FURL file is the key that gives an engine the ability to connect to a controller. ipcontroller-tc.furl - This ``.furl`` file is the key that a :class:`TaskClient` must use to + This FURL file is the key that a :class:`TaskClient` must use to connect to the task interface of a controller. ipcontroller-mec.furl - This ``.furl`` file is the key that a :class:`MultiEngineClient` must use to - connect to the multiengine interface of a controller. + This FURL file is the key that a :class:`MultiEngineClient` must use + to connect to the multiengine interface of a controller. -More details of how these ``.furl`` files are used are given below. +More details of how these FURL files are used are given below. A detailed description of the security model and its implementation in IPython can be found :ref:`here `. @@ -157,13 +152,15 @@ Getting Started To use IPython for parallel computing, you need to start one instance of the controller and one or more instances of the engine. Initially, it is best to simply start a controller and engines on a single host using the :command:`ipcluster` command. To start a controller and 4 engines on you localhost, just do:: - $ ipcluster -n 4 + $ ipcluster local -n 4 More details about starting the IPython controller and engines can be found :ref:`here ` Once you have started the IPython controller and one or more engines, you are ready to use the engines to do something useful. To make sure -everything is working correctly, try the following commands:: +everything is working correctly, try the following commands: + +.. sourcecode:: ipython In [1]: from IPython.kernel import client @@ -187,7 +184,7 @@ everything is working correctly, try the following commands:: [3] In [1]: print "Hello World" [3] Out[1]: Hello World -Remember, a client also needs to present a ``.furl`` file to the controller. How does this happen? When a multiengine client is created with no arguments, the client tries to find the corresponding ``.furl`` file in the local :file:`~./ipython/security` directory. If it finds it, you are set. If you have put the ``.furl`` file in a different location or it has a different name, create the client like this:: +Remember, a client also needs to present a FURL file to the controller. How does this happen? When a multiengine client is created with no arguments, the client tries to find the corresponding FURL file in the local :file:`~./ipython/security` directory. If it finds it, you are set. If you have put the FURL file in a different location or it has a different name, create the client like this:: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl') @@ -200,5 +197,9 @@ You are now ready to learn more about the :ref:`MultiEngine ` to using IPython for parallel computing. @@ -30,7 +28,9 @@ Creating a ``MultiEngineClient`` instance ========================================= The first step is to import the IPython :mod:`IPython.kernel.client` module -and then create a :class:`MultiEngineClient` instance:: +and then create a :class:`MultiEngineClient` instance: + +.. sourcecode:: ipython In [1]: from IPython.kernel import client @@ -38,13 +38,17 @@ and then create a :class:`MultiEngineClient` instance:: This form assumes that the :file:`ipcontroller-mec.furl` is in the :file:`~./ipython/security` directory on the client's host. If not, the -location of the ``.furl`` file must be given as an argument to the -constructor:: +location of the FURL file must be given as an argument to the +constructor: + +.. sourcecode:: ipython - In[2]: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl') + In [2]: mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl') To make sure there are engines connected to the controller, use can get a list -of engine ids:: +of engine ids: + +.. sourcecode:: ipython In [3]: mec.get_ids() Out[3]: [0, 1, 2, 3] @@ -62,7 +66,9 @@ Parallel map Python's builtin :func:`map` functions allows a function to be applied to a sequence element-by-element. This type of code is typically trivial to parallelize. In fact, the multiengine interface in IPython already has a -parallel version of :meth:`map` that works just like its serial counterpart:: +parallel version of :meth:`map` that works just like its serial counterpart: + +.. sourcecode:: ipython In [63]: serial_result = map(lambda x:x**10, range(32)) @@ -84,7 +90,9 @@ parallel version of :meth:`map` that works just like its serial counterpart:: Parallel function decorator --------------------------- -Parallel functions are just like normal function, but they can be called on sequences and *in parallel*. The multiengine interface provides a decorator that turns any Python function into a parallel function:: +Parallel functions are just like normal function, but they can be called on sequences and *in parallel*. The multiengine interface provides a decorator that turns any Python function into a parallel function: + +.. sourcecode:: ipython In [10]: @mec.parallel() ....: def f(x): @@ -110,7 +118,9 @@ Blocking execution In blocking mode, the :class:`MultiEngineClient` object (called ``mec`` in these examples) submits the command to the controller, which places the command in the engines' queues for execution. The :meth:`execute` call then -blocks until the engines are done executing the command:: +blocks until the engines are done executing the command: + +.. sourcecode:: ipython # The default is to run on all engines In [4]: mec.execute('a=5') @@ -130,7 +140,9 @@ blocks until the engines are done executing the command:: [3] In [2]: b=10 Python commands can be executed on specific engines by calling execute using -the ``targets`` keyword argument:: +the ``targets`` keyword argument: + +.. sourcecode:: ipython In [6]: mec.execute('c=a+b',targets=[0,2]) Out[6]: @@ -163,7 +175,9 @@ the ``targets`` keyword argument:: This example also shows one of the most important things about the IPython engines: they have a persistent user namespaces. The :meth:`execute` method -returns a Python ``dict`` that contains useful information:: +returns a Python ``dict`` that contains useful information: + +.. sourcecode:: ipython In [9]: result_dict = mec.execute('d=10; print d') @@ -184,7 +198,9 @@ then returns a :class:`PendingResult` object immediately. The :class:`PendingResult` object gives you a way of getting a result at a later time through its :meth:`get_result` method or :attr:`r` attribute. This allows you to quickly submit long running commands without blocking your local -Python/IPython session:: +Python/IPython session: + +.. sourcecode:: ipython # In blocking mode In [6]: mec.execute('import time') @@ -225,7 +241,9 @@ Python/IPython session:: Often, it is desirable to wait until a set of :class:`PendingResult` objects are done. For this, there is a the method :meth:`barrier`. This method takes a tuple of :class:`PendingResult` objects and blocks until all of the associated -results are ready:: +results are ready: + +.. sourcecode:: ipython In [72]: mec.block=False @@ -259,7 +277,9 @@ and :attr:`targets`: * If no keyword argument is provided, the instance attributes are used. * Keyword argument, if provided override the instance attributes. -The following examples demonstrate how to use the instance attributes:: +The following examples demonstrate how to use the instance attributes: + +.. sourcecode:: ipython In [16]: mec.targets = [0,2] @@ -305,7 +325,9 @@ that make it more pleasant to execute Python commands on the engines interactively. These are simply shortcuts to :meth:`execute` and :meth:`get_result`. The ``%px`` magic executes a single Python command on the engines specified by the :attr:`targets` attribute of the -:class:`MultiEngineClient` instance (by default this is ``'all'``):: +:class:`MultiEngineClient` instance (by default this is ``'all'``): + +.. sourcecode:: ipython # Make this MultiEngineClient active for parallel magic commands In [23]: mec.activate() @@ -352,7 +374,9 @@ engines specified by the :attr:`targets` attribute of the The ``%result`` magic gets and prints the stdin/stdout/stderr of the last command executed on each engine. It is simply a shortcut to the -:meth:`get_result` method:: +:meth:`get_result` method: + +.. sourcecode:: ipython In [29]: %result Out[29]: @@ -370,7 +394,9 @@ command executed on each engine. It is simply a shortcut to the [3] Out[9]: [ 0.83664764 -0.25602658] The ``%autopx`` magic switches to a mode where everything you type is executed -on the engines given by the :attr:`targets` attribute:: +on the engines given by the :attr:`targets` attribute: + +.. sourcecode:: ipython In [30]: mec.block=False @@ -423,7 +449,9 @@ are called :meth:`push` (sending an object to the engines) and :meth:`pull` Basic push and pull ------------------- -Here are some examples of how you use :meth:`push` and :meth:`pull`:: +Here are some examples of how you use :meth:`push` and :meth:`pull`: + +.. sourcecode:: ipython In [38]: mec.push(dict(a=1.03234,b=3453)) Out[38]: [None, None, None, None] @@ -460,7 +488,9 @@ Here are some examples of how you use :meth:`push` and :meth:`pull`:: [3] Out[13]: speed In non-blocking mode :meth:`push` and :meth:`pull` also return -:class:`PendingResult` objects:: +:class:`PendingResult` objects: + +.. sourcecode:: ipython In [47]: mec.block=False @@ -474,8 +504,9 @@ Push and pull for functions --------------------------- Functions can also be pushed and pulled using :meth:`push_function` and -:meth:`pull_function`:: +:meth:`pull_function`: +.. sourcecode:: ipython In [52]: mec.block=True @@ -518,7 +549,9 @@ Dictionary interface As a shorthand to :meth:`push` and :meth:`pull`, the :class:`MultiEngineClient` class implements some of the Python dictionary interface. This make the remote namespaces of the engines appear as a local -dictionary. Underneath, this uses :meth:`push` and :meth:`pull`:: +dictionary. Underneath, this uses :meth:`push` and :meth:`pull`: + +.. sourcecode:: ipython In [50]: mec.block=True @@ -536,7 +569,9 @@ follow that terminology. However, it is important to remember that in IPython's :class:`MultiEngineClient` class, :meth:`scatter` is from the interactive IPython session to the engines and :meth:`gather` is from the engines back to the interactive IPython session. For scatter/gather operations -between engines, MPI should be used:: +between engines, MPI should be used: + +.. sourcecode:: ipython In [58]: mec.scatter('a',range(16)) Out[58]: [None, None, None, None] @@ -569,7 +604,9 @@ How to do parallel list comprehensions In many cases list comprehensions are nicer than using the map function. While we don't have fully parallel list comprehensions, it is simple to get the -basic effect using :meth:`scatter` and :meth:`gather`:: +basic effect using :meth:`scatter` and :meth:`gather`: + +.. sourcecode:: ipython In [66]: mec.scatter('x',range(64)) Out[66]: [None, None, None, None] @@ -598,7 +635,9 @@ parallel command can actually raise multiple exceptions (one for each engine the command was run on). To express this idea, the MultiEngine interface has a :exc:`CompositeError` exception class that will be raised in most cases. The :exc:`CompositeError` class is a special type of exception that wraps one or -more other types of exceptions. Here is how it works:: +more other types of exceptions. Here is how it works: + +.. sourcecode:: ipython In [76]: mec.block=True @@ -628,7 +667,9 @@ more other types of exceptions. Here is how it works:: [2:execute]: ZeroDivisionError: integer division or modulo by zero [3:execute]: ZeroDivisionError: integer division or modulo by zero -Notice how the error message printed when :exc:`CompositeError` is raised has information about the individual exceptions that were raised on each engine. If you want, you can even raise one of these original exceptions:: +Notice how the error message printed when :exc:`CompositeError` is raised has information about the individual exceptions that were raised on each engine. If you want, you can even raise one of these original exceptions: + +.. sourcecode:: ipython In [80]: try: ....: mec.execute('1/0') @@ -652,7 +693,9 @@ Notice how the error message printed when :exc:`CompositeError` is raised has in If you are working in IPython, you can simple type ``%debug`` after one of these :exc:`CompositeError` exceptions is raised, and inspect the exception -instance:: +instance: + +.. sourcecode:: ipython In [81]: mec.execute('1/0') --------------------------------------------------------------------------- @@ -734,7 +777,9 @@ instance:: The above example appears to be broken right now because of a change in how we are using Twisted. -All of this same error handling magic even works in non-blocking mode:: +All of this same error handling magic even works in non-blocking mode: + +.. sourcecode:: ipython In [83]: mec.block=False diff --git a/docs/source/parallel/parallel_process.txt b/docs/source/parallel/parallel_process.txt new file mode 100644 index 0000000..660d06d --- /dev/null +++ b/docs/source/parallel/parallel_process.txt @@ -0,0 +1,251 @@ +.. _parallel_process: + +=========================================== +Starting the IPython controller and engines +=========================================== + +To use IPython for parallel computing, you need to start one instance of +the controller and one or more instances of the engine. The controller +and each engine can run on different machines or on the same machine. +Because of this, there are many different possibilities. + +Broadly speaking, there are two ways of going about starting a controller and engines: + +* In an automated manner using the :command:`ipcluster` command. +* In a more manual way using the :command:`ipcontroller` and + :command:`ipengine` commands. + +This document describes both of these methods. We recommend that new users start with the :command:`ipcluster` command as it simplifies many common usage cases. + +General considerations +====================== + +Before delving into the details about how you can start a controller and engines using the various methods, we outline some of the general issues that come up when starting the controller and engines. These things come up no matter which method you use to start your IPython cluster. + +Let's say that you want to start the controller on ``host0`` and engines on hosts ``host1``-``hostn``. The following steps are then required: + +1. Start the controller on ``host0`` by running :command:`ipcontroller` on + ``host0``. +2. Move the FURL file (:file:`ipcontroller-engine.furl`) created by the + controller from ``host0`` to hosts ``host1``-``hostn``. +3. Start the engines on hosts ``host1``-``hostn`` by running + :command:`ipengine`. This command has to be told where the FURL file + (:file:`ipcontroller-engine.furl`) is located. + +At this point, the controller and engines will be connected. By default, the +FURL files created by the controller are put into the +:file:`~/.ipython/security` directory. If the engines share a filesystem with +the controller, step 2 can be skipped as the engines will automatically look +at that location. + +The final step required required to actually use the running controller from a +client is to move the FURL files :file:`ipcontroller-mec.furl` and +:file:`ipcontroller-tc.furl` from ``host0`` to the host where the clients will +be run. If these file are put into the :file:`~/.ipython/security` directory of the client's host, they will be found automatically. Otherwise, the full path to them has to be passed to the client's constructor. + +Using :command:`ipcluster` +========================== + +The :command:`ipcluster` command provides a simple way of starting a controller and engines in the following situations: + +1. When the controller and engines are all run on localhost. This is useful + for testing or running on a multicore computer. +2. When engines are started using the :command:`mpirun` command that comes + with most MPI [MPI]_ implementations +3. When engines are started using the PBS [PBS]_ batch system. + +.. note:: + + It is also possible for advanced users to add support to + :command:`ipcluster` for starting controllers and engines using other + methods (like Sun's Grid Engine for example). + +.. note:: + + Currently :command:`ipcluster` requires that the + :file:`~/.ipython/security` directory live on a shared filesystem that is + seen by both the controller and engines. If you don't have a shared file + system you will need to use :command:`ipcontroller` and + :command:`ipengine` directly. + +Underneath the hood, :command:`ipcluster` just uses :command:`ipcontroller` +and :command:`ipengine` to perform the steps described above. + +Using :command:`ipcluster` in local mode +---------------------------------------- + +To start one controller and 4 engines on localhost, just do:: + + $ ipcluster local -n 4 + +To see other command line options for the local mode, do:: + + $ ipcluster local -h + +Using :command:`ipcluster` in mpirun mode +----------------------------------------- + +The mpirun mode is useful if you: + +1. Have MPI installed. +2. Your systems are configured to use the :command:`mpirun` command to start + processes. + +If these are satisfied, you can start an IPython cluster using:: + + $ ipcluster mpirun -n 4 + +This does the following: + +1. Starts the IPython controller on current host. +2. Uses :command:`mpirun` to start 4 engines. + +On newer MPI implementations (such as OpenMPI), this will work even if you don't make any calls to MPI or call :func:`MPI_Init`. However, older MPI implementations actually require each process to call :func:`MPI_Init` upon starting. The easiest way of having this done is to install the mpi4py [mpi4py]_ package and then call ipcluster with the ``--mpi`` option:: + + $ ipcluster mpirun -n 4 --mpi=mpi4py + +Unfortunately, even this won't work for some MPI implementations. If you are having problems with this, you will likely have to use a custom Python executable that itself calls :func:`MPI_Init` at the appropriate time. Fortunately, mpi4py comes with such a custom Python executable that is easy to install and use. However, this custom Python executable approach will not work with :command:`ipcluster` currently. + +Additional command line options for this mode can be found by doing:: + + $ ipcluster mpirun -h + +More details on using MPI with IPython can be found :ref:`here `. + + +Using :command:`ipcluster` in PBS mode +-------------------------------------- + +The PBS mode uses the Portable Batch System [PBS]_ to start the engines. To use this mode, you first need to create a PBS script template that will be used to start the engines. Here is a sample PBS script template: + +.. sourcecode:: bash + + #PBS -N ipython + #PBS -j oe + #PBS -l walltime=00:10:00 + #PBS -l nodes=${n/4}:ppn=4 + #PBS -q parallel + + cd $$PBS_O_WORKDIR + export PATH=$$HOME/usr/local/bin + export PYTHONPATH=$$HOME/usr/local/lib/python2.4/site-packages + /usr/local/bin/mpiexec -n ${n} ipengine --logfile=$$PBS_O_WORKDIR/ipengine + +There are a few important points about this template: + +1. This template will be rendered at runtime using IPython's :mod:`Itpl` + template engine. + +2. Instead of putting in the actual number of engines, use the notation + ``${n}`` to indicate the number of engines to be started. You can also uses + expressions like ``${n/4}`` in the template to indicate the number of + nodes. + +3. Because ``$`` is a special character used by the template engine, you must + escape any ``$`` by using ``$$``. This is important when referring to + environment variables in the template. + +4. Any options to :command:`ipengine` should be given in the batch script + template. + +5. Depending on the configuration of you system, you may have to set + environment variables in the script template. + +Once you have created such a script, save it with a name like :file:`pbs.template`. Now you are ready to start your job:: + + $ ipcluster pbs -n 128 --pbs-script=pbs.template + +Additional command line options for this mode can be found by doing:: + + $ ipcluster pbs -h + +Using the :command:`ipcontroller` and :command:`ipengine` commands +================================================================== + +It is also possible to use the :command:`ipcontroller` and :command:`ipengine` commands to start your controller and engines. This approach gives you full control over all aspects of the startup process. + +Starting the controller and engine on your local machine +-------------------------------------------------------- + +To use :command:`ipcontroller` and :command:`ipengine` to start things on your +local machine, do the following. + +First start the controller:: + + $ ipcontroller + +Next, start however many instances of the engine you want using (repeatedly) the command:: + + $ ipengine + +The engines should start and automatically connect to the controller using the FURL files in :file:`~./ipython/security`. You are now ready to use the controller and engines from IPython. + +.. warning:: + + The order of the above operations is very important. You *must* + start the controller before the engines, since the engines connect + to the controller as they get started. + +.. note:: + + On some platforms (OS X), to put the controller and engine into the + background you may need to give these commands in the form ``(ipcontroller + &)`` and ``(ipengine &)`` (with the parentheses) for them to work + properly. + +Starting the controller and engines on different hosts +------------------------------------------------------ + +When the controller and engines are running on different hosts, things are +slightly more complicated, but the underlying ideas are the same: + +1. Start the controller on a host using :command:`ipcontroller`. +2. Copy :file:`ipcontroller-engine.furl` from :file:`~./ipython/security` on the controller's host to the host where the engines will run. +3. Use :command:`ipengine` on the engine's hosts to start the engines. + +The only thing you have to be careful of is to tell :command:`ipengine` where the :file:`ipcontroller-engine.furl` file is located. There are two ways you can do this: + +* Put :file:`ipcontroller-engine.furl` in the :file:`~./ipython/security` + directory on the engine's host, where it will be found automatically. +* Call :command:`ipengine` with the ``--furl-file=full_path_to_the_file`` + flag. + +The ``--furl-file`` flag works like this:: + + $ ipengine --furl-file=/path/to/my/ipcontroller-engine.furl + +.. note:: + + If the controller's and engine's hosts all have a shared file system + (:file:`~./ipython/security` is the same on all of them), then things + will just work! + +Make FURL files persistent +--------------------------- + +At fist glance it may seem that that managing the FURL files is a bit annoying. Going back to the house and key analogy, copying the FURL around each time you start the controller is like having to make a new key every time you want to unlock the door and enter your house. As with your house, you want to be able to create the key (or FURL file) once, and then simply use it at any point in the future. + +This is possible. The only thing you have to do is decide what ports the controller will listen on for the engines and clients. This is done as follows:: + + $ ipcontroller -r --client-port=10101 --engine-port=10102 + +Then, just copy the furl files over the first time and you are set. You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to use the *same* ports. + +.. note:: + + You may ask the question: what ports does the controller listen on if you + don't tell is to use specific ones? The default is to use high random port + numbers. We do this for two reasons: i) to increase security through + obscurity and ii) to multiple controllers on a given host to start and + automatically use different ports. + +Log files +--------- + +All of the components of IPython have log files associated with them. +These log files can be extremely useful in debugging problems with +IPython and can be found in the directory :file:`~/.ipython/log`. Sending +the log files to us will often help us to debug any problems. + + +.. [PBS] Portable Batch System. http://www.openpbs.org/ diff --git a/docs/source/parallel/parallel_security.txt b/docs/source/parallel/parallel_security.txt index bacca35..abd876c 100644 --- a/docs/source/parallel/parallel_security.txt +++ b/docs/source/parallel/parallel_security.txt @@ -1,6 +1,5 @@ .. _parallelsecurity: - =========================== Security details of IPython =========================== @@ -73,31 +72,30 @@ engines and controller are fully encrypted and authenticated. This section describes the details of the encryption and authentication approached used within IPython. -IPython uses the `Foolscap `_ network -protocol for all communications between processes. Thus, the details of -IPython's security model are directly related to those of Foolscap. Thus, much -of the following discussion is actually just a discussion of the security that -is built in to Foolscap. +IPython uses the Foolscap network protocol [Foolscap]_ for all communications +between processes. Thus, the details of IPython's security model are directly +related to those of Foolscap. Thus, much of the following discussion is +actually just a discussion of the security that is built in to Foolscap. Encryption ---------- For encryption purposes, IPython and Foolscap use the well known Secure Socket -Layer (SSL) protocol (`RFC5246 `_). We use -the implementation of this protocol provided by the OpenSSL project through -the `pyOpenSSL `_ Python bindings to OpenSSL. +Layer (SSL) protocol [RFC5246]_. We use the implementation of this protocol +provided by the OpenSSL project through the pyOpenSSL [pyOpenSSL]_ Python +bindings to OpenSSL. Authentication -------------- IPython clients and engines must also authenticate themselves with the -controller. This is handled in a `capabilities based security model -`_. In this model, the -controller creates a strong cryptographic key or token that represents each -set of capability that the controller offers. Any party who has this key and -presents it to the controller has full access to the corresponding -capabilities of the controller. This model is analogous to using a physical -key to gain access to physical items (capabilities) behind a locked door. +controller. This is handled in a capabilities based security model +[Capability]_. In this model, the controller creates a strong cryptographic +key or token that represents each set of capability that the controller +offers. Any party who has this key and presents it to the controller has full +access to the corresponding capabilities of the controller. This model is +analogous to using a physical key to gain access to physical items +(capabilities) behind a locked door. For a capabilities based authentication system to prevent unauthorized access, two things must be ensured: @@ -118,7 +116,7 @@ controller creates a number of FURLs for different purposes: to execute. Upon starting, the controller creates these different FURLS and writes them -files in the user-read-only directory $HOME/.ipython/security. Thus, only the +files in the user-read-only directory :file:`$HOME/.ipython/security`. Thus, only the user who starts the controller has access to the FURLs. For an IPython client or engine to authenticate with a controller, it must @@ -361,3 +359,5 @@ the certificate of their peer. This mutual authentication is handled by the SSL handshake and is separate and independent from the additional authentication steps that the CLIENT and SERVER perform after an encrypted channel is established. + +.. [RFC5246] diff --git a/docs/source/parallel/parallel_task.txt b/docs/source/parallel/parallel_task.txt index a307439..14d4565 100644 --- a/docs/source/parallel/parallel_task.txt +++ b/docs/source/parallel/parallel_task.txt @@ -4,8 +4,6 @@ The IPython task interface ========================== -.. contents:: - The task interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the multiengine interface, in the task interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful. Best of all the user can use both of these interfaces running at the same time to take advantage or both of their strengths. When the user can break up the user's work into segments that do not depend on previous execution, the task interface is ideal. But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign tasks to engines explicitly. @@ -17,7 +15,7 @@ To follow along with this tutorial, you will need to start the IPython controller and four IPython engines. The simplest way of doing this is to use the :command:`ipcluster` command:: - $ ipcluster -n 4 + $ ipcluster local -n 4 For more detailed information about starting the controller and engines, see our :ref:`introduction ` to using IPython for parallel computing. @@ -26,7 +24,9 @@ Creating a ``TaskClient`` instance ========================================= The first step is to import the IPython :mod:`IPython.kernel.client` module -and then create a :class:`TaskClient` instance:: +and then create a :class:`TaskClient` instance: + +.. sourcecode:: ipython In [1]: from IPython.kernel import client @@ -34,10 +34,12 @@ and then create a :class:`TaskClient` instance:: This form assumes that the :file:`ipcontroller-tc.furl` is in the :file:`~./ipython/security` directory on the client's host. If not, the -location of the ``.furl`` file must be given as an argument to the -constructor:: +location of the FURL file must be given as an argument to the +constructor: + +.. sourcecode:: ipython - In[2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl') + In [2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl') Quick and easy parallelism ========================== @@ -47,7 +49,9 @@ In many cases, you simply want to apply a Python function to a sequence of objec Parallel map ------------ -The parallel :meth:`map` in the task interface is similar to that in the multiengine interface:: +The parallel :meth:`map` in the task interface is similar to that in the multiengine interface: + +.. sourcecode:: ipython In [63]: serial_result = map(lambda x:x**10, range(32)) @@ -59,7 +63,9 @@ The parallel :meth:`map` in the task interface is similar to that in the multien Parallel function decorator --------------------------- -Parallel functions are just like normal function, but they can be called on sequences and *in parallel*. The multiengine interface provides a decorator that turns any Python function into a parallel function:: +Parallel functions are just like normal function, but they can be called on sequences and *in parallel*. The multiengine interface provides a decorator that turns any Python function into a parallel function: + +.. sourcecode:: ipython In [10]: @tc.parallel() ....: def f(x):