Using various batch systems with :command:`ipcluster`

:command:`ipcluster` has a notion of Launchers that can start controllers and engines with various remote execution schemes. Currently supported models include :command:`ssh`, :command:`mpiexec`, PBS-style (Torque, SGE, LSF), and Windows HPC Server.

In general, these are configured by the :attr:`IPClusterEngines.engine_set_launcher_class`, and :attr:`IPClusterStart.controller_launcher_class` configurables, which can be the fully specified object name (e.g. 'IPython.parallel.apps.launcher.LocalControllerLauncher'), but if you are using IPython's builtin launchers, you can specify just the class name, or even just the prefix e.g:

c.IPClusterEngines.engine_launcher_class = 'SSH'
# equivalent to
c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher'
# both of which expand to
c.IPClusterEngines.engine_launcher_class = 'IPython.parallel.apps.launcher.SSHEngineSetLauncher'

The shortest form being of particular use on the command line, where all you need to do to get an IPython cluster running with engines started with MPI is:

$> ipcluster start --engines=MPI

Assuming that the default MPI config is sufficient.

Using :command:`ipcluster` in mpiexec/mpirun mode

The mpiexec/mpirun mode is useful if you:

1. Have MPI installed.
2. Your systems are configured to use the :command:`mpiexec` or :command:`mpirun` commands to start MPI processes.

If these are satisfied, you can create a new profile:

$ ipython profile create --parallel --profile=mpi

and edit the file :file:`IPYTHONDIR/profile_mpi/ipcluster_config.py`.

There, instruct ipcluster to use the MPI launchers by adding the lines:

c.IPClusterEngines.engine_launcher_class = 'MPIEngineSetLauncher'

If the default MPI configuration is correct, then you can now start your cluster, with:

$ ipcluster start -n 4 --profile=mpi

This does the following:

1. Starts the IPython controller on current host.
2. Uses :command:`mpiexec` to start 4 engines.

If you have a reason to also start the Controller with mpi, you can specify:

c.IPClusterStart.controller_launcher_class = 'MPIControllerLauncher'

On newer MPI implementations (such as OpenMPI), this will work even if you don't make any calls to MPI or call :func:`MPI_Init`. However, older MPI implementations actually require each process to call :func:`MPI_Init` upon starting. The easiest way of having this done is to install the mpi4py [mpi4py]_ package and then specify the c.MPI.use option in :file:`ipengine_config.py`:

c.MPI.use = 'mpi4py'

Unfortunately, even this won't work for some MPI implementations. If you are having problems with this, you will likely have to use a custom Python executable that itself calls :func:`MPI_Init` at the appropriate time. Fortunately, mpi4py comes with such a custom Python executable that is easy to install and use. However, this custom Python executable approach will not work with :command:`ipcluster` currently.

More details on using MPI with IPython can be found :ref:`here <parallelmpi>`.

Using :command:`ipcluster` in PBS mode

The PBS mode uses the Portable Batch System (PBS) to start the engines.

As usual, we will start by creating a fresh profile:

$ ipython profile create --parallel --profile=pbs

And in :file:`ipcluster_config.py`, we will select the PBS launchers for the controller and engines:

c.IPClusterStart.controller_launcher_class = 'PBSControllerLauncher'
c.IPClusterEngines.engine_launcher_class = 'PBSEngineSetLauncher'

IPython does provide simple default batch templates for PBS and SGE, but you may need to specify your own. Here is a sample PBS script template:

#PBS -N ipython
#PBS -j oe
#PBS -l walltime=00:10:00
#PBS -l nodes={n/4}:ppn=4
#PBS -q {queue}

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
export PYTHONPATH=$HOME/usr/local/lib/python2.7/site-packages
/usr/local/bin/mpiexec -n {n} ipengine --profile-dir={profile_dir}

There are a few important points about this template:

1. This template will be rendered at runtime using IPython's :class:`EvalFormatter`. This is simply a subclass of :class:`string.Formatter` that allows simple expressions on keys.
2. Instead of putting in the actual number of engines, use the notation {n} to indicate the number of engines to be started. You can also use expressions like {n/4} in the template to indicate the number of nodes. There will always be {n} and {profile_dir} variables passed to the formatter. These allow the batch system to know how many engines, and where the configuration files reside. The same is true for the batch queue, with the template variable {queue}.
3. Any options to :command:`ipengine` can be given in the batch script template, or in :file:`ipengine_config.py`.
4. Depending on the configuration of you system, you may have to set environment variables in the script template.

The controller template should be similar, but simpler:

#PBS -N ipython
#PBS -j oe
#PBS -l walltime=00:10:00
#PBS -l nodes=1:ppn=4
#PBS -q {queue}

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
export PYTHONPATH=$HOME/usr/local/lib/python2.7/site-packages
ipcontroller --profile-dir={profile_dir}

Once you have created these scripts, save them with names like :file:`pbs.engine.template`. Now you can load them into the :file:`ipcluster_config` with:

c.PBSEngineSetLauncher.batch_template_file = "pbs.engine.template"

c.PBSControllerLauncher.batch_template_file = "pbs.controller.template"

Alternately, you can just define the templates as strings inside :file:`ipcluster_config`.

Whether you are using your own templates or our defaults, the extra configurables available are the number of engines to launch ({n}, and the batch system queue to which the jobs are to be submitted ({queue})). These are configurables, and can be specified in :file:`ipcluster_config`:

c.PBSLauncher.queue = 'veryshort.q'
c.IPClusterEngines.n = 64

Note that assuming you are running PBS on a multi-node cluster, the Controller's default behavior of listening only on localhost is likely too restrictive. In this case, also assuming the nodes are safely behind a firewall, you can simply instruct the Controller to listen for connections on all its interfaces, by adding in :file:`ipcontroller_config`:

c.HubFactory.ip = '*'

You can now run the cluster with:

$ ipcluster start --profile=pbs -n 128

Additional configuration options can be found in the PBS section of :file:`ipcluster_config`.

Using :command:`ipcluster` in SSH mode

The SSH mode uses :command:`ssh` to execute :command:`ipengine` on remote nodes and :command:`ipcontroller` can be run remotely as well, or on localhost.

As usual, we start by creating a clean profile:

$ ipython profile create --parallel --profile=ssh

To use this mode, select the SSH launchers in :file:`ipcluster_config.py`:

c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher'
# and if the Controller is also to be remote:
c.IPClusterStart.controller_launcher_class = 'SSHControllerLauncher'

The controller's remote location and configuration can be specified:

# Set the user and hostname for the controller
# c.SSHControllerLauncher.hostname = 'controller.example.com'
# c.SSHControllerLauncher.user = os.environ.get('USER','username')

# Set the arguments to be passed to ipcontroller
# note that remotely launched ipcontroller will not get the contents of
# the local ipcontroller_config.py unless it resides on the *remote host*
# in the location specified by the `profile-dir` argument.
# c.SSHControllerLauncher.controller_args = ['--reuse', '--ip=*', '--profile-dir=/path/to/cd']

Engines are specified in a dictionary, by hostname and the number of engines to be run on that host.

c.SSHEngineSetLauncher.engines = { 'host1.example.com' : 2,
            'host2.example.com' : 5,
            'host3.example.com' : (1, ['--profile-dir=/home/different/location']),
            'host4.example.com' : 8 }

• The engines dict, where the keys are the host we want to run engines on and the value is the number of engines to run on that host.
• on host3, the value is a tuple, where the number of engines is first, and the arguments to be passed to :command:`ipengine` are the second element.

For engines without explicitly specified arguments, the default arguments are set in a single location:

c.SSHEngineSetLauncher.engine_args = ['--profile-dir=/path/to/profile_ssh']

Current limitations of the SSH mode of :command:`ipcluster` are:

• Untested and unsupported on Windows. Would require a working :command:`ssh` on Windows. Also, we are using shell scripts to setup and execute commands on remote hosts.

c.SSHLauncher.to_send = []
c.SSHLauncher.to_fetch = []

Starting the controller and engine on your local machine

To use :command:`ipcontroller` and :command:`ipengine` to start things on your local machine, do the following.

First start the controller:

$ ipcontroller

Next, start however many instances of the engine you want using (repeatedly) the command:

$ ipengine

The engines should start and automatically connect to the controller using the JSON files in :file:`IPYTHONDIR/profile_default/security`. You are now ready to use the controller and engines from IPython.

Starting the controller and engines on different hosts

When the controller and engines are running on different hosts, things are slightly more complicated, but the underlying ideas are the same:

1. Start the controller on a host using :command:`ipcontroller`. The controller must be instructed to listen on an interface visible to the engine machines, via the ip command-line argument or HubFactory.ip in :file:`ipcontroller_config.py`:

   $ ipcontroller --ip=192.168.1.16
   

   # in ipcontroller_config.py
   HubFactory.ip = '192.168.1.16'
   

2. Copy :file:`ipcontroller-engine.json` from :file:`IPYTHONDIR/profile_<name>/security` on the controller's host to the host where the engines will run.

3. Use :command:`ipengine` on the engine's hosts to start the engines.

The only thing you have to be careful of is to tell :command:`ipengine` where the :file:`ipcontroller-engine.json` file is located. There are two ways you can do this:

• Put :file:`ipcontroller-engine.json` in the :file:`IPYTHONDIR/profile_<name>/security` directory on the engine's host, where it will be found automatically.
• Call :command:`ipengine` with the --file=full_path_to_the_file flag.

The file flag works like this:

$ ipengine --file=/path/to/my/ipcontroller-engine.json

{
  "url":"tcp://192.168.1.123:56951",
  "exec_key":"26f4c040-587d-4a4e-b58b-030b96399584",
  "ssh":"user@example.com",
  "location":"192.168.1.123"
}

$> ipengine --profile=foo --ssh=my.login.node

[node1] $> ipcontroller --enginessh=node1
[node2] $> ipengine
[node3] $> ipcluster engines --n=4

An example using ipcontroller/engine with ssh

No configuration files are necessary to use ipcontroller/engine in an SSH environment without a shared filesystem. You simply need to make sure that the controller is listening on an interface visible to the engines, and move the connection file from the controller to the engines.

1. start the controller, listening on an ip-address visible to the engine machines:

   [controller.host] $ ipcontroller --ip=192.168.1.16
   
   [IPControllerApp] Using existing profile dir: u'/Users/me/.ipython/profile_default'
   [IPControllerApp] Hub listening on tcp://192.168.1.16:63320 for registration.
   [IPControllerApp] Hub using DB backend: 'IPython.parallel.controller.dictdb.DictDB'
   [IPControllerApp] hub::created hub
   [IPControllerApp] writing connection info to /Users/me/.ipython/profile_default/security/ipcontroller-client.json
   [IPControllerApp] writing connection info to /Users/me/.ipython/profile_default/security/ipcontroller-engine.json
   [IPControllerApp] task::using Python leastload Task scheduler
   [IPControllerApp] Heartmonitor started
   [IPControllerApp] Creating pid file: /Users/me/.ipython/profile_default/pid/ipcontroller.pid
   Scheduler started [leastload]
   

2. on each engine, fetch the connection file with scp:

   [engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ./
   

   Note

   The log output of ipcontroller above shows you where the json files were written. They will be in :file:`~/.ipython` under :file:`profile_default/security/ipcontroller-engine.json`

3. start the engines, using the connection file:

   [engine.host.n] $ ipengine --file=./ipcontroller-engine.json
   

A couple of notes:

• You can avoid having to fetch the connection file every time by adding --reuse flag to ipcontroller, which instructs the controller to read the previous connection file for connection info, rather than generate a new one with randomized ports.

• In step 2, if you fetch the connection file directly into the security dir of a profile, then you need not specify its path directly, only the profile (assumes the path exists, otherwise you must create it first):

  [engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ~/.ipython/profile_ssh/security/
  [engine.host.n] $ ipengine --profile=ssh
  

  Of course, if you fetch the file into the default profile, no arguments must be passed to ipengine at all.

• Note that ipengine did not specify the ip argument. In general, it is unlikely for any connection information to be specified at the command-line to ipengine, as all of this information should be contained in the connection file written by ipcontroller.

Make JSON files persistent

At fist glance it may seem that that managing the JSON files is a bit annoying. Going back to the house and key analogy, copying the JSON around each time you start the controller is like having to make a new key every time you want to unlock the door and enter your house. As with your house, you want to be able to create the key (or JSON file) once, and then simply use it at any point in the future.

To do this, the only thing you have to do is specify the --reuse flag, so that the connection information in the JSON files remains accurate:

$ ipcontroller --reuse

Then, just copy the JSON files over the first time and you are set. You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to reuse the file.

Log files

All of the components of IPython have log files associated with them. These log files can be extremely useful in debugging problems with IPython and can be found in the directory :file:`IPYTHONDIR/profile_<name>/log`. Sending the log files to us will often help us to debug any problems.

Configuring ipcontroller

The IPython Controller takes its configuration from the file :file:`ipcontroller_config.py` in the active profile directory.

c.HubFactory.ip = '*'

# allow connections on all interfaces from engines
# engines on the same node will use loopback, while engines
# from other nodes will use an external IP
c.HubFactory.ip = '*'

# you typically only need to specify the location when there are extra
# interfaces that may not be visible to peer nodes (e.g. VM interfaces)
c.HubFactory.location = '10.0.1.5'
# or to get an automatic value, try this:
import socket
hostname = socket.gethostname()
# alternate choices for hostname include `socket.getfqdn()`
# or `socket.gethostname() + '.local'`

ex_ip = socket.gethostbyname_ex(hostname)[-1][-1]
c.HubFactory.location = ex_ip

# now instruct clients to use the login node for SSH tunnels:
c.HubFactory.ssh_server = 'login.mycluster.net'

{
  "url":"tcp:\/\/*:43447",
  "exec_key":"9c7779e4-d08a-4c3b-ba8e-db1f80b562c1",
  "ssh":"login.mycluster.net",
  "location":"10.0.1.5"
}

# for a simple dict-based in-memory implementation, use dictdb
# This is the default and the fastest, since it doesn't involve the filesystem
c.HubFactory.db_class = 'IPython.parallel.controller.dictdb.DictDB'

# To use MongoDB:
c.HubFactory.db_class = 'IPython.parallel.controller.mongodb.MongoDB'

# and SQLite:
c.HubFactory.db_class = 'IPython.parallel.controller.sqlitedb.SQLiteDB'

# You can use NoDB to disable the database altogether, in case you don't need
# to reuse tasks or results, and want to keep memory consumption under control.
c.HubFactory.db_class = 'IPython.parallel.controller.dictdb.NoDB'

# To keep persistant task history in MongoDB:
c.MongoDB.database = 'tasks'

# and in SQLite:
c.SQLiteDB.table = 'tasks'

# positional args to pymongo.Connection
c.MongoDB.connection_args = []

# keyword args to pymongo.Connection
c.MongoDB.connection_kwargs = {}

Configuring ipengine

The IPython Engine takes its configuration from the file :file:`ipengine_config.py`

The Engine itself also has some amount of configuration. Most of this has to do with initializing MPI or connecting to the controller.

To instruct the Engine to initialize with an MPI environment set up by mpi4py, add:

c.MPI.use = 'mpi4py'

In this case, the Engine will use our default mpi4py init script to set up the MPI environment prior to exection. We have default init scripts for mpi4py and pytrilinos. If you want to specify your own code to be run at the beginning, specify c.MPI.init_script.

You can also specify a file or python command to be run at startup of the Engine:

c.IPEngineApp.startup_script = u'/path/to/my/startup.py'

c.IPEngineApp.startup_command = 'import numpy, scipy, mpi4py'

These commands/files will be run again, after each

It's also useful on systems with shared filesystems to run the engines in some scratch directory. This can be set with:

c.IPEngineApp.work_dir = u'/path/to/scratch/'