##// END OF EJS Templates
Move crash handling to the application level and simplify class structure....
Move crash handling to the application level and simplify class structure. Starting to try to take real advantage of the refactoring, to have generic crash handling. This also lets us initialize the app without needing all the self.attempt() wrappers, since now there's a good system-wide crash handler at the app level (not inside the shell instance). I didn't yet remove the attempt() method because we may have occasional uses for it (we still do, but in one place only). I also removed some extra class layers that weren't quite needed. Creating classes solely for the purpose of passing parameters makes the code (IMO) harder to understand, I kept getting lost in parts of the class hierarchy. I think these changes provide the same flexibility but with easier to follow code (less things to remember, basically). What I tried to do was to use argument passing instead of inheritance for all cases I saw where the inheritance wasn't really adding new functionality. In some cases, this actually allowed me to remove methods that were effectively duplicated in the subclasses.

File last commit:

r2342:9e3db2d3
r2403:08d6ac78
Show More
ipcluster_config.py
184 lines | 6.7 KiB | text/x-python | PythonLexer
Brian Granger
Adding files for the refactored kernel scripts.
r2304 import os
c = get_config()
Brian Granger
Work on default config files and docstrings....
r2310 #-----------------------------------------------------------------------------
# Select which launchers to use
#-----------------------------------------------------------------------------
# This allows you to control what method is used to start the controller
# and engines. The following methods are currently supported:
Brian Granger
More work on the launchers and Win HPC support.
r2333 # - Start as a regular process on localhost.
# - Start using mpiexec.
# - Start using the Windows HPC Server 2008 scheduler
# - Start using PBS
# - Start using SSH (currently broken)
Brian Granger
Work on default config files and docstrings....
r2310
# The selected launchers can be configured below.
Brian Granger
More work on the launchers and Win HPC support.
r2333 # Options are:
# - LocalControllerLauncher
# - MPIExecControllerLauncher
# - PBSControllerLauncher
# - WindowsHPCControllerLauncher
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.Global.controller_launcher = 'IPython.kernel.launcher.LocalControllerLauncher'
Brian Granger
More work on the launchers and Win HPC support.
r2333 # Options are:
# - LocalEngineSetLauncher
# - MPIExecEngineSetLauncher
# - PBSEngineSetLauncher
# - WindowsHPCEngineSetLauncher
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.Global.engine_launcher = 'IPython.kernel.launcher.LocalEngineSetLauncher'
Brian Granger
Work on default config files and docstrings....
r2310 #-----------------------------------------------------------------------------
# Global configuration
#-----------------------------------------------------------------------------
Brian Granger
More work on the launchers and Win HPC support.
r2333 # The default number of engines that will be started. This is overridden by
Brian Granger
Work on default config files and docstrings....
r2310 # the -n command line option: "ipcluster start -n 4"
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.Global.n = 2
Brian Granger
Work on default config files and docstrings....
r2310 # Log to a file in cluster_dir/log, otherwise just log to sys.stdout.
# c.Global.log_to_file = False
Brian Granger
Adding files for the refactored kernel scripts.
r2304
Brian Granger
Work on default config files and docstrings....
r2310 # Remove old logs from cluster_dir/log before starting.
# c.Global.clean_logs = True
Brian Granger
Adding files for the refactored kernel scripts.
r2304
Brian Granger
The cluster applications now have a working directory option.....
r2330 # The working directory for the process. The application will use os.chdir
# to change to this directory before starting.
Brian Granger
Fixing how the working directory is handled in kernel....
r2336 # c.Global.work_dir = os.getcwd()
Brian Granger
The cluster applications now have a working directory option.....
r2330
Brian Granger
More work on the launchers and Win HPC support.
r2333
Brian Granger
Work on default config files and docstrings....
r2310 #-----------------------------------------------------------------------------
Brian Granger
More work on the launchers and Win HPC support.
r2333 # Local process launchers
Brian Granger
Work on default config files and docstrings....
r2310 #-----------------------------------------------------------------------------
Brian Granger
Adding files for the refactored kernel scripts.
r2304
Brian Granger
Work on default config files and docstrings....
r2310 # The command line arguments to call the controller with.
# c.LocalControllerLauncher.controller_args = \
# ['--log-to-file','--log-level', '40']
Brian Granger
More work on the launchers and Win HPC support.
r2333 # The working directory for the controller
Brian Granger
Fixing how the working directory is handled in kernel....
r2336 # c.LocalEngineSetLauncher.work_dir = u''
Brian Granger
More work on the launchers and Win HPC support.
r2333
# Command line argument passed to the engines.
# c.LocalEngineSetLauncher.engine_args = ['--log-to-file','--log-level', '40']
#-----------------------------------------------------------------------------
# MPIExec launchers
#-----------------------------------------------------------------------------
Brian Granger
Work on default config files and docstrings....
r2310 # The mpiexec/mpirun command to use in started the controller.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.MPIExecControllerLauncher.mpi_cmd = ['mpiexec']
Brian Granger
Work on default config files and docstrings....
r2310
# Additional arguments to pass to the actual mpiexec command.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.MPIExecControllerLauncher.mpi_args = []
Brian Granger
Work on default config files and docstrings....
r2310 # The command line argument to call the controller with.
# c.MPIExecControllerLauncher.controller_args = \
# ['--log-to-file','--log-level', '40']
Brian Granger
More work on the launchers and Win HPC support.
r2333
# The mpiexec/mpirun command to use in started the controller.
# c.MPIExecEngineSetLauncher.mpi_cmd = ['mpiexec']
# Additional arguments to pass to the actual mpiexec command.
# c.MPIExecEngineSetLauncher.mpi_args = []
# Command line argument passed to the engines.
# c.MPIExecEngineSetLauncher.engine_args = ['--log-to-file','--log-level', '40']
# The default number of engines to start if not given elsewhere.
# c.MPIExecEngineSetLauncher.n = 1
#-----------------------------------------------------------------------------
# SSH launchers
#-----------------------------------------------------------------------------
# Todo
#-----------------------------------------------------------------------------
# Unix batch (PBS) schedulers launchers
#-----------------------------------------------------------------------------
Brian Granger
Work on default config files and docstrings....
r2310 # The command line program to use to submit a PBS job.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSControllerLauncher.submit_command = 'qsub'
Brian Granger
Work on default config files and docstrings....
r2310
# The command line program to use to delete a PBS job.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSControllerLauncher.delete_command = 'qdel'
Brian Granger
Work on default config files and docstrings....
r2310
# A regular expression that takes the output of qsub and find the job id.
Brian Granger
Fixing minor bugs in IPython.kernel....
r2342 # c.PBSControllerLauncher.job_id_regexp = r'\d+'
Brian Granger
Work on default config files and docstrings....
r2310
# The batch submission script used to start the controller. This is where
# environment variables would be setup, etc. This string is interpolated using
Brian Granger
Fixing how the working directory is handled in kernel....
r2336 # the Itpl module in IPython.external. Basically, you can use ${n} for the
# number of engine and ${cluster_dir} for the cluster_dir.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSControllerLauncher.batch_template = """"""
Brian Granger
Work on default config files and docstrings....
r2310 # The name of the instantiated batch script that will actually be used to
# submit the job. This will be written to the cluster directory.
# c.PBSControllerLauncher.batch_file_name = u'pbs_batch_script_controller'
# The command line program to use to submit a PBS job.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSEngineSetLauncher.submit_command = 'qsub'
Brian Granger
Work on default config files and docstrings....
r2310
# The command line program to use to delete a PBS job.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSEngineSetLauncher.delete_command = 'qdel'
Brian Granger
Work on default config files and docstrings....
r2310
# A regular expression that takes the output of qsub and find the job id.
Brian Granger
Fixing minor bugs in IPython.kernel....
r2342 # c.PBSEngineSetLauncher.job_id_regexp = r'\d+'
Brian Granger
Work on default config files and docstrings....
r2310
# The batch submission script used to start the engines. This is where
# environment variables would be setup, etc. This string is interpolated using
# the Itpl module in IPython.external. Basically, you can use ${n} for the
Brian Granger
Fixing how the working directory is handled in kernel....
r2336 # number of engine and ${cluster_dir} for the cluster_dir.
Brian Granger
Adding files for the refactored kernel scripts.
r2304 # c.PBSEngineSetLauncher.batch_template = """"""
Brian Granger
Work on default config files and docstrings....
r2310
# The name of the instantiated batch script that will actually be used to
# submit the job. This will be written to the cluster directory.
# c.PBSEngineSetLauncher.batch_file_name = u'pbs_batch_script_engines'
#-----------------------------------------------------------------------------
Brian Granger
More work on the launchers and Win HPC support.
r2333 # Windows HPC Server 2008 launcher configuration
Brian Granger
Work on default config files and docstrings....
r2310 #-----------------------------------------------------------------------------
Brian Granger
More work on the launchers and Win HPC support.
r2333 # c.IPControllerJob.job_name = 'IPController'
# c.IPControllerJob.is_exclusive = False
Brian Granger
Fixing minor bugs in IPython.kernel....
r2342 # c.IPControllerJob.username = r'USERDOMAIN\USERNAME'
Brian Granger
More work on the launchers and Win HPC support.
r2333 # c.IPControllerJob.priority = 'Highest'
# c.IPControllerJob.requested_nodes = ''
# c.IPControllerJob.project = 'MyProject'
# c.IPControllerTask.task_name = 'IPController'
# c.IPControllerTask.controller_cmd = [u'ipcontroller.exe']
# c.IPControllerTask.controller_args = ['--log-to-file', '--log-level', '40']
# c.IPControllerTask.environment_variables = {}
# c.WindowsHPCControllerLauncher.scheduler = 'HEADNODE'
# c.WindowsHPCControllerLauncher.job_file_name = u'ipcontroller_job.xml'
# c.IPEngineSetJob.job_name = 'IPEngineSet'
# c.IPEngineSetJob.is_exclusive = False
Brian Granger
Fixing minor bugs in IPython.kernel....
r2342 # c.IPEngineSetJob.username = r'USERDOMAIN\USERNAME'
Brian Granger
More work on the launchers and Win HPC support.
r2333 # c.IPEngineSetJob.priority = 'Highest'
# c.IPEngineSetJob.requested_nodes = ''
# c.IPEngineSetJob.project = 'MyProject'
# c.IPEngineTask.task_name = 'IPEngine'
# c.IPEngineTask.engine_cmd = [u'ipengine.exe']
# c.IPEngineTask.engine_args = ['--log-to-file', '--log-level', '40']
# c.IPEngineTask.environment_variables = {}
# c.WindowsHPCEngineSetLauncher.scheduler = 'HEADNODE'
# c.WindowsHPCEngineSetLauncher.job_file_name = u'ipengineset_job.xml'
Brian Granger
Work on default config files and docstrings....
r2310
Brian Granger
Adding files for the refactored kernel scripts.
r2304