Updating developer docs.

docs/source/development/config_blueprint.txt

@@ -2,6 +2,8 b''
 Notes on the IPython configuration system
 =========================================
 
+This document has some random notes on the configuration system.
+
 To start, an IPython process needs:
 
 * Configuration files

docs/source/development/notification_blueprint.txt

@@ -6,27 +6,50 b' IPython.kernel.core.notification blueprint'
 
 Overview
 ========
-The :mod:`IPython.kernel.core.notification` module will provide a simple implementation of a notification center and support for the observer pattern within the :mod:`IPython.kernel.core`. The main intended use case is to provide notification of Interpreter events to an observing frontend during the execution of a single block of code.
+
+The :mod:`IPython.kernel.core.notification` module will provide a simple
+implementation of a notification center and support for the observer pattern
+within the :mod:`IPython.kernel.core`. The main intended use case is to
+provide notification of Interpreter events to an observing frontend during the
+execution of a single block of code.
 
 Functional Requirements
 =======================
+
 The notification center must:
-    * Provide synchronous notification of events to all registered observers.
-    * Provide typed or labeled notification types
-    * Allow observers to register callbacks for individual or all notification types
-    * Allow observers to register callbacks for events from individual or all notifying objects
-    * Notification to the observer consists of the notification type, notifying object and user-supplied extra information [implementation: as keyword parameters to the registered callback]
-    * Perform as O(1) in the case of no registered observers.
-    * Permit out-of-process or cross-network extension.
+
+* Provide synchronous notification of events to all registered observers.
+
+* Provide typed or labeled notification types.
+
+* Allow observers to register callbacks for individual or all notification
+  types.
+
+* Allow observers to register callbacks for events from individual or all
+  notifying objects.
+
+* Notification to the observer consists of the notification type, notifying
+  object and user-supplied extra information [implementation: as keyword
+  parameters to the registered callback].
+
+* Perform as O(1) in the case of no registered observers.
+
+* Permit out-of-process or cross-network extension.
     
 What's not included
-==============================================================
+===================
+
 As written, the :mod:`IPython.kernel.core.notificaiton` module does not:
-    * Provide out-of-process or network notifications [these should be handled by a separate, Twisted aware module in :mod:`IPython.kernel`].
-    * Provide zope.interface-style interfaces for the notification system [these should also be provided by the :mod:`IPython.kernel` module]
+
+* Provide out-of-process or network notifications (these should be handled by
+  a separate, Twisted aware module in :mod:`IPython.kernel`).
+
+* Provide zope.interface-style interfaces for the notification system (these
+  should also be provided by the :mod:`IPython.kernel` module).
     
 Use Cases
 =========
+
 The following use cases describe the main intended uses of the notificaiton module and illustrate the main success scenario for each use case:
 
     1. Dwight Schroot is writing a frontend for the IPython project. His frontend is stuck in the stone age and must communicate synchronously with an IPython.kernel.core.Interpreter instance. Because code is executed in blocks by the Interpreter, Dwight's UI freezes every time he executes a long block of code. To keep track of the progress of his long running block, Dwight adds the following code to his frontend's set-up code::
@@ -40,9 +63,20 b' The following use cases describe the main intended uses of the notificaiton modu'
         def stdout_notification(self, type, notifying_object, out_string=None):
             self.writeStdOut(out_string)
     
-    If everything works, the Interpreter will (according to its published API) fire a notification via the :data:`IPython.kernel.core.notification.sharedCenter` of type :const:`STD_OUT_NOTIFICATION_TYPE` before writing anything to stdout [it's up to the Intereter implementation to figure out when to do this]. The notificaiton center will then call the registered callbacks for that event type (in this case, Dwight's frontend's stdout_notification method). Again, according to its API, the Interpreter provides an additional keyword argument when firing the notificaiton of out_string, a copy of the string it will write to stdout.
+If everything works, the Interpreter will (according to its published API)
+fire a notification via the
+:data:`IPython.kernel.core.notification.sharedCenter` of type
+:const:`STD_OUT_NOTIFICATION_TYPE` before writing anything to stdout [it's up
+to the Intereter implementation to figure out when to do this]. The
+notificaiton center will then call the registered callbacks for that event
+type (in this case, Dwight's frontend's stdout_notification method). Again,
+according to its API, the Interpreter provides an additional keyword argument
+when firing the notificaiton of out_string, a copy of the string it will write
+to stdout.
 
-    Like magic, Dwight's frontend is able to provide output, even during long-running calculations. Now if Jim could just convince Dwight to use Twisted...
+Like magic, Dwight's frontend is able to provide output, even during
+long-running calculations. Now if Jim could just convince Dwight to use
+Twisted...
 
     2. Boss Hog is writing a frontend for the IPython project. Because Boss Hog is stuck in the stone age, his frontend will be written in a new Fortran-like dialect of python and will run only from the command line. Because he doesn't need any fancy notification system and is used to worrying about every cycle on his rat-wheel powered mini, Boss Hog is adamant that the new notification system not produce any performance penalty. As they say in Hazard county, there's no such thing as a free lunch. If he wanted zero overhead, he should have kept using IPython 0.8. Instead, those tricky Duke boys slide in a suped-up bridge-out jumpin' awkwardly confederate-lovin' notification module that imparts only a constant (and small) performance penalty when the Interpreter (or any other object) fires an event for which there are no registered observers. Of course, the same notificaiton-enabled Interpreter can then be used in frontends that require notifications, thus saving the IPython project from a nasty civil war.
     

docs/source/development/roadmap.txt

@@ -6,97 +6,68 b' Development roadmap'
 
 IPython is an ambitious project that is still under heavy development.  However, we want IPython to become useful to as many people as possible, as quickly as possible.  To help us accomplish this, we are laying out a roadmap of where we are headed and what needs to happen to get there.  Hopefully, this will help the IPython developers figure out the best things to work on for each upcoming release.
 
-Speaking of releases, we are going to begin releasing a new version of IPython every four weeks.  We are hoping that a regular release schedule, along with a clear roadmap of where we are headed will propel the project forward.
+Work targeted to particular releases
+====================================
 
-Where are we headed
-===================
-
-Our goal with IPython is simple: to provide a *powerful*, *robust* and *easy to use* framework for parallel computing.  While there are other secondary goals you will hear us talking about at various times, this is the primary goal of IPython that frames the roadmap.
-
-Steps along the way
-===================
-
-Here we describe the various things that we need to work on to accomplish this goal.
-
-Setting up for regular release schedule
----------------------------------------
-
-We would like to begin to release IPython regularly (probably a 4 week release cycle).  To get ready for this, we need to revisit the development guidelines and put in information about releasing IPython.  
+Release 0.10
+------------
 
-Process startup and management
-------------------------------
+* Initial refactor of :command:`ipcluster`.
 
-IPython is implemented using a distributed set of processes that communicate using TCP/IP network channels.  Currently, users have to start each of the various processes separately using command line scripts.  This is both difficult and error prone.  Furthermore, there are a number of things that often need to be managed once the processes have been started, such as the sending of signals and the shutting down and cleaning up of processes.
+* Better TextMate integration.
 
-We need to build a system that makes it trivial for users to start and manage IPython processes.  This system should have the following properties:
+Release 0.11
+------------
 
-* It should possible to do everything through an extremely simple API that
-  users can call from their own Python script. No shell commands should be
-  needed.
-* This simple API should be configured using standard .ini files.
-* The system should make it possible to start processes using a number of
-  different approaches: SSH, PBS/Torque, Xgrid, Windows Server, mpirun, etc.
-* The controller and engine processes should each have a daemon for
-  monitoring, signaling and clean up.
-* The system should be secure.
-* The system should work under all the major operating systems, including
-  Windows.
+* Refactor of configuration system and command line options for
+  :command:`ipengine` and :command:`ipcontroller`. This will include the
+  creation of cluster directories that encapsulate all the configuration
+  files, log files and security related files for a particular cluster.
 
-Initial work has begun on the daemon infrastructure, and some of the needed logic is contained in the ipcluster script.
+* Merge in the daemon branch.
 
-Ease of use/high-level approaches to parallelism
-------------------------------------------------
+* Merge back in the core of the notebook.
 
-While our current API for clients is well designed, we can still do a lot better in designing a user-facing API that is super simple.  The main goal here is that it should take *almost no extra code* for users to get their code running in parallel.  For this to be possible, we need to tie into Python's standard idioms that enable efficient coding.  The biggest ones we are looking at are using context managers (i.e., Python 2.5's ``with`` statement) and decorators.  Initial work on this front has begun, but more work is needed.
+Release 0.12
+------------
 
-We also need to think about new models for expressing parallelism.  This is fun work as most of the foundation has already been established.
+* Integrate process startup with the daemons for full process management.
 
-Security
---------
-
-Currently, IPython has no built in security or security model.  Because we would like IPython to be usable on public computer systems and over wide area networks, we need to come up with a robust solution for security.  Here are some of the specific things that need to be included:
-
-* User authentication between all processes (engines, controller and clients).
-* Optional TSL/SSL based encryption of all communication channels.
-* A good way of picking network ports so multiple users on the same system can
-  run their own controller and engines without interfering with those of
-  others.
-* A clear model for security that enables users to evaluate the security risks
-  associated with using IPython in various manners.
-
-For the implementation of this, we plan on using Twisted's support for SSL and authentication.  One things that we really should look at is the `Foolscap`_ network protocol, which provides many of these things out of the box.
+Major areas of work
+===================
 
-.. _Foolscap: http://foolscap.lothar.com/trac
+Refactoring the main IPython core
+---------------------------------
 
-The security work needs to be done in conjunction with other network protocol stuff.
+Process management for :mod:`IPython.kernel`
+--------------------------------------------
 
-As of the 0.9 release of IPython, we are using Foolscap and we have implemented
-a full security model.
+Configuration system
+--------------------
 
-Latent performance issues
--------------------------
+Performance problems
+--------------------
 
 Currently, we have a number of performance issues that are waiting to bite users:
 
-* The controller store a large amount of state in Python dictionaries. Under
+* The controller stores a large amount of state in Python dictionaries. Under
   heavy usage, these dicts with get very large, causing memory usage problems.
   We need to develop more scalable solutions to this problem, such as using a
   sqlite database to store this state. This will also help the controller to
   be more fault tolerant.
-* Currently, the client to controller connections are done through XML-RPC
-  using HTTP 1.0. This is very inefficient as XML-RPC is a very verbose
-  protocol and each request must be handled with a new connection. We need to
-  move these network connections over to PB or Foolscap. Done!
+
 * We currently don't have a good way of handling large objects in the
   controller. The biggest problem is that because we don't have any way of
   streaming objects, we get lots of temporary copies in the low-level buffers.
   We need to implement a better serialization approach and true streaming
   support.
+
 * The controller currently unpickles and repickles objects. We need to use the
   [push|pull]_serialized methods instead.
-* Currently the controller is a bottleneck. We need the ability to scale the
-  controller by aggregating multiple controllers into one effective
-  controller.
+
+* Currently the controller is a bottleneck. The best approach for this is to
+  separate the controller itself into multiple processes, one for the core
+  controller and one each for the controller interfaces.