Merging branch created for work on dev docs.

docs/source/development/config_blueprint.txt

@@ -2,6 +2,8 b''
 Notes on the IPython configuration system
 =========================================
 
+This document has some random notes on the configuration system.
+
 To start, an IPython process needs:
 
 * Configuration files

docs/source/development/development.txt

@@ -8,143 +8,72 b' IPython development guidelines'
 Overview
 ========
 
-IPython is the next generation of IPython.  It is named such for two reasons:
-
-- Eventually, IPython will become IPython version 1.0.
-- This new code base needs to be able to co-exist with the existing IPython until
-  it is a full replacement for it.  Thus we needed a different name.  We couldn't
-  use ``ipython`` (lowercase) as some files systems are case insensitive.
-
-There are two, no three, main goals of the IPython effort:
-
-1. Clean up the existing codebase and write lots of tests.
-2. Separate the core functionality of IPython from the terminal to enable IPython
-   to be used from within a variety of GUI applications.
-3. Implement a system for interactive parallel computing.
-
-While the third goal may seem a bit unrelated to the main focus of IPython, it
-turns out that the technologies required for this goal are nearly identical
-with those required for goal two. This is the main reason the interactive
-parallel computing capabilities are being put into IPython proper. Currently
-the third of these goals is furthest along.
-
-This document describes IPython from the perspective of developers.  
-
-
-Project organization
-====================
-
-Subpackages
------------
-
-IPython is organized into semi self-contained subpackages.  Each of the
-subpackages will have its own:
-
-- **Dependencies**.  One of the most important things to keep in mind in
-  partitioning code amongst subpackages, is that they should be used to cleanly
-  encapsulate dependencies.
-  
-- **Tests**.  Each subpackage shoud have its own ``tests`` subdirectory that
-  contains all of the tests for that package.  For information about writing
-  tests for IPython, see the `Testing System`_ section of this document.
-  
-- **Configuration**.  Each subpackage should have its own ``config``
-  subdirectory that contains the configuration information for the components
-  of the subpackage.  For information about how the IPython configuration
-  system works, see the `Configuration System`_ section of this document.
-  
-- **Scripts**.  Each subpackage should have its own ``scripts`` subdirectory
-  that contains all of the command line scripts associated with the subpackage.
-
-Installation and dependencies
------------------------------
-   
-IPython will not use `setuptools`_ for installation. Instead, we will use
-standard ``setup.py`` scripts that use `distutils`_. While there are a number a
-extremely nice features that `setuptools`_ has (like namespace packages), the
-current implementation of `setuptools`_ has performance problems, particularly
-on shared file systems. In particular, when Python packages are installed on
-NSF file systems, import times become much too long (up towards 10 seconds).
-
-Because IPython is being used extensively in the context of high performance
-computing, where performance is critical but shared file systems are common, we
-feel these performance hits are not acceptable. Thus, until the performance
-problems associated with `setuptools`_ are addressed, we will stick with plain
-`distutils`_. We are hopeful that these problems will be addressed and that we
-will eventually begin using `setuptools`_. Because of this, we are trying to
-organize IPython in a way that will make the eventual transition to
-`setuptools`_ as painless as possible.
-
-Because we will be using `distutils`_, there will be no method for
-automatically installing dependencies.  Instead, we are following the approach
-of `Matplotlib`_ which can be summarized as follows:
-
-- Distinguish between required and optional dependencies.  However, the required
-  dependencies for IPython should be only the Python standard library.
-  
-- Upon installation check to see which optional dependencies are present and
-  tell the user which parts of IPython need which optional dependencies.
-
-It is absolutely critical that each subpackage of IPython has a clearly
-specified set of dependencies and that dependencies are not carelessly
-inherited from other IPython subpackages. Furthermore, tests that have certain
-dependencies should not fail if those dependencies are not present. Instead
-they should be skipped and print a message.
-
-.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
-.. _distutils: http://docs.python.org/lib/module-distutils.html
-.. _Matplotlib: http://matplotlib.sourceforge.net/
-
-Specific subpackages
---------------------
-
-``core``
-    This is the core functionality of IPython that is independent of the
-    terminal, network and GUIs.  Most of the code that is in the current
-    IPython trunk will be refactored, cleaned up and moved here.
-
-``kernel``
-    The enables the IPython core to be expose to a the network.  This is
-    also where all of the parallel computing capabilities are to be found.
-    
-``config``
-    The configuration package used by IPython.
+This document describes IPython from the perspective of developers. Most
+importantly, it gives information for people who want to contribute to the
+development of IPython. So if you want to help out, read on!
+
+How to contribute to IPython
+============================
+
+IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
+This makes it easy for people to contribute to the development of IPython.
+Here is a sketch of how to get going.
 
-``frontends``
-    The various frontends for IPython.  A frontend is the end-user application
-    that exposes the capabilities of IPython to the user.  The most basic
-    frontend will simply be a terminal based application that looks just like
-    today 's IPython.  Other frontends will likely be more powerful and based
-    on GUI toolkits.
+Install Bazaar and create a Launchpad account
+---------------------------------------------
 
-``notebook``
-    An application that allows users to work with IPython notebooks.
+First make sure you have installed Bazaar (see their `website
+<http://bazaar-vcs.org/>`_). To see that Bazaar is installed and knows about
+you, try the following::
 
-``tools``
-    This is where general utilities go.
+    $ bzr whoami
+    Joe Coder <[e-mail]>
 
+This should display your name and email. Next, you will want to create an
+account on the `Launchpad website <http://www.launchpad.net>`_ and setup your
+ssh keys. For more information of setting up your ssh keys, see `this link
+<https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair>`_.
 
-Version control
-===============
+Get the main IPython branch from Launchpad
+------------------------------------------
 
-In the past, IPython development has been done using `Subversion`__.  Recently,
-we made the transition to using `Bazaar`__ and `Launchpad`__.  This makes it
-much easier for people to contribute code to IPython.  Here is a sketch of how
-to use Bazaar for IPython development.  First, you should install Bazaar.
-After you have done that, make sure that it is working by getting the latest
-main branch of IPython::
+Now, you can get a copy of the main IPython development branch (we call this
+the "trunk")::
 
     $ bzr branch lp:ipython
 
-Now you can create a new branch for you to do your work in::
+Create a working branch
+-----------------------
+
+When working on IPython, you won't actually make edits directly to the
+:file:`lp:ipython` branch. Instead, you will create a separate branch for your
+changes. For now, let's assume you want to do your work in a branch named
+"ipython-mybranch". Create this branch by doing::
 
     $ bzr branch ipython ipython-mybranch
 
-The typical work cycle in this branch will be to make changes in
-``ipython-mybranch`` and then commit those changes using the commit command::
+When you actually create a branch, you will want to give it a name that
+reflects the nature of the work that you will be doing in it, like
+"install-docs-update".
+
+Make edits in your working branch
+---------------------------------
+
+Now you are ready to actually make edits in your :file:`ipython-mybranch`
+branch. Before doing this, it is helpful to install this branch so you can
+test your changes as you work. This is easiest if you have setuptools
+installed. Then, just do::
+
+    $ cd ipython-mybranch
+    $ python setupegg.py develop
+
+Now, make some changes. After a while, you will want to commit your changes.
+This let's Bazaar know that you like the changes you have made and gives you
+an opportunity to keep a nice record of what you have done. This looks like
+this::
 
     $ ...do work in ipython-mybranch...
-    $ bzr ci -m "the commit message goes here"
+    $ bzr commit -m "the commit message goes here"
 
 Please note that since we now don't use an old-style linear ChangeLog (that
 tends to cause problems with distributed version control systems), you should
@@ -154,48 +83,60 b' approach in the commit messages (including the second line being left'
 
   Single line summary of  changes being committed.
 
-  - more details when warranted ...
-  - including crediting outside contributors if they sent the
+  * more details when warranted ...
+  * including crediting outside contributors if they sent the
     code/bug/idea!
 
-If we couple this with a policy of making single commits for each reasonably
-atomic change, the bzr log should give an excellent view of the project, and
-the `--short` log option becomes a nice summary.
+As you work, you will repeat this edit/commit cycle many times. If you work on
+your branch for a long time, you will also want to get the latest changes from
+the :file:`lp:ipython` branch. This can be done with the following sequence of
+commands::
 
-While working with this branch, it is a good idea to merge in changes that have
-been made upstream in the parent branch.  This can be done by doing::
-
-    $ bzr pull
+    $ ls
+    ipython
+    ipython-mybranch
     
-If this command shows that the branches have diverged, then you should do a
-merge instead::
+    $ cd ipython
+    $ bzr pull
+    $ cd ../ipython-mybranch
+    $ bzr merge ../ipython
+    $ bzr commit -m "Merging changes from trunk"
 
-    $ bzr merge lp:ipython
+Along the way, you should also run the IPython test suite.  You can do this using the :command:`iptest` command::
 
-If you want others to be able to see your branch, you can create an account
-with launchpad and push the branch to your own workspace::
+    $ cd
+    $ iptest
 
-    $ bzr push bzr+ssh://<me>@bazaar.launchpad.net/~<me>/+junk/ipython-mybranch
+The :command:`iptest` command will also pick up and run any tests you have written.
 
-Finally, once the work in your branch is done, you can merge your changes back
-into the `ipython` branch by using merge::
+Post your branch and request a code review
+------------------------------------------
 
-    $ cd ipython
-    $ merge ../ipython-mybranch
-    [resolve any conflicts]
-    $ bzr ci -m "Fixing that bug"
-    $ bzr push
+Once you are done with your edits, you should post your branch on Launchpad so
+that other IPython developers can review the changes and help you merge your
+changes into the main development branch. To post your branch on Launchpad,
+do::
+
+    $ cd ipython-mybranch
+    $ bzr push lp:~yourusername/ipython/ipython-mybranch
+
+Then, go to the `IPython Launchpad site <www.launchpad.net/ipython>`_, and you
+should see your branch under the "Code" tab. If you click on your branch, you
+can provide a short description of the branch as well as mark its status. Most
+importantly, you should click the link that reads "Propose for merging into
+another branch". What does this do?
 
-But this will require you to have write permissions to the `ipython` branch.
-It you don't you can tell one of the IPython devs about your branch and they
-can do the merge for you.
+This let's the other IPython developers know that your branch is ready to be
+reviewed and merged into the main development branch. During this review
+process, other developers will give you feedback and help you get your code
+ready to be merged. What types of things will we be looking for:
 
-More information about Bazaar workflows can be found `here`__.
+* All code is documented.
+* All code has tests.
+* The entire IPython test suite passes.
 
-.. __: http://subversion.tigris.org/
-.. __: http://bazaar-vcs.org/
-.. __: http://www.launchpad.net/ipython
-.. __: http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/index.html
+Once your changes have been reviewed and approved, someone will merge them
+into the main development branch.
 
 Documentation
 =============
@@ -204,39 +145,31 b' Standalone documentation'
 ------------------------
 
 All standalone documentation should be written in plain text (``.txt``) files
-using `reStructuredText`_ for markup and formatting. All such documentation
-should be placed in the top level directory ``docs`` of the IPython source
-tree. Or, when appropriate, a suitably named subdirectory should be used. The
-documentation in this location will serve as the main source for IPython
-documentation and all existing documentation should be converted to this
-format.
+using reStructuredText [reStructuredText]_ for markup and formatting. All such
+documentation should be placed in directory :file:`docs/source` of the IPython
+source tree. The documentation in this location will serve as the main source
+for IPython documentation and all existing documentation should be converted
+to this format.
 
-In the future, the text files in the ``docs`` directory will be used to
-generate all forms of documentation for IPython. This include documentation on
-the IPython website as well as *pdf* documentation.
+To build the final documentation, we use Sphinx [Sphinx]_.  Once you have Sphinx installed, you can build the html docs yourself by doing::
 
-.. _reStructuredText: http://docutils.sourceforge.net/rst.html
+    $ cd ipython-mybranch/docs
+    $ make html
 
 Docstring format
 ----------------
 
-Good docstrings are very important. All new code will use `Epydoc`_ for
-generating API docs, so we will follow the `Epydoc`_ conventions. More
-specifically, we will use `reStructuredText`_ for markup and formatting, since
-it is understood by a wide variety of tools. This means that if in the future
-we have any reason to change from `Epydoc`_ to something else, we'll have fewer
-transition pains.
-
-Details about using `reStructuredText`_ for docstrings can be found `here
+Good docstrings are very important. All new code should have docstrings that
+are formatted using reStructuredText for markup and formatting, since it is
+understood by a wide variety of tools. Details about using reStructuredText
+for docstrings can be found `here
 <http://epydoc.sourceforge.net/manual-othermarkup.html>`_.
 
-.. _Epydoc: http://epydoc.sourceforge.net/
-
 Additional PEPs of interest regarding documentation of code:
 
-- `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
-- `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
-- `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_
+* `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
+* `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
+* `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_
 
 
 Coding conventions
@@ -248,17 +181,16 b' General'
 In general, we'll try to follow the standard Python style conventions as
 described here:
 
-- `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  
+* `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  
 
 
 Other comments:
 
-- In a large file, top level classes and functions should be 
+* In a large file, top level classes and functions should be 
   separated by 2-3 lines to make it easier to separate them visually.
-- Use 4 spaces for indentation.
-- Keep the ordering of methods the same in classes that have the same 
-  methods.  This is particularly true for classes that implement 
-  similar interfaces and for interfaces that are similar.
+* Use 4 spaces for indentation.
+* Keep the ordering of methods the same in classes that have the same 
+  methods.  This is particularly true for classes that implement an interface.
 
 Naming conventions
 ------------------
@@ -268,38 +200,33 b' Guide for Python Code`_.'
 
 For all new IPython code (and much existing code is being refactored), we'll use:
 
-- All ``lowercase`` module names.
+* All ``lowercase`` module names.
 
-- ``CamelCase`` for class names.
+* ``CamelCase`` for class names.
 
-- ``lowercase_with_underscores`` for methods, functions, variables and
+* ``lowercase_with_underscores`` for methods, functions, variables and
   attributes.
 
-This may be confusing as most of the existing IPython codebase uses a different
-convention (``lowerCamelCase`` for methods and attributes).  Slowly, we will
-move IPython over to the new convention, providing shadow names for backward
-compatibility in public interfaces.
-
-There are, however, some important exceptions to these rules.  In some cases,
+There are, however, some important exceptions to these rules. In some cases,
 IPython code will interface with packages (Twisted, Wx, Qt) that use other
-conventions.  At some level this makes it impossible to adhere to our own
-standards at all times.  In particular, when subclassing classes that use other
-naming conventions, you must follow their naming conventions.  To deal with
+conventions. At some level this makes it impossible to adhere to our own
+standards at all times. In particular, when subclassing classes that use other
+naming conventions, you must follow their naming conventions. To deal with
 cases like this, we propose the following policy:
 
-- If you are subclassing a class that uses different conventions, use its
+* If you are subclassing a class that uses different conventions, use its
   naming conventions throughout your subclass.  Thus, if you are creating a
   Twisted Protocol class, used Twisted's
   ``namingSchemeForMethodsAndAttributes.``
 
-- All IPython's official interfaces should use our conventions.  In some cases
+* All IPython's official interfaces should use our conventions.  In some cases
   this will mean that you need to provide shadow names (first implement
   ``fooBar`` and then ``foo_bar = fooBar``).  We want to avoid this at all
   costs, but it will probably be necessary at times.  But, please use this
   sparingly!
 
 Implementation-specific *private* methods will use
-``_single_underscore_prefix``.  Names with a leading double underscore will
+``_single_underscore_prefix``. Names with a leading double underscore will
 *only* be used in special cases, as they makes subclassing difficult (such
 names are not easily seen by child classes).
 
@@ -323,107 +250,38 b' specific ``IPY`` or ``ipy`` are preferred.'
 Testing system
 ==============
 
-It is extremely important that all code contributed to IPython has tests. Tests
-should be written as unittests, doctests or as entities that the `Nose`_
-testing package will find. Regardless of how the tests are written, we will use
-`Nose`_ for discovering and running the tests. `Nose`_ will be required to run
-the IPython test suite, but will not be required to simply use IPython.
+It is extremely important that all code contributed to IPython has tests.
+Tests should be written as unittests, doctests or as entities that the Nose
+[Nose]_ testing package will find. Regardless of how the tests are written, we
+will use Nose for discovering and running the tests. Nose will be required to
+run the IPython test suite, but will not be required to simply use IPython.
 
-.. _Nose: http://code.google.com/p/python-nose/
+Tests of Twisted using code need to follow two additional guidelines:
 
-Tests of `Twisted`__ using code should be written by subclassing the
-``TestCase`` class that comes with ``twisted.trial.unittest``. When this is
-done, `Nose`_ will be able to run the tests and the twisted reactor will be
-handled correctly.
+1. Twisted using tests should be written by subclassing the :class:`TestCase`
+   class that comes with :mod:`twisted.trial.unittest`. 
 
-.. __: http://www.twistedmatrix.com
+2. All :class:`Deferred` instances that are created in the test must be
+   properly chained and the final one *must* be the return value of the test
+   method.
 
-Each subpackage in IPython should have its own ``tests`` directory that
+When these two things are done, Nose will be able to run the tests and the
+twisted reactor will be handled correctly.
+
+Each subpackage in IPython should have its own :file:`tests` directory that
 contains all of the tests for that subpackage. This allows each subpackage to
 be self-contained. If a subpackage has any dependencies beyond the Python
 standard library, the tests for that subpackage should be skipped if the
 dependencies are not found. This is very important so users don't get tests
 failing simply because they don't have dependencies.
 
-We also need to look into use Noses ability to tag tests to allow a more
-modular approach of running tests.
+To run the IPython test suite, use the :command:`iptest` command that is installed with IPython::
 
-.. _devel_config:
+    $ iptest
 
-Configuration system
-====================
-
-IPython uses `.ini`_ files for configuration purposes. This represents a huge
-improvement over the configuration system used in IPython. IPython works with
-these files using the `ConfigObj`_ package, which IPython includes as
-``ipython1/external/configobj.py``.
-
-Currently, we are using raw `ConfigObj`_ objects themselves. Each subpackage of
-IPython should contain a ``config`` subdirectory that contains all of the
-configuration information for the subpackage. To see how configuration
-information is defined (along with defaults) see at the examples in
-``ipython1/kernel/config`` and ``ipython1/core/config``. Likewise, to see how
-the configuration information is used, see examples in
-``ipython1/kernel/scripts/ipengine.py``.
-
-Eventually, we will add a new layer on top of the raw `ConfigObj`_ objects. We
-are calling this new layer, ``tconfig``, as it will use a `Traits`_-like
-validation model.  We won't actually use `Traits`_, but will implement
-something similar in pure Python.  But, even in this new system, we will still
-use `ConfigObj`_ and `.ini`_ files underneath the hood. Talk to Fernando if you
-are interested in working on this part of IPython. The current prototype of
-``tconfig`` is located in the IPython sandbox.
-
-.. _.ini: http://docs.python.org/lib/module-ConfigParser.html
-.. _ConfigObj: http://www.voidspace.org.uk/python/configobj.html
-.. _Traits: http://code.enthought.com/traits/
-
-
-Installation and testing scenarios
-==================================
-
-This section outlines the various scenarios that we need to test before we
-release an IPython version.  These scenarios represent different ways of
-installing IPython and its dependencies.
-
-Installation scenarios under Linux and OS X
--------------------------------------------
-
-    1.  Install from tarball using ``python setup.py install``.
-        a.  With only readline+nose dependencies installed.
-        b.  With all dependencies installed (readline, zope.interface, Twisted,
-        foolscap, Sphinx, nose, pyOpenSSL).
-	    
-    2.  Install using easy_install.
-
-        a.  With only readline+nose dependencies installed.
-            i.  Default dependencies: ``easy_install ipython-0.9.beta3-py2.5.egg``
-            ii. Optional dependency sets: ``easy_install -f ipython-0.9.beta3-py2.5.egg IPython[kernel,doc,test,security]``
-
-        b.  With all dependencies already installed.
-	
-
-Installation scenarios under Win32
-----------------------------------
-
-	1.  Install everything from .exe installers
-	2.  easy_install?
-
-
-Tests to run for these scenarios
---------------------------------
-
-    1.  Run the full test suite.
-    2.  Start a controller and engines and try a few things by hand.
-        a.  Using ipcluster.
-        b.  Using ipcontroller/ipengine by hand.
-
-    3.  Run a few of the parallel examples.
-    4.  Try the kernel with and without security with and without PyOpenSSL
-        installed.
-    5.  Beat on the IPython terminal a bunch.
-    6.  Make sure that furl files are being put in proper locations.
+This command runs Nose with the proper options and extensions.
 
+.. _devel_config:
 
 Release checklist
 =================
@@ -434,9 +292,9 b' Most of the release process is automated by the :file:`release` script in the'
 #. Run the release script, which makes the tar.gz, eggs and Win32 .exe
    installer.  It posts them to the site and registers the release with PyPI.
 
-#. Updating the website with announcements and links to the updated changes.txt
-   in html form.  Remember to put a short note both on the news page of the site
-   and on launcphad.
+#. Updating the website with announcements and links to the updated
+   changes.txt in html form. Remember to put a short note both on the news
+   page of the site and on Launcphad.
 
 #. Drafting a short release announcement with i) highlights and ii) a link to
    the html changes.txt.
@@ -444,3 +302,9 b' Most of the release process is automated by the :file:`release` script in the'
 #. Make sure that the released version of the docs is live on the site.
 
 #. Celebrate!
+
+.. [Bazaar] Bazaar. http://bazaar-vcs.org/
+.. [Launchpad] Launchpad. http://www.launchpad.net/ipython
+.. [reStructuredText] reStructuredText.  http://docutils.sourceforge.net/rst.html
+.. [Sphinx] Sphinx. http://sphinx.pocoo.org/
+.. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/

docs/source/development/notification_blueprint.txt

@@ -6,27 +6,50 b' IPython.kernel.core.notification blueprint'
 
 Overview
 ========
-The :mod:`IPython.kernel.core.notification` module will provide a simple implementation of a notification center and support for the observer pattern within the :mod:`IPython.kernel.core`. The main intended use case is to provide notification of Interpreter events to an observing frontend during the execution of a single block of code.
+
+The :mod:`IPython.kernel.core.notification` module will provide a simple
+implementation of a notification center and support for the observer pattern
+within the :mod:`IPython.kernel.core`. The main intended use case is to
+provide notification of Interpreter events to an observing frontend during the
+execution of a single block of code.
 
 Functional Requirements
 =======================
+
 The notification center must:
-    * Provide synchronous notification of events to all registered observers.
-    * Provide typed or labeled notification types
-    * Allow observers to register callbacks for individual or all notification types
-    * Allow observers to register callbacks for events from individual or all notifying objects
-    * Notification to the observer consists of the notification type, notifying object and user-supplied extra information [implementation: as keyword parameters to the registered callback]
-    * Perform as O(1) in the case of no registered observers.
-    * Permit out-of-process or cross-network extension.
+
+* Provide synchronous notification of events to all registered observers.
+
+* Provide typed or labeled notification types.
+
+* Allow observers to register callbacks for individual or all notification
+  types.
+
+* Allow observers to register callbacks for events from individual or all
+  notifying objects.
+
+* Notification to the observer consists of the notification type, notifying
+  object and user-supplied extra information [implementation: as keyword
+  parameters to the registered callback].
+
+* Perform as O(1) in the case of no registered observers.
+
+* Permit out-of-process or cross-network extension.
     
 What's not included
-==============================================================
+===================
+
 As written, the :mod:`IPython.kernel.core.notificaiton` module does not:
-    * Provide out-of-process or network notifications [these should be handled by a separate, Twisted aware module in :mod:`IPython.kernel`].
-    * Provide zope.interface-style interfaces for the notification system [these should also be provided by the :mod:`IPython.kernel` module]
+
+* Provide out-of-process or network notifications (these should be handled by
+  a separate, Twisted aware module in :mod:`IPython.kernel`).
+
+* Provide zope.interface-style interfaces for the notification system (these
+  should also be provided by the :mod:`IPython.kernel` module).
     
 Use Cases
 =========
+
 The following use cases describe the main intended uses of the notificaiton module and illustrate the main success scenario for each use case:
 
     1. Dwight Schroot is writing a frontend for the IPython project. His frontend is stuck in the stone age and must communicate synchronously with an IPython.kernel.core.Interpreter instance. Because code is executed in blocks by the Interpreter, Dwight's UI freezes every time he executes a long block of code. To keep track of the progress of his long running block, Dwight adds the following code to his frontend's set-up code::
@@ -40,9 +63,20 b' The following use cases describe the main intended uses of the notificaiton modu'
         def stdout_notification(self, type, notifying_object, out_string=None):
             self.writeStdOut(out_string)
     
-    If everything works, the Interpreter will (according to its published API) fire a notification via the :data:`IPython.kernel.core.notification.sharedCenter` of type :const:`STD_OUT_NOTIFICATION_TYPE` before writing anything to stdout [it's up to the Intereter implementation to figure out when to do this]. The notificaiton center will then call the registered callbacks for that event type (in this case, Dwight's frontend's stdout_notification method). Again, according to its API, the Interpreter provides an additional keyword argument when firing the notificaiton of out_string, a copy of the string it will write to stdout.
+If everything works, the Interpreter will (according to its published API)
+fire a notification via the
+:data:`IPython.kernel.core.notification.sharedCenter` of type
+:const:`STD_OUT_NOTIFICATION_TYPE` before writing anything to stdout [it's up
+to the Intereter implementation to figure out when to do this]. The
+notificaiton center will then call the registered callbacks for that event
+type (in this case, Dwight's frontend's stdout_notification method). Again,
+according to its API, the Interpreter provides an additional keyword argument
+when firing the notificaiton of out_string, a copy of the string it will write
+to stdout.
 
-    Like magic, Dwight's frontend is able to provide output, even during long-running calculations. Now if Jim could just convince Dwight to use Twisted...
+Like magic, Dwight's frontend is able to provide output, even during
+long-running calculations. Now if Jim could just convince Dwight to use
+Twisted...
 
     2. Boss Hog is writing a frontend for the IPython project. Because Boss Hog is stuck in the stone age, his frontend will be written in a new Fortran-like dialect of python and will run only from the command line. Because he doesn't need any fancy notification system and is used to worrying about every cycle on his rat-wheel powered mini, Boss Hog is adamant that the new notification system not produce any performance penalty. As they say in Hazard county, there's no such thing as a free lunch. If he wanted zero overhead, he should have kept using IPython 0.8. Instead, those tricky Duke boys slide in a suped-up bridge-out jumpin' awkwardly confederate-lovin' notification module that imparts only a constant (and small) performance penalty when the Interpreter (or any other object) fires an event for which there are no registered observers. Of course, the same notificaiton-enabled Interpreter can then be used in frontends that require notifications, thus saving the IPython project from a nasty civil war.
     

docs/source/development/roadmap.txt

@@ -6,97 +6,68 b' Development roadmap'
 
 IPython is an ambitious project that is still under heavy development.  However, we want IPython to become useful to as many people as possible, as quickly as possible.  To help us accomplish this, we are laying out a roadmap of where we are headed and what needs to happen to get there.  Hopefully, this will help the IPython developers figure out the best things to work on for each upcoming release.
 
-Speaking of releases, we are going to begin releasing a new version of IPython every four weeks.  We are hoping that a regular release schedule, along with a clear roadmap of where we are headed will propel the project forward.
+Work targeted to particular releases
+====================================
 
-Where are we headed
-===================
-
-Our goal with IPython is simple: to provide a *powerful*, *robust* and *easy to use* framework for parallel computing.  While there are other secondary goals you will hear us talking about at various times, this is the primary goal of IPython that frames the roadmap.
-
-Steps along the way
-===================
-
-Here we describe the various things that we need to work on to accomplish this goal.
-
-Setting up for regular release schedule
----------------------------------------
-
-We would like to begin to release IPython regularly (probably a 4 week release cycle).  To get ready for this, we need to revisit the development guidelines and put in information about releasing IPython.  
+Release 0.10
+------------
 
-Process startup and management
-------------------------------
+* Initial refactor of :command:`ipcluster`.
 
-IPython is implemented using a distributed set of processes that communicate using TCP/IP network channels.  Currently, users have to start each of the various processes separately using command line scripts.  This is both difficult and error prone.  Furthermore, there are a number of things that often need to be managed once the processes have been started, such as the sending of signals and the shutting down and cleaning up of processes.
+* Better TextMate integration.
 
-We need to build a system that makes it trivial for users to start and manage IPython processes.  This system should have the following properties:
+Release 0.11
+------------
 
-* It should possible to do everything through an extremely simple API that
-  users can call from their own Python script. No shell commands should be
-  needed.
-* This simple API should be configured using standard .ini files.
-* The system should make it possible to start processes using a number of
-  different approaches: SSH, PBS/Torque, Xgrid, Windows Server, mpirun, etc.
-* The controller and engine processes should each have a daemon for
-  monitoring, signaling and clean up.
-* The system should be secure.
-* The system should work under all the major operating systems, including
-  Windows.
+* Refactor of configuration system and command line options for
+  :command:`ipengine` and :command:`ipcontroller`. This will include the
+  creation of cluster directories that encapsulate all the configuration
+  files, log files and security related files for a particular cluster.
 
-Initial work has begun on the daemon infrastructure, and some of the needed logic is contained in the ipcluster script.
+* Merge in the daemon branch.
 
-Ease of use/high-level approaches to parallelism
-------------------------------------------------
+* Merge back in the core of the notebook.
 
-While our current API for clients is well designed, we can still do a lot better in designing a user-facing API that is super simple.  The main goal here is that it should take *almost no extra code* for users to get their code running in parallel.  For this to be possible, we need to tie into Python's standard idioms that enable efficient coding.  The biggest ones we are looking at are using context managers (i.e., Python 2.5's ``with`` statement) and decorators.  Initial work on this front has begun, but more work is needed.
+Release 0.12
+------------
 
-We also need to think about new models for expressing parallelism.  This is fun work as most of the foundation has already been established.
+* Integrate process startup with the daemons for full process management.
 
-Security
---------
-
-Currently, IPython has no built in security or security model.  Because we would like IPython to be usable on public computer systems and over wide area networks, we need to come up with a robust solution for security.  Here are some of the specific things that need to be included:
-
-* User authentication between all processes (engines, controller and clients).
-* Optional TSL/SSL based encryption of all communication channels.
-* A good way of picking network ports so multiple users on the same system can
-  run their own controller and engines without interfering with those of
-  others.
-* A clear model for security that enables users to evaluate the security risks
-  associated with using IPython in various manners.
-
-For the implementation of this, we plan on using Twisted's support for SSL and authentication.  One things that we really should look at is the `Foolscap`_ network protocol, which provides many of these things out of the box.
+Major areas of work
+===================
 
-.. _Foolscap: http://foolscap.lothar.com/trac
+Refactoring the main IPython core
+---------------------------------
 
-The security work needs to be done in conjunction with other network protocol stuff.
+Process management for :mod:`IPython.kernel`
+--------------------------------------------
 
-As of the 0.9 release of IPython, we are using Foolscap and we have implemented
-a full security model.
+Configuration system
+--------------------
 
-Latent performance issues
--------------------------
+Performance problems
+--------------------
 
 Currently, we have a number of performance issues that are waiting to bite users:
 
-* The controller store a large amount of state in Python dictionaries. Under
+* The controller stores a large amount of state in Python dictionaries. Under
   heavy usage, these dicts with get very large, causing memory usage problems.
   We need to develop more scalable solutions to this problem, such as using a
   sqlite database to store this state. This will also help the controller to
   be more fault tolerant.
-* Currently, the client to controller connections are done through XML-RPC
-  using HTTP 1.0. This is very inefficient as XML-RPC is a very verbose
-  protocol and each request must be handled with a new connection. We need to
-  move these network connections over to PB or Foolscap. Done!
+
 * We currently don't have a good way of handling large objects in the
   controller. The biggest problem is that because we don't have any way of
   streaming objects, we get lots of temporary copies in the low-level buffers.
   We need to implement a better serialization approach and true streaming
   support.
+
 * The controller currently unpickles and repickles objects. We need to use the
   [push|pull]_serialized methods instead.
-* Currently the controller is a bottleneck. We need the ability to scale the
-  controller by aggregating multiple controllers into one effective
-  controller.
+
+* Currently the controller is a bottleneck. The best approach for this is to
+  separate the controller itself into multiple processes, one for the core
+  controller and one each for the controller interfaces.