Added a python3.0 porting section to the development.txt document

docs/source/development/development.txt

@@ -1,446 +1,461 b''
 .. _development:
 
 ==============================
 IPython development guidelines
 ==============================
 
 
 Overview
 ========
 
 IPython is the next generation of IPython.  It is named such for two reasons:
 
 - Eventually, IPython will become IPython version 1.0.
 - This new code base needs to be able to co-exist with the existing IPython until
   it is a full replacement for it.  Thus we needed a different name.  We couldn't
   use ``ipython`` (lowercase) as some files systems are case insensitive.
 
 There are two, no three, main goals of the IPython effort:
 
 1. Clean up the existing codebase and write lots of tests.
 2. Separate the core functionality of IPython from the terminal to enable IPython
    to be used from within a variety of GUI applications.
 3. Implement a system for interactive parallel computing.
 
 While the third goal may seem a bit unrelated to the main focus of IPython, it
 turns out that the technologies required for this goal are nearly identical
 with those required for goal two. This is the main reason the interactive
 parallel computing capabilities are being put into IPython proper. Currently
 the third of these goals is furthest along.
 
 This document describes IPython from the perspective of developers.  
 
 
 Project organization
 ====================
 
 Subpackages
 -----------
 
 IPython is organized into semi self-contained subpackages.  Each of the
 subpackages will have its own:
 
 - **Dependencies**.  One of the most important things to keep in mind in
   partitioning code amongst subpackages, is that they should be used to cleanly
   encapsulate dependencies.
   
 - **Tests**.  Each subpackage shoud have its own ``tests`` subdirectory that
   contains all of the tests for that package.  For information about writing
   tests for IPython, see the `Testing System`_ section of this document.
   
 - **Configuration**.  Each subpackage should have its own ``config``
   subdirectory that contains the configuration information for the components
   of the subpackage.  For information about how the IPython configuration
   system works, see the `Configuration System`_ section of this document.
   
 - **Scripts**.  Each subpackage should have its own ``scripts`` subdirectory
   that contains all of the command line scripts associated with the subpackage.
 
 Installation and dependencies
 -----------------------------
    
 IPython will not use `setuptools`_ for installation. Instead, we will use
 standard ``setup.py`` scripts that use `distutils`_. While there are a number a
 extremely nice features that `setuptools`_ has (like namespace packages), the
 current implementation of `setuptools`_ has performance problems, particularly
 on shared file systems. In particular, when Python packages are installed on
 NSF file systems, import times become much too long (up towards 10 seconds).
 
 Because IPython is being used extensively in the context of high performance
 computing, where performance is critical but shared file systems are common, we
 feel these performance hits are not acceptable. Thus, until the performance
 problems associated with `setuptools`_ are addressed, we will stick with plain
 `distutils`_. We are hopeful that these problems will be addressed and that we
 will eventually begin using `setuptools`_. Because of this, we are trying to
 organize IPython in a way that will make the eventual transition to
 `setuptools`_ as painless as possible.
 
 Because we will be using `distutils`_, there will be no method for
 automatically installing dependencies.  Instead, we are following the approach
 of `Matplotlib`_ which can be summarized as follows:
 
 - Distinguish between required and optional dependencies.  However, the required
   dependencies for IPython should be only the Python standard library.
   
 - Upon installation check to see which optional dependencies are present and
   tell the user which parts of IPython need which optional dependencies.
 
 It is absolutely critical that each subpackage of IPython has a clearly
 specified set of dependencies and that dependencies are not carelessly
 inherited from other IPython subpackages. Furthermore, tests that have certain
 dependencies should not fail if those dependencies are not present. Instead
 they should be skipped and print a message.
 
 .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
 .. _distutils: http://docs.python.org/lib/module-distutils.html
 .. _Matplotlib: http://matplotlib.sourceforge.net/
 
 Specific subpackages
 --------------------
 
 ``core``
     This is the core functionality of IPython that is independent of the
     terminal, network and GUIs.  Most of the code that is in the current
     IPython trunk will be refactored, cleaned up and moved here.
 
 ``kernel``
     The enables the IPython core to be expose to a the network.  This is
     also where all of the parallel computing capabilities are to be found.
     
 ``config``
     The configuration package used by IPython.
 
 ``frontends``
     The various frontends for IPython.  A frontend is the end-user application
     that exposes the capabilities of IPython to the user.  The most basic
     frontend will simply be a terminal based application that looks just like
     today 's IPython.  Other frontends will likely be more powerful and based
     on GUI toolkits.
 
 ``notebook``
     An application that allows users to work with IPython notebooks.
 
 ``tools``
     This is where general utilities go.
 
 
 Version control
 ===============
 
 In the past, IPython development has been done using `Subversion`__.  Recently,
 we made the transition to using `Bazaar`__ and `Launchpad`__.  This makes it
 much easier for people to contribute code to IPython.  Here is a sketch of how
 to use Bazaar for IPython development.  First, you should install Bazaar.
 After you have done that, make sure that it is working by getting the latest
 main branch of IPython::
 
     $ bzr branch lp:ipython
 
 Now you can create a new branch for you to do your work in::
 
     $ bzr branch ipython ipython-mybranch
 
 The typical work cycle in this branch will be to make changes in
 ``ipython-mybranch`` and then commit those changes using the commit command::
 
     $ ...do work in ipython-mybranch...
     $ bzr ci -m "the commit message goes here"
 
 Please note that since we now don't use an old-style linear ChangeLog (that
 tends to cause problems with distributed version control systems), you should
 ensure that your log messages are reasonably detailed.  Use a docstring-like
 approach in the commit messages (including the second line being left
 *blank*)::
 
   Single line summary of  changes being committed.
 
   - more details when warranted ...
   - including crediting outside contributors if they sent the
     code/bug/idea!
 
 If we couple this with a policy of making single commits for each reasonably
 atomic change, the bzr log should give an excellent view of the project, and
 the `--short` log option becomes a nice summary.
 
 While working with this branch, it is a good idea to merge in changes that have
 been made upstream in the parent branch.  This can be done by doing::
 
     $ bzr pull
     
 If this command shows that the branches have diverged, then you should do a
 merge instead::
 
     $ bzr merge lp:ipython
 
 If you want others to be able to see your branch, you can create an account
 with launchpad and push the branch to your own workspace::
 
     $ bzr push bzr+ssh://<me>@bazaar.launchpad.net/~<me>/+junk/ipython-mybranch
 
 Finally, once the work in your branch is done, you can merge your changes back
 into the `ipython` branch by using merge::
 
     $ cd ipython
     $ merge ../ipython-mybranch
     [resolve any conflicts]
     $ bzr ci -m "Fixing that bug"
     $ bzr push
 
 But this will require you to have write permissions to the `ipython` branch.
 It you don't you can tell one of the IPython devs about your branch and they
 can do the merge for you.
 
 More information about Bazaar workflows can be found `here`__.
 
 .. __: http://subversion.tigris.org/
 .. __: http://bazaar-vcs.org/
 .. __: http://www.launchpad.net/ipython
 .. __: http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/index.html
 
 Documentation
 =============
 
 Standalone documentation
 ------------------------
 
 All standalone documentation should be written in plain text (``.txt``) files
 using `reStructuredText`_ for markup and formatting. All such documentation
 should be placed in the top level directory ``docs`` of the IPython source
 tree. Or, when appropriate, a suitably named subdirectory should be used. The
 documentation in this location will serve as the main source for IPython
 documentation and all existing documentation should be converted to this
 format.
 
 In the future, the text files in the ``docs`` directory will be used to
 generate all forms of documentation for IPython. This include documentation on
 the IPython website as well as *pdf* documentation.
 
 .. _reStructuredText: http://docutils.sourceforge.net/rst.html
 
 Docstring format
 ----------------
 
 Good docstrings are very important. All new code will use `Epydoc`_ for
 generating API docs, so we will follow the `Epydoc`_ conventions. More
 specifically, we will use `reStructuredText`_ for markup and formatting, since
 it is understood by a wide variety of tools. This means that if in the future
 we have any reason to change from `Epydoc`_ to something else, we'll have fewer
 transition pains.
 
 Details about using `reStructuredText`_ for docstrings can be found `here
 <http://epydoc.sourceforge.net/manual-othermarkup.html>`_.
 
 .. _Epydoc: http://epydoc.sourceforge.net/
 
 Additional PEPs of interest regarding documentation of code:
 
 - `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
 - `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
 - `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_
 
 
 Coding conventions
 ==================
 
 General
 -------
 
 In general, we'll try to follow the standard Python style conventions as
 described here:
 
 - `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  
 
 
 Other comments:
 
 - In a large file, top level classes and functions should be 
   separated by 2-3 lines to make it easier to separate them visually.
 - Use 4 spaces for indentation.
 - Keep the ordering of methods the same in classes that have the same 
   methods.  This is particularly true for classes that implement 
   similar interfaces and for interfaces that are similar.
 
 Naming conventions
 ------------------
 
 In terms of naming conventions, we'll follow the guidelines from the `Style
 Guide for Python Code`_.
 
 For all new IPython code (and much existing code is being refactored), we'll use:
 
 - All ``lowercase`` module names.
 
 - ``CamelCase`` for class names.
 
 - ``lowercase_with_underscores`` for methods, functions, variables and
   attributes.
 
 This may be confusing as most of the existing IPython codebase uses a different
 convention (``lowerCamelCase`` for methods and attributes).  Slowly, we will
 move IPython over to the new convention, providing shadow names for backward
 compatibility in public interfaces.
 
 There are, however, some important exceptions to these rules.  In some cases,
 IPython code will interface with packages (Twisted, Wx, Qt) that use other
 conventions.  At some level this makes it impossible to adhere to our own
 standards at all times.  In particular, when subclassing classes that use other
 naming conventions, you must follow their naming conventions.  To deal with
 cases like this, we propose the following policy:
 
 - If you are subclassing a class that uses different conventions, use its
   naming conventions throughout your subclass.  Thus, if you are creating a
   Twisted Protocol class, used Twisted's
   ``namingSchemeForMethodsAndAttributes.``
 
 - All IPython's official interfaces should use our conventions.  In some cases
   this will mean that you need to provide shadow names (first implement
   ``fooBar`` and then ``foo_bar = fooBar``).  We want to avoid this at all
   costs, but it will probably be necessary at times.  But, please use this
   sparingly!
 
 Implementation-specific *private* methods will use
 ``_single_underscore_prefix``.  Names with a leading double underscore will
 *only* be used in special cases, as they makes subclassing difficult (such
 names are not easily seen by child classes).
 
 Occasionally some run-in lowercase names are used, but mostly for very short
 names or where we are implementing methods very similar to existing ones in a
 base class (like ``runlines()`` where ``runsource()`` and ``runcode()`` had
 established precedent).
 
 The old IPython codebase has a big mix of classes and modules prefixed with an
 explicit ``IP``. In Python this is mostly unnecessary, redundant and frowned
 upon, as namespaces offer cleaner prefixing. The only case where this approach
 is justified is for classes which are expected to be imported into external
 namespaces and a very generic name (like Shell) is too likely to clash with
 something else. We'll need to revisit this issue as we clean up and refactor
 the code, but in general we should remove as many unnecessary ``IP``/``ip``
 prefixes as possible. However, if a prefix seems absolutely necessary the more
 specific ``IPY`` or ``ipy`` are preferred.
 
 .. _devel_testing:
 
 Testing system
 ==============
 
 It is extremely important that all code contributed to IPython has tests. Tests
 should be written as unittests, doctests or as entities that the `Nose`_
 testing package will find. Regardless of how the tests are written, we will use
 `Nose`_ for discovering and running the tests. `Nose`_ will be required to run
 the IPython test suite, but will not be required to simply use IPython.
 
 .. _Nose: http://code.google.com/p/python-nose/
 
 Tests of `Twisted`__ using code should be written by subclassing the
 ``TestCase`` class that comes with ``twisted.trial.unittest``. When this is
 done, `Nose`_ will be able to run the tests and the twisted reactor will be
 handled correctly.
 
 .. __: http://www.twistedmatrix.com
 
 Each subpackage in IPython should have its own ``tests`` directory that
 contains all of the tests for that subpackage. This allows each subpackage to
 be self-contained. If a subpackage has any dependencies beyond the Python
 standard library, the tests for that subpackage should be skipped if the
 dependencies are not found. This is very important so users don't get tests
 failing simply because they don't have dependencies.
 
 We also need to look into use Noses ability to tag tests to allow a more
 modular approach of running tests.
 
 .. _devel_config:
 
 Configuration system
 ====================
 
 IPython uses `.ini`_ files for configuration purposes. This represents a huge
 improvement over the configuration system used in IPython. IPython works with
 these files using the `ConfigObj`_ package, which IPython includes as
 ``ipython1/external/configobj.py``.
 
 Currently, we are using raw `ConfigObj`_ objects themselves. Each subpackage of
 IPython should contain a ``config`` subdirectory that contains all of the
 configuration information for the subpackage. To see how configuration
 information is defined (along with defaults) see at the examples in
 ``ipython1/kernel/config`` and ``ipython1/core/config``. Likewise, to see how
 the configuration information is used, see examples in
 ``ipython1/kernel/scripts/ipengine.py``.
 
 Eventually, we will add a new layer on top of the raw `ConfigObj`_ objects. We
 are calling this new layer, ``tconfig``, as it will use a `Traits`_-like
 validation model.  We won't actually use `Traits`_, but will implement
 something similar in pure Python.  But, even in this new system, we will still
 use `ConfigObj`_ and `.ini`_ files underneath the hood. Talk to Fernando if you
 are interested in working on this part of IPython. The current prototype of
 ``tconfig`` is located in the IPython sandbox.
 
 .. _.ini: http://docs.python.org/lib/module-ConfigParser.html
 .. _ConfigObj: http://www.voidspace.org.uk/python/configobj.html
 .. _Traits: http://code.enthought.com/traits/
 
 
 Installation and testing scenarios
 ==================================
 
 This section outlines the various scenarios that we need to test before we
 release an IPython version.  These scenarios represent different ways of
 installing IPython and its dependencies.
 
 Installation scenarios under Linux and OS X
 -------------------------------------------
 
     1.  Install from tarball using ``python setup.py install``.
         a.  With only readline+nose dependencies installed.
         b.  With all dependencies installed (readline, zope.interface, Twisted,
         foolscap, Sphinx, nose, pyOpenSSL).
 	    
     2.  Install using easy_install.
 
         a.  With only readline+nose dependencies installed.
             i.  Default dependencies: ``easy_install ipython-0.9.beta3-py2.5.egg``
             ii. Optional dependency sets: ``easy_install -f ipython-0.9.beta3-py2.5.egg IPython[kernel,doc,test,security]``
 
         b.  With all dependencies already installed.
 	
 
 Installation scenarios under Win32
 ----------------------------------
 
 	1.  Install everything from .exe installers
 	2.  easy_install?
 
 
 Tests to run for these scenarios
 --------------------------------
 
     1.  Run the full test suite.
     2.  Start a controller and engines and try a few things by hand.
         a.  Using ipcluster.
         b.  Using ipcontroller/ipengine by hand.
 
     3.  Run a few of the parallel examples.
     4.  Try the kernel with and without security with and without PyOpenSSL
         installed.
     5.  Beat on the IPython terminal a bunch.
     6.  Make sure that furl files are being put in proper locations.
 
 
 Release checklist
 =================
 
 Most of the release process is automated by the :file:`release` script in the
 :file:`tools` directory.  This is just a handy reminder for the release manager.
 
 #. Run the release script, which makes the tar.gz, eggs and Win32 .exe
    installer.  It posts them to the site and registers the release with PyPI.
 
 #. Updating the website with announcements and links to the updated changes.txt
    in html form.  Remember to put a short note both on the news page of the site
    and on launcphad.
 
 #. Drafting a short release announcement with i) highlights and ii) a link to
    the html changes.txt.
 
 #. Make sure that the released version of the docs is live on the site.
 
 #. Celebrate!
+
+
+Porting to 3.0
+==============
+There are no definite plans for porting of IPython to python 3. The major 
+issue is the dependency on twisted framework for the networking/threading 
+stuff. It is possible that it the traditional IPython interactive console 
+could be ported more easily since it has no such dependency. Here are a few
+things that will need to be considered when doing such a port especially
+if we want to have a codebase that works directly on both 2.x and 3.x.
+
+	1. The syntax for exceptions changed (PEP 3110). The old 
+	   `except exc, var` changed to `except exc as var`. At last
+	   count there was 78 occurences of this usage in the codebase
+