docs/source/development/overview.txt

.. This file has a lot of bash but little python, set default role

.. highlight:: bash

.. _development:

==============================
 IPython development overview
==============================

Currently, IPython's development tree contains all of the code that used to be
part of IPython since the project's inception in 2001, plus all of the effort
that was known for a while (roughly 2004-2008) as ``IPython1``.  The IPyhton1
development was meant to be an effort to:

1. Clean up the existing codebase and write lots of tests.

2. Separate the core functionality of IPython from the terminal to enable
   IPython to be used from within a variety of GUI applications.
   
3. Implement a system for interactive parallel computing.

While the third goal may seem a bit unrelated to the main focus of IPython, it
turns out that the technologies required for this goal are nearly identical
with those required for goal two. This is the main reason the interactive
parallel computing capabilities are being put into IPython proper.

In the summer of 2008, the IPython1 code was merged back into the mainline, and
now there is a unified codebase.  While the above goals aren't completely
implemented for the older code, we do have a proper :ref:`testing system
<devel-testing>` in place (though this is still evolving), unified
documentation and partial implementations of the more separated components.

This document describes IPython from the perspective of developers.  


Project organization
====================

Subpackages
-----------

IPython is organized into semi self-contained subpackages.  Each of the
subpackages will have its own:

- **Dependencies**.  One of the most important things to keep in mind in
  partitioning code amongst subpackages, is that they should be used to cleanly
  encapsulate dependencies.
  
- **Tests**.  Each subpackage shoud have its own ``tests`` subdirectory that
  contains all of the tests for that package.  For information about writing
  tests for IPython, see the :ref:`Testing System <devel-testing>` section of
  this document.
  
- **Configuration**.  Each subpackage should have its own ``config``
  subdirectory that contains the configuration information for the components
  of the subpackage.  For information about how the IPython configuration
  system works, see the :ref:`Configuration System <devel-config>` section of
  this document.
  
- **Scripts**.  Each subpackage should have its own ``scripts`` subdirectory
  that contains all of the command line scripts associated with the subpackage.

  
Installation and dependencies
-----------------------------
   
IPython will not use `setuptools`_ for installation. Instead, we will use
standard ``setup.py`` scripts that use `distutils`_. While there are a number a
extremely nice features that `setuptools`_ has (like namespace packages), the
current implementation of `setuptools`_ has performance problems, particularly
on shared file systems. In particular, when Python packages are installed on
NSF file systems, import times become much too long (up towards 10 seconds).

Because IPython is being used extensively in the context of high performance
computing, where performance is critical but shared file systems are common, we
feel these performance hits are not acceptable. Thus, until the performance
problems associated with `setuptools`_ are addressed, we will stick with plain
`distutils`_. We are hopeful that these problems will be addressed and that we
will eventually begin using `setuptools`_. Because of this, we are trying to
organize IPython in a way that will make the eventual transition to
`setuptools`_ as painless as possible.

Because we will be using `distutils`_, there will be no method for
automatically installing dependencies.  Instead, we are following the approach
of `Matplotlib`_ which can be summarized as follows:

- Distinguish between required and optional dependencies.  However, the required
  dependencies for IPython should be only the Python standard library.
  
- Upon installation check to see which optional dependencies are present and
  tell the user which parts of IPython need which optional dependencies.

It is absolutely critical that each subpackage of IPython has a clearly
specified set of dependencies and that dependencies are not carelessly
inherited from other IPython subpackages. Furthermore, tests that have certain
dependencies should not fail if those dependencies are not present. Instead
they should be skipped and print a message.

.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
.. _distutils: http://docs.python.org/lib/module-distutils.html
.. _Matplotlib: http://matplotlib.sourceforge.net/

Specific subpackages
--------------------

``core``
    This is the core functionality of IPython that is independent of the
    terminal, network and GUIs.  Most of the code that is in the current
    IPython trunk will be refactored, cleaned up and moved here.

``kernel``
    The enables the IPython core to be expose to a the network.  This is
    also where all of the parallel computing capabilities are to be found.
    
``config``
    The configuration package used by IPython.

``frontends``
    The various frontends for IPython.  A frontend is the end-user application
    that exposes the capabilities of IPython to the user.  The most basic
    frontend will simply be a terminal based application that looks just like
    today 's IPython.  Other frontends will likely be more powerful and based
    on GUI toolkits.

``tools``
    This is where general utilities go.


Version control workflow
========================

All IPython development today is done using the `Bazaar`__ system for
distributed version control and the `Launchpad`__ site for code hosting and bug
tracking.  This makes it very easy for anyone to contribute code to IPython.

.. __: http://bazaar-vcs.org
.. __: http://launchpad.net/ipython

If you are interested in contributing to IPython, you should familiarize a bit
with how to use bazaar, but this document gives you a concise set of steps to
get started.  If you are going to become heavily involved with creating code
for IPython you'll eventually want to have a Launchpad account (free) so you
can publish your own code on the site for review and merging, but small
contributions can be simply sent via email to the IPython list as patches.

Start by creating what Bazaar calls a `shared repository`_::

    # You can choose any name you want instead of "ipython-repo"
    bzr init-repo ipython-repo

.. _shared repository: http://bazaar-vcs.org/SharedRepositoryTutorial

This creates an empty repository where a lot of related branches can be kept,
and they all reuse common storage.  This way, many operations are very fast and
take up less space.  Now, go to the newly created repository and make a local
branch of the public trunk::

    cd ipython-repo
    # This makes a local copy of the public trunk called "trunk-lp"
    bzr branch lp:ipython trunk-lp

The ``trunk-lp`` branch is meant to always be a pristine copy of the public
trunk.  From here, make a personal development copy of the public trunk, where
you'll make your changes::

    bzr branch trunk-lp trunk-dev

Now, your working area looks like this::
    
    maqroll[ipython-repo]> ls -a
    ./  ../  .bzr/  trunk-dev/  trunk-lp/

The ``.bzr`` directory is the Bazaar storage area, and you now have both the
reference copy of the public trunk and one working copy for your own
development.  You can later make further branches as needed (a common workflow
is to make branches to contain specific feature implementations until they are
ready to be merged).
    
The typical work cycle will be to make changes in ``trunk-dev`` and then commit
those changes as often as needed::

    cd trunk-dev
    # ... implement cool new feature...
    bzr commit -m "Commit message goes here."

Please note that since we now don't use an old-style linear ChangeLog (that
tends to cause problems with distributed version control systems), you should
ensure that your log messages are reasonably detailed.  For non-trivial
changes, use a docstring-like approach in the commit messages (including the
second line being left *blank*).  Type ``bzr commit`` *without* the ``-m``
switch, and Bazaar will open an editor where you can type a more detailed
message::

    Single line summary of changes being committed.

    - more details when warranted ...
    - including crediting outside contributors if they sent the
      code/bug/idea!

If we couple this with a policy of making single commits for each reasonably
atomic change, the bzr log should give an excellent view of the project, and
the ``--short`` log option becomes a nice summary.

As you work on the branch, it's a good idea to frequently keep your copy of the
trunk updated with respect to Launchpad.  This is done via::

    cd trunk-lp
    bzr pull

Bazaar can then merge any changes that were done to the public trunk into your
local branch via::

    cd trunk-dev
    bzr merge ../trunk-lp
    bzr commit -m"Merged upstream changes"

This workflow ensures that a local copy stays in sync with the public trunk,
while allowing for local development to be pushed back for public review.

Once your changes are ready for review, you can push them to Launchpad where
the IPython team can review them and give you feedback.  The first time, use::

    bzr push --remember \
    bzr+ssh://<you>@bazaar.launchpad.net/~<you>/ipython/trunk-dev

where ``<you>`` is your Launchpad user name. This will make this branch
publicly visible to all.  In subsequent uses you can simply run in the branch::

    bzr push

.. note::

    Before you can push your code to Launchpad, you need to configure your SSH
    keys.  You can do this by clicking on ``Change details`` for your profile
    and then clicking on the ``SSH Keys`` tab.  This should take you to a URL
    of the form ``https://launchpad.net/~<you>/+editsshkeys`` with instructions
    on how to upload your keys.
    
Once the branch is publicly visible, using the launchpad interface it can be
marked for merging with the link marked *"Propose for merging into another
branch"*, leaving the default choice of trunk as its target.  This way,
Launchpad tracks what changes of this branch are new with respect to the trunk,
others in the team can review it, and merge the changes into the trunk.

The IPython team has a policy of reviewing all code (both from core members of
the team and from external contributors) before merging it.  Code should be
clearly documented (as explained further in this guide), clear and well tested.
We will be happy to provide you with feedback for your code to be a great
contribution to the project, and we've found that peer review of code is an
excellent way to improve the quality of everyone's work.


Installation and testing scenarios
==================================

This section outlines the various scenarios that we need to test before we
release an IPython version.  These scenarios represent different ways of
installing IPython and its dependencies.


Installation scenarios under Linux and OS X
-------------------------------------------

1.  Install from tarball using ``python setup.py install``.

  * With only readline+nose dependencies installed.

  * With all dependencies installed (readline, zope.interface, Twisted,
    foolscap, Sphinx, nose, pyOpenSSL).

2.  Install using easy_install.

  * With only readline+nose dependencies installed.
    
    * Default dependencies: ``easy_install ipython-0.9.beta3-py2.5.egg``

    * Optional dependency sets:
      ``easy_install -f ipython-0.9.beta3-py2.5.egg IPython[kernel,doc,test,security]``

  * With all dependencies already installed.
	

Installation scenarios under Win32
----------------------------------

#. Install everything from .exe installers
#. easy_install?


Tests to run for these scenarios
--------------------------------

#. Run the full test suite.

#. Start a controller and engines and try a few things by hand.

   * Using ipcluster.
   * Using ipcontroller/ipengine by hand.

#. Run a few of the parallel examples.

#. Try the kernel with and without security with and without PyOpenSSL
   installed.

#. Beat on the IPython terminal a bunch.

#. Make sure that furl files are being put in proper locations.


Release checklist
=================

Most of the release process is automated by the :file:`release` script in the
:file:`tools` directory.  This is just a handy reminder for the release manager.

#. Run the release script, which makes the tar.gz, eggs and Win32 .exe
   installer.  It posts them to the site and registers the release with PyPI.

#. Updating the website with announcements and links to the updated changes.txt
   in html form.  Remember to put a short note both on the news page of the site
   and on launcphad.

#. Drafting a short release announcement with i) highlights and ii) a link to
   the html changes.txt.

#. Make sure that the released version of the docs is live on the site.

#. Celebrate!