docs/source/development/overview.txt

.. _development:

==============================
IPython development guidelines
==============================


Overview
========

This document describes IPython from the perspective of developers. Most
importantly, it gives information for people who want to contribute to the
development of IPython. So if you want to help out, read on!

How to contribute to IPython
============================

IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
This makes it easy for people to contribute to the development of IPython.
There are several ways in which you can join in.

If you have a small change that you want to send to the team, you can edit your
bazaar checkout of IPython (see below) in-place, and ask bazaar for the
differences::

    $ cd /path/to/your/copy/of/ipython
    $ bzr diff > my_fixes.diff

This produces a patch file with your fixes, which we can apply to the source
tree.  This file should then be attached to a ticket in our `bug tracker
<https://bugs.launchpad.net/ipython>`_, indicating what it does.

This model of creating small, self-contained patches works very well and there
are open source projects that do their entire development this way.  However,
in IPython we have found that for tracking larger changes, making use of
bazaar's full capabilities in conjunction with Launchpad's code hosting
services makes for a much better experience.

Making your own branch of IPython allows you to refine your changes over time,
track the development of the main team, and propose your own full version of
the code for others to use and review, with a minimum amount of fuss.  The next
parts of this document will explain how to do this.

Install Bazaar and create a Launchpad account
---------------------------------------------

First make sure you have installed Bazaar (see their `website
<http://bazaar-vcs.org/>`_). To see that Bazaar is installed and knows about
you, try the following::

    $ bzr whoami
    Joe Coder <[e-mail]>

This should display your name and email. Next, you will want to create an
account on the `Launchpad website <http://www.launchpad.net>`_ and setup your
ssh keys. For more information of setting up your ssh keys, see `this link
<https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair>`_.

Get the main IPython branch from Launchpad
------------------------------------------

Now, you can get a copy of the main IPython development branch (we call this
the "trunk")::

    $ bzr branch lp:ipython

Create a working branch
-----------------------

When working on IPython, you won't actually make edits directly to the
:file:`lp:ipython` branch. Instead, you will create a separate branch for your
changes. For now, let's assume you want to do your work in a branch named
"ipython-mybranch". Create this branch by doing::

    $ bzr branch ipython ipython-mybranch

When you actually create a branch, you will want to give it a name that
reflects the nature of the work that you will be doing in it, like
"install-docs-update".

Make edits in your working branch
---------------------------------

Now you are ready to actually make edits in your :file:`ipython-mybranch`
branch. Before doing this, it is helpful to install this branch so you can
test your changes as you work. This is easiest if you have setuptools
installed. Then, just do::

    $ cd ipython-mybranch
    $ python setupegg.py develop

Now, make some changes. After a while, you will want to commit your changes.
This let's Bazaar know that you like the changes you have made and gives you
an opportunity to keep a nice record of what you have done. This looks like
this::

    $ ...do work in ipython-mybranch...
    $ bzr commit -m "the commit message goes here"

Please note that since we now don't use an old-style linear ChangeLog (that
tends to cause problems with distributed version control systems), you should
ensure that your log messages are reasonably detailed.  Use a docstring-like
approach in the commit messages (including the second line being left
*blank*)::

  Single line summary of  changes being committed.

  * more details when warranted ...
  * including crediting outside contributors if they sent the
    code/bug/idea!

As you work, you will repeat this edit/commit cycle many times. If you work on
your branch for a long time, you will also want to get the latest changes from
the :file:`lp:ipython` branch. This can be done with the following sequence of
commands::

    $ ls
    ipython
    ipython-mybranch
    
    $ cd ipython
    $ bzr pull
    $ cd ../ipython-mybranch
    $ bzr merge ../ipython
    $ bzr commit -m "Merging changes from trunk"

Along the way, you should also run the IPython test suite.  You can do this
using the :command:`iptest` command (which is basically a customized version of
:command:`nosetests`)::

    $ cd
    $ iptest

The :command:`iptest` command will also pick up and run any tests you have
written.  See :ref:`_devel_testing` for further details on the testing system.


Post your branch and request a code review
------------------------------------------

Once you are done with your edits, you should post your branch on Launchpad so
that other IPython developers can review the changes and help you merge your
changes into the main development branch. To post your branch on Launchpad,
do::

    $ cd ipython-mybranch
    $ bzr push lp:~yourusername/ipython/ipython-mybranch

Then, go to the `IPython Launchpad site <www.launchpad.net/ipython>`_, and you
should see your branch under the "Code" tab. If you click on your branch, you
can provide a short description of the branch as well as mark its status. Most
importantly, you should click the link that reads "Propose for merging into
another branch". What does this do?

This let's the other IPython developers know that your branch is ready to be
reviewed and merged into the main development branch. During this review
process, other developers will give you feedback and help you get your code
ready to be merged. What types of things will we be looking for:

* All code is documented.
* All code has tests.
* The entire IPython test suite passes.

Once your changes have been reviewed and approved, someone will merge them
into the main development branch.


Some notes for core developers when merging third-party contributions
=====================================================================

Core developers, who ultimately merge any approved branch (from themselves,
another developer, or any third-party contribution) will typically use
:command:`bzr merge` to merge the branch into the trunk and push it to the main
Launcphad site.  This is a short list of things to keep in mind when doing this
process, so that the project history is easy to understand in the long run, and
that generating release notes is as painless and accurate as possible.

- When you merge any non-trivial functionality (from one small bug fix to a big
  feature branch), please remember to always edit the changes_ file
  accordingly.  This file has one main section for each release, and if you
  edit it as you go, noting what new features, bug fixes or API changes you
  have made, the release notes will be almost finished when they are needed
  later.  This is much easier if done when you merge the work, rather than
  weeks or months later by re-reading a massive Bazaar log.

- When big merges are done, the practice of putting a summary commit message in
  the merge is *extremely* useful.  It makes this kind of job much nicer,
  because that summary log message can be almost copy/pasted without changes,
  if it was well written, rather than dissecting the next-level messages from
  the individual commits.

- It's important that we remember to always credit who gave us something if
  it's not the committer.  In general, we have been fairly good on this front,
  this is just a reminder to keep things up.  As a note, if you are ever
  committing something that is completely (or almost so) a third-party
  contribution, do the commit as::

    $ bzr commit --author="Someone Else"

  This way it will show that name separately in the log, which makes it even
  easier to spot.  Obviously we often rework third party contributions
  extensively, but this is still good to keep in mind for cases when we don't
  touch the code too much.


Documentation
=============

Standalone documentation
------------------------

All standalone documentation should be written in plain text (``.txt``) files
using reStructuredText [reStructuredText]_ for markup and formatting. All such
documentation should be placed in directory :file:`docs/source` of the IPython
source tree. The documentation in this location will serve as the main source
for IPython documentation and all existing documentation should be converted
to this format.

To build the final documentation, we use Sphinx [Sphinx]_.  Once you have
Sphinx installed, you can build the html docs yourself by doing::

    $ cd ipython-mybranch/docs
    $ make html

Docstring format
----------------

Good docstrings are very important. All new code should have docstrings that
are formatted using reStructuredText for markup and formatting, since it is
understood by a wide variety of tools. Details about using reStructuredText
for docstrings can be found `here
<http://epydoc.sourceforge.net/manual-othermarkup.html>`_.

Additional PEPs of interest regarding documentation of code:

* `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
* `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
* `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_


Coding conventions
==================

General
-------

In general, we'll try to follow the standard Python style conventions as
described here:

* `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  


Other comments:

* In a large file, top level classes and functions should be 
  separated by 2-3 lines to make it easier to separate them visually.
* Use 4 spaces for indentation.
* Keep the ordering of methods the same in classes that have the same 
  methods.  This is particularly true for classes that implement an interface.

Naming conventions
------------------

In terms of naming conventions, we'll follow the guidelines from the `Style
Guide for Python Code`_.

For all new IPython code (and much existing code is being refactored), we'll
use:

* All ``lowercase`` module names.

* ``CamelCase`` for class names.

* ``lowercase_with_underscores`` for methods, functions, variables and
  attributes.

There are, however, some important exceptions to these rules. In some cases,
IPython code will interface with packages (Twisted, Wx, Qt) that use other
conventions. At some level this makes it impossible to adhere to our own
standards at all times. In particular, when subclassing classes that use other
naming conventions, you must follow their naming conventions. To deal with
cases like this, we propose the following policy:

* If you are subclassing a class that uses different conventions, use its
  naming conventions throughout your subclass.  Thus, if you are creating a
  Twisted Protocol class, used Twisted's
  ``namingSchemeForMethodsAndAttributes.``

* All IPython's official interfaces should use our conventions.  In some cases
  this will mean that you need to provide shadow names (first implement
  ``fooBar`` and then ``foo_bar = fooBar``).  We want to avoid this at all
  costs, but it will probably be necessary at times.  But, please use this
  sparingly!

Implementation-specific *private* methods will use
``_single_underscore_prefix``. Names with a leading double underscore will
*only* be used in special cases, as they makes subclassing difficult (such
names are not easily seen by child classes).

Occasionally some run-in lowercase names are used, but mostly for very short
names or where we are implementing methods very similar to existing ones in a
base class (like ``runlines()`` where ``runsource()`` and ``runcode()`` had
established precedent).

The old IPython codebase has a big mix of classes and modules prefixed with an
explicit ``IP``. In Python this is mostly unnecessary, redundant and frowned
upon, as namespaces offer cleaner prefixing. The only case where this approach
is justified is for classes which are expected to be imported into external
namespaces and a very generic name (like Shell) is too likely to clash with
something else. We'll need to revisit this issue as we clean up and refactor
the code, but in general we should remove as many unnecessary ``IP``/``ip``
prefixes as possible. However, if a prefix seems absolutely necessary the more
specific ``IPY`` or ``ipy`` are preferred.

.. _devel_testing:

Testing system
==============

It is extremely important that all code contributed to IPython has tests.
Tests should be written as unittests, doctests or as entities that the Nose
[Nose]_ testing package will find. Regardless of how the tests are written, we
will use Nose for discovering and running the tests. Nose will be required to
run the IPython test suite, but will not be required to simply use IPython.

Tests of Twisted using code need to follow two additional guidelines:

1. Twisted using tests should be written by subclassing the :class:`TestCase`
   class that comes with :mod:`twisted.trial.unittest`. 

2. All :class:`Deferred` instances that are created in the test must be
   properly chained and the final one *must* be the return value of the test
   method.

When these two things are done, Nose will be able to run the tests and the
twisted reactor will be handled correctly.

Each subpackage in IPython should have its own :file:`tests` directory that
contains all of the tests for that subpackage. This allows each subpackage to
be self-contained.  A good convention to follow is to have a file named
:file:`test_foo.py` for each module :file:`foo.py` in the package.  This makes
it easy to organize the tests, though like most conventions, it's OK to break
it if logic and common sense dictate otherwise.

If a subpackage has any dependencies beyond the Python standard library, the
tests for that subpackage should be skipped if the dependencies are not
found. This is very important so users don't get tests failing simply because
they don't have dependencies.  We ship a set of decorators in the
:mod:`IPython.testing` package to tag tests that may be platform-specific or
otherwise may have restrictions; if the existing ones don't fit your needs, add
a new decorator in that location so other tests can reuse it.

To run the IPython test suite, use the :command:`iptest` command that is
installed with IPython (if you are using IPython in-place, without installing
it, you can find this script in the :file:`scripts` directory)::

    $ iptest

This command colects all IPython tests into separate groups, and then calls
either Nose with the proper options and extensions, or Twisted's
:command:`trial`.  This ensures that tests that need the Twisted reactor
management facilities execute separate of Nose.  If any individual test group
fails, :command:`iptest` will print what you need to type so you can rerun that
particular test group alone for debugging.

By default, :command:`iptest` runs the entire IPython test
suite (skipping tests that may be platform-specific or which depend on tools
you may not have).  But you can also use it to run only one specific test file,
or a specific test function.  For example, this will run only the
:file:`test_magic` file from the test suite::

    $ iptest IPython.tests.test_magic
    ----------------------------------------------------------------------
    Ran 10 tests in 0.348s

    OK (SKIP=3)
    Deleting object: second_pass

while the ``path:function`` syntax allows you to select a specific function in
that file to run::

    $ iptest IPython.tests.test_magic:test_obj_del
    ----------------------------------------------------------------------
    Ran 1 test in 0.204s

    OK

Since :command:`iptest` is based on nosetests, you can pass it any regular
nosetests option.  For example, you can use ``--pdb`` or ``--pdb-failures`` to
automatically activate the interactive Pdb debugger on errors or failures.  See
the nosetests documentation for further details.


A few tips for writing tests
----------------------------

You can write tests either as normal test files, using all the conventions that
Nose recognizes, or as doctests.  Note that *all* IPython functions should have
at least one example that serves as a doctest, whenever technically feasible.
However, example doctests should only be in the main docstring if they are *a
good example*, i.e. if they convey useful information about the function.  If
you simply would like to write a test as a doctest, put it in a separate test
file and write a no-op function whose only purpose is its docstring.

Note, however, that in a file named :file:`test_X`, functions whose only test
is their docstring (as a doctest) and which have no test functionality of their
own, should be called *doctest_foo* instead of *test_foo*, otherwise they get
double-counted (the empty function call is counted as a test, which just
inflates tests numbers artificially).  This restriction does not apply to
functions in files with other names, due to how Nose discovers tests.

You can use IPython examples in your docstrings.  Those can make full use of
IPython functionality (magics, variable substitution, etc), but be careful to
keep them generic enough that they run identically on all Operating Systems.

The prompts in your doctests can be either of the plain Python ``>>>`` variety
or ``In [1]:`` IPython style.  Since this is the IPython system, after all, we
encourage you to use IPython prompts throughout, unless you are illustrating a
specific aspect of the normal prompts (such as the ``%doctest_mode`` magic).

If a test isn't safe to run inside the main nose process (e.g. because it loads
a GUI toolkit), consider running it in a subprocess and capturing its output
for evaluation and test decision later.  Here is an example of how to do it, by
relying on the builtin ``_ip`` object that contains the public IPython api as
defined in :mod:`IPython.ipapi`::

   def test_obj_del():
       """Test that object's __del__ methods are called on exit."""
       test_dir = os.path.dirname(__file__)
       del_file = os.path.join(test_dir,'obj_del.py')
       out = _ip.IP.getoutput('ipython %s' % del_file)
       nt.assert_equals(out,'object A deleted')



If a doctest contains input whose output you don't want to verify identically
via doctest (random output, an object id, etc), you can mark a docstring with
``#random``.  All of these test will have their code executed but no output
checking will be done::

       >>> 1+3
       junk goes here...  # random

       >>> 1+2
       again,  anything goes #random
       if multiline, the random mark is only needed once.

       >>> 1+2
       You can also put the random marker at the end:
       # random

       >>> 1+2
       # random
       .. or at the beginning.

In a case where you want an *entire* docstring to be executed but not verified
(this only serves to check that the code runs without crashing, so it should be
used very sparingly), you can put ``# all-random`` in the docstring.
       
.. _devel_config:

Release checklist
=================

Most of the release process is automated by the :file:`release` script in the
:file:`tools` directory.  This is just a handy reminder for the release manager.

#. First, run :file:`build_release`, which does all the file checking and
   building that the real release script will do.  This will let you do test
   installations, check that the build procedure runs OK, etc.  You may want to
   disable a few things like multi-version RPM building while testing, because
   otherwise the build takes really long.

#. Run the release script, which makes the tar.gz, eggs and Win32 .exe
   installer.  It posts them to the site and registers the release with PyPI.

#. Updating the website with announcements and links to the updated
   changes.txt in html form. Remember to put a short note both on the news
   page of the site and on Launcphad.

#. Drafting a short release announcement with i) highlights and ii) a link to
   the html changes.txt.

#. Make sure that the released version of the docs is live on the site.

#. Celebrate!

Porting to 3.0
==============

There are no definite plans for porting of IPython to python 3. The major 
issue is the dependency on twisted framework for the networking/threading 
stuff. It is possible that it the traditional IPython interactive console 
could be ported more easily since it has no such dependency. Here are a few
things that will need to be considered when doing such a port especially
if we want to have a codebase that works directly on both 2.x and 3.x.

	1. The syntax for exceptions changed (PEP 3110). The old 
	   `except exc, var` changed to `except exc as var`. At last
	   count there was 78 occurences of this usage in the codebase.  This
	   is  a particularly problematic issue, because it's not easy to
	   implement it in a 2.5-compatible way.

Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we
will likely at some point put out a release that requires strictly 2.6 and
abandons 2.5 compatibility.  This will then allow us to port the code to using
:func:`print` as a function, `except exc as var` syntax, etc.  But as of
version 0.11 at least, we will retain Python 2.5 compatibility.


.. [Bazaar] Bazaar. http://bazaar-vcs.org/
.. [Launchpad] Launchpad. http://www.launchpad.net/ipython
.. [reStructuredText] reStructuredText.  http://docutils.sourceforge.net/rst.html
.. [Sphinx] Sphinx. http://sphinx.pocoo.org/
.. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/