docs/source/development/overview.txt

.. _development:

==============================
IPython development guidelines
==============================


Overview
========

This document describes IPython from the perspective of developers. Most
importantly, it gives information for people who want to contribute to the
development of IPython. So if you want to help out, read on!

How to contribute to IPython
============================

IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
This makes it easy for people to contribute to the development of IPython.
Here is a sketch of how to get going.

Install Bazaar and create a Launchpad account
---------------------------------------------

First make sure you have installed Bazaar (see their `website
<http://bazaar-vcs.org/>`_). To see that Bazaar is installed and knows about
you, try the following::

    $ bzr whoami
    Joe Coder <[e-mail]>

This should display your name and email. Next, you will want to create an
account on the `Launchpad website <http://www.launchpad.net>`_ and setup your
ssh keys. For more information of setting up your ssh keys, see `this link
<https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair>`_.

Get the main IPython branch from Launchpad
------------------------------------------

Now, you can get a copy of the main IPython development branch (we call this
the "trunk")::

    $ bzr branch lp:ipython

Create a working branch
-----------------------

When working on IPython, you won't actually make edits directly to the
:file:`lp:ipython` branch. Instead, you will create a separate branch for your
changes. For now, let's assume you want to do your work in a branch named
"ipython-mybranch". Create this branch by doing::

    $ bzr branch ipython ipython-mybranch

When you actually create a branch, you will want to give it a name that
reflects the nature of the work that you will be doing in it, like
"install-docs-update".

Make edits in your working branch
---------------------------------

Now you are ready to actually make edits in your :file:`ipython-mybranch`
branch. Before doing this, it is helpful to install this branch so you can
test your changes as you work. This is easiest if you have setuptools
installed. Then, just do::

    $ cd ipython-mybranch
    $ python setupegg.py develop

Now, make some changes. After a while, you will want to commit your changes.
This let's Bazaar know that you like the changes you have made and gives you
an opportunity to keep a nice record of what you have done. This looks like
this::

    $ ...do work in ipython-mybranch...
    $ bzr commit -m "the commit message goes here"

Please note that since we now don't use an old-style linear ChangeLog (that
tends to cause problems with distributed version control systems), you should
ensure that your log messages are reasonably detailed.  Use a docstring-like
approach in the commit messages (including the second line being left
*blank*)::

  Single line summary of  changes being committed.

  * more details when warranted ...
  * including crediting outside contributors if they sent the
    code/bug/idea!

As you work, you will repeat this edit/commit cycle many times. If you work on
your branch for a long time, you will also want to get the latest changes from
the :file:`lp:ipython` branch. This can be done with the following sequence of
commands::

    $ ls
    ipython
    ipython-mybranch
    
    $ cd ipython
    $ bzr pull
    $ cd ../ipython-mybranch
    $ bzr merge ../ipython
    $ bzr commit -m "Merging changes from trunk"

Along the way, you should also run the IPython test suite.  You can do this using the :command:`iptest` command::

    $ cd
    $ iptest

The :command:`iptest` command will also pick up and run any tests you have written.

Post your branch and request a code review
------------------------------------------

Once you are done with your edits, you should post your branch on Launchpad so
that other IPython developers can review the changes and help you merge your
changes into the main development branch. To post your branch on Launchpad,
do::

    $ cd ipython-mybranch
    $ bzr push lp:~yourusername/ipython/ipython-mybranch

Then, go to the `IPython Launchpad site <www.launchpad.net/ipython>`_, and you
should see your branch under the "Code" tab. If you click on your branch, you
can provide a short description of the branch as well as mark its status. Most
importantly, you should click the link that reads "Propose for merging into
another branch". What does this do?

This let's the other IPython developers know that your branch is ready to be
reviewed and merged into the main development branch. During this review
process, other developers will give you feedback and help you get your code
ready to be merged. What types of things will we be looking for:

* All code is documented.
* All code has tests.
* The entire IPython test suite passes.

Once your changes have been reviewed and approved, someone will merge them
into the main development branch.

Documentation
=============

Standalone documentation
------------------------

All standalone documentation should be written in plain text (``.txt``) files
using reStructuredText [reStructuredText]_ for markup and formatting. All such
documentation should be placed in directory :file:`docs/source` of the IPython
source tree. The documentation in this location will serve as the main source
for IPython documentation and all existing documentation should be converted
to this format.

To build the final documentation, we use Sphinx [Sphinx]_.  Once you have Sphinx installed, you can build the html docs yourself by doing::

    $ cd ipython-mybranch/docs
    $ make html

Docstring format
----------------

Good docstrings are very important. All new code should have docstrings that
are formatted using reStructuredText for markup and formatting, since it is
understood by a wide variety of tools. Details about using reStructuredText
for docstrings can be found `here
<http://epydoc.sourceforge.net/manual-othermarkup.html>`_.

Additional PEPs of interest regarding documentation of code:

* `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
* `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
* `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_


Coding conventions
==================

General
-------

In general, we'll try to follow the standard Python style conventions as
described here:

* `Style Guide for Python Code  <http://www.python.org/peps/pep-0008.html>`_  


Other comments:

* In a large file, top level classes and functions should be 
  separated by 2-3 lines to make it easier to separate them visually.
* Use 4 spaces for indentation.
* Keep the ordering of methods the same in classes that have the same 
  methods.  This is particularly true for classes that implement an interface.

Naming conventions
------------------

In terms of naming conventions, we'll follow the guidelines from the `Style
Guide for Python Code`_.

For all new IPython code (and much existing code is being refactored), we'll
use:

* All ``lowercase`` module names.

* ``CamelCase`` for class names.

* ``lowercase_with_underscores`` for methods, functions, variables and
  attributes.

There are, however, some important exceptions to these rules. In some cases,
IPython code will interface with packages (Twisted, Wx, Qt) that use other
conventions. At some level this makes it impossible to adhere to our own
standards at all times. In particular, when subclassing classes that use other
naming conventions, you must follow their naming conventions. To deal with
cases like this, we propose the following policy:

* If you are subclassing a class that uses different conventions, use its
  naming conventions throughout your subclass.  Thus, if you are creating a
  Twisted Protocol class, used Twisted's
  ``namingSchemeForMethodsAndAttributes.``

* All IPython's official interfaces should use our conventions.  In some cases
  this will mean that you need to provide shadow names (first implement
  ``fooBar`` and then ``foo_bar = fooBar``).  We want to avoid this at all
  costs, but it will probably be necessary at times.  But, please use this
  sparingly!

Implementation-specific *private* methods will use
``_single_underscore_prefix``. Names with a leading double underscore will
*only* be used in special cases, as they makes subclassing difficult (such
names are not easily seen by child classes).

Occasionally some run-in lowercase names are used, but mostly for very short
names or where we are implementing methods very similar to existing ones in a
base class (like ``runlines()`` where ``runsource()`` and ``runcode()`` had
established precedent).

The old IPython codebase has a big mix of classes and modules prefixed with an
explicit ``IP``. In Python this is mostly unnecessary, redundant and frowned
upon, as namespaces offer cleaner prefixing. The only case where this approach
is justified is for classes which are expected to be imported into external
namespaces and a very generic name (like Shell) is too likely to clash with
something else. We'll need to revisit this issue as we clean up and refactor
the code, but in general we should remove as many unnecessary ``IP``/``ip``
prefixes as possible. However, if a prefix seems absolutely necessary the more
specific ``IPY`` or ``ipy`` are preferred.

.. _devel_testing:

Testing system
==============

It is extremely important that all code contributed to IPython has tests.
Tests should be written as unittests, doctests or as entities that the Nose
[Nose]_ testing package will find. Regardless of how the tests are written, we
will use Nose for discovering and running the tests. Nose will be required to
run the IPython test suite, but will not be required to simply use IPython.

Tests of Twisted using code need to follow two additional guidelines:

1. Twisted using tests should be written by subclassing the :class:`TestCase`
   class that comes with :mod:`twisted.trial.unittest`. 

2. All :class:`Deferred` instances that are created in the test must be
   properly chained and the final one *must* be the return value of the test
   method.

When these two things are done, Nose will be able to run the tests and the
twisted reactor will be handled correctly.

Each subpackage in IPython should have its own :file:`tests` directory that
contains all of the tests for that subpackage. This allows each subpackage to
be self-contained. If a subpackage has any dependencies beyond the Python
standard library, the tests for that subpackage should be skipped if the
dependencies are not found. This is very important so users don't get tests
failing simply because they don't have dependencies.

To run the IPython test suite, use the :command:`iptest` command that is
installed with IPython::

    $ iptest

This command runs Nose with the proper options and extensions.

A few tips for writing tests:

* You can use IPython examples in your docstrings, including all IPython
  prompts.  Rather than repeating it all here, see the files
  :file:`dtexample.py` and :file:`test_ipdoctest.py` in the
  :mod:`IPython.testing.plugin` module for examples of how you can use plain
  Python or IPython prompts, and what to do with examples whose output could be
  partly or completely random.

* Use the decorators shipped in the :mod:`IPython.testing` package to tag tests
  that may be platform-specific or otherwise may have restrictions.

* If a test isn't safe to run inside the main nose process (e.g. because it
  loads a GUI toolkit), consider running it in a subprocess and capturing its
  output for evaluation and test decision later.  Here is an example of how to
  do it, by relying on the builtin ``_ip`` object that contains the public
  IPython api as defined in :mod:`IPython.ipapi`::

   def test_obj_del():
       """Test that object's __del__ methods are called on exit."""
       test_dir = os.path.dirname(__file__)
       del_file = os.path.join(test_dir,'obj_del.py')
       out = _ip.IP.getoutput('ipython %s' % del_file)
       nt.assert_equals(out,'object A deleted')


* In a file named ``test_X``, functions whose only test is their docstring (as
  a doctest) and which have no test functionality of their own, should be
  called *doctest_foo* instead of *test_foo*, otherwise they get double-counted
  (the empty function call is counted as a test, which just inflates tests
  numbers artificially).  This restriction does not apply to functions in files
  with other names, due to how Nose discovers tests.

.. _devel_config:

Release checklist
=================

Most of the release process is automated by the :file:`release` script in the
:file:`tools` directory.  This is just a handy reminder for the release manager.

#. Run the release script, which makes the tar.gz, eggs and Win32 .exe
   installer.  It posts them to the site and registers the release with PyPI.

#. Updating the website with announcements and links to the updated
   changes.txt in html form. Remember to put a short note both on the news
   page of the site and on Launcphad.

#. Drafting a short release announcement with i) highlights and ii) a link to
   the html changes.txt.

#. Make sure that the released version of the docs is live on the site.

#. Celebrate!

Porting to 3.0
==============
There are no definite plans for porting of IPython to python 3. The major 
issue is the dependency on twisted framework for the networking/threading 
stuff. It is possible that it the traditional IPython interactive console 
could be ported more easily since it has no such dependency. Here are a few
things that will need to be considered when doing such a port especially
if we want to have a codebase that works directly on both 2.x and 3.x.

	1. The syntax for exceptions changed (PEP 3110). The old 
	   `except exc, var` changed to `except exc as var`. At last
	   count there was 78 occurences of this usage in the codebase
	   
.. [Bazaar] Bazaar. http://bazaar-vcs.org/
.. [Launchpad] Launchpad. http://www.launchpad.net/ipython
.. [reStructuredText] reStructuredText.  http://docutils.sourceforge.net/rst.html
.. [Sphinx] Sphinx. http://sphinx.pocoo.org/
.. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/