|
|
.. _development:
|
|
|
|
|
|
==============================
|
|
|
IPython development guidelines
|
|
|
==============================
|
|
|
|
|
|
|
|
|
Overview
|
|
|
========
|
|
|
|
|
|
This document describes IPython from the perspective of developers. Most
|
|
|
importantly, it gives information for people who want to contribute to the
|
|
|
development of IPython. So if you want to help out, read on!
|
|
|
|
|
|
How to contribute to IPython
|
|
|
============================
|
|
|
|
|
|
IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
|
|
|
This makes it easy for people to contribute to the development of IPython.
|
|
|
There are several ways in which you can join in.
|
|
|
|
|
|
If you have a small change that you want to send to the team, you can edit your
|
|
|
bazaar checkout of IPython (see below) in-place, and ask bazaar for the
|
|
|
differences::
|
|
|
|
|
|
$ cd /path/to/your/copy/of/ipython
|
|
|
$ bzr diff > my_fixes.diff
|
|
|
|
|
|
This produces a patch file with your fixes, which we can apply to the source
|
|
|
tree. This file should then be attached to a ticket in our `bug tracker
|
|
|
<https://bugs.launchpad.net/ipython>`_, indicating what it does.
|
|
|
|
|
|
This model of creating small, self-contained patches works very well and there
|
|
|
are open source projects that do their entire development this way. However,
|
|
|
in IPython we have found that for tracking larger changes, making use of
|
|
|
bazaar's full capabilities in conjunction with Launchpad's code hosting
|
|
|
services makes for a much better experience.
|
|
|
|
|
|
Making your own branch of IPython allows you to refine your changes over time,
|
|
|
track the development of the main team, and propose your own full version of
|
|
|
the code for others to use and review, with a minimum amount of fuss. The next
|
|
|
parts of this document will explain how to do this.
|
|
|
|
|
|
Install Bazaar and create a Launchpad account
|
|
|
---------------------------------------------
|
|
|
|
|
|
First make sure you have installed Bazaar (see their `website
|
|
|
<http://bazaar-vcs.org/>`_). To see that Bazaar is installed and knows about
|
|
|
you, try the following::
|
|
|
|
|
|
$ bzr whoami
|
|
|
Joe Coder <jcoder@gmail.com>
|
|
|
|
|
|
This should display your name and email. Next, you will want to create an
|
|
|
account on the `Launchpad website <http://www.launchpad.net>`_ and setup your
|
|
|
ssh keys. For more information of setting up your ssh keys, see `this link
|
|
|
<https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair>`_.
|
|
|
|
|
|
Get the main IPython branch from Launchpad
|
|
|
------------------------------------------
|
|
|
|
|
|
Now, you can get a copy of the main IPython development branch (we call this
|
|
|
the "trunk")::
|
|
|
|
|
|
$ bzr branch lp:ipython
|
|
|
|
|
|
Create a working branch
|
|
|
-----------------------
|
|
|
|
|
|
When working on IPython, you won't actually make edits directly to the
|
|
|
:file:`lp:ipython` branch. Instead, you will create a separate branch for your
|
|
|
changes. For now, let's assume you want to do your work in a branch named
|
|
|
"ipython-mybranch". Create this branch by doing::
|
|
|
|
|
|
$ bzr branch ipython ipython-mybranch
|
|
|
|
|
|
When you actually create a branch, you will want to give it a name that
|
|
|
reflects the nature of the work that you will be doing in it, like
|
|
|
"install-docs-update".
|
|
|
|
|
|
Make edits in your working branch
|
|
|
---------------------------------
|
|
|
|
|
|
Now you are ready to actually make edits in your :file:`ipython-mybranch`
|
|
|
branch. Before doing this, it is helpful to install this branch so you can
|
|
|
test your changes as you work. This is easiest if you have setuptools
|
|
|
installed. Then, just do::
|
|
|
|
|
|
$ cd ipython-mybranch
|
|
|
$ python setupegg.py develop
|
|
|
|
|
|
Now, make some changes. After a while, you will want to commit your changes.
|
|
|
This let's Bazaar know that you like the changes you have made and gives you
|
|
|
an opportunity to keep a nice record of what you have done. This looks like
|
|
|
this::
|
|
|
|
|
|
$ ...do work in ipython-mybranch...
|
|
|
$ bzr commit -m "the commit message goes here"
|
|
|
|
|
|
Please note that since we now don't use an old-style linear ChangeLog (that
|
|
|
tends to cause problems with distributed version control systems), you should
|
|
|
ensure that your log messages are reasonably detailed. Use a docstring-like
|
|
|
approach in the commit messages (including the second line being left
|
|
|
*blank*)::
|
|
|
|
|
|
Single line summary of changes being committed.
|
|
|
|
|
|
* more details when warranted ...
|
|
|
* including crediting outside contributors if they sent the
|
|
|
code/bug/idea!
|
|
|
|
|
|
As you work, you will repeat this edit/commit cycle many times. If you work on
|
|
|
your branch for a long time, you will also want to get the latest changes from
|
|
|
the :file:`lp:ipython` branch. This can be done with the following sequence of
|
|
|
commands::
|
|
|
|
|
|
$ ls
|
|
|
ipython
|
|
|
ipython-mybranch
|
|
|
|
|
|
$ cd ipython
|
|
|
$ bzr pull
|
|
|
$ cd ../ipython-mybranch
|
|
|
$ bzr merge ../ipython
|
|
|
$ bzr commit -m "Merging changes from trunk"
|
|
|
|
|
|
Along the way, you should also run the IPython test suite. You can do this
|
|
|
using the :command:`iptest` command (which is basically a customized version of
|
|
|
:command:`nosetests`)::
|
|
|
|
|
|
$ cd
|
|
|
$ iptest
|
|
|
|
|
|
The :command:`iptest` command will also pick up and run any tests you have
|
|
|
written. See :ref:`_devel_testing` for further details on the testing system.
|
|
|
|
|
|
|
|
|
Post your branch and request a code review
|
|
|
------------------------------------------
|
|
|
|
|
|
Once you are done with your edits, you should post your branch on Launchpad so
|
|
|
that other IPython developers can review the changes and help you merge your
|
|
|
changes into the main development branch. To post your branch on Launchpad,
|
|
|
do::
|
|
|
|
|
|
$ cd ipython-mybranch
|
|
|
$ bzr push lp:~yourusername/ipython/ipython-mybranch
|
|
|
|
|
|
Then, go to the `IPython Launchpad site <www.launchpad.net/ipython>`_, and you
|
|
|
should see your branch under the "Code" tab. If you click on your branch, you
|
|
|
can provide a short description of the branch as well as mark its status. Most
|
|
|
importantly, you should click the link that reads "Propose for merging into
|
|
|
another branch". What does this do?
|
|
|
|
|
|
This let's the other IPython developers know that your branch is ready to be
|
|
|
reviewed and merged into the main development branch. During this review
|
|
|
process, other developers will give you feedback and help you get your code
|
|
|
ready to be merged. What types of things will we be looking for:
|
|
|
|
|
|
* All code is documented.
|
|
|
* All code has tests.
|
|
|
* The entire IPython test suite passes.
|
|
|
|
|
|
Once your changes have been reviewed and approved, someone will merge them
|
|
|
into the main development branch.
|
|
|
|
|
|
Documentation
|
|
|
=============
|
|
|
|
|
|
Standalone documentation
|
|
|
------------------------
|
|
|
|
|
|
All standalone documentation should be written in plain text (``.txt``) files
|
|
|
using reStructuredText [reStructuredText]_ for markup and formatting. All such
|
|
|
documentation should be placed in directory :file:`docs/source` of the IPython
|
|
|
source tree. The documentation in this location will serve as the main source
|
|
|
for IPython documentation and all existing documentation should be converted
|
|
|
to this format.
|
|
|
|
|
|
To build the final documentation, we use Sphinx [Sphinx]_. Once you have
|
|
|
Sphinx installed, you can build the html docs yourself by doing::
|
|
|
|
|
|
$ cd ipython-mybranch/docs
|
|
|
$ make html
|
|
|
|
|
|
Docstring format
|
|
|
----------------
|
|
|
|
|
|
Good docstrings are very important. All new code should have docstrings that
|
|
|
are formatted using reStructuredText for markup and formatting, since it is
|
|
|
understood by a wide variety of tools. Details about using reStructuredText
|
|
|
for docstrings can be found `here
|
|
|
<http://epydoc.sourceforge.net/manual-othermarkup.html>`_.
|
|
|
|
|
|
Additional PEPs of interest regarding documentation of code:
|
|
|
|
|
|
* `Docstring Conventions <http://www.python.org/peps/pep-0257.html>`_
|
|
|
* `Docstring Processing System Framework <http://www.python.org/peps/pep-0256.html>`_
|
|
|
* `Docutils Design Specification <http://www.python.org/peps/pep-0258.html>`_
|
|
|
|
|
|
|
|
|
Coding conventions
|
|
|
==================
|
|
|
|
|
|
General
|
|
|
-------
|
|
|
|
|
|
In general, we'll try to follow the standard Python style conventions as
|
|
|
described here:
|
|
|
|
|
|
* `Style Guide for Python Code <http://www.python.org/peps/pep-0008.html>`_
|
|
|
|
|
|
|
|
|
Other comments:
|
|
|
|
|
|
* In a large file, top level classes and functions should be
|
|
|
separated by 2-3 lines to make it easier to separate them visually.
|
|
|
* Use 4 spaces for indentation.
|
|
|
* Keep the ordering of methods the same in classes that have the same
|
|
|
methods. This is particularly true for classes that implement an interface.
|
|
|
|
|
|
Naming conventions
|
|
|
------------------
|
|
|
|
|
|
In terms of naming conventions, we'll follow the guidelines from the `Style
|
|
|
Guide for Python Code`_.
|
|
|
|
|
|
For all new IPython code (and much existing code is being refactored), we'll
|
|
|
use:
|
|
|
|
|
|
* All ``lowercase`` module names.
|
|
|
|
|
|
* ``CamelCase`` for class names.
|
|
|
|
|
|
* ``lowercase_with_underscores`` for methods, functions, variables and
|
|
|
attributes.
|
|
|
|
|
|
There are, however, some important exceptions to these rules. In some cases,
|
|
|
IPython code will interface with packages (Twisted, Wx, Qt) that use other
|
|
|
conventions. At some level this makes it impossible to adhere to our own
|
|
|
standards at all times. In particular, when subclassing classes that use other
|
|
|
naming conventions, you must follow their naming conventions. To deal with
|
|
|
cases like this, we propose the following policy:
|
|
|
|
|
|
* If you are subclassing a class that uses different conventions, use its
|
|
|
naming conventions throughout your subclass. Thus, if you are creating a
|
|
|
Twisted Protocol class, used Twisted's
|
|
|
``namingSchemeForMethodsAndAttributes.``
|
|
|
|
|
|
* All IPython's official interfaces should use our conventions. In some cases
|
|
|
this will mean that you need to provide shadow names (first implement
|
|
|
``fooBar`` and then ``foo_bar = fooBar``). We want to avoid this at all
|
|
|
costs, but it will probably be necessary at times. But, please use this
|
|
|
sparingly!
|
|
|
|
|
|
Implementation-specific *private* methods will use
|
|
|
``_single_underscore_prefix``. Names with a leading double underscore will
|
|
|
*only* be used in special cases, as they makes subclassing difficult (such
|
|
|
names are not easily seen by child classes).
|
|
|
|
|
|
Occasionally some run-in lowercase names are used, but mostly for very short
|
|
|
names or where we are implementing methods very similar to existing ones in a
|
|
|
base class (like ``runlines()`` where ``runsource()`` and ``runcode()`` had
|
|
|
established precedent).
|
|
|
|
|
|
The old IPython codebase has a big mix of classes and modules prefixed with an
|
|
|
explicit ``IP``. In Python this is mostly unnecessary, redundant and frowned
|
|
|
upon, as namespaces offer cleaner prefixing. The only case where this approach
|
|
|
is justified is for classes which are expected to be imported into external
|
|
|
namespaces and a very generic name (like Shell) is too likely to clash with
|
|
|
something else. We'll need to revisit this issue as we clean up and refactor
|
|
|
the code, but in general we should remove as many unnecessary ``IP``/``ip``
|
|
|
prefixes as possible. However, if a prefix seems absolutely necessary the more
|
|
|
specific ``IPY`` or ``ipy`` are preferred.
|
|
|
|
|
|
.. _devel_testing:
|
|
|
|
|
|
Testing system
|
|
|
==============
|
|
|
|
|
|
It is extremely important that all code contributed to IPython has tests.
|
|
|
Tests should be written as unittests, doctests or as entities that the Nose
|
|
|
[Nose]_ testing package will find. Regardless of how the tests are written, we
|
|
|
will use Nose for discovering and running the tests. Nose will be required to
|
|
|
run the IPython test suite, but will not be required to simply use IPython.
|
|
|
|
|
|
Tests of Twisted using code need to follow two additional guidelines:
|
|
|
|
|
|
1. Twisted using tests should be written by subclassing the :class:`TestCase`
|
|
|
class that comes with :mod:`twisted.trial.unittest`.
|
|
|
|
|
|
2. All :class:`Deferred` instances that are created in the test must be
|
|
|
properly chained and the final one *must* be the return value of the test
|
|
|
method.
|
|
|
|
|
|
When these two things are done, Nose will be able to run the tests and the
|
|
|
twisted reactor will be handled correctly.
|
|
|
|
|
|
Each subpackage in IPython should have its own :file:`tests` directory that
|
|
|
contains all of the tests for that subpackage. This allows each subpackage to
|
|
|
be self-contained. A good convention to follow is to have a file named
|
|
|
:file:`test_foo.py` for each module :file:`foo.py` in the package. This makes
|
|
|
it easy to organize the tests, though like most conventions, it's OK to break
|
|
|
it if logic and common sense dictate otherwise.
|
|
|
|
|
|
If a subpackage has any dependencies beyond the Python standard library, the
|
|
|
tests for that subpackage should be skipped if the dependencies are not
|
|
|
found. This is very important so users don't get tests failing simply because
|
|
|
they don't have dependencies. We ship a set of decorators in the
|
|
|
:mod:`IPython.testing` package to tag tests that may be platform-specific or
|
|
|
otherwise may have restrictions; if the existing ones don't fit your needs, add
|
|
|
a new decorator in that location so other tests can reuse it.
|
|
|
|
|
|
To run the IPython test suite, use the :command:`iptest` command that is
|
|
|
installed with IPython (if you are using IPython in-place, without installing
|
|
|
it, you can find this script in the :file:`scripts` directory)::
|
|
|
|
|
|
$ iptest
|
|
|
|
|
|
This command runs Nose with the proper options and extensions. By default,
|
|
|
:command:`iptest` runs the entire IPython test suite (skipping tests that may
|
|
|
be platform-specific or which depend on tools you may not have). But you can
|
|
|
also use it to run only one specific test file, or a specific test function.
|
|
|
For example, this will run only the :file:`test_magic` file from the test
|
|
|
suite::
|
|
|
|
|
|
$ iptest IPython.tests.test_magic
|
|
|
----------------------------------------------------------------------
|
|
|
Ran 10 tests in 0.348s
|
|
|
|
|
|
OK (SKIP=3)
|
|
|
Deleting object: second_pass
|
|
|
|
|
|
while the ``path:function`` syntax allows you to select a specific function in
|
|
|
that file to run::
|
|
|
|
|
|
$ iptest IPython.tests.test_magic:test_obj_del
|
|
|
----------------------------------------------------------------------
|
|
|
Ran 1 test in 0.204s
|
|
|
|
|
|
OK
|
|
|
|
|
|
Since :command:`iptest` is based on nosetests, you can pass it any regular
|
|
|
nosetests option. For example, you can use ``--pdb`` or ``--pdb-failures`` to
|
|
|
automatically activate the interactive Pdb debugger on errors or failures. See
|
|
|
the nosetests documentation for further details.
|
|
|
|
|
|
.. warning::
|
|
|
|
|
|
Note that right now we have a nasty interaction between ipdoctest and
|
|
|
twisted. Until we figure this out, please use the following instructions to
|
|
|
ensure that at least you run all the tests.
|
|
|
|
|
|
Right now, if you now run::
|
|
|
|
|
|
$ iptest [any options] [any submodules]
|
|
|
|
|
|
it will NOT load ipdoctest but won't cause any Twisted problems.
|
|
|
|
|
|
Once you're happy that you didn't break Twisted, run::
|
|
|
|
|
|
$ iptest --with-ipdoctest [any options] [any submodules]
|
|
|
|
|
|
This MAY give a Twisted AlreadyCalledError exception at the end, but it will
|
|
|
also correctly load up all of the ipython-specific tests and doctests.
|
|
|
|
|
|
The above can be made easier with a trivial shell alias::
|
|
|
|
|
|
$ alias iptest2='iptest --with-ipdoctest'
|
|
|
|
|
|
So that you can run::
|
|
|
|
|
|
$ iptest ...
|
|
|
# Twisted happy
|
|
|
# iptest2 ...
|
|
|
# ignore possible Twisted error, this checks all the rest.
|
|
|
|
|
|
|
|
|
A few tips for writing tests
|
|
|
----------------------------
|
|
|
|
|
|
You can write tests either as normal test files, using all the conventions that
|
|
|
Nose recognizes, or as doctests. Note that *all* IPython functions should have
|
|
|
at least one example that serves as a doctest, whenever technically feasible.
|
|
|
However, example doctests should only be in the main docstring if they are *a
|
|
|
good example*, i.e. if they convey useful information about the function. If
|
|
|
you simply would like to write a test as a doctest, put it in a separate test
|
|
|
file and write a no-op function whose only purpose is its docstring.
|
|
|
|
|
|
Note, however, that in a file named :file:`test_X`, functions whose only test
|
|
|
is their docstring (as a doctest) and which have no test functionality of their
|
|
|
own, should be called *doctest_foo* instead of *test_foo*, otherwise they get
|
|
|
double-counted (the empty function call is counted as a test, which just
|
|
|
inflates tests numbers artificially). This restriction does not apply to
|
|
|
functions in files with other names, due to how Nose discovers tests.
|
|
|
|
|
|
You can use IPython examples in your docstrings. Those can make full use of
|
|
|
IPython functionality (magics, variable substitution, etc), but be careful to
|
|
|
keep them generic enough that they run identically on all Operating Systems.
|
|
|
|
|
|
The prompts in your doctests can be either of the plain Python ``>>>`` variety
|
|
|
or ``In [1]:`` IPython style. Since this is the IPython system, after all, we
|
|
|
encourage you to use IPython prompts throughout, unless you are illustrating a
|
|
|
specific aspect of the normal prompts (such as the ``%doctest_mode`` magic).
|
|
|
|
|
|
If a test isn't safe to run inside the main nose process (e.g. because it loads
|
|
|
a GUI toolkit), consider running it in a subprocess and capturing its output
|
|
|
for evaluation and test decision later. Here is an example of how to do it, by
|
|
|
relying on the builtin ``_ip`` object that contains the public IPython api as
|
|
|
defined in :mod:`IPython.ipapi`::
|
|
|
|
|
|
def test_obj_del():
|
|
|
"""Test that object's __del__ methods are called on exit."""
|
|
|
test_dir = os.path.dirname(__file__)
|
|
|
del_file = os.path.join(test_dir,'obj_del.py')
|
|
|
out = _ip.IP.getoutput('ipython %s' % del_file)
|
|
|
nt.assert_equals(out,'object A deleted')
|
|
|
|
|
|
|
|
|
|
|
|
If a doctest contains input whose output you don't want to verify identically
|
|
|
via doctest (random output, an object id, etc), you can mark a docstring with
|
|
|
``#random``. All of these test will have their code executed but no output
|
|
|
checking will be done::
|
|
|
|
|
|
>>> 1+3
|
|
|
junk goes here... # random
|
|
|
|
|
|
>>> 1+2
|
|
|
again, anything goes #random
|
|
|
if multiline, the random mark is only needed once.
|
|
|
|
|
|
>>> 1+2
|
|
|
You can also put the random marker at the end:
|
|
|
# random
|
|
|
|
|
|
>>> 1+2
|
|
|
# random
|
|
|
.. or at the beginning.
|
|
|
|
|
|
In a case where you want an *entire* docstring to be executed but not verified
|
|
|
(this only serves to check that the code runs without crashing, so it should be
|
|
|
used very sparingly), you can put ``# all-random`` in the docstring.
|
|
|
|
|
|
.. _devel_config:
|
|
|
|
|
|
Release checklist
|
|
|
=================
|
|
|
|
|
|
Most of the release process is automated by the :file:`release` script in the
|
|
|
:file:`tools` directory. This is just a handy reminder for the release manager.
|
|
|
|
|
|
#. Run the release script, which makes the tar.gz, eggs and Win32 .exe
|
|
|
installer. It posts them to the site and registers the release with PyPI.
|
|
|
|
|
|
#. Updating the website with announcements and links to the updated
|
|
|
changes.txt in html form. Remember to put a short note both on the news
|
|
|
page of the site and on Launcphad.
|
|
|
|
|
|
#. Drafting a short release announcement with i) highlights and ii) a link to
|
|
|
the html changes.txt.
|
|
|
|
|
|
#. Make sure that the released version of the docs is live on the site.
|
|
|
|
|
|
#. Celebrate!
|
|
|
|
|
|
Porting to 3.0
|
|
|
==============
|
|
|
|
|
|
There are no definite plans for porting of IPython to python 3. The major
|
|
|
issue is the dependency on twisted framework for the networking/threading
|
|
|
stuff. It is possible that it the traditional IPython interactive console
|
|
|
could be ported more easily since it has no such dependency. Here are a few
|
|
|
things that will need to be considered when doing such a port especially
|
|
|
if we want to have a codebase that works directly on both 2.x and 3.x.
|
|
|
|
|
|
1. The syntax for exceptions changed (PEP 3110). The old
|
|
|
`except exc, var` changed to `except exc as var`. At last
|
|
|
count there was 78 occurences of this usage in the codebase. This
|
|
|
is a particularly problematic issue, because it's not easy to
|
|
|
implement it in a 2.5-compatible way.
|
|
|
|
|
|
Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we
|
|
|
will likely at some point put out a release that requires strictly 2.6 and
|
|
|
abandons 2.5 compatibility. This will then allow us to port the code to using
|
|
|
:func:`print` as a function, `except exc as var` syntax, etc. But as of
|
|
|
version 0.11 at least, we will retain Python 2.5 compatibility.
|
|
|
|
|
|
|
|
|
.. [Bazaar] Bazaar. http://bazaar-vcs.org/
|
|
|
.. [Launchpad] Launchpad. http://www.launchpad.net/ipython
|
|
|
.. [reStructuredText] reStructuredText. http://docutils.sourceforge.net/rst.html
|
|
|
.. [Sphinx] Sphinx. http://sphinx.pocoo.org/
|
|
|
.. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/
|
|
|
|