Add information about our testing system, test writing tips, etc.

docs/source/development/overview.txt

@@ -17,7 +17,29 b' How to contribute to IPython'
 
 IPython development is done using Bazaar [Bazaar]_ and Launchpad [Launchpad]_.
 This makes it easy for people to contribute to the development of IPython.
-Here is a sketch of how to get going.
+There are several ways in which you can join in.
+
+If you have a small change that you want to send to the team, you can edit your
+bazaar checkout of IPython (see below) in-place, and ask bazaar for the
+differences::
+
+    $ cd /path/to/your/copy/of/ipython
+    $ bzr diff > my_fixes.diff
+
+This produces a patch file with your fixes, which we can apply to the source
+tree.  This file should then be attached to a ticket in our `bug tracker
+<https://bugs.launchpad.net/ipython>`_, indicating what it does.
+
+This model of creating small, self-contained patches works very well and there
+are open source projects that do their entire development this way.  However,
+in IPython we have found that for tracking larger changes, making use of
+bazaar's full capabilities in conjunction with Launchpad's code hosting
+services makes for a much better experience.
+
+Making your own branch of IPython allows you to refine your changes over time,
+track the development of the main team, and propose your own full version of
+the code for others to use and review, with a minimum amount of fuss.  The next
+parts of this document will explain how to do this.
 
 Install Bazaar and create a Launchpad account
 ---------------------------------------------
@@ -102,12 +124,16 b' commands::'
     $ bzr merge ../ipython
     $ bzr commit -m "Merging changes from trunk"
 
-Along the way, you should also run the IPython test suite.  You can do this using the :command:`iptest` command::
+Along the way, you should also run the IPython test suite.  You can do this
+using the :command:`iptest` command (which is basically a customized version of
+:command:`nosetests`)::
 
     $ cd
     $ iptest
 
-The :command:`iptest` command will also pick up and run any tests you have written.
+The :command:`iptest` command will also pick up and run any tests you have
+written.  See :ref:`_devel_testing` for further details on the testing system.
+
 
 Post your branch and request a code review
 ------------------------------------------
@@ -151,7 +177,8 b' source tree. The documentation in this location will serve as the main source'
 for IPython documentation and all existing documentation should be converted
 to this format.
 
-To build the final documentation, we use Sphinx [Sphinx]_.  Once you have Sphinx installed, you can build the html docs yourself by doing::
+To build the final documentation, we use Sphinx [Sphinx]_.  Once you have
+Sphinx installed, you can build the html docs yourself by doing::
 
     $ cd ipython-mybranch/docs
     $ make html
@@ -271,35 +298,85 b' twisted reactor will be handled correctly.'
 
 Each subpackage in IPython should have its own :file:`tests` directory that
 contains all of the tests for that subpackage. This allows each subpackage to
-be self-contained. If a subpackage has any dependencies beyond the Python
-standard library, the tests for that subpackage should be skipped if the
-dependencies are not found. This is very important so users don't get tests
-failing simply because they don't have dependencies.
+be self-contained.  A good convention to follow is to have a file named
+:file:`test_foo.py` for each module :file:`foo.py` in the package.  This makes
+it easy to organize the tests, though like most conventions, it's OK to break
+it if logic and common sense dictate otherwise.
+
+If a subpackage has any dependencies beyond the Python standard library, the
+tests for that subpackage should be skipped if the dependencies are not
+found. This is very important so users don't get tests failing simply because
+they don't have dependencies.  We ship a set of decorators in the
+:mod:`IPython.testing` package to tag tests that may be platform-specific or
+otherwise may have restrictions; if the existing ones don't fit your needs, add
+a new decorator in that location so other tests can reuse it.
 
 To run the IPython test suite, use the :command:`iptest` command that is
-installed with IPython::
+installed with IPython (if you are using IPython in-place, without installing
+it, you can find this script in the :file:`scripts` directory)::
 
     $ iptest
 
-This command runs Nose with the proper options and extensions.
-
-A few tips for writing tests:
-
-* You can use IPython examples in your docstrings, including all IPython
-  prompts.  Rather than repeating it all here, see the files
-  :file:`dtexample.py` and :file:`test_ipdoctest.py` in the
-  :mod:`IPython.testing.plugin` module for examples of how you can use plain
-  Python or IPython prompts, and what to do with examples whose output could be
-  partly or completely random.
-
-* Use the decorators shipped in the :mod:`IPython.testing` package to tag tests
-  that may be platform-specific or otherwise may have restrictions.
-
-* If a test isn't safe to run inside the main nose process (e.g. because it
-  loads a GUI toolkit), consider running it in a subprocess and capturing its
-  output for evaluation and test decision later.  Here is an example of how to
-  do it, by relying on the builtin ``_ip`` object that contains the public
-  IPython api as defined in :mod:`IPython.ipapi`::
+This command runs Nose with the proper options and extensions.  By default,
+:command:`iptest` runs the entire IPython test suite (skipping tests that may
+be platform-specific or which depend on tools you may not have).  But you can
+also use it to run only one specific test file, or a specific test function.
+For example, this will run only the :file:`test_magic` file from the test
+suite::
+
+    $ iptest IPython.tests.test_magic
+    ----------------------------------------------------------------------
+    Ran 10 tests in 0.348s
+
+    OK (SKIP=3)
+    Deleting object: second_pass
+
+while the ``path:function`` syntax allows you to select a specific function in
+that file to run::
+
+    $ iptest IPython.tests.test_magic:test_obj_del
+    ----------------------------------------------------------------------
+    Ran 1 test in 0.204s
+
+    OK
+
+Since :command:`iptest` is based on nosetests, you can pass it any regular
+nosetests option.  For example, you can use ``--pdb`` or ``--pdb-failures`` to
+automatically activate the interactive Pdb debugger on errors or failures.  See
+the nosetests documentation for further details.
+
+A few tips for writing tests
+----------------------------
+
+You can write tests either as normal test files, using all the conventions that
+Nose recognizes, or as doctests.  Note that *all* IPython functions should have
+at least one example that serves as a doctest, whenever technically feasible.
+However, example doctests should only be in the main docstring if they are *a
+good example*, i.e. if they convey useful information about the function.  If
+you simply would like to write a test as a doctest, put it in a separate test
+file and write a no-op function whose only purpose is its docstring.
+
+Note, however, that in a file named :file:`test_X`, functions whose only test
+is their docstring (as a doctest) and which have no test functionality of their
+own, should be called *doctest_foo* instead of *test_foo*, otherwise they get
+double-counted (the empty function call is counted as a test, which just
+inflates tests numbers artificially).  This restriction does not apply to
+functions in files with other names, due to how Nose discovers tests.
+
+You can use IPython examples in your docstrings.  Those can make full use of
+IPython functionality (magics, variable substitution, etc), but be careful to
+keep them generic enough that they run identically on all Operating Systems.
+
+The prompts in your doctests can be either of the plain Python ``>>>`` variety
+or ``In [1]:`` IPython style.  Since this is the IPython system, after all, we
+encourage you to use IPython prompts throughout, unless you are illustrating a
+specific aspect of the normal prompts (such as the ``%doctest_mode`` magic).
+
+If a test isn't safe to run inside the main nose process (e.g. because it loads
+a GUI toolkit), consider running it in a subprocess and capturing its output
+for evaluation and test decision later.  Here is an example of how to do it, by
+relying on the builtin ``_ip`` object that contains the public IPython api as
+defined in :mod:`IPython.ipapi`::
 
    def test_obj_del():
        """Test that object's __del__ methods are called on exit."""
@@ -309,13 +386,31 b' A few tips for writing tests:'
        nt.assert_equals(out,'object A deleted')
 
 
-* In a file named ``test_X``, functions whose only test is their docstring (as
-  a doctest) and which have no test functionality of their own, should be
-  called *doctest_foo* instead of *test_foo*, otherwise they get double-counted
-  (the empty function call is counted as a test, which just inflates tests
-  numbers artificially).  This restriction does not apply to functions in files
-  with other names, due to how Nose discovers tests.
 
+If a doctest contains input whose output you don't want to verify identically
+via doctest (random output, an object id, etc), you can mark a docstring with
+``#random``.  All of these test will have their code executed but no output
+checking will be done::
+
+       >>> 1+3
+       junk goes here...  # random
+
+       >>> 1+2
+       again,  anything goes #random
+       if multiline, the random mark is only needed once.
+
+       >>> 1+2
+       You can also put the random marker at the end:
+       # random
+
+       >>> 1+2
+       # random
+       .. or at the beginning.
+
+In a case where you want an *entire* docstring to be executed but not verified
+(this only serves to check that the code runs without crashing, so it should be
+used very sparingly), you can put ``# all-random`` in the docstring.
+       
 .. _devel_config:
 
 Release checklist
@@ -340,6 +435,7 b' Most of the release process is automated by the :file:`release` script in the'
 
 Porting to 3.0
 ==============
+
 There are no definite plans for porting of IPython to python 3. The major 
 issue is the dependency on twisted framework for the networking/threading 
 stuff. It is possible that it the traditional IPython interactive console