|
|
.. _testing:
|
|
|
|
|
|
==========================================
|
|
|
Testing IPython for users and developers
|
|
|
==========================================
|
|
|
|
|
|
Overview
|
|
|
========
|
|
|
|
|
|
It is extremely important that all code contributed to IPython has tests.
|
|
|
Tests should be written as unittests, doctests or other entities that the
|
|
|
IPython test system can detect. See below for more details on this.
|
|
|
|
|
|
Each subpackage in IPython should have its own :file:`tests` directory that
|
|
|
contains all of the tests for that subpackage. All of the files in the
|
|
|
:file:`tests` directory should have the word "tests" in them to enable
|
|
|
the testing framework to find them.
|
|
|
|
|
|
In docstrings, examples (either using IPython prompts like ``In [1]:`` or
|
|
|
'classic' python ``>>>`` ones) can and should be included. The testing system
|
|
|
will detect them as doctests and will run them; it offers control to skip parts
|
|
|
or all of a specific doctest if the example is meant to be informative but
|
|
|
shows non-reproducible information (like filesystem data).
|
|
|
|
|
|
If a subpackage has any dependencies beyond the Python standard library, the
|
|
|
tests for that subpackage should be skipped if the dependencies are not found.
|
|
|
This is very important so users don't get tests failing simply because they
|
|
|
don't have dependencies.
|
|
|
|
|
|
The testing system we use is a hybrid of nose_ and Twisted's trial_ test runner.
|
|
|
We use both because nose detects more things than Twisted and allows for more
|
|
|
flexible (and lighter-weight) ways of writing tests; in particular we've
|
|
|
developed a nose plugin that allows us to paste verbatim IPython sessions and
|
|
|
test them as doctests, which is extremely important for us. But the parts of
|
|
|
IPython that depend on Twisted must be tested using trial, because only trial
|
|
|
manages the Twisted reactor correctly.
|
|
|
|
|
|
.. _nose: http://code.google.com/p/python-nose
|
|
|
.. _trial: http://twistedmatrix.com/trac/wiki/TwistedTrial
|
|
|
|
|
|
|
|
|
For the impatient: running the tests
|
|
|
====================================
|
|
|
|
|
|
You can run IPython from the source download directory without even installing
|
|
|
it system-wide or having configure anything, by typing at the terminal:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
python ipython.py
|
|
|
|
|
|
and similarly, you can execute the built-in test suite with:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
python iptest.py
|
|
|
|
|
|
|
|
|
This script manages intelligently both nose and trial, choosing the correct
|
|
|
test system for each of IPython's components.
|
|
|
|
|
|
Once you have either installed it or at least configured your system to be
|
|
|
able to import IPython, you can run the tests with:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
python -c "import IPython; IPython.test()"
|
|
|
|
|
|
This should work as long as IPython can be imported, even if you haven't fully
|
|
|
installed the user-facing scripts yet (common in a development environment).
|
|
|
Once you have installed IPython, you will have available system-wide a script
|
|
|
called :file:`iptest` that does the exact same as the :file:`iptest.py` script
|
|
|
in the source directory, so you can then test simply with:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
iptest [args]
|
|
|
|
|
|
|
|
|
Regardless of how you run things, you should eventually see something like:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
**********************************************************************
|
|
|
Test suite completed for system with the following information:
|
|
|
IPython version: 0.11.bzr.r1340
|
|
|
BZR revision : 1340
|
|
|
Platform info : os.name -> posix, sys.platform -> linux2
|
|
|
: Linux-2.6.31-17-generic-i686-with-Ubuntu-9.10-karmic
|
|
|
Python info : 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
|
|
|
[GCC 4.4.1]
|
|
|
|
|
|
Running from an installed IPython: True
|
|
|
|
|
|
Tools and libraries available at test time:
|
|
|
curses foolscap gobject gtk pexpect twisted wx wx.aui zope.interface
|
|
|
|
|
|
Tools and libraries NOT available at test time:
|
|
|
objc
|
|
|
|
|
|
Ran 11 test groups in 36.244s
|
|
|
|
|
|
Status:
|
|
|
OK
|
|
|
|
|
|
If not, there will be a message indicating which test group failed and how to
|
|
|
rerun that group individually. For example, this tests the
|
|
|
:mod:`IPython.utils` subpackage, the :option:`-v` option shows progress
|
|
|
indicators:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
$ python iptest.py -v IPython.utils
|
|
|
..........................SS..SSS............................S.S...
|
|
|
.........................................................
|
|
|
----------------------------------------------------------------------
|
|
|
Ran 125 tests in 0.119s
|
|
|
|
|
|
OK (SKIP=7)
|
|
|
|
|
|
|
|
|
Because the IPython test machinery is based on nose, you can use all nose
|
|
|
options and syntax, typing ``iptest -h`` shows all available options. For
|
|
|
example, this lets you run the specific test :func:`test_rehashx` inside the
|
|
|
:mod:`test_magic` module:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
$ python iptest.py -vv IPython.core.tests.test_magic:test_rehashx
|
|
|
IPython.core.tests.test_magic.test_rehashx(True,) ... ok
|
|
|
IPython.core.tests.test_magic.test_rehashx(True,) ... ok
|
|
|
|
|
|
----------------------------------------------------------------------
|
|
|
Ran 2 tests in 0.100s
|
|
|
|
|
|
OK
|
|
|
|
|
|
When developing, the :option:`--pdb` and :option:`--pdb-failures` of nose are
|
|
|
particularly useful, these drop you into an interactive pdb session at the
|
|
|
point of the error or failure respectively.
|
|
|
|
|
|
To run Twisted-using tests, use the :command:`trial` command on a per file or
|
|
|
package basis:
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
|
trial IPython.kernel
|
|
|
|
|
|
|
|
|
For developers: writing tests
|
|
|
=============================
|
|
|
|
|
|
By now IPython has a reasonable test suite, so the best way to see what's
|
|
|
available is to look at the :file:`tests` directory in most subpackages. But
|
|
|
here are a few pointers to make the process easier.
|
|
|
|
|
|
|
|
|
Main tools: :mod:`IPython.testing`
|
|
|
----------------------------------
|
|
|
|
|
|
The :mod:`IPython.testing` package is where all of the machinery to test
|
|
|
IPython (rather than the tests for its various parts) lives. In particular,
|
|
|
the :mod:`iptest` module in there has all the smarts to control the test
|
|
|
process. In there, the :func:`make_exclude` function is used to build a
|
|
|
blacklist of exclusions, these are modules that do not get even imported for
|
|
|
tests. This is important so that things that would fail to even import because
|
|
|
of missing dependencies don't give errors to end users, as we stated above.
|
|
|
|
|
|
The :mod:`decorators` module contains a lot of useful decorators, especially
|
|
|
useful to mark individual tests that should be skipped under certain conditions
|
|
|
(rather than blacklisting the package altogether because of a missing major
|
|
|
dependency).
|
|
|
|
|
|
Our nose plugin for doctests
|
|
|
----------------------------
|
|
|
|
|
|
The :mod:`plugin` subpackage in testing contains a nose plugin called
|
|
|
:mod:`ipdoctest` that teaches nose about IPython syntax, so you can write
|
|
|
doctests with IPython prompts. You can also mark doctest output with ``#
|
|
|
random`` for the output corresponding to a single input to be ignored (stronger
|
|
|
than using ellipsis and useful to keep it as an example). If you want the
|
|
|
entire docstring to be executed but none of the output from any input to be
|
|
|
checked, you can use the ``# all-random`` marker. The
|
|
|
:mod:`IPython.testing.plugin.dtexample` module contains examples of how to use
|
|
|
these; for reference here is how to use ``# random``::
|
|
|
|
|
|
def ranfunc():
|
|
|
"""A function with some random output.
|
|
|
|
|
|
Normal examples are verified as usual:
|
|
|
>>> 1+3
|
|
|
4
|
|
|
|
|
|
But if you put '# random' in the output, it is ignored:
|
|
|
>>> 1+3
|
|
|
junk goes here... # random
|
|
|
|
|
|
>>> 1+2
|
|
|
again, anything goes #random
|
|
|
if multiline, the random mark is only needed once.
|
|
|
|
|
|
>>> 1+2
|
|
|
You can also put the random marker at the end:
|
|
|
# random
|
|
|
|
|
|
>>> 1+2
|
|
|
# random
|
|
|
.. or at the beginning.
|
|
|
|
|
|
More correct input is properly verified:
|
|
|
>>> ranfunc()
|
|
|
'ranfunc'
|
|
|
"""
|
|
|
return 'ranfunc'
|
|
|
|
|
|
and an example of ``# all-random``::
|
|
|
|
|
|
def random_all():
|
|
|
"""A function where we ignore the output of ALL examples.
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
# all-random
|
|
|
|
|
|
This mark tells the testing machinery that all subsequent examples
|
|
|
should be treated as random (ignoring their output). They are still
|
|
|
executed, so if a they raise an error, it will be detected as such,
|
|
|
but their output is completely ignored.
|
|
|
|
|
|
>>> 1+3
|
|
|
junk goes here...
|
|
|
|
|
|
>>> 1+3
|
|
|
klasdfj;
|
|
|
|
|
|
In [8]: print 'hello'
|
|
|
world # random
|
|
|
|
|
|
In [9]: iprand()
|
|
|
Out[9]: 'iprand'
|
|
|
"""
|
|
|
return 'iprand'
|
|
|
|
|
|
|
|
|
When writing docstrings, you can use the ``@skip_doctest`` decorator to
|
|
|
indicate that a docstring should *not* be treated as a doctest at all. The
|
|
|
difference between ``# all-random`` and ``@skip_doctest`` is that the former
|
|
|
executes the example but ignores output, while the latter doesn't execute any
|
|
|
code. ``@skip_doctest`` should be used for docstrings whose examples are
|
|
|
purely informational.
|
|
|
|
|
|
If a given docstring fails under certain conditions but otherwise is a good
|
|
|
doctest, you can use code like the following, that relies on the 'null'
|
|
|
decorator to leave the docstring intact where it works as a test::
|
|
|
|
|
|
# The docstring for full_path doctests differently on win32 (different path
|
|
|
# separator) so just skip the doctest there, and use a null decorator
|
|
|
# elsewhere:
|
|
|
|
|
|
doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco
|
|
|
|
|
|
@doctest_deco
|
|
|
def full_path(startPath,files):
|
|
|
"""Make full paths for all the listed files, based on startPath..."""
|
|
|
|
|
|
# function body follows...
|
|
|
|
|
|
With our nose plugin that understands IPython syntax, an extremely effective
|
|
|
way to write tests is to simply copy and paste an interactive session into a
|
|
|
docstring. You can writing this type of test, where your docstring is meant
|
|
|
*only* as a test, by prefixing the function name with ``doctest_`` and leaving
|
|
|
its body *absolutely empty* other than the docstring. In
|
|
|
:mod:`IPython.core.tests.test_magic` you can find several examples of this, but
|
|
|
for completeness sake, your code should look like this (a simple case)::
|
|
|
|
|
|
def doctest_time():
|
|
|
"""
|
|
|
In [10]: %time None
|
|
|
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
|
|
|
Wall time: 0.00 s
|
|
|
"""
|
|
|
|
|
|
This function is only analyzed for its docstring but it is not considered a
|
|
|
separate test, which is why its body should be empty.
|
|
|
|
|
|
|
|
|
Parametric tests done right
|
|
|
---------------------------
|
|
|
|
|
|
If you need to run multiple tests inside the same standalone function or method
|
|
|
of a :class:`unittest.TestCase` subclass, IPython provides the ``parametric``
|
|
|
decorator for this purpose. This is superior to how test generators work in
|
|
|
nose, because IPython's keeps intact your stack, which makes debugging vastly
|
|
|
easier. For example, these are some parametric tests both in class form and as
|
|
|
a standalone function (choose in each situation the style that best fits the
|
|
|
problem at hand, since both work)::
|
|
|
|
|
|
from IPython.testing import decorators as dec
|
|
|
|
|
|
def is_smaller(i,j):
|
|
|
assert i<j,"%s !< %s" % (i,j)
|
|
|
|
|
|
class Tester(ParametricTestCase):
|
|
|
|
|
|
def test_parametric(self):
|
|
|
yield is_smaller(3, 4)
|
|
|
x, y = 1, 2
|
|
|
yield is_smaller(x, y)
|
|
|
|
|
|
@dec.parametric
|
|
|
def test_par_standalone():
|
|
|
yield is_smaller(3, 4)
|
|
|
x, y = 1, 2
|
|
|
yield is_smaller(x, y)
|
|
|
|
|
|
|
|
|
Writing tests for Twisted-using code
|
|
|
------------------------------------
|
|
|
|
|
|
Tests of Twisted [Twisted]_ using code should be written by subclassing the
|
|
|
``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, all
|
|
|
:class:`Deferred` instances that are created in the test must be properly
|
|
|
chained and the final one *must* be the return value of the test method.
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
The best place to see how to use the testing tools, are the tests for these
|
|
|
tools themselves, which live in :mod:`IPython.testing.tests`.
|
|
|
|
|
|
|
|
|
Design requirements
|
|
|
===================
|
|
|
|
|
|
This section is a set of notes on the key points of the IPython testing needs,
|
|
|
that were used when writing the system and should be kept for reference as it
|
|
|
eveolves.
|
|
|
|
|
|
Testing IPython in full requires modifications to the default behavior of nose
|
|
|
and doctest, because the IPython prompt is not recognized to determine Python
|
|
|
input, and because IPython admits user input that is not valid Python (things
|
|
|
like ``%magics`` and ``!system commands``.
|
|
|
|
|
|
We basically need to be able to test the following types of code:
|
|
|
|
|
|
1. Pure Python files containing normal tests. These are not a problem, since
|
|
|
Nose will pick them up as long as they conform to the (flexible) conventions
|
|
|
used by nose to recognize tests.
|
|
|
|
|
|
2. Python files containing doctests. Here, we have two possibilities:
|
|
|
- The prompts are the usual ``>>>`` and the input is pure Python.
|
|
|
- The prompts are of the form ``In [1]:`` and the input can contain extended
|
|
|
IPython expressions.
|
|
|
|
|
|
In the first case, Nose will recognize the doctests as long as it is called
|
|
|
with the ``--with-doctest`` flag. But the second case will likely require
|
|
|
modifications or the writing of a new doctest plugin for Nose that is
|
|
|
IPython-aware.
|
|
|
|
|
|
3. ReStructuredText files that contain code blocks. For this type of file, we
|
|
|
have three distinct possibilities for the code blocks:
|
|
|
- They use ``>>>`` prompts.
|
|
|
- They use ``In [1]:`` prompts.
|
|
|
- They are standalone blocks of pure Python code without any prompts.
|
|
|
|
|
|
The first two cases are similar to the situation #2 above, except that in
|
|
|
this case the doctests must be extracted from input code blocks using
|
|
|
docutils instead of from the Python docstrings.
|
|
|
|
|
|
In the third case, we must have a convention for distinguishing code blocks
|
|
|
that are meant for execution from others that may be snippets of shell code
|
|
|
or other examples not meant to be run. One possibility is to assume that
|
|
|
all indented code blocks are meant for execution, but to have a special
|
|
|
docutils directive for input that should not be executed.
|
|
|
|
|
|
For those code blocks that we will execute, the convention used will simply
|
|
|
be that they get called and are considered successful if they run to
|
|
|
completion without raising errors. This is similar to what Nose does for
|
|
|
standalone test functions, and by putting asserts or other forms of
|
|
|
exception-raising statements it becomes possible to have literate examples
|
|
|
that double as lightweight tests.
|
|
|
|
|
|
4. Extension modules with doctests in function and method docstrings.
|
|
|
Currently Nose simply can't find these docstrings correctly, because the
|
|
|
underlying doctest DocTestFinder object fails there. Similarly to #2 above,
|
|
|
the docstrings could have either pure python or IPython prompts.
|
|
|
|
|
|
Of these, only 3-c (reST with standalone code blocks) is not implemented at
|
|
|
this point.
|
|
|
|
