testing.txt
393 lines
| 14.1 KiB
| text/plain
|
TextLexer
Brian Granger
|
r2276 | .. _testing: | ||
Fernando Perez
|
r2467 | ========================================== | ||
Testing IPython for users and developers | ||||
========================================== | ||||
Brian Granger
|
r2276 | |||
Overview | ||||
======== | ||||
It is extremely important that all code contributed to IPython has tests. | ||||
Tests should be written as unittests, doctests or other entities that the | ||||
IPython test system can detect. See below for more details on this. | ||||
Each subpackage in IPython should have its own :file:`tests` directory that | ||||
contains all of the tests for that subpackage. All of the files in the | ||||
:file:`tests` directory should have the word "tests" in them to enable | ||||
Fernando Perez
|
r2467 | the testing framework to find them. | ||
In docstrings, examples (either using IPython prompts like ``In [1]:`` or | ||||
'classic' python ``>>>`` ones) can and should be included. The testing system | ||||
will detect them as doctests and will run them; it offers control to skip parts | ||||
or all of a specific doctest if the example is meant to be informative but | ||||
shows non-reproducible information (like filesystem data). | ||||
Brian Granger
|
r2276 | |||
If a subpackage has any dependencies beyond the Python standard library, the | ||||
tests for that subpackage should be skipped if the dependencies are not found. | ||||
This is very important so users don't get tests failing simply because they | ||||
Fernando Perez
|
r2467 | don't have dependencies. | ||
The testing system we use is a hybrid of nose_ and Twisted's trial_ test runner. | ||||
We use both because nose detects more things than Twisted and allows for more | ||||
flexible (and lighter-weight) ways of writing tests; in particular we've | ||||
developed a nose plugin that allows us to paste verbatim IPython sessions and | ||||
test them as doctests, which is extremely important for us. But the parts of | ||||
IPython that depend on Twisted must be tested using trial, because only trial | ||||
manages the Twisted reactor correctly. | ||||
.. _nose: http://code.google.com/p/python-nose | ||||
.. _trial: http://twistedmatrix.com/trac/wiki/TwistedTrial | ||||
For the impatient: running the tests | ||||
==================================== | ||||
Fernando Perez
|
r2481 | You can run IPython from the source download directory without even installing | ||
it system-wide or having configure anything, by typing at the terminal: | ||||
.. code-block:: bash | ||||
python ipython.py | ||||
Fernando Perez
|
r3204 | In order to run the test suite, you must at least be able to import IPython, | ||
even if you haven't fully installed the user-facing scripts yet (common in a | ||||
development environment). You can then run the tests with: | ||||
Fernando Perez
|
r2481 | |||
Fernando Perez
|
r3204 | .. code-block:: bash | ||
Fernando Perez
|
r2481 | |||
Fernando Perez
|
r3204 | python -c "import IPython; IPython.test()" | ||
Fernando Perez
|
r2490 | |||
Fernando Perez
|
r3204 | Once you have installed IPython either via a full install or using: | ||
Fernando Perez
|
r2467 | |||
.. code-block:: bash | ||||
Fernando Perez
|
r3204 | python setup.py develop | ||
Fernando Perez
|
r2467 | |||
Fernando Perez
|
r3204 | you will have available a system-wide script called :file:`iptest` that runs | ||
the full test suite. You can then run the suite with: | ||||
Fernando Perez
|
r2490 | |||
.. code-block:: bash | ||||
iptest [args] | ||||
Fernando Perez
|
r2481 | |||
Regardless of how you run things, you should eventually see something like: | ||||
Fernando Perez
|
r2467 | |||
.. code-block:: bash | ||||
Fernando Perez
|
r2490 | ********************************************************************** | ||
Fernando Perez
|
r2496 | Test suite completed for system with the following information: | ||
Fernando Perez
|
r3204 | {'commit_hash': '144fdae', | ||
'commit_source': 'repository', | ||||
'ipython_path': '/home/fperez/usr/lib/python2.6/site-packages/IPython', | ||||
'ipython_version': '0.11.dev', | ||||
'os_name': 'posix', | ||||
'platform': 'Linux-2.6.35-22-generic-i686-with-Ubuntu-10.10-maverick', | ||||
'sys_executable': '/usr/bin/python', | ||||
'sys_platform': 'linux2', | ||||
'sys_version': '2.6.6 (r266:84292, Sep 15 2010, 15:52:39) \n[GCC 4.4.5]'} | ||||
Fernando Perez
|
r2496 | |||
Tools and libraries available at test time: | ||||
curses foolscap gobject gtk pexpect twisted wx wx.aui zope.interface | ||||
Fernando Perez
|
r3204 | Ran 9 test groups in 67.213s | ||
Fernando Perez
|
r2496 | |||
Status: | ||||
OK | ||||
Fernando Perez
|
r2467 | |||
Fernando Perez
|
r3204 | |||
Fernando Perez
|
r2490 | If not, there will be a message indicating which test group failed and how to | ||
rerun that group individually. For example, this tests the | ||||
:mod:`IPython.utils` subpackage, the :option:`-v` option shows progress | ||||
indicators: | ||||
Fernando Perez
|
r2467 | |||
.. code-block:: bash | ||||
Brian Granger
|
r2276 | |||
Fernando Perez
|
r3204 | $ iptest -v IPython.utils | ||
Fernando Perez
|
r2490 | ..........................SS..SSS............................S.S... | ||
......................................................... | ||||
---------------------------------------------------------------------- | ||||
Ran 125 tests in 0.119s | ||||
OK (SKIP=7) | ||||
Brian Granger
|
r2276 | |||
Fernando Perez
|
r2490 | Because the IPython test machinery is based on nose, you can use all nose | ||
options and syntax, typing ``iptest -h`` shows all available options. For | ||||
example, this lets you run the specific test :func:`test_rehashx` inside the | ||||
Fernando Perez
|
r2467 | :mod:`test_magic` module: | ||
Brian Granger
|
r2276 | |||
.. code-block:: bash | ||||
Fernando Perez
|
r3204 | $ iptest -vv IPython.core.tests.test_magic:test_rehashx | ||
Fernando Perez
|
r2490 | IPython.core.tests.test_magic.test_rehashx(True,) ... ok | ||
IPython.core.tests.test_magic.test_rehashx(True,) ... ok | ||||
Brian Granger
|
r2276 | |||
Fernando Perez
|
r2490 | ---------------------------------------------------------------------- | ||
Ran 2 tests in 0.100s | ||||
Fernando Perez
|
r2467 | |||
Fernando Perez
|
r2490 | OK | ||
Fernando Perez
|
r2467 | |||
When developing, the :option:`--pdb` and :option:`--pdb-failures` of nose are | ||||
particularly useful, these drop you into an interactive pdb session at the | ||||
point of the error or failure respectively. | ||||
To run Twisted-using tests, use the :command:`trial` command on a per file or | ||||
package basis: | ||||
Brian Granger
|
r2276 | |||
.. code-block:: bash | ||||
trial IPython.kernel | ||||
Fernando Perez
|
r3204 | .. note:: | ||
The system information summary printed above is accessible from the top | ||||
level package. If you encounter a problem with IPython, it's useful to | ||||
include this information when reporting on the mailing list; use:: | ||||
from IPython import sys_info | ||||
print sys_info() | ||||
and include the resulting information in your query. | ||||
Brian Granger
|
r2276 | |||
Fernando Perez
|
r2467 | For developers: writing tests | ||
============================= | ||||
By now IPython has a reasonable test suite, so the best way to see what's | ||||
available is to look at the :file:`tests` directory in most subpackages. But | ||||
here are a few pointers to make the process easier. | ||||
Main tools: :mod:`IPython.testing` | ||||
---------------------------------- | ||||
The :mod:`IPython.testing` package is where all of the machinery to test | ||||
IPython (rather than the tests for its various parts) lives. In particular, | ||||
the :mod:`iptest` module in there has all the smarts to control the test | ||||
process. In there, the :func:`make_exclude` function is used to build a | ||||
blacklist of exclusions, these are modules that do not get even imported for | ||||
tests. This is important so that things that would fail to even import because | ||||
of missing dependencies don't give errors to end users, as we stated above. | ||||
The :mod:`decorators` module contains a lot of useful decorators, especially | ||||
useful to mark individual tests that should be skipped under certain conditions | ||||
(rather than blacklisting the package altogether because of a missing major | ||||
dependency). | ||||
Our nose plugin for doctests | ||||
---------------------------- | ||||
The :mod:`plugin` subpackage in testing contains a nose plugin called | ||||
:mod:`ipdoctest` that teaches nose about IPython syntax, so you can write | ||||
doctests with IPython prompts. You can also mark doctest output with ``# | ||||
random`` for the output corresponding to a single input to be ignored (stronger | ||||
than using ellipsis and useful to keep it as an example). If you want the | ||||
entire docstring to be executed but none of the output from any input to be | ||||
checked, you can use the ``# all-random`` marker. The | ||||
:mod:`IPython.testing.plugin.dtexample` module contains examples of how to use | ||||
these; for reference here is how to use ``# random``:: | ||||
def ranfunc(): | ||||
"""A function with some random output. | ||||
Normal examples are verified as usual: | ||||
>>> 1+3 | ||||
4 | ||||
But if you put '# random' in the output, it is ignored: | ||||
>>> 1+3 | ||||
junk goes here... # random | ||||
>>> 1+2 | ||||
again, anything goes #random | ||||
if multiline, the random mark is only needed once. | ||||
>>> 1+2 | ||||
You can also put the random marker at the end: | ||||
# random | ||||
>>> 1+2 | ||||
# random | ||||
.. or at the beginning. | ||||
More correct input is properly verified: | ||||
>>> ranfunc() | ||||
'ranfunc' | ||||
""" | ||||
return 'ranfunc' | ||||
and an example of ``# all-random``:: | ||||
def random_all(): | ||||
"""A function where we ignore the output of ALL examples. | ||||
Examples: | ||||
# all-random | ||||
This mark tells the testing machinery that all subsequent examples | ||||
should be treated as random (ignoring their output). They are still | ||||
executed, so if a they raise an error, it will be detected as such, | ||||
but their output is completely ignored. | ||||
>>> 1+3 | ||||
junk goes here... | ||||
>>> 1+3 | ||||
klasdfj; | ||||
In [8]: print 'hello' | ||||
world # random | ||||
In [9]: iprand() | ||||
Out[9]: 'iprand' | ||||
""" | ||||
return 'iprand' | ||||
When writing docstrings, you can use the ``@skip_doctest`` decorator to | ||||
indicate that a docstring should *not* be treated as a doctest at all. The | ||||
Brian Granger
|
r2499 | difference between ``# all-random`` and ``@skip_doctest`` is that the former | ||
Fernando Perez
|
r2467 | executes the example but ignores output, while the latter doesn't execute any | ||
code. ``@skip_doctest`` should be used for docstrings whose examples are | ||||
purely informational. | ||||
If a given docstring fails under certain conditions but otherwise is a good | ||||
doctest, you can use code like the following, that relies on the 'null' | ||||
decorator to leave the docstring intact where it works as a test:: | ||||
# The docstring for full_path doctests differently on win32 (different path | ||||
# separator) so just skip the doctest there, and use a null decorator | ||||
# elsewhere: | ||||
doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco | ||||
@doctest_deco | ||||
def full_path(startPath,files): | ||||
"""Make full paths for all the listed files, based on startPath...""" | ||||
# function body follows... | ||||
With our nose plugin that understands IPython syntax, an extremely effective | ||||
way to write tests is to simply copy and paste an interactive session into a | ||||
docstring. You can writing this type of test, where your docstring is meant | ||||
*only* as a test, by prefixing the function name with ``doctest_`` and leaving | ||||
its body *absolutely empty* other than the docstring. In | ||||
:mod:`IPython.core.tests.test_magic` you can find several examples of this, but | ||||
for completeness sake, your code should look like this (a simple case):: | ||||
def doctest_time(): | ||||
""" | ||||
In [10]: %time None | ||||
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s | ||||
Wall time: 0.00 s | ||||
""" | ||||
This function is only analyzed for its docstring but it is not considered a | ||||
separate test, which is why its body should be empty. | ||||
Parametric tests done right | ||||
--------------------------- | ||||
If you need to run multiple tests inside the same standalone function or method | ||||
of a :class:`unittest.TestCase` subclass, IPython provides the ``parametric`` | ||||
decorator for this purpose. This is superior to how test generators work in | ||||
nose, because IPython's keeps intact your stack, which makes debugging vastly | ||||
easier. For example, these are some parametric tests both in class form and as | ||||
a standalone function (choose in each situation the style that best fits the | ||||
problem at hand, since both work):: | ||||
from IPython.testing import decorators as dec | ||||
def is_smaller(i,j): | ||||
assert i<j,"%s !< %s" % (i,j) | ||||
class Tester(ParametricTestCase): | ||||
def test_parametric(self): | ||||
yield is_smaller(3, 4) | ||||
x, y = 1, 2 | ||||
yield is_smaller(x, y) | ||||
@dec.parametric | ||||
def test_par_standalone(): | ||||
yield is_smaller(3, 4) | ||||
x, y = 1, 2 | ||||
yield is_smaller(x, y) | ||||
Writing tests for Twisted-using code | ||||
------------------------------------ | ||||
Tests of Twisted [Twisted]_ using code should be written by subclassing the | ||||
``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, all | ||||
:class:`Deferred` instances that are created in the test must be properly | ||||
chained and the final one *must* be the return value of the test method. | ||||
.. note:: | ||||
The best place to see how to use the testing tools, are the tests for these | ||||
tools themselves, which live in :mod:`IPython.testing.tests`. | ||||
Design requirements | ||||
=================== | ||||
This section is a set of notes on the key points of the IPython testing needs, | ||||
that were used when writing the system and should be kept for reference as it | ||||
eveolves. | ||||
Testing IPython in full requires modifications to the default behavior of nose | ||||
and doctest, because the IPython prompt is not recognized to determine Python | ||||
input, and because IPython admits user input that is not valid Python (things | ||||
like ``%magics`` and ``!system commands``. | ||||
We basically need to be able to test the following types of code: | ||||
1. Pure Python files containing normal tests. These are not a problem, since | ||||
Nose will pick them up as long as they conform to the (flexible) conventions | ||||
used by nose to recognize tests. | ||||
Thomas Kluyver
|
r4198 | 2. Python files containing doctests. Here, we have two possibilities: | ||
Fernando Perez
|
r2467 | - The prompts are the usual ``>>>`` and the input is pure Python. | ||
- The prompts are of the form ``In [1]:`` and the input can contain extended | ||||
IPython expressions. | ||||
In the first case, Nose will recognize the doctests as long as it is called | ||||
with the ``--with-doctest`` flag. But the second case will likely require | ||||
modifications or the writing of a new doctest plugin for Nose that is | ||||
IPython-aware. | ||||
3. ReStructuredText files that contain code blocks. For this type of file, we | ||||
have three distinct possibilities for the code blocks: | ||||
Thomas Kluyver
|
r4198 | |||
Fernando Perez
|
r2467 | - They use ``>>>`` prompts. | ||
- They use ``In [1]:`` prompts. | ||||
- They are standalone blocks of pure Python code without any prompts. | ||||
The first two cases are similar to the situation #2 above, except that in | ||||
this case the doctests must be extracted from input code blocks using | ||||
docutils instead of from the Python docstrings. | ||||
In the third case, we must have a convention for distinguishing code blocks | ||||
that are meant for execution from others that may be snippets of shell code | ||||
or other examples not meant to be run. One possibility is to assume that | ||||
all indented code blocks are meant for execution, but to have a special | ||||
docutils directive for input that should not be executed. | ||||
For those code blocks that we will execute, the convention used will simply | ||||
be that they get called and are considered successful if they run to | ||||
completion without raising errors. This is similar to what Nose does for | ||||
standalone test functions, and by putting asserts or other forms of | ||||
exception-raising statements it becomes possible to have literate examples | ||||
that double as lightweight tests. | ||||
4. Extension modules with doctests in function and method docstrings. | ||||
Currently Nose simply can't find these docstrings correctly, because the | ||||
underlying doctest DocTestFinder object fails there. Similarly to #2 above, | ||||
the docstrings could have either pure python or IPython prompts. | ||||
Brian Granger
|
r2276 | |||
Fernando Perez
|
r2467 | Of these, only 3-c (reST with standalone code blocks) is not implemented at | ||
this point. | ||||