From 44dc248743ac90c28a3d28fa9effe987d9b98a8e 2010-01-13 09:50:02 From: Fernando Perez Date: 2010-01-13 09:50:02 Subject: [PATCH] Wrote up description with examples of our testing system. --- diff --git a/docs/source/development/testing.txt b/docs/source/development/testing.txt index 1a5971f..8d496cd 100644 --- a/docs/source/development/testing.txt +++ b/docs/source/development/testing.txt @@ -1,8 +1,8 @@ .. _testing: -========================= -Writing and running tests -========================= +========================================== + Testing IPython for users and developers +========================================== Overview ======== @@ -14,41 +14,342 @@ IPython test system can detect. See below for more details on this. Each subpackage in IPython should have its own :file:`tests` directory that contains all of the tests for that subpackage. All of the files in the :file:`tests` directory should have the word "tests" in them to enable -the testing framework to find them. +the testing framework to find them. + +In docstrings, examples (either using IPython prompts like ``In [1]:`` or +'classic' python ``>>>`` ones) can and should be included. The testing system +will detect them as doctests and will run them; it offers control to skip parts +or all of a specific doctest if the example is meant to be informative but +shows non-reproducible information (like filesystem data). If a subpackage has any dependencies beyond the Python standard library, the tests for that subpackage should be skipped if the dependencies are not found. This is very important so users don't get tests failing simply because they -don't have dependencies. We are still figuring out the best way for this -to be handled. +don't have dependencies. + +The testing system we use is a hybrid of nose_ and Twisted's trial_ test runner. +We use both because nose detects more things than Twisted and allows for more +flexible (and lighter-weight) ways of writing tests; in particular we've +developed a nose plugin that allows us to paste verbatim IPython sessions and +test them as doctests, which is extremely important for us. But the parts of +IPython that depend on Twisted must be tested using trial, because only trial +manages the Twisted reactor correctly. + +.. _nose: http://code.google.com/p/python-nose +.. _trial: http://twistedmatrix.com/trac/wiki/TwistedTrial + + +For the impatient: running the tests +==================================== + +The simplest way to test IPython is to type at the command line: + +.. code-block:: bash + + python -c "import IPython; IPython.test()" + +This should work as long as IPython can be imported, even if you haven't fully +installed the user-facing scripts yet (common in a development environment). +After a lot of output, you should see something like: + +.. code-block:: bash + + ************************************************************************ + Ran 10 test groups in 35.228s + + OK + +If not, there will be a message indicating which test group failed and how to +rerun that group individually. + +But IPython ships with an entry point script called :file:`iptest` that offers +fine-grain control over the test process and is particularly useful for +developers; this script also manages intelligently both nose and trial, +choosing the correct test system for each of IPython's components. Running +:file:`iptest` without arguments gives output identical to that above, but with +it, you can also run specific tests with fine control. The :file:`iptest` +script is installed with IPython, but if you are running from a source tree, +you can find it in the :file:`IPython/scripts` directory and you can run +directly from there. + +For example, this tests the :mod:`IPython.utils` subpackage, the :option:`-v` +option shows progress indicators: + +.. code-block:: bash -Status -====== + maqroll[ipython]> cd IPython/scripts/ + maqroll[scripts]> ./iptest -v IPython.utils + ..........................SS..SSS............................S.S......... + ................................................... + ---------------------------------------------------------------------- + Ran 125 tests in 0.070s -Currently IPython's testing system is being reworked. In the meantime, -we recommend the following testing practices: + OK (SKIP=7) -* To run regular tests, use the :command:`nosetests` command that Nose [Nose]_ - provides on a per file basis: +Because :file:`iptest` is based on nose, you can use all nose options and +syntax, typing ``iptest -h`` shows all available options. For example, this +lets you run the specific test :func:`test_rehashx` inside the +:mod:`test_magic` module: .. code-block:: bash - nosetests -vvs IPython.core.tests.test_component + maqroll[scripts]> ./iptest -vv IPython.core.tests.test_magic:test_rehashx + IPython.core.tests.test_magic.test_rehashx(True,) ... ok + IPython.core.tests.test_magic.test_rehashx(True,) ... ok -* To run Twisted-using tests, use the :command:`trial` command on a per file - basis: + ---------------------------------------------------------------------- + Ran 2 tests in 0.101s + + OK + +When developing, the :option:`--pdb` and :option:`--pdb-failures` of nose are +particularly useful, these drop you into an interactive pdb session at the +point of the error or failure respectively. + +To run Twisted-using tests, use the :command:`trial` command on a per file or +package basis: .. code-block:: bash trial IPython.kernel -* For now, regular tests (of non-Twisted using code) should be written as - unit tests. They should be subclasses of :class:`unittest.TestCase`. -* Tests of Twisted [Twisted]_ using code should be written by subclassing the - ``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, - all :class:`Deferred` instances that are created in the test must be - properly chained and the final one *must* be the return value of the test - method. +For developers: writing tests +============================= + +By now IPython has a reasonable test suite, so the best way to see what's +available is to look at the :file:`tests` directory in most subpackages. But +here are a few pointers to make the process easier. + + +Main tools: :mod:`IPython.testing` +---------------------------------- + +The :mod:`IPython.testing` package is where all of the machinery to test +IPython (rather than the tests for its various parts) lives. In particular, +the :mod:`iptest` module in there has all the smarts to control the test +process. In there, the :func:`make_exclude` function is used to build a +blacklist of exclusions, these are modules that do not get even imported for +tests. This is important so that things that would fail to even import because +of missing dependencies don't give errors to end users, as we stated above. + +The :mod:`decorators` module contains a lot of useful decorators, especially +useful to mark individual tests that should be skipped under certain conditions +(rather than blacklisting the package altogether because of a missing major +dependency). + +Our nose plugin for doctests +---------------------------- + +The :mod:`plugin` subpackage in testing contains a nose plugin called +:mod:`ipdoctest` that teaches nose about IPython syntax, so you can write +doctests with IPython prompts. You can also mark doctest output with ``# +random`` for the output corresponding to a single input to be ignored (stronger +than using ellipsis and useful to keep it as an example). If you want the +entire docstring to be executed but none of the output from any input to be +checked, you can use the ``# all-random`` marker. The +:mod:`IPython.testing.plugin.dtexample` module contains examples of how to use +these; for reference here is how to use ``# random``:: + + def ranfunc(): + """A function with some random output. + + Normal examples are verified as usual: + >>> 1+3 + 4 + + But if you put '# random' in the output, it is ignored: + >>> 1+3 + junk goes here... # random + + >>> 1+2 + again, anything goes #random + if multiline, the random mark is only needed once. + + >>> 1+2 + You can also put the random marker at the end: + # random + + >>> 1+2 + # random + .. or at the beginning. + + More correct input is properly verified: + >>> ranfunc() + 'ranfunc' + """ + return 'ranfunc' + +and an example of ``# all-random``:: + + def random_all(): + """A function where we ignore the output of ALL examples. + + Examples: + + # all-random + + This mark tells the testing machinery that all subsequent examples + should be treated as random (ignoring their output). They are still + executed, so if a they raise an error, it will be detected as such, + but their output is completely ignored. + + >>> 1+3 + junk goes here... + + >>> 1+3 + klasdfj; + + In [8]: print 'hello' + world # random + + In [9]: iprand() + Out[9]: 'iprand' + """ + return 'iprand' + + +When writing docstrings, you can use the ``@skip_doctest`` decorator to +indicate that a docstring should *not* be treated as a doctest at all. The +difference betwee ``# all-random`` and ``@skip_doctest`` is that the former +executes the example but ignores output, while the latter doesn't execute any +code. ``@skip_doctest`` should be used for docstrings whose examples are +purely informational. + +If a given docstring fails under certain conditions but otherwise is a good +doctest, you can use code like the following, that relies on the 'null' +decorator to leave the docstring intact where it works as a test:: + + # The docstring for full_path doctests differently on win32 (different path + # separator) so just skip the doctest there, and use a null decorator + # elsewhere: + + doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco + + @doctest_deco + def full_path(startPath,files): + """Make full paths for all the listed files, based on startPath...""" + + # function body follows... + +With our nose plugin that understands IPython syntax, an extremely effective +way to write tests is to simply copy and paste an interactive session into a +docstring. You can writing this type of test, where your docstring is meant +*only* as a test, by prefixing the function name with ``doctest_`` and leaving +its body *absolutely empty* other than the docstring. In +:mod:`IPython.core.tests.test_magic` you can find several examples of this, but +for completeness sake, your code should look like this (a simple case):: + + def doctest_time(): + """ + In [10]: %time None + CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s + Wall time: 0.00 s + """ + +This function is only analyzed for its docstring but it is not considered a +separate test, which is why its body should be empty. + + +Parametric tests done right +--------------------------- + +If you need to run multiple tests inside the same standalone function or method +of a :class:`unittest.TestCase` subclass, IPython provides the ``parametric`` +decorator for this purpose. This is superior to how test generators work in +nose, because IPython's keeps intact your stack, which makes debugging vastly +easier. For example, these are some parametric tests both in class form and as +a standalone function (choose in each situation the style that best fits the +problem at hand, since both work):: + + from IPython.testing import decorators as dec + + def is_smaller(i,j): + assert i>>`` and the input is pure Python. + - The prompts are of the form ``In [1]:`` and the input can contain extended + IPython expressions. + + In the first case, Nose will recognize the doctests as long as it is called + with the ``--with-doctest`` flag. But the second case will likely require + modifications or the writing of a new doctest plugin for Nose that is + IPython-aware. + +3. ReStructuredText files that contain code blocks. For this type of file, we + have three distinct possibilities for the code blocks: + - They use ``>>>`` prompts. + - They use ``In [1]:`` prompts. + - They are standalone blocks of pure Python code without any prompts. + + The first two cases are similar to the situation #2 above, except that in + this case the doctests must be extracted from input code blocks using + docutils instead of from the Python docstrings. + + In the third case, we must have a convention for distinguishing code blocks + that are meant for execution from others that may be snippets of shell code + or other examples not meant to be run. One possibility is to assume that + all indented code blocks are meant for execution, but to have a special + docutils directive for input that should not be executed. + + For those code blocks that we will execute, the convention used will simply + be that they get called and are considered successful if they run to + completion without raising errors. This is similar to what Nose does for + standalone test functions, and by putting asserts or other forms of + exception-raising statements it becomes possible to have literate examples + that double as lightweight tests. + +4. Extension modules with doctests in function and method docstrings. + Currently Nose simply can't find these docstrings correctly, because the + underlying doctest DocTestFinder object fails there. Similarly to #2 above, + the docstrings could have either pure python or IPython prompts. -.. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/ +Of these, only 3-c (reST with standalone code blocks) is not implemented at +this point.