Added diagnostics printout at the end of the test suite.

This will make it easier for us to understand problem reports from users.

IPython/core/release.py

@@ -1,121 +1,121
 # -*- coding: utf-8 -*-
 """Release data for the IPython project."""
 
 #*****************************************************************************
 #       Copyright (C) 2008-2009  The IPython Development Team
 #       Copyright (C) 2001-2008 Fernando Perez <[e-mail]>
 #       Copyright (c) 2001 Janko Hauser <[e-mail]> and Nathaniel Gray
 #       <[e-mail]>
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
 #*****************************************************************************
 
 # Name of the package for release purposes.  This is the name which labels
 # the tarballs and RPMs made by distutils, so it's best to lowercase it.
 name = 'ipython'
 
 # For versions with substrings (like 0.6.16.svn), use an extra . to separate
 # the new substring.  We have to avoid using either dashes or underscores,
 # because bdist_rpm does not accept dashes (an RPM) convention, and
 # bdist_deb does not accept underscores (a Debian convention).
 
 development = True    # change this to False to do a release
 version_base = '0.11'
 branch = 'ipython'
-revision = '1340'
+revision = '1346'
 
 if development:
     if branch == 'ipython':
         version = '%s.bzr.r%s' % (version_base, revision)
     else:
         version = '%s.bzr.r%s.%s' % (version_base, revision, branch)
 else:
     version = version_base
 
 
 description = "An interactive computing environment for Python"
 
 long_description = \
 """
 The goal of IPython is to create a comprehensive environment for
 interactive and exploratory computing.  To support this goal, IPython
 has two main components:
 
 * An enhanced interactive Python shell.
 
 * An architecture for interactive parallel computing.
 
 The enhanced interactive Python shell has the following main features:
 
 * Comprehensive object introspection.
 
 * Input history, persistent across sessions.
 
 * Caching of output results during a session with automatically generated
   references.
 
 * Readline based name completion.
 
 * Extensible system of 'magic' commands for controlling the environment and
   performing many tasks related either to IPython or the operating system.
 
 * Configuration system with easy switching between different setups (simpler
   than changing $PYTHONSTARTUP environment variables every time).
 
 * Session logging and reloading.
 
 * Extensible syntax processing for special purpose situations.
 
 * Access to the system shell with user-extensible alias system.
 
 * Easily embeddable in other Python programs and wxPython GUIs.
 
 * Integrated access to the pdb debugger and the Python profiler. 
 
 The parallel computing architecture has the following main features:
 
 * Quickly parallelize Python code from an interactive Python/IPython session.
 
 * A flexible and dynamic process model that be deployed on anything from 
   multicore workstations to supercomputers.
 
 * An architecture that supports many different styles of parallelism, from
   message passing to task farming.
 
 * Both blocking and fully asynchronous interfaces.
 
 * High level APIs that enable many things to be parallelized in a few lines
   of code.
 
 * Share live parallel jobs with other users securely.  
 
 * Dynamically load balanced task farming system.
 
 * Robust error handling in parallel code.
 
 The latest development version is always available from IPython's `Launchpad 
 site <http://launchpad.net/ipython>`_.
 """
 
 license = 'BSD'
 
 authors = {'Fernando' : ('Fernando Perez','[e-mail]'),
            'Janko'    : ('Janko Hauser','[e-mail]'),
            'Nathan'   : ('Nathaniel Gray','[e-mail]'),
            'Ville'    : ('Ville Vainio','[e-mail]'),
            'Brian'    : ('Brian E Granger', '[e-mail]'),
            'Min'      : ('Min Ragan-Kelley', '[e-mail]')
            }
 
 author = 'The IPython Development Team'
 
 author_email = '[e-mail]'
 
 url = 'http://ipython.scipy.org'
 
 download_url = 'http://ipython.scipy.org/dist'
 
 platforms = ['Linux','Mac OSX','Windows XP/2000/NT','Windows 95/98/ME']
 
 keywords = ['Interactive','Interpreter','Shell','Parallel','Distributed']

IPython/testing/iptest.py

@@ -1,450 +1,488
 # -*- coding: utf-8 -*-
 """IPython Test Suite Runner.
 
 This module provides a main entry point to a user script to test IPython
 itself from the command line. There are two ways of running this script:
 
 1. With the syntax `iptest all`.  This runs our entire test suite by
    calling this script (with different arguments) or trial recursively.  This
    causes modules and package to be tested in different processes, using nose
    or trial where appropriate.
 2. With the regular nose syntax, like `iptest -vvs IPython`.  In this form
    the script simply calls nose, but with special command line flags and
    plugins loaded.
 
 For now, this script requires that both nose and twisted are installed.  This
 will change in the future.
 """
 
 #-----------------------------------------------------------------------------
 # Module imports
 #-----------------------------------------------------------------------------
 
 # Stdlib
 import os
 import os.path as path
+import platform
 import signal
 import sys
 import subprocess
 import tempfile
 import time
 import warnings
 
 
 # Ugly,  but necessary hack to ensure the test suite finds our version of
 # IPython and not a possibly different one that may exist system-wide.
 # Note that this must be done here, so the imports that come next work
 # correctly even if IPython isn't installed yet.
 p = os.path
 ippath = p.abspath(p.join(p.dirname(__file__),'..','..'))
 sys.path.insert(0, ippath)
 
 # Note: monkeypatch!
 # We need to monkeypatch a small problem in nose itself first, before importing
 # it for actual use.  This should get into nose upstream, but its release cycle
 # is slow and we need it for our parametric tests to work correctly.
 from IPython.testing import nosepatch
 # Now, proceed to import nose itself
 import nose.plugins.builtin
 from nose.core import TestProgram
 
 # Our own imports
+from IPython.core import release
 from IPython.utils import genutils
 from IPython.utils.platutils import find_cmd, FindCmdError
 from IPython.testing import globalipapp
 from IPython.testing import tools
 from IPython.testing.plugin.ipdoctest import IPythonDoctest
 
 pjoin = path.join
 
 
 #-----------------------------------------------------------------------------
 # Globals
 #-----------------------------------------------------------------------------
 
 # By default, we assume IPython has been installed.  But if the test suite is
 # being run from a source tree that has NOT been installed yet, this flag can
 # be set to False by the entry point scripts, to let us know that we must call
 # the source tree versions of the scripts which manipulate sys.path instead of
 # assuming that things exist system-wide.
 INSTALLED = True
 
 #-----------------------------------------------------------------------------
 # Warnings control
 #-----------------------------------------------------------------------------
 # Twisted generates annoying warnings with Python 2.6, as will do other code
 # that imports 'sets' as of today
 warnings.filterwarnings('ignore', 'the sets module is deprecated',
                         DeprecationWarning )
 
 # This one also comes from Twisted
 warnings.filterwarnings('ignore', 'the sha module is deprecated',
                         DeprecationWarning)
 
 # Wx on Fedora11 spits these out
 warnings.filterwarnings('ignore', 'wxPython/wxWidgets release number mismatch',
                         UserWarning)
 
 #-----------------------------------------------------------------------------
 # Logic for skipping doctests
 #-----------------------------------------------------------------------------
 
 def test_for(mod):
     """Test to see if mod is importable."""
     try:
         __import__(mod)
     except (ImportError, RuntimeError):
         # GTK reports Runtime error if it can't be initialized even if  it's
         # importable.
         return False
     else:
         return True
 
-have_curses = test_for('_curses')
-have_wx = test_for('wx')
-have_wx_aui = test_for('wx.aui')
-have_zi = test_for('zope.interface')
-have_twisted = test_for('twisted')
-have_foolscap = test_for('foolscap')
-have_objc = test_for('objc')
-have_pexpect = test_for('pexpect')
-have_gtk = test_for('gtk')
-have_gobject = test_for('gobject')
+# Global dict where we can store information on what we have and what we don't
+# have available at test run time
+have = {}
+
+have['curses'] = test_for('_curses')
+have['wx'] = test_for('wx')
+have['wx.aui'] = test_for('wx.aui')
+have['zope.interface'] = test_for('zope.interface')
+have['twisted'] = test_for('twisted')
+have['foolscap'] = test_for('foolscap')
+have['objc'] = test_for('objc')
+have['pexpect'] = test_for('pexpect')
+have['gtk'] = test_for('gtk')
+have['gobject'] = test_for('gobject')
 
 #-----------------------------------------------------------------------------
 # Functions and classes
 #-----------------------------------------------------------------------------
 
+def report():
+    """Return a string with a summary report of test-related variables."""
+
+    out = [ genutils.sys_info() ]
+
+    out.append('\nRunning from an installed IPython: %s\n' % INSTALLED)
+
+    avail = []
+    not_avail = []
+    
+    for k, is_avail in have.items():
+        if is_avail:
+            avail.append(k)
+        else:
+            not_avail.append(k)
+
+    if avail:
+        out.append('\nTools and libraries available at test time:\n')
+        avail.sort()
+        out.append('   ' + ' '.join(avail)+'\n')
+
+    if not_avail:
+        out.append('\nTools and libraries NOT available at test time:\n')
+        not_avail.sort()
+        out.append('   ' + ' '.join(not_avail)+'\n')
+        
+    return ''.join(out)
+
+
 def make_exclude():
     """Make patterns of modules and packages to exclude from testing.
     
     For the IPythonDoctest plugin, we need to exclude certain patterns that
     cause testing problems.  We should strive to minimize the number of
     skipped modules, since this means untested code.  As the testing
     machinery solidifies, this list should eventually become empty.
     These modules and packages will NOT get scanned by nose at all for tests.
     """
     # Simple utility to make IPython paths more readably, we need a lot of
     # these below
     ipjoin = lambda *paths: pjoin('IPython', *paths)
     
     exclusions = [ipjoin('external'),
                   ipjoin('frontend', 'process', 'winprocess.py'),
                   # Deprecated old Shell and iplib modules, skip to avoid
                   # warnings
                   ipjoin('Shell'),
                   ipjoin('iplib'),
                   pjoin('IPython_doctest_plugin'),
                   ipjoin('quarantine'),
                   ipjoin('deathrow'),
                   ipjoin('testing', 'attic'),
                   # This guy is probably attic material
                   ipjoin('testing', 'mkdoctests'),
                   # Testing inputhook will need a lot of thought, to figure out
                   # how to have tests that don't lock up with the gui event
                   # loops in the picture
                   ipjoin('lib', 'inputhook'),
                   # Config files aren't really importable stand-alone
                   ipjoin('config', 'default'),
                   ipjoin('config', 'profile'),
                   ]
 
-    if not have_wx:
+    if not have['wx']:
         exclusions.append(ipjoin('gui'))
         exclusions.append(ipjoin('frontend', 'wx'))
         exclusions.append(ipjoin('lib', 'inputhookwx'))
 
-    if not have_gtk or not have_gobject:
+    if not have['gtk'] or not have['gobject']:
         exclusions.append(ipjoin('lib', 'inputhookgtk'))
 
-    if not have_wx_aui:
+    if not have['wx.aui']:
         exclusions.append(ipjoin('gui', 'wx', 'wxIPython'))
 
-    if not have_objc:
+    if not have['objc']:
         exclusions.append(ipjoin('frontend', 'cocoa'))
 
     if not sys.platform == 'win32':
         exclusions.append(ipjoin('utils', 'platutils_win32'))
 
     # These have to be skipped on win32 because the use echo, rm, cd, etc.
     # See ticket https://bugs.launchpad.net/bugs/366982
     if sys.platform == 'win32':
         exclusions.append(ipjoin('testing', 'plugin', 'test_exampleip'))
         exclusions.append(ipjoin('testing', 'plugin', 'dtexample'))
 
     if not os.name == 'posix':
         exclusions.append(ipjoin('utils', 'platutils_posix'))
 
-    if not have_pexpect:
+    if not have['pexpect']:
         exclusions.extend([ipjoin('scripts', 'irunner'),
                            ipjoin('lib', 'irunner')])
 
     # This is scary.  We still have things in frontend and testing that
     # are being tested by nose that use twisted.  We need to rethink
     # how we are isolating dependencies in testing.
-    if not (have_twisted and have_zi and have_foolscap):
+    if not (have['twisted'] and have['zope.interface'] and have['foolscap']):
         exclusions.extend(
             [ipjoin('frontend', 'asyncfrontendbase'),
              ipjoin('frontend', 'prefilterfrontend'),
              ipjoin('frontend', 'frontendbase'),
              ipjoin('frontend', 'linefrontendbase'),
              ipjoin('frontend', 'tests', 'test_linefrontend'),
              ipjoin('frontend', 'tests', 'test_frontendbase'),
              ipjoin('frontend', 'tests', 'test_prefilterfrontend'),
              ipjoin('frontend', 'tests', 'test_asyncfrontendbase'),
              ipjoin('testing', 'parametric'),
              ipjoin('testing', 'util'),
              ipjoin('testing', 'tests', 'test_decorators_trial'),
              ] )
 
     # This is needed for the reg-exp to match on win32 in the ipdoctest plugin.
     if sys.platform == 'win32':
         exclusions = [s.replace('\\','\\\\') for s in exclusions]
 
     return exclusions
 
 
 class IPTester(object):
     """Call that calls iptest or trial in a subprocess.
     """
     #: string, name of test runner that will be called
     runner = None
     #: list, parameters for test runner
     params = None
     #: list, arguments of system call to be made to call test runner
     call_args = None
     #: list, process ids of subprocesses we start (for cleanup)
     pids = None
     
     def __init__(self, runner='iptest', params=None):
         """Create new test runner."""
         p = os.path
         if runner == 'iptest':
             if INSTALLED:
                 self.runner = tools.cmd2argv(
                     p.abspath(find_cmd('iptest'))) + sys.argv[1:]
             else:
                 # Find our own 'iptest' script OS-level entry point.  Don't
                 # look system-wide, so we are sure we pick up *this one*.  And
                 # pass through to subprocess call our own sys.argv
                 ippath = p.abspath(p.join(p.dirname(__file__),'..','..'))
                 script = p.join(ippath, 'iptest.py')
                 self.runner = tools.cmd2argv(script) + sys.argv[1:]
                 
         else:
             # For trial, it needs to be installed system-wide
             self.runner = tools.cmd2argv(p.abspath(find_cmd('trial')))
         if params is None:
             params = []
         if isinstance(params, str):
             params = [params]
         self.params = params
 
         # Assemble call
         self.call_args = self.runner+self.params
 
         # Store pids of anything we start to clean up on deletion, if possible
         # (on posix only, since win32 has no os.kill)
         self.pids = []
 
     if sys.platform == 'win32':
         def _run_cmd(self):
             # On Windows, use os.system instead of subprocess.call, because I
             # was having problems with subprocess and I just don't know enough
             # about win32 to debug this reliably.  Os.system may be the 'old
             # fashioned' way to do it, but it works just fine.  If someone
             # later can clean this up that's fine, as long as the tests run
             # reliably in win32.
             return os.system(' '.join(self.call_args))
     else:
         def _run_cmd(self):
             #print >> sys.stderr, '*** CMD:', ' '.join(self.call_args) # dbg
             subp = subprocess.Popen(self.call_args)
             self.pids.append(subp.pid)
             # If this fails, the pid will be left in self.pids and cleaned up
             # later, but if the wait call succeeds, then we can clear the
             # stored pid.
             retcode = subp.wait()
             self.pids.pop()
             return retcode
         
     def run(self):
         """Run the stored commands"""
         try:
             return self._run_cmd()
         except:
             import traceback
             traceback.print_exc()
             return 1  # signal failure
 
     def __del__(self):
         """Cleanup on exit by killing any leftover processes."""
 
         if not hasattr(os, 'kill'):
             return
         
         for pid in self.pids:
             try:
                 print 'Cleaning stale PID:', pid
                 os.kill(pid, signal.SIGKILL)
             except OSError:
                 # This is just a best effort, if we fail or the process was
                 # really gone, ignore it.
                 pass     
 
 
 def make_runners():
     """Define the top-level packages that need to be tested.
     """
 
     # Packages to be tested via nose, that only depend on the stdlib
     nose_pkg_names = ['config', 'core', 'extensions', 'frontend', 'lib',
                      'scripts', 'testing', 'utils' ]
     # The machinery in kernel needs twisted for real testing
     trial_pkg_names = []
 
-    if have_wx:
+    if have['wx']:
         nose_pkg_names.append('gui')
 
     # And add twisted ones if conditions are met
-    if have_zi and have_twisted and have_foolscap:
+    if have['zope.interface'] and have['twisted'] and have['foolscap']:
         # Note that we list the kernel here, though the bulk of it is
         # twisted-based, because nose picks up doctests that twisted doesn't.
         nose_pkg_names.append('kernel')
         trial_pkg_names.append('kernel')
 
     # For debugging this code, only load quick stuff
     #nose_pkg_names = ['core', 'extensions']  # dbg
     #trial_pkg_names = []  # dbg 
 
     # Make fully qualified package names prepending 'IPython.' to our name lists
     nose_packages = ['IPython.%s' % m for m in nose_pkg_names ]
     trial_packages = ['IPython.%s' % m for m in trial_pkg_names ]
 
     # Make runners
     runners = [ (v, IPTester('iptest', params=v)) for v in nose_packages ]
     runners.extend([ (v, IPTester('trial', params=v)) for v in trial_packages ])
     
     return runners
 
 
 def run_iptest():
     """Run the IPython test suite using nose.
     
     This function is called when this script is **not** called with the form
     `iptest all`.  It simply calls nose with appropriate command line flags
     and accepts all of the standard nose arguments.
     """
 
     warnings.filterwarnings('ignore', 
         'This will be removed soon.  Use IPython.testing.util instead')
 
     argv = sys.argv + [ '--detailed-errors',  # extra info in tracebacks
                                                 
                         # Loading ipdoctest causes problems with Twisted, but
                         # our test suite runner now separates things and runs
                         # all Twisted tests with trial.
                         '--with-ipdoctest',
                         '--ipdoctest-tests','--ipdoctest-extension=txt',
                         
                         # We add --exe because of setuptools' imbecility (it
                         # blindly does chmod +x on ALL files).  Nose does the
                         # right thing and it tries to avoid executables,
                         # setuptools unfortunately forces our hand here.  This
                         # has been discussed on the distutils list and the
                         # setuptools devs refuse to fix this problem!
                         '--exe',
                         ]
 
     if nose.__version__ >= '0.11':
         # I don't fully understand why we need this one, but depending on what
         # directory the test suite is run from, if we don't give it, 0 tests
         # get run.  Specifically, if the test suite is run from the source dir
         # with an argument (like 'iptest.py IPython.core', 0 tests are run,
         # even if the same call done in this directory works fine).  It appears
         # that if the requested package is in the current dir, nose bails early
         # by default.  Since it's otherwise harmless, leave it in by default
         # for nose >= 0.11, though unfortunately nose 0.10 doesn't support it.
         argv.append('--traverse-namespace')
 
     # Construct list of plugins, omitting the existing doctest plugin, which
     # ours replaces (and extends).
     plugins = [IPythonDoctest(make_exclude())]
     for p in nose.plugins.builtin.plugins:
         plug = p()
         if plug.name == 'doctest':
             continue
         plugins.append(plug)
 
     # We need a global ipython running in this process
     globalipapp.start_ipython()
     # Now nose can run
     TestProgram(argv=argv, plugins=plugins)
 
 
 def run_iptestall():
     """Run the entire IPython test suite by calling nose and trial.
     
     This function constructs :class:`IPTester` instances for all IPython
     modules and package and then runs each of them.  This causes the modules
     and packages of IPython to be tested each in their own subprocess using
     nose or twisted.trial appropriately.
     """
 
     runners = make_runners()
 
     # Run the test runners in a temporary dir so we can nuke it when finished
     # to clean up any junk files left over by accident.  This also makes it
     # robust against being run in non-writeable directories by mistake, as the
     # temp dir will always be user-writeable.
     curdir = os.getcwd()
     testdir = tempfile.gettempdir()
     os.chdir(testdir)
 
     # Run all test runners, tracking execution time
     failed = []
     t_start = time.time()
     try:
         for (name, runner) in runners:
             print '*'*70
             print 'IPython test group:',name
             res = runner.run()
             if res:
                 failed.append( (name, runner) )
     finally:
         os.chdir(curdir)
     t_end = time.time()
     t_tests = t_end - t_start
     nrunners = len(runners)
     nfail = len(failed)
     # summarize results
     print
     print '*'*70
+    print 'Test suite completed for system with the following information:'
+    print report()
     print 'Ran %s test groups in %.3fs' % (nrunners, t_tests)
     print
+    print 'Status:'
     if not failed:
         print 'OK'
     else:
         # If anything went wrong, point out what command to rerun manually to
         # see the actual errors and individual summary
         print 'ERROR - %s out of %s test groups failed.' % (nfail, nrunners)
         for name, failed_runner in failed:
             print '-'*40
             print 'Runner failed:',name
             print 'You may wish to rerun this one individually, with:'
             print ' '.join(failed_runner.call_args)
             print
 
 
 def main():
     for arg in sys.argv[1:]:
         if arg.startswith('IPython'):
             # This is in-process
             run_iptest()
     else:
         # This starts subprocesses
         run_iptestall()
 
 
 if __name__ == '__main__':
     main()

docs/source/development/testing.txt

@@ -1,372 +1,388
 .. _testing:
 
 ==========================================
  Testing IPython for users and developers
 ==========================================
 
 Overview
 ========
 
 It is extremely important that all code contributed to IPython has tests.
 Tests should be written as unittests, doctests or other entities that the
 IPython test system can detect.  See below for more details on this.
 
 Each subpackage in IPython should have its own :file:`tests` directory that
 contains all of the tests for that subpackage. All of the files in the
 :file:`tests` directory should have the word "tests" in them to enable
 the testing framework to find them.
 
 In docstrings, examples (either using IPython prompts like ``In [1]:`` or
 'classic' python ``>>>`` ones) can and should be included.  The testing system
 will detect them as doctests and will run them; it offers control to skip parts
 or all of a specific doctest if the example is meant to be informative but
 shows non-reproducible information (like filesystem data).
 
 If a subpackage has any dependencies beyond the Python standard library, the
 tests for that subpackage should be skipped if the dependencies are not found.
 This is very important so users don't get tests failing simply because they
 don't have dependencies.
 
 The testing system we use is a hybrid of nose_ and Twisted's trial_ test runner.
 We use both because nose detects more things than Twisted and allows for more
 flexible (and lighter-weight) ways of writing tests; in particular we've
 developed a nose plugin that allows us to paste verbatim IPython sessions and
 test them as doctests, which is extremely important for us.  But the parts of
 IPython that depend on Twisted must be tested using trial, because only trial
 manages the Twisted reactor correctly.
 
 .. _nose: http://code.google.com/p/python-nose
 .. _trial: http://twistedmatrix.com/trac/wiki/TwistedTrial
 
 
 For the impatient: running the tests
 ====================================
 
 You can run IPython from the source download directory without even installing
 it system-wide or having configure anything, by typing at the terminal:
 
 .. code-block:: bash
 
    python ipython.py
 
 and similarly, you can execute the built-in test suite with:
    
 .. code-block:: bash
 
    python iptest.py
 
 
 This script manages intelligently both nose and trial, choosing the correct
 test system for each of IPython's components.
 
 Once you have either installed it or at least configured your system to be
 able to import IPython, you can run the tests with:
 
 .. code-block:: bash
 
   python -c "import IPython; IPython.test()"
 
 This should work as long as IPython can be imported, even if you haven't fully
 installed the user-facing scripts yet (common in a development environment).
 Once you have installed IPython, you will have available system-wide a script
 called :file:`iptest` that does the exact same as the :file:`iptest.py` script
 in the source directory, so you can then test simply with:
 
 .. code-block:: bash
 
    iptest  [args]
 
 
 Regardless of how you run things, you should eventually see something like:
 
 .. code-block:: bash
 
    **********************************************************************
-   Ran 11 test groups in 64.117s
+   Test suite completed for system with the following information:
+   IPython version: 0.11.bzr.r1340
+   BZR revision   : 1340
+   Platform info  : os.name -> posix, sys.platform -> linux2
+		  : Linux-2.6.31-17-generic-i686-with-Ubuntu-9.10-karmic
+   Python info    : 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
+   [GCC 4.4.1]
 
-   OK
+   Running from an installed IPython: True
+
+   Tools and libraries available at test time:
+      curses foolscap gobject gtk pexpect twisted wx wx.aui zope.interface
+
+   Tools and libraries NOT available at test time:
+      objc
 
+   Ran 11 test groups in 36.244s
+
+   Status:
+   OK
 
 If not, there will be a message indicating which test group failed and how to
 rerun that group individually.  For example, this tests the
 :mod:`IPython.utils` subpackage, the :option:`-v` option shows progress
 indicators:
 
 .. code-block:: bash
 
    $ python iptest.py -v IPython.utils
    ..........................SS..SSS............................S.S...
    .........................................................
    ----------------------------------------------------------------------
    Ran 125 tests in 0.119s
 
    OK (SKIP=7)
 
 
 Because the IPython test machinery is based on nose, you can use all nose
 options and syntax, typing ``iptest -h`` shows all available options.  For
 example, this lets you run the specific test :func:`test_rehashx` inside the
 :mod:`test_magic` module:
 
 .. code-block:: bash
 
    $ python iptest.py -vv IPython.core.tests.test_magic:test_rehashx
    IPython.core.tests.test_magic.test_rehashx(True,) ... ok
    IPython.core.tests.test_magic.test_rehashx(True,) ... ok
 
    ----------------------------------------------------------------------
    Ran 2 tests in 0.100s
 
    OK
 
 When developing, the :option:`--pdb` and :option:`--pdb-failures` of nose are
 particularly useful, these drop you into an interactive pdb session at the
 point of the error or failure respectively.
 
 To run Twisted-using tests, use the :command:`trial` command on a per file or
 package basis:
 
 .. code-block:: bash
 
     trial IPython.kernel
 
 
 For developers: writing tests
 =============================
 
 By now IPython has a reasonable test suite, so the best way to see what's
 available is to look at the :file:`tests` directory in most subpackages.  But
 here are a few pointers to make the process easier.
 
 
 Main tools: :mod:`IPython.testing`
 ----------------------------------
 
 The :mod:`IPython.testing` package is where all of the machinery to test
 IPython (rather than the tests for its various parts) lives.  In particular,
 the :mod:`iptest` module in there has all the smarts to control the test
 process.  In there, the :func:`make_exclude` function is used to build a
 blacklist of exclusions, these are modules that do not get even imported for
 tests.  This is important so that things that would fail to even import because
 of missing dependencies don't give errors to end users, as we stated above.
 
 The :mod:`decorators` module contains a lot of useful decorators, especially
 useful to mark individual tests that should be skipped under certain conditions
 (rather than blacklisting the package altogether because of a missing major
 dependency).
 
 Our nose plugin for doctests
 ----------------------------
 
 The :mod:`plugin` subpackage in testing contains a nose plugin called
 :mod:`ipdoctest` that teaches nose about IPython syntax, so you can write
 doctests with IPython prompts.  You can also mark doctest output with ``#
 random`` for the output corresponding to a single input to be ignored (stronger
 than using ellipsis and useful to keep it as an example).  If you want the
 entire docstring to be executed but none of the output from any input to be
 checked, you can use the ``# all-random`` marker.  The
 :mod:`IPython.testing.plugin.dtexample` module contains examples of how to use
 these; for reference here is how to use ``# random``::
 
     def ranfunc():
 	"""A function with some random output.
 
 	   Normal examples are verified as usual:
 	   >>> 1+3
 	   4
 
 	   But if you put '# random' in the output, it is ignored:
 	   >>> 1+3
 	   junk goes here...  # random
 
 	   >>> 1+2
 	   again,  anything goes #random
 	   if multiline, the random mark is only needed once.
 
 	   >>> 1+2
 	   You can also put the random marker at the end:
 	   # random
 
 	   >>> 1+2
 	   # random
 	   .. or at the beginning.
 
 	   More correct input is properly verified:
 	   >>> ranfunc()
 	   'ranfunc'
 	"""
 	return 'ranfunc'
 
 and an example of ``# all-random``::
 	
     def random_all():
 	"""A function where we ignore the output of ALL examples.
 
 	Examples:
 
 	  # all-random
 
 	  This mark tells the testing machinery that all subsequent examples
 	  should be treated as random (ignoring their output).  They are still
 	  executed, so if a they raise an error, it will be detected as such,
 	  but their output is completely ignored.
 
 	  >>> 1+3
 	  junk goes here...
 
 	  >>> 1+3
 	  klasdfj;
 
 	In [8]: print 'hello'
 	world  # random
 
 	In [9]: iprand()
 	Out[9]: 'iprand'
 	"""
 	return 'iprand'
 
 
 When writing docstrings, you can use the ``@skip_doctest`` decorator to
 indicate that a docstring should *not* be treated as a doctest at all.  The
 difference betwee ``# all-random`` and ``@skip_doctest`` is that the former
 executes the example but ignores output, while the latter doesn't execute any
 code.  ``@skip_doctest`` should be used for docstrings whose examples are
 purely informational.
 
 If a given docstring fails under certain conditions but otherwise is a good
 doctest, you can use code like the following, that relies on the 'null'
 decorator to leave the docstring intact where it works as a test::
 
   # The docstring for full_path doctests differently on win32 (different path
   # separator) so just skip the doctest there, and use a null decorator
   # elsewhere:
   
   doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco
 
   @doctest_deco
   def full_path(startPath,files):
       """Make full paths for all the listed files, based on startPath..."""
 
       # function body follows...
 
 With our nose plugin that understands IPython syntax, an extremely effective
 way to write tests is to simply copy and paste an interactive session into a
 docstring.  You can writing this type of test, where your docstring is meant
 *only* as a test, by prefixing the function name with ``doctest_`` and leaving
 its body *absolutely empty* other than the docstring.  In
 :mod:`IPython.core.tests.test_magic` you can find several examples of this, but
 for completeness sake, your code should look like this (a simple case)::
 
     def doctest_time():
 	"""
 	In [10]: %time None
 	CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
 	Wall time: 0.00 s
 	"""
 	
 This function is only analyzed for its docstring but it is not considered a
 separate test, which is why its body should be empty.
 
 
 Parametric tests done right
 ---------------------------
 
 If you need to run multiple tests inside the same standalone function or method
 of a :class:`unittest.TestCase` subclass, IPython provides the ``parametric``
 decorator for this purpose.  This is superior to how test generators work in
 nose, because IPython's keeps intact your stack, which makes debugging vastly
 easier.  For example, these are some parametric tests both in class form and as
 a standalone function (choose in each situation the style that best fits the
 problem at hand, since both work)::
 
   from IPython.testing import decorators as dec
 
   def is_smaller(i,j):
       assert i<j,"%s !< %s" % (i,j)
 
   class Tester(ParametricTestCase):
 
       def test_parametric(self):
 	  yield is_smaller(3, 4)
 	  x, y = 1, 2
 	  yield is_smaller(x, y)
 
   @dec.parametric
   def test_par_standalone():
       yield is_smaller(3, 4)
       x, y = 1, 2
       yield is_smaller(x, y)
       
 
 Writing tests for Twisted-using code
 ------------------------------------
       
 Tests of Twisted [Twisted]_ using code should be written by subclassing the
 ``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, all
 :class:`Deferred` instances that are created in the test must be properly
 chained and the final one *must* be the return value of the test method.
 
 .. note::
   
   The best place to see how to use the testing tools, are the tests for these
   tools themselves, which live in :mod:`IPython.testing.tests`.
   
     
 Design requirements
 ===================
 
 This section is a set of notes on the key points of the IPython testing needs,
 that were used when writing the system and should be kept for reference as it
 eveolves.
 
 Testing IPython in full requires modifications to the default behavior of nose
 and doctest, because the IPython prompt is not recognized to determine Python
 input, and because IPython admits user input that is not valid Python (things
 like ``%magics`` and ``!system commands``.
 
 We basically need to be able to test the following types of code:
 
 1. Pure Python files containing normal tests.  These are not a problem, since
    Nose will pick them up as long as they conform to the (flexible) conventions
    used by nose to recognize tests.
 
 2. Python files containing doctests.  Here, we have two possibilities:
    - The prompts are the usual ``>>>`` and the input is pure Python.
    - The prompts are of the form ``In [1]:`` and the input can contain extended
      IPython expressions.
 
    In the first case, Nose will recognize the doctests as long as it is called
    with the ``--with-doctest`` flag.  But the second case will likely require
    modifications or the writing of a new doctest plugin for Nose that is
    IPython-aware.
 
 3. ReStructuredText files that contain code blocks.  For this type of file, we
    have three distinct possibilities for the code blocks:
    - They use ``>>>`` prompts.
    - They use ``In [1]:`` prompts.
    - They are standalone blocks of pure Python code without any prompts.
 
    The first two cases are similar to the situation #2 above, except that in
    this case the doctests must be extracted from input code blocks using
    docutils instead of from the Python docstrings.
 
    In the third case, we must have a convention for distinguishing code blocks
    that are meant for execution from others that may be snippets of shell code
    or other examples not meant to be run.  One possibility is to assume that
    all indented code blocks are meant for execution, but to have a special
    docutils directive for input that should not be executed.
 
    For those code blocks that we will execute, the convention used will simply
    be that they get called and are considered successful if they run to
    completion without raising errors.  This is similar to what Nose does for
    standalone test functions, and by putting asserts or other forms of
    exception-raising statements it becomes possible to have literate examples
    that double as lightweight tests.
 
 4. Extension modules with doctests in function and method docstrings.
    Currently Nose simply can't find these docstrings correctly, because the
    underlying doctest DocTestFinder object fails there.  Similarly to #2 above,
    the docstrings could have either pure python or IPython prompts.
 
 Of these, only 3-c (reST with standalone code blocks) is not implemented at
 this point.