Show More
@@ -1,8 +1,8 b'' | |||
|
1 | 1 | .. _testing: |
|
2 | 2 | |
|
3 | ========================= | |
|
4 | Writing and running tests | |
|
5 | ========================= | |
|
3 | ========================================== | |
|
4 | Testing IPython for users and developers | |
|
5 | ========================================== | |
|
6 | 6 | |
|
7 | 7 | Overview |
|
8 | 8 | ======== |
@@ -14,41 +14,342 b' IPython test system can detect. See below for more details on this.' | |||
|
14 | 14 | Each subpackage in IPython should have its own :file:`tests` directory that |
|
15 | 15 | contains all of the tests for that subpackage. All of the files in the |
|
16 | 16 | :file:`tests` directory should have the word "tests" in them to enable |
|
17 |
the testing framework to find them. |
|
|
17 | the testing framework to find them. | |
|
18 | ||
|
19 | In docstrings, examples (either using IPython prompts like ``In [1]:`` or | |
|
20 | 'classic' python ``>>>`` ones) can and should be included. The testing system | |
|
21 | will detect them as doctests and will run them; it offers control to skip parts | |
|
22 | or all of a specific doctest if the example is meant to be informative but | |
|
23 | shows non-reproducible information (like filesystem data). | |
|
18 | 24 | |
|
19 | 25 | If a subpackage has any dependencies beyond the Python standard library, the |
|
20 | 26 | tests for that subpackage should be skipped if the dependencies are not found. |
|
21 | 27 | This is very important so users don't get tests failing simply because they |
|
22 | don't have dependencies. We are still figuring out the best way for this | |
|
23 | to be handled. | |
|
28 | don't have dependencies. | |
|
29 | ||
|
30 | The testing system we use is a hybrid of nose_ and Twisted's trial_ test runner. | |
|
31 | We use both because nose detects more things than Twisted and allows for more | |
|
32 | flexible (and lighter-weight) ways of writing tests; in particular we've | |
|
33 | developed a nose plugin that allows us to paste verbatim IPython sessions and | |
|
34 | test them as doctests, which is extremely important for us. But the parts of | |
|
35 | IPython that depend on Twisted must be tested using trial, because only trial | |
|
36 | manages the Twisted reactor correctly. | |
|
37 | ||
|
38 | .. _nose: http://code.google.com/p/python-nose | |
|
39 | .. _trial: http://twistedmatrix.com/trac/wiki/TwistedTrial | |
|
40 | ||
|
41 | ||
|
42 | For the impatient: running the tests | |
|
43 | ==================================== | |
|
44 | ||
|
45 | The simplest way to test IPython is to type at the command line: | |
|
46 | ||
|
47 | .. code-block:: bash | |
|
48 | ||
|
49 | python -c "import IPython; IPython.test()" | |
|
50 | ||
|
51 | This should work as long as IPython can be imported, even if you haven't fully | |
|
52 | installed the user-facing scripts yet (common in a development environment). | |
|
53 | After a lot of output, you should see something like: | |
|
54 | ||
|
55 | .. code-block:: bash | |
|
56 | ||
|
57 | ************************************************************************ | |
|
58 | Ran 10 test groups in 35.228s | |
|
59 | ||
|
60 | OK | |
|
61 | ||
|
62 | If not, there will be a message indicating which test group failed and how to | |
|
63 | rerun that group individually. | |
|
64 | ||
|
65 | But IPython ships with an entry point script called :file:`iptest` that offers | |
|
66 | fine-grain control over the test process and is particularly useful for | |
|
67 | developers; this script also manages intelligently both nose and trial, | |
|
68 | choosing the correct test system for each of IPython's components. Running | |
|
69 | :file:`iptest` without arguments gives output identical to that above, but with | |
|
70 | it, you can also run specific tests with fine control. The :file:`iptest` | |
|
71 | script is installed with IPython, but if you are running from a source tree, | |
|
72 | you can find it in the :file:`IPython/scripts` directory and you can run | |
|
73 | directly from there. | |
|
74 | ||
|
75 | For example, this tests the :mod:`IPython.utils` subpackage, the :option:`-v` | |
|
76 | option shows progress indicators: | |
|
77 | ||
|
78 | .. code-block:: bash | |
|
24 | 79 | |
|
25 | Status | |
|
26 | ====== | |
|
80 | maqroll[ipython]> cd IPython/scripts/ | |
|
81 | maqroll[scripts]> ./iptest -v IPython.utils | |
|
82 | ..........................SS..SSS............................S.S......... | |
|
83 | ................................................... | |
|
84 | ---------------------------------------------------------------------- | |
|
85 | Ran 125 tests in 0.070s | |
|
27 | 86 | |
|
28 | Currently IPython's testing system is being reworked. In the meantime, | |
|
29 | we recommend the following testing practices: | |
|
87 | OK (SKIP=7) | |
|
30 | 88 | |
|
31 | * To run regular tests, use the :command:`nosetests` command that Nose [Nose]_ | |
|
32 | provides on a per file basis: | |
|
89 | Because :file:`iptest` is based on nose, you can use all nose options and | |
|
90 | syntax, typing ``iptest -h`` shows all available options. For example, this | |
|
91 | lets you run the specific test :func:`test_rehashx` inside the | |
|
92 | :mod:`test_magic` module: | |
|
33 | 93 | |
|
34 | 94 | .. code-block:: bash |
|
35 | 95 | |
|
36 |
|
|
|
96 | maqroll[scripts]> ./iptest -vv IPython.core.tests.test_magic:test_rehashx | |
|
97 | IPython.core.tests.test_magic.test_rehashx(True,) ... ok | |
|
98 | IPython.core.tests.test_magic.test_rehashx(True,) ... ok | |
|
37 | 99 | |
|
38 | * To run Twisted-using tests, use the :command:`trial` command on a per file | |
|
39 | basis: | |
|
100 | ---------------------------------------------------------------------- | |
|
101 | Ran 2 tests in 0.101s | |
|
102 | ||
|
103 | OK | |
|
104 | ||
|
105 | When developing, the :option:`--pdb` and :option:`--pdb-failures` of nose are | |
|
106 | particularly useful, these drop you into an interactive pdb session at the | |
|
107 | point of the error or failure respectively. | |
|
108 | ||
|
109 | To run Twisted-using tests, use the :command:`trial` command on a per file or | |
|
110 | package basis: | |
|
40 | 111 | |
|
41 | 112 | .. code-block:: bash |
|
42 | 113 | |
|
43 | 114 | trial IPython.kernel |
|
44 | 115 | |
|
45 | * For now, regular tests (of non-Twisted using code) should be written as | |
|
46 | unit tests. They should be subclasses of :class:`unittest.TestCase`. | |
|
47 | 116 | |
|
48 | * Tests of Twisted [Twisted]_ using code should be written by subclassing the | |
|
49 | ``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, | |
|
50 | all :class:`Deferred` instances that are created in the test must be | |
|
51 | properly chained and the final one *must* be the return value of the test | |
|
52 | method. | |
|
117 | For developers: writing tests | |
|
118 | ============================= | |
|
119 | ||
|
120 | By now IPython has a reasonable test suite, so the best way to see what's | |
|
121 | available is to look at the :file:`tests` directory in most subpackages. But | |
|
122 | here are a few pointers to make the process easier. | |
|
123 | ||
|
124 | ||
|
125 | Main tools: :mod:`IPython.testing` | |
|
126 | ---------------------------------- | |
|
127 | ||
|
128 | The :mod:`IPython.testing` package is where all of the machinery to test | |
|
129 | IPython (rather than the tests for its various parts) lives. In particular, | |
|
130 | the :mod:`iptest` module in there has all the smarts to control the test | |
|
131 | process. In there, the :func:`make_exclude` function is used to build a | |
|
132 | blacklist of exclusions, these are modules that do not get even imported for | |
|
133 | tests. This is important so that things that would fail to even import because | |
|
134 | of missing dependencies don't give errors to end users, as we stated above. | |
|
135 | ||
|
136 | The :mod:`decorators` module contains a lot of useful decorators, especially | |
|
137 | useful to mark individual tests that should be skipped under certain conditions | |
|
138 | (rather than blacklisting the package altogether because of a missing major | |
|
139 | dependency). | |
|
140 | ||
|
141 | Our nose plugin for doctests | |
|
142 | ---------------------------- | |
|
143 | ||
|
144 | The :mod:`plugin` subpackage in testing contains a nose plugin called | |
|
145 | :mod:`ipdoctest` that teaches nose about IPython syntax, so you can write | |
|
146 | doctests with IPython prompts. You can also mark doctest output with ``# | |
|
147 | random`` for the output corresponding to a single input to be ignored (stronger | |
|
148 | than using ellipsis and useful to keep it as an example). If you want the | |
|
149 | entire docstring to be executed but none of the output from any input to be | |
|
150 | checked, you can use the ``# all-random`` marker. The | |
|
151 | :mod:`IPython.testing.plugin.dtexample` module contains examples of how to use | |
|
152 | these; for reference here is how to use ``# random``:: | |
|
153 | ||
|
154 | def ranfunc(): | |
|
155 | """A function with some random output. | |
|
156 | ||
|
157 | Normal examples are verified as usual: | |
|
158 | >>> 1+3 | |
|
159 | 4 | |
|
160 | ||
|
161 | But if you put '# random' in the output, it is ignored: | |
|
162 | >>> 1+3 | |
|
163 | junk goes here... # random | |
|
164 | ||
|
165 | >>> 1+2 | |
|
166 | again, anything goes #random | |
|
167 | if multiline, the random mark is only needed once. | |
|
168 | ||
|
169 | >>> 1+2 | |
|
170 | You can also put the random marker at the end: | |
|
171 | # random | |
|
172 | ||
|
173 | >>> 1+2 | |
|
174 | # random | |
|
175 | .. or at the beginning. | |
|
176 | ||
|
177 | More correct input is properly verified: | |
|
178 | >>> ranfunc() | |
|
179 | 'ranfunc' | |
|
180 | """ | |
|
181 | return 'ranfunc' | |
|
182 | ||
|
183 | and an example of ``# all-random``:: | |
|
184 | ||
|
185 | def random_all(): | |
|
186 | """A function where we ignore the output of ALL examples. | |
|
187 | ||
|
188 | Examples: | |
|
189 | ||
|
190 | # all-random | |
|
191 | ||
|
192 | This mark tells the testing machinery that all subsequent examples | |
|
193 | should be treated as random (ignoring their output). They are still | |
|
194 | executed, so if a they raise an error, it will be detected as such, | |
|
195 | but their output is completely ignored. | |
|
196 | ||
|
197 | >>> 1+3 | |
|
198 | junk goes here... | |
|
199 | ||
|
200 | >>> 1+3 | |
|
201 | klasdfj; | |
|
202 | ||
|
203 | In [8]: print 'hello' | |
|
204 | world # random | |
|
205 | ||
|
206 | In [9]: iprand() | |
|
207 | Out[9]: 'iprand' | |
|
208 | """ | |
|
209 | return 'iprand' | |
|
210 | ||
|
211 | ||
|
212 | When writing docstrings, you can use the ``@skip_doctest`` decorator to | |
|
213 | indicate that a docstring should *not* be treated as a doctest at all. The | |
|
214 | difference betwee ``# all-random`` and ``@skip_doctest`` is that the former | |
|
215 | executes the example but ignores output, while the latter doesn't execute any | |
|
216 | code. ``@skip_doctest`` should be used for docstrings whose examples are | |
|
217 | purely informational. | |
|
218 | ||
|
219 | If a given docstring fails under certain conditions but otherwise is a good | |
|
220 | doctest, you can use code like the following, that relies on the 'null' | |
|
221 | decorator to leave the docstring intact where it works as a test:: | |
|
222 | ||
|
223 | # The docstring for full_path doctests differently on win32 (different path | |
|
224 | # separator) so just skip the doctest there, and use a null decorator | |
|
225 | # elsewhere: | |
|
226 | ||
|
227 | doctest_deco = dec.skip_doctest if sys.platform == 'win32' else dec.null_deco | |
|
228 | ||
|
229 | @doctest_deco | |
|
230 | def full_path(startPath,files): | |
|
231 | """Make full paths for all the listed files, based on startPath...""" | |
|
232 | ||
|
233 | # function body follows... | |
|
234 | ||
|
235 | With our nose plugin that understands IPython syntax, an extremely effective | |
|
236 | way to write tests is to simply copy and paste an interactive session into a | |
|
237 | docstring. You can writing this type of test, where your docstring is meant | |
|
238 | *only* as a test, by prefixing the function name with ``doctest_`` and leaving | |
|
239 | its body *absolutely empty* other than the docstring. In | |
|
240 | :mod:`IPython.core.tests.test_magic` you can find several examples of this, but | |
|
241 | for completeness sake, your code should look like this (a simple case):: | |
|
242 | ||
|
243 | def doctest_time(): | |
|
244 | """ | |
|
245 | In [10]: %time None | |
|
246 | CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s | |
|
247 | Wall time: 0.00 s | |
|
248 | """ | |
|
249 | ||
|
250 | This function is only analyzed for its docstring but it is not considered a | |
|
251 | separate test, which is why its body should be empty. | |
|
252 | ||
|
253 | ||
|
254 | Parametric tests done right | |
|
255 | --------------------------- | |
|
256 | ||
|
257 | If you need to run multiple tests inside the same standalone function or method | |
|
258 | of a :class:`unittest.TestCase` subclass, IPython provides the ``parametric`` | |
|
259 | decorator for this purpose. This is superior to how test generators work in | |
|
260 | nose, because IPython's keeps intact your stack, which makes debugging vastly | |
|
261 | easier. For example, these are some parametric tests both in class form and as | |
|
262 | a standalone function (choose in each situation the style that best fits the | |
|
263 | problem at hand, since both work):: | |
|
264 | ||
|
265 | from IPython.testing import decorators as dec | |
|
266 | ||
|
267 | def is_smaller(i,j): | |
|
268 | assert i<j,"%s !< %s" % (i,j) | |
|
269 | ||
|
270 | class Tester(ParametricTestCase): | |
|
271 | ||
|
272 | def test_parametric(self): | |
|
273 | yield is_smaller(3, 4) | |
|
274 | x, y = 1, 2 | |
|
275 | yield is_smaller(x, y) | |
|
276 | ||
|
277 | @dec.parametric | |
|
278 | def test_par_standalone(): | |
|
279 | yield is_smaller(3, 4) | |
|
280 | x, y = 1, 2 | |
|
281 | yield is_smaller(x, y) | |
|
282 | ||
|
283 | ||
|
284 | Writing tests for Twisted-using code | |
|
285 | ------------------------------------ | |
|
286 | ||
|
287 | Tests of Twisted [Twisted]_ using code should be written by subclassing the | |
|
288 | ``TestCase`` class that comes with ``twisted.trial.unittest``. Furthermore, all | |
|
289 | :class:`Deferred` instances that are created in the test must be properly | |
|
290 | chained and the final one *must* be the return value of the test method. | |
|
291 | ||
|
292 | .. note:: | |
|
293 | ||
|
294 | The best place to see how to use the testing tools, are the tests for these | |
|
295 | tools themselves, which live in :mod:`IPython.testing.tests`. | |
|
296 | ||
|
297 | ||
|
298 | Design requirements | |
|
299 | =================== | |
|
300 | ||
|
301 | This section is a set of notes on the key points of the IPython testing needs, | |
|
302 | that were used when writing the system and should be kept for reference as it | |
|
303 | eveolves. | |
|
304 | ||
|
305 | Testing IPython in full requires modifications to the default behavior of nose | |
|
306 | and doctest, because the IPython prompt is not recognized to determine Python | |
|
307 | input, and because IPython admits user input that is not valid Python (things | |
|
308 | like ``%magics`` and ``!system commands``. | |
|
309 | ||
|
310 | We basically need to be able to test the following types of code: | |
|
311 | ||
|
312 | 1. Pure Python files containing normal tests. These are not a problem, since | |
|
313 | Nose will pick them up as long as they conform to the (flexible) conventions | |
|
314 | used by nose to recognize tests. | |
|
315 | ||
|
316 | 2. Python files containing doctests. Here, we have two possibilities: | |
|
317 | - The prompts are the usual ``>>>`` and the input is pure Python. | |
|
318 | - The prompts are of the form ``In [1]:`` and the input can contain extended | |
|
319 | IPython expressions. | |
|
320 | ||
|
321 | In the first case, Nose will recognize the doctests as long as it is called | |
|
322 | with the ``--with-doctest`` flag. But the second case will likely require | |
|
323 | modifications or the writing of a new doctest plugin for Nose that is | |
|
324 | IPython-aware. | |
|
325 | ||
|
326 | 3. ReStructuredText files that contain code blocks. For this type of file, we | |
|
327 | have three distinct possibilities for the code blocks: | |
|
328 | - They use ``>>>`` prompts. | |
|
329 | - They use ``In [1]:`` prompts. | |
|
330 | - They are standalone blocks of pure Python code without any prompts. | |
|
331 | ||
|
332 | The first two cases are similar to the situation #2 above, except that in | |
|
333 | this case the doctests must be extracted from input code blocks using | |
|
334 | docutils instead of from the Python docstrings. | |
|
335 | ||
|
336 | In the third case, we must have a convention for distinguishing code blocks | |
|
337 | that are meant for execution from others that may be snippets of shell code | |
|
338 | or other examples not meant to be run. One possibility is to assume that | |
|
339 | all indented code blocks are meant for execution, but to have a special | |
|
340 | docutils directive for input that should not be executed. | |
|
341 | ||
|
342 | For those code blocks that we will execute, the convention used will simply | |
|
343 | be that they get called and are considered successful if they run to | |
|
344 | completion without raising errors. This is similar to what Nose does for | |
|
345 | standalone test functions, and by putting asserts or other forms of | |
|
346 | exception-raising statements it becomes possible to have literate examples | |
|
347 | that double as lightweight tests. | |
|
348 | ||
|
349 | 4. Extension modules with doctests in function and method docstrings. | |
|
350 | Currently Nose simply can't find these docstrings correctly, because the | |
|
351 | underlying doctest DocTestFinder object fails there. Similarly to #2 above, | |
|
352 | the docstrings could have either pure python or IPython prompts. | |
|
53 | 353 | |
|
54 | .. [Nose] Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/ | |
|
354 | Of these, only 3-c (reST with standalone code blocks) is not implemented at | |
|
355 | this point. |
General Comments 0
You need to be logged in to leave comments.
Login now