##// END OF EJS Templates
Final documentation changes from code review with Brian, ready to merge.
Fernando Perez -
Show More
@@ -1,817 +1,839 b''
1 1 """Analysis of text input into executable blocks.
2 2
3 3 The main class in this module, :class:`InputSplitter`, is designed to break
4 4 input from either interactive, line-by-line environments or block-based ones,
5 5 into standalone blocks that can be executed by Python as 'single' statements
6 6 (thus triggering sys.displayhook).
7 7
8 8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 9 with full support for the extended IPython syntax (magics, system calls, etc).
10 10
11 11 For more details, see the class docstring below.
12 12
13 Syntax Transformations
14 ----------------------
15
16 One of the main jobs of the code in this file is to apply all syntax
17 transformations that make up 'the IPython language', i.e. magics, shell
18 escapes, etc. All transformations should be implemented as *fully stateless*
19 entities, that simply take one line as their input and return a line.
20 Internally for implementation purposes they may be a normal function or a
21 callable object, but the only input they receive will be a single line and they
22 should only return a line, without holding any data-dependent state between
23 calls.
24
25 As an example, the EscapedTransformer is a class so we can more clearly group
26 together the functionality of dispatching to individual functions based on the
27 starting escape character, but the only method for public use is its call
28 method.
29
30
13 31 ToDo
14 32 ----
15 33
34 - Should we make push() actually raise an exception once push_accepts_more()
35 returns False?
36
16 37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
17 38 at least just attributes of a class so not really very exposed.
18 39
19 40 - Think about the best way to support dynamic things: automagic, autocall,
20 41 macros, etc.
21 42
22 43 - Think of a better heuristic for the application of the transforms in
23 44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
24 45 track indentation change events (indent, dedent, nothing) and apply them only
25 46 if the indentation went up, but not otherwise.
26 47
27 48 - Think of the cleanest way for supporting user-specified transformations (the
28 49 user prefilters we had before).
29 50
30 51 Authors
31 52 -------
32 53
33 54 * Fernando Perez
34 55 * Brian Granger
35 56 """
36 57 #-----------------------------------------------------------------------------
37 58 # Copyright (C) 2010 The IPython Development Team
38 59 #
39 60 # Distributed under the terms of the BSD License. The full license is in
40 61 # the file COPYING, distributed as part of this software.
41 62 #-----------------------------------------------------------------------------
42 63
43 64 #-----------------------------------------------------------------------------
44 65 # Imports
45 66 #-----------------------------------------------------------------------------
46 67 # stdlib
47 68 import codeop
48 69 import re
49 70 import sys
50 71
51 72 # IPython modules
52 73 from IPython.utils.text import make_quoted_expr
53 74
54 75 #-----------------------------------------------------------------------------
55 76 # Globals
56 77 #-----------------------------------------------------------------------------
57 78
58 79 # The escape sequences that define the syntax transformations IPython will
59 80 # apply to user input. These can NOT be just changed here: many regular
60 81 # expressions and other parts of the code may use their hardcoded values, and
61 82 # for all intents and purposes they constitute the 'IPython syntax', so they
62 83 # should be considered fixed.
63 84
64 85 ESC_SHELL = '!'
65 86 ESC_SH_CAP = '!!'
66 87 ESC_HELP = '?'
67 88 ESC_HELP2 = '??'
68 89 ESC_MAGIC = '%'
69 90 ESC_QUOTE = ','
70 91 ESC_QUOTE2 = ';'
71 92 ESC_PAREN = '/'
72 93
73 94 #-----------------------------------------------------------------------------
74 95 # Utilities
75 96 #-----------------------------------------------------------------------------
76 97
77 98 # FIXME: These are general-purpose utilities that later can be moved to the
78 99 # general ward. Kept here for now because we're being very strict about test
79 100 # coverage with this code, and this lets us ensure that we keep 100% coverage
80 101 # while developing.
81 102
82 103 # compiled regexps for autoindent management
83 104 dedent_re = re.compile(r'^\s+raise|^\s+return|^\s+pass')
84 105 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
85 106
86 107
87 108 def num_ini_spaces(s):
88 109 """Return the number of initial spaces in a string.
89 110
90 111 Note that tabs are counted as a single space. For now, we do *not* support
91 112 mixing of tabs and spaces in the user's input.
92 113
93 114 Parameters
94 115 ----------
95 116 s : string
96 117
97 118 Returns
98 119 -------
99 120 n : int
100 121 """
101 122
102 123 ini_spaces = ini_spaces_re.match(s)
103 124 if ini_spaces:
104 125 return ini_spaces.end()
105 126 else:
106 127 return 0
107 128
108 129
109 130 def remove_comments(src):
110 131 """Remove all comments from input source.
111 132
112 133 Note: comments are NOT recognized inside of strings!
113 134
114 135 Parameters
115 136 ----------
116 137 src : string
117 138 A single or multiline input string.
118 139
119 140 Returns
120 141 -------
121 142 String with all Python comments removed.
122 143 """
123 144
124 145 return re.sub('#.*', '', src)
125 146
126 147
127 148 def get_input_encoding():
128 149 """Return the default standard input encoding.
129 150
130 151 If sys.stdin has no encoding, 'ascii' is returned."""
131 152 # There are strange environments for which sys.stdin.encoding is None. We
132 153 # ensure that a valid encoding is returned.
133 154 encoding = getattr(sys.stdin, 'encoding', None)
134 155 if encoding is None:
135 156 encoding = 'ascii'
136 157 return encoding
137 158
138 159 #-----------------------------------------------------------------------------
139 160 # Classes and functions for normal Python syntax handling
140 161 #-----------------------------------------------------------------------------
141 162
142 163 class InputSplitter(object):
143 164 """An object that can split Python source input in executable blocks.
144 165
145 166 This object is designed to be used in one of two basic modes:
146 167
147 168 1. By feeding it python source line-by-line, using :meth:`push`. In this
148 169 mode, it will return on each push whether the currently pushed code
149 170 could be executed already. In addition, it provides a method called
150 171 :meth:`push_accepts_more` that can be used to query whether more input
151 172 can be pushed into a single interactive block.
152 173
153 174 2. By calling :meth:`split_blocks` with a single, multiline Python string,
154 175 that is then split into blocks each of which can be executed
155 176 interactively as a single statement.
156 177
157 178 This is a simple example of how an interactive terminal-based client can use
158 179 this tool::
159 180
160 181 isp = InputSplitter()
161 182 while isp.push_accepts_more():
162 183 indent = ' '*isp.indent_spaces
163 184 prompt = '>>> ' + indent
164 185 line = indent + raw_input(prompt)
165 186 isp.push(line)
166 187 print 'Input source was:\n', isp.source_reset(),
167 188 """
168 189 # Number of spaces of indentation computed from input that has been pushed
169 190 # so far. This is the attributes callers should query to get the current
170 191 # indentation level, in order to provide auto-indent facilities.
171 192 indent_spaces = 0
172 193 # String, indicating the default input encoding. It is computed by default
173 194 # at initialization time via get_input_encoding(), but it can be reset by a
174 195 # client with specific knowledge of the encoding.
175 196 encoding = ''
176 197 # String where the current full source input is stored, properly encoded.
177 198 # Reading this attribute is the normal way of querying the currently pushed
178 199 # source code, that has been properly encoded.
179 200 source = ''
180 201 # Code object corresponding to the current source. It is automatically
181 202 # synced to the source, so it can be queried at any time to obtain the code
182 203 # object; it will be None if the source doesn't compile to valid Python.
183 204 code = None
184 205 # Input mode
185 206 input_mode = 'append'
186 207
187 208 # Private attributes
188 209
189 210 # List with lines of input accumulated so far
190 211 _buffer = None
191 212 # Command compiler
192 213 _compile = None
193 214 # Mark when input has changed indentation all the way back to flush-left
194 215 _full_dedent = False
195 216 # Boolean indicating whether the current block is complete
196 217 _is_complete = None
197 218
198 219 def __init__(self, input_mode=None):
199 220 """Create a new InputSplitter instance.
200 221
201 222 Parameters
202 223 ----------
203 224 input_mode : str
204 225
205 226 One of 'append', 'replace', default is 'append'. This controls how
206 227 new inputs are used: in 'append' mode, they are appended to the
207 228 existing buffer and the whole buffer is compiled; in 'replace' mode,
208 229 each new input completely replaces all prior inputs. Replace mode is
209 230 thus equivalent to prepending a full reset() to every push() call.
210 231
211 232 In practice, line-oriented clients likely want to use 'append' mode
212 233 while block-oriented ones will want to use 'replace'.
213 234 """
214 235 self._buffer = []
215 236 self._compile = codeop.CommandCompiler()
216 237 self.encoding = get_input_encoding()
217 238 self.input_mode = InputSplitter.input_mode if input_mode is None \
218 239 else input_mode
219 240
220 241 def reset(self):
221 242 """Reset the input buffer and associated state."""
222 243 self.indent_spaces = 0
223 244 self._buffer[:] = []
224 245 self.source = ''
225 246 self.code = None
226 247 self._is_complete = False
227 248 self._full_dedent = False
228 249
229 250 def source_reset(self):
230 251 """Return the input source and perform a full reset.
231 252 """
232 253 out = self.source
233 254 self.reset()
234 255 return out
235 256
236 257 def push(self, lines):
237 258 """Push one ore more lines of input.
238 259
239 260 This stores the given lines and returns a status code indicating
240 261 whether the code forms a complete Python block or not.
241 262
242 263 Any exceptions generated in compilation are swallowed, but if an
243 264 exception was produced, the method returns True.
244 265
245 266 Parameters
246 267 ----------
247 268 lines : string
248 269 One or more lines of Python input.
249 270
250 271 Returns
251 272 -------
252 273 is_complete : boolean
253 274 True if the current input source (the result of the current input
254 275 plus prior inputs) forms a complete Python execution block. Note that
255 276 this value is also stored as a private attribute (_is_complete), so it
256 277 can be queried at any time.
257 278 """
258 279 if self.input_mode == 'replace':
259 280 self.reset()
260 281
261 282 # If the source code has leading blanks, add 'if 1:\n' to it
262 283 # this allows execution of indented pasted code. It is tempting
263 284 # to add '\n' at the end of source to run commands like ' a=1'
264 285 # directly, but this fails for more complicated scenarios
265 286 if not self._buffer and lines[:1] in [' ', '\t']:
266 287 lines = 'if 1:\n%s' % lines
267 288
268 289 self._store(lines)
269 290 source = self.source
270 291
271 292 # Before calling _compile(), reset the code object to None so that if an
272 293 # exception is raised in compilation, we don't mislead by having
273 294 # inconsistent code/source attributes.
274 295 self.code, self._is_complete = None, None
275 296
276 297 self._update_indent(lines)
277 298 try:
278 299 self.code = self._compile(source)
279 300 # Invalid syntax can produce any of a number of different errors from
280 301 # inside the compiler, so we have to catch them all. Syntax errors
281 302 # immediately produce a 'ready' block, so the invalid Python can be
282 303 # sent to the kernel for evaluation with possible ipython
283 304 # special-syntax conversion.
284 305 except (SyntaxError, OverflowError, ValueError, TypeError,
285 306 MemoryError):
286 307 self._is_complete = True
287 308 else:
288 309 # Compilation didn't produce any exceptions (though it may not have
289 310 # given a complete code object)
290 311 self._is_complete = self.code is not None
291 312
292 313 return self._is_complete
293 314
294 315 def push_accepts_more(self):
295 316 """Return whether a block of interactive input can accept more input.
296 317
297 318 This method is meant to be used by line-oriented frontends, who need to
298 319 guess whether a block is complete or not based solely on prior and
299 320 current input lines. The InputSplitter considers it has a complete
300 321 interactive block and will not accept more input only when either a
301 322 SyntaxError is raised, or *all* of the following are true:
302 323
303 324 1. The input compiles to a complete statement.
304 325
305 326 2. The indentation level is flush-left (because if we are indented,
306 327 like inside a function definition or for loop, we need to keep
307 328 reading new input).
308 329
309 330 3. There is one extra line consisting only of whitespace.
310 331
311 332 Because of condition #3, this method should be used only by
312 333 *line-oriented* frontends, since it means that intermediate blank lines
313 334 are not allowed in function definitions (or any other indented block).
314 335
315 336 Block-oriented frontends that have a separate keyboard event to
316 337 indicate execution should use the :meth:`split_blocks` method instead.
317 338
318 339 If the current input produces a syntax error, this method immediately
319 340 returns False but does *not* raise the syntax error exception, as
320 341 typically clients will want to send invalid syntax to an execution
321 342 backend which might convert the invalid syntax into valid Python via
322 343 one of the dynamic IPython mechanisms.
323 344 """
324 345
325 346 if not self._is_complete:
326 347 return True
327 348
328 349 if self.indent_spaces==0:
329 350 return False
330 351
331 352 last_line = self.source.splitlines()[-1]
332 353 return bool(last_line and not last_line.isspace())
333 354
334 355 def split_blocks(self, lines):
335 356 """Split a multiline string into multiple input blocks.
336 357
337 358 Note: this method starts by performing a full reset().
338 359
339 360 Parameters
340 361 ----------
341 362 lines : str
342 363 A possibly multiline string.
343 364
344 365 Returns
345 366 -------
346 367 blocks : list
347 368 A list of strings, each possibly multiline. Each string corresponds
348 369 to a single block that can be compiled in 'single' mode (unless it
349 370 has a syntax error)."""
350 371
351 372 # This code is fairly delicate. If you make any changes here, make
352 373 # absolutely sure that you do run the full test suite and ALL tests
353 374 # pass.
354 375
355 376 self.reset()
356 377 blocks = []
357 378
358 379 # Reversed copy so we can use pop() efficiently and consume the input
359 380 # as a stack
360 381 lines = lines.splitlines()[::-1]
361 382 # Outer loop over all input
362 383 while lines:
363 384 # Inner loop to build each block
364 385 while True:
365 386 # Safety exit from inner loop
366 387 if not lines:
367 388 break
368 389 # Grab next line but don't push it yet
369 390 next_line = lines.pop()
370 391 # Blank/empty lines are pushed as-is
371 392 if not next_line or next_line.isspace():
372 393 self.push(next_line)
373 394 continue
374 395
375 396 # Check indentation changes caused by the *next* line
376 397 indent_spaces, _full_dedent = self._find_indent(next_line)
377 398
378 399 # If the next line causes a dedent, it can be for two differnt
379 400 # reasons: either an explicit de-dent by the user or a
380 401 # return/raise/pass statement. These MUST be handled
381 402 # separately:
382 403 #
383 404 # 1. the first case is only detected when the actual explicit
384 405 # dedent happens, and that would be the *first* line of a *new*
385 406 # block. Thus, we must put the line back into the input buffer
386 407 # so that it starts a new block on the next pass.
387 408 #
388 409 # 2. the second case is detected in the line before the actual
389 410 # dedent happens, so , we consume the line and we can break out
390 411 # to start a new block.
391 412
392 413 # Case 1, explicit dedent causes a break
393 414 if _full_dedent and not next_line.startswith(' '):
394 415 lines.append(next_line)
395 416 break
396 417
397 418 # Otherwise any line is pushed
398 419 self.push(next_line)
399 420
400 421 # Case 2, full dedent with full block ready:
401 422 if _full_dedent or \
402 423 self.indent_spaces==0 and not self.push_accepts_more():
403 424 break
404 425 # Form the new block with the current source input
405 426 blocks.append(self.source_reset())
406 427
407 428 return blocks
408 429
409 430 #------------------------------------------------------------------------
410 431 # Private interface
411 432 #------------------------------------------------------------------------
412 433
413 434 def _find_indent(self, line):
414 435 """Compute the new indentation level for a single line.
415 436
416 437 Parameters
417 438 ----------
418 439 line : str
419 440 A single new line of non-whitespace, non-comment Python input.
420 441
421 442 Returns
422 443 -------
423 444 indent_spaces : int
424 445 New value for the indent level (it may be equal to self.indent_spaces
425 446 if indentation doesn't change.
426 447
427 448 full_dedent : boolean
428 449 Whether the new line causes a full flush-left dedent.
429 450 """
430 451 indent_spaces = self.indent_spaces
431 452 full_dedent = self._full_dedent
432 453
433 454 inisp = num_ini_spaces(line)
434 455 if inisp < indent_spaces:
435 456 indent_spaces = inisp
436 457 if indent_spaces <= 0:
437 458 #print 'Full dedent in text',self.source # dbg
438 459 full_dedent = True
439 460
440 461 if line[-1] == ':':
441 462 indent_spaces += 4
442 463 elif dedent_re.match(line):
443 464 indent_spaces -= 4
444 465 if indent_spaces <= 0:
445 466 full_dedent = True
446 467
447 468 # Safety
448 469 if indent_spaces < 0:
449 470 indent_spaces = 0
450 471 #print 'safety' # dbg
451 472
452 473 return indent_spaces, full_dedent
453 474
454 475 def _update_indent(self, lines):
455 476 for line in remove_comments(lines).splitlines():
456 477 if line and not line.isspace():
457 478 self.indent_spaces, self._full_dedent = self._find_indent(line)
458 479
459 480 def _store(self, lines):
460 481 """Store one or more lines of input.
461 482
462 483 If input lines are not newline-terminated, a newline is automatically
463 484 appended."""
464 485
465 486 if lines.endswith('\n'):
466 487 self._buffer.append(lines)
467 488 else:
468 489 self._buffer.append(lines+'\n')
469 490 self._set_source()
470 491
471 492 def _set_source(self):
472 493 self.source = ''.join(self._buffer).encode(self.encoding)
473 494
474 495
475 496 #-----------------------------------------------------------------------------
476 497 # Functions and classes for IPython-specific syntactic support
477 498 #-----------------------------------------------------------------------------
478 499
479 500 # RegExp for splitting line contents into pre-char//first word-method//rest.
480 501 # For clarity, each group in on one line.
481 502
482 503 line_split = re.compile("""
483 504 ^(\s*) # any leading space
484 505 ([,;/%]|!!?|\?\??) # escape character or characters
485 506 \s*([\w\.]*) # function/method part (mix of \w and '.')
486 507 (\s+.*$|$) # rest of line
487 508 """, re.VERBOSE)
488 509
489 510
490 511 def split_user_input(line):
491 512 """Split user input into early whitespace, esc-char, function part and rest.
492 513
493 514 This is currently handles lines with '=' in them in a very inconsistent
494 515 manner.
495 516
496 517 Examples
497 518 ========
498 519 >>> split_user_input('x=1')
499 520 ('', '', 'x=1', '')
500 521 >>> split_user_input('?')
501 522 ('', '?', '', '')
502 523 >>> split_user_input('??')
503 524 ('', '??', '', '')
504 525 >>> split_user_input(' ?')
505 526 (' ', '?', '', '')
506 527 >>> split_user_input(' ??')
507 528 (' ', '??', '', '')
508 529 >>> split_user_input('??x')
509 530 ('', '??', 'x', '')
510 531 >>> split_user_input('?x=1')
511 532 ('', '', '?x=1', '')
512 533 >>> split_user_input('!ls')
513 534 ('', '!', 'ls', '')
514 535 >>> split_user_input(' !ls')
515 536 (' ', '!', 'ls', '')
516 537 >>> split_user_input('!!ls')
517 538 ('', '!!', 'ls', '')
518 539 >>> split_user_input(' !!ls')
519 540 (' ', '!!', 'ls', '')
520 541 >>> split_user_input(',ls')
521 542 ('', ',', 'ls', '')
522 543 >>> split_user_input(';ls')
523 544 ('', ';', 'ls', '')
524 545 >>> split_user_input(' ;ls')
525 546 (' ', ';', 'ls', '')
526 547 >>> split_user_input('f.g(x)')
527 548 ('', '', 'f.g(x)', '')
528 549 >>> split_user_input('f.g (x)')
529 550 ('', '', 'f.g', '(x)')
530 551 """
531 552 match = line_split.match(line)
532 553 if match:
533 554 lspace, esc, fpart, rest = match.groups()
534 555 else:
535 556 # print "match failed for line '%s'" % line
536 557 try:
537 558 fpart, rest = line.split(None, 1)
538 559 except ValueError:
539 560 # print "split failed for line '%s'" % line
540 561 fpart, rest = line,''
541 562 lspace = re.match('^(\s*)(.*)', line).groups()[0]
542 563 esc = ''
543 564
544 565 # fpart has to be a valid python identifier, so it better be only pure
545 566 # ascii, no unicode:
546 567 try:
547 568 fpart = fpart.encode('ascii')
548 569 except UnicodeEncodeError:
549 570 lspace = unicode(lspace)
550 571 rest = fpart + u' ' + rest
551 572 fpart = u''
552 573
553 574 #print 'line:<%s>' % line # dbg
554 575 #print 'esc <%s> fpart <%s> rest <%s>' % (esc,fpart.strip(),rest) # dbg
555 576 return lspace, esc, fpart.strip(), rest.lstrip()
556 577
557 578
558 579 # The escaped translators ALL receive a line where their own escape has been
559 580 # stripped. Only '?' is valid at the end of the line, all others can only be
560 581 # placed at the start.
561 582
562 583 class LineInfo(object):
563 584 """A single line of input and associated info.
564 585
565 586 This is a utility class that mostly wraps the output of
566 587 :func:`split_user_input` into a convenient object to be passed around
567 588 during input transformations.
568 589
569 590 Includes the following as properties:
570 591
571 592 line
572 593 The original, raw line
573 594
574 595 lspace
575 596 Any early whitespace before actual text starts.
576 597
577 598 esc
578 599 The initial esc character (or characters, for double-char escapes like
579 600 '??' or '!!').
580 601
581 602 fpart
582 603 The 'function part', which is basically the maximal initial sequence
583 604 of valid python identifiers and the '.' character. This is what is
584 605 checked for alias and magic transformations, used for auto-calling,
585 606 etc.
586 607
587 608 rest
588 609 Everything else on the line.
589 610 """
590 611 def __init__(self, line):
591 612 self.line = line
592 613 self.lspace, self.esc, self.fpart, self.rest = \
593 614 split_user_input(line)
594 615
595 616 def __str__(self):
596 617 return "LineInfo [%s|%s|%s|%s]" % (self.lspace, self.esc,
597 618 self.fpart, self.rest)
598 619
599 620
600 621 # Transformations of the special syntaxes that don't rely on an explicit escape
601 622 # character but instead on patterns on the input line
602 623
603 624 # The core transformations are implemented as standalone functions that can be
604 625 # tested and validated in isolation. Each of these uses a regexp, we
605 626 # pre-compile these and keep them close to each function definition for clarity
606 627
607 628 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
608 629 r'\s*=\s*!\s*(?P<cmd>.*)')
609 630
610 631 def transform_assign_system(line):
611 632 """Handle the `files = !ls` syntax."""
612 633 # FIXME: This transforms the line to use %sc, but we've listed that magic
613 634 # as deprecated. We should then implement this functionality in a
614 635 # standalone api that we can transform to, without going through a
615 636 # deprecated magic.
616 637 m = _assign_system_re.match(line)
617 638 if m is not None:
618 639 cmd = m.group('cmd')
619 640 lhs = m.group('lhs')
620 641 expr = make_quoted_expr("sc -l = %s" % cmd)
621 642 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
622 643 return new_line
623 644 return line
624 645
625 646
626 647 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
627 648 r'\s*=\s*%\s*(?P<cmd>.*)')
628 649
629 650 def transform_assign_magic(line):
630 651 """Handle the `a = %who` syntax."""
631 652 m = _assign_magic_re.match(line)
632 653 if m is not None:
633 654 cmd = m.group('cmd')
634 655 lhs = m.group('lhs')
635 656 expr = make_quoted_expr(cmd)
636 657 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
637 658 return new_line
638 659 return line
639 660
640 661
641 662 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
642 663
643 664 def transform_classic_prompt(line):
644 665 """Handle inputs that start with '>>> ' syntax."""
645 666
646 667 if not line or line.isspace():
647 668 return line
648 669 m = _classic_prompt_re.match(line)
649 670 if m:
650 671 return line[len(m.group(0)):]
651 672 else:
652 673 return line
653 674
654 675
655 676 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
656 677
657 678 def transform_ipy_prompt(line):
658 679 """Handle inputs that start classic IPython prompt syntax."""
659 680
660 681 if not line or line.isspace():
661 682 return line
662 683 m = _ipy_prompt_re.match(line)
663 684 if m:
664 685 return line[len(m.group(0)):]
665 686 else:
666 687 return line
667 688
668 689
669 690 class EscapedTransformer(object):
670 691 """Class to transform lines that are explicitly escaped out."""
671 692
672 693 def __init__(self):
673 tr = { ESC_SHELL : self.tr_system,
674 ESC_SH_CAP : self.tr_system2,
675 ESC_HELP : self.tr_help,
676 ESC_HELP2 : self.tr_help,
677 ESC_MAGIC : self.tr_magic,
678 ESC_QUOTE : self.tr_quote,
679 ESC_QUOTE2 : self.tr_quote2,
680 ESC_PAREN : self.tr_paren }
694 tr = { ESC_SHELL : self._tr_system,
695 ESC_SH_CAP : self._tr_system2,
696 ESC_HELP : self._tr_help,
697 ESC_HELP2 : self._tr_help,
698 ESC_MAGIC : self._tr_magic,
699 ESC_QUOTE : self._tr_quote,
700 ESC_QUOTE2 : self._tr_quote2,
701 ESC_PAREN : self._tr_paren }
681 702 self.tr = tr
682 703
683 704 # Support for syntax transformations that use explicit escapes typed by the
684 705 # user at the beginning of a line
685 706 @staticmethod
686 def tr_system(line_info):
707 def _tr_system(line_info):
687 708 "Translate lines escaped with: !"
688 709 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
689 710 return '%sget_ipython().system(%s)' % (line_info.lspace,
690 711 make_quoted_expr(cmd))
691 712
692 713 @staticmethod
693 def tr_system2(line_info):
714 def _tr_system2(line_info):
694 715 "Translate lines escaped with: !!"
695 716 cmd = line_info.line.lstrip()[2:]
696 717 return '%sget_ipython().getoutput(%s)' % (line_info.lspace,
697 718 make_quoted_expr(cmd))
698 719
699 720 @staticmethod
700 def tr_help(line_info):
721 def _tr_help(line_info):
701 722 "Translate lines escaped with: ?/??"
702 723 # A naked help line should just fire the intro help screen
703 724 if not line_info.line[1:]:
704 725 return 'get_ipython().show_usage()'
705 726
706 727 # There may be one or two '?' at the end, move them to the front so that
707 728 # the rest of the logic can assume escapes are at the start
708 729 line = line_info.line
709 730 if line.endswith('?'):
710 731 line = line[-1] + line[:-1]
711 732 if line.endswith('?'):
712 733 line = line[-1] + line[:-1]
713 734 line_info = LineInfo(line)
714 735
715 736 # From here on, simply choose which level of detail to get.
716 737 if line_info.esc == '?':
717 738 pinfo = 'pinfo'
718 739 elif line_info.esc == '??':
719 740 pinfo = 'pinfo2'
720 741
721 742 tpl = '%sget_ipython().magic("%s %s")'
722 743 return tpl % (line_info.lspace, pinfo,
723 744 ' '.join([line_info.fpart, line_info.rest]).strip())
724 745
725 746 @staticmethod
726 def tr_magic(line_info):
747 def _tr_magic(line_info):
727 748 "Translate lines escaped with: %"
728 749 tpl = '%sget_ipython().magic(%s)'
729 750 cmd = make_quoted_expr(' '.join([line_info.fpart,
730 751 line_info.rest]).strip())
731 752 return tpl % (line_info.lspace, cmd)
732 753
733 754 @staticmethod
734 def tr_quote(line_info):
755 def _tr_quote(line_info):
735 756 "Translate lines escaped with: ,"
736 757 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
737 758 '", "'.join(line_info.rest.split()) )
738 759
739 760 @staticmethod
740 def tr_quote2(line_info):
761 def _tr_quote2(line_info):
741 762 "Translate lines escaped with: ;"
742 763 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
743 764 line_info.rest)
744 765
745 766 @staticmethod
746 def tr_paren(line_info):
767 def _tr_paren(line_info):
747 768 "Translate lines escaped with: /"
748 769 return '%s%s(%s)' % (line_info.lspace, line_info.fpart,
749 770 ", ".join(line_info.rest.split()))
750 771
751 772 def __call__(self, line):
752 773 """Class to transform lines that are explicitly escaped out.
753 774
754 This calls the above tr_* static methods for the actual line
775 This calls the above _tr_* static methods for the actual line
755 776 translations."""
756 777
757 778 # Empty lines just get returned unmodified
758 779 if not line or line.isspace():
759 780 return line
760 781
761 782 # Get line endpoints, where the escapes can be
762 783 line_info = LineInfo(line)
763 784
764 785 # If the escape is not at the start, only '?' needs to be special-cased.
765 786 # All other escapes are only valid at the start
766 787 if not line_info.esc in self.tr:
767 788 if line.endswith(ESC_HELP):
768 return self.tr_help(line_info)
789 return self._tr_help(line_info)
769 790 else:
770 791 # If we don't recognize the escape, don't modify the line
771 792 return line
772 793
773 794 return self.tr[line_info.esc](line_info)
774 795
796
775 797 # A function-looking object to be used by the rest of the code. The purpose of
776 798 # the class in this case is to organize related functionality, more than to
777 799 # manage state.
778 800 transform_escaped = EscapedTransformer()
779 801
780 802
781 803 class IPythonInputSplitter(InputSplitter):
782 804 """An input splitter that recognizes all of IPython's special syntax."""
783 805
784 806 def push(self, lines):
785 807 """Push one or more lines of IPython input.
786 808 """
787 809 if not lines:
788 810 return super(IPythonInputSplitter, self).push(lines)
789 811
790 812 lines_list = lines.splitlines()
791 813
792 814 transforms = [transform_escaped, transform_assign_system,
793 815 transform_assign_magic, transform_ipy_prompt,
794 816 transform_classic_prompt]
795 817
796 818 # Transform logic
797 819 #
798 820 # We only apply the line transformers to the input if we have either no
799 821 # input yet, or complete input, or if the last line of the buffer ends
800 822 # with ':' (opening an indented block). This prevents the accidental
801 823 # transformation of escapes inside multiline expressions like
802 824 # triple-quoted strings or parenthesized expressions.
803 825 #
804 826 # The last heuristic, while ugly, ensures that the first line of an
805 827 # indented block is correctly transformed.
806 828 #
807 829 # FIXME: try to find a cleaner approach for this last bit.
808 830
809 831 for line in lines_list:
810 832 if self._is_complete or not self._buffer or \
811 833 (self._buffer and self._buffer[-1].rstrip().endswith(':')):
812 834 for f in transforms:
813 835 line = f(line)
814 836
815 837 out = super(IPythonInputSplitter, self).push(line)
816 838
817 839 return out
@@ -1,621 +1,622 b''
1 1 # -*- coding: utf-8 -*-
2 2 """Tests for the inputsplitter module.
3 3 """
4 4 #-----------------------------------------------------------------------------
5 5 # Copyright (C) 2010 The IPython Development Team
6 6 #
7 7 # Distributed under the terms of the BSD License. The full license is in
8 8 # the file COPYING, distributed as part of this software.
9 9 #-----------------------------------------------------------------------------
10 10
11 11 #-----------------------------------------------------------------------------
12 12 # Imports
13 13 #-----------------------------------------------------------------------------
14 14 # stdlib
15 15 import unittest
16 16 import sys
17 17
18 18 # Third party
19 19 import nose.tools as nt
20 20
21 21 # Our own
22 22 from IPython.core import inputsplitter as isp
23 23
24 24 #-----------------------------------------------------------------------------
25 25 # Semi-complete examples (also used as tests)
26 26 #-----------------------------------------------------------------------------
27 27
28 28 # Note: at the bottom, there's a slightly more complete version of this that
29 29 # can be useful during development of code here.
30 30
31 31 def mini_interactive_loop(raw_input):
32 32 """Minimal example of the logic of an interactive interpreter loop.
33 33
34 34 This serves as an example, and it is used by the test system with a fake
35 35 raw_input that simulates interactive input."""
36 36
37 37 from IPython.core.inputsplitter import InputSplitter
38 38
39 39 isp = InputSplitter()
40 40 # In practice, this input loop would be wrapped in an outside loop to read
41 41 # input indefinitely, until some exit/quit command was issued. Here we
42 42 # only illustrate the basic inner loop.
43 43 while isp.push_accepts_more():
44 44 indent = ' '*isp.indent_spaces
45 45 prompt = '>>> ' + indent
46 46 line = indent + raw_input(prompt)
47 47 isp.push(line)
48 48
49 49 # Here we just return input so we can use it in a test suite, but a real
50 50 # interpreter would instead send it for execution somewhere.
51 51 src = isp.source_reset()
52 52 #print 'Input source was:\n', src # dbg
53 53 return src
54 54
55 55 #-----------------------------------------------------------------------------
56 56 # Test utilities, just for local use
57 57 #-----------------------------------------------------------------------------
58 58
59 59 def assemble(block):
60 60 """Assemble a block into multi-line sub-blocks."""
61 61 return ['\n'.join(sub_block)+'\n' for sub_block in block]
62 62
63 63
64 64 def pseudo_input(lines):
65 65 """Return a function that acts like raw_input but feeds the input list."""
66 66 ilines = iter(lines)
67 67 def raw_in(prompt):
68 68 try:
69 69 return next(ilines)
70 70 except StopIteration:
71 71 return ''
72 72 return raw_in
73 73
74 74 #-----------------------------------------------------------------------------
75 75 # Tests
76 76 #-----------------------------------------------------------------------------
77 77 def test_spaces():
78 78 tests = [('', 0),
79 79 (' ', 1),
80 80 ('\n', 0),
81 81 (' \n', 1),
82 82 ('x', 0),
83 83 (' x', 1),
84 84 (' x',2),
85 85 (' x',4),
86 86 # Note: tabs are counted as a single whitespace!
87 87 ('\tx', 1),
88 88 ('\t x', 2),
89 89 ]
90 90
91 91 for s, nsp in tests:
92 92 nt.assert_equal(isp.num_ini_spaces(s), nsp)
93 93
94 94
95 95 def test_remove_comments():
96 96 tests = [('text', 'text'),
97 97 ('text # comment', 'text '),
98 98 ('text # comment\n', 'text \n'),
99 99 ('text # comment \n', 'text \n'),
100 100 ('line # c \nline\n','line \nline\n'),
101 101 ('line # c \nline#c2 \nline\nline #c\n\n',
102 102 'line \nline\nline\nline \n\n'),
103 103 ]
104 104
105 105 for inp, out in tests:
106 106 nt.assert_equal(isp.remove_comments(inp), out)
107 107
108 108
109 109 def test_get_input_encoding():
110 110 encoding = isp.get_input_encoding()
111 111 nt.assert_true(isinstance(encoding, basestring))
112 112 # simple-minded check that at least encoding a simple string works with the
113 113 # encoding we got.
114 114 nt.assert_equal('test'.encode(encoding), 'test')
115 115
116 116
117 117 class NoInputEncodingTestCase(unittest.TestCase):
118 118 def setUp(self):
119 119 self.old_stdin = sys.stdin
120 120 class X: pass
121 121 fake_stdin = X()
122 122 sys.stdin = fake_stdin
123 123
124 124 def test(self):
125 125 # Verify that if sys.stdin has no 'encoding' attribute we do the right
126 126 # thing
127 127 enc = isp.get_input_encoding()
128 128 self.assertEqual(enc, 'ascii')
129 129
130 130 def tearDown(self):
131 131 sys.stdin = self.old_stdin
132 132
133 133
134 134 class InputSplitterTestCase(unittest.TestCase):
135 135 def setUp(self):
136 136 self.isp = isp.InputSplitter()
137 137
138 138 def test_reset(self):
139 139 isp = self.isp
140 140 isp.push('x=1')
141 141 isp.reset()
142 142 self.assertEqual(isp._buffer, [])
143 143 self.assertEqual(isp.indent_spaces, 0)
144 144 self.assertEqual(isp.source, '')
145 145 self.assertEqual(isp.code, None)
146 146 self.assertEqual(isp._is_complete, False)
147 147
148 148 def test_source(self):
149 149 self.isp._store('1')
150 150 self.isp._store('2')
151 151 self.assertEqual(self.isp.source, '1\n2\n')
152 152 self.assertTrue(len(self.isp._buffer)>0)
153 153 self.assertEqual(self.isp.source_reset(), '1\n2\n')
154 154 self.assertEqual(self.isp._buffer, [])
155 155 self.assertEqual(self.isp.source, '')
156 156
157 157 def test_indent(self):
158 158 isp = self.isp # shorthand
159 159 isp.push('x=1')
160 160 self.assertEqual(isp.indent_spaces, 0)
161 161 isp.push('if 1:\n x=1')
162 162 self.assertEqual(isp.indent_spaces, 4)
163 163 isp.push('y=2\n')
164 164 self.assertEqual(isp.indent_spaces, 0)
165 165 isp.push('if 1:')
166 166 self.assertEqual(isp.indent_spaces, 4)
167 167 isp.push(' x=1')
168 168 self.assertEqual(isp.indent_spaces, 4)
169 169 # Blank lines shouldn't change the indent level
170 170 isp.push(' '*2)
171 171 self.assertEqual(isp.indent_spaces, 4)
172 172
173 173 def test_indent2(self):
174 174 isp = self.isp
175 175 # When a multiline statement contains parens or multiline strings, we
176 176 # shouldn't get confused.
177 177 isp.push("if 1:")
178 178 isp.push(" x = (1+\n 2)")
179 179 self.assertEqual(isp.indent_spaces, 4)
180 180
181 181 def test_dedent(self):
182 182 isp = self.isp # shorthand
183 183 isp.push('if 1:')
184 184 self.assertEqual(isp.indent_spaces, 4)
185 185 isp.push(' pass')
186 186 self.assertEqual(isp.indent_spaces, 0)
187 187
188 188 def test_push(self):
189 189 isp = self.isp
190 190 self.assertTrue(isp.push('x=1'))
191 191
192 192 def test_push2(self):
193 193 isp = self.isp
194 194 self.assertFalse(isp.push('if 1:'))
195 195 for line in [' x=1', '# a comment', ' y=2']:
196 196 self.assertTrue(isp.push(line))
197 197
198 198 def test_push3(self):
199 199 """Test input with leading whitespace"""
200 200 isp = self.isp
201 201 isp.push(' x=1')
202 202 isp.push(' y=2')
203 203 self.assertEqual(isp.source, 'if 1:\n x=1\n y=2\n')
204 204
205 205 def test_replace_mode(self):
206 206 isp = self.isp
207 207 isp.input_mode = 'replace'
208 208 isp.push('x=1')
209 209 self.assertEqual(isp.source, 'x=1\n')
210 210 isp.push('x=2')
211 211 self.assertEqual(isp.source, 'x=2\n')
212 212
213 213 def test_push_accepts_more(self):
214 214 isp = self.isp
215 215 isp.push('x=1')
216 216 self.assertFalse(isp.push_accepts_more())
217 217
218 218 def test_push_accepts_more2(self):
219 219 isp = self.isp
220 220 isp.push('if 1:')
221 221 self.assertTrue(isp.push_accepts_more())
222 222 isp.push(' x=1')
223 223 self.assertTrue(isp.push_accepts_more())
224 224 isp.push('')
225 225 self.assertFalse(isp.push_accepts_more())
226 226
227 227 def test_push_accepts_more3(self):
228 228 isp = self.isp
229 229 isp.push("x = (2+\n3)")
230 230 self.assertFalse(isp.push_accepts_more())
231 231
232 232 def test_push_accepts_more4(self):
233 233 isp = self.isp
234 234 # When a multiline statement contains parens or multiline strings, we
235 235 # shouldn't get confused.
236 236 # FIXME: we should be able to better handle de-dents in statements like
237 237 # multiline strings and multiline expressions (continued with \ or
238 238 # parens). Right now we aren't handling the indentation tracking quite
239 239 # correctly with this, though in practice it may not be too much of a
240 240 # problem. We'll need to see.
241 241 isp.push("if 1:")
242 242 isp.push(" x = (2+")
243 243 isp.push(" 3)")
244 244 self.assertTrue(isp.push_accepts_more())
245 245 isp.push(" y = 3")
246 246 self.assertTrue(isp.push_accepts_more())
247 247 isp.push('')
248 248 self.assertFalse(isp.push_accepts_more())
249 249
250 250 def test_syntax_error(self):
251 251 isp = self.isp
252 252 # Syntax errors immediately produce a 'ready' block, so the invalid
253 253 # Python can be sent to the kernel for evaluation with possible ipython
254 254 # special-syntax conversion.
255 255 isp.push('run foo')
256 256 self.assertFalse(isp.push_accepts_more())
257 257
258 258 def check_split(self, block_lines, compile=True):
259 259 blocks = assemble(block_lines)
260 260 lines = ''.join(blocks)
261 261 oblock = self.isp.split_blocks(lines)
262 262 self.assertEqual(oblock, blocks)
263 263 if compile:
264 264 for block in blocks:
265 265 self.isp._compile(block)
266 266
267 267 def test_split(self):
268 268 # All blocks of input we want to test in a list. The format for each
269 269 # block is a list of lists, with each inner lists consisting of all the
270 270 # lines (as single-lines) that should make up a sub-block.
271 271
272 272 # Note: do NOT put here sub-blocks that don't compile, as the
273 273 # check_split() routine makes a final verification pass to check that
274 274 # each sub_block, as returned by split_blocks(), does compile
275 275 # correctly.
276 276 all_blocks = [ [['x=1']],
277 277
278 278 [['x=1'],
279 279 ['y=2']],
280 280
281 281 [['x=1'],
282 282 ['# a comment'],
283 283 ['y=11']],
284 284
285 285 [['if 1:',
286 286 ' x=1'],
287 287 ['y=3']],
288 288
289 289 [['def f(x):',
290 290 ' return x'],
291 291 ['x=1']],
292 292
293 293 [['def f(x):',
294 294 ' x+=1',
295 295 ' ',
296 296 ' return x'],
297 297 ['x=1']],
298 298
299 299 [['def f(x):',
300 300 ' if x>0:',
301 301 ' y=1',
302 302 ' # a comment',
303 303 ' else:',
304 304 ' y=4',
305 305 ' ',
306 306 ' return y'],
307 307 ['x=1'],
308 308 ['if 1:',
309 309 ' y=11'] ],
310 310
311 311 [['for i in range(10):'
312 312 ' x=i**2']],
313 313
314 314 [['for i in range(10):'
315 315 ' x=i**2'],
316 316 ['z = 1']],
317 317 ]
318 318 for block_lines in all_blocks:
319 319 self.check_split(block_lines)
320 320
321 321 def test_split_syntax_errors(self):
322 322 # Block splitting with invalid syntax
323 323 all_blocks = [ [['a syntax error']],
324 324
325 325 [['x=1'],
326 326 ['a syntax error']],
327 327
328 328 [['for i in range(10):'
329 329 ' an error']],
330 330
331 331 ]
332 332 for block_lines in all_blocks:
333 333 self.check_split(block_lines, compile=False)
334 334
335 335
336 336 class InteractiveLoopTestCase(unittest.TestCase):
337 337 """Tests for an interactive loop like a python shell.
338 338 """
339 339 def check_ns(self, lines, ns):
340 340 """Validate that the given input lines produce the resulting namespace.
341 341
342 342 Note: the input lines are given exactly as they would be typed in an
343 343 auto-indenting environment, as mini_interactive_loop above already does
344 344 auto-indenting and prepends spaces to the input.
345 345 """
346 346 src = mini_interactive_loop(pseudo_input(lines))
347 347 test_ns = {}
348 348 exec src in test_ns
349 349 # We can't check that the provided ns is identical to the test_ns,
350 350 # because Python fills test_ns with extra keys (copyright, etc). But
351 351 # we can check that the given dict is *contained* in test_ns
352 352 for k,v in ns.items():
353 353 self.assertEqual(test_ns[k], v)
354 354
355 355 def test_simple(self):
356 356 self.check_ns(['x=1'], dict(x=1))
357 357
358 358 def test_simple2(self):
359 359 self.check_ns(['if 1:', 'x=2'], dict(x=2))
360 360
361 361 def test_xy(self):
362 362 self.check_ns(['x=1; y=2'], dict(x=1, y=2))
363 363
364 364 def test_abc(self):
365 365 self.check_ns(['if 1:','a=1','b=2','c=3'], dict(a=1, b=2, c=3))
366 366
367 367 def test_multi(self):
368 368 self.check_ns(['x =(1+','1+','2)'], dict(x=4))
369 369
370 370
371 371 def test_LineInfo():
372 372 """Simple test for LineInfo construction and str()"""
373 373 linfo = isp.LineInfo(' %cd /home')
374 374 nt.assert_equals(str(linfo), 'LineInfo [ |%|cd|/home]')
375 375
376 376
377 377 def test_split_user_input():
378 378 """Unicode test - split_user_input already has good doctests"""
379 379 line = u"PΓ©rez Fernando"
380 380 parts = isp.split_user_input(line)
381 381 parts_expected = (u'', u'', u'', line)
382 382 nt.assert_equal(parts, parts_expected)
383 383
384 384
385 385 # Transformer tests
386 386 def transform_checker(tests, func):
387 387 """Utility to loop over test inputs"""
388 388 for inp, tr in tests:
389 389 nt.assert_equals(func(inp), tr)
390 390
391 391 # Data for all the syntax tests in the form of lists of pairs of
392 392 # raw/transformed input. We store it here as a global dict so that we can use
393 393 # it both within single-function tests and also to validate the behavior of the
394 394 # larger objects
395 395
396 396 syntax = \
397 397 dict(assign_system =
398 398 [('a =! ls', 'a = get_ipython().magic("sc -l = ls")'),
399 399 ('b = !ls', 'b = get_ipython().magic("sc -l = ls")'),
400 400 ('x=1', 'x=1'), # normal input is unmodified
401 401 (' ',' '), # blank lines are kept intact
402 402 ],
403 403
404 404 assign_magic =
405 405 [('a =% who', 'a = get_ipython().magic("who")'),
406 406 ('b = %who', 'b = get_ipython().magic("who")'),
407 407 ('x=1', 'x=1'), # normal input is unmodified
408 408 (' ',' '), # blank lines are kept intact
409 409 ],
410 410
411 411 classic_prompt =
412 412 [('>>> x=1', 'x=1'),
413 413 ('x=1', 'x=1'), # normal input is unmodified
414 414 (' ',' '), # blank lines are kept intact
415 415 ],
416 416
417 417 ipy_prompt =
418 418 [('In [1]: x=1', 'x=1'),
419 419 ('x=1', 'x=1'), # normal input is unmodified
420 420 (' ',' '), # blank lines are kept intact
421 421 ],
422 422
423 423 # Tests for the escape transformer to leave normal code alone
424 424 escaped_noesc =
425 425 [ (' ', ' '),
426 426 ('x=1', 'x=1'),
427 427 ],
428 428
429 429 # System calls
430 430 escaped_shell =
431 431 [ ('!ls', 'get_ipython().system("ls")'),
432 432 # Double-escape shell, this means to capture the output of the
433 433 # subprocess and return it
434 434 ('!!ls', 'get_ipython().getoutput("ls")'),
435 435 ],
436 436
437 437 # Help/object info
438 438 escaped_help =
439 439 [ ('?', 'get_ipython().show_usage()'),
440 440 ('?x1', 'get_ipython().magic("pinfo x1")'),
441 441 ('??x2', 'get_ipython().magic("pinfo2 x2")'),
442 442 ('x3?', 'get_ipython().magic("pinfo x3")'),
443 443 ('x4??', 'get_ipython().magic("pinfo2 x4")'),
444 444 ],
445 445
446 446 # Explicit magic calls
447 447 escaped_magic =
448 448 [ ('%cd', 'get_ipython().magic("cd")'),
449 449 ('%cd /home', 'get_ipython().magic("cd /home")'),
450 450 (' %magic', ' get_ipython().magic("magic")'),
451 451 ],
452 452
453 453 # Quoting with separate arguments
454 454 escaped_quote =
455 455 [ (',f', 'f("")'),
456 456 (',f x', 'f("x")'),
457 457 (' ,f y', ' f("y")'),
458 458 (',f a b', 'f("a", "b")'),
459 459 ],
460 460
461 461 # Quoting with single argument
462 462 escaped_quote2 =
463 463 [ (';f', 'f("")'),
464 464 (';f x', 'f("x")'),
465 465 (' ;f y', ' f("y")'),
466 466 (';f a b', 'f("a b")'),
467 467 ],
468 468
469 469 # Simply apply parens
470 470 escaped_paren =
471 471 [ ('/f', 'f()'),
472 472 ('/f x', 'f(x)'),
473 473 (' /f y', ' f(y)'),
474 474 ('/f a b', 'f(a, b)'),
475 475 ],
476 476
477 477 # More complex multiline tests
478 478 ## escaped_multiline =
479 479 ## [()],
480 480 )
481 481
482 482 # multiline syntax examples. Each of these should be a list of lists, with
483 483 # each entry itself having pairs of raw/transformed input. The union (with
484 484 # '\n'.join() of the transformed inputs is what the splitter should produce
485 485 # when fed the raw lines one at a time via push.
486 486 syntax_ml = \
487 487 dict(classic_prompt =
488 488 [ [('>>> for i in range(10):','for i in range(10):'),
489 489 ('... print i',' print i'),
490 490 ('... ', ''),
491 491 ],
492 492 ],
493 493
494 494 ipy_prompt =
495 495 [ [('In [24]: for i in range(10):','for i in range(10):'),
496 496 (' ....: print i',' print i'),
497 497 (' ....: ', ''),
498 498 ],
499 499 ],
500 500 )
501 501
502 502
503 503 def test_assign_system():
504 504 transform_checker(syntax['assign_system'], isp.transform_assign_system)
505 505
506 506
507 507 def test_assign_magic():
508 508 transform_checker(syntax['assign_magic'], isp.transform_assign_magic)
509 509
510 510
511 511 def test_classic_prompt():
512 512 transform_checker(syntax['classic_prompt'], isp.transform_classic_prompt)
513 513 for example in syntax_ml['classic_prompt']:
514 514 transform_checker(example, isp.transform_classic_prompt)
515 515
516 516
517 517 def test_ipy_prompt():
518 518 transform_checker(syntax['ipy_prompt'], isp.transform_ipy_prompt)
519 519 for example in syntax_ml['ipy_prompt']:
520 520 transform_checker(example, isp.transform_ipy_prompt)
521 521
522 522
523 523 def test_escaped_noesc():
524 524 transform_checker(syntax['escaped_noesc'], isp.transform_escaped)
525 525
526 526
527 527 def test_escaped_shell():
528 528 transform_checker(syntax['escaped_shell'], isp.transform_escaped)
529 529
530 530
531 531 def test_escaped_help():
532 532 transform_checker(syntax['escaped_help'], isp.transform_escaped)
533 533
534 534
535 535 def test_escaped_magic():
536 536 transform_checker(syntax['escaped_magic'], isp.transform_escaped)
537 537
538 538
539 539 def test_escaped_quote():
540 540 transform_checker(syntax['escaped_quote'], isp.transform_escaped)
541 541
542 542
543 543 def test_escaped_quote2():
544 544 transform_checker(syntax['escaped_quote2'], isp.transform_escaped)
545 545
546 546
547 547 def test_escaped_paren():
548 548 transform_checker(syntax['escaped_paren'], isp.transform_escaped)
549 549
550 550
551 551 class IPythonInputTestCase(InputSplitterTestCase):
552 552 """By just creating a new class whose .isp is a different instance, we
553 553 re-run the same test battery on the new input splitter.
554 554
555 555 In addition, this runs the tests over the syntax and syntax_ml dicts that
556 556 were tested by individual functions, as part of the OO interface.
557 557 """
558 558 def setUp(self):
559 559 self.isp = isp.IPythonInputSplitter()
560 560
561 561 def test_syntax(self):
562 562 """Call all single-line syntax tests from the main object"""
563 563 isp = self.isp
564 564 for example in syntax.itervalues():
565 565 for raw, out_t in example:
566 566 if raw.startswith(' '):
567 567 continue
568 568
569 569 isp.push(raw)
570 570 out = isp.source_reset().rstrip()
571 571 self.assertEqual(out, out_t)
572 572
573 573 def test_syntax_multiline(self):
574 574 isp = self.isp
575 575 for example in syntax_ml.itervalues():
576 576 out_t_parts = []
577 577 for line_pairs in example:
578 578 for raw, out_t_part in line_pairs:
579 579 isp.push(raw)
580 580 out_t_parts.append(out_t_part)
581 581
582 582 out = isp.source_reset().rstrip()
583 583 out_t = '\n'.join(out_t_parts).rstrip()
584 584 self.assertEqual(out, out_t)
585 585
586
586
587 587 #-----------------------------------------------------------------------------
588 588 # Main - use as a script
589 589 #-----------------------------------------------------------------------------
590 590
591 591 if __name__ == '__main__':
592 592 # A simple demo for interactive experimentation. This code will not get
593 593 # picked up by any test suite. Useful mostly for illustration and during
594 594 # development.
595 595 from IPython.core.inputsplitter import InputSplitter, IPythonInputSplitter
596 596
597 597 # configure here the syntax to use, prompt and whether to autoindent
598 598 #isp, start_prompt = InputSplitter(), '>>> '
599 599 isp, start_prompt = IPythonInputSplitter(), 'In> '
600 600
601 601 autoindent = True
602 602 #autoindent = False
603 603
604 604 try:
605 605 while True:
606 606 prompt = start_prompt
607 607 while isp.push_accepts_more():
608 608 indent = ' '*isp.indent_spaces
609 609 if autoindent:
610 610 line = indent + raw_input(prompt+indent)
611 611 else:
612 612 line = raw_input(prompt)
613 613 isp.push(line)
614 614 prompt = '... '
615 615
616 616 # Here we just return input so we can use it in a test suite, but a
617 617 # real interpreter would instead send it for execution somewhere.
618 #src = isp.source; raise EOFError # dbg
618 619 src = isp.source_reset()
619 620 print 'Input source was:\n', src
620 621 except EOFError:
621 622 print 'Bye'
General Comments 0
You need to be logged in to leave comments. Login now