##// END OF EJS Templates
Add a few more fixes to cell/line input code, switch approaches....
Fernando Perez -
Show More
@@ -1,993 +1,1004 b''
1 1 """Analysis of text input into executable blocks.
2 2
3 3 The main class in this module, :class:`InputSplitter`, is designed to break
4 4 input from either interactive, line-by-line environments or block-based ones,
5 5 into standalone blocks that can be executed by Python as 'single' statements
6 6 (thus triggering sys.displayhook).
7 7
8 8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 9 with full support for the extended IPython syntax (magics, system calls, etc).
10 10
11 11 For more details, see the class docstring below.
12 12
13 13 Syntax Transformations
14 14 ----------------------
15 15
16 16 One of the main jobs of the code in this file is to apply all syntax
17 17 transformations that make up 'the IPython language', i.e. magics, shell
18 18 escapes, etc. All transformations should be implemented as *fully stateless*
19 19 entities, that simply take one line as their input and return a line.
20 20 Internally for implementation purposes they may be a normal function or a
21 21 callable object, but the only input they receive will be a single line and they
22 22 should only return a line, without holding any data-dependent state between
23 23 calls.
24 24
25 25 As an example, the EscapedTransformer is a class so we can more clearly group
26 26 together the functionality of dispatching to individual functions based on the
27 27 starting escape character, but the only method for public use is its call
28 28 method.
29 29
30 30
31 31 ToDo
32 32 ----
33 33
34 34 - Should we make push() actually raise an exception once push_accepts_more()
35 35 returns False?
36 36
37 37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
38 38 at least just attributes of a class so not really very exposed.
39 39
40 40 - Think about the best way to support dynamic things: automagic, autocall,
41 41 macros, etc.
42 42
43 43 - Think of a better heuristic for the application of the transforms in
44 44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
45 45 track indentation change events (indent, dedent, nothing) and apply them only
46 46 if the indentation went up, but not otherwise.
47 47
48 48 - Think of the cleanest way for supporting user-specified transformations (the
49 49 user prefilters we had before).
50 50
51 51 Authors
52 52 -------
53 53
54 54 * Fernando Perez
55 55 * Brian Granger
56 56 """
57 57 #-----------------------------------------------------------------------------
58 58 # Copyright (C) 2010 The IPython Development Team
59 59 #
60 60 # Distributed under the terms of the BSD License. The full license is in
61 61 # the file COPYING, distributed as part of this software.
62 62 #-----------------------------------------------------------------------------
63 63 from __future__ import print_function
64 64
65 65 #-----------------------------------------------------------------------------
66 66 # Imports
67 67 #-----------------------------------------------------------------------------
68 68 # stdlib
69 69 import ast
70 70 import codeop
71 71 import re
72 72 import sys
73 73 import tokenize
74 74 from StringIO import StringIO
75 75
76 76 # IPython modules
77 77 from IPython.core.splitinput import split_user_input, LineInfo
78 78 from IPython.utils.py3compat import cast_unicode
79 79
80 80 #-----------------------------------------------------------------------------
81 81 # Globals
82 82 #-----------------------------------------------------------------------------
83 83
84 84 # The escape sequences that define the syntax transformations IPython will
85 85 # apply to user input. These can NOT be just changed here: many regular
86 86 # expressions and other parts of the code may use their hardcoded values, and
87 87 # for all intents and purposes they constitute the 'IPython syntax', so they
88 88 # should be considered fixed.
89 89
90 90 ESC_SHELL = '!' # Send line to underlying system shell
91 91 ESC_SH_CAP = '!!' # Send line to system shell and capture output
92 92 ESC_HELP = '?' # Find information about object
93 93 ESC_HELP2 = '??' # Find extra-detailed information about object
94 94 ESC_MAGIC = '%' # Call magic function
95 95 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
96 96 ESC_QUOTE2 = ';' # Quote all args as a single string, call
97 97 ESC_PAREN = '/' # Call first argument with rest of line as arguments
98 98
99 99 #-----------------------------------------------------------------------------
100 100 # Utilities
101 101 #-----------------------------------------------------------------------------
102 102
103 103 # FIXME: These are general-purpose utilities that later can be moved to the
104 104 # general ward. Kept here for now because we're being very strict about test
105 105 # coverage with this code, and this lets us ensure that we keep 100% coverage
106 106 # while developing.
107 107
108 108 # compiled regexps for autoindent management
109 109 dedent_re = re.compile('|'.join([
110 110 r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
111 111 r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
112 112 r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
113 113 r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
114 114 r'^\s+pass\s*$' # pass (optionally followed by trailing spaces)
115 115 ]))
116 116 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
117 117
118 118 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
119 119 # before pure comments
120 120 comment_line_re = re.compile('^\s*\#')
121 121
122 122
123 123 def num_ini_spaces(s):
124 124 """Return the number of initial spaces in a string.
125 125
126 126 Note that tabs are counted as a single space. For now, we do *not* support
127 127 mixing of tabs and spaces in the user's input.
128 128
129 129 Parameters
130 130 ----------
131 131 s : string
132 132
133 133 Returns
134 134 -------
135 135 n : int
136 136 """
137 137
138 138 ini_spaces = ini_spaces_re.match(s)
139 139 if ini_spaces:
140 140 return ini_spaces.end()
141 141 else:
142 142 return 0
143 143
144 last_blank_re = re.compile(r'^.*\n\s+$', re.MULTILINE)
145
146 144 def last_blank(src):
147 145 """Determine if the input source ends in a blank.
148 146
149 147 A blank is either a newline or a line consisting of whitespace.
150 148
151 149 Parameters
152 150 ----------
153 151 src : string
154 152 A single or multiline string.
155 153 """
156 return src == '\n' or bool(last_blank_re.match(src))
154 if not src: return False
155 ll = src.splitlines()[-1]
156 return (ll == '') or ll.isspace()
157 157
158 158
159 last_two_blanks_re = re.compile(r'^\n\s*\n\s*$', re.MULTILINE)
160 last_two_blanks_re2 = re.compile(r'^.+\n\s*\n\s+$', re.MULTILINE)
159 last_two_blanks_re = re.compile(r'\n\s*\n\s*$', re.MULTILINE)
160 last_two_blanks_re2 = re.compile(r'.+\n\s*\n\s+$', re.MULTILINE)
161 161
162 162 def last_two_blanks(src):
163 163 """Determine if the input source ends in two blanks.
164 164
165 165 A blank is either a newline or a line consisting of whitespace.
166 166
167 167 Parameters
168 168 ----------
169 169 src : string
170 170 A single or multiline string.
171 171 """
172 return (bool(last_two_blanks_re.match(src)) or
173 bool(last_two_blanks_re2.match(src)) )
172 if not src: return False
173 # The logic here is tricky: I couldn't get a regexp to work and pass all
174 # the tests, so I took a different approach: split the source by lines,
175 # grab the last two and prepend '###\n' as a stand-in for whatever was in
176 # the body before the last two lines. Then, with that structure, it's
177 # possible to analyze with two regexps. Not the most elegant solution, but
178 # it works. If anyone tries to change this logic, make sure to validate
179 # the whole test suite first!
180 new_src = '\n'.join(['###\n'] + src.splitlines()[-2:])
181 return (bool(last_two_blanks_re.match(new_src)) or
182 bool(last_two_blanks_re2.match(new_src)) )
174 183
175 184
176 185 def remove_comments(src):
177 186 """Remove all comments from input source.
178 187
179 188 Note: comments are NOT recognized inside of strings!
180 189
181 190 Parameters
182 191 ----------
183 192 src : string
184 193 A single or multiline input string.
185 194
186 195 Returns
187 196 -------
188 197 String with all Python comments removed.
189 198 """
190 199
191 200 return re.sub('#.*', '', src)
192 201
193 202 def has_comment(src):
194 203 """Indicate whether an input line has (i.e. ends in, or is) a comment.
195 204
196 205 This uses tokenize, so it can distinguish comments from # inside strings.
197 206
198 207 Parameters
199 208 ----------
200 209 src : string
201 210 A single line input string.
202 211
203 212 Returns
204 213 -------
205 214 Boolean: True if source has a comment.
206 215 """
207 216 readline = StringIO(src).readline
208 217 toktypes = set()
209 218 try:
210 219 for t in tokenize.generate_tokens(readline):
211 220 toktypes.add(t[0])
212 221 except tokenize.TokenError:
213 222 pass
214 223 return(tokenize.COMMENT in toktypes)
215 224
216 225
217 226 def get_input_encoding():
218 227 """Return the default standard input encoding.
219 228
220 229 If sys.stdin has no encoding, 'ascii' is returned."""
221 230 # There are strange environments for which sys.stdin.encoding is None. We
222 231 # ensure that a valid encoding is returned.
223 232 encoding = getattr(sys.stdin, 'encoding', None)
224 233 if encoding is None:
225 234 encoding = 'ascii'
226 235 return encoding
227 236
228 237 #-----------------------------------------------------------------------------
229 238 # Classes and functions for normal Python syntax handling
230 239 #-----------------------------------------------------------------------------
231 240
232 241 class InputSplitter(object):
233 242 """An object that can accumulate lines of Python source before execution.
234 243
235 244 This object is designed to be fed python source line-by-line, using
236 245 :meth:`push`. It will return on each push whether the currently pushed
237 246 code could be executed already. In addition, it provides a method called
238 247 :meth:`push_accepts_more` that can be used to query whether more input
239 248 can be pushed into a single interactive block.
240 249
241 250 This is a simple example of how an interactive terminal-based client can use
242 251 this tool::
243 252
244 253 isp = InputSplitter()
245 254 while isp.push_accepts_more():
246 255 indent = ' '*isp.indent_spaces
247 256 prompt = '>>> ' + indent
248 257 line = indent + raw_input(prompt)
249 258 isp.push(line)
250 259 print 'Input source was:\n', isp.source_reset(),
251 260 """
252 261 # Number of spaces of indentation computed from input that has been pushed
253 262 # so far. This is the attributes callers should query to get the current
254 263 # indentation level, in order to provide auto-indent facilities.
255 264 indent_spaces = 0
256 265 # String, indicating the default input encoding. It is computed by default
257 266 # at initialization time via get_input_encoding(), but it can be reset by a
258 267 # client with specific knowledge of the encoding.
259 268 encoding = ''
260 269 # String where the current full source input is stored, properly encoded.
261 270 # Reading this attribute is the normal way of querying the currently pushed
262 271 # source code, that has been properly encoded.
263 272 source = ''
264 273 # Code object corresponding to the current source. It is automatically
265 274 # synced to the source, so it can be queried at any time to obtain the code
266 275 # object; it will be None if the source doesn't compile to valid Python.
267 276 code = None
268 277 # Input mode
269 278 input_mode = 'line'
270 279
271 280 # Private attributes
272 281
273 282 # List with lines of input accumulated so far
274 283 _buffer = None
275 284 # Command compiler
276 285 _compile = None
277 286 # Mark when input has changed indentation all the way back to flush-left
278 287 _full_dedent = False
279 288 # Boolean indicating whether the current block is complete
280 289 _is_complete = None
281 290
282 291 def __init__(self, input_mode=None):
283 292 """Create a new InputSplitter instance.
284 293
285 294 Parameters
286 295 ----------
287 296 input_mode : str
288 297
289 298 One of ['line', 'cell']; default is 'line'.
290 299
291 300 The input_mode parameter controls how new inputs are used when fed via
292 301 the :meth:`push` method:
293 302
294 303 - 'line': meant for line-oriented clients, inputs are appended one at a
295 304 time to the internal buffer and the whole buffer is compiled.
296 305
297 306 - 'cell': meant for clients that can edit multi-line 'cells' of text at
298 307 a time. A cell can contain one or more blocks that can be compile in
299 308 'single' mode by Python. In this mode, each new input new input
300 309 completely replaces all prior inputs. Cell mode is thus equivalent
301 310 to prepending a full reset() to every push() call.
302 311 """
303 312 self._buffer = []
304 313 self._compile = codeop.CommandCompiler()
305 314 self.encoding = get_input_encoding()
306 315 self.input_mode = InputSplitter.input_mode if input_mode is None \
307 316 else input_mode
308 317
309 318 def reset(self):
310 319 """Reset the input buffer and associated state."""
311 320 self.indent_spaces = 0
312 321 self._buffer[:] = []
313 322 self.source = ''
314 323 self.code = None
315 324 self._is_complete = False
316 325 self._full_dedent = False
317 326
318 327 def source_reset(self):
319 328 """Return the input source and perform a full reset.
320 329 """
321 330 out = self.source
322 331 self.reset()
323 332 return out
324 333
325 334 def push(self, lines):
326 335 """Push one or more lines of input.
327 336
328 337 This stores the given lines and returns a status code indicating
329 338 whether the code forms a complete Python block or not.
330 339
331 340 Any exceptions generated in compilation are swallowed, but if an
332 341 exception was produced, the method returns True.
333 342
334 343 Parameters
335 344 ----------
336 345 lines : string
337 346 One or more lines of Python input.
338 347
339 348 Returns
340 349 -------
341 350 is_complete : boolean
342 351 True if the current input source (the result of the current input
343 352 plus prior inputs) forms a complete Python execution block. Note that
344 353 this value is also stored as a private attribute (_is_complete), so it
345 354 can be queried at any time.
346 355 """
347 356 if self.input_mode == 'cell':
348 357 self.reset()
349 358
350 359 self._store(lines)
351 360 source = self.source
352 361
353 362 # Before calling _compile(), reset the code object to None so that if an
354 363 # exception is raised in compilation, we don't mislead by having
355 364 # inconsistent code/source attributes.
356 365 self.code, self._is_complete = None, None
357 366
358 367 # Honor termination lines properly
359 368 if source.rstrip().endswith('\\'):
360 369 return False
361 370
362 371 self._update_indent(lines)
363 372 try:
364 373 self.code = self._compile(source, symbol="exec")
365 374 # Invalid syntax can produce any of a number of different errors from
366 375 # inside the compiler, so we have to catch them all. Syntax errors
367 376 # immediately produce a 'ready' block, so the invalid Python can be
368 377 # sent to the kernel for evaluation with possible ipython
369 378 # special-syntax conversion.
370 379 except (SyntaxError, OverflowError, ValueError, TypeError,
371 380 MemoryError):
372 381 self._is_complete = True
373 382 else:
374 383 # Compilation didn't produce any exceptions (though it may not have
375 384 # given a complete code object)
376 385 self._is_complete = self.code is not None
377 386
378 387 return self._is_complete
379 388
380 389 def push_accepts_more(self):
381 390 """Return whether a block of interactive input can accept more input.
382 391
383 392 This method is meant to be used by line-oriented frontends, who need to
384 393 guess whether a block is complete or not based solely on prior and
385 394 current input lines. The InputSplitter considers it has a complete
386 395 interactive block and will not accept more input only when either a
387 396 SyntaxError is raised, or *all* of the following are true:
388 397
389 398 1. The input compiles to a complete statement.
390 399
391 400 2. The indentation level is flush-left (because if we are indented,
392 401 like inside a function definition or for loop, we need to keep
393 402 reading new input).
394 403
395 404 3. There is one extra line consisting only of whitespace.
396 405
397 406 Because of condition #3, this method should be used only by
398 407 *line-oriented* frontends, since it means that intermediate blank lines
399 408 are not allowed in function definitions (or any other indented block).
400 409
401 410 If the current input produces a syntax error, this method immediately
402 411 returns False but does *not* raise the syntax error exception, as
403 412 typically clients will want to send invalid syntax to an execution
404 413 backend which might convert the invalid syntax into valid Python via
405 414 one of the dynamic IPython mechanisms.
406 415 """
407 416
408 417 # With incomplete input, unconditionally accept more
409 418 if not self._is_complete:
410 419 return True
411 420
412 421 # If we already have complete input and we're flush left, the answer
413 422 # depends. In line mode, if there hasn't been any indentation,
414 423 # that's it. If we've come back from some indentation, we need
415 424 # the blank final line to finish.
416 425 # In cell mode, we need to check how many blocks the input so far
417 426 # compiles into, because if there's already more than one full
418 427 # independent block of input, then the client has entered full
419 428 # 'cell' mode and is feeding lines that each is complete. In this
420 429 # case we should then keep accepting. The Qt terminal-like console
421 430 # does precisely this, to provide the convenience of terminal-like
422 431 # input of single expressions, but allowing the user (with a
423 432 # separate keystroke) to switch to 'cell' mode and type multiple
424 433 # expressions in one shot.
425 434 if self.indent_spaces==0:
426 435 if self.input_mode=='line':
427 436 if not self._full_dedent:
428 437 return False
429 438 else:
430 439 try:
431 440 code_ast = ast.parse(u''.join(self._buffer))
432 441 except Exception:
433 442 return False
434 443 else:
435 444 if len(code_ast.body) == 1:
436 445 return False
437 446
438 447 # When input is complete, then termination is marked by an extra blank
439 448 # line at the end.
440 449 last_line = self.source.splitlines()[-1]
441 450 return bool(last_line and not last_line.isspace())
442 451
443 452 #------------------------------------------------------------------------
444 453 # Private interface
445 454 #------------------------------------------------------------------------
446 455
447 456 def _find_indent(self, line):
448 457 """Compute the new indentation level for a single line.
449 458
450 459 Parameters
451 460 ----------
452 461 line : str
453 462 A single new line of non-whitespace, non-comment Python input.
454 463
455 464 Returns
456 465 -------
457 466 indent_spaces : int
458 467 New value for the indent level (it may be equal to self.indent_spaces
459 468 if indentation doesn't change.
460 469
461 470 full_dedent : boolean
462 471 Whether the new line causes a full flush-left dedent.
463 472 """
464 473 indent_spaces = self.indent_spaces
465 474 full_dedent = self._full_dedent
466 475
467 476 inisp = num_ini_spaces(line)
468 477 if inisp < indent_spaces:
469 478 indent_spaces = inisp
470 479 if indent_spaces <= 0:
471 480 #print 'Full dedent in text',self.source # dbg
472 481 full_dedent = True
473 482
474 483 if line.rstrip()[-1] == ':':
475 484 indent_spaces += 4
476 485 elif dedent_re.match(line):
477 486 indent_spaces -= 4
478 487 if indent_spaces <= 0:
479 488 full_dedent = True
480 489
481 490 # Safety
482 491 if indent_spaces < 0:
483 492 indent_spaces = 0
484 493 #print 'safety' # dbg
485 494
486 495 return indent_spaces, full_dedent
487 496
488 497 def _update_indent(self, lines):
489 498 for line in remove_comments(lines).splitlines():
490 499 if line and not line.isspace():
491 500 self.indent_spaces, self._full_dedent = self._find_indent(line)
492 501
493 502 def _store(self, lines, buffer=None, store='source'):
494 503 """Store one or more lines of input.
495 504
496 505 If input lines are not newline-terminated, a newline is automatically
497 506 appended."""
498 507
499 508 if buffer is None:
500 509 buffer = self._buffer
501 510
502 511 if lines.endswith('\n'):
503 512 buffer.append(lines)
504 513 else:
505 514 buffer.append(lines+'\n')
506 515 setattr(self, store, self._set_source(buffer))
507 516
508 517 def _set_source(self, buffer):
509 518 return u''.join(buffer)
510 519
511 520
512 521 #-----------------------------------------------------------------------------
513 522 # Functions and classes for IPython-specific syntactic support
514 523 #-----------------------------------------------------------------------------
515 524
516 525 # The escaped translators ALL receive a line where their own escape has been
517 526 # stripped. Only '?' is valid at the end of the line, all others can only be
518 527 # placed at the start.
519 528
520 529 # Transformations of the special syntaxes that don't rely on an explicit escape
521 530 # character but instead on patterns on the input line
522 531
523 532 # The core transformations are implemented as standalone functions that can be
524 533 # tested and validated in isolation. Each of these uses a regexp, we
525 534 # pre-compile these and keep them close to each function definition for clarity
526 535
527 536 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
528 537 r'\s*=\s*!\s*(?P<cmd>.*)')
529 538
530 539 def transform_assign_system(line):
531 540 """Handle the `files = !ls` syntax."""
532 541 m = _assign_system_re.match(line)
533 542 if m is not None:
534 543 cmd = m.group('cmd')
535 544 lhs = m.group('lhs')
536 545 new_line = '%s = get_ipython().getoutput(%r)' % (lhs, cmd)
537 546 return new_line
538 547 return line
539 548
540 549
541 550 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
542 551 r'\s*=\s*%\s*(?P<cmd>.*)')
543 552
544 553 def transform_assign_magic(line):
545 554 """Handle the `a = %who` syntax."""
546 555 m = _assign_magic_re.match(line)
547 556 if m is not None:
548 557 cmd = m.group('cmd')
549 558 lhs = m.group('lhs')
550 559 new_line = '%s = get_ipython().magic(%r)' % (lhs, cmd)
551 560 return new_line
552 561 return line
553 562
554 563
555 564 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
556 565
557 566 def transform_classic_prompt(line):
558 567 """Handle inputs that start with '>>> ' syntax."""
559 568
560 569 if not line or line.isspace():
561 570 return line
562 571 m = _classic_prompt_re.match(line)
563 572 if m:
564 573 return line[len(m.group(0)):]
565 574 else:
566 575 return line
567 576
568 577
569 578 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
570 579
571 580 def transform_ipy_prompt(line):
572 581 """Handle inputs that start classic IPython prompt syntax."""
573 582
574 583 if not line or line.isspace():
575 584 return line
576 585 #print 'LINE: %r' % line # dbg
577 586 m = _ipy_prompt_re.match(line)
578 587 if m:
579 588 #print 'MATCH! %r -> %r' % (line, line[len(m.group(0)):]) # dbg
580 589 return line[len(m.group(0)):]
581 590 else:
582 591 return line
583 592
584 593
585 594 def _make_help_call(target, esc, lspace, next_input=None):
586 595 """Prepares a pinfo(2)/psearch call from a target name and the escape
587 596 (i.e. ? or ??)"""
588 597 method = 'pinfo2' if esc == '??' \
589 598 else 'psearch' if '*' in target \
590 599 else 'pinfo'
591 600 arg = " ".join([method, target])
592 601
593 602 if next_input is None:
594 603 return '%sget_ipython().magic(%r)' % (lspace, arg)
595 604 else:
596 605 return '%sget_ipython().set_next_input(%r);get_ipython().magic(%r)' % \
597 606 (lspace, next_input, arg)
598 607
599 608
600 609 _initial_space_re = re.compile(r'\s*')
601 610
602 611 _help_end_re = re.compile(r"""(%?
603 612 [a-zA-Z_*][\w*]* # Variable name
604 613 (\.[a-zA-Z_*][\w*]*)* # .etc.etc
605 614 )
606 615 (\?\??)$ # ? or ??""",
607 616 re.VERBOSE)
608 617
609 618
610 619 def transform_help_end(line):
611 620 """Translate lines with ?/?? at the end"""
612 621 m = _help_end_re.search(line)
613 622 if m is None or has_comment(line):
614 623 return line
615 624 target = m.group(1)
616 625 esc = m.group(3)
617 626 lspace = _initial_space_re.match(line).group(0)
618 627
619 628 # If we're mid-command, put it back on the next prompt for the user.
620 629 next_input = line.rstrip('?') if line.strip() != m.group(0) else None
621 630
622 631 return _make_help_call(target, esc, lspace, next_input)
623 632
624 633
625 634 class EscapedTransformer(object):
626 635 """Class to transform lines that are explicitly escaped out."""
627 636
628 637 def __init__(self):
629 638 tr = { ESC_SHELL : self._tr_system,
630 639 ESC_SH_CAP : self._tr_system2,
631 640 ESC_HELP : self._tr_help,
632 641 ESC_HELP2 : self._tr_help,
633 642 ESC_MAGIC : self._tr_magic,
634 643 ESC_QUOTE : self._tr_quote,
635 644 ESC_QUOTE2 : self._tr_quote2,
636 645 ESC_PAREN : self._tr_paren }
637 646 self.tr = tr
638 647
639 648 # Support for syntax transformations that use explicit escapes typed by the
640 649 # user at the beginning of a line
641 650 @staticmethod
642 651 def _tr_system(line_info):
643 652 "Translate lines escaped with: !"
644 653 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
645 654 return '%sget_ipython().system(%r)' % (line_info.pre, cmd)
646 655
647 656 @staticmethod
648 657 def _tr_system2(line_info):
649 658 "Translate lines escaped with: !!"
650 659 cmd = line_info.line.lstrip()[2:]
651 660 return '%sget_ipython().getoutput(%r)' % (line_info.pre, cmd)
652 661
653 662 @staticmethod
654 663 def _tr_help(line_info):
655 664 "Translate lines escaped with: ?/??"
656 665 # A naked help line should just fire the intro help screen
657 666 if not line_info.line[1:]:
658 667 return 'get_ipython().show_usage()'
659 668
660 669 return _make_help_call(line_info.ifun, line_info.esc, line_info.pre)
661 670
662 671 @staticmethod
663 672 def _tr_magic(line_info):
664 673 "Translate lines escaped with: %"
665 674 tpl = '%sget_ipython().magic(%r)'
666 675 cmd = ' '.join([line_info.ifun, line_info.the_rest]).strip()
667 676 return tpl % (line_info.pre, cmd)
668 677
669 678 @staticmethod
670 679 def _tr_quote(line_info):
671 680 "Translate lines escaped with: ,"
672 681 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
673 682 '", "'.join(line_info.the_rest.split()) )
674 683
675 684 @staticmethod
676 685 def _tr_quote2(line_info):
677 686 "Translate lines escaped with: ;"
678 687 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
679 688 line_info.the_rest)
680 689
681 690 @staticmethod
682 691 def _tr_paren(line_info):
683 692 "Translate lines escaped with: /"
684 693 return '%s%s(%s)' % (line_info.pre, line_info.ifun,
685 694 ", ".join(line_info.the_rest.split()))
686 695
687 696 def __call__(self, line):
688 697 """Class to transform lines that are explicitly escaped out.
689 698
690 699 This calls the above _tr_* static methods for the actual line
691 700 translations."""
692 701
693 702 # Empty lines just get returned unmodified
694 703 if not line or line.isspace():
695 704 return line
696 705
697 706 # Get line endpoints, where the escapes can be
698 707 line_info = LineInfo(line)
699 708
700 709 if not line_info.esc in self.tr:
701 710 # If we don't recognize the escape, don't modify the line
702 711 return line
703 712
704 713 return self.tr[line_info.esc](line_info)
705 714
706 715
707 716 # A function-looking object to be used by the rest of the code. The purpose of
708 717 # the class in this case is to organize related functionality, more than to
709 718 # manage state.
710 719 transform_escaped = EscapedTransformer()
711 720
712 721
713 722 class IPythonInputSplitter(InputSplitter):
714 723 """An input splitter that recognizes all of IPython's special syntax."""
715 724
716 725 # String with raw, untransformed input.
717 726 source_raw = ''
718 727
719 728 cell_magic_parts = []
720 729
721 730 cell_magic_mode = False
722 731
723 732 # Private attributes
724 733
725 734 # List with lines of raw input accumulated so far.
726 735 _buffer_raw = None
727 736
728 737 def __init__(self, input_mode=None):
729 738 super(IPythonInputSplitter, self).__init__(input_mode)
730 739 self._buffer_raw = []
731 740 self._validate = True
732 741
733 742 def reset(self):
734 743 """Reset the input buffer and associated state."""
735 744 super(IPythonInputSplitter, self).reset()
736 745 self._buffer_raw[:] = []
737 746 self.source_raw = ''
738 747 self.cell_magic_parts = []
739 748 self.cell_magic_mode = False
740 749
741 750 def source_raw_reset(self):
742 751 """Return input and raw source and perform a full reset.
743 752 """
744 753 out = self.source
745 754 out_r = self.source_raw
746 755 self.reset()
747 756 return out, out_r
748 757
749 758 def push_accepts_more(self):
750 759 if self.cell_magic_mode:
751 760 return not self._is_complete
752 761 else:
753 762 return super(IPythonInputSplitter, self).push_accepts_more()
754 763
755 764 def _push_line_mode(self, lines):
756 765 """Push in line mode.
757 766
758 767 This means that we only get individual 'lines' with each call, though
759 768 in practice each input may be multiline. But this is in contrast to
760 769 cell mode, which feeds the entirety of the cell from the start with
761 770 each call.
762 771 """
763 772 # cell magic support
764 773 #print('#'*10)
765 774 #print(lines+'\n---') # dbg
766 775 #print (repr(lines)+'\n+++')
767 776 #print('raw', self._buffer_raw, 'validate', self.cell_magic_mode)
768 777 # Only trigger this block if we're at a 'fresh' pumping start.
769 778 if lines.startswith('%%') and (not self.cell_magic_mode) and \
770 779 not self._buffer_raw:
771 780 # Cell magics bypass all further transformations
772 781 self.cell_magic_mode = True
773 782 first, _, body = lines.partition('\n')
774 783 magic_name, _, line = first.partition(' ')
775 784 magic_name = magic_name.lstrip(ESC_MAGIC)
776 785 # We store the body of the cell and create a call to a method that
777 786 # will use this stored value. This is ugly, but it's a first cut to
778 787 # get it all working, as right now changing the return API of our
779 788 # methods would require major refactoring.
780 789 self.cell_magic_parts = [body]
781 790 tpl = 'get_ipython()._cell_magic(%r, %r)'
782 791 tlines = tpl % (magic_name, line)
783 792 self._store(tlines)
784 793 self._store(lines, self._buffer_raw, 'source_raw')
785 794 self._is_complete = False
786 795 return False
787 796
788 797 if self.cell_magic_mode:
798 #print('c2 lines', repr(lines)) # dbg
789 799 # Find out if the last stored block has a whitespace line as its
790 800 # last line and also this line is whitespace, case in which we're
791 801 # done (two contiguous blank lines signal termination). Note that
792 802 # the storage logic *enforces* that every stored block is
793 803 # newline-terminated, so we grab everything but the last character
794 804 # so we can have the body of the block alone.
795 805 last_block = self.cell_magic_parts[-1]
796 806 self._is_complete = last_blank(last_block) and lines.isspace()
797 807 # Only store the raw input. For lines beyond the first one, we
798 808 # only store them for history purposes, and for execution we want
799 809 # the caller to only receive the _cell_magic() call.
800 810 self._store(lines, self._buffer_raw, 'source_raw')
801 811 self.cell_magic_parts.append(lines)
802 812 return self._is_complete
803 813
804 814 lines_list = lines.splitlines()
805 815
806 816 transforms = [transform_ipy_prompt, transform_classic_prompt,
807 817 transform_help_end, transform_escaped,
808 818 transform_assign_system, transform_assign_magic]
809 819
810 820 # Transform logic
811 821 #
812 822 # We only apply the line transformers to the input if we have either no
813 823 # input yet, or complete input, or if the last line of the buffer ends
814 824 # with ':' (opening an indented block). This prevents the accidental
815 825 # transformation of escapes inside multiline expressions like
816 826 # triple-quoted strings or parenthesized expressions.
817 827 #
818 828 # The last heuristic, while ugly, ensures that the first line of an
819 829 # indented block is correctly transformed.
820 830 #
821 831 # FIXME: try to find a cleaner approach for this last bit.
822 832
823 833 # Store raw source before applying any transformations to it. Note
824 834 # that this must be done *after* the reset() call that would otherwise
825 835 # flush the buffer.
826 836 self._store(lines, self._buffer_raw, 'source_raw')
827 837
828 838 push = super(IPythonInputSplitter, self).push
829 839 buf = self._buffer
830 840 for line in lines_list:
831 841 if self._is_complete or not buf or \
832 842 (buf and buf[-1].rstrip().endswith((':', ','))):
833 843 for f in transforms:
834 844 line = f(line)
835 845
836 846 out = push(line)
837 847 return out
838 848
839 849
840 850 def _push_cell_mode(self, lines):
841 851 """Push in cell mode.
842 852
843 853 This means that we get the entire cell with each call. Between resets,
844 854 the calls simply add more text to the input."""
845
855 print('lines', repr(lines)) # dbg
846 856 if lines.startswith('%%'):
847 857 # Cell magics bypass all further transformations
848 858 self.cell_magic_mode = True
849 859 first, _, body = lines.partition('\n')
850 860 magic_name, _, line = first.partition(' ')
851 861 magic_name = magic_name.lstrip(ESC_MAGIC)
852 862 # We store the body of the cell and create a call to a method that
853 863 # will use this stored value. This is ugly, but it's a first cut to
854 864 # get it all working, as right now changing the return API of our
855 865 # methods would require major refactoring.
856 866 self.cell_magic_parts = [body]
857 867 tpl = 'get_ipython()._cell_magic(%r, %r)'
858 868 tlines = tpl % (magic_name, line)
859 869 self._store(tlines)
860 870 self._store(lines, self._buffer_raw, 'source_raw')
861 871 self._is_complete = last_two_blanks(lines)
872 print('IC', self._is_complete) # dbg
862 873 return self._is_complete
863 874
864 875 lines_list = lines.splitlines()
865 876
866 877 transforms = [transform_ipy_prompt, transform_classic_prompt,
867 878 transform_help_end, transform_escaped,
868 879 transform_assign_system, transform_assign_magic]
869 880
870 881 # Transform logic
871 882 #
872 883 # We only apply the line transformers to the input if we have either no
873 884 # input yet, or complete input, or if the last line of the buffer ends
874 885 # with ':' (opening an indented block). This prevents the accidental
875 886 # transformation of escapes inside multiline expressions like
876 887 # triple-quoted strings or parenthesized expressions.
877 888 #
878 889 # The last heuristic, while ugly, ensures that the first line of an
879 890 # indented block is correctly transformed.
880 891 #
881 892 # FIXME: try to find a cleaner approach for this last bit.
882 893
883 894 # In cell mode, since we're going to pump the parent class by hand line
884 895 # by line, we need to temporarily switch out to 'line' mode, do a
885 896 # single manual reset and then feed the lines one by one. Note that
886 897 # this only matters if the input has more than one line.
887 898 self.reset()
888 899 self.input_mode = 'line'
889 900
890 901 # Store raw source before applying any transformations to it. Note
891 902 # that this must be done *after* the reset() call that would otherwise
892 903 # flush the buffer.
893 904 self._store(lines, self._buffer_raw, 'source_raw')
894 905
895 906 try:
896 907 push = super(IPythonInputSplitter, self).push
897 908 buf = self._buffer
898 909 for line in lines_list:
899 910 if self._is_complete or not buf or \
900 911 (buf and buf[-1].rstrip().endswith((':', ','))):
901 912 for f in transforms:
902 913 line = f(line)
903 914
904 915 out = push(line)
905 916 finally:
906 917 self.input_mode = 'cell'
907 918 return out
908 919
909 920 def push(self, lines):
910 921 """Push one or more lines of IPython input.
911 922
912 923 This stores the given lines and returns a status code indicating
913 924 whether the code forms a complete Python block or not, after processing
914 925 all input lines for special IPython syntax.
915 926
916 927 Any exceptions generated in compilation are swallowed, but if an
917 928 exception was produced, the method returns True.
918 929
919 930 Parameters
920 931 ----------
921 932 lines : string
922 933 One or more lines of Python input.
923 934
924 935 Returns
925 936 -------
926 937 is_complete : boolean
927 938 True if the current input source (the result of the current input
928 939 plus prior inputs) forms a complete Python execution block. Note that
929 940 this value is also stored as a private attribute (_is_complete), so it
930 941 can be queried at any time.
931 942 """
932 943 if not lines:
933 944 return super(IPythonInputSplitter, self).push(lines)
934 945
935 946 # We must ensure all input is pure unicode
936 947 lines = cast_unicode(lines, self.encoding)
937 948
938 949 if self.input_mode == 'line':
939 950 return self._push_line_mode(lines)
940 951 else:
941 952 return self._push_cell_mode(lines)
942 953
943 954 lines_list = lines.splitlines()
944 955
945 956 transforms = [transform_ipy_prompt, transform_classic_prompt,
946 957 transform_help_end, transform_escaped,
947 958 transform_assign_system, transform_assign_magic]
948 959
949 960 # Transform logic
950 961 #
951 962 # We only apply the line transformers to the input if we have either no
952 963 # input yet, or complete input, or if the last line of the buffer ends
953 964 # with ':' (opening an indented block). This prevents the accidental
954 965 # transformation of escapes inside multiline expressions like
955 966 # triple-quoted strings or parenthesized expressions.
956 967 #
957 968 # The last heuristic, while ugly, ensures that the first line of an
958 969 # indented block is correctly transformed.
959 970 #
960 971 # FIXME: try to find a cleaner approach for this last bit.
961 972
962 973 # If we were in 'block' mode, since we're going to pump the parent
963 974 # class by hand line by line, we need to temporarily switch out to
964 975 # 'line' mode, do a single manual reset and then feed the lines one
965 976 # by one. Note that this only matters if the input has more than one
966 977 # line.
967 978 changed_input_mode = False
968 979
969 980 if self.input_mode == 'cell':
970 981 self.reset()
971 982 changed_input_mode = True
972 983 saved_input_mode = 'cell'
973 984 self.input_mode = 'line'
974 985
975 986 # Store raw source before applying any transformations to it. Note
976 987 # that this must be done *after* the reset() call that would otherwise
977 988 # flush the buffer.
978 989 self._store(lines, self._buffer_raw, 'source_raw')
979 990
980 991 try:
981 992 push = super(IPythonInputSplitter, self).push
982 993 buf = self._buffer
983 994 for line in lines_list:
984 995 if self._is_complete or not buf or \
985 996 (buf and buf[-1].rstrip().endswith((':', ','))):
986 997 for f in transforms:
987 998 line = f(line)
988 999
989 1000 out = push(line)
990 1001 finally:
991 1002 if changed_input_mode:
992 1003 self.input_mode = saved_input_mode
993 1004 return out
@@ -1,789 +1,794 b''
1 1 # -*- coding: utf-8 -*-
2 2 """Tests for the inputsplitter module.
3 3
4 4 Authors
5 5 -------
6 6 * Fernando Perez
7 7 * Robert Kern
8 8 """
9 9 #-----------------------------------------------------------------------------
10 10 # Copyright (C) 2010-2011 The IPython Development Team
11 11 #
12 12 # Distributed under the terms of the BSD License. The full license is in
13 13 # the file COPYING, distributed as part of this software.
14 14 #-----------------------------------------------------------------------------
15 15
16 16 #-----------------------------------------------------------------------------
17 17 # Imports
18 18 #-----------------------------------------------------------------------------
19 19 # stdlib
20 20 import unittest
21 21 import sys
22 22
23 23 # Third party
24 24 import nose.tools as nt
25 25
26 26 # Our own
27 27 from IPython.core import inputsplitter as isp
28 28 from IPython.testing import tools as tt
29 29 from IPython.utils import py3compat
30 30
31 31 #-----------------------------------------------------------------------------
32 32 # Semi-complete examples (also used as tests)
33 33 #-----------------------------------------------------------------------------
34 34
35 35 # Note: at the bottom, there's a slightly more complete version of this that
36 36 # can be useful during development of code here.
37 37
38 38 def mini_interactive_loop(input_func):
39 39 """Minimal example of the logic of an interactive interpreter loop.
40 40
41 41 This serves as an example, and it is used by the test system with a fake
42 42 raw_input that simulates interactive input."""
43 43
44 44 from IPython.core.inputsplitter import InputSplitter
45 45
46 46 isp = InputSplitter()
47 47 # In practice, this input loop would be wrapped in an outside loop to read
48 48 # input indefinitely, until some exit/quit command was issued. Here we
49 49 # only illustrate the basic inner loop.
50 50 while isp.push_accepts_more():
51 51 indent = ' '*isp.indent_spaces
52 52 prompt = '>>> ' + indent
53 53 line = indent + input_func(prompt)
54 54 isp.push(line)
55 55
56 56 # Here we just return input so we can use it in a test suite, but a real
57 57 # interpreter would instead send it for execution somewhere.
58 58 src = isp.source_reset()
59 59 #print 'Input source was:\n', src # dbg
60 60 return src
61 61
62 62 #-----------------------------------------------------------------------------
63 63 # Test utilities, just for local use
64 64 #-----------------------------------------------------------------------------
65 65
66 66 def assemble(block):
67 67 """Assemble a block into multi-line sub-blocks."""
68 68 return ['\n'.join(sub_block)+'\n' for sub_block in block]
69 69
70 70
71 71 def pseudo_input(lines):
72 72 """Return a function that acts like raw_input but feeds the input list."""
73 73 ilines = iter(lines)
74 74 def raw_in(prompt):
75 75 try:
76 76 return next(ilines)
77 77 except StopIteration:
78 78 return ''
79 79 return raw_in
80 80
81 81 #-----------------------------------------------------------------------------
82 82 # Tests
83 83 #-----------------------------------------------------------------------------
84 84 def test_spaces():
85 85 tests = [('', 0),
86 86 (' ', 1),
87 87 ('\n', 0),
88 88 (' \n', 1),
89 89 ('x', 0),
90 90 (' x', 1),
91 91 (' x',2),
92 92 (' x',4),
93 93 # Note: tabs are counted as a single whitespace!
94 94 ('\tx', 1),
95 95 ('\t x', 2),
96 96 ]
97 97 tt.check_pairs(isp.num_ini_spaces, tests)
98 98
99 99
100 100 def test_remove_comments():
101 101 tests = [('text', 'text'),
102 102 ('text # comment', 'text '),
103 103 ('text # comment\n', 'text \n'),
104 104 ('text # comment \n', 'text \n'),
105 105 ('line # c \nline\n','line \nline\n'),
106 106 ('line # c \nline#c2 \nline\nline #c\n\n',
107 107 'line \nline\nline\nline \n\n'),
108 108 ]
109 109 tt.check_pairs(isp.remove_comments, tests)
110 110
111 111 def test_has_comment():
112 112 tests = [('text', False),
113 113 ('text #comment', True),
114 114 ('text #comment\n', True),
115 115 ('#comment', True),
116 116 ('#comment\n', True),
117 117 ('a = "#string"', False),
118 118 ('a = "#string" # comment', True),
119 119 ('a #comment not "string"', True),
120 120 ]
121 121 tt.check_pairs(isp.has_comment, tests)
122 122
123 123
124 124 def test_get_input_encoding():
125 125 encoding = isp.get_input_encoding()
126 126 nt.assert_true(isinstance(encoding, basestring))
127 127 # simple-minded check that at least encoding a simple string works with the
128 128 # encoding we got.
129 129 nt.assert_equal(u'test'.encode(encoding), b'test')
130 130
131 131
132 132 class NoInputEncodingTestCase(unittest.TestCase):
133 133 def setUp(self):
134 134 self.old_stdin = sys.stdin
135 135 class X: pass
136 136 fake_stdin = X()
137 137 sys.stdin = fake_stdin
138 138
139 139 def test(self):
140 140 # Verify that if sys.stdin has no 'encoding' attribute we do the right
141 141 # thing
142 142 enc = isp.get_input_encoding()
143 143 self.assertEqual(enc, 'ascii')
144 144
145 145 def tearDown(self):
146 146 sys.stdin = self.old_stdin
147 147
148 148
149 149 class InputSplitterTestCase(unittest.TestCase):
150 150 def setUp(self):
151 151 self.isp = isp.InputSplitter()
152 152
153 153 def test_reset(self):
154 154 isp = self.isp
155 155 isp.push('x=1')
156 156 isp.reset()
157 157 self.assertEqual(isp._buffer, [])
158 158 self.assertEqual(isp.indent_spaces, 0)
159 159 self.assertEqual(isp.source, '')
160 160 self.assertEqual(isp.code, None)
161 161 self.assertEqual(isp._is_complete, False)
162 162
163 163 def test_source(self):
164 164 self.isp._store('1')
165 165 self.isp._store('2')
166 166 self.assertEqual(self.isp.source, '1\n2\n')
167 167 self.assertTrue(len(self.isp._buffer)>0)
168 168 self.assertEqual(self.isp.source_reset(), '1\n2\n')
169 169 self.assertEqual(self.isp._buffer, [])
170 170 self.assertEqual(self.isp.source, '')
171 171
172 172 def test_indent(self):
173 173 isp = self.isp # shorthand
174 174 isp.push('x=1')
175 175 self.assertEqual(isp.indent_spaces, 0)
176 176 isp.push('if 1:\n x=1')
177 177 self.assertEqual(isp.indent_spaces, 4)
178 178 isp.push('y=2\n')
179 179 self.assertEqual(isp.indent_spaces, 0)
180 180
181 181 def test_indent2(self):
182 182 # In cell mode, inputs must be fed in whole blocks, so skip this test
183 183 if self.isp.input_mode == 'cell': return
184 184
185 185 isp = self.isp
186 186 isp.push('if 1:')
187 187 self.assertEqual(isp.indent_spaces, 4)
188 188 isp.push(' x=1')
189 189 self.assertEqual(isp.indent_spaces, 4)
190 190 # Blank lines shouldn't change the indent level
191 191 isp.push(' '*2)
192 192 self.assertEqual(isp.indent_spaces, 4)
193 193
194 194 def test_indent3(self):
195 195 # In cell mode, inputs must be fed in whole blocks, so skip this test
196 196 if self.isp.input_mode == 'cell': return
197 197
198 198 isp = self.isp
199 199 # When a multiline statement contains parens or multiline strings, we
200 200 # shouldn't get confused.
201 201 isp.push("if 1:")
202 202 isp.push(" x = (1+\n 2)")
203 203 self.assertEqual(isp.indent_spaces, 4)
204 204
205 205 def test_indent4(self):
206 206 # In cell mode, inputs must be fed in whole blocks, so skip this test
207 207 if self.isp.input_mode == 'cell': return
208 208
209 209 isp = self.isp
210 210 # whitespace after ':' should not screw up indent level
211 211 isp.push('if 1: \n x=1')
212 212 self.assertEqual(isp.indent_spaces, 4)
213 213 isp.push('y=2\n')
214 214 self.assertEqual(isp.indent_spaces, 0)
215 215 isp.push('if 1:\t\n x=1')
216 216 self.assertEqual(isp.indent_spaces, 4)
217 217 isp.push('y=2\n')
218 218 self.assertEqual(isp.indent_spaces, 0)
219 219
220 220 def test_dedent_pass(self):
221 221 isp = self.isp # shorthand
222 222 # should NOT cause dedent
223 223 isp.push('if 1:\n passes = 5')
224 224 self.assertEqual(isp.indent_spaces, 4)
225 225 isp.push('if 1:\n pass')
226 226 self.assertEqual(isp.indent_spaces, 0)
227 227 isp.push('if 1:\n pass ')
228 228 self.assertEqual(isp.indent_spaces, 0)
229 229
230 230 def test_dedent_raise(self):
231 231 isp = self.isp # shorthand
232 232 # should NOT cause dedent
233 233 isp.push('if 1:\n raised = 4')
234 234 self.assertEqual(isp.indent_spaces, 4)
235 235 isp.push('if 1:\n raise TypeError()')
236 236 self.assertEqual(isp.indent_spaces, 0)
237 237 isp.push('if 1:\n raise')
238 238 self.assertEqual(isp.indent_spaces, 0)
239 239 isp.push('if 1:\n raise ')
240 240 self.assertEqual(isp.indent_spaces, 0)
241 241
242 242 def test_dedent_return(self):
243 243 isp = self.isp # shorthand
244 244 # should NOT cause dedent
245 245 isp.push('if 1:\n returning = 4')
246 246 self.assertEqual(isp.indent_spaces, 4)
247 247 isp.push('if 1:\n return 5 + 493')
248 248 self.assertEqual(isp.indent_spaces, 0)
249 249 isp.push('if 1:\n return')
250 250 self.assertEqual(isp.indent_spaces, 0)
251 251 isp.push('if 1:\n return ')
252 252 self.assertEqual(isp.indent_spaces, 0)
253 253 isp.push('if 1:\n return(0)')
254 254 self.assertEqual(isp.indent_spaces, 0)
255 255
256 256 def test_push(self):
257 257 isp = self.isp
258 258 self.assertTrue(isp.push('x=1'))
259 259
260 260 def test_push2(self):
261 261 isp = self.isp
262 262 self.assertFalse(isp.push('if 1:'))
263 263 for line in [' x=1', '# a comment', ' y=2']:
264 264 self.assertTrue(isp.push(line))
265 265
266 266 def test_push3(self):
267 267 isp = self.isp
268 268 isp.push('if True:')
269 269 isp.push(' a = 1')
270 270 self.assertFalse(isp.push('b = [1,'))
271 271
272 272 def test_replace_mode(self):
273 273 isp = self.isp
274 274 isp.input_mode = 'cell'
275 275 isp.push('x=1')
276 276 self.assertEqual(isp.source, 'x=1\n')
277 277 isp.push('x=2')
278 278 self.assertEqual(isp.source, 'x=2\n')
279 279
280 280 def test_push_accepts_more(self):
281 281 isp = self.isp
282 282 isp.push('x=1')
283 283 self.assertFalse(isp.push_accepts_more())
284 284
285 285 def test_push_accepts_more2(self):
286 286 # In cell mode, inputs must be fed in whole blocks, so skip this test
287 287 if self.isp.input_mode == 'cell': return
288 288
289 289 isp = self.isp
290 290 isp.push('if 1:')
291 291 self.assertTrue(isp.push_accepts_more())
292 292 isp.push(' x=1')
293 293 self.assertTrue(isp.push_accepts_more())
294 294 isp.push('')
295 295 self.assertFalse(isp.push_accepts_more())
296 296
297 297 def test_push_accepts_more3(self):
298 298 isp = self.isp
299 299 isp.push("x = (2+\n3)")
300 300 self.assertFalse(isp.push_accepts_more())
301 301
302 302 def test_push_accepts_more4(self):
303 303 # In cell mode, inputs must be fed in whole blocks, so skip this test
304 304 if self.isp.input_mode == 'cell': return
305 305
306 306 isp = self.isp
307 307 # When a multiline statement contains parens or multiline strings, we
308 308 # shouldn't get confused.
309 309 # FIXME: we should be able to better handle de-dents in statements like
310 310 # multiline strings and multiline expressions (continued with \ or
311 311 # parens). Right now we aren't handling the indentation tracking quite
312 312 # correctly with this, though in practice it may not be too much of a
313 313 # problem. We'll need to see.
314 314 isp.push("if 1:")
315 315 isp.push(" x = (2+")
316 316 isp.push(" 3)")
317 317 self.assertTrue(isp.push_accepts_more())
318 318 isp.push(" y = 3")
319 319 self.assertTrue(isp.push_accepts_more())
320 320 isp.push('')
321 321 self.assertFalse(isp.push_accepts_more())
322 322
323 323 def test_push_accepts_more5(self):
324 324 # In cell mode, inputs must be fed in whole blocks, so skip this test
325 325 if self.isp.input_mode == 'cell': return
326 326
327 327 isp = self.isp
328 328 isp.push('try:')
329 329 isp.push(' a = 5')
330 330 isp.push('except:')
331 331 isp.push(' raise')
332 332 self.assertTrue(isp.push_accepts_more())
333 333
334 334 def test_continuation(self):
335 335 isp = self.isp
336 336 isp.push("import os, \\")
337 337 self.assertTrue(isp.push_accepts_more())
338 338 isp.push("sys")
339 339 self.assertFalse(isp.push_accepts_more())
340 340
341 341 def test_syntax_error(self):
342 342 isp = self.isp
343 343 # Syntax errors immediately produce a 'ready' block, so the invalid
344 344 # Python can be sent to the kernel for evaluation with possible ipython
345 345 # special-syntax conversion.
346 346 isp.push('run foo')
347 347 self.assertFalse(isp.push_accepts_more())
348 348
349 349 def test_unicode(self):
350 350 self.isp.push(u"PΓ©rez")
351 351 self.isp.push(u'\xc3\xa9')
352 352 self.isp.push(u"u'\xc3\xa9'")
353 353
354 354 class InteractiveLoopTestCase(unittest.TestCase):
355 355 """Tests for an interactive loop like a python shell.
356 356 """
357 357 def check_ns(self, lines, ns):
358 358 """Validate that the given input lines produce the resulting namespace.
359 359
360 360 Note: the input lines are given exactly as they would be typed in an
361 361 auto-indenting environment, as mini_interactive_loop above already does
362 362 auto-indenting and prepends spaces to the input.
363 363 """
364 364 src = mini_interactive_loop(pseudo_input(lines))
365 365 test_ns = {}
366 366 exec src in test_ns
367 367 # We can't check that the provided ns is identical to the test_ns,
368 368 # because Python fills test_ns with extra keys (copyright, etc). But
369 369 # we can check that the given dict is *contained* in test_ns
370 370 for k,v in ns.iteritems():
371 371 self.assertEqual(test_ns[k], v)
372 372
373 373 def test_simple(self):
374 374 self.check_ns(['x=1'], dict(x=1))
375 375
376 376 def test_simple2(self):
377 377 self.check_ns(['if 1:', 'x=2'], dict(x=2))
378 378
379 379 def test_xy(self):
380 380 self.check_ns(['x=1; y=2'], dict(x=1, y=2))
381 381
382 382 def test_abc(self):
383 383 self.check_ns(['if 1:','a=1','b=2','c=3'], dict(a=1, b=2, c=3))
384 384
385 385 def test_multi(self):
386 386 self.check_ns(['x =(1+','1+','2)'], dict(x=4))
387 387
388 388
389 389 def test_LineInfo():
390 390 """Simple test for LineInfo construction and str()"""
391 391 linfo = isp.LineInfo(' %cd /home')
392 392 nt.assert_equals(str(linfo), 'LineInfo [ |%|cd|/home]')
393 393
394 394 # Transformer tests
395 395 def transform_checker(tests, func):
396 396 """Utility to loop over test inputs"""
397 397 for inp, tr in tests:
398 398 nt.assert_equals(func(inp), tr)
399 399
400 400 # Data for all the syntax tests in the form of lists of pairs of
401 401 # raw/transformed input. We store it here as a global dict so that we can use
402 402 # it both within single-function tests and also to validate the behavior of the
403 403 # larger objects
404 404
405 405 syntax = \
406 406 dict(assign_system =
407 407 [(i,py3compat.u_format(o)) for i,o in \
408 408 [(u'a =! ls', "a = get_ipython().getoutput({u}'ls')"),
409 409 (u'b = !ls', "b = get_ipython().getoutput({u}'ls')"),
410 410 ('x=1', 'x=1'), # normal input is unmodified
411 411 (' ',' '), # blank lines are kept intact
412 412 ]],
413 413
414 414 assign_magic =
415 415 [(i,py3compat.u_format(o)) for i,o in \
416 416 [(u'a =% who', "a = get_ipython().magic({u}'who')"),
417 417 (u'b = %who', "b = get_ipython().magic({u}'who')"),
418 418 ('x=1', 'x=1'), # normal input is unmodified
419 419 (' ',' '), # blank lines are kept intact
420 420 ]],
421 421
422 422 classic_prompt =
423 423 [('>>> x=1', 'x=1'),
424 424 ('x=1', 'x=1'), # normal input is unmodified
425 425 (' ', ' '), # blank lines are kept intact
426 426 ('... ', ''), # continuation prompts
427 427 ],
428 428
429 429 ipy_prompt =
430 430 [('In [1]: x=1', 'x=1'),
431 431 ('x=1', 'x=1'), # normal input is unmodified
432 432 (' ',' '), # blank lines are kept intact
433 433 (' ....: ', ''), # continuation prompts
434 434 ],
435 435
436 436 # Tests for the escape transformer to leave normal code alone
437 437 escaped_noesc =
438 438 [ (' ', ' '),
439 439 ('x=1', 'x=1'),
440 440 ],
441 441
442 442 # System calls
443 443 escaped_shell =
444 444 [(i,py3compat.u_format(o)) for i,o in \
445 445 [ (u'!ls', "get_ipython().system({u}'ls')"),
446 446 # Double-escape shell, this means to capture the output of the
447 447 # subprocess and return it
448 448 (u'!!ls', "get_ipython().getoutput({u}'ls')"),
449 449 ]],
450 450
451 451 # Help/object info
452 452 escaped_help =
453 453 [(i,py3compat.u_format(o)) for i,o in \
454 454 [ (u'?', 'get_ipython().show_usage()'),
455 455 (u'?x1', "get_ipython().magic({u}'pinfo x1')"),
456 456 (u'??x2', "get_ipython().magic({u}'pinfo2 x2')"),
457 457 (u'?a.*s', "get_ipython().magic({u}'psearch a.*s')"),
458 458 (u'?%hist', "get_ipython().magic({u}'pinfo %hist')"),
459 459 (u'?abc = qwe', "get_ipython().magic({u}'pinfo abc')"),
460 460 ]],
461 461
462 462 end_help =
463 463 [(i,py3compat.u_format(o)) for i,o in \
464 464 [ (u'x3?', "get_ipython().magic({u}'pinfo x3')"),
465 465 (u'x4??', "get_ipython().magic({u}'pinfo2 x4')"),
466 466 (u'%hist?', "get_ipython().magic({u}'pinfo %hist')"),
467 467 (u'f*?', "get_ipython().magic({u}'psearch f*')"),
468 468 (u'ax.*aspe*?', "get_ipython().magic({u}'psearch ax.*aspe*')"),
469 469 (u'a = abc?', "get_ipython().set_next_input({u}'a = abc');"
470 470 "get_ipython().magic({u}'pinfo abc')"),
471 471 (u'a = abc.qe??', "get_ipython().set_next_input({u}'a = abc.qe');"
472 472 "get_ipython().magic({u}'pinfo2 abc.qe')"),
473 473 (u'a = *.items?', "get_ipython().set_next_input({u}'a = *.items');"
474 474 "get_ipython().magic({u}'psearch *.items')"),
475 475 (u'plot(a?', "get_ipython().set_next_input({u}'plot(a');"
476 476 "get_ipython().magic({u}'pinfo a')"),
477 477 (u'a*2 #comment?', 'a*2 #comment?'),
478 478 ]],
479 479
480 480 # Explicit magic calls
481 481 escaped_magic =
482 482 [(i,py3compat.u_format(o)) for i,o in \
483 483 [ (u'%cd', "get_ipython().magic({u}'cd')"),
484 484 (u'%cd /home', "get_ipython().magic({u}'cd /home')"),
485 485 # Backslashes need to be escaped.
486 486 (u'%cd C:\\User', "get_ipython().magic({u}'cd C:\\\\User')"),
487 487 (u' %magic', " get_ipython().magic({u}'magic')"),
488 488 ]],
489 489
490 490 # Quoting with separate arguments
491 491 escaped_quote =
492 492 [ (',f', 'f("")'),
493 493 (',f x', 'f("x")'),
494 494 (' ,f y', ' f("y")'),
495 495 (',f a b', 'f("a", "b")'),
496 496 ],
497 497
498 498 # Quoting with single argument
499 499 escaped_quote2 =
500 500 [ (';f', 'f("")'),
501 501 (';f x', 'f("x")'),
502 502 (' ;f y', ' f("y")'),
503 503 (';f a b', 'f("a b")'),
504 504 ],
505 505
506 506 # Simply apply parens
507 507 escaped_paren =
508 508 [ ('/f', 'f()'),
509 509 ('/f x', 'f(x)'),
510 510 (' /f y', ' f(y)'),
511 511 ('/f a b', 'f(a, b)'),
512 512 ],
513 513
514 514 # Check that we transform prompts before other transforms
515 515 mixed =
516 516 [(i,py3compat.u_format(o)) for i,o in \
517 517 [ (u'In [1]: %lsmagic', "get_ipython().magic({u}'lsmagic')"),
518 518 (u'>>> %lsmagic', "get_ipython().magic({u}'lsmagic')"),
519 519 (u'In [2]: !ls', "get_ipython().system({u}'ls')"),
520 520 (u'In [3]: abs?', "get_ipython().magic({u}'pinfo abs')"),
521 521 (u'In [4]: b = %who', "b = get_ipython().magic({u}'who')"),
522 522 ]],
523 523 )
524 524
525 525 # multiline syntax examples. Each of these should be a list of lists, with
526 526 # each entry itself having pairs of raw/transformed input. The union (with
527 527 # '\n'.join() of the transformed inputs is what the splitter should produce
528 528 # when fed the raw lines one at a time via push.
529 529 syntax_ml = \
530 530 dict(classic_prompt =
531 531 [ [('>>> for i in range(10):','for i in range(10):'),
532 532 ('... print i',' print i'),
533 533 ('... ', ''),
534 534 ],
535 535 ],
536 536
537 537 ipy_prompt =
538 538 [ [('In [24]: for i in range(10):','for i in range(10):'),
539 539 (' ....: print i',' print i'),
540 540 (' ....: ', ''),
541 541 ],
542 542 ],
543 543
544 544 multiline_datastructure =
545 545 [ [('>>> a = [1,','a = [1,'),
546 546 ('... 2]','2]'),
547 547 ],
548 548 ],
549 549 )
550 550
551 551
552 552 def test_assign_system():
553 553 tt.check_pairs(isp.transform_assign_system, syntax['assign_system'])
554 554
555 555
556 556 def test_assign_magic():
557 557 tt.check_pairs(isp.transform_assign_magic, syntax['assign_magic'])
558 558
559 559
560 560 def test_classic_prompt():
561 561 transform_checker(syntax['classic_prompt'], isp.transform_classic_prompt)
562 562 for example in syntax_ml['classic_prompt']:
563 563 transform_checker(example, isp.transform_classic_prompt)
564 564
565 565
566 566 def test_ipy_prompt():
567 567 transform_checker(syntax['ipy_prompt'], isp.transform_ipy_prompt)
568 568 for example in syntax_ml['ipy_prompt']:
569 569 transform_checker(example, isp.transform_ipy_prompt)
570 570
571 571 def test_end_help():
572 572 tt.check_pairs(isp.transform_help_end, syntax['end_help'])
573 573
574 574 def test_escaped_noesc():
575 575 tt.check_pairs(isp.transform_escaped, syntax['escaped_noesc'])
576 576
577 577
578 578 def test_escaped_shell():
579 579 tt.check_pairs(isp.transform_escaped, syntax['escaped_shell'])
580 580
581 581
582 582 def test_escaped_help():
583 583 tt.check_pairs(isp.transform_escaped, syntax['escaped_help'])
584 584
585 585
586 586 def test_escaped_magic():
587 587 tt.check_pairs(isp.transform_escaped, syntax['escaped_magic'])
588 588
589 589
590 590 def test_escaped_quote():
591 591 tt.check_pairs(isp.transform_escaped, syntax['escaped_quote'])
592 592
593 593
594 594 def test_escaped_quote2():
595 595 tt.check_pairs(isp.transform_escaped, syntax['escaped_quote2'])
596 596
597 597
598 598 def test_escaped_paren():
599 599 tt.check_pairs(isp.transform_escaped, syntax['escaped_paren'])
600 600
601 601
602 602 def test_last_blank():
603 603 nt.assert_false(isp.last_blank(''))
604 604 nt.assert_false(isp.last_blank('abc'))
605 605 nt.assert_false(isp.last_blank('abc\n'))
606 606 nt.assert_false(isp.last_blank('abc\na'))
607 607
608 608 nt.assert_true(isp.last_blank('\n'))
609 609 nt.assert_true(isp.last_blank('\n '))
610 610 nt.assert_true(isp.last_blank('abc\n '))
611 611 nt.assert_true(isp.last_blank('abc\n\n'))
612 nt.assert_true(isp.last_blank('abc\nd\n\n'))
613 nt.assert_true(isp.last_blank('abc\nd\ne\n\n'))
614 nt.assert_true(isp.last_blank('abc \n \n \n\n'))
612 615
613 616
614 617 def test_last_two_blanks():
615 618 nt.assert_false(isp.last_two_blanks(''))
616 619 nt.assert_false(isp.last_two_blanks('abc'))
617 620 nt.assert_false(isp.last_two_blanks('abc\n'))
618 621 nt.assert_false(isp.last_two_blanks('abc\n\na'))
619 622 nt.assert_false(isp.last_two_blanks('abc\n \n'))
620 623 nt.assert_false(isp.last_two_blanks('abc\n\n'))
621 624
622 625 nt.assert_true(isp.last_two_blanks('\n\n'))
623 626 nt.assert_true(isp.last_two_blanks('\n\n '))
624 627 nt.assert_true(isp.last_two_blanks('\n \n'))
625 628 nt.assert_true(isp.last_two_blanks('abc\n\n '))
626 629 nt.assert_true(isp.last_two_blanks('abc\n\n\n'))
627 630 nt.assert_true(isp.last_two_blanks('abc\n\n \n'))
628 631 nt.assert_true(isp.last_two_blanks('abc\n\n \n '))
629 632 nt.assert_true(isp.last_two_blanks('abc\n\n \n \n'))
633 nt.assert_true(isp.last_two_blanks('abc\nd\n\n\n'))
634 nt.assert_true(isp.last_two_blanks('abc\nd\ns\nds\n\n\n'))
630 635
631 636
632 637 def test_cell_magics_line_mode():
633 638
634 639 cell = """\
635 640 %%cellm line
636 641 body
637 642 """
638 643 sp = isp.IPythonInputSplitter(input_mode='line')
639 644 sp.push(cell)
640 645 nt.assert_equal(sp.cell_magic_parts, ['body\n'])
641 646 out = sp.source
642 647 ref = u"get_ipython()._cell_magic(u'cellm', u'line')\n"
643 648 nt.assert_equal(out, ref)
644 649
645 650 sp.reset()
646 651
647 652 sp.push('%%cellm line2\n')
648 653 nt.assert_true(sp.push_accepts_more()) #1
649 654 sp.push('\n')
650 655 nt.assert_true(sp.push_accepts_more()) #2
651 656 sp.push('\n')
652 657 nt.assert_false(sp.push_accepts_more()) #3
653 658
654 659
655 660 def test_cell_magics_cell_mode():
656 661
657 662 cell = """\
658 663 %%cellm line
659 664 body
660 665 """
661 666 sp = isp.IPythonInputSplitter(input_mode='cell')
662 667 sp.push(cell)
663 668 nt.assert_equal(sp.cell_magic_parts, ['body\n'])
664 669 out = sp.source
665 670 ref = u"get_ipython()._cell_magic(u'cellm', u'line')\n"
666 671 nt.assert_equal(out, ref)
667 672
668 673 sp.reset()
669 674
670 675 src = '%%cellm line2\n'
671 676 sp.push(src)
672 677 nt.assert_true(sp.push_accepts_more()) #1
673 678 src += '\n'
674 679 sp.push(src)
675 680 nt.assert_true(sp.push_accepts_more()) #2
676 681 src += '\n'
677 682 sp.push(src)
678 683 nt.assert_false(sp.push_accepts_more()) #3
679 684
680 685
681 686 class IPythonInputTestCase(InputSplitterTestCase):
682 687 """By just creating a new class whose .isp is a different instance, we
683 688 re-run the same test battery on the new input splitter.
684 689
685 690 In addition, this runs the tests over the syntax and syntax_ml dicts that
686 691 were tested by individual functions, as part of the OO interface.
687 692
688 693 It also makes some checks on the raw buffer storage.
689 694 """
690 695
691 696 def setUp(self):
692 697 self.isp = isp.IPythonInputSplitter(input_mode='line')
693 698
694 699 def test_syntax(self):
695 700 """Call all single-line syntax tests from the main object"""
696 701 isp = self.isp
697 702 for example in syntax.itervalues():
698 703 for raw, out_t in example:
699 704 if raw.startswith(' '):
700 705 continue
701 706
702 707 isp.push(raw)
703 708 out, out_raw = isp.source_raw_reset()
704 709 self.assertEqual(out.rstrip(), out_t,
705 710 tt.pair_fail_msg.format("inputsplitter",raw, out_t, out))
706 711 self.assertEqual(out_raw.rstrip(), raw.rstrip())
707 712
708 713 def test_syntax_multiline(self):
709 714 isp = self.isp
710 715 for example in syntax_ml.itervalues():
711 716 out_t_parts = []
712 717 raw_parts = []
713 718 for line_pairs in example:
714 719 for lraw, out_t_part in line_pairs:
715 720 isp.push(lraw)
716 721 out_t_parts.append(out_t_part)
717 722 raw_parts.append(lraw)
718 723
719 724 out, out_raw = isp.source_raw_reset()
720 725 out_t = '\n'.join(out_t_parts).rstrip()
721 726 raw = '\n'.join(raw_parts).rstrip()
722 727 self.assertEqual(out.rstrip(), out_t)
723 728 self.assertEqual(out_raw.rstrip(), raw)
724 729
725 730
726 731 class BlockIPythonInputTestCase(IPythonInputTestCase):
727 732
728 733 # Deactivate tests that don't make sense for the block mode
729 734 test_push3 = test_split = lambda s: None
730 735
731 736 def setUp(self):
732 737 self.isp = isp.IPythonInputSplitter(input_mode='cell')
733 738
734 739 def test_syntax_multiline(self):
735 740 isp = self.isp
736 741 for example in syntax_ml.itervalues():
737 742 raw_parts = []
738 743 out_t_parts = []
739 744 for line_pairs in example:
740 745 for raw, out_t_part in line_pairs:
741 746 raw_parts.append(raw)
742 747 out_t_parts.append(out_t_part)
743 748
744 749 raw = '\n'.join(raw_parts)
745 750 out_t = '\n'.join(out_t_parts)
746 751
747 752 isp.push(raw)
748 753 out, out_raw = isp.source_raw_reset()
749 754 # Match ignoring trailing whitespace
750 755 self.assertEqual(out.rstrip(), out_t.rstrip())
751 756 self.assertEqual(out_raw.rstrip(), raw.rstrip())
752 757
753 758
754 759 #-----------------------------------------------------------------------------
755 760 # Main - use as a script, mostly for developer experiments
756 761 #-----------------------------------------------------------------------------
757 762
758 763 if __name__ == '__main__':
759 764 # A simple demo for interactive experimentation. This code will not get
760 765 # picked up by any test suite.
761 766 from IPython.core.inputsplitter import InputSplitter, IPythonInputSplitter
762 767
763 768 # configure here the syntax to use, prompt and whether to autoindent
764 769 #isp, start_prompt = InputSplitter(), '>>> '
765 770 isp, start_prompt = IPythonInputSplitter(), 'In> '
766 771
767 772 autoindent = True
768 773 #autoindent = False
769 774
770 775 try:
771 776 while True:
772 777 prompt = start_prompt
773 778 while isp.push_accepts_more():
774 779 indent = ' '*isp.indent_spaces
775 780 if autoindent:
776 781 line = indent + raw_input(prompt+indent)
777 782 else:
778 783 line = raw_input(prompt)
779 784 isp.push(line)
780 785 prompt = '... '
781 786
782 787 # Here we just return input so we can use it in a test suite, but a
783 788 # real interpreter would instead send it for execution somewhere.
784 789 #src = isp.source; raise EOFError # dbg
785 790 src, raw = isp.source_raw_reset()
786 791 print 'Input source was:\n', src
787 792 print 'Raw source was:\n', raw
788 793 except EOFError:
789 794 print 'Bye'
General Comments 0
You need to be logged in to leave comments. Login now