##// END OF EJS Templates
Working implementation of cell mode with regular expressions - cleaner.
Fernando Perez -
Show More
@@ -1,910 +1,991 b''
1 1 """Analysis of text input into executable blocks.
2 2
3 3 The main class in this module, :class:`InputSplitter`, is designed to break
4 4 input from either interactive, line-by-line environments or block-based ones,
5 5 into standalone blocks that can be executed by Python as 'single' statements
6 6 (thus triggering sys.displayhook).
7 7
8 8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 9 with full support for the extended IPython syntax (magics, system calls, etc).
10 10
11 11 For more details, see the class docstring below.
12 12
13 13 Syntax Transformations
14 14 ----------------------
15 15
16 16 One of the main jobs of the code in this file is to apply all syntax
17 17 transformations that make up 'the IPython language', i.e. magics, shell
18 18 escapes, etc. All transformations should be implemented as *fully stateless*
19 19 entities, that simply take one line as their input and return a line.
20 20 Internally for implementation purposes they may be a normal function or a
21 21 callable object, but the only input they receive will be a single line and they
22 22 should only return a line, without holding any data-dependent state between
23 23 calls.
24 24
25 25 As an example, the EscapedTransformer is a class so we can more clearly group
26 26 together the functionality of dispatching to individual functions based on the
27 27 starting escape character, but the only method for public use is its call
28 28 method.
29 29
30 30
31 31 ToDo
32 32 ----
33 33
34 34 - Should we make push() actually raise an exception once push_accepts_more()
35 35 returns False?
36 36
37 37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
38 38 at least just attributes of a class so not really very exposed.
39 39
40 40 - Think about the best way to support dynamic things: automagic, autocall,
41 41 macros, etc.
42 42
43 43 - Think of a better heuristic for the application of the transforms in
44 44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
45 45 track indentation change events (indent, dedent, nothing) and apply them only
46 46 if the indentation went up, but not otherwise.
47 47
48 48 - Think of the cleanest way for supporting user-specified transformations (the
49 49 user prefilters we had before).
50 50
51 51 Authors
52 52 -------
53 53
54 54 * Fernando Perez
55 55 * Brian Granger
56 56 """
57 57 #-----------------------------------------------------------------------------
58 58 # Copyright (C) 2010 The IPython Development Team
59 59 #
60 60 # Distributed under the terms of the BSD License. The full license is in
61 61 # the file COPYING, distributed as part of this software.
62 62 #-----------------------------------------------------------------------------
63 63 from __future__ import print_function
64 64
65 65 #-----------------------------------------------------------------------------
66 66 # Imports
67 67 #-----------------------------------------------------------------------------
68 68 # stdlib
69 69 import ast
70 70 import codeop
71 71 import re
72 72 import sys
73 73 import tokenize
74 74 from StringIO import StringIO
75 75
76 76 # IPython modules
77 77 from IPython.core.splitinput import split_user_input, LineInfo
78 78 from IPython.utils.py3compat import cast_unicode
79 79
80 80 #-----------------------------------------------------------------------------
81 81 # Globals
82 82 #-----------------------------------------------------------------------------
83 83
84 84 # The escape sequences that define the syntax transformations IPython will
85 85 # apply to user input. These can NOT be just changed here: many regular
86 86 # expressions and other parts of the code may use their hardcoded values, and
87 87 # for all intents and purposes they constitute the 'IPython syntax', so they
88 88 # should be considered fixed.
89 89
90 90 ESC_SHELL = '!' # Send line to underlying system shell
91 91 ESC_SH_CAP = '!!' # Send line to system shell and capture output
92 92 ESC_HELP = '?' # Find information about object
93 93 ESC_HELP2 = '??' # Find extra-detailed information about object
94 94 ESC_MAGIC = '%' # Call magic function
95 95 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
96 96 ESC_QUOTE2 = ';' # Quote all args as a single string, call
97 97 ESC_PAREN = '/' # Call first argument with rest of line as arguments
98 98
99 99 #-----------------------------------------------------------------------------
100 100 # Utilities
101 101 #-----------------------------------------------------------------------------
102 102
103 103 # FIXME: These are general-purpose utilities that later can be moved to the
104 104 # general ward. Kept here for now because we're being very strict about test
105 105 # coverage with this code, and this lets us ensure that we keep 100% coverage
106 106 # while developing.
107 107
108 108 # compiled regexps for autoindent management
109 109 dedent_re = re.compile('|'.join([
110 110 r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
111 111 r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
112 112 r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
113 113 r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
114 114 r'^\s+pass\s*$' # pass (optionally followed by trailing spaces)
115 115 ]))
116 116 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
117 117
118 118 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
119 119 # before pure comments
120 120 comment_line_re = re.compile('^\s*\#')
121 121
122 122
123 123 def num_ini_spaces(s):
124 124 """Return the number of initial spaces in a string.
125 125
126 126 Note that tabs are counted as a single space. For now, we do *not* support
127 127 mixing of tabs and spaces in the user's input.
128 128
129 129 Parameters
130 130 ----------
131 131 s : string
132 132
133 133 Returns
134 134 -------
135 135 n : int
136 136 """
137 137
138 138 ini_spaces = ini_spaces_re.match(s)
139 139 if ini_spaces:
140 140 return ini_spaces.end()
141 141 else:
142 142 return 0
143 143
144 last_blank_re = re.compile(r'^.*\n\s+$', re.MULTILINE)
144 145
145 146 def last_blank(src):
146 147 """Determine if the input source ends in a blank.
147 148
148 149 A blank is either a newline or a line consisting of whitespace.
149 150
150 151 Parameters
151 152 ----------
152 153 src : string
153 154 A single or multiline string.
154 155 """
155 if not src: return False
156 ll = src.splitlines()[-1]
157 return (ll == '') or ll.isspace()
156 return src == '\n' or bool(last_blank_re.match(src))
157
158
159 last_two_blanks_re = re.compile(r'^.*\n\s*\n\s*$', re.MULTILINE)
160
161 def last_two_blanks(src):
162 """Determine if the input source ends in two blanks.
163
164 A blank is either a newline or a line consisting of whitespace.
165
166 Parameters
167 ----------
168 src : string
169 A single or multiline string.
170 """
171 return bool(last_two_blanks_re.match(src))
158 172
159 173
160 174 def remove_comments(src):
161 175 """Remove all comments from input source.
162 176
163 177 Note: comments are NOT recognized inside of strings!
164 178
165 179 Parameters
166 180 ----------
167 181 src : string
168 182 A single or multiline input string.
169 183
170 184 Returns
171 185 -------
172 186 String with all Python comments removed.
173 187 """
174 188
175 189 return re.sub('#.*', '', src)
176 190
177 191 def has_comment(src):
178 192 """Indicate whether an input line has (i.e. ends in, or is) a comment.
179 193
180 194 This uses tokenize, so it can distinguish comments from # inside strings.
181 195
182 196 Parameters
183 197 ----------
184 198 src : string
185 199 A single line input string.
186 200
187 201 Returns
188 202 -------
189 203 Boolean: True if source has a comment.
190 204 """
191 205 readline = StringIO(src).readline
192 206 toktypes = set()
193 207 try:
194 208 for t in tokenize.generate_tokens(readline):
195 209 toktypes.add(t[0])
196 210 except tokenize.TokenError:
197 211 pass
198 212 return(tokenize.COMMENT in toktypes)
199 213
200 214
201 215 def get_input_encoding():
202 216 """Return the default standard input encoding.
203 217
204 218 If sys.stdin has no encoding, 'ascii' is returned."""
205 219 # There are strange environments for which sys.stdin.encoding is None. We
206 220 # ensure that a valid encoding is returned.
207 221 encoding = getattr(sys.stdin, 'encoding', None)
208 222 if encoding is None:
209 223 encoding = 'ascii'
210 224 return encoding
211 225
212 226 #-----------------------------------------------------------------------------
213 227 # Classes and functions for normal Python syntax handling
214 228 #-----------------------------------------------------------------------------
215 229
216 230 class InputSplitter(object):
217 231 """An object that can accumulate lines of Python source before execution.
218 232
219 233 This object is designed to be fed python source line-by-line, using
220 234 :meth:`push`. It will return on each push whether the currently pushed
221 235 code could be executed already. In addition, it provides a method called
222 236 :meth:`push_accepts_more` that can be used to query whether more input
223 237 can be pushed into a single interactive block.
224 238
225 239 This is a simple example of how an interactive terminal-based client can use
226 240 this tool::
227 241
228 242 isp = InputSplitter()
229 243 while isp.push_accepts_more():
230 244 indent = ' '*isp.indent_spaces
231 245 prompt = '>>> ' + indent
232 246 line = indent + raw_input(prompt)
233 247 isp.push(line)
234 248 print 'Input source was:\n', isp.source_reset(),
235 249 """
236 250 # Number of spaces of indentation computed from input that has been pushed
237 251 # so far. This is the attributes callers should query to get the current
238 252 # indentation level, in order to provide auto-indent facilities.
239 253 indent_spaces = 0
240 254 # String, indicating the default input encoding. It is computed by default
241 255 # at initialization time via get_input_encoding(), but it can be reset by a
242 256 # client with specific knowledge of the encoding.
243 257 encoding = ''
244 258 # String where the current full source input is stored, properly encoded.
245 259 # Reading this attribute is the normal way of querying the currently pushed
246 260 # source code, that has been properly encoded.
247 261 source = ''
248 262 # Code object corresponding to the current source. It is automatically
249 263 # synced to the source, so it can be queried at any time to obtain the code
250 264 # object; it will be None if the source doesn't compile to valid Python.
251 265 code = None
252 266 # Input mode
253 267 input_mode = 'line'
254 268
255 269 # Private attributes
256 270
257 271 # List with lines of input accumulated so far
258 272 _buffer = None
259 273 # Command compiler
260 274 _compile = None
261 275 # Mark when input has changed indentation all the way back to flush-left
262 276 _full_dedent = False
263 277 # Boolean indicating whether the current block is complete
264 278 _is_complete = None
265 279
266 280 def __init__(self, input_mode=None):
267 281 """Create a new InputSplitter instance.
268 282
269 283 Parameters
270 284 ----------
271 285 input_mode : str
272 286
273 287 One of ['line', 'cell']; default is 'line'.
274 288
275 289 The input_mode parameter controls how new inputs are used when fed via
276 290 the :meth:`push` method:
277 291
278 292 - 'line': meant for line-oriented clients, inputs are appended one at a
279 293 time to the internal buffer and the whole buffer is compiled.
280 294
281 295 - 'cell': meant for clients that can edit multi-line 'cells' of text at
282 296 a time. A cell can contain one or more blocks that can be compile in
283 297 'single' mode by Python. In this mode, each new input new input
284 298 completely replaces all prior inputs. Cell mode is thus equivalent
285 299 to prepending a full reset() to every push() call.
286 300 """
287 301 self._buffer = []
288 302 self._compile = codeop.CommandCompiler()
289 303 self.encoding = get_input_encoding()
290 304 self.input_mode = InputSplitter.input_mode if input_mode is None \
291 305 else input_mode
292 306
293 307 def reset(self):
294 308 """Reset the input buffer and associated state."""
295 309 self.indent_spaces = 0
296 310 self._buffer[:] = []
297 311 self.source = ''
298 312 self.code = None
299 313 self._is_complete = False
300 314 self._full_dedent = False
301 315
302 316 def source_reset(self):
303 317 """Return the input source and perform a full reset.
304 318 """
305 319 out = self.source
306 320 self.reset()
307 321 return out
308 322
309 323 def push(self, lines):
310 324 """Push one or more lines of input.
311 325
312 326 This stores the given lines and returns a status code indicating
313 327 whether the code forms a complete Python block or not.
314 328
315 329 Any exceptions generated in compilation are swallowed, but if an
316 330 exception was produced, the method returns True.
317 331
318 332 Parameters
319 333 ----------
320 334 lines : string
321 335 One or more lines of Python input.
322 336
323 337 Returns
324 338 -------
325 339 is_complete : boolean
326 340 True if the current input source (the result of the current input
327 341 plus prior inputs) forms a complete Python execution block. Note that
328 342 this value is also stored as a private attribute (_is_complete), so it
329 343 can be queried at any time.
330 344 """
331 345 if self.input_mode == 'cell':
332 346 self.reset()
333 347
334 348 self._store(lines)
335 349 source = self.source
336 350
337 351 # Before calling _compile(), reset the code object to None so that if an
338 352 # exception is raised in compilation, we don't mislead by having
339 353 # inconsistent code/source attributes.
340 354 self.code, self._is_complete = None, None
341 355
342 356 # Honor termination lines properly
343 357 if source.rstrip().endswith('\\'):
344 358 return False
345 359
346 360 self._update_indent(lines)
347 361 try:
348 362 self.code = self._compile(source, symbol="exec")
349 363 # Invalid syntax can produce any of a number of different errors from
350 364 # inside the compiler, so we have to catch them all. Syntax errors
351 365 # immediately produce a 'ready' block, so the invalid Python can be
352 366 # sent to the kernel for evaluation with possible ipython
353 367 # special-syntax conversion.
354 368 except (SyntaxError, OverflowError, ValueError, TypeError,
355 369 MemoryError):
356 370 self._is_complete = True
357 371 else:
358 372 # Compilation didn't produce any exceptions (though it may not have
359 373 # given a complete code object)
360 374 self._is_complete = self.code is not None
361 375
362 376 return self._is_complete
363 377
364 378 def push_accepts_more(self):
365 379 """Return whether a block of interactive input can accept more input.
366 380
367 381 This method is meant to be used by line-oriented frontends, who need to
368 382 guess whether a block is complete or not based solely on prior and
369 383 current input lines. The InputSplitter considers it has a complete
370 384 interactive block and will not accept more input only when either a
371 385 SyntaxError is raised, or *all* of the following are true:
372 386
373 387 1. The input compiles to a complete statement.
374 388
375 389 2. The indentation level is flush-left (because if we are indented,
376 390 like inside a function definition or for loop, we need to keep
377 391 reading new input).
378 392
379 393 3. There is one extra line consisting only of whitespace.
380 394
381 395 Because of condition #3, this method should be used only by
382 396 *line-oriented* frontends, since it means that intermediate blank lines
383 397 are not allowed in function definitions (or any other indented block).
384 398
385 399 If the current input produces a syntax error, this method immediately
386 400 returns False but does *not* raise the syntax error exception, as
387 401 typically clients will want to send invalid syntax to an execution
388 402 backend which might convert the invalid syntax into valid Python via
389 403 one of the dynamic IPython mechanisms.
390 404 """
391 405
392 406 # With incomplete input, unconditionally accept more
393 407 if not self._is_complete:
394 408 return True
395 409
396 410 # If we already have complete input and we're flush left, the answer
397 411 # depends. In line mode, if there hasn't been any indentation,
398 412 # that's it. If we've come back from some indentation, we need
399 413 # the blank final line to finish.
400 414 # In cell mode, we need to check how many blocks the input so far
401 415 # compiles into, because if there's already more than one full
402 416 # independent block of input, then the client has entered full
403 417 # 'cell' mode and is feeding lines that each is complete. In this
404 418 # case we should then keep accepting. The Qt terminal-like console
405 419 # does precisely this, to provide the convenience of terminal-like
406 420 # input of single expressions, but allowing the user (with a
407 421 # separate keystroke) to switch to 'cell' mode and type multiple
408 422 # expressions in one shot.
409 423 if self.indent_spaces==0:
410 424 if self.input_mode=='line':
411 425 if not self._full_dedent:
412 426 return False
413 427 else:
414 428 try:
415 429 code_ast = ast.parse(u''.join(self._buffer))
416 430 except Exception:
417 431 return False
418 432 else:
419 433 if len(code_ast.body) == 1:
420 434 return False
421 435
422 436 # When input is complete, then termination is marked by an extra blank
423 437 # line at the end.
424 438 last_line = self.source.splitlines()[-1]
425 439 return bool(last_line and not last_line.isspace())
426 440
427 441 #------------------------------------------------------------------------
428 442 # Private interface
429 443 #------------------------------------------------------------------------
430 444
431 445 def _find_indent(self, line):
432 446 """Compute the new indentation level for a single line.
433 447
434 448 Parameters
435 449 ----------
436 450 line : str
437 451 A single new line of non-whitespace, non-comment Python input.
438 452
439 453 Returns
440 454 -------
441 455 indent_spaces : int
442 456 New value for the indent level (it may be equal to self.indent_spaces
443 457 if indentation doesn't change.
444 458
445 459 full_dedent : boolean
446 460 Whether the new line causes a full flush-left dedent.
447 461 """
448 462 indent_spaces = self.indent_spaces
449 463 full_dedent = self._full_dedent
450 464
451 465 inisp = num_ini_spaces(line)
452 466 if inisp < indent_spaces:
453 467 indent_spaces = inisp
454 468 if indent_spaces <= 0:
455 469 #print 'Full dedent in text',self.source # dbg
456 470 full_dedent = True
457 471
458 472 if line.rstrip()[-1] == ':':
459 473 indent_spaces += 4
460 474 elif dedent_re.match(line):
461 475 indent_spaces -= 4
462 476 if indent_spaces <= 0:
463 477 full_dedent = True
464 478
465 479 # Safety
466 480 if indent_spaces < 0:
467 481 indent_spaces = 0
468 482 #print 'safety' # dbg
469 483
470 484 return indent_spaces, full_dedent
471 485
472 486 def _update_indent(self, lines):
473 487 for line in remove_comments(lines).splitlines():
474 488 if line and not line.isspace():
475 489 self.indent_spaces, self._full_dedent = self._find_indent(line)
476 490
477 491 def _store(self, lines, buffer=None, store='source'):
478 492 """Store one or more lines of input.
479 493
480 494 If input lines are not newline-terminated, a newline is automatically
481 495 appended."""
482 496
483 497 if buffer is None:
484 498 buffer = self._buffer
485 499
486 500 if lines.endswith('\n'):
487 501 buffer.append(lines)
488 502 else:
489 503 buffer.append(lines+'\n')
490 504 setattr(self, store, self._set_source(buffer))
491 505
492 506 def _set_source(self, buffer):
493 507 return u''.join(buffer)
494 508
495 509
496 510 #-----------------------------------------------------------------------------
497 511 # Functions and classes for IPython-specific syntactic support
498 512 #-----------------------------------------------------------------------------
499 513
500 514 # The escaped translators ALL receive a line where their own escape has been
501 515 # stripped. Only '?' is valid at the end of the line, all others can only be
502 516 # placed at the start.
503 517
504 518 # Transformations of the special syntaxes that don't rely on an explicit escape
505 519 # character but instead on patterns on the input line
506 520
507 521 # The core transformations are implemented as standalone functions that can be
508 522 # tested and validated in isolation. Each of these uses a regexp, we
509 523 # pre-compile these and keep them close to each function definition for clarity
510 524
511 525 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
512 526 r'\s*=\s*!\s*(?P<cmd>.*)')
513 527
514 528 def transform_assign_system(line):
515 529 """Handle the `files = !ls` syntax."""
516 530 m = _assign_system_re.match(line)
517 531 if m is not None:
518 532 cmd = m.group('cmd')
519 533 lhs = m.group('lhs')
520 534 new_line = '%s = get_ipython().getoutput(%r)' % (lhs, cmd)
521 535 return new_line
522 536 return line
523 537
524 538
525 539 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
526 540 r'\s*=\s*%\s*(?P<cmd>.*)')
527 541
528 542 def transform_assign_magic(line):
529 543 """Handle the `a = %who` syntax."""
530 544 m = _assign_magic_re.match(line)
531 545 if m is not None:
532 546 cmd = m.group('cmd')
533 547 lhs = m.group('lhs')
534 548 new_line = '%s = get_ipython().magic(%r)' % (lhs, cmd)
535 549 return new_line
536 550 return line
537 551
538 552
539 553 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
540 554
541 555 def transform_classic_prompt(line):
542 556 """Handle inputs that start with '>>> ' syntax."""
543 557
544 558 if not line or line.isspace():
545 559 return line
546 560 m = _classic_prompt_re.match(line)
547 561 if m:
548 562 return line[len(m.group(0)):]
549 563 else:
550 564 return line
551 565
552 566
553 567 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
554 568
555 569 def transform_ipy_prompt(line):
556 570 """Handle inputs that start classic IPython prompt syntax."""
557 571
558 572 if not line or line.isspace():
559 573 return line
560 574 #print 'LINE: %r' % line # dbg
561 575 m = _ipy_prompt_re.match(line)
562 576 if m:
563 577 #print 'MATCH! %r -> %r' % (line, line[len(m.group(0)):]) # dbg
564 578 return line[len(m.group(0)):]
565 579 else:
566 580 return line
567 581
568 582
569 583 def _make_help_call(target, esc, lspace, next_input=None):
570 584 """Prepares a pinfo(2)/psearch call from a target name and the escape
571 585 (i.e. ? or ??)"""
572 586 method = 'pinfo2' if esc == '??' \
573 587 else 'psearch' if '*' in target \
574 588 else 'pinfo'
575 589 arg = " ".join([method, target])
576 590
577 591 if next_input is None:
578 592 return '%sget_ipython().magic(%r)' % (lspace, arg)
579 593 else:
580 594 return '%sget_ipython().set_next_input(%r);get_ipython().magic(%r)' % \
581 595 (lspace, next_input, arg)
582 596
583 597
584 598 _initial_space_re = re.compile(r'\s*')
585 599
586 600 _help_end_re = re.compile(r"""(%?
587 601 [a-zA-Z_*][\w*]* # Variable name
588 602 (\.[a-zA-Z_*][\w*]*)* # .etc.etc
589 603 )
590 604 (\?\??)$ # ? or ??""",
591 605 re.VERBOSE)
592 606
593 607
594 608 def transform_help_end(line):
595 609 """Translate lines with ?/?? at the end"""
596 610 m = _help_end_re.search(line)
597 611 if m is None or has_comment(line):
598 612 return line
599 613 target = m.group(1)
600 614 esc = m.group(3)
601 615 lspace = _initial_space_re.match(line).group(0)
602 616
603 617 # If we're mid-command, put it back on the next prompt for the user.
604 618 next_input = line.rstrip('?') if line.strip() != m.group(0) else None
605 619
606 620 return _make_help_call(target, esc, lspace, next_input)
607 621
608 622
609 623 class EscapedTransformer(object):
610 624 """Class to transform lines that are explicitly escaped out."""
611 625
612 626 def __init__(self):
613 627 tr = { ESC_SHELL : self._tr_system,
614 628 ESC_SH_CAP : self._tr_system2,
615 629 ESC_HELP : self._tr_help,
616 630 ESC_HELP2 : self._tr_help,
617 631 ESC_MAGIC : self._tr_magic,
618 632 ESC_QUOTE : self._tr_quote,
619 633 ESC_QUOTE2 : self._tr_quote2,
620 634 ESC_PAREN : self._tr_paren }
621 635 self.tr = tr
622 636
623 637 # Support for syntax transformations that use explicit escapes typed by the
624 638 # user at the beginning of a line
625 639 @staticmethod
626 640 def _tr_system(line_info):
627 641 "Translate lines escaped with: !"
628 642 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
629 643 return '%sget_ipython().system(%r)' % (line_info.pre, cmd)
630 644
631 645 @staticmethod
632 646 def _tr_system2(line_info):
633 647 "Translate lines escaped with: !!"
634 648 cmd = line_info.line.lstrip()[2:]
635 649 return '%sget_ipython().getoutput(%r)' % (line_info.pre, cmd)
636 650
637 651 @staticmethod
638 652 def _tr_help(line_info):
639 653 "Translate lines escaped with: ?/??"
640 654 # A naked help line should just fire the intro help screen
641 655 if not line_info.line[1:]:
642 656 return 'get_ipython().show_usage()'
643 657
644 658 return _make_help_call(line_info.ifun, line_info.esc, line_info.pre)
645 659
646 660 @staticmethod
647 661 def _tr_magic(line_info):
648 662 "Translate lines escaped with: %"
649 663 tpl = '%sget_ipython().magic(%r)'
650 664 cmd = ' '.join([line_info.ifun, line_info.the_rest]).strip()
651 665 return tpl % (line_info.pre, cmd)
652 666
653 667 @staticmethod
654 668 def _tr_quote(line_info):
655 669 "Translate lines escaped with: ,"
656 670 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
657 671 '", "'.join(line_info.the_rest.split()) )
658 672
659 673 @staticmethod
660 674 def _tr_quote2(line_info):
661 675 "Translate lines escaped with: ;"
662 676 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
663 677 line_info.the_rest)
664 678
665 679 @staticmethod
666 680 def _tr_paren(line_info):
667 681 "Translate lines escaped with: /"
668 682 return '%s%s(%s)' % (line_info.pre, line_info.ifun,
669 683 ", ".join(line_info.the_rest.split()))
670 684
671 685 def __call__(self, line):
672 686 """Class to transform lines that are explicitly escaped out.
673 687
674 688 This calls the above _tr_* static methods for the actual line
675 689 translations."""
676 690
677 691 # Empty lines just get returned unmodified
678 692 if not line or line.isspace():
679 693 return line
680 694
681 695 # Get line endpoints, where the escapes can be
682 696 line_info = LineInfo(line)
683 697
684 698 if not line_info.esc in self.tr:
685 699 # If we don't recognize the escape, don't modify the line
686 700 return line
687 701
688 702 return self.tr[line_info.esc](line_info)
689 703
690 704
691 705 # A function-looking object to be used by the rest of the code. The purpose of
692 706 # the class in this case is to organize related functionality, more than to
693 707 # manage state.
694 708 transform_escaped = EscapedTransformer()
695 709
696 710
697 711 class IPythonInputSplitter(InputSplitter):
698 712 """An input splitter that recognizes all of IPython's special syntax."""
699 713
700 714 # String with raw, untransformed input.
701 715 source_raw = ''
702 716
703 717 cell_magic_parts = []
704 718
705 719 cell_magic_mode = False
706 720
707 721 # Private attributes
708 722
709 723 # List with lines of raw input accumulated so far.
710 724 _buffer_raw = None
711 725
712 726 def __init__(self, input_mode=None):
713 727 super(IPythonInputSplitter, self).__init__(input_mode)
714 728 self._buffer_raw = []
715 729 self._validate = True
716 730
717 731 def reset(self):
718 732 """Reset the input buffer and associated state."""
719 733 super(IPythonInputSplitter, self).reset()
720 734 self._buffer_raw[:] = []
721 735 self.source_raw = ''
722 736 self.cell_magic_parts = []
723 737 self.cell_magic_mode = False
724 738
725 739 def source_raw_reset(self):
726 740 """Return input and raw source and perform a full reset.
727 741 """
728 742 out = self.source
729 743 out_r = self.source_raw
730 744 self.reset()
731 745 return out, out_r
732 746
733 747 def push_accepts_more(self):
734 748 if self.cell_magic_mode:
735 749 return not self._is_complete
736 750 else:
737 751 return super(IPythonInputSplitter, self).push_accepts_more()
738 752
739 753 def _push_line_mode(self, lines):
740 754 """Push in line mode.
741 755
742 756 This means that we only get individual 'lines' with each call, though
743 757 in practice each input may be multiline. But this is in contrast to
744 758 cell mode, which feeds the entirety of the cell from the start with
745 759 each call.
746 760 """
747 761 # cell magic support
748 762 #print('#'*10)
749 763 #print(lines+'\n---') # dbg
750 764 #print (repr(lines)+'\n+++')
751 765 #print('raw', self._buffer_raw, 'validate', self.cell_magic_mode)
752 766 # Only trigger this block if we're at a 'fresh' pumping start.
753 767 if lines.startswith('%%') and (not self.cell_magic_mode) and \
754 768 not self._buffer_raw:
755 769 # Cell magics bypass all further transformations
756 770 self.cell_magic_mode = True
757 771 first, _, body = lines.partition('\n')
758 772 magic_name, _, line = first.partition(' ')
759 773 magic_name = magic_name.lstrip(ESC_MAGIC)
760 774 # We store the body of the cell and create a call to a method that
761 775 # will use this stored value. This is ugly, but it's a first cut to
762 776 # get it all working, as right now changing the return API of our
763 777 # methods would require major refactoring.
764 778 self.cell_magic_parts = [body]
765 779 tpl = 'get_ipython()._cell_magic(%r, %r)'
766 780 tlines = tpl % (magic_name, line)
767 781 self._store(tlines)
768 782 self._store(lines, self._buffer_raw, 'source_raw')
769 783 self._is_complete = False
770 784 return False
771 785
772 786 if self.cell_magic_mode:
773 787 # Find out if the last stored block has a whitespace line as its
774 788 # last line and also this line is whitespace, case in which we're
775 789 # done (two contiguous blank lines signal termination). Note that
776 790 # the storage logic *enforces* that every stored block is
777 791 # newline-terminated, so we grab everything but the last character
778 792 # so we can have the body of the block alone.
779 793 last_block = self.cell_magic_parts[-1]
780 794 self._is_complete = last_blank(last_block) and lines.isspace()
781 795 # Only store the raw input. For lines beyond the first one, we
782 796 # only store them for history purposes, and for execution we want
783 797 # the caller to only receive the _cell_magic() call.
784 798 self._store(lines, self._buffer_raw, 'source_raw')
785 799 self.cell_magic_parts.append(lines)
786 800 return self._is_complete
787 801
788 802 lines_list = lines.splitlines()
789 803
790 804 transforms = [transform_ipy_prompt, transform_classic_prompt,
791 805 transform_help_end, transform_escaped,
792 806 transform_assign_system, transform_assign_magic]
793 807
794 808 # Transform logic
795 809 #
796 810 # We only apply the line transformers to the input if we have either no
797 811 # input yet, or complete input, or if the last line of the buffer ends
798 812 # with ':' (opening an indented block). This prevents the accidental
799 813 # transformation of escapes inside multiline expressions like
800 814 # triple-quoted strings or parenthesized expressions.
801 815 #
802 816 # The last heuristic, while ugly, ensures that the first line of an
803 817 # indented block is correctly transformed.
804 818 #
805 819 # FIXME: try to find a cleaner approach for this last bit.
806 820
807 821 # Store raw source before applying any transformations to it. Note
808 822 # that this must be done *after* the reset() call that would otherwise
809 823 # flush the buffer.
810 824 self._store(lines, self._buffer_raw, 'source_raw')
811 825
812 826 push = super(IPythonInputSplitter, self).push
813 827 buf = self._buffer
814 828 for line in lines_list:
815 829 if self._is_complete or not buf or \
816 830 (buf and buf[-1].rstrip().endswith((':', ','))):
817 831 for f in transforms:
818 832 line = f(line)
819 833
820 834 out = push(line)
821 835 return out
822 836
837
838 def _push_cell_mode(self, lines):
839 """Push in cell mode.
840
841 This means that we get the entire cell with each call. Between resets,
842 the calls simply add more text to the input."""
843
844 if lines.startswith('%%'):
845 # Cell magics bypass all further transformations
846 self.cell_magic_mode = True
847 first, _, body = lines.partition('\n')
848 magic_name, _, line = first.partition(' ')
849 magic_name = magic_name.lstrip(ESC_MAGIC)
850 # We store the body of the cell and create a call to a method that
851 # will use this stored value. This is ugly, but it's a first cut to
852 # get it all working, as right now changing the return API of our
853 # methods would require major refactoring.
854 self.cell_magic_parts = [body]
855 tpl = 'get_ipython()._cell_magic(%r, %r)'
856 tlines = tpl % (magic_name, line)
857 self._store(tlines)
858 self._store(lines, self._buffer_raw, 'source_raw')
859 self._is_complete = last_two_blanks(lines)
860 return self._is_complete
861
862 lines_list = lines.splitlines()
863
864 transforms = [transform_ipy_prompt, transform_classic_prompt,
865 transform_help_end, transform_escaped,
866 transform_assign_system, transform_assign_magic]
867
868 # Transform logic
869 #
870 # We only apply the line transformers to the input if we have either no
871 # input yet, or complete input, or if the last line of the buffer ends
872 # with ':' (opening an indented block). This prevents the accidental
873 # transformation of escapes inside multiline expressions like
874 # triple-quoted strings or parenthesized expressions.
875 #
876 # The last heuristic, while ugly, ensures that the first line of an
877 # indented block is correctly transformed.
878 #
879 # FIXME: try to find a cleaner approach for this last bit.
880
881 # In cell mode, since we're going to pump the parent class by hand line
882 # by line, we need to temporarily switch out to 'line' mode, do a
883 # single manual reset and then feed the lines one by one. Note that
884 # this only matters if the input has more than one line.
885 self.reset()
886 self.input_mode = 'line'
887
888 # Store raw source before applying any transformations to it. Note
889 # that this must be done *after* the reset() call that would otherwise
890 # flush the buffer.
891 self._store(lines, self._buffer_raw, 'source_raw')
892
893 try:
894 push = super(IPythonInputSplitter, self).push
895 buf = self._buffer
896 for line in lines_list:
897 if self._is_complete or not buf or \
898 (buf and buf[-1].rstrip().endswith((':', ','))):
899 for f in transforms:
900 line = f(line)
901
902 out = push(line)
903 finally:
904 self.input_mode = 'cell'
905 return out
906
823 907 def push(self, lines):
824 908 """Push one or more lines of IPython input.
825 909
826 910 This stores the given lines and returns a status code indicating
827 911 whether the code forms a complete Python block or not, after processing
828 912 all input lines for special IPython syntax.
829 913
830 914 Any exceptions generated in compilation are swallowed, but if an
831 915 exception was produced, the method returns True.
832 916
833 917 Parameters
834 918 ----------
835 919 lines : string
836 920 One or more lines of Python input.
837 921
838 922 Returns
839 923 -------
840 924 is_complete : boolean
841 925 True if the current input source (the result of the current input
842 926 plus prior inputs) forms a complete Python execution block. Note that
843 927 this value is also stored as a private attribute (_is_complete), so it
844 928 can be queried at any time.
845 929 """
846 print('mode:', self.input_mode)
847 print('lines:',repr(lines))
848 930 if not lines:
849 931 return super(IPythonInputSplitter, self).push(lines)
850 932
851 933 # We must ensure all input is pure unicode
852 934 lines = cast_unicode(lines, self.encoding)
853 935
854 936 if self.input_mode == 'line':
855 937 return self._push_line_mode(lines)
856
857 ## else:
858 ## return self._push_cell_mode(lines)
938 else:
939 return self._push_cell_mode(lines)
859 940
860 941 lines_list = lines.splitlines()
861 942
862 943 transforms = [transform_ipy_prompt, transform_classic_prompt,
863 944 transform_help_end, transform_escaped,
864 945 transform_assign_system, transform_assign_magic]
865 946
866 947 # Transform logic
867 948 #
868 949 # We only apply the line transformers to the input if we have either no
869 950 # input yet, or complete input, or if the last line of the buffer ends
870 951 # with ':' (opening an indented block). This prevents the accidental
871 952 # transformation of escapes inside multiline expressions like
872 953 # triple-quoted strings or parenthesized expressions.
873 954 #
874 955 # The last heuristic, while ugly, ensures that the first line of an
875 956 # indented block is correctly transformed.
876 957 #
877 958 # FIXME: try to find a cleaner approach for this last bit.
878 959
879 960 # If we were in 'block' mode, since we're going to pump the parent
880 961 # class by hand line by line, we need to temporarily switch out to
881 962 # 'line' mode, do a single manual reset and then feed the lines one
882 963 # by one. Note that this only matters if the input has more than one
883 964 # line.
884 965 changed_input_mode = False
885 966
886 967 if self.input_mode == 'cell':
887 968 self.reset()
888 969 changed_input_mode = True
889 970 saved_input_mode = 'cell'
890 971 self.input_mode = 'line'
891 972
892 973 # Store raw source before applying any transformations to it. Note
893 974 # that this must be done *after* the reset() call that would otherwise
894 975 # flush the buffer.
895 976 self._store(lines, self._buffer_raw, 'source_raw')
896 977
897 978 try:
898 979 push = super(IPythonInputSplitter, self).push
899 980 buf = self._buffer
900 981 for line in lines_list:
901 982 if self._is_complete or not buf or \
902 983 (buf and buf[-1].rstrip().endswith((':', ','))):
903 984 for f in transforms:
904 985 line = f(line)
905 986
906 987 out = push(line)
907 988 finally:
908 989 if changed_input_mode:
909 990 self.input_mode = saved_input_mode
910 991 return out
@@ -1,745 +1,760 b''
1 1 # -*- coding: utf-8 -*-
2 2 """Tests for the inputsplitter module.
3 3
4 4 Authors
5 5 -------
6 6 * Fernando Perez
7 7 * Robert Kern
8 8 """
9 9 #-----------------------------------------------------------------------------
10 10 # Copyright (C) 2010-2011 The IPython Development Team
11 11 #
12 12 # Distributed under the terms of the BSD License. The full license is in
13 13 # the file COPYING, distributed as part of this software.
14 14 #-----------------------------------------------------------------------------
15 15
16 16 #-----------------------------------------------------------------------------
17 17 # Imports
18 18 #-----------------------------------------------------------------------------
19 19 # stdlib
20 20 import unittest
21 21 import sys
22 22
23 23 # Third party
24 24 import nose.tools as nt
25 25
26 26 # Our own
27 27 from IPython.core import inputsplitter as isp
28 28 from IPython.testing import tools as tt
29 29 from IPython.utils import py3compat
30 30
31 31 #-----------------------------------------------------------------------------
32 32 # Semi-complete examples (also used as tests)
33 33 #-----------------------------------------------------------------------------
34 34
35 35 # Note: at the bottom, there's a slightly more complete version of this that
36 36 # can be useful during development of code here.
37 37
38 38 def mini_interactive_loop(input_func):
39 39 """Minimal example of the logic of an interactive interpreter loop.
40 40
41 41 This serves as an example, and it is used by the test system with a fake
42 42 raw_input that simulates interactive input."""
43 43
44 44 from IPython.core.inputsplitter import InputSplitter
45 45
46 46 isp = InputSplitter()
47 47 # In practice, this input loop would be wrapped in an outside loop to read
48 48 # input indefinitely, until some exit/quit command was issued. Here we
49 49 # only illustrate the basic inner loop.
50 50 while isp.push_accepts_more():
51 51 indent = ' '*isp.indent_spaces
52 52 prompt = '>>> ' + indent
53 53 line = indent + input_func(prompt)
54 54 isp.push(line)
55 55
56 56 # Here we just return input so we can use it in a test suite, but a real
57 57 # interpreter would instead send it for execution somewhere.
58 58 src = isp.source_reset()
59 59 #print 'Input source was:\n', src # dbg
60 60 return src
61 61
62 62 #-----------------------------------------------------------------------------
63 63 # Test utilities, just for local use
64 64 #-----------------------------------------------------------------------------
65 65
66 66 def assemble(block):
67 67 """Assemble a block into multi-line sub-blocks."""
68 68 return ['\n'.join(sub_block)+'\n' for sub_block in block]
69 69
70 70
71 71 def pseudo_input(lines):
72 72 """Return a function that acts like raw_input but feeds the input list."""
73 73 ilines = iter(lines)
74 74 def raw_in(prompt):
75 75 try:
76 76 return next(ilines)
77 77 except StopIteration:
78 78 return ''
79 79 return raw_in
80 80
81 81 #-----------------------------------------------------------------------------
82 82 # Tests
83 83 #-----------------------------------------------------------------------------
84 84 def test_spaces():
85 85 tests = [('', 0),
86 86 (' ', 1),
87 87 ('\n', 0),
88 88 (' \n', 1),
89 89 ('x', 0),
90 90 (' x', 1),
91 91 (' x',2),
92 92 (' x',4),
93 93 # Note: tabs are counted as a single whitespace!
94 94 ('\tx', 1),
95 95 ('\t x', 2),
96 96 ]
97 97 tt.check_pairs(isp.num_ini_spaces, tests)
98 98
99 99
100 100 def test_remove_comments():
101 101 tests = [('text', 'text'),
102 102 ('text # comment', 'text '),
103 103 ('text # comment\n', 'text \n'),
104 104 ('text # comment \n', 'text \n'),
105 105 ('line # c \nline\n','line \nline\n'),
106 106 ('line # c \nline#c2 \nline\nline #c\n\n',
107 107 'line \nline\nline\nline \n\n'),
108 108 ]
109 109 tt.check_pairs(isp.remove_comments, tests)
110 110
111 111 def test_has_comment():
112 112 tests = [('text', False),
113 113 ('text #comment', True),
114 114 ('text #comment\n', True),
115 115 ('#comment', True),
116 116 ('#comment\n', True),
117 117 ('a = "#string"', False),
118 118 ('a = "#string" # comment', True),
119 119 ('a #comment not "string"', True),
120 120 ]
121 121 tt.check_pairs(isp.has_comment, tests)
122 122
123 123
124 124 def test_get_input_encoding():
125 125 encoding = isp.get_input_encoding()
126 126 nt.assert_true(isinstance(encoding, basestring))
127 127 # simple-minded check that at least encoding a simple string works with the
128 128 # encoding we got.
129 129 nt.assert_equal(u'test'.encode(encoding), b'test')
130 130
131 131
132 132 class NoInputEncodingTestCase(unittest.TestCase):
133 133 def setUp(self):
134 134 self.old_stdin = sys.stdin
135 135 class X: pass
136 136 fake_stdin = X()
137 137 sys.stdin = fake_stdin
138 138
139 139 def test(self):
140 140 # Verify that if sys.stdin has no 'encoding' attribute we do the right
141 141 # thing
142 142 enc = isp.get_input_encoding()
143 143 self.assertEqual(enc, 'ascii')
144 144
145 145 def tearDown(self):
146 146 sys.stdin = self.old_stdin
147 147
148 148
149 149 class InputSplitterTestCase(unittest.TestCase):
150 150 def setUp(self):
151 151 self.isp = isp.InputSplitter()
152 152
153 153 def test_reset(self):
154 154 isp = self.isp
155 155 isp.push('x=1')
156 156 isp.reset()
157 157 self.assertEqual(isp._buffer, [])
158 158 self.assertEqual(isp.indent_spaces, 0)
159 159 self.assertEqual(isp.source, '')
160 160 self.assertEqual(isp.code, None)
161 161 self.assertEqual(isp._is_complete, False)
162 162
163 163 def test_source(self):
164 164 self.isp._store('1')
165 165 self.isp._store('2')
166 166 self.assertEqual(self.isp.source, '1\n2\n')
167 167 self.assertTrue(len(self.isp._buffer)>0)
168 168 self.assertEqual(self.isp.source_reset(), '1\n2\n')
169 169 self.assertEqual(self.isp._buffer, [])
170 170 self.assertEqual(self.isp.source, '')
171 171
172 172 def test_indent(self):
173 173 isp = self.isp # shorthand
174 174 isp.push('x=1')
175 175 self.assertEqual(isp.indent_spaces, 0)
176 176 isp.push('if 1:\n x=1')
177 177 self.assertEqual(isp.indent_spaces, 4)
178 178 isp.push('y=2\n')
179 179 self.assertEqual(isp.indent_spaces, 0)
180 180
181 181 def test_indent2(self):
182 182 # In cell mode, inputs must be fed in whole blocks, so skip this test
183 183 if self.isp.input_mode == 'cell': return
184 184
185 185 isp = self.isp
186 186 isp.push('if 1:')
187 187 self.assertEqual(isp.indent_spaces, 4)
188 188 isp.push(' x=1')
189 189 self.assertEqual(isp.indent_spaces, 4)
190 190 # Blank lines shouldn't change the indent level
191 191 isp.push(' '*2)
192 192 self.assertEqual(isp.indent_spaces, 4)
193 193
194 194 def test_indent3(self):
195 195 # In cell mode, inputs must be fed in whole blocks, so skip this test
196 196 if self.isp.input_mode == 'cell': return
197 197
198 198 isp = self.isp
199 199 # When a multiline statement contains parens or multiline strings, we
200 200 # shouldn't get confused.
201 201 isp.push("if 1:")
202 202 isp.push(" x = (1+\n 2)")
203 203 self.assertEqual(isp.indent_spaces, 4)
204 204
205 205 def test_indent4(self):
206 206 # In cell mode, inputs must be fed in whole blocks, so skip this test
207 207 if self.isp.input_mode == 'cell': return
208 208
209 209 isp = self.isp
210 210 # whitespace after ':' should not screw up indent level
211 211 isp.push('if 1: \n x=1')
212 212 self.assertEqual(isp.indent_spaces, 4)
213 213 isp.push('y=2\n')
214 214 self.assertEqual(isp.indent_spaces, 0)
215 215 isp.push('if 1:\t\n x=1')
216 216 self.assertEqual(isp.indent_spaces, 4)
217 217 isp.push('y=2\n')
218 218 self.assertEqual(isp.indent_spaces, 0)
219 219
220 220 def test_dedent_pass(self):
221 221 isp = self.isp # shorthand
222 222 # should NOT cause dedent
223 223 isp.push('if 1:\n passes = 5')
224 224 self.assertEqual(isp.indent_spaces, 4)
225 225 isp.push('if 1:\n pass')
226 226 self.assertEqual(isp.indent_spaces, 0)
227 227 isp.push('if 1:\n pass ')
228 228 self.assertEqual(isp.indent_spaces, 0)
229 229
230 230 def test_dedent_raise(self):
231 231 isp = self.isp # shorthand
232 232 # should NOT cause dedent
233 233 isp.push('if 1:\n raised = 4')
234 234 self.assertEqual(isp.indent_spaces, 4)
235 235 isp.push('if 1:\n raise TypeError()')
236 236 self.assertEqual(isp.indent_spaces, 0)
237 237 isp.push('if 1:\n raise')
238 238 self.assertEqual(isp.indent_spaces, 0)
239 239 isp.push('if 1:\n raise ')
240 240 self.assertEqual(isp.indent_spaces, 0)
241 241
242 242 def test_dedent_return(self):
243 243 isp = self.isp # shorthand
244 244 # should NOT cause dedent
245 245 isp.push('if 1:\n returning = 4')
246 246 self.assertEqual(isp.indent_spaces, 4)
247 247 isp.push('if 1:\n return 5 + 493')
248 248 self.assertEqual(isp.indent_spaces, 0)
249 249 isp.push('if 1:\n return')
250 250 self.assertEqual(isp.indent_spaces, 0)
251 251 isp.push('if 1:\n return ')
252 252 self.assertEqual(isp.indent_spaces, 0)
253 253 isp.push('if 1:\n return(0)')
254 254 self.assertEqual(isp.indent_spaces, 0)
255 255
256 256 def test_push(self):
257 257 isp = self.isp
258 258 self.assertTrue(isp.push('x=1'))
259 259
260 260 def test_push2(self):
261 261 isp = self.isp
262 262 self.assertFalse(isp.push('if 1:'))
263 263 for line in [' x=1', '# a comment', ' y=2']:
264 264 self.assertTrue(isp.push(line))
265 265
266 266 def test_push3(self):
267 267 isp = self.isp
268 268 isp.push('if True:')
269 269 isp.push(' a = 1')
270 270 self.assertFalse(isp.push('b = [1,'))
271 271
272 272 def test_replace_mode(self):
273 273 isp = self.isp
274 274 isp.input_mode = 'cell'
275 275 isp.push('x=1')
276 276 self.assertEqual(isp.source, 'x=1\n')
277 277 isp.push('x=2')
278 278 self.assertEqual(isp.source, 'x=2\n')
279 279
280 280 def test_push_accepts_more(self):
281 281 isp = self.isp
282 282 isp.push('x=1')
283 283 self.assertFalse(isp.push_accepts_more())
284 284
285 285 def test_push_accepts_more2(self):
286 286 # In cell mode, inputs must be fed in whole blocks, so skip this test
287 287 if self.isp.input_mode == 'cell': return
288 288
289 289 isp = self.isp
290 290 isp.push('if 1:')
291 291 self.assertTrue(isp.push_accepts_more())
292 292 isp.push(' x=1')
293 293 self.assertTrue(isp.push_accepts_more())
294 294 isp.push('')
295 295 self.assertFalse(isp.push_accepts_more())
296 296
297 297 def test_push_accepts_more3(self):
298 298 isp = self.isp
299 299 isp.push("x = (2+\n3)")
300 300 self.assertFalse(isp.push_accepts_more())
301 301
302 302 def test_push_accepts_more4(self):
303 303 # In cell mode, inputs must be fed in whole blocks, so skip this test
304 304 if self.isp.input_mode == 'cell': return
305 305
306 306 isp = self.isp
307 307 # When a multiline statement contains parens or multiline strings, we
308 308 # shouldn't get confused.
309 309 # FIXME: we should be able to better handle de-dents in statements like
310 310 # multiline strings and multiline expressions (continued with \ or
311 311 # parens). Right now we aren't handling the indentation tracking quite
312 312 # correctly with this, though in practice it may not be too much of a
313 313 # problem. We'll need to see.
314 314 isp.push("if 1:")
315 315 isp.push(" x = (2+")
316 316 isp.push(" 3)")
317 317 self.assertTrue(isp.push_accepts_more())
318 318 isp.push(" y = 3")
319 319 self.assertTrue(isp.push_accepts_more())
320 320 isp.push('')
321 321 self.assertFalse(isp.push_accepts_more())
322 322
323 323 def test_push_accepts_more5(self):
324 324 # In cell mode, inputs must be fed in whole blocks, so skip this test
325 325 if self.isp.input_mode == 'cell': return
326 326
327 327 isp = self.isp
328 328 isp.push('try:')
329 329 isp.push(' a = 5')
330 330 isp.push('except:')
331 331 isp.push(' raise')
332 332 self.assertTrue(isp.push_accepts_more())
333 333
334 334 def test_continuation(self):
335 335 isp = self.isp
336 336 isp.push("import os, \\")
337 337 self.assertTrue(isp.push_accepts_more())
338 338 isp.push("sys")
339 339 self.assertFalse(isp.push_accepts_more())
340 340
341 341 def test_syntax_error(self):
342 342 isp = self.isp
343 343 # Syntax errors immediately produce a 'ready' block, so the invalid
344 344 # Python can be sent to the kernel for evaluation with possible ipython
345 345 # special-syntax conversion.
346 346 isp.push('run foo')
347 347 self.assertFalse(isp.push_accepts_more())
348 348
349 349 def test_unicode(self):
350 350 self.isp.push(u"PΓ©rez")
351 351 self.isp.push(u'\xc3\xa9')
352 352 self.isp.push(u"u'\xc3\xa9'")
353 353
354 354 class InteractiveLoopTestCase(unittest.TestCase):
355 355 """Tests for an interactive loop like a python shell.
356 356 """
357 357 def check_ns(self, lines, ns):
358 358 """Validate that the given input lines produce the resulting namespace.
359 359
360 360 Note: the input lines are given exactly as they would be typed in an
361 361 auto-indenting environment, as mini_interactive_loop above already does
362 362 auto-indenting and prepends spaces to the input.
363 363 """
364 364 src = mini_interactive_loop(pseudo_input(lines))
365 365 test_ns = {}
366 366 exec src in test_ns
367 367 # We can't check that the provided ns is identical to the test_ns,
368 368 # because Python fills test_ns with extra keys (copyright, etc). But
369 369 # we can check that the given dict is *contained* in test_ns
370 370 for k,v in ns.iteritems():
371 371 self.assertEqual(test_ns[k], v)
372 372
373 373 def test_simple(self):
374 374 self.check_ns(['x=1'], dict(x=1))
375 375
376 376 def test_simple2(self):
377 377 self.check_ns(['if 1:', 'x=2'], dict(x=2))
378 378
379 379 def test_xy(self):
380 380 self.check_ns(['x=1; y=2'], dict(x=1, y=2))
381 381
382 382 def test_abc(self):
383 383 self.check_ns(['if 1:','a=1','b=2','c=3'], dict(a=1, b=2, c=3))
384 384
385 385 def test_multi(self):
386 386 self.check_ns(['x =(1+','1+','2)'], dict(x=4))
387 387
388 388
389 389 def test_LineInfo():
390 390 """Simple test for LineInfo construction and str()"""
391 391 linfo = isp.LineInfo(' %cd /home')
392 392 nt.assert_equals(str(linfo), 'LineInfo [ |%|cd|/home]')
393 393
394 394 # Transformer tests
395 395 def transform_checker(tests, func):
396 396 """Utility to loop over test inputs"""
397 397 for inp, tr in tests:
398 398 nt.assert_equals(func(inp), tr)
399 399
400 400 # Data for all the syntax tests in the form of lists of pairs of
401 401 # raw/transformed input. We store it here as a global dict so that we can use
402 402 # it both within single-function tests and also to validate the behavior of the
403 403 # larger objects
404 404
405 405 syntax = \
406 406 dict(assign_system =
407 407 [(i,py3compat.u_format(o)) for i,o in \
408 408 [(u'a =! ls', "a = get_ipython().getoutput({u}'ls')"),
409 409 (u'b = !ls', "b = get_ipython().getoutput({u}'ls')"),
410 410 ('x=1', 'x=1'), # normal input is unmodified
411 411 (' ',' '), # blank lines are kept intact
412 412 ]],
413 413
414 414 assign_magic =
415 415 [(i,py3compat.u_format(o)) for i,o in \
416 416 [(u'a =% who', "a = get_ipython().magic({u}'who')"),
417 417 (u'b = %who', "b = get_ipython().magic({u}'who')"),
418 418 ('x=1', 'x=1'), # normal input is unmodified
419 419 (' ',' '), # blank lines are kept intact
420 420 ]],
421 421
422 422 classic_prompt =
423 423 [('>>> x=1', 'x=1'),
424 424 ('x=1', 'x=1'), # normal input is unmodified
425 425 (' ', ' '), # blank lines are kept intact
426 426 ('... ', ''), # continuation prompts
427 427 ],
428 428
429 429 ipy_prompt =
430 430 [('In [1]: x=1', 'x=1'),
431 431 ('x=1', 'x=1'), # normal input is unmodified
432 432 (' ',' '), # blank lines are kept intact
433 433 (' ....: ', ''), # continuation prompts
434 434 ],
435 435
436 436 # Tests for the escape transformer to leave normal code alone
437 437 escaped_noesc =
438 438 [ (' ', ' '),
439 439 ('x=1', 'x=1'),
440 440 ],
441 441
442 442 # System calls
443 443 escaped_shell =
444 444 [(i,py3compat.u_format(o)) for i,o in \
445 445 [ (u'!ls', "get_ipython().system({u}'ls')"),
446 446 # Double-escape shell, this means to capture the output of the
447 447 # subprocess and return it
448 448 (u'!!ls', "get_ipython().getoutput({u}'ls')"),
449 449 ]],
450 450
451 451 # Help/object info
452 452 escaped_help =
453 453 [(i,py3compat.u_format(o)) for i,o in \
454 454 [ (u'?', 'get_ipython().show_usage()'),
455 455 (u'?x1', "get_ipython().magic({u}'pinfo x1')"),
456 456 (u'??x2', "get_ipython().magic({u}'pinfo2 x2')"),
457 457 (u'?a.*s', "get_ipython().magic({u}'psearch a.*s')"),
458 458 (u'?%hist', "get_ipython().magic({u}'pinfo %hist')"),
459 459 (u'?abc = qwe', "get_ipython().magic({u}'pinfo abc')"),
460 460 ]],
461 461
462 462 end_help =
463 463 [(i,py3compat.u_format(o)) for i,o in \
464 464 [ (u'x3?', "get_ipython().magic({u}'pinfo x3')"),
465 465 (u'x4??', "get_ipython().magic({u}'pinfo2 x4')"),
466 466 (u'%hist?', "get_ipython().magic({u}'pinfo %hist')"),
467 467 (u'f*?', "get_ipython().magic({u}'psearch f*')"),
468 468 (u'ax.*aspe*?', "get_ipython().magic({u}'psearch ax.*aspe*')"),
469 469 (u'a = abc?', "get_ipython().set_next_input({u}'a = abc');"
470 470 "get_ipython().magic({u}'pinfo abc')"),
471 471 (u'a = abc.qe??', "get_ipython().set_next_input({u}'a = abc.qe');"
472 472 "get_ipython().magic({u}'pinfo2 abc.qe')"),
473 473 (u'a = *.items?', "get_ipython().set_next_input({u}'a = *.items');"
474 474 "get_ipython().magic({u}'psearch *.items')"),
475 475 (u'plot(a?', "get_ipython().set_next_input({u}'plot(a');"
476 476 "get_ipython().magic({u}'pinfo a')"),
477 477 (u'a*2 #comment?', 'a*2 #comment?'),
478 478 ]],
479 479
480 480 # Explicit magic calls
481 481 escaped_magic =
482 482 [(i,py3compat.u_format(o)) for i,o in \
483 483 [ (u'%cd', "get_ipython().magic({u}'cd')"),
484 484 (u'%cd /home', "get_ipython().magic({u}'cd /home')"),
485 485 # Backslashes need to be escaped.
486 486 (u'%cd C:\\User', "get_ipython().magic({u}'cd C:\\\\User')"),
487 487 (u' %magic', " get_ipython().magic({u}'magic')"),
488 488 ]],
489 489
490 490 # Quoting with separate arguments
491 491 escaped_quote =
492 492 [ (',f', 'f("")'),
493 493 (',f x', 'f("x")'),
494 494 (' ,f y', ' f("y")'),
495 495 (',f a b', 'f("a", "b")'),
496 496 ],
497 497
498 498 # Quoting with single argument
499 499 escaped_quote2 =
500 500 [ (';f', 'f("")'),
501 501 (';f x', 'f("x")'),
502 502 (' ;f y', ' f("y")'),
503 503 (';f a b', 'f("a b")'),
504 504 ],
505 505
506 506 # Simply apply parens
507 507 escaped_paren =
508 508 [ ('/f', 'f()'),
509 509 ('/f x', 'f(x)'),
510 510 (' /f y', ' f(y)'),
511 511 ('/f a b', 'f(a, b)'),
512 512 ],
513 513
514 514 # Check that we transform prompts before other transforms
515 515 mixed =
516 516 [(i,py3compat.u_format(o)) for i,o in \
517 517 [ (u'In [1]: %lsmagic', "get_ipython().magic({u}'lsmagic')"),
518 518 (u'>>> %lsmagic', "get_ipython().magic({u}'lsmagic')"),
519 519 (u'In [2]: !ls', "get_ipython().system({u}'ls')"),
520 520 (u'In [3]: abs?', "get_ipython().magic({u}'pinfo abs')"),
521 521 (u'In [4]: b = %who', "b = get_ipython().magic({u}'who')"),
522 522 ]],
523 523 )
524 524
525 525 # multiline syntax examples. Each of these should be a list of lists, with
526 526 # each entry itself having pairs of raw/transformed input. The union (with
527 527 # '\n'.join() of the transformed inputs is what the splitter should produce
528 528 # when fed the raw lines one at a time via push.
529 529 syntax_ml = \
530 530 dict(classic_prompt =
531 531 [ [('>>> for i in range(10):','for i in range(10):'),
532 532 ('... print i',' print i'),
533 533 ('... ', ''),
534 534 ],
535 535 ],
536 536
537 537 ipy_prompt =
538 538 [ [('In [24]: for i in range(10):','for i in range(10):'),
539 539 (' ....: print i',' print i'),
540 540 (' ....: ', ''),
541 541 ],
542 542 ],
543 543
544 544 multiline_datastructure =
545 545 [ [('>>> a = [1,','a = [1,'),
546 546 ('... 2]','2]'),
547 547 ],
548 548 ],
549 549 )
550 550
551 551
552 552 def test_assign_system():
553 553 tt.check_pairs(isp.transform_assign_system, syntax['assign_system'])
554 554
555 555
556 556 def test_assign_magic():
557 557 tt.check_pairs(isp.transform_assign_magic, syntax['assign_magic'])
558 558
559 559
560 560 def test_classic_prompt():
561 561 transform_checker(syntax['classic_prompt'], isp.transform_classic_prompt)
562 562 for example in syntax_ml['classic_prompt']:
563 563 transform_checker(example, isp.transform_classic_prompt)
564 564
565 565
566 566 def test_ipy_prompt():
567 567 transform_checker(syntax['ipy_prompt'], isp.transform_ipy_prompt)
568 568 for example in syntax_ml['ipy_prompt']:
569 569 transform_checker(example, isp.transform_ipy_prompt)
570 570
571 571 def test_end_help():
572 572 tt.check_pairs(isp.transform_help_end, syntax['end_help'])
573 573
574 574 def test_escaped_noesc():
575 575 tt.check_pairs(isp.transform_escaped, syntax['escaped_noesc'])
576 576
577 577
578 578 def test_escaped_shell():
579 579 tt.check_pairs(isp.transform_escaped, syntax['escaped_shell'])
580 580
581 581
582 582 def test_escaped_help():
583 583 tt.check_pairs(isp.transform_escaped, syntax['escaped_help'])
584 584
585 585
586 586 def test_escaped_magic():
587 587 tt.check_pairs(isp.transform_escaped, syntax['escaped_magic'])
588 588
589 589
590 590 def test_escaped_quote():
591 591 tt.check_pairs(isp.transform_escaped, syntax['escaped_quote'])
592 592
593 593
594 594 def test_escaped_quote2():
595 595 tt.check_pairs(isp.transform_escaped, syntax['escaped_quote2'])
596 596
597 597
598 598 def test_escaped_paren():
599 599 tt.check_pairs(isp.transform_escaped, syntax['escaped_paren'])
600 600
601 601
602 602 def test_last_blank():
603 603 nt.assert_false(isp.last_blank(''))
604 604 nt.assert_false(isp.last_blank('abc'))
605 605 nt.assert_false(isp.last_blank('abc\n'))
606 606 nt.assert_false(isp.last_blank('abc\na'))
607
607 608 nt.assert_true(isp.last_blank('\n'))
608 609 nt.assert_true(isp.last_blank('\n '))
609 610 nt.assert_true(isp.last_blank('abc\n '))
610 611 nt.assert_true(isp.last_blank('abc\n\n'))
611 612
612 613
614 def test_last_two_blanks():
615 nt.assert_false(isp.last_two_blanks(''))
616 nt.assert_false(isp.last_two_blanks('abc'))
617 nt.assert_false(isp.last_two_blanks('abc\n'))
618 nt.assert_false(isp.last_two_blanks('abc\n\na'))
619
620 nt.assert_true(isp.last_two_blanks('\n\n'))
621 nt.assert_true(isp.last_two_blanks('\n\n '))
622 nt.assert_true(isp.last_two_blanks('\n \n'))
623 nt.assert_true(isp.last_two_blanks('abc\n \n'))
624 nt.assert_true(isp.last_two_blanks('abc\n\n '))
625 nt.assert_true(isp.last_two_blanks('abc\n\n'))
626
627
613 628 def test_cell_magics():
614 629 from IPython.core import magic
615 630
616 631 cell = """\
617 632 %%cellm line
618 633 body
619 634 """
620 635 sp = isp.IPythonInputSplitter(input_mode='line')
621 636 sp.push(cell)
622 637 nt.assert_equal(sp.cell_magic_parts, ['body\n'])
623 638 out = sp.source
624 639 ref = u"get_ipython()._cell_magic(u'cellm', u'line')\n"
625 640 nt.assert_equal(out, ref)
626 641
627 642 sp.reset()
628 643
629 644 sp.push('%%cellm line2\n')
630 645 nt.assert_true(sp.push_accepts_more()) #1
631 646 sp.push('\n')
632 647 nt.assert_true(sp.push_accepts_more()) #2
633 648 sp.push('\n')
634 649 nt.assert_false(sp.push_accepts_more()) #3
635 650
636 651
637 652 class IPythonInputTestCase(InputSplitterTestCase):
638 653 """By just creating a new class whose .isp is a different instance, we
639 654 re-run the same test battery on the new input splitter.
640 655
641 656 In addition, this runs the tests over the syntax and syntax_ml dicts that
642 657 were tested by individual functions, as part of the OO interface.
643 658
644 659 It also makes some checks on the raw buffer storage.
645 660 """
646 661
647 662 def setUp(self):
648 663 self.isp = isp.IPythonInputSplitter(input_mode='line')
649 664
650 665 def test_syntax(self):
651 666 """Call all single-line syntax tests from the main object"""
652 667 isp = self.isp
653 668 for example in syntax.itervalues():
654 669 for raw, out_t in example:
655 670 if raw.startswith(' '):
656 671 continue
657 672
658 673 isp.push(raw)
659 674 out, out_raw = isp.source_raw_reset()
660 675 self.assertEqual(out.rstrip(), out_t,
661 676 tt.pair_fail_msg.format("inputsplitter",raw, out_t, out))
662 677 self.assertEqual(out_raw.rstrip(), raw.rstrip())
663 678
664 679 def test_syntax_multiline(self):
665 680 isp = self.isp
666 681 for example in syntax_ml.itervalues():
667 682 out_t_parts = []
668 683 raw_parts = []
669 684 for line_pairs in example:
670 685 for lraw, out_t_part in line_pairs:
671 686 isp.push(lraw)
672 687 out_t_parts.append(out_t_part)
673 688 raw_parts.append(lraw)
674 689
675 690 out, out_raw = isp.source_raw_reset()
676 691 out_t = '\n'.join(out_t_parts).rstrip()
677 692 raw = '\n'.join(raw_parts).rstrip()
678 693 self.assertEqual(out.rstrip(), out_t)
679 694 self.assertEqual(out_raw.rstrip(), raw)
680 695
681 696
682 697 class BlockIPythonInputTestCase(IPythonInputTestCase):
683 698
684 699 # Deactivate tests that don't make sense for the block mode
685 700 test_push3 = test_split = lambda s: None
686 701
687 702 def setUp(self):
688 703 self.isp = isp.IPythonInputSplitter(input_mode='cell')
689 704
690 705 def test_syntax_multiline(self):
691 706 isp = self.isp
692 707 for example in syntax_ml.itervalues():
693 708 raw_parts = []
694 709 out_t_parts = []
695 710 for line_pairs in example:
696 711 for raw, out_t_part in line_pairs:
697 712 raw_parts.append(raw)
698 713 out_t_parts.append(out_t_part)
699 714
700 715 raw = '\n'.join(raw_parts)
701 716 out_t = '\n'.join(out_t_parts)
702 717
703 718 isp.push(raw)
704 719 out, out_raw = isp.source_raw_reset()
705 720 # Match ignoring trailing whitespace
706 721 self.assertEqual(out.rstrip(), out_t.rstrip())
707 722 self.assertEqual(out_raw.rstrip(), raw.rstrip())
708 723
709 724
710 725 #-----------------------------------------------------------------------------
711 726 # Main - use as a script, mostly for developer experiments
712 727 #-----------------------------------------------------------------------------
713 728
714 729 if __name__ == '__main__':
715 730 # A simple demo for interactive experimentation. This code will not get
716 731 # picked up by any test suite.
717 732 from IPython.core.inputsplitter import InputSplitter, IPythonInputSplitter
718 733
719 734 # configure here the syntax to use, prompt and whether to autoindent
720 735 #isp, start_prompt = InputSplitter(), '>>> '
721 736 isp, start_prompt = IPythonInputSplitter(), 'In> '
722 737
723 738 autoindent = True
724 739 #autoindent = False
725 740
726 741 try:
727 742 while True:
728 743 prompt = start_prompt
729 744 while isp.push_accepts_more():
730 745 indent = ' '*isp.indent_spaces
731 746 if autoindent:
732 747 line = indent + raw_input(prompt+indent)
733 748 else:
734 749 line = raw_input(prompt)
735 750 isp.push(line)
736 751 prompt = '... '
737 752
738 753 # Here we just return input so we can use it in a test suite, but a
739 754 # real interpreter would instead send it for execution somewhere.
740 755 #src = isp.source; raise EOFError # dbg
741 756 src, raw = isp.source_raw_reset()
742 757 print 'Input source was:\n', src
743 758 print 'Raw source was:\n', raw
744 759 except EOFError:
745 760 print 'Bye'
General Comments 0
You need to be logged in to leave comments. Login now