##// END OF EJS Templates
Small edits for clarity and local optimization.
Fernando Perez -
Show More
@@ -1,772 +1,773 b''
1 1 """Analysis of text input into executable blocks.
2 2
3 3 The main class in this module, :class:`InputSplitter`, is designed to break
4 4 input from either interactive, line-by-line environments or block-based ones,
5 5 into standalone blocks that can be executed by Python as 'single' statements
6 6 (thus triggering sys.displayhook).
7 7
8 8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 9 with full support for the extended IPython syntax (magics, system calls, etc).
10 10
11 11 For more details, see the class docstring below.
12 12
13 13 Syntax Transformations
14 14 ----------------------
15 15
16 16 One of the main jobs of the code in this file is to apply all syntax
17 17 transformations that make up 'the IPython language', i.e. magics, shell
18 18 escapes, etc. All transformations should be implemented as *fully stateless*
19 19 entities, that simply take one line as their input and return a line.
20 20 Internally for implementation purposes they may be a normal function or a
21 21 callable object, but the only input they receive will be a single line and they
22 22 should only return a line, without holding any data-dependent state between
23 23 calls.
24 24
25 25 As an example, the EscapedTransformer is a class so we can more clearly group
26 26 together the functionality of dispatching to individual functions based on the
27 27 starting escape character, but the only method for public use is its call
28 28 method.
29 29
30 30
31 31 ToDo
32 32 ----
33 33
34 34 - Should we make push() actually raise an exception once push_accepts_more()
35 35 returns False?
36 36
37 37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
38 38 at least just attributes of a class so not really very exposed.
39 39
40 40 - Think about the best way to support dynamic things: automagic, autocall,
41 41 macros, etc.
42 42
43 43 - Think of a better heuristic for the application of the transforms in
44 44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
45 45 track indentation change events (indent, dedent, nothing) and apply them only
46 46 if the indentation went up, but not otherwise.
47 47
48 48 - Think of the cleanest way for supporting user-specified transformations (the
49 49 user prefilters we had before).
50 50
51 51 Authors
52 52 -------
53 53
54 54 * Fernando Perez
55 55 * Brian Granger
56 56 """
57 57 #-----------------------------------------------------------------------------
58 58 # Copyright (C) 2010 The IPython Development Team
59 59 #
60 60 # Distributed under the terms of the BSD License. The full license is in
61 61 # the file COPYING, distributed as part of this software.
62 62 #-----------------------------------------------------------------------------
63 63 from __future__ import print_function
64 64
65 65 #-----------------------------------------------------------------------------
66 66 # Imports
67 67 #-----------------------------------------------------------------------------
68 68 # stdlib
69 69 import ast
70 70 import codeop
71 71 import re
72 72 import sys
73 73 import tokenize
74 74 from StringIO import StringIO
75 75
76 76 # IPython modules
77 77 from IPython.core.splitinput import split_user_input, LineInfo
78 78 from IPython.utils.text import make_quoted_expr
79 79 from IPython.utils.py3compat import cast_unicode
80 80
81 81 #-----------------------------------------------------------------------------
82 82 # Globals
83 83 #-----------------------------------------------------------------------------
84 84
85 85 # The escape sequences that define the syntax transformations IPython will
86 86 # apply to user input. These can NOT be just changed here: many regular
87 87 # expressions and other parts of the code may use their hardcoded values, and
88 88 # for all intents and purposes they constitute the 'IPython syntax', so they
89 89 # should be considered fixed.
90 90
91 91 ESC_SHELL = '!' # Send line to underlying system shell
92 92 ESC_SH_CAP = '!!' # Send line to system shell and capture output
93 93 ESC_HELP = '?' # Find information about object
94 94 ESC_HELP2 = '??' # Find extra-detailed information about object
95 95 ESC_MAGIC = '%' # Call magic function
96 96 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
97 97 ESC_QUOTE2 = ';' # Quote all args as a single string, call
98 98 ESC_PAREN = '/' # Call first argument with rest of line as arguments
99 99
100 100 #-----------------------------------------------------------------------------
101 101 # Utilities
102 102 #-----------------------------------------------------------------------------
103 103
104 104 # FIXME: These are general-purpose utilities that later can be moved to the
105 105 # general ward. Kept here for now because we're being very strict about test
106 106 # coverage with this code, and this lets us ensure that we keep 100% coverage
107 107 # while developing.
108 108
109 109 # compiled regexps for autoindent management
110 110 dedent_re = re.compile('|'.join([
111 111 r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
112 112 r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
113 113 r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
114 114 r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
115 115 r'^\s+pass\s*$' # pass (optionally followed by trailing spaces)
116 116 ]))
117 117 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
118 118
119 119 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
120 120 # before pure comments
121 121 comment_line_re = re.compile('^\s*\#')
122 122
123 123
124 124 def num_ini_spaces(s):
125 125 """Return the number of initial spaces in a string.
126 126
127 127 Note that tabs are counted as a single space. For now, we do *not* support
128 128 mixing of tabs and spaces in the user's input.
129 129
130 130 Parameters
131 131 ----------
132 132 s : string
133 133
134 134 Returns
135 135 -------
136 136 n : int
137 137 """
138 138
139 139 ini_spaces = ini_spaces_re.match(s)
140 140 if ini_spaces:
141 141 return ini_spaces.end()
142 142 else:
143 143 return 0
144 144
145 145
146 146 def remove_comments(src):
147 147 """Remove all comments from input source.
148 148
149 149 Note: comments are NOT recognized inside of strings!
150 150
151 151 Parameters
152 152 ----------
153 153 src : string
154 154 A single or multiline input string.
155 155
156 156 Returns
157 157 -------
158 158 String with all Python comments removed.
159 159 """
160 160
161 161 return re.sub('#.*', '', src)
162 162
163 163 def has_comment(src):
164 164 """Indicate whether an input line has (i.e. ends in, or is) a comment.
165 165
166 166 This uses tokenize, so it can distinguish comments from # inside strings.
167 167
168 168 Parameters
169 169 ----------
170 170 src : string
171 171 A single line input string.
172 172
173 173 Returns
174 174 -------
175 175 Boolean: True if source has a comment.
176 176 """
177 177 readline = StringIO(src).readline
178 178 toktypes = set()
179 179 try:
180 180 for t in tokenize.generate_tokens(readline):
181 181 toktypes.add(t[0])
182 182 except tokenize.TokenError:
183 183 pass
184 184 return(tokenize.COMMENT in toktypes)
185 185
186 186
187 187 def get_input_encoding():
188 188 """Return the default standard input encoding.
189 189
190 190 If sys.stdin has no encoding, 'ascii' is returned."""
191 191 # There are strange environments for which sys.stdin.encoding is None. We
192 192 # ensure that a valid encoding is returned.
193 193 encoding = getattr(sys.stdin, 'encoding', None)
194 194 if encoding is None:
195 195 encoding = 'ascii'
196 196 return encoding
197 197
198 198 #-----------------------------------------------------------------------------
199 199 # Classes and functions for normal Python syntax handling
200 200 #-----------------------------------------------------------------------------
201 201
202 202 class InputSplitter(object):
203 203 """An object that can accumulate lines of Python source before execution.
204 204
205 205 This object is designed to be fed python source line-by-line, using
206 206 :meth:`push`. It will return on each push whether the currently pushed
207 207 code could be executed already. In addition, it provides a method called
208 208 :meth:`push_accepts_more` that can be used to query whether more input
209 209 can be pushed into a single interactive block.
210 210
211 211 This is a simple example of how an interactive terminal-based client can use
212 212 this tool::
213 213
214 214 isp = InputSplitter()
215 215 while isp.push_accepts_more():
216 216 indent = ' '*isp.indent_spaces
217 217 prompt = '>>> ' + indent
218 218 line = indent + raw_input(prompt)
219 219 isp.push(line)
220 220 print 'Input source was:\n', isp.source_reset(),
221 221 """
222 222 # Number of spaces of indentation computed from input that has been pushed
223 223 # so far. This is the attributes callers should query to get the current
224 224 # indentation level, in order to provide auto-indent facilities.
225 225 indent_spaces = 0
226 226 # String, indicating the default input encoding. It is computed by default
227 227 # at initialization time via get_input_encoding(), but it can be reset by a
228 228 # client with specific knowledge of the encoding.
229 229 encoding = ''
230 230 # String where the current full source input is stored, properly encoded.
231 231 # Reading this attribute is the normal way of querying the currently pushed
232 232 # source code, that has been properly encoded.
233 233 source = ''
234 234 # Code object corresponding to the current source. It is automatically
235 235 # synced to the source, so it can be queried at any time to obtain the code
236 236 # object; it will be None if the source doesn't compile to valid Python.
237 237 code = None
238 238 # Input mode
239 239 input_mode = 'line'
240 240
241 241 # Private attributes
242 242
243 243 # List with lines of input accumulated so far
244 244 _buffer = None
245 245 # Command compiler
246 246 _compile = None
247 247 # Mark when input has changed indentation all the way back to flush-left
248 248 _full_dedent = False
249 249 # Boolean indicating whether the current block is complete
250 250 _is_complete = None
251 251
252 252 def __init__(self, input_mode=None):
253 253 """Create a new InputSplitter instance.
254 254
255 255 Parameters
256 256 ----------
257 257 input_mode : str
258 258
259 259 One of ['line', 'cell']; default is 'line'.
260 260
261 261 The input_mode parameter controls how new inputs are used when fed via
262 262 the :meth:`push` method:
263 263
264 264 - 'line': meant for line-oriented clients, inputs are appended one at a
265 265 time to the internal buffer and the whole buffer is compiled.
266 266
267 267 - 'cell': meant for clients that can edit multi-line 'cells' of text at
268 268 a time. A cell can contain one or more blocks that can be compile in
269 269 'single' mode by Python. In this mode, each new input new input
270 270 completely replaces all prior inputs. Cell mode is thus equivalent
271 271 to prepending a full reset() to every push() call.
272 272 """
273 273 self._buffer = []
274 274 self._compile = codeop.CommandCompiler()
275 275 self.encoding = get_input_encoding()
276 276 self.input_mode = InputSplitter.input_mode if input_mode is None \
277 277 else input_mode
278 278
279 279 def reset(self):
280 280 """Reset the input buffer and associated state."""
281 281 self.indent_spaces = 0
282 282 self._buffer[:] = []
283 283 self.source = ''
284 284 self.code = None
285 285 self._is_complete = False
286 286 self._full_dedent = False
287 287
288 288 def source_reset(self):
289 289 """Return the input source and perform a full reset.
290 290 """
291 291 out = self.source
292 292 self.reset()
293 293 return out
294 294
295 295 def push(self, lines):
296 296 """Push one or more lines of input.
297 297
298 298 This stores the given lines and returns a status code indicating
299 299 whether the code forms a complete Python block or not.
300 300
301 301 Any exceptions generated in compilation are swallowed, but if an
302 302 exception was produced, the method returns True.
303 303
304 304 Parameters
305 305 ----------
306 306 lines : string
307 307 One or more lines of Python input.
308 308
309 309 Returns
310 310 -------
311 311 is_complete : boolean
312 312 True if the current input source (the result of the current input
313 313 plus prior inputs) forms a complete Python execution block. Note that
314 314 this value is also stored as a private attribute (_is_complete), so it
315 315 can be queried at any time.
316 316 """
317 317 if self.input_mode == 'cell':
318 318 self.reset()
319 319
320 320 self._store(lines)
321 321 source = self.source
322 322
323 323 # Before calling _compile(), reset the code object to None so that if an
324 324 # exception is raised in compilation, we don't mislead by having
325 325 # inconsistent code/source attributes.
326 326 self.code, self._is_complete = None, None
327 327
328 328 # Honor termination lines properly
329 329 if source.rstrip().endswith('\\'):
330 330 return False
331 331
332 332 self._update_indent(lines)
333 333 try:
334 334 self.code = self._compile(source, symbol="exec")
335 335 # Invalid syntax can produce any of a number of different errors from
336 336 # inside the compiler, so we have to catch them all. Syntax errors
337 337 # immediately produce a 'ready' block, so the invalid Python can be
338 338 # sent to the kernel for evaluation with possible ipython
339 339 # special-syntax conversion.
340 340 except (SyntaxError, OverflowError, ValueError, TypeError,
341 341 MemoryError):
342 342 self._is_complete = True
343 343 else:
344 344 # Compilation didn't produce any exceptions (though it may not have
345 345 # given a complete code object)
346 346 self._is_complete = self.code is not None
347 347
348 348 return self._is_complete
349 349
350 350 def push_accepts_more(self):
351 351 """Return whether a block of interactive input can accept more input.
352 352
353 353 This method is meant to be used by line-oriented frontends, who need to
354 354 guess whether a block is complete or not based solely on prior and
355 355 current input lines. The InputSplitter considers it has a complete
356 356 interactive block and will not accept more input only when either a
357 357 SyntaxError is raised, or *all* of the following are true:
358 358
359 359 1. The input compiles to a complete statement.
360 360
361 361 2. The indentation level is flush-left (because if we are indented,
362 362 like inside a function definition or for loop, we need to keep
363 363 reading new input).
364 364
365 365 3. There is one extra line consisting only of whitespace.
366 366
367 367 Because of condition #3, this method should be used only by
368 368 *line-oriented* frontends, since it means that intermediate blank lines
369 369 are not allowed in function definitions (or any other indented block).
370 370
371 371 If the current input produces a syntax error, this method immediately
372 372 returns False but does *not* raise the syntax error exception, as
373 373 typically clients will want to send invalid syntax to an execution
374 374 backend which might convert the invalid syntax into valid Python via
375 375 one of the dynamic IPython mechanisms.
376 376 """
377 377
378 378 # With incomplete input, unconditionally accept more
379 379 if not self._is_complete:
380 380 return True
381 381
382 382 # If we already have complete input and we're flush left, the answer
383 383 # depends. In line mode, if there hasn't been any indentation,
384 384 # that's it. If we've come back from some indentation, we need
385 385 # the blank final line to finish.
386 386 # In cell mode, we need to check how many blocks the input so far
387 387 # compiles into, because if there's already more than one full
388 388 # independent block of input, then the client has entered full
389 389 # 'cell' mode and is feeding lines that each is complete. In this
390 390 # case we should then keep accepting. The Qt terminal-like console
391 391 # does precisely this, to provide the convenience of terminal-like
392 392 # input of single expressions, but allowing the user (with a
393 393 # separate keystroke) to switch to 'cell' mode and type multiple
394 394 # expressions in one shot.
395 395 if self.indent_spaces==0:
396 396 if self.input_mode=='line':
397 397 if not self._full_dedent:
398 398 return False
399 399 else:
400 400 try:
401 401 code_ast = ast.parse(u''.join(self._buffer))
402 402 except Exception:
403 403 return False
404 404 else:
405 405 if len(code_ast.body) == 1:
406 406 return False
407 407
408 408 # When input is complete, then termination is marked by an extra blank
409 409 # line at the end.
410 410 last_line = self.source.splitlines()[-1]
411 411 return bool(last_line and not last_line.isspace())
412 412
413 413 #------------------------------------------------------------------------
414 414 # Private interface
415 415 #------------------------------------------------------------------------
416 416
417 417 def _find_indent(self, line):
418 418 """Compute the new indentation level for a single line.
419 419
420 420 Parameters
421 421 ----------
422 422 line : str
423 423 A single new line of non-whitespace, non-comment Python input.
424 424
425 425 Returns
426 426 -------
427 427 indent_spaces : int
428 428 New value for the indent level (it may be equal to self.indent_spaces
429 429 if indentation doesn't change.
430 430
431 431 full_dedent : boolean
432 432 Whether the new line causes a full flush-left dedent.
433 433 """
434 434 indent_spaces = self.indent_spaces
435 435 full_dedent = self._full_dedent
436 436
437 437 inisp = num_ini_spaces(line)
438 438 if inisp < indent_spaces:
439 439 indent_spaces = inisp
440 440 if indent_spaces <= 0:
441 441 #print 'Full dedent in text',self.source # dbg
442 442 full_dedent = True
443 443
444 444 if line.rstrip()[-1] == ':':
445 445 indent_spaces += 4
446 446 elif dedent_re.match(line):
447 447 indent_spaces -= 4
448 448 if indent_spaces <= 0:
449 449 full_dedent = True
450 450
451 451 # Safety
452 452 if indent_spaces < 0:
453 453 indent_spaces = 0
454 454 #print 'safety' # dbg
455 455
456 456 return indent_spaces, full_dedent
457 457
458 458 def _update_indent(self, lines):
459 459 for line in remove_comments(lines).splitlines():
460 460 if line and not line.isspace():
461 461 self.indent_spaces, self._full_dedent = self._find_indent(line)
462 462
463 463 def _store(self, lines, buffer=None, store='source'):
464 464 """Store one or more lines of input.
465 465
466 466 If input lines are not newline-terminated, a newline is automatically
467 467 appended."""
468 468
469 469 if buffer is None:
470 470 buffer = self._buffer
471 471
472 472 if lines.endswith('\n'):
473 473 buffer.append(lines)
474 474 else:
475 475 buffer.append(lines+'\n')
476 476 setattr(self, store, self._set_source(buffer))
477 477
478 478 def _set_source(self, buffer):
479 479 return u''.join(buffer)
480 480
481 481
482 482 #-----------------------------------------------------------------------------
483 483 # Functions and classes for IPython-specific syntactic support
484 484 #-----------------------------------------------------------------------------
485 485
486 486 # The escaped translators ALL receive a line where their own escape has been
487 487 # stripped. Only '?' is valid at the end of the line, all others can only be
488 488 # placed at the start.
489 489
490 490 # Transformations of the special syntaxes that don't rely on an explicit escape
491 491 # character but instead on patterns on the input line
492 492
493 493 # The core transformations are implemented as standalone functions that can be
494 494 # tested and validated in isolation. Each of these uses a regexp, we
495 495 # pre-compile these and keep them close to each function definition for clarity
496 496
497 497 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
498 498 r'\s*=\s*!\s*(?P<cmd>.*)')
499 499
500 500 def transform_assign_system(line):
501 501 """Handle the `files = !ls` syntax."""
502 502 m = _assign_system_re.match(line)
503 503 if m is not None:
504 504 cmd = m.group('cmd')
505 505 lhs = m.group('lhs')
506 506 expr = make_quoted_expr(cmd)
507 507 new_line = '%s = get_ipython().getoutput(%s)' % (lhs, expr)
508 508 return new_line
509 509 return line
510 510
511 511
512 512 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
513 513 r'\s*=\s*%\s*(?P<cmd>.*)')
514 514
515 515 def transform_assign_magic(line):
516 516 """Handle the `a = %who` syntax."""
517 517 m = _assign_magic_re.match(line)
518 518 if m is not None:
519 519 cmd = m.group('cmd')
520 520 lhs = m.group('lhs')
521 521 expr = make_quoted_expr(cmd)
522 522 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
523 523 return new_line
524 524 return line
525 525
526 526
527 527 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
528 528
529 529 def transform_classic_prompt(line):
530 530 """Handle inputs that start with '>>> ' syntax."""
531 531
532 532 if not line or line.isspace():
533 533 return line
534 534 m = _classic_prompt_re.match(line)
535 535 if m:
536 536 return line[len(m.group(0)):]
537 537 else:
538 538 return line
539 539
540 540
541 541 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
542 542
543 543 def transform_ipy_prompt(line):
544 544 """Handle inputs that start classic IPython prompt syntax."""
545 545
546 546 if not line or line.isspace():
547 547 return line
548 548 #print 'LINE: %r' % line # dbg
549 549 m = _ipy_prompt_re.match(line)
550 550 if m:
551 551 #print 'MATCH! %r -> %r' % (line, line[len(m.group(0)):]) # dbg
552 552 return line[len(m.group(0)):]
553 553 else:
554 554 return line
555 555
556 556
557 557 def _make_help_call(target, esc, lspace, next_input=None):
558 558 """Prepares a pinfo(2)/psearch call from a target name and the escape
559 559 (i.e. ? or ??)"""
560 560 method = 'pinfo2' if esc == '??' \
561 561 else 'psearch' if '*' in target \
562 562 else 'pinfo'
563 563 arg = make_quoted_expr(" ".join([method, target]))
564 564
565 565 if next_input:
566 566 tpl = '%sget_ipython().magic(%s, next_input=%s)'
567 567 return tpl % (lspace, arg, make_quoted_expr(next_input))
568 568 else:
569 569 return '%sget_ipython().magic(%s)' % (lspace, arg)
570 570
571 571 _initial_space_re = re.compile(r'\s*')
572 572 _help_end_re = re.compile(r"""(%?
573 573 [a-zA-Z_*][\w*]* # Variable name
574 574 (\.[a-zA-Z_*][\w*]*)* # .etc.etc
575 575 )
576 576 (\?\??)$ # ? or ??""",
577 577 re.VERBOSE)
578 578 def transform_help_end(line):
579 579 """Translate lines with ?/?? at the end"""
580 580 m = _help_end_re.search(line)
581 581 if m is None or has_comment(line):
582 582 return line
583 583 target = m.group(1)
584 584 esc = m.group(3)
585 585 lspace = _initial_space_re.match(line).group(0)
586 586
587 587 # If we're mid-command, put it back on the next prompt for the user.
588 588 next_input = line.rstrip('?') if line.strip() != m.group(0) else None
589 589
590 590 return _make_help_call(target, esc, lspace, next_input)
591 591
592 592
593 593 class EscapedTransformer(object):
594 594 """Class to transform lines that are explicitly escaped out."""
595 595
596 596 def __init__(self):
597 597 tr = { ESC_SHELL : self._tr_system,
598 598 ESC_SH_CAP : self._tr_system2,
599 599 ESC_HELP : self._tr_help,
600 600 ESC_HELP2 : self._tr_help,
601 601 ESC_MAGIC : self._tr_magic,
602 602 ESC_QUOTE : self._tr_quote,
603 603 ESC_QUOTE2 : self._tr_quote2,
604 604 ESC_PAREN : self._tr_paren }
605 605 self.tr = tr
606 606
607 607 # Support for syntax transformations that use explicit escapes typed by the
608 608 # user at the beginning of a line
609 609 @staticmethod
610 610 def _tr_system(line_info):
611 611 "Translate lines escaped with: !"
612 612 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
613 613 return '%sget_ipython().system(%s)' % (line_info.pre,
614 614 make_quoted_expr(cmd))
615 615
616 616 @staticmethod
617 617 def _tr_system2(line_info):
618 618 "Translate lines escaped with: !!"
619 619 cmd = line_info.line.lstrip()[2:]
620 620 return '%sget_ipython().getoutput(%s)' % (line_info.pre,
621 621 make_quoted_expr(cmd))
622 622
623 623 @staticmethod
624 624 def _tr_help(line_info):
625 625 "Translate lines escaped with: ?/??"
626 626 # A naked help line should just fire the intro help screen
627 627 if not line_info.line[1:]:
628 628 return 'get_ipython().show_usage()'
629 629
630 630 return _make_help_call(line_info.ifun, line_info.esc, line_info.pre)
631 631
632 632 @staticmethod
633 633 def _tr_magic(line_info):
634 634 "Translate lines escaped with: %"
635 635 tpl = '%sget_ipython().magic(%s)'
636 636 cmd = make_quoted_expr(' '.join([line_info.ifun,
637 637 line_info.the_rest]).strip())
638 638 return tpl % (line_info.pre, cmd)
639 639
640 640 @staticmethod
641 641 def _tr_quote(line_info):
642 642 "Translate lines escaped with: ,"
643 643 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
644 644 '", "'.join(line_info.the_rest.split()) )
645 645
646 646 @staticmethod
647 647 def _tr_quote2(line_info):
648 648 "Translate lines escaped with: ;"
649 649 return '%s%s("%s")' % (line_info.pre, line_info.ifun,
650 650 line_info.the_rest)
651 651
652 652 @staticmethod
653 653 def _tr_paren(line_info):
654 654 "Translate lines escaped with: /"
655 655 return '%s%s(%s)' % (line_info.pre, line_info.ifun,
656 656 ", ".join(line_info.the_rest.split()))
657 657
658 658 def __call__(self, line):
659 659 """Class to transform lines that are explicitly escaped out.
660 660
661 661 This calls the above _tr_* static methods for the actual line
662 662 translations."""
663 663
664 664 # Empty lines just get returned unmodified
665 665 if not line or line.isspace():
666 666 return line
667 667
668 668 # Get line endpoints, where the escapes can be
669 669 line_info = LineInfo(line)
670 670
671 671 if not line_info.esc in self.tr:
672 672 # If we don't recognize the escape, don't modify the line
673 673 return line
674 674
675 675 return self.tr[line_info.esc](line_info)
676 676
677 677
678 678 # A function-looking object to be used by the rest of the code. The purpose of
679 679 # the class in this case is to organize related functionality, more than to
680 680 # manage state.
681 681 transform_escaped = EscapedTransformer()
682 682
683 683
684 684 class IPythonInputSplitter(InputSplitter):
685 685 """An input splitter that recognizes all of IPython's special syntax."""
686 686
687 687 # String with raw, untransformed input.
688 688 source_raw = ''
689 689
690 690 # Private attributes
691 691
692 692 # List with lines of raw input accumulated so far.
693 693 _buffer_raw = None
694 694
695 695 def __init__(self, input_mode=None):
696 696 InputSplitter.__init__(self, input_mode)
697 697 self._buffer_raw = []
698 698
699 699 def reset(self):
700 700 """Reset the input buffer and associated state."""
701 701 InputSplitter.reset(self)
702 702 self._buffer_raw[:] = []
703 703 self.source_raw = ''
704 704
705 705 def source_raw_reset(self):
706 706 """Return input and raw source and perform a full reset.
707 707 """
708 708 out = self.source
709 709 out_r = self.source_raw
710 710 self.reset()
711 711 return out, out_r
712 712
713 713 def push(self, lines):
714 714 """Push one or more lines of IPython input.
715 715 """
716 716 if not lines:
717 717 return super(IPythonInputSplitter, self).push(lines)
718 718
719 719 # We must ensure all input is pure unicode
720 720 lines = cast_unicode(lines, self.encoding)
721 721
722 722 lines_list = lines.splitlines()
723 723
724 724 transforms = [transform_ipy_prompt, transform_classic_prompt,
725 725 transform_help_end, transform_escaped,
726 726 transform_assign_system, transform_assign_magic]
727 727
728 728 # Transform logic
729 729 #
730 730 # We only apply the line transformers to the input if we have either no
731 731 # input yet, or complete input, or if the last line of the buffer ends
732 732 # with ':' (opening an indented block). This prevents the accidental
733 733 # transformation of escapes inside multiline expressions like
734 734 # triple-quoted strings or parenthesized expressions.
735 735 #
736 736 # The last heuristic, while ugly, ensures that the first line of an
737 737 # indented block is correctly transformed.
738 738 #
739 739 # FIXME: try to find a cleaner approach for this last bit.
740 740
741 741 # If we were in 'block' mode, since we're going to pump the parent
742 742 # class by hand line by line, we need to temporarily switch out to
743 743 # 'line' mode, do a single manual reset and then feed the lines one
744 744 # by one. Note that this only matters if the input has more than one
745 745 # line.
746 746 changed_input_mode = False
747 747
748 748 if self.input_mode == 'cell':
749 749 self.reset()
750 750 changed_input_mode = True
751 751 saved_input_mode = 'cell'
752 752 self.input_mode = 'line'
753 753
754 754 # Store raw source before applying any transformations to it. Note
755 755 # that this must be done *after* the reset() call that would otherwise
756 756 # flush the buffer.
757 757 self._store(lines, self._buffer_raw, 'source_raw')
758 758
759 759 try:
760 760 push = super(IPythonInputSplitter, self).push
761 buf = self._buffer
761 762 for line in lines_list:
762 if self._is_complete or not self._buffer or \
763 (self._buffer and (self._buffer[-1].rstrip().endswith(':') or\
764 self._buffer[-1].rstrip().endswith(','))):
763 if self._is_complete or not buf or \
764 (buf and (buf[-1].rstrip().endswith(':') or
765 buf[-1].rstrip().endswith(',')) ):
765 766 for f in transforms:
766 767 line = f(line)
767 768
768 769 out = push(line)
769 770 finally:
770 771 if changed_input_mode:
771 772 self.input_mode = saved_input_mode
772 773 return out
General Comments 0
You need to be logged in to leave comments. Login now