##// END OF EJS Templates
Fix crash with 'x?' when input line had unusual patterns.
Fernando Perez -
Show More
@@ -1,987 +1,986 b''
1 """Analysis of text input into executable blocks.
1 """Analysis of text input into executable blocks.
2
2
3 The main class in this module, :class:`InputSplitter`, is designed to break
3 The main class in this module, :class:`InputSplitter`, is designed to break
4 input from either interactive, line-by-line environments or block-based ones,
4 input from either interactive, line-by-line environments or block-based ones,
5 into standalone blocks that can be executed by Python as 'single' statements
5 into standalone blocks that can be executed by Python as 'single' statements
6 (thus triggering sys.displayhook).
6 (thus triggering sys.displayhook).
7
7
8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 with full support for the extended IPython syntax (magics, system calls, etc).
9 with full support for the extended IPython syntax (magics, system calls, etc).
10
10
11 For more details, see the class docstring below.
11 For more details, see the class docstring below.
12
12
13 Syntax Transformations
13 Syntax Transformations
14 ----------------------
14 ----------------------
15
15
16 One of the main jobs of the code in this file is to apply all syntax
16 One of the main jobs of the code in this file is to apply all syntax
17 transformations that make up 'the IPython language', i.e. magics, shell
17 transformations that make up 'the IPython language', i.e. magics, shell
18 escapes, etc. All transformations should be implemented as *fully stateless*
18 escapes, etc. All transformations should be implemented as *fully stateless*
19 entities, that simply take one line as their input and return a line.
19 entities, that simply take one line as their input and return a line.
20 Internally for implementation purposes they may be a normal function or a
20 Internally for implementation purposes they may be a normal function or a
21 callable object, but the only input they receive will be a single line and they
21 callable object, but the only input they receive will be a single line and they
22 should only return a line, without holding any data-dependent state between
22 should only return a line, without holding any data-dependent state between
23 calls.
23 calls.
24
24
25 As an example, the EscapedTransformer is a class so we can more clearly group
25 As an example, the EscapedTransformer is a class so we can more clearly group
26 together the functionality of dispatching to individual functions based on the
26 together the functionality of dispatching to individual functions based on the
27 starting escape character, but the only method for public use is its call
27 starting escape character, but the only method for public use is its call
28 method.
28 method.
29
29
30
30
31 ToDo
31 ToDo
32 ----
32 ----
33
33
34 - Should we make push() actually raise an exception once push_accepts_more()
34 - Should we make push() actually raise an exception once push_accepts_more()
35 returns False?
35 returns False?
36
36
37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
38 at least just attributes of a class so not really very exposed.
38 at least just attributes of a class so not really very exposed.
39
39
40 - Think about the best way to support dynamic things: automagic, autocall,
40 - Think about the best way to support dynamic things: automagic, autocall,
41 macros, etc.
41 macros, etc.
42
42
43 - Think of a better heuristic for the application of the transforms in
43 - Think of a better heuristic for the application of the transforms in
44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
45 track indentation change events (indent, dedent, nothing) and apply them only
45 track indentation change events (indent, dedent, nothing) and apply them only
46 if the indentation went up, but not otherwise.
46 if the indentation went up, but not otherwise.
47
47
48 - Think of the cleanest way for supporting user-specified transformations (the
48 - Think of the cleanest way for supporting user-specified transformations (the
49 user prefilters we had before).
49 user prefilters we had before).
50
50
51 Authors
51 Authors
52 -------
52 -------
53
53
54 * Fernando Perez
54 * Fernando Perez
55 * Brian Granger
55 * Brian Granger
56 """
56 """
57 #-----------------------------------------------------------------------------
57 #-----------------------------------------------------------------------------
58 # Copyright (C) 2010 The IPython Development Team
58 # Copyright (C) 2010 The IPython Development Team
59 #
59 #
60 # Distributed under the terms of the BSD License. The full license is in
60 # Distributed under the terms of the BSD License. The full license is in
61 # the file COPYING, distributed as part of this software.
61 # the file COPYING, distributed as part of this software.
62 #-----------------------------------------------------------------------------
62 #-----------------------------------------------------------------------------
63 from __future__ import print_function
63 from __future__ import print_function
64
64
65 #-----------------------------------------------------------------------------
65 #-----------------------------------------------------------------------------
66 # Imports
66 # Imports
67 #-----------------------------------------------------------------------------
67 #-----------------------------------------------------------------------------
68 # stdlib
68 # stdlib
69 import codeop
69 import codeop
70 import re
70 import re
71 import sys
71 import sys
72
72
73 # IPython modules
73 # IPython modules
74 from IPython.utils.text import make_quoted_expr
74 from IPython.utils.text import make_quoted_expr
75
75
76 #-----------------------------------------------------------------------------
76 #-----------------------------------------------------------------------------
77 # Globals
77 # Globals
78 #-----------------------------------------------------------------------------
78 #-----------------------------------------------------------------------------
79
79
80 # The escape sequences that define the syntax transformations IPython will
80 # The escape sequences that define the syntax transformations IPython will
81 # apply to user input. These can NOT be just changed here: many regular
81 # apply to user input. These can NOT be just changed here: many regular
82 # expressions and other parts of the code may use their hardcoded values, and
82 # expressions and other parts of the code may use their hardcoded values, and
83 # for all intents and purposes they constitute the 'IPython syntax', so they
83 # for all intents and purposes they constitute the 'IPython syntax', so they
84 # should be considered fixed.
84 # should be considered fixed.
85
85
86 ESC_SHELL = '!' # Send line to underlying system shell
86 ESC_SHELL = '!' # Send line to underlying system shell
87 ESC_SH_CAP = '!!' # Send line to system shell and capture output
87 ESC_SH_CAP = '!!' # Send line to system shell and capture output
88 ESC_HELP = '?' # Find information about object
88 ESC_HELP = '?' # Find information about object
89 ESC_HELP2 = '??' # Find extra-detailed information about object
89 ESC_HELP2 = '??' # Find extra-detailed information about object
90 ESC_MAGIC = '%' # Call magic function
90 ESC_MAGIC = '%' # Call magic function
91 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
91 ESC_QUOTE = ',' # Split args on whitespace, quote each as string and call
92 ESC_QUOTE2 = ';' # Quote all args as a single string, call
92 ESC_QUOTE2 = ';' # Quote all args as a single string, call
93 ESC_PAREN = '/' # Call first argument with rest of line as arguments
93 ESC_PAREN = '/' # Call first argument with rest of line as arguments
94
94
95 #-----------------------------------------------------------------------------
95 #-----------------------------------------------------------------------------
96 # Utilities
96 # Utilities
97 #-----------------------------------------------------------------------------
97 #-----------------------------------------------------------------------------
98
98
99 # FIXME: These are general-purpose utilities that later can be moved to the
99 # FIXME: These are general-purpose utilities that later can be moved to the
100 # general ward. Kept here for now because we're being very strict about test
100 # general ward. Kept here for now because we're being very strict about test
101 # coverage with this code, and this lets us ensure that we keep 100% coverage
101 # coverage with this code, and this lets us ensure that we keep 100% coverage
102 # while developing.
102 # while developing.
103
103
104 # compiled regexps for autoindent management
104 # compiled regexps for autoindent management
105 dedent_re = re.compile(r'^\s+raise|^\s+return|^\s+pass')
105 dedent_re = re.compile(r'^\s+raise|^\s+return|^\s+pass')
106 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
106 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
107
107
108 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
108 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
109 # before pure comments
109 # before pure comments
110 comment_line_re = re.compile('^\s*\#')
110 comment_line_re = re.compile('^\s*\#')
111
111
112
112
113 def num_ini_spaces(s):
113 def num_ini_spaces(s):
114 """Return the number of initial spaces in a string.
114 """Return the number of initial spaces in a string.
115
115
116 Note that tabs are counted as a single space. For now, we do *not* support
116 Note that tabs are counted as a single space. For now, we do *not* support
117 mixing of tabs and spaces in the user's input.
117 mixing of tabs and spaces in the user's input.
118
118
119 Parameters
119 Parameters
120 ----------
120 ----------
121 s : string
121 s : string
122
122
123 Returns
123 Returns
124 -------
124 -------
125 n : int
125 n : int
126 """
126 """
127
127
128 ini_spaces = ini_spaces_re.match(s)
128 ini_spaces = ini_spaces_re.match(s)
129 if ini_spaces:
129 if ini_spaces:
130 return ini_spaces.end()
130 return ini_spaces.end()
131 else:
131 else:
132 return 0
132 return 0
133
133
134
134
135 def remove_comments(src):
135 def remove_comments(src):
136 """Remove all comments from input source.
136 """Remove all comments from input source.
137
137
138 Note: comments are NOT recognized inside of strings!
138 Note: comments are NOT recognized inside of strings!
139
139
140 Parameters
140 Parameters
141 ----------
141 ----------
142 src : string
142 src : string
143 A single or multiline input string.
143 A single or multiline input string.
144
144
145 Returns
145 Returns
146 -------
146 -------
147 String with all Python comments removed.
147 String with all Python comments removed.
148 """
148 """
149
149
150 return re.sub('#.*', '', src)
150 return re.sub('#.*', '', src)
151
151
152
152
153 def get_input_encoding():
153 def get_input_encoding():
154 """Return the default standard input encoding.
154 """Return the default standard input encoding.
155
155
156 If sys.stdin has no encoding, 'ascii' is returned."""
156 If sys.stdin has no encoding, 'ascii' is returned."""
157 # There are strange environments for which sys.stdin.encoding is None. We
157 # There are strange environments for which sys.stdin.encoding is None. We
158 # ensure that a valid encoding is returned.
158 # ensure that a valid encoding is returned.
159 encoding = getattr(sys.stdin, 'encoding', None)
159 encoding = getattr(sys.stdin, 'encoding', None)
160 if encoding is None:
160 if encoding is None:
161 encoding = 'ascii'
161 encoding = 'ascii'
162 return encoding
162 return encoding
163
163
164 #-----------------------------------------------------------------------------
164 #-----------------------------------------------------------------------------
165 # Classes and functions for normal Python syntax handling
165 # Classes and functions for normal Python syntax handling
166 #-----------------------------------------------------------------------------
166 #-----------------------------------------------------------------------------
167
167
168 # HACK! This implementation, written by Robert K a while ago using the
168 # HACK! This implementation, written by Robert K a while ago using the
169 # compiler module, is more robust than the other one below, but it expects its
169 # compiler module, is more robust than the other one below, but it expects its
170 # input to be pure python (no ipython syntax). For now we're using it as a
170 # input to be pure python (no ipython syntax). For now we're using it as a
171 # second-pass splitter after the first pass transforms the input to pure
171 # second-pass splitter after the first pass transforms the input to pure
172 # python.
172 # python.
173
173
174 def split_blocks(python):
174 def split_blocks(python):
175 """ Split multiple lines of code into discrete commands that can be
175 """ Split multiple lines of code into discrete commands that can be
176 executed singly.
176 executed singly.
177
177
178 Parameters
178 Parameters
179 ----------
179 ----------
180 python : str
180 python : str
181 Pure, exec'able Python code.
181 Pure, exec'able Python code.
182
182
183 Returns
183 Returns
184 -------
184 -------
185 commands : list of str
185 commands : list of str
186 Separate commands that can be exec'ed independently.
186 Separate commands that can be exec'ed independently.
187 """
187 """
188
188
189 import compiler
189 import compiler
190
190
191 # compiler.parse treats trailing spaces after a newline as a
191 # compiler.parse treats trailing spaces after a newline as a
192 # SyntaxError. This is different than codeop.CommandCompiler, which
192 # SyntaxError. This is different than codeop.CommandCompiler, which
193 # will compile the trailng spaces just fine. We simply strip any
193 # will compile the trailng spaces just fine. We simply strip any
194 # trailing whitespace off. Passing a string with trailing whitespace
194 # trailing whitespace off. Passing a string with trailing whitespace
195 # to exec will fail however. There seems to be some inconsistency in
195 # to exec will fail however. There seems to be some inconsistency in
196 # how trailing whitespace is handled, but this seems to work.
196 # how trailing whitespace is handled, but this seems to work.
197 python_ori = python # save original in case we bail on error
197 python_ori = python # save original in case we bail on error
198 python = python.strip()
198 python = python.strip()
199
199
200 # The compiler module does not like unicode. We need to convert
200 # The compiler module does not like unicode. We need to convert
201 # it encode it:
201 # it encode it:
202 if isinstance(python, unicode):
202 if isinstance(python, unicode):
203 # Use the utf-8-sig BOM so the compiler detects this a UTF-8
203 # Use the utf-8-sig BOM so the compiler detects this a UTF-8
204 # encode string.
204 # encode string.
205 python = '\xef\xbb\xbf' + python.encode('utf-8')
205 python = '\xef\xbb\xbf' + python.encode('utf-8')
206
206
207 # The compiler module will parse the code into an abstract syntax tree.
207 # The compiler module will parse the code into an abstract syntax tree.
208 # This has a bug with str("a\nb"), but not str("""a\nb""")!!!
208 # This has a bug with str("a\nb"), but not str("""a\nb""")!!!
209 try:
209 try:
210 ast = compiler.parse(python)
210 ast = compiler.parse(python)
211 except:
211 except:
212 return [python_ori]
212 return [python_ori]
213
213
214 # Uncomment to help debug the ast tree
214 # Uncomment to help debug the ast tree
215 # for n in ast.node:
215 # for n in ast.node:
216 # print n.lineno,'->',n
216 # print n.lineno,'->',n
217
217
218 # Each separate command is available by iterating over ast.node. The
218 # Each separate command is available by iterating over ast.node. The
219 # lineno attribute is the line number (1-indexed) beginning the commands
219 # lineno attribute is the line number (1-indexed) beginning the commands
220 # suite.
220 # suite.
221 # lines ending with ";" yield a Discard Node that doesn't have a lineno
221 # lines ending with ";" yield a Discard Node that doesn't have a lineno
222 # attribute. These nodes can and should be discarded. But there are
222 # attribute. These nodes can and should be discarded. But there are
223 # other situations that cause Discard nodes that shouldn't be discarded.
223 # other situations that cause Discard nodes that shouldn't be discarded.
224 # We might eventually discover other cases where lineno is None and have
224 # We might eventually discover other cases where lineno is None and have
225 # to put in a more sophisticated test.
225 # to put in a more sophisticated test.
226 linenos = [x.lineno-1 for x in ast.node if x.lineno is not None]
226 linenos = [x.lineno-1 for x in ast.node if x.lineno is not None]
227
227
228 # When we finally get the slices, we will need to slice all the way to
228 # When we finally get the slices, we will need to slice all the way to
229 # the end even though we don't have a line number for it. Fortunately,
229 # the end even though we don't have a line number for it. Fortunately,
230 # None does the job nicely.
230 # None does the job nicely.
231 linenos.append(None)
231 linenos.append(None)
232
232
233 # Same problem at the other end: sometimes the ast tree has its
233 # Same problem at the other end: sometimes the ast tree has its
234 # first complete statement not starting on line 0. In this case
234 # first complete statement not starting on line 0. In this case
235 # we might miss part of it. This fixes ticket 266993. Thanks Gael!
235 # we might miss part of it. This fixes ticket 266993. Thanks Gael!
236 linenos[0] = 0
236 linenos[0] = 0
237
237
238 lines = python.splitlines()
238 lines = python.splitlines()
239
239
240 # Create a list of atomic commands.
240 # Create a list of atomic commands.
241 cmds = []
241 cmds = []
242 for i, j in zip(linenos[:-1], linenos[1:]):
242 for i, j in zip(linenos[:-1], linenos[1:]):
243 cmd = lines[i:j]
243 cmd = lines[i:j]
244 if cmd:
244 if cmd:
245 cmds.append('\n'.join(cmd)+'\n')
245 cmds.append('\n'.join(cmd)+'\n')
246
246
247 return cmds
247 return cmds
248
248
249
249
250 class InputSplitter(object):
250 class InputSplitter(object):
251 """An object that can split Python source input in executable blocks.
251 """An object that can split Python source input in executable blocks.
252
252
253 This object is designed to be used in one of two basic modes:
253 This object is designed to be used in one of two basic modes:
254
254
255 1. By feeding it python source line-by-line, using :meth:`push`. In this
255 1. By feeding it python source line-by-line, using :meth:`push`. In this
256 mode, it will return on each push whether the currently pushed code
256 mode, it will return on each push whether the currently pushed code
257 could be executed already. In addition, it provides a method called
257 could be executed already. In addition, it provides a method called
258 :meth:`push_accepts_more` that can be used to query whether more input
258 :meth:`push_accepts_more` that can be used to query whether more input
259 can be pushed into a single interactive block.
259 can be pushed into a single interactive block.
260
260
261 2. By calling :meth:`split_blocks` with a single, multiline Python string,
261 2. By calling :meth:`split_blocks` with a single, multiline Python string,
262 that is then split into blocks each of which can be executed
262 that is then split into blocks each of which can be executed
263 interactively as a single statement.
263 interactively as a single statement.
264
264
265 This is a simple example of how an interactive terminal-based client can use
265 This is a simple example of how an interactive terminal-based client can use
266 this tool::
266 this tool::
267
267
268 isp = InputSplitter()
268 isp = InputSplitter()
269 while isp.push_accepts_more():
269 while isp.push_accepts_more():
270 indent = ' '*isp.indent_spaces
270 indent = ' '*isp.indent_spaces
271 prompt = '>>> ' + indent
271 prompt = '>>> ' + indent
272 line = indent + raw_input(prompt)
272 line = indent + raw_input(prompt)
273 isp.push(line)
273 isp.push(line)
274 print 'Input source was:\n', isp.source_reset(),
274 print 'Input source was:\n', isp.source_reset(),
275 """
275 """
276 # Number of spaces of indentation computed from input that has been pushed
276 # Number of spaces of indentation computed from input that has been pushed
277 # so far. This is the attributes callers should query to get the current
277 # so far. This is the attributes callers should query to get the current
278 # indentation level, in order to provide auto-indent facilities.
278 # indentation level, in order to provide auto-indent facilities.
279 indent_spaces = 0
279 indent_spaces = 0
280 # String, indicating the default input encoding. It is computed by default
280 # String, indicating the default input encoding. It is computed by default
281 # at initialization time via get_input_encoding(), but it can be reset by a
281 # at initialization time via get_input_encoding(), but it can be reset by a
282 # client with specific knowledge of the encoding.
282 # client with specific knowledge of the encoding.
283 encoding = ''
283 encoding = ''
284 # String where the current full source input is stored, properly encoded.
284 # String where the current full source input is stored, properly encoded.
285 # Reading this attribute is the normal way of querying the currently pushed
285 # Reading this attribute is the normal way of querying the currently pushed
286 # source code, that has been properly encoded.
286 # source code, that has been properly encoded.
287 source = ''
287 source = ''
288 # Code object corresponding to the current source. It is automatically
288 # Code object corresponding to the current source. It is automatically
289 # synced to the source, so it can be queried at any time to obtain the code
289 # synced to the source, so it can be queried at any time to obtain the code
290 # object; it will be None if the source doesn't compile to valid Python.
290 # object; it will be None if the source doesn't compile to valid Python.
291 code = None
291 code = None
292 # Input mode
292 # Input mode
293 input_mode = 'line'
293 input_mode = 'line'
294
294
295 # Private attributes
295 # Private attributes
296
296
297 # List with lines of input accumulated so far
297 # List with lines of input accumulated so far
298 _buffer = None
298 _buffer = None
299 # Command compiler
299 # Command compiler
300 _compile = None
300 _compile = None
301 # Mark when input has changed indentation all the way back to flush-left
301 # Mark when input has changed indentation all the way back to flush-left
302 _full_dedent = False
302 _full_dedent = False
303 # Boolean indicating whether the current block is complete
303 # Boolean indicating whether the current block is complete
304 _is_complete = None
304 _is_complete = None
305
305
306 def __init__(self, input_mode=None):
306 def __init__(self, input_mode=None):
307 """Create a new InputSplitter instance.
307 """Create a new InputSplitter instance.
308
308
309 Parameters
309 Parameters
310 ----------
310 ----------
311 input_mode : str
311 input_mode : str
312
312
313 One of ['line', 'cell']; default is 'line'.
313 One of ['line', 'cell']; default is 'line'.
314
314
315 The input_mode parameter controls how new inputs are used when fed via
315 The input_mode parameter controls how new inputs are used when fed via
316 the :meth:`push` method:
316 the :meth:`push` method:
317
317
318 - 'line': meant for line-oriented clients, inputs are appended one at a
318 - 'line': meant for line-oriented clients, inputs are appended one at a
319 time to the internal buffer and the whole buffer is compiled.
319 time to the internal buffer and the whole buffer is compiled.
320
320
321 - 'cell': meant for clients that can edit multi-line 'cells' of text at
321 - 'cell': meant for clients that can edit multi-line 'cells' of text at
322 a time. A cell can contain one or more blocks that can be compile in
322 a time. A cell can contain one or more blocks that can be compile in
323 'single' mode by Python. In this mode, each new input new input
323 'single' mode by Python. In this mode, each new input new input
324 completely replaces all prior inputs. Cell mode is thus equivalent
324 completely replaces all prior inputs. Cell mode is thus equivalent
325 to prepending a full reset() to every push() call.
325 to prepending a full reset() to every push() call.
326 """
326 """
327 self._buffer = []
327 self._buffer = []
328 self._compile = codeop.CommandCompiler()
328 self._compile = codeop.CommandCompiler()
329 self.encoding = get_input_encoding()
329 self.encoding = get_input_encoding()
330 self.input_mode = InputSplitter.input_mode if input_mode is None \
330 self.input_mode = InputSplitter.input_mode if input_mode is None \
331 else input_mode
331 else input_mode
332
332
333 def reset(self):
333 def reset(self):
334 """Reset the input buffer and associated state."""
334 """Reset the input buffer and associated state."""
335 self.indent_spaces = 0
335 self.indent_spaces = 0
336 self._buffer[:] = []
336 self._buffer[:] = []
337 self.source = ''
337 self.source = ''
338 self.code = None
338 self.code = None
339 self._is_complete = False
339 self._is_complete = False
340 self._full_dedent = False
340 self._full_dedent = False
341
341
342 def source_reset(self):
342 def source_reset(self):
343 """Return the input source and perform a full reset.
343 """Return the input source and perform a full reset.
344 """
344 """
345 out = self.source
345 out = self.source
346 self.reset()
346 self.reset()
347 return out
347 return out
348
348
349 def push(self, lines):
349 def push(self, lines):
350 """Push one ore more lines of input.
350 """Push one ore more lines of input.
351
351
352 This stores the given lines and returns a status code indicating
352 This stores the given lines and returns a status code indicating
353 whether the code forms a complete Python block or not.
353 whether the code forms a complete Python block or not.
354
354
355 Any exceptions generated in compilation are swallowed, but if an
355 Any exceptions generated in compilation are swallowed, but if an
356 exception was produced, the method returns True.
356 exception was produced, the method returns True.
357
357
358 Parameters
358 Parameters
359 ----------
359 ----------
360 lines : string
360 lines : string
361 One or more lines of Python input.
361 One or more lines of Python input.
362
362
363 Returns
363 Returns
364 -------
364 -------
365 is_complete : boolean
365 is_complete : boolean
366 True if the current input source (the result of the current input
366 True if the current input source (the result of the current input
367 plus prior inputs) forms a complete Python execution block. Note that
367 plus prior inputs) forms a complete Python execution block. Note that
368 this value is also stored as a private attribute (_is_complete), so it
368 this value is also stored as a private attribute (_is_complete), so it
369 can be queried at any time.
369 can be queried at any time.
370 """
370 """
371 if self.input_mode == 'cell':
371 if self.input_mode == 'cell':
372 self.reset()
372 self.reset()
373
373
374 # If the source code has leading blanks, add 'if 1:\n' to it
374 # If the source code has leading blanks, add 'if 1:\n' to it
375 # this allows execution of indented pasted code. It is tempting
375 # this allows execution of indented pasted code. It is tempting
376 # to add '\n' at the end of source to run commands like ' a=1'
376 # to add '\n' at the end of source to run commands like ' a=1'
377 # directly, but this fails for more complicated scenarios
377 # directly, but this fails for more complicated scenarios
378
378
379 if not self._buffer and lines[:1] in [' ', '\t'] and \
379 if not self._buffer and lines[:1] in [' ', '\t'] and \
380 not comment_line_re.match(lines):
380 not comment_line_re.match(lines):
381 lines = 'if 1:\n%s' % lines
381 lines = 'if 1:\n%s' % lines
382
382
383 self._store(lines)
383 self._store(lines)
384 source = self.source
384 source = self.source
385
385
386 # Before calling _compile(), reset the code object to None so that if an
386 # Before calling _compile(), reset the code object to None so that if an
387 # exception is raised in compilation, we don't mislead by having
387 # exception is raised in compilation, we don't mislead by having
388 # inconsistent code/source attributes.
388 # inconsistent code/source attributes.
389 self.code, self._is_complete = None, None
389 self.code, self._is_complete = None, None
390
390
391 # Honor termination lines properly
391 # Honor termination lines properly
392 if source.rstrip().endswith('\\'):
392 if source.rstrip().endswith('\\'):
393 return False
393 return False
394
394
395 self._update_indent(lines)
395 self._update_indent(lines)
396 try:
396 try:
397 self.code = self._compile(source)
397 self.code = self._compile(source)
398 # Invalid syntax can produce any of a number of different errors from
398 # Invalid syntax can produce any of a number of different errors from
399 # inside the compiler, so we have to catch them all. Syntax errors
399 # inside the compiler, so we have to catch them all. Syntax errors
400 # immediately produce a 'ready' block, so the invalid Python can be
400 # immediately produce a 'ready' block, so the invalid Python can be
401 # sent to the kernel for evaluation with possible ipython
401 # sent to the kernel for evaluation with possible ipython
402 # special-syntax conversion.
402 # special-syntax conversion.
403 except (SyntaxError, OverflowError, ValueError, TypeError,
403 except (SyntaxError, OverflowError, ValueError, TypeError,
404 MemoryError):
404 MemoryError):
405 self._is_complete = True
405 self._is_complete = True
406 else:
406 else:
407 # Compilation didn't produce any exceptions (though it may not have
407 # Compilation didn't produce any exceptions (though it may not have
408 # given a complete code object)
408 # given a complete code object)
409 self._is_complete = self.code is not None
409 self._is_complete = self.code is not None
410
410
411 return self._is_complete
411 return self._is_complete
412
412
413 def push_accepts_more(self):
413 def push_accepts_more(self):
414 """Return whether a block of interactive input can accept more input.
414 """Return whether a block of interactive input can accept more input.
415
415
416 This method is meant to be used by line-oriented frontends, who need to
416 This method is meant to be used by line-oriented frontends, who need to
417 guess whether a block is complete or not based solely on prior and
417 guess whether a block is complete or not based solely on prior and
418 current input lines. The InputSplitter considers it has a complete
418 current input lines. The InputSplitter considers it has a complete
419 interactive block and will not accept more input only when either a
419 interactive block and will not accept more input only when either a
420 SyntaxError is raised, or *all* of the following are true:
420 SyntaxError is raised, or *all* of the following are true:
421
421
422 1. The input compiles to a complete statement.
422 1. The input compiles to a complete statement.
423
423
424 2. The indentation level is flush-left (because if we are indented,
424 2. The indentation level is flush-left (because if we are indented,
425 like inside a function definition or for loop, we need to keep
425 like inside a function definition or for loop, we need to keep
426 reading new input).
426 reading new input).
427
427
428 3. There is one extra line consisting only of whitespace.
428 3. There is one extra line consisting only of whitespace.
429
429
430 Because of condition #3, this method should be used only by
430 Because of condition #3, this method should be used only by
431 *line-oriented* frontends, since it means that intermediate blank lines
431 *line-oriented* frontends, since it means that intermediate blank lines
432 are not allowed in function definitions (or any other indented block).
432 are not allowed in function definitions (or any other indented block).
433
433
434 Block-oriented frontends that have a separate keyboard event to
434 Block-oriented frontends that have a separate keyboard event to
435 indicate execution should use the :meth:`split_blocks` method instead.
435 indicate execution should use the :meth:`split_blocks` method instead.
436
436
437 If the current input produces a syntax error, this method immediately
437 If the current input produces a syntax error, this method immediately
438 returns False but does *not* raise the syntax error exception, as
438 returns False but does *not* raise the syntax error exception, as
439 typically clients will want to send invalid syntax to an execution
439 typically clients will want to send invalid syntax to an execution
440 backend which might convert the invalid syntax into valid Python via
440 backend which might convert the invalid syntax into valid Python via
441 one of the dynamic IPython mechanisms.
441 one of the dynamic IPython mechanisms.
442 """
442 """
443
443
444 # With incomplete input, unconditionally accept more
444 # With incomplete input, unconditionally accept more
445 if not self._is_complete:
445 if not self._is_complete:
446 return True
446 return True
447
447
448 # If we already have complete input and we're flush left, the answer
448 # If we already have complete input and we're flush left, the answer
449 # depends. In line mode, we're done. But in cell mode, we need to
449 # depends. In line mode, we're done. But in cell mode, we need to
450 # check how many blocks the input so far compiles into, because if
450 # check how many blocks the input so far compiles into, because if
451 # there's already more than one full independent block of input, then
451 # there's already more than one full independent block of input, then
452 # the client has entered full 'cell' mode and is feeding lines that
452 # the client has entered full 'cell' mode and is feeding lines that
453 # each is complete. In this case we should then keep accepting.
453 # each is complete. In this case we should then keep accepting.
454 # The Qt terminal-like console does precisely this, to provide the
454 # The Qt terminal-like console does precisely this, to provide the
455 # convenience of terminal-like input of single expressions, but
455 # convenience of terminal-like input of single expressions, but
456 # allowing the user (with a separate keystroke) to switch to 'cell'
456 # allowing the user (with a separate keystroke) to switch to 'cell'
457 # mode and type multiple expressions in one shot.
457 # mode and type multiple expressions in one shot.
458 if self.indent_spaces==0:
458 if self.indent_spaces==0:
459 if self.input_mode=='line':
459 if self.input_mode=='line':
460 return False
460 return False
461 else:
461 else:
462 nblocks = len(split_blocks(''.join(self._buffer)))
462 nblocks = len(split_blocks(''.join(self._buffer)))
463 if nblocks==1:
463 if nblocks==1:
464 return False
464 return False
465
465
466 # When input is complete, then termination is marked by an extra blank
466 # When input is complete, then termination is marked by an extra blank
467 # line at the end.
467 # line at the end.
468 last_line = self.source.splitlines()[-1]
468 last_line = self.source.splitlines()[-1]
469 return bool(last_line and not last_line.isspace())
469 return bool(last_line and not last_line.isspace())
470
470
471 def split_blocks(self, lines):
471 def split_blocks(self, lines):
472 """Split a multiline string into multiple input blocks.
472 """Split a multiline string into multiple input blocks.
473
473
474 Note: this method starts by performing a full reset().
474 Note: this method starts by performing a full reset().
475
475
476 Parameters
476 Parameters
477 ----------
477 ----------
478 lines : str
478 lines : str
479 A possibly multiline string.
479 A possibly multiline string.
480
480
481 Returns
481 Returns
482 -------
482 -------
483 blocks : list
483 blocks : list
484 A list of strings, each possibly multiline. Each string corresponds
484 A list of strings, each possibly multiline. Each string corresponds
485 to a single block that can be compiled in 'single' mode (unless it
485 to a single block that can be compiled in 'single' mode (unless it
486 has a syntax error)."""
486 has a syntax error)."""
487
487
488 # This code is fairly delicate. If you make any changes here, make
488 # This code is fairly delicate. If you make any changes here, make
489 # absolutely sure that you do run the full test suite and ALL tests
489 # absolutely sure that you do run the full test suite and ALL tests
490 # pass.
490 # pass.
491
491
492 self.reset()
492 self.reset()
493 blocks = []
493 blocks = []
494
494
495 # Reversed copy so we can use pop() efficiently and consume the input
495 # Reversed copy so we can use pop() efficiently and consume the input
496 # as a stack
496 # as a stack
497 lines = lines.splitlines()[::-1]
497 lines = lines.splitlines()[::-1]
498 # Outer loop over all input
498 # Outer loop over all input
499 while lines:
499 while lines:
500 #print 'Current lines:', lines # dbg
500 #print 'Current lines:', lines # dbg
501 # Inner loop to build each block
501 # Inner loop to build each block
502 while True:
502 while True:
503 # Safety exit from inner loop
503 # Safety exit from inner loop
504 if not lines:
504 if not lines:
505 break
505 break
506 # Grab next line but don't push it yet
506 # Grab next line but don't push it yet
507 next_line = lines.pop()
507 next_line = lines.pop()
508 # Blank/empty lines are pushed as-is
508 # Blank/empty lines are pushed as-is
509 if not next_line or next_line.isspace():
509 if not next_line or next_line.isspace():
510 self.push(next_line)
510 self.push(next_line)
511 continue
511 continue
512
512
513 # Check indentation changes caused by the *next* line
513 # Check indentation changes caused by the *next* line
514 indent_spaces, _full_dedent = self._find_indent(next_line)
514 indent_spaces, _full_dedent = self._find_indent(next_line)
515
515
516 # If the next line causes a dedent, it can be for two differnt
516 # If the next line causes a dedent, it can be for two differnt
517 # reasons: either an explicit de-dent by the user or a
517 # reasons: either an explicit de-dent by the user or a
518 # return/raise/pass statement. These MUST be handled
518 # return/raise/pass statement. These MUST be handled
519 # separately:
519 # separately:
520 #
520 #
521 # 1. the first case is only detected when the actual explicit
521 # 1. the first case is only detected when the actual explicit
522 # dedent happens, and that would be the *first* line of a *new*
522 # dedent happens, and that would be the *first* line of a *new*
523 # block. Thus, we must put the line back into the input buffer
523 # block. Thus, we must put the line back into the input buffer
524 # so that it starts a new block on the next pass.
524 # so that it starts a new block on the next pass.
525 #
525 #
526 # 2. the second case is detected in the line before the actual
526 # 2. the second case is detected in the line before the actual
527 # dedent happens, so , we consume the line and we can break out
527 # dedent happens, so , we consume the line and we can break out
528 # to start a new block.
528 # to start a new block.
529
529
530 # Case 1, explicit dedent causes a break.
530 # Case 1, explicit dedent causes a break.
531 # Note: check that we weren't on the very last line, else we'll
531 # Note: check that we weren't on the very last line, else we'll
532 # enter an infinite loop adding/removing the last line.
532 # enter an infinite loop adding/removing the last line.
533 if _full_dedent and lines and not next_line.startswith(' '):
533 if _full_dedent and lines and not next_line.startswith(' '):
534 lines.append(next_line)
534 lines.append(next_line)
535 break
535 break
536
536
537 # Otherwise any line is pushed
537 # Otherwise any line is pushed
538 self.push(next_line)
538 self.push(next_line)
539
539
540 # Case 2, full dedent with full block ready:
540 # Case 2, full dedent with full block ready:
541 if _full_dedent or \
541 if _full_dedent or \
542 self.indent_spaces==0 and not self.push_accepts_more():
542 self.indent_spaces==0 and not self.push_accepts_more():
543 break
543 break
544 # Form the new block with the current source input
544 # Form the new block with the current source input
545 blocks.append(self.source_reset())
545 blocks.append(self.source_reset())
546
546
547 #return blocks
547 #return blocks
548 # HACK!!! Now that our input is in blocks but guaranteed to be pure
548 # HACK!!! Now that our input is in blocks but guaranteed to be pure
549 # python syntax, feed it back a second time through the AST-based
549 # python syntax, feed it back a second time through the AST-based
550 # splitter, which is more accurate than ours.
550 # splitter, which is more accurate than ours.
551 return split_blocks(''.join(blocks))
551 return split_blocks(''.join(blocks))
552
552
553 #------------------------------------------------------------------------
553 #------------------------------------------------------------------------
554 # Private interface
554 # Private interface
555 #------------------------------------------------------------------------
555 #------------------------------------------------------------------------
556
556
557 def _find_indent(self, line):
557 def _find_indent(self, line):
558 """Compute the new indentation level for a single line.
558 """Compute the new indentation level for a single line.
559
559
560 Parameters
560 Parameters
561 ----------
561 ----------
562 line : str
562 line : str
563 A single new line of non-whitespace, non-comment Python input.
563 A single new line of non-whitespace, non-comment Python input.
564
564
565 Returns
565 Returns
566 -------
566 -------
567 indent_spaces : int
567 indent_spaces : int
568 New value for the indent level (it may be equal to self.indent_spaces
568 New value for the indent level (it may be equal to self.indent_spaces
569 if indentation doesn't change.
569 if indentation doesn't change.
570
570
571 full_dedent : boolean
571 full_dedent : boolean
572 Whether the new line causes a full flush-left dedent.
572 Whether the new line causes a full flush-left dedent.
573 """
573 """
574 indent_spaces = self.indent_spaces
574 indent_spaces = self.indent_spaces
575 full_dedent = self._full_dedent
575 full_dedent = self._full_dedent
576
576
577 inisp = num_ini_spaces(line)
577 inisp = num_ini_spaces(line)
578 if inisp < indent_spaces:
578 if inisp < indent_spaces:
579 indent_spaces = inisp
579 indent_spaces = inisp
580 if indent_spaces <= 0:
580 if indent_spaces <= 0:
581 #print 'Full dedent in text',self.source # dbg
581 #print 'Full dedent in text',self.source # dbg
582 full_dedent = True
582 full_dedent = True
583
583
584 if line[-1] == ':':
584 if line[-1] == ':':
585 indent_spaces += 4
585 indent_spaces += 4
586 elif dedent_re.match(line):
586 elif dedent_re.match(line):
587 indent_spaces -= 4
587 indent_spaces -= 4
588 if indent_spaces <= 0:
588 if indent_spaces <= 0:
589 full_dedent = True
589 full_dedent = True
590
590
591 # Safety
591 # Safety
592 if indent_spaces < 0:
592 if indent_spaces < 0:
593 indent_spaces = 0
593 indent_spaces = 0
594 #print 'safety' # dbg
594 #print 'safety' # dbg
595
595
596 return indent_spaces, full_dedent
596 return indent_spaces, full_dedent
597
597
598 def _update_indent(self, lines):
598 def _update_indent(self, lines):
599 for line in remove_comments(lines).splitlines():
599 for line in remove_comments(lines).splitlines():
600 if line and not line.isspace():
600 if line and not line.isspace():
601 self.indent_spaces, self._full_dedent = self._find_indent(line)
601 self.indent_spaces, self._full_dedent = self._find_indent(line)
602
602
603 def _store(self, lines):
603 def _store(self, lines):
604 """Store one or more lines of input.
604 """Store one or more lines of input.
605
605
606 If input lines are not newline-terminated, a newline is automatically
606 If input lines are not newline-terminated, a newline is automatically
607 appended."""
607 appended."""
608
608
609 if lines.endswith('\n'):
609 if lines.endswith('\n'):
610 self._buffer.append(lines)
610 self._buffer.append(lines)
611 else:
611 else:
612 self._buffer.append(lines+'\n')
612 self._buffer.append(lines+'\n')
613 self._set_source()
613 self._set_source()
614
614
615 def _set_source(self):
615 def _set_source(self):
616 self.source = ''.join(self._buffer).encode(self.encoding)
616 self.source = ''.join(self._buffer).encode(self.encoding)
617
617
618
618
619 #-----------------------------------------------------------------------------
619 #-----------------------------------------------------------------------------
620 # Functions and classes for IPython-specific syntactic support
620 # Functions and classes for IPython-specific syntactic support
621 #-----------------------------------------------------------------------------
621 #-----------------------------------------------------------------------------
622
622
623 # RegExp for splitting line contents into pre-char//first word-method//rest.
623 # RegExp for splitting line contents into pre-char//first word-method//rest.
624 # For clarity, each group in on one line.
624 # For clarity, each group in on one line.
625
625
626 line_split = re.compile("""
626 line_split = re.compile("""
627 ^(\s*) # any leading space
627 ^(\s*) # any leading space
628 ([,;/%]|!!?|\?\??) # escape character or characters
628 ([,;/%]|!!?|\?\??) # escape character or characters
629 \s*(%?[\w\.\*]*) # function/method, possibly with leading %
629 \s*(%?[\w\.\*]*) # function/method, possibly with leading %
630 # to correctly treat things like '?%magic'
630 # to correctly treat things like '?%magic'
631 (\s+.*$|$) # rest of line
631 (\s+.*$|$) # rest of line
632 """, re.VERBOSE)
632 """, re.VERBOSE)
633
633
634
634
635 def split_user_input(line):
635 def split_user_input(line):
636 """Split user input into early whitespace, esc-char, function part and rest.
636 """Split user input into early whitespace, esc-char, function part and rest.
637
637
638 This is currently handles lines with '=' in them in a very inconsistent
638 This is currently handles lines with '=' in them in a very inconsistent
639 manner.
639 manner.
640
640
641 Examples
641 Examples
642 ========
642 ========
643 >>> split_user_input('x=1')
643 >>> split_user_input('x=1')
644 ('', '', 'x=1', '')
644 ('', '', 'x=1', '')
645 >>> split_user_input('?')
645 >>> split_user_input('?')
646 ('', '?', '', '')
646 ('', '?', '', '')
647 >>> split_user_input('??')
647 >>> split_user_input('??')
648 ('', '??', '', '')
648 ('', '??', '', '')
649 >>> split_user_input(' ?')
649 >>> split_user_input(' ?')
650 (' ', '?', '', '')
650 (' ', '?', '', '')
651 >>> split_user_input(' ??')
651 >>> split_user_input(' ??')
652 (' ', '??', '', '')
652 (' ', '??', '', '')
653 >>> split_user_input('??x')
653 >>> split_user_input('??x')
654 ('', '??', 'x', '')
654 ('', '??', 'x', '')
655 >>> split_user_input('?x=1')
655 >>> split_user_input('?x=1')
656 ('', '', '?x=1', '')
656 ('', '', '?x=1', '')
657 >>> split_user_input('!ls')
657 >>> split_user_input('!ls')
658 ('', '!', 'ls', '')
658 ('', '!', 'ls', '')
659 >>> split_user_input(' !ls')
659 >>> split_user_input(' !ls')
660 (' ', '!', 'ls', '')
660 (' ', '!', 'ls', '')
661 >>> split_user_input('!!ls')
661 >>> split_user_input('!!ls')
662 ('', '!!', 'ls', '')
662 ('', '!!', 'ls', '')
663 >>> split_user_input(' !!ls')
663 >>> split_user_input(' !!ls')
664 (' ', '!!', 'ls', '')
664 (' ', '!!', 'ls', '')
665 >>> split_user_input(',ls')
665 >>> split_user_input(',ls')
666 ('', ',', 'ls', '')
666 ('', ',', 'ls', '')
667 >>> split_user_input(';ls')
667 >>> split_user_input(';ls')
668 ('', ';', 'ls', '')
668 ('', ';', 'ls', '')
669 >>> split_user_input(' ;ls')
669 >>> split_user_input(' ;ls')
670 (' ', ';', 'ls', '')
670 (' ', ';', 'ls', '')
671 >>> split_user_input('f.g(x)')
671 >>> split_user_input('f.g(x)')
672 ('', '', 'f.g(x)', '')
672 ('', '', 'f.g(x)', '')
673 >>> split_user_input('f.g (x)')
673 >>> split_user_input('f.g (x)')
674 ('', '', 'f.g', '(x)')
674 ('', '', 'f.g', '(x)')
675 >>> split_user_input('?%hist')
675 >>> split_user_input('?%hist')
676 ('', '?', '%hist', '')
676 ('', '?', '%hist', '')
677 >>> split_user_input('?x*')
677 >>> split_user_input('?x*')
678 ('', '?', 'x*', '')
678 ('', '?', 'x*', '')
679 """
679 """
680 match = line_split.match(line)
680 match = line_split.match(line)
681 if match:
681 if match:
682 lspace, esc, fpart, rest = match.groups()
682 lspace, esc, fpart, rest = match.groups()
683 else:
683 else:
684 # print "match failed for line '%s'" % line
684 # print "match failed for line '%s'" % line
685 try:
685 try:
686 fpart, rest = line.split(None, 1)
686 fpart, rest = line.split(None, 1)
687 except ValueError:
687 except ValueError:
688 # print "split failed for line '%s'" % line
688 # print "split failed for line '%s'" % line
689 fpart, rest = line,''
689 fpart, rest = line,''
690 lspace = re.match('^(\s*)(.*)', line).groups()[0]
690 lspace = re.match('^(\s*)(.*)', line).groups()[0]
691 esc = ''
691 esc = ''
692
692
693 # fpart has to be a valid python identifier, so it better be only pure
693 # fpart has to be a valid python identifier, so it better be only pure
694 # ascii, no unicode:
694 # ascii, no unicode:
695 try:
695 try:
696 fpart = fpart.encode('ascii')
696 fpart = fpart.encode('ascii')
697 except UnicodeEncodeError:
697 except UnicodeEncodeError:
698 lspace = unicode(lspace)
698 lspace = unicode(lspace)
699 rest = fpart + u' ' + rest
699 rest = fpart + u' ' + rest
700 fpart = u''
700 fpart = u''
701
701
702 #print 'line:<%s>' % line # dbg
702 #print 'line:<%s>' % line # dbg
703 #print 'esc <%s> fpart <%s> rest <%s>' % (esc,fpart.strip(),rest) # dbg
703 #print 'esc <%s> fpart <%s> rest <%s>' % (esc,fpart.strip(),rest) # dbg
704 return lspace, esc, fpart.strip(), rest.lstrip()
704 return lspace, esc, fpart.strip(), rest.lstrip()
705
705
706
706
707 # The escaped translators ALL receive a line where their own escape has been
707 # The escaped translators ALL receive a line where their own escape has been
708 # stripped. Only '?' is valid at the end of the line, all others can only be
708 # stripped. Only '?' is valid at the end of the line, all others can only be
709 # placed at the start.
709 # placed at the start.
710
710
711 class LineInfo(object):
711 class LineInfo(object):
712 """A single line of input and associated info.
712 """A single line of input and associated info.
713
713
714 This is a utility class that mostly wraps the output of
714 This is a utility class that mostly wraps the output of
715 :func:`split_user_input` into a convenient object to be passed around
715 :func:`split_user_input` into a convenient object to be passed around
716 during input transformations.
716 during input transformations.
717
717
718 Includes the following as properties:
718 Includes the following as properties:
719
719
720 line
720 line
721 The original, raw line
721 The original, raw line
722
722
723 lspace
723 lspace
724 Any early whitespace before actual text starts.
724 Any early whitespace before actual text starts.
725
725
726 esc
726 esc
727 The initial esc character (or characters, for double-char escapes like
727 The initial esc character (or characters, for double-char escapes like
728 '??' or '!!').
728 '??' or '!!').
729
729
730 fpart
730 fpart
731 The 'function part', which is basically the maximal initial sequence
731 The 'function part', which is basically the maximal initial sequence
732 of valid python identifiers and the '.' character. This is what is
732 of valid python identifiers and the '.' character. This is what is
733 checked for alias and magic transformations, used for auto-calling,
733 checked for alias and magic transformations, used for auto-calling,
734 etc.
734 etc.
735
735
736 rest
736 rest
737 Everything else on the line.
737 Everything else on the line.
738 """
738 """
739 def __init__(self, line):
739 def __init__(self, line):
740 self.line = line
740 self.line = line
741 self.lspace, self.esc, self.fpart, self.rest = \
741 self.lspace, self.esc, self.fpart, self.rest = \
742 split_user_input(line)
742 split_user_input(line)
743
743
744 def __str__(self):
744 def __str__(self):
745 return "LineInfo [%s|%s|%s|%s]" % (self.lspace, self.esc,
745 return "LineInfo [%s|%s|%s|%s]" % (self.lspace, self.esc,
746 self.fpart, self.rest)
746 self.fpart, self.rest)
747
747
748
748
749 # Transformations of the special syntaxes that don't rely on an explicit escape
749 # Transformations of the special syntaxes that don't rely on an explicit escape
750 # character but instead on patterns on the input line
750 # character but instead on patterns on the input line
751
751
752 # The core transformations are implemented as standalone functions that can be
752 # The core transformations are implemented as standalone functions that can be
753 # tested and validated in isolation. Each of these uses a regexp, we
753 # tested and validated in isolation. Each of these uses a regexp, we
754 # pre-compile these and keep them close to each function definition for clarity
754 # pre-compile these and keep them close to each function definition for clarity
755
755
756 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
756 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
757 r'\s*=\s*!\s*(?P<cmd>.*)')
757 r'\s*=\s*!\s*(?P<cmd>.*)')
758
758
759 def transform_assign_system(line):
759 def transform_assign_system(line):
760 """Handle the `files = !ls` syntax."""
760 """Handle the `files = !ls` syntax."""
761 m = _assign_system_re.match(line)
761 m = _assign_system_re.match(line)
762 if m is not None:
762 if m is not None:
763 cmd = m.group('cmd')
763 cmd = m.group('cmd')
764 lhs = m.group('lhs')
764 lhs = m.group('lhs')
765 expr = make_quoted_expr(cmd)
765 expr = make_quoted_expr(cmd)
766 new_line = '%s = get_ipython().getoutput(%s)' % (lhs, expr)
766 new_line = '%s = get_ipython().getoutput(%s)' % (lhs, expr)
767 return new_line
767 return new_line
768 return line
768 return line
769
769
770
770
771 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
771 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
772 r'\s*=\s*%\s*(?P<cmd>.*)')
772 r'\s*=\s*%\s*(?P<cmd>.*)')
773
773
774 def transform_assign_magic(line):
774 def transform_assign_magic(line):
775 """Handle the `a = %who` syntax."""
775 """Handle the `a = %who` syntax."""
776 m = _assign_magic_re.match(line)
776 m = _assign_magic_re.match(line)
777 if m is not None:
777 if m is not None:
778 cmd = m.group('cmd')
778 cmd = m.group('cmd')
779 lhs = m.group('lhs')
779 lhs = m.group('lhs')
780 expr = make_quoted_expr(cmd)
780 expr = make_quoted_expr(cmd)
781 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
781 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
782 return new_line
782 return new_line
783 return line
783 return line
784
784
785
785
786 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
786 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
787
787
788 def transform_classic_prompt(line):
788 def transform_classic_prompt(line):
789 """Handle inputs that start with '>>> ' syntax."""
789 """Handle inputs that start with '>>> ' syntax."""
790
790
791 if not line or line.isspace():
791 if not line or line.isspace():
792 return line
792 return line
793 m = _classic_prompt_re.match(line)
793 m = _classic_prompt_re.match(line)
794 if m:
794 if m:
795 return line[len(m.group(0)):]
795 return line[len(m.group(0)):]
796 else:
796 else:
797 return line
797 return line
798
798
799
799
800 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
800 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
801
801
802 def transform_ipy_prompt(line):
802 def transform_ipy_prompt(line):
803 """Handle inputs that start classic IPython prompt syntax."""
803 """Handle inputs that start classic IPython prompt syntax."""
804
804
805 if not line or line.isspace():
805 if not line or line.isspace():
806 return line
806 return line
807 #print 'LINE: %r' % line # dbg
807 #print 'LINE: %r' % line # dbg
808 m = _ipy_prompt_re.match(line)
808 m = _ipy_prompt_re.match(line)
809 if m:
809 if m:
810 #print 'MATCH! %r -> %r' % (line, line[len(m.group(0)):]) # dbg
810 #print 'MATCH! %r -> %r' % (line, line[len(m.group(0)):]) # dbg
811 return line[len(m.group(0)):]
811 return line[len(m.group(0)):]
812 else:
812 else:
813 return line
813 return line
814
814
815
815
816 class EscapedTransformer(object):
816 class EscapedTransformer(object):
817 """Class to transform lines that are explicitly escaped out."""
817 """Class to transform lines that are explicitly escaped out."""
818
818
819 def __init__(self):
819 def __init__(self):
820 tr = { ESC_SHELL : self._tr_system,
820 tr = { ESC_SHELL : self._tr_system,
821 ESC_SH_CAP : self._tr_system2,
821 ESC_SH_CAP : self._tr_system2,
822 ESC_HELP : self._tr_help,
822 ESC_HELP : self._tr_help,
823 ESC_HELP2 : self._tr_help,
823 ESC_HELP2 : self._tr_help,
824 ESC_MAGIC : self._tr_magic,
824 ESC_MAGIC : self._tr_magic,
825 ESC_QUOTE : self._tr_quote,
825 ESC_QUOTE : self._tr_quote,
826 ESC_QUOTE2 : self._tr_quote2,
826 ESC_QUOTE2 : self._tr_quote2,
827 ESC_PAREN : self._tr_paren }
827 ESC_PAREN : self._tr_paren }
828 self.tr = tr
828 self.tr = tr
829
829
830 # Support for syntax transformations that use explicit escapes typed by the
830 # Support for syntax transformations that use explicit escapes typed by the
831 # user at the beginning of a line
831 # user at the beginning of a line
832 @staticmethod
832 @staticmethod
833 def _tr_system(line_info):
833 def _tr_system(line_info):
834 "Translate lines escaped with: !"
834 "Translate lines escaped with: !"
835 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
835 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
836 return '%sget_ipython().system(%s)' % (line_info.lspace,
836 return '%sget_ipython().system(%s)' % (line_info.lspace,
837 make_quoted_expr(cmd))
837 make_quoted_expr(cmd))
838
838
839 @staticmethod
839 @staticmethod
840 def _tr_system2(line_info):
840 def _tr_system2(line_info):
841 "Translate lines escaped with: !!"
841 "Translate lines escaped with: !!"
842 cmd = line_info.line.lstrip()[2:]
842 cmd = line_info.line.lstrip()[2:]
843 return '%sget_ipython().getoutput(%s)' % (line_info.lspace,
843 return '%sget_ipython().getoutput(%s)' % (line_info.lspace,
844 make_quoted_expr(cmd))
844 make_quoted_expr(cmd))
845
845
846 @staticmethod
846 @staticmethod
847 def _tr_help(line_info):
847 def _tr_help(line_info):
848 "Translate lines escaped with: ?/??"
848 "Translate lines escaped with: ?/??"
849 # A naked help line should just fire the intro help screen
849 # A naked help line should just fire the intro help screen
850 if not line_info.line[1:]:
850 if not line_info.line[1:]:
851 return 'get_ipython().show_usage()'
851 return 'get_ipython().show_usage()'
852
852
853 # There may be one or two '?' at the end, move them to the front so that
853 # There may be one or two '?' at the end, move them to the front so that
854 # the rest of the logic can assume escapes are at the start
854 # the rest of the logic can assume escapes are at the start
855 l_ori = line_info
855 l_ori = line_info
856 line = line_info.line
856 line = line_info.line
857 if line.endswith('?'):
857 if line.endswith('?'):
858 line = line[-1] + line[:-1]
858 line = line[-1] + line[:-1]
859 if line.endswith('?'):
859 if line.endswith('?'):
860 line = line[-1] + line[:-1]
860 line = line[-1] + line[:-1]
861 line_info = LineInfo(line)
861 line_info = LineInfo(line)
862
862
863 # From here on, simply choose which level of detail to get, and
863 # From here on, simply choose which level of detail to get, and
864 # special-case the psearch syntax
864 # special-case the psearch syntax
865 pinfo = 'pinfo' # default
865 if '*' in line_info.line:
866 if '*' in line_info.line:
866 pinfo = 'psearch'
867 pinfo = 'psearch'
867 elif line_info.esc == '?':
868 pinfo = 'pinfo'
869 elif line_info.esc == '??':
868 elif line_info.esc == '??':
870 pinfo = 'pinfo2'
869 pinfo = 'pinfo2'
871
870
872 tpl = '%sget_ipython().magic("%s %s")'
871 tpl = '%sget_ipython().magic("%s %s")'
873 return tpl % (line_info.lspace, pinfo,
872 return tpl % (line_info.lspace, pinfo,
874 ' '.join([line_info.fpart, line_info.rest]).strip())
873 ' '.join([line_info.fpart, line_info.rest]).strip())
875
874
876 @staticmethod
875 @staticmethod
877 def _tr_magic(line_info):
876 def _tr_magic(line_info):
878 "Translate lines escaped with: %"
877 "Translate lines escaped with: %"
879 tpl = '%sget_ipython().magic(%s)'
878 tpl = '%sget_ipython().magic(%s)'
880 cmd = make_quoted_expr(' '.join([line_info.fpart,
879 cmd = make_quoted_expr(' '.join([line_info.fpart,
881 line_info.rest]).strip())
880 line_info.rest]).strip())
882 return tpl % (line_info.lspace, cmd)
881 return tpl % (line_info.lspace, cmd)
883
882
884 @staticmethod
883 @staticmethod
885 def _tr_quote(line_info):
884 def _tr_quote(line_info):
886 "Translate lines escaped with: ,"
885 "Translate lines escaped with: ,"
887 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
886 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
888 '", "'.join(line_info.rest.split()) )
887 '", "'.join(line_info.rest.split()) )
889
888
890 @staticmethod
889 @staticmethod
891 def _tr_quote2(line_info):
890 def _tr_quote2(line_info):
892 "Translate lines escaped with: ;"
891 "Translate lines escaped with: ;"
893 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
892 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
894 line_info.rest)
893 line_info.rest)
895
894
896 @staticmethod
895 @staticmethod
897 def _tr_paren(line_info):
896 def _tr_paren(line_info):
898 "Translate lines escaped with: /"
897 "Translate lines escaped with: /"
899 return '%s%s(%s)' % (line_info.lspace, line_info.fpart,
898 return '%s%s(%s)' % (line_info.lspace, line_info.fpart,
900 ", ".join(line_info.rest.split()))
899 ", ".join(line_info.rest.split()))
901
900
902 def __call__(self, line):
901 def __call__(self, line):
903 """Class to transform lines that are explicitly escaped out.
902 """Class to transform lines that are explicitly escaped out.
904
903
905 This calls the above _tr_* static methods for the actual line
904 This calls the above _tr_* static methods for the actual line
906 translations."""
905 translations."""
907
906
908 # Empty lines just get returned unmodified
907 # Empty lines just get returned unmodified
909 if not line or line.isspace():
908 if not line or line.isspace():
910 return line
909 return line
911
910
912 # Get line endpoints, where the escapes can be
911 # Get line endpoints, where the escapes can be
913 line_info = LineInfo(line)
912 line_info = LineInfo(line)
914
913
915 # If the escape is not at the start, only '?' needs to be special-cased.
914 # If the escape is not at the start, only '?' needs to be special-cased.
916 # All other escapes are only valid at the start
915 # All other escapes are only valid at the start
917 if not line_info.esc in self.tr:
916 if not line_info.esc in self.tr:
918 if line.endswith(ESC_HELP):
917 if line.endswith(ESC_HELP):
919 return self._tr_help(line_info)
918 return self._tr_help(line_info)
920 else:
919 else:
921 # If we don't recognize the escape, don't modify the line
920 # If we don't recognize the escape, don't modify the line
922 return line
921 return line
923
922
924 return self.tr[line_info.esc](line_info)
923 return self.tr[line_info.esc](line_info)
925
924
926
925
927 # A function-looking object to be used by the rest of the code. The purpose of
926 # A function-looking object to be used by the rest of the code. The purpose of
928 # the class in this case is to organize related functionality, more than to
927 # the class in this case is to organize related functionality, more than to
929 # manage state.
928 # manage state.
930 transform_escaped = EscapedTransformer()
929 transform_escaped = EscapedTransformer()
931
930
932
931
933 class IPythonInputSplitter(InputSplitter):
932 class IPythonInputSplitter(InputSplitter):
934 """An input splitter that recognizes all of IPython's special syntax."""
933 """An input splitter that recognizes all of IPython's special syntax."""
935
934
936 def push(self, lines):
935 def push(self, lines):
937 """Push one or more lines of IPython input.
936 """Push one or more lines of IPython input.
938 """
937 """
939 if not lines:
938 if not lines:
940 return super(IPythonInputSplitter, self).push(lines)
939 return super(IPythonInputSplitter, self).push(lines)
941
940
942 lines_list = lines.splitlines()
941 lines_list = lines.splitlines()
943
942
944 transforms = [transform_escaped, transform_assign_system,
943 transforms = [transform_escaped, transform_assign_system,
945 transform_assign_magic, transform_ipy_prompt,
944 transform_assign_magic, transform_ipy_prompt,
946 transform_classic_prompt]
945 transform_classic_prompt]
947
946
948 # Transform logic
947 # Transform logic
949 #
948 #
950 # We only apply the line transformers to the input if we have either no
949 # We only apply the line transformers to the input if we have either no
951 # input yet, or complete input, or if the last line of the buffer ends
950 # input yet, or complete input, or if the last line of the buffer ends
952 # with ':' (opening an indented block). This prevents the accidental
951 # with ':' (opening an indented block). This prevents the accidental
953 # transformation of escapes inside multiline expressions like
952 # transformation of escapes inside multiline expressions like
954 # triple-quoted strings or parenthesized expressions.
953 # triple-quoted strings or parenthesized expressions.
955 #
954 #
956 # The last heuristic, while ugly, ensures that the first line of an
955 # The last heuristic, while ugly, ensures that the first line of an
957 # indented block is correctly transformed.
956 # indented block is correctly transformed.
958 #
957 #
959 # FIXME: try to find a cleaner approach for this last bit.
958 # FIXME: try to find a cleaner approach for this last bit.
960
959
961 # If we were in 'block' mode, since we're going to pump the parent
960 # If we were in 'block' mode, since we're going to pump the parent
962 # class by hand line by line, we need to temporarily switch out to
961 # class by hand line by line, we need to temporarily switch out to
963 # 'line' mode, do a single manual reset and then feed the lines one
962 # 'line' mode, do a single manual reset and then feed the lines one
964 # by one. Note that this only matters if the input has more than one
963 # by one. Note that this only matters if the input has more than one
965 # line.
964 # line.
966 changed_input_mode = False
965 changed_input_mode = False
967
966
968 if len(lines_list)>1 and self.input_mode == 'cell':
967 if len(lines_list)>1 and self.input_mode == 'cell':
969 self.reset()
968 self.reset()
970 changed_input_mode = True
969 changed_input_mode = True
971 saved_input_mode = 'cell'
970 saved_input_mode = 'cell'
972 self.input_mode = 'line'
971 self.input_mode = 'line'
973
972
974 try:
973 try:
975 push = super(IPythonInputSplitter, self).push
974 push = super(IPythonInputSplitter, self).push
976 for line in lines_list:
975 for line in lines_list:
977 if self._is_complete or not self._buffer or \
976 if self._is_complete or not self._buffer or \
978 (self._buffer and self._buffer[-1].rstrip().endswith(':')):
977 (self._buffer and self._buffer[-1].rstrip().endswith(':')):
979 for f in transforms:
978 for f in transforms:
980 line = f(line)
979 line = f(line)
981
980
982 out = push(line)
981 out = push(line)
983 finally:
982 finally:
984 if changed_input_mode:
983 if changed_input_mode:
985 self.input_mode = saved_input_mode
984 self.input_mode = saved_input_mode
986
985
987 return out
986 return out
General Comments 0
You need to be logged in to leave comments. Login now