##// END OF EJS Templates
Final documentation changes from code review with Brian, ready to merge.
Fernando Perez -
Show More
@@ -1,817 +1,839 b''
1 """Analysis of text input into executable blocks.
1 """Analysis of text input into executable blocks.
2
2
3 The main class in this module, :class:`InputSplitter`, is designed to break
3 The main class in this module, :class:`InputSplitter`, is designed to break
4 input from either interactive, line-by-line environments or block-based ones,
4 input from either interactive, line-by-line environments or block-based ones,
5 into standalone blocks that can be executed by Python as 'single' statements
5 into standalone blocks that can be executed by Python as 'single' statements
6 (thus triggering sys.displayhook).
6 (thus triggering sys.displayhook).
7
7
8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
8 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
9 with full support for the extended IPython syntax (magics, system calls, etc).
9 with full support for the extended IPython syntax (magics, system calls, etc).
10
10
11 For more details, see the class docstring below.
11 For more details, see the class docstring below.
12
12
13 Syntax Transformations
14 ----------------------
15
16 One of the main jobs of the code in this file is to apply all syntax
17 transformations that make up 'the IPython language', i.e. magics, shell
18 escapes, etc. All transformations should be implemented as *fully stateless*
19 entities, that simply take one line as their input and return a line.
20 Internally for implementation purposes they may be a normal function or a
21 callable object, but the only input they receive will be a single line and they
22 should only return a line, without holding any data-dependent state between
23 calls.
24
25 As an example, the EscapedTransformer is a class so we can more clearly group
26 together the functionality of dispatching to individual functions based on the
27 starting escape character, but the only method for public use is its call
28 method.
29
30
13 ToDo
31 ToDo
14 ----
32 ----
15
33
34 - Should we make push() actually raise an exception once push_accepts_more()
35 returns False?
36
16 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
37 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
17 at least just attributes of a class so not really very exposed.
38 at least just attributes of a class so not really very exposed.
18
39
19 - Think about the best way to support dynamic things: automagic, autocall,
40 - Think about the best way to support dynamic things: automagic, autocall,
20 macros, etc.
41 macros, etc.
21
42
22 - Think of a better heuristic for the application of the transforms in
43 - Think of a better heuristic for the application of the transforms in
23 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
44 IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
24 track indentation change events (indent, dedent, nothing) and apply them only
45 track indentation change events (indent, dedent, nothing) and apply them only
25 if the indentation went up, but not otherwise.
46 if the indentation went up, but not otherwise.
26
47
27 - Think of the cleanest way for supporting user-specified transformations (the
48 - Think of the cleanest way for supporting user-specified transformations (the
28 user prefilters we had before).
49 user prefilters we had before).
29
50
30 Authors
51 Authors
31 -------
52 -------
32
53
33 * Fernando Perez
54 * Fernando Perez
34 * Brian Granger
55 * Brian Granger
35 """
56 """
36 #-----------------------------------------------------------------------------
57 #-----------------------------------------------------------------------------
37 # Copyright (C) 2010 The IPython Development Team
58 # Copyright (C) 2010 The IPython Development Team
38 #
59 #
39 # Distributed under the terms of the BSD License. The full license is in
60 # Distributed under the terms of the BSD License. The full license is in
40 # the file COPYING, distributed as part of this software.
61 # the file COPYING, distributed as part of this software.
41 #-----------------------------------------------------------------------------
62 #-----------------------------------------------------------------------------
42
63
43 #-----------------------------------------------------------------------------
64 #-----------------------------------------------------------------------------
44 # Imports
65 # Imports
45 #-----------------------------------------------------------------------------
66 #-----------------------------------------------------------------------------
46 # stdlib
67 # stdlib
47 import codeop
68 import codeop
48 import re
69 import re
49 import sys
70 import sys
50
71
51 # IPython modules
72 # IPython modules
52 from IPython.utils.text import make_quoted_expr
73 from IPython.utils.text import make_quoted_expr
53
74
54 #-----------------------------------------------------------------------------
75 #-----------------------------------------------------------------------------
55 # Globals
76 # Globals
56 #-----------------------------------------------------------------------------
77 #-----------------------------------------------------------------------------
57
78
58 # The escape sequences that define the syntax transformations IPython will
79 # The escape sequences that define the syntax transformations IPython will
59 # apply to user input. These can NOT be just changed here: many regular
80 # apply to user input. These can NOT be just changed here: many regular
60 # expressions and other parts of the code may use their hardcoded values, and
81 # expressions and other parts of the code may use their hardcoded values, and
61 # for all intents and purposes they constitute the 'IPython syntax', so they
82 # for all intents and purposes they constitute the 'IPython syntax', so they
62 # should be considered fixed.
83 # should be considered fixed.
63
84
64 ESC_SHELL = '!'
85 ESC_SHELL = '!'
65 ESC_SH_CAP = '!!'
86 ESC_SH_CAP = '!!'
66 ESC_HELP = '?'
87 ESC_HELP = '?'
67 ESC_HELP2 = '??'
88 ESC_HELP2 = '??'
68 ESC_MAGIC = '%'
89 ESC_MAGIC = '%'
69 ESC_QUOTE = ','
90 ESC_QUOTE = ','
70 ESC_QUOTE2 = ';'
91 ESC_QUOTE2 = ';'
71 ESC_PAREN = '/'
92 ESC_PAREN = '/'
72
93
73 #-----------------------------------------------------------------------------
94 #-----------------------------------------------------------------------------
74 # Utilities
95 # Utilities
75 #-----------------------------------------------------------------------------
96 #-----------------------------------------------------------------------------
76
97
77 # FIXME: These are general-purpose utilities that later can be moved to the
98 # FIXME: These are general-purpose utilities that later can be moved to the
78 # general ward. Kept here for now because we're being very strict about test
99 # general ward. Kept here for now because we're being very strict about test
79 # coverage with this code, and this lets us ensure that we keep 100% coverage
100 # coverage with this code, and this lets us ensure that we keep 100% coverage
80 # while developing.
101 # while developing.
81
102
82 # compiled regexps for autoindent management
103 # compiled regexps for autoindent management
83 dedent_re = re.compile(r'^\s+raise|^\s+return|^\s+pass')
104 dedent_re = re.compile(r'^\s+raise|^\s+return|^\s+pass')
84 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
105 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
85
106
86
107
87 def num_ini_spaces(s):
108 def num_ini_spaces(s):
88 """Return the number of initial spaces in a string.
109 """Return the number of initial spaces in a string.
89
110
90 Note that tabs are counted as a single space. For now, we do *not* support
111 Note that tabs are counted as a single space. For now, we do *not* support
91 mixing of tabs and spaces in the user's input.
112 mixing of tabs and spaces in the user's input.
92
113
93 Parameters
114 Parameters
94 ----------
115 ----------
95 s : string
116 s : string
96
117
97 Returns
118 Returns
98 -------
119 -------
99 n : int
120 n : int
100 """
121 """
101
122
102 ini_spaces = ini_spaces_re.match(s)
123 ini_spaces = ini_spaces_re.match(s)
103 if ini_spaces:
124 if ini_spaces:
104 return ini_spaces.end()
125 return ini_spaces.end()
105 else:
126 else:
106 return 0
127 return 0
107
128
108
129
109 def remove_comments(src):
130 def remove_comments(src):
110 """Remove all comments from input source.
131 """Remove all comments from input source.
111
132
112 Note: comments are NOT recognized inside of strings!
133 Note: comments are NOT recognized inside of strings!
113
134
114 Parameters
135 Parameters
115 ----------
136 ----------
116 src : string
137 src : string
117 A single or multiline input string.
138 A single or multiline input string.
118
139
119 Returns
140 Returns
120 -------
141 -------
121 String with all Python comments removed.
142 String with all Python comments removed.
122 """
143 """
123
144
124 return re.sub('#.*', '', src)
145 return re.sub('#.*', '', src)
125
146
126
147
127 def get_input_encoding():
148 def get_input_encoding():
128 """Return the default standard input encoding.
149 """Return the default standard input encoding.
129
150
130 If sys.stdin has no encoding, 'ascii' is returned."""
151 If sys.stdin has no encoding, 'ascii' is returned."""
131 # There are strange environments for which sys.stdin.encoding is None. We
152 # There are strange environments for which sys.stdin.encoding is None. We
132 # ensure that a valid encoding is returned.
153 # ensure that a valid encoding is returned.
133 encoding = getattr(sys.stdin, 'encoding', None)
154 encoding = getattr(sys.stdin, 'encoding', None)
134 if encoding is None:
155 if encoding is None:
135 encoding = 'ascii'
156 encoding = 'ascii'
136 return encoding
157 return encoding
137
158
138 #-----------------------------------------------------------------------------
159 #-----------------------------------------------------------------------------
139 # Classes and functions for normal Python syntax handling
160 # Classes and functions for normal Python syntax handling
140 #-----------------------------------------------------------------------------
161 #-----------------------------------------------------------------------------
141
162
142 class InputSplitter(object):
163 class InputSplitter(object):
143 """An object that can split Python source input in executable blocks.
164 """An object that can split Python source input in executable blocks.
144
165
145 This object is designed to be used in one of two basic modes:
166 This object is designed to be used in one of two basic modes:
146
167
147 1. By feeding it python source line-by-line, using :meth:`push`. In this
168 1. By feeding it python source line-by-line, using :meth:`push`. In this
148 mode, it will return on each push whether the currently pushed code
169 mode, it will return on each push whether the currently pushed code
149 could be executed already. In addition, it provides a method called
170 could be executed already. In addition, it provides a method called
150 :meth:`push_accepts_more` that can be used to query whether more input
171 :meth:`push_accepts_more` that can be used to query whether more input
151 can be pushed into a single interactive block.
172 can be pushed into a single interactive block.
152
173
153 2. By calling :meth:`split_blocks` with a single, multiline Python string,
174 2. By calling :meth:`split_blocks` with a single, multiline Python string,
154 that is then split into blocks each of which can be executed
175 that is then split into blocks each of which can be executed
155 interactively as a single statement.
176 interactively as a single statement.
156
177
157 This is a simple example of how an interactive terminal-based client can use
178 This is a simple example of how an interactive terminal-based client can use
158 this tool::
179 this tool::
159
180
160 isp = InputSplitter()
181 isp = InputSplitter()
161 while isp.push_accepts_more():
182 while isp.push_accepts_more():
162 indent = ' '*isp.indent_spaces
183 indent = ' '*isp.indent_spaces
163 prompt = '>>> ' + indent
184 prompt = '>>> ' + indent
164 line = indent + raw_input(prompt)
185 line = indent + raw_input(prompt)
165 isp.push(line)
186 isp.push(line)
166 print 'Input source was:\n', isp.source_reset(),
187 print 'Input source was:\n', isp.source_reset(),
167 """
188 """
168 # Number of spaces of indentation computed from input that has been pushed
189 # Number of spaces of indentation computed from input that has been pushed
169 # so far. This is the attributes callers should query to get the current
190 # so far. This is the attributes callers should query to get the current
170 # indentation level, in order to provide auto-indent facilities.
191 # indentation level, in order to provide auto-indent facilities.
171 indent_spaces = 0
192 indent_spaces = 0
172 # String, indicating the default input encoding. It is computed by default
193 # String, indicating the default input encoding. It is computed by default
173 # at initialization time via get_input_encoding(), but it can be reset by a
194 # at initialization time via get_input_encoding(), but it can be reset by a
174 # client with specific knowledge of the encoding.
195 # client with specific knowledge of the encoding.
175 encoding = ''
196 encoding = ''
176 # String where the current full source input is stored, properly encoded.
197 # String where the current full source input is stored, properly encoded.
177 # Reading this attribute is the normal way of querying the currently pushed
198 # Reading this attribute is the normal way of querying the currently pushed
178 # source code, that has been properly encoded.
199 # source code, that has been properly encoded.
179 source = ''
200 source = ''
180 # Code object corresponding to the current source. It is automatically
201 # Code object corresponding to the current source. It is automatically
181 # synced to the source, so it can be queried at any time to obtain the code
202 # synced to the source, so it can be queried at any time to obtain the code
182 # object; it will be None if the source doesn't compile to valid Python.
203 # object; it will be None if the source doesn't compile to valid Python.
183 code = None
204 code = None
184 # Input mode
205 # Input mode
185 input_mode = 'append'
206 input_mode = 'append'
186
207
187 # Private attributes
208 # Private attributes
188
209
189 # List with lines of input accumulated so far
210 # List with lines of input accumulated so far
190 _buffer = None
211 _buffer = None
191 # Command compiler
212 # Command compiler
192 _compile = None
213 _compile = None
193 # Mark when input has changed indentation all the way back to flush-left
214 # Mark when input has changed indentation all the way back to flush-left
194 _full_dedent = False
215 _full_dedent = False
195 # Boolean indicating whether the current block is complete
216 # Boolean indicating whether the current block is complete
196 _is_complete = None
217 _is_complete = None
197
218
198 def __init__(self, input_mode=None):
219 def __init__(self, input_mode=None):
199 """Create a new InputSplitter instance.
220 """Create a new InputSplitter instance.
200
221
201 Parameters
222 Parameters
202 ----------
223 ----------
203 input_mode : str
224 input_mode : str
204
225
205 One of 'append', 'replace', default is 'append'. This controls how
226 One of 'append', 'replace', default is 'append'. This controls how
206 new inputs are used: in 'append' mode, they are appended to the
227 new inputs are used: in 'append' mode, they are appended to the
207 existing buffer and the whole buffer is compiled; in 'replace' mode,
228 existing buffer and the whole buffer is compiled; in 'replace' mode,
208 each new input completely replaces all prior inputs. Replace mode is
229 each new input completely replaces all prior inputs. Replace mode is
209 thus equivalent to prepending a full reset() to every push() call.
230 thus equivalent to prepending a full reset() to every push() call.
210
231
211 In practice, line-oriented clients likely want to use 'append' mode
232 In practice, line-oriented clients likely want to use 'append' mode
212 while block-oriented ones will want to use 'replace'.
233 while block-oriented ones will want to use 'replace'.
213 """
234 """
214 self._buffer = []
235 self._buffer = []
215 self._compile = codeop.CommandCompiler()
236 self._compile = codeop.CommandCompiler()
216 self.encoding = get_input_encoding()
237 self.encoding = get_input_encoding()
217 self.input_mode = InputSplitter.input_mode if input_mode is None \
238 self.input_mode = InputSplitter.input_mode if input_mode is None \
218 else input_mode
239 else input_mode
219
240
220 def reset(self):
241 def reset(self):
221 """Reset the input buffer and associated state."""
242 """Reset the input buffer and associated state."""
222 self.indent_spaces = 0
243 self.indent_spaces = 0
223 self._buffer[:] = []
244 self._buffer[:] = []
224 self.source = ''
245 self.source = ''
225 self.code = None
246 self.code = None
226 self._is_complete = False
247 self._is_complete = False
227 self._full_dedent = False
248 self._full_dedent = False
228
249
229 def source_reset(self):
250 def source_reset(self):
230 """Return the input source and perform a full reset.
251 """Return the input source and perform a full reset.
231 """
252 """
232 out = self.source
253 out = self.source
233 self.reset()
254 self.reset()
234 return out
255 return out
235
256
236 def push(self, lines):
257 def push(self, lines):
237 """Push one ore more lines of input.
258 """Push one ore more lines of input.
238
259
239 This stores the given lines and returns a status code indicating
260 This stores the given lines and returns a status code indicating
240 whether the code forms a complete Python block or not.
261 whether the code forms a complete Python block or not.
241
262
242 Any exceptions generated in compilation are swallowed, but if an
263 Any exceptions generated in compilation are swallowed, but if an
243 exception was produced, the method returns True.
264 exception was produced, the method returns True.
244
265
245 Parameters
266 Parameters
246 ----------
267 ----------
247 lines : string
268 lines : string
248 One or more lines of Python input.
269 One or more lines of Python input.
249
270
250 Returns
271 Returns
251 -------
272 -------
252 is_complete : boolean
273 is_complete : boolean
253 True if the current input source (the result of the current input
274 True if the current input source (the result of the current input
254 plus prior inputs) forms a complete Python execution block. Note that
275 plus prior inputs) forms a complete Python execution block. Note that
255 this value is also stored as a private attribute (_is_complete), so it
276 this value is also stored as a private attribute (_is_complete), so it
256 can be queried at any time.
277 can be queried at any time.
257 """
278 """
258 if self.input_mode == 'replace':
279 if self.input_mode == 'replace':
259 self.reset()
280 self.reset()
260
281
261 # If the source code has leading blanks, add 'if 1:\n' to it
282 # If the source code has leading blanks, add 'if 1:\n' to it
262 # this allows execution of indented pasted code. It is tempting
283 # this allows execution of indented pasted code. It is tempting
263 # to add '\n' at the end of source to run commands like ' a=1'
284 # to add '\n' at the end of source to run commands like ' a=1'
264 # directly, but this fails for more complicated scenarios
285 # directly, but this fails for more complicated scenarios
265 if not self._buffer and lines[:1] in [' ', '\t']:
286 if not self._buffer and lines[:1] in [' ', '\t']:
266 lines = 'if 1:\n%s' % lines
287 lines = 'if 1:\n%s' % lines
267
288
268 self._store(lines)
289 self._store(lines)
269 source = self.source
290 source = self.source
270
291
271 # Before calling _compile(), reset the code object to None so that if an
292 # Before calling _compile(), reset the code object to None so that if an
272 # exception is raised in compilation, we don't mislead by having
293 # exception is raised in compilation, we don't mislead by having
273 # inconsistent code/source attributes.
294 # inconsistent code/source attributes.
274 self.code, self._is_complete = None, None
295 self.code, self._is_complete = None, None
275
296
276 self._update_indent(lines)
297 self._update_indent(lines)
277 try:
298 try:
278 self.code = self._compile(source)
299 self.code = self._compile(source)
279 # Invalid syntax can produce any of a number of different errors from
300 # Invalid syntax can produce any of a number of different errors from
280 # inside the compiler, so we have to catch them all. Syntax errors
301 # inside the compiler, so we have to catch them all. Syntax errors
281 # immediately produce a 'ready' block, so the invalid Python can be
302 # immediately produce a 'ready' block, so the invalid Python can be
282 # sent to the kernel for evaluation with possible ipython
303 # sent to the kernel for evaluation with possible ipython
283 # special-syntax conversion.
304 # special-syntax conversion.
284 except (SyntaxError, OverflowError, ValueError, TypeError,
305 except (SyntaxError, OverflowError, ValueError, TypeError,
285 MemoryError):
306 MemoryError):
286 self._is_complete = True
307 self._is_complete = True
287 else:
308 else:
288 # Compilation didn't produce any exceptions (though it may not have
309 # Compilation didn't produce any exceptions (though it may not have
289 # given a complete code object)
310 # given a complete code object)
290 self._is_complete = self.code is not None
311 self._is_complete = self.code is not None
291
312
292 return self._is_complete
313 return self._is_complete
293
314
294 def push_accepts_more(self):
315 def push_accepts_more(self):
295 """Return whether a block of interactive input can accept more input.
316 """Return whether a block of interactive input can accept more input.
296
317
297 This method is meant to be used by line-oriented frontends, who need to
318 This method is meant to be used by line-oriented frontends, who need to
298 guess whether a block is complete or not based solely on prior and
319 guess whether a block is complete or not based solely on prior and
299 current input lines. The InputSplitter considers it has a complete
320 current input lines. The InputSplitter considers it has a complete
300 interactive block and will not accept more input only when either a
321 interactive block and will not accept more input only when either a
301 SyntaxError is raised, or *all* of the following are true:
322 SyntaxError is raised, or *all* of the following are true:
302
323
303 1. The input compiles to a complete statement.
324 1. The input compiles to a complete statement.
304
325
305 2. The indentation level is flush-left (because if we are indented,
326 2. The indentation level is flush-left (because if we are indented,
306 like inside a function definition or for loop, we need to keep
327 like inside a function definition or for loop, we need to keep
307 reading new input).
328 reading new input).
308
329
309 3. There is one extra line consisting only of whitespace.
330 3. There is one extra line consisting only of whitespace.
310
331
311 Because of condition #3, this method should be used only by
332 Because of condition #3, this method should be used only by
312 *line-oriented* frontends, since it means that intermediate blank lines
333 *line-oriented* frontends, since it means that intermediate blank lines
313 are not allowed in function definitions (or any other indented block).
334 are not allowed in function definitions (or any other indented block).
314
335
315 Block-oriented frontends that have a separate keyboard event to
336 Block-oriented frontends that have a separate keyboard event to
316 indicate execution should use the :meth:`split_blocks` method instead.
337 indicate execution should use the :meth:`split_blocks` method instead.
317
338
318 If the current input produces a syntax error, this method immediately
339 If the current input produces a syntax error, this method immediately
319 returns False but does *not* raise the syntax error exception, as
340 returns False but does *not* raise the syntax error exception, as
320 typically clients will want to send invalid syntax to an execution
341 typically clients will want to send invalid syntax to an execution
321 backend which might convert the invalid syntax into valid Python via
342 backend which might convert the invalid syntax into valid Python via
322 one of the dynamic IPython mechanisms.
343 one of the dynamic IPython mechanisms.
323 """
344 """
324
345
325 if not self._is_complete:
346 if not self._is_complete:
326 return True
347 return True
327
348
328 if self.indent_spaces==0:
349 if self.indent_spaces==0:
329 return False
350 return False
330
351
331 last_line = self.source.splitlines()[-1]
352 last_line = self.source.splitlines()[-1]
332 return bool(last_line and not last_line.isspace())
353 return bool(last_line and not last_line.isspace())
333
354
334 def split_blocks(self, lines):
355 def split_blocks(self, lines):
335 """Split a multiline string into multiple input blocks.
356 """Split a multiline string into multiple input blocks.
336
357
337 Note: this method starts by performing a full reset().
358 Note: this method starts by performing a full reset().
338
359
339 Parameters
360 Parameters
340 ----------
361 ----------
341 lines : str
362 lines : str
342 A possibly multiline string.
363 A possibly multiline string.
343
364
344 Returns
365 Returns
345 -------
366 -------
346 blocks : list
367 blocks : list
347 A list of strings, each possibly multiline. Each string corresponds
368 A list of strings, each possibly multiline. Each string corresponds
348 to a single block that can be compiled in 'single' mode (unless it
369 to a single block that can be compiled in 'single' mode (unless it
349 has a syntax error)."""
370 has a syntax error)."""
350
371
351 # This code is fairly delicate. If you make any changes here, make
372 # This code is fairly delicate. If you make any changes here, make
352 # absolutely sure that you do run the full test suite and ALL tests
373 # absolutely sure that you do run the full test suite and ALL tests
353 # pass.
374 # pass.
354
375
355 self.reset()
376 self.reset()
356 blocks = []
377 blocks = []
357
378
358 # Reversed copy so we can use pop() efficiently and consume the input
379 # Reversed copy so we can use pop() efficiently and consume the input
359 # as a stack
380 # as a stack
360 lines = lines.splitlines()[::-1]
381 lines = lines.splitlines()[::-1]
361 # Outer loop over all input
382 # Outer loop over all input
362 while lines:
383 while lines:
363 # Inner loop to build each block
384 # Inner loop to build each block
364 while True:
385 while True:
365 # Safety exit from inner loop
386 # Safety exit from inner loop
366 if not lines:
387 if not lines:
367 break
388 break
368 # Grab next line but don't push it yet
389 # Grab next line but don't push it yet
369 next_line = lines.pop()
390 next_line = lines.pop()
370 # Blank/empty lines are pushed as-is
391 # Blank/empty lines are pushed as-is
371 if not next_line or next_line.isspace():
392 if not next_line or next_line.isspace():
372 self.push(next_line)
393 self.push(next_line)
373 continue
394 continue
374
395
375 # Check indentation changes caused by the *next* line
396 # Check indentation changes caused by the *next* line
376 indent_spaces, _full_dedent = self._find_indent(next_line)
397 indent_spaces, _full_dedent = self._find_indent(next_line)
377
398
378 # If the next line causes a dedent, it can be for two differnt
399 # If the next line causes a dedent, it can be for two differnt
379 # reasons: either an explicit de-dent by the user or a
400 # reasons: either an explicit de-dent by the user or a
380 # return/raise/pass statement. These MUST be handled
401 # return/raise/pass statement. These MUST be handled
381 # separately:
402 # separately:
382 #
403 #
383 # 1. the first case is only detected when the actual explicit
404 # 1. the first case is only detected when the actual explicit
384 # dedent happens, and that would be the *first* line of a *new*
405 # dedent happens, and that would be the *first* line of a *new*
385 # block. Thus, we must put the line back into the input buffer
406 # block. Thus, we must put the line back into the input buffer
386 # so that it starts a new block on the next pass.
407 # so that it starts a new block on the next pass.
387 #
408 #
388 # 2. the second case is detected in the line before the actual
409 # 2. the second case is detected in the line before the actual
389 # dedent happens, so , we consume the line and we can break out
410 # dedent happens, so , we consume the line and we can break out
390 # to start a new block.
411 # to start a new block.
391
412
392 # Case 1, explicit dedent causes a break
413 # Case 1, explicit dedent causes a break
393 if _full_dedent and not next_line.startswith(' '):
414 if _full_dedent and not next_line.startswith(' '):
394 lines.append(next_line)
415 lines.append(next_line)
395 break
416 break
396
417
397 # Otherwise any line is pushed
418 # Otherwise any line is pushed
398 self.push(next_line)
419 self.push(next_line)
399
420
400 # Case 2, full dedent with full block ready:
421 # Case 2, full dedent with full block ready:
401 if _full_dedent or \
422 if _full_dedent or \
402 self.indent_spaces==0 and not self.push_accepts_more():
423 self.indent_spaces==0 and not self.push_accepts_more():
403 break
424 break
404 # Form the new block with the current source input
425 # Form the new block with the current source input
405 blocks.append(self.source_reset())
426 blocks.append(self.source_reset())
406
427
407 return blocks
428 return blocks
408
429
409 #------------------------------------------------------------------------
430 #------------------------------------------------------------------------
410 # Private interface
431 # Private interface
411 #------------------------------------------------------------------------
432 #------------------------------------------------------------------------
412
433
413 def _find_indent(self, line):
434 def _find_indent(self, line):
414 """Compute the new indentation level for a single line.
435 """Compute the new indentation level for a single line.
415
436
416 Parameters
437 Parameters
417 ----------
438 ----------
418 line : str
439 line : str
419 A single new line of non-whitespace, non-comment Python input.
440 A single new line of non-whitespace, non-comment Python input.
420
441
421 Returns
442 Returns
422 -------
443 -------
423 indent_spaces : int
444 indent_spaces : int
424 New value for the indent level (it may be equal to self.indent_spaces
445 New value for the indent level (it may be equal to self.indent_spaces
425 if indentation doesn't change.
446 if indentation doesn't change.
426
447
427 full_dedent : boolean
448 full_dedent : boolean
428 Whether the new line causes a full flush-left dedent.
449 Whether the new line causes a full flush-left dedent.
429 """
450 """
430 indent_spaces = self.indent_spaces
451 indent_spaces = self.indent_spaces
431 full_dedent = self._full_dedent
452 full_dedent = self._full_dedent
432
453
433 inisp = num_ini_spaces(line)
454 inisp = num_ini_spaces(line)
434 if inisp < indent_spaces:
455 if inisp < indent_spaces:
435 indent_spaces = inisp
456 indent_spaces = inisp
436 if indent_spaces <= 0:
457 if indent_spaces <= 0:
437 #print 'Full dedent in text',self.source # dbg
458 #print 'Full dedent in text',self.source # dbg
438 full_dedent = True
459 full_dedent = True
439
460
440 if line[-1] == ':':
461 if line[-1] == ':':
441 indent_spaces += 4
462 indent_spaces += 4
442 elif dedent_re.match(line):
463 elif dedent_re.match(line):
443 indent_spaces -= 4
464 indent_spaces -= 4
444 if indent_spaces <= 0:
465 if indent_spaces <= 0:
445 full_dedent = True
466 full_dedent = True
446
467
447 # Safety
468 # Safety
448 if indent_spaces < 0:
469 if indent_spaces < 0:
449 indent_spaces = 0
470 indent_spaces = 0
450 #print 'safety' # dbg
471 #print 'safety' # dbg
451
472
452 return indent_spaces, full_dedent
473 return indent_spaces, full_dedent
453
474
454 def _update_indent(self, lines):
475 def _update_indent(self, lines):
455 for line in remove_comments(lines).splitlines():
476 for line in remove_comments(lines).splitlines():
456 if line and not line.isspace():
477 if line and not line.isspace():
457 self.indent_spaces, self._full_dedent = self._find_indent(line)
478 self.indent_spaces, self._full_dedent = self._find_indent(line)
458
479
459 def _store(self, lines):
480 def _store(self, lines):
460 """Store one or more lines of input.
481 """Store one or more lines of input.
461
482
462 If input lines are not newline-terminated, a newline is automatically
483 If input lines are not newline-terminated, a newline is automatically
463 appended."""
484 appended."""
464
485
465 if lines.endswith('\n'):
486 if lines.endswith('\n'):
466 self._buffer.append(lines)
487 self._buffer.append(lines)
467 else:
488 else:
468 self._buffer.append(lines+'\n')
489 self._buffer.append(lines+'\n')
469 self._set_source()
490 self._set_source()
470
491
471 def _set_source(self):
492 def _set_source(self):
472 self.source = ''.join(self._buffer).encode(self.encoding)
493 self.source = ''.join(self._buffer).encode(self.encoding)
473
494
474
495
475 #-----------------------------------------------------------------------------
496 #-----------------------------------------------------------------------------
476 # Functions and classes for IPython-specific syntactic support
497 # Functions and classes for IPython-specific syntactic support
477 #-----------------------------------------------------------------------------
498 #-----------------------------------------------------------------------------
478
499
479 # RegExp for splitting line contents into pre-char//first word-method//rest.
500 # RegExp for splitting line contents into pre-char//first word-method//rest.
480 # For clarity, each group in on one line.
501 # For clarity, each group in on one line.
481
502
482 line_split = re.compile("""
503 line_split = re.compile("""
483 ^(\s*) # any leading space
504 ^(\s*) # any leading space
484 ([,;/%]|!!?|\?\??) # escape character or characters
505 ([,;/%]|!!?|\?\??) # escape character or characters
485 \s*([\w\.]*) # function/method part (mix of \w and '.')
506 \s*([\w\.]*) # function/method part (mix of \w and '.')
486 (\s+.*$|$) # rest of line
507 (\s+.*$|$) # rest of line
487 """, re.VERBOSE)
508 """, re.VERBOSE)
488
509
489
510
490 def split_user_input(line):
511 def split_user_input(line):
491 """Split user input into early whitespace, esc-char, function part and rest.
512 """Split user input into early whitespace, esc-char, function part and rest.
492
513
493 This is currently handles lines with '=' in them in a very inconsistent
514 This is currently handles lines with '=' in them in a very inconsistent
494 manner.
515 manner.
495
516
496 Examples
517 Examples
497 ========
518 ========
498 >>> split_user_input('x=1')
519 >>> split_user_input('x=1')
499 ('', '', 'x=1', '')
520 ('', '', 'x=1', '')
500 >>> split_user_input('?')
521 >>> split_user_input('?')
501 ('', '?', '', '')
522 ('', '?', '', '')
502 >>> split_user_input('??')
523 >>> split_user_input('??')
503 ('', '??', '', '')
524 ('', '??', '', '')
504 >>> split_user_input(' ?')
525 >>> split_user_input(' ?')
505 (' ', '?', '', '')
526 (' ', '?', '', '')
506 >>> split_user_input(' ??')
527 >>> split_user_input(' ??')
507 (' ', '??', '', '')
528 (' ', '??', '', '')
508 >>> split_user_input('??x')
529 >>> split_user_input('??x')
509 ('', '??', 'x', '')
530 ('', '??', 'x', '')
510 >>> split_user_input('?x=1')
531 >>> split_user_input('?x=1')
511 ('', '', '?x=1', '')
532 ('', '', '?x=1', '')
512 >>> split_user_input('!ls')
533 >>> split_user_input('!ls')
513 ('', '!', 'ls', '')
534 ('', '!', 'ls', '')
514 >>> split_user_input(' !ls')
535 >>> split_user_input(' !ls')
515 (' ', '!', 'ls', '')
536 (' ', '!', 'ls', '')
516 >>> split_user_input('!!ls')
537 >>> split_user_input('!!ls')
517 ('', '!!', 'ls', '')
538 ('', '!!', 'ls', '')
518 >>> split_user_input(' !!ls')
539 >>> split_user_input(' !!ls')
519 (' ', '!!', 'ls', '')
540 (' ', '!!', 'ls', '')
520 >>> split_user_input(',ls')
541 >>> split_user_input(',ls')
521 ('', ',', 'ls', '')
542 ('', ',', 'ls', '')
522 >>> split_user_input(';ls')
543 >>> split_user_input(';ls')
523 ('', ';', 'ls', '')
544 ('', ';', 'ls', '')
524 >>> split_user_input(' ;ls')
545 >>> split_user_input(' ;ls')
525 (' ', ';', 'ls', '')
546 (' ', ';', 'ls', '')
526 >>> split_user_input('f.g(x)')
547 >>> split_user_input('f.g(x)')
527 ('', '', 'f.g(x)', '')
548 ('', '', 'f.g(x)', '')
528 >>> split_user_input('f.g (x)')
549 >>> split_user_input('f.g (x)')
529 ('', '', 'f.g', '(x)')
550 ('', '', 'f.g', '(x)')
530 """
551 """
531 match = line_split.match(line)
552 match = line_split.match(line)
532 if match:
553 if match:
533 lspace, esc, fpart, rest = match.groups()
554 lspace, esc, fpart, rest = match.groups()
534 else:
555 else:
535 # print "match failed for line '%s'" % line
556 # print "match failed for line '%s'" % line
536 try:
557 try:
537 fpart, rest = line.split(None, 1)
558 fpart, rest = line.split(None, 1)
538 except ValueError:
559 except ValueError:
539 # print "split failed for line '%s'" % line
560 # print "split failed for line '%s'" % line
540 fpart, rest = line,''
561 fpart, rest = line,''
541 lspace = re.match('^(\s*)(.*)', line).groups()[0]
562 lspace = re.match('^(\s*)(.*)', line).groups()[0]
542 esc = ''
563 esc = ''
543
564
544 # fpart has to be a valid python identifier, so it better be only pure
565 # fpart has to be a valid python identifier, so it better be only pure
545 # ascii, no unicode:
566 # ascii, no unicode:
546 try:
567 try:
547 fpart = fpart.encode('ascii')
568 fpart = fpart.encode('ascii')
548 except UnicodeEncodeError:
569 except UnicodeEncodeError:
549 lspace = unicode(lspace)
570 lspace = unicode(lspace)
550 rest = fpart + u' ' + rest
571 rest = fpart + u' ' + rest
551 fpart = u''
572 fpart = u''
552
573
553 #print 'line:<%s>' % line # dbg
574 #print 'line:<%s>' % line # dbg
554 #print 'esc <%s> fpart <%s> rest <%s>' % (esc,fpart.strip(),rest) # dbg
575 #print 'esc <%s> fpart <%s> rest <%s>' % (esc,fpart.strip(),rest) # dbg
555 return lspace, esc, fpart.strip(), rest.lstrip()
576 return lspace, esc, fpart.strip(), rest.lstrip()
556
577
557
578
558 # The escaped translators ALL receive a line where their own escape has been
579 # The escaped translators ALL receive a line where their own escape has been
559 # stripped. Only '?' is valid at the end of the line, all others can only be
580 # stripped. Only '?' is valid at the end of the line, all others can only be
560 # placed at the start.
581 # placed at the start.
561
582
562 class LineInfo(object):
583 class LineInfo(object):
563 """A single line of input and associated info.
584 """A single line of input and associated info.
564
585
565 This is a utility class that mostly wraps the output of
586 This is a utility class that mostly wraps the output of
566 :func:`split_user_input` into a convenient object to be passed around
587 :func:`split_user_input` into a convenient object to be passed around
567 during input transformations.
588 during input transformations.
568
589
569 Includes the following as properties:
590 Includes the following as properties:
570
591
571 line
592 line
572 The original, raw line
593 The original, raw line
573
594
574 lspace
595 lspace
575 Any early whitespace before actual text starts.
596 Any early whitespace before actual text starts.
576
597
577 esc
598 esc
578 The initial esc character (or characters, for double-char escapes like
599 The initial esc character (or characters, for double-char escapes like
579 '??' or '!!').
600 '??' or '!!').
580
601
581 fpart
602 fpart
582 The 'function part', which is basically the maximal initial sequence
603 The 'function part', which is basically the maximal initial sequence
583 of valid python identifiers and the '.' character. This is what is
604 of valid python identifiers and the '.' character. This is what is
584 checked for alias and magic transformations, used for auto-calling,
605 checked for alias and magic transformations, used for auto-calling,
585 etc.
606 etc.
586
607
587 rest
608 rest
588 Everything else on the line.
609 Everything else on the line.
589 """
610 """
590 def __init__(self, line):
611 def __init__(self, line):
591 self.line = line
612 self.line = line
592 self.lspace, self.esc, self.fpart, self.rest = \
613 self.lspace, self.esc, self.fpart, self.rest = \
593 split_user_input(line)
614 split_user_input(line)
594
615
595 def __str__(self):
616 def __str__(self):
596 return "LineInfo [%s|%s|%s|%s]" % (self.lspace, self.esc,
617 return "LineInfo [%s|%s|%s|%s]" % (self.lspace, self.esc,
597 self.fpart, self.rest)
618 self.fpart, self.rest)
598
619
599
620
600 # Transformations of the special syntaxes that don't rely on an explicit escape
621 # Transformations of the special syntaxes that don't rely on an explicit escape
601 # character but instead on patterns on the input line
622 # character but instead on patterns on the input line
602
623
603 # The core transformations are implemented as standalone functions that can be
624 # The core transformations are implemented as standalone functions that can be
604 # tested and validated in isolation. Each of these uses a regexp, we
625 # tested and validated in isolation. Each of these uses a regexp, we
605 # pre-compile these and keep them close to each function definition for clarity
626 # pre-compile these and keep them close to each function definition for clarity
606
627
607 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
628 _assign_system_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
608 r'\s*=\s*!\s*(?P<cmd>.*)')
629 r'\s*=\s*!\s*(?P<cmd>.*)')
609
630
610 def transform_assign_system(line):
631 def transform_assign_system(line):
611 """Handle the `files = !ls` syntax."""
632 """Handle the `files = !ls` syntax."""
612 # FIXME: This transforms the line to use %sc, but we've listed that magic
633 # FIXME: This transforms the line to use %sc, but we've listed that magic
613 # as deprecated. We should then implement this functionality in a
634 # as deprecated. We should then implement this functionality in a
614 # standalone api that we can transform to, without going through a
635 # standalone api that we can transform to, without going through a
615 # deprecated magic.
636 # deprecated magic.
616 m = _assign_system_re.match(line)
637 m = _assign_system_re.match(line)
617 if m is not None:
638 if m is not None:
618 cmd = m.group('cmd')
639 cmd = m.group('cmd')
619 lhs = m.group('lhs')
640 lhs = m.group('lhs')
620 expr = make_quoted_expr("sc -l = %s" % cmd)
641 expr = make_quoted_expr("sc -l = %s" % cmd)
621 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
642 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
622 return new_line
643 return new_line
623 return line
644 return line
624
645
625
646
626 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
647 _assign_magic_re = re.compile(r'(?P<lhs>(\s*)([\w\.]+)((\s*,\s*[\w\.]+)*))'
627 r'\s*=\s*%\s*(?P<cmd>.*)')
648 r'\s*=\s*%\s*(?P<cmd>.*)')
628
649
629 def transform_assign_magic(line):
650 def transform_assign_magic(line):
630 """Handle the `a = %who` syntax."""
651 """Handle the `a = %who` syntax."""
631 m = _assign_magic_re.match(line)
652 m = _assign_magic_re.match(line)
632 if m is not None:
653 if m is not None:
633 cmd = m.group('cmd')
654 cmd = m.group('cmd')
634 lhs = m.group('lhs')
655 lhs = m.group('lhs')
635 expr = make_quoted_expr(cmd)
656 expr = make_quoted_expr(cmd)
636 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
657 new_line = '%s = get_ipython().magic(%s)' % (lhs, expr)
637 return new_line
658 return new_line
638 return line
659 return line
639
660
640
661
641 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
662 _classic_prompt_re = re.compile(r'^([ \t]*>>> |^[ \t]*\.\.\. )')
642
663
643 def transform_classic_prompt(line):
664 def transform_classic_prompt(line):
644 """Handle inputs that start with '>>> ' syntax."""
665 """Handle inputs that start with '>>> ' syntax."""
645
666
646 if not line or line.isspace():
667 if not line or line.isspace():
647 return line
668 return line
648 m = _classic_prompt_re.match(line)
669 m = _classic_prompt_re.match(line)
649 if m:
670 if m:
650 return line[len(m.group(0)):]
671 return line[len(m.group(0)):]
651 else:
672 else:
652 return line
673 return line
653
674
654
675
655 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
676 _ipy_prompt_re = re.compile(r'^([ \t]*In \[\d+\]: |^[ \t]*\ \ \ \.\.\.+: )')
656
677
657 def transform_ipy_prompt(line):
678 def transform_ipy_prompt(line):
658 """Handle inputs that start classic IPython prompt syntax."""
679 """Handle inputs that start classic IPython prompt syntax."""
659
680
660 if not line or line.isspace():
681 if not line or line.isspace():
661 return line
682 return line
662 m = _ipy_prompt_re.match(line)
683 m = _ipy_prompt_re.match(line)
663 if m:
684 if m:
664 return line[len(m.group(0)):]
685 return line[len(m.group(0)):]
665 else:
686 else:
666 return line
687 return line
667
688
668
689
669 class EscapedTransformer(object):
690 class EscapedTransformer(object):
670 """Class to transform lines that are explicitly escaped out."""
691 """Class to transform lines that are explicitly escaped out."""
671
692
672 def __init__(self):
693 def __init__(self):
673 tr = { ESC_SHELL : self.tr_system,
694 tr = { ESC_SHELL : self._tr_system,
674 ESC_SH_CAP : self.tr_system2,
695 ESC_SH_CAP : self._tr_system2,
675 ESC_HELP : self.tr_help,
696 ESC_HELP : self._tr_help,
676 ESC_HELP2 : self.tr_help,
697 ESC_HELP2 : self._tr_help,
677 ESC_MAGIC : self.tr_magic,
698 ESC_MAGIC : self._tr_magic,
678 ESC_QUOTE : self.tr_quote,
699 ESC_QUOTE : self._tr_quote,
679 ESC_QUOTE2 : self.tr_quote2,
700 ESC_QUOTE2 : self._tr_quote2,
680 ESC_PAREN : self.tr_paren }
701 ESC_PAREN : self._tr_paren }
681 self.tr = tr
702 self.tr = tr
682
703
683 # Support for syntax transformations that use explicit escapes typed by the
704 # Support for syntax transformations that use explicit escapes typed by the
684 # user at the beginning of a line
705 # user at the beginning of a line
685 @staticmethod
706 @staticmethod
686 def tr_system(line_info):
707 def _tr_system(line_info):
687 "Translate lines escaped with: !"
708 "Translate lines escaped with: !"
688 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
709 cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
689 return '%sget_ipython().system(%s)' % (line_info.lspace,
710 return '%sget_ipython().system(%s)' % (line_info.lspace,
690 make_quoted_expr(cmd))
711 make_quoted_expr(cmd))
691
712
692 @staticmethod
713 @staticmethod
693 def tr_system2(line_info):
714 def _tr_system2(line_info):
694 "Translate lines escaped with: !!"
715 "Translate lines escaped with: !!"
695 cmd = line_info.line.lstrip()[2:]
716 cmd = line_info.line.lstrip()[2:]
696 return '%sget_ipython().getoutput(%s)' % (line_info.lspace,
717 return '%sget_ipython().getoutput(%s)' % (line_info.lspace,
697 make_quoted_expr(cmd))
718 make_quoted_expr(cmd))
698
719
699 @staticmethod
720 @staticmethod
700 def tr_help(line_info):
721 def _tr_help(line_info):
701 "Translate lines escaped with: ?/??"
722 "Translate lines escaped with: ?/??"
702 # A naked help line should just fire the intro help screen
723 # A naked help line should just fire the intro help screen
703 if not line_info.line[1:]:
724 if not line_info.line[1:]:
704 return 'get_ipython().show_usage()'
725 return 'get_ipython().show_usage()'
705
726
706 # There may be one or two '?' at the end, move them to the front so that
727 # There may be one or two '?' at the end, move them to the front so that
707 # the rest of the logic can assume escapes are at the start
728 # the rest of the logic can assume escapes are at the start
708 line = line_info.line
729 line = line_info.line
709 if line.endswith('?'):
730 if line.endswith('?'):
710 line = line[-1] + line[:-1]
731 line = line[-1] + line[:-1]
711 if line.endswith('?'):
732 if line.endswith('?'):
712 line = line[-1] + line[:-1]
733 line = line[-1] + line[:-1]
713 line_info = LineInfo(line)
734 line_info = LineInfo(line)
714
735
715 # From here on, simply choose which level of detail to get.
736 # From here on, simply choose which level of detail to get.
716 if line_info.esc == '?':
737 if line_info.esc == '?':
717 pinfo = 'pinfo'
738 pinfo = 'pinfo'
718 elif line_info.esc == '??':
739 elif line_info.esc == '??':
719 pinfo = 'pinfo2'
740 pinfo = 'pinfo2'
720
741
721 tpl = '%sget_ipython().magic("%s %s")'
742 tpl = '%sget_ipython().magic("%s %s")'
722 return tpl % (line_info.lspace, pinfo,
743 return tpl % (line_info.lspace, pinfo,
723 ' '.join([line_info.fpart, line_info.rest]).strip())
744 ' '.join([line_info.fpart, line_info.rest]).strip())
724
745
725 @staticmethod
746 @staticmethod
726 def tr_magic(line_info):
747 def _tr_magic(line_info):
727 "Translate lines escaped with: %"
748 "Translate lines escaped with: %"
728 tpl = '%sget_ipython().magic(%s)'
749 tpl = '%sget_ipython().magic(%s)'
729 cmd = make_quoted_expr(' '.join([line_info.fpart,
750 cmd = make_quoted_expr(' '.join([line_info.fpart,
730 line_info.rest]).strip())
751 line_info.rest]).strip())
731 return tpl % (line_info.lspace, cmd)
752 return tpl % (line_info.lspace, cmd)
732
753
733 @staticmethod
754 @staticmethod
734 def tr_quote(line_info):
755 def _tr_quote(line_info):
735 "Translate lines escaped with: ,"
756 "Translate lines escaped with: ,"
736 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
757 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
737 '", "'.join(line_info.rest.split()) )
758 '", "'.join(line_info.rest.split()) )
738
759
739 @staticmethod
760 @staticmethod
740 def tr_quote2(line_info):
761 def _tr_quote2(line_info):
741 "Translate lines escaped with: ;"
762 "Translate lines escaped with: ;"
742 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
763 return '%s%s("%s")' % (line_info.lspace, line_info.fpart,
743 line_info.rest)
764 line_info.rest)
744
765
745 @staticmethod
766 @staticmethod
746 def tr_paren(line_info):
767 def _tr_paren(line_info):
747 "Translate lines escaped with: /"
768 "Translate lines escaped with: /"
748 return '%s%s(%s)' % (line_info.lspace, line_info.fpart,
769 return '%s%s(%s)' % (line_info.lspace, line_info.fpart,
749 ", ".join(line_info.rest.split()))
770 ", ".join(line_info.rest.split()))
750
771
751 def __call__(self, line):
772 def __call__(self, line):
752 """Class to transform lines that are explicitly escaped out.
773 """Class to transform lines that are explicitly escaped out.
753
774
754 This calls the above tr_* static methods for the actual line
775 This calls the above _tr_* static methods for the actual line
755 translations."""
776 translations."""
756
777
757 # Empty lines just get returned unmodified
778 # Empty lines just get returned unmodified
758 if not line or line.isspace():
779 if not line or line.isspace():
759 return line
780 return line
760
781
761 # Get line endpoints, where the escapes can be
782 # Get line endpoints, where the escapes can be
762 line_info = LineInfo(line)
783 line_info = LineInfo(line)
763
784
764 # If the escape is not at the start, only '?' needs to be special-cased.
785 # If the escape is not at the start, only '?' needs to be special-cased.
765 # All other escapes are only valid at the start
786 # All other escapes are only valid at the start
766 if not line_info.esc in self.tr:
787 if not line_info.esc in self.tr:
767 if line.endswith(ESC_HELP):
788 if line.endswith(ESC_HELP):
768 return self.tr_help(line_info)
789 return self._tr_help(line_info)
769 else:
790 else:
770 # If we don't recognize the escape, don't modify the line
791 # If we don't recognize the escape, don't modify the line
771 return line
792 return line
772
793
773 return self.tr[line_info.esc](line_info)
794 return self.tr[line_info.esc](line_info)
774
795
796
775 # A function-looking object to be used by the rest of the code. The purpose of
797 # A function-looking object to be used by the rest of the code. The purpose of
776 # the class in this case is to organize related functionality, more than to
798 # the class in this case is to organize related functionality, more than to
777 # manage state.
799 # manage state.
778 transform_escaped = EscapedTransformer()
800 transform_escaped = EscapedTransformer()
779
801
780
802
781 class IPythonInputSplitter(InputSplitter):
803 class IPythonInputSplitter(InputSplitter):
782 """An input splitter that recognizes all of IPython's special syntax."""
804 """An input splitter that recognizes all of IPython's special syntax."""
783
805
784 def push(self, lines):
806 def push(self, lines):
785 """Push one or more lines of IPython input.
807 """Push one or more lines of IPython input.
786 """
808 """
787 if not lines:
809 if not lines:
788 return super(IPythonInputSplitter, self).push(lines)
810 return super(IPythonInputSplitter, self).push(lines)
789
811
790 lines_list = lines.splitlines()
812 lines_list = lines.splitlines()
791
813
792 transforms = [transform_escaped, transform_assign_system,
814 transforms = [transform_escaped, transform_assign_system,
793 transform_assign_magic, transform_ipy_prompt,
815 transform_assign_magic, transform_ipy_prompt,
794 transform_classic_prompt]
816 transform_classic_prompt]
795
817
796 # Transform logic
818 # Transform logic
797 #
819 #
798 # We only apply the line transformers to the input if we have either no
820 # We only apply the line transformers to the input if we have either no
799 # input yet, or complete input, or if the last line of the buffer ends
821 # input yet, or complete input, or if the last line of the buffer ends
800 # with ':' (opening an indented block). This prevents the accidental
822 # with ':' (opening an indented block). This prevents the accidental
801 # transformation of escapes inside multiline expressions like
823 # transformation of escapes inside multiline expressions like
802 # triple-quoted strings or parenthesized expressions.
824 # triple-quoted strings or parenthesized expressions.
803 #
825 #
804 # The last heuristic, while ugly, ensures that the first line of an
826 # The last heuristic, while ugly, ensures that the first line of an
805 # indented block is correctly transformed.
827 # indented block is correctly transformed.
806 #
828 #
807 # FIXME: try to find a cleaner approach for this last bit.
829 # FIXME: try to find a cleaner approach for this last bit.
808
830
809 for line in lines_list:
831 for line in lines_list:
810 if self._is_complete or not self._buffer or \
832 if self._is_complete or not self._buffer or \
811 (self._buffer and self._buffer[-1].rstrip().endswith(':')):
833 (self._buffer and self._buffer[-1].rstrip().endswith(':')):
812 for f in transforms:
834 for f in transforms:
813 line = f(line)
835 line = f(line)
814
836
815 out = super(IPythonInputSplitter, self).push(line)
837 out = super(IPythonInputSplitter, self).push(line)
816
838
817 return out
839 return out
@@ -1,621 +1,622 b''
1 # -*- coding: utf-8 -*-
1 # -*- coding: utf-8 -*-
2 """Tests for the inputsplitter module.
2 """Tests for the inputsplitter module.
3 """
3 """
4 #-----------------------------------------------------------------------------
4 #-----------------------------------------------------------------------------
5 # Copyright (C) 2010 The IPython Development Team
5 # Copyright (C) 2010 The IPython Development Team
6 #
6 #
7 # Distributed under the terms of the BSD License. The full license is in
7 # Distributed under the terms of the BSD License. The full license is in
8 # the file COPYING, distributed as part of this software.
8 # the file COPYING, distributed as part of this software.
9 #-----------------------------------------------------------------------------
9 #-----------------------------------------------------------------------------
10
10
11 #-----------------------------------------------------------------------------
11 #-----------------------------------------------------------------------------
12 # Imports
12 # Imports
13 #-----------------------------------------------------------------------------
13 #-----------------------------------------------------------------------------
14 # stdlib
14 # stdlib
15 import unittest
15 import unittest
16 import sys
16 import sys
17
17
18 # Third party
18 # Third party
19 import nose.tools as nt
19 import nose.tools as nt
20
20
21 # Our own
21 # Our own
22 from IPython.core import inputsplitter as isp
22 from IPython.core import inputsplitter as isp
23
23
24 #-----------------------------------------------------------------------------
24 #-----------------------------------------------------------------------------
25 # Semi-complete examples (also used as tests)
25 # Semi-complete examples (also used as tests)
26 #-----------------------------------------------------------------------------
26 #-----------------------------------------------------------------------------
27
27
28 # Note: at the bottom, there's a slightly more complete version of this that
28 # Note: at the bottom, there's a slightly more complete version of this that
29 # can be useful during development of code here.
29 # can be useful during development of code here.
30
30
31 def mini_interactive_loop(raw_input):
31 def mini_interactive_loop(raw_input):
32 """Minimal example of the logic of an interactive interpreter loop.
32 """Minimal example of the logic of an interactive interpreter loop.
33
33
34 This serves as an example, and it is used by the test system with a fake
34 This serves as an example, and it is used by the test system with a fake
35 raw_input that simulates interactive input."""
35 raw_input that simulates interactive input."""
36
36
37 from IPython.core.inputsplitter import InputSplitter
37 from IPython.core.inputsplitter import InputSplitter
38
38
39 isp = InputSplitter()
39 isp = InputSplitter()
40 # In practice, this input loop would be wrapped in an outside loop to read
40 # In practice, this input loop would be wrapped in an outside loop to read
41 # input indefinitely, until some exit/quit command was issued. Here we
41 # input indefinitely, until some exit/quit command was issued. Here we
42 # only illustrate the basic inner loop.
42 # only illustrate the basic inner loop.
43 while isp.push_accepts_more():
43 while isp.push_accepts_more():
44 indent = ' '*isp.indent_spaces
44 indent = ' '*isp.indent_spaces
45 prompt = '>>> ' + indent
45 prompt = '>>> ' + indent
46 line = indent + raw_input(prompt)
46 line = indent + raw_input(prompt)
47 isp.push(line)
47 isp.push(line)
48
48
49 # Here we just return input so we can use it in a test suite, but a real
49 # Here we just return input so we can use it in a test suite, but a real
50 # interpreter would instead send it for execution somewhere.
50 # interpreter would instead send it for execution somewhere.
51 src = isp.source_reset()
51 src = isp.source_reset()
52 #print 'Input source was:\n', src # dbg
52 #print 'Input source was:\n', src # dbg
53 return src
53 return src
54
54
55 #-----------------------------------------------------------------------------
55 #-----------------------------------------------------------------------------
56 # Test utilities, just for local use
56 # Test utilities, just for local use
57 #-----------------------------------------------------------------------------
57 #-----------------------------------------------------------------------------
58
58
59 def assemble(block):
59 def assemble(block):
60 """Assemble a block into multi-line sub-blocks."""
60 """Assemble a block into multi-line sub-blocks."""
61 return ['\n'.join(sub_block)+'\n' for sub_block in block]
61 return ['\n'.join(sub_block)+'\n' for sub_block in block]
62
62
63
63
64 def pseudo_input(lines):
64 def pseudo_input(lines):
65 """Return a function that acts like raw_input but feeds the input list."""
65 """Return a function that acts like raw_input but feeds the input list."""
66 ilines = iter(lines)
66 ilines = iter(lines)
67 def raw_in(prompt):
67 def raw_in(prompt):
68 try:
68 try:
69 return next(ilines)
69 return next(ilines)
70 except StopIteration:
70 except StopIteration:
71 return ''
71 return ''
72 return raw_in
72 return raw_in
73
73
74 #-----------------------------------------------------------------------------
74 #-----------------------------------------------------------------------------
75 # Tests
75 # Tests
76 #-----------------------------------------------------------------------------
76 #-----------------------------------------------------------------------------
77 def test_spaces():
77 def test_spaces():
78 tests = [('', 0),
78 tests = [('', 0),
79 (' ', 1),
79 (' ', 1),
80 ('\n', 0),
80 ('\n', 0),
81 (' \n', 1),
81 (' \n', 1),
82 ('x', 0),
82 ('x', 0),
83 (' x', 1),
83 (' x', 1),
84 (' x',2),
84 (' x',2),
85 (' x',4),
85 (' x',4),
86 # Note: tabs are counted as a single whitespace!
86 # Note: tabs are counted as a single whitespace!
87 ('\tx', 1),
87 ('\tx', 1),
88 ('\t x', 2),
88 ('\t x', 2),
89 ]
89 ]
90
90
91 for s, nsp in tests:
91 for s, nsp in tests:
92 nt.assert_equal(isp.num_ini_spaces(s), nsp)
92 nt.assert_equal(isp.num_ini_spaces(s), nsp)
93
93
94
94
95 def test_remove_comments():
95 def test_remove_comments():
96 tests = [('text', 'text'),
96 tests = [('text', 'text'),
97 ('text # comment', 'text '),
97 ('text # comment', 'text '),
98 ('text # comment\n', 'text \n'),
98 ('text # comment\n', 'text \n'),
99 ('text # comment \n', 'text \n'),
99 ('text # comment \n', 'text \n'),
100 ('line # c \nline\n','line \nline\n'),
100 ('line # c \nline\n','line \nline\n'),
101 ('line # c \nline#c2 \nline\nline #c\n\n',
101 ('line # c \nline#c2 \nline\nline #c\n\n',
102 'line \nline\nline\nline \n\n'),
102 'line \nline\nline\nline \n\n'),
103 ]
103 ]
104
104
105 for inp, out in tests:
105 for inp, out in tests:
106 nt.assert_equal(isp.remove_comments(inp), out)
106 nt.assert_equal(isp.remove_comments(inp), out)
107
107
108
108
109 def test_get_input_encoding():
109 def test_get_input_encoding():
110 encoding = isp.get_input_encoding()
110 encoding = isp.get_input_encoding()
111 nt.assert_true(isinstance(encoding, basestring))
111 nt.assert_true(isinstance(encoding, basestring))
112 # simple-minded check that at least encoding a simple string works with the
112 # simple-minded check that at least encoding a simple string works with the
113 # encoding we got.
113 # encoding we got.
114 nt.assert_equal('test'.encode(encoding), 'test')
114 nt.assert_equal('test'.encode(encoding), 'test')
115
115
116
116
117 class NoInputEncodingTestCase(unittest.TestCase):
117 class NoInputEncodingTestCase(unittest.TestCase):
118 def setUp(self):
118 def setUp(self):
119 self.old_stdin = sys.stdin
119 self.old_stdin = sys.stdin
120 class X: pass
120 class X: pass
121 fake_stdin = X()
121 fake_stdin = X()
122 sys.stdin = fake_stdin
122 sys.stdin = fake_stdin
123
123
124 def test(self):
124 def test(self):
125 # Verify that if sys.stdin has no 'encoding' attribute we do the right
125 # Verify that if sys.stdin has no 'encoding' attribute we do the right
126 # thing
126 # thing
127 enc = isp.get_input_encoding()
127 enc = isp.get_input_encoding()
128 self.assertEqual(enc, 'ascii')
128 self.assertEqual(enc, 'ascii')
129
129
130 def tearDown(self):
130 def tearDown(self):
131 sys.stdin = self.old_stdin
131 sys.stdin = self.old_stdin
132
132
133
133
134 class InputSplitterTestCase(unittest.TestCase):
134 class InputSplitterTestCase(unittest.TestCase):
135 def setUp(self):
135 def setUp(self):
136 self.isp = isp.InputSplitter()
136 self.isp = isp.InputSplitter()
137
137
138 def test_reset(self):
138 def test_reset(self):
139 isp = self.isp
139 isp = self.isp
140 isp.push('x=1')
140 isp.push('x=1')
141 isp.reset()
141 isp.reset()
142 self.assertEqual(isp._buffer, [])
142 self.assertEqual(isp._buffer, [])
143 self.assertEqual(isp.indent_spaces, 0)
143 self.assertEqual(isp.indent_spaces, 0)
144 self.assertEqual(isp.source, '')
144 self.assertEqual(isp.source, '')
145 self.assertEqual(isp.code, None)
145 self.assertEqual(isp.code, None)
146 self.assertEqual(isp._is_complete, False)
146 self.assertEqual(isp._is_complete, False)
147
147
148 def test_source(self):
148 def test_source(self):
149 self.isp._store('1')
149 self.isp._store('1')
150 self.isp._store('2')
150 self.isp._store('2')
151 self.assertEqual(self.isp.source, '1\n2\n')
151 self.assertEqual(self.isp.source, '1\n2\n')
152 self.assertTrue(len(self.isp._buffer)>0)
152 self.assertTrue(len(self.isp._buffer)>0)
153 self.assertEqual(self.isp.source_reset(), '1\n2\n')
153 self.assertEqual(self.isp.source_reset(), '1\n2\n')
154 self.assertEqual(self.isp._buffer, [])
154 self.assertEqual(self.isp._buffer, [])
155 self.assertEqual(self.isp.source, '')
155 self.assertEqual(self.isp.source, '')
156
156
157 def test_indent(self):
157 def test_indent(self):
158 isp = self.isp # shorthand
158 isp = self.isp # shorthand
159 isp.push('x=1')
159 isp.push('x=1')
160 self.assertEqual(isp.indent_spaces, 0)
160 self.assertEqual(isp.indent_spaces, 0)
161 isp.push('if 1:\n x=1')
161 isp.push('if 1:\n x=1')
162 self.assertEqual(isp.indent_spaces, 4)
162 self.assertEqual(isp.indent_spaces, 4)
163 isp.push('y=2\n')
163 isp.push('y=2\n')
164 self.assertEqual(isp.indent_spaces, 0)
164 self.assertEqual(isp.indent_spaces, 0)
165 isp.push('if 1:')
165 isp.push('if 1:')
166 self.assertEqual(isp.indent_spaces, 4)
166 self.assertEqual(isp.indent_spaces, 4)
167 isp.push(' x=1')
167 isp.push(' x=1')
168 self.assertEqual(isp.indent_spaces, 4)
168 self.assertEqual(isp.indent_spaces, 4)
169 # Blank lines shouldn't change the indent level
169 # Blank lines shouldn't change the indent level
170 isp.push(' '*2)
170 isp.push(' '*2)
171 self.assertEqual(isp.indent_spaces, 4)
171 self.assertEqual(isp.indent_spaces, 4)
172
172
173 def test_indent2(self):
173 def test_indent2(self):
174 isp = self.isp
174 isp = self.isp
175 # When a multiline statement contains parens or multiline strings, we
175 # When a multiline statement contains parens or multiline strings, we
176 # shouldn't get confused.
176 # shouldn't get confused.
177 isp.push("if 1:")
177 isp.push("if 1:")
178 isp.push(" x = (1+\n 2)")
178 isp.push(" x = (1+\n 2)")
179 self.assertEqual(isp.indent_spaces, 4)
179 self.assertEqual(isp.indent_spaces, 4)
180
180
181 def test_dedent(self):
181 def test_dedent(self):
182 isp = self.isp # shorthand
182 isp = self.isp # shorthand
183 isp.push('if 1:')
183 isp.push('if 1:')
184 self.assertEqual(isp.indent_spaces, 4)
184 self.assertEqual(isp.indent_spaces, 4)
185 isp.push(' pass')
185 isp.push(' pass')
186 self.assertEqual(isp.indent_spaces, 0)
186 self.assertEqual(isp.indent_spaces, 0)
187
187
188 def test_push(self):
188 def test_push(self):
189 isp = self.isp
189 isp = self.isp
190 self.assertTrue(isp.push('x=1'))
190 self.assertTrue(isp.push('x=1'))
191
191
192 def test_push2(self):
192 def test_push2(self):
193 isp = self.isp
193 isp = self.isp
194 self.assertFalse(isp.push('if 1:'))
194 self.assertFalse(isp.push('if 1:'))
195 for line in [' x=1', '# a comment', ' y=2']:
195 for line in [' x=1', '# a comment', ' y=2']:
196 self.assertTrue(isp.push(line))
196 self.assertTrue(isp.push(line))
197
197
198 def test_push3(self):
198 def test_push3(self):
199 """Test input with leading whitespace"""
199 """Test input with leading whitespace"""
200 isp = self.isp
200 isp = self.isp
201 isp.push(' x=1')
201 isp.push(' x=1')
202 isp.push(' y=2')
202 isp.push(' y=2')
203 self.assertEqual(isp.source, 'if 1:\n x=1\n y=2\n')
203 self.assertEqual(isp.source, 'if 1:\n x=1\n y=2\n')
204
204
205 def test_replace_mode(self):
205 def test_replace_mode(self):
206 isp = self.isp
206 isp = self.isp
207 isp.input_mode = 'replace'
207 isp.input_mode = 'replace'
208 isp.push('x=1')
208 isp.push('x=1')
209 self.assertEqual(isp.source, 'x=1\n')
209 self.assertEqual(isp.source, 'x=1\n')
210 isp.push('x=2')
210 isp.push('x=2')
211 self.assertEqual(isp.source, 'x=2\n')
211 self.assertEqual(isp.source, 'x=2\n')
212
212
213 def test_push_accepts_more(self):
213 def test_push_accepts_more(self):
214 isp = self.isp
214 isp = self.isp
215 isp.push('x=1')
215 isp.push('x=1')
216 self.assertFalse(isp.push_accepts_more())
216 self.assertFalse(isp.push_accepts_more())
217
217
218 def test_push_accepts_more2(self):
218 def test_push_accepts_more2(self):
219 isp = self.isp
219 isp = self.isp
220 isp.push('if 1:')
220 isp.push('if 1:')
221 self.assertTrue(isp.push_accepts_more())
221 self.assertTrue(isp.push_accepts_more())
222 isp.push(' x=1')
222 isp.push(' x=1')
223 self.assertTrue(isp.push_accepts_more())
223 self.assertTrue(isp.push_accepts_more())
224 isp.push('')
224 isp.push('')
225 self.assertFalse(isp.push_accepts_more())
225 self.assertFalse(isp.push_accepts_more())
226
226
227 def test_push_accepts_more3(self):
227 def test_push_accepts_more3(self):
228 isp = self.isp
228 isp = self.isp
229 isp.push("x = (2+\n3)")
229 isp.push("x = (2+\n3)")
230 self.assertFalse(isp.push_accepts_more())
230 self.assertFalse(isp.push_accepts_more())
231
231
232 def test_push_accepts_more4(self):
232 def test_push_accepts_more4(self):
233 isp = self.isp
233 isp = self.isp
234 # When a multiline statement contains parens or multiline strings, we
234 # When a multiline statement contains parens or multiline strings, we
235 # shouldn't get confused.
235 # shouldn't get confused.
236 # FIXME: we should be able to better handle de-dents in statements like
236 # FIXME: we should be able to better handle de-dents in statements like
237 # multiline strings and multiline expressions (continued with \ or
237 # multiline strings and multiline expressions (continued with \ or
238 # parens). Right now we aren't handling the indentation tracking quite
238 # parens). Right now we aren't handling the indentation tracking quite
239 # correctly with this, though in practice it may not be too much of a
239 # correctly with this, though in practice it may not be too much of a
240 # problem. We'll need to see.
240 # problem. We'll need to see.
241 isp.push("if 1:")
241 isp.push("if 1:")
242 isp.push(" x = (2+")
242 isp.push(" x = (2+")
243 isp.push(" 3)")
243 isp.push(" 3)")
244 self.assertTrue(isp.push_accepts_more())
244 self.assertTrue(isp.push_accepts_more())
245 isp.push(" y = 3")
245 isp.push(" y = 3")
246 self.assertTrue(isp.push_accepts_more())
246 self.assertTrue(isp.push_accepts_more())
247 isp.push('')
247 isp.push('')
248 self.assertFalse(isp.push_accepts_more())
248 self.assertFalse(isp.push_accepts_more())
249
249
250 def test_syntax_error(self):
250 def test_syntax_error(self):
251 isp = self.isp
251 isp = self.isp
252 # Syntax errors immediately produce a 'ready' block, so the invalid
252 # Syntax errors immediately produce a 'ready' block, so the invalid
253 # Python can be sent to the kernel for evaluation with possible ipython
253 # Python can be sent to the kernel for evaluation with possible ipython
254 # special-syntax conversion.
254 # special-syntax conversion.
255 isp.push('run foo')
255 isp.push('run foo')
256 self.assertFalse(isp.push_accepts_more())
256 self.assertFalse(isp.push_accepts_more())
257
257
258 def check_split(self, block_lines, compile=True):
258 def check_split(self, block_lines, compile=True):
259 blocks = assemble(block_lines)
259 blocks = assemble(block_lines)
260 lines = ''.join(blocks)
260 lines = ''.join(blocks)
261 oblock = self.isp.split_blocks(lines)
261 oblock = self.isp.split_blocks(lines)
262 self.assertEqual(oblock, blocks)
262 self.assertEqual(oblock, blocks)
263 if compile:
263 if compile:
264 for block in blocks:
264 for block in blocks:
265 self.isp._compile(block)
265 self.isp._compile(block)
266
266
267 def test_split(self):
267 def test_split(self):
268 # All blocks of input we want to test in a list. The format for each
268 # All blocks of input we want to test in a list. The format for each
269 # block is a list of lists, with each inner lists consisting of all the
269 # block is a list of lists, with each inner lists consisting of all the
270 # lines (as single-lines) that should make up a sub-block.
270 # lines (as single-lines) that should make up a sub-block.
271
271
272 # Note: do NOT put here sub-blocks that don't compile, as the
272 # Note: do NOT put here sub-blocks that don't compile, as the
273 # check_split() routine makes a final verification pass to check that
273 # check_split() routine makes a final verification pass to check that
274 # each sub_block, as returned by split_blocks(), does compile
274 # each sub_block, as returned by split_blocks(), does compile
275 # correctly.
275 # correctly.
276 all_blocks = [ [['x=1']],
276 all_blocks = [ [['x=1']],
277
277
278 [['x=1'],
278 [['x=1'],
279 ['y=2']],
279 ['y=2']],
280
280
281 [['x=1'],
281 [['x=1'],
282 ['# a comment'],
282 ['# a comment'],
283 ['y=11']],
283 ['y=11']],
284
284
285 [['if 1:',
285 [['if 1:',
286 ' x=1'],
286 ' x=1'],
287 ['y=3']],
287 ['y=3']],
288
288
289 [['def f(x):',
289 [['def f(x):',
290 ' return x'],
290 ' return x'],
291 ['x=1']],
291 ['x=1']],
292
292
293 [['def f(x):',
293 [['def f(x):',
294 ' x+=1',
294 ' x+=1',
295 ' ',
295 ' ',
296 ' return x'],
296 ' return x'],
297 ['x=1']],
297 ['x=1']],
298
298
299 [['def f(x):',
299 [['def f(x):',
300 ' if x>0:',
300 ' if x>0:',
301 ' y=1',
301 ' y=1',
302 ' # a comment',
302 ' # a comment',
303 ' else:',
303 ' else:',
304 ' y=4',
304 ' y=4',
305 ' ',
305 ' ',
306 ' return y'],
306 ' return y'],
307 ['x=1'],
307 ['x=1'],
308 ['if 1:',
308 ['if 1:',
309 ' y=11'] ],
309 ' y=11'] ],
310
310
311 [['for i in range(10):'
311 [['for i in range(10):'
312 ' x=i**2']],
312 ' x=i**2']],
313
313
314 [['for i in range(10):'
314 [['for i in range(10):'
315 ' x=i**2'],
315 ' x=i**2'],
316 ['z = 1']],
316 ['z = 1']],
317 ]
317 ]
318 for block_lines in all_blocks:
318 for block_lines in all_blocks:
319 self.check_split(block_lines)
319 self.check_split(block_lines)
320
320
321 def test_split_syntax_errors(self):
321 def test_split_syntax_errors(self):
322 # Block splitting with invalid syntax
322 # Block splitting with invalid syntax
323 all_blocks = [ [['a syntax error']],
323 all_blocks = [ [['a syntax error']],
324
324
325 [['x=1'],
325 [['x=1'],
326 ['a syntax error']],
326 ['a syntax error']],
327
327
328 [['for i in range(10):'
328 [['for i in range(10):'
329 ' an error']],
329 ' an error']],
330
330
331 ]
331 ]
332 for block_lines in all_blocks:
332 for block_lines in all_blocks:
333 self.check_split(block_lines, compile=False)
333 self.check_split(block_lines, compile=False)
334
334
335
335
336 class InteractiveLoopTestCase(unittest.TestCase):
336 class InteractiveLoopTestCase(unittest.TestCase):
337 """Tests for an interactive loop like a python shell.
337 """Tests for an interactive loop like a python shell.
338 """
338 """
339 def check_ns(self, lines, ns):
339 def check_ns(self, lines, ns):
340 """Validate that the given input lines produce the resulting namespace.
340 """Validate that the given input lines produce the resulting namespace.
341
341
342 Note: the input lines are given exactly as they would be typed in an
342 Note: the input lines are given exactly as they would be typed in an
343 auto-indenting environment, as mini_interactive_loop above already does
343 auto-indenting environment, as mini_interactive_loop above already does
344 auto-indenting and prepends spaces to the input.
344 auto-indenting and prepends spaces to the input.
345 """
345 """
346 src = mini_interactive_loop(pseudo_input(lines))
346 src = mini_interactive_loop(pseudo_input(lines))
347 test_ns = {}
347 test_ns = {}
348 exec src in test_ns
348 exec src in test_ns
349 # We can't check that the provided ns is identical to the test_ns,
349 # We can't check that the provided ns is identical to the test_ns,
350 # because Python fills test_ns with extra keys (copyright, etc). But
350 # because Python fills test_ns with extra keys (copyright, etc). But
351 # we can check that the given dict is *contained* in test_ns
351 # we can check that the given dict is *contained* in test_ns
352 for k,v in ns.items():
352 for k,v in ns.items():
353 self.assertEqual(test_ns[k], v)
353 self.assertEqual(test_ns[k], v)
354
354
355 def test_simple(self):
355 def test_simple(self):
356 self.check_ns(['x=1'], dict(x=1))
356 self.check_ns(['x=1'], dict(x=1))
357
357
358 def test_simple2(self):
358 def test_simple2(self):
359 self.check_ns(['if 1:', 'x=2'], dict(x=2))
359 self.check_ns(['if 1:', 'x=2'], dict(x=2))
360
360
361 def test_xy(self):
361 def test_xy(self):
362 self.check_ns(['x=1; y=2'], dict(x=1, y=2))
362 self.check_ns(['x=1; y=2'], dict(x=1, y=2))
363
363
364 def test_abc(self):
364 def test_abc(self):
365 self.check_ns(['if 1:','a=1','b=2','c=3'], dict(a=1, b=2, c=3))
365 self.check_ns(['if 1:','a=1','b=2','c=3'], dict(a=1, b=2, c=3))
366
366
367 def test_multi(self):
367 def test_multi(self):
368 self.check_ns(['x =(1+','1+','2)'], dict(x=4))
368 self.check_ns(['x =(1+','1+','2)'], dict(x=4))
369
369
370
370
371 def test_LineInfo():
371 def test_LineInfo():
372 """Simple test for LineInfo construction and str()"""
372 """Simple test for LineInfo construction and str()"""
373 linfo = isp.LineInfo(' %cd /home')
373 linfo = isp.LineInfo(' %cd /home')
374 nt.assert_equals(str(linfo), 'LineInfo [ |%|cd|/home]')
374 nt.assert_equals(str(linfo), 'LineInfo [ |%|cd|/home]')
375
375
376
376
377 def test_split_user_input():
377 def test_split_user_input():
378 """Unicode test - split_user_input already has good doctests"""
378 """Unicode test - split_user_input already has good doctests"""
379 line = u"PΓ©rez Fernando"
379 line = u"PΓ©rez Fernando"
380 parts = isp.split_user_input(line)
380 parts = isp.split_user_input(line)
381 parts_expected = (u'', u'', u'', line)
381 parts_expected = (u'', u'', u'', line)
382 nt.assert_equal(parts, parts_expected)
382 nt.assert_equal(parts, parts_expected)
383
383
384
384
385 # Transformer tests
385 # Transformer tests
386 def transform_checker(tests, func):
386 def transform_checker(tests, func):
387 """Utility to loop over test inputs"""
387 """Utility to loop over test inputs"""
388 for inp, tr in tests:
388 for inp, tr in tests:
389 nt.assert_equals(func(inp), tr)
389 nt.assert_equals(func(inp), tr)
390
390
391 # Data for all the syntax tests in the form of lists of pairs of
391 # Data for all the syntax tests in the form of lists of pairs of
392 # raw/transformed input. We store it here as a global dict so that we can use
392 # raw/transformed input. We store it here as a global dict so that we can use
393 # it both within single-function tests and also to validate the behavior of the
393 # it both within single-function tests and also to validate the behavior of the
394 # larger objects
394 # larger objects
395
395
396 syntax = \
396 syntax = \
397 dict(assign_system =
397 dict(assign_system =
398 [('a =! ls', 'a = get_ipython().magic("sc -l = ls")'),
398 [('a =! ls', 'a = get_ipython().magic("sc -l = ls")'),
399 ('b = !ls', 'b = get_ipython().magic("sc -l = ls")'),
399 ('b = !ls', 'b = get_ipython().magic("sc -l = ls")'),
400 ('x=1', 'x=1'), # normal input is unmodified
400 ('x=1', 'x=1'), # normal input is unmodified
401 (' ',' '), # blank lines are kept intact
401 (' ',' '), # blank lines are kept intact
402 ],
402 ],
403
403
404 assign_magic =
404 assign_magic =
405 [('a =% who', 'a = get_ipython().magic("who")'),
405 [('a =% who', 'a = get_ipython().magic("who")'),
406 ('b = %who', 'b = get_ipython().magic("who")'),
406 ('b = %who', 'b = get_ipython().magic("who")'),
407 ('x=1', 'x=1'), # normal input is unmodified
407 ('x=1', 'x=1'), # normal input is unmodified
408 (' ',' '), # blank lines are kept intact
408 (' ',' '), # blank lines are kept intact
409 ],
409 ],
410
410
411 classic_prompt =
411 classic_prompt =
412 [('>>> x=1', 'x=1'),
412 [('>>> x=1', 'x=1'),
413 ('x=1', 'x=1'), # normal input is unmodified
413 ('x=1', 'x=1'), # normal input is unmodified
414 (' ',' '), # blank lines are kept intact
414 (' ',' '), # blank lines are kept intact
415 ],
415 ],
416
416
417 ipy_prompt =
417 ipy_prompt =
418 [('In [1]: x=1', 'x=1'),
418 [('In [1]: x=1', 'x=1'),
419 ('x=1', 'x=1'), # normal input is unmodified
419 ('x=1', 'x=1'), # normal input is unmodified
420 (' ',' '), # blank lines are kept intact
420 (' ',' '), # blank lines are kept intact
421 ],
421 ],
422
422
423 # Tests for the escape transformer to leave normal code alone
423 # Tests for the escape transformer to leave normal code alone
424 escaped_noesc =
424 escaped_noesc =
425 [ (' ', ' '),
425 [ (' ', ' '),
426 ('x=1', 'x=1'),
426 ('x=1', 'x=1'),
427 ],
427 ],
428
428
429 # System calls
429 # System calls
430 escaped_shell =
430 escaped_shell =
431 [ ('!ls', 'get_ipython().system("ls")'),
431 [ ('!ls', 'get_ipython().system("ls")'),
432 # Double-escape shell, this means to capture the output of the
432 # Double-escape shell, this means to capture the output of the
433 # subprocess and return it
433 # subprocess and return it
434 ('!!ls', 'get_ipython().getoutput("ls")'),
434 ('!!ls', 'get_ipython().getoutput("ls")'),
435 ],
435 ],
436
436
437 # Help/object info
437 # Help/object info
438 escaped_help =
438 escaped_help =
439 [ ('?', 'get_ipython().show_usage()'),
439 [ ('?', 'get_ipython().show_usage()'),
440 ('?x1', 'get_ipython().magic("pinfo x1")'),
440 ('?x1', 'get_ipython().magic("pinfo x1")'),
441 ('??x2', 'get_ipython().magic("pinfo2 x2")'),
441 ('??x2', 'get_ipython().magic("pinfo2 x2")'),
442 ('x3?', 'get_ipython().magic("pinfo x3")'),
442 ('x3?', 'get_ipython().magic("pinfo x3")'),
443 ('x4??', 'get_ipython().magic("pinfo2 x4")'),
443 ('x4??', 'get_ipython().magic("pinfo2 x4")'),
444 ],
444 ],
445
445
446 # Explicit magic calls
446 # Explicit magic calls
447 escaped_magic =
447 escaped_magic =
448 [ ('%cd', 'get_ipython().magic("cd")'),
448 [ ('%cd', 'get_ipython().magic("cd")'),
449 ('%cd /home', 'get_ipython().magic("cd /home")'),
449 ('%cd /home', 'get_ipython().magic("cd /home")'),
450 (' %magic', ' get_ipython().magic("magic")'),
450 (' %magic', ' get_ipython().magic("magic")'),
451 ],
451 ],
452
452
453 # Quoting with separate arguments
453 # Quoting with separate arguments
454 escaped_quote =
454 escaped_quote =
455 [ (',f', 'f("")'),
455 [ (',f', 'f("")'),
456 (',f x', 'f("x")'),
456 (',f x', 'f("x")'),
457 (' ,f y', ' f("y")'),
457 (' ,f y', ' f("y")'),
458 (',f a b', 'f("a", "b")'),
458 (',f a b', 'f("a", "b")'),
459 ],
459 ],
460
460
461 # Quoting with single argument
461 # Quoting with single argument
462 escaped_quote2 =
462 escaped_quote2 =
463 [ (';f', 'f("")'),
463 [ (';f', 'f("")'),
464 (';f x', 'f("x")'),
464 (';f x', 'f("x")'),
465 (' ;f y', ' f("y")'),
465 (' ;f y', ' f("y")'),
466 (';f a b', 'f("a b")'),
466 (';f a b', 'f("a b")'),
467 ],
467 ],
468
468
469 # Simply apply parens
469 # Simply apply parens
470 escaped_paren =
470 escaped_paren =
471 [ ('/f', 'f()'),
471 [ ('/f', 'f()'),
472 ('/f x', 'f(x)'),
472 ('/f x', 'f(x)'),
473 (' /f y', ' f(y)'),
473 (' /f y', ' f(y)'),
474 ('/f a b', 'f(a, b)'),
474 ('/f a b', 'f(a, b)'),
475 ],
475 ],
476
476
477 # More complex multiline tests
477 # More complex multiline tests
478 ## escaped_multiline =
478 ## escaped_multiline =
479 ## [()],
479 ## [()],
480 )
480 )
481
481
482 # multiline syntax examples. Each of these should be a list of lists, with
482 # multiline syntax examples. Each of these should be a list of lists, with
483 # each entry itself having pairs of raw/transformed input. The union (with
483 # each entry itself having pairs of raw/transformed input. The union (with
484 # '\n'.join() of the transformed inputs is what the splitter should produce
484 # '\n'.join() of the transformed inputs is what the splitter should produce
485 # when fed the raw lines one at a time via push.
485 # when fed the raw lines one at a time via push.
486 syntax_ml = \
486 syntax_ml = \
487 dict(classic_prompt =
487 dict(classic_prompt =
488 [ [('>>> for i in range(10):','for i in range(10):'),
488 [ [('>>> for i in range(10):','for i in range(10):'),
489 ('... print i',' print i'),
489 ('... print i',' print i'),
490 ('... ', ''),
490 ('... ', ''),
491 ],
491 ],
492 ],
492 ],
493
493
494 ipy_prompt =
494 ipy_prompt =
495 [ [('In [24]: for i in range(10):','for i in range(10):'),
495 [ [('In [24]: for i in range(10):','for i in range(10):'),
496 (' ....: print i',' print i'),
496 (' ....: print i',' print i'),
497 (' ....: ', ''),
497 (' ....: ', ''),
498 ],
498 ],
499 ],
499 ],
500 )
500 )
501
501
502
502
503 def test_assign_system():
503 def test_assign_system():
504 transform_checker(syntax['assign_system'], isp.transform_assign_system)
504 transform_checker(syntax['assign_system'], isp.transform_assign_system)
505
505
506
506
507 def test_assign_magic():
507 def test_assign_magic():
508 transform_checker(syntax['assign_magic'], isp.transform_assign_magic)
508 transform_checker(syntax['assign_magic'], isp.transform_assign_magic)
509
509
510
510
511 def test_classic_prompt():
511 def test_classic_prompt():
512 transform_checker(syntax['classic_prompt'], isp.transform_classic_prompt)
512 transform_checker(syntax['classic_prompt'], isp.transform_classic_prompt)
513 for example in syntax_ml['classic_prompt']:
513 for example in syntax_ml['classic_prompt']:
514 transform_checker(example, isp.transform_classic_prompt)
514 transform_checker(example, isp.transform_classic_prompt)
515
515
516
516
517 def test_ipy_prompt():
517 def test_ipy_prompt():
518 transform_checker(syntax['ipy_prompt'], isp.transform_ipy_prompt)
518 transform_checker(syntax['ipy_prompt'], isp.transform_ipy_prompt)
519 for example in syntax_ml['ipy_prompt']:
519 for example in syntax_ml['ipy_prompt']:
520 transform_checker(example, isp.transform_ipy_prompt)
520 transform_checker(example, isp.transform_ipy_prompt)
521
521
522
522
523 def test_escaped_noesc():
523 def test_escaped_noesc():
524 transform_checker(syntax['escaped_noesc'], isp.transform_escaped)
524 transform_checker(syntax['escaped_noesc'], isp.transform_escaped)
525
525
526
526
527 def test_escaped_shell():
527 def test_escaped_shell():
528 transform_checker(syntax['escaped_shell'], isp.transform_escaped)
528 transform_checker(syntax['escaped_shell'], isp.transform_escaped)
529
529
530
530
531 def test_escaped_help():
531 def test_escaped_help():
532 transform_checker(syntax['escaped_help'], isp.transform_escaped)
532 transform_checker(syntax['escaped_help'], isp.transform_escaped)
533
533
534
534
535 def test_escaped_magic():
535 def test_escaped_magic():
536 transform_checker(syntax['escaped_magic'], isp.transform_escaped)
536 transform_checker(syntax['escaped_magic'], isp.transform_escaped)
537
537
538
538
539 def test_escaped_quote():
539 def test_escaped_quote():
540 transform_checker(syntax['escaped_quote'], isp.transform_escaped)
540 transform_checker(syntax['escaped_quote'], isp.transform_escaped)
541
541
542
542
543 def test_escaped_quote2():
543 def test_escaped_quote2():
544 transform_checker(syntax['escaped_quote2'], isp.transform_escaped)
544 transform_checker(syntax['escaped_quote2'], isp.transform_escaped)
545
545
546
546
547 def test_escaped_paren():
547 def test_escaped_paren():
548 transform_checker(syntax['escaped_paren'], isp.transform_escaped)
548 transform_checker(syntax['escaped_paren'], isp.transform_escaped)
549
549
550
550
551 class IPythonInputTestCase(InputSplitterTestCase):
551 class IPythonInputTestCase(InputSplitterTestCase):
552 """By just creating a new class whose .isp is a different instance, we
552 """By just creating a new class whose .isp is a different instance, we
553 re-run the same test battery on the new input splitter.
553 re-run the same test battery on the new input splitter.
554
554
555 In addition, this runs the tests over the syntax and syntax_ml dicts that
555 In addition, this runs the tests over the syntax and syntax_ml dicts that
556 were tested by individual functions, as part of the OO interface.
556 were tested by individual functions, as part of the OO interface.
557 """
557 """
558 def setUp(self):
558 def setUp(self):
559 self.isp = isp.IPythonInputSplitter()
559 self.isp = isp.IPythonInputSplitter()
560
560
561 def test_syntax(self):
561 def test_syntax(self):
562 """Call all single-line syntax tests from the main object"""
562 """Call all single-line syntax tests from the main object"""
563 isp = self.isp
563 isp = self.isp
564 for example in syntax.itervalues():
564 for example in syntax.itervalues():
565 for raw, out_t in example:
565 for raw, out_t in example:
566 if raw.startswith(' '):
566 if raw.startswith(' '):
567 continue
567 continue
568
568
569 isp.push(raw)
569 isp.push(raw)
570 out = isp.source_reset().rstrip()
570 out = isp.source_reset().rstrip()
571 self.assertEqual(out, out_t)
571 self.assertEqual(out, out_t)
572
572
573 def test_syntax_multiline(self):
573 def test_syntax_multiline(self):
574 isp = self.isp
574 isp = self.isp
575 for example in syntax_ml.itervalues():
575 for example in syntax_ml.itervalues():
576 out_t_parts = []
576 out_t_parts = []
577 for line_pairs in example:
577 for line_pairs in example:
578 for raw, out_t_part in line_pairs:
578 for raw, out_t_part in line_pairs:
579 isp.push(raw)
579 isp.push(raw)
580 out_t_parts.append(out_t_part)
580 out_t_parts.append(out_t_part)
581
581
582 out = isp.source_reset().rstrip()
582 out = isp.source_reset().rstrip()
583 out_t = '\n'.join(out_t_parts).rstrip()
583 out_t = '\n'.join(out_t_parts).rstrip()
584 self.assertEqual(out, out_t)
584 self.assertEqual(out, out_t)
585
585
586
586
587 #-----------------------------------------------------------------------------
587 #-----------------------------------------------------------------------------
588 # Main - use as a script
588 # Main - use as a script
589 #-----------------------------------------------------------------------------
589 #-----------------------------------------------------------------------------
590
590
591 if __name__ == '__main__':
591 if __name__ == '__main__':
592 # A simple demo for interactive experimentation. This code will not get
592 # A simple demo for interactive experimentation. This code will not get
593 # picked up by any test suite. Useful mostly for illustration and during
593 # picked up by any test suite. Useful mostly for illustration and during
594 # development.
594 # development.
595 from IPython.core.inputsplitter import InputSplitter, IPythonInputSplitter
595 from IPython.core.inputsplitter import InputSplitter, IPythonInputSplitter
596
596
597 # configure here the syntax to use, prompt and whether to autoindent
597 # configure here the syntax to use, prompt and whether to autoindent
598 #isp, start_prompt = InputSplitter(), '>>> '
598 #isp, start_prompt = InputSplitter(), '>>> '
599 isp, start_prompt = IPythonInputSplitter(), 'In> '
599 isp, start_prompt = IPythonInputSplitter(), 'In> '
600
600
601 autoindent = True
601 autoindent = True
602 #autoindent = False
602 #autoindent = False
603
603
604 try:
604 try:
605 while True:
605 while True:
606 prompt = start_prompt
606 prompt = start_prompt
607 while isp.push_accepts_more():
607 while isp.push_accepts_more():
608 indent = ' '*isp.indent_spaces
608 indent = ' '*isp.indent_spaces
609 if autoindent:
609 if autoindent:
610 line = indent + raw_input(prompt+indent)
610 line = indent + raw_input(prompt+indent)
611 else:
611 else:
612 line = raw_input(prompt)
612 line = raw_input(prompt)
613 isp.push(line)
613 isp.push(line)
614 prompt = '... '
614 prompt = '... '
615
615
616 # Here we just return input so we can use it in a test suite, but a
616 # Here we just return input so we can use it in a test suite, but a
617 # real interpreter would instead send it for execution somewhere.
617 # real interpreter would instead send it for execution somewhere.
618 #src = isp.source; raise EOFError # dbg
618 src = isp.source_reset()
619 src = isp.source_reset()
619 print 'Input source was:\n', src
620 print 'Input source was:\n', src
620 except EOFError:
621 except EOFError:
621 print 'Bye'
622 print 'Bye'
General Comments 0
You need to be logged in to leave comments. Login now