##// END OF EJS Templates
fix event names
fix event names

File last commit:

r11124:9567c77a
r11226:c4ac215c
Show More
inputsplitter.py
657 lines | 23.5 KiB | text/x-python | PythonLexer
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """Analysis of text input into executable blocks.
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 The main class in this module, :class:`InputSplitter`, is designed to break
input from either interactive, line-by-line environments or block-based ones,
into standalone blocks that can be executed by Python as 'single' statements
(thus triggering sys.displayhook).
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 A companion, :class:`IPythonInputSplitter`, provides the same functionality but
with full support for the extended IPython syntax (magics, system calls, etc).
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 For more details, see the class docstring below.
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780
Fernando Perez
Final documentation changes from code review with Brian, ready to merge.
r2828 Syntax Transformations
----------------------
One of the main jobs of the code in this file is to apply all syntax
transformations that make up 'the IPython language', i.e. magics, shell
escapes, etc. All transformations should be implemented as *fully stateless*
entities, that simply take one line as their input and return a line.
Internally for implementation purposes they may be a normal function or a
callable object, but the only input they receive will be a single line and they
should only return a line, without holding any data-dependent state between
calls.
As an example, the EscapedTransformer is a class so we can more clearly group
together the functionality of dispatching to individual functions based on the
starting escape character, but the only method for public use is its call
method.
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 ToDo
----
Fernando Perez
Final documentation changes from code review with Brian, ready to merge.
r2828 - Should we make push() actually raise an exception once push_accepts_more()
returns False?
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 - Naming cleanups. The tr_* names aren't the most elegant, though now they are
at least just attributes of a class so not really very exposed.
- Think about the best way to support dynamic things: automagic, autocall,
macros, etc.
- Think of a better heuristic for the application of the transforms in
IPythonInputSplitter.push() than looking at the buffer ending in ':'. Idea:
track indentation change events (indent, dedent, nothing) and apply them only
if the indentation went up, but not otherwise.
- Think of the cleanest way for supporting user-specified transformations (the
user prefilters we had before).
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780 Authors
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 -------
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780
* Fernando Perez
* Brian Granger
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """
#-----------------------------------------------------------------------------
Fernando Perez
First implementation of cell magics that goes via inputsplitter....
r6976 # Copyright (C) 2010 The IPython Development Team
Fernando Perez
Created tool for handling interactive input blocks....
r2628 #
# Distributed under the terms of the BSD License. The full license is in
# the file COPYING, distributed as part of this software.
#-----------------------------------------------------------------------------
#-----------------------------------------------------------------------------
# Imports
#-----------------------------------------------------------------------------
# stdlib
Thomas Kluyver
Fix up so tests pass again. input_splitter now uses ast module instead of compiler, bringing it closer to the Python 3 implementation.
r3454 import ast
Fernando Perez
Created tool for handling interactive input blocks....
r2628 import codeop
import re
import sys
Fernando Perez
First pass of input syntax transformation support
r2719 # IPython modules
Thomas Kluyver
More Python 3 compatibility fixes in core.
r4745 from IPython.utils.py3compat import cast_unicode
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 from IPython.core.inputtransformer import (leading_indent,
classic_prompt,
ipy_prompt,
cellmagic,
Thomas Kluyver
Prototype transformer to assemble logical lines
r10105 assemble_logical_lines,
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 help_end,
Thomas Kluyver
Simplify input transformers...
r10107 escaped_commands,
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 assign_from_magic,
assign_from_system,
Thomas Kluyver
Revised input transformation framework.
r10106 assemble_python_lines,
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 )
Thomas Kluyver
Remove unused imports from IPython.core
r11124 # These are available in this module for backwards compatibility.
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 from IPython.core.inputtransformer import (ESC_SHELL, ESC_SH_CAP, ESC_HELP,
ESC_HELP2, ESC_MAGIC, ESC_MAGIC2,
ESC_QUOTE, ESC_QUOTE2, ESC_PAREN, ESC_SEQUENCES)
Matthias BUSSONNIER
take #!%... prefix into account for completion...
r7554
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780 #-----------------------------------------------------------------------------
Fernando Perez
Created tool for handling interactive input blocks....
r2628 # Utilities
#-----------------------------------------------------------------------------
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780 # FIXME: These are general-purpose utilities that later can be moved to the
# general ward. Kept here for now because we're being very strict about test
# coverage with this code, and this lets us ensure that we keep 100% coverage
# while developing.
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633
Fernando Perez
Created tool for handling interactive input blocks....
r2628 # compiled regexps for autoindent management
David Warde-Farley
Fix behaviour of dedent triggering. Closes gh-142.
r3704 dedent_re = re.compile('|'.join([
r'^\s+raise(\s.*)?$', # raise statement (+ space + other stuff, maybe)
r'^\s+raise\([^\)]*\).*$', # wacky raise with immediate open paren
r'^\s+return(\s.*)?$', # normal return (+ space + other stuff, maybe)
r'^\s+return\([^\)]*\).*$', # wacky return with immediate open paren
Aaron Meurer
`yield`, `break`, and `continue` automatically dedent...
r7824 r'^\s+pass\s*$', # pass (optionally followed by trailing spaces)
r'^\s+break\s*$', # break (optionally followed by trailing spaces)
r'^\s+continue\s*$', # continue (optionally followed by trailing spaces)
David Warde-Farley
Fix behaviour of dedent triggering. Closes gh-142.
r3704 ]))
Fernando Perez
Created tool for handling interactive input blocks....
r2628 ini_spaces_re = re.compile(r'^([ \t\r\f\v]+)')
Fernando Perez
Fix bug where 'if 1:' was being added to comment-only code....
r2979 # regexp to match pure comment lines so we don't accidentally insert 'if 1:'
# before pure comments
comment_line_re = re.compile('^\s*\#')
Fernando Perez
Created tool for handling interactive input blocks....
r2628
def num_ini_spaces(s):
"""Return the number of initial spaces in a string.
Note that tabs are counted as a single space. For now, we do *not* support
mixing of tabs and spaces in the user's input.
Parameters
----------
s : string
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663
Returns
-------
n : int
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """
ini_spaces = ini_spaces_re.match(s)
if ini_spaces:
return ini_spaces.end()
else:
return 0
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978 def last_blank(src):
"""Determine if the input source ends in a blank.
A blank is either a newline or a line consisting of whitespace.
Parameters
----------
src : string
A single or multiline string.
"""
Fernando Perez
Add a few more fixes to cell/line input code, switch approaches....
r6981 if not src: return False
ll = src.splitlines()[-1]
return (ll == '') or ll.isspace()
Fernando Perez
Working implementation of cell mode with regular expressions - cleaner.
r6979
Fernando Perez
Add a few more fixes to cell/line input code, switch approaches....
r6981 last_two_blanks_re = re.compile(r'\n\s*\n\s*$', re.MULTILINE)
last_two_blanks_re2 = re.compile(r'.+\n\s*\n\s+$', re.MULTILINE)
Fernando Perez
Working implementation of cell mode with regular expressions - cleaner.
r6979
def last_two_blanks(src):
"""Determine if the input source ends in two blanks.
A blank is either a newline or a line consisting of whitespace.
Parameters
----------
src : string
A single or multiline string.
"""
Fernando Perez
Add a few more fixes to cell/line input code, switch approaches....
r6981 if not src: return False
# The logic here is tricky: I couldn't get a regexp to work and pass all
# the tests, so I took a different approach: split the source by lines,
# grab the last two and prepend '###\n' as a stand-in for whatever was in
# the body before the last two lines. Then, with that structure, it's
# possible to analyze with two regexps. Not the most elegant solution, but
# it works. If anyone tries to change this logic, make sure to validate
# the whole test suite first!
new_src = '\n'.join(['###\n'] + src.splitlines()[-2:])
return (bool(last_two_blanks_re.match(new_src)) or
bool(last_two_blanks_re2.match(new_src)) )
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978
Fernando Perez
Created tool for handling interactive input blocks....
r2628 def remove_comments(src):
"""Remove all comments from input source.
Note: comments are NOT recognized inside of strings!
Parameters
----------
src : string
A single or multiline input string.
Returns
-------
String with all Python comments removed.
"""
return re.sub('#.*', '', src)
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Created tool for handling interactive input blocks....
r2628
def get_input_encoding():
Fernando Perez
Add test for missing input encoding. Back to 100% coverage.
r2718 """Return the default standard input encoding.
If sys.stdin has no encoding, 'ascii' is returned."""
epatters
Made blockbreakers' input encoding detection more robust to strange...
r2674 # There are strange environments for which sys.stdin.encoding is None. We
# ensure that a valid encoding is returned.
encoding = getattr(sys.stdin, 'encoding', None)
if encoding is None:
encoding = 'ascii'
return encoding
Fernando Perez
Created tool for handling interactive input blocks....
r2628
#-----------------------------------------------------------------------------
Fernando Perez
Completed first pass of inputsplitter with IPython syntax....
r2780 # Classes and functions for normal Python syntax handling
Fernando Perez
Created tool for handling interactive input blocks....
r2628 #-----------------------------------------------------------------------------
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 class InputSplitter(object):
Thomas Kluyver
Update docstring for InputSplitter.
r4077 """An object that can accumulate lines of Python source before execution.
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663
Thomas Kluyver
Update docstring for InputSplitter.
r4077 This object is designed to be fed python source line-by-line, using
:meth:`push`. It will return on each push whether the currently pushed
code could be executed already. In addition, it provides a method called
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 :meth:`push_accepts_more` that can be used to query whether more input
can be pushed into a single interactive block.
This is a simple example of how an interactive terminal-based client can use
this tool::
isp = InputSplitter()
while isp.push_accepts_more():
indent = ' '*isp.indent_spaces
prompt = '>>> ' + indent
line = indent + raw_input(prompt)
isp.push(line)
print 'Input source was:\n', isp.source_reset(),
"""
# Number of spaces of indentation computed from input that has been pushed
# so far. This is the attributes callers should query to get the current
# indentation level, in order to provide auto-indent facilities.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 indent_spaces = 0
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # String, indicating the default input encoding. It is computed by default
# at initialization time via get_input_encoding(), but it can be reset by a
# client with specific knowledge of the encoding.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 encoding = ''
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # String where the current full source input is stored, properly encoded.
# Reading this attribute is the normal way of querying the currently pushed
# source code, that has been properly encoded.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 source = ''
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # Code object corresponding to the current source. It is automatically
# synced to the source, so it can be queried at any time to obtain the code
# object; it will be None if the source doesn't compile to valid Python.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 code = None
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 # Private attributes
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # List with lines of input accumulated so far
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 _buffer = None
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # Command compiler
_compile = None
# Mark when input has changed indentation all the way back to flush-left
_full_dedent = False
# Boolean indicating whether the current block is complete
_is_complete = None
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 def __init__(self):
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 """Create a new InputSplitter instance.
Fernando Perez
Add support for append/replace mode after discussion with Evan....
r2634 """
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 self._buffer = []
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self._compile = codeop.CommandCompiler()
Fernando Perez
Created tool for handling interactive input blocks....
r2628 self.encoding = get_input_encoding()
def reset(self):
"""Reset the input buffer and associated state."""
self.indent_spaces = 0
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 self._buffer[:] = []
Fernando Perez
Created tool for handling interactive input blocks....
r2628 self.source = ''
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 self.code = None
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self._is_complete = False
self._full_dedent = False
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Change get_source() api as per code review, to source_reset()....
r2636 def source_reset(self):
"""Return the input source and perform a full reset.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """
out = self.source
Fernando Perez
Change get_source() api as per code review, to source_reset()....
r2636 self.reset()
Fernando Perez
Created tool for handling interactive input blocks....
r2628 return out
def push(self, lines):
Robert Kern
Fix bug where bare strings would be silently ignored in input....
r3293 """Push one or more lines of input.
Fernando Perez
Created tool for handling interactive input blocks....
r2628
This stores the given lines and returns a status code indicating
whether the code forms a complete Python block or not.
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 Any exceptions generated in compilation are swallowed, but if an
exception was produced, the method returns True.
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Parameters
----------
lines : string
One or more lines of Python input.
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Created tool for handling interactive input blocks....
r2628 Returns
-------
is_complete : boolean
True if the current input source (the result of the current input
Jason Grout
Fix RST misformatting.
r7643 plus prior inputs) forms a complete Python execution block. Note that
this value is also stored as a private attribute (``_is_complete``), so it
can be queried at any time.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 self._store(lines)
Fernando Perez
Created tool for handling interactive input blocks....
r2628 source = self.source
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 # Before calling _compile(), reset the code object to None so that if an
Fernando Perez
Created tool for handling interactive input blocks....
r2628 # exception is raised in compilation, we don't mislead by having
# inconsistent code/source attributes.
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self.code, self._is_complete = None, None
Fernando Perez
Completed full block splitting for block-based frontends.
r2645
Fernando Perez
Fix bug with lines ending in continuation markers (\)....
r3013 # Honor termination lines properly
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823 if source.endswith('\\\n'):
Fernando Perez
Fix bug with lines ending in continuation markers (\)....
r3013 return False
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 self._update_indent(lines)
Fernando Perez
push() now swallows syntax errors and immediately produces a 'ready'...
r2635 try:
Thomas Kluyver
Add exec flag so inputsplitter handles multiline cells better.
r3748 self.code = self._compile(source, symbol="exec")
Fernando Perez
push() now swallows syntax errors and immediately produces a 'ready'...
r2635 # Invalid syntax can produce any of a number of different errors from
# inside the compiler, so we have to catch them all. Syntax errors
# immediately produce a 'ready' block, so the invalid Python can be
# sent to the kernel for evaluation with possible ipython
# special-syntax conversion.
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 except (SyntaxError, OverflowError, ValueError, TypeError,
MemoryError):
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self._is_complete = True
Fernando Perez
push() now swallows syntax errors and immediately produces a 'ready'...
r2635 else:
# Compilation didn't produce any exceptions (though it may not have
# given a complete code object)
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self._is_complete = self.code is not None
Fernando Perez
push() now swallows syntax errors and immediately produces a 'ready'...
r2635
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 return self._is_complete
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 def push_accepts_more(self):
"""Return whether a block of interactive input can accept more input.
Fernando Perez
Created tool for handling interactive input blocks....
r2628
This method is meant to be used by line-oriented frontends, who need to
guess whether a block is complete or not based solely on prior and
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 current input lines. The InputSplitter considers it has a complete
Thomas Kluyver
Fix test failure in IPython.lib
r10255 interactive block and will not accept more input when either:
* A SyntaxError is raised
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Thomas Kluyver
Fix test failure in IPython.lib
r10255 * The code is complete and consists of a single line or a single
non-compound statement
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Thomas Kluyver
Fix test failure in IPython.lib
r10255 * The code is complete and has a blank line at the end
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 If the current input produces a syntax error, this method immediately
returns False but does *not* raise the syntax error exception, as
typically clients will want to send invalid syntax to an execution
backend which might convert the invalid syntax into valid Python via
one of the dynamic IPython mechanisms.
Fernando Perez
Created tool for handling interactive input blocks....
r2628 """
Fernando Perez
Implement support for 'cell' mode with Ctrl-Enter....
r3004
# With incomplete input, unconditionally accept more
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 # A syntax error also sets _is_complete to True - see push()
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 if not self._is_complete:
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 #print("Not complete") # debug
Fernando Perez
Created tool for handling interactive input blocks....
r2628 return True
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251
# The user can make any (complete) input execute by leaving a blank line
last_line = self.source.splitlines()[-1]
if (not last_line) or last_line.isspace():
#print("Blank line") # debug
return False
Thomas Kluyver
Fix test failure in IPython.lib
r10255 # If there's just a single line or AST node, and we're flush left, as is
# the case after a simple statement such as 'a=1', we want to execute it
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 # straight away.
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 if self.indent_spaces==0:
Thomas Kluyver
Fix test failure in IPython.lib
r10255 if len(self.source.splitlines()) <= 1:
return False
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 try:
code_ast = ast.parse(u''.join(self._buffer))
except Exception:
#print("Can't parse AST") # debug
return False
Fernando Perez
Implement support for 'cell' mode with Ctrl-Enter....
r3004 else:
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 if len(code_ast.body) == 1 and \
not hasattr(code_ast.body[0], 'body'):
#print("Simple statement") # debug
Fernando Perez
Implement support for 'cell' mode with Ctrl-Enter....
r3004 return False
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 # General fallback - accept more code
return True
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 #------------------------------------------------------------------------
# Private interface
#------------------------------------------------------------------------
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 def _find_indent(self, line):
"""Compute the new indentation level for a single line.
Parameters
----------
line : str
A single new line of non-whitespace, non-comment Python input.
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 Returns
-------
indent_spaces : int
New value for the indent level (it may be equal to self.indent_spaces
if indentation doesn't change.
full_dedent : boolean
Whether the new line causes a full flush-left dedent.
"""
indent_spaces = self.indent_spaces
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 full_dedent = self._full_dedent
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 inisp = num_ini_spaces(line)
if inisp < indent_spaces:
indent_spaces = inisp
if indent_spaces <= 0:
#print 'Full dedent in text',self.source # dbg
full_dedent = True
Paul Ivanov
fix trailing whitespace from reseting indentation
r4204 if line.rstrip()[-1] == ':':
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 indent_spaces += 4
elif dedent_re.match(line):
indent_spaces -= 4
if indent_spaces <= 0:
full_dedent = True
# Safety
if indent_spaces < 0:
indent_spaces = 0
#print 'safety' # dbg
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 return indent_spaces, full_dedent
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Completed full block splitting for block-based frontends.
r2645 def _update_indent(self, lines):
for line in remove_comments(lines).splitlines():
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 if line and not line.isspace():
Fernando Perez
Renamed to inputsplitter, added more tests and examples....
r2663 self.indent_spaces, self._full_dedent = self._find_indent(line)
Fernando Perez
Created tool for handling interactive input blocks....
r2628
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 def _store(self, lines, buffer=None, store='source'):
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 """Store one or more lines of input.
If input lines are not newline-terminated, a newline is automatically
appended."""
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 if buffer is None:
buffer = self._buffer
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 if lines.endswith('\n'):
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 buffer.append(lines)
Fernando Perez
Split blockbreaker tests into a separate file and clean up api....
r2633 else:
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 buffer.append(lines+'\n')
setattr(self, store, self._set_source(buffer))
Fernando Perez
Completed full block splitting for block-based frontends.
r2645
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 def _set_source(self, buffer):
Thomas Kluyver
Further fixes and tweaks for inputsplitter.
r3455 return u''.join(buffer)
Fernando Perez
First pass of input syntax transformation support
r2719
class IPythonInputSplitter(InputSplitter):
"""An input splitter that recognizes all of IPython's special syntax."""
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 # String with raw, untransformed input.
source_raw = ''
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093
# Flag to track when a transformer has stored input that it hasn't given
# back yet.
transformer_accumulating = False
Thomas Kluyver
Revised input transformation framework.
r10106
# Flag to track when assemble_python_lines has stored input that it hasn't
# given back yet.
within_python_line = False
Fernando Perez
First implementation of cell magics that goes via inputsplitter....
r6976
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 # Private attributes
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 # List with lines of raw input accumulated so far.
_buffer_raw = None
Thomas Kluyver
Make line_input_checker=True the default for InputSplitter....
r10493 def __init__(self, line_input_checker=True, physical_line_transforms=None,
Thomas Kluyver
Revised input transformation framework.
r10106 logical_line_transforms=None, python_line_transforms=None):
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 super(IPythonInputSplitter, self).__init__()
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 self._buffer_raw = []
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978 self._validate = True
Thomas Kluyver
Revised input transformation framework.
r10106
Thomas Kluyver
Fix for starting the Qt console
r10113 if physical_line_transforms is not None:
self.physical_line_transforms = physical_line_transforms
else:
self.physical_line_transforms = [leading_indent(),
classic_prompt(),
ipy_prompt(),
]
Thomas Kluyver
Revised input transformation framework.
r10106
self.assemble_logical_lines = assemble_logical_lines()
Thomas Kluyver
Fix for starting the Qt console
r10113 if logical_line_transforms is not None:
self.logical_line_transforms = logical_line_transforms
else:
Thomas Kluyver
Allow IPythonInputSplitter to accept cell magics containing blank lines
r10252 self.logical_line_transforms = [cellmagic(end_on_blank_line=line_input_checker),
Thomas Kluyver
Fix for starting the Qt console
r10113 help_end(),
escaped_commands(),
assign_from_magic(),
assign_from_system(),
]
Thomas Kluyver
Revised input transformation framework.
r10106
self.assemble_python_lines = assemble_python_lines()
Thomas Kluyver
Fix for starting the Qt console
r10113 if python_line_transforms is not None:
self.python_line_transforms = python_line_transforms
else:
# We don't use any of these at present
self.python_line_transforms = []
Thomas Kluyver
Revised input transformation framework.
r10106
@property
def transforms(self):
"Quick access to all transformers."
return self.physical_line_transforms + \
[self.assemble_logical_lines] + self.logical_line_transforms + \
[self.assemble_python_lines] + self.python_line_transforms
@property
def transforms_in_use(self):
"""Transformers, excluding logical line transformers if we're in a
Python line."""
Thomas Kluyver
Fix for \ at end of comment, and add tests
r10112 t = self.physical_line_transforms[:]
Thomas Kluyver
Revised input transformation framework.
r10106 if not self.within_python_line:
Thomas Kluyver
Fix for \ at end of comment, and add tests
r10112 t += [self.assemble_logical_lines] + self.logical_line_transforms
Thomas Kluyver
Revised input transformation framework.
r10106 return t + [self.assemble_python_lines] + self.python_line_transforms
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 def reset(self):
"""Reset the input buffer and associated state."""
Fernando Perez
First implementation of cell magics that goes via inputsplitter....
r6976 super(IPythonInputSplitter, self).reset()
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 self._buffer_raw[:] = []
self.source_raw = ''
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 self.transformer_accumulating = False
Thomas Kluyver
Fix for \ at end of comment, and add tests
r10112 self.within_python_line = False
Thomas Kluyver
Inputsplitter flushes transformers before retrieving source.
r10096 for t in self.transforms:
t.reset()
def flush_transformers(self):
Thomas Kluyver
Revised input transformation framework.
r10106 def _flush(transform, out):
if out is not None:
tmp = transform.push(out)
return tmp or transform.reset() or None
else:
return transform.reset() or None
Thomas Kluyver
Inputsplitter flushes transformers before retrieving source.
r10096 out = None
Thomas Kluyver
Revised input transformation framework.
r10106 for t in self.transforms_in_use:
out = _flush(t, out)
if out is not None:
Thomas Kluyver
Inputsplitter flushes transformers before retrieving source.
r10096 self._store(out)
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080
def source_raw_reset(self):
"""Return input and raw source and perform a full reset.
"""
Thomas Kluyver
Inputsplitter flushes transformers before retrieving source.
r10096 self.flush_transformers()
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 out = self.source
out_r = self.source_raw
self.reset()
return out, out_r
Thomas Kluyver
Inputsplitter flushes transformers before retrieving source.
r10096
def source_reset(self):
self.flush_transformers()
return super(IPythonInputSplitter, self).source_reset()
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978 def push_accepts_more(self):
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 if self.transformer_accumulating:
return True
Fernando Perez
First working version of cell magics in inputsplitter in line mode....
r6978 else:
return super(IPythonInputSplitter, self).push_accepts_more()
Fernando Perez
Add a transform_cell convenience method to the inputsplitter object.
r7487 def transform_cell(self, cell):
"""Process and translate a cell of input.
"""
self.reset()
self.push(cell)
return self.source_reset()
Fernando Perez
First pass of input syntax transformation support
r2719 def push(self, lines):
"""Push one or more lines of IPython input.
Fernando Perez
First implementation of cell magics that goes via inputsplitter....
r6976
This stores the given lines and returns a status code indicating
whether the code forms a complete Python block or not, after processing
all input lines for special IPython syntax.
Any exceptions generated in compilation are swallowed, but if an
exception was produced, the method returns True.
Parameters
----------
lines : string
One or more lines of Python input.
Returns
-------
is_complete : boolean
True if the current input source (the result of the current input
plus prior inputs) forms a complete Python execution block. Note that
this value is also stored as a private attribute (_is_complete), so it
can be queried at any time.
Fernando Perez
First pass of input syntax transformation support
r2719 """
Fernando Perez
Final cleanups responding to Brian's code review....
r2782
Fernando Perez
Stop-gap fix for crash with unicode input....
r3126 # We must ensure all input is pure unicode
Thomas Kluyver
More Python 3 compatibility fixes in core.
r4745 lines = cast_unicode(lines, self.encoding)
Thomas Kluyver
Fix inputsplitter to pass empty lines to transformers
r10094
# ''.splitlines() --> [], but we need to push the empty line to transformers
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 lines_list = lines.splitlines()
Thomas Kluyver
Fix inputsplitter to pass empty lines to transformers
r10094 if not lines_list:
lines_list = ['']
Fernando Perez
Final cleanups responding to Brian's code review....
r2782
Fernando Perez
Add support for accessing raw data to inputsplitter....
r3080 # Store raw source before applying any transformations to it. Note
# that this must be done *after* the reset() call that would otherwise
# flush the buffer.
self._store(lines, self._buffer_raw, 'source_raw')
Aaron Meurer
Line continuations now terminate after one blank line (#2108)...
r7823
Thomas Kluyver
Simplify InputSplitter by stripping out input_mode distinction
r10251 for line in lines_list:
out = self.push_line(line)
Fernando Perez
Final cleanups responding to Brian's code review....
r2782 return out
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093
def push_line(self, line):
buf = self._buffer
Thomas Kluyver
Revised input transformation framework.
r10106
def _accumulating(dbg):
#print(dbg)
self.transformer_accumulating = True
return False
for transformer in self.physical_line_transforms:
line = transformer.push(line)
if line is None:
return _accumulating(transformer)
if not self.within_python_line:
Thomas Kluyver
Fix for \ at end of comment, and add tests
r10112 line = self.assemble_logical_lines.push(line)
if line is None:
return _accumulating('acc logical line')
Thomas Kluyver
Revised input transformation framework.
r10106 for transformer in self.logical_line_transforms:
line = transformer.push(line)
if line is None:
return _accumulating(transformer)
line = self.assemble_python_lines.push(line)
if line is None:
self.within_python_line = True
return _accumulating('acc python line')
else:
self.within_python_line = False
for transformer in self.python_line_transforms:
Thomas Kluyver
Prototype transformer to assemble logical lines
r10105 line = transformer.push(line)
if line is None:
Thomas Kluyver
Revised input transformation framework.
r10106 return _accumulating(transformer)
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093
Thomas Kluyver
Revised input transformation framework.
r10106 #print("transformers clear") #debug
Thomas Kluyver
Update inputsplitter to use new input transformers
r10093 self.transformer_accumulating = False
return super(IPythonInputSplitter, self).push(line)