Merge branch 'newkernel' into qtnewkernel

IPython/lib/guisupport.py

@@ -0,0 +1,140 b''
+#!/usr/bin/env python
+# coding: utf-8
+"""
+Support for creating GUI apps and starting event loops.
+
+IPython's GUI integration allows interative plotting and GUI usage in IPython
+session. IPython has two different types of GUI integration:
+
+1. The terminal based IPython supports GUI event loops through Python's
+   PyOS_InputHook. PyOS_InputHook is a hook that Python calls periodically
+   whenever raw_input is waiting for a user to type code. We implement GUI
+   support in the terminal by setting PyOS_InputHook to a function that 
+   iterates the event loop for a short while. It is important to note that
+   in this situation, the real GUI event loop is NOT run in the normal
+   manner, so you can't use the normal means to detect that it is running.
+2. In the two process IPython kernel/frontend, the GUI event loop is run in 
+   the kernel. In this case, the event loop is run in the normal manner by
+   calling the function or method of the GUI toolkit that starts the event
+   loop.
+
+In addition to starting the GUI event loops in one of these two ways, IPython
+will *always* create an appropriate GUI application object when GUi
+integration is enabled.
+
+If you want your GUI apps to run in IPython you need to do two things:
+
+1. Test to see if there is already an existing main application object. If
+   there is, you should use it. If there is not an existing application object
+   you should create one.
+2. Test to see if the GUI event loop is running. If it is, you should not
+   start it. If the event loop is not running you may start it.
+
+This module contains functions for each toolkit that perform these things
+in a consistent manner. Because of how PyOS_InputHook runs the event loop
+you cannot detect if the event loop is running using the traditional calls
+(such as ``wx.GetApp.IsMainLoopRunning()`` in wxPython). If PyOS_InputHook is
+set These methods will return a false negative. That is, they will say the
+event loop is not running, when is actually is. To work around this limitation
+we proposed the following informal protocol:
+
+* Whenever someone starts the event loop, they *must* set the ``_in_event_loop``
+  attribute of the main application object to ``True``. This should be done
+  regardless of how the event loop is actually run.
+* Whenever someone stops the event loop, they *must* set the ``_in_event_loop``
+  attribute of the main application object to ``False``.
+* If you want to see if the event loop is running, you *must* use ``hasattr``
+  to see if ``_in_event_loop`` attribute has been set. If it is set, you
+  *must* use its value. If it has not been set, you can query the toolkit
+  in the normal manner.
+
+The functions below implement this logic for each GUI toolkit. If you need
+to create custom application subclasses, you will likely have to modify this
+code for your own purposes. This code can be copied into your own project
+so you don't have to depend on IPython.
+
+"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2010  The IPython Development Team
+#
+#  Distributed under the terms of the BSD License.  The full license is in
+#  the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# wx
+#-----------------------------------------------------------------------------
+
+def get_app_wx(*args, **kwargs):
+    """Create a new wx app or return an exiting one."""
+    import wx
+    app = wx.GetApp()
+    if app is None:
+        app = wx.PySimpleApp(*args, **kwargs)
+    return app
+
+def is_event_loop_running_wx(app=None):
+    """Is the wx event loop running."""
+    if app is None:
+        app = get_app_wx()
+    if hasattr(app, '_in_event_loop'):
+        return app._in_event_loop
+    else:
+        return app.IsMainLoopRunning()
+
+def start_event_loop_wx(app=None):
+    """Start the wx event loop in a consistent manner."""
+    if app is None:
+        app = get_app_wx()
+    if not is_event_loop_running_wx(app):
+        app._in_event_loop = True
+        app.MainLoop()
+        app._in_event_loop = False
+    else:
+        app._in_event_loop = True
+
+#-----------------------------------------------------------------------------
+# qt4
+#-----------------------------------------------------------------------------
+
+def get_app_qt4(*args, **kwargs):
+    """Create a new qt4 app or return an existing one."""
+    from PyQt4 import QtGui
+    app = QtGui.QApplication.instance()
+    if app is None:
+        app = QtGui.QApplication(*args, **kwargs)
+    return app
+
+def is_event_loop_running_qt4(app=None):
+    """Is the qt4 event loop running."""
+    if app is None:
+        app = get_app_qt4()
+    if hasattr(app, '_in_event_loop'):
+        return app._in_event_loop
+    else:
+        # Does qt4 provide a other way to detect this?
+        return False
+
+def start_event_loop_qt4(app=None):
+    """Start the qt4 event loop in a consistent manner."""
+    if app is None:
+        app = get_app_qt4()
+    if not is_event_loop_running_qt4(app):
+        app._in_event_loop = True
+        app.exec_()
+        app._in_event_loop = False
+    else:
+        app._in_event_loop = True
+
+#-----------------------------------------------------------------------------
+# Tk
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# gtk
+#-----------------------------------------------------------------------------

IPython/zmq/pylab/backend_payload_svg.py

@@ -1,30 +1,39 b''
+"""Produce SVG versions of active plots for display by the rich Qt frontend.
+"""
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
 # Standard library imports
 from cStringIO import StringIO
 
 # System library imports.
 from matplotlib.backends.backend_svg import new_figure_manager
 from matplotlib._pylab_helpers import Gcf
 
 # Local imports.
 from backend_payload import add_plot_payload
 
+#-----------------------------------------------------------------------------
+# Functions
+#-----------------------------------------------------------------------------
 
 def show():
     """ Deliver a SVG payload.
     """
-    figure_manager = Gcf.get_active()
-    if figure_manager is not None:
+    for figure_manager in Gcf.get_all_fig_managers():
         # Make the background transparent.
         # figure_manager.canvas.figure.patch.set_alpha(0.0)
         # Set the background to white instead so it looks good on black.
         figure_manager.canvas.figure.set_facecolor('white')
         figure_manager.canvas.figure.set_edgecolor('white')
         data = svg_from_canvas(figure_manager.canvas)
         add_plot_payload('svg', data)
 
+
 def svg_from_canvas(canvas):
     """ Return a string containing the SVG representation of a FigureCanvasSvg.
     """
     string_io = StringIO()
     canvas.print_svg(string_io)
     return string_io.getvalue()

IPython/zmq/zmqshell.py

@@ -1,388 +1,427 b''
+"""A ZMQ-based subclass of InteractiveShell.
+
+This code is meant to ease the refactoring of the base InteractiveShell into
+something with a cleaner architecture for 2-process use, without actually
+breaking InteractiveShell itself.  So we're doing something a bit ugly, where
+we subclass and override what we want to fix.  Once this is working well, we
+can go back to the base class and refactor the code for a cleaner inheritance
+implementation that doesn't rely on so much monkeypatching.
+
+But this lets us maintain a fully working IPython as we develop the new
+machinery.  This should thus be thought of as scaffolding.
+"""
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+from __future__ import print_function
+
+# Stdlib
 import inspect
+import os
 import re
 import sys
+
 from subprocess import Popen, PIPE
 
+# Our own
 from IPython.core.interactiveshell import (
     InteractiveShell, InteractiveShellABC
 )
 from IPython.core.displayhook import DisplayHook
 from IPython.core.macro import Macro
 from IPython.utils.path import get_py_filename
 from IPython.utils.text import StringTypes
 from IPython.utils.traitlets import Instance, Type, Dict
 from IPython.utils.warn import warn
 from IPython.zmq.session import extract_header
 from IPython.core.payloadpage import install_payload_page
 from session import Session
 
+#-----------------------------------------------------------------------------
+# Globals and side-effects
+#-----------------------------------------------------------------------------
+
 # Install the payload version of page.
 install_payload_page()
 
+#-----------------------------------------------------------------------------
+# Functions and classes
+#-----------------------------------------------------------------------------
 
 class ZMQDisplayHook(DisplayHook):
 
     session = Instance(Session)
     pub_socket = Instance('zmq.Socket')
     parent_header = Dict({})
 
     def set_parent(self, parent):
         """Set the parent for outbound messages."""
         self.parent_header = extract_header(parent)
 
     def start_displayhook(self):
         self.msg = self.session.msg(u'pyout', {}, parent=self.parent_header)
 
     def write_output_prompt(self):
         """Write the output prompt."""
         if self.do_full_cache:
             self.msg['content']['output_sep'] = self.output_sep
             self.msg['content']['prompt_string'] = str(self.prompt_out)
             self.msg['content']['prompt_number'] = self.prompt_count
             self.msg['content']['output_sep2'] = self.output_sep2
 
     def write_result_repr(self, result_repr):
         self.msg['content']['data'] = result_repr
 
     def finish_displayhook(self):
         """Finish up all displayhook activities."""
         self.pub_socket.send_json(self.msg)
         self.msg = None
 
 
 class ZMQInteractiveShell(InteractiveShell):
     """A subclass of InteractiveShell for ZMQ."""
 
     displayhook_class = Type(ZMQDisplayHook)
 
     def system(self, cmd):
-        cmd = self.var_expand(cmd, depth=2)
+        cmd = self.var_expand(cmd, depth=2).strip()
+        
+        # Runnning a bacgkrounded process from within the gui isn't supported
+        # because we do p.wait() at the end.  So instead of silently blocking
+        # we simply refuse to run in this mode, to avoid surprising the user.
+        if cmd.endswith('&'):
+            raise OSError("Background processes not supported.")
+        
         sys.stdout.flush()
         sys.stderr.flush()
         p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
         for line in p.stdout.read().split('\n'):
             if len(line) > 0:
-                print line
+                print(line)
         for line in p.stderr.read().split('\n'):
             if len(line) > 0:
-                print line
+                print(line, file=sys.stderr)
         p.wait()
 
     def init_io(self):
         # This will just use sys.stdout and sys.stderr. If you want to
         # override sys.stdout and sys.stderr themselves, you need to do that
         # *before* instantiating this class, because Term holds onto 
         # references to the underlying streams.
         import IPython.utils.io
         Term = IPython.utils.io.IOTerm()
         IPython.utils.io.Term = Term
 
     def magic_edit(self,parameter_s='',last_call=['','']):
         """Bring up an editor and execute the resulting code.
 
         Usage:
           %edit [options] [args]
 
         %edit runs IPython's editor hook.  The default version of this hook is
         set to call the __IPYTHON__.rc.editor command.  This is read from your
         environment variable $EDITOR.  If this isn't found, it will default to
         vi under Linux/Unix and to notepad under Windows.  See the end of this
         docstring for how to change the editor hook.
 
         You can also set the value of this editor via the command line option
         '-editor' or in your ipythonrc file. This is useful if you wish to use
         specifically for IPython an editor different from your typical default
         (and for Windows users who typically don't set environment variables).
 
         This command allows you to conveniently edit multi-line code right in
         your IPython session.
         
         If called without arguments, %edit opens up an empty editor with a
         temporary file and will execute the contents of this file when you
         close it (don't forget to save it!).
 
 
         Options:
 
         -n <number>: open the editor at a specified line number.  By default,
         the IPython editor hook uses the unix syntax 'editor +N filename', but
         you can configure this by providing your own modified hook if your
         favorite editor supports line-number specifications with a different
         syntax.
         
         -p: this will call the editor with the same data as the previous time
         it was used, regardless of how long ago (in your current session) it
         was.
 
         -r: use 'raw' input.  This option only applies to input taken from the
         user's history.  By default, the 'processed' history is used, so that
         magics are loaded in their transformed version to valid Python.  If
         this option is given, the raw input as typed as the command line is
         used instead.  When you exit the editor, it will be executed by
         IPython's own processor.
         
         -x: do not execute the edited code immediately upon exit. This is
         mainly useful if you are editing programs which need to be called with
         command line arguments, which you can then do using %run.
 
 
         Arguments:
 
         If arguments are given, the following possibilites exist:
 
         - The arguments are numbers or pairs of colon-separated numbers (like
         1 4:8 9). These are interpreted as lines of previous input to be
         loaded into the editor. The syntax is the same of the %macro command.
 
         - If the argument doesn't start with a number, it is evaluated as a
         variable and its contents loaded into the editor. You can thus edit
         any string which contains python code (including the result of
         previous edits).
 
         - If the argument is the name of an object (other than a string),
         IPython will try to locate the file where it was defined and open the
         editor at the point where it is defined. You can use `%edit function`
         to load an editor exactly at the point where 'function' is defined,
         edit it and have the file be executed automatically.
 
         If the object is a macro (see %macro for details), this opens up your
         specified editor with a temporary file containing the macro's data.
         Upon exit, the macro is reloaded with the contents of the file.
 
         Note: opening at an exact line is only supported under Unix, and some
         editors (like kedit and gedit up to Gnome 2.8) do not understand the
         '+NUMBER' parameter necessary for this feature. Good editors like
         (X)Emacs, vi, jed, pico and joe all do.
 
         - If the argument is not found as a variable, IPython will look for a
         file with that name (adding .py if necessary) and load it into the
         editor. It will execute its contents with execfile() when you exit,
         loading any code in the file into your interactive namespace.
 
         After executing your code, %edit will return as output the code you
         typed in the editor (except when it was an existing file). This way
         you can reload the code in further invocations of %edit as a variable,
         via _<NUMBER> or Out[<NUMBER>], where <NUMBER> is the prompt number of
         the output.
 
         Note that %edit is also available through the alias %ed.
 
         This is an example of creating a simple function inside the editor and
         then modifying it. First, start up the editor:
 
         In [1]: ed
         Editing... done. Executing edited code...
         Out[1]: 'def foo():n    print "foo() was defined in an editing session"n'
 
         We can then call the function foo():
         
         In [2]: foo()
         foo() was defined in an editing session
 
         Now we edit foo.  IPython automatically loads the editor with the
         (temporary) file where foo() was previously defined:
         
         In [3]: ed foo
         Editing... done. Executing edited code...
 
         And if we call foo() again we get the modified version:
         
         In [4]: foo()
         foo() has now been changed!
 
         Here is an example of how to edit a code snippet successive
         times. First we call the editor:
 
         In [5]: ed
         Editing... done. Executing edited code...
         hello
         Out[5]: "print 'hello'n"
 
         Now we call it again with the previous output (stored in _):
 
         In [6]: ed _
         Editing... done. Executing edited code...
         hello world
         Out[6]: "print 'hello world'n"
 
         Now we call it with the output #8 (stored in _8, also as Out[8]):
 
         In [7]: ed _8
         Editing... done. Executing edited code...
         hello again
         Out[7]: "print 'hello again'n"
 
 
         Changing the default editor hook:
 
         If you wish to write your own editor hook, you can put it in a
         configuration file which you load at startup time.  The default hook
         is defined in the IPython.core.hooks module, and you can use that as a
         starting example for further modifications.  That file also has
         general instructions on how to set a new hook for use once you've
         defined it."""
         
         # FIXME: This function has become a convoluted mess.  It needs a
         # ground-up rewrite with clean, simple logic.
 
         def make_filename(arg):
             "Make a filename from the given args"
             try:
                 filename = get_py_filename(arg)
             except IOError:
                 if args.endswith('.py'):
                     filename = arg
                 else:
                     filename = None
             return filename
 
         # custom exceptions
         class DataIsObject(Exception): pass
 
         opts,args = self.parse_options(parameter_s,'prn:')
         # Set a few locals from the options for convenience:
         opts_p = opts.has_key('p')
         opts_r = opts.has_key('r')
         
         # Default line number value
         lineno = opts.get('n',None)
         if lineno is not None:
             try:
                 lineno = int(lineno)
             except:
                 warn("The -n argument must be an integer.")
                 return
 
         if opts_p:
             args = '_%s' % last_call[0]
             if not self.shell.user_ns.has_key(args):
                 args = last_call[1]
             
         # use last_call to remember the state of the previous call, but don't
         # let it be clobbered by successive '-p' calls.
         try:
             last_call[0] = self.shell.displayhook.prompt_count
             if not opts_p:
                 last_call[1] = parameter_s
         except:
             pass
 
         # by default this is done with temp files, except when the given
         # arg is a filename
         use_temp = 1
 
         if re.match(r'\d',args):
             # Mode where user specifies ranges of lines, like in %macro.
             # This means that you can't edit files whose names begin with
             # numbers this way. Tough.
             ranges = args.split()
             data = ''.join(self.extract_input_slices(ranges,opts_r))
         elif args.endswith('.py'):
             filename = make_filename(args)
             data = ''
             use_temp = 0
         elif args:
             try:
                 # Load the parameter given as a variable. If not a string,
                 # process it as an object instead (below)
 
                 #print '*** args',args,'type',type(args)  # dbg
                 data = eval(args,self.shell.user_ns)
                 if not type(data) in StringTypes:
                     raise DataIsObject
 
             except (NameError,SyntaxError):
                 # given argument is not a variable, try as a filename
                 filename = make_filename(args)
                 if filename is None:
                     warn("Argument given (%s) can't be found as a variable "
                          "or as a filename." % args)
                     return
 
                 data = ''
                 use_temp = 0
             except DataIsObject:
 
                 # macros have a special edit function
                 if isinstance(data,Macro):
                     self._edit_macro(args,data)
                     return
                                 
                 # For objects, try to edit the file where they are defined
                 try:
                     filename = inspect.getabsfile(data)
                     if 'fakemodule' in filename.lower() and inspect.isclass(data):                     
                         # class created by %edit? Try to find source
                         # by looking for method definitions instead, the
                         # __module__ in those classes is FakeModule.
                         attrs = [getattr(data, aname) for aname in dir(data)]
                         for attr in attrs:
                             if not inspect.ismethod(attr):
                                 continue
                             filename = inspect.getabsfile(attr)
                             if filename and 'fakemodule' not in filename.lower():
                                 # change the attribute to be the edit target instead
                                 data = attr 
                                 break
                     
                     datafile = 1
                 except TypeError:
                     filename = make_filename(args)
                     datafile = 1
                     warn('Could not find file where `%s` is defined.\n'
                          'Opening a file named `%s`' % (args,filename))
                 # Now, make sure we can actually read the source (if it was in
                 # a temp file it's gone by now).
                 if datafile:
                     try:
                         if lineno is None:
                             lineno = inspect.getsourcelines(data)[1]
                     except IOError:
                         filename = make_filename(args)
                         if filename is None:
                             warn('The file `%s` where `%s` was defined cannot '
                                  'be read.' % (filename,data))
                             return
                 use_temp = 0
         else:
             data = ''
 
         if use_temp:
             filename = self.shell.mktempfile(data)
-            print 'IPython will make a temporary file named:',filename
+            print('IPython will make a temporary file named:', filename)
+
+        # Make sure we send to the client an absolute path, in case the working
+        # directory of client and kernel don't match
+        filename = os.path.abspath(filename)
 
         payload = {
             'source' : 'IPython.zmq.zmqshell.ZMQInteractiveShell.edit_magic',
             'filename' : filename,
             'line_number' : lineno
         }
         self.payload_manager.write_payload(payload)
 
 
     def _showtraceback(self, etype, evalue, stb):
 
         exc_content = {
             u'status' : u'error',
             u'traceback' : stb,
             u'ename' : unicode(etype.__name__),
             u'evalue' : unicode(evalue)
         }
 
         dh = self.displayhook
         exc_msg = dh.session.msg(u'pyerr', exc_content, dh.parent_header)
         # Send exception info over pub socket for other clients than the caller
         # to pick up
         dh.pub_socket.send_json(exc_msg)
 
         # FIXME - Hack: store exception info in shell object.  Right now, the
         # caller is reading this info after the fact, we need to fix this logic
         # to remove this hack.
         self._reply_content = exc_content
         # /FIXME
         
         return exc_content
 
     def runlines(self, lines, clean=False):
         return InteractiveShell.runlines(self, lines, clean)
 
 InteractiveShellABC.register(ZMQInteractiveShell)

docs/Makefile

@@ -1,104 +1,104 b''
 # Makefile for Sphinx documentation
 #
 
 # You can set these variables from the command line.
 SPHINXOPTS    =
 SPHINXBUILD   = sphinx-build
 PAPER         =
 SRCDIR        = source
 
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4
 PAPEROPT_letter = -D latex_paper_size=letter
 ALLSPHINXOPTS   = -d build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) $(SRCDIR)
 
 .PHONY: help clean html web pickle htmlhelp latex changes linkcheck api
 
 default: html
 
 help:
 	@echo "Please use \`make <target>' where <target> is one of"
 	@echo "  html      to make standalone HTML files"
 	@echo "  pickle    to make pickle files (usable by e.g. sphinx-web)"
 	@echo "  htmlhelp  to make HTML files and a HTML help project"
 	@echo "  latex     to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
 	@echo "  changes   to make an overview over all changed/added/deprecated items"
 	@echo "  linkcheck to check all external links for integrity"
 	@echo
 	@echo "Compound utility targets:"
 	@echo "pdf          latex and then runs the PDF generation"
 	@echo "all          html and pdf"
 	@echo "dist         all, and then puts the results in dist/"
 	@echo "gitwash-update update git workflow from source repo"
 
 clean:
 	-rm -rf build/* dist/* $(SRCDIR)/api/generated
 
 pdf: latex
 	cd build/latex && make all-pdf
 
 all: html pdf
 
 dist: all
 	mkdir -p dist
 	rm -rf dist/*
 	ln build/latex/ipython.pdf dist/
 	cp -al build/html dist/
 	@echo "Build finished.  Final docs are in dist/"
 
 html: api
 	mkdir -p build/html build/doctrees
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) build/html
 	@echo
 	@echo "Build finished. The HTML pages are in build/html."
 
 api: source/api/generated/gen.txt
 
 source/api/generated/gen.txt:
 	python autogen_api.py
 	@echo "Build API docs finished."
 
 pickle:
 	mkdir -p build/pickle build/doctrees
 	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) build/pickle
 	@echo
 	@echo "Build finished; now you can process the pickle files or run"
 	@echo "  sphinx-web build/pickle"
 	@echo "to start the sphinx-web server."
 
 web: pickle
 
 htmlhelp:
 	mkdir -p build/htmlhelp build/doctrees
 	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) build/htmlhelp
 	@echo
 	@echo "Build finished; now you can run HTML Help Workshop with the" \
 	      ".hhp project file in build/htmlhelp."
 
 latex: api
 	mkdir -p build/latex build/doctrees
 	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) build/latex
 	@echo
 	@echo "Build finished; the LaTeX files are in build/latex."
 	@echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \
 	      "run these through (pdf)latex."
 
 changes:
 	mkdir -p build/changes build/doctrees
 	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) build/changes
 	@echo
 	@echo "The overview file is in build/changes."
 
 linkcheck:
 	mkdir -p build/linkcheck build/doctrees
 	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) build/linkcheck
 	@echo
 	@echo "Link check complete; look for any errors in the above output " \
 	      "or in build/linkcheck/output.txt."
 
 gitwash-update:
 	python ../tools/gitwash_dumper.py source/development ipython
 	cd source/development/gitwash && rename 's/.rst/.txt/' *.rst
 
 nightly: dist
-	rsync -avH --delete dist/ ipython:www/doc/nightly
  No newline at end of file
+	rsync -avH --delete dist/ ipython:www/doc/nightly

docs/source/development/messaging.txt

@@ -1,587 +1,708 b''
 .. _messaging:
 
 ======================
  Messaging in IPython
 ======================
 
 
 Introduction
 ============
 
 This document explains the basic communications design and messaging
 specification for how the various IPython objects interact over a network
 transport.  The current implementation uses the ZeroMQ_ library for messaging
 within and between hosts.
 
 .. Note::
 
    This document should be considered the authoritative description of the
    IPython messaging protocol, and all developers are strongly encouraged to
    keep it updated as the implementation evolves, so that we have a single
    common reference for all protocol details.
    
 The basic design is explained in the following diagram:
 
 .. image:: frontend-kernel.png
    :width: 450px
    :alt: IPython kernel/frontend messaging architecture.
    :align: center
    :target: ../_images/frontend-kernel.png
 
 A single kernel can be simultaneously connected to one or more frontends.  The
 kernel has three sockets that serve the following functions:
 
 1. REQ: this socket is connected to a *single* frontend at a time, and it allows
    the kernel to request input from a frontend when :func:`raw_input` is called.
    The frontend holding the matching REP socket acts as a 'virtual keyboard'
    for the kernel while this communication is happening (illustrated in the
    figure by the black outline around the central keyboard).  In practice,
    frontends may display such kernel requests using a special input widget or
    otherwise indicating that the user is to type input for the kernel instead
    of normal commands in the frontend.
 
 2. XREP: this single sockets allows multiple incoming connections from
    frontends, and this is the socket where requests for code execution, object
    information, prompts, etc. are made to the kernel by any frontend.  The
    communication on this socket is a sequence of request/reply actions from
    each frontend and the kernel.
 
 3. PUB: this socket is the 'broadcast channel' where the kernel publishes all
    side effects (stdout, stderr, etc.) as well as the requests coming from any
    client over the XREP socket and its own requests on the REP socket.  There
    are a number of actions in Python which generate side effects: :func:`print`
    writes to ``sys.stdout``, errors generate tracebacks, etc.  Additionally, in
    a multi-client scenario, we want all frontends to be able to know what each
    other has sent to the kernel (this can be useful in collaborative scenarios,
    for example).  This socket allows both side effects and the information
    about communications taking place with one client over the XREQ/XREP channel
    to be made available to all clients in a uniform manner.
 
    All messages are tagged with enough information (details below) for clients
    to know which messages come from their own interaction with the kernel and
    which ones are from other clients, so they can display each type
    appropriately.
 
 The actual format of the messages allowed on each of these channels is
 specified below.  Messages are dicts of dicts with string keys and values that
 are reasonably representable in JSON.  Our current implementation uses JSON
 explicitly as its message format, but this shouldn't be considered a permanent
 feature.  As we've discovered that JSON has non-trivial performance issues due
 to excessive copying, we may in the future move to a pure pickle-based raw
 message format.  However, it should be possible to easily convert from the raw
 objects to JSON, since we may have non-python clients (e.g. a web frontend).
 As long as it's easy to make a JSON version of the objects that is a faithful
 representation of all the data, we can communicate with such clients.
 
 .. Note::
 
    Not all of these have yet been fully fleshed out, but the key ones are, see
    kernel and frontend files for actual implementation details.
 
    
 Python functional API
 =====================
 
 As messages are dicts, they map naturally to a ``func(**kw)`` call form.  We
 should develop, at a few key points, functional forms of all the requests that
 take arguments in this manner and automatically construct the necessary dict
 for sending.
 
 
 General Message Format
 ======================
 
 All messages send or received by any IPython process should have the following
 generic structure::
     
     {
       # The message header contains a pair of unique identifiers for the
       # originating session and the actual message id, in addition to the
       # username for the process that generated the message.  This is useful in
       # collaborative settings where multiple users may be interacting with the
       # same kernel simultaneously, so that frontends can label the various
       # messages in a meaningful way.
       'header' : { 'msg_id' : uuid,
                    'username' : str,
            'session' : uuid
          },
 
       # In a chain of messages, the header from the parent is copied so that
       # clients can track where messages come from.
       'parent_header' : dict,
 
       # All recognized message type strings are listed below.
       'msg_type' : str,
 
       # The actual content of the message must be a dict, whose structure
       # depends on the message type.x
       'content' : dict,
     }
 
 For each message type, the actual content will differ and all existing message
 types are specified in what follows of this document.
 
 
 Messages on the XREP/XREQ socket
 ================================
 
 .. _execute:
 
 Execute
 -------
 
-The execution request contains a single string, but this may be a multiline
-string.  The kernel is responsible for splitting this into possibly more than
-one block and deciding whether to compile these in 'single' or 'exec' mode.
-We're still sorting out this policy.  The current inputsplitter is capable of
-splitting the input for blocks that can all be run as 'single', but in the long
-run it may prove cleaner to only use 'single' mode for truly single-line
-inputs, and run all multiline input in 'exec' mode.  This would preserve the
-natural behavior of single-line inputs while allowing long cells to behave more
-likea a script.  This design will be refined as we complete the implementation.
+This message type is used by frontends to ask the kernel to execute code on
+behalf of the user, in a namespace reserved to the user's variables (and thus
+separate from the kernel's own internal code and variables).
 
 Message type: ``execute_request``::
 
     content = {
         # Source code to be executed by the kernel, one or more lines.
     'code' : str,
 
     # A boolean flag which, if True, signals the kernel to execute this
     # code as quietly as possible.  This means that the kernel will compile
-    # the code with 'exec' instead of 'single' (so sys.displayhook will not
-    # fire), and will *not*:
+    # the code witIPython/core/tests/h 'exec' instead of 'single' (so
+    # sys.displayhook will not fire), and will *not*:
     #   - broadcast exceptions on the PUB socket
     #   - do any logging
     #   - populate any history
+    #
     # The default is False.
     'silent' : bool,
+
+    # A list of variable names from the user's namespace to be retrieved.  What
+    # returns is a JSON string of the variable's repr(), not a python object.
+    'user_variables' : list,
+
+    # Similarly, a dict mapping names to expressions to be evaluated in the
+    # user's dict.
+    'user_expressions' : dict,
     }
 
-Upon execution, the kernel *always* sends a reply, with a status code
-indicating what happened and additional data depending on the outcome.
+The ``code`` field contains a single string, but this may be a multiline
+string.  The kernel is responsible for splitting this into possibly more than
+one block and deciding whether to compile these in 'single' or 'exec' mode.
+We're still sorting out this policy.  The current inputsplitter is capable of
+splitting the input for blocks that can all be run as 'single', but in the long
+run it may prove cleaner to only use 'single' mode for truly single-line
+inputs, and run all multiline input in 'exec' mode.  This would preserve the
+natural behavior of single-line inputs while allowing long cells to behave more
+likea a script.  This design will be refined as we complete the implementation.
+
+The ``user_`` fields deserve a detailed explanation.  In the past, IPython had
+the notion of a prompt string that allowed arbitrary code to be evaluated, and
+this was put to good use by many in creating prompts that displayed system
+status, path information, and even more esoteric uses like remote instrument
+status aqcuired over the network.  But now that IPython has a clean separation
+between the kernel and the clients, the notion of embedding 'prompt'
+maninpulations into the kernel itself feels awkward.  Prompts should be a
+frontend-side feature, and it should be even possible for different frontends
+to display different prompts while interacting with the same kernel.
+
+We have therefore abandoned the idea of a 'prompt string' to be evaluated by
+the kernel, and instead provide the ability to retrieve from the user's
+namespace information after the execution of the main ``code``, with two fields
+of the execution request:
+
+- ``user_variables``: If only variables from the user's namespace are needed, a
+  list of variable names can be passed and a dict with these names as keys and
+  their :func:`repr()` as values will be returned.
+
+- ``user_expressions``: For more complex expressions that require function
+  evaluations, a dict can be provided with string keys and arbitrary python
+  expressions as values.  The return message will contain also a dict with the
+  same keys and the :func:`repr()` of the evaluated expressions as value.
+
+With this information, frontends can display any status information they wish
+in the form that best suits each frontend (a status line, a popup, inline for a
+terminal, etc).
+
+.. Note::
+
+   In order to obtain the current execution counter for the purposes of
+   displaying input prompts, frontends simply make an execution request with an
+   empty code string and ``silent=True``.
+
+Execution semantics
+    Upon completion of the execution request, the kernel *always* sends a
+    reply, with a status code indicating what happened and additional data
+    depending on the outcome.
+
+    The ``code`` field is executed first, and then the ``user_variables`` and
+    ``user_expressions`` are computed.  This ensures that any error in the
+    latter don't harm the main code execution.
+
+    Any error in retrieving the ``user_variables`` or evaluating the
+    ``user_expressions`` will result in a simple error message in the return
+    fields of the form::
+
+       [ERROR] ExceptionType: Exception message
+
+    The user can simply send the same variable name or expression for
+    evaluation to see a regular traceback.
+
+Execution counter (old prompt number)
+    The kernel has a single, monotonically increasing counter of all execution
+    requests that are made with ``silent=False``.  This counter is used to
+    populate the ``In[n]``, ``Out[n]`` and ``_n`` variables, so clients will
+    likely want to display it in some form to the user, which will typically
+    (but not necessarily) be done in the prompts.  The value of this counter
+    will be returned as the ``execution_count`` field of all ``execute_reply```
+    messages.
 
 Message type: ``execute_reply``::
 
     content = {
       # One of: 'ok' OR 'error' OR 'abort'
       'status' : str,
 
-      # This has the same structure as the output of a prompt request, but is
-      # for the client to set up the *next* prompt (with identical limitations
-      # to a prompt request)
-      'next_prompt' : {
-            'prompt_string' : str,
-            'prompt_number' : int,
-            'input_sep'     : str
-      },
-
-      # The prompt number of the actual execution for this code, which may be
-      # different from the one used when the code was typed, which was the
-      # 'next_prompt' field of the *previous* request.  They will differ in the
-      # case where there is more than one client talking simultaneously to a
-      # kernel, since the numbers can go out of sync.  GUI clients can use this
-      # to correct the previously written number in-place, terminal ones may
-      # re-print a corrected one if desired.
-      'prompt_number' : int,
+      # The global kernel counter that increases by one with each non-silent
+      # executed request.  This will typically be used by clients to display
+      # prompt numbers to the user.  If the request was a silent one, this will
+      # be the current value of the counter in the kernel.
+      'execution_count' : int,
+
+      # If the state_template was provided, this will contain the evaluated
+      # form of the template.
+      'state' : str,
     }
 
 When status is 'ok', the following extra fields are present::
 
     {
-      # The kernel will often transform the input provided to it.  This
-      # contains the transformed code, which is what was actually executed.
+      # The kernel will often transform the input provided to it.  If the
+      # '---->' transform had been applied, this is filled, otherwise it's the
+      # empty string.  So transformations like magics don't appear here, only
+      # autocall ones.
+    
       'transformed_code' : str,
 
       # The execution payload is a dict with string keys that may have been
       # produced by the code being executed.  It is retrieved by the kernel at
       # the end of the execution and sent back to the front end, which can take
       # action on it as needed.  See main text for further details.
       'payload' : dict,
     }
 
 .. admonition:: Execution payloads
     
    The notion of an 'execution payload' is different from a return value of a
    given set of code, which normally is just displayed on the pyout stream
    through the PUB socket.  The idea of a payload is to allow special types of
    code, typically magics, to populate a data container in the IPython kernel
    that will be shipped back to the caller via this channel.  The kernel will
    have an API for this, probably something along the lines of::
 
        ip.exec_payload_add(key, value)
 
    though this API is still in the design stages.  The data returned in this
    payload will allow frontends to present special views of what just happened.
 
    
 When status is 'error', the following extra fields are present::
 
     {
       'exc_name' : str,   # Exception name, as a string
       'exc_value' : str,  # Exception value, as a string
 
       # The traceback will contain a list of frames, represented each as a
       # string.  For now we'll stick to the existing design of ultraTB, which
       # controls exception level of detail statefully.  But eventually we'll
       # want to grow into a model where more information is collected and
       # packed into the traceback object, with clients deciding how little or
       # how much of it to unpack.  But for now, let's start with a simple list
       # of strings, since that requires only minimal changes to ultratb as
       # written.
       'traceback' : list,
     }
 
 
 When status is 'abort', there are for now no additional data fields.  This
 happens when the kernel was interrupted by a signal.
 
+Kernel attribute access
+-----------------------
 
-Prompt
-------
+While this protocol does not specify full RPC access to arbitrary methods of
+the kernel object, the kernel does allow read (and in some cases write) access
+to certain attributes.
 
-A simple request for a current prompt string.
+The policy for which attributes can be read is: any attribute of the kernel, or
+its sub-objects, that belongs to a :class:`Configurable` object and has been
+declared at the class-level with Traits validation, is in principle accessible
+as long as its name does not begin with a leading underscore.  The attribute
+itself will have metadata indicating whether it allows remote read and/or write
+access.  The message spec follows for attribute read and write requests.
 
-Message type: ``prompt_request``::
+Message type: ``getattr_request``::
 
-    content = {}
+    content = {
+        # The (possibly dotted) name of the attribute
+	'name' : str,
+    }
+
+When a ``getattr_request`` fails, there are two possible error types:
+
+- AttributeError: this type of error was raised when trying to access the
+  given name by the kernel itself.  This means that the attribute likely
+  doesn't exist.
+
+- AccessError: the attribute exists but its value is not readable remotely.
 
-In the reply, the prompt string comes back with the prompt number placeholder
-*unevaluated*.  The message format is:
-    
-Message type: ``prompt_reply``::
+
+Message type: ``getattr_reply``::
+
+    content = {
+        # One of ['ok', 'AttributeError', 'AccessError'].
+        'status' : str,
+	# If status is 'ok', a JSON object.
+	'value' : object,
+    }
+
+Message type: ``setattr_request``::
 
     content = {
-      'prompt_string' : str,
-      'prompt_number' : int,
-      'input_sep'     : str
+        # The (possibly dotted) name of the attribute
+	'name' : str,
+	
+	# A JSON-encoded object, that will be validated by the Traits
+	# information in the kernel
+	'value' : object,
     }
 
-Clients can produce a prompt with ``prompt_string.format(prompt_number)``, but
-they should be aware that the actual prompt number for that input could change
-later, in the case where multiple clients are interacting with a single
-kernel. 
+When a ``setattr_request`` fails, there are also two possible error types with
+similar meanings  as those of the ``getattr_request`` case, but for writing.
+    
+Message type: ``setattr_reply``::
+
+    content = {
+        # One of ['ok', 'AttributeError', 'AccessError'].
+        'status' : str,
+    }
 
 
 Object information
 ------------------
 
 One of IPython's most used capabilities is the introspection of Python objects
 in the user's namespace, typically invoked via the ``?`` and ``??`` characters
 (which in reality are shorthands for the ``%pinfo`` magic).  This is used often
 enough that it warrants an explicit message type, especially because frontends
 may want to get object information in response to user keystrokes (like Tab or
 F1) besides from the user explicitly typing code like ``x??``.
 
 Message type: ``object_info_request``::
 
     content = {
         # The (possibly dotted) name of the object to be searched in all
-    # relevant namespaces
-    'name' : str,
+	# relevant namespaces
+        'name' : str,
 
-    # The level of detail desired.  The default (0) is equivalent to typing
-    # 'x?' at the prompt, 1 is equivalent to 'x??'.
-    'detail_level' : int,
+    	# The level of detail desired.  The default (0) is equivalent to typing
+	# 'x?' at the prompt, 1 is equivalent to 'x??'.
+	'detail_level' : int,
     }
 
 The returned information will be a dictionary with keys very similar to the
 field names that IPython prints at the terminal.
     
 Message type: ``object_info_reply``::
 
     content = {
         # Flags for magics and system aliases
     'ismagic' : bool,
     'isalias' : bool,
 
     # The name of the namespace where the object was found ('builtin',
     # 'magics', 'alias', 'interactive', etc.)
     'namespace' : str,
 
     # The type name will be type.__name__ for normal Python objects, but it
     # can also be a string like 'Magic function' or 'System alias'
     'type_name' : str,
 
     'string_form' : str,
 
     # For objects with a __class__ attribute this will be set
     'base_class' : str,
 
     # For objects with a __len__ attribute this will be set
     'length' : int,
 
     # If the object is a function, class or method whose file we can find,
     # we give its full path
     'file' : str,
 
     # For pure Python callable objects, we can reconstruct the object
-    # definition line which provides its call signature
+    # definition line which provides its call signature.  For convenience this
+    # is returned as a single 'definition' field, but below the raw parts that
+    # compose it are also returned as the argspec field.
     'definition' : str,
 
+    # The individual parts that together form the definition string.  Clients
+    # with rich display capabilities may use this to provide a richer and more
+    # precise representation of the definition line (e.g. by highlighting
+    # arguments based on the user's cursor position).  For non-callable
+    # objects, this field is empty.
+    'argspec' : { # The names of all the arguments
+                  args : list,
+		  # The name of the varargs (*args), if any
+                  varargs : str,
+		  # The name of the varkw (**kw), if any
+		  varkw : str,
+		  # The values (as strings) of all default arguments.  Note
+		  # that these must be matched *in reverse* with the 'args'
+		  # list above, since the first positional args have no default
+		  # value at all.
+		  func_defaults : list,
+		  },
+
     # For instances, provide the constructor signature (the definition of
     # the __init__ method):
     'init_definition' : str,
     
     # Docstrings: for any object (function, method, module, package) with a
     # docstring, we show it.  But in addition, we may provide additional
     # docstrings.  For example, for instances we will show the constructor
     # and class docstrings as well, if available.
     'docstring' : str,
 
     # For instances, provide the constructor and class docstrings
     'init_docstring' : str,
     'class_docstring' : str,
 
     # If detail_level was 1, we also try to find the source code that
     # defines the object, if possible.  The string 'None' will indicate
     # that no source was found.
     'source' : str,
     }
 
     
 Complete
 --------
 
 Message type: ``complete_request``::
 
     content = {
         # The text to be completed, such as 'a.is'
     'text' : str,
 
     # The full line, such as 'print a.is'.  This allows completers to
     # make decisions that may require information about more than just the
     # current word.
     'line' : str,
 
     # The entire block of text where the line is.  This may be useful in the
     # case of multiline completions where more context may be needed.  Note: if
     # in practice this field proves unnecessary, remove it to lighten the
     # messages.
     
     'block' : str,
 
     # The position of the cursor where the user hit 'TAB' on the line.
     'cursor_pos' : int,
     }
 
 Message type: ``complete_reply``::
 
     content = {
         # The list of all matches to the completion request, such as
     # ['a.isalnum', 'a.isalpha'] for the above example.
     'matches' : list
     }
 
     
 History
 -------
 
 For clients to explicitly request history from a kernel.  The kernel has all
 the actual execution history stored in a single location, so clients can
 request it from the kernel when needed.
 
 Message type: ``history_request``::
 
     content = {
     
       # If True, also return output history in the resulting dict.
       'output' : bool,
 
       # If True, return the raw input history, else the transformed input.
       'raw' : bool,
 
       # This parameter can be one of: A number,  a pair of numbers, None
       # If not given, last 40 are returned.
       #  - number n: return the last n entries.
       #  - pair n1, n2: return entries in the range(n1, n2).
       #  - None: return all history
       'index' : n or (n1, n2) or None,
     }
 
 Message type: ``history_reply``::
 
     content = {
       # A dict with prompt numbers as keys and either (input, output) or input
       # as the value depending on whether output was True or False,
       # respectively.
       'history' : dict,
     }
+    
 Messages on the PUB/SUB socket
 ==============================
 
 Streams (stdout,  stderr, etc)
 ------------------------------
 
 Message type: ``stream``::
 
     content = {
         # The name of the stream is one of 'stdin', 'stdout', 'stderr'
     'name' : str,
     
     # The data is an arbitrary string to be written to that stream
     'data' : str,
     }
 
 When a kernel receives a raw_input call, it should also broadcast it on the pub
 socket with the names 'stdin' and 'stdin_reply'.  This will allow other clients
 to monitor/display kernel interactions and possibly replay them to their user
 or otherwise expose them.
 
 Python inputs
 -------------
 
 These messages are the re-broadcast of the ``execute_request``.
 
 Message type: ``pyin``::
 
     content = {
         # Source code to be executed, one or more lines
     'code' : str
     }
 
 Python outputs
 --------------
 
 When Python produces output from code that has been compiled in with the
 'single' flag to :func:`compile`, any expression that produces a value (such as
 ``1+1``) is passed to ``sys.displayhook``, which is a callable that can do with
 this value whatever it wants.  The default behavior of ``sys.displayhook`` in
 the Python interactive prompt is to print to ``sys.stdout`` the :func:`repr` of
 the value as long as it is not ``None`` (which isn't printed at all).  In our
 case, the kernel instantiates as ``sys.displayhook`` an object which has
 similar behavior, but which instead of printing to stdout, broadcasts these
 values as ``pyout`` messages for clients to display appropriately.
 
 Message type: ``pyout``::
 
     content = {
         # The data is typically the repr() of the object.
     'data' : str,
     
-    # The prompt number for this execution is also provided so that clients
-    # can display it, since IPython automatically creates variables called
-    # _N (for prompt N).
-    'prompt_number' : int,
+    # The counter for this execution is also provided so that clients can
+    # display it, since IPython automatically creates variables called _N (for
+    # prompt N).
+    'execution_count' : int,
     }
     
 Python errors
 -------------
 
 When an error occurs during code execution
 
 Message type: ``pyerr``::
 
     content = {
        # Similar content to the execute_reply messages for the 'error' case,
        # except the 'status' field is omitted.
     }
 
 Kernel crashes
 --------------
 
 When the kernel has an unexpected exception, caught by the last-resort
 sys.excepthook, we should broadcast the crash handler's output before exiting.
 This will allow clients to notice that a kernel died, inform the user and
 propose further actions.
 
 Message type: ``crash``::
 
     content = {
        # Similarly to the 'error' case for execute_reply messages, this will
        # contain exc_name, exc_type and traceback fields.
 
        # An additional field with supplementary information such as where to
        # send the crash message
        'info' : str,
     }
 
 
 Future ideas
 ------------
     
 Other potential message types, currently unimplemented, listed below as ideas.
     
 Message type: ``file``::
 
     content = {
     'path' : 'cool.jpg',
     'mimetype' : str,
     'data' : str,
     }
 
     
 Messages on the REQ/REP socket
 ==============================
 
 This is a socket that goes in the opposite direction: from the kernel to a
 *single* frontend, and its purpose is to allow ``raw_input`` and similar
 operations that read from ``sys.stdin`` on the kernel to be fulfilled by the
 client.  For now we will keep these messages as simple as possible, since they
 basically only mean to convey the ``raw_input(prompt)`` call.
 
 Message type: ``input_request``::
 
     content = { 'prompt' : str }
 
 Message type: ``input_reply``::
 
     content = { 'value' : str }
     
 .. Note::
 
    We do not explicitly try to forward the raw ``sys.stdin`` object, because in
    practice the kernel should behave like an interactive program.  When a
    program is opened on the console, the keyboard effectively takes over the
    ``stdin`` file descriptor, and it can't be used for raw reading anymore.
    Since the IPython kernel effectively behaves like a console program (albeit
    one whose "keyboard" is actually living in a separate process and
    transported over the zmq connection), raw ``stdin`` isn't expected to be
    available.
 
    
 Heartbeat for kernels
 =====================
 
 Initially we had considered using messages like those above over ZMQ for a
 kernel 'heartbeat' (a way to detect quickly and reliably whether a kernel is
 alive at all, even if it may be busy executing user code).  But this has the
 problem that if the kernel is locked inside extension code, it wouldn't execute
 the python heartbeat code.  But it turns out that we can implement a basic
 heartbeat with pure ZMQ, without using any Python messaging at all.
 
 The monitor sends out a single zmq message (right now, it is a str of the
 monitor's lifetime in seconds), and gets the same message right back, prefixed
 with the zmq identity of the XREQ socket in the heartbeat process. This can be
 a uuid, or even a full message, but there doesn't seem to be a need for packing
 up a message when the sender and receiver are the exact same Python object.
 
 The model is this::
 
     monitor.send(str(self.lifetime)) # '1.2345678910'
 
 and the monitor receives some number of messages of the form::
 
     ['uuid-abcd-dead-beef', '1.2345678910']
 
 where the first part is the zmq.IDENTITY of the heart's XREQ on the engine, and
 the rest is the message sent by the monitor.  No Python code ever has any
 access to the message between the monitor's send, and the monitor's recv.
 
 
 ToDo
 ====
 
 Missing things include:
 
 * Important: finish thinking through the payload concept and API.
 
 * Important: ensure that we have a good solution for magics like %edit.  It's
   likely that with the payload concept we can build a full solution, but not
   100% clear yet.
 
 * Finishing the details of the heartbeat protocol.
 
 * Signal handling: specify what kind of information kernel should broadcast (or
   not) when it receives signals.
 
 .. include:: ../links.rst