Merge Omar's Zmq proposal, formatting cleanups.

docs/source/development/ipythonzmq.txt

@@ -0,0 +1,116 b''
+.. _ipythonzmq:
+
+=====================================================
+ Porting IPython to a two process model using zeromq
+=====================================================
+
+Abstract
+--------
+
+IPython's execution in a command-line environment will be ported to a two
+process model using the zeromq library for inter-process communication. this
+will:
+
+- prevent an interpreter crash from destroying the user session,
+- allow multiple clients to interact simultaneously with a single interpreter
+- allow IPython to reuse code for local execution and distributed computing (dc)
+- give us a path for python3 support, since zeromq supports python3 while
+  twisted (what we use today for dc) does not.
+
+
+Project description
+-------------------
+
+Currently IPython provides a command-line client that executes all code in a
+single process, and a set of tools for distributed and parallel computing that
+execute code in multiple processes (possibly but not necessarily on different
+hosts), using the twisted asynchronous framework for communication between
+nodes. for a number of reasons, it is desirable to unify the architecture of
+the local execution with that of distributed computing, since ultimately many
+of the underlying abstractions are similar and should be reused.  in
+particular, we would like to:
+
+- have even for a single user a 2-process model, so that the environment where
+  code is being input runs in a different process from that which executes the
+  code. this would prevent a crash of the python interpreter executing code
+  (because of a segmentation fault in a compiled extension or an improper
+  access to a c library via ctypes, for example) from destroying the user
+  session.
+
+- have the same kernel used for executing code locally be available over the
+  network for distributed computing. currently the twisted-using IPython
+  engines for distributed computing do not share any code with the command-line
+  client, which means that many of the additional features of IPython (tab
+  completion, object introspection, magic functions, etc) are not available
+  while using the distributed computing system.  once the regular command-line
+  environment is ported to allowing such a 2-process model, this newly
+  decoupled kernel could form the core of a distributed computing IPython
+  engine and all capabilities would be available throughout the system.
+
+- have a route to python3 support. twisted is a large and complex library that
+  does currently not support python3, and as indicated by the twisted
+  developers it may take a while before it is ported
+  (http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3).
+  for IPython, this means that while we could port the command-line
+  environment, a large swath of IPython would be left 2.x-only, a highly
+  undesirable situation.  for this reason, the search for an alternative to
+  twisted has been active for a while, and recently we've identified the zeromq
+  (http://www.zeromq.org, zmq for short) library as a viable candidate. zmq is
+  a fast, simple messaging library written in c++, for which one of the IPython
+  developers has written python bindings using cython
+  (http://www.zeromq.org/bindings:python). since cython already knows how to
+  generate python3-compliant bindings with a simple command-line switch, zmq
+  can be used with python3 when needed.
+
+As part of the zmq python bindings, the IPython developers have already
+developed a simple prototype of such a two-process kernel/frontend system
+(details below). I propose to start from this example and port today's IPython
+code to operate in a similar manner. IPython's command-line program (the main
+'ipython' script) executes both user interaction and the user's code in the
+same process. This project will thus require breaking up IPython into the parts
+that correspond to the kernel and the parts that are meant to interact with the
+user, and making these two components communicate over the network using zmq
+instead of accessing local attributes and methods of a single global object.
+
+Once this port is complete, the resulting tools will be the foundation (though
+as part of this proposal i do not expect to undertake either of these tasks) to
+allow the distributed computing parts of IPython to use the same code as the
+command-line client, and for the whole system to be ported to python3.  so
+while i do not intend to tackle here the removal of twisted and the unification
+of the local and distributed parts of IPython, my proposal is a necessary step
+before those are possible.
+
+Project details
+---------------
+
+As part of the zeromq bindings, the IPython developers have already developed a
+simple prototype example that provides a python execution kernel (with none of
+IPython's code or features, just plain code execution) that listens on zmq
+sockets, and a frontend based on the interactiveconsole class of the code.py
+module from the python standard library. this example is capable of executing
+code, propagating errors, performing tab-completion over the network and having
+multiple frontends connect and disconnect simultaneously to a single kernel,
+with all inputs and outputs being made available to all connected clients
+(thanks to zqm's pub sockets that provide multicasting capabilities for the
+kernel and to which the frontends subscribe via a sub socket).
+
+We have all example code in:
+
+* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel/kernel.py
+* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel/completer.py
+* http://github.com/fperez/pyzmq/blob/completer/examples/kernel/frontend.py
+
+all of this code already works, and can be seen in this example directory from
+the zmq python bindings:
+
+* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel
+
+Based on this work, i expect to write a stable system for IPython kernel with
+IPython standards, error control,crash recovery system and general
+configuration options, also standardize defaults ports or auth system for
+remote connection etc.
+
+The crash recovery system, is a IPython kernel module for when it fails
+unexpectedly, you can retrieve the information from the section, this will be
+based on a log and a lock file to indicate when the kernel was not closed in a
+proper way.

docs/source/development/index.txt

@@ -18,3 +18,4 b" IPython developer's guide"
    notification_blueprint.txt
    ipgraph.txt
    ipython_qt.txt
+   ipythonzmq.txt

docs/source/development/ipython_qt.txt

@@ -5,32 +5,59 b' IPython Qt interface'
 ====================
 
 Abstract
-------------
+--------
 
-This is about the implementation of a Qt-based Graphical User Interface (GUI) to execute Python code with an interpreter that runs in a separate process and the two systems (GUI frontend and interpreter kernel) communicating via the ZeroMQ Messaging library.  The bulk of the implementation will be done without dependencies on IPython (only on Zmq). Once the key features are ready, IPython-specific features can be added using the IPython codebase.
+This is about the implementation of a Qt-based Graphical User Interface (GUI)
+to execute Python code with an interpreter that runs in a separate process and
+the two systems (GUI frontend and interpreter kernel) communicating via the
+ZeroMQ Messaging library.  The bulk of the implementation will be done without
+dependencies on IPython (only on Zmq). Once the key features are ready,
+IPython-specific features can be added using the IPython codebase.
 
 
 Project details
--------------------
+---------------
 
-For a long time there has been demand for a graphical user interface for IPython, and the project already ships Wx-based prototypes thereof.  But these run all code in a single process, making them extremely brittle, as a crash of the Python interpreter kills the entire user session.  Here I propose to build a Qt-based GUI that will communicate with a separate process for the code execution, so that if the interpreter kernel dies, the frontend can continue to function after restarting a new kernel (and offering the user the option to re-execute all inputs, which the frontend can know).
+For a long time there has been demand for a graphical user interface for
+IPython, and the project already ships Wx-based prototypes thereof.  But these
+run all code in a single process, making them extremely brittle, as a crash of
+the Python interpreter kills the entire user session.  Here I propose to build
+a Qt-based GUI that will communicate with a separate process for the code
+execution, so that if the interpreter kernel dies, the frontend can continue to
+function after restarting a new kernel (and offering the user the option to
+re-execute all inputs, which the frontend can know).
 
-This GUI will allow for the easy editing of multi-line input and the convenient re-editing of previous blocks of input, which can be displayed in a 2-d workspace instead of a line-driven one like today's IPython. This makes it much easier to incrementally build and tune a code, by combining the rapid feedback cycle of IPython with the ability to edit multiline code with good graphical support.
+This GUI will allow for the easy editing of multi-line input and the convenient
+re-editing of previous blocks of input, which can be displayed in a 2-d
+workspace instead of a line-driven one like today's IPython. This makes it much
+easier to incrementally build and tune a code, by combining the rapid feedback
+cycle of IPython with the ability to edit multiline code with good graphical
+support.
 
 
 2-process model pyzmq base
-++++++++++++++++++++++++++
-Since the necessity of a user to keep his data safe, the design is based in a 2-process model that will be achieved with a simple client/server system with `pyzmq <http://www.zeromq.org/bindings:python>`_, so the GUI session do not crash if the the kernel process does. This will be achieved using this test `code <http://github.com/fperez/pyzmq/blob/completer/examples/kernel/kernel.py>`_ and customizing it to the necessities of the GUI such as queue management  with  discrimination for different frontends connected to the same kernel and tab completion. A piece of drafted code for the kernel (server) should look like this::
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since the necessity of a user to keep his data safe, the design is based in a
+2-process model that will be achieved with a simple client/server system with
+`pyzmq <http://www.zeromq.org/bindings:python>`_, so the GUI session do not
+crash if the the kernel process does. This will be achieved using this test
+`code
+<http://github.com/fperez/pyzmq/blob/completer/examples/kernel/kernel.py>`_ and
+customizing it to the necessities of the GUI such as queue management with
+discrimination for different frontends connected to the same kernel and tab
+completion. A piece of drafted code for the kernel (server) should look like
+this::
 
     def main():
         c = zmq.Context(1, 1)
         rep_conn = connection % port_base                              
-        pub_conn = connection % (port_base+1)                                                                 
+        pub_conn = connection % (port_base+1)
         print >>sys.__stdout__, "Starting the kernel..."
-        print >>sys.__stdout__, "On:",rep_conn, pub_conn                                    
-        session = Session(username=u'kernel')                                     
-        reply_socket = c.socket(zmq.XREP)                                                  
-        reply_socket.bind(rep_conn)                                                             
+        print >>sys.__stdout__, "On:",rep_conn, pub_conn
+        session = Session(username=u'kernel')
+        reply_socket = c.socket(zmq.XREP)
+        reply_socket.bind(rep_conn)
         pub_socket = c.socket(zmq.PUB)
         pub_socket.bind(pub_conn)                                         
         stdout = OutStream(session, pub_socket, u'stdout')         
@@ -38,24 +65,50 b' Since the necessity of a user to keep his data safe, the design is based in a 2-'
         sys.stdout = stdout                                                 
         sys.stderr = stderr                     
         display_hook = DisplayHook(session, pub_socket)
-        sys.displayhook = display_hook                                                                   
+        sys.displayhook = display_hook
         kernel = Kernel(session, reply_socket, pub_socket)
 
-This kernel will use two queues (output and input), the input queue will have the id of the process(frontend) making the request, type(execute, complete, help, etc) and id of the request itself and the string of code to be executed, the output queue will have basically the same information just that the string is the to be displayed. This model is because the kernel needs to maintain control of timeouts when multiple requests are sent and keep them indexed.
+This kernel will use two queues (output and input), the input queue will have
+the id of the process(frontend) making the request, type(execute, complete,
+help, etc) and id of the request itself and the string of code to be executed,
+the output queue will have basically the same information just that the string
+is the to be displayed. This model is because the kernel needs to maintain
+control of timeouts when multiple requests are sent and keep them indexed.
 
 Qt based GUI
-++++++++++++
-Design of the interface is going to be based in cells of code executed on the previous defined kernel. It will also have GUI facilities such toolboxes, tooltips to autocomplete code and function summary, highlighting  and autoindentation.
-It will have the cell kind of multiline edition mode so each block of code can be edited and executed independently, this can be achieved queuing QTextEdit objects (the cell) giving them format so we can discriminate outputs from inputs.
-One of the main characteristics will be the debug support that will show the requested outputs as the debugger (that will be on a popup widget) "walks" through the code, this design is to be reviewed with the mentor.
-`This <http://gfif.udea.edu.co/IPythonQt_snapshot.png>`_ is a tentative view of the main window.
-
-The GUI will check continuously the output queue from the kernel for new information to handle. This information have to be handled with care since any output will come at anytime and possibly in a different order than requested or maybe not appear at all, this could be possible due to a variety of reasons(for example tab completion request while the kernel is busy processing another frontend's request). This is, if the kernel is busy it won't be possible to fulfill the request for a while so the GUI will be prepared to abandon waiting for the reply if the user moves on or a certain timeout expires.
+~~~~~~~~~~~~
+
+Design of the interface is going to be based in cells of code executed on the
+previous defined kernel. It will also have GUI facilities such toolboxes,
+tooltips to autocomplete code and function summary, highlighting and
+autoindentation.  It will have the cell kind of multiline edition mode so each
+block of code can be edited and executed independently, this can be achieved
+queuing QTextEdit objects (the cell) giving them format so we can discriminate
+outputs from inputs.  One of the main characteristics will be the debug support
+that will show the requested outputs as the debugger (that will be on a popup
+widget) "walks" through the code, this design is to be reviewed with the
+mentor.  `This <http://gfif.udea.edu.co/IPythonQt_snapshot.png>`_ is a
+tentative view of the main window.
+
+The GUI will check continuously the output queue from the kernel for new
+information to handle. This information have to be handled with care since any
+output will come at anytime and possibly in a different order than requested or
+maybe not appear at all, this could be possible due to a variety of reasons(for
+example tab completion request while the kernel is busy processing another
+frontend's request). This is, if the kernel is busy it won't be possible to
+fulfill the request for a while so the GUI will be prepared to abandon waiting
+for the reply if the user moves on or a certain timeout expires.
 
 
 POSSIBLE FUTURE DIRECTIONS
 ---------------------------
 
-The near future will bring the feature of saving and loading sessions, also importing and exporting to different formats like rst, html, pdf and python/ipython code, a discussion about this is taking place in the ipython-dev mailing list. Also the interaction with a remote kernel and distributed computation which is an IPython's project already in development.
+The near future will bring the feature of saving and loading sessions, also
+importing and exporting to different formats like rst, html, pdf and
+python/ipython code, a discussion about this is taking place in the ipython-dev
+mailing list. Also the interaction with a remote kernel and distributed
+computation which is an IPython's project already in development.
 
-The idea of a mathematica-like help widget (i.e. there will be parts of it that will execute as a native session of IPythonQt) is still to be discussed in the development mailing list but it's definitively a great idea.
  No newline at end of file
+The idea of a mathematica-like help widget (i.e. there will be parts of it that
+will execute as a native session of IPythonQt) is still to be discussed in the
+development mailing list but it's definitively a great idea.