diff --git a/docs/source/development/index.txt b/docs/source/development/index.txt index 2a800dd..d1de11b 100644 --- a/docs/source/development/index.txt +++ b/docs/source/development/index.txt @@ -13,14 +13,8 @@ IPython developer's guide doc_guide.txt testing.txt release.txt - roadmap.txt reorg.txt messaging.txt parallel_messages.txt parallel_connections.txt - magic_blueprint.txt - ipgraph.txt - ipython_qt.txt - ipythonzmq.txt - notebook_todo.txt ipython_directive.txt diff --git a/docs/source/development/ipgraph.txt b/docs/source/development/ipgraph.txt deleted file mode 100644 index 65862ef..0000000 --- a/docs/source/development/ipgraph.txt +++ /dev/null @@ -1,61 +0,0 @@ -==================================================== -Notes on code execution in :class:`InteractiveShell` -==================================================== - -Overview -======== - -This section contains information and notes about the code execution -system in :class:`InteractiveShell`. This system needs to be refactored -and we are keeping notes about this process here. - -Current design -============== - -Here is a script that shows the relationships between the various -methods in :class:`InteractiveShell` that manage code execution:: - - import networkx as nx - import matplotlib.pyplot as plt - - exec_init_cmd = 'exec_init_cmd' - interact = 'interact' - runlines = 'runlines' - runsource = 'runsource' - runcode = 'runcode' - push_line = 'push_line' - mainloop = 'mainloop' - embed_mainloop = 'embed_mainloop' - ri = 'raw_input' - prefilter = 'prefilter' - - g = nx.DiGraph() - - g.add_node(exec_init_cmd) - g.add_node(interact) - g.add_node(runlines) - g.add_node(runsource) - g.add_node(push_line) - g.add_node(mainloop) - g.add_node(embed_mainloop) - g.add_node(ri) - g.add_node(prefilter) - - g.add_edge(exec_init_cmd, push_line) - g.add_edge(exec_init_cmd, prefilter) - g.add_edge(mainloop, exec_init_cmd) - g.add_edge(mainloop, interact) - g.add_edge(embed_mainloop, interact) - g.add_edge(interact, ri) - g.add_edge(interact, push_line) - g.add_edge(push_line, runsource) - g.add_edge(runlines, push_line) - g.add_edge(runlines, prefilter) - g.add_edge(runsource, runcode) - g.add_edge(ri, prefilter) - - nx.draw_spectral(g, node_size=100, alpha=0.6, node_color='r', - font_size=10, node_shape='o') - plt.show() - - diff --git a/docs/source/development/ipython_qt.txt b/docs/source/development/ipython_qt.txt deleted file mode 100644 index 05e928f..0000000 --- a/docs/source/development/ipython_qt.txt +++ /dev/null @@ -1,114 +0,0 @@ -.. _ipython_qt: - -==================== -IPython Qt interface -==================== - -Abstract --------- - -This is about the implementation of a Qt-based Graphical User Interface (GUI) -to execute Python code with an interpreter that runs in a separate process and -the two systems (GUI frontend and interpreter kernel) communicating via the -ZeroMQ Messaging library. The bulk of the implementation will be done without -dependencies on IPython (only on Zmq). Once the key features are ready, -IPython-specific features can be added using the IPython codebase. - - -Project details ---------------- - -For a long time there has been demand for a graphical user interface for -IPython, and the project already ships Wx-based prototypes thereof. But these -run all code in a single process, making them extremely brittle, as a crash of -the Python interpreter kills the entire user session. Here I propose to build -a Qt-based GUI that will communicate with a separate process for the code -execution, so that if the interpreter kernel dies, the frontend can continue to -function after restarting a new kernel (and offering the user the option to -re-execute all inputs, which the frontend can know). - -This GUI will allow for the easy editing of multi-line input and the convenient -re-editing of previous blocks of input, which can be displayed in a 2-d -workspace instead of a line-driven one like today's IPython. This makes it much -easier to incrementally build and tune a code, by combining the rapid feedback -cycle of IPython with the ability to edit multiline code with good graphical -support. - - -2-process model pyzmq base -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Since the necessity of a user to keep his data safe, the design is based in a -2-process model that will be achieved with a simple client/server system with -`pyzmq <http://www.zeromq.org/bindings:python>`_, so the GUI session do not -crash if the the kernel process does. This will be achieved using this test -`code -<http://github.com/fperez/pyzmq/blob/completer/examples/kernel/kernel.py>`_ and -customizing it to the necessities of the GUI such as queue management with -discrimination for different frontends connected to the same kernel and tab -completion. A piece of drafted code for the kernel (server) should look like -this:: - - def main(): - c = zmq.Context(1, 1) - rep_conn = connection % port_base - pub_conn = connection % (port_base+1) - print >>sys.__stdout__, "Starting the kernel..." - print >>sys.__stdout__, "On:",rep_conn, pub_conn - session = Session(username=u'kernel') - reply_socket = c.socket(zmq.ROUTER) - reply_socket.bind(rep_conn) - pub_socket = c.socket(zmq.PUB) - pub_socket.bind(pub_conn) - stdout = OutStream(session, pub_socket, u'stdout') - stderr = OutStream(session, pub_socket, u'stderr') - sys.stdout = stdout - sys.stderr = stderr - display_hook = DisplayHook(session, pub_socket) - sys.displayhook = display_hook - kernel = Kernel(session, reply_socket, pub_socket) - -This kernel will use two queues (output and input), the input queue will have -the id of the process(frontend) making the request, type(execute, complete, -help, etc) and id of the request itself and the string of code to be executed, -the output queue will have basically the same information just that the string -is the to be displayed. This model is because the kernel needs to maintain -control of timeouts when multiple requests are sent and keep them indexed. - -Qt based GUI -~~~~~~~~~~~~ - -Design of the interface is going to be based in cells of code executed on the -previous defined kernel. It will also have GUI facilities such toolboxes, -tooltips to autocomplete code and function summary, highlighting and -autoindentation. It will have the cell kind of multiline edition mode so each -block of code can be edited and executed independently, this can be achieved -queuing QTextEdit objects (the cell) giving them format so we can discriminate -outputs from inputs. One of the main characteristics will be the debug support -that will show the requested outputs as the debugger (that will be on a popup -widget) "walks" through the code, this design is to be reviewed with the -mentor. `This <http://gfif.udea.edu.co/IPythonQt_snapshot.png>`_ is a -tentative view of the main window. - -The GUI will check continuously the output queue from the kernel for new -information to handle. This information have to be handled with care since any -output will come at anytime and possibly in a different order than requested or -maybe not appear at all, this could be possible due to a variety of reasons(for -example tab completion request while the kernel is busy processing another -frontend's request). This is, if the kernel is busy it won't be possible to -fulfill the request for a while so the GUI will be prepared to abandon waiting -for the reply if the user moves on or a certain timeout expires. - - -POSSIBLE FUTURE DIRECTIONS ---------------------------- - -The near future will bring the feature of saving and loading sessions, also -importing and exporting to different formats like rst, html, pdf and -python/ipython code, a discussion about this is taking place in the ipython-dev -mailing list. Also the interaction with a remote kernel and distributed -computation which is an IPython's project already in development. - -The idea of a mathematica-like help widget (i.e. there will be parts of it that -will execute as a native session of IPythonQt) is still to be discussed in the -development mailing list but it's definitively a great idea. diff --git a/docs/source/development/ipythonzmq.txt b/docs/source/development/ipythonzmq.txt deleted file mode 100644 index 0454653..0000000 --- a/docs/source/development/ipythonzmq.txt +++ /dev/null @@ -1,116 +0,0 @@ -.. _ipythonzmq: - -===================================================== - Porting IPython to a two process model using zeromq -===================================================== - -Abstract --------- - -IPython's execution in a command-line environment will be ported to a two -process model using the zeromq library for inter-process communication. this -will: - -- prevent an interpreter crash from destroying the user session, -- allow multiple clients to interact simultaneously with a single interpreter -- allow IPython to reuse code for local execution and distributed computing (dc) -- give us a path for python3 support, since zeromq supports python3 while - twisted (what we use today for dc) does not. - - -Project description -------------------- - -Currently IPython provides a command-line client that executes all code in a -single process, and a set of tools for distributed and parallel computing that -execute code in multiple processes (possibly but not necessarily on different -hosts), using the twisted asynchronous framework for communication between -nodes. for a number of reasons, it is desirable to unify the architecture of -the local execution with that of distributed computing, since ultimately many -of the underlying abstractions are similar and should be reused. in -particular, we would like to: - -- have even for a single user a 2-process model, so that the environment where - code is being input runs in a different process from that which executes the - code. this would prevent a crash of the python interpreter executing code - (because of a segmentation fault in a compiled extension or an improper - access to a c library via ctypes, for example) from destroying the user - session. - -- have the same kernel used for executing code locally be available over the - network for distributed computing. currently the twisted-using IPython - engines for distributed computing do not share any code with the command-line - client, which means that many of the additional features of IPython (tab - completion, object introspection, magic functions, etc) are not available - while using the distributed computing system. once the regular command-line - environment is ported to allowing such a 2-process model, this newly - decoupled kernel could form the core of a distributed computing IPython - engine and all capabilities would be available throughout the system. - -- have a route to python3 support. twisted is a large and complex library that - does currently not support python3, and as indicated by the twisted - developers it may take a while before it is ported - (http://stackoverflow.com/questions/172306/how-are-you-planning-on-handling-the-migration-to-python-3). - for IPython, this means that while we could port the command-line - environment, a large swath of IPython would be left 2.x-only, a highly - undesirable situation. for this reason, the search for an alternative to - twisted has been active for a while, and recently we've identified the zeromq - (http://www.zeromq.org, zmq for short) library as a viable candidate. zmq is - a fast, simple messaging library written in c++, for which one of the IPython - developers has written python bindings using cython - (http://www.zeromq.org/bindings:python). since cython already knows how to - generate python3-compliant bindings with a simple command-line switch, zmq - can be used with python3 when needed. - -As part of the zmq python bindings, the IPython developers have already -developed a simple prototype of such a two-process kernel/frontend system -(details below). I propose to start from this example and port today's IPython -code to operate in a similar manner. IPython's command-line program (the main -'ipython' script) executes both user interaction and the user's code in the -same process. This project will thus require breaking up IPython into the parts -that correspond to the kernel and the parts that are meant to interact with the -user, and making these two components communicate over the network using zmq -instead of accessing local attributes and methods of a single global object. - -Once this port is complete, the resulting tools will be the foundation (though -as part of this proposal i do not expect to undertake either of these tasks) to -allow the distributed computing parts of IPython to use the same code as the -command-line client, and for the whole system to be ported to python3. so -while i do not intend to tackle here the removal of twisted and the unification -of the local and distributed parts of IPython, my proposal is a necessary step -before those are possible. - -Project details ---------------- - -As part of the zeromq bindings, the IPython developers have already developed a -simple prototype example that provides a python execution kernel (with none of -IPython's code or features, just plain code execution) that listens on zmq -sockets, and a frontend based on the interactiveconsole class of the code.py -module from the python standard library. this example is capable of executing -code, propagating errors, performing tab-completion over the network and having -multiple frontends connect and disconnect simultaneously to a single kernel, -with all inputs and outputs being made available to all connected clients -(thanks to zqm's pub sockets that provide multicasting capabilities for the -kernel and to which the frontends subscribe via a sub socket). - -We have all example code in: - -* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel/kernel.py -* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel/completer.py -* http://github.com/fperez/pyzmq/blob/completer/examples/kernel/frontend.py - -all of this code already works, and can be seen in this example directory from -the zmq python bindings: - -* http://github.com/ellisonbg/pyzmq/blob/completer/examples/kernel - -Based on this work, i expect to write a stable system for IPython kernel with -IPython standards, error control,crash recovery system and general -configuration options, also standardize defaults ports or auth system for -remote connection etc. - -The crash recovery system, is a IPython kernel module for when it fails -unexpectedly, you can retrieve the information from the section, this will be -based on a log and a lock file to indicate when the kernel was not closed in a -proper way. diff --git a/docs/source/development/magic_blueprint.txt b/docs/source/development/magic_blueprint.txt deleted file mode 100644 index 2a82f2f..0000000 --- a/docs/source/development/magic_blueprint.txt +++ /dev/null @@ -1,103 +0,0 @@ -============================== - The magic commands subsystem -============================== - -.. warning:: - - These are *preliminary* notes and thoughts on the magic system, kept here - for reference so we can come up with a good design now that the major core - refactoring has made so much progress. Do not consider yet any part of this - document final. - -Two entry points: - -- m.line_eval(self,parameter_s): like today -- m.block_eval(self,code_block): for whole-block evaluation. - -This would allow us to have magics that take input, and whose single line form -can even take input and call block_eval later (like %cpaste does, but with a -generalized interface). - -Constructor -=========== - -Suggested syntax:: - - class MyMagic(BaseMagic): - requires_shell = True/False - def __init__(self,shell=None): - - -Registering magics -================== - -Today, ipapi provides an *expose_magic()* function for making simple magics. -We will probably extend this (in a backwards-compatible manner if possible) to -allow the simplest cases to work as today, while letting users register more -complex ones. - -Use cases:: - - def func(arg): pass # note signature, no 'self' - ip.expose_magic('name',func) - - def func_line(arg): pass - def func_block(arg):pass - ip.expose_magic('name',func_line,func_block) - - class mymagic(BaseMagic): - """Magic docstring, used in help messages. - """ - def line_eval(self,arg): pass - def block_eval(self,arg): pass - - ip.register_magic(mymagic) - - -The BaseMagic class will offer common functionality to all, including things -like options handling (via argparse). - - -Call forms: line and block -========================== - -Block-oriented environments will call line_eval() for the first line of input -(the call line starting with '%') and will then feed the rest of the block to -block_eval() if the magic in question has a block mode. - -In line environments, by default %foo -> foo.line_eval(), but no block call is -made. Specific implementations of line_eval can decide to then call block_eval -if they want to provide for whole-block input in line-oriented environments. - -The api might be adapted for this decision to be made automatically by the -frontend... - - -Precompiled magics for rapid loading -==================================== - -For IPython itself, we'll have a module of 'core' magic functions that do not -require run-time registration. These will be the ones contained today in -Magic.py, plus any others we deem worthy of being available by default. This -is a trick to enable faster startup, since once we move to a model where each -magic can in principle be registered at runtime, creating a lot of them can -easily swamp startup time. - -The trick is to make a module with a top-level class object that contains -explicit references to all the 'core' magics in its dict. This way, the magic -table can be quickly updated at interpreter startup with a single call, by -doing something along the lines of:: - - self.magic_table.update(static_magics.__dict__) - -The point will be to be able to bypass the explicit calling of whatever -register_magic() API we end up making for users to declare their own magics. -So ultimately one should be able to do either:: - - ip.register_magic(mymagic) # for one function - -or:: - - ip.load_magics(static_magics) # for a bunch of them - -I still need to clarify exactly how this should work though. diff --git a/docs/source/development/notebook_todo.txt b/docs/source/development/notebook_todo.txt deleted file mode 100644 index 39ca0d5..0000000 --- a/docs/source/development/notebook_todo.txt +++ /dev/null @@ -1,56 +0,0 @@ -Notebook todo -============= - -* Style the login page consistently with the rest of the site. -* Style the "Log Out" and username links in the header. -* Do a review of the header design and decide what to do about save widget moving - to the Notebook section of the L panel. -* Show last saved time next to save widget. -* Make the header logo a link to "/". -* Add a better divider line between the header and the content area. - - Fix spacing on notebook page. -* Organize templates to use inheritance and includes. - -* Implement better restart logic. - - Have LocalKernel monitor the hb port and always to restarts. - - Have the WebSocket still monitor the hb and notify the user of restarts. - -* Create unrendered rst cells. -* Users should be able to edit the contents of any cell in a global ACE editor. -* Add JSON'd metadata to the .py format notebooks. -* Implement white space between cells for insert. -* Implement a notebook reload button. -* Indicate visual difference between html and markdown cell. -* Export should save first. -* Add ability to merge and split cells. -* Add Ctrl-Z for undo delete cell. -* Fix horizontal overflow and scrolling of output_area. -* Add per cells controls on the R side of each cell. -* Users should be able to drag a .py file to a cell and have it imported into that cell. - -* Add reconnect logic in the javascript kernel. -* Add logic for failed ajax requests. With this, investigate returning JSON data to more - completely describe the HTTP error codes. -* Test web services against hostile attacks. -* Add optional html sanitizing. -* Add timestamp to cells. ISO8601. IPython.utils.jsonutil.ISO8601. Save as - submitted/started/completed/received. See http://webcloud.se/log/JavaScript-and-ISO-8601/ -* Try to figure out the issue with jQuery and <script> tags. See - http://stackoverflow.com/questions/610995/jquery-cant-append-script-element - -CodeMirror related ------------------- - -* Focus should only be called when the editor is on the page and visible. -* Refresh needs to be called when the editor is shown after hiding. -* Right now focus, then setValue causes the arrow keys to lock up. If that bug is - not fixed, we need to possible move to passing the input to the CodeCell - constructor. -* Implement a top-level refresh methods on Cells and the Notebook that can be called - after page/notebook load. -* Make insert_code_cell_* methods not call select always. Probably move to a model - where those methods take an options object. -* Notebook loading should be done without calls to select/focus/refresh. A single - refresh pass should be done after everything has been made visible. -* Remove \u0000 from placeholders after the relevant CM bug is fixed. - diff --git a/docs/source/development/roadmap.txt b/docs/source/development/roadmap.txt deleted file mode 100644 index 9c3e865..0000000 --- a/docs/source/development/roadmap.txt +++ /dev/null @@ -1,95 +0,0 @@ -.. _roadmap: - -=================== -Development roadmap -=================== - -IPython is an ambitious project that is still under heavy development. -However, we want IPython to become useful to as many people as possible, as -quickly as possible. To help us accomplish this, we are laying out a roadmap -of where we are headed and what needs to happen to get there. Hopefully, this -will help the IPython developers figure out the best things to work on for -each upcoming release. - -Work targeted to particular releases -==================================== - -Release 0.11 ------------- - -* Full module and package reorganization (done). - -* Removal of the threaded shells and new implementation of GUI support - based on ``PyOSInputHook`` (done). - -* Refactor the configuration system (done). - -* Prepare to refactor IPython's core by creating a new component and - application system (done). - -* Start to refactor IPython's core by turning everything into configurables - (mostly done). - -Release 0.12 ------------- - -* Continue to refactor IPython's core by turning everything into configurables. - -* Merge draft html notebook (started). - -* Add two-process Terminal frontend using ZMQ architecture. - - -Major areas of work -=================== - -Refactoring the main IPython core ---------------------------------- - -During the summer of 2009, we began refactoring IPython's core. The main -thrust in this work was to make the IPython core into a set of loosely coupled -components. The base configurable class for this is -:class:`IPython.core.configurable.Configurable`. This section outlines the -status of this work. - -Parts of the IPython core that have been turned into configurables: - -* The main :class:`~IPython.core.interactiveshell.InteractiveShell` class. -* The aliases (:mod:`IPython.core.alias`). -* History management (:mod:`IPython.core.history`). -* Plugins (:mod:`IPython.core.plugin`). -* The display and builtin traps (:mod:`IPython.core.display_trap` and - :mod:`IPython.core.builtin_trap`). -* The prefilter machinery (:mod:`IPython.core.prefilter`). - -Parts of the IPython core that still need to be Configurable: - -* Magics. -* Prompts. -* Tab completers. -* Exception handling. -* Anything else. - -Process management for :mod:`IPython.parallel` --------------------------------------------- - -A few things need to be done to improve how processes are started -up and managed for the parallel computing side of IPython: - -* Improve the SSH launcher so it is at least back to the levels of 0.10.2 -* We need to add support for other batch systems (LSF, Condor, etc.). - - -Porting to 3.0 -============== - -Dropping 2.5 support was a major step towards working with Python 3 due to -future syntax support in 2.6. IPython 0.11 requires 2.6 now, so 0.10.2 will -be the last IPython release that supports Python 2.5. - -We currently have a `separate repo <https://github.com/ipython/ipython-py3k>`_ -tracking IPython development that works with Python 3. The core of IPython does -work, but the parallel computing code in :mod:`IPython.parallel` does not yet -work in Python 3, though the only major dependency of the parallel code, pyzmq, -does work on Python 3. -