docs/source/development/roadmap.txt

.. _roadmap:

===================
Development roadmap
===================

IPython is an ambitious project that is still under heavy development.
However, we want IPython to become useful to as many people as possible, as
quickly as possible. To help us accomplish this, we are laying out a roadmap
of where we are headed and what needs to happen to get there. Hopefully, this
will help the IPython developers figure out the best things to work on for
each upcoming release.

Work targeted to particular releases
====================================

Release 0.11
------------

* Full module and package reorganization (done).

* Removal of the threaded shells and new implementation of GUI support
  based on ``PyOSInputHook`` (done).

* Refactor the configuration system (done).

* Prepare to refactor IPython's core by creating a new component and 
  application system (done).

* Start to refactor IPython's core by turning everything into components
  (started).

Release 0.12
------------

* Continue to refactor IPython's core by turning everything into components.


Major areas of work
===================

Refactoring the main IPython core
---------------------------------

During the summer of 2009, we began refactoring IPython's core. The main
thrust in this work was to make the IPython core into a set of loosely coupled
components. The base component class for this is
:class:`IPython.core.component.Component`. This section outlines the status of
this work.

Parts of the IPython core that have been turned into components:

* The main :class:`~IPython.core.iplib.InteractiveShell` class.
* The aliases (:mod:`IPython.core.alias`).
* The display and builtin traps (:mod:`IPython.core.display_trap` and
  :mod:`IPython.core.builtin_trap`).
* The prefilter machinery (:mod:`IPython.core.prefilter`).

Parts of the IPythoncore that still need to be turned into components:

* Magics.
* Input and output history management.
* Prompts.
* Tab completers.
* Logging.
* Exception handling.
* Anything else.

Process management for :mod:`IPython.kernel`
--------------------------------------------

A number of things need to be done to improve how processes are started
up and managed for the parallel computing side of IPython:

* All of the processes need to use the new configuration system, components
  and application.
* We need to add support for other batch systems.

Performance problems
--------------------

Currently, we have a number of performance issues in :mod:`IPython.kernel`:

* The controller stores a large amount of state in Python dictionaries. Under
  heavy usage, these dicts with get very large, causing memory usage problems.
  We need to develop more scalable solutions to this problem. This will also
  help the controller to be more fault tolerant.

* We currently don't have a good way of handling large objects in the
  controller. The biggest problem is that because we don't have any way of
  streaming objects, we get lots of temporary copies in the low-level buffers.
  We need to implement a better serialization approach and true streaming
  support.

* The controller currently unpickles and repickles objects. We need to use the
  [push|pull]_serialized methods instead.

* Currently the controller is a bottleneck. The best approach for this is to
  separate the controller itself into multiple processes, one for the core
  controller and one each for the controller interfaces.

Porting to 3.0
==============

There are no definite plans for porting of IPython to Python 3. The major 
issue is the dependency on Twisted framework for the networking/threading 
stuff. It is possible that it the traditional IPython interactive console 
could be ported more easily since it has no such dependency. Here are a few
things that will need to be considered when doing such a port especially
if we want to have a codebase that works directly on both 2.x and 3.x.

1. The syntax for exceptions changed (PEP 3110). The old `except exc, var`
   changed to `except exc as var`. At last count there was 78 occurrences of this
   usage in the code base. This is a particularly problematic issue, because it's
   not easy to implement it in a 2.5-compatible way.

Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we
will likely at some point put out a release that requires strictly 2.6 and
abandons 2.5 compatibility.  This will then allow us to port the code to using
:func:`print` as a function, `except exc as var` syntax, etc.  But as of
version 0.11 at least, we will retain Python 2.5 compatibility.