docs/source/whatsnew/development.txt

================================================
Development version
================================================

The changes listed here are a brief summary of the substantial work on IPython
since the 0.10.x release series. For more details, please consult the actual
source.

Main `ipython` branch
=====================

Refactoring
-----------

As of the 0.11 version of IPython, a signifiant portion of the core has been
refactored.  This refactoring is founded on a number of new abstractions.
The main new classes that implement these abstractions are:

* :class:`IPython.utils.traitlets.HasTraits`.
* :class:`IPython.config.configurable.Configurable`.
* :class:`IPython.config.application.Application`.
* :class:`IPython.config.loader.ConfigLoader`.
* :class:`IPython.config.loader.Config`

We are still in the process of writing developer focused documentation about
these classes, but for now our :ref:`configuration documentation
<config_overview>` contains a high level overview of the concepts that these
classes express.

The biggest user-visible change is likely the move to using the config system to
determine the command-line arguments for IPython applications. The benefit of
this is that *all* configurable values in IPython are exposed on the
command-line, but the syntax for specifying values has changed. The gist is that
assigning values is pure Python assignment, so there is always an '=', and never
a leading '-', nor a space separating key from value. Flags exist, to set
multiple values or boolean flags, and these are always prefixed with '--', and
never take arguments.

ZMQ architecture
----------------

There is a new GUI framework for IPython, based on a client-server model in
which multiple clients can communicate with one IPython kernel, using the
ZeroMQ messaging framework. There is already a Qt console client, which can
be started by calling ``ipython qtconsole``. The protocol is :ref:`documented
<messaging>`.

The parallel computing framework has also been rewritten using ZMQ. The
protocol is described :ref:`here <parallel_messages>`, and the code is in the
new :mod:`IPython.parallel` module.

Python 3 support
----------------

A Python 3 version of IPython has been prepared. For the time being, this is
maintained separately and updated from the main codebase. Its code can be found
`here <https://github.com/ipython/ipython-py3k>`_. Note that the parallel
computing features do not yet work in Python 3.

Unicode
-------

Entering non-ascii characters in unicode literals (``u"€ø"``) now works properly
on all platforms. However, entering these in byte/string literals (``"€ø"``)
will not work as expected on Windows (or any platform where the terminal encoding
is not UTF-8, as it typically is for Linux & Mac OS X). You can use escape sequences
(``"\xe9\x82"``) to get bytes above 128, or use unicode literals and encode
them. This is a limitation of Python 2 which we cannot easily work around.

New features
------------

* Added ``Bytes`` traitlet, removing ``Str``.  All 'string' traitlets should
  either be ``Unicode`` if a real string, or ``Bytes`` if a C-string. This
  removes ambiguity and helps the Python 3 transition.

* New magic ``%loadpy`` loads a python file from disk or web URL into
  the current input buffer.

* New magic ``%pastebin`` for sharing code via the 'Lodge it' pastebin.

* New magic ``%precision`` for controlling float and numpy pretty printing.

* IPython applications initiate logging, so any object can gain access to
  a the logger of the currently running Application with:

.. sourcecode:: python

    from IPython.config.application import Application
    logger = Application.instance().log

* You can now get help on an object halfway through typing a command. For
  instance, typing ``a = zip?`` shows the details of :func:`zip`. It also
  leaves the command at the next prompt so you can carry on with it.

* The input history is now written to an SQLite database. The API for
  retrieving items from the history has also been redesigned.

* The :mod:`IPython.extensions.pretty` extension has been moved out of
  quarantine and fully updated to the new extension API.

* New magics for loading/unloading/reloading extensions have been added:
  ``%load_ext``, ``%unload_ext`` and ``%reload_ext``.

* The configuration system and configuration files are brand new. See the
  configuration system :ref:`documentation <config_index>` for more details.

* The :class:`~IPython.core.interactiveshell.InteractiveShell` class is now a
  :class:`~IPython.config.configurable.Configurable` subclass and has traitlets that
  determine the defaults and runtime environment. The ``__init__`` method has
  also been refactored so this class can be instantiated and run without the
  old :mod:`ipmaker` module.

* The methods of :class:`~IPython.core.interactiveshell.InteractiveShell` have
  been organized into sections to make it easier to turn more sections
  of functionality into components.

* The embedded shell has been refactored into a truly standalone subclass of
  :class:`InteractiveShell` called :class:`InteractiveShellEmbed`.  All
  embedding logic has been taken out of the base class and put into the 
  embedded subclass.

* Added methods of :class:`~IPython.core.interactiveshell.InteractiveShell` to
  help it cleanup after itself. The :meth:`cleanup` method controls this. We
  couldn't do this in :meth:`__del__` because we have cycles in our object
  graph that prevent it from being called.

* Created a new module :mod:`IPython.utils.importstring` for resolving
  strings like ``foo.bar.Bar`` to the actual class.

* Completely refactored the :mod:`IPython.core.prefilter` module into
  :class:`~IPython.config.configurable.Configurable` subclasses. Added a new layer
  into the prefilter system, called "transformations" that all new prefilter
  logic should use (rather than the older "checker/handler" approach).

* Aliases are now components (:mod:`IPython.core.alias`).

* We are now using an internally shipped version of
  :mod:`~IPython.external.argparse` to parse command line options for
  :command:`ipython`.

* New top level :func:`~IPython.frontend.terminal.embed.embed` function that can
  be called to embed IPython at any place in user's code. One the first call it
  will create an :class:`~IPython.frontend.terminal.embed.InteractiveShellEmbed`
  instance and call it. In later calls, it just calls the previously created
  :class:`~IPython.frontend.terminal.embed.InteractiveShellEmbed`.

* Created a configuration system (:mod:`IPython.config.configurable`) that is
  based on :mod:`IPython.utils.traitlets`. Configurables are arranged into a
  runtime containment tree (not inheritance) that i) automatically propagates
  configuration information and ii) allows singletons to discover each other in
  a loosely coupled manner. In the future all parts of IPython will be
  subclasses of :class:`~IPython.config.configurable.Configurable`. All IPython
  developers should become familiar with the config system.

* Created a new :class:`~IPython.config.loader.Config` for holding
  configuration information. This is a dict like class with a few extras: i)
  it supports attribute style access, ii) it has a merge function that merges
  two :class:`~IPython.config.loader.Config` instances recursively and iii) it
  will automatically create sub-:class:`~IPython.config.loader.Config`
  instances for attributes that start with an uppercase character.

* Created new configuration loaders in :mod:`IPython.config.loader`. These
  loaders provide a unified loading interface for all configuration
  information including command line arguments and configuration files. We
  have two default implementations based on :mod:`argparse` and plain python
  files.  These are used to implement the new configuration system.

* Created a top-level :class:`Application` class in
  :mod:`IPython.core.application` that is designed to encapsulate the starting
  of any basic Python program. An application loads and merges all the
  configuration objects, constructs the main application, configures and
  initiates logging, and creates and configures any :class:`Configurable`
  instances and then starts the application running. An extended
  :class:`BaseIPythonApplication` class adds logic for handling the
  IPython directory as well as profiles, and all IPython entry points
  extend it.

* The :class:`Type` and :class:`Instance` traitlets now handle classes given
  as strings, like ``foo.bar.Bar``. This is needed for forward declarations.
  But, this was implemented in a careful way so that string to class
  resolution is done at a single point, when the parent
  :class:`~IPython.utils.traitlets.HasTraitlets` is instantiated.

* :mod:`IPython.utils.ipstruct` has been refactored to be a subclass of 
  dict.  It also now has full docstrings and doctests.
* Created a Trait's like implementation in :mod:`IPython.utils.traitlets`.
  This is a pure Python, lightweight version of a library that is similar to
  :mod:`enthought.traits`.  We are using this for validation, defaults and
  notification in our new component system.  Although it is not API compatible
  with :mod:`enthought.traits`, we plan on moving in this direction so that
  eventually our implementation could be replaced by a (yet to exist) pure
  Python version of :mod:`enthought.traits`.

* Added a new module :mod:`IPython.lib.inputhook` to manage the integration
  with GUI event loops using `PyOS_InputHook`.  See the docstrings in this
  module or the main IPython docs for details.

* For users, GUI event loop integration is now handled through the new
  :command:`%gui` magic command.  Type ``%gui?`` at an IPython prompt for
  documentation.

* For developers :mod:`IPython.lib.inputhook` provides a simple interface
  for managing the event loops in their interactive GUI applications.
  Examples can be found in our :file:`docs/examples/lib` directory.

Backwards incompatible changes
------------------------------

* The Twisted-based :mod:`IPython.kernel` has been removed, and completely
  rewritten as :mod:`IPython.parallel`, using ZeroMQ.

* Profiles are now directories. Instead of a profile being a single config file,
  profiles are now self-contained directories. By default, profiles get their
  own IPython history, log files, and everything. To create a new profile, do
  ``ipython profile create <name>``.

* All IPython applications have been rewritten to use
  :class:`~IPython.config.loader.KeyValueConfigLoader`. This means that
  command-line options have changed. Now, all configurable values are accessible
  from the command-line with the same syntax as in a configuration file.

* The command line options ``-wthread``, ``-qthread`` and
  ``-gthread`` have been removed. Use ``gui=wx``, ``gui=qt``, ``gui=gtk``
  instead.

* The extension loading functions have been renamed to
  :func:`load_ipython_extension` and :func:`unload_ipython_extension`.

* :class:`~IPython.core.interactiveshell.InteractiveShell` no longer takes an
  ``embedded`` argument. Instead just use the
  :class:`~IPython.core.interactiveshell.InteractiveShellEmbed` class.

* ``__IPYTHON__`` is no longer injected into ``__builtin__``.

* :meth:`Struct.__init__` no longer takes `None` as its first argument.  It
  must be a :class:`dict` or :class:`Struct`.

* :meth:`~IPython.core.interactiveshell.InteractiveShell.ipmagic` has been
  renamed :meth:`~IPython.core.interactiveshell.InteractiveShell.magic.`

* The functions :func:`ipmagic` and :func:`ipalias` have been removed from
  :mod:`__builtins__`.

* The references to the global
  :class:`~IPython.core.interactivehell.InteractiveShell` instance (``_ip``, and
  ``__IP``) have been removed from the user's namespace. They are replaced by a
  new function called :func:`get_ipython` that returns the current
  :class:`~IPython.core.interactiveshell.InteractiveShell` instance. This
  function is injected into the user's namespace and is now the main way of
  accessing the running IPython.

* Old style configuration files :file:`ipythonrc` and :file:`ipy_user_conf.py`
  are no longer supported. Users should migrate there configuration files to
  the new format described :ref:`here <config_overview>` and :ref:`here
  <configuring_ipython>`.

* The old IPython extension API that relied on :func:`ipapi` has been
  completely removed. The new extension API is described :ref:`here
  <configuring_ipython>`.

* Support for ``qt3`` has been dropped.  Users who need this should use
  previous versions of IPython.

* Removed :mod:`shellglobals` as it was obsolete.

* Removed all the threaded shells in :mod:`IPython.core.shell`. These are no
  longer needed because of the new capabilities in
  :mod:`IPython.lib.inputhook`.

* New top-level sub-packages have been created: :mod:`IPython.core`, 
  :mod:`IPython.lib`, :mod:`IPython.utils`, :mod:`IPython.deathrow`,
  :mod:`IPython.quarantine`.  All existing top-level modules have been
  moved to appropriate sub-packages.  All internal import statements
  have been updated and tests have been added.  The build system (setup.py
  and friends) have been updated.  See :ref:`this section <module_reorg>` of the
  documentation for descriptions of these new sub-packages.

* :mod:`IPython.ipapi` has been moved to :mod:`IPython.core.ipapi`.
  :mod:`IPython.Shell` and :mod:`IPython.iplib` have been split and removed as
  part of the refactor.

* :mod:`Extensions` has been moved to :mod:`extensions` and all existing
  extensions have been moved to either :mod:`IPython.quarantine` or
  :mod:`IPython.deathrow`. :mod:`IPython.quarantine` contains modules that we
  plan on keeping but that need to be updated. :mod:`IPython.deathrow`
  contains modules that are either dead or that should be maintained as third
  party libraries. More details about this can be found :ref:`here
  <module_reorg>`.

* Previous IPython GUIs in :mod:`IPython.frontend` and :mod:`IPython.gui` are
  likely broken, and have been removed to :mod:`IPython.deathrow` because of the
  refactoring in the core. With proper updates, these should still work.


Known Regressions
-----------------

We do our best to improve IPython, but there are some known regressions in 0.11 relative
to 0.10.2.

* The machinery that adds functionality to the 'sh' profile for using IPython as your
  system shell has not been updated to use the new APIs.  As a result, only the aesthetic
  (prompt) changes are still implemented. We intend to fix this by 0.12.

* The installation of scripts on Windows was broken without setuptools, so we now
  depend on setuptools on Windows.  We hope to fix setuptools-less installation,
  and then remove the setuptools dependency.