Improve what's new document, closer to release form...

docs/source/whatsnew/version0.11.txt

@@ -1,90 +1,100 b''
-===============
-0.11 Series
-===============
+=============
+ 0.11 Series
+=============
 
 Release 0.11
 ============
 
-IPython 0.11 is a *major* overhaul of IPython, two years in the making.  Most of
-the code base has been rewritten or at least reorganized, breaking backward compatibility
-with several APIs in previous versions.  It is the first major release in two years, and
-probably the most significant change to IPython since its inception.
-As a result of the significant changes, we do plan to have a relatively quick
-succession of releases, as people discover new bugs and regressions.
+IPython 0.11 is a *major* overhaul of IPython, two years in the making.  Most
+of the code base has been rewritten or at least reorganized, breaking backward
+compatibility with several APIs in previous versions.  It is the first major
+release in two years, and probably the most significant change to IPython since
+its inception.  We plan to have a relatively quick succession of releases, as
+people discover new bugs and regressions.  Once we iron out any significant
+bugs in this process and settle down the new APIs, this series will become
+IPython 1.0.  We encourage feedback now on the core APIs, which we hope to
+maintain stable during the 1.0 series.
 
+Chris Fonnesbeck recently wrote an `excellent post`_ that highlights some of
+the core new features, with examples and screenshots.  We encourage you
+to read it as it provides a great high-level overview that complements the
+detailed feature breakdown in this document.
 
-Authors
--------
+.. _excellent post: http://stronginference.com/weblog/2011/7/15/innovations-in-ipython.html
 
-Many users and developers contributed code, features, bug reports and ideas to
-this release.  Please do not hesitate in contacting us if we've failed to
-acknowledge your contribution here.  In particular, for this release we have
-contribution from the following people, a mix of new and regular names (in
-alphabetical order by first name):
+A quick summary of the major changes (see below for details):
 
+* *Standalone Qt console*: a new rich console has been added to IPython, started
+  with `ipython qtconsole`.  In this application we have tried to retain the
+  feel of a terminal for fast and efficient workflows, while adding many
+  features that a plain text terminal simply can not support, such as inline
+  figures, full multiline editing with syntax highlighting, tooltips for
+  function calls and much more.
 
-* Andy Wilson <wilson.andrew.j+github-at-gmail.com>
-* Aenugu Sai Kiran Reddy <saikrn08-at-gmail.com>
-* Antonio Cuni <antocuni>
-* Barry Wark <barrywark-at-gmail.com>
-* Beetoju Anuradha <anu.beethoju-at-gmail.com>
-* Brad Reisfeld 
-* Brian Granger <ellisonbg-at-gmail.com>
-* Cody Precord 
-* Darren Dale <dsdale24-at-gmail.com>
-* Dav Clark <davclark-at-berkeley.edu>
-* David Warde-Farley <wardefar-at-iro.umontreal.ca>
-* Eric Firing <efiring-at-hawaii.edu>
-* Erik Tollerud <erik.tollerud-at-gmail.com>
-* Evan Patterson <ejpatters-at-gmail.com>
-* Fernando Perez <Fernando.Perez-at-berkeley.edu>
-* Gael Varoquaux <gael.varoquaux-at-normalesup.org>
-* Gerardo <muzgash-at-Muzpelheim>
-* Jason Grout <jason.grout-at-drake.edu>
-* Jens Hedegaard Nielsen <jenshnielsen-at-gmail.com>
-* Justin Riley <justin.t.riley-at-gmail.com>
-* Jörgen Stenarson <jorgen.stenarson-at-bostream.nu>
-* Kiorky 
-* Laurent Dufrechou <laurent.dufrechou-at-gmail.com>
-* Luis Pedro Coelho <luis-at-luispedro.org>
-* Mani chandra <mchandra-at-iitk.ac.in>
-* Mark E. Smith 
-* Mark Voorhies <mark.voorhies-at-ucsf.edu>
-* Martin Spacek <git-at-mspacek.mm.st>
-* Michael Droettboom <mdroe-at-stsci.edu>
-* Min RK <benjaminrk-at-gmail.com>
-* Nick Tarleton <nick-at-quixey.com>
-* Nicolas Rougier <Nicolas.rougier-at-inria.fr>
-* Omar Andres Zapata Mesa <andresete.chaos-at-gmail.com>
-* Paul Ivanov <pivanov314-at-gmail.com>
-* Pauli Virtanen <pauli.virtanen-at-iki.fi>
-* Prabhu Ramachandran 
-* Ramana <sramana9-at-gmail.com>
-* Robert Kern <robert.kern-at-gmail.com>
-* Sathesh Chandra <satheshchandra88-at-gmail.com>
-* Satrajit Ghosh <satra-at-mit.edu>
-* Sebastian Busch 
-* Stefan van der Walt <bzr-at-mentat.za.net>
-* Stephan Peijnik <debian-at-sp.or.at>
-* Steven Bethard 
-* Thomas Kluyver <takowl-at-gmail.com>
-* Thomas Spura <tomspur-at-fedoraproject.org>
-* Tom Fetherston <tfetherston-at-aol.com>
-* Tom MacWright 
-* Ville M. Vainio <vivainio-at-gmail.com>
-* Vishal Vatsa <vishal.vatsa-at-gmail.com>
-* Vishnu S G <sgvishnu777-at-gmail.com>
-* Walter Doerwald <walter-at-livinglogic.de>
-* dan.milstein 
-* muzuiget <muzuiget-at-gmail.com>
-* tzanko 
-* vankayala sowjanya <hai.sowjanya-at-gmail.com>
+* *High-level parallel computing with ZeroMQ*. Using the same architecture that
+  our Qt console is based on, we have completely rewritten our high-level
+  parallel computing machinery that in prior versions used the Twisted
+  networking framework.  While this change will require users to update their
+  codes, the improvements in performance, memory control and internal
+  consistency across our codebase convinced us it was a price worth paying.  We
+  have tried to explain how to best proceed with this update, and will be happy
+  to answer questions that may arise.
 
-.. note::
+* *New model for GUI/plotting support in the terminal*.  Now instead of the
+  various `-Xthread` flags we had before, GUI support is provided without the
+  use of any threads, by directly integrating GUI event loops with Python's
+  `PyOS_InputHook` API.  A new command-line flag `--gui` controls GUI support,
+  and it can also be enabled after IPython startup via the new `%gui` magic.
 
-    This list was generated with the output of 
-    ``git log dev-0.11 HEAD --format='* %aN <%aE>' | sed 's/@/\-at\-/' | sed 's/<>//' | sort -u``
-    after some cleanup.  If you should be on this list, please add yourself.
+* *A two-process architecture.* The Qt console is the first example of using a
+  new model that splits IPython between a kernel process where code is executed
+  and a client that handles user interaction.  We plan on also providing
+  terminal and web-browser based clients using this infrastructure in future
+  releases.  This model allows multiple clients to interact with an IPython
+  process through a well-documented protocol using the ZeroMQ networking
+  library.
+
+* *Refactoring.* the entire codebase has been refactored, in order to make it
+  more modular and easier to contribute to.  IPython has traditionally been a
+  hard project to participate because the old codebase was very monolithic.  We
+  hope this (ongoing) restructuring will make it easier for new developers to
+  join us.
+
+* *Improved unicode support*. We closed many bugs related to unicode input.
+
+* *Python 3*. IPython now runs on Python 3.x.
+
+* *New profile model*. Profiles are now directories that contain all relevant
+   infromation for that session, and thus better isolate IPython use-cases.
+
+* *SQLite storage for history*. All history is now stored in a SQLite database,
+   providing support for multiple simultaneous sessions that won't clobber each
+   other as well as the ability to perform queries on all stored data.
+
+* *New configuration system*. All parts of IPython are now configured via a
+   mechanism inspired by the Enthought Traits library.  Any configurable
+   element can have its attributes set either via files that now use real
+   Python syntax or from the command-line.
+
+
+Over 60 separate authors have contributed to this release, see :ref:`below
+<credits>` for a full list.
+
+
+Plotting and GUI support in the terminal
+----------------------------------------
+
+blah
+
+
+Qt Console
+----------
+
+blah
+
+
+High-level parallel computing with ZeroMQ
+-----------------------------------------
 
 
 Refactoring
@@ -139,15 +149,16 b' working.'
 Unicode
 -------
 
-Entering non-ascii characters in unicode literals (``u"€ø"``) now works properly
-on all platforms. However, entering these in byte/string literals (``"€ø"``)
-will not work as expected on Windows (or any platform where the terminal encoding
-is not UTF-8, as it typically is for Linux & Mac OS X). You can use escape sequences
-(``"\xe9\x82"``) to get bytes above 128, or use unicode literals and encode
-them. This is a limitation of Python 2 which we cannot easily work around.
+Entering non-ascii characters in unicode literals (``u"€ø"``) now works
+properly on all platforms. However, entering these in byte/string literals
+(``"€ø"``) will not work as expected on Windows (or any platform where the
+terminal encoding is not UTF-8, as it typically is for Linux & Mac OS X). You
+can use escape sequences (``"\xe9\x82"``) to get bytes above 128, or use
+unicode literals and encode them. This is a limitation of Python 2 which we
+cannot easily work around.
 
-New features
-------------
+Additional new features
+-----------------------
 
 * Added ``Bytes`` traitlet, removing ``Str``.  All 'string' traitlets should
   either be ``Unicode`` if a real string, or ``Bytes`` if a C-string. This
@@ -185,10 +196,10 b' New features'
   configuration system :ref:`documentation <config_index>` for more details.
 
 * The :class:`~IPython.core.interactiveshell.InteractiveShell` class is now a
-  :class:`~IPython.config.configurable.Configurable` subclass and has traitlets that
-  determine the defaults and runtime environment. The ``__init__`` method has
-  also been refactored so this class can be instantiated and run without the
-  old :mod:`ipmaker` module.
+  :class:`~IPython.config.configurable.Configurable` subclass and has traitlets
+  that determine the defaults and runtime environment. The ``__init__`` method
+  has also been refactored so this class can be instantiated and run without
+  the old :mod:`ipmaker` module.
 
 * The methods of :class:`~IPython.core.interactiveshell.InteractiveShell` have
   been organized into sections to make it easier to turn more sections
@@ -208,18 +219,15 b' New features'
   strings like ``foo.bar.Bar`` to the actual class.
 
 * Completely refactored the :mod:`IPython.core.prefilter` module into
-  :class:`~IPython.config.configurable.Configurable` subclasses. Added a new layer
-  into the prefilter system, called "transformations" that all new prefilter
-  logic should use (rather than the older "checker/handler" approach).
+  :class:`~IPython.config.configurable.Configurable` subclasses. Added a new
+  layer into the prefilter system, called "transformations" that all new
+  prefilter logic should use (rather than the older "checker/handler"
+  approach).
 
 * Aliases are now components (:mod:`IPython.core.alias`).
 
-* We are now using an internally shipped version of
-  :mod:`~IPython.external.argparse` to parse command line options for
-  :command:`ipython`.
-
 * New top level :func:`~IPython.frontend.terminal.embed.embed` function that can
-  be called to embed IPython at any place in user's code. One the first call it
+  be called to embed IPython at any place in user's code. On the first call it
   will create an :class:`~IPython.frontend.terminal.embed.InteractiveShellEmbed`
   instance and call it. In later calls, it just calls the previously created
   :class:`~IPython.frontend.terminal.embed.InteractiveShellEmbed`.
@@ -263,13 +271,14 b' New features'
 
 * :mod:`IPython.utils.ipstruct` has been refactored to be a subclass of 
   dict.  It also now has full docstrings and doctests.
-* Created a Trait's like implementation in :mod:`IPython.utils.traitlets`.
-  This is a pure Python, lightweight version of a library that is similar to
-  :mod:`enthought.traits`.  We are using this for validation, defaults and
-  notification in our new component system.  Although it is not API compatible
-  with :mod:`enthought.traits`, we plan on moving in this direction so that
-  eventually our implementation could be replaced by a (yet to exist) pure
-  Python version of :mod:`enthought.traits`.
+
+* Created a Traits like implementation in :mod:`IPython.utils.traitlets`.  This
+  is a pure Python, lightweight version of a library that is similar to
+  Enthought's Traits project, but has no dependencies on Enthought's code.  We
+  are using this for validation, defaults and notification in our new component
+  system.  Although it is not 100% API compatible with Enthought's Traits, we
+  plan on moving in this direction so that eventually our implementation could
+  be replaced by a (yet to exist) pure Python version of Enthought Traits.
 
 * Added a new module :mod:`IPython.lib.inputhook` to manage the integration
   with GUI event loops using `PyOS_InputHook`.  See the docstrings in this
@@ -362,10 +371,9 b' Backwards incompatible changes'
 * :mod:`Extensions` has been moved to :mod:`extensions` and all existing
   extensions have been moved to either :mod:`IPython.quarantine` or
   :mod:`IPython.deathrow`. :mod:`IPython.quarantine` contains modules that we
-  plan on keeping but that need to be updated. :mod:`IPython.deathrow`
-  contains modules that are either dead or that should be maintained as third
-  party libraries. More details about this can be found :ref:`here
-  <module_reorg>`.
+  plan on keeping but that need to be updated. :mod:`IPython.deathrow` contains
+  modules that are either dead or that should be maintained as third party
+  libraries. More details about this can be found :ref:`here <module_reorg>`.
 
 * Previous IPython GUIs in :mod:`IPython.frontend` and :mod:`IPython.gui` are
   likely broken, and have been removed to :mod:`IPython.deathrow` because of the
@@ -375,17 +383,167 b' Backwards incompatible changes'
 Known Regressions
 -----------------
 
-We do our best to improve IPython, but there are some known regressions in 0.11 relative
-to 0.10.2.
+We do our best to improve IPython, but there are some known regressions in 0.11
+relative to 0.10.2.  First of all, there are features that have yet to be
+ported to the new APIs, and in order to ensure that all of the installed code
+runs for our users, we have moved them to two separate directories in the
+source distribution, `quarantine` and `deathrow`.  Finally, we have some other
+miscellaneous regressions that we hope to fix as soon as possible.  We now
+describe all of these in more detail.
+
+Quarantine
+~~~~~~~~~~
+
+These are tools and extensions that we consider relatively easy to update to
+the new classes and APIs, but that we simply haven't had time for.  Any user
+who is interested in one of these is encouraged to help us by porting it and
+submitting a pull request on our `development site`_.
+
+.. _development site:: http://github.com/ipython/ipython
+
+Currently, the quarantine directory contains::
+
+    clearcmd.py            ipy_fsops.py            ipy_signals.py
+    envpersist.py          ipy_gnuglobal.py        ipy_synchronize_with.py
+    ext_rescapture.py      ipy_greedycompleter.py  ipy_system_conf.py
+    InterpreterExec.py     ipy_jot.py              ipy_which.py
+    ipy_app_completers.py  ipy_lookfor.py          ipy_winpdb.py
+    ipy_autoreload.py      ipy_profile_doctest.py  ipy_workdir.py
+    ipy_completers.py      ipy_pydb.py             jobctrl.py
+    ipy_editors.py         ipy_rehashdir.py        ledit.py
+    ipy_exportdb.py        ipy_render.py           pspersistence.py
+    ipy_extutil.py         ipy_server.py           win32clip.py
+
+Deathrow
+~~~~~~~~
+
+These packages may be harder to update or make most sense as third-party
+libraries.  Some of them are completely obsolete and have been already replaced
+by better functionality (we simply haven't had the time to carefully weed them
+out so they are kept here for now). Others simply require fixes to code that
+the current core team may not be familiar with.  If a tool you were used to is
+included here, we encourage you to contact the dev list and we can discuss
+whether it makes sense to keep it in IPython (if it can be maintained).
+
+Currently, the deathrow directory contains::
+
+    astyle.py              ipy_defaults.py          ipy_vimserver.py
+    dtutils.py             ipy_kitcfg.py            numeric_formats.py
+    Gnuplot2.py            ipy_legacy.py            numutils.py
+    GnuplotInteractive.py  ipy_p4.py                outputtrap.py
+    GnuplotRuntime.py      ipy_profile_none.py      PhysicalQInput.py
+    ibrowse.py             ipy_profile_numpy.py     PhysicalQInteractive.py
+    igrid.py               ipy_profile_scipy.py     quitter.py*
+    ipipe.py               ipy_profile_sh.py        scitedirector.py
+    iplib.py               ipy_profile_zope.py      Shell.py
+    ipy_constants.py       ipy_traits_completer.py  twshell.py
+
+
+Other
+~~~~~
+
+* The machinery that adds functionality to the 'sh' profile for using IPython
+  as your system shell has not been updated to use the new APIs.  As a result,
+  only the aesthetic (prompt) changes are still implemented. We intend to fix
+  this by 0.12.
+
+* The installation of scripts on Windows was broken without setuptools, so we
+  now depend on setuptools on Windows.  We hope to fix setuptools-less
+  installation, and then remove the setuptools dependency.
+
+
+Removed Features
+----------------
+
+As part of the updating of IPython, we have removed a few features for the
+purpsoes of cleaning up the codebase and interfaces.  These removals are
+permanent, but for any item listed below, equivalent functionality is
+available.
+
+* The magics Exit and Quit have been dropped as ways to exit IPython. Instead,
+  the lowercase forms of both work either as a bare name (``exit``) or a
+  function call (``exit()``).  You can assign these to other names using
+  exec_lines in the config file.
 
-* The machinery that adds functionality to the 'sh' profile for using IPython as your
-  system shell has not been updated to use the new APIs.  As a result, only the aesthetic
-  (prompt) changes are still implemented. We intend to fix this by 0.12.
 
-* The installation of scripts on Windows was broken without setuptools, so we now
-  depend on setuptools on Windows.  We hope to fix setuptools-less installation,
-  and then remove the setuptools dependency.
+.. credits_::
+
+Credits
+-------
+
+Many users and developers contributed code, features, bug reports and ideas to
+this release.  Please do not hesitate in contacting us if we've failed to
+acknowledge your contribution here.  In particular, for this release we have
+contribution from the following people, a mix of new and regular names (in
+alphabetical order by first name):
+
+* Aenugu Sai Kiran Reddy <saikrn08-at-gmail.com>
+* andy wilson <wilson.andrew.j+github-at-gmail.com>
+* Antonio Cuni <antocuni>
+* Barry Wark <barrywark-at-gmail.com>
+* Beetoju Anuradha <anu.beethoju-at-gmail.com>
+* Benjamin Ragan-Kelley <minrk-at-Mercury.local>
+* Brad Reisfeld
+* Brian E. Granger <ellisonbg-at-gmail.com>
+* Christoph Gohlke <cgohlke-at-uci.edu>
+* Cody Precord
+* dan.milstein
+* Darren Dale <dsdale24-at-gmail.com>
+* Dav Clark <davclark-at-berkeley.edu>
+* David Warde-Farley <wardefar-at-iro.umontreal.ca>
+* epatters <ejpatters-at-gmail.com>
+* epatters <epatters-at-caltech.edu>
+* epatters <epatters-at-enthought.com>
+* Eric Firing <efiring-at-hawaii.edu>
+* Erik Tollerud <erik.tollerud-at-gmail.com>
+* Evan Patterson <epatters-at-enthought.com>
+* Fernando Perez <Fernando.Perez-at-berkeley.edu>
+* Gael Varoquaux <gael.varoquaux-at-normalesup.org>
+* Gerardo <muzgash-at-Muzpelheim>
+* Jason Grout <jason.grout-at-drake.edu>
+* John Hunter <jdh2358-at-gmail.com>
+* Jens Hedegaard Nielsen <jenshnielsen-at-gmail.com>
+* Johann Cohen-Tanugi <johann.cohentanugi-at-gmail.com>
+* Jörgen Stenarson <jorgen.stenarson-at-bostream.nu>
+* Justin Riley <justin.t.riley-at-gmail.com>
+* Kiorky
+* Laurent Dufrechou <laurent.dufrechou-at-gmail.com>
+* Luis Pedro Coelho <lpc-at-cmu.edu>
+* Mani chandra <mchandra-at-iitk.ac.in>
+* Mark E. Smith
+* Mark Voorhies <mark.voorhies-at-ucsf.edu>
+* Martin Spacek <git-at-mspacek.mm.st>
+* Michael Droettboom <mdroe-at-stsci.edu>
+* MinRK <benjaminrk-at-gmail.com>
+* muzuiget <muzuiget-at-gmail.com>
+* Nick Tarleton <nick-at-quixey.com>
+* Nicolas Rougier <Nicolas.rougier-at-inria.fr>
+* Omar Andres Zapata Mesa <andresete.chaos-at-gmail.com>
+* Paul Ivanov <pivanov314-at-gmail.com>
+* Pauli Virtanen <pauli.virtanen-at-iki.fi>
+* Prabhu Ramachandran
+* Ramana <sramana9-at-gmail.com>
+* Robert Kern <robert.kern-at-gmail.com>
+* Sathesh Chandra <satheshchandra88-at-gmail.com>
+* Satrajit Ghosh <satra-at-mit.edu>
+* Sebastian Busch
+* Stefan van der Walt <bzr-at-mentat.za.net>
+* Stephan Peijnik <debian-at-sp.or.at>
+* Steven Bethard
+* Thomas Kluyver <takowl-at-gmail.com>
+* Thomas Spura <tomspur-at-fedoraproject.org>
+* Tom Fetherston <tfetherston-at-aol.com>
+* Tom MacWright
+* tzanko
+* vankayala sowjanya <hai.sowjanya-at-gmail.com>
+* Vivian De Smedt <vds2212-at-VIVIAN>
+* Ville M. Vainio <vivainio-at-gmail.com>
+* Vishal Vatsa <vishal.vatsa-at-gmail.com>
+* Vishnu S G <sgvishnu777-at-gmail.com>
+* Walter Doerwald <walter-at-livinglogic.de>
+
+.. note::
 
-* Capitalised Exit and Quit have been dropped ways to exit IPython. The lowercase forms
-  of both work either as a bare name (``exit``) or a function call (``exit()``).
-  You can assign these to other names using exec_lines in the config file.
+    This list was generated with the output of
+    ``git log dev-0.11 HEAD --format='* %aN <%aE>' | sed 's/@/\-at\-/' | sed 's/<>//' | sort -u``
+    after some cleanup.  If you should be on this list, please add yourself.