Add 0.13 release notes - in progress.

docs/source/whatsnew/version0.13.txt

@@ -0,0 +1,577
+=============
+ 0.13 Series
+=============
+
+Release 0.13
+============
+
+IPython 0.13 contains several major new features, as well as a large amount of
+bug and regression fixes.  The previous version (0.12) was released on December
+19 2011, so this release cycle was roughly 6 months long, during which we
+closed a total of 373 pull requests and 742 issues, with contributions from 62
+authors comprising over 1740 commits.
+
+The amount of work included in this release is so large, that we can only cover
+here the main highlights; please see our :ref:`detailed release statistics
+<issues_list_013>` for links to every issue and pull request closed on github
+
+
+Major Notebook improvements: new user interface and more
+--------------------------------------------------------
+
+The IPython Notebook, which has proven since its release to be wildly popular,
+has seen a massive amount of work in this release cycle, leading to a
+significantly improved user experience as well as many new features.
+
+The first user-visible change is a reorganization of the user interface; the
+left panel has been removed and was replaced by a real menu system and a
+toolbar with icons.  Both the toolbar and the header above the menu can be
+collapsed to leave an unobstructed working area:
+
+.. image:: ../_static/ipy_013_notebook_spectrogram.png
+    :width: 400px
+    :alt: New user interface for Notebook
+    :align: center
+    :target: ../_static/ipy_013_notebook_spectrogram.png
+
+The notebook handles very long outputs much better than before (this was a
+serious usability issue when running processes that generated massive amounts
+of output).  Now, in the presence of outputs longer than ~100 lines, the
+notebook will automatically collapse to a scrollable area and the entire left
+part of this area controls the display: one click in this area will expand the
+output region completely, and a double-click will hide it completely.  This
+figure shows both the scrolled and hidden modes:
+
+.. image:: ../_static/ipy_013_notebook_long_out.png
+    :width: 400px
+    :alt: Scrolling and hiding of long output in the notebook.
+    :align: center
+    :target: ../_static/ipy_013_notebook_long_out.png
+
+.. note::
+
+   The auto-folding of long outputs is disabled in Firefox due to bugs in its
+   scrolling behavior.  See :ghpull:`2047` for details.
+
+Uploading notebooks to the dashboard is now easier: in addition to drag and
+drop (which can be finicky sometimes), you can now click on the upload text and
+use a regular file dialog box to select notebooks to upload. Furthermore, the
+notebook dashboard now auto-refreshes its contents and offers buttons to shut
+down any running kernels (:ghpull:`1739`):
+
+.. image:: ../_static/ipy_013_dashboard.png
+    :width: 400px
+    :alt: Improved dashboard
+    :align: center
+    :target: ../_static/ipy_013_dashboard.png
+
+
+Cluster management
+~~~~~~~~~~~~~~~~~~
+
+The notebook dashboard can now also start and stop clusters, and you can
+override the number of engines started.  There is a new tab in the dashboard
+user interface:
+
+.. image:: ../_static/ipy_013_dashboard_cluster.png
+    :width: 400px
+    :alt: Cluster management from the notebook dashboard
+    :align: center
+    :target: ../_static/ipy_013_dashboard_cluster.png
+
+This tab allows you, for each profile you have configured, to start and stop a
+cluster (and optionally override the default number of engines corresponding to
+that configuration).  While this hides all error reporting, once you have a
+configuration that you know works smoothly, it is a very convenient interface
+for controlling your parallel resources.
+
+
+New notebook format
+~~~~~~~~~~~~~~~~~~~
+
+The notebooks saved now use version 3 of our format, which supports heading
+levels as well as the concept of 'raw' text cells that are not rendered as
+Markdown.  These will be useful with converters_ we are developing, to pass raw
+markup (say LaTeX).  That conversion code is still under heavy development and
+not quite ready for prime time, but we welcome help on this front so that we
+can merge it for full production use as soon as possible.
+
+.. _converters: https://github.com/ipython/nbconvert
+
+.. note::
+
+   v3 notebooks can *not* be read by older versions of IPython, but we provide
+   a `simple script`__ that you can use in case you need to export a v3
+   notebook to share with a v2 user.
+
+.. _simple script: https://gist.github.com/1935808
+
+
+JavaScript refactoring
+~~~~~~~~~~~~~~~~~~~~~~
+  
+All the client-side JavaScript has been decoupled to ease reuse of parts of the
+machinery without having to build a full-blown notebook. This will make it much
+easier to communicate with an IPython kernel from existing web pages and to
+integrate single cells into other sites, without loading the full notebook
+document-like UI. :ghpull:`1711`.
+    
+This refactoring also enables the possibility of writing dynamic javascript
+widgets that are returned from Python code and that present an interactive view
+to the user, with callbacks in Javascript executing calls to the Kernel.  This
+will enable many interactive elements to be added by users in notebooks.
+
+An example of this capability has been provided as a proof of concept in
+`docs/examples/widgets` that lets you directly communicate with one or more
+parallel engines, acting as a mini-console for parallel debugging and
+introspection.
+
+    
+Improved tooltips
+~~~~~~~~~~~~~~~~~
+
+The object tooltips have gained some new functionality. By pressing tab several
+times, you can expand them to see more of a docstring, keep them visible as you
+fill in a function's parameters, or transfer the information to the pager at the
+bottom of the screen. For the details, look at the example notebook
+:file:`01_notebook_introduction.ipynb`.
+
+.. figure:: ../_static/ipy_013_notebook_tooltip.png
+    :width: 400px
+    :alt: Improved tooltips in the notebook.
+    :align: center
+    :target: ../_static/ipy_013_notebook_tooltip.png
+
+    The new notebook tooltips.
+
+Other improvements to the Notebook
+----------------------------------
+   
+* The notebook pager (the area at the bottom) is now resizeable by dragging its
+  divider handle, a feature that had been requested many times by just about
+  anyone who had used the notebook system.  :ghpull:`1705`.
+
+* If a notebook directory is specified with ``--notebook-dir`` (or with the
+  corresponding configuration flag ``NotebookManager.notebook_dir``), all
+  kernels start in this directory.
+
+* It is now possible to open notebooks directly from the command line; for
+  example: ``ipython notebook path/`` will automatically set ``path/`` as the
+  notebook directory, and ``ipython notebook path/foo.ipynb`` will further
+  start with the ``foo.ipynb`` notebook opened.  :ghpull:`1686`.
+  
+* Fix codemirror clearing of cells with ``Ctrl-Z``; :ghpull:`1965`.
+  
+* Text (markdown) cells now line wrap correctly in the notebook, making them
+  much easier to edit :ghpull:`1330`.
+  
+* PNG and JPEG figures returned from plots can be interactively resized in the
+  notebook, by dragging them from their lower left corner. :ghpull:`1832`.
+
+* Clear In[] prompt numbers on "Clear All Output".  For more
+  version-control-friendly `.ipynb` files, this strips the `In[]` prompt
+  numbers when doing a "Clear all output".  This reduces the amount of noise in
+  commit-to-commit diffs that would otherwise show the (highly variable) prompt
+  number changes. :ghpull:`1621`.
+
+* The notebook server now requires *two* consecutive ``Ctrl-C`` to stop within 5
+  seconds (or an interactive confirmation).  This makes it less likely that you
+  will accidentally kill a long-running server by typing ``Ctrl-C`` in the
+  wrong terminal.  :ghpull:`1609`.
+
+* Using ``Ctrl-S`` (or ``Cmd-S`` on a Mac) actually saves the notebook rather
+  than providing the fairly useless browser html save dialog.  :ghpull:`1334`.
+  
+* Allow accessing local files from the notebook (in urls), by serving any local
+  file as the url ``files/<relativepath``.  This makes it possible to, for
+  example, embed local images in a notebook.  :ghpull:`1211`.
+
+      
+Cell magics
+-----------
+
+We have completely refactored the magic system, finally moving the magic
+objects to standalone, independent objects instead of being the mixin class
+we'd had since the beginning of IPython (:ghpull:`1732`).  Now, a separate base
+class is provided in :class:`IPython.core.magic.Magics` that users can subclass
+to create their own magics.  Decorators are also provided to create magics from
+simple functions without the need for object orientation.
+    
+All builtin magics now exist in a few subclasses that group together related
+functionality, and the new :mod:`IPython.core.magics` package has been created
+to organize this into smaller files.
+    
+This cleanup was the last major piece of deep refactoring needed from the
+original 2001 codebase.
+    
+We have also introduced a new type of magic function, prefixed with `%%`
+instead of `%`, which operates at the cell level.  A cell magic receives two
+arguments: the line it is called on (like a line magic) and the body of the
+cell below it.
+    
+Cell magics are most natural in the notebook, but they also work in the
+terminal and qt console, with the usual approach of using a blank line to
+signal cell termination.
+    
+For example, to time the execution of several statements::
+
+    %%timeit x = 0   # setup
+    for i in range(100000):
+        x += i**2
+
+This is particularly useful to integrate code in another language, and cell
+magics already exist for shell scripts, Cython, R and Octave. Using ``%%script
+/usr/bin/foo``, you can run a cell in any interpreter that accepts code via
+stdin.
+
+Another handy cell magic makes it easy to write short text files: ``%%file
+~/save/to/here.txt``.
+
+The following cell magics are now included by default; all those that use
+special interpreters (Perl, Ruby, bash, etc., assume you have the requisite
+interpreter installed):
+
+* ``%%!``: run cell body with the underlying OS shell; this is similar to
+  prefixing every line in the cell with ``!``.
+  
+* ``%%bash``: run cell body under bash.
+  
+* ``%%capture``: capture the output of the code in the cell (and stderr as
+  well).  Useful to run codes that produce too much output that you don't even
+  want scrolled.
+  
+* ``%%file``: save cell body as a file.
+  
+* ``%%perl``: run cell body using Perl.
+  
+* ``%%prun``: run cell body with profiler (cell extension of ``%prun``).
+  
+* ``%%python3``: run cell body using Python 3.
+  
+* ``%%ruby``: run cell body using Ruby.
+  
+* ``%%script``: run cell body with the script specified in the first line.
+  
+* ``%%sh``: run cell body using sh.
+  
+* ``%%sx``: capture cell output running the code with the system shell (cell
+  extension of ``%sx``).
+  
+* ``%%system``: run cell with system shell (similar to ``%%!``)
+  
+* ``%%timeit``: time the execution of the cell (extension of ``%timeit``).
+
+This is what some of the script-related magics look like in action:
+
+.. image:: ../_static/ipy_013_notebook_script_cells.png
+    :width: 400px
+    :alt: Cluster management from the notebook dashboard
+    :align: center
+    :target: ../_static/ipy_013_notebook_script_cells.png
+  
+In addition, we have also a number of :ref:`extensions <extensions_overview>`
+that provide specialized magics.  These typically require additional software
+to run and must be manually loaded via ``%load_ext <extension name>``, but are
+extremely useful.  In particular, we want to highlight a few:
+
+
+
+
+Tab completer improvements
+--------------------------
+
+Useful tab-completion based on live inspection of objects is one of the most
+popular features of IPython. To make this process even more user-friendly, the
+completers of both the Qt console and the Notebook have been reworked.
+
+The Qt console comes with a new ncurses-like tab completer, activated by
+default, which lets you cycle through the available completions by pressing tab,
+or select a completion with the arrow keys (:ghpull:`1851`).
+
+.. figure:: ../_static/ipy_013_qtconsole_completer.png
+    :width: 400px
+    :alt: ncurses-like completer, with highlighted selection.
+    :align: center
+    :target: ../_static/ipy_013_qtconsole_completer.png
+
+    The new improved Qt console's ncurses-like completer allows to easily
+    navigate thought long list of completions.
+
+In the notebook, completions are now sourced both from object introspection and
+analysis of surrounding code, so limited completions can be offered for
+variables defined in the current cell, or while the kernel is busy 
+(:ghpull:`1711`).
+
+
+We have implemented a new configurable flag to control tab completion on
+modules that provide the ``__all__`` attribute::
+
+  IPCompleter.limit_to__all__= Boolean.
+
+This instructs the completer to honor ``__all__`` for the completion.
+Specifically, when completing on ``object.<tab>``, if True: only those names
+in ``obj.__all__`` will be included.  When False [default]: the ``__all__``
+attribute is ignored. :ghpull:`1529`.
+
+
+Improvements to the Qt console
+------------------------------
+
+* changes for easier integration into other projects such as Spyder_.
+
+.. _spyder: https://code.google.com/p/spyderlib
+
+* Improved menus with a new Magic menu that is organized by magic groups (this
+  was made possible by the reorganization of the magic system
+  internals). :ghpull:`1782`.
+
+* Allow for restarting kernels without clearing the qtconsole, while leaving a
+  visible indication that the kernel has restarted. :ghpull:`1681`.
+  
+* Allow the native display of jpeg image in the qtconsole. :ghpull:`1643`.
+
+  
+Parallel
+--------
+
+The parallel tools have been improved and fine-tuned on multiple fronts.  Now,
+the creation of an :class:`IPython.parallel.Client` object automatically
+activates a line and cell magic function ``px`` that sends its code to all the
+engines. Further magics can be easily created with the :meth:`Client.activate`
+method, to conveniently execute code on any subset of engines. :ghpull:`1893`.
+
+The ``%%px`` cell magic can also be given an optional targets argument, as well
+as a ``--out`` argument for storing its output.
+
+A new magic has also been added, ``%pxconfig``, that lets you configure various
+defaults of the parallel magics.  As usual, type  ``%pxconfig?`` for details.
+
+The exception reporting in parallel contexts has been improved to be easier to
+read.  Now, IPython directly reports the remote exceptions without showing any
+of the internal execution parts:
+
+.. image::  ../_static/ipy_013_par_tb.png
+    :width: 400px
+    :alt: Improved parallel exceptions.
+    :align: center
+    :target: ../_static/ipy_013_par_tb.png
+
+
+The parallel tools now default to using ``NoDB`` as the storage backend for
+intermediate results.  This means that the default usage case will have a
+significantly reduced memory footprint, though certain advanced features are
+not available with this backend.  For more details, see :ref:`parallel_db`.
+
+The parallel magics now display all output, so you can do parallel plotting or
+other actions with complex display.  The ``px`` magic has now both line and cell
+modes, and in cell mode finer control has been added about how to collate
+output from multiple engines. :ghpull:`1768`.
+
+Incremental improvements to SSH launchers:
+    
+* add to_send/fetch steps for moving connection files around.
+  
+* add SSHProxyEngineSetLauncher, for invoking to `ipcluster engines` on a
+  remote host. This can be used to start a set of engines via PBS/SGE/MPI
+  *remotely*.
+    
+This makes the SSHLauncher usable on machines without shared filesystems.
+    
+When sending files, the destination directory must *already exist* - that is,
+`ipython profile create` may be necessary on the remote system, before the
+security dir exists for putting the connection file the first
+time. :ghpull:`1634`.
+
+Add sugar methods/properties to AsyncResult that are generically useful
+(:ghpull:`1548`):
+    
+    * ``ar.wall_time`` = received - submitted
+    * ``ar.serial_time`` = sum of serial computation time
+    * ``ar.elapsed`` = time since submission (wall_time if done)
+    * ``ar.progress`` = (int) number of sub-tasks that have completed
+    * ``len(ar)`` = # of tasks
+    * ``ar.wait_interactive()``: prints progress
+    
+Added ``Client.spin_thread(interval)`` / ``stop_spin_thread()`` for running
+spin in a background thread, to keep zmq queue clear.  This can be used to
+ensure that timing information is as accurate as possible (at the cost of
+having a background thread active).
+
+Set TaskScheduler.hwm default to 1 instead of 0.  1 has more
+predictable/intuitive behavior, if often slower, and thus a more logical
+default.  Users whose workloads require maximum throughput and are largely
+homogeneous in time per task can make the optimization themselves, but now the
+behavior will be less surprising to new users. :ghpull:`1294`.
+    
+
+Kernel/Engine unification
+-------------------------
+
+:func:`IPython.parallel.bind_kernel`
+      
+Add IPython.embed_kernel() as a public API.
+Embedding an IPython kernel in an application is useful when you want to use
+IPython.embed() but don't have a terminal attached on stdin and stdout. 1640
+
+
+
+Official Public API
+-------------------
+
+We have begun organizing our API for easier public use, with an eye towards an
+official IPython 1.0 release which will firmly maintain this API compatible for
+its entire lifecycle.  There is now an :mod:`IPython.display` module that
+aggregates all display routines, and the :mod:`IPython.config` namespace has
+all public configuration tools.  We will continue improving our public API
+layout so that users only need to import names one level deeper than the main
+``IPython`` package to access all public namespaces.
+
+
+IPython notebook file icons
+---------------------------
+
+The directory ``docs/resources`` in the source distribution contains SVG and
+PNG versions of our file icons, as well as an ``Info.plist.example`` file with
+instructions to install them on Mac OSX.  This is a first draft of our icons,
+and we encourage contributions from users with graphic talent to improve them
+in the future:
+
+.. image:: ../../resources/ipynb_icon_128x128.png
+   :alt:  IPython notebook file icon.
+
+	  
+New top-level `locate` command
+------------------------------
+
+Add `locate` entry points; these would be useful for quickly locating IPython
+directories and profiles from other (non-Python) applications. :ghpull:`1762`.
+    
+Examples::
+    
+    $> ipython locate
+    /Users/me/.ipython
+  
+    $> ipython locate profile foo
+    /Users/me/.ipython/profile_foo
+  
+    $> ipython locate profile
+    /Users/me/.ipython/profile_default
+  
+    $> ipython locate profile dne
+    [ProfileLocate] Profile u'dne' not found.
+
+	
+Other new features and improvements
+-----------------------------------
+
+* **%install_ext**: A new magic function to install an IPython extension from
+  a URL. E.g. ``%install_ext
+  https://bitbucket.org/birkenfeld/ipython-physics/raw/default/physics.py``.
+
+* The ``%loadpy`` magic is no longer restricted to Python files, and has been
+  renamed ``%load``. The old name remains as an alias.
+
+* New command line arguments will help external programs find IPython folders:
+  ``ipython locate`` finds the user's IPython directory, and ``ipython locate
+  profile foo`` finds the folder for the 'foo' profile (if it exists).
+
+* The :envvar:`IPYTHON_DIR` environment variable, introduced in the Great
+  Reorganization of 0.11 and existing only in versions 0.11-0.13, has been
+  deprecated. As described in :ghpull:`1167`, the complexity and confusion of
+  migrating to this variable is not worth the aesthetic improvement. Please use
+  the historical :envvar:`IPYTHONDIR` environment variable instead.
+
+* The default value of *interactivity* passed from
+  :meth:`~IPython.core.interactiveshell.InteractiveShell.run_cell` to
+  :meth:`~IPython.core.interactiveshell.InteractiveShell.run_ast_nodes`
+  is now configurable.
+
+* New ``%alias_magic`` function to conveniently create aliases of existing
+  magics, if you prefer to have shorter names for personal use.
+
+* We ship unminified versions of the JavaScript libraries we use, to better
+  comply with Debian's packaging policies.
+
+* Simplify the information presented by ``obj?/obj??`` to eliminate a few
+  redundant fields when possible.  :ghpull:`2038`.
+
+* Improved continuous integration for IPython.  We now have automated test runs
+  on `Shining Panda <https://jenkins.shiningpanda.com/ipython>`_ and `Travis-CI
+  <http://travis-ci.org/#!/ipython/ipython>`_, as well as `Tox support
+  <http://tox.testrun.org>`_.
+
+* The `vim-ipython`_ functionality (externally developed) has been updated to
+  the latest version.
+
+.. _vim-ipython: https://github.com/ivanov/vim-ipython
+
+* The ``%save`` magic now has a ``-f`` flag to force overwriting, which makes
+  it much more usable in the notebook where it is not possible to reply to
+  interactive questions from the kernel. :ghpull:`1937`.
+
+* Use dvipng to format sympy.Matrix, enabling display of matrices in the Qt
+  console with the sympy printing extension. :ghpull:`1861`.
+
+* Our messaging protocol now has a reasonable test suite, helping ensure that
+  we don't accidentally deviate from the spec and possibly break third-party
+  applications that may have been using it.  We encourage users to contribute
+  more stringent tests to this part of the test suite.  :ghpull:`1627`.
+
+* Use LaTeX to display, on output, various built-in types with the SymPy
+  printing extension. :ghpull:`1399`.
+
+* Add Gtk3 event loop integration and example. :ghpull:`1588`.
+
+* ``clear_output`` improvements, which allow things like progress bars and other
+  simple animations to work well in the notebook (:ghpull:`1563`):
+    
+    * `clear_output()` clears the line, even in terminal IPython, the QtConsole
+      and plain Python as well, by printing `\r` to streams.
+    
+    * `clear_output()` avoids the flicker in the notebook by adding a delay,
+      and firing immediately upon the next actual display message.
+    
+    * `display_javascript` hides its `output_area` element, so using display to
+      run a bunch of javascript doesn't result in ever-growing vertical space.
+
+* Add simple support for running inside a virtualenv.  While this doesn't
+  supplant proper installation (as users should do), it helps ad-hoc calling of
+  IPython from inside a virtualenv. :ghpull:`1388`.
+
+  
+Major Bugs fixed
+----------------
+
+In this cycle, we have :ref:`closed over 740 issues <issues_list_013>`, but a
+few major ones merit special mention:
+
+* The ``%pastebin`` magic has been updated to point to gist.github.com, since
+  unfortunately http://paste.pocoo.org has closed down. We also added a -d flag
+  for the user to provide a gist description string. :ghpull:`1670`.
+
+* Fix ``%paste`` that would reject certain valid inputs. :ghpull:`1258`.
+
+* Fix sending and receiving of Numpy structured arrays (those with composite
+  dtypes, often used as recarrays). :ghpull:`2034`.
+
+* Reconnect when the websocket connection closes unexpectedly. :ghpull:`1577`.
+
+* Fix truncated representation of objects in the debugger by showing at least
+  80 characters' worth of information.  :ghpull:`1793`.
+
+* Fix logger to be Unicode-aware: logging could crash ipython if there was
+  unicode in the input. :ghpull:`1792`.
+
+* Fix images missing from XML/SVG export in the Qt console. :ghpull:`1449`.
+
+* Fix deepreload on Python 3. :ghpull:`1625`, as well as having a much cleaner
+  and more robust implementation of deepreload in general. :ghpull:`1457`.
+
+
+Backwards incompatible changes
+------------------------------
+
+* The exception :exc:`IPython.core.error.TryNext` previously accepted
+  arguments and keyword arguments to be passed to the next implementation
+  of the hook. This feature was removed as it made error message propagation
+  difficult and violated the principle of loose coupling.