docs/source/development/reorg.txt

=============================
IPython module reorganization
=============================

Currently, IPython has many top-level modules that serve many different purposes.  The lack of organization make it very difficult for developers to work on IPython and understand its design.  This document contains notes about how we will reorganize the modules into sub-packages.

.. warning::

    This effort will possibly break third party packages that use IPython as
    a library or hack on the IPython internals.

.. warning::

    This effort will result in the removal from IPython of certain modules
    that are not used anymore, don't currently work, are unmaintained, etc.

Current subpackges
==================

IPython currently has the following sub-packages:

* :mod:`IPython.config`

* :mod:`IPython.Extensions`

* :mod:`IPython.external`

* :mod:`IPython.frontend`

* :mod:`IPython.gui`

* :mod:`IPython.kernel`

* :mod:`IPython.testing`

* :mod:`IPython.tests`

* :mod:`IPython.tools`

* :mod:`IPython.UserConfig`

New Subpackages to be created
=============================

We propose to create the following new sub-packages:

* :mod:`IPython.core`. This sub-package will contain the core of the IPython
  interpreter, but none of its extended capabilities.

* :mod:`IPython.lib`. IPython has many extended capabilities that are not part
  of the IPython core. These things go here and sit on top of the IPython
  core.

* :mod:`IPython.python`. This sub-package will contain anything that might
  eventually be found in the Python standard library. These things should only
  depend on Python and the standard library.  Each sub-module in this
  sub-package should contain functions and objects that serve a single
  purpose. Similar in purpose to :mod:`twisted.python`. Could also call this
  :mod:`IPython.tools` or something similar.

* :mod:`IPython.sandbox`. This is for code that is untested and/or rotting and
  needs to be removed from IPython. Eventually all this code will either i) be
  revived with tests and docs and re-included into IPython or 2) be put into a
  separate top-level (not IPython) package that we keep around.

Where things will be moved
==========================

* :mod:`ColorANSI.py`.  Move to :mod:`IPython.core`.

* :mod:`ConfigLoader.py`.  Move to :mod:`IPython.config`.

* :mod:`CrashHandler.py`.  Move to :mod:`IPython.core`.

* :mod:`DPyGetOpt.py`.  Move the :mod:`IPython.sandbox` and replace with newer options parser.

* :mod:`Debugger.py`.  Move to :mod:`IPython.core`.

* :mod:`Extensions`. This needs to be gone through separately. By default
  everything here should simply be put into :mod:`IPython.core`. Of course, as
  we do this, we will realize that some things belong in places like
  :mod:`IPython.core`, :mod:`IPython.lib` and :mod:`IPython.python`.

* :mod:`FakeModule.py`.  Move to :mod:`IPython.core`.

* :mod:`Gnuplot2.py`.  Move to :mod:`IPython.sandbox`.

* :mod:`GnuplotInteractive.py`.  Move to :mod:`IPython.sandbox`.

* :mod:`GnuplotRuntime.py`.  Move to :mod:`IPython.sandbox`.

* :mod:`Itpl.py`.  Remove.  Version already in :mod:`IPython.external`.

* :mod:`Logger.py`.  Move to :mod:`IPython.core`.

* :mod:`Magic.py`.  Move to :mod:`IPython.core`.

* :mod:`OInspect.py`.  Move to :mod:`IPython.core`.

* :mod:`OutputTrap.py`.  Move to :mod:`IPython.core`.

* :mod:`Prompts.py`.  Move to :mod:`IPython.core` or :mod:`IPython.frontend`.

* :mod:`PyColorize.py`.  Replace with pygments?  If not, move to :mod:`IPython.core`.

* :mod:`Release.py`.  Move to ???.

* :mod:`Shell.py`.  Move to :mod:`IPython.core` or :mod:`IPython.frontend`.

* :mod:`UserConfig`.  Move to a subdirectory of :mod:`IPython.config`.

* :mod:`background_jobs.py`.  Move to :mod:`IPython.lib`.

* :mod:`completer.py`.  Move to :mod:`IPython.core`.

* :mod:`config`.  Good where it is!

* :mod:`deep_reload.py`.  Move to :mod:`IPython.lib`.

* :mod:`demo.py`.  Move to :mod:`IPython.lib`.

* :mod:`dtutils.py`.  Move to :mod:`IPython.testing` or :mod:`IPython.lib`.

* :mod:`excolors.py`.  Move to :mod:`IPython.core` or :mod:`IPython.config`.

* :mod:`external`.  Good where it is!

* :mod:`frontend`.  Good where it is!

* :mod:`generics.py`.  Move to :mod:`IPython.python`.

* :mod:`genutils.py`.  Move to :mod:`IPython.python` and break up into different modules.

* :mod:`gui`.  Eventually this should be moved to a subdir of :mod:`IPython.python`

* :mod:`history.py`.  Move to :mod:`IPython.core`.

* :mod:`hooks.py`.  Move to :mod:`IPython.core`.

* :mod:`ipapi.py`.  Move to :mod:`IPython.core`.

* :mod:`iplib.py`.  Move to :mod:`IPython.core`.

* :mod:`ipmaker.py`:  Move to :mod:`IPython.core`.

* :mod:`ipstruct.py`.  Move to :mod:`IPython.python`.

* :mod:`irunner.py`.  Move to :mod:`IPython.scripts`.

* :mod:`kernel`.  Good where it is.

* :mod:`macro.py`.  Move to :mod:`IPython.core`.

* :mod:`numutils.py`.  Move to :mod:`IPython.sandbox`.

* :mod:`platutils.py`.  Move to :mod:`IPython.python`.

* :mod:`platutils_dummy.py`.  Move to :mod:`IPython.python`.

* :mod:`platutils_posix.py`.  Move to :mod:`IPython.python`.

* :mod:`platutils_win32.py`.  Move to :mod:`IPython.python`.

* :mod:`prefilter.py`:  Move to :mod:`IPython.core`.

* :mod:`rlineimpl.py`.  Move to :mod:`IPython.core`.

* :mod:`shadowns.py`.  Move to :mod:`IPython.core`.

* :mod:`shellglobals.py`.  Move to :mod:`IPython.core`.

* :mod:`strdispatch.py`.  Move to :mod:`IPython.python`.

* :mod:`testing`.  Good where it is.

* :mod:`tests`.  Good where it is.

* :mod:`tools`.  Things in here need to be looked at and moved elsewhere.

* :mod:`twshell.py`.  Move to :mod:`IPython.sandbox`.

* :mod:`ultraTB.py`.  Move to :mod:`IPython.core`.

* :mod:`upgrade_dir.py`.  Move to :mod:`IPython.python`.

* :mod:`usage.py`.  Move to :mod:`IPython.core`.

* :mod:`wildcard.py`.  Move to :mod:`IPython.python` or :mod:`IPython.core`.

* :mod:`winconsole.py`.  Move to :mod:`IPython.lib`.

Other things
============

When these files are moved around, a number of other things will happen at the same time:

1. Test files will be created for each module in IPython. Minimally, all
   modules will be imported as a part of the test. This will serve as a 
   test of the module reorganization. These tests will be put into new
   :file:`tests` subdirectories that each package will have.

2. Modules will be renamed to comply with PEP 8 naming conventions:  all
   lowercase and no special characters like ``-`` or ``_``.

3. Existing tests will be moved to the appropriate :file:`tests`
   subdirectories.