docs/source/development/reorg.txt

=============================
IPython module reorganization
=============================

Currently, IPython has many top-level modules that serve many different purposes.
The lack of organization make it very difficult for developers to work on IPython
and understand its design. This document contains notes about how we will reorganize
the modules into sub-packages.

.. warning::

    This effort will possibly break third party packages that use IPython as
    a library or hack on the IPython internals.

.. warning::

    This effort will result in the removal from IPython of certain modules
    that are not used anymore, don't currently work, are unmaintained, etc.


Current subpackges
==================

IPython currently has the following sub-packages:

* :mod:`IPython.config`

* :mod:`IPython.Extensions`

* :mod:`IPython.external`

* :mod:`IPython.frontend`

* :mod:`IPython.gui`

* :mod:`IPython.kernel`

* :mod:`IPython.testing`

* :mod:`IPython.tests`

* :mod:`IPython.tools`

* :mod:`IPython.UserConfig`

New Subpackages to be created
=============================

We propose to create the following new sub-packages:

* :mod:`IPython.core`. This sub-package will contain the core of the IPython
  interpreter, but none of its extended capabilities.

* :mod:`IPython.lib`. IPython has many extended capabilities that are not part
  of the IPython core. These things will go here.  Any better names than 
  :mod:`IPython.lib`?

* :mod:`IPython.python`. This sub-package will contain anything that might
  eventually be found in the Python standard library, like things in
  :mod:`genutils`. Each sub-module in this sub-package should contain functions 
  and classes that serve a single purpose. Similar in purpose to
  :mod:`twisted.python`. Could also call this :mod:`IPython.tools`,
  :mod:`IPython.utils` or something similar.

* :mod:`IPython.sandbox`. This is for code that is untested and/or rotting and
  needs to be removed from IPython. Eventually all this code will either i) be
  revived with tests and docs and re-included into IPython or 2) be removed from
  IPython proper, but put into a separate top-level (not IPython) package that we
  keep around.

Where things will be moved
==========================

* :file:`ColorANSI.py`.  Move to :file:`IPython/core/coloransi.py`.  Maybe move to
  :mod:`IPython.lib` or :mod:`IPython.python`?

* :file:`ConfigLoader.py`.  Move to :file:`IPython/config/configloader.py`.

* :file:`CrashHandler.py`.  Move to :file:`IPython/core/crashhandler`.

* :file:`DPyGetOpt.py`.  Move to :mod:`IPython.sandbox` and replace with newer options parser.

* :file:`Debugger.py`.  Move to :file:`IPython/core/debugger.py`.

* :file:`Extensions`. This needs to be gone through separately.  Minimally,
  the package should be renamed to :file:`extensions`.

* :file:`FakeModule.py`.  Move to :file:`IPython/core/fakemodule.py`.

* :file:`Gnuplot2.py`.  Move to :file:`IPython.sandbox`.

* :file:`GnuplotInteractive.py`.  Move to :file:`IPython.sandbox`.

* :file:`GnuplotRuntime.py`.  Move to :file:`IPython.sandbox`.

* :file:`Itpl.py`.  Remove.  Version already in :file:`IPython.external`.

* :file:`Logger.py`.  Move to :file:`IPython/core/logger.py`.

* :file:`Magic.py`.  Move to :file:`IPython/core/magic.py`.

* :file:`OInspect.py`.  Move to :file:`IPython/core/oinspect.py`.

* :file:`OutputTrap.py`.  Move to :file:`IPython/core/outputtrap.py`.

* :file:`Prompts.py`.  Move to :file:`IPython/core/prompts.py` or
  :file:`IPython/frontend/prompts.py`.

* :file:`PyColorize.py`. Replace with pygments? If not, move to
  :file:`IPython/core/pycolorize.py`. Maybe move to :mod:`IPython.lib` or
  :mod:`IPython.python`?

* :file:`Release.py`.  Move to ??? or remove?

* :file:`Shell.py`.  Move to :file:`IPython.core.shell.py` or
  :file:`IPython/frontend/shell.py`.

* :file:`UserConfig`.  Move to a subdirectory of :file:`IPython.config`.

* :file:`background_jobs.py`.  Move to :file:`IPython/lib/backgroundjobs.py`.

* :file:`completer.py`.  Move to :file:`IPython/core/completer.py`.

* :file:`config`.  Good where it is!

* :file:`deep_reload.py`.  Move to :file:`IPython/lib/deepreload.py`.

* :file:`demo.py`.  Move to :file:`IPython/lib/demo.py`.

* :file:`dtutils.py`.  Remove or move to :file:`IPython.testing` or
  :file:`IPython.lib`.

* :file:`excolors.py`.  Move to :file:`IPython.core` or :file:`IPython.config`.
  Maybe move to :mod:`IPython.lib` or :mod:`IPython.python`?

* :file:`external`.  Good where it is!

* :file:`frontend`.  Good where it is!

* :file:`generics.py`.  Move to :file:`IPython.python`.

* :file:`genutils.py`.  Move to :file:`IPython.python` and break up into different
  modules.

* :file:`gui`.  Eventually this should be moved to a subdir of   
  :file:`IPython.frontend`.

* :file:`history.py`.  Move to :file:`IPython.core`.

* :file:`hooks.py`.  Move to :file:`IPython.core`.

* :file:`ipapi.py`.  Move to :file:`IPython.core`.

* :file:`iplib.py`.  Move to :file:`IPython.core`.

* :file:`ipmaker.py`:  Move to :file:`IPython.core`.

* :file:`ipstruct.py`.  Move to :file:`IPython.python`.

* :file:`irunner.py`.  Move to :file:`IPython.scripts`.

* :file:`kernel`.  Good where it is.

* :file:`macro.py`.  Move to :file:`IPython.core`.

* :file:`numutils.py`.  Move to :file:`IPython.sandbox`.

* :file:`platutils.py`.  Move to :file:`IPython.python`.

* :file:`platutils_dummy.py`.  Move to :file:`IPython.python`.

* :file:`platutils_posix.py`.  Move to :file:`IPython.python`.

* :file:`platutils_win32.py`.  Move to :file:`IPython.python`.

* :file:`prefilter.py`:  Move to :file:`IPython.core`.

* :file:`rlineimpl.py`.  Move to :file:`IPython.core`.

* :file:`shadowns.py`.  Move to :file:`IPython.core`.

* :file:`shellglobals.py`.  Move to :file:`IPython.core`.

* :file:`strdispatch.py`.  Move to :file:`IPython.python`.

* :file:`testing`.  Good where it is.

* :file:`tests`.  Good where it is.

* :file:`tools`.  Things in here need to be looked at and moved elsewhere like
  :file:`IPython.python`.

* :file:`twshell.py`.  Move to :file:`IPython.sandbox`.

* :file:`ultraTB.py`.  Move to :file:`IPython/core/ultratb.py`.

* :file:`upgrade_dir.py`.  Move to :file:`IPython/python/upgradedir.py`.

* :file:`usage.py`.  Move to :file:`IPython.core`.

* :file:`wildcard.py`.  Move to :file:`IPython.python` or :file:`IPython.core`.

* :file:`winconsole.py`.  Move to :file:`IPython.lib`.

Other things
============

When these files are moved around, a number of other things will happen at the same time:

1. Test files will be created for each module in IPython. Minimally, all
   modules will be imported as a part of the test. This will serve as a 
   test of the module reorganization. These tests will be put into new
   :file:`tests` subdirectories that each package will have.

2. PyFlakes and other code checkers will be run to look for problems.

3. Modules will be renamed to comply with PEP 8 naming conventions:  all
   lowercase and no special characters like ``-`` or ``_``.

4. Existing tests will be moved to the appropriate :file:`tests`
   subdirectories.