.. This file has a lot of bash but little python, set default role .. highlight:: bash .. _development: ============================== IPython development overview ============================== Currently, IPython's development tree contains all of the code that used to be part of IPython since the project's inception in 2001, plus all of the effort that was known for a while (roughly 2004-2008) as ``IPython1``. The IPyhton1 development was meant to be an effort to: 1. Clean up the existing codebase and write lots of tests. 2. Separate the core functionality of IPython from the terminal to enable IPython to be used from within a variety of GUI applications. 3. Implement a system for interactive parallel computing. While the third goal may seem a bit unrelated to the main focus of IPython, it turns out that the technologies required for this goal are nearly identical with those required for goal two. This is the main reason the interactive parallel computing capabilities are being put into IPython proper. In the summer of 2008, the IPython1 code was merged back into the mainline, and now there is a unified codebase. While the above goals aren't completely implemented for the older code, we do have a proper :ref:`testing system ` in place (though this is still evolving), unified documentation and partial implementations of the more separated components. This document describes IPython from the perspective of developers. Project organization ==================== Subpackages ----------- IPython is organized into semi self-contained subpackages. Each of the subpackages will have its own: - **Dependencies**. One of the most important things to keep in mind in partitioning code amongst subpackages, is that they should be used to cleanly encapsulate dependencies. - **Tests**. Each subpackage shoud have its own ``tests`` subdirectory that contains all of the tests for that package. For information about writing tests for IPython, see the :ref:`Testing System ` section of this document. - **Configuration**. Each subpackage should have its own ``config`` subdirectory that contains the configuration information for the components of the subpackage. For information about how the IPython configuration system works, see the :ref:`Configuration System ` section of this document. - **Scripts**. Each subpackage should have its own ``scripts`` subdirectory that contains all of the command line scripts associated with the subpackage. Installation and dependencies ----------------------------- IPython will not use `setuptools`_ for installation. Instead, we will use standard ``setup.py`` scripts that use `distutils`_. While there are a number a extremely nice features that `setuptools`_ has (like namespace packages), the current implementation of `setuptools`_ has performance problems, particularly on shared file systems. In particular, when Python packages are installed on NSF file systems, import times become much too long (up towards 10 seconds). Because IPython is being used extensively in the context of high performance computing, where performance is critical but shared file systems are common, we feel these performance hits are not acceptable. Thus, until the performance problems associated with `setuptools`_ are addressed, we will stick with plain `distutils`_. We are hopeful that these problems will be addressed and that we will eventually begin using `setuptools`_. Because of this, we are trying to organize IPython in a way that will make the eventual transition to `setuptools`_ as painless as possible. Because we will be using `distutils`_, there will be no method for automatically installing dependencies. Instead, we are following the approach of `Matplotlib`_ which can be summarized as follows: - Distinguish between required and optional dependencies. However, the required dependencies for IPython should be only the Python standard library. - Upon installation check to see which optional dependencies are present and tell the user which parts of IPython need which optional dependencies. It is absolutely critical that each subpackage of IPython has a clearly specified set of dependencies and that dependencies are not carelessly inherited from other IPython subpackages. Furthermore, tests that have certain dependencies should not fail if those dependencies are not present. Instead they should be skipped and print a message. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools .. _distutils: http://docs.python.org/lib/module-distutils.html .. _Matplotlib: http://matplotlib.sourceforge.net/ Specific subpackages -------------------- ``core`` This is the core functionality of IPython that is independent of the terminal, network and GUIs. Most of the code that is in the current IPython trunk will be refactored, cleaned up and moved here. ``kernel`` The enables the IPython core to be expose to a the network. This is also where all of the parallel computing capabilities are to be found. ``config`` The configuration package used by IPython. ``frontends`` The various frontends for IPython. A frontend is the end-user application that exposes the capabilities of IPython to the user. The most basic frontend will simply be a terminal based application that looks just like today 's IPython. Other frontends will likely be more powerful and based on GUI toolkits. ``tools`` This is where general utilities go. Version control workflow ======================== All IPython development today is done using the `Bazaar`__ system for distributed version control and the `Launchpad`__ site for code hosting and bug tracking. This makes it very easy for anyone to contribute code to IPython. .. __: http://bazaar-vcs.org .. __: http://launchpad.net/ipython If you are interested in contributing to IPython, you should familiarize a bit with how to use bazaar, but this document gives you a concise set of steps to get started. If you are going to become heavily involved with creating code for IPython you'll eventually want to have a Launchpad account (free) so you can publish your own code on the site for review and merging, but small contributions can be simply sent via email to the IPython list as patches. Start by creating what Bazaar calls a `shared repository`_:: # You can choose any name you want instead of "ipython-repo" bzr init-repo ipython-repo .. _shared repository: http://bazaar-vcs.org/SharedRepositoryTutorial This creates an empty repository where a lot of related branches can be kept, and they all reuse common storage. This way, many operations are very fast and take up less space. Now, go to the newly created repository and make a local branch of the public trunk:: cd ipython-repo # This makes a local copy of the public trunk called "trunk-lp" bzr branch lp:ipython trunk-lp The ``trunk-lp`` branch is meant to always be a pristine copy of the public trunk. From here, make a personal development copy of the public trunk, where you'll make your changes:: bzr branch trunk-lp trunk-dev Now, your working area looks like this:: maqroll[ipython-repo]> ls -a ./ ../ .bzr/ trunk-dev/ trunk-lp/ The ``.bzr`` directory is the Bazaar storage area, and you now have both the reference copy of the public trunk and one working copy for your own development. You can later make further branches as needed (a common workflow is to make branches to contain specific feature implementations until they are ready to be merged). The typical work cycle will be to make changes in ``trunk-dev`` and then commit those changes as often as needed:: cd trunk-dev # ... implement cool new feature... bzr commit -m "Commit message goes here." Please note that since we now don't use an old-style linear ChangeLog (that tends to cause problems with distributed version control systems), you should ensure that your log messages are reasonably detailed. For non-trivial changes, use a docstring-like approach in the commit messages (including the second line being left *blank*). Type ``bzr commit`` *without* the ``-m`` switch, and Bazaar will open an editor where you can type a more detailed message:: Single line summary of changes being committed. - more details when warranted ... - including crediting outside contributors if they sent the code/bug/idea! If we couple this with a policy of making single commits for each reasonably atomic change, the bzr log should give an excellent view of the project, and the ``--short`` log option becomes a nice summary. As you work on the branch, it's a good idea to frequently keep your copy of the trunk updated with respect to Launchpad. This is done via:: cd trunk-lp bzr pull Bazaar can then merge any changes that were done to the public trunk into your local branch via:: cd trunk-dev bzr merge ../trunk-lp bzr commit -m"Merged upstream changes" This workflow ensures that a local copy stays in sync with the public trunk, while allowing for local development to be pushed back for public review. Once your changes are ready for review, you can push them to Launchpad where the IPython team can review them and give you feedback. The first time, use:: bzr push --remember \ bzr+ssh://@bazaar.launchpad.net/~/ipython/trunk-dev where ```` is your Launchpad user name. This will make this branch publicly visible to all. In subsequent uses you can simply run in the branch:: bzr push .. note:: Before you can push your code to Launchpad, you need to configure your SSH keys. You can do this by clicking on ``Change details`` for your profile and then clicking on the ``SSH Keys`` tab. This should take you to a URL of the form ``https://launchpad.net/~/+editsshkeys`` with instructions on how to upload your keys. Once the branch is publicly visible, using the launchpad interface it can be marked for merging with the link marked *"Propose for merging into another branch"*, leaving the default choice of trunk as its target. This way, Launchpad tracks what changes of this branch are new with respect to the trunk, others in the team can review it, and merge the changes into the trunk. The IPython team has a policy of reviewing all code (both from core members of the team and from external contributors) before merging it. Code should be clearly documented (as explained further in this guide), clear and well tested. We will be happy to provide you with feedback for your code to be a great contribution to the project, and we've found that peer review of code is an excellent way to improve the quality of everyone's work. Installation and testing scenarios ================================== This section outlines the various scenarios that we need to test before we release an IPython version. These scenarios represent different ways of installing IPython and its dependencies. Installation scenarios under Linux and OS X ------------------------------------------- 1. Install from tarball using ``python setup.py install``. * With only readline+nose dependencies installed. * With all dependencies installed (readline, zope.interface, Twisted, foolscap, Sphinx, nose, pyOpenSSL). 2. Install using easy_install. * With only readline+nose dependencies installed. * Default dependencies: ``easy_install ipython-0.9.beta3-py2.5.egg`` * Optional dependency sets: ``easy_install -f ipython-0.9.beta3-py2.5.egg IPython[kernel,doc,test,security]`` * With all dependencies already installed. Installation scenarios under Win32 ---------------------------------- #. Install everything from .exe installers #. easy_install? Tests to run for these scenarios -------------------------------- #. Run the full test suite. #. Start a controller and engines and try a few things by hand. * Using ipcluster. * Using ipcontroller/ipengine by hand. #. Run a few of the parallel examples. #. Try the kernel with and without security with and without PyOpenSSL installed. #. Beat on the IPython terminal a bunch. #. Make sure that furl files are being put in proper locations. Release checklist ================= Most of the release process is automated by the :file:`release` script in the :file:`tools` directory. This is just a handy reminder for the release manager. #. Run the release script, which makes the tar.gz, eggs and Win32 .exe installer. It posts them to the site and registers the release with PyPI. #. Updating the website with announcements and links to the updated changes.txt in html form. Remember to put a short note both on the news page of the site and on launcphad. #. Drafting a short release announcement with i) highlights and ii) a link to the html changes.txt. #. Make sure that the released version of the docs is live on the site. #. Celebrate!