doc/ipnb_google_soc.lyx

#LyX 1.3 created this file. For more info see http://www.lyx.org/
\lyxformat 221
\textclass article
\begin_preamble
\usepackage{hyperref}

\usepackage{color}
\definecolor{orange}{cmyk}{0,0.4,0.8,0.2}
\definecolor{brown}{cmyk}{0,0.75,0.75,0.35}

% Use and configure listings package for nicely formatted code
\usepackage{listings}
\lstset{
  language=Python,
  basicstyle=\small\ttfamily,
  commentstyle=\ttfamily\color{blue},
  stringstyle=\ttfamily\color{brown},
  showstringspaces=false,
  breaklines=true,
  postbreak = \space\dots
}
\end_preamble
\language english
\inputencoding auto
\fontscheme palatino
\graphics default
\paperfontsize 11
\spacing single 
\papersize Default
\paperpackage a4
\use_geometry 1
\use_amsmath 0
\use_natbib 0
\use_numerical_citations 0
\paperorientation portrait
\leftmargin 1in
\topmargin 0.9in
\rightmargin 1in
\bottommargin 0.9in
\secnumdepth 3
\tocdepth 3
\paragraph_separation skip
\defskip medskip
\quotes_language english
\quotes_times 2
\papercolumns 1
\papersides 1
\paperpagestyle default

\layout Title

Interactive Notebooks for Python
\newline 

\size small 
An IPython project for Google's Summer of Code 2005
\layout Author

Fernando P
\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
'{e}
\end_inset 

rez
\begin_inset Foot
collapsed true

\layout Standard


\family typewriter 
\size small 
[e-mail]
\end_inset 


\layout Abstract

This project aims to develop a file format and interactive support for documents
 which can combine Python code with rich text and embedded graphics.
 The initial requirements only aim at being able to edit such documents
 with a normal programming editor, with final rendering to PDF or HTML being
 done by calling an external program.
 The editing component would have to be integrated with IPython.
 
\layout Abstract

This document was written by the IPython developer; it is made available
 to students looking for projects of interest and for inclusion in their
 application.
\layout Section

Project overview
\layout Standard

Python's interactive interpreter is one of the language's most appealing
 features for certain types of usage, yet the basic shell which ships with
 the language is very limited.
 Over the last few years, IPython
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://ipython.scipy.org}

\end_inset 


\end_inset 

 has become the de facto standard interactive shell in the scientific computing
 community, and it enjoys wide popularity with general audiences.
 All the major Linux distributions (Fedora Core via Extras, SUSE, Debian)
 and OS X (via fink) carry IPython, and Windows users report using it as
 a viable system shell.
\layout Standard

However, IPython is currently a command-line only application, based on
 the readline library and hence with single-line editing capabilities.
 While this kind of usage is sufficient for many contexts, there are usage
 cases where integration in a graphical user interface (GUI) is desirable.
 
\layout Standard

In particular, we wish to have an interface where users can execute Python
 code, input regular text (neither code nor comments) and keep inline graphics,
 which we will call 
\emph on 
Python notebooks
\emph default 
.
 This kind of system is very popular in scientific computing; well known
 implementations can be found in Mathematica
\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
texttrademark
\end_inset 

\SpecialChar ~

\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://www.wolfram.com/products/mathematica}

\end_inset 


\end_inset 

 and Maple
\begin_inset ERT
status Collapsed

\layout Standard

\backslash 
texttrademark
\end_inset 

\SpecialChar ~

\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://www.maplesoft.com}

\end_inset 


\end_inset 

, among others.
 However, these are proprietary (and quite expensive) systems aimed at an
 audience of mathematicians, scientists and engineers.
\layout Standard

The full-blown implementation of a graphical shell supporting this kind
 of work model is probably too ambitious for a summer project.
 Simultaneous support for rich-text editing, embedded graphics and syntax-highli
ghted code is extremely complex, and likely to require far more effort than
 can be mustered by an individual developer for a short-term project.
\layout Standard

This project will thus aim to build the necessary base infrastructure to
 be able to edit such documents from a plain text editor, and to render
 them to suitable formats for printing or online distribution, such as HTML,
 PDF or PostScript.
 This model follows that for the production of LaTeX documents, which can
 be edited with any text editor.
\layout Standard

Such documents would be extremely useful for many audiences beyond scientists:
 one can use them to produce additionally documented code, to explore a
 problem with Python and maintain all relevant information in the same place,
 as a way to distribute enhanced Python-based educational materials, etc.
\layout Standard

Demand for such a system exists, as evidenced by repeated requests made
 to me by IPython users over the last few years.
 Unfortunately IPython is only a spare-time project for me, and I have not
 had the time to devote to this, despite being convinced of its long term
 value and wide appeal.
\layout Standard

If this project is successful, the infrastructure laid out by it will be
 immediately useful for Python users wishing to maintain `literate' programs
 which include rich formatting.
 In addition, this will open the door for the future development of graphical
 shells which can render such documents in real time: this is exactly the
 development model successfully followed by the LyX
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://www.lyx.org}

\end_inset 


\end_inset 

 document processing system.
\layout Section

Implementation effort
\layout Subsection

Specific goals
\layout Standard

This is a brief outline of the main points of this project.
 The next section provides details on all of them.
 The student(s) working on the project would need to:
\layout Enumerate

Make design decisions for the internal file structure to enable valid Python
 notebooks.
 
\layout Enumerate

Implement the rendering library, capable of processing an input notebook
 through reST or LaTeX and producing HTML or PDF output, as well as exporting
 a `pure-code' Python file stripped of all markup calls.
\layout Enumerate

Study existing programming editor widgets to find the most suitable one
 for extending with an IPython connector for interactive execution of the
 notebooks.
\layout Subsection

Complexity level
\layout Standard

This project is relatively complicated.
 While I will gladly assist the student with design and implementation issues,
 it will require a fair amount of thinking in terms of overall library architect
ure.
 The actual implementation does not require any sophisticated concepts,
 but rather a reasonably broad knowledge of a wide set of topics (markup,
 interaction with external programs and libraries, namespace tricks to provide
 runtime changes in the effect of the markup calls, etc.)
\layout Standard

While raw novices are welcome to try, I suspect that it may be a bit too
 much for them.
 Students wanting to apply should keep in mind, if the money is an important
 consideration, that Google only gives the $4500 reward upon 
\emph on 
successful completion
\emph default 
 of the project.
 So don't bite more than you can chew.
 Obviously if this doesn't matter, anyone is welcome to participate, since
 the project can be a very interesting learning experience, and it will
 provide a genuinely useful tool for many.
\layout Section

Technical details
\layout Subsection

The files
\layout Standard

A basic requirement of this project will be that the Python notebooks shall
 be valid Python source files, typically with a 
\family typewriter 
.py
\family default 
 extension.
 A renderer program can be used to process the markup calls in them and
 generate output.
 If run at a regular command line, these files should execute like normal
 Python files.
 But when run via a special rendering script, the result should be a properly
 formatted file.
 Output formats could be PDF or HTML depending on user-supplied options.
\layout Standard

A reST markup mode should be implemented, as reST is already widely used
 in the Python community and is a very simple format to write.
 The following is a sketch of what such files could look like using reST
 markup:
\layout Standard


\begin_inset ERT
status Open

\layout Standard

\backslash 
lstinputlisting{nbexample.py}
\end_inset 


\layout Standard

Additionally, a LaTeX markup mode should also be implemented.
 Here's a mockup example of what code using the LaTeX mode could look like.
\layout Standard


\begin_inset ERT
status Open

\layout Standard

\backslash 
lstinputlisting{nbexample_latex.py}
\end_inset 


\layout Standard

At this point, it must be noted that the code above is simply a sketch of
 these ideas, not a finalized design.
 An important part of this project will be to think about what the best
 API and structure for this problem should be.
\layout Subsection

From notebooks to PDF, HTML or Python
\layout Standard

Once a clean API for markup has been specified, converters will be written
 to take a python source file which uses notebook constructs, and generate
 final output in printable formats, such as HTML or PDF.
 For example, if 
\family typewriter 
nbfile.py
\family default 
 is a python notebook, then
\layout LyX-Code

$ pynb --export=pdf nbfile.py
\layout Standard

should produce 
\family typewriter 
nbfile.pdf
\family default 
, while
\layout LyX-Code

$ pynb --export=html nbfile.py
\layout Standard

would produce an HTML version.
 The actual rendering will be done by calling appropriate utilities, such
 as the reST toolchain or LaTeX, depending on the markup used by the file.
\layout Standard

Additionally, while the notebooks will be valid Python files, if executed
 on their own, all the markup calls will still return their results, which
 are not really needed when the file is being treated as pure code.
 For this reason, a module to execute these files turning the markup calls
 into no-ops should be written.
 Using Python 2.4's -m switch, one can then use something like
\layout LyX-Code

$ python -m notebook nbfile.py
\layout Standard

and the notebook file 
\family typewriter 
nbfile.py
\family default 
 will be executed without any overhead introduced by the markup (other than
 making calls to functions which return immediately).
 Finally, an exporter to clean code can be trivially implemented, so that:
\layout LyX-Code

$ pynb --export=python nbfile.py nbcode.py
\layout Standard

would export only the code in 
\family typewriter 
nbfile.py
\family default 
 to 
\family typewriter 
nbcode.py
\family default 
, removing the markup completely.
 This can be used to generate final production versions of large modules
 implemented as notebooks, if one wants to eliminate the markup overhead.
\layout Subsection

The editing environment
\layout Standard

The first and most important part of the project should be the definition
 of a clean API and the implementation of the exporter modules as indicated
 above.
 Ultimately, such files can be developed using any text editor, since they
 are nothing more than regular Python code.
\layout Standard

But once these goals are reached, further integration with an editor will
 be done, without the need for a full-blown GUI shell.
 In fact, already today the (X)Emacs editors can provide for interactive
 usage of such files.
 Using python-mode in (X)Emacs, one can pass highlighted regions of a file
 for execution to an underlying python process, and the results are printed
 in the python window.
 With recent versions of python-mode, IPython can be used instead of the
 plain python interpreter, so that IPython's extended interactive capabilities
 become available within (X)Emacs (improved tracebacks, automatic debugger
 integration, variable information, easy filesystem access to Python, etc).
\layout Standard

But even with IPython integration, the usage within (X)Emacs is not ideal
 for a notebook environment, since the python process buffer is separate
 from the python file.
 Therefore, the next stage of the project will be to enable tighter integration
 between the editing and execution environments.
 The basic idea is to provide an easy way to mark regions of the file to
 be executed interactively, and to have the output inserted automatically
 into the file.
 The following listing is a mockup of what the resulting file could look
 like
\layout Standard


\begin_inset ERT
status Open

\layout Standard

\backslash 
lstinputlisting{nbexample_output.py}
\end_inset 


\layout Standard

Basically, the editor will execute 
\family typewriter 
add(2,3)
\family default 
 and insert the string representation of the output into the file, so it
 can be used for rendering later.
 
\layout Section

Available resources
\layout Standard

IPython currently has all the necessary infrastructure for code execution,
 albeit in a rather messy code base.
 Most I/O is already abstracted out, a necessary condition for embedding
 in a GUI (since you are not writing to stdout/err but to the GUI's text
 area).
 
\layout Standard

For interaction with an editor, it will be necessary to identify a good
 programming editor with a Python-compatible license, which can be extended
 to communicate with the underlying IPython engine.
 IDLE, the Tk-based IDE which ships with Python, should obviously be considered.
 The Scintilla editing component
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://www.scintilla.org}

\end_inset 


\end_inset 

 may also be a viable candidate.
 
\layout Standard

It will also be interesting to look at the LyX editor
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://www.lyx.org}

\end_inset 


\end_inset 

, which already offers a Python client
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://wiki.lyx.org/Tools/PyClient}

\end_inset 


\end_inset 

.
 Since LyX has very sophisticated LaTeX support, this is a very interesting
 direction to consider for the future (though LyX makes a poor programming
 editor).
\layout Section

Support offered to the students
\layout Standard

The IPython project already has an established Open Source infrastructure,
 including CVS repositories, a bug tracker and mailing lists.
 As the main author and sole maintainer of IPython, I will personally assist
 the student(s) funded with architectural and design guidance, preferably
 on the public development mailing list.
 I expect them to start working by submitting patches until they show, by
 the quality of their work, that they can be granted CVS write access.
 I expect most actual implementation work to be done by the students, though
 I will provide assistance if they need it with a specific technical issue.
\layout Standard

If more than one applicant is accepted to work on this project, there is
 more than enough work to be done which can be coordinated between them.
\layout Section

Licensing and copyright
\layout Standard

IPython is licensed under BSD terms, and copyright of all sources rests
 with the original authors of the core modules.
 Over the years, all external contributions have been small enough patches
 that they have been simply folded into the main source tree without additional
 copyright attributions, though explicit credit has always been given to
 all contributors.
\layout Standard

I expect the students participating in this project to contribute enough
 standalone code that they can retain the copyright to it if they so desire,
 as long as they accept all their work to be licensed under BSD terms.
\layout Section

Acknowledgements
\layout Standard

I'd like to thank John D.
 Hunter, the author of matplotlib
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://matplotlib.sf.net}

\end_inset 


\end_inset 

, for lengthy discussions which helped clarify much of this project.
 In particular, the important decision of embedding the notebook markup
 calls in true Python functions instead of specially-tagged strings or comments
 was an idea I thank him for pushing hard enough to convince me of using.
 
\layout Standard

My conversations with Brian Granger, the author of PyXG
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://hammonds.scu.edu/~classes/pyxg.html}

\end_inset 


\end_inset 

 and braid
\begin_inset Foot
collapsed true

\layout Standard


\begin_inset LatexCommand \htmlurl{http://hammonds.scu.edu/~classes/braid.html}

\end_inset 


\end_inset 

, have also been very useful in clarifying details of the necessary underlying
 infrastructure and future evolution of IPython for this kind of system.
\layout Standard

Thank you also to the IPython users who have, in the past, discussed this
 topic with me either in private or on the IPython or Scipy lists.
\the_end