##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

add Importing Notebooks example
add Importing Notebooks example
MinRK - - Load All Authors

File last commit:

r11699:327c104f
r11699:327c104f
Show More
Importing Notebooks.ipynb
795 lines | 24.0 KiB | text/plain | TextLexer
/ examples / notebooks / Importing Notebooks.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Importing IPython Notebooks as Modules

It is a common problem that people want to import code from IPython Notebooks. This is made difficult by the fact that Notebooks are not plain Python files, and thus cannot be imported by the regular Python machinery.

There is a flag in the notebook server that provides a certain workaround for a small set of cases, but I think it's gross so I won't even discuss it.

Fortunately, Python provides some fairly sophisticated hooks into the import machinery, so we can actually make IPython notebooks importable without much difficulty, and only using public APIs.

Forgive me if some of this is gross or wrong, I haven't really written import hooks before.

In [1]:
import io, os, sys, types
In [2]:
from IPython.nbformat import current
from IPython.core.interactiveshell import InteractiveShell

Import hooks typically take the form of two objects:

  1. a Module Loader, which takes a module name (e.g. 'IPython.display'), and returns a Module
  2. a Module Finder, which figures out whether a module might exist, and tells Python what Loader to use
In [3]:
def find_notebook(fullname, path=None):
    """find a notebook, given its fully qualified name and an optional path
    
    This turns "foo.bar" into "foo/bar.ipynb"
    and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
    does not exist.
    """
    name = fullname.rsplit('.', 1)[-1]
    if not path:
        path = ['']
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        if os.path.isfile(nb_path):
            return nb_path
        # let import Notebook_Name find "Notebook Name.ipynb"
        nb_path = nb_path.replace("_", " ")
        if os.path.isfile(nb_path):
            return nb_path

Notebook Loader

Here we have our Notebook Loader. It's actually quite simple - once we figure out the filename of the module, all it does is:

  1. load the notebook document into memory
  2. create an empty Module
  3. execute every cell in the Module namespace

Since IPython cells can have extended syntax, the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them. If all of your notebook cells are pure-Python, this step is unnecessary.

In [4]:
class NotebookLoader(object):
    """Module Loader for IPython Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path
    
    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)
        
        print ("importing IPython notebook from %s" % path)
                                       
        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = current.read(f, 'json')
        
        
        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        sys.modules[fullname] = mod
        
        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__
        
        try:
          for cell in nb.worksheets[0].cells:
            if cell.cell_type == 'code' and cell.language == 'python':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.input)
                # run the code in themodule
                exec code in mod.__dict__
        finally:
            self.shell.user_ns = save_user_ns
        return mod

The Module Finder

The finder is a simple object that tells you whether a name can be imported, and returns the appropriate loader. All this one does is check, when you do:

import mynotebook

it checks whether mynotebook.ipynb exists. If a notebook is found, then it returns a NotebookLoader.

Any extra logic is just for resolving paths within packages.

In [5]:
class NotebookFinder(object):
    """Module finder that locates IPython Notebooks"""
    def __init__(self):
        self.loaders = {}
    
    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return
        
        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)
        
        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

Register the hook

Now we register the NotebookFinder with sys.meta_path

In [6]:
sys.meta_path.append(NotebookFinder())

After this point, my notebooks should be importable.

Let's look at what we have in the CWD:

In [7]:
ls nbimp

__init__.py       __init__.pyc      bs.ipynb          mynotebook.ipynb  nbs/

So I should be able to import nbimp.mynotebook.

Aside: displaying notebooks

Here is some simple code to display the contents of a notebook with syntax highlighting, etc.

In [8]:
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from IPython.display import display, HTML

formatter = HtmlFormatter()
lexer = PythonLexer()

# publish the CSS for pygments highlighting
display(HTML("""
<style type='text/css'>
%s
</style>
""" % formatter.get_style_defs()
))
In [9]:
def show_notebook(fname):
    """display a short summary of the cells of a notebook"""
    with io.open(fname, 'r', encoding='utf-8') as f:
        nb = current.read(f, 'json')
    html = []
    for cell in nb.worksheets[0].cells:
        html.append("<h4>%s cell</h4>" % cell.cell_type)
        if cell.cell_type == 'code':
            html.append(highlight(cell.input, lexer, formatter))
        else:
            html.append("<pre>%s</pre>" % cell.source)
    display(HTML('\n'.join(html)))

show_notebook(os.path.join("nbimp", "mynotebook.ipynb"))

heading cell

My Notebook

code cell

def foo():
    return "foo"

code cell

def has_ip_syntax():
    listing = !ls
    return listing

code cell

def whatsmyname():
    return __name__

So my notebook has a heading cell and some code cells, one of which contains some IPython syntax.

Let's see what happens when we import it

In [10]:
from nbimp import mynotebook

importing IPython notebook from nbimp/mynotebook.ipynb

Hooray, it imported! Does it work?

In [11]:
mynotebook.foo()
Out[11]:
'foo'

Hooray again!

Even the function that contains IPython syntax works:

In [12]:
mynotebook.has_ip_syntax()
Out[12]:
['Animations Using clear_output.ipynb',
 'Cell Magics.ipynb',
 'Custom Display Logic.ipynb',
 'Cython Magics.ipynb',
 'Data Publication API.ipynb',
 'Frontend-Kernel Model.ipynb',
 'Importing Notebooks.ipynb',
 'Octave Magic.ipynb',
 'Part 1 - Running Code.ipynb',
 'Part 2 - Basic Output.ipynb',
 'Part 3 - Plotting with Matplotlib.ipynb',
 'Part 4 - Markdown Cells.ipynb',
 'Part 5 - Rich Display System.ipynb',
 'Progress Bars.ipynb',
 'R Magics.ipynb',
 'README.md',
 'Script Magics.ipynb',
 'SymPy Examples.ipynb',
 'Trapezoid Rule.ipynb',
 'Typesetting Math Using MathJax.ipynb',
 'animation.m4v',
 'foo.pyx',
 'lnum.py',
 'logo',
 'nbimp',
 'python-logo.svg',
 'test.html']

Notebooks in packages

We also have a notebook inside the nb package, so let's make sure that works as well.

In [13]:
ls nbimp/nbs

__init__.py   __init__.pyc  other.ipynb

Note that the __init__.py is necessary for nb to be considered a package, just like usual.

In [14]:
show_notebook(os.path.join("nbimp", "nbs", "other.ipynb"))

markdown cell

This notebook just defines `bar`

code cell

def bar(x):
    return "bar" * x
In [15]:
from nbimp.nbs import other
other.bar(5)

importing IPython notebook from nbimp/nbs/other.ipynb
Out[15]:
'barbarbarbarbar'

So now we have importable notebooks, from both the local directory and inside packages.

I can even put a notebook inside IPython, to further demonstrate that this is working properly:

In [16]:
import shutil
from IPython.utils.path import get_ipython_package_dir

utils = os.path.join(get_ipython_package_dir(), 'utils')
shutil.copy(os.path.join("nbimp", "mynotebook.ipynb"),
            os.path.join(utils, "inside_ipython.ipynb")
)

and import the notebook from IPython.utils

In [17]:
from IPython.utils import inside_ipython
inside_ipython.whatsmyname()

importing IPython notebook from /Users/minrk/dev/ip/mine/IPython/utils/inside_ipython.ipynb
Out[17]:
'IPython.utils.inside_ipython'

Even Cython magics

With a bit of extra magic for handling the IPython interactive namespace during load, even magics like %%cython can be used:

In [18]:
import Cython_Magics

importing IPython notebook from Cython Magics.ipynb
1000000 loops, best of 3: 439 ns per loop
sin(1)= 0.841470984808
In [19]:
Cython_Magics.black_scholes(100.0, 100.0, 1.0, 0.3, 0.03, 0.0, -1)
Out[19]:
10.327861752731728