examples/Notebook/Converting Notebooks With nbconvert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NbConvert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command line usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`NbConvert` is the library, and the command line tool that allow to convert from notebook to other formats.\n",
    "It is a technological preview in 1.0 but is already usable and highly configurable.\n",
    "It ships already with many default available formats : `html`, `latex`, `markdown`, `python`, `rst` and  `slides`\n",
    "which are fully base on Jinja templating engine, so writing a converter for your custom format or tweeking the existing \n",
    "one should be extra simple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can invoke nbconvert by doing\n",
    "\n",
    "```bash\n",
    "$ ipython nbconvert <options and arguments>\n",
    "```\n",
    "\n",
    "Call `ipython nbconvert` with the `--help` flag or no aruments to get basic help on how to use it.\n",
    "For more information about configuration use the `--help-all` flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be converting `Custom Display Logic.ipynb`. \n",
    "Be sure to have runed some of the cells in it to have output otherwise you will only see input in nbconvert.\n",
    "Nbconvert **do not execute the code** in the notebook files, it only converts what is inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Using existing profile dir: u'/Users/bussonniermatthias/.ipython/profile_default'\n",
      "[NbConvertApp] Converting notebook 04 - Custom Display Logic.ipynb to html\n",
      "[NbConvertApp] Support files will be in 04 - Custom Display Logic_files/\n",
      "[NbConvertApp] Loaded template html_full.tpl\n",
      "[NbConvertApp] Writing 221081 bytes to 04 - Custom Display Logic.html\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ipython nbconvert  '04 - Custom Display Logic.ipynb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Html is the default value (that can be configured) , so the verbose form would be "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Using existing profile dir: u'/Users/bussonniermatthias/.ipython/profile_default'\n",
      "[NbConvertApp] Converting notebook 04 - Custom Display Logic.ipynb to html\n",
      "[NbConvertApp] Support files will be in 04 - Custom Display Logic_files/\n",
      "[NbConvertApp] Loaded template html_full.tpl\n",
      "[NbConvertApp] Writing 221081 bytes to 04 - Custom Display Logic.html\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ipython nbconvert  --to=html '04 - Custom Display Logic.ipynb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also convert to latex, which will take care of extractin the embeded base64 encoded png, or the svg and call inkscape to convert those svg to pdf if necessary :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Using existing profile dir: u'/Users/bussonniermatthias/.ipython/profile_default'\n",
      "[NbConvertApp] Converting notebook 04 - Custom Display Logic.ipynb to latex\n",
      "[NbConvertApp] Support files will be in 04 - Custom Display Logic_files/\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26432): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26472): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26512): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26552): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26592): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "[NbConvertApp] Loaded template latex_article.tplx\n",
      "[NbConvertApp] Writing 41196 bytes to 04 - Custom Display Logic.tex\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ipython nbconvert  --to=latex '04 - Custom Display Logic.ipynb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should just have to compile the generated `.tex` file. If you get the required packages installed, if should compile out of the box.\n",
    "\n",
    "For convenience we allow to run extra action after the conversion has been done, in particular for `latex` we have a `pdf` post-processor. \n",
    "You can define the postprocessor tu run with the `--post` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Using existing profile dir: u'/Users/bussonniermatthias/.ipython/profile_default'\n",
      "[NbConvertApp] Converting notebook 04 - Custom Display Logic.ipynb to latex\n",
      "[NbConvertApp] Support files will be in 04 - Custom Display Logic_files/\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26658): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26698): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26738): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26778): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "Setting Language: .UTF-8\n",
      "\n",
      "(process:26818): Gtk-WARNING **: Locale not supported by C library.\n",
      "\tUsing the fallback 'C' locale.\n",
      "[NbConvertApp] Loaded template latex_article.tplx\n",
      "[NbConvertApp] Writing 41196 bytes to 04 - Custom Display Logic.tex\n",
      "[NbConvertApp] Building PDF: ['pdflatex', '04 - Custom Display Logic.tex']\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ipython nbconvert  --to=latex '04 - Custom Display Logic.ipynb' --post=pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at `04 - Custom Display Logic.pdf`, toward the end, where we compared `display()` vs `display_html()` and returning the object.\n",
    "See how the cell where we use `display_html` was not able to display the circle, whereas the two other ones were able to select one of the oher representation they know how to display."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customizing template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's look at the first 20 lines of the `python` exporter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# 1. Implementing special display methods such as `_repr_html_`.\n",
      "# 2. Registering a display function for a particular type.\n",
      "# \n",
      "# In this Notebook we show how both approaches work.\n",
      "\n",
      "# Before we get started, we will import the various display functions for displaying the different formats we will create.\n",
      "\n",
      "# In[54]:\n",
      "\n",
      "from IPython.display import display\n",
      "from IPython.display import (\n",
      "    display_html, display_jpeg, display_png,\n",
      "    display_javascript, display_svg, display_latex\n",
      ")\n",
      "\n",
      "\n",
      "### Implementing special display methods\n",
      "\n",
      "# The main idea of the first approach is that you have to implement special display methods, one for each representation you want to use. Here is a list of the names of the special methods and the values they must return:\n",
      "# \n"
     ]
    }
   ],
   "source": [
    "pyfile = !ipython nbconvert --to python '04 - Custom Display Logic.ipynb' --stdout\n",
    "for l in pyfile[20:40]:\n",
    "    print l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the non-code cell are exported to the file. To have a cleaner script, we will export only the code contained in the code cells.\n",
    "\n",
    "To do so, we will inherit the python template, and overwrite the markdown blocks to be empty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting simplepython.tpl\n"
     ]
    }
   ],
   "source": [
    "%%writefile simplepython.tpl\n",
    "{% extends 'python.tpl'%}\n",
    "\n",
    "{% block markdowncell -%}\n",
    "{% endblock markdowncell %}\n",
    "\n",
    "## we also want to get rig of header cell\n",
    "{% block headingcell -%}\n",
    "{% endblock headingcell %}\n",
    "\n",
    "## and let's change the appearance of input prompt\n",
    "{% block in_prompt %}\n",
    "# This was input cell with prompt number : {{ cell.prompt_number if cell.prompt_number else ' ' }}\n",
    "{%- endblock in_prompt %}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# This was input cell with prompt number : 54\n",
      "from IPython.display import display\n",
      "from IPython.display import (\n",
      "    display_html, display_jpeg, display_png,\n",
      "    display_javascript, display_svg, display_latex\n",
      ")\n",
      "\n",
      "\n",
      "# This was input cell with prompt number : 55\n",
      "get_ipython().magic(u'load soln/mycircle.py')\n",
      "\n",
      "\n",
      "# This was input cell with prompt number : 56\n",
      "class MyCircle(object):\n",
      "    \n",
      "    def _repr_html_(self):\n",
      "        return \"&#x25CB; (<b>html</b>)\"\n",
      "\n",
      "    def _repr_svg_(self):\n",
      "        return \"\"\"<svg width='100px' height='100px'>\n",
      "           <circle cx=\"50\" cy=\"50\" r=\"20\" stroke=\"black\" stroke-width=\"1\" fill=\"blue\"/>\n",
      "        </svg>\"\"\"\n",
      "    \n",
      "    def _repr_latex_(self):\n",
      "        return r\"$\\bigcirc \\LaTeX$\"\n",
      "\n",
      "    def _repr_javascript_(self):\n",
      "        return \"alert('I am a circle!');\"\n",
      "\n",
      "\n",
      "# This was input cell with prompt number : 57\n",
      "c = MyCircle()\n",
      "\n",
      "\n",
      "# This was input cell with prompt number : 58\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "pyfile = !ipython nbconvert --to python '04 - Custom Display Logic.ipynb' --stdout --template=simplepython.tpl\n",
    "\n",
    "for l in pyfile[4:40]:\n",
    "    print l\n",
    "print '...'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll let you read Jinja manual for the exact syntax of the template."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Template that use cells metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebook fileformat support attaching arbitrary JSON metadata to each cell of a notebook. In this part we will use those metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you need to choose another notebook you want to convert to html, and tag some of the cell with metadata.\n",
    "You can see the file `soln/celldiff.js` for a solution on how to tag, or follow the javascript tutorial to see how to do that. Use what we have written there to tag cells of some notebooks to `Easy`|`Medium`|`Hard`|`<None>`, and convert this notebook using your template."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you might need the following : \n",
    "```\n",
    "{% extends 'html_full.tpl'%}\n",
    "{% block any_cell %}\n",
    "{{ super() }}\n",
    "<div style=\"background-color:red\">\n",
    "<div style='background-color:orange'>\n",
    "```\n",
    "\n",
    "`metadata` might not exist, be sure to :\n",
    "\n",
    "`cell['metadata'].get('example',{}).get('difficulty','')`\n",
    "\n",
    "tip: use `%%writefile` to edit the template in the notebook :-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# ipython nbconvert --to html <your chosen notebook.ipynb> --template=<your template file>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%loadpy soln/coloreddiff.tpl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ipython nbconvert --to html '04 - Custom Display Logic.ipynb' --template=soln/coloreddiff.tpl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get rid of all command line flags."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of all of IPython nbconvert can be configured using profiles and passing the `--profile` flag. \n",
    "Moreover if a `config.py` file exist in current working directory nbconvert will use that, or read the config file you give to it with the `--config=<file>` flag. \n",
    "\n",
    "In the end, if you are often running nbconvert on the sam project, `$ ipython nbconvert` should be enough to get you up and ready."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}