docs/examples/parallel/Parallel Magics.ipynb

{
 "metadata": {
  "name": "Parallel Magics"
 },
 "nbformat": 3,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "source": [
      "Using Parallel Magics"
     ]
    },
    {
     "cell_type": "markdown",
     "source": [
      "IPython has a few magics for working with your engines.",
      "",
      "This assumes you have started an IPython cluster, either with the notebook interface,",
      "or the `ipcluster/controller/engine` commands."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython import parallel",
      "rc = parallel.Client()",
      "dv = rc[:]",
      "dv.block = True",
      "dv"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "The parallel magics come from the `parallelmagics` IPython extension.",
      "The magics are set to work with a particular View object,",
      "so to activate them, you call the `activate()` method on a particular view:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "dv.activate()"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "Now we can execute code remotely with `%px`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a=5"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px print a"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with dv.sync_imports():",
      "    import sys"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px print >> sys.stderr, \"ERROR\""
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "You don't have to wait for results:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "dv.block = False"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px import time",
      "%px time.sleep(5)",
      "%px time.time()"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "But you will notice that this didn't output the result of the last command.",
      "For this, we have `%result`, which displays the output of the latest request:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%result"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "Remember, an IPython engine is IPython, so you can do magics remotely as well!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv.block = True",
      "%px %pylab inline"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "`%%px` can also be used as a cell magic, for submitting whole blocks.",
      "This one acceps `--block` and `--noblock` flags to specify",
      "the blocking behavior, though the default is unchanged.",
      ""
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "dv.scatter('id', dv.targets, flatten=True)",
      "dv['stride'] = len(dv)"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --noblock",
      "x = linspace(0,pi,1000)",
      "for n in range(id,12, stride):",
      "    print n",
      "    plt.plot(x,sin(n*x))",
      "plt.title(\"Plot %i\" % id)"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%result"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:",
      "",
      "The choices are:",
      "",
      "* `engine` - all of an engine's output is collected together",
      "* `type` - where stdout of each engine is grouped, etc. (the default)",
      "* `order` - same as `type`, but individual displaypub outputs are interleaved.",
      "  That is, it will output the first plot from each engine, then the second from each,",
      "  etc."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --group-outputs=engine",
      "x = linspace(0,pi,1000)",
      "for n in range(id,12, stride):",
      "    print n",
      "    plt.figure()",
      "    plt.plot(x,sin(n*x))",
      "plt.title(\"Plot %i\" % id)"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "source": [
      "When you specify 'order', then individual display outputs (e.g. plots) will be interleaved:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --group-outputs=order",
      "x = linspace(0,pi,1000)",
      "for n in range(id,12, stride):",
      "    print n",
      "    plt.figure()",
      "    plt.plot(x,sin(n*x))",
      "plt.title(\"Plot %i\" % id)"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "source": [
      "Single-engine views"
     ]
    },
    {
     "cell_type": "markdown",
     "source": [
      "When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "def generate_output():",
      "    \"\"\"function for testing output",
      "    ",
      "    publishes two outputs of each type, and returns something",
      "    \"\"\"",
      "    ",
      "    import sys,os",
      "    from IPython.core.display import display, HTML, Math",
      "    ",
      "    print \"stdout\"",
      "    print >> sys.stderr, \"stderr\"",
      "    ",
      "    display(HTML(\"<b>HTML</b>\"))",
      "    ",
      "    print \"stdout2\"",
      "    print >> sys.stderr, \"stderr2\"",
      "    ",
      "    display(Math(r\"\\alpha=\\beta\"))",
      "    ",
      "    return os.getpid()",
      "",
      "dv['generate_output'] = generate_output"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "e0 = rc[-1]",
      "e0.block = True",
      "e0.activate()"
     ],
     "language": "python",
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px generate_output()"
     ],
     "language": "python",
     "outputs": []
    }
   ]
  }
 ]
}