examples/Parallel Computing/Parallel Magics.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Using Parallel Magics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "IPython has a few magics for working with your engines.\n",
      "\n",
      "This assumes you have started an IPython cluster, either with the notebook interface,\n",
      "or the `ipcluster/controller/engine` commands."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython import parallel\n",
      "rc = parallel.Client()\n",
      "dv = rc[:]\n",
      "rc.ids"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`.  \n",
      "These magics are initially associated with a DirectView always associated with all currently registered engines."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can execute code remotely with `%px`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a=5"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px print(a)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px a"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with dv.sync_imports():\n",
      "    import sys"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px from __future__ import print_function\n",
      "%px print(\"ERROR\", file=sys.stderr)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You don't have to wait for results.  The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxconfig --noblock"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px import time\n",
      "%px time.sleep(5)\n",
      "%px time.time()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "But you will notice that this didn't output the result of the last command.\n",
      "For this, we have `%pxresult`, which displays the output of the latest request:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember, an IPython engine is IPython, so you can do magics remotely as well!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxconfig --block\n",
      "%px %matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "import numpy as np\n",
      "import matplotlib.pyplot as plt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`%%px` can also be used as a cell magic, for submitting whole blocks.\n",
      "This one acceps `--block` and `--noblock` flags to specify\n",
      "the blocking behavior, though the default is unchanged.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv.scatter('id', dv.targets, flatten=True)\n",
      "dv['stride'] = len(dv)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --noblock\n",
      "x = np.linspace(0,np.pi,1000)\n",
      "for n in range(id,12, stride):\n",
      "    print(n)\n",
      "    plt.plot(x,np.sin(n*x))\n",
      "plt.title(\"Plot %i\" % id)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:\n",
      "\n",
      "The choices are:\n",
      "\n",
      "* `engine` - all of an engine's output is collected together\n",
      "* `type` - where stdout of each engine is grouped, etc. (the default)\n",
      "* `order` - same as `type`, but individual displaypub outputs are interleaved.\n",
      "  That is, it will output the first plot from each engine, then the second from each,\n",
      "  etc."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --group-outputs=engine\n",
      "x = np.linspace(0,np.pi,1000)\n",
      "for n in range(id+1,12, stride):\n",
      "    print(n)\n",
      "    plt.figure()\n",
      "    plt.plot(x,np.sin(n*x))\n",
      "    plt.title(\"Plot %i\" % n)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.\n",
      "\n",
      "`%pxresult` takes the same output-ordering arguments as `%%px`, \n",
      "so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult --group-outputs=order"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Single-engine views"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from __future__ import print_function\n",
      "\n",
      "def generate_output():\n",
      "    \"\"\"function for testing output\n",
      "    \n",
      "    publishes two outputs of each type, and returns something\n",
      "    \"\"\"\n",
      "    \n",
      "    import sys,os\n",
      "    from IPython.display import display, HTML, Math\n",
      "    \n",
      "    print(\"stdout\")\n",
      "    print(\"stderr\", file=sys.stderr)\n",
      "    \n",
      "    display(HTML(\"<b>HTML</b>\"))\n",
      "    \n",
      "    print(\"stdout2\")\n",
      "    print(\"stderr2\", file=sys.stderr)\n",
      "    \n",
      "    display(Math(r\"\\alpha=\\beta\"))\n",
      "    \n",
      "    return os.getpid()\n",
      "\n",
      "dv['generate_output'] = generate_output"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can also have more than one set of parallel magics registered at a time.\n",
      "\n",
      "The `View.activate()` method takes a suffix argument, which is added to `'px'`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "e0 = rc[-1]\n",
      "e0.block = True\n",
      "e0.activate('0')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px0 generate_output()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%px generate_output()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As mentioned above, we can redisplay those same results with various grouping:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult --group-outputs order"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pxresult --group-outputs engine"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Parallel Exceptions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you raise exceptions with the parallel exception,\n",
      "the CompositeError raised locally will display your remote traceback."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "from numpy.random import random\n",
      "A = random((100,100,'invalid shape'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Remote Cell Magics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px\n",
      "%%timeit\n",
      "from numpy.random import random\n",
      "from numpy.linalg import norm\n",
      "A = random((100,100))\n",
      "norm(A, 2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Local Execution"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As of IPython 1.0, you can instruct `%%px` to also execute the cell locally.\n",
      "This is useful for interactive definitions,\n",
      "or if you want to load a data source everywhere,\n",
      "not just on the engines."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%px --local\n",
      "import os\n",
      "thispid = os.getpid()\n",
      "print(thispid)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}