examples/Parallel Computing/Parallel Magics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Parallel Magics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IPython has a few magics for working with your engines.\n",
    "\n",
    "This assumes you have started an IPython cluster, either with the notebook interface,\n",
    "or the `ipcluster/controller/engine` commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython import parallel\n",
    "rc = parallel.Client()\n",
    "dv = rc[:]\n",
    "rc.ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`.  \n",
    "These magics are initially associated with a DirectView always associated with all currently registered engines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can execute code remotely with `%px`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px a=5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with dv.sync_imports():\n",
    "    import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px from __future__ import print_function\n",
    "%px print(\"ERROR\", file=sys.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't have to wait for results.  The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxconfig --noblock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px import time\n",
    "%px time.sleep(5)\n",
    "%px time.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you will notice that this didn't output the result of the last command.\n",
    "For this, we have `%pxresult`, which displays the output of the latest request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxresult"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, an IPython engine is IPython, so you can do magics remotely as well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxconfig --block\n",
    "%px %matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`%%px` can also be used as a cell magic, for submitting whole blocks.\n",
    "This one acceps `--block` and `--noblock` flags to specify\n",
    "the blocking behavior, though the default is unchanged.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dv.scatter('id', dv.targets, flatten=True)\n",
    "dv['stride'] = len(dv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px --noblock\n",
    "x = np.linspace(0,np.pi,1000)\n",
    "for n in range(id,12, stride):\n",
    "    print(n)\n",
    "    plt.plot(x,np.sin(n*x))\n",
    "plt.title(\"Plot %i\" % id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxresult"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:\n",
    "\n",
    "The choices are:\n",
    "\n",
    "* `engine` - all of an engine's output is collected together\n",
    "* `type` - where stdout of each engine is grouped, etc. (the default)\n",
    "* `order` - same as `type`, but individual displaypub outputs are interleaved.\n",
    "  That is, it will output the first plot from each engine, then the second from each,\n",
    "  etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px --group-outputs=engine\n",
    "x = np.linspace(0,np.pi,1000)\n",
    "for n in range(id+1,12, stride):\n",
    "    print(n)\n",
    "    plt.figure()\n",
    "    plt.plot(x,np.sin(n*x))\n",
    "    plt.title(\"Plot %i\" % n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.\n",
    "\n",
    "`%pxresult` takes the same output-ordering arguments as `%%px`, \n",
    "so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxresult --group-outputs=order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single-engine views"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "def generate_output():\n",
    "    \"\"\"function for testing output\n",
    "    \n",
    "    publishes two outputs of each type, and returns something\n",
    "    \"\"\"\n",
    "    \n",
    "    import sys,os\n",
    "    from IPython.display import display, HTML, Math\n",
    "    \n",
    "    print(\"stdout\")\n",
    "    print(\"stderr\", file=sys.stderr)\n",
    "    \n",
    "    display(HTML(\"<b>HTML</b>\"))\n",
    "    \n",
    "    print(\"stdout2\")\n",
    "    print(\"stderr2\", file=sys.stderr)\n",
    "    \n",
    "    display(Math(r\"\\alpha=\\beta\"))\n",
    "    \n",
    "    return os.getpid()\n",
    "\n",
    "dv['generate_output'] = generate_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also have more than one set of parallel magics registered at a time.\n",
    "\n",
    "The `View.activate()` method takes a suffix argument, which is added to `'px'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "e0 = rc[-1]\n",
    "e0.block = True\n",
    "e0.activate('0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px0 generate_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%px generate_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, we can redisplay those same results with various grouping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxresult --group-outputs order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pxresult --group-outputs engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel Exceptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you raise exceptions with the parallel exception,\n",
    "the CompositeError raised locally will display your remote traceback."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "from numpy.random import random\n",
    "A = random((100,100,'invalid shape'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remote Cell Magics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "%%timeit\n",
    "from numpy.random import random\n",
    "from numpy.linalg import norm\n",
    "A = random((100,100))\n",
    "norm(A, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of IPython 1.0, you can instruct `%%px` to also execute the cell locally.\n",
    "This is useful for interactive definitions,\n",
    "or if you want to load a data source everywhere,\n",
    "not just on the engines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%px --local\n",
    "import os\n",
    "thispid = os.getpid()\n",
    "print(thispid)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}