upstream/ipython Files · docs/examples/lib/BackgroundJobs.ipynb

Merge pull request from minrk/parallelmagics...

Merge pull request from minrk/parallelmagics Update parallel magics They now display all output, so you can do parallel plotting or other actions with complex display. The `px` magic has now both line and cell modes, and in cell mode finer control has been added about how to collate output from multiple engines. Tests, docs and example notebook added.

Fernando Perez - - Load All Authors

File last commit:

r4941:eac11eb3


                r7060:a1360828

Download file

             BackgroundJobs.ipynb
        
                    192 lines
            
             | 17.8 KiB
            
                | text/plain
            
             |
                TextLexer

/ docs / examples / lib / BackgroundJobs.ipynb

History | Source | Raw |Copy content |Copy permalink

Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	{
		"worksheets": [
		{
		"cells": [
		{
		"source": "# Simple interactive bacgkround jobs with IPython\n\nWe start by loading the `backgroundjobs` library and defining a few trivial functions to illustrate things with.",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 35,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "from IPython.lib import backgroundjobs as bg\n\nimport sys\nimport time\n\ndef sleepfunc(interval=2, a, kw):\n args = dict(interval=interval,\n args=a,\n kwargs=kw)\n time.sleep(interval)\n return args\n\ndef diefunc(interval=2, a, **kw):\n time.sleep(interval)\n raise Exception(\"Dead job with interval %s\" % interval)\n\ndef printfunc(interval=1, reps=5):\n for n in range(reps):\n time.sleep(interval)\n print 'In the background...', n\n sys.stdout.flush()\n print 'All done!'\n sys.stdout.flush()"
		},
		{
		"source": "Now, we can create a job manager (called simply `jobs`) and use it to submit new jobs.\n<br>\nRun the cell below and wait a few seconds for the whole thing to finish, until you see the \"All done!\" printout.",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Starting job # 0 in a separate thread.\nStarting job # 2 in a separate thread.\nStarting job # 3 in a separate thread.\nStarting job # 4 in a separate thread.\nStarting job # 5 in a separate thread.\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 0\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 1\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 2\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "All done!\n"
		}
		],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 36,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "jobs = bg.BackgroundJobManager()\n\n# Start a few jobs, the first one will have ID # 0\njobs.new(sleepfunc, 4)\njobs.new(sleepfunc, kw={'reps':2})\njobs.new('printfunc(1,3)')\n\n# This makes a couple of jobs which will die. Let's keep a reference to\n# them for easier traceback reporting later\ndiejob1 = jobs.new(diefunc, 1)\ndiejob2 = jobs.new(diefunc, 2)"
		},
		{
		"source": "You can check the status of your jobs at any time:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
Fernando Perez Add basic test suite to background jobs library.	r4941	"text": "Completed jobs:\n0 : <function sleepfunc at 0x30e1578>\n2 : <function sleepfunc at 0x30e1578>\n3 : printfunc(1,3)\n\nDead jobs:\n4 : <function diefunc at 0x304d488>\n5 : <function diefunc at 0x304d488>\n\n"
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	}
		],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 37,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "jobs.status()"
		},
		{
		"source": "For any completed job, you can get its result easily:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
Fernando Perez Add basic test suite to background jobs library.	r4941	"outputs": [],
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 43,
		"input": "jobs[0].result\nj0 = jobs[0]\nj0.join?"
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	},
		{
		"source": "You can get the traceback of any dead job. Run the line\nbelow again interactively until it prints a traceback (check the status\nof the job):\n",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Status of diejob1: Dead (Exception), call jobs.traceback() for details\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n"
		}
		],
		"collapsed": false,
		"prompt_number": 34,
		"input": "print \"Status of diejob1:\", diejob1.status\ndiejob1.traceback() # jobs.traceback(4) would also work here, with the job number"
		},
		{
		"source": "This will print all tracebacks for all dead jobs:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Traceback for: <BackgroundJob #4: <function diefunc at 0x30df758>>\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n\nTraceback for: <BackgroundJob #5: <function diefunc at 0x30df758>>\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 2\n\n"
		}
		],
		"collapsed": false,
		"prompt_number": 33,
		"input": "jobs.traceback()"
		},
		{
		"source": "The job manager can be flushed of all completed jobs at any time:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "No jobs to flush.\n"
		}
		],
		"collapsed": false,
		"prompt_number": 25,
		"input": "jobs.flush()"
		},
		{
		"source": "After that, the status is simply empty:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [],
		"collapsed": true,
		"prompt_number": 27,
		"input": "jobs.status()"
		},
		{
Fernando Perez Add basic test suite to background jobs library.	r4941	"source": "It's easy to wait on a job:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Starting job # 7 in a separate thread.\nWill wait for j now...\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Result from j:\n"
		},
		{
		"output_type": "pyout",
		"prompt_number": 46,
		"text": "{'args': (), 'interval': 2, 'kwargs': {}}"
		}
		],
		"collapsed": false,
		"prompt_number": 46,
		"input": "j = jobs.new(sleepfunc, 2)\nprint \"Will wait for j now...\"\nsys.stdout.flush()\nj.join()\nprint \"Result from j:\"\nj.result"
		},
		{
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "",
		"cell_type": "code",
		"collapsed": true,
		"language": "python",
		"outputs": []
		}
		]
		}
		],
		"metadata": {
		"name": "BackgroundJobs"
		},
		"nbformat": 2
		}

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages