upstream/ipython Files · docs/examples/lib/BackgroundJobs.ipynb

Update deepreload to use a rewritten knee.py. Fixes dreload(numpy)....

Update deepreload to use a rewritten knee.py. Fixes dreload(numpy). knee.py, a Python re-implementation of hierarchical module import was removed from the standard library because it no longer functioned properly. deepreload.py is little more than a hacked version of knee.py which overrides __builtin__.__import__ to ensure that each module is re-imported once (before just referring to sys.modules as usual). In addition, `os.path` was added to the default excluded modules, since somehow it has an entry in sys.modules without `os' being a package.

Fernando Perez - - Load All Authors

File last commit:

r4941:eac11eb3


                r6531:c0ad0135

Download file

             BackgroundJobs.ipynb
        
                    192 lines
            
             | 17.8 KiB
            
                | text/plain
            
             |
                TextLexer

/ docs / examples / lib / BackgroundJobs.ipynb

History | Source | Raw |Copy content |Copy permalink

Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	{
		"worksheets": [
		{
		"cells": [
		{
		"source": "# Simple interactive bacgkround jobs with IPython\n\nWe start by loading the `backgroundjobs` library and defining a few trivial functions to illustrate things with.",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 35,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "from IPython.lib import backgroundjobs as bg\n\nimport sys\nimport time\n\ndef sleepfunc(interval=2, a, kw):\n args = dict(interval=interval,\n args=a,\n kwargs=kw)\n time.sleep(interval)\n return args\n\ndef diefunc(interval=2, a, **kw):\n time.sleep(interval)\n raise Exception(\"Dead job with interval %s\" % interval)\n\ndef printfunc(interval=1, reps=5):\n for n in range(reps):\n time.sleep(interval)\n print 'In the background...', n\n sys.stdout.flush()\n print 'All done!'\n sys.stdout.flush()"
		},
		{
		"source": "Now, we can create a job manager (called simply `jobs`) and use it to submit new jobs.\n<br>\nRun the cell below and wait a few seconds for the whole thing to finish, until you see the \"All done!\" printout.",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Starting job # 0 in a separate thread.\nStarting job # 2 in a separate thread.\nStarting job # 3 in a separate thread.\nStarting job # 4 in a separate thread.\nStarting job # 5 in a separate thread.\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 0\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 1\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "In the background... 2\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "All done!\n"
		}
		],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 36,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "jobs = bg.BackgroundJobManager()\n\n# Start a few jobs, the first one will have ID # 0\njobs.new(sleepfunc, 4)\njobs.new(sleepfunc, kw={'reps':2})\njobs.new('printfunc(1,3)')\n\n# This makes a couple of jobs which will die. Let's keep a reference to\n# them for easier traceback reporting later\ndiejob1 = jobs.new(diefunc, 1)\ndiejob2 = jobs.new(diefunc, 2)"
		},
		{
		"source": "You can check the status of your jobs at any time:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
Fernando Perez Add basic test suite to background jobs library.	r4941	"text": "Completed jobs:\n0 : <function sleepfunc at 0x30e1578>\n2 : <function sleepfunc at 0x30e1578>\n3 : printfunc(1,3)\n\nDead jobs:\n4 : <function diefunc at 0x304d488>\n5 : <function diefunc at 0x304d488>\n\n"
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	}
		],
		"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 37,
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "jobs.status()"
		},
		{
		"source": "For any completed job, you can get its result easily:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
Fernando Perez Add basic test suite to background jobs library.	r4941	"outputs": [],
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"collapsed": false,
Fernando Perez Add basic test suite to background jobs library.	r4941	"prompt_number": 43,
		"input": "jobs[0].result\nj0 = jobs[0]\nj0.join?"
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	},
		{
		"source": "You can get the traceback of any dead job. Run the line\nbelow again interactively until it prints a traceback (check the status\nof the job):\n",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Status of diejob1: Dead (Exception), call jobs.traceback() for details\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n"
		}
		],
		"collapsed": false,
		"prompt_number": 34,
		"input": "print \"Status of diejob1:\", diejob1.status\ndiejob1.traceback() # jobs.traceback(4) would also work here, with the job number"
		},
		{
		"source": "This will print all tracebacks for all dead jobs:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Traceback for: <BackgroundJob #4: <function diefunc at 0x30df758>>\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n\nTraceback for: <BackgroundJob #5: <function diefunc at 0x30df758>>\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span> Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\"> 462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 463</span> <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--> 464</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\"></span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/<ipython-input-15-54795a097787></span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, a, *kw)</span>\n<span class=\"ansigreen\"> 14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\"></span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 15</span> time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---> 16</span><span class=\"ansiyellow\"> </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">"Dead job with interval %s"</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\"> 18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 2\n\n"
		}
		],
		"collapsed": false,
		"prompt_number": 33,
		"input": "jobs.traceback()"
		},
		{
		"source": "The job manager can be flushed of all completed jobs at any time:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "No jobs to flush.\n"
		}
		],
		"collapsed": false,
		"prompt_number": 25,
		"input": "jobs.flush()"
		},
		{
		"source": "After that, the status is simply empty:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [],
		"collapsed": true,
		"prompt_number": 27,
		"input": "jobs.status()"
		},
		{
Fernando Perez Add basic test suite to background jobs library.	r4941	"source": "It's easy to wait on a job:",
		"cell_type": "markdown"
		},
		{
		"cell_type": "code",
		"language": "python",
		"outputs": [
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Starting job # 7 in a separate thread.\nWill wait for j now...\n"
		},
		{
		"output_type": "stream",
		"stream": "stdout",
		"text": "Result from j:\n"
		},
		{
		"output_type": "pyout",
		"prompt_number": 46,
		"text": "{'args': (), 'interval': 2, 'kwargs': {}}"
		}
		],
		"collapsed": false,
		"prompt_number": 46,
		"input": "j = jobs.new(sleepfunc, 2)\nprint \"Will wait for j now...\"\nsys.stdout.flush()\nj.join()\nprint \"Result from j:\"\nj.result"
		},
		{
Fernando Perez Add a notebook illustrating the backgroundjobs library.	r4940	"input": "",
		"cell_type": "code",
		"collapsed": true,
		"language": "python",
		"outputs": []
		}
		]
		}
		],
		"metadata": {
		"name": "BackgroundJobs"
		},
		"nbformat": 2
		}

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages