##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

add BlockingInProcKM to kernel.inprocess
add BlockingInProcKM to kernel.inprocess
Brian Granger - - Load All Authors

File last commit:

r9201:ef712b0b
r10303:bb774a03
Show More
BackgroundJobs.ipynb
380 lines | 16.9 KiB | text/plain | TextLexer
/ examples / lib / BackgroundJobs.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Simple interactive bacgkround jobs with IPython

We start by loading the backgroundjobs library and defining a few trivial functions to illustrate things with.

In [35]:
from IPython.lib import backgroundjobs as bg

import sys
import time

def sleepfunc(interval=2, *a, **kw):
    args = dict(interval=interval,
                args=a,
                kwargs=kw)
    time.sleep(interval)
    return args

def diefunc(interval=2, *a, **kw):
    time.sleep(interval)
    raise Exception("Dead job with interval %s" % interval)

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print 'In the background...', n
        sys.stdout.flush()
    print 'All done!'
    sys.stdout.flush()

Now, we can create a job manager (called simply jobs) and use it to submit new jobs.


Run the cell below and wait a few seconds for the whole thing to finish, until you see the "All done!" printout.
In [36]:
jobs = bg.BackgroundJobManager()

# Start a few jobs, the first one will have ID # 0
jobs.new(sleepfunc, 4)
jobs.new(sleepfunc, kw={'reps':2})
jobs.new('printfunc(1,3)')

# This makes a couple of jobs which will die.  Let's keep a reference to
# them for easier traceback reporting later
diejob1 = jobs.new(diefunc, 1)
diejob2 = jobs.new(diefunc, 2)

Starting job # 0 in a separate thread.
Starting job # 2 in a separate thread.
Starting job # 3 in a separate thread.
Starting job # 4 in a separate thread.
Starting job # 5 in a separate thread.

In the background... 0

In the background... 1

In the background... 2

All done!

You can check the status of your jobs at any time:

In [37]:
jobs.status()

Completed jobs:
0 : <function sleepfunc at 0x30e1578>
2 : <function sleepfunc at 0x30e1578>
3 : printfunc(1,3)

Dead jobs:
4 : <function diefunc at 0x304d488>
5 : <function diefunc at 0x304d488>

For any completed job, you can get its result easily:

In [43]:
jobs[0].result
j0 = jobs[0]
j0.join?

You can get the traceback of any dead job. Run the line below again interactively until it prints a traceback (check the status of the job):

In [34]:
print "Status of diejob1:", diejob1.status
diejob1.traceback()  # jobs.traceback(4) would also work here, with the job number

Status of diejob1: Dead (Exception), call jobs.traceback() for details
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

This will print all tracebacks for all dead jobs:

In [33]:
jobs.traceback()

Traceback for: &lt;BackgroundJob #4: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

Traceback for: &lt;BackgroundJob #5: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 2

The job manager can be flushed of all completed jobs at any time:

In [25]:
jobs.flush()

No jobs to flush.

After that, the status is simply empty:

In [27]:
jobs.status()

It's easy to wait on a job:

In [46]:
j = jobs.new(sleepfunc, 2)
print "Will wait for j now..."
sys.stdout.flush()
j.join()
print "Result from j:"
j.result

Starting job # 7 in a separate thread.
Will wait for j now...

Result from j:
Out[46]:
{&apos;args&apos;: (), &apos;interval&apos;: 2, &apos;kwargs&apos;: {}}
In [ ]: