##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

Merge pull request #1369 from minrk/EngineError...
Merge pull request #1369 from minrk/EngineError load header with engine id when engine dies in TaskScheduler This ensures that the metadata dict on the Client has the engine_uuid of the engine on which the task failed. Previously, this entry would remain empty. It is identical to code elsewhere (Hub, Client) for constructing the dummy reply when engines die.
Fernando Perez - - Load All Authors

File last commit:

r4941:eac11eb3
r6098:0291d619 merge
Show More
BackgroundJobs.ipynb
192 lines | 17.8 KiB | text/plain | TextLexer
/ docs / examples / lib / BackgroundJobs.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Simple interactive bacgkround jobs with IPython

We start by loading the backgroundjobs library and defining a few trivial functions to illustrate things with.

In [35]:
from IPython.lib import backgroundjobs as bg

import sys
import time

def sleepfunc(interval=2, *a, **kw):
    args = dict(interval=interval,
                args=a,
                kwargs=kw)
    time.sleep(interval)
    return args

def diefunc(interval=2, *a, **kw):
    time.sleep(interval)
    raise Exception("Dead job with interval %s" % interval)

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print 'In the background...', n
        sys.stdout.flush()
    print 'All done!'
    sys.stdout.flush()

Now, we can create a job manager (called simply jobs) and use it to submit new jobs.


Run the cell below and wait a few seconds for the whole thing to finish, until you see the "All done!" printout.
In [36]:
jobs = bg.BackgroundJobManager()

# Start a few jobs, the first one will have ID # 0
jobs.new(sleepfunc, 4)
jobs.new(sleepfunc, kw={'reps':2})
jobs.new('printfunc(1,3)')

# This makes a couple of jobs which will die.  Let's keep a reference to
# them for easier traceback reporting later
diejob1 = jobs.new(diefunc, 1)
diejob2 = jobs.new(diefunc, 2)

Starting job # 0 in a separate thread.
Starting job # 2 in a separate thread.
Starting job # 3 in a separate thread.
Starting job # 4 in a separate thread.
Starting job # 5 in a separate thread.

In the background... 0

In the background... 1

In the background... 2

All done!

You can check the status of your jobs at any time:

In [37]:
jobs.status()

Completed jobs:
0 : <function sleepfunc at 0x30e1578>
2 : <function sleepfunc at 0x30e1578>
3 : printfunc(1,3)

Dead jobs:
4 : <function diefunc at 0x304d488>
5 : <function diefunc at 0x304d488>

For any completed job, you can get its result easily:

In [43]:
jobs[0].result
j0 = jobs[0]
j0.join?

You can get the traceback of any dead job. Run the line below again interactively until it prints a traceback (check the status of the job):

In [34]:
print "Status of diejob1:", diejob1.status
diejob1.traceback()  # jobs.traceback(4) would also work here, with the job number

Status of diejob1: Dead (Exception), call jobs.traceback() for details
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

This will print all tracebacks for all dead jobs:

In [33]:
jobs.traceback()

Traceback for: &lt;BackgroundJob #4: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

Traceback for: &lt;BackgroundJob #5: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 2

The job manager can be flushed of all completed jobs at any time:

In [25]:
jobs.flush()

No jobs to flush.

After that, the status is simply empty:

In [27]:
jobs.status()

It's easy to wait on a job:

In [46]:
j = jobs.new(sleepfunc, 2)
print "Will wait for j now..."
sys.stdout.flush()
j.join()
print "Result from j:"
j.result

Starting job # 7 in a separate thread.
Will wait for j now...

Result from j:
Out[46]:
{&apos;args&apos;: (), &apos;interval&apos;: 2, &apos;kwargs&apos;: {}}
In [ ]: