##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

Strip prompts even if the prompt isn't present on the first line....
Strip prompts even if the prompt isn't present on the first line. Users (including me) often copy and paste examples from just after the first prompt. This checks for prompts in the first two lines of a block, but if they aren't found there, it will ignore prompt-like prefixes in later lines, to minimise interference with multi-line strings etc. Closes gh-3206
Brian Granger - - Load All Authors

File last commit:

r9201:ef712b0b
r10652:e3df40cf
Show More
BackgroundJobs.ipynb
380 lines | 16.9 KiB | text/plain | TextLexer
/ examples / lib / BackgroundJobs.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Simple interactive bacgkround jobs with IPython

We start by loading the backgroundjobs library and defining a few trivial functions to illustrate things with.

In [35]:
from IPython.lib import backgroundjobs as bg

import sys
import time

def sleepfunc(interval=2, *a, **kw):
    args = dict(interval=interval,
                args=a,
                kwargs=kw)
    time.sleep(interval)
    return args

def diefunc(interval=2, *a, **kw):
    time.sleep(interval)
    raise Exception("Dead job with interval %s" % interval)

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print 'In the background...', n
        sys.stdout.flush()
    print 'All done!'
    sys.stdout.flush()

Now, we can create a job manager (called simply jobs) and use it to submit new jobs.


Run the cell below and wait a few seconds for the whole thing to finish, until you see the "All done!" printout.
In [36]:
jobs = bg.BackgroundJobManager()

# Start a few jobs, the first one will have ID # 0
jobs.new(sleepfunc, 4)
jobs.new(sleepfunc, kw={'reps':2})
jobs.new('printfunc(1,3)')

# This makes a couple of jobs which will die.  Let's keep a reference to
# them for easier traceback reporting later
diejob1 = jobs.new(diefunc, 1)
diejob2 = jobs.new(diefunc, 2)

Starting job # 0 in a separate thread.
Starting job # 2 in a separate thread.
Starting job # 3 in a separate thread.
Starting job # 4 in a separate thread.
Starting job # 5 in a separate thread.

In the background... 0

In the background... 1

In the background... 2

All done!

You can check the status of your jobs at any time:

In [37]:
jobs.status()

Completed jobs:
0 : <function sleepfunc at 0x30e1578>
2 : <function sleepfunc at 0x30e1578>
3 : printfunc(1,3)

Dead jobs:
4 : <function diefunc at 0x304d488>
5 : <function diefunc at 0x304d488>

For any completed job, you can get its result easily:

In [43]:
jobs[0].result
j0 = jobs[0]
j0.join?

You can get the traceback of any dead job. Run the line below again interactively until it prints a traceback (check the status of the job):

In [34]:
print "Status of diejob1:", diejob1.status
diejob1.traceback()  # jobs.traceback(4) would also work here, with the job number

Status of diejob1: Dead (Exception), call jobs.traceback() for details
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

This will print all tracebacks for all dead jobs:

In [33]:
jobs.traceback()

Traceback for: &lt;BackgroundJob #4: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 1

Traceback for: &lt;BackgroundJob #5: &lt;function diefunc at 0x30df758&gt;&gt;
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">Exception</span>                                 Traceback (most recent call last)
<span class="ansigreen">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class="ansicyan">call</span><span class="ansiblue">(self)</span>
<span class="ansigreen">    462</span> <span class="ansiyellow"></span>
<span class="ansigreen">    463</span>     <span class="ansigreen">def</span> call<span class="ansiyellow">(</span>self<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--&gt; 464</span><span class="ansiyellow">         </span><span class="ansigreen">return</span> self<span class="ansiyellow">.</span>func<span class="ansiyellow">(</span><span class="ansiyellow">*</span>self<span class="ansiyellow">.</span>args<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>self<span class="ansiyellow">.</span>kwargs<span class="ansiyellow">)</span><span class="ansiyellow"></span>

<span class="ansigreen">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class="ansicyan">diefunc</span><span class="ansiblue">(interval, *a, **kw)</span>
<span class="ansigreen">     14</span> <span class="ansigreen">def</span> diefunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">2</span><span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>kw<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">     15</span>     time<span class="ansiyellow">.</span>sleep<span class="ansiyellow">(</span>interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">---&gt; 16</span><span class="ansiyellow">     </span><span class="ansigreen">raise</span> Exception<span class="ansiyellow">(</span><span class="ansiblue">&quot;Dead job with interval %s&quot;</span> <span class="ansiyellow">%</span> interval<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">     17</span> <span class="ansiyellow"></span>
<span class="ansigreen">     18</span> <span class="ansigreen">def</span> printfunc<span class="ansiyellow">(</span>interval<span class="ansiyellow">=</span><span class="ansicyan">1</span><span class="ansiyellow">,</span> reps<span class="ansiyellow">=</span><span class="ansicyan">5</span><span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>

<span class="ansired">Exception</span>: Dead job with interval 2

The job manager can be flushed of all completed jobs at any time:

In [25]:
jobs.flush()

No jobs to flush.

After that, the status is simply empty:

In [27]:
jobs.status()

It's easy to wait on a job:

In [46]:
j = jobs.new(sleepfunc, 2)
print "Will wait for j now..."
sys.stdout.flush()
j.join()
print "Result from j:"
j.result

Starting job # 7 in a separate thread.
Will wait for j now...

Result from j:
Out[46]:
{&apos;args&apos;: (), &apos;interval&apos;: 2, &apos;kwargs&apos;: {}}
In [ ]: