##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

Statically type OInfo. (#13973)...
Statically type OInfo. (#13973) In view of working with #13860, some cleanup inspect to be properly typed, and using stricter datastructure. Instead of dict we now use dataclasses, this will make sure that fields type and access can be stricter and verified not only at runtime, but by mypy
Min RK - - Load All Authors

File last commit:

r20547:8f4e2b41
r28166:29b451fc merge
Show More
Background Jobs.ipynb
404 lines | 15.1 KiB | text/plain | TextLexer
/ examples / IPython Kernel / Background Jobs.ipynb
History | Annotation | Raw |Copy content |Copy permalink

Simple interactive bacgkround jobs with IPython

We start by loading the backgroundjobs library and defining a few trivial functions to illustrate things with.

In [1]:
from IPython.lib import backgroundjobs as bg

import sys
import time

def sleepfunc(interval=2, *a, **kw):
    args = dict(interval=interval,
                args=a,
                kwargs=kw)
    time.sleep(interval)
    return args

def diefunc(interval=2, *a, **kw):
    time.sleep(interval)
    raise Exception("Dead job with interval %s" % interval)

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print('In the background... %i' % n)
        sys.stdout.flush()
    print('All done!')
    sys.stdout.flush()

Now, we can create a job manager (called simply jobs) and use it to submit new jobs.

Run the cell below, it will show when the jobs start. Wait a few seconds until you see the 'all done' completion message:

In [2]:
jobs = bg.BackgroundJobManager()

# Start a few jobs, the first one will have ID # 0
jobs.new(sleepfunc, 4)
jobs.new(sleepfunc, kw={'reps':2})
jobs.new('printfunc(1,3)')

Starting job # 0 in a separate thread.
Starting job # 2 in a separate thread.
Starting job # 3 in a separate thread.
Out[2]:
<BackgroundJob #3: printfunc(1,3)>
In the background... 0
In the background... 1
In the background... 2
All done!

You can check the status of your jobs at any time:

In [3]:
jobs.status()

Completed jobs:
0 : <function sleepfunc at 0x10521f2f0>
2 : <function sleepfunc at 0x10521f2f0>
3 : printfunc(1,3)

For any completed job, you can get its result easily:

In [4]:
jobs[0].result
Out[4]:
{'args': (), 'interval': 4, 'kwargs': {}}

Errors and tracebacks

The jobs manager tries to help you with debugging:

In [5]:
# This makes a couple of jobs which will die.  Let's keep a reference to
# them for easier traceback reporting later
diejob1 = jobs.new(diefunc, 1)
diejob2 = jobs.new(diefunc, 2)

Starting job # 4 in a separate thread.
Starting job # 5 in a separate thread.

You can get the traceback of any dead job. Run the line below again interactively until it prints a traceback (check the status of the job):

In [6]:
print("Status of diejob1: %s" % diejob1.status)
diejob1.traceback()  # jobs.traceback(4) would also work here, with the job number

Status of diejob1: Dead (Exception), call jobs.traceback() for details
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/minrk/dev/ip/mine/IPython/lib/backgroundjobs.py in call(self)
    489 
    490     def call(self):
--> 491         return self.func(*self.args, **self.kwargs)

<ipython-input-1-169e49434ce0> in diefunc(interval, *a, **kw)
     13 def diefunc(interval=2, *a, **kw):
     14     time.sleep(interval)
---> 15     raise Exception("Dead job with interval %s" % interval)
     16 
     17 def printfunc(interval=1, reps=5):

Exception: Dead job with interval 1

This will print all tracebacks for all dead jobs:

In [7]:
jobs.traceback()

Traceback for: <BackgroundJob #4: <function diefunc at 0x10521f7b8>>
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/minrk/dev/ip/mine/IPython/lib/backgroundjobs.py in call(self)
    489 
    490     def call(self):
--> 491         return self.func(*self.args, **self.kwargs)

<ipython-input-1-169e49434ce0> in diefunc(interval, *a, **kw)
     13 def diefunc(interval=2, *a, **kw):
     14     time.sleep(interval)
---> 15     raise Exception("Dead job with interval %s" % interval)
     16 
     17 def printfunc(interval=1, reps=5):

Exception: Dead job with interval 1

Traceback for: <BackgroundJob #5: <function diefunc at 0x10521f7b8>>
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/minrk/dev/ip/mine/IPython/lib/backgroundjobs.py in call(self)
    489 
    490     def call(self):
--> 491         return self.func(*self.args, **self.kwargs)

<ipython-input-1-169e49434ce0> in diefunc(interval, *a, **kw)
     13 def diefunc(interval=2, *a, **kw):
     14     time.sleep(interval)
---> 15     raise Exception("Dead job with interval %s" % interval)
     16 
     17 def printfunc(interval=1, reps=5):

Exception: Dead job with interval 2

The job manager can be flushed of all completed jobs at any time:

In [8]:
jobs.flush()

Flushing 3 Completed jobs.
Flushing 2 Dead jobs.

After that, the status is simply empty:

In [9]:
jobs.status()

Jobs have a .join method that lets you wait on their thread for completion:

In [10]:
j = jobs.new(sleepfunc, 2)
j.join?

Starting job # 0 in a separate thread.

Exercise

  1. Start a new job that calls sleepfunc with a 5-second wait
  2. Print a short message that indicates you are waiting (note: you'll need to flush stdout to see that print output appear).
  3. Wait on the job and then print its result.