Using dill to pickle anything¶
IPython.parallel doesn't do much in the way of serialization. It has custom zero-copy handling of numpy arrays, but other than that, it doesn't do anything other than the bare minimum to make basic interactively defined functions and classes sendable.
There are a few projects that extend pickle to make just about anything sendable, and one of these is dill.
To install dill:
pip install --pre dill
First, as always, we create a task function, this time with a closure
def make_closure(a):
"""make a function with a closure, and return it"""
def has_closure(b):
return a * b
return has_closure
closed = make_closure(5)
closed(2)
import pickle
Without help, pickle can't deal with closures
pickle.dumps(closed)
But after we import dill, magic happens
import dill
pickle.dumps(closed)[:64] + '...'
So from now on, pretty much everything is pickleable.
Now use this in IPython.parallel¶
As usual, we start by creating our Client and View
from IPython import parallel
rc = parallel.Client()
view = rc.load_balanced_view()
Now let's try sending our function with a closure:
view.apply_sync(closed, 3)
Oops, no dice. For IPython to work with dill,
there are one or two more steps. IPython will do these for you if you call DirectView.use_dill:
rc[:].use_dill()
This is equivalent to
from IPython.utils.pickleutil import use_dill
use_dill()
rc[:].apply(use_dill)
Now let's try again
view.apply_sync(closed, 3)
Yay! Now we can use dill to allow IPython.parallel to send anything.
And that's it! We can send closures and other previously non-pickleables to our engines.
Let's give it a try now:
remote_closure = view.apply_sync(make_closure, 4)
remote_closure(5)
But wait, there's more!
At this point, we can send/recv all kinds of stuff
def outer(a):
def inner(b):
def inner_again(c):
return c * b * a
return inner_again
return inner
So outer returns a function with a closure, which returns a function with a closure.
Now, we can resolve the first closure on the engine, the second here, and the third on a different engine, after passing through a lambda we define here and call there, just for good measure.
view.apply_sync(lambda f: f(3),view.apply_sync(outer, 1)(2))
And for good measure, let's test that normal execution still works:
%px foo = 5
print(rc[:]['foo'])
rc[:]['bar'] = lambda : 2 * foo
rc[:].apply_sync(parallel.Reference('bar'))
And test that the @interactive decorator works
%%file testdill.py
from IPython.parallel import interactive
@interactive
class C(object):
a = 5
@interactive
class D(C):
b = 10
@interactive
def foo(a):
return a * b
import testdill
v = rc[-1]
v['D'] = testdill.D
d = v.apply_sync(lambda : D())
print d.a, d.b
v['b'] = 10
v.apply_sync(testdill.foo, 5)