Show More
@@ -204,6 +204,7 b" intersphinx_mapping = {'python': ('http://docs.python.org/2/', None)," | |||
|
204 | 204 | 'rpy2': ('http://rpy.sourceforge.net/rpy2/doc-2.4/html/', None), |
|
205 | 205 | 'traitlets': ('http://traitlets.readthedocs.org/en/latest/', None), |
|
206 | 206 | 'jupyterclient': ('http://jupyter-client.readthedocs.org/en/latest/', None), |
|
207 | 'ipyparallel': ('http://ipyparallel.readthedocs.org/en/latest/', None), | |
|
207 | 208 | } |
|
208 | 209 | |
|
209 | 210 | # Options for LaTeX output |
@@ -25,7 +25,7 b' the terminal, and third party interfaces\xe2\x80\x94use the IPython Kernel. This is a' | |||
|
25 | 25 | separate process which is responsible for running user code, and things like |
|
26 | 26 | computing possible completions. Frontends communicate with it using JSON |
|
27 | 27 | messages sent over `ZeroMQ <http://zeromq.org/>`_ sockets; the protocol they use is described in |
|
28 | :doc:`messaging`. | |
|
28 | :ref:`jupyterclient:messaging`. | |
|
29 | 29 | |
|
30 | 30 | The core execution machinery for the kernel is shared with terminal IPython: |
|
31 | 31 | |
@@ -57,48 +57,7 b' likely to be better maintained by the community using them, like' | |||
|
57 | 57 | |
|
58 | 58 | .. seealso:: |
|
59 | 59 | |
|
60 | :doc:`kernels` | |
|
60 | :ref:`jupyterclient:kernels` | |
|
61 | 61 | |
|
62 | 62 | :doc:`wrapperkernels` |
|
63 | 63 | |
|
64 | Notebooks | |
|
65 | --------- | |
|
66 | ||
|
67 | The Notebook frontend does something extra. In addition to running your code, it | |
|
68 | stores code and output, together with markdown notes, in an editable document | |
|
69 | called a notebook. When you save it, this is sent from your browser to the | |
|
70 | notebook server, which saves it on disk as a JSON file with a ``.ipynb`` | |
|
71 | extension. | |
|
72 | ||
|
73 | .. image:: figs/notebook_components.png | |
|
74 | ||
|
75 | The notebook server, not the kernel, is responsible for saving and loading | |
|
76 | notebooks, so you can edit notebooks even if you don't have the kernel for that | |
|
77 | language—you just won't be able to run code. The kernel doesn't know anything | |
|
78 | about the notebook document: it just gets sent cells of code to execute when the | |
|
79 | user runs them. | |
|
80 | ||
|
81 | Exporting to other formats | |
|
82 | `````````````````````````` | |
|
83 | ||
|
84 | The Nbconvert tool in IPython converts notebook files to other formats, such as | |
|
85 | HTML, LaTeX, or reStructuredText. This conversion goes through a series of steps: | |
|
86 | ||
|
87 | .. image:: figs/nbconvert.png | |
|
88 | ||
|
89 | 1. Preprocessors modify the notebook in memory. E.g. ExecutePreprocessor runs | |
|
90 | the code in the notebook and updates the output. | |
|
91 | 2. An exporter converts the notebook to another file format. Most of the | |
|
92 | exporters use templates for this. | |
|
93 | 3. Postprocessors work on the file produced by exporting. | |
|
94 | ||
|
95 | The `nbviewer <http://nbviewer.ipython.org/>`_ website uses nbconvert with the | |
|
96 | HTML exporter. When you give it a URL, it fetches the notebook from that URL, | |
|
97 | converts it to HTML, and serves that HTML to you. | |
|
98 | ||
|
99 | IPython.parallel | |
|
100 | ---------------- | |
|
101 | ||
|
102 | IPython also includes a parallel computing framework, ``IPython.parallel``. This | |
|
103 | allows you to control many individual engines, which are an extended version of | |
|
104 | the IPython kernel described above. For more details, see :doc:`/parallel/index`. |
@@ -20,11 +20,8 b' on the IPython GitHub wiki.' | |||
|
20 | 20 | :maxdepth: 1 |
|
21 | 21 | |
|
22 | 22 | how_ipython_works |
|
23 | kernels | |
|
24 | 23 | wrapperkernels |
|
25 | 24 | execution |
|
26 | parallel_messages | |
|
27 | parallel_connections | |
|
28 | 25 | lexer |
|
29 | 26 | pycompat |
|
30 | 27 | config |
@@ -1,143 +1,8 b'' | |||
|
1 | :orphan: | |
|
2 | ||
|
1 | 3 | ========================== |
|
2 | 4 | Making kernels for IPython |
|
3 | 5 | ========================== |
|
4 | 6 | |
|
5 | A 'kernel' is a program that runs and introspects the user's code. IPython | |
|
6 | includes a kernel for Python code, and people have written kernels for | |
|
7 | `several other languages <https://github.com/ipython/ipython/wiki/Projects-using-IPython#list-of-some-ipython-compatible-kernels>`_. | |
|
8 | ||
|
9 | When IPython starts a kernel, it passes it a connection file. This specifies | |
|
10 | how to set up communications with the frontend. | |
|
11 | ||
|
12 | There are two options for writing a kernel: | |
|
13 | ||
|
14 | 1. You can reuse the IPython kernel machinery to handle the communications, and | |
|
15 | just describe how to execute your code. This is much simpler if the target | |
|
16 | language can be driven from Python. See :doc:`wrapperkernels` for details. | |
|
17 | 2. You can implement the kernel machinery in your target language. This is more | |
|
18 | work initially, but the people using your kernel might be more likely to | |
|
19 | contribute to it if it's in the language they know. | |
|
20 | ||
|
21 | Connection files | |
|
22 | ================ | |
|
23 | ||
|
24 | Your kernel will be given the path to a connection file when it starts (see | |
|
25 | :ref:`kernelspecs` for how to specify the command line arguments for your kernel). | |
|
26 | This file, which is accessible only to the current user, will contain a JSON | |
|
27 | dictionary looking something like this:: | |
|
28 | ||
|
29 | { | |
|
30 | "control_port": 50160, | |
|
31 | "shell_port": 57503, | |
|
32 | "transport": "tcp", | |
|
33 | "signature_scheme": "hmac-sha256", | |
|
34 | "stdin_port": 52597, | |
|
35 | "hb_port": 42540, | |
|
36 | "ip": "127.0.0.1", | |
|
37 | "iopub_port": 40885, | |
|
38 | "key": "a0436f6c-1916-498b-8eb9-e81ab9368e84" | |
|
39 | } | |
|
40 | ||
|
41 | The ``transport``, ``ip`` and five ``_port`` fields specify five ports which the | |
|
42 | kernel should bind to using `ZeroMQ <http://zeromq.org/>`_. For instance, the | |
|
43 | address of the shell socket in the example above would be:: | |
|
44 | ||
|
45 | tcp://127.0.0.1:57503 | |
|
46 | ||
|
47 | New ports are chosen at random for each kernel started. | |
|
48 | ||
|
49 | ``signature_scheme`` and ``key`` are used to cryptographically sign messages, so | |
|
50 | that other users on the system can't send code to run in this kernel. See | |
|
51 | :ref:`wire_protocol` for the details of how this signature is calculated. | |
|
52 | ||
|
53 | Handling messages | |
|
54 | ================= | |
|
55 | ||
|
56 | After reading the connection file and binding to the necessary sockets, the | |
|
57 | kernel should go into an event loop, listening on the hb (heartbeat), control | |
|
58 | and shell sockets. | |
|
59 | ||
|
60 | :ref:`Heartbeat <kernel_heartbeat>` messages should be echoed back immediately | |
|
61 | on the same socket - the frontend uses this to check that the kernel is still | |
|
62 | alive. | |
|
63 | ||
|
64 | Messages on the control and shell sockets should be parsed, and their signature | |
|
65 | validated. See :ref:`wire_protocol` for how to do this. | |
|
66 | ||
|
67 | The kernel will send messages on the iopub socket to display output, and on the | |
|
68 | stdin socket to prompt the user for textual input. | |
|
69 | ||
|
70 | .. seealso:: | |
|
71 | ||
|
72 | :doc:`messaging` | |
|
73 | Details of the different sockets and the messages that come over them | |
|
74 | ||
|
75 | `Creating Language Kernels for IPython <http://andrew.gibiansky.com/blog/ipython/ipython-kernels/>`_ | |
|
76 | A blog post by the author of `IHaskell <https://github.com/gibiansky/IHaskell>`_, | |
|
77 | a Haskell kernel | |
|
78 | ||
|
79 | `simple_kernel <https://github.com/dsblank/simple_kernel>`_ | |
|
80 | A simple example implementation of the kernel machinery in Python | |
|
81 | ||
|
82 | ||
|
83 | .. _kernelspecs: | |
|
84 | ||
|
85 | Kernel specs | |
|
86 | ============ | |
|
87 | ||
|
88 | A kernel identifies itself to IPython by creating a directory, the name of which | |
|
89 | is used as an identifier for the kernel. These may be created in a number of | |
|
90 | locations: | |
|
91 | ||
|
92 | +--------+--------------------------------------+-----------------------------------+ | |
|
93 | | | Unix | Windows | | |
|
94 | +========+======================================+===================================+ | |
|
95 | | System | ``/usr/share/jupyter/kernels`` | ``%PROGRAMDATA%\jupyter\kernels`` | | |
|
96 | | | | | | |
|
97 | | | ``/usr/local/share/jupyter/kernels`` | | | |
|
98 | +--------+--------------------------------------+-----------------------------------+ | |
|
99 | | User | ``~/.ipython/kernels`` | | |
|
100 | +--------+--------------------------------------+-----------------------------------+ | |
|
101 | ||
|
102 | The user location takes priority over the system locations, and the case of the | |
|
103 | names is ignored, so selecting kernels works the same way whether or not the | |
|
104 | filesystem is case sensitive. | |
|
105 | ||
|
106 | Inside the directory, the most important file is *kernel.json*. This should be a | |
|
107 | JSON serialised dictionary containing the following keys and values: | |
|
108 | ||
|
109 | - **argv**: A list of command line arguments used to start the kernel. The text | |
|
110 | ``{connection_file}`` in any argument will be replaced with the path to the | |
|
111 | connection file. | |
|
112 | - **display_name**: The kernel's name as it should be displayed in the UI. | |
|
113 | Unlike the kernel name used in the API, this can contain arbitrary unicode | |
|
114 | characters. | |
|
115 | - **language**: The name of the language of the kernel. | |
|
116 | When loading notebooks, if no matching kernelspec key (may differ across machines) | |
|
117 | is found, a kernel with a matching `language` will be used. | |
|
118 | This allows a notebook written on any Python or Julia kernel to be properly associated | |
|
119 | with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's. | |
|
120 | - **env** (optional): A dictionary of environment variables to set for the kernel. | |
|
121 | These will be added to the current environment variables before the kernel is | |
|
122 | started. | |
|
123 | ||
|
124 | For example, the kernel.json file for IPython looks like this:: | |
|
125 | ||
|
126 | { | |
|
127 | "argv": ["python3", "-m", "IPython.kernel", | |
|
128 | "-f", "{connection_file}"], | |
|
129 | "display_name": "Python 3", | |
|
130 | "language": "python" | |
|
131 | } | |
|
132 | ||
|
133 | To see the available kernel specs, run:: | |
|
134 | ||
|
135 | ipython kernelspec list | |
|
136 | ||
|
137 | To start the terminal console or the Qt console with a specific kernel:: | |
|
138 | ||
|
139 | ipython console --kernel bash | |
|
140 | ipython qtconsole --kernel bash | |
|
141 | ||
|
142 | To use different kernels in the notebook, select a different kernel from the | |
|
143 | dropdown menu in the top-right of the UI. | |
|
7 | Kernels are now part of Jupyter - see | |
|
8 | :ref:`jupyterclient:kernels` for the documentation. |
@@ -1,154 +1,8 b'' | |||
|
1 | .. _parallel_connections: | |
|
1 | :orphan: | |
|
2 | 2 | |
|
3 | 3 | ============================================== |
|
4 | 4 | Connection Diagrams of The IPython ZMQ Cluster |
|
5 | 5 | ============================================== |
|
6 | 6 | |
|
7 | This is a quick summary and illustration of the connections involved in the ZeroMQ based | |
|
8 | IPython cluster for parallel computing. | |
|
9 | ||
|
10 | All Connections | |
|
11 | =============== | |
|
12 | ||
|
13 | The IPython cluster consists of a Controller, and one or more each of clients and engines. | |
|
14 | The goal of the Controller is to manage and monitor the connections and communications | |
|
15 | between the clients and the engines. The Controller is no longer a single process entity, | |
|
16 | but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers. | |
|
17 | ||
|
18 | It is important for security/practicality reasons that all connections be inbound to the | |
|
19 | controller processes. The arrows in the figures indicate the direction of the | |
|
20 | connection. | |
|
21 | ||
|
22 | ||
|
23 | .. figure:: figs/allconnections.png | |
|
24 | :width: 432px | |
|
25 | :alt: IPython cluster connections | |
|
26 | :align: center | |
|
27 | ||
|
28 | All the connections involved in connecting one client to one engine. | |
|
29 | ||
|
30 | The Controller consists of 1-5 processes. Central to the cluster is the **Hub**, which monitors | |
|
31 | engine state, execution traffic, and handles registration and notification. The Hub includes a | |
|
32 | Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4 | |
|
33 | **Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally | |
|
34 | threads) that relay messages very fast, but also send a copy of each message along a side socket | |
|
35 | to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay | |
|
36 | explicitly addressed messages from clients to engines, and their replies back up. The Balanced | |
|
37 | queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue | |
|
38 | device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ | |
|
39 | devices, but with additional internal logic. stdout/err are also propagated from the Engines to | |
|
40 | the clients via a PUB/SUB MonitoredQueue. | |
|
41 | ||
|
42 | ||
|
43 | Registration | |
|
44 | ------------ | |
|
45 | ||
|
46 | .. figure:: figs/queryfade.png | |
|
47 | :width: 432px | |
|
48 | :alt: IPython Registration connections | |
|
49 | :align: center | |
|
50 | ||
|
51 | Engines and Clients only need to know where the Query ``ROUTER`` is located to start | |
|
52 | connecting. | |
|
53 | ||
|
54 | Once a controller is launched, the only information needed for connecting clients and/or | |
|
55 | engines is the IP/port of the Hub's ``ROUTER`` socket called the Registrar. This socket | |
|
56 | handles connections from both clients and engines, and replies with the remaining | |
|
57 | information necessary to establish the remaining connections. Clients use this same socket for | |
|
58 | querying the Hub for state information. | |
|
59 | ||
|
60 | Heartbeat | |
|
61 | --------- | |
|
62 | ||
|
63 | .. figure:: figs/hbfade.png | |
|
64 | :width: 432px | |
|
65 | :alt: IPython Heartbeat connections | |
|
66 | :align: center | |
|
67 | ||
|
68 | The heartbeat sockets. | |
|
69 | ||
|
70 | The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor | |
|
71 | publishes a distinct message periodically via a ``PUB`` socket. Each engine has a | |
|
72 | ``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``DEALER`` socket for output. | |
|
73 | The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``DEALER`` is | |
|
74 | connected to the ``ROUTER`` labeled *pong*. This results in the same message being relayed | |
|
75 | back to the Heartbeat Monitor with the addition of the ``DEALER`` prefix. The Heartbeat | |
|
76 | Monitor receives all the replies via an ``ROUTER`` socket, and identifies which hearts are | |
|
77 | still beating by the ``zmq.IDENTITY`` prefix of the ``DEALER`` sockets, which information | |
|
78 | the Hub uses to notify clients of any changes in the available engines. | |
|
79 | ||
|
80 | Schedulers | |
|
81 | ---------- | |
|
82 | ||
|
83 | .. figure:: figs/queuefade.png | |
|
84 | :width: 432px | |
|
85 | :alt: IPython Queue connections | |
|
86 | :align: center | |
|
87 | ||
|
88 | Control message scheduler on the left, execution (apply) schedulers on the right. | |
|
89 | ||
|
90 | The controller has at least three Schedulers. These devices are primarily for | |
|
91 | relaying messages between clients and engines, but the Hub needs to see those | |
|
92 | messages for its own purposes. Since no Python code may exist between the two sockets in a | |
|
93 | queue, all messages sent through these queues (both directions) are also sent via a | |
|
94 | ``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without | |
|
95 | interfering with it. | |
|
96 | ||
|
97 | For tasks, the engine need not be specified. Messages sent to the ``ROUTER`` socket from the | |
|
98 | client side are assigned to an engine via ZMQ's ``DEALER`` round-robin load balancing. | |
|
99 | Engine replies are directed to specific clients via the IDENTITY of the client, which is | |
|
100 | received as a prefix at the Engine. | |
|
101 | ||
|
102 | For Multiplexing, ``ROUTER`` is used for both in and output sockets in the device. Clients must | |
|
103 | specify the destination by the ``zmq.IDENTITY`` of the ``ROUTER`` socket connected to | |
|
104 | the downstream end of the device. | |
|
105 | ||
|
106 | At the Kernel level, both of these ``ROUTER`` sockets are treated in the same way as the ``REP`` | |
|
107 | socket in the serial version (except using ZMQStreams instead of explicit sockets). | |
|
108 | ||
|
109 | Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified) | |
|
110 | manner. The sockets on the Client and Engine are the same for these two actions, but the | |
|
111 | scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of | |
|
112 | the upstream sockets in each MonitoredQueue. | |
|
113 | ||
|
114 | IOPub | |
|
115 | ----- | |
|
116 | ||
|
117 | .. figure:: figs/iopubfade.png | |
|
118 | :width: 432px | |
|
119 | :alt: IOPub connections | |
|
120 | :align: center | |
|
121 | ||
|
122 | stdout/err are published via a ``PUB/SUB`` MonitoredQueue | |
|
123 | ||
|
124 | ||
|
125 | On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB`` | |
|
126 | sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all | |
|
127 | messages. They are then republished via another ``PUB`` socket, which can be | |
|
128 | subscribed by the clients. | |
|
129 | ||
|
130 | Client connections | |
|
131 | ------------------ | |
|
132 | ||
|
133 | .. figure:: figs/queryfade.png | |
|
134 | :width: 432px | |
|
135 | :alt: IPython client query connections | |
|
136 | :align: center | |
|
137 | ||
|
138 | Clients connect to an ``ROUTER`` socket to query the hub. | |
|
139 | ||
|
140 | The hub's registrar ``ROUTER`` socket also listens for queries from clients as to queue status, | |
|
141 | and control instructions. Clients connect to this socket via an ``DEALER`` during registration. | |
|
142 | ||
|
143 | .. figure:: figs/notiffade.png | |
|
144 | :width: 432px | |
|
145 | :alt: IPython Registration connections | |
|
146 | :align: center | |
|
147 | ||
|
148 | Engine registration events are published via a ``PUB`` socket. | |
|
149 | ||
|
150 | The Hub publishes all registration/unregistration events via a ``PUB`` socket. This | |
|
151 | allows clients to stay up to date with what engines are available by subscribing to the | |
|
152 | feed with a ``SUB`` socket. Other processes could selectively subscribe to just | |
|
153 | registration or unregistration events. | |
|
154 | ||
|
7 | IPython parallel has moved to ipyparallel - | |
|
8 | see :ref:`ipyparallel:parallel_connections` for the documentation. |
@@ -1,367 +1,8 b'' | |||
|
1 | .. _parallel_messages: | |
|
1 | :orphan: | |
|
2 | 2 | |
|
3 | ================================ | |
|
3 | 4 | Messaging for Parallel Computing |
|
4 | 5 | ================================ |
|
5 | 6 | |
|
6 | This is an extension of the :ref:`messaging <messaging>` doc. Diagrams of the connections | |
|
7 | can be found in the :ref:`parallel connections <parallel_connections>` doc. | |
|
8 | ||
|
9 | ||
|
10 | ZMQ messaging is also used in the parallel computing IPython system. All messages to/from | |
|
11 | kernels remain the same as the single kernel model, and are forwarded through a ZMQ Queue | |
|
12 | device. The controller receives all messages and replies in these channels, and saves | |
|
13 | results for future use. | |
|
14 | ||
|
15 | The Controller | |
|
16 | -------------- | |
|
17 | ||
|
18 | The controller is the central collection of processes in the IPython parallel computing | |
|
19 | model. It has two major components: | |
|
20 | ||
|
21 | * The Hub | |
|
22 | * A collection of Schedulers | |
|
23 | ||
|
24 | The Hub | |
|
25 | ------- | |
|
26 | ||
|
27 | The Hub is the central process for monitoring the state of the engines, and all task | |
|
28 | requests and results. It has no role in execution and does no relay of messages, so | |
|
29 | large blocking requests or database actions in the Hub do not have the ability to impede | |
|
30 | job submission and results. | |
|
31 | ||
|
32 | Registration (``ROUTER``) | |
|
33 | ************************* | |
|
34 | ||
|
35 | The first function of the Hub is to facilitate and monitor connections of clients | |
|
36 | and engines. Both client and engine registration are handled by the same socket, so only | |
|
37 | one ip/port pair is needed to connect any number of connections and clients. | |
|
38 | ||
|
39 | Engines register with the ``zmq.IDENTITY`` of their two ``DEALER`` sockets, one for the | |
|
40 | queue, which receives execute requests, and one for the heartbeat, which is used to | |
|
41 | monitor the survival of the Engine process. | |
|
42 | ||
|
43 | Message type: ``registration_request``:: | |
|
44 | ||
|
45 | content = { | |
|
46 | 'uuid' : 'abcd-1234-...', # the zmq.IDENTITY of the engine's sockets | |
|
47 | } | |
|
48 | ||
|
49 | .. note:: | |
|
50 | ||
|
51 | these are always the same, at least for now. | |
|
52 | ||
|
53 | The Controller replies to an Engine's registration request with the engine's integer ID, | |
|
54 | and all the remaining connection information for connecting the heartbeat process, and | |
|
55 | kernel queue socket(s). The message status will be an error if the Engine requests IDs that | |
|
56 | already in use. | |
|
57 | ||
|
58 | Message type: ``registration_reply``:: | |
|
59 | ||
|
60 | content = { | |
|
61 | 'status' : 'ok', # or 'error' | |
|
62 | # if ok: | |
|
63 | 'id' : 0, # int, the engine id | |
|
64 | } | |
|
65 | ||
|
66 | Clients use the same socket as engines to start their connections. Connection requests | |
|
67 | from clients need no information: | |
|
68 | ||
|
69 | Message type: ``connection_request``:: | |
|
70 | ||
|
71 | content = {} | |
|
72 | ||
|
73 | The reply to a Client registration request contains the connection information for the | |
|
74 | multiplexer and load balanced queues, as well as the address for direct hub | |
|
75 | queries. If any of these addresses is `None`, that functionality is not available. | |
|
76 | ||
|
77 | Message type: ``connection_reply``:: | |
|
78 | ||
|
79 | content = { | |
|
80 | 'status' : 'ok', # or 'error' | |
|
81 | } | |
|
82 | ||
|
83 | Heartbeat | |
|
84 | ********* | |
|
85 | ||
|
86 | The hub uses a heartbeat system to monitor engines, and track when they become | |
|
87 | unresponsive. As described in :ref:`messaging <messaging>`, and shown in :ref:`connections | |
|
88 | <parallel_connections>`. | |
|
89 | ||
|
90 | Notification (``PUB``) | |
|
91 | ********************** | |
|
92 | ||
|
93 | The hub publishes all engine registration/unregistration events on a ``PUB`` socket. | |
|
94 | This allows clients to have up-to-date engine ID sets without polling. Registration | |
|
95 | notifications contain both the integer engine ID and the queue ID, which is necessary for | |
|
96 | sending messages via the Multiplexer Queue and Control Queues. | |
|
97 | ||
|
98 | Message type: ``registration_notification``:: | |
|
99 | ||
|
100 | content = { | |
|
101 | 'id' : 0, # engine ID that has been registered | |
|
102 | 'uuid' : 'engine_id' # the IDENT for the engine's sockets | |
|
103 | } | |
|
104 | ||
|
105 | Message type : ``unregistration_notification``:: | |
|
106 | ||
|
107 | content = { | |
|
108 | 'id' : 0 # engine ID that has been unregistered | |
|
109 | 'uuid' : 'engine_id' # the IDENT for the engine's sockets | |
|
110 | } | |
|
111 | ||
|
112 | ||
|
113 | Client Queries (``ROUTER``) | |
|
114 | *************************** | |
|
115 | ||
|
116 | The hub monitors and logs all queue traffic, so that clients can retrieve past | |
|
117 | results or monitor pending tasks. This information may reside in-memory on the Hub, or | |
|
118 | on disk in a database (SQLite and MongoDB are currently supported). These requests are | |
|
119 | handled by the same socket as registration. | |
|
120 | ||
|
121 | ||
|
122 | :func:`queue_request` requests can specify multiple engines to query via the `targets` | |
|
123 | element. A verbose flag can be passed, to determine whether the result should be the list | |
|
124 | of `msg_ids` in the queue or simply the length of each list. | |
|
125 | ||
|
126 | Message type: ``queue_request``:: | |
|
127 | ||
|
128 | content = { | |
|
129 | 'verbose' : True, # whether return should be lists themselves or just lens | |
|
130 | 'targets' : [0,3,1] # list of ints | |
|
131 | } | |
|
132 | ||
|
133 | The content of a reply to a :func:`queue_request` request is a dict, keyed by the engine | |
|
134 | IDs. Note that they will be the string representation of the integer keys, since JSON | |
|
135 | cannot handle number keys. The three keys of each dict are:: | |
|
136 | ||
|
137 | 'completed' : messages submitted via any queue that ran on the engine | |
|
138 | 'queue' : jobs submitted via MUX queue, whose results have not been received | |
|
139 | 'tasks' : tasks that are known to have been submitted to the engine, but | |
|
140 | have not completed. Note that with the pure zmq scheduler, this will | |
|
141 | always be 0/[]. | |
|
142 | ||
|
143 | Message type: ``queue_reply``:: | |
|
144 | ||
|
145 | content = { | |
|
146 | 'status' : 'ok', # or 'error' | |
|
147 | # if verbose=False: | |
|
148 | '0' : {'completed' : 1, 'queue' : 7, 'tasks' : 0}, | |
|
149 | # if verbose=True: | |
|
150 | '1' : {'completed' : ['abcd-...','1234-...'], 'queue' : ['58008-'], 'tasks' : []}, | |
|
151 | } | |
|
152 | ||
|
153 | Clients can request individual results directly from the hub. This is primarily for | |
|
154 | gathering results of executions not submitted by the requesting client, as the client | |
|
155 | will have all its own results already. Requests are made by msg_id, and can contain one or | |
|
156 | more msg_id. An additional boolean key 'statusonly' can be used to not request the | |
|
157 | results, but simply poll the status of the jobs. | |
|
158 | ||
|
159 | Message type: ``result_request``:: | |
|
160 | ||
|
161 | content = { | |
|
162 | 'msg_ids' : ['uuid','...'], # list of strs | |
|
163 | 'targets' : [1,2,3], # list of int ids or uuids | |
|
164 | 'statusonly' : False, # bool | |
|
165 | } | |
|
166 | ||
|
167 | The :func:`result_request` reply contains the content objects of the actual execution | |
|
168 | reply messages. If `statusonly=True`, then there will be only the 'pending' and | |
|
169 | 'completed' lists. | |
|
170 | ||
|
171 | ||
|
172 | Message type: ``result_reply``:: | |
|
173 | ||
|
174 | content = { | |
|
175 | 'status' : 'ok', # else error | |
|
176 | # if ok: | |
|
177 | 'acbd-...' : msg, # the content dict is keyed by msg_ids, | |
|
178 | # values are the result messages | |
|
179 | # there will be none of these if `statusonly=True` | |
|
180 | 'pending' : ['msg_id','...'], # msg_ids still pending | |
|
181 | 'completed' : ['msg_id','...'], # list of completed msg_ids | |
|
182 | } | |
|
183 | buffers = ['bufs','...'] # the buffers that contained the results of the objects. | |
|
184 | # this will be empty if no messages are complete, or if | |
|
185 | # statusonly is True. | |
|
186 | ||
|
187 | For memory management purposes, Clients can also instruct the hub to forget the | |
|
188 | results of messages. This can be done by message ID or engine ID. Individual messages are | |
|
189 | dropped by msg_id, and all messages completed on an engine are dropped by engine ID. This | |
|
190 | may no longer be necessary with the mongodb-based message logging backend. | |
|
191 | ||
|
192 | If the msg_ids element is the string ``'all'`` instead of a list, then all completed | |
|
193 | results are forgotten. | |
|
194 | ||
|
195 | Message type: ``purge_request``:: | |
|
196 | ||
|
197 | content = { | |
|
198 | 'msg_ids' : ['id1', 'id2',...], # list of msg_ids or 'all' | |
|
199 | 'engine_ids' : [0,2,4] # list of engine IDs | |
|
200 | } | |
|
201 | ||
|
202 | The reply to a purge request is simply the status 'ok' if the request succeeded, or an | |
|
203 | explanation of why it failed, such as requesting the purge of a nonexistent or pending | |
|
204 | message. | |
|
205 | ||
|
206 | Message type: ``purge_reply``:: | |
|
207 | ||
|
208 | content = { | |
|
209 | 'status' : 'ok', # or 'error' | |
|
210 | } | |
|
211 | ||
|
212 | ||
|
213 | Schedulers | |
|
214 | ---------- | |
|
215 | ||
|
216 | There are three basic schedulers: | |
|
217 | ||
|
218 | * Task Scheduler | |
|
219 | * MUX Scheduler | |
|
220 | * Control Scheduler | |
|
221 | ||
|
222 | The MUX and Control schedulers are simple MonitoredQueue ØMQ devices, with ``ROUTER`` | |
|
223 | sockets on either side. This allows the queue to relay individual messages to particular | |
|
224 | targets via ``zmq.IDENTITY`` routing. The Task scheduler may be a MonitoredQueue ØMQ | |
|
225 | device, in which case the client-facing socket is ``ROUTER``, and the engine-facing socket | |
|
226 | is ``DEALER``. The result of this is that client-submitted messages are load-balanced via | |
|
227 | the ``DEALER`` socket, but the engine's replies to each message go to the requesting client. | |
|
228 | ||
|
229 | Raw ``DEALER`` scheduling is quite primitive, and doesn't allow message introspection, so | |
|
230 | there are also Python Schedulers that can be used. These Schedulers behave in much the | |
|
231 | same way as a MonitoredQueue does from the outside, but have rich internal logic to | |
|
232 | determine destinations, as well as handle dependency graphs Their sockets are always | |
|
233 | ``ROUTER`` on both sides. | |
|
234 | ||
|
235 | The Python task schedulers have an additional message type, which informs the Hub of | |
|
236 | the destination of a task as soon as that destination is known. | |
|
237 | ||
|
238 | Message type: ``task_destination``:: | |
|
239 | ||
|
240 | content = { | |
|
241 | 'msg_id' : 'abcd-1234-...', # the msg's uuid | |
|
242 | 'engine_id' : '1234-abcd-...', # the destination engine's zmq.IDENTITY | |
|
243 | } | |
|
244 | ||
|
245 | :func:`apply` | |
|
246 | ************* | |
|
247 | ||
|
248 | In terms of message classes, the MUX scheduler and Task scheduler relay the exact same | |
|
249 | message types. Their only difference lies in how the destination is selected. | |
|
250 | ||
|
251 | The `Namespace <http://gist.github.com/483294>`_ model suggests that execution be able to | |
|
252 | use the model:: | |
|
253 | ||
|
254 | ns.apply(f, *args, **kwargs) | |
|
255 | ||
|
256 | which takes `f`, a function in the user's namespace, and executes ``f(*args, **kwargs)`` | |
|
257 | on a remote engine, returning the result (or, for non-blocking, information facilitating | |
|
258 | later retrieval of the result). This model, unlike the execute message which just uses a | |
|
259 | code string, must be able to send arbitrary (pickleable) Python objects. And ideally, copy | |
|
260 | as little data as we can. The `buffers` property of a Message was introduced for this | |
|
261 | purpose. | |
|
262 | ||
|
263 | Utility method :func:`build_apply_message` in :mod:`IPython.kernel.zmq.serialize` wraps a | |
|
264 | function signature and builds a sendable buffer format for minimal data copying (exactly | |
|
265 | zero copies of numpy array data or buffers or large strings). | |
|
266 | ||
|
267 | Message type: ``apply_request``:: | |
|
268 | ||
|
269 | metadata = { | |
|
270 | 'after' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict() | |
|
271 | 'follow' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict() | |
|
272 | } | |
|
273 | content = {} | |
|
274 | buffers = ['...'] # at least 3 in length | |
|
275 | # as built by build_apply_message(f,args,kwargs) | |
|
276 | ||
|
277 | after/follow represent task dependencies. 'after' corresponds to a time dependency. The | |
|
278 | request will not arrive at an engine until the 'after' dependency tasks have completed. | |
|
279 | 'follow' corresponds to a location dependency. The task will be submitted to the same | |
|
280 | engine as these msg_ids (see :class:`Dependency` docs for details). | |
|
281 | ||
|
282 | Message type: ``apply_reply``:: | |
|
283 | ||
|
284 | content = { | |
|
285 | 'status' : 'ok' # 'ok' or 'error' | |
|
286 | # other error info here, as in other messages | |
|
287 | } | |
|
288 | buffers = ['...'] # either 1 or 2 in length | |
|
289 | # a serialization of the return value of f(*args,**kwargs) | |
|
290 | # only populated if status is 'ok' | |
|
291 | ||
|
292 | All engine execution and data movement is performed via apply messages. | |
|
293 | ||
|
294 | Control Messages | |
|
295 | ---------------- | |
|
296 | ||
|
297 | Messages that interact with the engines, but are not meant to execute code, are submitted | |
|
298 | via the Control queue. These messages have high priority, and are thus received and | |
|
299 | handled before any execution requests. | |
|
300 | ||
|
301 | Clients may want to clear the namespace on the engine. There are no arguments nor | |
|
302 | information involved in this request, so the content is empty. | |
|
303 | ||
|
304 | Message type: ``clear_request``:: | |
|
305 | ||
|
306 | content = {} | |
|
307 | ||
|
308 | Message type: ``clear_reply``:: | |
|
309 | ||
|
310 | content = { | |
|
311 | 'status' : 'ok' # 'ok' or 'error' | |
|
312 | # other error info here, as in other messages | |
|
313 | } | |
|
314 | ||
|
315 | Clients may want to abort tasks that have not yet run. This can by done by message id, or | |
|
316 | all enqueued messages can be aborted if None is specified. | |
|
317 | ||
|
318 | Message type: ``abort_request``:: | |
|
319 | ||
|
320 | content = { | |
|
321 | 'msg_ids' : ['1234-...', '...'] # list of msg_ids or None | |
|
322 | } | |
|
323 | ||
|
324 | Message type: ``abort_reply``:: | |
|
325 | ||
|
326 | content = { | |
|
327 | 'status' : 'ok' # 'ok' or 'error' | |
|
328 | # other error info here, as in other messages | |
|
329 | } | |
|
330 | ||
|
331 | The last action a client may want to do is shutdown the kernel. If a kernel receives a | |
|
332 | shutdown request, then it aborts all queued messages, replies to the request, and exits. | |
|
333 | ||
|
334 | Message type: ``shutdown_request``:: | |
|
335 | ||
|
336 | content = {} | |
|
337 | ||
|
338 | Message type: ``shutdown_reply``:: | |
|
339 | ||
|
340 | content = { | |
|
341 | 'status' : 'ok' # 'ok' or 'error' | |
|
342 | # other error info here, as in other messages | |
|
343 | } | |
|
344 | ||
|
345 | ||
|
346 | Implementation | |
|
347 | -------------- | |
|
348 | ||
|
349 | There are a few differences in implementation between the `StreamSession` object used in | |
|
350 | the newparallel branch and the `Session` object, the main one being that messages are | |
|
351 | sent in parts, rather than as a single serialized object. `StreamSession` objects also | |
|
352 | take pack/unpack functions, which are to be used when serializing/deserializing objects. | |
|
353 | These can be any functions that translate to/from formats that ZMQ sockets can send | |
|
354 | (buffers,bytes, etc.). | |
|
355 | ||
|
356 | Split Sends | |
|
357 | *********** | |
|
358 | ||
|
359 | Previously, messages were bundled as a single json object and one call to | |
|
360 | :func:`socket.send_json`. Since the hub inspects all messages, and doesn't need to | |
|
361 | see the content of the messages, which can be large, messages are now serialized and sent in | |
|
362 | pieces. All messages are sent in at least 4 parts: the header, the parent header, the metadata and the content. | |
|
363 | This allows the controller to unpack and inspect the (always small) header, | |
|
364 | without spending time unpacking the content unless the message is bound for the | |
|
365 | controller. Buffers are added on to the end of the message, and can be any objects that | |
|
366 | present the buffer interface. | |
|
367 | ||
|
7 | IPython parallel has moved to ipyparallel - | |
|
8 | see :ref:`ipyparallel:parallel_messages` for the documentation. |
@@ -18,7 +18,7 b' such as bash.' | |||
|
18 | 18 | Required steps |
|
19 | 19 | -------------- |
|
20 | 20 | |
|
21 |
Subclass :class:` |
|
|
21 | Subclass :class:`ipykernel.kernelbase.Kernel`, and implement the | |
|
22 | 22 | following methods and attributes: |
|
23 | 23 | |
|
24 | 24 | .. class:: MyKernel |
@@ -61,13 +61,13 b' following methods and attributes:' | |||
|
61 | 61 | |
|
62 | 62 | Your method should return a dict containing the fields described in |
|
63 | 63 | :ref:`execution_results`. To display output, it can send messages |
|
64 |
using :meth:`~ |
|
|
64 | using :meth:`~ipykernel.kernelbase.Kernel.send_response`. | |
|
65 | 65 | See :doc:`messaging` for details of the different message types. |
|
66 | 66 | |
|
67 | 67 | To launch your kernel, add this at the end of your module:: |
|
68 | 68 | |
|
69 | 69 | if __name__ == '__main__': |
|
70 |
from |
|
|
70 | from ipykernel.kernelapp import IPKernelApp | |
|
71 | 71 | IPKernelApp.launch_instance(kernel_class=MyKernel) |
|
72 | 72 | |
|
73 | 73 | Example |
@@ -75,7 +75,7 b' Example' | |||
|
75 | 75 | |
|
76 | 76 | ``echokernel.py`` will simply echo any input it's given to stdout:: |
|
77 | 77 | |
|
78 |
from |
|
|
78 | from ipykernel.kernelbase import Kernel | |
|
79 | 79 | |
|
80 | 80 | class EchoKernel(Kernel): |
|
81 | 81 | implementation = 'Echo' |
@@ -99,7 +99,7 b' Example' | |||
|
99 | 99 | } |
|
100 | 100 | |
|
101 | 101 | if __name__ == '__main__': |
|
102 |
from |
|
|
102 | from ipykernel.kernelapp import IPKernelApp | |
|
103 | 103 | IPKernelApp.launch_instance(kernel_class=EchoKernel) |
|
104 | 104 | |
|
105 | 105 | Here's the Kernel spec ``kernel.json`` file for this:: |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed | |
This diff has been collapsed as it changes many lines, (4012 lines changed) Show them Hide them |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed | |
This diff has been collapsed as it changes many lines, (596 lines changed) Show them Hide them |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed, binary diff hidden |
|
1 | NO CONTENT: file was removed, binary diff hidden |
General Comments 0
You need to be logged in to leave comments.
Login now