Show More
@@ -1,159 +1,154 b'' | |||||
1 | .. _parallel_connections: |
|
1 | .. _parallel_connections: | |
2 |
|
2 | |||
3 | ============================================== |
|
3 | ============================================== | |
4 | Connection Diagrams of The IPython ZMQ Cluster |
|
4 | Connection Diagrams of The IPython ZMQ Cluster | |
5 | ============================================== |
|
5 | ============================================== | |
6 |
|
6 | |||
7 | This is a quick summary and illustration of the connections involved in the ZeroMQ based |
|
7 | This is a quick summary and illustration of the connections involved in the ZeroMQ based | |
8 | IPython cluster for parallel computing. |
|
8 | IPython cluster for parallel computing. | |
9 |
|
9 | |||
10 | All Connections |
|
10 | All Connections | |
11 | =============== |
|
11 | =============== | |
12 |
|
12 | |||
13 | The Parallel Computing code is currently under development in IPython's newparallel_ |
|
|||
14 | branch on GitHub. |
|
|||
15 |
|
||||
16 | .. _newparallel: http://github.com/ipython/ipython/tree/newparallel |
|
|||
17 |
|
||||
18 | The IPython cluster consists of a Controller, and one or more each of clients and engines. |
|
13 | The IPython cluster consists of a Controller, and one or more each of clients and engines. | |
19 | The goal of the Controller is to manage and monitor the connections and communications |
|
14 | The goal of the Controller is to manage and monitor the connections and communications | |
20 | between the clients and the engines. The Controller is no longer a single process entity, |
|
15 | between the clients and the engines. The Controller is no longer a single process entity, | |
21 |
but rather a collection of processes - specifically one Hub, and |
|
16 | but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers. | |
22 |
|
17 | |||
23 | It is important for security/practicality reasons that all connections be inbound to the |
|
18 | It is important for security/practicality reasons that all connections be inbound to the | |
24 | controller processes. The arrows in the figures indicate the direction of the |
|
19 | controller processes. The arrows in the figures indicate the direction of the | |
25 | connection. |
|
20 | connection. | |
26 |
|
21 | |||
27 |
|
22 | |||
28 | .. figure:: figs/allconnections.png |
|
23 | .. figure:: figs/allconnections.png | |
29 | :width: 432px |
|
24 | :width: 432px | |
30 | :alt: IPython cluster connections |
|
25 | :alt: IPython cluster connections | |
31 | :align: center |
|
26 | :align: center | |
32 |
|
27 | |||
33 | All the connections involved in connecting one client to one engine. |
|
28 | All the connections involved in connecting one client to one engine. | |
34 |
|
29 | |||
35 | The Controller consists of 1-4 processes. Central to the cluster is the **Hub**, which monitors |
|
30 | The Controller consists of 1-4 processes. Central to the cluster is the **Hub**, which monitors | |
36 | engine state, execution traffic, and handles registration and notification. The Hub includes a |
|
31 | engine state, execution traffic, and handles registration and notification. The Hub includes a | |
37 | Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4 |
|
32 | Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4 | |
38 | **Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally |
|
33 | **Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally | |
39 | threads) that relay messages very fast, but also send a copy of each message along a side socket |
|
34 | threads) that relay messages very fast, but also send a copy of each message along a side socket | |
40 | to the Hub. The MUX queue and Control queue are MonitoredQueue ΓMQ devices which relay |
|
35 | to the Hub. The MUX queue and Control queue are MonitoredQueue ΓMQ devices which relay | |
41 | explicitly addressed messages from clients to engines, and their replies back up. The Balanced |
|
36 | explicitly addressed messages from clients to engines, and their replies back up. The Balanced | |
42 | queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue |
|
37 | queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue | |
43 | device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ |
|
38 | device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ | |
44 | devices, but with additional internal logic. stdout/err are also propagated from the Engines to |
|
39 | devices, but with additional internal logic. stdout/err are also propagated from the Engines to | |
45 | the clients via a PUB/SUB MonitoredQueue. |
|
40 | the clients via a PUB/SUB MonitoredQueue. | |
46 |
|
41 | |||
47 |
|
42 | |||
48 | Registration |
|
43 | Registration | |
49 | ------------ |
|
44 | ------------ | |
50 |
|
45 | |||
51 | .. figure:: figs/queryfade.png |
|
46 | .. figure:: figs/queryfade.png | |
52 | :width: 432px |
|
47 | :width: 432px | |
53 | :alt: IPython Registration connections |
|
48 | :alt: IPython Registration connections | |
54 | :align: center |
|
49 | :align: center | |
55 |
|
50 | |||
56 | Engines and Clients only need to know where the Query ``XREP`` is located to start |
|
51 | Engines and Clients only need to know where the Query ``XREP`` is located to start | |
57 | connecting. |
|
52 | connecting. | |
58 |
|
53 | |||
59 | Once a controller is launched, the only information needed for connecting clients and/or |
|
54 | Once a controller is launched, the only information needed for connecting clients and/or | |
60 | engines is the IP/port of the Hub's ``XREP`` socket called the Registrar. This socket |
|
55 | engines is the IP/port of the Hub's ``XREP`` socket called the Registrar. This socket | |
61 | handles connections from both clients and engines, and replies with the remaining |
|
56 | handles connections from both clients and engines, and replies with the remaining | |
62 | information necessary to establish the remaining connections. Clients use this same socket for |
|
57 | information necessary to establish the remaining connections. Clients use this same socket for | |
63 | querying the Hub for state information. |
|
58 | querying the Hub for state information. | |
64 |
|
59 | |||
65 | Heartbeat |
|
60 | Heartbeat | |
66 | --------- |
|
61 | --------- | |
67 |
|
62 | |||
68 | .. figure:: figs/hbfade.png |
|
63 | .. figure:: figs/hbfade.png | |
69 | :width: 432px |
|
64 | :width: 432px | |
70 | :alt: IPython Heartbeat connections |
|
65 | :alt: IPython Heartbeat connections | |
71 | :align: center |
|
66 | :align: center | |
72 |
|
67 | |||
73 | The heartbeat sockets. |
|
68 | The heartbeat sockets. | |
74 |
|
69 | |||
75 | The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor |
|
70 | The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor | |
76 | publishes a distinct message periodically via a ``PUB`` socket. Each engine has a |
|
71 | publishes a distinct message periodically via a ``PUB`` socket. Each engine has a | |
77 | ``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``XREQ`` socket for output. |
|
72 | ``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``XREQ`` socket for output. | |
78 | The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``XREQ`` is |
|
73 | The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``XREQ`` is | |
79 | connected to the ``XREP`` labeled *pong*. This results in the same message being relayed |
|
74 | connected to the ``XREP`` labeled *pong*. This results in the same message being relayed | |
80 | back to the Heartbeat Monitor with the addition of the ``XREQ`` prefix. The Heartbeat |
|
75 | back to the Heartbeat Monitor with the addition of the ``XREQ`` prefix. The Heartbeat | |
81 | Monitor receives all the replies via an ``XREP`` socket, and identifies which hearts are |
|
76 | Monitor receives all the replies via an ``XREP`` socket, and identifies which hearts are | |
82 | still beating by the ``zmq.IDENTITY`` prefix of the ``XREQ`` sockets, which information |
|
77 | still beating by the ``zmq.IDENTITY`` prefix of the ``XREQ`` sockets, which information | |
83 | the Hub uses to notify clients of any changes in the available engines. |
|
78 | the Hub uses to notify clients of any changes in the available engines. | |
84 |
|
79 | |||
85 | Schedulers |
|
80 | Schedulers | |
86 | ---------- |
|
81 | ---------- | |
87 |
|
82 | |||
88 | .. figure:: figs/queuefade.png |
|
83 | .. figure:: figs/queuefade.png | |
89 | :width: 432px |
|
84 | :width: 432px | |
90 | :alt: IPython Queue connections |
|
85 | :alt: IPython Queue connections | |
91 | :align: center |
|
86 | :align: center | |
92 |
|
87 | |||
93 | Control message scheduler on the left, execution (apply) schedulers on the right. |
|
88 | Control message scheduler on the left, execution (apply) schedulers on the right. | |
94 |
|
89 | |||
95 | The controller has at least three Schedulers. These devices are primarily for |
|
90 | The controller has at least three Schedulers. These devices are primarily for | |
96 | relaying messages between clients and engines, but the Hub needs to see those |
|
91 | relaying messages between clients and engines, but the Hub needs to see those | |
97 | messages for its own purposes. Since no Python code may exist between the two sockets in a |
|
92 | messages for its own purposes. Since no Python code may exist between the two sockets in a | |
98 | queue, all messages sent through these queues (both directions) are also sent via a |
|
93 | queue, all messages sent through these queues (both directions) are also sent via a | |
99 | ``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without |
|
94 | ``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without | |
100 | interfering with it. |
|
95 | interfering with it. | |
101 |
|
96 | |||
102 | For tasks, the engine need not be specified. Messages sent to the ``XREP`` socket from the |
|
97 | For tasks, the engine need not be specified. Messages sent to the ``XREP`` socket from the | |
103 | client side are assigned to an engine via ZMQ's ``XREQ`` round-robin load balancing. |
|
98 | client side are assigned to an engine via ZMQ's ``XREQ`` round-robin load balancing. | |
104 | Engine replies are directed to specific clients via the IDENTITY of the client, which is |
|
99 | Engine replies are directed to specific clients via the IDENTITY of the client, which is | |
105 | received as a prefix at the Engine. |
|
100 | received as a prefix at the Engine. | |
106 |
|
101 | |||
107 | For Multiplexing, ``XREP`` is used for both in and output sockets in the device. Clients must |
|
102 | For Multiplexing, ``XREP`` is used for both in and output sockets in the device. Clients must | |
108 | specify the destination by the ``zmq.IDENTITY`` of the ``XREP`` socket connected to |
|
103 | specify the destination by the ``zmq.IDENTITY`` of the ``XREP`` socket connected to | |
109 | the downstream end of the device. |
|
104 | the downstream end of the device. | |
110 |
|
105 | |||
111 | At the Kernel level, both of these ``XREP`` sockets are treated in the same way as the ``REP`` |
|
106 | At the Kernel level, both of these ``XREP`` sockets are treated in the same way as the ``REP`` | |
112 | socket in the serial version (except using ZMQStreams instead of explicit sockets). |
|
107 | socket in the serial version (except using ZMQStreams instead of explicit sockets). | |
113 |
|
108 | |||
114 | Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified) |
|
109 | Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified) | |
115 | manner. The sockets on the Client and Engine are the same for these two actions, but the |
|
110 | manner. The sockets on the Client and Engine are the same for these two actions, but the | |
116 | scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of |
|
111 | scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of | |
117 | the upstream sockets in each MonitoredQueue. |
|
112 | the upstream sockets in each MonitoredQueue. | |
118 |
|
113 | |||
119 | IOPub |
|
114 | IOPub | |
120 | ----- |
|
115 | ----- | |
121 |
|
116 | |||
122 | .. figure:: figs/iopubfade.png |
|
117 | .. figure:: figs/iopubfade.png | |
123 | :width: 432px |
|
118 | :width: 432px | |
124 | :alt: IOPub connections |
|
119 | :alt: IOPub connections | |
125 | :align: center |
|
120 | :align: center | |
126 |
|
121 | |||
127 | stdout/err are published via a ``PUB/SUB`` MonitoredQueue |
|
122 | stdout/err are published via a ``PUB/SUB`` MonitoredQueue | |
128 |
|
123 | |||
129 |
|
124 | |||
130 | On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB`` |
|
125 | On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB`` | |
131 | sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all |
|
126 | sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all | |
132 | messages. They are then republished via another ``PUB`` socket, which can be |
|
127 | messages. They are then republished via another ``PUB`` socket, which can be | |
133 | subscribed by the clients. |
|
128 | subscribed by the clients. | |
134 |
|
129 | |||
135 | Client connections |
|
130 | Client connections | |
136 | ------------------ |
|
131 | ------------------ | |
137 |
|
132 | |||
138 | .. figure:: figs/queryfade.png |
|
133 | .. figure:: figs/queryfade.png | |
139 | :width: 432px |
|
134 | :width: 432px | |
140 | :alt: IPython client query connections |
|
135 | :alt: IPython client query connections | |
141 | :align: center |
|
136 | :align: center | |
142 |
|
137 | |||
143 | Clients connect to an ``XREP`` socket to query the hub. |
|
138 | Clients connect to an ``XREP`` socket to query the hub. | |
144 |
|
139 | |||
145 | The hub's registrar ``XREP`` socket also listens for queries from clients as to queue status, |
|
140 | The hub's registrar ``XREP`` socket also listens for queries from clients as to queue status, | |
146 | and control instructions. Clients connect to this socket via an ``XREQ`` during registration. |
|
141 | and control instructions. Clients connect to this socket via an ``XREQ`` during registration. | |
147 |
|
142 | |||
148 | .. figure:: figs/notiffade.png |
|
143 | .. figure:: figs/notiffade.png | |
149 | :width: 432px |
|
144 | :width: 432px | |
150 | :alt: IPython Registration connections |
|
145 | :alt: IPython Registration connections | |
151 | :align: center |
|
146 | :align: center | |
152 |
|
147 | |||
153 | Engine registration events are published via a ``PUB`` socket. |
|
148 | Engine registration events are published via a ``PUB`` socket. | |
154 |
|
149 | |||
155 | The Hub publishes all registration/unregistration events via a ``PUB`` socket. This |
|
150 | The Hub publishes all registration/unregistration events via a ``PUB`` socket. This | |
156 | allows clients to stay up to date with what engines are available by subscribing to the |
|
151 | allows clients to stay up to date with what engines are available by subscribing to the | |
157 | feed with a ``SUB`` socket. Other processes could selectively subscribe to just |
|
152 | feed with a ``SUB`` socket. Other processes could selectively subscribe to just | |
158 | registration or unregistration events. |
|
153 | registration or unregistration events. | |
159 |
|
154 |
General Comments 0
You need to be logged in to leave comments.
Login now