##// END OF EJS Templates
upstream » ipython
RSS

Clone from https://github.com/ipython/ipython.git

remove link to newparallel branch in connections doc
MinRK -
Show More
Show file before | Show file after | Hide comments
@@ -1,159 +1,154 b''
1 .. _parallel_connections:
1 .. _parallel_connections:
2
2
3 ==============================================
3 ==============================================
4 Connection Diagrams of The IPython ZMQ Cluster
4 Connection Diagrams of The IPython ZMQ Cluster
5 ==============================================
5 ==============================================
6
6
7 This is a quick summary and illustration of the connections involved in the ZeroMQ based
7 This is a quick summary and illustration of the connections involved in the ZeroMQ based
8 IPython cluster for parallel computing.
8 IPython cluster for parallel computing.
9
9
10 All Connections
10 All Connections
11 ===============
11 ===============
12
12
13 The Parallel Computing code is currently under development in IPython's newparallel_
14 branch on GitHub.
15
16 .. _newparallel: http://github.com/ipython/ipython/tree/newparallel
17
18 The IPython cluster consists of a Controller, and one or more each of clients and engines.
13 The IPython cluster consists of a Controller, and one or more each of clients and engines.
19 The goal of the Controller is to manage and monitor the connections and communications
14 The goal of the Controller is to manage and monitor the connections and communications
20 between the clients and the engines. The Controller is no longer a single process entity,
15 between the clients and the engines. The Controller is no longer a single process entity,
21 but rather a collection of processes - specifically one Hub, and 3 (or more) Schedulers.
16 but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers.
22
17
23 It is important for security/practicality reasons that all connections be inbound to the
18 It is important for security/practicality reasons that all connections be inbound to the
24 controller processes. The arrows in the figures indicate the direction of the
19 controller processes. The arrows in the figures indicate the direction of the
25 connection.
20 connection.
26
21
27
22
28 .. figure:: figs/allconnections.png
23 .. figure:: figs/allconnections.png
29 :width: 432px
24 :width: 432px
30 :alt: IPython cluster connections
25 :alt: IPython cluster connections
31 :align: center
26 :align: center
32
27
33 All the connections involved in connecting one client to one engine.
28 All the connections involved in connecting one client to one engine.
34
29
35 The Controller consists of 1-4 processes. Central to the cluster is the **Hub**, which monitors
30 The Controller consists of 1-4 processes. Central to the cluster is the **Hub**, which monitors
36 engine state, execution traffic, and handles registration and notification. The Hub includes a
31 engine state, execution traffic, and handles registration and notification. The Hub includes a
37 Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
32 Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
38 **Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally
33 **Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally
39 threads) that relay messages very fast, but also send a copy of each message along a side socket
34 threads) that relay messages very fast, but also send a copy of each message along a side socket
40 to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
35 to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
41 explicitly addressed messages from clients to engines, and their replies back up. The Balanced
36 explicitly addressed messages from clients to engines, and their replies back up. The Balanced
42 queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
37 queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
43 device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
38 device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
44 devices, but with additional internal logic. stdout/err are also propagated from the Engines to
39 devices, but with additional internal logic. stdout/err are also propagated from the Engines to
45 the clients via a PUB/SUB MonitoredQueue.
40 the clients via a PUB/SUB MonitoredQueue.
46
41
47
42
48 Registration
43 Registration
49 ------------
44 ------------
50
45
51 .. figure:: figs/queryfade.png
46 .. figure:: figs/queryfade.png
52 :width: 432px
47 :width: 432px
53 :alt: IPython Registration connections
48 :alt: IPython Registration connections
54 :align: center
49 :align: center
55
50
56 Engines and Clients only need to know where the Query ``XREP`` is located to start
51 Engines and Clients only need to know where the Query ``XREP`` is located to start
57 connecting.
52 connecting.
58
53
59 Once a controller is launched, the only information needed for connecting clients and/or
54 Once a controller is launched, the only information needed for connecting clients and/or
60 engines is the IP/port of the Hub's ``XREP`` socket called the Registrar. This socket
55 engines is the IP/port of the Hub's ``XREP`` socket called the Registrar. This socket
61 handles connections from both clients and engines, and replies with the remaining
56 handles connections from both clients and engines, and replies with the remaining
62 information necessary to establish the remaining connections. Clients use this same socket for
57 information necessary to establish the remaining connections. Clients use this same socket for
63 querying the Hub for state information.
58 querying the Hub for state information.
64
59
65 Heartbeat
60 Heartbeat
66 ---------
61 ---------
67
62
68 .. figure:: figs/hbfade.png
63 .. figure:: figs/hbfade.png
69 :width: 432px
64 :width: 432px
70 :alt: IPython Heartbeat connections
65 :alt: IPython Heartbeat connections
71 :align: center
66 :align: center
72
67
73 The heartbeat sockets.
68 The heartbeat sockets.
74
69
75 The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
70 The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
76 publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
71 publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
77 ``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``XREQ`` socket for output.
72 ``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``XREQ`` socket for output.
78 The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``XREQ`` is
73 The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``XREQ`` is
79 connected to the ``XREP`` labeled *pong*. This results in the same message being relayed
74 connected to the ``XREP`` labeled *pong*. This results in the same message being relayed
80 back to the Heartbeat Monitor with the addition of the ``XREQ`` prefix. The Heartbeat
75 back to the Heartbeat Monitor with the addition of the ``XREQ`` prefix. The Heartbeat
81 Monitor receives all the replies via an ``XREP`` socket, and identifies which hearts are
76 Monitor receives all the replies via an ``XREP`` socket, and identifies which hearts are
82 still beating by the ``zmq.IDENTITY`` prefix of the ``XREQ`` sockets, which information
77 still beating by the ``zmq.IDENTITY`` prefix of the ``XREQ`` sockets, which information
83 the Hub uses to notify clients of any changes in the available engines.
78 the Hub uses to notify clients of any changes in the available engines.
84
79
85 Schedulers
80 Schedulers
86 ----------
81 ----------
87
82
88 .. figure:: figs/queuefade.png
83 .. figure:: figs/queuefade.png
89 :width: 432px
84 :width: 432px
90 :alt: IPython Queue connections
85 :alt: IPython Queue connections
91 :align: center
86 :align: center
92
87
93 Control message scheduler on the left, execution (apply) schedulers on the right.
88 Control message scheduler on the left, execution (apply) schedulers on the right.
94
89
95 The controller has at least three Schedulers. These devices are primarily for
90 The controller has at least three Schedulers. These devices are primarily for
96 relaying messages between clients and engines, but the Hub needs to see those
91 relaying messages between clients and engines, but the Hub needs to see those
97 messages for its own purposes. Since no Python code may exist between the two sockets in a
92 messages for its own purposes. Since no Python code may exist between the two sockets in a
98 queue, all messages sent through these queues (both directions) are also sent via a
93 queue, all messages sent through these queues (both directions) are also sent via a
99 ``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
94 ``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
100 interfering with it.
95 interfering with it.
101
96
102 For tasks, the engine need not be specified. Messages sent to the ``XREP`` socket from the
97 For tasks, the engine need not be specified. Messages sent to the ``XREP`` socket from the
103 client side are assigned to an engine via ZMQ's ``XREQ`` round-robin load balancing.
98 client side are assigned to an engine via ZMQ's ``XREQ`` round-robin load balancing.
104 Engine replies are directed to specific clients via the IDENTITY of the client, which is
99 Engine replies are directed to specific clients via the IDENTITY of the client, which is
105 received as a prefix at the Engine.
100 received as a prefix at the Engine.
106
101
107 For Multiplexing, ``XREP`` is used for both in and output sockets in the device. Clients must
102 For Multiplexing, ``XREP`` is used for both in and output sockets in the device. Clients must
108 specify the destination by the ``zmq.IDENTITY`` of the ``XREP`` socket connected to
103 specify the destination by the ``zmq.IDENTITY`` of the ``XREP`` socket connected to
109 the downstream end of the device.
104 the downstream end of the device.
110
105
111 At the Kernel level, both of these ``XREP`` sockets are treated in the same way as the ``REP``
106 At the Kernel level, both of these ``XREP`` sockets are treated in the same way as the ``REP``
112 socket in the serial version (except using ZMQStreams instead of explicit sockets).
107 socket in the serial version (except using ZMQStreams instead of explicit sockets).
113
108
114 Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
109 Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
115 manner. The sockets on the Client and Engine are the same for these two actions, but the
110 manner. The sockets on the Client and Engine are the same for these two actions, but the
116 scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
111 scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
117 the upstream sockets in each MonitoredQueue.
112 the upstream sockets in each MonitoredQueue.
118
113
119 IOPub
114 IOPub
120 -----
115 -----
121
116
122 .. figure:: figs/iopubfade.png
117 .. figure:: figs/iopubfade.png
123 :width: 432px
118 :width: 432px
124 :alt: IOPub connections
119 :alt: IOPub connections
125 :align: center
120 :align: center
126
121
127 stdout/err are published via a ``PUB/SUB`` MonitoredQueue
122 stdout/err are published via a ``PUB/SUB`` MonitoredQueue
128
123
129
124
130 On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
125 On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
131 sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
126 sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
132 messages. They are then republished via another ``PUB`` socket, which can be
127 messages. They are then republished via another ``PUB`` socket, which can be
133 subscribed by the clients.
128 subscribed by the clients.
134
129
135 Client connections
130 Client connections
136 ------------------
131 ------------------
137
132
138 .. figure:: figs/queryfade.png
133 .. figure:: figs/queryfade.png
139 :width: 432px
134 :width: 432px
140 :alt: IPython client query connections
135 :alt: IPython client query connections
141 :align: center
136 :align: center
142
137
143 Clients connect to an ``XREP`` socket to query the hub.
138 Clients connect to an ``XREP`` socket to query the hub.
144
139
145 The hub's registrar ``XREP`` socket also listens for queries from clients as to queue status,
140 The hub's registrar ``XREP`` socket also listens for queries from clients as to queue status,
146 and control instructions. Clients connect to this socket via an ``XREQ`` during registration.
141 and control instructions. Clients connect to this socket via an ``XREQ`` during registration.
147
142
148 .. figure:: figs/notiffade.png
143 .. figure:: figs/notiffade.png
149 :width: 432px
144 :width: 432px
150 :alt: IPython Registration connections
145 :alt: IPython Registration connections
151 :align: center
146 :align: center
152
147
153 Engine registration events are published via a ``PUB`` socket.
148 Engine registration events are published via a ``PUB`` socket.
154
149
155 The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
150 The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
156 allows clients to stay up to date with what engines are available by subscribing to the
151 allows clients to stay up to date with what engines are available by subscribing to the
157 feed with a ``SUB`` socket. Other processes could selectively subscribe to just
152 feed with a ``SUB`` socket. Other processes could selectively subscribe to just
158 registration or unregistration events.
153 registration or unregistration events.
159
154
General Comments 0
You need to be logged in to leave comments. Login now