|
|
@@
-1,154
+1,154
b''
|
|
|
1
|
.. _parallel_connections:
|
|
1
|
.. _parallel_connections:
|
|
|
2
|
|
|
2
|
|
|
|
3
|
==============================================
|
|
3
|
==============================================
|
|
|
4
|
Connection Diagrams of The IPython ZMQ Cluster
|
|
4
|
Connection Diagrams of The IPython ZMQ Cluster
|
|
|
5
|
==============================================
|
|
5
|
==============================================
|
|
|
6
|
|
|
6
|
|
|
|
7
|
This is a quick summary and illustration of the connections involved in the ZeroMQ based
|
|
7
|
This is a quick summary and illustration of the connections involved in the ZeroMQ based
|
|
|
8
|
IPython cluster for parallel computing.
|
|
8
|
IPython cluster for parallel computing.
|
|
|
9
|
|
|
9
|
|
|
|
10
|
All Connections
|
|
10
|
All Connections
|
|
|
11
|
===============
|
|
11
|
===============
|
|
|
12
|
|
|
12
|
|
|
|
13
|
The IPython cluster consists of a Controller, and one or more each of clients and engines.
|
|
13
|
The IPython cluster consists of a Controller, and one or more each of clients and engines.
|
|
|
14
|
The goal of the Controller is to manage and monitor the connections and communications
|
|
14
|
The goal of the Controller is to manage and monitor the connections and communications
|
|
|
15
|
between the clients and the engines. The Controller is no longer a single process entity,
|
|
15
|
between the clients and the engines. The Controller is no longer a single process entity,
|
|
|
16
|
but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers.
|
|
16
|
but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers.
|
|
|
17
|
|
|
17
|
|
|
|
18
|
It is important for security/practicality reasons that all connections be inbound to the
|
|
18
|
It is important for security/practicality reasons that all connections be inbound to the
|
|
|
19
|
controller processes. The arrows in the figures indicate the direction of the
|
|
19
|
controller processes. The arrows in the figures indicate the direction of the
|
|
|
20
|
connection.
|
|
20
|
connection.
|
|
|
21
|
|
|
21
|
|
|
|
22
|
|
|
22
|
|
|
|
23
|
.. figure:: figs/allconnections.png
|
|
23
|
.. figure:: figs/allconnections.png
|
|
|
24
|
:width: 432px
|
|
24
|
:width: 432px
|
|
|
25
|
:alt: IPython cluster connections
|
|
25
|
:alt: IPython cluster connections
|
|
|
26
|
:align: center
|
|
26
|
:align: center
|
|
|
27
|
|
|
27
|
|
|
|
28
|
All the connections involved in connecting one client to one engine.
|
|
28
|
All the connections involved in connecting one client to one engine.
|
|
|
29
|
|
|
29
|
|
|
|
30
|
The Controller consists of 1-4 processes. Central to the cluster is the **Hub**, which monitors
|
|
30
|
The Controller consists of 1-5 processes. Central to the cluster is the **Hub**, which monitors
|
|
|
31
|
engine state, execution traffic, and handles registration and notification. The Hub includes a
|
|
31
|
engine state, execution traffic, and handles registration and notification. The Hub includes a
|
|
|
32
|
Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
|
|
32
|
Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
|
|
|
33
|
**Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally
|
|
33
|
**Schedulers**. These devices are very small pure-C MonitoredQueue processes (or optionally
|
|
|
34
|
threads) that relay messages very fast, but also send a copy of each message along a side socket
|
|
34
|
threads) that relay messages very fast, but also send a copy of each message along a side socket
|
|
|
35
|
to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
|
|
35
|
to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
|
|
|
36
|
explicitly addressed messages from clients to engines, and their replies back up. The Balanced
|
|
36
|
explicitly addressed messages from clients to engines, and their replies back up. The Balanced
|
|
|
37
|
queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
|
|
37
|
queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
|
|
|
38
|
device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
|
|
38
|
device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
|
|
|
39
|
devices, but with additional internal logic. stdout/err are also propagated from the Engines to
|
|
39
|
devices, but with additional internal logic. stdout/err are also propagated from the Engines to
|
|
|
40
|
the clients via a PUB/SUB MonitoredQueue.
|
|
40
|
the clients via a PUB/SUB MonitoredQueue.
|
|
|
41
|
|
|
41
|
|
|
|
42
|
|
|
42
|
|
|
|
43
|
Registration
|
|
43
|
Registration
|
|
|
44
|
------------
|
|
44
|
------------
|
|
|
45
|
|
|
45
|
|
|
|
46
|
.. figure:: figs/queryfade.png
|
|
46
|
.. figure:: figs/queryfade.png
|
|
|
47
|
:width: 432px
|
|
47
|
:width: 432px
|
|
|
48
|
:alt: IPython Registration connections
|
|
48
|
:alt: IPython Registration connections
|
|
|
49
|
:align: center
|
|
49
|
:align: center
|
|
|
50
|
|
|
50
|
|
|
|
51
|
Engines and Clients only need to know where the Query ``ROUTER`` is located to start
|
|
51
|
Engines and Clients only need to know where the Query ``ROUTER`` is located to start
|
|
|
52
|
connecting.
|
|
52
|
connecting.
|
|
|
53
|
|
|
53
|
|
|
|
54
|
Once a controller is launched, the only information needed for connecting clients and/or
|
|
54
|
Once a controller is launched, the only information needed for connecting clients and/or
|
|
|
55
|
engines is the IP/port of the Hub's ``ROUTER`` socket called the Registrar. This socket
|
|
55
|
engines is the IP/port of the Hub's ``ROUTER`` socket called the Registrar. This socket
|
|
|
56
|
handles connections from both clients and engines, and replies with the remaining
|
|
56
|
handles connections from both clients and engines, and replies with the remaining
|
|
|
57
|
information necessary to establish the remaining connections. Clients use this same socket for
|
|
57
|
information necessary to establish the remaining connections. Clients use this same socket for
|
|
|
58
|
querying the Hub for state information.
|
|
58
|
querying the Hub for state information.
|
|
|
59
|
|
|
59
|
|
|
|
60
|
Heartbeat
|
|
60
|
Heartbeat
|
|
|
61
|
---------
|
|
61
|
---------
|
|
|
62
|
|
|
62
|
|
|
|
63
|
.. figure:: figs/hbfade.png
|
|
63
|
.. figure:: figs/hbfade.png
|
|
|
64
|
:width: 432px
|
|
64
|
:width: 432px
|
|
|
65
|
:alt: IPython Heartbeat connections
|
|
65
|
:alt: IPython Heartbeat connections
|
|
|
66
|
:align: center
|
|
66
|
:align: center
|
|
|
67
|
|
|
67
|
|
|
|
68
|
The heartbeat sockets.
|
|
68
|
The heartbeat sockets.
|
|
|
69
|
|
|
69
|
|
|
|
70
|
The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
|
|
70
|
The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
|
|
|
71
|
publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
|
|
71
|
publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
|
|
|
72
|
``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``DEALER`` socket for output.
|
|
72
|
``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``DEALER`` socket for output.
|
|
|
73
|
The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``DEALER`` is
|
|
73
|
The ``SUB`` socket is connected to the ``PUB`` socket labeled *ping*, and the ``DEALER`` is
|
|
|
74
|
connected to the ``ROUTER`` labeled *pong*. This results in the same message being relayed
|
|
74
|
connected to the ``ROUTER`` labeled *pong*. This results in the same message being relayed
|
|
|
75
|
back to the Heartbeat Monitor with the addition of the ``DEALER`` prefix. The Heartbeat
|
|
75
|
back to the Heartbeat Monitor with the addition of the ``DEALER`` prefix. The Heartbeat
|
|
|
76
|
Monitor receives all the replies via an ``ROUTER`` socket, and identifies which hearts are
|
|
76
|
Monitor receives all the replies via an ``ROUTER`` socket, and identifies which hearts are
|
|
|
77
|
still beating by the ``zmq.IDENTITY`` prefix of the ``DEALER`` sockets, which information
|
|
77
|
still beating by the ``zmq.IDENTITY`` prefix of the ``DEALER`` sockets, which information
|
|
|
78
|
the Hub uses to notify clients of any changes in the available engines.
|
|
78
|
the Hub uses to notify clients of any changes in the available engines.
|
|
|
79
|
|
|
79
|
|
|
|
80
|
Schedulers
|
|
80
|
Schedulers
|
|
|
81
|
----------
|
|
81
|
----------
|
|
|
82
|
|
|
82
|
|
|
|
83
|
.. figure:: figs/queuefade.png
|
|
83
|
.. figure:: figs/queuefade.png
|
|
|
84
|
:width: 432px
|
|
84
|
:width: 432px
|
|
|
85
|
:alt: IPython Queue connections
|
|
85
|
:alt: IPython Queue connections
|
|
|
86
|
:align: center
|
|
86
|
:align: center
|
|
|
87
|
|
|
87
|
|
|
|
88
|
Control message scheduler on the left, execution (apply) schedulers on the right.
|
|
88
|
Control message scheduler on the left, execution (apply) schedulers on the right.
|
|
|
89
|
|
|
89
|
|
|
|
90
|
The controller has at least three Schedulers. These devices are primarily for
|
|
90
|
The controller has at least three Schedulers. These devices are primarily for
|
|
|
91
|
relaying messages between clients and engines, but the Hub needs to see those
|
|
91
|
relaying messages between clients and engines, but the Hub needs to see those
|
|
|
92
|
messages for its own purposes. Since no Python code may exist between the two sockets in a
|
|
92
|
messages for its own purposes. Since no Python code may exist between the two sockets in a
|
|
|
93
|
queue, all messages sent through these queues (both directions) are also sent via a
|
|
93
|
queue, all messages sent through these queues (both directions) are also sent via a
|
|
|
94
|
``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
|
|
94
|
``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
|
|
|
95
|
interfering with it.
|
|
95
|
interfering with it.
|
|
|
96
|
|
|
96
|
|
|
|
97
|
For tasks, the engine need not be specified. Messages sent to the ``ROUTER`` socket from the
|
|
97
|
For tasks, the engine need not be specified. Messages sent to the ``ROUTER`` socket from the
|
|
|
98
|
client side are assigned to an engine via ZMQ's ``DEALER`` round-robin load balancing.
|
|
98
|
client side are assigned to an engine via ZMQ's ``DEALER`` round-robin load balancing.
|
|
|
99
|
Engine replies are directed to specific clients via the IDENTITY of the client, which is
|
|
99
|
Engine replies are directed to specific clients via the IDENTITY of the client, which is
|
|
|
100
|
received as a prefix at the Engine.
|
|
100
|
received as a prefix at the Engine.
|
|
|
101
|
|
|
101
|
|
|
|
102
|
For Multiplexing, ``ROUTER`` is used for both in and output sockets in the device. Clients must
|
|
102
|
For Multiplexing, ``ROUTER`` is used for both in and output sockets in the device. Clients must
|
|
|
103
|
specify the destination by the ``zmq.IDENTITY`` of the ``ROUTER`` socket connected to
|
|
103
|
specify the destination by the ``zmq.IDENTITY`` of the ``ROUTER`` socket connected to
|
|
|
104
|
the downstream end of the device.
|
|
104
|
the downstream end of the device.
|
|
|
105
|
|
|
105
|
|
|
|
106
|
At the Kernel level, both of these ``ROUTER`` sockets are treated in the same way as the ``REP``
|
|
106
|
At the Kernel level, both of these ``ROUTER`` sockets are treated in the same way as the ``REP``
|
|
|
107
|
socket in the serial version (except using ZMQStreams instead of explicit sockets).
|
|
107
|
socket in the serial version (except using ZMQStreams instead of explicit sockets).
|
|
|
108
|
|
|
108
|
|
|
|
109
|
Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
|
|
109
|
Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
|
|
|
110
|
manner. The sockets on the Client and Engine are the same for these two actions, but the
|
|
110
|
manner. The sockets on the Client and Engine are the same for these two actions, but the
|
|
|
111
|
scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
|
|
111
|
scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
|
|
|
112
|
the upstream sockets in each MonitoredQueue.
|
|
112
|
the upstream sockets in each MonitoredQueue.
|
|
|
113
|
|
|
113
|
|
|
|
114
|
IOPub
|
|
114
|
IOPub
|
|
|
115
|
-----
|
|
115
|
-----
|
|
|
116
|
|
|
116
|
|
|
|
117
|
.. figure:: figs/iopubfade.png
|
|
117
|
.. figure:: figs/iopubfade.png
|
|
|
118
|
:width: 432px
|
|
118
|
:width: 432px
|
|
|
119
|
:alt: IOPub connections
|
|
119
|
:alt: IOPub connections
|
|
|
120
|
:align: center
|
|
120
|
:align: center
|
|
|
121
|
|
|
121
|
|
|
|
122
|
stdout/err are published via a ``PUB/SUB`` MonitoredQueue
|
|
122
|
stdout/err are published via a ``PUB/SUB`` MonitoredQueue
|
|
|
123
|
|
|
123
|
|
|
|
124
|
|
|
124
|
|
|
|
125
|
On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
|
|
125
|
On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
|
|
|
126
|
sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
|
|
126
|
sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
|
|
|
127
|
messages. They are then republished via another ``PUB`` socket, which can be
|
|
127
|
messages. They are then republished via another ``PUB`` socket, which can be
|
|
|
128
|
subscribed by the clients.
|
|
128
|
subscribed by the clients.
|
|
|
129
|
|
|
129
|
|
|
|
130
|
Client connections
|
|
130
|
Client connections
|
|
|
131
|
------------------
|
|
131
|
------------------
|
|
|
132
|
|
|
132
|
|
|
|
133
|
.. figure:: figs/queryfade.png
|
|
133
|
.. figure:: figs/queryfade.png
|
|
|
134
|
:width: 432px
|
|
134
|
:width: 432px
|
|
|
135
|
:alt: IPython client query connections
|
|
135
|
:alt: IPython client query connections
|
|
|
136
|
:align: center
|
|
136
|
:align: center
|
|
|
137
|
|
|
137
|
|
|
|
138
|
Clients connect to an ``ROUTER`` socket to query the hub.
|
|
138
|
Clients connect to an ``ROUTER`` socket to query the hub.
|
|
|
139
|
|
|
139
|
|
|
|
140
|
The hub's registrar ``ROUTER`` socket also listens for queries from clients as to queue status,
|
|
140
|
The hub's registrar ``ROUTER`` socket also listens for queries from clients as to queue status,
|
|
|
141
|
and control instructions. Clients connect to this socket via an ``DEALER`` during registration.
|
|
141
|
and control instructions. Clients connect to this socket via an ``DEALER`` during registration.
|
|
|
142
|
|
|
142
|
|
|
|
143
|
.. figure:: figs/notiffade.png
|
|
143
|
.. figure:: figs/notiffade.png
|
|
|
144
|
:width: 432px
|
|
144
|
:width: 432px
|
|
|
145
|
:alt: IPython Registration connections
|
|
145
|
:alt: IPython Registration connections
|
|
|
146
|
:align: center
|
|
146
|
:align: center
|
|
|
147
|
|
|
147
|
|
|
|
148
|
Engine registration events are published via a ``PUB`` socket.
|
|
148
|
Engine registration events are published via a ``PUB`` socket.
|
|
|
149
|
|
|
149
|
|
|
|
150
|
The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
|
|
150
|
The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
|
|
|
151
|
allows clients to stay up to date with what engines are available by subscribing to the
|
|
151
|
allows clients to stay up to date with what engines are available by subscribing to the
|
|
|
152
|
feed with a ``SUB`` socket. Other processes could selectively subscribe to just
|
|
152
|
feed with a ``SUB`` socket. Other processes could selectively subscribe to just
|
|
|
153
|
registration or unregistration events.
|
|
153
|
registration or unregistration events.
|
|
|
154
|
|
|
154
|
|
