##// END OF EJS Templates
Update info in parallel docs.
Thomas Kluyver -
Show More
@@ -1,156 +1,151 b''
1 1 .. _parallelmpi:
2 2
3 3 =======================
4 4 Using MPI with IPython
5 5 =======================
6 6
7 .. note::
8
9 Not adapted to zmq yet
10 This is out of date wrt ipcluster in general as well
11
12 7 Often, a parallel algorithm will require moving data between the engines. One
13 8 way of accomplishing this is by doing a pull and then a push using the
14 9 multiengine client. However, this will be slow as all the data has to go
15 10 through the controller to the client and then back through the controller, to
16 11 its final destination.
17 12
18 13 A much better way of moving data between engines is to use a message passing
19 14 library, such as the Message Passing Interface (MPI) [MPI]_. IPython's
20 15 parallel computing architecture has been designed from the ground up to
21 16 integrate with MPI. This document describes how to use MPI with IPython.
22 17
23 18 Additional installation requirements
24 19 ====================================
25 20
26 21 If you want to use MPI with IPython, you will need to install:
27 22
28 23 * A standard MPI implementation such as OpenMPI [OpenMPI]_ or MPICH.
29 24 * The mpi4py [mpi4py]_ package.
30 25
31 26 .. note::
32 27
33 28 The mpi4py package is not a strict requirement. However, you need to
34 29 have *some* way of calling MPI from Python. You also need some way of
35 30 making sure that :func:`MPI_Init` is called when the IPython engines start
36 31 up. There are a number of ways of doing this and a good number of
37 32 associated subtleties. We highly recommend just using mpi4py as it
38 33 takes care of most of these problems. If you want to do something
39 34 different, let us know and we can help you get started.
40 35
41 36 Starting the engines with MPI enabled
42 37 =====================================
43 38
44 39 To use code that calls MPI, there are typically two things that MPI requires.
45 40
46 41 1. The process that wants to call MPI must be started using
47 42 :command:`mpiexec` or a batch system (like PBS) that has MPI support.
48 43 2. Once the process starts, it must call :func:`MPI_Init`.
49 44
50 45 There are a couple of ways that you can start the IPython engines and get
51 46 these things to happen.
52 47
53 48 Automatic starting using :command:`mpiexec` and :command:`ipcluster`
54 49 --------------------------------------------------------------------
55 50
56 51 The easiest approach is to use the `MPIExec` Launchers in :command:`ipcluster`,
57 52 which will first start a controller and then a set of engines using
58 53 :command:`mpiexec`::
59 54
60 55 $ ipcluster start --n=4 --elauncher=MPIExecEngineSetLauncher
61 56
62 57 This approach is best as interrupting :command:`ipcluster` will automatically
63 58 stop and clean up the controller and engines.
64 59
65 60 Manual starting using :command:`mpiexec`
66 61 ----------------------------------------
67 62
68 63 If you want to start the IPython engines using the :command:`mpiexec`, just
69 64 do::
70 65
71 66 $ mpiexec n=4 ipengine --mpi=mpi4py
72 67
73 68 This requires that you already have a controller running and that the FURL
74 69 files for the engines are in place. We also have built in support for
75 70 PyTrilinos [PyTrilinos]_, which can be used (assuming is installed) by
76 71 starting the engines with::
77 72
78 73 $ mpiexec n=4 ipengine --mpi=pytrilinos
79 74
80 75 Automatic starting using PBS and :command:`ipcluster`
81 76 ------------------------------------------------------
82 77
83 78 The :command:`ipcluster` command also has built-in integration with PBS. For
84 79 more information on this approach, see our documentation on :ref:`ipcluster
85 80 <parallel_process>`.
86 81
87 82 Actually using MPI
88 83 ==================
89 84
90 85 Once the engines are running with MPI enabled, you are ready to go. You can
91 86 now call any code that uses MPI in the IPython engines. And, all of this can
92 87 be done interactively. Here we show a simple example that uses mpi4py
93 88 [mpi4py]_ version 1.1.0 or later.
94 89
95 90 First, lets define a simply function that uses MPI to calculate the sum of a
96 91 distributed array. Save the following text in a file called :file:`psum.py`:
97 92
98 93 .. sourcecode:: python
99 94
100 95 from mpi4py import MPI
101 96 import numpy as np
102 97
103 98 def psum(a):
104 99 s = np.sum(a)
105 100 rcvBuf = np.array(0.0,'d')
106 101 MPI.COMM_WORLD.Allreduce([s, MPI.DOUBLE],
107 102 [rcvBuf, MPI.DOUBLE],
108 103 op=MPI.SUM)
109 104 return rcvBuf
110 105
111 106 Now, start an IPython cluster::
112 107
113 108 $ ipcluster start --profile=mpi --n=4
114 109
115 110 .. note::
116 111
117 112 It is assumed here that the mpi profile has been set up, as described :ref:`here
118 113 <parallel_process>`.
119 114
120 115 Finally, connect to the cluster and use this function interactively. In this
121 116 case, we create a random array on each engine and sum up all the random arrays
122 117 using our :func:`psum` function:
123 118
124 119 .. sourcecode:: ipython
125 120
126 121 In [1]: from IPython.parallel import Client
127 122
128 123 In [2]: %load_ext parallel_magic
129 124
130 125 In [3]: c = Client(profile='mpi')
131 126
132 127 In [4]: view = c[:]
133 128
134 129 In [5]: view.activate()
135 130
136 131 # run the contents of the file on each engine:
137 132 In [6]: view.run('psum.py')
138 133
139 134 In [6]: px a = np.random.rand(100)
140 135 Parallel execution on engines: [0,1,2,3]
141 136
142 137 In [8]: px s = psum(a)
143 138 Parallel execution on engines: [0,1,2,3]
144 139
145 140 In [9]: view['s']
146 141 Out[9]: [187.451545803,187.451545803,187.451545803,187.451545803]
147 142
148 143 Any Python code that makes calls to MPI can be used in this manner, including
149 144 compiled C, C++ and Fortran libraries that have been exposed to Python.
150 145
151 146 .. [MPI] Message Passing Interface. http://www-unix.mcs.anl.gov/mpi/
152 147 .. [mpi4py] MPI for Python. mpi4py: http://mpi4py.scipy.org/
153 148 .. [OpenMPI] Open MPI. http://www.open-mpi.org/
154 149 .. [PyTrilinos] PyTrilinos. http://trilinos.sandia.gov/packages/pytrilinos/
155 150
156 151
@@ -1,324 +1,255 b''
1 1 .. _parallelsecurity:
2 2
3 3 ===========================
4 4 Security details of IPython
5 5 ===========================
6 6
7 7 .. note::
8 8
9 9 This section is not thorough, and IPython.zmq needs a thorough security
10 10 audit.
11 11
12 12 IPython's :mod:`IPython.zmq` package exposes the full power of the
13 13 Python interpreter over a TCP/IP network for the purposes of parallel
14 14 computing. This feature brings up the important question of IPython's security
15 15 model. This document gives details about this model and how it is implemented
16 16 in IPython's architecture.
17 17
18 18 Process and network topology
19 19 ============================
20 20
21 21 To enable parallel computing, IPython has a number of different processes that
22 22 run. These processes are discussed at length in the IPython documentation and
23 23 are summarized here:
24 24
25 25 * The IPython *engine*. This process is a full blown Python
26 26 interpreter in which user code is executed. Multiple
27 27 engines are started to make parallel computing possible.
28 28 * The IPython *hub*. This process monitors a set of
29 29 engines and schedulers, and keeps track of the state of the processes. It listens
30 30 for registration connections from engines and clients, and monitor connections
31 31 from schedulers.
32 32 * The IPython *schedulers*. This is a set of processes that relay commands and results
33 33 between clients and engines. They are typically on the same machine as the controller,
34 34 and listen for connections from engines and clients, but connect to the Hub.
35 35 * The IPython *client*. This process is typically an
36 36 interactive Python process that is used to coordinate the
37 37 engines to get a parallel computation done.
38 38
39 39 Collectively, these processes are called the IPython *cluster*, and the hub and schedulers
40 40 together are referred to as the *controller*.
41 41
42 42
43 43 These processes communicate over any transport supported by ZeroMQ (tcp,pgm,infiniband,ipc)
44 44 with a well defined topology. The IPython hub and schedulers listen on sockets. Upon
45 45 starting, an engine connects to a hub and registers itself, which then informs the engine
46 46 of the connection information for the schedulers, and the engine then connects to the
47 47 schedulers. These engine/hub and engine/scheduler connections persist for the
48 48 lifetime of each engine.
49 49
50 50 The IPython client also connects to the controller processes using a number of socket
51 51 connections. As of writing, this is one socket per scheduler (4), and 3 connections to the
52 52 hub for a total of 7. These connections persist for the lifetime of the client only.
53 53
54 54 A given IPython controller and set of engines engines typically has a relatively
55 55 short lifetime. Typically this lifetime corresponds to the duration of a single parallel
56 56 simulation performed by a single user. Finally, the hub, schedulers, engines, and client
57 57 processes typically execute with the permissions of that same user. More specifically, the
58 58 controller and engines are *not* executed as root or with any other superuser permissions.
59 59
60 60 Application logic
61 61 =================
62 62
63 63 When running the IPython kernel to perform a parallel computation, a user
64 64 utilizes the IPython client to send Python commands and data through the
65 65 IPython schedulers to the IPython engines, where those commands are executed
66 66 and the data processed. The design of IPython ensures that the client is the
67 67 only access point for the capabilities of the engines. That is, the only way
68 68 of addressing the engines is through a client.
69 69
70 70 A user can utilize the client to instruct the IPython engines to execute
71 71 arbitrary Python commands. These Python commands can include calls to the
72 72 system shell, access the filesystem, etc., as required by the user's
73 73 application code. From this perspective, when a user runs an IPython engine on
74 74 a host, that engine has the same capabilities and permissions as the user
75 75 themselves (as if they were logged onto the engine's host with a terminal).
76 76
77 77 Secure network connections
78 78 ==========================
79 79
80 80 Overview
81 81 --------
82 82
83 83 ZeroMQ provides exactly no security. For this reason, users of IPython must be very
84 84 careful in managing connections, because an open TCP/IP socket presents access to
85 85 arbitrary execution as the user on the engine machines. As a result, the default behavior
86 86 of controller processes is to only listen for clients on the loopback interface, and the
87 87 client must establish SSH tunnels to connect to the controller processes.
88 88
89 89 .. warning::
90 90
91 91 If the controller's loopback interface is untrusted, then IPython should be considered
92 92 vulnerable, and this extends to the loopback of all connected clients, which have
93 93 opened a loopback port that is redirected to the controller's loopback port.
94 94
95 95
96 96 SSH
97 97 ---
98 98
99 99 Since ZeroMQ provides no security, SSH tunnels are the primary source of secure
100 100 connections. A connector file, such as `ipcontroller-client.json`, will contain
101 101 information for connecting to the controller, possibly including the address of an
102 102 ssh-server through with the client is to tunnel. The Client object then creates tunnels
103 103 using either [OpenSSH]_ or [Paramiko]_, depending on the platform. If users do not wish to
104 104 use OpenSSH or Paramiko, or the tunneling utilities are insufficient, then they may
105 105 construct the tunnels themselves, and simply connect clients and engines as if the
106 106 controller were on loopback on the connecting machine.
107 107
108 108 .. note::
109 109
110 110 There is not currently tunneling available for engines.
111 111
112 112 Authentication
113 113 --------------
114 114
115 115 To protect users of shared machines, [HMAC]_ digests are used to sign messages, using a
116 116 shared key.
117 117
118 118 The Session object that handles the message protocol uses a unique key to verify valid
119 119 messages. This can be any value specified by the user, but the default behavior is a
120 120 pseudo-random 128-bit number, as generated by `uuid.uuid4()`. This key is used to
121 121 initialize an HMAC object, which digests all messages, and includes that digest as a
122 122 signature and part of the message. Every message that is unpacked (on Controller, Engine,
123 123 and Client) will also be digested by the receiver, ensuring that the sender's key is the
124 124 same as the receiver's. No messages that do not contain this key are acted upon in any
125 125 way. The key itself is never sent over the network.
126 126
127 127 There is exactly one shared key per cluster - it must be the same everywhere. Typically,
128 128 the controller creates this key, and stores it in the private connection files
129 129 `ipython-{engine|client}.json`. These files are typically stored in the
130 130 `~/.ipython/profile_<name>/security` directory, and are maintained as readable only by the
131 131 owner, just as is common practice with a user's keys in their `.ssh` directory.
132 132
133 133 .. warning::
134 134
135 135 It is important to note that the key authentication, as emphasized by the use of
136 136 a uuid rather than generating a key with a cryptographic library, provides a
137 137 defense against *accidental* messages more than it does against malicious attacks.
138 138 If loopback is compromised, it would be trivial for an attacker to intercept messages
139 139 and deduce the key, as there is no encryption.
140 140
141 141
142 142
143 143 Specific security vulnerabilities
144 144 =================================
145 145
146 146 There are a number of potential security vulnerabilities present in IPython's
147 147 architecture. In this section we discuss those vulnerabilities and detail how
148 148 the security architecture described above prevents them from being exploited.
149 149
150 150 Unauthorized clients
151 151 --------------------
152 152
153 153 The IPython client can instruct the IPython engines to execute arbitrary
154 154 Python code with the permissions of the user who started the engines. If an
155 155 attacker were able to connect their own hostile IPython client to the IPython
156 156 controller, they could instruct the engines to execute code.
157 157
158 158
159 159 On the first level, this attack is prevented by requiring access to the controller's
160 160 ports, which are recommended to only be open on loopback if the controller is on an
161 161 untrusted local network. If the attacker does have access to the Controller's ports, then
162 162 the attack is prevented by the capabilities based client authentication of the execution
163 163 key. The relevant authentication information is encoded into the JSON file that clients
164 164 must present to gain access to the IPython controller. By limiting the distribution of
165 165 those keys, a user can grant access to only authorized persons, just as with SSH keys.
166 166
167 167 It is highly unlikely that an execution key could be guessed by an attacker
168 168 in a brute force guessing attack. A given instance of the IPython controller
169 169 only runs for a relatively short amount of time (on the order of hours). Thus
170 170 an attacker would have only a limited amount of time to test a search space of
171 171 size 2**128. For added security, users can have arbitrarily long keys.
172 172
173 173 .. warning::
174 174
175 175 If the attacker has gained enough access to intercept loopback connections on *either* the
176 176 controller or client, then a duplicate message can be sent. To protect against this,
177 177 recipients only allow each signature once, and consider duplicates invalid. However,
178 178 the duplicate message could be sent to *another* recipient using the same key,
179 179 and it would be considered valid.
180 180
181 181
182 182 Unauthorized engines
183 183 --------------------
184 184
185 185 If an attacker were able to connect a hostile engine to a user's controller,
186 186 the user might unknowingly send sensitive code or data to the hostile engine.
187 187 This attacker's engine would then have full access to that code and data.
188 188
189 189 This type of attack is prevented in the same way as the unauthorized client
190 190 attack, through the usage of the capabilities based authentication scheme.
191 191
192 192 Unauthorized controllers
193 193 ------------------------
194 194
195 195 It is also possible that an attacker could try to convince a user's IPython
196 196 client or engine to connect to a hostile IPython controller. That controller
197 197 would then have full access to the code and data sent between the IPython
198 198 client and the IPython engines.
199 199
200 200 Again, this attack is prevented through the capabilities in a connection file, which
201 201 ensure that a client or engine connects to the correct controller. It is also important to
202 202 note that the connection files also encode the IP address and port that the controller is
203 203 listening on, so there is little chance of mistakenly connecting to a controller running
204 204 on a different IP address and port.
205 205
206 206 When starting an engine or client, a user must specify the key to use
207 207 for that connection. Thus, in order to introduce a hostile controller, the
208 208 attacker must convince the user to use the key associated with the
209 209 hostile controller. As long as a user is diligent in only using keys from
210 210 trusted sources, this attack is not possible.
211 211
212 212 .. note::
213 213
214 214 I may be wrong, the unauthorized controller may be easier to fake than this.
215 215
216 216 Other security measures
217 217 =======================
218 218
219 219 A number of other measures are taken to further limit the security risks
220 220 involved in running the IPython kernel.
221 221
222 222 First, by default, the IPython controller listens on random port numbers.
223 223 While this can be overridden by the user, in the default configuration, an
224 224 attacker would have to do a port scan to even find a controller to attack.
225 225 When coupled with the relatively short running time of a typical controller
226 226 (on the order of hours), an attacker would have to work extremely hard and
227 227 extremely *fast* to even find a running controller to attack.
228 228
229 229 Second, much of the time, especially when run on supercomputers or clusters,
230 230 the controller is running behind a firewall. Thus, for engines or client to
231 231 connect to the controller:
232 232
233 233 * The different processes have to all be behind the firewall.
234 234
235 235 or:
236 236
237 237 * The user has to use SSH port forwarding to tunnel the
238 238 connections through the firewall.
239 239
240 240 In either case, an attacker is presented with additional barriers that prevent
241 241 attacking or even probing the system.
242 242
243 243 Summary
244 244 =======
245 245
246 246 IPython's architecture has been carefully designed with security in mind. The
247 247 capabilities based authentication model, in conjunction with SSH tunneled
248 248 TCP/IP channels, address the core potential vulnerabilities in the system,
249 249 while still enabling user's to use the system in open networks.
250 250
251 Other questions
252 ===============
253
254 .. note::
255
256 this does not apply to ZMQ, but I am sure there will be questions.
257
258 About keys
259 ----------
260
261 Can you clarify the roles of the certificate and its keys versus the FURL,
262 which is also called a key?
263
264 The certificate created by IPython processes is a standard public key x509
265 certificate, that is used by the SSL handshake protocol to setup encrypted
266 channel between the controller and the IPython engine or client. This public
267 and private key associated with this certificate are used only by the SSL
268 handshake protocol in setting up this encrypted channel.
269
270 The FURL serves a completely different and independent purpose from the
271 key pair associated with the certificate. When we refer to a FURL as a
272 key, we are using the word "key" in the capabilities based security model
273 sense. This has nothing to do with "key" in the public/private key sense used
274 in the SSL protocol.
275
276 With that said the FURL is used as an cryptographic key, to grant
277 IPython engines and clients access to particular capabilities that the
278 controller offers.
279
280 Self signed certificates
281 ------------------------
282
283 Is the controller creating a self-signed certificate? Is this created for per
284 instance/session, one-time-setup or each-time the controller is started?
285
286 The Foolscap network protocol, which handles the SSL protocol details, creates
287 a self-signed x509 certificate using OpenSSL for each IPython process. The
288 lifetime of the certificate is handled differently for the IPython controller
289 and the engines/client.
290
291 For the IPython engines and client, the certificate is only held in memory for
292 the lifetime of its process. It is never written to disk.
293
294 For the controller, the certificate can be created anew each time the
295 controller starts or it can be created once and reused each time the
296 controller starts. If at any point, the certificate is deleted, a new one is
297 created the next time the controller starts.
298
299 SSL private key
300 ---------------
301
302 How the private key (associated with the certificate) is distributed?
303
304 In the usual implementation of the SSL protocol, the private key is never
305 distributed. We follow this standard always.
306
307 SSL versus Foolscap authentication
308 ----------------------------------
309
310 Many SSL connections only perform one sided authentication (the server to the
311 client). How is the client authentication in IPython's system related to SSL
312 authentication?
313
314 We perform a two way SSL handshake in which both parties request and verify
315 the certificate of their peer. This mutual authentication is handled by the
316 SSL handshake and is separate and independent from the additional
317 authentication steps that the CLIENT and SERVER perform after an encrypted
318 channel is established.
319
320 251 .. [RFC5246] <http://tools.ietf.org/html/rfc5246>
321 252
322 253 .. [OpenSSH] <http://www.openssh.com/>
323 254 .. [Paramiko] <http://www.lag.net/paramiko/>
324 255 .. [HMAC] <http://tools.ietf.org/html/rfc2104.html>
General Comments 0
You need to be logged in to leave comments. Login now