##// END OF EJS Templates
Merge pull request #5693 from mohanraj-r/patch-2...
Min RK -
r16401:dd096556 merge
parent child Browse files
Show More
@@ -1,307 +1,307 b''
1 .. _parallel_overview:
1 .. _parallel_overview:
2
2
3 ============================
3 ============================
4 Overview and getting started
4 Overview and getting started
5 ============================
5 ============================
6
6
7
7
8 Examples
8 Examples
9 ========
9 ========
10
10
11 We have various example scripts and notebooks for using IPython.parallel in our
11 We have various example scripts and notebooks for using IPython.parallel in our
12 :file:`examples/parallel` directory, or they can be found `on GitHub`__.
12 :file:`examples/Parallel%20Computing` directory, or they can be viewed `using nbviewer`__.
13 Some of these are covered in more detail in the :ref:`examples
13 Some of these are covered in more detail in the :ref:`examples
14 <parallel_examples>` section.
14 <parallel_examples>` section.
15
15
16 .. __: https://github.com/ipython/ipython/tree/master/examples/parallel
16 .. __: http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Parallel%20Computing/Index.ipynb
17
17
18 Introduction
18 Introduction
19 ============
19 ============
20
20
21 This section gives an overview of IPython's sophisticated and powerful
21 This section gives an overview of IPython's sophisticated and powerful
22 architecture for parallel and distributed computing. This architecture
22 architecture for parallel and distributed computing. This architecture
23 abstracts out parallelism in a very general way, which enables IPython to
23 abstracts out parallelism in a very general way, which enables IPython to
24 support many different styles of parallelism including:
24 support many different styles of parallelism including:
25
25
26 * Single program, multiple data (SPMD) parallelism.
26 * Single program, multiple data (SPMD) parallelism.
27 * Multiple program, multiple data (MPMD) parallelism.
27 * Multiple program, multiple data (MPMD) parallelism.
28 * Message passing using MPI.
28 * Message passing using MPI.
29 * Task farming.
29 * Task farming.
30 * Data parallel.
30 * Data parallel.
31 * Combinations of these approaches.
31 * Combinations of these approaches.
32 * Custom user defined approaches.
32 * Custom user defined approaches.
33
33
34 Most importantly, IPython enables all types of parallel applications to
34 Most importantly, IPython enables all types of parallel applications to
35 be developed, executed, debugged and monitored *interactively*. Hence,
35 be developed, executed, debugged and monitored *interactively*. Hence,
36 the ``I`` in IPython. The following are some example usage cases for IPython:
36 the ``I`` in IPython. The following are some example usage cases for IPython:
37
37
38 * Quickly parallelize algorithms that are embarrassingly parallel
38 * Quickly parallelize algorithms that are embarrassingly parallel
39 using a number of simple approaches. Many simple things can be
39 using a number of simple approaches. Many simple things can be
40 parallelized interactively in one or two lines of code.
40 parallelized interactively in one or two lines of code.
41
41
42 * Steer traditional MPI applications on a supercomputer from an
42 * Steer traditional MPI applications on a supercomputer from an
43 IPython session on your laptop.
43 IPython session on your laptop.
44
44
45 * Analyze and visualize large datasets (that could be remote and/or
45 * Analyze and visualize large datasets (that could be remote and/or
46 distributed) interactively using IPython and tools like
46 distributed) interactively using IPython and tools like
47 matplotlib/TVTK.
47 matplotlib/TVTK.
48
48
49 * Develop, test and debug new parallel algorithms
49 * Develop, test and debug new parallel algorithms
50 (that may use MPI) interactively.
50 (that may use MPI) interactively.
51
51
52 * Tie together multiple MPI jobs running on different systems into
52 * Tie together multiple MPI jobs running on different systems into
53 one giant distributed and parallel system.
53 one giant distributed and parallel system.
54
54
55 * Start a parallel job on your cluster and then have a remote
55 * Start a parallel job on your cluster and then have a remote
56 collaborator connect to it and pull back data into their
56 collaborator connect to it and pull back data into their
57 local IPython session for plotting and analysis.
57 local IPython session for plotting and analysis.
58
58
59 * Run a set of tasks on a set of CPUs using dynamic load balancing.
59 * Run a set of tasks on a set of CPUs using dynamic load balancing.
60
60
61 .. tip::
61 .. tip::
62
62
63 At the SciPy 2011 conference in Austin, Min Ragan-Kelley presented a
63 At the SciPy 2011 conference in Austin, Min Ragan-Kelley presented a
64 complete 4-hour tutorial on the use of these features, and all the materials
64 complete 4-hour tutorial on the use of these features, and all the materials
65 for the tutorial are now `available online`__. That tutorial provides an
65 for the tutorial are now `available online`__. That tutorial provides an
66 excellent, hands-on oriented complement to the reference documentation
66 excellent, hands-on oriented complement to the reference documentation
67 presented here.
67 presented here.
68
68
69 .. __: http://minrk.github.com/scipy-tutorial-2011
69 .. __: http://minrk.github.com/scipy-tutorial-2011
70
70
71 Architecture overview
71 Architecture overview
72 =====================
72 =====================
73
73
74 .. figure:: figs/wideView.png
74 .. figure:: figs/wideView.png
75 :width: 300px
75 :width: 300px
76
76
77
77
78 The IPython architecture consists of four components:
78 The IPython architecture consists of four components:
79
79
80 * The IPython engine.
80 * The IPython engine.
81 * The IPython hub.
81 * The IPython hub.
82 * The IPython schedulers.
82 * The IPython schedulers.
83 * The controller client.
83 * The controller client.
84
84
85 These components live in the :mod:`IPython.parallel` package and are
85 These components live in the :mod:`IPython.parallel` package and are
86 installed with IPython. They do, however, have additional dependencies
86 installed with IPython. They do, however, have additional dependencies
87 that must be installed. For more information, see our
87 that must be installed. For more information, see our
88 :ref:`installation documentation <install_index>`.
88 :ref:`installation documentation <install_index>`.
89
89
90 .. TODO: include zmq in install_index
90 .. TODO: include zmq in install_index
91
91
92 IPython engine
92 IPython engine
93 ---------------
93 ---------------
94
94
95 The IPython engine is a Python instance that takes Python commands over a
95 The IPython engine is a Python instance that takes Python commands over a
96 network connection. Eventually, the IPython engine will be a full IPython
96 network connection. Eventually, the IPython engine will be a full IPython
97 interpreter, but for now, it is a regular Python interpreter. The engine
97 interpreter, but for now, it is a regular Python interpreter. The engine
98 can also handle incoming and outgoing Python objects sent over a network
98 can also handle incoming and outgoing Python objects sent over a network
99 connection. When multiple engines are started, parallel and distributed
99 connection. When multiple engines are started, parallel and distributed
100 computing becomes possible. An important feature of an IPython engine is
100 computing becomes possible. An important feature of an IPython engine is
101 that it blocks while user code is being executed. Read on for how the
101 that it blocks while user code is being executed. Read on for how the
102 IPython controller solves this problem to expose a clean asynchronous API
102 IPython controller solves this problem to expose a clean asynchronous API
103 to the user.
103 to the user.
104
104
105 IPython controller
105 IPython controller
106 ------------------
106 ------------------
107
107
108 The IPython controller processes provide an interface for working with a set of engines.
108 The IPython controller processes provide an interface for working with a set of engines.
109 At a general level, the controller is a collection of processes to which IPython engines
109 At a general level, the controller is a collection of processes to which IPython engines
110 and clients can connect. The controller is composed of a :class:`Hub` and a collection of
110 and clients can connect. The controller is composed of a :class:`Hub` and a collection of
111 :class:`Schedulers`. These Schedulers are typically run in separate processes but on the
111 :class:`Schedulers`. These Schedulers are typically run in separate processes but on the
112 same machine as the Hub, but can be run anywhere from local threads or on remote machines.
112 same machine as the Hub, but can be run anywhere from local threads or on remote machines.
113
113
114 The controller also provides a single point of contact for users who wish to
114 The controller also provides a single point of contact for users who wish to
115 utilize the engines connected to the controller. There are different ways of
115 utilize the engines connected to the controller. There are different ways of
116 working with a controller. In IPython, all of these models are implemented via
116 working with a controller. In IPython, all of these models are implemented via
117 the :meth:`.View.apply` method, after
117 the :meth:`.View.apply` method, after
118 constructing :class:`.View` objects to represent subsets of engines. The two
118 constructing :class:`.View` objects to represent subsets of engines. The two
119 primary models for interacting with engines are:
119 primary models for interacting with engines are:
120
120
121 * A **Direct** interface, where engines are addressed explicitly.
121 * A **Direct** interface, where engines are addressed explicitly.
122 * A **LoadBalanced** interface, where the Scheduler is trusted with assigning work to
122 * A **LoadBalanced** interface, where the Scheduler is trusted with assigning work to
123 appropriate engines.
123 appropriate engines.
124
124
125 Advanced users can readily extend the View models to enable other
125 Advanced users can readily extend the View models to enable other
126 styles of parallelism.
126 styles of parallelism.
127
127
128 .. note::
128 .. note::
129
129
130 A single controller and set of engines can be used with multiple models
130 A single controller and set of engines can be used with multiple models
131 simultaneously. This opens the door for lots of interesting things.
131 simultaneously. This opens the door for lots of interesting things.
132
132
133
133
134 The Hub
134 The Hub
135 *******
135 *******
136
136
137 The center of an IPython cluster is the Hub. This is the process that keeps
137 The center of an IPython cluster is the Hub. This is the process that keeps
138 track of engine connections, schedulers, clients, as well as all task requests and
138 track of engine connections, schedulers, clients, as well as all task requests and
139 results. The primary role of the Hub is to facilitate queries of the cluster state, and
139 results. The primary role of the Hub is to facilitate queries of the cluster state, and
140 minimize the necessary information required to establish the many connections involved in
140 minimize the necessary information required to establish the many connections involved in
141 connecting new clients and engines.
141 connecting new clients and engines.
142
142
143
143
144 Schedulers
144 Schedulers
145 **********
145 **********
146
146
147 All actions that can be performed on the engine go through a Scheduler. While the engines
147 All actions that can be performed on the engine go through a Scheduler. While the engines
148 themselves block when user code is run, the schedulers hide that from the user to provide
148 themselves block when user code is run, the schedulers hide that from the user to provide
149 a fully asynchronous interface to a set of engines.
149 a fully asynchronous interface to a set of engines.
150
150
151
151
152 IPython client and views
152 IPython client and views
153 ------------------------
153 ------------------------
154
154
155 There is one primary object, the :class:`~.parallel.Client`, for connecting to a cluster.
155 There is one primary object, the :class:`~.parallel.Client`, for connecting to a cluster.
156 For each execution model, there is a corresponding :class:`~.parallel.View`. These views
156 For each execution model, there is a corresponding :class:`~.parallel.View`. These views
157 allow users to interact with a set of engines through the interface. Here are the two default
157 allow users to interact with a set of engines through the interface. Here are the two default
158 views:
158 views:
159
159
160 * The :class:`DirectView` class for explicit addressing.
160 * The :class:`DirectView` class for explicit addressing.
161 * The :class:`LoadBalancedView` class for destination-agnostic scheduling.
161 * The :class:`LoadBalancedView` class for destination-agnostic scheduling.
162
162
163 Security
163 Security
164 --------
164 --------
165
165
166 IPython uses ZeroMQ for networking, which has provided many advantages, but
166 IPython uses ZeroMQ for networking, which has provided many advantages, but
167 one of the setbacks is its utter lack of security [ZeroMQ]_. By default, no IPython
167 one of the setbacks is its utter lack of security [ZeroMQ]_. By default, no IPython
168 connections are encrypted, but open ports only listen on localhost. The only
168 connections are encrypted, but open ports only listen on localhost. The only
169 source of security for IPython is via ssh-tunnel. IPython supports both shell
169 source of security for IPython is via ssh-tunnel. IPython supports both shell
170 (`openssh`) and `paramiko` based tunnels for connections. There is a key necessary
170 (`openssh`) and `paramiko` based tunnels for connections. There is a key necessary
171 to submit requests, but due to the lack of encryption, it does not provide
171 to submit requests, but due to the lack of encryption, it does not provide
172 significant security if loopback traffic is compromised.
172 significant security if loopback traffic is compromised.
173
173
174 In our architecture, the controller is the only process that listens on
174 In our architecture, the controller is the only process that listens on
175 network ports, and is thus the main point of vulnerability. The standard model
175 network ports, and is thus the main point of vulnerability. The standard model
176 for secure connections is to designate that the controller listen on
176 for secure connections is to designate that the controller listen on
177 localhost, and use ssh-tunnels to connect clients and/or
177 localhost, and use ssh-tunnels to connect clients and/or
178 engines.
178 engines.
179
179
180 To connect and authenticate to the controller an engine or client needs
180 To connect and authenticate to the controller an engine or client needs
181 some information that the controller has stored in a JSON file.
181 some information that the controller has stored in a JSON file.
182 Thus, the JSON files need to be copied to a location where
182 Thus, the JSON files need to be copied to a location where
183 the clients and engines can find them. Typically, this is the
183 the clients and engines can find them. Typically, this is the
184 :file:`~/.ipython/profile_default/security` directory on the host where the
184 :file:`~/.ipython/profile_default/security` directory on the host where the
185 client/engine is running (which could be a different host than the controller).
185 client/engine is running (which could be a different host than the controller).
186 Once the JSON files are copied over, everything should work fine.
186 Once the JSON files are copied over, everything should work fine.
187
187
188 Currently, there are two JSON files that the controller creates:
188 Currently, there are two JSON files that the controller creates:
189
189
190 ipcontroller-engine.json
190 ipcontroller-engine.json
191 This JSON file has the information necessary for an engine to connect
191 This JSON file has the information necessary for an engine to connect
192 to a controller.
192 to a controller.
193
193
194 ipcontroller-client.json
194 ipcontroller-client.json
195 The client's connection information. This may not differ from the engine's,
195 The client's connection information. This may not differ from the engine's,
196 but since the controller may listen on different ports for clients and
196 but since the controller may listen on different ports for clients and
197 engines, it is stored separately.
197 engines, it is stored separately.
198
198
199 ipcontroller-client.json will look something like this, under default localhost
199 ipcontroller-client.json will look something like this, under default localhost
200 circumstances:
200 circumstances:
201
201
202 .. sourcecode:: python
202 .. sourcecode:: python
203
203
204 {
204 {
205 "url":"tcp:\/\/127.0.0.1:54424",
205 "url":"tcp:\/\/127.0.0.1:54424",
206 "exec_key":"a361fe89-92fc-4762-9767-e2f0a05e3130",
206 "exec_key":"a361fe89-92fc-4762-9767-e2f0a05e3130",
207 "ssh":"",
207 "ssh":"",
208 "location":"10.19.1.135"
208 "location":"10.19.1.135"
209 }
209 }
210
210
211 If, however, you are running the controller on a work node on a cluster, you will likely
211 If, however, you are running the controller on a work node on a cluster, you will likely
212 need to use ssh tunnels to connect clients from your laptop to it. You will also
212 need to use ssh tunnels to connect clients from your laptop to it. You will also
213 probably need to instruct the controller to listen for engines coming from other work nodes
213 probably need to instruct the controller to listen for engines coming from other work nodes
214 on the cluster. An example of ipcontroller-client.json, as created by::
214 on the cluster. An example of ipcontroller-client.json, as created by::
215
215
216 $> ipcontroller --ip=* --ssh=login.mycluster.com
216 $> ipcontroller --ip=* --ssh=login.mycluster.com
217
217
218
218
219 .. sourcecode:: python
219 .. sourcecode:: python
220
220
221 {
221 {
222 "url":"tcp:\/\/*:54424",
222 "url":"tcp:\/\/*:54424",
223 "exec_key":"a361fe89-92fc-4762-9767-e2f0a05e3130",
223 "exec_key":"a361fe89-92fc-4762-9767-e2f0a05e3130",
224 "ssh":"login.mycluster.com",
224 "ssh":"login.mycluster.com",
225 "location":"10.0.0.2"
225 "location":"10.0.0.2"
226 }
226 }
227
227
228 More details of how these JSON files are used are given below.
228 More details of how these JSON files are used are given below.
229
229
230 A detailed description of the security model and its implementation in IPython
230 A detailed description of the security model and its implementation in IPython
231 can be found :ref:`here <parallelsecurity>`.
231 can be found :ref:`here <parallelsecurity>`.
232
232
233 .. warning::
233 .. warning::
234
234
235 Even at its most secure, the Controller listens on ports on localhost, and
235 Even at its most secure, the Controller listens on ports on localhost, and
236 every time you make a tunnel, you open a localhost port on the connecting
236 every time you make a tunnel, you open a localhost port on the connecting
237 machine that points to the Controller. If localhost on the Controller's
237 machine that points to the Controller. If localhost on the Controller's
238 machine, or the machine of any client or engine, is untrusted, then your
238 machine, or the machine of any client or engine, is untrusted, then your
239 Controller is insecure. There is no way around this with ZeroMQ.
239 Controller is insecure. There is no way around this with ZeroMQ.
240
240
241
241
242
242
243 Getting Started
243 Getting Started
244 ===============
244 ===============
245
245
246 To use IPython for parallel computing, you need to start one instance of the
246 To use IPython for parallel computing, you need to start one instance of the
247 controller and one or more instances of the engine. Initially, it is best to
247 controller and one or more instances of the engine. Initially, it is best to
248 simply start a controller and engines on a single host using the
248 simply start a controller and engines on a single host using the
249 :command:`ipcluster` command. To start a controller and 4 engines on your
249 :command:`ipcluster` command. To start a controller and 4 engines on your
250 localhost, just do::
250 localhost, just do::
251
251
252 $ ipcluster start -n 4
252 $ ipcluster start -n 4
253
253
254 More details about starting the IPython controller and engines can be found
254 More details about starting the IPython controller and engines can be found
255 :ref:`here <parallel_process>`
255 :ref:`here <parallel_process>`
256
256
257 Once you have started the IPython controller and one or more engines, you
257 Once you have started the IPython controller and one or more engines, you
258 are ready to use the engines to do something useful. To make sure
258 are ready to use the engines to do something useful. To make sure
259 everything is working correctly, try the following commands:
259 everything is working correctly, try the following commands:
260
260
261 .. sourcecode:: ipython
261 .. sourcecode:: ipython
262
262
263 In [1]: from IPython.parallel import Client
263 In [1]: from IPython.parallel import Client
264
264
265 In [2]: c = Client()
265 In [2]: c = Client()
266
266
267 In [4]: c.ids
267 In [4]: c.ids
268 Out[4]: set([0, 1, 2, 3])
268 Out[4]: set([0, 1, 2, 3])
269
269
270 In [5]: c[:].apply_sync(lambda : "Hello, World")
270 In [5]: c[:].apply_sync(lambda : "Hello, World")
271 Out[5]: [ 'Hello, World', 'Hello, World', 'Hello, World', 'Hello, World' ]
271 Out[5]: [ 'Hello, World', 'Hello, World', 'Hello, World', 'Hello, World' ]
272
272
273
273
274 When a client is created with no arguments, the client tries to find the corresponding JSON file
274 When a client is created with no arguments, the client tries to find the corresponding JSON file
275 in the local `~/.ipython/profile_default/security` directory. Or if you specified a profile,
275 in the local `~/.ipython/profile_default/security` directory. Or if you specified a profile,
276 you can use that with the Client. This should cover most cases:
276 you can use that with the Client. This should cover most cases:
277
277
278 .. sourcecode:: ipython
278 .. sourcecode:: ipython
279
279
280 In [2]: c = Client(profile='myprofile')
280 In [2]: c = Client(profile='myprofile')
281
281
282 If you have put the JSON file in a different location or it has a different name, create the
282 If you have put the JSON file in a different location or it has a different name, create the
283 client like this:
283 client like this:
284
284
285 .. sourcecode:: ipython
285 .. sourcecode:: ipython
286
286
287 In [2]: c = Client('/path/to/my/ipcontroller-client.json')
287 In [2]: c = Client('/path/to/my/ipcontroller-client.json')
288
288
289 Remember, a client needs to be able to see the Hub's ports to connect. So if they are on a
289 Remember, a client needs to be able to see the Hub's ports to connect. So if they are on a
290 different machine, you may need to use an ssh server to tunnel access to that machine,
290 different machine, you may need to use an ssh server to tunnel access to that machine,
291 then you would connect to it with:
291 then you would connect to it with:
292
292
293 .. sourcecode:: ipython
293 .. sourcecode:: ipython
294
294
295 In [2]: c = Client('/path/to/my/ipcontroller-client.json', sshserver='me@myhub.example.com')
295 In [2]: c = Client('/path/to/my/ipcontroller-client.json', sshserver='me@myhub.example.com')
296
296
297 Where 'myhub.example.com' is the url or IP address of the machine on
297 Where 'myhub.example.com' is the url or IP address of the machine on
298 which the Hub process is running (or another machine that has direct access to the Hub's ports).
298 which the Hub process is running (or another machine that has direct access to the Hub's ports).
299
299
300 The SSH server may already be specified in ipcontroller-client.json, if the controller was
300 The SSH server may already be specified in ipcontroller-client.json, if the controller was
301 instructed at its launch time.
301 instructed at its launch time.
302
302
303 You are now ready to learn more about the :ref:`Direct
303 You are now ready to learn more about the :ref:`Direct
304 <parallel_multiengine>` and :ref:`LoadBalanced <parallel_task>` interfaces to the
304 <parallel_multiengine>` and :ref:`LoadBalanced <parallel_task>` interfaces to the
305 controller.
305 controller.
306
306
307 .. [ZeroMQ] ZeroMQ. http://www.zeromq.org
307 .. [ZeroMQ] ZeroMQ. http://www.zeromq.org
General Comments 0
You need to be logged in to leave comments. Login now