##// END OF EJS Templates
add label to the two-process section...
Paul Ivanov -
Show More
@@ -1,269 +1,271 b''
1 .. _overview:
1 .. _overview:
2
2
3 ============
3 ============
4 Introduction
4 Introduction
5 ============
5 ============
6
6
7 Overview
7 Overview
8 ========
8 ========
9
9
10 One of Python's most useful features is its interactive interpreter.
10 One of Python's most useful features is its interactive interpreter.
11 This system allows very fast testing of ideas without the overhead of
11 This system allows very fast testing of ideas without the overhead of
12 creating test files as is typical in most programming languages.
12 creating test files as is typical in most programming languages.
13 However, the interpreter supplied with the standard Python distribution
13 However, the interpreter supplied with the standard Python distribution
14 is somewhat limited for extended interactive use.
14 is somewhat limited for extended interactive use.
15
15
16 The goal of IPython is to create a comprehensive environment for
16 The goal of IPython is to create a comprehensive environment for
17 interactive and exploratory computing. To support this goal, IPython
17 interactive and exploratory computing. To support this goal, IPython
18 has three main components:
18 has three main components:
19
19
20 * An enhanced interactive Python shell.
20 * An enhanced interactive Python shell.
21 * A decoupled two-process communication model, which allows for multiple
21 * A decoupled two-process communication model, which allows for multiple
22 clients to connect to a computation kernel, most notably the web-based
22 clients to connect to a computation kernel, most notably the web-based
23 :ref:`notebook <htmlnotebook>`
23 :ref:`notebook <htmlnotebook>`
24 * An architecture for interactive parallel computing.
24 * An architecture for interactive parallel computing.
25
25
26 All of IPython is open source (released under the revised BSD license).
26 All of IPython is open source (released under the revised BSD license).
27
27
28 Enhanced interactive Python shell
28 Enhanced interactive Python shell
29 =================================
29 =================================
30
30
31 IPython's interactive shell (:command:`ipython`), has the following goals,
31 IPython's interactive shell (:command:`ipython`), has the following goals,
32 amongst others:
32 amongst others:
33
33
34 1. Provide an interactive shell superior to Python's default. IPython
34 1. Provide an interactive shell superior to Python's default. IPython
35 has many features for object introspection, system shell access,
35 has many features for object introspection, system shell access,
36 and its own special command system for adding functionality when
36 and its own special command system for adding functionality when
37 working interactively. It tries to be a very efficient environment
37 working interactively. It tries to be a very efficient environment
38 both for Python code development and for exploration of problems
38 both for Python code development and for exploration of problems
39 using Python objects (in situations like data analysis).
39 using Python objects (in situations like data analysis).
40
40
41 2. Serve as an embeddable, ready to use interpreter for your own
41 2. Serve as an embeddable, ready to use interpreter for your own
42 programs. IPython can be started with a single call from inside
42 programs. IPython can be started with a single call from inside
43 another program, providing access to the current namespace. This
43 another program, providing access to the current namespace. This
44 can be very useful both for debugging purposes and for situations
44 can be very useful both for debugging purposes and for situations
45 where a blend of batch-processing and interactive exploration are
45 where a blend of batch-processing and interactive exploration are
46 needed. New in the 0.9 version of IPython is a reusable wxPython
46 needed. New in the 0.9 version of IPython is a reusable wxPython
47 based IPython widget.
47 based IPython widget.
48
48
49 3. Offer a flexible framework which can be used as the base
49 3. Offer a flexible framework which can be used as the base
50 environment for other systems with Python as the underlying
50 environment for other systems with Python as the underlying
51 language. Specifically scientific environments like Mathematica,
51 language. Specifically scientific environments like Mathematica,
52 IDL and Matlab inspired its design, but similar ideas can be
52 IDL and Matlab inspired its design, but similar ideas can be
53 useful in many fields.
53 useful in many fields.
54
54
55 4. Allow interactive testing of threaded graphical toolkits. IPython
55 4. Allow interactive testing of threaded graphical toolkits. IPython
56 has support for interactive, non-blocking control of GTK, Qt and
56 has support for interactive, non-blocking control of GTK, Qt and
57 WX applications via special threading flags. The normal Python
57 WX applications via special threading flags. The normal Python
58 shell can only do this for Tkinter applications.
58 shell can only do this for Tkinter applications.
59
59
60 Main features of the interactive shell
60 Main features of the interactive shell
61 --------------------------------------
61 --------------------------------------
62
62
63 * Dynamic object introspection. One can access docstrings, function
63 * Dynamic object introspection. One can access docstrings, function
64 definition prototypes, source code, source files and other details
64 definition prototypes, source code, source files and other details
65 of any object accessible to the interpreter with a single
65 of any object accessible to the interpreter with a single
66 keystroke (:samp:`?`, and using :samp:`??` provides additional detail).
66 keystroke (:samp:`?`, and using :samp:`??` provides additional detail).
67
67
68 * Searching through modules and namespaces with :samp:`*` wildcards, both
68 * Searching through modules and namespaces with :samp:`*` wildcards, both
69 when using the :samp:`?` system and via the :samp:`%psearch` command.
69 when using the :samp:`?` system and via the :samp:`%psearch` command.
70
70
71 * Completion in the local namespace, by typing :kbd:`TAB` at the prompt.
71 * Completion in the local namespace, by typing :kbd:`TAB` at the prompt.
72 This works for keywords, modules, methods, variables and files in the
72 This works for keywords, modules, methods, variables and files in the
73 current directory. This is supported via the readline library, and
73 current directory. This is supported via the readline library, and
74 full access to configuring readline's behavior is provided.
74 full access to configuring readline's behavior is provided.
75 Custom completers can be implemented easily for different purposes
75 Custom completers can be implemented easily for different purposes
76 (system commands, magic arguments etc.)
76 (system commands, magic arguments etc.)
77
77
78 * Numbered input/output prompts with command history (persistent
78 * Numbered input/output prompts with command history (persistent
79 across sessions and tied to each profile), full searching in this
79 across sessions and tied to each profile), full searching in this
80 history and caching of all input and output.
80 history and caching of all input and output.
81
81
82 * User-extensible 'magic' commands. A set of commands prefixed with
82 * User-extensible 'magic' commands. A set of commands prefixed with
83 :samp:`%` is available for controlling IPython itself and provides
83 :samp:`%` is available for controlling IPython itself and provides
84 directory control, namespace information and many aliases to
84 directory control, namespace information and many aliases to
85 common system shell commands.
85 common system shell commands.
86
86
87 * Alias facility for defining your own system aliases.
87 * Alias facility for defining your own system aliases.
88
88
89 * Complete system shell access. Lines starting with :samp:`!` are passed
89 * Complete system shell access. Lines starting with :samp:`!` are passed
90 directly to the system shell, and using :samp:`!!` or :samp:`var = !cmd`
90 directly to the system shell, and using :samp:`!!` or :samp:`var = !cmd`
91 captures shell output into python variables for further use.
91 captures shell output into python variables for further use.
92
92
93 * Background execution of Python commands in a separate thread.
93 * Background execution of Python commands in a separate thread.
94 IPython has an internal job manager called jobs, and a
94 IPython has an internal job manager called jobs, and a
95 convenience backgrounding magic function called :samp:`%bg`.
95 convenience backgrounding magic function called :samp:`%bg`.
96
96
97 * The ability to expand python variables when calling the system shell. In a
97 * The ability to expand python variables when calling the system shell. In a
98 shell command, any python variable prefixed with :samp:`$` is expanded. A
98 shell command, any python variable prefixed with :samp:`$` is expanded. A
99 double :samp:`$$` allows passing a literal :samp:`$` to the shell (for access
99 double :samp:`$$` allows passing a literal :samp:`$` to the shell (for access
100 to shell and environment variables like :envvar:`PATH`).
100 to shell and environment variables like :envvar:`PATH`).
101
101
102 * Filesystem navigation, via a magic :samp:`%cd` command, along with a
102 * Filesystem navigation, via a magic :samp:`%cd` command, along with a
103 persistent bookmark system (using :samp:`%bookmark`) for fast access to
103 persistent bookmark system (using :samp:`%bookmark`) for fast access to
104 frequently visited directories.
104 frequently visited directories.
105
105
106 * A lightweight persistence framework via the :samp:`%store` command, which
106 * A lightweight persistence framework via the :samp:`%store` command, which
107 allows you to save arbitrary Python variables. These get restored
107 allows you to save arbitrary Python variables. These get restored
108 automatically when your session restarts.
108 automatically when your session restarts.
109
109
110 * Automatic indentation (optional) of code as you type (through the
110 * Automatic indentation (optional) of code as you type (through the
111 readline library).
111 readline library).
112
112
113 * Macro system for quickly re-executing multiple lines of previous
113 * Macro system for quickly re-executing multiple lines of previous
114 input with a single name. Macros can be stored persistently via
114 input with a single name. Macros can be stored persistently via
115 :samp:`%store` and edited via :samp:`%edit`.
115 :samp:`%store` and edited via :samp:`%edit`.
116
116
117 * Session logging (you can then later use these logs as code in your
117 * Session logging (you can then later use these logs as code in your
118 programs). Logs can optionally timestamp all input, and also store
118 programs). Logs can optionally timestamp all input, and also store
119 session output (marked as comments, so the log remains valid
119 session output (marked as comments, so the log remains valid
120 Python source code).
120 Python source code).
121
121
122 * Session restoring: logs can be replayed to restore a previous
122 * Session restoring: logs can be replayed to restore a previous
123 session to the state where you left it.
123 session to the state where you left it.
124
124
125 * Verbose and colored exception traceback printouts. Easier to parse
125 * Verbose and colored exception traceback printouts. Easier to parse
126 visually, and in verbose mode they produce a lot of useful
126 visually, and in verbose mode they produce a lot of useful
127 debugging information (basically a terminal version of the cgitb
127 debugging information (basically a terminal version of the cgitb
128 module).
128 module).
129
129
130 * Auto-parentheses: callable objects can be executed without
130 * Auto-parentheses: callable objects can be executed without
131 parentheses: :samp:`sin 3` is automatically converted to :samp:`sin(3)`.
131 parentheses: :samp:`sin 3` is automatically converted to :samp:`sin(3)`.
132
132
133 * Auto-quoting: using :samp:`,`, or :samp:`;` as the first character forces
133 * Auto-quoting: using :samp:`,`, or :samp:`;` as the first character forces
134 auto-quoting of the rest of the line: :samp:`,my_function a b` becomes
134 auto-quoting of the rest of the line: :samp:`,my_function a b` becomes
135 automatically :samp:`my_function("a","b")`, while :samp:`;my_function a b`
135 automatically :samp:`my_function("a","b")`, while :samp:`;my_function a b`
136 becomes :samp:`my_function("a b")`.
136 becomes :samp:`my_function("a b")`.
137
137
138 * Extensible input syntax. You can define filters that pre-process
138 * Extensible input syntax. You can define filters that pre-process
139 user input to simplify input in special situations. This allows
139 user input to simplify input in special situations. This allows
140 for example pasting multi-line code fragments which start with
140 for example pasting multi-line code fragments which start with
141 :samp:`>>>` or :samp:`...` such as those from other python sessions or the
141 :samp:`>>>` or :samp:`...` such as those from other python sessions or the
142 standard Python documentation.
142 standard Python documentation.
143
143
144 * Flexible configuration system. It uses a configuration file which
144 * Flexible configuration system. It uses a configuration file which
145 allows permanent setting of all command-line options, module
145 allows permanent setting of all command-line options, module
146 loading, code and file execution. The system allows recursive file
146 loading, code and file execution. The system allows recursive file
147 inclusion, so you can have a base file with defaults and layers
147 inclusion, so you can have a base file with defaults and layers
148 which load other customizations for particular projects.
148 which load other customizations for particular projects.
149
149
150 * Embeddable. You can call IPython as a python shell inside your own
150 * Embeddable. You can call IPython as a python shell inside your own
151 python programs. This can be used both for debugging code or for
151 python programs. This can be used both for debugging code or for
152 providing interactive abilities to your programs with knowledge
152 providing interactive abilities to your programs with knowledge
153 about the local namespaces (very useful in debugging and data
153 about the local namespaces (very useful in debugging and data
154 analysis situations).
154 analysis situations).
155
155
156 * Easy debugger access. You can set IPython to call up an enhanced version of
156 * Easy debugger access. You can set IPython to call up an enhanced version of
157 the Python debugger (pdb) every time there is an uncaught exception. This
157 the Python debugger (pdb) every time there is an uncaught exception. This
158 drops you inside the code which triggered the exception with all the data
158 drops you inside the code which triggered the exception with all the data
159 live and it is possible to navigate the stack to rapidly isolate the source
159 live and it is possible to navigate the stack to rapidly isolate the source
160 of a bug. The :samp:`%run` magic command (with the :samp:`-d` option) can run
160 of a bug. The :samp:`%run` magic command (with the :samp:`-d` option) can run
161 any script under pdb's control, automatically setting initial breakpoints for
161 any script under pdb's control, automatically setting initial breakpoints for
162 you. This version of pdb has IPython-specific improvements, including
162 you. This version of pdb has IPython-specific improvements, including
163 tab-completion and traceback coloring support. For even easier debugger
163 tab-completion and traceback coloring support. For even easier debugger
164 access, try :samp:`%debug` after seeing an exception. winpdb is also
164 access, try :samp:`%debug` after seeing an exception. winpdb is also
165 supported, see ipy_winpdb extension.
165 supported, see ipy_winpdb extension.
166
166
167 * Profiler support. You can run single statements (similar to
167 * Profiler support. You can run single statements (similar to
168 :samp:`profile.run()`) or complete programs under the profiler's control.
168 :samp:`profile.run()`) or complete programs under the profiler's control.
169 While this is possible with standard cProfile or profile modules,
169 While this is possible with standard cProfile or profile modules,
170 IPython wraps this functionality with magic commands (see :samp:`%prun`
170 IPython wraps this functionality with magic commands (see :samp:`%prun`
171 and :samp:`%run -p`) convenient for rapid interactive work.
171 and :samp:`%run -p`) convenient for rapid interactive work.
172
172
173 * Doctest support. The special :samp:`%doctest_mode` command toggles a mode
173 * Doctest support. The special :samp:`%doctest_mode` command toggles a mode
174 that allows you to paste existing doctests (with leading :samp:`>>>`
174 that allows you to paste existing doctests (with leading :samp:`>>>`
175 prompts and whitespace) and uses doctest-compatible prompts and
175 prompts and whitespace) and uses doctest-compatible prompts and
176 output, so you can use IPython sessions as doctest code.
176 output, so you can use IPython sessions as doctest code.
177
177
178 .. _ipythonzmq:
179
178 Decoupled two-process model
180 Decoupled two-process model
179 ==============================
181 ==============================
180
182
181 IPython has abstracted and extended the notion of a traditional
183 IPython has abstracted and extended the notion of a traditional
182 *Read-Evaluate-Print Loop* (REPL) environment by decoupling the *evaluation*
184 *Read-Evaluate-Print Loop* (REPL) environment by decoupling the *evaluation*
183 into its own process. We call this process a kernel: it receives execution
185 into its own process. We call this process a kernel: it receives execution
184 instructions from clients and communicates the results back to them.
186 instructions from clients and communicates the results back to them.
185
187
186 This decoupling allows us to have several clients connected to the same
188 This decoupling allows us to have several clients connected to the same
187 kernel, and even allows clients and kernels to live on different machines.
189 kernel, and even allows clients and kernels to live on different machines.
188 With the exclusion of the traditional single process terminal-based IPython
190 With the exclusion of the traditional single process terminal-based IPython
189 (what you start if you run ``ipython`` without any subcommands), all
191 (what you start if you run ``ipython`` without any subcommands), all
190 other IPython machinery uses this two-process model. This includes ``ipython
192 other IPython machinery uses this two-process model. This includes ``ipython
191 console``, ``ipython qtconsole``, and ``ipython notebook``.
193 console``, ``ipython qtconsole``, and ``ipython notebook``.
192
194
193 As an example, this means that when you start ``ipython qtconsole``, you're
195 As an example, this means that when you start ``ipython qtconsole``, you're
194 really starting two processes, a kernel and a Qt-based client can send
196 really starting two processes, a kernel and a Qt-based client can send
195 commands to and receive results from that kernel. If there is already a kernel
197 commands to and receive results from that kernel. If there is already a kernel
196 running that you want to connect to, you can pass the ``--existing`` flag
198 running that you want to connect to, you can pass the ``--existing`` flag
197 which will skip initiating a new kernel and connect to the most recent kernel,
199 which will skip initiating a new kernel and connect to the most recent kernel,
198 instead. To connect to a specific kernel once you have several kernels
200 instead. To connect to a specific kernel once you have several kernels
199 running, use the ``%connect_info`` magic to get the unique connection file,
201 running, use the ``%connect_info`` magic to get the unique connection file,
200 which will be something like ``--existing kernel-19732.json`` but with
202 which will be something like ``--existing kernel-19732.json`` but with
201 different numbers which correspond to the Process ID of the kernel.
203 different numbers which correspond to the Process ID of the kernel.
202
204
203 You can read more about using :ref:`ipython qtconsole <qtconsole>`, and
205 You can read more about using :ref:`ipython qtconsole <qtconsole>`, and
204 :ref:`ipython notebook <htmlnotebook>`. There is also a :ref:`message spec
206 :ref:`ipython notebook <htmlnotebook>`. There is also a :ref:`message spec
205 <messaging>` which documents the protocol for communication between kernels
207 <messaging>` which documents the protocol for communication between kernels
206 and clients.
208 and clients.
207
209
208
210
209 Interactive parallel computing
211 Interactive parallel computing
210 ==============================
212 ==============================
211
213
212 Increasingly, parallel computer hardware, such as multicore CPUs, clusters and
214 Increasingly, parallel computer hardware, such as multicore CPUs, clusters and
213 supercomputers, is becoming ubiquitous. Over the last 8 years, we have
215 supercomputers, is becoming ubiquitous. Over the last 8 years, we have
214 developed an architecture within IPython that allows such hardware to be used
216 developed an architecture within IPython that allows such hardware to be used
215 quickly and easily from Python. Moreover, this architecture is designed to
217 quickly and easily from Python. Moreover, this architecture is designed to
216 support interactive and collaborative parallel computing.
218 support interactive and collaborative parallel computing.
217
219
218 The main features of this system are:
220 The main features of this system are:
219
221
220 * Quickly parallelize Python code from an interactive Python/IPython session.
222 * Quickly parallelize Python code from an interactive Python/IPython session.
221
223
222 * A flexible and dynamic process model that be deployed on anything from
224 * A flexible and dynamic process model that be deployed on anything from
223 multicore workstations to supercomputers.
225 multicore workstations to supercomputers.
224
226
225 * An architecture that supports many different styles of parallelism, from
227 * An architecture that supports many different styles of parallelism, from
226 message passing to task farming. And all of these styles can be handled
228 message passing to task farming. And all of these styles can be handled
227 interactively.
229 interactively.
228
230
229 * Both blocking and fully asynchronous interfaces.
231 * Both blocking and fully asynchronous interfaces.
230
232
231 * High level APIs that enable many things to be parallelized in a few lines
233 * High level APIs that enable many things to be parallelized in a few lines
232 of code.
234 of code.
233
235
234 * Write parallel code that will run unchanged on everything from multicore
236 * Write parallel code that will run unchanged on everything from multicore
235 workstations to supercomputers.
237 workstations to supercomputers.
236
238
237 * Full integration with Message Passing libraries (MPI).
239 * Full integration with Message Passing libraries (MPI).
238
240
239 * Capabilities based security model with full encryption of network connections.
241 * Capabilities based security model with full encryption of network connections.
240
242
241 * Share live parallel jobs with other users securely. We call this
243 * Share live parallel jobs with other users securely. We call this
242 collaborative parallel computing.
244 collaborative parallel computing.
243
245
244 * Dynamically load balanced task farming system.
246 * Dynamically load balanced task farming system.
245
247
246 * Robust error handling. Python exceptions raised in parallel execution are
248 * Robust error handling. Python exceptions raised in parallel execution are
247 gathered and presented to the top-level code.
249 gathered and presented to the top-level code.
248
250
249 For more information, see our :ref:`overview <parallel_index>` of using IPython
251 For more information, see our :ref:`overview <parallel_index>` of using IPython
250 for parallel computing.
252 for parallel computing.
251
253
252 Portability and Python requirements
254 Portability and Python requirements
253 -----------------------------------
255 -----------------------------------
254
256
255 As of the 0.11 release, IPython works with Python 2.6 and 2.7. Versions 0.9 and
257 As of the 0.11 release, IPython works with Python 2.6 and 2.7. Versions 0.9 and
256 0.10 worked with Python 2.4 and above. IPython now also supports Python 3,
258 0.10 worked with Python 2.4 and above. IPython now also supports Python 3,
257 although for now the code for this is separate, and kept up to date with the
259 although for now the code for this is separate, and kept up to date with the
258 main IPython repository. In the future, these will converge to a single codebase
260 main IPython repository. In the future, these will converge to a single codebase
259 which can be automatically translated using 2to3.
261 which can be automatically translated using 2to3.
260
262
261 IPython is known to work on the following operating systems:
263 IPython is known to work on the following operating systems:
262
264
263 * Linux
265 * Linux
264 * Most other Unix-like OSs (AIX, Solaris, BSD, etc.)
266 * Most other Unix-like OSs (AIX, Solaris, BSD, etc.)
265 * Mac OS X
267 * Mac OS X
266 * Windows (CygWin, XP, Vista, etc.)
268 * Windows (CygWin, XP, Vista, etc.)
267
269
268 See :ref:`here <install_index>` for instructions on how to install IPython.
270 See :ref:`here <install_index>` for instructions on how to install IPython.
269
271
General Comments 0
You need to be logged in to leave comments. Login now