upstream/ipython Commit - r21636:ec0fabd0

remove a few more docs that have been moved to other repos

Min RK -

r21636:ec0fabd0

parent child

Collapse all files

The requested changes are too big and content was truncated. Show full diff

docs/source/conf.py

0 +1 0

                                    'rpy2': ('http://rpy.sourceforge.net/rpy2/doc-2.4/html/', None),
                                    'traitlets': ('http://traitlets.readthedocs.org/en/latest/', None),
                                    'jupyterclient': ('http://jupyter-client.readthedocs.org/en/latest/', None),
+                                   'ipyparallel': ('http://ipyparallel.readthedocs.org/en/latest/', None),
                                   }
             # Options for LaTeX output

docs/source/development/how_ipython_works.rst

0 +2 -43

@@ -25,7 +25,7 b' the terminal, and third party interfaces\xe2\x80\x94use the IPython Kernel. This is a'
25	separate process which is responsible for running user code, and things like	25	separate process which is responsible for running user code, and things like
26	computing possible completions. Frontends communicate with it using JSON	26	computing possible completions. Frontends communicate with it using JSON
27	messages sent over `ZeroMQ <http://zeromq.org/>`_ sockets; the protocol they use is described in	27	messages sent over `ZeroMQ <http://zeromq.org/>`_ sockets; the protocol they use is described in
28	:doc:`messaging`.	28	:ref:`jupyterclient:messaging`.
29		29
30	The core execution machinery for the kernel is shared with terminal IPython:	30	The core execution machinery for the kernel is shared with terminal IPython:
31		31
@@ -57,48 +57,7 b' likely to be better maintained by the community using them, like'
57		57
58	.. seealso::	58	.. seealso::
59		59
60	:doc:`kernels`	60	:ref:`jupyterclient:kernels`
61		61
62	:doc:`wrapperkernels`	62	:doc:`wrapperkernels`
63		63
64	Notebooks
65	---------
66
67	The Notebook frontend does something extra. In addition to running your code, it
68	stores code and output, together with markdown notes, in an editable document
69	called a notebook. When you save it, this is sent from your browser to the
70	notebook server, which saves it on disk as a JSON file with a ``.ipynb``
71	extension.
72
73	.. image:: figs/notebook_components.png
74
75	The notebook server, not the kernel, is responsible for saving and loading
76	notebooks, so you can edit notebooks even if you don't have the kernel for that
77	language—you just won't be able to run code. The kernel doesn't know anything
78	about the notebook document: it just gets sent cells of code to execute when the
79	user runs them.
80
81	Exporting to other formats
82	``````````````````````````
83
84	The Nbconvert tool in IPython converts notebook files to other formats, such as
85	HTML, LaTeX, or reStructuredText. This conversion goes through a series of steps:
86
87	.. image:: figs/nbconvert.png
88
89	1. Preprocessors modify the notebook in memory. E.g. ExecutePreprocessor runs
90	the code in the notebook and updates the output.
91	2. An exporter converts the notebook to another file format. Most of the
92	exporters use templates for this.
93	3. Postprocessors work on the file produced by exporting.
94
95	The `nbviewer <http://nbviewer.ipython.org/>`_ website uses nbconvert with the
96	HTML exporter. When you give it a URL, it fetches the notebook from that URL,
97	converts it to HTML, and serves that HTML to you.
98
99	IPython.parallel
100	----------------
101
102	IPython also includes a parallel computing framework, ``IPython.parallel``. This
103	allows you to control many individual engines, which are an extended version of
104	the IPython kernel described above. For more details, see :doc:`/parallel/index`.

docs/source/development/index.rst

0 0 -3

                :maxdepth: 1
                how_ipython_works
-               kernels
                wrapperkernels
                execution
-               parallel_messages
-               parallel_connections
                lexer
                pycompat
                config

docs/source/development/kernels.rst

0 +4 -139

@@ -1,143 +1,8 b''
		1	:orphan:
		2
1	==========================	3	==========================
2	Making kernels for IPython	4	Making kernels for IPython
3	==========================	5	==========================
4		6
5	A 'kernel' is a program that runs and introspects the user's code. IPython	7	Kernels are now part of Jupyter - see
6	includes a kernel for Python code, and people have written kernels for	8	:ref:`jupyterclient:kernels` for the documentation.
7	`several other languages <https://github.com/ipython/ipython/wiki/Projects-using-IPython#list-of-some-ipython-compatible-kernels>`_.
8
9	When IPython starts a kernel, it passes it a connection file. This specifies
10	how to set up communications with the frontend.
11
12	There are two options for writing a kernel:
13
14	1. You can reuse the IPython kernel machinery to handle the communications, and
15	just describe how to execute your code. This is much simpler if the target
16	language can be driven from Python. See :doc:`wrapperkernels` for details.
17	2. You can implement the kernel machinery in your target language. This is more
18	work initially, but the people using your kernel might be more likely to
19	contribute to it if it's in the language they know.
20
21	Connection files
22	================
23
24	Your kernel will be given the path to a connection file when it starts (see
25	:ref:`kernelspecs` for how to specify the command line arguments for your kernel).
26	This file, which is accessible only to the current user, will contain a JSON
27	dictionary looking something like this::
28
29	{
30	"control_port": 50160,
31	"shell_port": 57503,
32	"transport": "tcp",
33	"signature_scheme": "hmac-sha256",
34	"stdin_port": 52597,
35	"hb_port": 42540,
36	"ip": "127.0.0.1",
37	"iopub_port": 40885,
38	"key": "a0436f6c-1916-498b-8eb9-e81ab9368e84"
39	}
40
41	The ``transport``, ``ip`` and five ``_port`` fields specify five ports which the
42	kernel should bind to using `ZeroMQ <http://zeromq.org/>`_. For instance, the
43	address of the shell socket in the example above would be::
44
45	tcp://127.0.0.1:57503
46
47	New ports are chosen at random for each kernel started.
48
49	``signature_scheme`` and ``key`` are used to cryptographically sign messages, so
50	that other users on the system can't send code to run in this kernel. See
51	:ref:`wire_protocol` for the details of how this signature is calculated.
52
53	Handling messages
54	=================
55
56	After reading the connection file and binding to the necessary sockets, the
57	kernel should go into an event loop, listening on the hb (heartbeat), control
58	and shell sockets.
59
60	:ref:`Heartbeat <kernel_heartbeat>` messages should be echoed back immediately
61	on the same socket - the frontend uses this to check that the kernel is still
62	alive.
63
64	Messages on the control and shell sockets should be parsed, and their signature
65	validated. See :ref:`wire_protocol` for how to do this.
66
67	The kernel will send messages on the iopub socket to display output, and on the
68	stdin socket to prompt the user for textual input.
69
70	.. seealso::
71
72	:doc:`messaging`
73	Details of the different sockets and the messages that come over them
74
75	`Creating Language Kernels for IPython <http://andrew.gibiansky.com/blog/ipython/ipython-kernels/>`_
76	A blog post by the author of `IHaskell <https://github.com/gibiansky/IHaskell>`_,
77	a Haskell kernel
78
79	`simple_kernel <https://github.com/dsblank/simple_kernel>`_
80	A simple example implementation of the kernel machinery in Python
81
82
83	.. _kernelspecs:
84
85	Kernel specs
86	============
87
88	A kernel identifies itself to IPython by creating a directory, the name of which
89	is used as an identifier for the kernel. These may be created in a number of
90	locations:
91
92	+--------+--------------------------------------+-----------------------------------+
93	\| \| Unix \| Windows \|
94	+========+======================================+===================================+
95	\| System \| ``/usr/share/jupyter/kernels`` \| ``%PROGRAMDATA%\jupyter\kernels`` \|
96	\| \| \| \|
97	\| \| ``/usr/local/share/jupyter/kernels`` \| \|
98	+--------+--------------------------------------+-----------------------------------+
99	\| User \| ``~/.ipython/kernels`` \|
100	+--------+--------------------------------------+-----------------------------------+
101
102	The user location takes priority over the system locations, and the case of the
103	names is ignored, so selecting kernels works the same way whether or not the
104	filesystem is case sensitive.
105
106	Inside the directory, the most important file is kernel.json. This should be a
107	JSON serialised dictionary containing the following keys and values:
108
109	- argv: A list of command line arguments used to start the kernel. The text
110	``{connection_file}`` in any argument will be replaced with the path to the
111	connection file.
112	- display_name: The kernel's name as it should be displayed in the UI.
113	Unlike the kernel name used in the API, this can contain arbitrary unicode
114	characters.
115	- language: The name of the language of the kernel.
116	When loading notebooks, if no matching kernelspec key (may differ across machines)
117	is found, a kernel with a matching `language` will be used.
118	This allows a notebook written on any Python or Julia kernel to be properly associated
119	with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's.
120	- env (optional): A dictionary of environment variables to set for the kernel.
121	These will be added to the current environment variables before the kernel is
122	started.
123
124	For example, the kernel.json file for IPython looks like this::
125
126	{
127	"argv": ["python3", "-m", "IPython.kernel",
128	"-f", "{connection_file}"],
129	"display_name": "Python 3",
130	"language": "python"
131	}
132
133	To see the available kernel specs, run::
134
135	ipython kernelspec list
136
137	To start the terminal console or the Qt console with a specific kernel::
138
139	ipython console --kernel bash
140	ipython qtconsole --kernel bash
141
142	To use different kernels in the notebook, select a different kernel from the
143	dropdown menu in the top-right of the UI.

docs/source/development/parallel_connections.rst

0 +3 -149

@@ -1,154 +1,8 b''
1	.. _parallel_connections:	1	:orphan:
2		2
3	==============================================	3	==============================================
4	Connection Diagrams of The IPython ZMQ Cluster	4	Connection Diagrams of The IPython ZMQ Cluster
5	==============================================	5	==============================================
6		6
7	This is a quick summary and illustration of the connections involved in the ZeroMQ based	7	IPython parallel has moved to ipyparallel -
8	IPython cluster for parallel computing.	8	see :ref:`ipyparallel:parallel_connections` for the documentation.
9
10	All Connections
11	===============
12
13	The IPython cluster consists of a Controller, and one or more each of clients and engines.
14	The goal of the Controller is to manage and monitor the connections and communications
15	between the clients and the engines. The Controller is no longer a single process entity,
16	but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers.
17
18	It is important for security/practicality reasons that all connections be inbound to the
19	controller processes. The arrows in the figures indicate the direction of the
20	connection.
21
22
23	.. figure:: figs/allconnections.png
24	:width: 432px
25	:alt: IPython cluster connections
26	:align: center
27
28	All the connections involved in connecting one client to one engine.
29
30	The Controller consists of 1-5 processes. Central to the cluster is the Hub, which monitors
31	engine state, execution traffic, and handles registration and notification. The Hub includes a
32	Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
33	Schedulers. These devices are very small pure-C MonitoredQueue processes (or optionally
34	threads) that relay messages very fast, but also send a copy of each message along a side socket
35	to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
36	explicitly addressed messages from clients to engines, and their replies back up. The Balanced
37	queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
38	device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
39	devices, but with additional internal logic. stdout/err are also propagated from the Engines to
40	the clients via a PUB/SUB MonitoredQueue.
41
42
43	Registration
44	------------
45
46	.. figure:: figs/queryfade.png
47	:width: 432px
48	:alt: IPython Registration connections
49	:align: center
50
51	Engines and Clients only need to know where the Query ``ROUTER`` is located to start
52	connecting.
53
54	Once a controller is launched, the only information needed for connecting clients and/or
55	engines is the IP/port of the Hub's ``ROUTER`` socket called the Registrar. This socket
56	handles connections from both clients and engines, and replies with the remaining
57	information necessary to establish the remaining connections. Clients use this same socket for
58	querying the Hub for state information.
59
60	Heartbeat
61	---------
62
63	.. figure:: figs/hbfade.png
64	:width: 432px
65	:alt: IPython Heartbeat connections
66	:align: center
67
68	The heartbeat sockets.
69
70	The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
71	publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
72	``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``DEALER`` socket for output.
73	The ``SUB`` socket is connected to the ``PUB`` socket labeled ping, and the ``DEALER`` is
74	connected to the ``ROUTER`` labeled pong. This results in the same message being relayed
75	back to the Heartbeat Monitor with the addition of the ``DEALER`` prefix. The Heartbeat
76	Monitor receives all the replies via an ``ROUTER`` socket, and identifies which hearts are
77	still beating by the ``zmq.IDENTITY`` prefix of the ``DEALER`` sockets, which information
78	the Hub uses to notify clients of any changes in the available engines.
79
80	Schedulers
81	----------
82
83	.. figure:: figs/queuefade.png
84	:width: 432px
85	:alt: IPython Queue connections
86	:align: center
87
88	Control message scheduler on the left, execution (apply) schedulers on the right.
89
90	The controller has at least three Schedulers. These devices are primarily for
91	relaying messages between clients and engines, but the Hub needs to see those
92	messages for its own purposes. Since no Python code may exist between the two sockets in a
93	queue, all messages sent through these queues (both directions) are also sent via a
94	``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
95	interfering with it.
96
97	For tasks, the engine need not be specified. Messages sent to the ``ROUTER`` socket from the
98	client side are assigned to an engine via ZMQ's ``DEALER`` round-robin load balancing.
99	Engine replies are directed to specific clients via the IDENTITY of the client, which is
100	received as a prefix at the Engine.
101
102	For Multiplexing, ``ROUTER`` is used for both in and output sockets in the device. Clients must
103	specify the destination by the ``zmq.IDENTITY`` of the ``ROUTER`` socket connected to
104	the downstream end of the device.
105
106	At the Kernel level, both of these ``ROUTER`` sockets are treated in the same way as the ``REP``
107	socket in the serial version (except using ZMQStreams instead of explicit sockets).
108
109	Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
110	manner. The sockets on the Client and Engine are the same for these two actions, but the
111	scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
112	the upstream sockets in each MonitoredQueue.
113
114	IOPub
115	-----
116
117	.. figure:: figs/iopubfade.png
118	:width: 432px
119	:alt: IOPub connections
120	:align: center
121
122	stdout/err are published via a ``PUB/SUB`` MonitoredQueue
123
124
125	On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
126	sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
127	messages. They are then republished via another ``PUB`` socket, which can be
128	subscribed by the clients.
129
130	Client connections
131	------------------
132
133	.. figure:: figs/queryfade.png
134	:width: 432px
135	:alt: IPython client query connections
136	:align: center
137
138	Clients connect to an ``ROUTER`` socket to query the hub.
139
140	The hub's registrar ``ROUTER`` socket also listens for queries from clients as to queue status,
141	and control instructions. Clients connect to this socket via an ``DEALER`` during registration.
142
143	.. figure:: figs/notiffade.png
144	:width: 432px
145	:alt: IPython Registration connections
146	:align: center
147
148	Engine registration events are published via a ``PUB`` socket.
149
150	The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
151	allows clients to stay up to date with what engines are available by subscribing to the
152	feed with a ``SUB`` socket. Other processes could selectively subscribe to just
153	registration or unregistration events.
154

docs/source/development/parallel_messages.rst

0 +4 -363

@@ -1,367 +1,8 b''
1	.. _parallel_messages:	1	:orphan:
2		2
		3	================================
3	Messaging for Parallel Computing	4	Messaging for Parallel Computing
4	================================	5	================================
5		6
6	This is an extension of the :ref:`messaging <messaging>` doc. Diagrams of the connections	7	IPython parallel has moved to ipyparallel -
7	can be found in the :ref:`parallel connections <parallel_connections>` doc.	8	see :ref:`ipyparallel:parallel_messages` for the documentation.
8
9
10	ZMQ messaging is also used in the parallel computing IPython system. All messages to/from
11	kernels remain the same as the single kernel model, and are forwarded through a ZMQ Queue
12	device. The controller receives all messages and replies in these channels, and saves
13	results for future use.
14
15	The Controller
16	--------------
17
18	The controller is the central collection of processes in the IPython parallel computing
19	model. It has two major components:
20
21	* The Hub
22	* A collection of Schedulers
23
24	The Hub
25	-------
26
27	The Hub is the central process for monitoring the state of the engines, and all task
28	requests and results. It has no role in execution and does no relay of messages, so
29	large blocking requests or database actions in the Hub do not have the ability to impede
30	job submission and results.
31
32	Registration (``ROUTER``)
33	*************************
34
35	The first function of the Hub is to facilitate and monitor connections of clients
36	and engines. Both client and engine registration are handled by the same socket, so only
37	one ip/port pair is needed to connect any number of connections and clients.
38
39	Engines register with the ``zmq.IDENTITY`` of their two ``DEALER`` sockets, one for the
40	queue, which receives execute requests, and one for the heartbeat, which is used to
41	monitor the survival of the Engine process.
42
43	Message type: ``registration_request``::
44
45	content = {
46	'uuid' : 'abcd-1234-...', # the zmq.IDENTITY of the engine's sockets
47	}
48
49	.. note::
50
51	these are always the same, at least for now.
52
53	The Controller replies to an Engine's registration request with the engine's integer ID,
54	and all the remaining connection information for connecting the heartbeat process, and
55	kernel queue socket(s). The message status will be an error if the Engine requests IDs that
56	already in use.
57
58	Message type: ``registration_reply``::
59
60	content = {
61	'status' : 'ok', # or 'error'
62	# if ok:
63	'id' : 0, # int, the engine id
64	}
65
66	Clients use the same socket as engines to start their connections. Connection requests
67	from clients need no information:
68
69	Message type: ``connection_request``::
70
71	content = {}
72
73	The reply to a Client registration request contains the connection information for the
74	multiplexer and load balanced queues, as well as the address for direct hub
75	queries. If any of these addresses is `None`, that functionality is not available.
76
77	Message type: ``connection_reply``::
78
79	content = {
80	'status' : 'ok', # or 'error'
81	}
82
83	Heartbeat
84	*********
85
86	The hub uses a heartbeat system to monitor engines, and track when they become
87	unresponsive. As described in :ref:`messaging <messaging>`, and shown in :ref:`connections
88	<parallel_connections>`.
89
90	Notification (``PUB``)
91	**********************
92
93	The hub publishes all engine registration/unregistration events on a ``PUB`` socket.
94	This allows clients to have up-to-date engine ID sets without polling. Registration
95	notifications contain both the integer engine ID and the queue ID, which is necessary for
96	sending messages via the Multiplexer Queue and Control Queues.
97
98	Message type: ``registration_notification``::
99
100	content = {
101	'id' : 0, # engine ID that has been registered
102	'uuid' : 'engine_id' # the IDENT for the engine's sockets
103	}
104
105	Message type : ``unregistration_notification``::
106
107	content = {
108	'id' : 0 # engine ID that has been unregistered
109	'uuid' : 'engine_id' # the IDENT for the engine's sockets
110	}
111
112
113	Client Queries (``ROUTER``)
114	***************************
115
116	The hub monitors and logs all queue traffic, so that clients can retrieve past
117	results or monitor pending tasks. This information may reside in-memory on the Hub, or
118	on disk in a database (SQLite and MongoDB are currently supported). These requests are
119	handled by the same socket as registration.
120
121
122	:func:`queue_request` requests can specify multiple engines to query via the `targets`
123	element. A verbose flag can be passed, to determine whether the result should be the list
124	of `msg_ids` in the queue or simply the length of each list.
125
126	Message type: ``queue_request``::
127
128	content = {
129	'verbose' : True, # whether return should be lists themselves or just lens
130	'targets' : [0,3,1] # list of ints
131	}
132
133	The content of a reply to a :func:`queue_request` request is a dict, keyed by the engine
134	IDs. Note that they will be the string representation of the integer keys, since JSON
135	cannot handle number keys. The three keys of each dict are::
136
137	'completed' : messages submitted via any queue that ran on the engine
138	'queue' : jobs submitted via MUX queue, whose results have not been received
139	'tasks' : tasks that are known to have been submitted to the engine, but
140	have not completed. Note that with the pure zmq scheduler, this will
141	always be 0/[].
142
143	Message type: ``queue_reply``::
144
145	content = {
146	'status' : 'ok', # or 'error'
147	# if verbose=False:
148	'0' : {'completed' : 1, 'queue' : 7, 'tasks' : 0},
149	# if verbose=True:
150	'1' : {'completed' : ['abcd-...','1234-...'], 'queue' : ['58008-'], 'tasks' : []},
151	}
152
153	Clients can request individual results directly from the hub. This is primarily for
154	gathering results of executions not submitted by the requesting client, as the client
155	will have all its own results already. Requests are made by msg_id, and can contain one or
156	more msg_id. An additional boolean key 'statusonly' can be used to not request the
157	results, but simply poll the status of the jobs.
158
159	Message type: ``result_request``::
160
161	content = {
162	'msg_ids' : ['uuid','...'], # list of strs
163	'targets' : [1,2,3], # list of int ids or uuids
164	'statusonly' : False, # bool
165	}
166
167	The :func:`result_request` reply contains the content objects of the actual execution
168	reply messages. If `statusonly=True`, then there will be only the 'pending' and
169	'completed' lists.
170
171
172	Message type: ``result_reply``::
173
174	content = {
175	'status' : 'ok', # else error
176	# if ok:
177	'acbd-...' : msg, # the content dict is keyed by msg_ids,
178	# values are the result messages
179	# there will be none of these if `statusonly=True`
180	'pending' : ['msg_id','...'], # msg_ids still pending
181	'completed' : ['msg_id','...'], # list of completed msg_ids
182	}
183	buffers = ['bufs','...'] # the buffers that contained the results of the objects.
184	# this will be empty if no messages are complete, or if
185	# statusonly is True.
186
187	For memory management purposes, Clients can also instruct the hub to forget the
188	results of messages. This can be done by message ID or engine ID. Individual messages are
189	dropped by msg_id, and all messages completed on an engine are dropped by engine ID. This
190	may no longer be necessary with the mongodb-based message logging backend.
191
192	If the msg_ids element is the string ``'all'`` instead of a list, then all completed
193	results are forgotten.
194
195	Message type: ``purge_request``::
196
197	content = {
198	'msg_ids' : ['id1', 'id2',...], # list of msg_ids or 'all'
199	'engine_ids' : [0,2,4] # list of engine IDs
200	}
201
202	The reply to a purge request is simply the status 'ok' if the request succeeded, or an
203	explanation of why it failed, such as requesting the purge of a nonexistent or pending
204	message.
205
206	Message type: ``purge_reply``::
207
208	content = {
209	'status' : 'ok', # or 'error'
210	}
211
212
213	Schedulers
214	----------
215
216	There are three basic schedulers:
217
218	* Task Scheduler
219	* MUX Scheduler
220	* Control Scheduler
221
222	The MUX and Control schedulers are simple MonitoredQueue ØMQ devices, with ``ROUTER``
223	sockets on either side. This allows the queue to relay individual messages to particular
224	targets via ``zmq.IDENTITY`` routing. The Task scheduler may be a MonitoredQueue ØMQ
225	device, in which case the client-facing socket is ``ROUTER``, and the engine-facing socket
226	is ``DEALER``. The result of this is that client-submitted messages are load-balanced via
227	the ``DEALER`` socket, but the engine's replies to each message go to the requesting client.
228
229	Raw ``DEALER`` scheduling is quite primitive, and doesn't allow message introspection, so
230	there are also Python Schedulers that can be used. These Schedulers behave in much the
231	same way as a MonitoredQueue does from the outside, but have rich internal logic to
232	determine destinations, as well as handle dependency graphs Their sockets are always
233	``ROUTER`` on both sides.
234
235	The Python task schedulers have an additional message type, which informs the Hub of
236	the destination of a task as soon as that destination is known.
237
238	Message type: ``task_destination``::
239
240	content = {
241	'msg_id' : 'abcd-1234-...', # the msg's uuid
242	'engine_id' : '1234-abcd-...', # the destination engine's zmq.IDENTITY
243	}
244
245	:func:`apply`
246	*************
247
248	In terms of message classes, the MUX scheduler and Task scheduler relay the exact same
249	message types. Their only difference lies in how the destination is selected.
250
251	The `Namespace <http://gist.github.com/483294>`_ model suggests that execution be able to
252	use the model::
253
254	ns.apply(f, args, *kwargs)
255
256	which takes `f`, a function in the user's namespace, and executes ``f(args, *kwargs)``
257	on a remote engine, returning the result (or, for non-blocking, information facilitating
258	later retrieval of the result). This model, unlike the execute message which just uses a
259	code string, must be able to send arbitrary (pickleable) Python objects. And ideally, copy
260	as little data as we can. The `buffers` property of a Message was introduced for this
261	purpose.
262
263	Utility method :func:`build_apply_message` in :mod:`IPython.kernel.zmq.serialize` wraps a
264	function signature and builds a sendable buffer format for minimal data copying (exactly
265	zero copies of numpy array data or buffers or large strings).
266
267	Message type: ``apply_request``::
268
269	metadata = {
270	'after' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict()
271	'follow' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict()
272	}
273	content = {}
274	buffers = ['...'] # at least 3 in length
275	# as built by build_apply_message(f,args,kwargs)
276
277	after/follow represent task dependencies. 'after' corresponds to a time dependency. The
278	request will not arrive at an engine until the 'after' dependency tasks have completed.
279	'follow' corresponds to a location dependency. The task will be submitted to the same
280	engine as these msg_ids (see :class:`Dependency` docs for details).
281
282	Message type: ``apply_reply``::
283
284	content = {
285	'status' : 'ok' # 'ok' or 'error'
286	# other error info here, as in other messages
287	}
288	buffers = ['...'] # either 1 or 2 in length
289	# a serialization of the return value of f(args,*kwargs)
290	# only populated if status is 'ok'
291
292	All engine execution and data movement is performed via apply messages.
293
294	Control Messages
295	----------------
296
297	Messages that interact with the engines, but are not meant to execute code, are submitted
298	via the Control queue. These messages have high priority, and are thus received and
299	handled before any execution requests.
300
301	Clients may want to clear the namespace on the engine. There are no arguments nor
302	information involved in this request, so the content is empty.
303
304	Message type: ``clear_request``::
305
306	content = {}
307
308	Message type: ``clear_reply``::
309
310	content = {
311	'status' : 'ok' # 'ok' or 'error'
312	# other error info here, as in other messages
313	}
314
315	Clients may want to abort tasks that have not yet run. This can by done by message id, or
316	all enqueued messages can be aborted if None is specified.
317
318	Message type: ``abort_request``::
319
320	content = {
321	'msg_ids' : ['1234-...', '...'] # list of msg_ids or None
322	}
323
324	Message type: ``abort_reply``::
325
326	content = {
327	'status' : 'ok' # 'ok' or 'error'
328	# other error info here, as in other messages
329	}
330
331	The last action a client may want to do is shutdown the kernel. If a kernel receives a
332	shutdown request, then it aborts all queued messages, replies to the request, and exits.
333
334	Message type: ``shutdown_request``::
335
336	content = {}
337
338	Message type: ``shutdown_reply``::
339
340	content = {
341	'status' : 'ok' # 'ok' or 'error'
342	# other error info here, as in other messages
343	}
344
345
346	Implementation
347	--------------
348
349	There are a few differences in implementation between the `StreamSession` object used in
350	the newparallel branch and the `Session` object, the main one being that messages are
351	sent in parts, rather than as a single serialized object. `StreamSession` objects also
352	take pack/unpack functions, which are to be used when serializing/deserializing objects.
353	These can be any functions that translate to/from formats that ZMQ sockets can send
354	(buffers,bytes, etc.).
355
356	Split Sends
357	***********
358
359	Previously, messages were bundled as a single json object and one call to
360	:func:`socket.send_json`. Since the hub inspects all messages, and doesn't need to
361	see the content of the messages, which can be large, messages are now serialized and sent in
362	pieces. All messages are sent in at least 4 parts: the header, the parent header, the metadata and the content.
363	This allows the controller to unpack and inspect the (always small) header,
364	without spending time unpacking the content unless the message is bound for the
365	controller. Buffers are added on to the end of the message, and can be any objects that
366	present the buffer interface.
367

docs/source/development/wrapperkernels.rst

0 +5 -5

             Required steps
             --------------
-            Subclass :class:`IPython.kernel.zmq.kernelbase.Kernel`, and implement the
+            Subclass :class:`ipykernel.kernelbase.Kernel`, and implement the
             following methods and attributes:
             .. class:: MyKernel
                  Your method should return a dict containing the fields described in
                  :ref:`execution_results`. To display output, it can send messages
-                 using :meth:`~IPython.kernel.zmq.kernelbase.Kernel.send_response`.
+                 using :meth:`~ipykernel.kernelbase.Kernel.send_response`.
                  See :doc:`messaging` for details of the different message types.
             To launch your kernel, add this at the end of your module::
                 if __name__ == '__main__':
-                    from IPython.kernel.zmq.kernelapp import IPKernelApp
+                    from ipykernel.kernelapp import IPKernelApp
                     IPKernelApp.launch_instance(kernel_class=MyKernel)
             Example
             ``echokernel.py`` will simply echo any input it's given to stdout::
-                from IPython.kernel.zmq.kernelbase import Kernel
+                from ipykernel.kernelbase import Kernel
                 class EchoKernel(Kernel):
                     implementation = 'Echo'
                                }
                 if __name__ == '__main__':
-                    from IPython.kernel.zmq.kernelapp import IPKernelApp
+                    from ipykernel.kernelapp import IPKernelApp
                     IPKernelApp.launch_instance(kernel_class=EchoKernel)
             Here's the Kernel spec ``kernel.json`` file for this::

docs/source/development/figs/allconnections.png

0 removed binary 0 0