upstream/ipython Commit - r21636:ec0fabd0

remove a few more docs that have been moved to other repos

Min RK -

r21636:ec0fabd0

parent child

docs/source/conf.py

0 +1 0

                                     'rpy2': ('http://rpy.sourceforge.net/rpy2/doc-2.4/html/', None),
                                     'traitlets': ('http://traitlets.readthedocs.org/en/latest/', None),
                                     'jupyterclient': ('http://jupyter-client.readthedocs.org/en/latest/', None),
+                                    'ipyparallel': ('http://ipyparallel.readthedocs.org/en/latest/', None),
                                    }
              # Options for LaTeX output

docs/source/development/how_ipython_works.rst

0 +2 -43

		@@ -25,7 +25,7 b' the terminal, and third party interfaces\xe2\x80\x94use the IPython Kernel. This is a'
25	25	separate process which is responsible for running user code, and things like
26	26	computing possible completions. Frontends communicate with it using JSON
27	27	messages sent over `ZeroMQ <http://zeromq.org/>`_ sockets; the protocol they use is described in
28		:doc:`messaging`.
	28	:ref:`jupyterclient:messaging`.
29	29
30	30	The core execution machinery for the kernel is shared with terminal IPython:
31	31
		@@ -57,48 +57,7 b' likely to be better maintained by the community using them, like'
57	57
58	58	.. seealso::
59	59
60		:doc:`kernels`
	60	:ref:`jupyterclient:kernels`
61	61
62	62	:doc:`wrapperkernels`
63	63
64		Notebooks
65		---------
66
67		The Notebook frontend does something extra. In addition to running your code, it
68		stores code and output, together with markdown notes, in an editable document
69		called a notebook. When you save it, this is sent from your browser to the
70		notebook server, which saves it on disk as a JSON file with a ``.ipynb``
71		extension.
72
73		.. image:: figs/notebook_components.png
74
75		The notebook server, not the kernel, is responsible for saving and loading
76		notebooks, so you can edit notebooks even if you don't have the kernel for that
77		language—you just won't be able to run code. The kernel doesn't know anything
78		about the notebook document: it just gets sent cells of code to execute when the
79		user runs them.
80
81		Exporting to other formats
82		``````````````````````````
83
84		The Nbconvert tool in IPython converts notebook files to other formats, such as
85		HTML, LaTeX, or reStructuredText. This conversion goes through a series of steps:
86
87		.. image:: figs/nbconvert.png
88
89		1. Preprocessors modify the notebook in memory. E.g. ExecutePreprocessor runs
90		the code in the notebook and updates the output.
91		2. An exporter converts the notebook to another file format. Most of the
92		exporters use templates for this.
93		3. Postprocessors work on the file produced by exporting.
94
95		The `nbviewer <http://nbviewer.ipython.org/>`_ website uses nbconvert with the
96		HTML exporter. When you give it a URL, it fetches the notebook from that URL,
97		converts it to HTML, and serves that HTML to you.
98
99		IPython.parallel
100		----------------
101
102		IPython also includes a parallel computing framework, ``IPython.parallel``. This
103		allows you to control many individual engines, which are an extended version of
104		the IPython kernel described above. For more details, see :doc:`/parallel/index`.

docs/source/development/index.rst

0 0 -3

                 :maxdepth: 1
                 how_ipython_works
-                kernels
                 wrapperkernels
                 execution
-                parallel_messages
-                parallel_connections
                 lexer
                 pycompat
                 config

docs/source/development/kernels.rst

0 +4 -139

		@@ -1,143 +1,8 b''
	1	:orphan:
	2
1	3	==========================
2	4	Making kernels for IPython
3	5	==========================
4	6
5		A 'kernel' is a program that runs and introspects the user's code. IPython
6		includes a kernel for Python code, and people have written kernels for
7		`several other languages <https://github.com/ipython/ipython/wiki/Projects-using-IPython#list-of-some-ipython-compatible-kernels>`_.
8
9		When IPython starts a kernel, it passes it a connection file. This specifies
10		how to set up communications with the frontend.
11
12		There are two options for writing a kernel:
13
14		1. You can reuse the IPython kernel machinery to handle the communications, and
15		just describe how to execute your code. This is much simpler if the target
16		language can be driven from Python. See :doc:`wrapperkernels` for details.
17		2. You can implement the kernel machinery in your target language. This is more
18		work initially, but the people using your kernel might be more likely to
19		contribute to it if it's in the language they know.
20
21		Connection files
22		================
23
24		Your kernel will be given the path to a connection file when it starts (see
25		:ref:`kernelspecs` for how to specify the command line arguments for your kernel).
26		This file, which is accessible only to the current user, will contain a JSON
27		dictionary looking something like this::
28
29		{
30		"control_port": 50160,
31		"shell_port": 57503,
32		"transport": "tcp",
33		"signature_scheme": "hmac-sha256",
34		"stdin_port": 52597,
35		"hb_port": 42540,
36		"ip": "127.0.0.1",
37		"iopub_port": 40885,
38		"key": "a0436f6c-1916-498b-8eb9-e81ab9368e84"
39		}
40
41		The ``transport``, ``ip`` and five ``_port`` fields specify five ports which the
42		kernel should bind to using `ZeroMQ <http://zeromq.org/>`_. For instance, the
43		address of the shell socket in the example above would be::
44
45		tcp://127.0.0.1:57503
46
47		New ports are chosen at random for each kernel started.
48
49		``signature_scheme`` and ``key`` are used to cryptographically sign messages, so
50		that other users on the system can't send code to run in this kernel. See
51		:ref:`wire_protocol` for the details of how this signature is calculated.
52
53		Handling messages
54		=================
55
56		After reading the connection file and binding to the necessary sockets, the
57		kernel should go into an event loop, listening on the hb (heartbeat), control
58		and shell sockets.
59
60		:ref:`Heartbeat <kernel_heartbeat>` messages should be echoed back immediately
61		on the same socket - the frontend uses this to check that the kernel is still
62		alive.
63
64		Messages on the control and shell sockets should be parsed, and their signature
65		validated. See :ref:`wire_protocol` for how to do this.
66
67		The kernel will send messages on the iopub socket to display output, and on the
68		stdin socket to prompt the user for textual input.
69
70		.. seealso::
71
72		:doc:`messaging`
73		Details of the different sockets and the messages that come over them
74
75		`Creating Language Kernels for IPython <http://andrew.gibiansky.com/blog/ipython/ipython-kernels/>`_
76		A blog post by the author of `IHaskell <https://github.com/gibiansky/IHaskell>`_,
77		a Haskell kernel
78
79		`simple_kernel <https://github.com/dsblank/simple_kernel>`_
80		A simple example implementation of the kernel machinery in Python
81
82
83		.. _kernelspecs:
84
85		Kernel specs
86		============
87
88		A kernel identifies itself to IPython by creating a directory, the name of which
89		is used as an identifier for the kernel. These may be created in a number of
90		locations:
91
92		+--------+--------------------------------------+-----------------------------------+
93		\| \| Unix \| Windows \|
94		+========+======================================+===================================+
95		\| System \| ``/usr/share/jupyter/kernels`` \| ``%PROGRAMDATA%\jupyter\kernels`` \|
96		\| \| \| \|
97		\| \| ``/usr/local/share/jupyter/kernels`` \| \|
98		+--------+--------------------------------------+-----------------------------------+
99		\| User \| ``~/.ipython/kernels`` \|
100		+--------+--------------------------------------+-----------------------------------+
101
102		The user location takes priority over the system locations, and the case of the
103		names is ignored, so selecting kernels works the same way whether or not the
104		filesystem is case sensitive.
105
106		Inside the directory, the most important file is kernel.json. This should be a
107		JSON serialised dictionary containing the following keys and values:
108
109		- argv: A list of command line arguments used to start the kernel. The text
110		``{connection_file}`` in any argument will be replaced with the path to the
111		connection file.
112		- display_name: The kernel's name as it should be displayed in the UI.
113		Unlike the kernel name used in the API, this can contain arbitrary unicode
114		characters.
115		- language: The name of the language of the kernel.
116		When loading notebooks, if no matching kernelspec key (may differ across machines)
117		is found, a kernel with a matching `language` will be used.
118		This allows a notebook written on any Python or Julia kernel to be properly associated
119		with the user's Python or Julia kernel, even if they aren't listed under the same name as the author's.
120		- env (optional): A dictionary of environment variables to set for the kernel.
121		These will be added to the current environment variables before the kernel is
122		started.
123
124		For example, the kernel.json file for IPython looks like this::
125
126		{
127		"argv": ["python3", "-m", "IPython.kernel",
128		"-f", "{connection_file}"],
129		"display_name": "Python 3",
130		"language": "python"
131		}
132
133		To see the available kernel specs, run::
134
135		ipython kernelspec list
136
137		To start the terminal console or the Qt console with a specific kernel::
138
139		ipython console --kernel bash
140		ipython qtconsole --kernel bash
141
142		To use different kernels in the notebook, select a different kernel from the
143		dropdown menu in the top-right of the UI.
	7	Kernels are now part of Jupyter - see
	8	:ref:`jupyterclient:kernels` for the documentation.

docs/source/development/parallel_connections.rst

0 +3 -149

		@@ -1,154 +1,8 b''
1		.. _parallel_connections:
	1	:orphan:
2	2
3	3	==============================================
4	4	Connection Diagrams of The IPython ZMQ Cluster
5	5	==============================================
6	6
7		This is a quick summary and illustration of the connections involved in the ZeroMQ based
8		IPython cluster for parallel computing.
9
10		All Connections
11		===============
12
13		The IPython cluster consists of a Controller, and one or more each of clients and engines.
14		The goal of the Controller is to manage and monitor the connections and communications
15		between the clients and the engines. The Controller is no longer a single process entity,
16		but rather a collection of processes - specifically one Hub, and 4 (or more) Schedulers.
17
18		It is important for security/practicality reasons that all connections be inbound to the
19		controller processes. The arrows in the figures indicate the direction of the
20		connection.
21
22
23		.. figure:: figs/allconnections.png
24		:width: 432px
25		:alt: IPython cluster connections
26		:align: center
27
28		All the connections involved in connecting one client to one engine.
29
30		The Controller consists of 1-5 processes. Central to the cluster is the Hub, which monitors
31		engine state, execution traffic, and handles registration and notification. The Hub includes a
32		Heartbeat Monitor for keeping track of engines that are alive. Outside the Hub are 4
33		Schedulers. These devices are very small pure-C MonitoredQueue processes (or optionally
34		threads) that relay messages very fast, but also send a copy of each message along a side socket
35		to the Hub. The MUX queue and Control queue are MonitoredQueue ØMQ devices which relay
36		explicitly addressed messages from clients to engines, and their replies back up. The Balanced
37		queue performs load-balancing destination-agnostic scheduling. It may be a MonitoredQueue
38		device, but may also be a Python Scheduler that behaves externally in an identical fashion to MQ
39		devices, but with additional internal logic. stdout/err are also propagated from the Engines to
40		the clients via a PUB/SUB MonitoredQueue.
41
42
43		Registration
44		------------
45
46		.. figure:: figs/queryfade.png
47		:width: 432px
48		:alt: IPython Registration connections
49		:align: center
50
51		Engines and Clients only need to know where the Query ``ROUTER`` is located to start
52		connecting.
53
54		Once a controller is launched, the only information needed for connecting clients and/or
55		engines is the IP/port of the Hub's ``ROUTER`` socket called the Registrar. This socket
56		handles connections from both clients and engines, and replies with the remaining
57		information necessary to establish the remaining connections. Clients use this same socket for
58		querying the Hub for state information.
59
60		Heartbeat
61		---------
62
63		.. figure:: figs/hbfade.png
64		:width: 432px
65		:alt: IPython Heartbeat connections
66		:align: center
67
68		The heartbeat sockets.
69
70		The heartbeat process has been described elsewhere. To summarize: the Heartbeat Monitor
71		publishes a distinct message periodically via a ``PUB`` socket. Each engine has a
72		``zmq.FORWARDER`` device with a ``SUB`` socket for input, and ``DEALER`` socket for output.
73		The ``SUB`` socket is connected to the ``PUB`` socket labeled ping, and the ``DEALER`` is
74		connected to the ``ROUTER`` labeled pong. This results in the same message being relayed
75		back to the Heartbeat Monitor with the addition of the ``DEALER`` prefix. The Heartbeat
76		Monitor receives all the replies via an ``ROUTER`` socket, and identifies which hearts are
77		still beating by the ``zmq.IDENTITY`` prefix of the ``DEALER`` sockets, which information
78		the Hub uses to notify clients of any changes in the available engines.
79
80		Schedulers
81		----------
82
83		.. figure:: figs/queuefade.png
84		:width: 432px
85		:alt: IPython Queue connections
86		:align: center
87
88		Control message scheduler on the left, execution (apply) schedulers on the right.
89
90		The controller has at least three Schedulers. These devices are primarily for
91		relaying messages between clients and engines, but the Hub needs to see those
92		messages for its own purposes. Since no Python code may exist between the two sockets in a
93		queue, all messages sent through these queues (both directions) are also sent via a
94		``PUB`` socket to a monitor, which allows the Hub to monitor queue traffic without
95		interfering with it.
96
97		For tasks, the engine need not be specified. Messages sent to the ``ROUTER`` socket from the
98		client side are assigned to an engine via ZMQ's ``DEALER`` round-robin load balancing.
99		Engine replies are directed to specific clients via the IDENTITY of the client, which is
100		received as a prefix at the Engine.
101
102		For Multiplexing, ``ROUTER`` is used for both in and output sockets in the device. Clients must
103		specify the destination by the ``zmq.IDENTITY`` of the ``ROUTER`` socket connected to
104		the downstream end of the device.
105
106		At the Kernel level, both of these ``ROUTER`` sockets are treated in the same way as the ``REP``
107		socket in the serial version (except using ZMQStreams instead of explicit sockets).
108
109		Execution can be done in a load-balanced (engine-agnostic) or multiplexed (engine-specified)
110		manner. The sockets on the Client and Engine are the same for these two actions, but the
111		scheduler used determines the actual behavior. This routing is done via the ``zmq.IDENTITY`` of
112		the upstream sockets in each MonitoredQueue.
113
114		IOPub
115		-----
116
117		.. figure:: figs/iopubfade.png
118		:width: 432px
119		:alt: IOPub connections
120		:align: center
121
122		stdout/err are published via a ``PUB/SUB`` MonitoredQueue
123
124
125		On the kernels, stdout/stderr are captured and published via a ``PUB`` socket. These ``PUB``
126		sockets all connect to a ``SUB`` socket input of a MonitoredQueue, which subscribes to all
127		messages. They are then republished via another ``PUB`` socket, which can be
128		subscribed by the clients.
129
130		Client connections
131		------------------
132
133		.. figure:: figs/queryfade.png
134		:width: 432px
135		:alt: IPython client query connections
136		:align: center
137
138		Clients connect to an ``ROUTER`` socket to query the hub.
139
140		The hub's registrar ``ROUTER`` socket also listens for queries from clients as to queue status,
141		and control instructions. Clients connect to this socket via an ``DEALER`` during registration.
142
143		.. figure:: figs/notiffade.png
144		:width: 432px
145		:alt: IPython Registration connections
146		:align: center
147
148		Engine registration events are published via a ``PUB`` socket.
149
150		The Hub publishes all registration/unregistration events via a ``PUB`` socket. This
151		allows clients to stay up to date with what engines are available by subscribing to the
152		feed with a ``SUB`` socket. Other processes could selectively subscribe to just
153		registration or unregistration events.
154
	7	IPython parallel has moved to ipyparallel -
	8	see :ref:`ipyparallel:parallel_connections` for the documentation.

docs/source/development/parallel_messages.rst

0 +4 -363

		@@ -1,367 +1,8 b''
1		.. _parallel_messages:
	1	:orphan:
2	2
	3	================================
3	4	Messaging for Parallel Computing
4	5	================================
5	6
6		This is an extension of the :ref:`messaging <messaging>` doc. Diagrams of the connections
7		can be found in the :ref:`parallel connections <parallel_connections>` doc.
8
9
10		ZMQ messaging is also used in the parallel computing IPython system. All messages to/from
11		kernels remain the same as the single kernel model, and are forwarded through a ZMQ Queue
12		device. The controller receives all messages and replies in these channels, and saves
13		results for future use.
14
15		The Controller
16		--------------
17
18		The controller is the central collection of processes in the IPython parallel computing
19		model. It has two major components:
20
21		* The Hub
22		* A collection of Schedulers
23
24		The Hub
25		-------
26
27		The Hub is the central process for monitoring the state of the engines, and all task
28		requests and results. It has no role in execution and does no relay of messages, so
29		large blocking requests or database actions in the Hub do not have the ability to impede
30		job submission and results.
31
32		Registration (``ROUTER``)
33		*************************
34
35		The first function of the Hub is to facilitate and monitor connections of clients
36		and engines. Both client and engine registration are handled by the same socket, so only
37		one ip/port pair is needed to connect any number of connections and clients.
38
39		Engines register with the ``zmq.IDENTITY`` of their two ``DEALER`` sockets, one for the
40		queue, which receives execute requests, and one for the heartbeat, which is used to
41		monitor the survival of the Engine process.
42
43		Message type: ``registration_request``::
44
45		content = {
46		'uuid' : 'abcd-1234-...', # the zmq.IDENTITY of the engine's sockets
47		}
48
49		.. note::
50
51		these are always the same, at least for now.
52
53		The Controller replies to an Engine's registration request with the engine's integer ID,
54		and all the remaining connection information for connecting the heartbeat process, and
55		kernel queue socket(s). The message status will be an error if the Engine requests IDs that
56		already in use.
57
58		Message type: ``registration_reply``::
59
60		content = {
61		'status' : 'ok', # or 'error'
62		# if ok:
63		'id' : 0, # int, the engine id
64		}
65
66		Clients use the same socket as engines to start their connections. Connection requests
67		from clients need no information:
68
69		Message type: ``connection_request``::
70
71		content = {}
72
73		The reply to a Client registration request contains the connection information for the
74		multiplexer and load balanced queues, as well as the address for direct hub
75		queries. If any of these addresses is `None`, that functionality is not available.
76
77		Message type: ``connection_reply``::
78
79		content = {
80		'status' : 'ok', # or 'error'
81		}
82
83		Heartbeat
84		*********
85
86		The hub uses a heartbeat system to monitor engines, and track when they become
87		unresponsive. As described in :ref:`messaging <messaging>`, and shown in :ref:`connections
88		<parallel_connections>`.
89
90		Notification (``PUB``)
91		**********************
92
93		The hub publishes all engine registration/unregistration events on a ``PUB`` socket.
94		This allows clients to have up-to-date engine ID sets without polling. Registration
95		notifications contain both the integer engine ID and the queue ID, which is necessary for
96		sending messages via the Multiplexer Queue and Control Queues.
97
98		Message type: ``registration_notification``::
99
100		content = {
101		'id' : 0, # engine ID that has been registered
102		'uuid' : 'engine_id' # the IDENT for the engine's sockets
103		}
104
105		Message type : ``unregistration_notification``::
106
107		content = {
108		'id' : 0 # engine ID that has been unregistered
109		'uuid' : 'engine_id' # the IDENT for the engine's sockets
110		}
111
112
113		Client Queries (``ROUTER``)
114		***************************
115
116		The hub monitors and logs all queue traffic, so that clients can retrieve past
117		results or monitor pending tasks. This information may reside in-memory on the Hub, or
118		on disk in a database (SQLite and MongoDB are currently supported). These requests are
119		handled by the same socket as registration.
120
121
122		:func:`queue_request` requests can specify multiple engines to query via the `targets`
123		element. A verbose flag can be passed, to determine whether the result should be the list
124		of `msg_ids` in the queue or simply the length of each list.
125
126		Message type: ``queue_request``::
127
128		content = {
129		'verbose' : True, # whether return should be lists themselves or just lens
130		'targets' : [0,3,1] # list of ints
131		}
132
133		The content of a reply to a :func:`queue_request` request is a dict, keyed by the engine
134		IDs. Note that they will be the string representation of the integer keys, since JSON
135		cannot handle number keys. The three keys of each dict are::
136
137		'completed' : messages submitted via any queue that ran on the engine
138		'queue' : jobs submitted via MUX queue, whose results have not been received
139		'tasks' : tasks that are known to have been submitted to the engine, but
140		have not completed. Note that with the pure zmq scheduler, this will
141		always be 0/[].
142
143		Message type: ``queue_reply``::
144
145		content = {
146		'status' : 'ok', # or 'error'
147		# if verbose=False:
148		'0' : {'completed' : 1, 'queue' : 7, 'tasks' : 0},
149		# if verbose=True:
150		'1' : {'completed' : ['abcd-...','1234-...'], 'queue' : ['58008-'], 'tasks' : []},
151		}
152
153		Clients can request individual results directly from the hub. This is primarily for
154		gathering results of executions not submitted by the requesting client, as the client
155		will have all its own results already. Requests are made by msg_id, and can contain one or
156		more msg_id. An additional boolean key 'statusonly' can be used to not request the
157		results, but simply poll the status of the jobs.
158
159		Message type: ``result_request``::
160
161		content = {
162		'msg_ids' : ['uuid','...'], # list of strs
163		'targets' : [1,2,3], # list of int ids or uuids
164		'statusonly' : False, # bool
165		}
166
167		The :func:`result_request` reply contains the content objects of the actual execution
168		reply messages. If `statusonly=True`, then there will be only the 'pending' and
169		'completed' lists.
170
171
172		Message type: ``result_reply``::
173
174		content = {
175		'status' : 'ok', # else error
176		# if ok:
177		'acbd-...' : msg, # the content dict is keyed by msg_ids,
178		# values are the result messages
179		# there will be none of these if `statusonly=True`
180		'pending' : ['msg_id','...'], # msg_ids still pending
181		'completed' : ['msg_id','...'], # list of completed msg_ids
182		}
183		buffers = ['bufs','...'] # the buffers that contained the results of the objects.
184		# this will be empty if no messages are complete, or if
185		# statusonly is True.
186
187		For memory management purposes, Clients can also instruct the hub to forget the
188		results of messages. This can be done by message ID or engine ID. Individual messages are
189		dropped by msg_id, and all messages completed on an engine are dropped by engine ID. This
190		may no longer be necessary with the mongodb-based message logging backend.
191
192		If the msg_ids element is the string ``'all'`` instead of a list, then all completed
193		results are forgotten.
194
195		Message type: ``purge_request``::
196
197		content = {
198		'msg_ids' : ['id1', 'id2',...], # list of msg_ids or 'all'
199		'engine_ids' : [0,2,4] # list of engine IDs
200		}
201
202		The reply to a purge request is simply the status 'ok' if the request succeeded, or an
203		explanation of why it failed, such as requesting the purge of a nonexistent or pending
204		message.
205
206		Message type: ``purge_reply``::
207
208		content = {
209		'status' : 'ok', # or 'error'
210		}
211
212
213		Schedulers
214		----------
215
216		There are three basic schedulers:
217
218		* Task Scheduler
219		* MUX Scheduler
220		* Control Scheduler
221
222		The MUX and Control schedulers are simple MonitoredQueue ØMQ devices, with ``ROUTER``
223		sockets on either side. This allows the queue to relay individual messages to particular
224		targets via ``zmq.IDENTITY`` routing. The Task scheduler may be a MonitoredQueue ØMQ
225		device, in which case the client-facing socket is ``ROUTER``, and the engine-facing socket
226		is ``DEALER``. The result of this is that client-submitted messages are load-balanced via
227		the ``DEALER`` socket, but the engine's replies to each message go to the requesting client.
228
229		Raw ``DEALER`` scheduling is quite primitive, and doesn't allow message introspection, so
230		there are also Python Schedulers that can be used. These Schedulers behave in much the
231		same way as a MonitoredQueue does from the outside, but have rich internal logic to
232		determine destinations, as well as handle dependency graphs Their sockets are always
233		``ROUTER`` on both sides.
234
235		The Python task schedulers have an additional message type, which informs the Hub of
236		the destination of a task as soon as that destination is known.
237
238		Message type: ``task_destination``::
239
240		content = {
241		'msg_id' : 'abcd-1234-...', # the msg's uuid
242		'engine_id' : '1234-abcd-...', # the destination engine's zmq.IDENTITY
243		}
244
245		:func:`apply`
246		*************
247
248		In terms of message classes, the MUX scheduler and Task scheduler relay the exact same
249		message types. Their only difference lies in how the destination is selected.
250
251		The `Namespace <http://gist.github.com/483294>`_ model suggests that execution be able to
252		use the model::
253
254		ns.apply(f, args, *kwargs)
255
256		which takes `f`, a function in the user's namespace, and executes ``f(args, *kwargs)``
257		on a remote engine, returning the result (or, for non-blocking, information facilitating
258		later retrieval of the result). This model, unlike the execute message which just uses a
259		code string, must be able to send arbitrary (pickleable) Python objects. And ideally, copy
260		as little data as we can. The `buffers` property of a Message was introduced for this
261		purpose.
262
263		Utility method :func:`build_apply_message` in :mod:`IPython.kernel.zmq.serialize` wraps a
264		function signature and builds a sendable buffer format for minimal data copying (exactly
265		zero copies of numpy array data or buffers or large strings).
266
267		Message type: ``apply_request``::
268
269		metadata = {
270		'after' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict()
271		'follow' : ['msg_id',...], # list of msg_ids or output of Dependency.as_dict()
272		}
273		content = {}
274		buffers = ['...'] # at least 3 in length
275		# as built by build_apply_message(f,args,kwargs)
276
277		after/follow represent task dependencies. 'after' corresponds to a time dependency. The
278		request will not arrive at an engine until the 'after' dependency tasks have completed.
279		'follow' corresponds to a location dependency. The task will be submitted to the same
280		engine as these msg_ids (see :class:`Dependency` docs for details).
281
282		Message type: ``apply_reply``::
283
284		content = {
285		'status' : 'ok' # 'ok' or 'error'
286		# other error info here, as in other messages
287		}
288		buffers = ['...'] # either 1 or 2 in length
289		# a serialization of the return value of f(args,*kwargs)
290		# only populated if status is 'ok'
291
292		All engine execution and data movement is performed via apply messages.
293
294		Control Messages
295		----------------
296
297		Messages that interact with the engines, but are not meant to execute code, are submitted
298		via the Control queue. These messages have high priority, and are thus received and
299		handled before any execution requests.
300
301		Clients may want to clear the namespace on the engine. There are no arguments nor
302		information involved in this request, so the content is empty.
303
304		Message type: ``clear_request``::
305
306		content = {}
307
308		Message type: ``clear_reply``::
309
310		content = {
311		'status' : 'ok' # 'ok' or 'error'
312		# other error info here, as in other messages
313		}
314
315		Clients may want to abort tasks that have not yet run. This can by done by message id, or
316		all enqueued messages can be aborted if None is specified.
317
318		Message type: ``abort_request``::
319
320		content = {
321		'msg_ids' : ['1234-...', '...'] # list of msg_ids or None
322		}
323
324		Message type: ``abort_reply``::
325
326		content = {
327		'status' : 'ok' # 'ok' or 'error'
328		# other error info here, as in other messages
329		}
330
331		The last action a client may want to do is shutdown the kernel. If a kernel receives a
332		shutdown request, then it aborts all queued messages, replies to the request, and exits.
333
334		Message type: ``shutdown_request``::
335
336		content = {}
337
338		Message type: ``shutdown_reply``::
339
340		content = {
341		'status' : 'ok' # 'ok' or 'error'
342		# other error info here, as in other messages
343		}
344
345
346		Implementation
347		--------------
348
349		There are a few differences in implementation between the `StreamSession` object used in
350		the newparallel branch and the `Session` object, the main one being that messages are
351		sent in parts, rather than as a single serialized object. `StreamSession` objects also
352		take pack/unpack functions, which are to be used when serializing/deserializing objects.
353		These can be any functions that translate to/from formats that ZMQ sockets can send
354		(buffers,bytes, etc.).
355
356		Split Sends
357		***********
358
359		Previously, messages were bundled as a single json object and one call to
360		:func:`socket.send_json`. Since the hub inspects all messages, and doesn't need to
361		see the content of the messages, which can be large, messages are now serialized and sent in
362		pieces. All messages are sent in at least 4 parts: the header, the parent header, the metadata and the content.
363		This allows the controller to unpack and inspect the (always small) header,
364		without spending time unpacking the content unless the message is bound for the
365		controller. Buffers are added on to the end of the message, and can be any objects that
366		present the buffer interface.
367
	7	IPython parallel has moved to ipyparallel -
	8	see :ref:`ipyparallel:parallel_messages` for the documentation.

docs/source/development/wrapperkernels.rst

0 +5 -5

              Required steps
              --------------
-             Subclass :class:`IPython.kernel.zmq.kernelbase.Kernel`, and implement the
+             Subclass :class:`ipykernel.kernelbase.Kernel`, and implement the
              following methods and attributes:
              .. class:: MyKernel
                   Your method should return a dict containing the fields described in
                   :ref:`execution_results`. To display output, it can send messages
-                  using :meth:`~IPython.kernel.zmq.kernelbase.Kernel.send_response`.
+                  using :meth:`~ipykernel.kernelbase.Kernel.send_response`.
                   See :doc:`messaging` for details of the different message types.
              To launch your kernel, add this at the end of your module::
                  if __name__ == '__main__':
-                     from IPython.kernel.zmq.kernelapp import IPKernelApp
+                     from ipykernel.kernelapp import IPKernelApp
                      IPKernelApp.launch_instance(kernel_class=MyKernel)
              Example
              ``echokernel.py`` will simply echo any input it's given to stdout::
-                 from IPython.kernel.zmq.kernelbase import Kernel
+                 from ipykernel.kernelbase import Kernel
                  class EchoKernel(Kernel):
                      implementation = 'Echo'
                                 }
                  if __name__ == '__main__':
-                     from IPython.kernel.zmq.kernelapp import IPKernelApp
+                     from ipykernel.kernelapp import IPKernelApp
                      IPKernelApp.launch_instance(kernel_class=EchoKernel)
              Here's the Kernel spec ``kernel.json`` file for this::

docs/source/development/figs/allconnections.png

0 removed binary 0 0