From abffd96eaa9c76cde52b531a4e9cd76f35dc3b3b 2014-07-22 18:14:14
From: Thomas Kluyver <takowl@gmail.com>
Date: 2014-07-22 18:14:14
Subject: [PATCH] Merge pull request #6179 from takluyver/master
Add docs on writing kernels
---
diff --git a/docs/source/development/kernels.rst b/docs/source/development/kernels.rst
new file mode 100644
index 0000000..4077d3b
--- /dev/null
+++ b/docs/source/development/kernels.rst
@@ -0,0 +1,152 @@
+==========================
+Making kernels for IPython
+==========================
+
+A 'kernel' is a program that runs and introspects the user's code. IPython
+includes a kernel for Python code, and people have written kernels for
+`several other languages <https://github.com/ipython/ipython/wiki/Projects-using-IPython#list-of-some-ipython-compatible-kernels>`_.
+
+When IPython starts a kernel, it passes it a connection file. This specifies
+how to set up communications with the frontend.
+
+There are two options for writing a kernel:
+
+1. You can reuse the IPython kernel machinery to handle the communications, and
+ just describe how to execute your code. This is much simpler if the target
+ language can be driven from Python. See :doc:`wrapperkernels` for details.
+2. You can implement the kernel machinery in your target language. This is more
+ work initially, but the people using your kernel might be more likely to
+ contribute to it if it's in the language they know.
+
+Connection files
+================
+
+Your kernel will be given the path to a connection file when it starts (see
+:ref:`kernelspecs` for how to specify the command line arguments for your kernel).
+This file, which is accessible only to the current user, will contain a JSON
+dictionary looking something like this::
+
+ {
+ "control_port": 50160,
+ "shell_port": 57503,
+ "transport": "tcp",
+ "signature_scheme": "hmac-sha256",
+ "stdin_port": 52597,
+ "hb_port": 42540,
+ "ip": "127.0.0.1",
+ "iopub_port": 40885,
+ "key": "a0436f6c-1916-498b-8eb9-e81ab9368e84"
+ }
+
+The ``transport``, ``ip`` and five ``_port`` fields specify five ports which the
+kernel should bind to using `ZeroMQ <http://zeromq.org/>`_. For instance, the
+address of the shell socket in the example above would be::
+
+ tcp://127.0.0.1:57503
+
+New ports are chosen at random for each kernel started.
+
+``signature_scheme`` and ``key`` are used to cryptographically sign messages, so
+that other users on the system can't send code to run in this kernel. See
+:ref:`wire_protocol` for the details of how this signature is calculated.
+
+Handling messages
+=================
+
+After reading the connection file and binding to the necessary sockets, the
+kernel should go into an event loop, listening on the hb (heartbeat), control
+and shell sockets.
+
+:ref:`Heartbeat <kernel_heartbeat>` messages should be echoed back immediately
+on the same socket - the frontend uses this to check that the kernel is still
+alive.
+
+Messages on the control and shell sockets should be parsed, and their signature
+validated. See :ref:`wire_protocol` for how to do this.
+
+The kernel will send messages on the iopub socket to display output, and on the
+stdin socket to prompt the user for textual input.
+
+.. seealso::
+
+ :doc:`messaging`
+ Details of the different sockets and the messages that come over them
+
+ `Creating Language Kernels for IPython <http://andrew.gibiansky.com/blog/ipython/ipython-kernels/>`_
+ A blog post by the author of `IHaskell <https://github.com/gibiansky/IHaskell>`_,
+ a Haskell kernel
+
+ `simple_kernel <https://github.com/dsblank/simple_kernel>`_
+ A simple example implementation of the kernel machinery in Python
+
+
+.. _kernelspecs:
+
+Kernel specs
+============
+
+A kernel identifies itself to IPython by creating a directory, the name of which
+is used as an identifier for the kernel. These may be created in a number of
+locations:
+
++--------+--------------------------------------+-----------------------------------+
+| | Unix | Windows |
++========+======================================+===================================+
+| System | ``/usr/share/ipython/kernels`` | ``%PROGRAMDATA%\ipython\kernels`` |
+| | | |
+| | ``/usr/local/share/ipython/kernels`` | |
++--------+--------------------------------------+-----------------------------------+
+| User | ``~/.ipython/kernels`` |
++--------+--------------------------------------+-----------------------------------+
+
+The user location takes priority over the system locations, and the case of the
+names is ignored, so selecting kernels works the same way whether or not the
+filesystem is case sensitive.
+
+Inside the directory, the most important file is *kernel.json*. This should be a
+JSON serialised dictionary containing the following keys and values:
+
+- **argv**: A list of command line arguments used to start the kernel. The text
+ ``{connection_file}`` in any argument will be replaced with the path to the
+ connection file.
+- **display_name**: The kernel's name as it should be displayed in the UI.
+ Unlike the kernel name used in the API, this can contain arbitrary unicode
+ characters.
+- **language**: The programming language which this kernel runs. This will be
+ stored in notebook metadata. This may be used by syntax highlighters to guess
+ how to parse code in a notebook, and frontends may eventually use it to
+ identify alternative kernels that can run some code.
+- **codemirror_mode** (optional): The `codemirror mode <http://codemirror.net/mode/index.html>`_
+ to use for code in this language. This can be a string or a dictionary, as
+ passed to codemirror config. The string from *language* will be used if this is
+ not provided.
+- **env** (optional): A dictionary of environment variables to set for the kernel.
+ These will be added to the current environment variables before the kernel is
+ started.
+- **help_links** (optional): A list of dictionaries, each with keys 'text' and
+ 'url'. These will be displayed in the help menu in the notebook UI.
+
+For example, the kernel.json file for IPython looks like this::
+
+ {
+ "argv": ["python3", "-c", "from IPython.kernel.zmq.kernelapp import main; main()",
+ "-f", "{connection_file}"],
+ "codemirror_mode": {
+ "version": 3,
+ "name": "ipython"
+ },
+ "display_name": "IPython (Python 3)",
+ "language": "python"
+ }
+
+To see the available kernel specs, run::
+
+ ipython kernelspec list
+
+To start the terminal console or the Qt console with a specific kernel::
+
+ ipython console --kernel bash
+ ipython qtconsole --kernel bash
+
+To use different kernels in the notebook, select a different kernel from the
+dropdown menu in the top-right of the UI.
diff --git a/docs/source/development/messaging.rst b/docs/source/development/messaging.rst
index 0d31575..f43b137 100644
--- a/docs/source/development/messaging.rst
+++ b/docs/source/development/messaging.rst
@@ -26,7 +26,7 @@ within and between hosts.
IPython messaging protocol, and all developers are strongly encouraged to
keep it updated as the implementation evolves, so that we have a single
common reference for all protocol details.
-
+
The basic design is explained in the following diagram:
.. image:: figs/frontend-kernel.png
@@ -126,6 +126,8 @@ A message is defined by the following four-dictionary structure::
``version`` key added to the header.
+.. _wire_protocol:
+
The Wire Protocol
=================
@@ -971,6 +973,7 @@ When ``password`` is True, the frontend should not echo the input as it is enter
transported over the zmq connection), raw ``stdin`` isn't expected to be
available.
+.. _kernel_heartbeat:
Heartbeat for kernels
=====================
diff --git a/docs/source/development/wrapperkernels.rst b/docs/source/development/wrapperkernels.rst
index 5efcbef..e345333 100644
--- a/docs/source/development/wrapperkernels.rst
+++ b/docs/source/development/wrapperkernels.rst
@@ -10,6 +10,11 @@ This is useful for languages that have Python bindings, such as `Octave
where the REPL can be controlled in a tty using `pexpect <http://pexpect.readthedocs.org/en/latest/>`_,
such as bash.
+.. seealso::
+
+ `bash_kernel <https://github.com/takluyver/bash_kernel>`_
+ A simple kernel for bash, written using this machinery
+
Required steps
--------------