diff --git a/docs/source/development/kernels.rst b/docs/source/development/kernels.rst new file mode 100644 index 0000000..4077d3b --- /dev/null +++ b/docs/source/development/kernels.rst @@ -0,0 +1,152 @@ +========================== +Making kernels for IPython +========================== + +A 'kernel' is a program that runs and introspects the user's code. IPython +includes a kernel for Python code, and people have written kernels for +`several other languages `_. + +When IPython starts a kernel, it passes it a connection file. This specifies +how to set up communications with the frontend. + +There are two options for writing a kernel: + +1. You can reuse the IPython kernel machinery to handle the communications, and + just describe how to execute your code. This is much simpler if the target + language can be driven from Python. See :doc:`wrapperkernels` for details. +2. You can implement the kernel machinery in your target language. This is more + work initially, but the people using your kernel might be more likely to + contribute to it if it's in the language they know. + +Connection files +================ + +Your kernel will be given the path to a connection file when it starts (see +:ref:`kernelspecs` for how to specify the command line arguments for your kernel). +This file, which is accessible only to the current user, will contain a JSON +dictionary looking something like this:: + + { + "control_port": 50160, + "shell_port": 57503, + "transport": "tcp", + "signature_scheme": "hmac-sha256", + "stdin_port": 52597, + "hb_port": 42540, + "ip": "127.0.0.1", + "iopub_port": 40885, + "key": "a0436f6c-1916-498b-8eb9-e81ab9368e84" + } + +The ``transport``, ``ip`` and five ``_port`` fields specify five ports which the +kernel should bind to using `ZeroMQ `_. For instance, the +address of the shell socket in the example above would be:: + + tcp://127.0.0.1:57503 + +New ports are chosen at random for each kernel started. + +``signature_scheme`` and ``key`` are used to cryptographically sign messages, so +that other users on the system can't send code to run in this kernel. See +:ref:`wire_protocol` for the details of how this signature is calculated. + +Handling messages +================= + +After reading the connection file and binding to the necessary sockets, the +kernel should go into an event loop, listening on the hb (heartbeat), control +and shell sockets. + +:ref:`Heartbeat ` messages should be echoed back immediately +on the same socket - the frontend uses this to check that the kernel is still +alive. + +Messages on the control and shell sockets should be parsed, and their signature +validated. See :ref:`wire_protocol` for how to do this. + +The kernel will send messages on the iopub socket to display output, and on the +stdin socket to prompt the user for textual input. + +.. seealso:: + + :doc:`messaging` + Details of the different sockets and the messages that come over them + + `Creating Language Kernels for IPython `_ + A blog post by the author of `IHaskell `_, + a Haskell kernel + + `simple_kernel `_ + A simple example implementation of the kernel machinery in Python + + +.. _kernelspecs: + +Kernel specs +============ + +A kernel identifies itself to IPython by creating a directory, the name of which +is used as an identifier for the kernel. These may be created in a number of +locations: + ++--------+--------------------------------------+-----------------------------------+ +| | Unix | Windows | ++========+======================================+===================================+ +| System | ``/usr/share/ipython/kernels`` | ``%PROGRAMDATA%\ipython\kernels`` | +| | | | +| | ``/usr/local/share/ipython/kernels`` | | ++--------+--------------------------------------+-----------------------------------+ +| User | ``~/.ipython/kernels`` | ++--------+--------------------------------------+-----------------------------------+ + +The user location takes priority over the system locations, and the case of the +names is ignored, so selecting kernels works the same way whether or not the +filesystem is case sensitive. + +Inside the directory, the most important file is *kernel.json*. This should be a +JSON serialised dictionary containing the following keys and values: + +- **argv**: A list of command line arguments used to start the kernel. The text + ``{connection_file}`` in any argument will be replaced with the path to the + connection file. +- **display_name**: The kernel's name as it should be displayed in the UI. + Unlike the kernel name used in the API, this can contain arbitrary unicode + characters. +- **language**: The programming language which this kernel runs. This will be + stored in notebook metadata. This may be used by syntax highlighters to guess + how to parse code in a notebook, and frontends may eventually use it to + identify alternative kernels that can run some code. +- **codemirror_mode** (optional): The `codemirror mode `_ + to use for code in this language. This can be a string or a dictionary, as + passed to codemirror config. The string from *language* will be used if this is + not provided. +- **env** (optional): A dictionary of environment variables to set for the kernel. + These will be added to the current environment variables before the kernel is + started. +- **help_links** (optional): A list of dictionaries, each with keys 'text' and + 'url'. These will be displayed in the help menu in the notebook UI. + +For example, the kernel.json file for IPython looks like this:: + + { + "argv": ["python3", "-c", "from IPython.kernel.zmq.kernelapp import main; main()", + "-f", "{connection_file}"], + "codemirror_mode": { + "version": 3, + "name": "ipython" + }, + "display_name": "IPython (Python 3)", + "language": "python" + } + +To see the available kernel specs, run:: + + ipython kernelspec list + +To start the terminal console or the Qt console with a specific kernel:: + + ipython console --kernel bash + ipython qtconsole --kernel bash + +To use different kernels in the notebook, select a different kernel from the +dropdown menu in the top-right of the UI. diff --git a/docs/source/development/messaging.rst b/docs/source/development/messaging.rst index 0d31575..f43b137 100644 --- a/docs/source/development/messaging.rst +++ b/docs/source/development/messaging.rst @@ -26,7 +26,7 @@ within and between hosts. IPython messaging protocol, and all developers are strongly encouraged to keep it updated as the implementation evolves, so that we have a single common reference for all protocol details. - + The basic design is explained in the following diagram: .. image:: figs/frontend-kernel.png @@ -126,6 +126,8 @@ A message is defined by the following four-dictionary structure:: ``version`` key added to the header. +.. _wire_protocol: + The Wire Protocol ================= @@ -971,6 +973,7 @@ When ``password`` is True, the frontend should not echo the input as it is enter transported over the zmq connection), raw ``stdin`` isn't expected to be available. +.. _kernel_heartbeat: Heartbeat for kernels ===================== diff --git a/docs/source/development/wrapperkernels.rst b/docs/source/development/wrapperkernels.rst index 5efcbef..e345333 100644 --- a/docs/source/development/wrapperkernels.rst +++ b/docs/source/development/wrapperkernels.rst @@ -10,6 +10,11 @@ This is useful for languages that have Python bindings, such as `Octave where the REPL can be controlled in a tty using `pexpect `_, such as bash. +.. seealso:: + + `bash_kernel `_ + A simple kernel for bash, written using this machinery + Required steps --------------