wireproto: define and expose types of wire command arguments

Exposing the set of argument names is cool. But with wire protocol
version 2, we're using CBOR to transport arguments and this allows us
to have typing for arguments.

Typed arguments are much nicer because they will cut down on transfer
overhead and processing overhead for decoding values.

This commit teaches @wireprotocommand to accept a dictionary for
arguments. The arguments registered for version 2 transports are
canonically stored as dictionaries rather than a space-delimited string.

It is an error to defined arguments with a dictionary for commands using
version 1 transports. This reinforces my intent to fully decouple command
handlers for version 2 transports.

Differential Revision: https://phab.mercurial-scm.org/D3202

mercurial/help/internals/wireprotocol.txt

@@ -1,1774 +1,1779 b''
 The Mercurial wire protocol is a request-response based protocol
 with multiple wire representations.
 
 Each request is modeled as a command name, a dictionary of arguments, and
 optional raw input. Command arguments and their types are intrinsic
 properties of commands. So is the response type of the command. This means
 clients can't always send arbitrary arguments to servers and servers can't
 return multiple response types.
 
 The protocol is synchronous and does not support multiplexing (concurrent
 commands).
 
 Handshake
 =========
 
 It is required or common for clients to perform a *handshake* when connecting
 to a server. The handshake serves the following purposes:
 
 * Negotiating protocol/transport level options
 * Allows the client to learn about server capabilities to influence
   future requests
 * Ensures the underlying transport channel is in a *clean* state
 
 An important goal of the handshake is to allow clients to use more modern
 wire protocol features. By default, clients must assume they are talking
 to an old version of Mercurial server (possibly even the very first
 implementation). So, clients should not attempt to call or utilize modern
 wire protocol features until they have confirmation that the server
 supports them. The handshake implementation is designed to allow both
 ends to utilize the latest set of features and capabilities with as
 few round trips as possible.
 
 The handshake mechanism varies by transport and protocol and is documented
 in the sections below.
 
 HTTP Protocol
 =============
 
 Handshake
 ---------
 
 The client sends a ``capabilities`` command request (``?cmd=capabilities``)
 as soon as HTTP requests may be issued.
 
 The server responds with a capabilities string, which the client parses to
 learn about the server's abilities.
 
 HTTP Version 1 Transport
 ------------------------
 
 Commands are issued as HTTP/1.0 or HTTP/1.1 requests. Commands are
 sent to the base URL of the repository with the command name sent in
 the ``cmd`` query string parameter. e.g.
 ``https://example.com/repo?cmd=capabilities``. The HTTP method is ``GET``
 or ``POST`` depending on the command and whether there is a request
 body.
 
 Command arguments can be sent multiple ways.
 
 The simplest is part of the URL query string using ``x-www-form-urlencoded``
 encoding (see Python's ``urllib.urlencode()``. However, many servers impose
 length limitations on the URL. So this mechanism is typically only used if
 the server doesn't support other mechanisms.
 
 If the server supports the ``httpheader`` capability, command arguments can
 be sent in HTTP request headers named ``X-HgArg-<N>`` where ``<N>`` is an
 integer starting at 1. A ``x-www-form-urlencoded`` representation of the
 arguments is obtained. This full string is then split into chunks and sent
 in numbered ``X-HgArg-<N>`` headers. The maximum length of each HTTP header
 is defined by the server in the ``httpheader`` capability value, which defaults
 to ``1024``. The server reassembles the encoded arguments string by
 concatenating the ``X-HgArg-<N>`` headers then URL decodes them into a
 dictionary.
 
 The list of ``X-HgArg-<N>`` headers should be added to the ``Vary`` request
 header to instruct caches to take these headers into consideration when caching
 requests.
 
 If the server supports the ``httppostargs`` capability, the client
 may send command arguments in the HTTP request body as part of an
 HTTP POST request. The command arguments will be URL encoded just like
 they would for sending them via HTTP headers. However, no splitting is
 performed: the raw arguments are included in the HTTP request body.
 
 The client sends a ``X-HgArgs-Post`` header with the string length of the
 encoded arguments data. Additional data may be included in the HTTP
 request body immediately following the argument data. The offset of the
 non-argument data is defined by the ``X-HgArgs-Post`` header. The
 ``X-HgArgs-Post`` header is not required if there is no argument data.
 
 Additional command data can be sent as part of the HTTP request body. The
 default ``Content-Type`` when sending data is ``application/mercurial-0.1``.
 A ``Content-Length`` header is currently always sent.
 
 Example HTTP requests::
 
     GET /repo?cmd=capabilities
     X-HgArg-1: foo=bar&baz=hello%20world
 
 The request media type should be chosen based on server support. If the
 ``httpmediatype`` server capability is present, the client should send
 the newest mutually supported media type. If this capability is absent,
 the client must assume the server only supports the
 ``application/mercurial-0.1`` media type.
 
 The ``Content-Type`` HTTP response header identifies the response as coming
 from Mercurial and can also be used to signal an error has occurred.
 
 The ``application/mercurial-*`` media types indicate a generic Mercurial
 data type.
 
 The ``application/mercurial-0.1`` media type is raw Mercurial data. It is the
 predecessor of the format below.
 
 The ``application/mercurial-0.2`` media type is compression framed Mercurial
 data. The first byte of the payload indicates the length of the compression
 format identifier that follows. Next are N bytes indicating the compression
 format. e.g. ``zlib``. The remaining bytes are compressed according to that
 compression format. The decompressed data behaves the same as with
 ``application/mercurial-0.1``.
 
 The ``application/hg-error`` media type indicates a generic error occurred.
 The content of the HTTP response body typically holds text describing the
 error.
 
 The ``application/hg-changegroup`` media type indicates a changegroup response
 type.
 
 Clients also accept the ``text/plain`` media type. All other media
 types should cause the client to error.
 
 Behavior of media types is further described in the ``Content Negotiation``
 section below.
 
 Clients should issue a ``User-Agent`` request header that identifies the client.
 The server should not use the ``User-Agent`` for feature detection.
 
 A command returning a ``string`` response issues a
 ``application/mercurial-0.*`` media type and the HTTP response body contains
 the raw string value (after compression decoding, if used). A
 ``Content-Length`` header is typically issued, but not required.
 
 A command returning a ``stream`` response issues a
 ``application/mercurial-0.*`` media type and the HTTP response is typically
 using *chunked transfer* (``Transfer-Encoding: chunked``).
 
 HTTP Version 2 Transport
 ------------------------
 
 **Experimental - feature under active development**
 
 Version 2 of the HTTP protocol is exposed under the ``/api/*`` URL space.
 It's final API name is not yet formalized.
 
 Commands are triggered by sending HTTP POST requests against URLs of the
 form ``<permission>/<command>``, where ``<permission>`` is ``ro`` or
 ``rw``, meaning read-only and read-write, respectively and ``<command>``
 is a named wire protocol command.
 
 Non-POST request methods MUST be rejected by the server with an HTTP
 405 response.
 
 Commands that modify repository state in meaningful ways MUST NOT be
 exposed under the ``ro`` URL prefix. All available commands MUST be
 available under the ``rw`` URL prefix.
 
 Server adminstrators MAY implement blanket HTTP authentication keyed
 off the URL prefix. For example, a server may require authentication
 for all ``rw/*`` URLs and let unauthenticated requests to ``ro/*``
 URL proceed. A server MAY issue an HTTP 401, 403, or 407 response
 in accordance with RFC 7235. Clients SHOULD recognize the HTTP Basic
 (RFC 7617) and Digest (RFC 7616) authentication schemes. Clients SHOULD
 make an attempt to recognize unknown schemes using the
 ``WWW-Authenticate`` response header on a 401 response, as defined by
 RFC 7235.
 
 Read-only commands are accessible under ``rw/*`` URLs so clients can
 signal the intent of the operation very early in the connection
 lifecycle. For example, a ``push`` operation - which consists of
 various read-only commands mixed with at least one read-write command -
 can perform all commands against ``rw/*`` URLs so that any server-side
 authentication requirements are discovered upon attempting the first
 command - not potentially several commands into the exchange. This
 allows clients to fail faster or prompt for credentials as soon as the
 exchange takes place. This provides a better end-user experience.
 
 Requests to unknown commands or URLS result in an HTTP 404.
 TODO formally define response type, how error is communicated, etc.
 
 HTTP request and response bodies use the *Unified Frame-Based Protocol*
 (defined below) for media exchange. The entirety of the HTTP message
 body is 0 or more frames as defined by this protocol.
 
 Clients and servers MUST advertise the ``TBD`` media type via the
 ``Content-Type`` request and response headers. In addition, clients MUST
 advertise this media type value in their ``Accept`` request header in all
 requests.
 TODO finalize the media type. For now, it is defined in wireprotoserver.py.
 
 Servers receiving requests without an ``Accept`` header SHOULD respond with
 an HTTP 406.
 
 Servers receiving requests with an invalid ``Content-Type`` header SHOULD
 respond with an HTTP 415.
 
 The command to run is specified in the POST payload as defined by the
 *Unified Frame-Based Protocol*. This is redundant with data already
 encoded in the URL. This is by design, so server operators can have
 better understanding about server activity from looking merely at
 HTTP access logs.
 
 In most circumstances, the command specified in the URL MUST match
 the command specified in the frame-based payload or the server will
 respond with an error. The exception to this is the special
 ``multirequest`` URL. (See below.) In addition, HTTP requests
 are limited to one command invocation. The exception is the special
 ``multirequest`` URL.
 
 The ``multirequest`` command endpoints (``ro/multirequest`` and
 ``rw/multirequest``) are special in that they allow the execution of
 *any* command and allow the execution of multiple commands. If the
 HTTP request issues multiple commands across multiple frames, all
 issued commands will be processed by the server. Per the defined
 behavior of the *Unified Frame-Based Protocol*, commands may be
 issued interleaved and responses may come back in a different order
 than they were issued. Clients MUST be able to deal with this.
 
 SSH Protocol
 ============
 
 Handshake
 ---------
 
 For all clients, the handshake consists of the client sending 1 or more
 commands to the server using version 1 of the transport. Servers respond
 to commands they know how to respond to and send an empty response (``0\n``)
 for unknown commands (per standard behavior of version 1 of the transport).
 Clients then typically look for a response to the newest sent command to
 determine which transport version to use and what the available features for
 the connection and server are.
 
 Preceding any response from client-issued commands, the server may print
 non-protocol output. It is common for SSH servers to print banners, message
 of the day announcements, etc when clients connect. It is assumed that any
 such *banner* output will precede any Mercurial server output. So clients
 must be prepared to handle server output on initial connect that isn't
 in response to any client-issued command and doesn't conform to Mercurial's
 wire protocol. This *banner* output should only be on stdout. However,
 some servers may send output on stderr.
 
 Pre 0.9.1 clients issue a ``between`` command with the ``pairs`` argument
 having the value
 ``0000000000000000000000000000000000000000-0000000000000000000000000000000000000000``.
 
 The ``between`` command has been supported since the original Mercurial
 SSH server. Requesting the empty range will return a ``\n`` string response,
 which will be encoded as ``1\n\n`` (value length of ``1`` followed by a newline
 followed by the value, which happens to be a newline).
 
 For pre 0.9.1 clients and all servers, the exchange looks like::
 
    c: between\n
    c: pairs 81\n
    c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
    s: 1\n
    s: \n
 
 0.9.1+ clients send a ``hello`` command (with no arguments) before the
 ``between`` command. The response to this command allows clients to
 discover server capabilities and settings.
 
 An example exchange between 0.9.1+ clients and a ``hello`` aware server looks
 like::
 
    c: hello\n
    c: between\n
    c: pairs 81\n
    c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
    s: 324\n
    s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n
    s: 1\n
    s: \n
 
 And a similar scenario but with servers sending a banner on connect::
 
    c: hello\n
    c: between\n
    c: pairs 81\n
    c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
    s: welcome to the server\n
    s: if you find any issues, email [e-mail]\n
    s: 324\n
    s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n
    s: 1\n
    s: \n
 
 Note that output from the ``hello`` command is terminated by a ``\n``. This is
 part of the response payload and not part of the wire protocol adding a newline
 after responses. In other words, the length of the response contains the
 trailing ``\n``.
 
 Clients supporting version 2 of the SSH transport send a line beginning
 with ``upgrade`` before the ``hello`` and ``between`` commands. The line
 (which isn't a well-formed command line because it doesn't consist of a
 single command name) serves to both communicate the client's intent to
 switch to transport version 2 (transports are version 1 by default) as
 well as to advertise the client's transport-level capabilities so the
 server may satisfy that request immediately.
 
 The upgrade line has the form:
 
     upgrade <token> <transport capabilities>
 
 That is the literal string ``upgrade`` followed by a space, followed by
 a randomly generated string, followed by a space, followed by a string
 denoting the client's transport capabilities.
 
 The token can be anything. However, a random UUID is recommended. (Use
 of version 4 UUIDs is recommended because version 1 UUIDs can leak the
 client's MAC address.)
 
 The transport capabilities string is a URL/percent encoded string
 containing key-value pairs defining the client's transport-level
 capabilities. The following capabilities are defined:
 
 proto
    A comma-delimited list of transport protocol versions the client
    supports. e.g. ``ssh-v2``.
 
 If the server does not recognize the ``upgrade`` line, it should issue
 an empty response and continue processing the ``hello`` and ``between``
 commands. Here is an example handshake between a version 2 aware client
 and a non version 2 aware server:
 
    c: upgrade 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a proto=ssh-v2
    c: hello\n
    c: between\n
    c: pairs 81\n
    c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
    s: 0\n
    s: 324\n
    s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n
    s: 1\n
    s: \n
 
 (The initial ``0\n`` line from the server indicates an empty response to
 the unknown ``upgrade ..`` command/line.)
 
 If the server recognizes the ``upgrade`` line and is willing to satisfy that
 upgrade request, it replies to with a payload of the following form:
 
    upgraded <token> <transport name>\n
 
 This line is the literal string ``upgraded``, a space, the token that was
 specified by the client in its ``upgrade ...`` request line, a space, and the
 name of the transport protocol that was chosen by the server. The transport
 name MUST match one of the names the client specified in the ``proto`` field
 of its ``upgrade ...`` request line.
 
 If a server issues an ``upgraded`` response, it MUST also read and ignore
 the lines associated with the ``hello`` and ``between`` command requests
 that were issued by the server. It is assumed that the negotiated transport
 will respond with equivalent requested information following the transport
 handshake.
 
 All data following the ``\n`` terminating the ``upgraded`` line is the
 domain of the negotiated transport. It is common for the data immediately
 following to contain additional metadata about the state of the transport and
 the server. However, this isn't strictly speaking part of the transport
 handshake and isn't covered by this section.
 
 Here is an example handshake between a version 2 aware client and a version
 2 aware server:
 
    c:  upgrade 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a proto=ssh-v2
    c:  hello\n
    c:  between\n
    c:  pairs 81\n
    c:  0000000000000000000000000000000000000000-0000000000000000000000000000000000000000
    s: upgraded 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a ssh-v2\n
    s: <additional transport specific data>
 
 The client-issued token that is echoed in the response provides a more
 resilient mechanism for differentiating *banner* output from Mercurial
 output. In version 1, properly formatted banner output could get confused
 for Mercurial server output. By submitting a randomly generated token
 that is then present in the response, the client can look for that token
 in response lines and have reasonable certainty that the line did not
 originate from a *banner* message.
 
 SSH Version 1 Transport
 -----------------------
 
 The SSH transport (version 1) is a custom text-based protocol suitable for
 use over any bi-directional stream transport. It is most commonly used with
 SSH.
 
 A SSH transport server can be started with ``hg serve --stdio``. The stdin,
 stderr, and stdout file descriptors of the started process are used to exchange
 data. When Mercurial connects to a remote server over SSH, it actually starts
 a ``hg serve --stdio`` process on the remote server.
 
 Commands are issued by sending the command name followed by a trailing newline
 ``\n`` to the server. e.g. ``capabilities\n``.
 
 Command arguments are sent in the following format::
 
     <argument> <length>\n<value>
 
 That is, the argument string name followed by a space followed by the
 integer length of the value (expressed as a string) followed by a newline
 (``\n``) followed by the raw argument value.
 
 Dictionary arguments are encoded differently::
 
     <argument> <# elements>\n
     <key1> <length1>\n<value1>
     <key2> <length2>\n<value2>
     ...
 
 Non-argument data is sent immediately after the final argument value. It is
 encoded in chunks::
 
     <length>\n<data>
 
 Each command declares a list of supported arguments and their types. If a
 client sends an unknown argument to the server, the server should abort
 immediately. The special argument ``*`` in a command's definition indicates
 that all argument names are allowed.
 
 The definition of supported arguments and types is initially made when a
 new command is implemented. The client and server must initially independently
 agree on the arguments and their types. This initial set of arguments can be
 supplemented through the presence of *capabilities* advertised by the server.
 
 Each command has a defined expected response type.
 
 A ``string`` response type is a length framed value. The response consists of
 the string encoded integer length of a value followed by a newline (``\n``)
 followed by the value. Empty values are allowed (and are represented as
 ``0\n``).
 
 A ``stream`` response type consists of raw bytes of data. There is no framing.
 
 A generic error response type is also supported. It consists of a an error
 message written to ``stderr`` followed by ``\n-\n``. In addition, ``\n`` is
 written to ``stdout``.
 
 If the server receives an unknown command, it will send an empty ``string``
 response.
 
 The server terminates if it receives an empty command (a ``\n`` character).
 
 If the server announces support for the ``protocaps`` capability, the client
 should issue a ``protocaps`` command after the initial handshake to annonunce
 its own capabilities. The client capabilities are persistent.
 
 SSH Version 2 Transport
 -----------------------
 
 **Experimental and under development**
 
 Version 2 of the SSH transport behaves identically to version 1 of the SSH
 transport with the exception of handshake semantics. See above for how
 version 2 of the SSH transport is negotiated.
 
 Immediately following the ``upgraded`` line signaling a switch to version
 2 of the SSH protocol, the server automatically sends additional details
 about the capabilities of the remote server. This has the form:
 
    <integer length of value>\n
    capabilities: ...\n
 
 e.g.
 
    s: upgraded 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a ssh-v2\n
    s: 240\n
    s: capabilities: known getbundle batch ...\n
 
 Following capabilities advertisement, the peers communicate using version
 1 of the SSH transport.
 
 Unified Frame-Based Protocol
 ============================
 
 **Experimental and under development**
 
 The *Unified Frame-Based Protocol* is a communications protocol between
 Mercurial peers. The protocol aims to be mostly transport agnostic
 (works similarly on HTTP, SSH, etc).
 
 To operate the protocol, a bi-directional, half-duplex pipe supporting
 ordered sends and receives is required. That is, each peer has one pipe
 for sending data and another for receiving.
 
 All data is read and written in atomic units called *frames*. These
 are conceptually similar to TCP packets. Higher-level functionality
 is built on the exchange and processing of frames.
 
 All frames are associated with a *stream*. A *stream* provides a
 unidirectional grouping of frames. Streams facilitate two goals:
 content encoding and parallelism. There is a dedicated section on
 streams below.
 
 The protocol is request-response based: the client issues requests to
 the server, which issues replies to those requests. Server-initiated
 messaging is not currently supported, but this specification carves
 out room to implement it.
 
 All frames are associated with a numbered request. Frames can thus
 be logically grouped by their request ID.
 
 Frames begin with an 8 octet header followed by a variable length
 payload::
 
     +------------------------------------------------+
     |                 Length (24)                    |
     +--------------------------------+---------------+
     |         Request ID (16)        | Stream ID (8) |
     +------------------+-------------+---------------+
     | Stream Flags (8) |
     +-----------+------+
     | Type (4)  |
     +-----------+
     | Flags (4) |
     +===========+===================================================|
     |                     Frame Payload (0...)                    ...
     +---------------------------------------------------------------+
 
 The length of the frame payload is expressed as an unsigned 24 bit
 little endian integer. Values larger than 65535 MUST NOT be used unless
 given permission by the server as part of the negotiated capabilities
 during the handshake. The frame header is not part of the advertised
 frame length. The payload length is the over-the-wire length. If there
 is content encoding applied to the payload as part of the frame's stream,
 the length is the output of that content encoding, not the input.
 
 The 16-bit ``Request ID`` field denotes the integer request identifier,
 stored as an unsigned little endian integer. Odd numbered requests are
 client-initiated. Even numbered requests are server-initiated. This
 refers to where the *request* was initiated - not where the *frame* was
 initiated, so servers will send frames with odd ``Request ID`` in
 response to client-initiated requests. Implementations are advised to
 start ordering request identifiers at ``1`` and ``0``, increment by
 ``2``, and wrap around if all available numbers have been exhausted.
 
 The 8-bit ``Stream ID`` field denotes the stream that the frame is
 associated with. Frames belonging to a stream may have content
 encoding applied and the receiver may need to decode the raw frame
 payload to obtain the original data. Odd numbered IDs are
 client-initiated. Even numbered IDs are server-initiated.
 
 The 8-bit ``Stream Flags`` field defines stream processing semantics.
 See the section on streams below.
 
 The 4-bit ``Type`` field denotes the type of frame being sent.
 
 The 4-bit ``Flags`` field defines special, per-type attributes for
 the frame.
 
 The sections below define the frame types and their behavior.
 
 Command Request (``0x01``)
 --------------------------
 
 This frame contains a request to run a command.
 
 The payload consists of a CBOR map defining the command request. The
 bytestring keys of that map are:
 
 name
    Name of the command that should be executed (bytestring).
 args
    Map of bytestring keys to various value types containing the named
    arguments to this command.
 
    Each command defines its own set of argument names and their expected
    types.
 
 This frame type MUST ONLY be sent from clients to servers: it is illegal
 for a server to send this frame to a client.
 
 The following flag values are defined for this type:
 
 0x01
    New command request. When set, this frame represents the beginning
    of a new request to run a command. The ``Request ID`` attached to this
    frame MUST NOT be active.
 0x02
    Command request continuation. When set, this frame is a continuation
    from a previous command request frame for its ``Request ID``. This
    flag is set when the CBOR data for a command request does not fit
    in a single frame.
 0x04
    Additional frames expected. When set, the command request didn't fit
    into a single frame and additional CBOR data follows in a subsequent
    frame.
 0x08
    Command data frames expected. When set, command data frames are
    expected to follow the final command request frame for this request.
 
 ``0x01`` MUST be set on the initial command request frame for a
 ``Request ID``.
 
 ``0x01`` or ``0x02`` MUST be set to indicate this frame's role in
 a series of command request frames.
 
 If command data frames are to be sent, ``0x10`` MUST be set on ALL
 command request frames.
 
 Command Data (``0x03``)
 -----------------------
 
 This frame contains raw data for a command.
 
 Most commands can be executed by specifying arguments. However,
 arguments have an upper bound to their length. For commands that
 accept data that is beyond this length or whose length isn't known
 when the command is initially sent, they will need to stream
 arbitrary data to the server. This frame type facilitates the sending
 of this data.
 
 The payload of this frame type consists of a stream of raw data to be
 consumed by the command handler on the server. The format of the data
 is command specific.
 
 The following flag values are defined for this type:
 
 0x01
    Command data continuation. When set, the data for this command
    continues into a subsequent frame.
 
 0x02
    End of data. When set, command data has been fully sent to the
    server. The command has been fully issued and no new data for this
    command will be sent. The next frame will belong to a new command.
 
 Response Data (``0x04``)
 ------------------------
 
 This frame contains raw response data to an issued command.
 
 The following flag values are defined for this type:
 
 0x01
    Data continuation. When set, an additional frame containing response data
    will follow.
 0x02
    End of data. When set, the response data has been fully sent and
    no additional frames for this response will be sent.
 0x04
    CBOR data. When set, the frame payload consists of CBOR data.
 
 The ``0x01`` flag is mutually exclusive with the ``0x02`` flag.
 
 Error Response (``0x05``)
 -------------------------
 
 An error occurred when processing a request. This could indicate
 a protocol-level failure or an application level failure depending
 on the flags for this message type.
 
 The payload for this type is an error message that should be
 displayed to the user.
 
 The following flag values are defined for this type:
 
 0x01
    The error occurred at the transport/protocol level. If set, the
    connection should be closed.
 0x02
    The error occurred at the application level. e.g. invalid command.
 
 Human Output Side-Channel (``0x06``)
 ------------------------------------
 
 This frame contains a message that is intended to be displayed to
 people. Whereas most frames communicate machine readable data, this
 frame communicates textual data that is intended to be shown to
 humans.
 
 The frame consists of a series of *formatting requests*. Each formatting
 request consists of a formatting string, arguments for that formatting
 string, and labels to apply to that formatting string.
 
 A formatting string is a printf()-like string that allows variable
 substitution within the string. Labels allow the rendered text to be
 *decorated*. Assuming use of the canonical Mercurial code base, a
 formatting string can be the input to the ``i18n._`` function. This
 allows messages emitted from the server to be localized. So even if
 the server has different i18n settings, people could see messages in
 their *native* settings. Similarly, the use of labels allows
 decorations like coloring and underlining to be applied using the
 client's configured rendering settings.
 
 Formatting strings are similar to ``printf()`` strings or how
 Python's ``%`` operator works. The only supported formatting sequences
 are ``%s`` and ``%%``. ``%s`` will be replaced by whatever the string
 at that position resolves to. ``%%`` will be replaced by ``%``. All
 other 2-byte sequences beginning with ``%`` represent a literal
 ``%`` followed by that character. However, future versions of the
 wire protocol reserve the right to allow clients to opt in to receiving
 formatting strings with additional formatters, hence why ``%%`` is
 required to represent the literal ``%``.
 
 The frame payload consists of a CBOR array of CBOR maps. Each map
 defines an *atom* of text data to print. Each *atom* has the following
 bytestring keys:
 
 msg
    (bytestring) The formatting string. Content MUST be ASCII.
 args (optional)
    Array of bytestrings defining arguments to the formatting string.
 labels (optional)
    Array of bytestrings defining labels to apply to this atom.
 
 All data to be printed MUST be encoded into a single frame: this frame
 does not support spanning data across multiple frames.
 
 All textual data encoded in these frames is assumed to be line delimited.
 The last atom in the frame SHOULD end with a newline (``\n``). If it
 doesn't, clients MAY add a newline to facilitate immediate printing.
 
 Progress Update (``0x07``)
 --------------------------
 
 This frame holds the progress of an operation on the peer. Consumption
 of these frames allows clients to display progress bars, estimated
 completion times, etc.
 
 Each frame defines the progress of a single operation on the peer. The
 payload consists of a CBOR map with the following bytestring keys:
 
 topic
    Topic name (string)
 pos
    Current numeric position within the topic (integer)
 total
    Total/end numeric position of this topic (unsigned integer)
 label (optional)
    Unit label (string)
 item (optional)
    Item name (string)
 
 Progress state is created when a frame is received referencing a
 *topic* that isn't currently tracked. Progress tracking for that
 *topic* is finished when a frame is received reporting the current
 position of that topic as ``-1``.
 
 Multiple *topics* may be active at any given time.
 
 Rendering of progress information is not mandated or governed by this
 specification: implementations MAY render progress information however
 they see fit, including not at all.
 
 The string data describing the topic SHOULD be static strings to
 facilitate receivers localizing that string data. The emitter
 MUST normalize all string data to valid UTF-8 and receivers SHOULD
 validate that received data conforms to UTF-8. The topic name
 SHOULD be ASCII.
 
 Stream Encoding Settings (``0x08``)
 -----------------------------------
 
 This frame type holds information defining the content encoding
 settings for a *stream*.
 
 This frame type is likely consumed by the protocol layer and is not
 passed on to applications.
 
 This frame type MUST ONLY occur on frames having the *Beginning of Stream*
 ``Stream Flag`` set.
 
 The payload of this frame defines what content encoding has (possibly)
 been applied to the payloads of subsequent frames in this stream.
 
 The payload begins with an 8-bit integer defining the length of the
 encoding *profile*, followed by the string name of that profile, which
 must be an ASCII string. All bytes that follow can be used by that
 profile for supplemental settings definitions. See the section below
 on defined encoding profiles.
 
 Stream States and Flags
 -----------------------
 
 Streams can be in two states: *open* and *closed*. An *open* stream
 is active and frames attached to that stream could arrive at any time.
 A *closed* stream is not active. If a frame attached to a *closed*
 stream arrives, that frame MUST have an appropriate stream flag
 set indicating beginning of stream. All streams are in the *closed*
 state by default.
 
 The ``Stream Flags`` field denotes a set of bit flags for defining
 the relationship of this frame within a stream. The following flags
 are defined:
 
 0x01
    Beginning of stream. The first frame in the stream MUST set this
    flag. When received, the ``Stream ID`` this frame is attached to
    becomes ``open``.
 
 0x02
    End of stream. The last frame in a stream MUST set this flag. When
    received, the ``Stream ID`` this frame is attached to becomes
    ``closed``. Any content encoding context associated with this stream
    can be destroyed after processing the payload of this frame.
 
 0x04
    Apply content encoding. When set, any content encoding settings
    defined by the stream should be applied when attempting to read
    the frame. When not set, the frame payload isn't encoded.
 
 Streams
 -------
 
 Streams - along with ``Request IDs`` - facilitate grouping of frames.
 But the purpose of each is quite different and the groupings they
 constitute are independent.
 
 A ``Request ID`` is essentially a tag. It tells you which logical
 request a frame is associated with.
 
 A *stream* is a sequence of frames grouped for the express purpose
 of applying a stateful encoding or for denoting sub-groups of frames.
 
 Unlike ``Request ID``s which span the request and response, a stream
 is unidirectional and stream IDs are independent from client to
 server.
 
 There is no strict hierarchical relationship between ``Request IDs``
 and *streams*. A stream can contain frames having multiple
 ``Request IDs``. Frames belonging to the same ``Request ID`` can
 span multiple streams.
 
 One goal of streams is to facilitate content encoding. A stream can
 define an encoding to be applied to frame payloads. For example, the
 payload transmitted over the wire may contain output from a
 zstandard compression operation and the receiving end may decompress
 that payload to obtain the original data.
 
 The other goal of streams is to facilitate concurrent execution. For
 example, a server could spawn 4 threads to service a request that can
 be easily parallelized. Each of those 4 threads could write into its
 own stream. Those streams could then in turn be delivered to 4 threads
 on the receiving end, with each thread consuming its stream in near
 isolation. The *main* thread on both ends merely does I/O and
 encodes/decodes frame headers: the bulk of the work is done by worker
 threads.
 
 In addition, since content encoding is defined per stream, each
 *worker thread* could perform potentially CPU bound work concurrently
 with other threads. This approach of applying encoding at the
 sub-protocol / stream level eliminates a potential resource constraint
 on the protocol stream as a whole (it is common for the throughput of
 a compression engine to be smaller than the throughput of a network).
 
 Having multiple streams - each with their own encoding settings - also
 facilitates the use of advanced data compression techniques. For
 example, a transmitter could see that it is generating data faster
 and slower than the receiving end is consuming it and adjust its
 compression settings to trade CPU for compression ratio accordingly.
 
 While streams can define a content encoding, not all frames within
 that stream must use that content encoding. This can be useful when
 data is being served from caches and being derived dynamically. A
 cache could pre-compressed data so the server doesn't have to
 recompress it. The ability to pick and choose which frames are
 compressed allows servers to easily send data to the wire without
 involving potentially expensive encoding overhead.
 
 Content Encoding Profiles
 -------------------------
 
 Streams can have named content encoding *profiles* associated with
 them. A profile defines a shared understanding of content encoding
 settings and behavior.
 
 The following profiles are defined:
 
 TBD
 
 Issuing Commands
 ----------------
 
 A client can request that a remote run a command by sending it
 frames defining that command. This logical stream is composed of
 1 or more ``Command Request`` frames and and 0 or more ``Command Data``
 frames.
 
 All frames composing a single command request MUST be associated with
 the same ``Request ID``.
 
 Clients MAY send additional command requests without waiting on the
 response to a previous command request. If they do so, they MUST ensure
 that the ``Request ID`` field of outbound frames does not conflict
 with that of an active ``Request ID`` whose response has not yet been
 fully received.
 
 Servers MAY respond to commands in a different order than they were
 sent over the wire. Clients MUST be prepared to deal with this. Servers
 also MAY start executing commands in a different order than they were
 received, or MAY execute multiple commands concurrently.
 
 If there is a dependency between commands or a race condition between
 commands executing (e.g. a read-only command that depends on the results
 of a command that mutates the repository), then clients MUST NOT send
 frames issuing a command until a response to all dependent commands has
 been received.
 TODO think about whether we should express dependencies between commands
 to avoid roundtrip latency.
 
 A command is defined by a command name, 0 or more command arguments,
 and optional command data.
 
 Arguments are the recommended mechanism for transferring fixed sets of
 parameters to a command. Data is appropriate for transferring variable
 data. Thinking in terms of HTTP, arguments would be headers and data
 would be the message body.
 
 It is recommended for servers to delay the dispatch of a command
 until all argument have been received. Servers MAY impose limits on the
 maximum argument size.
 TODO define failure mechanism.
 
 Servers MAY dispatch to commands immediately once argument data
 is available or delay until command data is received in full.
 
 Capabilities
 ============
 
 Servers advertise supported wire protocol features. This allows clients to
 probe for server features before blindly calling a command or passing a
 specific argument.
 
 The server's features are exposed via a *capabilities* string. This is a
 space-delimited string of tokens/features. Some features are single words
 like ``lookup`` or ``batch``. Others are complicated key-value pairs
 advertising sub-features. e.g. ``httpheader=2048``. When complex, non-word
 values are used, each feature name can define its own encoding of sub-values.
 Comma-delimited and ``x-www-form-urlencoded`` values are common.
 
 The following document capabilities defined by the canonical Mercurial server
 implementation.
 
 batch
 -----
 
 Whether the server supports the ``batch`` command.
 
 This capability/command was introduced in Mercurial 1.9 (released July 2011).
 
 branchmap
 ---------
 
 Whether the server supports the ``branchmap`` command.
 
 This capability/command was introduced in Mercurial 1.3 (released July 2009).
 
 bundle2-exp
 -----------
 
 Precursor to ``bundle2`` capability that was used before bundle2 was a
 stable feature.
 
 This capability was introduced in Mercurial 3.0 behind an experimental
 flag. This capability should not be observed in the wild.
 
 bundle2
 -------
 
 Indicates whether the server supports the ``bundle2`` data exchange format.
 
 The value of the capability is a URL quoted, newline (``\n``) delimited
 list of keys or key-value pairs.
 
 A key is simply a URL encoded string.
 
 A key-value pair is a URL encoded key separated from a URL encoded value by
 an ``=``. If the value is a list, elements are delimited by a ``,`` after
 URL encoding.
 
 For example, say we have the values::
 
   {'HG20': [], 'changegroup': ['01', '02'], 'digests': ['sha1', 'sha512']}
 
 We would first construct a string::
 
   HG20\nchangegroup=01,02\ndigests=sha1,sha512
 
 We would then URL quote this string::
 
   HG20%0Achangegroup%3D01%2C02%0Adigests%3Dsha1%2Csha512
 
 This capability was introduced in Mercurial 3.4 (released May 2015).
 
 changegroupsubset
 -----------------
 
 Whether the server supports the ``changegroupsubset`` command.
 
 This capability was introduced in Mercurial 0.9.2 (released December
 2006).
 
 This capability was introduced at the same time as the ``lookup``
 capability/command.
 
 compression
 -----------
 
 Declares support for negotiating compression formats.
 
 Presence of this capability indicates the server supports dynamic selection
 of compression formats based on the client request.
 
 Servers advertising this capability are required to support the
 ``application/mercurial-0.2`` media type in response to commands returning
 streams. Servers may support this media type on any command.
 
 The value of the capability is a comma-delimited list of strings declaring
 supported compression formats. The order of the compression formats is in
 server-preferred order, most preferred first.
 
 The identifiers used by the official Mercurial distribution are:
 
 bzip2
    bzip2
 none
    uncompressed / raw data
 zlib
    zlib (no gzip header)
 zstd
    zstd
 
 This capability was introduced in Mercurial 4.1 (released February 2017).
 
 getbundle
 ---------
 
 Whether the server supports the ``getbundle`` command.
 
 This capability was introduced in Mercurial 1.9 (released July 2011).
 
 httpheader
 ----------
 
 Whether the server supports receiving command arguments via HTTP request
 headers.
 
 The value of the capability is an integer describing the max header
 length that clients should send. Clients should ignore any content after a
 comma in the value, as this is reserved for future use.
 
 This capability was introduced in Mercurial 1.9 (released July 2011).
 
 httpmediatype
 -------------
 
 Indicates which HTTP media types (``Content-Type`` header) the server is
 capable of receiving and sending.
 
 The value of the capability is a comma-delimited list of strings identifying
 support for media type and transmission direction. The following strings may
 be present:
 
 0.1rx
    Indicates server support for receiving ``application/mercurial-0.1`` media
    types.
 
 0.1tx
    Indicates server support for sending ``application/mercurial-0.1`` media
    types.
 
 0.2rx
    Indicates server support for receiving ``application/mercurial-0.2`` media
    types.
 
 0.2tx
    Indicates server support for sending ``application/mercurial-0.2`` media
    types.
 
 minrx=X
    Minimum media type version the server is capable of receiving. Value is a
    string like ``0.2``.
 
    This capability can be used by servers to limit connections from legacy
    clients not using the latest supported media type. However, only clients
    with knowledge of this capability will know to consult this value. This
    capability is present so the client may issue a more user-friendly error
    when the server has locked out a legacy client.
 
 mintx=X
    Minimum media type version the server is capable of sending. Value is a
    string like ``0.1``.
 
 Servers advertising support for the ``application/mercurial-0.2`` media type
 should also advertise the ``compression`` capability.
 
 This capability was introduced in Mercurial 4.1 (released February 2017).
 
 httppostargs
 ------------
 
 **Experimental**
 
 Indicates that the server supports and prefers clients send command arguments
 via a HTTP POST request as part of the request body.
 
 This capability was introduced in Mercurial 3.8 (released May 2016).
 
 known
 -----
 
 Whether the server supports the ``known`` command.
 
 This capability/command was introduced in Mercurial 1.9 (released July 2011).
 
 lookup
 ------
 
 Whether the server supports the ``lookup`` command.
 
 This capability was introduced in Mercurial 0.9.2 (released December
 2006).
 
 This capability was introduced at the same time as the ``changegroupsubset``
 capability/command.
 
 partial-pull
 ------------
 
 Indicates that the client can deal with partial answers to pull requests
 by repeating the request.
 
 If this parameter is not advertised, the server will not send pull bundles.
 
 This client capability was introduced in Mercurial 4.6.
 
 protocaps
 ---------
 
 Whether the server supports the ``protocaps`` command for SSH V1 transport.
 
 This capability was introduced in Mercurial 4.6.
 
 pushkey
 -------
 
 Whether the server supports the ``pushkey`` and ``listkeys`` commands.
 
 This capability was introduced in Mercurial 1.6 (released July 2010).
 
 standardbundle
 --------------
 
 **Unsupported**
 
 This capability was introduced during the Mercurial 0.9.2 development cycle in
 2006. It was never present in a release, as it was replaced by the ``unbundle``
 capability. This capability should not be encountered in the wild.
 
 stream-preferred
 ----------------
 
 If present the server prefers that clients clone using the streaming clone
 protocol (``hg clone --stream``) rather than the standard
 changegroup/bundle based protocol.
 
 This capability was introduced in Mercurial 2.2 (released May 2012).
 
 streamreqs
 ----------
 
 Indicates whether the server supports *streaming clones* and the *requirements*
 that clients must support to receive it.
 
 If present, the server supports the ``stream_out`` command, which transmits
 raw revlogs from the repository instead of changegroups. This provides a faster
 cloning mechanism at the expense of more bandwidth used.
 
 The value of this capability is a comma-delimited list of repo format
 *requirements*. These are requirements that impact the reading of data in
 the ``.hg/store`` directory. An example value is
 ``streamreqs=generaldelta,revlogv1`` indicating the server repo requires
 the ``revlogv1`` and ``generaldelta`` requirements.
 
 If the only format requirement is ``revlogv1``, the server may expose the
 ``stream`` capability instead of the ``streamreqs`` capability.
 
 This capability was introduced in Mercurial 1.7 (released November 2010).
 
 stream
 ------
 
 Whether the server supports *streaming clones* from ``revlogv1`` repos.
 
 If present, the server supports the ``stream_out`` command, which transmits
 raw revlogs from the repository instead of changegroups. This provides a faster
 cloning mechanism at the expense of more bandwidth used.
 
 This capability was introduced in Mercurial 0.9.1 (released July 2006).
 
 When initially introduced, the value of the capability was the numeric
 revlog revision. e.g. ``stream=1``. This indicates the changegroup is using
 ``revlogv1``. This simple integer value wasn't powerful enough, so the
 ``streamreqs`` capability was invented to handle cases where the repo
 requirements have more than just ``revlogv1``. Newer servers omit the
 ``=1`` since it was the only value supported and the value of ``1`` can
 be implied by clients.
 
 unbundlehash
 ------------
 
 Whether the ``unbundle`` commands supports receiving a hash of all the
 heads instead of a list.
 
 For more, see the documentation for the ``unbundle`` command.
 
 This capability was introduced in Mercurial 1.9 (released July 2011).
 
 unbundle
 --------
 
 Whether the server supports pushing via the ``unbundle`` command.
 
 This capability/command has been present since Mercurial 0.9.1 (released
 July 2006).
 
 Mercurial 0.9.2 (released December 2006) added values to the capability
 indicating which bundle types the server supports receiving. This value is a
 comma-delimited list. e.g. ``HG10GZ,HG10BZ,HG10UN``. The order of values
 reflects the priority/preference of that type, where the first value is the
 most preferred type.
 
 Content Negotiation
 ===================
 
 The wire protocol has some mechanisms to help peers determine what content
 types and encoding the other side will accept. Historically, these mechanisms
 have been built into commands themselves because most commands only send a
 well-defined response type and only certain commands needed to support
 functionality like compression.
 
 Currently, only the HTTP version 1 transport supports content negotiation
 at the protocol layer.
 
 HTTP requests advertise supported response formats via the ``X-HgProto-<N>``
 request header, where ``<N>`` is an integer starting at 1 allowing the logical
 value to span multiple headers. This value consists of a list of
 space-delimited parameters. Each parameter denotes a feature or capability.
 
 The following parameters are defined:
 
 0.1
    Indicates the client supports receiving ``application/mercurial-0.1``
    responses.
 
 0.2
    Indicates the client supports receiving ``application/mercurial-0.2``
    responses.
 
 comp
    Indicates compression formats the client can decode. Value is a list of
    comma delimited strings identifying compression formats ordered from
    most preferential to least preferential. e.g. ``comp=zstd,zlib,none``.
 
    This parameter does not have an effect if only the ``0.1`` parameter
    is defined, as support for ``application/mercurial-0.2`` or greater is
    required to use arbitrary compression formats.
 
    If this parameter is not advertised, the server interprets this as
    equivalent to ``zlib,none``.
 
 Clients may choose to only send this header if the ``httpmediatype``
 server capability is present, as currently all server-side features
 consulting this header require the client to opt in to new protocol features
 advertised via the ``httpmediatype`` capability.
 
 A server that doesn't receive an ``X-HgProto-<N>`` header should infer a
 value of ``0.1``. This is compatible with legacy clients.
 
 A server receiving a request indicating support for multiple media type
 versions may respond with any of the supported media types. Not all servers
 may support all media types on all commands.
 
 Commands
 ========
 
 This section contains a list of all wire protocol commands implemented by
 the canonical Mercurial server.
 
 batch
 -----
 
 Issue multiple commands while sending a single command request. The purpose
 of this command is to allow a client to issue multiple commands while avoiding
 multiple round trips to the server therefore enabling commands to complete
 quicker.
 
 The command accepts a ``cmds`` argument that contains a list of commands to
 execute.
 
 The value of ``cmds`` is a ``;`` delimited list of strings. Each string has the
 form ``<command> <arguments>``. That is, the command name followed by a space
 followed by an argument string.
 
 The argument string is a ``,`` delimited list of ``<key>=<value>`` values
 corresponding to command arguments. Both the argument name and value are
 escaped using a special substitution map::
 
    : -> :c
    , -> :o
    ; -> :s
    = -> :e
 
 The response type for this command is ``string``. The value contains a
 ``;`` delimited list of responses for each requested command. Each value
 in this list is escaped using the same substitution map used for arguments.
 
 If an error occurs, the generic error response may be sent.
 
 between
 -------
 
 (Legacy command used for discovery in old clients)
 
 Obtain nodes between pairs of nodes.
 
 The ``pairs`` arguments contains a space-delimited list of ``-`` delimited
 hex node pairs. e.g.::
 
    a072279d3f7fd3a4aa7ffa1a5af8efc573e1c896-6dc58916e7c070f678682bfe404d2e2d68291a18
 
 Return type is a ``string``. Value consists of lines corresponding to each
 requested range. Each line contains a space-delimited list of hex nodes.
 A newline ``\n`` terminates each line, including the last one.
 
 branchmap
 ---------
 
 Obtain heads in named branches.
 
 Accepts no arguments. Return type is a ``string``.
 
 Return value contains lines with URL encoded branch names followed by a space
 followed by a space-delimited list of hex nodes of heads on that branch.
 e.g.::
 
     default a072279d3f7fd3a4aa7ffa1a5af8efc573e1c896 6dc58916e7c070f678682bfe404d2e2d68291a18
     stable baae3bf31522f41dd5e6d7377d0edd8d1cf3fccc
 
 There is no trailing newline.
 
 branches
 --------
 
 (Legacy command used for discovery in old clients. Clients with ``getbundle``
 use the ``known`` and ``heads`` commands instead.)
 
 Obtain ancestor changesets of specific nodes back to a branch point.
 
 Despite the name, this command has nothing to do with Mercurial named branches.
 Instead, it is related to DAG branches.
 
 The command accepts a ``nodes`` argument, which is a string of space-delimited
 hex nodes.
 
 For each node requested, the server will find the first ancestor node that is
 a DAG root or is a merge.
 
 Return type is a ``string``. Return value contains lines with result data for
 each requested node. Each line contains space-delimited nodes followed by a
 newline (``\n``). The 4 nodes reported on each line correspond to the requested
 node, the ancestor node found, and its 2 parent nodes (which may be the null
 node).
 
 capabilities
 ------------
 
 Obtain the capabilities string for the repo.
 
 Unlike the ``hello`` command, the capabilities string is not prefixed.
 There is no trailing newline.
 
 This command does not accept any arguments. Return type is a ``string``.
 
 This command was introduced in Mercurial 0.9.1 (released July 2006).
 
 changegroup
 -----------
 
 (Legacy command: use ``getbundle`` instead)
 
 Obtain a changegroup version 1 with data for changesets that are
 descendants of client-specified changesets.
 
 The ``roots`` arguments contains a list of space-delimited hex nodes.
 
 The server responds with a changegroup version 1 containing all
 changesets between the requested root/base nodes and the repo's head nodes
 at the time of the request.
 
 The return type is a ``stream``.
 
 changegroupsubset
 -----------------
 
 (Legacy command: use ``getbundle`` instead)
 
 Obtain a changegroup version 1 with data for changesetsets between
 client specified base and head nodes.
 
 The ``bases`` argument contains a list of space-delimited hex nodes.
 The ``heads`` argument contains a list of space-delimited hex nodes.
 
 The server responds with a changegroup version 1 containing all
 changesets between the requested base and head nodes at the time of the
 request.
 
 The return type is a ``stream``.
 
 clonebundles
 ------------
 
 Obtains a manifest of bundle URLs available to seed clones.
 
 Each returned line contains a URL followed by metadata. See the
 documentation in the ``clonebundles`` extension for more.
 
 The return type is a ``string``.
 
 getbundle
 ---------
 
 Obtain a bundle containing repository data.
 
 This command accepts the following arguments:
 
 heads
    List of space-delimited hex nodes of heads to retrieve.
 common
    List of space-delimited hex nodes that the client has in common with the
    server.
 obsmarkers
    Boolean indicating whether to include obsolescence markers as part
    of the response. Only works with bundle2.
 bundlecaps
    Comma-delimited set of strings defining client bundle capabilities.
 listkeys
    Comma-delimited list of strings of ``pushkey`` namespaces. For each
    namespace listed, a bundle2 part will be included with the content of
    that namespace.
 cg
    Boolean indicating whether changegroup data is requested.
 cbattempted
    Boolean indicating whether the client attempted to use the *clone bundles*
    feature before performing this request.
 bookmarks
    Boolean indicating whether bookmark data is requested.
 phases
    Boolean indicating whether phases data is requested.
 
 The return type on success is a ``stream`` where the value is bundle.
 On the HTTP version 1 transport, the response is zlib compressed.
 
 If an error occurs, a generic error response can be sent.
 
 Unless the client sends a false value for the ``cg`` argument, the returned
 bundle contains a changegroup with the nodes between the specified ``common``
 and ``heads`` nodes. Depending on the command arguments, the type and content
 of the returned bundle can vary significantly.
 
 The default behavior is for the server to send a raw changegroup version
 ``01`` response.
 
 If the ``bundlecaps`` provided by the client contain a value beginning
 with ``HG2``, a bundle2 will be returned. The bundle2 data may contain
 additional repository data, such as ``pushkey`` namespace values.
 
 heads
 -----
 
 Returns a list of space-delimited hex nodes of repository heads followed
 by a newline. e.g.
 ``a9eeb3adc7ddb5006c088e9eda61791c777cbf7c 31f91a3da534dc849f0d6bfc00a395a97cf218a1\n``
 
 This command does not accept any arguments. The return type is a ``string``.
 
 hello
 -----
 
 Returns lines describing interesting things about the server in an RFC-822
 like format.
 
 Currently, the only line defines the server capabilities. It has the form::
 
     capabilities: <value>
 
 See above for more about the capabilities string.
 
 SSH clients typically issue this command as soon as a connection is
 established.
 
 This command does not accept any arguments. The return type is a ``string``.
 
 This command was introduced in Mercurial 0.9.1 (released July 2006).
 
 listkeys
 --------
 
 List values in a specified ``pushkey`` namespace.
 
 The ``namespace`` argument defines the pushkey namespace to operate on.
 
 The return type is a ``string``. The value is an encoded dictionary of keys.
 
 Key-value pairs are delimited by newlines (``\n``). Within each line, keys and
 values are separated by a tab (``\t``). Keys and values are both strings.
 
 lookup
 ------
 
 Try to resolve a value to a known repository revision.
 
 The ``key`` argument is converted from bytes to an
 ``encoding.localstr`` instance then passed into
 ``localrepository.__getitem__`` in an attempt to resolve it.
 
 The return type is a ``string``.
 
 Upon successful resolution, returns ``1 <hex node>\n``. On failure,
 returns ``0 <error string>\n``. e.g.::
 
    1 273ce12ad8f155317b2c078ec75a4eba507f1fba\n
 
    0 unknown revision 'foo'\n
 
 known
 -----
 
 Determine whether multiple nodes are known.
 
 The ``nodes`` argument is a list of space-delimited hex nodes to check
 for existence.
 
 The return type is ``string``.
 
 Returns a string consisting of ``0``s and ``1``s indicating whether nodes
 are known. If the Nth node specified in the ``nodes`` argument is known,
 a ``1`` will be returned at byte offset N. If the node isn't known, ``0``
 will be present at byte offset N.
 
 There is no trailing newline.
 
 protocaps
 ---------
 
 Notify the server about the client capabilities in the SSH V1 transport
 protocol.
 
 The ``caps`` argument is a space-delimited list of capabilities.
 
 The server will reply with the string ``OK``.
 
 pushkey
 -------
 
 Set a value using the ``pushkey`` protocol.
 
 Accepts arguments ``namespace``, ``key``, ``old``, and ``new``, which
 correspond to the pushkey namespace to operate on, the key within that
 namespace to change, the old value (which may be empty), and the new value.
 All arguments are string types.
 
 The return type is a ``string``. The value depends on the transport protocol.
 
 The SSH version 1 transport sends a string encoded integer followed by a
 newline (``\n``) which indicates operation result. The server may send
 additional output on the ``stderr`` stream that should be displayed to the
 user.
 
 The HTTP version 1 transport sends a string encoded integer followed by a
 newline followed by additional server output that should be displayed to
 the user. This may include output from hooks, etc.
 
 The integer result varies by namespace. ``0`` means an error has occurred
 and there should be additional output to display to the user.
 
 stream_out
 ----------
 
 Obtain *streaming clone* data.
 
 The return type is either a ``string`` or a ``stream``, depending on
 whether the request was fulfilled properly.
 
 A return value of ``1\n`` indicates the server is not configured to serve
 this data. If this is seen by the client, they may not have verified the
 ``stream`` capability is set before making the request.
 
 A return value of ``2\n`` indicates the server was unable to lock the
 repository to generate data.
 
 All other responses are a ``stream`` of bytes. The first line of this data
 contains 2 space-delimited integers corresponding to the path count and
 payload size, respectively::
 
     <path count> <payload size>\n
 
 The ``<payload size>`` is the total size of path data: it does not include
 the size of the per-path header lines.
 
 Following that header are ``<path count>`` entries. Each entry consists of a
 line with metadata followed by raw revlog data. The line consists of::
 
     <store path>\0<size>\n
 
 The ``<store path>`` is the encoded store path of the data that follows.
 ``<size>`` is the amount of data for this store path/revlog that follows the
 newline.
 
 There is no trailer to indicate end of data. Instead, the client should stop
 reading after ``<path count>`` entries are consumed.
 
 unbundle
 --------
 
 Send a bundle containing data (usually changegroup data) to the server.
 
 Accepts the argument ``heads``, which is a space-delimited list of hex nodes
 corresponding to server repository heads observed by the client. This is used
 to detect race conditions and abort push operations before a server performs
 too much work or a client transfers too much data.
 
 The request payload consists of a bundle to be applied to the repository,
 similarly to as if :hg:`unbundle` were called.
 
 In most scenarios, a special ``push response`` type is returned. This type
 contains an integer describing the change in heads as a result of the
 operation. A value of ``0`` indicates nothing changed. ``1`` means the number
 of heads remained the same. Values ``2`` and larger indicate the number of
 added heads minus 1. e.g. ``3`` means 2 heads were added. Negative values
 indicate the number of fewer heads, also off by 1. e.g. ``-2`` means there
 is 1 fewer head.
 
 The encoding of the ``push response`` type varies by transport.
 
 For the SSH version 1 transport, this type is composed of 2 ``string``
 responses: an empty response (``0\n``) followed by the integer result value.
 e.g. ``1\n2``. So the full response might be ``0\n1\n2``.
 
 For the HTTP version 1 transport, the response is a ``string`` type composed
 of an integer result value followed by a newline (``\n``) followed by string
 content holding server output that should be displayed on the client (output
 hooks, etc).
 
 In some cases, the server may respond with a ``bundle2`` bundle. In this
 case, the response type is ``stream``. For the HTTP version 1 transport, the
 response is zlib compressed.
 
 The server may also respond with a generic error type, which contains a string
 indicating the failure.
 
 Frame-Based Protocol Commands
 =============================
 
 **Experimental and under active development**
 
 This section documents the wire protocol commands exposed to transports
 using the frame-based protocol. The set of commands exposed through
 these transports is distinct from the set of commands exposed to legacy
 transports.
 
 The frame-based protocol uses CBOR to encode command execution requests.
 All command arguments must be mapped to a specific or set of CBOR data
 types.
 
 The response to many commands is also CBOR. There is no common response
 format: each command defines its own response format.
 
 TODO require node type be specified, as N bytes of binary node value
 could be ambiguous once SHA-1 is replaced.
 
 branchmap
 ---------
 
 Obtain heads in named branches.
 
 Receives no arguments.
 
 The response is a map with bytestring keys defining the branch name.
 Values are arrays of bytestring defining raw changeset nodes.
 
 capabilities
 ------------
 
 Obtain the server's capabilities.
 
 Receives no arguments.
 
 This command is typically called only as part of the handshake during
 initial connection establishment.
 
 The response is a map with bytestring keys defining server information.
 
 The defined keys are:
 
 commands
    A map defining available wire protocol commands on this server.
 
    Keys in the map are the names of commands that can be invoked. Values
    are maps defining information about that command. The bytestring keys
    are:
 
       args
-         An array of argument names accepted by this command.
+         A map of argument names and their expected types.
+
+         Types are defined as a representative value for the expected type.
+         e.g. an argument expecting a boolean type will have its value
+         set to true. An integer type will have its value set to 42. The
+         actual values are arbitrary and may not have meaning.
       permissions
          An array of permissions required to execute this command.
 
 compression
    An array of maps defining available compression format support.
 
    The array is sorted from most preferred to least preferred.
 
    Each entry has the following bytestring keys:
 
       name
          Name of the compression engine. e.g. ``zstd`` or ``zlib``.
 
 heads
 -----
 
 Obtain DAG heads in the repository.
 
 The command accepts the following arguments:
 
 publiconly (optional)
    (boolean) If set, operate on the DAG for public phase changesets only.
    Non-public (i.e. draft) phase DAG heads will not be returned.
 
 The response is a CBOR array of bytestrings defining changeset nodes
 of DAG heads. The array can be empty if the repository is empty or no
 changesets satisfied the request.
 
 TODO consider exposing phase of heads in response
 
 known
 -----
 
 Determine whether a series of changeset nodes is known to the server.
 
 The command accepts the following arguments:
 
 nodes
    (array of bytestrings) List of changeset nodes whose presence to
    query.
 
 The response is a bytestring where each byte contains a 0 or 1 for the
 corresponding requested node at the same index.
 
 TODO use a bit array for even more compact response
 
 listkeys
 --------
 
 List values in a specified ``pushkey`` namespace.
 
 The command receives the following arguments:
 
 namespace
    (bytestring) Pushkey namespace to query.
 
 The response is a map with bytestring keys and values.
 
 TODO consider using binary to represent nodes in certain pushkey namespaces.

mercurial/wireproto.py

@@ -1,1350 +1,1377 b''
 # wireproto.py - generic wire protocol support functions
 #
 # Copyright 2005-2010 Matt Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 import hashlib
 import os
 import tempfile
 
 from .i18n import _
 from .node import (
     bin,
     hex,
     nullid,
 )
 
 from . import (
     bundle2,
     changegroup as changegroupmod,
     discovery,
     encoding,
     error,
     exchange,
     peer,
     pushkey as pushkeymod,
     pycompat,
     repository,
     streamclone,
     util,
     wireprototypes,
 )
 
 from .utils import (
     procutil,
     stringutil,
 )
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 bundle2requiredmain = _('incompatible Mercurial client; bundle2 required')
 bundle2requiredhint = _('see https://www.mercurial-scm.org/wiki/'
                         'IncompatibleClient')
 bundle2required = '%s\n(%s)\n' % (bundle2requiredmain, bundle2requiredhint)
 
 class remoteiterbatcher(peer.iterbatcher):
     def __init__(self, remote):
         super(remoteiterbatcher, self).__init__()
         self._remote = remote
 
     def __getattr__(self, name):
         # Validate this method is batchable, since submit() only supports
         # batchable methods.
         fn = getattr(self._remote, name)
         if not getattr(fn, 'batchable', None):
             raise error.ProgrammingError('Attempted to batch a non-batchable '
                                          'call to %r' % name)
 
         return super(remoteiterbatcher, self).__getattr__(name)
 
     def submit(self):
         """Break the batch request into many patch calls and pipeline them.
 
         This is mostly valuable over http where request sizes can be
         limited, but can be used in other places as well.
         """
         # 2-tuple of (command, arguments) that represents what will be
         # sent over the wire.
         requests = []
 
         # 4-tuple of (command, final future, @batchable generator, remote
         # future).
         results = []
 
         for command, args, opts, finalfuture in self.calls:
             mtd = getattr(self._remote, command)
             batchable = mtd.batchable(mtd.__self__, *args, **opts)
 
             commandargs, fremote = next(batchable)
             assert fremote
             requests.append((command, commandargs))
             results.append((command, finalfuture, batchable, fremote))
 
         if requests:
             self._resultiter = self._remote._submitbatch(requests)
 
         self._results = results
 
     def results(self):
         for command, finalfuture, batchable, remotefuture in self._results:
             # Get the raw result, set it in the remote future, feed it
             # back into the @batchable generator so it can be decoded, and
             # set the result on the final future to this value.
             remoteresult = next(self._resultiter)
             remotefuture.set(remoteresult)
             finalfuture.set(next(batchable))
 
             # Verify our @batchable generators only emit 2 values.
             try:
                 next(batchable)
             except StopIteration:
                 pass
             else:
                 raise error.ProgrammingError('%s @batchable generator emitted '
                                              'unexpected value count' % command)
 
             yield finalfuture.value
 
 # Forward a couple of names from peer to make wireproto interactions
 # slightly more sensible.
 batchable = peer.batchable
 future = peer.future
 
 # list of nodes encoding / decoding
 
 def decodelist(l, sep=' '):
     if l:
         return [bin(v) for v in  l.split(sep)]
     return []
 
 def encodelist(l, sep=' '):
     try:
         return sep.join(map(hex, l))
     except TypeError:
         raise
 
 # batched call argument encoding
 
 def escapearg(plain):
     return (plain
             .replace(':', ':c')
             .replace(',', ':o')
             .replace(';', ':s')
             .replace('=', ':e'))
 
 def unescapearg(escaped):
     return (escaped
             .replace(':e', '=')
             .replace(':s', ';')
             .replace(':o', ',')
             .replace(':c', ':'))
 
 def encodebatchcmds(req):
     """Return a ``cmds`` argument value for the ``batch`` command."""
     cmds = []
     for op, argsdict in req:
         # Old servers didn't properly unescape argument names. So prevent
         # the sending of argument names that may not be decoded properly by
         # servers.
         assert all(escapearg(k) == k for k in argsdict)
 
         args = ','.join('%s=%s' % (escapearg(k), escapearg(v))
                         for k, v in argsdict.iteritems())
         cmds.append('%s %s' % (op, args))
 
     return ';'.join(cmds)
 
 def clientcompressionsupport(proto):
     """Returns a list of compression methods supported by the client.
 
     Returns a list of the compression methods supported by the client
     according to the protocol capabilities. If no such capability has
     been announced, fallback to the default of zlib and uncompressed.
     """
     for cap in proto.getprotocaps():
         if cap.startswith('comp='):
             return cap[5:].split(',')
     return ['zlib', 'none']
 
 # mapping of options accepted by getbundle and their types
 #
 # Meant to be extended by extensions. It is extensions responsibility to ensure
 # such options are properly processed in exchange.getbundle.
 #
 # supported types are:
 #
 # :nodes: list of binary nodes
 # :csv:   list of comma-separated values
 # :scsv:  list of comma-separated values return as set
 # :plain: string with no transformation needed.
 gboptsmap = {'heads':  'nodes',
              'bookmarks': 'boolean',
              'common': 'nodes',
              'obsmarkers': 'boolean',
              'phases': 'boolean',
              'bundlecaps': 'scsv',
              'listkeys': 'csv',
              'cg': 'boolean',
              'cbattempted': 'boolean',
              'stream': 'boolean',
 }
 
 # client side
 
 class wirepeer(repository.legacypeer):
     """Client-side interface for communicating with a peer repository.
 
     Methods commonly call wire protocol commands of the same name.
 
     See also httppeer.py and sshpeer.py for protocol-specific
     implementations of this interface.
     """
     # Begin of ipeercommands interface.
 
     def iterbatch(self):
         return remoteiterbatcher(self)
 
     @batchable
     def lookup(self, key):
         self.requirecap('lookup', _('look up remote revision'))
         f = future()
         yield {'key': encoding.fromlocal(key)}, f
         d = f.value
         success, data = d[:-1].split(" ", 1)
         if int(success):
             yield bin(data)
         else:
             self._abort(error.RepoError(data))
 
     @batchable
     def heads(self):
         f = future()
         yield {}, f
         d = f.value
         try:
             yield decodelist(d[:-1])
         except ValueError:
             self._abort(error.ResponseError(_("unexpected response:"), d))
 
     @batchable
     def known(self, nodes):
         f = future()
         yield {'nodes': encodelist(nodes)}, f
         d = f.value
         try:
             yield [bool(int(b)) for b in d]
         except ValueError:
             self._abort(error.ResponseError(_("unexpected response:"), d))
 
     @batchable
     def branchmap(self):
         f = future()
         yield {}, f
         d = f.value
         try:
             branchmap = {}
             for branchpart in d.splitlines():
                 branchname, branchheads = branchpart.split(' ', 1)
                 branchname = encoding.tolocal(urlreq.unquote(branchname))
                 branchheads = decodelist(branchheads)
                 branchmap[branchname] = branchheads
             yield branchmap
         except TypeError:
             self._abort(error.ResponseError(_("unexpected response:"), d))
 
     @batchable
     def listkeys(self, namespace):
         if not self.capable('pushkey'):
             yield {}, None
         f = future()
         self.ui.debug('preparing listkeys for "%s"\n' % namespace)
         yield {'namespace': encoding.fromlocal(namespace)}, f
         d = f.value
         self.ui.debug('received listkey for "%s": %i bytes\n'
                       % (namespace, len(d)))
         yield pushkeymod.decodekeys(d)
 
     @batchable
     def pushkey(self, namespace, key, old, new):
         if not self.capable('pushkey'):
             yield False, None
         f = future()
         self.ui.debug('preparing pushkey for "%s:%s"\n' % (namespace, key))
         yield {'namespace': encoding.fromlocal(namespace),
                'key': encoding.fromlocal(key),
                'old': encoding.fromlocal(old),
                'new': encoding.fromlocal(new)}, f
         d = f.value
         d, output = d.split('\n', 1)
         try:
             d = bool(int(d))
         except ValueError:
             raise error.ResponseError(
                 _('push failed (unexpected response):'), d)
         for l in output.splitlines(True):
             self.ui.status(_('remote: '), l)
         yield d
 
     def stream_out(self):
         return self._callstream('stream_out')
 
     def getbundle(self, source, **kwargs):
         kwargs = pycompat.byteskwargs(kwargs)
         self.requirecap('getbundle', _('look up remote changes'))
         opts = {}
         bundlecaps = kwargs.get('bundlecaps') or set()
         for key, value in kwargs.iteritems():
             if value is None:
                 continue
             keytype = gboptsmap.get(key)
             if keytype is None:
                 raise error.ProgrammingError(
                     'Unexpectedly None keytype for key %s' % key)
             elif keytype == 'nodes':
                 value = encodelist(value)
             elif keytype == 'csv':
                 value = ','.join(value)
             elif keytype == 'scsv':
                 value = ','.join(sorted(value))
             elif keytype == 'boolean':
                 value = '%i' % bool(value)
             elif keytype != 'plain':
                 raise KeyError('unknown getbundle option type %s'
                                % keytype)
             opts[key] = value
         f = self._callcompressable("getbundle", **pycompat.strkwargs(opts))
         if any((cap.startswith('HG2') for cap in bundlecaps)):
             return bundle2.getunbundler(self.ui, f)
         else:
             return changegroupmod.cg1unpacker(f, 'UN')
 
     def unbundle(self, cg, heads, url):
         '''Send cg (a readable file-like object representing the
         changegroup to push, typically a chunkbuffer object) to the
         remote server as a bundle.
 
         When pushing a bundle10 stream, return an integer indicating the
         result of the push (see changegroup.apply()).
 
         When pushing a bundle20 stream, return a bundle20 stream.
 
         `url` is the url the client thinks it's pushing to, which is
         visible to hooks.
         '''
 
         if heads != ['force'] and self.capable('unbundlehash'):
             heads = encodelist(['hashed',
                                 hashlib.sha1(''.join(sorted(heads))).digest()])
         else:
             heads = encodelist(heads)
 
         if util.safehasattr(cg, 'deltaheader'):
             # this a bundle10, do the old style call sequence
             ret, output = self._callpush("unbundle", cg, heads=heads)
             if ret == "":
                 raise error.ResponseError(
                     _('push failed:'), output)
             try:
                 ret = int(ret)
             except ValueError:
                 raise error.ResponseError(
                     _('push failed (unexpected response):'), ret)
 
             for l in output.splitlines(True):
                 self.ui.status(_('remote: '), l)
         else:
             # bundle2 push. Send a stream, fetch a stream.
             stream = self._calltwowaystream('unbundle', cg, heads=heads)
             ret = bundle2.getunbundler(self.ui, stream)
         return ret
 
     # End of ipeercommands interface.
 
     # Begin of ipeerlegacycommands interface.
 
     def branches(self, nodes):
         n = encodelist(nodes)
         d = self._call("branches", nodes=n)
         try:
             br = [tuple(decodelist(b)) for b in d.splitlines()]
             return br
         except ValueError:
             self._abort(error.ResponseError(_("unexpected response:"), d))
 
     def between(self, pairs):
         batch = 8 # avoid giant requests
         r = []
         for i in xrange(0, len(pairs), batch):
             n = " ".join([encodelist(p, '-') for p in pairs[i:i + batch]])
             d = self._call("between", pairs=n)
             try:
                 r.extend(l and decodelist(l) or [] for l in d.splitlines())
             except ValueError:
                 self._abort(error.ResponseError(_("unexpected response:"), d))
         return r
 
     def changegroup(self, nodes, kind):
         n = encodelist(nodes)
         f = self._callcompressable("changegroup", roots=n)
         return changegroupmod.cg1unpacker(f, 'UN')
 
     def changegroupsubset(self, bases, heads, kind):
         self.requirecap('changegroupsubset', _('look up remote changes'))
         bases = encodelist(bases)
         heads = encodelist(heads)
         f = self._callcompressable("changegroupsubset",
                                    bases=bases, heads=heads)
         return changegroupmod.cg1unpacker(f, 'UN')
 
     # End of ipeerlegacycommands interface.
 
     def _submitbatch(self, req):
         """run batch request <req> on the server
 
         Returns an iterator of the raw responses from the server.
         """
         ui = self.ui
         if ui.debugflag and ui.configbool('devel', 'debug.peer-request'):
             ui.debug('devel-peer-request: batched-content\n')
             for op, args in req:
                 msg = 'devel-peer-request:    - %s (%d arguments)\n'
                 ui.debug(msg % (op, len(args)))
 
         rsp = self._callstream("batch", cmds=encodebatchcmds(req))
         chunk = rsp.read(1024)
         work = [chunk]
         while chunk:
             while ';' not in chunk and chunk:
                 chunk = rsp.read(1024)
                 work.append(chunk)
             merged = ''.join(work)
             while ';' in merged:
                 one, merged = merged.split(';', 1)
                 yield unescapearg(one)
             chunk = rsp.read(1024)
             work = [merged, chunk]
         yield unescapearg(''.join(work))
 
     def _submitone(self, op, args):
         return self._call(op, **pycompat.strkwargs(args))
 
     def debugwireargs(self, one, two, three=None, four=None, five=None):
         # don't pass optional arguments left at their default value
         opts = {}
         if three is not None:
             opts[r'three'] = three
         if four is not None:
             opts[r'four'] = four
         return self._call('debugwireargs', one=one, two=two, **opts)
 
     def _call(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a simple string.
 
         returns the server reply as a string."""
         raise NotImplementedError()
 
     def _callstream(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a stream. Note that if the
         command doesn't return a stream, _callstream behaves
         differently for ssh and http peers.
 
         returns the server reply as a file like object.
         """
         raise NotImplementedError()
 
     def _callcompressable(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a stream.
 
         The stream may have been compressed in some implementations. This
         function takes care of the decompression. This is the only difference
         with _callstream.
 
         returns the server reply as a file like object.
         """
         raise NotImplementedError()
 
     def _callpush(self, cmd, fp, **args):
         """execute a <cmd> on server
 
         The command is expected to be related to a push. Push has a special
         return method.
 
         returns the server reply as a (ret, output) tuple. ret is either
         empty (error) or a stringified int.
         """
         raise NotImplementedError()
 
     def _calltwowaystream(self, cmd, fp, **args):
         """execute <cmd> on server
 
         The command will send a stream to the server and get a stream in reply.
         """
         raise NotImplementedError()
 
     def _abort(self, exception):
         """clearly abort the wire protocol connection and raise the exception
         """
         raise NotImplementedError()
 
 # server side
 
 # wire protocol command can either return a string or one of these classes.
 
 def getdispatchrepo(repo, proto, command):
     """Obtain the repo used for processing wire protocol commands.
 
     The intent of this function is to serve as a monkeypatch point for
     extensions that need commands to operate on different repo views under
     specialized circumstances.
     """
     return repo.filtered('served')
 
 def dispatch(repo, proto, command):
     repo = getdispatchrepo(repo, proto, command)
 
     transportversion = wireprototypes.TRANSPORTS[proto.name]['version']
     commandtable = commandsv2 if transportversion == 2 else commands
     func, spec = commandtable[command]
 
     args = proto.getargs(spec)
 
     # Version 1 protocols define arguments as a list. Version 2 uses a dict.
     if isinstance(args, list):
         return func(repo, proto, *args)
     elif isinstance(args, dict):
         return func(repo, proto, **args)
     else:
         raise error.ProgrammingError('unexpected type returned from '
                                      'proto.getargs(): %s' % type(args))
 
 def options(cmd, keys, others):
     opts = {}
     for k in keys:
         if k in others:
             opts[k] = others[k]
             del others[k]
     if others:
         procutil.stderr.write("warning: %s ignored unexpected arguments %s\n"
                               % (cmd, ",".join(others)))
     return opts
 
 def bundle1allowed(repo, action):
     """Whether a bundle1 operation is allowed from the server.
 
     Priority is:
 
     1. server.bundle1gd.<action> (if generaldelta active)
     2. server.bundle1.<action>
     3. server.bundle1gd (if generaldelta active)
     4. server.bundle1
     """
     ui = repo.ui
     gd = 'generaldelta' in repo.requirements
 
     if gd:
         v = ui.configbool('server', 'bundle1gd.%s' % action)
         if v is not None:
             return v
 
     v = ui.configbool('server', 'bundle1.%s' % action)
     if v is not None:
         return v
 
     if gd:
         v = ui.configbool('server', 'bundle1gd')
         if v is not None:
             return v
 
     return ui.configbool('server', 'bundle1')
 
 def supportedcompengines(ui, role):
     """Obtain the list of supported compression engines for a request."""
     assert role in (util.CLIENTROLE, util.SERVERROLE)
 
     compengines = util.compengines.supportedwireengines(role)
 
     # Allow config to override default list and ordering.
     if role == util.SERVERROLE:
         configengines = ui.configlist('server', 'compressionengines')
         config = 'server.compressionengines'
     else:
         # This is currently implemented mainly to facilitate testing. In most
         # cases, the server should be in charge of choosing a compression engine
         # because a server has the most to lose from a sub-optimal choice. (e.g.
         # CPU DoS due to an expensive engine or a network DoS due to poor
         # compression ratio).
         configengines = ui.configlist('experimental',
                                       'clientcompressionengines')
         config = 'experimental.clientcompressionengines'
 
     # No explicit config. Filter out the ones that aren't supposed to be
     # advertised and return default ordering.
     if not configengines:
         attr = 'serverpriority' if role == util.SERVERROLE else 'clientpriority'
         return [e for e in compengines
                 if getattr(e.wireprotosupport(), attr) > 0]
 
     # If compression engines are listed in the config, assume there is a good
     # reason for it (like server operators wanting to achieve specific
     # performance characteristics). So fail fast if the config references
     # unusable compression engines.
     validnames = set(e.name() for e in compengines)
     invalidnames = set(e for e in configengines if e not in validnames)
     if invalidnames:
         raise error.Abort(_('invalid compression engine defined in %s: %s') %
                           (config, ', '.join(sorted(invalidnames))))
 
     compengines = [e for e in compengines if e.name() in configengines]
     compengines = sorted(compengines,
                          key=lambda e: configengines.index(e.name()))
 
     if not compengines:
         raise error.Abort(_('%s config option does not specify any known '
                             'compression engines') % config,
                           hint=_('usable compression engines: %s') %
                           ', '.sorted(validnames))
 
     return compengines
 
 class commandentry(object):
     """Represents a declared wire protocol command."""
     def __init__(self, func, args='', transports=None,
                  permission='push'):
         self.func = func
         self.args = args
         self.transports = transports or set()
         self.permission = permission
 
     def _merge(self, func, args):
         """Merge this instance with an incoming 2-tuple.
 
         This is called when a caller using the old 2-tuple API attempts
         to replace an instance. The incoming values are merged with
         data not captured by the 2-tuple and a new instance containing
         the union of the two objects is returned.
         """
         return commandentry(func, args=args, transports=set(self.transports),
                             permission=self.permission)
 
     # Old code treats instances as 2-tuples. So expose that interface.
     def __iter__(self):
         yield self.func
         yield self.args
 
     def __getitem__(self, i):
         if i == 0:
             return self.func
         elif i == 1:
             return self.args
         else:
             raise IndexError('can only access elements 0 and 1')
 
 class commanddict(dict):
     """Container for registered wire protocol commands.
 
     It behaves like a dict. But __setitem__ is overwritten to allow silent
     coercion of values from 2-tuples for API compatibility.
     """
     def __setitem__(self, k, v):
         if isinstance(v, commandentry):
             pass
         # Cast 2-tuples to commandentry instances.
         elif isinstance(v, tuple):
             if len(v) != 2:
                 raise ValueError('command tuples must have exactly 2 elements')
 
             # It is common for extensions to wrap wire protocol commands via
             # e.g. ``wireproto.commands[x] = (newfn, args)``. Because callers
             # doing this aren't aware of the new API that uses objects to store
             # command entries, we automatically merge old state with new.
             if k in self:
                 v = self[k]._merge(v[0], v[1])
             else:
                 # Use default values from @wireprotocommand.
                 v = commandentry(v[0], args=v[1],
                                  transports=set(wireprototypes.TRANSPORTS),
                                  permission='push')
         else:
             raise ValueError('command entries must be commandentry instances '
                              'or 2-tuples')
 
         return super(commanddict, self).__setitem__(k, v)
 
     def commandavailable(self, command, proto):
         """Determine if a command is available for the requested protocol."""
         assert proto.name in wireprototypes.TRANSPORTS
 
         entry = self.get(command)
 
         if not entry:
             return False
 
         if proto.name not in entry.transports:
             return False
 
         return True
 
 # Constants specifying which transports a wire protocol command should be
 # available on. For use with @wireprotocommand.
 POLICY_ALL = 'all'
 POLICY_V1_ONLY = 'v1-only'
 POLICY_V2_ONLY = 'v2-only'
 
 # For version 1 transports.
 commands = commanddict()
 
 # For version 2 transports.
 commandsv2 = commanddict()
 
 def wireprotocommand(name, args='', transportpolicy=POLICY_ALL,
                      permission='push'):
     """Decorator to declare a wire protocol command.
 
     ``name`` is the name of the wire protocol command being provided.
 
-    ``args`` is a space-delimited list of named arguments that the command
-    accepts. ``*`` is a special value that says to accept all arguments.
+    ``args`` defines the named arguments accepted by the command. It is
+    ideally a dict mapping argument names to their types. For backwards
+    compatibility, it can be a space-delimited list of argument names. For
+    version 1 transports, ``*`` denotes a special value that says to accept
+    all named arguments.
 
     ``transportpolicy`` is a POLICY_* constant denoting which transports
     this wire protocol command should be exposed to. By default, commands
     are exposed to all wire protocol transports.
 
     ``permission`` defines the permission type needed to run this command.
     Can be ``push`` or ``pull``. These roughly map to read-write and read-only,
     respectively. Default is to assume command requires ``push`` permissions
     because otherwise commands not declaring their permissions could modify
     a repository that is supposed to be read-only.
     """
     if transportpolicy == POLICY_ALL:
         transports = set(wireprototypes.TRANSPORTS)
         transportversions = {1, 2}
     elif transportpolicy == POLICY_V1_ONLY:
         transports = {k for k, v in wireprototypes.TRANSPORTS.items()
                       if v['version'] == 1}
         transportversions = {1}
     elif transportpolicy == POLICY_V2_ONLY:
         transports = {k for k, v in wireprototypes.TRANSPORTS.items()
                       if v['version'] == 2}
         transportversions = {2}
     else:
         raise error.ProgrammingError('invalid transport policy value: %s' %
                                      transportpolicy)
 
     # Because SSHv2 is a mirror of SSHv1, we allow "batch" commands through to
     # SSHv2.
     # TODO undo this hack when SSH is using the unified frame protocol.
     if name == b'batch':
         transports.add(wireprototypes.SSHV2)
 
     if permission not in ('push', 'pull'):
         raise error.ProgrammingError('invalid wire protocol permission; '
                                      'got %s; expected "push" or "pull"' %
                                      permission)
 
+    if 1 in transportversions and not isinstance(args, bytes):
+        raise error.ProgrammingError('arguments for version 1 commands must '
+                                     'be declared as bytes')
+
+    if isinstance(args, bytes):
+        dictargs = {arg: b'legacy' for arg in args.split()}
+    elif isinstance(args, dict):
+        dictargs = args
+    else:
+        raise ValueError('args must be bytes or a dict')
+
     def register(func):
         if 1 in transportversions:
             if name in commands:
                 raise error.ProgrammingError('%s command already registered '
                                              'for version 1' % name)
             commands[name] = commandentry(func, args=args,
                                           transports=transports,
                                           permission=permission)
         if 2 in transportversions:
             if name in commandsv2:
                 raise error.ProgrammingError('%s command already registered '
                                              'for version 2' % name)
-            commandsv2[name] = commandentry(func, args=args,
+
+            commandsv2[name] = commandentry(func, args=dictargs,
                                             transports=transports,
                                             permission=permission)
 
         return func
     return register
 
 # TODO define a more appropriate permissions type to use for this.
 @wireprotocommand('batch', 'cmds *', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def batch(repo, proto, cmds, others):
     repo = repo.filtered("served")
     res = []
     for pair in cmds.split(';'):
         op, args = pair.split(' ', 1)
         vals = {}
         for a in args.split(','):
             if a:
                 n, v = a.split('=')
                 vals[unescapearg(n)] = unescapearg(v)
         func, spec = commands[op]
 
         # Validate that client has permissions to perform this command.
         perm = commands[op].permission
         assert perm in ('push', 'pull')
         proto.checkperm(perm)
 
         if spec:
             keys = spec.split()
             data = {}
             for k in keys:
                 if k == '*':
                     star = {}
                     for key in vals.keys():
                         if key not in keys:
                             star[key] = vals[key]
                     data['*'] = star
                 else:
                     data[k] = vals[k]
             result = func(repo, proto, *[data[k] for k in keys])
         else:
             result = func(repo, proto)
         if isinstance(result, wireprototypes.ooberror):
             return result
 
         # For now, all batchable commands must return bytesresponse or
         # raw bytes (for backwards compatibility).
         assert isinstance(result, (wireprototypes.bytesresponse, bytes))
         if isinstance(result, wireprototypes.bytesresponse):
             result = result.data
         res.append(escapearg(result))
 
     return wireprototypes.bytesresponse(';'.join(res))
 
 @wireprotocommand('between', 'pairs', transportpolicy=POLICY_V1_ONLY,
                   permission='pull')
 def between(repo, proto, pairs):
     pairs = [decodelist(p, '-') for p in pairs.split(" ")]
     r = []
     for b in repo.between(pairs):
         r.append(encodelist(b) + "\n")
 
     return wireprototypes.bytesresponse(''.join(r))
 
 @wireprotocommand('branchmap', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def branchmap(repo, proto):
     branchmap = repo.branchmap()
     heads = []
     for branch, nodes in branchmap.iteritems():
         branchname = urlreq.quote(encoding.fromlocal(branch))
         branchnodes = encodelist(nodes)
         heads.append('%s %s' % (branchname, branchnodes))
 
     return wireprototypes.bytesresponse('\n'.join(heads))
 
 @wireprotocommand('branches', 'nodes', transportpolicy=POLICY_V1_ONLY,
                   permission='pull')
 def branches(repo, proto, nodes):
     nodes = decodelist(nodes)
     r = []
     for b in repo.branches(nodes):
         r.append(encodelist(b) + "\n")
 
     return wireprototypes.bytesresponse(''.join(r))
 
 @wireprotocommand('clonebundles', '', permission='pull')
 def clonebundles(repo, proto):
     """Server command for returning info for available bundles to seed clones.
 
     Clients will parse this response and determine what bundle to fetch.
 
     Extensions may wrap this command to filter or dynamically emit data
     depending on the request. e.g. you could advertise URLs for the closest
     data center given the client's IP address.
     """
     return wireprototypes.bytesresponse(
         repo.vfs.tryread('clonebundles.manifest'))
 
 wireprotocaps = ['lookup', 'branchmap', 'pushkey',
                  'known', 'getbundle', 'unbundlehash']
 
 def _capabilities(repo, proto):
     """return a list of capabilities for a repo
 
     This function exists to allow extensions to easily wrap capabilities
     computation
 
     - returns a lists: easy to alter
     - change done here will be propagated to both `capabilities` and `hello`
       command without any other action needed.
     """
     # copy to prevent modification of the global list
     caps = list(wireprotocaps)
 
     # Command of same name as capability isn't exposed to version 1 of
     # transports. So conditionally add it.
     if commands.commandavailable('changegroupsubset', proto):
         caps.append('changegroupsubset')
 
     if streamclone.allowservergeneration(repo):
         if repo.ui.configbool('server', 'preferuncompressed'):
             caps.append('stream-preferred')
         requiredformats = repo.requirements & repo.supportedformats
         # if our local revlogs are just revlogv1, add 'stream' cap
         if not requiredformats - {'revlogv1'}:
             caps.append('stream')
         # otherwise, add 'streamreqs' detailing our local revlog format
         else:
             caps.append('streamreqs=%s' % ','.join(sorted(requiredformats)))
     if repo.ui.configbool('experimental', 'bundle2-advertise'):
         capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role='server'))
         caps.append('bundle2=' + urlreq.quote(capsblob))
     caps.append('unbundle=%s' % ','.join(bundle2.bundlepriority))
 
     return proto.addcapabilities(repo, caps)
 
 # If you are writing an extension and consider wrapping this function. Wrap
 # `_capabilities` instead.
 @wireprotocommand('capabilities', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def capabilities(repo, proto):
     caps = _capabilities(repo, proto)
     return wireprototypes.bytesresponse(' '.join(sorted(caps)))
 
 @wireprotocommand('changegroup', 'roots', transportpolicy=POLICY_V1_ONLY,
                   permission='pull')
 def changegroup(repo, proto, roots):
     nodes = decodelist(roots)
     outgoing = discovery.outgoing(repo, missingroots=nodes,
                                   missingheads=repo.heads())
     cg = changegroupmod.makechangegroup(repo, outgoing, '01', 'serve')
     gen = iter(lambda: cg.read(32768), '')
     return wireprototypes.streamres(gen=gen)
 
 @wireprotocommand('changegroupsubset', 'bases heads',
                   transportpolicy=POLICY_V1_ONLY,
                   permission='pull')
 def changegroupsubset(repo, proto, bases, heads):
     bases = decodelist(bases)
     heads = decodelist(heads)
     outgoing = discovery.outgoing(repo, missingroots=bases,
                                   missingheads=heads)
     cg = changegroupmod.makechangegroup(repo, outgoing, '01', 'serve')
     gen = iter(lambda: cg.read(32768), '')
     return wireprototypes.streamres(gen=gen)
 
 @wireprotocommand('debugwireargs', 'one two *',
                   permission='pull', transportpolicy=POLICY_V1_ONLY)
 def debugwireargs(repo, proto, one, two, others):
     # only accept optional args from the known set
     opts = options('debugwireargs', ['three', 'four'], others)
     return wireprototypes.bytesresponse(repo.debugwireargs(
         one, two, **pycompat.strkwargs(opts)))
 
 def find_pullbundle(repo, proto, opts, clheads, heads, common):
     """Return a file object for the first matching pullbundle.
 
     Pullbundles are specified in .hg/pullbundles.manifest similar to
     clonebundles.
     For each entry, the bundle specification is checked for compatibility:
     - Client features vs the BUNDLESPEC.
     - Revisions shared with the clients vs base revisions of the bundle.
       A bundle can be applied only if all its base revisions are known by
       the client.
     - At least one leaf of the bundle's DAG is missing on the client.
     - Every leaf of the bundle's DAG is part of node set the client wants.
       E.g. do not send a bundle of all changes if the client wants only
       one specific branch of many.
     """
     def decodehexstring(s):
         return set([h.decode('hex') for h in s.split(';')])
 
     manifest = repo.vfs.tryread('pullbundles.manifest')
     if not manifest:
         return None
     res = exchange.parseclonebundlesmanifest(repo, manifest)
     res = exchange.filterclonebundleentries(repo, res)
     if not res:
         return None
     cl = repo.changelog
     heads_anc = cl.ancestors([cl.rev(rev) for rev in heads], inclusive=True)
     common_anc = cl.ancestors([cl.rev(rev) for rev in common], inclusive=True)
     compformats = clientcompressionsupport(proto)
     for entry in res:
         if 'COMPRESSION' in entry and entry['COMPRESSION'] not in compformats:
             continue
         # No test yet for VERSION, since V2 is supported by any client
         # that advertises partial pulls
         if 'heads' in entry:
             try:
                 bundle_heads = decodehexstring(entry['heads'])
             except TypeError:
                 # Bad heads entry
                 continue
             if bundle_heads.issubset(common):
                 continue # Nothing new
             if all(cl.rev(rev) in common_anc for rev in bundle_heads):
                 continue # Still nothing new
             if any(cl.rev(rev) not in heads_anc and
                    cl.rev(rev) not in common_anc for rev in bundle_heads):
                 continue
         if 'bases' in entry:
             try:
                 bundle_bases = decodehexstring(entry['bases'])
             except TypeError:
                 # Bad bases entry
                 continue
             if not all(cl.rev(rev) in common_anc for rev in bundle_bases):
                 continue
         path = entry['URL']
         repo.ui.debug('sending pullbundle "%s"\n' % path)
         try:
             return repo.vfs.open(path)
         except IOError:
             repo.ui.debug('pullbundle "%s" not accessible\n' % path)
             continue
     return None
 
 @wireprotocommand('getbundle', '*', permission='pull')
 def getbundle(repo, proto, others):
     opts = options('getbundle', gboptsmap.keys(), others)
     for k, v in opts.iteritems():
         keytype = gboptsmap[k]
         if keytype == 'nodes':
             opts[k] = decodelist(v)
         elif keytype == 'csv':
             opts[k] = list(v.split(','))
         elif keytype == 'scsv':
             opts[k] = set(v.split(','))
         elif keytype == 'boolean':
             # Client should serialize False as '0', which is a non-empty string
             # so it evaluates as a True bool.
             if v == '0':
                 opts[k] = False
             else:
                 opts[k] = bool(v)
         elif keytype != 'plain':
             raise KeyError('unknown getbundle option type %s'
                            % keytype)
 
     if not bundle1allowed(repo, 'pull'):
         if not exchange.bundle2requested(opts.get('bundlecaps')):
             if proto.name == 'http-v1':
                 return wireprototypes.ooberror(bundle2required)
             raise error.Abort(bundle2requiredmain,
                               hint=bundle2requiredhint)
 
     prefercompressed = True
 
     try:
         clheads = set(repo.changelog.heads())
         heads = set(opts.get('heads', set()))
         common = set(opts.get('common', set()))
         common.discard(nullid)
         if (repo.ui.configbool('server', 'pullbundle') and
             'partial-pull' in proto.getprotocaps()):
             # Check if a pre-built bundle covers this request.
             bundle = find_pullbundle(repo, proto, opts, clheads, heads, common)
             if bundle:
                 return wireprototypes.streamres(gen=util.filechunkiter(bundle),
                                                 prefer_uncompressed=True)
 
         if repo.ui.configbool('server', 'disablefullbundle'):
             # Check to see if this is a full clone.
             changegroup = opts.get('cg', True)
             if changegroup and not common and clheads == heads:
                 raise error.Abort(
                     _('server has pull-based clones disabled'),
                     hint=_('remove --pull if specified or upgrade Mercurial'))
 
         info, chunks = exchange.getbundlechunks(repo, 'serve',
                                                 **pycompat.strkwargs(opts))
         prefercompressed = info.get('prefercompressed', True)
     except error.Abort as exc:
         # cleanly forward Abort error to the client
         if not exchange.bundle2requested(opts.get('bundlecaps')):
             if proto.name == 'http-v1':
                 return wireprototypes.ooberror(pycompat.bytestr(exc) + '\n')
             raise # cannot do better for bundle1 + ssh
         # bundle2 request expect a bundle2 reply
         bundler = bundle2.bundle20(repo.ui)
         manargs = [('message', pycompat.bytestr(exc))]
         advargs = []
         if exc.hint is not None:
             advargs.append(('hint', exc.hint))
         bundler.addpart(bundle2.bundlepart('error:abort',
                                            manargs, advargs))
         chunks = bundler.getchunks()
         prefercompressed = False
 
     return wireprototypes.streamres(
         gen=chunks, prefer_uncompressed=not prefercompressed)
 
 @wireprotocommand('heads', permission='pull', transportpolicy=POLICY_V1_ONLY)
 def heads(repo, proto):
     h = repo.heads()
     return wireprototypes.bytesresponse(encodelist(h) + '\n')
 
 @wireprotocommand('hello', permission='pull', transportpolicy=POLICY_V1_ONLY)
 def hello(repo, proto):
     """Called as part of SSH handshake to obtain server info.
 
     Returns a list of lines describing interesting things about the
     server, in an RFC822-like format.
 
     Currently, the only one defined is ``capabilities``, which consists of a
     line of space separated tokens describing server abilities:
 
         capabilities: <token0> <token1> <token2>
     """
     caps = capabilities(repo, proto).data
     return wireprototypes.bytesresponse('capabilities: %s\n' % caps)
 
 @wireprotocommand('listkeys', 'namespace', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def listkeys(repo, proto, namespace):
     d = sorted(repo.listkeys(encoding.tolocal(namespace)).items())
     return wireprototypes.bytesresponse(pushkeymod.encodekeys(d))
 
 @wireprotocommand('lookup', 'key', permission='pull')
 def lookup(repo, proto, key):
     try:
         k = encoding.tolocal(key)
         n = repo.lookup(k)
         r = hex(n)
         success = 1
     except Exception as inst:
         r = stringutil.forcebytestr(inst)
         success = 0
     return wireprototypes.bytesresponse('%d %s\n' % (success, r))
 
 @wireprotocommand('known', 'nodes *', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def known(repo, proto, nodes, others):
     v = ''.join(b and '1' or '0' for b in repo.known(decodelist(nodes)))
     return wireprototypes.bytesresponse(v)
 
 @wireprotocommand('protocaps', 'caps', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def protocaps(repo, proto, caps):
     if proto.name == wireprototypes.SSHV1:
         proto._protocaps = set(caps.split(' '))
     return wireprototypes.bytesresponse('OK')
 
 @wireprotocommand('pushkey', 'namespace key old new', permission='push')
 def pushkey(repo, proto, namespace, key, old, new):
     # compatibility with pre-1.8 clients which were accidentally
     # sending raw binary nodes rather than utf-8-encoded hex
     if len(new) == 20 and stringutil.escapestr(new) != new:
         # looks like it could be a binary node
         try:
             new.decode('utf-8')
             new = encoding.tolocal(new) # but cleanly decodes as UTF-8
         except UnicodeDecodeError:
             pass # binary, leave unmodified
     else:
         new = encoding.tolocal(new) # normal path
 
     with proto.mayberedirectstdio() as output:
         r = repo.pushkey(encoding.tolocal(namespace), encoding.tolocal(key),
                          encoding.tolocal(old), new) or False
 
     output = output.getvalue() if output else ''
     return wireprototypes.bytesresponse('%d\n%s' % (int(r), output))
 
 @wireprotocommand('stream_out', permission='pull',
                   transportpolicy=POLICY_V1_ONLY)
 def stream(repo, proto):
     '''If the server supports streaming clone, it advertises the "stream"
     capability with a value representing the version and flags of the repo
     it is serving. Client checks to see if it understands the format.
     '''
     return wireprototypes.streamreslegacy(
         streamclone.generatev1wireproto(repo))
 
 @wireprotocommand('unbundle', 'heads', permission='push')
 def unbundle(repo, proto, heads):
     their_heads = decodelist(heads)
 
     with proto.mayberedirectstdio() as output:
         try:
             exchange.check_heads(repo, their_heads, 'preparing changes')
             cleanup = lambda: None
             try:
                 payload = proto.getpayload()
                 if repo.ui.configbool('server', 'streamunbundle'):
                     def cleanup():
                         # Ensure that the full payload is consumed, so
                         # that the connection doesn't contain trailing garbage.
                         for p in payload:
                             pass
                     fp = util.chunkbuffer(payload)
                 else:
                     # write bundle data to temporary file as it can be big
                     fp, tempname = None, None
                     def cleanup():
                         if fp:
                             fp.close()
                         if tempname:
                             os.unlink(tempname)
                     fd, tempname = tempfile.mkstemp(prefix='hg-unbundle-')
                     repo.ui.debug('redirecting incoming bundle to %s\n' %
                         tempname)
                     fp = os.fdopen(fd, pycompat.sysstr('wb+'))
                     r = 0
                     for p in payload:
                         fp.write(p)
                     fp.seek(0)
 
                 gen = exchange.readbundle(repo.ui, fp, None)
                 if (isinstance(gen, changegroupmod.cg1unpacker)
                     and not bundle1allowed(repo, 'push')):
                     if proto.name == 'http-v1':
                         # need to special case http because stderr do not get to
                         # the http client on failed push so we need to abuse
                         # some other error type to make sure the message get to
                         # the user.
                         return wireprototypes.ooberror(bundle2required)
                     raise error.Abort(bundle2requiredmain,
                                       hint=bundle2requiredhint)
 
                 r = exchange.unbundle(repo, gen, their_heads, 'serve',
                                       proto.client())
                 if util.safehasattr(r, 'addpart'):
                     # The return looks streamable, we are in the bundle2 case
                     # and should return a stream.
                     return wireprototypes.streamreslegacy(gen=r.getchunks())
                 return wireprototypes.pushres(
                     r, output.getvalue() if output else '')
 
             finally:
                 cleanup()
 
         except (error.BundleValueError, error.Abort, error.PushRaced) as exc:
             # handle non-bundle2 case first
             if not getattr(exc, 'duringunbundle2', False):
                 try:
                     raise
                 except error.Abort:
                     # The old code we moved used procutil.stderr directly.
                     # We did not change it to minimise code change.
                     # This need to be moved to something proper.
                     # Feel free to do it.
                     procutil.stderr.write("abort: %s\n" % exc)
                     if exc.hint is not None:
                         procutil.stderr.write("(%s)\n" % exc.hint)
                     procutil.stderr.flush()
                     return wireprototypes.pushres(
                         0, output.getvalue() if output else '')
                 except error.PushRaced:
                     return wireprototypes.pusherr(
                         pycompat.bytestr(exc),
                         output.getvalue() if output else '')
 
             bundler = bundle2.bundle20(repo.ui)
             for out in getattr(exc, '_bundle2salvagedoutput', ()):
                 bundler.addpart(out)
             try:
                 try:
                     raise
                 except error.PushkeyFailed as exc:
                     # check client caps
                     remotecaps = getattr(exc, '_replycaps', None)
                     if (remotecaps is not None
                             and 'pushkey' not in remotecaps.get('error', ())):
                         # no support remote side, fallback to Abort handler.
                         raise
                     part = bundler.newpart('error:pushkey')
                     part.addparam('in-reply-to', exc.partid)
                     if exc.namespace is not None:
                         part.addparam('namespace', exc.namespace,
                                       mandatory=False)
                     if exc.key is not None:
                         part.addparam('key', exc.key, mandatory=False)
                     if exc.new is not None:
                         part.addparam('new', exc.new, mandatory=False)
                     if exc.old is not None:
                         part.addparam('old', exc.old, mandatory=False)
                     if exc.ret is not None:
                         part.addparam('ret', exc.ret, mandatory=False)
             except error.BundleValueError as exc:
                 errpart = bundler.newpart('error:unsupportedcontent')
                 if exc.parttype is not None:
                     errpart.addparam('parttype', exc.parttype)
                 if exc.params:
                     errpart.addparam('params', '\0'.join(exc.params))
             except error.Abort as exc:
                 manargs = [('message', stringutil.forcebytestr(exc))]
                 advargs = []
                 if exc.hint is not None:
                     advargs.append(('hint', exc.hint))
                 bundler.addpart(bundle2.bundlepart('error:abort',
                                                    manargs, advargs))
             except error.PushRaced as exc:
                 bundler.newpart('error:pushraced',
                                 [('message', stringutil.forcebytestr(exc))])
             return wireprototypes.streamreslegacy(gen=bundler.getchunks())
 
 # Wire protocol version 2 commands only past this point.
 
 def _capabilitiesv2(repo, proto):
     """Obtain the set of capabilities for version 2 transports.
 
     These capabilities are distinct from the capabilities for version 1
     transports.
     """
     compression = []
     for engine in supportedcompengines(repo.ui, util.SERVERROLE):
         compression.append({
             b'name': engine.wireprotosupport().name,
         })
 
     caps = {
         'commands': {},
         'compression': compression,
     }
 
     for command, entry in commandsv2.items():
         caps['commands'][command] = {
-            'args': sorted(entry.args.split()) if entry.args else [],
+            'args': entry.args,
             'permissions': [entry.permission],
         }
 
     return proto.addcapabilities(repo, caps)
 
 @wireprotocommand('branchmap', permission='pull',
                   transportpolicy=POLICY_V2_ONLY)
 def branchmapv2(repo, proto):
     branchmap = {encoding.fromlocal(k): v
                  for k, v in repo.branchmap().iteritems()}
 
     return wireprototypes.cborresponse(branchmap)
 
 @wireprotocommand('capabilities', permission='pull',
                   transportpolicy=POLICY_V2_ONLY)
 def capabilitiesv2(repo, proto):
     caps = _capabilitiesv2(repo, proto)
 
     return wireprototypes.cborresponse(caps)
 
-@wireprotocommand('heads', args='publiconly', permission='pull',
+@wireprotocommand('heads',
+                  args={
+                      'publiconly': False,
+                  },
+                  permission='pull',
                   transportpolicy=POLICY_V2_ONLY)
 def headsv2(repo, proto, publiconly=False):
     if publiconly:
         repo = repo.filtered('immutable')
 
     return wireprototypes.cborresponse(repo.heads())
 
-@wireprotocommand('known', 'nodes', permission='pull',
+@wireprotocommand('known',
+                  args={
+                      'nodes': [b'deadbeef'],
+                  },
+                  permission='pull',
                   transportpolicy=POLICY_V2_ONLY)
 def knownv2(repo, proto, nodes=None):
     nodes = nodes or []
     result = b''.join(b'1' if n else b'0' for n in repo.known(nodes))
     return wireprototypes.cborresponse(result)
 
-@wireprotocommand('listkeys', 'namespace', permission='pull',
+@wireprotocommand('listkeys',
+                  args={
+                      'namespace': b'ns',
+                  },
+                  permission='pull',
                   transportpolicy=POLICY_V2_ONLY)
 def listkeysv2(repo, proto, namespace=None):
     keys = repo.listkeys(encoding.tolocal(namespace))
     keys = {encoding.fromlocal(k): encoding.fromlocal(v)
             for k, v in keys.iteritems()}
 
     return wireprototypes.cborresponse(keys)

mercurial/wireprotoserver.py

@@ -1,1077 +1,1078 b''
 # Copyright 21 May 2005 - (c) 2005 Jake Edge <[e-mail]>
 # Copyright 2005-2007 Matt Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 import contextlib
 import struct
 import sys
 import threading
 
 from .i18n import _
 from .thirdparty import (
     cbor,
 )
 from .thirdparty.zope import (
     interface as zi,
 )
 from . import (
     encoding,
     error,
     hook,
     pycompat,
     util,
     wireproto,
     wireprotoframing,
     wireprototypes,
 )
 from .utils import (
     procutil,
 )
 
 stringio = util.stringio
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 HTTP_OK = 200
 
 HGTYPE = 'application/mercurial-0.1'
 HGTYPE2 = 'application/mercurial-0.2'
 HGERRTYPE = 'application/hg-error'
 FRAMINGTYPE = b'application/mercurial-exp-framing-0003'
 
 HTTPV2 = wireprototypes.HTTPV2
 SSHV1 = wireprototypes.SSHV1
 SSHV2 = wireprototypes.SSHV2
 
 def decodevaluefromheaders(req, headerprefix):
     """Decode a long value from multiple HTTP request headers.
 
     Returns the value as a bytes, not a str.
     """
     chunks = []
     i = 1
     while True:
         v = req.headers.get(b'%s-%d' % (headerprefix, i))
         if v is None:
             break
         chunks.append(pycompat.bytesurl(v))
         i += 1
 
     return ''.join(chunks)
 
 @zi.implementer(wireprototypes.baseprotocolhandler)
 class httpv1protocolhandler(object):
     def __init__(self, req, ui, checkperm):
         self._req = req
         self._ui = ui
         self._checkperm = checkperm
         self._protocaps = None
 
     @property
     def name(self):
         return 'http-v1'
 
     def getargs(self, args):
         knownargs = self._args()
         data = {}
         keys = args.split()
         for k in keys:
             if k == '*':
                 star = {}
                 for key in knownargs.keys():
                     if key != 'cmd' and key not in keys:
                         star[key] = knownargs[key][0]
                 data['*'] = star
             else:
                 data[k] = knownargs[k][0]
         return [data[k] for k in keys]
 
     def _args(self):
         args = self._req.qsparams.asdictoflists()
         postlen = int(self._req.headers.get(b'X-HgArgs-Post', 0))
         if postlen:
             args.update(urlreq.parseqs(
                 self._req.bodyfh.read(postlen), keep_blank_values=True))
             return args
 
         argvalue = decodevaluefromheaders(self._req, b'X-HgArg')
         args.update(urlreq.parseqs(argvalue, keep_blank_values=True))
         return args
 
     def getprotocaps(self):
         if self._protocaps is None:
             value = decodevaluefromheaders(self._req, r'X-HgProto')
             self._protocaps = set(value.split(' '))
         return self._protocaps
 
     def getpayload(self):
         # Existing clients *always* send Content-Length.
         length = int(self._req.headers[b'Content-Length'])
 
         # If httppostargs is used, we need to read Content-Length
         # minus the amount that was consumed by args.
         length -= int(self._req.headers.get(b'X-HgArgs-Post', 0))
         return util.filechunkiter(self._req.bodyfh, limit=length)
 
     @contextlib.contextmanager
     def mayberedirectstdio(self):
         oldout = self._ui.fout
         olderr = self._ui.ferr
 
         out = util.stringio()
 
         try:
             self._ui.fout = out
             self._ui.ferr = out
             yield out
         finally:
             self._ui.fout = oldout
             self._ui.ferr = olderr
 
     def client(self):
         return 'remote:%s:%s:%s' % (
             self._req.urlscheme,
             urlreq.quote(self._req.remotehost or ''),
             urlreq.quote(self._req.remoteuser or ''))
 
     def addcapabilities(self, repo, caps):
         caps.append(b'batch')
 
         caps.append('httpheader=%d' %
                     repo.ui.configint('server', 'maxhttpheaderlen'))
         if repo.ui.configbool('experimental', 'httppostargs'):
             caps.append('httppostargs')
 
         # FUTURE advertise 0.2rx once support is implemented
         # FUTURE advertise minrx and mintx after consulting config option
         caps.append('httpmediatype=0.1rx,0.1tx,0.2tx')
 
         compengines = wireproto.supportedcompengines(repo.ui, util.SERVERROLE)
         if compengines:
             comptypes = ','.join(urlreq.quote(e.wireprotosupport().name)
                                  for e in compengines)
             caps.append('compression=%s' % comptypes)
 
         return caps
 
     def checkperm(self, perm):
         return self._checkperm(perm)
 
 # This method exists mostly so that extensions like remotefilelog can
 # disable a kludgey legacy method only over http. As of early 2018,
 # there are no other known users, so with any luck we can discard this
 # hook if remotefilelog becomes a first-party extension.
 def iscmd(cmd):
     return cmd in wireproto.commands
 
 def handlewsgirequest(rctx, req, res, checkperm):
     """Possibly process a wire protocol request.
 
     If the current request is a wire protocol request, the request is
     processed by this function.
 
     ``req`` is a ``parsedrequest`` instance.
     ``res`` is a ``wsgiresponse`` instance.
 
     Returns a bool indicating if the request was serviced. If set, the caller
     should stop processing the request, as a response has already been issued.
     """
     # Avoid cycle involving hg module.
     from .hgweb import common as hgwebcommon
 
     repo = rctx.repo
 
     # HTTP version 1 wire protocol requests are denoted by a "cmd" query
     # string parameter. If it isn't present, this isn't a wire protocol
     # request.
     if 'cmd' not in req.qsparams:
         return False
 
     cmd = req.qsparams['cmd']
 
     # The "cmd" request parameter is used by both the wire protocol and hgweb.
     # While not all wire protocol commands are available for all transports,
     # if we see a "cmd" value that resembles a known wire protocol command, we
     # route it to a protocol handler. This is better than routing possible
     # wire protocol requests to hgweb because it prevents hgweb from using
     # known wire protocol commands and it is less confusing for machine
     # clients.
     if not iscmd(cmd):
         return False
 
     # The "cmd" query string argument is only valid on the root path of the
     # repo. e.g. ``/?cmd=foo``, ``/repo?cmd=foo``. URL paths within the repo
     # like ``/blah?cmd=foo`` are not allowed. So don't recognize the request
     # in this case. We send an HTTP 404 for backwards compatibility reasons.
     if req.dispatchpath:
         res.status = hgwebcommon.statusmessage(404)
         res.headers['Content-Type'] = HGTYPE
         # TODO This is not a good response to issue for this request. This
         # is mostly for BC for now.
         res.setbodybytes('0\n%s\n' % b'Not Found')
         return True
 
     proto = httpv1protocolhandler(req, repo.ui,
                                   lambda perm: checkperm(rctx, req, perm))
 
     # The permissions checker should be the only thing that can raise an
     # ErrorResponse. It is kind of a layer violation to catch an hgweb
     # exception here. So consider refactoring into a exception type that
     # is associated with the wire protocol.
     try:
         _callhttp(repo, req, res, proto, cmd)
     except hgwebcommon.ErrorResponse as e:
         for k, v in e.headers:
             res.headers[k] = v
         res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
         # TODO This response body assumes the failed command was
         # "unbundle." That assumption is not always valid.
         res.setbodybytes('0\n%s\n' % pycompat.bytestr(e))
 
     return True
 
 def handlewsgiapirequest(rctx, req, res, checkperm):
     """Handle requests to /api/*."""
     assert req.dispatchparts[0] == b'api'
 
     repo = rctx.repo
 
     # This whole URL space is experimental for now. But we want to
     # reserve the URL space. So, 404 all URLs if the feature isn't enabled.
     if not repo.ui.configbool('experimental', 'web.apiserver'):
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('Experimental API server endpoint not enabled'))
         return
 
     # The URL space is /api/<protocol>/*. The structure of URLs under varies
     # by <protocol>.
 
     # Registered APIs are made available via config options of the name of
     # the protocol.
     availableapis = set()
     for k, v in API_HANDLERS.items():
         section, option = v['config']
         if repo.ui.configbool(section, option):
             availableapis.add(k)
 
     # Requests to /api/ list available APIs.
     if req.dispatchparts == [b'api']:
         res.status = b'200 OK'
         res.headers[b'Content-Type'] = b'text/plain'
         lines = [_('APIs can be accessed at /api/<name>, where <name> can be '
                    'one of the following:\n')]
         if availableapis:
             lines.extend(sorted(availableapis))
         else:
             lines.append(_('(no available APIs)\n'))
         res.setbodybytes(b'\n'.join(lines))
         return
 
     proto = req.dispatchparts[1]
 
     if proto not in API_HANDLERS:
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('Unknown API: %s\nKnown APIs: %s') % (
             proto, b', '.join(sorted(availableapis))))
         return
 
     if proto not in availableapis:
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('API %s not enabled\n') % proto)
         return
 
     API_HANDLERS[proto]['handler'](rctx, req, res, checkperm,
                                    req.dispatchparts[2:])
 
 def _handlehttpv2request(rctx, req, res, checkperm, urlparts):
     from .hgweb import common as hgwebcommon
 
     # URL space looks like: <permissions>/<command>, where <permission> can
     # be ``ro`` or ``rw`` to signal read-only or read-write, respectively.
 
     # Root URL does nothing meaningful... yet.
     if not urlparts:
         res.status = b'200 OK'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('HTTP version 2 API handler'))
         return
 
     if len(urlparts) == 1:
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('do not know how to process %s\n') %
                          req.dispatchpath)
         return
 
     permission, command = urlparts[0:2]
 
     if permission not in (b'ro', b'rw'):
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('unknown permission: %s') % permission)
         return
 
     if req.method != 'POST':
         res.status = b'405 Method Not Allowed'
         res.headers[b'Allow'] = b'POST'
         res.setbodybytes(_('commands require POST requests'))
         return
 
     # At some point we'll want to use our own API instead of recycling the
     # behavior of version 1 of the wire protocol...
     # TODO return reasonable responses - not responses that overload the
     # HTTP status line message for error reporting.
     try:
         checkperm(rctx, req, 'pull' if permission == b'ro' else 'push')
     except hgwebcommon.ErrorResponse as e:
         res.status = hgwebcommon.statusmessage(e.code, pycompat.bytestr(e))
         for k, v in e.headers:
             res.headers[k] = v
         res.setbodybytes('permission denied')
         return
 
     # We have a special endpoint to reflect the request back at the client.
     if command == b'debugreflect':
         _processhttpv2reflectrequest(rctx.repo.ui, rctx.repo, req, res)
         return
 
     # Extra commands that we handle that aren't really wire protocol
     # commands. Think extra hard before making this hackery available to
     # extension.
     extracommands = {'multirequest'}
 
     if command not in wireproto.commandsv2 and command not in extracommands:
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('unknown wire protocol command: %s\n') % command)
         return
 
     repo = rctx.repo
     ui = repo.ui
 
     proto = httpv2protocolhandler(req, ui)
 
     if (not wireproto.commandsv2.commandavailable(command, proto)
         and command not in extracommands):
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('invalid wire protocol command: %s') % command)
         return
 
     # TODO consider cases where proxies may add additional Accept headers.
     if req.headers.get(b'Accept') != FRAMINGTYPE:
         res.status = b'406 Not Acceptable'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('client MUST specify Accept header with value: %s\n')
                            % FRAMINGTYPE)
         return
 
     if req.headers.get(b'Content-Type') != FRAMINGTYPE:
         res.status = b'415 Unsupported Media Type'
         # TODO we should send a response with appropriate media type,
         # since client does Accept it.
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('client MUST send Content-Type header with '
                            'value: %s\n') % FRAMINGTYPE)
         return
 
     _processhttpv2request(ui, repo, req, res, permission, command, proto)
 
 def _processhttpv2reflectrequest(ui, repo, req, res):
     """Reads unified frame protocol request and dumps out state to client.
 
     This special endpoint can be used to help debug the wire protocol.
 
     Instead of routing the request through the normal dispatch mechanism,
     we instead read all frames, decode them, and feed them into our state
     tracker. We then dump the log of all that activity back out to the
     client.
     """
     import json
 
     # Reflection APIs have a history of being abused, accidentally disclosing
     # sensitive data, etc. So we have a config knob.
     if not ui.configbool('experimental', 'web.api.debugreflect'):
         res.status = b'404 Not Found'
         res.headers[b'Content-Type'] = b'text/plain'
         res.setbodybytes(_('debugreflect service not available'))
         return
 
     # We assume we have a unified framing protocol request body.
 
     reactor = wireprotoframing.serverreactor()
     states = []
 
     while True:
         frame = wireprotoframing.readframe(req.bodyfh)
 
         if not frame:
             states.append(b'received: <no frame>')
             break
 
         states.append(b'received: %d %d %d %s' % (frame.typeid, frame.flags,
                                                   frame.requestid,
                                                   frame.payload))
 
         action, meta = reactor.onframerecv(frame)
         states.append(json.dumps((action, meta), sort_keys=True,
                                  separators=(', ', ': ')))
 
     action, meta = reactor.oninputeof()
     meta['action'] = action
     states.append(json.dumps(meta, sort_keys=True, separators=(', ',': ')))
 
     res.status = b'200 OK'
     res.headers[b'Content-Type'] = b'text/plain'
     res.setbodybytes(b'\n'.join(states))
 
 def _processhttpv2request(ui, repo, req, res, authedperm, reqcommand, proto):
     """Post-validation handler for HTTPv2 requests.
 
     Called when the HTTP request contains unified frame-based protocol
     frames for evaluation.
     """
     # TODO Some HTTP clients are full duplex and can receive data before
     # the entire request is transmitted. Figure out a way to indicate support
     # for that so we can opt into full duplex mode.
     reactor = wireprotoframing.serverreactor(deferoutput=True)
     seencommand = False
 
     outstream = reactor.makeoutputstream()
 
     while True:
         frame = wireprotoframing.readframe(req.bodyfh)
         if not frame:
             break
 
         action, meta = reactor.onframerecv(frame)
 
         if action == 'wantframe':
             # Need more data before we can do anything.
             continue
         elif action == 'runcommand':
             sentoutput = _httpv2runcommand(ui, repo, req, res, authedperm,
                                            reqcommand, reactor, outstream,
                                            meta, issubsequent=seencommand)
 
             if sentoutput:
                 return
 
             seencommand = True
 
         elif action == 'error':
             # TODO define proper error mechanism.
             res.status = b'200 OK'
             res.headers[b'Content-Type'] = b'text/plain'
             res.setbodybytes(meta['message'] + b'\n')
             return
         else:
             raise error.ProgrammingError(
                 'unhandled action from frame processor: %s' % action)
 
     action, meta = reactor.oninputeof()
     if action == 'sendframes':
         # We assume we haven't started sending the response yet. If we're
         # wrong, the response type will raise an exception.
         res.status = b'200 OK'
         res.headers[b'Content-Type'] = FRAMINGTYPE
         res.setbodygen(meta['framegen'])
     elif action == 'noop':
         pass
     else:
         raise error.ProgrammingError('unhandled action from frame processor: %s'
                                      % action)
 
 def _httpv2runcommand(ui, repo, req, res, authedperm, reqcommand, reactor,
                       outstream, command, issubsequent):
     """Dispatch a wire protocol command made from HTTPv2 requests.
 
     The authenticated permission (``authedperm``) along with the original
     command from the URL (``reqcommand``) are passed in.
     """
     # We already validated that the session has permissions to perform the
     # actions in ``authedperm``. In the unified frame protocol, the canonical
     # command to run is expressed in a frame. However, the URL also requested
     # to run a specific command. We need to be careful that the command we
     # run doesn't have permissions requirements greater than what was granted
     # by ``authedperm``.
     #
     # Our rule for this is we only allow one command per HTTP request and
     # that command must match the command in the URL. However, we make
     # an exception for the ``multirequest`` URL. This URL is allowed to
     # execute multiple commands. We double check permissions of each command
     # as it is invoked to ensure there is no privilege escalation.
     # TODO consider allowing multiple commands to regular command URLs
     # iff each command is the same.
 
     proto = httpv2protocolhandler(req, ui, args=command['args'])
 
     if reqcommand == b'multirequest':
         if not wireproto.commandsv2.commandavailable(command['command'], proto):
             # TODO proper error mechanism
             res.status = b'200 OK'
             res.headers[b'Content-Type'] = b'text/plain'
             res.setbodybytes(_('wire protocol command not available: %s') %
                              command['command'])
             return True
 
         # TODO don't use assert here, since it may be elided by -O.
         assert authedperm in (b'ro', b'rw')
         wirecommand = wireproto.commandsv2[command['command']]
         assert wirecommand.permission in ('push', 'pull')
 
         if authedperm == b'ro' and wirecommand.permission != 'pull':
             # TODO proper error mechanism
             res.status = b'403 Forbidden'
             res.headers[b'Content-Type'] = b'text/plain'
             res.setbodybytes(_('insufficient permissions to execute '
                                'command: %s') % command['command'])
             return True
 
         # TODO should we also call checkperm() here? Maybe not if we're going
         # to overhaul that API. The granted scope from the URL check should
         # be good enough.
 
     else:
         # Don't allow multiple commands outside of ``multirequest`` URL.
         if issubsequent:
             # TODO proper error mechanism
             res.status = b'200 OK'
             res.headers[b'Content-Type'] = b'text/plain'
             res.setbodybytes(_('multiple commands cannot be issued to this '
                                'URL'))
             return True
 
         if reqcommand != command['command']:
             # TODO define proper error mechanism
             res.status = b'200 OK'
             res.headers[b'Content-Type'] = b'text/plain'
             res.setbodybytes(_('command in frame must match command in URL'))
             return True
 
     rsp = wireproto.dispatch(repo, proto, command['command'])
 
     res.status = b'200 OK'
     res.headers[b'Content-Type'] = FRAMINGTYPE
 
     if isinstance(rsp, wireprototypes.bytesresponse):
         action, meta = reactor.onbytesresponseready(outstream,
                                                     command['requestid'],
                                                     rsp.data)
     elif isinstance(rsp, wireprototypes.cborresponse):
         encoded = cbor.dumps(rsp.value, canonical=True)
         action, meta = reactor.onbytesresponseready(outstream,
                                                     command['requestid'],
                                                     encoded,
                                                     iscbor=True)
     else:
         action, meta = reactor.onapplicationerror(
             _('unhandled response type from wire proto command'))
 
     if action == 'sendframes':
         res.setbodygen(meta['framegen'])
         return True
     elif action == 'noop':
         return False
     else:
         raise error.ProgrammingError('unhandled event from reactor: %s' %
                                      action)
 
 # Maps API name to metadata so custom API can be registered.
 API_HANDLERS = {
     HTTPV2: {
         'config': ('experimental', 'web.api.http-v2'),
         'handler': _handlehttpv2request,
     },
 }
 
 @zi.implementer(wireprototypes.baseprotocolhandler)
 class httpv2protocolhandler(object):
     def __init__(self, req, ui, args=None):
         self._req = req
         self._ui = ui
         self._args = args
 
     @property
     def name(self):
         return HTTPV2
 
     def getargs(self, args):
         data = {}
-        for k in args.split():
+        for k, typ in args.items():
             if k == '*':
                 raise NotImplementedError('do not support * args')
             elif k in self._args:
+                # TODO consider validating value types.
                 data[k] = self._args[k]
 
         return data
 
     def getprotocaps(self):
         # Protocol capabilities are currently not implemented for HTTP V2.
         return set()
 
     def getpayload(self):
         raise NotImplementedError
 
     @contextlib.contextmanager
     def mayberedirectstdio(self):
         raise NotImplementedError
 
     def client(self):
         raise NotImplementedError
 
     def addcapabilities(self, repo, caps):
         return caps
 
     def checkperm(self, perm):
         raise NotImplementedError
 
 def _httpresponsetype(ui, proto, prefer_uncompressed):
     """Determine the appropriate response type and compression settings.
 
     Returns a tuple of (mediatype, compengine, engineopts).
     """
     # Determine the response media type and compression engine based
     # on the request parameters.
 
     if '0.2' in proto.getprotocaps():
         # All clients are expected to support uncompressed data.
         if prefer_uncompressed:
             return HGTYPE2, util._noopengine(), {}
 
         # Now find an agreed upon compression format.
         compformats = wireproto.clientcompressionsupport(proto)
         for engine in wireproto.supportedcompengines(ui, util.SERVERROLE):
             if engine.wireprotosupport().name in compformats:
                 opts = {}
                 level = ui.configint('server', '%slevel' % engine.name())
                 if level is not None:
                     opts['level'] = level
 
                 return HGTYPE2, engine, opts
 
         # No mutually supported compression format. Fall back to the
         # legacy protocol.
 
     # Don't allow untrusted settings because disabling compression or
     # setting a very high compression level could lead to flooding
     # the server's network or CPU.
     opts = {'level': ui.configint('server', 'zliblevel')}
     return HGTYPE, util.compengines['zlib'], opts
 
 def _callhttp(repo, req, res, proto, cmd):
     # Avoid cycle involving hg module.
     from .hgweb import common as hgwebcommon
 
     def genversion2(gen, engine, engineopts):
         # application/mercurial-0.2 always sends a payload header
         # identifying the compression engine.
         name = engine.wireprotosupport().name
         assert 0 < len(name) < 256
         yield struct.pack('B', len(name))
         yield name
 
         for chunk in gen:
             yield chunk
 
     def setresponse(code, contenttype, bodybytes=None, bodygen=None):
         if code == HTTP_OK:
             res.status = '200 Script output follows'
         else:
             res.status = hgwebcommon.statusmessage(code)
 
         res.headers['Content-Type'] = contenttype
 
         if bodybytes is not None:
             res.setbodybytes(bodybytes)
         if bodygen is not None:
             res.setbodygen(bodygen)
 
     if not wireproto.commands.commandavailable(cmd, proto):
         setresponse(HTTP_OK, HGERRTYPE,
                     _('requested wire protocol command is not available over '
                       'HTTP'))
         return
 
     proto.checkperm(wireproto.commands[cmd].permission)
 
     rsp = wireproto.dispatch(repo, proto, cmd)
 
     if isinstance(rsp, bytes):
         setresponse(HTTP_OK, HGTYPE, bodybytes=rsp)
     elif isinstance(rsp, wireprototypes.bytesresponse):
         setresponse(HTTP_OK, HGTYPE, bodybytes=rsp.data)
     elif isinstance(rsp, wireprototypes.streamreslegacy):
         setresponse(HTTP_OK, HGTYPE, bodygen=rsp.gen)
     elif isinstance(rsp, wireprototypes.streamres):
         gen = rsp.gen
 
         # This code for compression should not be streamres specific. It
         # is here because we only compress streamres at the moment.
         mediatype, engine, engineopts = _httpresponsetype(
             repo.ui, proto, rsp.prefer_uncompressed)
         gen = engine.compressstream(gen, engineopts)
 
         if mediatype == HGTYPE2:
             gen = genversion2(gen, engine, engineopts)
 
         setresponse(HTTP_OK, mediatype, bodygen=gen)
     elif isinstance(rsp, wireprototypes.pushres):
         rsp = '%d\n%s' % (rsp.res, rsp.output)
         setresponse(HTTP_OK, HGTYPE, bodybytes=rsp)
     elif isinstance(rsp, wireprototypes.pusherr):
         rsp = '0\n%s\n' % rsp.res
         res.drain = True
         setresponse(HTTP_OK, HGTYPE, bodybytes=rsp)
     elif isinstance(rsp, wireprototypes.ooberror):
         setresponse(HTTP_OK, HGERRTYPE, bodybytes=rsp.message)
     else:
         raise error.ProgrammingError('hgweb.protocol internal failure', rsp)
 
 def _sshv1respondbytes(fout, value):
     """Send a bytes response for protocol version 1."""
     fout.write('%d\n' % len(value))
     fout.write(value)
     fout.flush()
 
 def _sshv1respondstream(fout, source):
     write = fout.write
     for chunk in source.gen:
         write(chunk)
     fout.flush()
 
 def _sshv1respondooberror(fout, ferr, rsp):
     ferr.write(b'%s\n-\n' % rsp)
     ferr.flush()
     fout.write(b'\n')
     fout.flush()
 
 @zi.implementer(wireprototypes.baseprotocolhandler)
 class sshv1protocolhandler(object):
     """Handler for requests services via version 1 of SSH protocol."""
     def __init__(self, ui, fin, fout):
         self._ui = ui
         self._fin = fin
         self._fout = fout
         self._protocaps = set()
 
     @property
     def name(self):
         return wireprototypes.SSHV1
 
     def getargs(self, args):
         data = {}
         keys = args.split()
         for n in xrange(len(keys)):
             argline = self._fin.readline()[:-1]
             arg, l = argline.split()
             if arg not in keys:
                 raise error.Abort(_("unexpected parameter %r") % arg)
             if arg == '*':
                 star = {}
                 for k in xrange(int(l)):
                     argline = self._fin.readline()[:-1]
                     arg, l = argline.split()
                     val = self._fin.read(int(l))
                     star[arg] = val
                 data['*'] = star
             else:
                 val = self._fin.read(int(l))
                 data[arg] = val
         return [data[k] for k in keys]
 
     def getprotocaps(self):
         return self._protocaps
 
     def getpayload(self):
         # We initially send an empty response. This tells the client it is
         # OK to start sending data. If a client sees any other response, it
         # interprets it as an error.
         _sshv1respondbytes(self._fout, b'')
 
         # The file is in the form:
         #
         # <chunk size>\n<chunk>
         # ...
         # 0\n
         count = int(self._fin.readline())
         while count:
             yield self._fin.read(count)
             count = int(self._fin.readline())
 
     @contextlib.contextmanager
     def mayberedirectstdio(self):
         yield None
 
     def client(self):
         client = encoding.environ.get('SSH_CLIENT', '').split(' ', 1)[0]
         return 'remote:ssh:' + client
 
     def addcapabilities(self, repo, caps):
         if self.name == wireprototypes.SSHV1:
             caps.append(b'protocaps')
         caps.append(b'batch')
         return caps
 
     def checkperm(self, perm):
         pass
 
 class sshv2protocolhandler(sshv1protocolhandler):
     """Protocol handler for version 2 of the SSH protocol."""
 
     @property
     def name(self):
         return wireprototypes.SSHV2
 
     def addcapabilities(self, repo, caps):
         return caps
 
 def _runsshserver(ui, repo, fin, fout, ev):
     # This function operates like a state machine of sorts. The following
     # states are defined:
     #
     # protov1-serving
     #    Server is in protocol version 1 serving mode. Commands arrive on
     #    new lines. These commands are processed in this state, one command
     #    after the other.
     #
     # protov2-serving
     #    Server is in protocol version 2 serving mode.
     #
     # upgrade-initial
     #    The server is going to process an upgrade request.
     #
     # upgrade-v2-filter-legacy-handshake
     #    The protocol is being upgraded to version 2. The server is expecting
     #    the legacy handshake from version 1.
     #
     # upgrade-v2-finish
     #    The upgrade to version 2 of the protocol is imminent.
     #
     # shutdown
     #    The server is shutting down, possibly in reaction to a client event.
     #
     # And here are their transitions:
     #
     # protov1-serving -> shutdown
     #    When server receives an empty request or encounters another
     #    error.
     #
     # protov1-serving -> upgrade-initial
     #    An upgrade request line was seen.
     #
     # upgrade-initial -> upgrade-v2-filter-legacy-handshake
     #    Upgrade to version 2 in progress. Server is expecting to
     #    process a legacy handshake.
     #
     # upgrade-v2-filter-legacy-handshake -> shutdown
     #    Client did not fulfill upgrade handshake requirements.
     #
     # upgrade-v2-filter-legacy-handshake -> upgrade-v2-finish
     #    Client fulfilled version 2 upgrade requirements. Finishing that
     #    upgrade.
     #
     # upgrade-v2-finish -> protov2-serving
     #    Protocol upgrade to version 2 complete. Server can now speak protocol
     #    version 2.
     #
     # protov2-serving -> protov1-serving
     #    Ths happens by default since protocol version 2 is the same as
     #    version 1 except for the handshake.
 
     state = 'protov1-serving'
     proto = sshv1protocolhandler(ui, fin, fout)
     protoswitched = False
 
     while not ev.is_set():
         if state == 'protov1-serving':
             # Commands are issued on new lines.
             request = fin.readline()[:-1]
 
             # Empty lines signal to terminate the connection.
             if not request:
                 state = 'shutdown'
                 continue
 
             # It looks like a protocol upgrade request. Transition state to
             # handle it.
             if request.startswith(b'upgrade '):
                 if protoswitched:
                     _sshv1respondooberror(fout, ui.ferr,
                                           b'cannot upgrade protocols multiple '
                                           b'times')
                     state = 'shutdown'
                     continue
 
                 state = 'upgrade-initial'
                 continue
 
             available = wireproto.commands.commandavailable(request, proto)
 
             # This command isn't available. Send an empty response and go
             # back to waiting for a new command.
             if not available:
                 _sshv1respondbytes(fout, b'')
                 continue
 
             rsp = wireproto.dispatch(repo, proto, request)
 
             if isinstance(rsp, bytes):
                 _sshv1respondbytes(fout, rsp)
             elif isinstance(rsp, wireprototypes.bytesresponse):
                 _sshv1respondbytes(fout, rsp.data)
             elif isinstance(rsp, wireprototypes.streamres):
                 _sshv1respondstream(fout, rsp)
             elif isinstance(rsp, wireprototypes.streamreslegacy):
                 _sshv1respondstream(fout, rsp)
             elif isinstance(rsp, wireprototypes.pushres):
                 _sshv1respondbytes(fout, b'')
                 _sshv1respondbytes(fout, b'%d' % rsp.res)
             elif isinstance(rsp, wireprototypes.pusherr):
                 _sshv1respondbytes(fout, rsp.res)
             elif isinstance(rsp, wireprototypes.ooberror):
                 _sshv1respondooberror(fout, ui.ferr, rsp.message)
             else:
                 raise error.ProgrammingError('unhandled response type from '
                                              'wire protocol command: %s' % rsp)
 
         # For now, protocol version 2 serving just goes back to version 1.
         elif state == 'protov2-serving':
             state = 'protov1-serving'
             continue
 
         elif state == 'upgrade-initial':
             # We should never transition into this state if we've switched
             # protocols.
             assert not protoswitched
             assert proto.name == wireprototypes.SSHV1
 
             # Expected: upgrade <token> <capabilities>
             # If we get something else, the request is malformed. It could be
             # from a future client that has altered the upgrade line content.
             # We treat this as an unknown command.
             try:
                 token, caps = request.split(b' ')[1:]
             except ValueError:
                 _sshv1respondbytes(fout, b'')
                 state = 'protov1-serving'
                 continue
 
             # Send empty response if we don't support upgrading protocols.
             if not ui.configbool('experimental', 'sshserver.support-v2'):
                 _sshv1respondbytes(fout, b'')
                 state = 'protov1-serving'
                 continue
 
             try:
                 caps = urlreq.parseqs(caps)
             except ValueError:
                 _sshv1respondbytes(fout, b'')
                 state = 'protov1-serving'
                 continue
 
             # We don't see an upgrade request to protocol version 2. Ignore
             # the upgrade request.
             wantedprotos = caps.get(b'proto', [b''])[0]
             if SSHV2 not in wantedprotos:
                 _sshv1respondbytes(fout, b'')
                 state = 'protov1-serving'
                 continue
 
             # It looks like we can honor this upgrade request to protocol 2.
             # Filter the rest of the handshake protocol request lines.
             state = 'upgrade-v2-filter-legacy-handshake'
             continue
 
         elif state == 'upgrade-v2-filter-legacy-handshake':
             # Client should have sent legacy handshake after an ``upgrade``
             # request. Expected lines:
             #
             #    hello
             #    between
             #    pairs 81
             #    0000...-0000...
 
             ok = True
             for line in (b'hello', b'between', b'pairs 81'):
                 request = fin.readline()[:-1]
 
                 if request != line:
                     _sshv1respondooberror(fout, ui.ferr,
                                           b'malformed handshake protocol: '
                                           b'missing %s' % line)
                     ok = False
                     state = 'shutdown'
                     break
 
             if not ok:
                 continue
 
             request = fin.read(81)
             if request != b'%s-%s' % (b'0' * 40, b'0' * 40):
                 _sshv1respondooberror(fout, ui.ferr,
                                       b'malformed handshake protocol: '
                                       b'missing between argument value')
                 state = 'shutdown'
                 continue
 
             state = 'upgrade-v2-finish'
             continue
 
         elif state == 'upgrade-v2-finish':
             # Send the upgrade response.
             fout.write(b'upgraded %s %s\n' % (token, SSHV2))
             servercaps = wireproto.capabilities(repo, proto)
             rsp = b'capabilities: %s' % servercaps.data
             fout.write(b'%d\n%s\n' % (len(rsp), rsp))
             fout.flush()
 
             proto = sshv2protocolhandler(ui, fin, fout)
             protoswitched = True
 
             state = 'protov2-serving'
             continue
 
         elif state == 'shutdown':
             break
 
         else:
             raise error.ProgrammingError('unhandled ssh server state: %s' %
                                          state)
 
 class sshserver(object):
     def __init__(self, ui, repo, logfh=None):
         self._ui = ui
         self._repo = repo
         self._fin = ui.fin
         self._fout = ui.fout
 
         # Log write I/O to stdout and stderr if configured.
         if logfh:
             self._fout = util.makeloggingfileobject(
                 logfh, self._fout, 'o', logdata=True)
             ui.ferr = util.makeloggingfileobject(
                 logfh, ui.ferr, 'e', logdata=True)
 
         hook.redirect(True)
         ui.fout = repo.ui.fout = ui.ferr
 
         # Prevent insertion/deletion of CRs
         procutil.setbinary(self._fin)
         procutil.setbinary(self._fout)
 
     def serve_forever(self):
         self.serveuntil(threading.Event())
         sys.exit(0)
 
     def serveuntil(self, ev):
         """Serve until a threading.Event is set."""
         _runsshserver(self._ui, self._repo, self._fin, self._fout, ev)

tests/test-wireproto-command-capabilities.t

@@ -1,40 +1,40 b''
   $ . $TESTDIR/wireprotohelpers.sh
 
   $ hg init server
   $ enablehttpv2 server
   $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
   $ cat hg.pid > $DAEMON_PIDS
 
 capabilities request returns an array of capability strings
 
   $ sendhttpv2peer << EOF
   > command capabilities
   > EOF
   creating http peer for wire protocol version 2
   sending capabilities command
   s>     POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
   s>     Accept-Encoding: identity\r\n
   s>     accept: application/mercurial-exp-framing-0003\r\n
   s>     content-type: application/mercurial-exp-framing-0003\r\n
   s>     content-length: 27\r\n
   s>     host: $LOCALIP:$HGPORT\r\n (glob)
   s>     user-agent: Mercurial debugwireproto\r\n
   s>     \r\n
   s>     \x13\x00\x00\x01\x00\x01\x01\x11\xa1DnameLcapabilities
   s> makefile('rb', None)
   s>     HTTP/1.1 200 OK\r\n
   s>     Server: testing stub value\r\n
   s>     Date: $HTTP_DATE$\r\n
   s>     Content-Type: application/mercurial-exp-framing-0003\r\n
   s>     Transfer-Encoding: chunked\r\n
   s>     \r\n
   s>     *\r\n (glob)
   s>     *\x00\x01\x00\x02\x01F (glob)
-  s>     \xa2Hcommands\xaaEheads\xa2Dargs\x81JpubliconlyKpermissions\x81DpullEknown\xa2Dargs\x81EnodesKpermissions\x81DpullFlookup\xa2Dargs\x81CkeyKpermissions\x81DpullGpushkey\xa2Dargs\x84CkeyInamespaceCnewColdKpermissions\x81DpushHlistkeys\xa2Dargs\x81InamespaceKpermissions\x81DpullHunbundle\xa2Dargs\x81EheadsKpermissions\x81DpushIbranchmap\xa2Dargs\x80Kpermissions\x81DpullIgetbundle\xa2Dargs\x81A*Kpermissions\x81DpullLcapabilities\xa2Dargs\x80Kpermissions\x81DpullLclonebundles\xa2Dargs\x80Kpermissions\x81DpullKcompression\x82\xa1DnameDzstd\xa1DnameDzlib
+  s>     \xa2Hcommands\xaaEheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyFlegacyKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyFlegacyCnewFlegacyColdFlegacyInamespaceFlegacyKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullHunbundle\xa2Dargs\xa1EheadsFlegacyKpermissions\x81DpushIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullIgetbundle\xa2Dargs\xa1A*FlegacyKpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLclonebundles\xa2Dargs\xa0Kpermissions\x81DpullKcompression\x82\xa1DnameDzstd\xa1DnameDzlib
   s>     \r\n
   received frame(size=*; request=1; stream=2; streamflags=stream-begin; type=bytes-response; flags=eos|cbor) (glob)
   s>     0\r\n
   s>     \r\n
-  response: [{b'commands': {b'branchmap': {b'args': [], b'permissions': [b'pull']}, b'capabilities': {b'args': [], b'permissions': [b'pull']}, b'clonebundles': {b'args': [], b'permissions': [b'pull']}, b'getbundle': {b'args': [b'*'], b'permissions': [b'pull']}, b'heads': {b'args': [b'publiconly'], b'permissions': [b'pull']}, b'known': {b'args': [b'nodes'], b'permissions': [b'pull']}, b'listkeys': {b'args': [b'namespace'], b'permissions': [b'pull']}, b'lookup': {b'args': [b'key'], b'permissions': [b'pull']}, b'pushkey': {b'args': [b'key', b'namespace', b'new', b'old'], b'permissions': [b'push']}, b'unbundle': {b'args': [b'heads'], b'permissions': [b'push']}}, b'compression': [{b'name': b'zstd'}, {b'name': b'zlib'}]}]
+  response: [{b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'clonebundles': {b'args': {}, b'permissions': [b'pull']}, b'getbundle': {b'args': {b'*': b'legacy'}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'legacy'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'legacy', b'namespace': b'legacy', b'new': b'legacy', b'old': b'legacy'}, b'permissions': [b'push']}, b'unbundle': {b'args': {b'heads': b'legacy'}, b'permissions': [b'push']}}, b'compression': [{b'name': b'zstd'}, {b'name': b'zlib'}]}]
 
   $ cat error.log