wireprotov2: add phases to "changesetdata" command

This commit teaches the "changesetdata" wire protocol command
to emit the phase state for each changeset.

This is a different approach from existing phase transfer in a
few ways. Previously, if there are no new revisions (or we're
not using bundle2), we perform a "listkeys" request to retrieve
phase heads. And when revision data is being transferred
with bundle2, phases data is encoded in a standalone bundle2 part.
In both cases, phases data is logically decoupled from the changeset
data and is encountered/applied after changeset revision data
is received.

The new wire protocol purposefully tries to more tightly associate
changeset metadata (phases, bookmarks, obsolescence markers, etc)
with the changeset revision and index data itself, rather than
have it live as a separate entity that must be fetched and
processed separately. I reckon that one reason we didn't do this
before was it was difficult to add new data types/fields without
breaking existing consumers. By using CBOR maps to transfer
changeset data and putting clients in control of what fields are
requested / present in those maps, we can easily add additional
changeset data while maintaining backwards compatibility. I believe
this to be a superior approach to the problem.

That being said, for performance reasons, we may need to resort
to alternative mechanisms for transferring data like phases. But
for now, I think giving the wire protocol the ability to transfer
changeset metadata next to the changeset itself is a powerful feature
because it is a raw, changeset-centric data API. And if you build
simple APIs for accessing the fundamental units of repository data,
you enable client-side experimentation (partial clone, etc). If it
turns out that we need specialized APIs or mechanisms for transferring
data like phases, we can build in those APIs later. For now, I'd
like to see how far we can get on simple APIs.

It's worth noting that when phase data is being requested, the
server will also emit changeset records for nodes in the bases
specified by the "noderange" argument. This is to ensure that
phase-only updates for nodes the client has are available to the
client, even if no new changesets will be transferred.

Differential Revision: https://phab.mercurial-scm.org/D4483

mercurial/help/internals/wireprotocolv2.txt

@@ -113,6 +113,9 b' fields'
    parents
       Parent revisions.
 
+   phase
+      The phase state of a revision.
+
    revision
       The raw, revision data for the changelog entry. The hash of this data
       will match the revision's node value.
@@ -126,7 +129,8 b' This map has the following bytestring ke'
 
 totalitems
    (unsigned integer) Total number of changelog revisions whose data is being
-   transferred.
+   transferred. This maps to the set of revisions in the requested node
+   range, not the total number of records that follow (see below for why).
 
 Following the map header is a series of 0 or more CBOR values. If values
 are present, the first value will always be a map describing a single changeset
@@ -144,6 +148,11 b' parents (optional)'
    (array of bytestrings) The nodes representing the parent revisions of this
    revision. Only present if ``parents`` data is being requested.
 
+phase (optional)
+   (bytestring) The phase that a revision is in. Recognized values are
+   ``secret``, ``draft``, and ``public``. Only present if ``phase`` data
+   is being requested.
+
 revisionsize (optional)
    (unsigned integer) Indicates the size of raw revision data that follows this
    map. The following data contains a serialized form of the changeset data,
@@ -160,10 +169,19 b' If nodes are requested via ``nodes``, th'
 
 Nodes from ``nodes`` are emitted before nodes from ``noderange``.
 
+The set of changeset revisions emitted may not match the exact set of
+changesets requested. Furthermore, the set of keys present on each
+map may vary. This is to facilitate emitting changeset updates as well
+as new revisions.
+
+For example, if the request wants ``phase`` and ``revision`` data,
+the response may contain entries for each changeset in the common nodes
+set with the ``phase`` key and without the ``revision`` key in order
+to reflect a phase-only update.
+
 TODO support different revision selection mechanisms (e.g. non-public, specific
 revisions)
 TODO support different hash "namespaces" for revisions (e.g. sha-1 versus other)
-TODO support emitting phases data
 TODO support emitting bookmarks data
 TODO support emitting obsolescence data
 TODO support filtering based on relevant paths (narrow clone)

mercurial/wireprotov2server.py

@@ -511,6 +511,7 b' def changesetdata(repo, proto, noderange'
             # list.
 
     seen.clear()
+    publishing = repo.publishing()
 
     if outgoing:
         repo.hook('preoutgoing', throw=True, source='serve')
@@ -519,6 +520,19 b' def changesetdata(repo, proto, noderange'
         b'totalitems': len(outgoing),
     }
 
+    # The phases of nodes already transferred to the client may have changed
+    # since the client last requested data. We send phase-only records
+    # for these revisions, if requested.
+    if b'phase' in fields and noderange is not None:
+        # TODO skip nodes whose phase will be reflected by a node in the
+        # outgoing set. This is purely an optimization to reduce data
+        # size.
+        for node in noderange[0]:
+            yield {
+                b'node': node,
+                b'phase': b'public' if publishing else repo[node].phasestr()
+            }
+
     # It is already topologically sorted by revision number.
     for node in outgoing:
         d = {
@@ -528,6 +542,13 b' def changesetdata(repo, proto, noderange'
         if b'parents' in fields:
             d[b'parents'] = cl.parents(node)
 
+        if b'phase' in fields:
+            if publishing:
+                d[b'phase'] = b'public'
+            else:
+                ctx = repo[node]
+                d[b'phase'] = ctx.phasestr()
+
         revisiondata = None
 
         if b'revision' in fields:

tests/test-wireproto-command-changesetdata.t

@@ -3,6 +3,10 b''
   $ hg init server
   $ enablehttpv2 server
   $ cd server
+  $ cat >> .hg/hgrc << EOF
+  > [phases]
+  > publish = false
+  > EOF
   $ echo a0 > a
   $ echo b0 > b
 
@@ -13,6 +17,7 b''
   $ hg commit -m 'commit 1'
   $ echo b2 > b
   $ hg commit -m 'commit 2'
+  $ hg phase --public -r .
 
   $ hg -q up -r 0
   $ echo a2 > a
@@ -365,6 +370,57 b' Parents data is transferred upon request'
     }
   ]
 
+Phase data is transferred upon request
+
+  $ sendhttpv2peer << EOF
+  > command changesetdata
+  >     fields eval:[b'phase']
+  >     nodes eval:[b'\x0b\xb8\xad\x89\x4a\x15\xb1\x53\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f\x2a\x4b\x28\xdd']
+  > EOF
+  creating http peer for wire protocol version 2
+  sending changesetdata command
+  s>     POST /api/exp-http-v2-0001/ro/changesetdata HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 76\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     D\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa2Ffields\x81EphaseEnodes\x81T\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddDnameMchangesetdata
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     3d\r\n
+  s>     5\x00\x00\x01\x00\x02\x001
+  s>     \xa1Jtotalitems\x01\xa2DnodeT\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddEphaseFpublic
+  s>     \r\n
+  received frame(size=53; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: gen[
+    {
+      b'totalitems': 1
+    },
+    {
+      b'node': b'\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xdd',
+      b'phase': b'public'
+    }
+  ]
+
 Revision data is transferred upon request
 
   $ sendhttpv2peer << EOF
@@ -483,4 +539,101 b' Multiple fields can be transferred'
     b'1b74476799ec8318045db759b1b4bcc9b839d0aa\ntest\n0 0\na\n\ncommit 3'
   ]
 
+Base nodes have just their metadata (e.g. phase) transferred
+
+  $ sendhttpv2peer << EOF
+  > command changesetdata
+  >     fields eval:[b'phase', b'parents', b'revision']
+  >     noderange eval:[[b'\x33\x90\xef\x85\x00\x73\xfb\xc2\xf0\xdf\xff\x22\x44\x34\x2c\x8e\x92\x29\x01\x3a'], [b'\x0b\xb8\xad\x89\x4a\x15\xb1\x53\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f\x2a\x4b\x28\xdd', b'\xea\xe5\xf8\x2c\x2e\x62\x23\x68\xd2\x7d\xae\xcb\x76\xb7\xe3\x93\xd0\xf2\x42\x11']]
+  > EOF
+  creating http peer for wire protocol version 2
+  sending changesetdata command
+  s>     POST /api/exp-http-v2-0001/ro/changesetdata HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 141\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     \x85\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa2Ffields\x83EphaseGparentsHrevisionInoderange\x82\x81T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:\x82T\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddT\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11DnameMchangesetdata
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     239\r\n
+  s>     1\x02\x00\x01\x00\x02\x001
+  s>     \xa1Jtotalitems\x03\xa2DnodeT3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:EphaseFpublic\xa4DnodeTu\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1Gparents\x82T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseFpublicLrevisionsize\x18?X?7f144aea0ba742713887b564d57e9d12f12ff382\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     b\n
+  s>     \n
+  s>     commit 1\xa4DnodeT\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddGparents\x82Tu\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseFpublicLrevisionsize\x18=X=37f0a2d1c28ffe4b879109a7d1bbf8f07b3c763b\n
+  s>     test\n
+  s>     0 0\n
+  s>     b\n
+  s>     \n
+  s>     commit 2\xa4DnodeT\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11Gparents\x82T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseEdraftLrevisionsize\x18=X=1b74476799ec8318045db759b1b4bcc9b839d0aa\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 3
+  s>     \r\n
+  received frame(size=561; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: gen[
+    {
+      b'totalitems': 3
+    },
+    {
+      b'node': b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+      b'phase': b'public'
+    },
+    {
+      b'node': b'u\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1',
+      b'parents': [
+        b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'public',
+      b'revisionsize': 63
+    },
+    b'7f144aea0ba742713887b564d57e9d12f12ff382\ntest\n0 0\na\nb\n\ncommit 1',
+    {
+      b'node': b'\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xdd',
+      b'parents': [
+        b'u\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'public',
+      b'revisionsize': 61
+    },
+    b'37f0a2d1c28ffe4b879109a7d1bbf8f07b3c763b\ntest\n0 0\nb\n\ncommit 2',
+    {
+      b'node': b'\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11',
+      b'parents': [
+        b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'draft',
+      b'revisionsize': 61
+    },
+    b'1b74476799ec8318045db759b1b4bcc9b839d0aa\ntest\n0 0\na\n\ncommit 3'
+  ]
+
   $ cat error.log