clonebundles: introduce a new write protocol command

The old wire command does not take argument so we need a new command. We make
that new command take argument to prepare it for the future (like narrow
support).

hgext/clonebundles.py

@@ -1,1036 +1,1037 b''
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 """advertise pre-generated bundles to seed clones
 
 "clonebundles" is a server-side extension used to advertise the existence
 of pre-generated, externally hosted bundle files to clients that are
 cloning so that cloning can be faster, more reliable, and require less
 resources on the server. "pullbundles" is a related feature for sending
 pre-generated bundle files to clients as part of pull operations.
 
 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
 the server, in response to a client's request to clone, dynamically generates
 a bundle containing the entire repository content and sends it to the client.
 There is no caching on the server and the server will have to redundantly
 generate the same outgoing bundle in response to each clone request. For
 servers with large repositories or with high clone volume, the load from
 clones can make scaling the server challenging and costly.
 
 This extension provides server operators the ability to offload
 potentially expensive clone load to an external service. Pre-generated
 bundles also allow using more CPU intensive compression, reducing the
 effective bandwidth requirements.
 
 Here's how clone bundles work:
 
 1. A server operator establishes a mechanism for making bundle files available
    on a hosting service where Mercurial clients can fetch them.
 2. A manifest file listing available bundle URLs and some optional metadata
    is added to the Mercurial repository on the server.
 3. A client initiates a clone against a clone bundles aware server.
 4. The client sees the server is advertising clone bundles and fetches the
    manifest listing available bundles.
 5. The client filters and sorts the available bundles based on what it
    supports and prefers.
 6. The client downloads and applies an available bundle from the
    server-specified URL.
 7. The client reconnects to the original server and performs the equivalent
    of :hg:`pull` to retrieve all repository data not in the bundle. (The
    repository could have been updated between when the bundle was created
    and when the client started the clone.) This may use "pullbundles".
 
 Instead of the server generating full repository bundles for every clone
 request, it generates full bundles once and they are subsequently reused to
 bootstrap new clones. The server may still transfer data at clone time.
 However, this is only data that has been added/changed since the bundle was
 created. For large, established repositories, this can reduce server load for
 clones to less than 1% of original.
 
 Here's how pullbundles work:
 
 1. A manifest file listing available bundles and describing the revisions
    is added to the Mercurial repository on the server.
 2. A new-enough client informs the server that it supports partial pulls
    and initiates a pull.
 3. If the server has pull bundles enabled and sees the client advertising
    partial pulls, it checks for a matching pull bundle in the manifest.
    A bundle matches if the format is supported by the client, the client
    has the required revisions already and needs something from the bundle.
 4. If there is at least one matching bundle, the server sends it to the client.
 5. The client applies the bundle and notices that the server reply was
    incomplete. It initiates another pull.
 
 To work, this extension requires the following of server operators:
 
 * Generating bundle files of repository content (typically periodically,
   such as once per day).
 * Clone bundles: A file server that clients have network access to and that
   Python knows how to talk to through its normal URL handling facility
   (typically an HTTP/HTTPS server).
 * A process for keeping the bundles manifest in sync with available bundle
   files.
 
 Strictly speaking, using a static file hosting server isn't required: a server
 operator could use a dynamic service for retrieving bundle data. However,
 static file hosting services are simple and scalable and should be sufficient
 for most needs.
 
 Bundle files can be generated with the :hg:`bundle` command. Typically
 :hg:`bundle --all` is used to produce a bundle of the entire repository.
 
 The bundlespec option `stream` (see :hg:`help bundlespec`)
 can be used to produce a special *streaming clonebundle*, typically using
 :hg:`bundle --all --type="none-streamv2"`.
 These are bundle files that are extremely efficient
 to produce and consume (read: fast). However, they are larger than
 traditional bundle formats and require that clients support the exact set
 of repository data store formats in use by the repository that created them.
 Typically, a newer server can serve data that is compatible with older clients.
 However, *streaming clone bundles* don't have this guarantee. **Server
 operators need to be aware that newer versions of Mercurial may produce
 streaming clone bundles incompatible with older Mercurial versions.**
 
 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
 file containing the list of available bundle files suitable for seeding
 clones. If this file does not exist, the repository will not advertise the
 existence of clone bundles when clients connect. For pull bundles,
 ``.hg/pullbundles.manifest`` is used.
 
 The manifest file contains a newline (\\n) delimited list of entries.
 
 Each line in this file defines an available bundle. Lines have the format:
 
     <URL> [<key>=<value>[ <key>=<value>]]
 
 That is, a URL followed by an optional, space-delimited list of key=value
 pairs describing additional properties of this bundle. Both keys and values
 are URI encoded.
 
 For pull bundles, the URL is a path under the ``.hg`` directory of the
 repository.
 
 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
 All non-uppercase keys can be used by site installations. An example use
 for custom properties is to use the *datacenter* attribute to define which
 data center a file is hosted in. Clients could then prefer a server in the
 data center closest to them.
 
 The following reserved keys are currently defined:
 
 BUNDLESPEC
    A "bundle specification" string that describes the type of the bundle.
 
    These are string values that are accepted by the "--type" argument of
    :hg:`bundle`.
 
    The values are parsed in strict mode, which means they must be of the
    "<compression>-<type>" form. See
    mercurial.exchange.parsebundlespec() for more details.
 
    :hg:`debugbundle --spec` can be used to print the bundle specification
    string for a bundle file. The output of this command can be used verbatim
    for the value of ``BUNDLESPEC`` (it is already escaped).
 
    Clients will automatically filter out specifications that are unknown or
    unsupported so they won't attempt to download something that likely won't
    apply.
 
    The actual value doesn't impact client behavior beyond filtering:
    clients will still sniff the bundle type from the header of downloaded
    files.
 
    **Use of this key is highly recommended**, as it allows clients to
    easily skip unsupported bundles. If this key is not defined, an old
    client may attempt to apply a bundle that it is incapable of reading.
 
 REQUIRESNI
    Whether Server Name Indication (SNI) is required to connect to the URL.
    SNI allows servers to use multiple certificates on the same IP. It is
    somewhat common in CDNs and other hosting providers. Older Python
    versions do not support SNI. Defining this attribute enables clients
    with older Python versions to filter this entry without experiencing
    an opaque SSL failure at connection time.
 
    If this is defined, it is important to advertise a non-SNI fallback
    URL or clients running old Python releases may not be able to clone
    with the clonebundles facility.
 
    Value should be "true".
 
 REQUIREDRAM
    Value specifies expected memory requirements to decode the payload.
    Values can have suffixes for common bytes sizes. e.g. "64MB".
 
    This key is often used with zstd-compressed bundles using a high
    compression level / window size, which can require 100+ MB of memory
    to decode.
 
 heads
    Used for pull bundles. This contains the ``;`` separated changeset
    hashes of the heads of the bundle content.
 
 bases
    Used for pull bundles. This contains the ``;`` separated changeset
    hashes of the roots of the bundle content. This can be skipped if
    the bundle was created without ``--base``.
 
 Manifests can contain multiple entries. Assuming metadata is defined, clients
 will filter entries from the manifest that they don't support. The remaining
 entries are optionally sorted by client preferences
 (``ui.clonebundleprefers`` config option). The client then attempts
 to fetch the bundle at the first URL in the remaining list.
 
 **Errors when downloading a bundle will fail the entire clone operation:
 clients do not automatically fall back to a traditional clone.** The reason
 for this is that if a server is using clone bundles, it is probably doing so
 because the feature is necessary to help it scale. In other words, there
 is an assumption that clone load will be offloaded to another service and
 that the Mercurial server isn't responsible for serving this clone load.
 If that other service experiences issues and clients start mass falling back to
 the original Mercurial server, the added clone load could overwhelm the server
 due to unexpected load and effectively take it offline. Not having clients
 automatically fall back to cloning from the original server mitigates this
 scenario.
 
 Because there is no automatic Mercurial server fallback on failure of the
 bundle hosting service, it is important for server operators to view the bundle
 hosting service as an extension of the Mercurial server in terms of
 availability and service level agreements: if the bundle hosting service goes
 down, so does the ability for clients to clone. Note: clients will see a
 message informing them how to bypass the clone bundles facility when a failure
 occurs. So server operators should prepare for some people to follow these
 instructions when a failure occurs, thus driving more load to the original
 Mercurial server when the bundle hosting service fails.
 
 
 inline clonebundles
 -------------------
 
 It is possible to transmit clonebundles inline in case repositories are
 accessed over SSH. This avoids having to setup an external HTTPS server
 and results in the same access control as already present for the SSH setup.
 
 Inline clonebundles should be placed into the `.hg/bundle-cache` directory.
 A clonebundle at `.hg/bundle-cache/mybundle.bundle` is referred to
 in the `clonebundles.manifest` file as `peer-bundle-cache://mybundle.bundle`.
 
 
 auto-generation of clone bundles
 --------------------------------
 
 It is possible to set Mercurial to automatically re-generate clone bundles when
 enough new content is available.
 
 Mercurial will take care of the process asynchronously. The defined list of
 bundle-type will be generated, uploaded, and advertised. Older bundles will get
 decommissioned as newer ones replace them.
 
 Bundles Generation:
 ...................
 
 The extension can generate multiple variants of the clone bundle. Each
 different variant will be defined by the "bundle-spec" they use::
 
     [clone-bundles]
     auto-generate.formats= zstd-v2, gzip-v2
 
 See `hg help bundlespec` for details about available options.
 
 By default, new bundles are generated when 5% of the repository contents or at
 least 1000 revisions are not contained in the cached bundles. This option can
 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
 
     [clone-bundles]
     trigger.below-bundled-ratio=0.95
     trigger.revs=1000
 
 This logic can be manually triggered using the `admin::clone-bundles-refresh`
 command, or automatically on each repository change if
 `clone-bundles.auto-generate.on-change` is set to `yes`.
 
     [clone-bundles]
     auto-generate.on-change=yes
     auto-generate.formats= zstd-v2, gzip-v2
 
 Bundles Upload and Serving:
 ...........................
 
 The generated bundles need to be made available to users through a "public" URL.
 This should be donne through `clone-bundles.upload-command` configuration. The
 value of this command should be a shell command. It will have access to the
 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
 basename in the "public" URL is accessible at::
 
   [clone-bundles]
   upload-command=sftp put $HGCB_BUNDLE_PATH \
       sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
 
 If the file was already uploaded, the command must still succeed.
 
 After upload, the file should be available at an url defined by
 `clone-bundles.url-template`.
 
   [clone-bundles]
   url-template=https://bundles.host/cache/clone-bundles/{basename}
 
 Old bundles cleanup:
 ....................
 
 When new bundles are generated, the older ones are no longer necessary and can
 be removed from storage. This is done through the `clone-bundles.delete-command`
 configuration. The command is given the url of the artifact to delete through
 the `$HGCB_BUNDLE_URL` environment variable.
 
   [clone-bundles]
   delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
 
 If the file was already deleted, the command must still succeed.
 """
 
 
 import os
 import weakref
 
 from mercurial.i18n import _
 
 from mercurial import (
     bundlecaches,
     commands,
     error,
     extensions,
     localrepo,
     lock,
     node,
     registrar,
     util,
     wireprotov1server,
 )
 
 
 from mercurial.utils import (
     procutil,
 )
 
 testedwith = b'ships-with-hg-core'
 
 
 def capabilities(orig, repo, proto):
     caps = orig(repo, proto)
 
     # Only advertise if a manifest exists. This does add some I/O to requests.
     # But this should be cheaper than a wasted network round trip due to
     # missing file.
     if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
         caps.append(b'clonebundles')
+        caps.append(b'clonebundles_manifest')
 
     return caps
 
 
 def extsetup(ui):
     extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities)
 
 
 # logic for bundle auto-generation
 
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
 configitem(b'clone-bundles', b'auto-generate.formats', default=list)
 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
 configitem(b'clone-bundles', b'trigger.revs', default=1000)
 
 configitem(b'clone-bundles', b'upload-command', default=None)
 
 configitem(b'clone-bundles', b'delete-command', default=None)
 
 configitem(b'clone-bundles', b'url-template', default=None)
 
 configitem(b'devel', b'debug.clonebundles', default=False)
 
 
 # category for the post-close transaction hooks
 CAT_POSTCLOSE = b"clonebundles-autobundles"
 
 # template for bundle file names
 BUNDLE_MASK = (
     b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
 )
 
 
 # file in .hg/ use to track clonebundles being auto-generated
 AUTO_GEN_FILE = b'clonebundles.auto-gen'
 
 
 class BundleBase(object):
     """represents the core of properties that matters for us in a bundle
 
     :bundle_type: the bundlespec (see hg help bundlespec)
     :revs:        the number of revisions in the repo at bundle creation time
     :tip_rev:     the rev-num of the tip revision
     :tip_node:    the node id of the tip-most revision in the bundle
 
     :ready:       True if the bundle is ready to be served
     """
 
     ready = False
 
     def __init__(self, bundle_type, revs, tip_rev, tip_node):
         self.bundle_type = bundle_type
         self.revs = revs
         self.tip_rev = tip_rev
         self.tip_node = tip_node
 
     def valid_for(self, repo):
         """is this bundle applicable to the current repository
 
         This is useful for detecting bundles made irrelevant by stripping.
         """
         tip_node = node.bin(self.tip_node)
         return repo.changelog.index.get_rev(tip_node) == self.tip_rev
 
     def __eq__(self, other):
         left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
         right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
         return left == right
 
     def __neq__(self, other):
         return not self == other
 
     def __cmp__(self, other):
         if self == other:
             return 0
         return -1
 
 
 class RequestedBundle(BundleBase):
     """A bundle that should be generated.
 
     Additional attributes compared to BundleBase
     :heads:       list of head revisions (as rev-num)
     :op_id:       a "unique" identifier for the operation triggering the change
     """
 
     def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
         self.head_revs = head_revs
         self.op_id = op_id
         super(RequestedBundle, self).__init__(
             bundle_type,
             revs,
             tip_rev,
             tip_node,
         )
 
     @property
     def suggested_filename(self):
         """A filename that can be used for the generated bundle"""
         data = {
             b'bundle_type': self.bundle_type,
             b'revs': self.revs,
             b'heads': self.head_revs,
             b'tip_rev': self.tip_rev,
             b'tip_node': self.tip_node,
             b'tip_short': self.tip_node[:12],
             b'op_id': self.op_id,
         }
         return BUNDLE_MASK % data
 
     def generate_bundle(self, repo, file_path):
         """generate the bundle at `filepath`"""
         commands.bundle(
             repo.ui,
             repo,
             file_path,
             base=[b"null"],
             rev=self.head_revs,
             type=self.bundle_type,
             quiet=True,
         )
 
     def generating(self, file_path, hostname=None, pid=None):
         """return a GeneratingBundle object from this object"""
         if pid is None:
             pid = os.getpid()
         if hostname is None:
             hostname = lock._getlockprefix()
         return GeneratingBundle(
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             hostname,
             pid,
             file_path,
         )
 
 
 class GeneratingBundle(BundleBase):
     """A bundle being generated
 
     extra attributes compared to BundleBase:
 
     :hostname: the hostname of the machine generating the bundle
     :pid:      the pid of the process generating the bundle
     :filepath: the target filename of the bundle
 
     These attributes exist to help detect stalled generation processes.
     """
 
     ready = False
 
     def __init__(
         self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
     ):
         self.hostname = hostname
         self.pid = pid
         self.filepath = filepath
         super(GeneratingBundle, self).__init__(
             bundle_type, revs, tip_rev, tip_node
         )
 
     @classmethod
     def from_line(cls, line):
         """create an object by deserializing a line from AUTO_GEN_FILE"""
         assert line.startswith(b'PENDING-v1 ')
         (
             __,
             bundle_type,
             revs,
             tip_rev,
             tip_node,
             hostname,
             pid,
             filepath,
         ) = line.split()
         hostname = util.urlreq.unquote(hostname)
         filepath = util.urlreq.unquote(filepath)
         revs = int(revs)
         tip_rev = int(tip_rev)
         pid = int(pid)
         return cls(
             bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
         )
 
     def to_line(self):
         """serialize the object to include as a line in AUTO_GEN_FILE"""
         templ = b"PENDING-v1 %s %d %d %s %s %d %s"
         data = (
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             util.urlreq.quote(self.hostname),
             self.pid,
             util.urlreq.quote(self.filepath),
         )
         return templ % data
 
     def __eq__(self, other):
         if not super(GeneratingBundle, self).__eq__(other):
             return False
         left = (self.hostname, self.pid, self.filepath)
         right = (other.hostname, other.pid, other.filepath)
         return left == right
 
     def uploaded(self, url, basename):
         """return a GeneratedBundle from this object"""
         return GeneratedBundle(
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             url,
             basename,
         )
 
 
 class GeneratedBundle(BundleBase):
     """A bundle that is done being generated and can be served
 
     extra attributes compared to BundleBase:
 
     :file_url: the url where the bundle is available.
     :basename: the "basename" used to upload (useful for deletion)
 
     These attributes exist to generate a bundle manifest
     (.hg/pullbundles.manifest)
     """
 
     ready = True
 
     def __init__(
         self, bundle_type, revs, tip_rev, tip_node, file_url, basename
     ):
         self.file_url = file_url
         self.basename = basename
         super(GeneratedBundle, self).__init__(
             bundle_type, revs, tip_rev, tip_node
         )
 
     @classmethod
     def from_line(cls, line):
         """create an object by deserializing a line from AUTO_GEN_FILE"""
         assert line.startswith(b'DONE-v1 ')
         (
             __,
             bundle_type,
             revs,
             tip_rev,
             tip_node,
             file_url,
             basename,
         ) = line.split()
         revs = int(revs)
         tip_rev = int(tip_rev)
         file_url = util.urlreq.unquote(file_url)
         return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
 
     def to_line(self):
         """serialize the object to include as a line in AUTO_GEN_FILE"""
         templ = b"DONE-v1 %s %d %d %s %s %s"
         data = (
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             util.urlreq.quote(self.file_url),
             self.basename,
         )
         return templ % data
 
     def manifest_line(self):
         """serialize the object to include as a line in pullbundles.manifest"""
         templ = b"%s BUNDLESPEC=%s REQUIRESNI=true"
         return templ % (self.file_url, self.bundle_type)
 
     def __eq__(self, other):
         if not super(GeneratedBundle, self).__eq__(other):
             return False
         return self.file_url == other.file_url
 
 
 def parse_auto_gen(content):
     """parse the AUTO_GEN_FILE to return a list of Bundle object"""
     bundles = []
     for line in content.splitlines():
         if line.startswith(b'PENDING-v1 '):
             bundles.append(GeneratingBundle.from_line(line))
         elif line.startswith(b'DONE-v1 '):
             bundles.append(GeneratedBundle.from_line(line))
     return bundles
 
 
 def dumps_auto_gen(bundles):
     """serialize a list of Bundle as a AUTO_GEN_FILE content"""
     lines = []
     for b in bundles:
         lines.append(b"%s\n" % b.to_line())
     lines.sort()
     return b"".join(lines)
 
 
 def read_auto_gen(repo):
     """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
     data = repo.vfs.tryread(AUTO_GEN_FILE)
     if not data:
         return []
     return parse_auto_gen(data)
 
 
 def write_auto_gen(repo, bundles):
     """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
     assert repo._cb_lock_ref is not None
     data = dumps_auto_gen(bundles)
     with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
         f.write(data)
 
 
 def generate_manifest(bundles):
     """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
     bundles = list(bundles)
     bundles.sort(key=lambda b: b.bundle_type)
     lines = []
     for b in bundles:
         lines.append(b"%s\n" % b.manifest_line())
     return b"".join(lines)
 
 
 def update_ondisk_manifest(repo):
     """update the clonebundle manifest with latest url"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
 
         per_types = {}
         for b in bundles:
             if not (b.ready and b.valid_for(repo)):
                 continue
             current = per_types.get(b.bundle_type)
             if current is not None and current.revs >= b.revs:
                 continue
             per_types[b.bundle_type] = b
         manifest = generate_manifest(per_types.values())
         with repo.vfs(
             bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
         ) as f:
             f.write(manifest)
 
 
 def update_bundle_list(repo, new_bundles=(), del_bundles=()):
     """modify the repo's AUTO_GEN_FILE
 
     This method also regenerates the clone bundle manifest when needed"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         if del_bundles:
             bundles = [b for b in bundles if b not in del_bundles]
         new_bundles = [b for b in new_bundles if b not in bundles]
         bundles.extend(new_bundles)
         write_auto_gen(repo, bundles)
         all_changed = []
         all_changed.extend(new_bundles)
         all_changed.extend(del_bundles)
         if any(b.ready for b in all_changed):
             update_ondisk_manifest(repo)
 
 
 def cleanup_tmp_bundle(repo, target):
     """remove a GeneratingBundle file and entry"""
     assert not target.ready
     with repo.clonebundles_lock():
         repo.vfs.tryunlink(target.filepath)
         update_bundle_list(repo, del_bundles=[target])
 
 
 def finalize_one_bundle(repo, target):
     """upload a generated bundle and advertise it in the clonebundles.manifest"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         if target in bundles and target.valid_for(repo):
             result = upload_bundle(repo, target)
             update_bundle_list(repo, new_bundles=[result])
     cleanup_tmp_bundle(repo, target)
 
 
 def find_outdated_bundles(repo, bundles):
     """finds outdated bundles"""
     olds = []
     per_types = {}
     for b in bundles:
         if not b.valid_for(repo):
             olds.append(b)
             continue
         l = per_types.setdefault(b.bundle_type, [])
         l.append(b)
     for key in sorted(per_types):
         all = per_types[key]
         if len(all) > 1:
             all.sort(key=lambda b: b.revs, reverse=True)
             olds.extend(all[1:])
     return olds
 
 
 def collect_garbage(repo):
     """finds outdated bundles and get them deleted"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         olds = find_outdated_bundles(repo, bundles)
         for o in olds:
             delete_bundle(repo, o)
         update_bundle_list(repo, del_bundles=olds)
 
 
 def upload_bundle(repo, bundle):
     """upload the result of a GeneratingBundle and return a GeneratedBundle
 
     The upload is done using the `clone-bundles.upload-command`
     """
     cmd = repo.ui.config(b'clone-bundles', b'upload-command')
     url = repo.ui.config(b'clone-bundles', b'url-template')
     basename = repo.vfs.basename(bundle.filepath)
     filepath = procutil.shellquote(bundle.filepath)
     variables = {
         b'HGCB_BUNDLE_PATH': filepath,
         b'HGCB_BUNDLE_BASENAME': basename,
     }
     env = procutil.shellenviron(environ=variables)
     ret = repo.ui.system(cmd, environ=env)
     if ret:
         raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
     url = (
         url.decode('utf8')
         .format(basename=basename.decode('utf8'))
         .encode('utf8')
     )
     return bundle.uploaded(url, basename)
 
 
 def delete_bundle(repo, bundle):
     """delete a bundle from storage"""
     assert bundle.ready
     msg = b'clone-bundles: deleting bundle %s\n'
     msg %= bundle.basename
     if repo.ui.configbool(b'devel', b'debug.clonebundles'):
         repo.ui.write(msg)
     else:
         repo.ui.debug(msg)
 
     cmd = repo.ui.config(b'clone-bundles', b'delete-command')
     variables = {
         b'HGCB_BUNDLE_URL': bundle.file_url,
         b'HGCB_BASENAME': bundle.basename,
     }
     env = procutil.shellenviron(environ=variables)
     ret = repo.ui.system(cmd, environ=env)
     if ret:
         raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
 
 
 def auto_bundle_needed_actions(repo, bundles, op_id):
     """find the list of bundles that need action
 
     returns a list of RequestedBundle objects that need to be generated and
     uploaded."""
     create_bundles = []
     delete_bundles = []
     repo = repo.filtered(b"immutable")
     targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
     ratio = float(
         repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
     )
     abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
     revs = len(repo.changelog)
     generic_data = {
         'revs': revs,
         'head_revs': repo.changelog.headrevs(),
         'tip_rev': repo.changelog.tiprev(),
         'tip_node': node.hex(repo.changelog.tip()),
         'op_id': op_id,
     }
     for t in targets:
         if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
             data = generic_data.copy()
             data['bundle_type'] = t
             b = RequestedBundle(**data)
             create_bundles.append(b)
     delete_bundles.extend(find_outdated_bundles(repo, bundles))
     return create_bundles, delete_bundles
 
 
 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
     """consider the current cached content and trigger new bundles if needed"""
     threshold = max((revs * ratio), (revs - abs_revs))
     for b in bundles:
         if not b.valid_for(repo) or b.bundle_type != bundle_type:
             continue
         if b.revs > threshold:
             return False
     return True
 
 
 def start_one_bundle(repo, bundle):
     """start the generation of a single bundle file
 
     the `bundle` argument should be a RequestedBundle object.
 
     This data is passed to the `debugmakeclonebundles` "as is".
     """
     data = util.pickle.dumps(bundle)
     cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
     env = procutil.shellenviron()
     msg = b'clone-bundles: starting bundle generation: %s\n'
     stdout = None
     stderr = None
     waits = []
     record_wait = None
     if repo.ui.configbool(b'devel', b'debug.clonebundles'):
         stdout = procutil.stdout
         stderr = procutil.stderr
         repo.ui.write(msg % bundle.bundle_type)
         record_wait = waits.append
     else:
         repo.ui.debug(msg % bundle.bundle_type)
     bg = procutil.runbgcommand
     bg(
         cmd,
         env,
         stdin_bytes=data,
         stdout=stdout,
         stderr=stderr,
         record_wait=record_wait,
     )
     for f in waits:
         f()
 
 
 INTERNAL_CMD = b'debug::internal-make-clone-bundles'
 
 
 @command(INTERNAL_CMD, [], b'')
 def debugmakeclonebundles(ui, repo):
     """Internal command to auto-generate debug bundles"""
     requested_bundle = util.pickle.load(procutil.stdin)
     procutil.stdin.close()
 
     collect_garbage(repo)
 
     fname = requested_bundle.suggested_filename
     fpath = repo.vfs.makedirs(b'tmp-bundles')
     fpath = repo.vfs.join(b'tmp-bundles', fname)
     bundle = requested_bundle.generating(fpath)
     update_bundle_list(repo, new_bundles=[bundle])
 
     requested_bundle.generate_bundle(repo, fpath)
 
     repo.invalidate()
     finalize_one_bundle(repo, bundle)
 
 
 def make_auto_bundler(source_repo):
     reporef = weakref.ref(source_repo)
 
     def autobundle(tr):
         repo = reporef()
         assert repo is not None
         bundles = read_auto_gen(repo)
         new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
         for data in new:
             start_one_bundle(repo, data)
         return None
 
     return autobundle
 
 
 def reposetup(ui, repo):
     """install the two pieces needed for automatic clonebundle generation
 
     - add a "post-close" hook that fires bundling when needed
     - introduce a clone-bundle lock to let multiple processes meddle with the
       state files.
     """
     if not repo.local():
         return
 
     class autobundlesrepo(repo.__class__):
         def transaction(self, *args, **kwargs):
             tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
             enabled = repo.ui.configbool(
                 b'clone-bundles',
                 b'auto-generate.on-change',
             )
             targets = repo.ui.configlist(
                 b'clone-bundles', b'auto-generate.formats'
             )
             if enabled and targets:
                 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
             return tr
 
         @localrepo.unfilteredmethod
         def clonebundles_lock(self, wait=True):
             '''Lock the repository file related to clone bundles'''
             if not util.safehasattr(self, '_cb_lock_ref'):
                 self._cb_lock_ref = None
             l = self._currentlock(self._cb_lock_ref)
             if l is not None:
                 l.lock()
                 return l
 
             l = self._lock(
                 vfs=self.vfs,
                 lockname=b"clonebundleslock",
                 wait=wait,
                 releasefn=None,
                 acquirefn=None,
                 desc=_(b'repository %s') % self.origroot,
             )
             self._cb_lock_ref = weakref.ref(l)
             return l
 
     repo._wlockfreeprefix.add(AUTO_GEN_FILE)
     repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
     repo.__class__ = autobundlesrepo
 
 
 @command(
     b'admin::clone-bundles-refresh',
     [
         (
             b'',
             b'background',
             False,
             _(b'start bundle generation in the background'),
         ),
     ],
     b'',
 )
 def cmd_admin_clone_bundles_refresh(
     ui,
     repo: localrepo.localrepository,
     background=False,
 ):
     """generate clone bundles according to the configuration
 
     This runs the logic for automatic generation, removing outdated bundles and
     generating new ones if necessary. See :hg:`help -e clone-bundles` for
     details about how to configure this feature.
     """
     debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
     bundles = read_auto_gen(repo)
     op_id = b"%d_acbr" % os.getpid()
     create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
 
     # if some bundles are scheduled for creation in the background, they will
     # deal with garbage collection too, so no need to synchroniously do it.
     #
     # However if no bundles are scheduled for creation, we need to explicitly do
     # it here.
     if not (background and create):
         # we clean up outdated bundles before generating new ones to keep the
         # last two versions of the bundle around for a while and avoid having to
         # deal with clients that just got served a manifest.
         for o in delete:
             delete_bundle(repo, o)
         update_bundle_list(repo, del_bundles=delete)
 
     if create:
         fpath = repo.vfs.makedirs(b'tmp-bundles')
 
     if background:
         for requested_bundle in create:
             start_one_bundle(repo, requested_bundle)
     else:
         for requested_bundle in create:
             if debug:
                 msg = b'clone-bundles: starting bundle generation: %s\n'
                 repo.ui.write(msg % requested_bundle.bundle_type)
             fname = requested_bundle.suggested_filename
             fpath = repo.vfs.join(b'tmp-bundles', fname)
             generating_bundle = requested_bundle.generating(fpath)
             update_bundle_list(repo, new_bundles=[generating_bundle])
             requested_bundle.generate_bundle(repo, fpath)
             result = upload_bundle(repo, generating_bundle)
             update_bundle_list(repo, new_bundles=[result])
             update_ondisk_manifest(repo)
             cleanup_tmp_bundle(repo, generating_bundle)
 
 
 @command(b'admin::clone-bundles-clear', [], b'')
 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
     """remove existing clone bundle caches
 
     See `hg help admin::clone-bundles-refresh` for details on how to regenerate
     them.
 
     This command will only affect bundles currently available, it will not
     affect bundles being asynchronously generated.
     """
     bundles = read_auto_gen(repo)
     delete = [b for b in bundles if b.ready]
     for o in delete:
         delete_bundle(repo, o)
     update_bundle_list(repo, del_bundles=delete)

mercurial/wireprotov1peer.py

@@ -1,661 +1,664 b''
 # wireprotov1peer.py - Client-side functionality for wire protocol version 1.
 #
 # Copyright 2005-2010 Olivia Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 
 import sys
 import weakref
 
 from concurrent import futures
 from .i18n import _
 from .node import bin
 from .pycompat import (
     getattr,
     setattr,
 )
 from . import (
     bundle2,
     changegroup as changegroupmod,
     encoding,
     error,
     pushkey as pushkeymod,
     pycompat,
     util,
     wireprototypes,
 )
 from .interfaces import (
     repository,
     util as interfaceutil,
 )
 from .utils import hashutil
 
 urlreq = util.urlreq
 
 
 def batchable(f):
     """annotation for batchable methods
 
     Such methods must implement a coroutine as follows:
 
     @batchable
     def sample(self, one, two=None):
         # Build list of encoded arguments suitable for your wire protocol:
         encoded_args = [('one', encode(one),), ('two', encode(two),)]
         # Return it, along with a function that will receive the result
         # from the batched request.
         return encoded_args, decode
 
     The decorator returns a function which wraps this coroutine as a plain
     method, but adds the original method as an attribute called "batchable",
     which is used by remotebatch to split the call into separate encoding and
     decoding phases.
     """
 
     def plain(*args, **opts):
         encoded_args_or_res, decode = f(*args, **opts)
         if not decode:
             return encoded_args_or_res  # a local result in this case
         self = args[0]
         cmd = pycompat.bytesurl(f.__name__)  # ensure cmd is ascii bytestr
         encoded_res = self._submitone(cmd, encoded_args_or_res)
         return decode(encoded_res)
 
     setattr(plain, 'batchable', f)
     setattr(plain, '__name__', f.__name__)
     return plain
 
 
 def encodebatchcmds(req):
     """Return a ``cmds`` argument value for the ``batch`` command."""
     escapearg = wireprototypes.escapebatcharg
 
     cmds = []
     for op, argsdict in req:
         # Old servers didn't properly unescape argument names. So prevent
         # the sending of argument names that may not be decoded properly by
         # servers.
         assert all(escapearg(k) == k for k in argsdict)
 
         args = b','.join(
             b'%s=%s' % (escapearg(k), escapearg(v)) for k, v in argsdict.items()
         )
         cmds.append(b'%s %s' % (op, args))
 
     return b';'.join(cmds)
 
 
 class unsentfuture(futures.Future):
     """A Future variation to represent an unsent command.
 
     Because we buffer commands and don't submit them immediately, calling
     ``result()`` on an unsent future could deadlock. Futures for buffered
     commands are represented by this type, which wraps ``result()`` to
     call ``sendcommands()``.
     """
 
     def result(self, timeout=None):
         if self.done():
             return futures.Future.result(self, timeout)
 
         self._peerexecutor.sendcommands()
 
         # This looks like it will infinitely recurse. However,
         # sendcommands() should modify __class__. This call serves as a check
         # on that.
         return self.result(timeout)
 
 
 @interfaceutil.implementer(repository.ipeercommandexecutor)
 class peerexecutor:
     def __init__(self, peer):
         self._peer = peer
         self._sent = False
         self._closed = False
         self._calls = []
         self._futures = weakref.WeakSet()
         self._responseexecutor = None
         self._responsef = None
 
     def __enter__(self):
         return self
 
     def __exit__(self, exctype, excvalee, exctb):
         self.close()
 
     def callcommand(self, command, args):
         if self._sent:
             raise error.ProgrammingError(
                 b'callcommand() cannot be used after commands are sent'
             )
 
         if self._closed:
             raise error.ProgrammingError(
                 b'callcommand() cannot be used after close()'
             )
 
         # Commands are dispatched through methods on the peer.
         fn = getattr(self._peer, pycompat.sysstr(command), None)
 
         if not fn:
             raise error.ProgrammingError(
                 b'cannot call command %s: method of same name not available '
                 b'on peer' % command
             )
 
         # Commands are either batchable or they aren't. If a command
         # isn't batchable, we send it immediately because the executor
         # can no longer accept new commands after a non-batchable command.
         # If a command is batchable, we queue it for later. But we have
         # to account for the case of a non-batchable command arriving after
         # a batchable one and refuse to service it.
 
         def addcall():
             f = futures.Future()
             self._futures.add(f)
             self._calls.append((command, args, fn, f))
             return f
 
         if getattr(fn, 'batchable', False):
             f = addcall()
 
             # But since we don't issue it immediately, we wrap its result()
             # to trigger sending so we avoid deadlocks.
             f.__class__ = unsentfuture
             f._peerexecutor = self
         else:
             if self._calls:
                 raise error.ProgrammingError(
                     b'%s is not batchable and cannot be called on a command '
                     b'executor along with other commands' % command
                 )
 
             f = addcall()
 
             # Non-batchable commands can never coexist with another command
             # in this executor. So send the command immediately.
             self.sendcommands()
 
         return f
 
     def sendcommands(self):
         if self._sent:
             return
 
         if not self._calls:
             return
 
         self._sent = True
 
         # Unhack any future types so caller seens a clean type and to break
         # cycle between us and futures.
         for f in self._futures:
             if isinstance(f, unsentfuture):
                 f.__class__ = futures.Future
                 f._peerexecutor = None
 
         calls = self._calls
         # Mainly to destroy references to futures.
         self._calls = None
 
         # Simple case of a single command. We call it synchronously.
         if len(calls) == 1:
             command, args, fn, f = calls[0]
 
             # Future was cancelled. Ignore it.
             if not f.set_running_or_notify_cancel():
                 return
 
             try:
                 result = fn(**pycompat.strkwargs(args))
             except Exception:
                 pycompat.future_set_exception_info(f, sys.exc_info()[1:])
             else:
                 f.set_result(result)
 
             return
 
         # Batch commands are a bit harder. First, we have to deal with the
         # @batchable coroutine. That's a bit annoying. Furthermore, we also
         # need to preserve streaming. i.e. it should be possible for the
         # futures to resolve as data is coming in off the wire without having
         # to wait for the final byte of the final response. We do this by
         # spinning up a thread to read the responses.
 
         requests = []
         states = []
 
         for command, args, fn, f in calls:
             # Future was cancelled. Ignore it.
             if not f.set_running_or_notify_cancel():
                 continue
 
             try:
                 encoded_args_or_res, decode = fn.batchable(
                     fn.__self__, **pycompat.strkwargs(args)
                 )
             except Exception:
                 pycompat.future_set_exception_info(f, sys.exc_info()[1:])
                 return
 
             if not decode:
                 f.set_result(encoded_args_or_res)
             else:
                 requests.append((command, encoded_args_or_res))
                 states.append((command, f, batchable, decode))
 
         if not requests:
             return
 
         # This will emit responses in order they were executed.
         wireresults = self._peer._submitbatch(requests)
 
         # The use of a thread pool executor here is a bit weird for something
         # that only spins up a single thread. However, thread management is
         # hard and it is easy to encounter race conditions, deadlocks, etc.
         # concurrent.futures already solves these problems and its thread pool
         # executor has minimal overhead. So we use it.
         self._responseexecutor = futures.ThreadPoolExecutor(1)
         self._responsef = self._responseexecutor.submit(
             self._readbatchresponse, states, wireresults
         )
 
     def close(self):
         self.sendcommands()
 
         if self._closed:
             return
 
         self._closed = True
 
         if not self._responsef:
             return
 
         # We need to wait on our in-flight response and then shut down the
         # executor once we have a result.
         try:
             self._responsef.result()
         finally:
             self._responseexecutor.shutdown(wait=True)
             self._responsef = None
             self._responseexecutor = None
 
             # If any of our futures are still in progress, mark them as
             # errored. Otherwise a result() could wait indefinitely.
             for f in self._futures:
                 if not f.done():
                     f.set_exception(
                         error.ResponseError(
                             _(b'unfulfilled batch command response'), None
                         )
                     )
 
             self._futures = None
 
     def _readbatchresponse(self, states, wireresults):
         # Executes in a thread to read data off the wire.
 
         for command, f, batchable, decode in states:
             # Grab raw result off the wire and teach the internal future
             # about it.
             try:
                 remoteresult = next(wireresults)
             except StopIteration:
                 # This can happen in particular because next(batchable)
                 # in the previous iteration can call peer._abort, which
                 # may close the peer.
                 f.set_exception(
                     error.ResponseError(
                         _(b'unfulfilled batch command response'), None
                     )
                 )
             else:
                 try:
                     result = decode(remoteresult)
                 except Exception:
                     pycompat.future_set_exception_info(f, sys.exc_info()[1:])
                 else:
                     f.set_result(result)
 
 
 @interfaceutil.implementer(
     repository.ipeercommands, repository.ipeerlegacycommands
 )
 class wirepeer(repository.peer):
     """Client-side interface for communicating with a peer repository.
 
     Methods commonly call wire protocol commands of the same name.
 
     See also httppeer.py and sshpeer.py for protocol-specific
     implementations of this interface.
     """
 
     def commandexecutor(self):
         return peerexecutor(self)
 
     # Begin of ipeercommands interface.
 
     def clonebundles(self):
-        self.requirecap(b'clonebundles', _(b'clone bundles'))
-        return self._call(b'clonebundles')
+        if self.capable(b'clonebundles_manifest'):
+            return self._call(b'clonebundles_manifest')
+        else:
+            self.requirecap(b'clonebundles', _(b'clone bundles'))
+            return self._call(b'clonebundles')
 
     def _finish_inline_clone_bundle(self, stream):
         pass  # allow override for httppeer
 
     def get_cached_bundle_inline(self, path):
         stream = self._callstream(b"get_cached_bundle_inline", path=path)
         length = util.uvarintdecodestream(stream)
 
         # SSH streams will block if reading more than length
         for chunk in util.filechunkiter(stream, limit=length):
             yield chunk
 
         self._finish_inline_clone_bundle(stream)
 
     @batchable
     def lookup(self, key):
         self.requirecap(b'lookup', _(b'look up remote revision'))
 
         def decode(d):
             success, data = d[:-1].split(b" ", 1)
             if int(success):
                 return bin(data)
             else:
                 self._abort(error.RepoError(data))
 
         return {b'key': encoding.fromlocal(key)}, decode
 
     @batchable
     def heads(self):
         def decode(d):
             try:
                 return wireprototypes.decodelist(d[:-1])
             except ValueError:
                 self._abort(error.ResponseError(_(b"unexpected response:"), d))
 
         return {}, decode
 
     @batchable
     def known(self, nodes):
         def decode(d):
             try:
                 return [bool(int(b)) for b in pycompat.iterbytestr(d)]
             except ValueError:
                 self._abort(error.ResponseError(_(b"unexpected response:"), d))
 
         return {b'nodes': wireprototypes.encodelist(nodes)}, decode
 
     @batchable
     def branchmap(self):
         def decode(d):
             try:
                 branchmap = {}
                 for branchpart in d.splitlines():
                     branchname, branchheads = branchpart.split(b' ', 1)
                     branchname = encoding.tolocal(urlreq.unquote(branchname))
                     branchheads = wireprototypes.decodelist(branchheads)
                     branchmap[branchname] = branchheads
                 return branchmap
             except TypeError:
                 self._abort(error.ResponseError(_(b"unexpected response:"), d))
 
         return {}, decode
 
     @batchable
     def listkeys(self, namespace):
         if not self.capable(b'pushkey'):
             return {}, None
         self.ui.debug(b'preparing listkeys for "%s"\n' % namespace)
 
         def decode(d):
             self.ui.debug(
                 b'received listkey for "%s": %i bytes\n' % (namespace, len(d))
             )
             return pushkeymod.decodekeys(d)
 
         return {b'namespace': encoding.fromlocal(namespace)}, decode
 
     @batchable
     def pushkey(self, namespace, key, old, new):
         if not self.capable(b'pushkey'):
             return False, None
         self.ui.debug(b'preparing pushkey for "%s:%s"\n' % (namespace, key))
 
         def decode(d):
             d, output = d.split(b'\n', 1)
             try:
                 d = bool(int(d))
             except ValueError:
                 raise error.ResponseError(
                     _(b'push failed (unexpected response):'), d
                 )
             for l in output.splitlines(True):
                 self.ui.status(_(b'remote: '), l)
             return d
 
         return {
             b'namespace': encoding.fromlocal(namespace),
             b'key': encoding.fromlocal(key),
             b'old': encoding.fromlocal(old),
             b'new': encoding.fromlocal(new),
         }, decode
 
     def stream_out(self):
         return self._callstream(b'stream_out')
 
     def getbundle(self, source, **kwargs):
         kwargs = pycompat.byteskwargs(kwargs)
         self.requirecap(b'getbundle', _(b'look up remote changes'))
         opts = {}
         bundlecaps = kwargs.get(b'bundlecaps') or set()
         for key, value in kwargs.items():
             if value is None:
                 continue
             keytype = wireprototypes.GETBUNDLE_ARGUMENTS.get(key)
             if keytype is None:
                 raise error.ProgrammingError(
                     b'Unexpectedly None keytype for key %s' % key
                 )
             elif keytype == b'nodes':
                 value = wireprototypes.encodelist(value)
             elif keytype == b'csv':
                 value = b','.join(value)
             elif keytype == b'scsv':
                 value = b','.join(sorted(value))
             elif keytype == b'boolean':
                 value = b'%i' % bool(value)
             elif keytype != b'plain':
                 raise KeyError(b'unknown getbundle option type %s' % keytype)
             opts[key] = value
         f = self._callcompressable(b"getbundle", **pycompat.strkwargs(opts))
         if any((cap.startswith(b'HG2') for cap in bundlecaps)):
             return bundle2.getunbundler(self.ui, f)
         else:
             return changegroupmod.cg1unpacker(f, b'UN')
 
     def unbundle(self, bundle, heads, url):
         """Send cg (a readable file-like object representing the
         changegroup to push, typically a chunkbuffer object) to the
         remote server as a bundle.
 
         When pushing a bundle10 stream, return an integer indicating the
         result of the push (see changegroup.apply()).
 
         When pushing a bundle20 stream, return a bundle20 stream.
 
         `url` is the url the client thinks it's pushing to, which is
         visible to hooks.
         """
 
         if heads != [b'force'] and self.capable(b'unbundlehash'):
             heads = wireprototypes.encodelist(
                 [b'hashed', hashutil.sha1(b''.join(sorted(heads))).digest()]
             )
         else:
             heads = wireprototypes.encodelist(heads)
 
         if util.safehasattr(bundle, 'deltaheader'):
             # this a bundle10, do the old style call sequence
             ret, output = self._callpush(b"unbundle", bundle, heads=heads)
             if ret == b"":
                 raise error.ResponseError(_(b'push failed:'), output)
             try:
                 ret = int(ret)
             except ValueError:
                 raise error.ResponseError(
                     _(b'push failed (unexpected response):'), ret
                 )
 
             for l in output.splitlines(True):
                 self.ui.status(_(b'remote: '), l)
         else:
             # bundle2 push. Send a stream, fetch a stream.
             stream = self._calltwowaystream(b'unbundle', bundle, heads=heads)
             ret = bundle2.getunbundler(self.ui, stream)
         return ret
 
     # End of ipeercommands interface.
 
     # Begin of ipeerlegacycommands interface.
 
     def branches(self, nodes):
         n = wireprototypes.encodelist(nodes)
         d = self._call(b"branches", nodes=n)
         try:
             br = [tuple(wireprototypes.decodelist(b)) for b in d.splitlines()]
             return br
         except ValueError:
             self._abort(error.ResponseError(_(b"unexpected response:"), d))
 
     def between(self, pairs):
         batch = 8  # avoid giant requests
         r = []
         for i in range(0, len(pairs), batch):
             n = b" ".join(
                 [
                     wireprototypes.encodelist(p, b'-')
                     for p in pairs[i : i + batch]
                 ]
             )
             d = self._call(b"between", pairs=n)
             try:
                 r.extend(
                     l and wireprototypes.decodelist(l) or []
                     for l in d.splitlines()
                 )
             except ValueError:
                 self._abort(error.ResponseError(_(b"unexpected response:"), d))
         return r
 
     def changegroup(self, nodes, source):
         n = wireprototypes.encodelist(nodes)
         f = self._callcompressable(b"changegroup", roots=n)
         return changegroupmod.cg1unpacker(f, b'UN')
 
     def changegroupsubset(self, bases, heads, source):
         self.requirecap(b'changegroupsubset', _(b'look up remote changes'))
         bases = wireprototypes.encodelist(bases)
         heads = wireprototypes.encodelist(heads)
         f = self._callcompressable(
             b"changegroupsubset", bases=bases, heads=heads
         )
         return changegroupmod.cg1unpacker(f, b'UN')
 
     # End of ipeerlegacycommands interface.
 
     def _submitbatch(self, req):
         """run batch request <req> on the server
 
         Returns an iterator of the raw responses from the server.
         """
         ui = self.ui
         if ui.debugflag and ui.configbool(b'devel', b'debug.peer-request'):
             ui.debug(b'devel-peer-request: batched-content\n')
             for op, args in req:
                 msg = b'devel-peer-request:    - %s (%d arguments)\n'
                 ui.debug(msg % (op, len(args)))
 
         unescapearg = wireprototypes.unescapebatcharg
 
         rsp = self._callstream(b"batch", cmds=encodebatchcmds(req))
         chunk = rsp.read(1024)
         work = [chunk]
         while chunk:
             while b';' not in chunk and chunk:
                 chunk = rsp.read(1024)
                 work.append(chunk)
             merged = b''.join(work)
             while b';' in merged:
                 one, merged = merged.split(b';', 1)
                 yield unescapearg(one)
             chunk = rsp.read(1024)
             work = [merged, chunk]
         yield unescapearg(b''.join(work))
 
     def _submitone(self, op, args):
         return self._call(op, **pycompat.strkwargs(args))
 
     def debugwireargs(self, one, two, three=None, four=None, five=None):
         # don't pass optional arguments left at their default value
         opts = {}
         if three is not None:
             opts['three'] = three
         if four is not None:
             opts['four'] = four
         return self._call(b'debugwireargs', one=one, two=two, **opts)
 
     def _call(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a simple string.
 
         returns the server reply as a string."""
         raise NotImplementedError()
 
     def _callstream(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a stream. Note that if the
         command doesn't return a stream, _callstream behaves
         differently for ssh and http peers.
 
         returns the server reply as a file like object.
         """
         raise NotImplementedError()
 
     def _callcompressable(self, cmd, **args):
         """execute <cmd> on the server
 
         The command is expected to return a stream.
 
         The stream may have been compressed in some implementations. This
         function takes care of the decompression. This is the only difference
         with _callstream.
 
         returns the server reply as a file like object.
         """
         raise NotImplementedError()
 
     def _callpush(self, cmd, fp, **args):
         """execute a <cmd> on server
 
         The command is expected to be related to a push. Push has a special
         return method.
 
         returns the server reply as a (ret, output) tuple. ret is either
         empty (error) or a stringified int.
         """
         raise NotImplementedError()
 
     def _calltwowaystream(self, cmd, fp, **args):
         """execute <cmd> on server
 
         The command will send a stream to the server and get a stream in reply.
         """
         raise NotImplementedError()
 
     def _abort(self, exception):
         """clearly abort the wire protocol connection and raise the exception"""
         raise NotImplementedError()

mercurial/wireprotov1server.py

@@ -1,795 +1,804 b''
 # wireprotov1server.py - Wire protocol version 1 server functionality
 #
 # Copyright 2005-2010 Olivia Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 
 import binascii
 import os
 
 from .i18n import _
 from .node import hex
 from .pycompat import getattr
 
 from . import (
     bundle2,
     bundlecaches,
     changegroup as changegroupmod,
     discovery,
     encoding,
     error,
     exchange,
     hook,
     pushkey as pushkeymod,
     pycompat,
     repoview,
     requirements as requirementsmod,
     streamclone,
     util,
     wireprototypes,
 )
 
 from .utils import (
     procutil,
     stringutil,
 )
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 bundle2requiredmain = _(b'incompatible Mercurial client; bundle2 required')
 bundle2requiredhint = _(
     b'see https://www.mercurial-scm.org/wiki/IncompatibleClient'
 )
 bundle2required = b'%s\n(%s)\n' % (bundle2requiredmain, bundle2requiredhint)
 
 
 def clientcompressionsupport(proto):
     """Returns a list of compression methods supported by the client.
 
     Returns a list of the compression methods supported by the client
     according to the protocol capabilities. If no such capability has
     been announced, fallback to the default of zlib and uncompressed.
     """
     for cap in proto.getprotocaps():
         if cap.startswith(b'comp='):
             return cap[5:].split(b',')
     return [b'zlib', b'none']
 
 
 # wire protocol command can either return a string or one of these classes.
 
 
 def getdispatchrepo(repo, proto, command, accesshidden=False):
     """Obtain the repo used for processing wire protocol commands.
 
     The intent of this function is to serve as a monkeypatch point for
     extensions that need commands to operate on different repo views under
     specialized circumstances.
     """
     viewconfig = repo.ui.config(b'server', b'view')
 
     # Only works if the filter actually supports being upgraded to show hidden
     # changesets.
     if (
         accesshidden
         and viewconfig is not None
         and viewconfig + b'.hidden' in repoview.filtertable
     ):
         viewconfig += b'.hidden'
 
     return repo.filtered(viewconfig)
 
 
 def dispatch(repo, proto, command, accesshidden=False):
     repo = getdispatchrepo(repo, proto, command, accesshidden=accesshidden)
 
     func, spec = commands[command]
     args = proto.getargs(spec)
 
     return func(repo, proto, *args)
 
 
 def options(cmd, keys, others):
     opts = {}
     for k in keys:
         if k in others:
             opts[k] = others[k]
             del others[k]
     if others:
         procutil.stderr.write(
             b"warning: %s ignored unexpected arguments %s\n"
             % (cmd, b",".join(others))
         )
     return opts
 
 
 def bundle1allowed(repo, action):
     """Whether a bundle1 operation is allowed from the server.
 
     Priority is:
 
     1. server.bundle1gd.<action> (if generaldelta active)
     2. server.bundle1.<action>
     3. server.bundle1gd (if generaldelta active)
     4. server.bundle1
     """
     ui = repo.ui
     gd = requirementsmod.GENERALDELTA_REQUIREMENT in repo.requirements
 
     if gd:
         v = ui.configbool(b'server', b'bundle1gd.%s' % action)
         if v is not None:
             return v
 
     v = ui.configbool(b'server', b'bundle1.%s' % action)
     if v is not None:
         return v
 
     if gd:
         v = ui.configbool(b'server', b'bundle1gd')
         if v is not None:
             return v
 
     return ui.configbool(b'server', b'bundle1')
 
 
 commands = wireprototypes.commanddict()
 
 
 def wireprotocommand(name, args=None, permission=b'push'):
     """Decorator to declare a wire protocol command.
 
     ``name`` is the name of the wire protocol command being provided.
 
     ``args`` defines the named arguments accepted by the command. It is
     a space-delimited list of argument names. ``*`` denotes a special value
     that says to accept all named arguments.
 
     ``permission`` defines the permission type needed to run this command.
     Can be ``push`` or ``pull``. These roughly map to read-write and read-only,
     respectively. Default is to assume command requires ``push`` permissions
     because otherwise commands not declaring their permissions could modify
     a repository that is supposed to be read-only.
     """
     transports = {
         k for k, v in wireprototypes.TRANSPORTS.items() if v[b'version'] == 1
     }
 
     if permission not in (b'push', b'pull'):
         raise error.ProgrammingError(
             b'invalid wire protocol permission; '
             b'got %s; expected "push" or "pull"' % permission
         )
 
     if args is None:
         args = b''
 
     if not isinstance(args, bytes):
         raise error.ProgrammingError(
             b'arguments for version 1 commands must be declared as bytes'
         )
 
     def register(func):
         if name in commands:
             raise error.ProgrammingError(
                 b'%s command already registered for version 1' % name
             )
         commands[name] = wireprototypes.commandentry(
             func, args=args, transports=transports, permission=permission
         )
 
         return func
 
     return register
 
 
 # TODO define a more appropriate permissions type to use for this.
 @wireprotocommand(b'batch', b'cmds *', permission=b'pull')
 def batch(repo, proto, cmds, others):
     unescapearg = wireprototypes.unescapebatcharg
     res = []
     for pair in cmds.split(b';'):
         op, args = pair.split(b' ', 1)
         vals = {}
         for a in args.split(b','):
             if a:
                 n, v = a.split(b'=')
                 vals[unescapearg(n)] = unescapearg(v)
         func, spec = commands[op]
 
         # Validate that client has permissions to perform this command.
         perm = commands[op].permission
         assert perm in (b'push', b'pull')
         proto.checkperm(perm)
 
         if spec:
             keys = spec.split()
             data = {}
             for k in keys:
                 if k == b'*':
                     star = {}
                     for key in vals.keys():
                         if key not in keys:
                             star[key] = vals[key]
                     data[b'*'] = star
                 else:
                     data[k] = vals[k]
             result = func(repo, proto, *[data[k] for k in keys])
         else:
             result = func(repo, proto)
         if isinstance(result, wireprototypes.ooberror):
             return result
 
         # For now, all batchable commands must return bytesresponse or
         # raw bytes (for backwards compatibility).
         assert isinstance(result, (wireprototypes.bytesresponse, bytes))
         if isinstance(result, wireprototypes.bytesresponse):
             result = result.data
         res.append(wireprototypes.escapebatcharg(result))
 
     return wireprototypes.bytesresponse(b';'.join(res))
 
 
 @wireprotocommand(b'between', b'pairs', permission=b'pull')
 def between(repo, proto, pairs):
     pairs = [wireprototypes.decodelist(p, b'-') for p in pairs.split(b" ")]
     r = []
     for b in repo.between(pairs):
         r.append(wireprototypes.encodelist(b) + b"\n")
 
     return wireprototypes.bytesresponse(b''.join(r))
 
 
 @wireprotocommand(b'branchmap', permission=b'pull')
 def branchmap(repo, proto):
     branchmap = repo.branchmap()
     heads = []
     for branch, nodes in branchmap.items():
         branchname = urlreq.quote(encoding.fromlocal(branch))
         branchnodes = wireprototypes.encodelist(nodes)
         heads.append(b'%s %s' % (branchname, branchnodes))
 
     return wireprototypes.bytesresponse(b'\n'.join(heads))
 
 
 @wireprotocommand(b'branches', b'nodes', permission=b'pull')
 def branches(repo, proto, nodes):
     nodes = wireprototypes.decodelist(nodes)
     r = []
     for b in repo.branches(nodes):
         r.append(wireprototypes.encodelist(b) + b"\n")
 
     return wireprototypes.bytesresponse(b''.join(r))
 
 
 @wireprotocommand(b'get_cached_bundle_inline', b'path', permission=b'pull')
 def get_cached_bundle_inline(repo, proto, path):
     """
     Server command to send a clonebundle to the client
     """
     if hook.hashook(repo.ui, b'pretransmit-inline-clone-bundle'):
         hook.hook(
             repo.ui,
             repo,
             b'pretransmit-inline-clone-bundle',
             throw=True,
             clonebundlepath=path,
         )
 
     bundle_dir = repo.vfs.join(bundlecaches.BUNDLE_CACHE_DIR)
     clonebundlepath = repo.vfs.join(bundle_dir, path)
     if not repo.vfs.exists(clonebundlepath):
         raise error.Abort(b'clonebundle %s does not exist' % path)
 
     clonebundles_dir = os.path.realpath(bundle_dir)
     if not os.path.realpath(clonebundlepath).startswith(clonebundles_dir):
         raise error.Abort(b'clonebundle %s is using an illegal path' % path)
 
     def generator(vfs, bundle_path):
         with vfs(bundle_path) as f:
             length = os.fstat(f.fileno())[6]
             yield util.uvarintencode(length)
             for chunk in util.filechunkiter(f):
                 yield chunk
 
     stream = generator(repo.vfs, clonebundlepath)
     return wireprototypes.streamres(gen=stream, prefer_uncompressed=True)
 
 
 @wireprotocommand(b'clonebundles', b'', permission=b'pull')
 def clonebundles(repo, proto):
+    """A legacy version of clonebundles_manifest
+
+    This version filtered out new url scheme (like peer-bundle-cache://) to
+    avoid confusion in older clients.
+    """
+    manifest_contents = bundlecaches.get_manifest(repo)
+    # Filter out peer-bundle-cache:// entries
+    modified_manifest = []
+    for line in manifest_contents.splitlines():
+        if line.startswith(bundlecaches.CLONEBUNDLESCHEME):
+            continue
+        modified_manifest.append(line)
+    return wireprototypes.bytesresponse(b'\n'.join(modified_manifest))
+
+
+@wireprotocommand(b'clonebundles_manifest', b'*', permission=b'pull')
+def clonebundles_2(repo, proto, args):
     """Server command for returning info for available bundles to seed clones.
 
     Clients will parse this response and determine what bundle to fetch.
 
     Extensions may wrap this command to filter or dynamically emit data
     depending on the request. e.g. you could advertise URLs for the closest
     data center given the client's IP address.
 
     The only filter on the server side is filtering out inline clonebundles
     in case a client does not support them.
     Otherwise, older clients would retrieve and error out on those.
     """
     manifest_contents = bundlecaches.get_manifest(repo)
-    clientcapabilities = proto.getprotocaps()
-    if b'inlineclonebundles' in clientcapabilities:
-        return wireprototypes.bytesresponse(manifest_contents)
-    modified_manifest = []
-    for line in manifest_contents.splitlines():
-        if line.startswith(bundlecaches.CLONEBUNDLESCHEME):
-            continue
-        modified_manifest.append(line)
-    return wireprototypes.bytesresponse(b'\n'.join(modified_manifest))
+    return wireprototypes.bytesresponse(manifest_contents)
 
 
 wireprotocaps = [
     b'lookup',
     b'branchmap',
     b'pushkey',
     b'known',
     b'getbundle',
     b'unbundlehash',
 ]
 
 
 def _capabilities(repo, proto):
     """return a list of capabilities for a repo
 
     This function exists to allow extensions to easily wrap capabilities
     computation
 
     - returns a lists: easy to alter
     - change done here will be propagated to both `capabilities` and `hello`
       command without any other action needed.
     """
     # copy to prevent modification of the global list
     caps = list(wireprotocaps)
 
     # Command of same name as capability isn't exposed to version 1 of
     # transports. So conditionally add it.
     if commands.commandavailable(b'changegroupsubset', proto):
         caps.append(b'changegroupsubset')
 
     if streamclone.allowservergeneration(repo):
         if repo.ui.configbool(b'server', b'preferuncompressed'):
             caps.append(b'stream-preferred')
         requiredformats = streamclone.streamed_requirements(repo)
         # if our local revlogs are just revlogv1, add 'stream' cap
         if not requiredformats - {requirementsmod.REVLOGV1_REQUIREMENT}:
             caps.append(b'stream')
         # otherwise, add 'streamreqs' detailing our local revlog format
         else:
             caps.append(b'streamreqs=%s' % b','.join(sorted(requiredformats)))
     if repo.ui.configbool(b'experimental', b'bundle2-advertise'):
         capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role=b'server'))
         caps.append(b'bundle2=' + urlreq.quote(capsblob))
     caps.append(b'unbundle=%s' % b','.join(bundle2.bundlepriority))
 
     if repo.ui.configbool(b'experimental', b'narrow'):
         caps.append(wireprototypes.NARROWCAP)
         if repo.ui.configbool(b'experimental', b'narrowservebrokenellipses'):
             caps.append(wireprototypes.ELLIPSESCAP)
 
     return proto.addcapabilities(repo, caps)
 
 
 # If you are writing an extension and consider wrapping this function. Wrap
 # `_capabilities` instead.
 @wireprotocommand(b'capabilities', permission=b'pull')
 def capabilities(repo, proto):
     caps = _capabilities(repo, proto)
     return wireprototypes.bytesresponse(b' '.join(sorted(caps)))
 
 
 @wireprotocommand(b'changegroup', b'roots', permission=b'pull')
 def changegroup(repo, proto, roots):
     nodes = wireprototypes.decodelist(roots)
     outgoing = discovery.outgoing(
         repo, missingroots=nodes, ancestorsof=repo.heads()
     )
     cg = changegroupmod.makechangegroup(repo, outgoing, b'01', b'serve')
     gen = iter(lambda: cg.read(32768), b'')
     return wireprototypes.streamres(gen=gen)
 
 
 @wireprotocommand(b'changegroupsubset', b'bases heads', permission=b'pull')
 def changegroupsubset(repo, proto, bases, heads):
     bases = wireprototypes.decodelist(bases)
     heads = wireprototypes.decodelist(heads)
     outgoing = discovery.outgoing(repo, missingroots=bases, ancestorsof=heads)
     cg = changegroupmod.makechangegroup(repo, outgoing, b'01', b'serve')
     gen = iter(lambda: cg.read(32768), b'')
     return wireprototypes.streamres(gen=gen)
 
 
 @wireprotocommand(b'debugwireargs', b'one two *', permission=b'pull')
 def debugwireargs(repo, proto, one, two, others):
     # only accept optional args from the known set
     opts = options(b'debugwireargs', [b'three', b'four'], others)
     return wireprototypes.bytesresponse(
         repo.debugwireargs(one, two, **pycompat.strkwargs(opts))
     )
 
 
 def find_pullbundle(repo, proto, opts, clheads, heads, common):
     """Return a file object for the first matching pullbundle.
 
     Pullbundles are specified in .hg/pullbundles.manifest similar to
     clonebundles.
     For each entry, the bundle specification is checked for compatibility:
     - Client features vs the BUNDLESPEC.
     - Revisions shared with the clients vs base revisions of the bundle.
       A bundle can be applied only if all its base revisions are known by
       the client.
     - At least one leaf of the bundle's DAG is missing on the client.
     - Every leaf of the bundle's DAG is part of node set the client wants.
       E.g. do not send a bundle of all changes if the client wants only
       one specific branch of many.
     """
 
     def decodehexstring(s):
         return {binascii.unhexlify(h) for h in s.split(b';')}
 
     manifest = repo.vfs.tryread(b'pullbundles.manifest')
     if not manifest:
         return None
     res = bundlecaches.parseclonebundlesmanifest(repo, manifest)
     res = bundlecaches.filterclonebundleentries(repo, res, pullbundles=True)
     if not res:
         return None
     cl = repo.unfiltered().changelog
     heads_anc = cl.ancestors([cl.rev(rev) for rev in heads], inclusive=True)
     common_anc = cl.ancestors([cl.rev(rev) for rev in common], inclusive=True)
     compformats = clientcompressionsupport(proto)
     for entry in res:
         comp = entry.get(b'COMPRESSION')
         altcomp = util.compengines._bundlenames.get(comp)
         if comp and comp not in compformats and altcomp not in compformats:
             continue
         # No test yet for VERSION, since V2 is supported by any client
         # that advertises partial pulls
         if b'heads' in entry:
             try:
                 bundle_heads = decodehexstring(entry[b'heads'])
             except TypeError:
                 # Bad heads entry
                 continue
             if bundle_heads.issubset(common):
                 continue  # Nothing new
             if all(cl.rev(rev) in common_anc for rev in bundle_heads):
                 continue  # Still nothing new
             if any(
                 cl.rev(rev) not in heads_anc and cl.rev(rev) not in common_anc
                 for rev in bundle_heads
             ):
                 continue
         if b'bases' in entry:
             try:
                 bundle_bases = decodehexstring(entry[b'bases'])
             except TypeError:
                 # Bad bases entry
                 continue
             if not all(cl.rev(rev) in common_anc for rev in bundle_bases):
                 continue
         path = entry[b'URL']
         repo.ui.debug(b'sending pullbundle "%s"\n' % path)
         try:
             return repo.vfs.open(path)
         except IOError:
             repo.ui.debug(b'pullbundle "%s" not accessible\n' % path)
             continue
     return None
 
 
 @wireprotocommand(b'getbundle', b'*', permission=b'pull')
 def getbundle(repo, proto, others):
     opts = options(
         b'getbundle', wireprototypes.GETBUNDLE_ARGUMENTS.keys(), others
     )
     for k, v in opts.items():
         keytype = wireprototypes.GETBUNDLE_ARGUMENTS[k]
         if keytype == b'nodes':
             opts[k] = wireprototypes.decodelist(v)
         elif keytype == b'csv':
             opts[k] = list(v.split(b','))
         elif keytype == b'scsv':
             opts[k] = set(v.split(b','))
         elif keytype == b'boolean':
             # Client should serialize False as '0', which is a non-empty string
             # so it evaluates as a True bool.
             if v == b'0':
                 opts[k] = False
             else:
                 opts[k] = bool(v)
         elif keytype != b'plain':
             raise KeyError(b'unknown getbundle option type %s' % keytype)
 
     if not bundle1allowed(repo, b'pull'):
         if not exchange.bundle2requested(opts.get(b'bundlecaps')):
             if proto.name == b'http-v1':
                 return wireprototypes.ooberror(bundle2required)
             raise error.Abort(bundle2requiredmain, hint=bundle2requiredhint)
 
     try:
         clheads = set(repo.changelog.heads())
         heads = set(opts.get(b'heads', set()))
         common = set(opts.get(b'common', set()))
         common.discard(repo.nullid)
         if (
             repo.ui.configbool(b'server', b'pullbundle')
             and b'partial-pull' in proto.getprotocaps()
         ):
             # Check if a pre-built bundle covers this request.
             bundle = find_pullbundle(repo, proto, opts, clheads, heads, common)
             if bundle:
                 return wireprototypes.streamres(
                     gen=util.filechunkiter(bundle), prefer_uncompressed=True
                 )
 
         if repo.ui.configbool(b'server', b'disablefullbundle'):
             # Check to see if this is a full clone.
             changegroup = opts.get(b'cg', True)
             if changegroup and not common and clheads == heads:
                 raise error.Abort(
                     _(b'server has pull-based clones disabled'),
                     hint=_(b'remove --pull if specified or upgrade Mercurial'),
                 )
 
         info, chunks = exchange.getbundlechunks(
             repo, b'serve', **pycompat.strkwargs(opts)
         )
         prefercompressed = info.get(b'prefercompressed', True)
     except error.Abort as exc:
         # cleanly forward Abort error to the client
         if not exchange.bundle2requested(opts.get(b'bundlecaps')):
             if proto.name == b'http-v1':
                 return wireprototypes.ooberror(exc.message + b'\n')
             raise  # cannot do better for bundle1 + ssh
         # bundle2 request expect a bundle2 reply
         bundler = bundle2.bundle20(repo.ui)
         manargs = [(b'message', exc.message)]
         advargs = []
         if exc.hint is not None:
             advargs.append((b'hint', exc.hint))
         bundler.addpart(bundle2.bundlepart(b'error:abort', manargs, advargs))
         chunks = bundler.getchunks()
         prefercompressed = False
 
     return wireprototypes.streamres(
         gen=chunks, prefer_uncompressed=not prefercompressed
     )
 
 
 @wireprotocommand(b'heads', permission=b'pull')
 def heads(repo, proto):
     h = repo.heads()
     return wireprototypes.bytesresponse(wireprototypes.encodelist(h) + b'\n')
 
 
 @wireprotocommand(b'hello', permission=b'pull')
 def hello(repo, proto):
     """Called as part of SSH handshake to obtain server info.
 
     Returns a list of lines describing interesting things about the
     server, in an RFC822-like format.
 
     Currently, the only one defined is ``capabilities``, which consists of a
     line of space separated tokens describing server abilities:
 
         capabilities: <token0> <token1> <token2>
     """
     caps = capabilities(repo, proto).data
     return wireprototypes.bytesresponse(b'capabilities: %s\n' % caps)
 
 
 @wireprotocommand(b'listkeys', b'namespace', permission=b'pull')
 def listkeys(repo, proto, namespace):
     d = sorted(repo.listkeys(encoding.tolocal(namespace)).items())
     return wireprototypes.bytesresponse(pushkeymod.encodekeys(d))
 
 
 @wireprotocommand(b'lookup', b'key', permission=b'pull')
 def lookup(repo, proto, key):
     try:
         k = encoding.tolocal(key)
         n = repo.lookup(k)
         r = hex(n)
         success = 1
     except Exception as inst:
         r = stringutil.forcebytestr(inst)
         success = 0
     return wireprototypes.bytesresponse(b'%d %s\n' % (success, r))
 
 
 @wireprotocommand(b'known', b'nodes *', permission=b'pull')
 def known(repo, proto, nodes, others):
     v = b''.join(
         b and b'1' or b'0' for b in repo.known(wireprototypes.decodelist(nodes))
     )
     return wireprototypes.bytesresponse(v)
 
 
 @wireprotocommand(b'protocaps', b'caps', permission=b'pull')
 def protocaps(repo, proto, caps):
     if proto.name == wireprototypes.SSHV1:
         proto._protocaps = set(caps.split(b' '))
     return wireprototypes.bytesresponse(b'OK')
 
 
 @wireprotocommand(b'pushkey', b'namespace key old new', permission=b'push')
 def pushkey(repo, proto, namespace, key, old, new):
     # compatibility with pre-1.8 clients which were accidentally
     # sending raw binary nodes rather than utf-8-encoded hex
     if len(new) == 20 and stringutil.escapestr(new) != new:
         # looks like it could be a binary node
         try:
             new.decode('utf-8')
             new = encoding.tolocal(new)  # but cleanly decodes as UTF-8
         except UnicodeDecodeError:
             pass  # binary, leave unmodified
     else:
         new = encoding.tolocal(new)  # normal path
 
     with proto.mayberedirectstdio() as output:
         r = (
             repo.pushkey(
                 encoding.tolocal(namespace),
                 encoding.tolocal(key),
                 encoding.tolocal(old),
                 new,
             )
             or False
         )
 
     output = output.getvalue() if output else b''
     return wireprototypes.bytesresponse(b'%d\n%s' % (int(r), output))
 
 
 @wireprotocommand(b'stream_out', permission=b'pull')
 def stream(repo, proto):
     """If the server supports streaming clone, it advertises the "stream"
     capability with a value representing the version and flags of the repo
     it is serving. Client checks to see if it understands the format.
     """
     return wireprototypes.streamreslegacy(streamclone.generatev1wireproto(repo))
 
 
 @wireprotocommand(b'unbundle', b'heads', permission=b'push')
 def unbundle(repo, proto, heads):
     their_heads = wireprototypes.decodelist(heads)
 
     with proto.mayberedirectstdio() as output:
         try:
             exchange.check_heads(repo, their_heads, b'preparing changes')
             cleanup = lambda: None
             try:
                 payload = proto.getpayload()
                 if repo.ui.configbool(b'server', b'streamunbundle'):
 
                     def cleanup():
                         # Ensure that the full payload is consumed, so
                         # that the connection doesn't contain trailing garbage.
                         for p in payload:
                             pass
 
                     fp = util.chunkbuffer(payload)
                 else:
                     # write bundle data to temporary file as it can be big
                     fp, tempname = None, None
 
                     def cleanup():
                         if fp:
                             fp.close()
                         if tempname:
                             os.unlink(tempname)
 
                     fd, tempname = pycompat.mkstemp(prefix=b'hg-unbundle-')
                     repo.ui.debug(
                         b'redirecting incoming bundle to %s\n' % tempname
                     )
                     fp = os.fdopen(fd, pycompat.sysstr(b'wb+'))
                     for p in payload:
                         fp.write(p)
                     fp.seek(0)
 
                 gen = exchange.readbundle(repo.ui, fp, None)
                 if isinstance(
                     gen, changegroupmod.cg1unpacker
                 ) and not bundle1allowed(repo, b'push'):
                     if proto.name == b'http-v1':
                         # need to special case http because stderr do not get to
                         # the http client on failed push so we need to abuse
                         # some other error type to make sure the message get to
                         # the user.
                         return wireprototypes.ooberror(bundle2required)
                     raise error.Abort(
                         bundle2requiredmain, hint=bundle2requiredhint
                     )
 
                 r = exchange.unbundle(
                     repo, gen, their_heads, b'serve', proto.client()
                 )
                 if util.safehasattr(r, 'addpart'):
                     # The return looks streamable, we are in the bundle2 case
                     # and should return a stream.
                     return wireprototypes.streamreslegacy(gen=r.getchunks())
                 return wireprototypes.pushres(
                     r, output.getvalue() if output else b''
                 )
 
             finally:
                 cleanup()
 
         except (error.BundleValueError, error.Abort, error.PushRaced) as exc:
             # handle non-bundle2 case first
             if not getattr(exc, 'duringunbundle2', False):
                 try:
                     raise
                 except error.Abort as exc:
                     # The old code we moved used procutil.stderr directly.
                     # We did not change it to minimise code change.
                     # This need to be moved to something proper.
                     # Feel free to do it.
                     procutil.stderr.write(exc.format())
                     procutil.stderr.flush()
                     return wireprototypes.pushres(
                         0, output.getvalue() if output else b''
                     )
                 except error.PushRaced:
                     return wireprototypes.pusherr(
                         pycompat.bytestr(exc),
                         output.getvalue() if output else b'',
                     )
 
             bundler = bundle2.bundle20(repo.ui)
             for out in getattr(exc, '_bundle2salvagedoutput', ()):
                 bundler.addpart(out)
             try:
                 try:
                     raise
                 except error.PushkeyFailed as exc:
                     # check client caps
                     remotecaps = getattr(exc, '_replycaps', None)
                     if (
                         remotecaps is not None
                         and b'pushkey' not in remotecaps.get(b'error', ())
                     ):
                         # no support remote side, fallback to Abort handler.
                         raise
                     part = bundler.newpart(b'error:pushkey')
                     part.addparam(b'in-reply-to', exc.partid)
                     if exc.namespace is not None:
                         part.addparam(
                             b'namespace', exc.namespace, mandatory=False
                         )
                     if exc.key is not None:
                         part.addparam(b'key', exc.key, mandatory=False)
                     if exc.new is not None:
                         part.addparam(b'new', exc.new, mandatory=False)
                     if exc.old is not None:
                         part.addparam(b'old', exc.old, mandatory=False)
                     if exc.ret is not None:
                         part.addparam(b'ret', exc.ret, mandatory=False)
             except error.BundleValueError as exc:
                 errpart = bundler.newpart(b'error:unsupportedcontent')
                 if exc.parttype is not None:
                     errpart.addparam(b'parttype', exc.parttype)
                 if exc.params:
                     errpart.addparam(b'params', b'\0'.join(exc.params))
             except error.Abort as exc:
                 manargs = [(b'message', exc.message)]
                 advargs = []
                 if exc.hint is not None:
                     advargs.append((b'hint', exc.hint))
                 bundler.addpart(
                     bundle2.bundlepart(b'error:abort', manargs, advargs)
                 )
             except error.PushRaced as exc:
                 bundler.newpart(
                     b'error:pushraced',
                     [(b'message', stringutil.forcebytestr(exc))],
                 )
             return wireprototypes.streamreslegacy(gen=bundler.getchunks())

tests/test-clonebundles.t

@@ -1,838 +1,837 b''
 #require no-reposimplestore no-chg
 
 Set up a server
 
   $ hg init server
   $ cd server
   $ cat >> .hg/hgrc << EOF
   > [extensions]
   > clonebundles =
   > EOF
 
   $ touch foo
   $ hg -q commit -A -m 'add foo'
   $ touch bar
   $ hg -q commit -A -m 'add bar'
 
   $ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
   $ cat hg.pid >> $DAEMON_PIDS
   $ cd ..
 
 Missing manifest should not result in server lookup
 
   $ hg --verbose clone -U http://localhost:$HGPORT no-manifest
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
   (sent 3 HTTP requests and * bytes; received * bytes in responses) (glob)
 
   $ cat server/access.log
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
   $LOCALIP - - [$LOGDATE$] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
   $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&$USUAL_BUNDLE_CAPS$&cg=1&common=0000000000000000000000000000000000000000&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob)
 
 Empty manifest file results in retrieval
 (the extension only checks if the manifest file exists)
 
   $ touch server/.hg/clonebundles.manifest
   $ hg --verbose clone -U http://localhost:$HGPORT empty-manifest
   no clone bundles available on remote; falling back to regular clone
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
   (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
 
 Manifest file with invalid URL aborts
 
   $ echo 'http://does.not.exist/bundle.hg' > server/.hg/clonebundles.manifest
   $ hg clone http://localhost:$HGPORT 404-url
   applying clone bundle from http://does.not.exist/bundle.hg
   error fetching bundle: (.* not known|(\[Errno -?\d+] )?([Nn]o address associated with (host)?name|Temporary failure in name resolution|Name does not resolve)) (re) (no-windows !)
   error fetching bundle: [Errno 1100*] getaddrinfo failed (glob) (windows !)
   abort: error applying bundle
   (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
   [255]
 
 Manifest file with URL with unknown scheme skips the URL
   $ echo 'weirdscheme://does.not.exist/bundle.hg' > server/.hg/clonebundles.manifest
   $ hg clone http://localhost:$HGPORT unknown-scheme
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
   updating to branch default
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 Server is not running aborts
 
   $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
   $ hg clone http://localhost:$HGPORT server-not-runner
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
   error fetching bundle: (.* refused.*|Protocol not supported|(.* )?\$EADDRNOTAVAIL\$|.* No route to host) (re)
   abort: error applying bundle
   (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
   [255]
 
 Server returns 404
 
   $ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
   $ cat http.pid >> $DAEMON_PIDS
   $ hg clone http://localhost:$HGPORT running-404
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
   HTTP error fetching bundle: HTTP Error 404: File not found
   abort: error applying bundle
   (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
   [255]
 
 We can override failure to fall back to regular clone
 
   $ hg --config ui.clonebundlefallback=true clone -U http://localhost:$HGPORT 404-fallback
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
   HTTP error fetching bundle: HTTP Error 404: File not found
   falling back to normal clone
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
 
 Bundle with partial content works
 
   $ hg -R server bundle --type gzip-v1 --base null -r 53245c60e682 partial.hg
   1 changesets found
 
 We verify exact bundle content as an extra check against accidental future
 changes. If this output changes, we could break old clients.
 
   $ f --size --hexdump partial.hg
   partial.hg: size=207
   0000: 48 47 31 30 47 5a 78 9c 63 60 60 98 17 ac 12 93 |HG10GZx.c``.....|
   0010: f0 ac a9 23 45 70 cb bf 0d 5f 59 4e 4a 7f 79 21 |...#Ep..._YNJ.y!|
   0020: 9b cc 40 24 20 a0 d7 ce 2c d1 38 25 cd 24 25 d5 |..@$ ...,.8%.$%.|
   0030: d8 c2 22 cd 38 d9 24 cd 22 d5 c8 22 cd 24 cd 32 |..".8.$."..".$.2|
   0040: d1 c2 d0 c4 c8 d2 32 d1 38 39 29 c9 34 cd d4 80 |......2.89).4...|
   0050: ab 24 b5 b8 84 cb 40 c1 80 2b 2d 3f 9f 8b 2b 31 |.$....@..+-?..+1|
   0060: 25 45 01 c8 80 9a d2 9b 65 fb e5 9e 45 bf 8d 7f |%E......e...E...|
   0070: 9f c6 97 9f 2b 44 34 67 d9 ec 8e 0f a0 92 0b 75 |....+D4g.......u|
   0080: 41 d6 24 59 18 a4 a4 9a a6 18 1a 5b 98 9b 5a 98 |A.$Y.......[..Z.|
   0090: 9a 18 26 9b a6 19 98 1a 99 99 26 a6 18 9a 98 24 |..&.......&....$|
   00a0: 26 59 a6 25 5a 98 a5 18 a6 24 71 41 35 b1 43 dc |&Y.%Z....$qA5.C.|
   00b0: 16 b2 83 f7 e9 45 8b d2 56 c7 a3 1f 82 52 d7 8a |.....E..V....R..|
   00c0: 78 ed fc d5 76 f1 36 35 dc 05 00 36 ed 5e c7    |x...v.65...6.^.|
 
   $ echo "http://localhost:$HGPORT1/partial.hg" > server/.hg/clonebundles.manifest
   $ hg clone -U http://localhost:$HGPORT partial-bundle
   applying clone bundle from http://localhost:$HGPORT1/partial.hg
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   finished applying clone bundle
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   new changesets aaff8d2ffbbf
   1 local changesets published
 
 Incremental pull doesn't fetch bundle
 
   $ hg clone -r 53245c60e682 -U http://localhost:$HGPORT partial-clone
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   new changesets 53245c60e682
 
   $ cd partial-clone
   $ hg pull
   pulling from http://localhost:$HGPORT/
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   new changesets aaff8d2ffbbf
   (run 'hg update' to get a working copy)
   $ cd ..
 
 Bundle with full content works
 
   $ hg -R server bundle --type gzip-v2 --base null -r tip full.hg
   2 changesets found
 
 Again, we perform an extra check against bundle content changes. If this content
 changes, clone bundles produced by new Mercurial versions may not be readable
 by old clients.
 
   $ f --size --hexdump full.hg
   full.hg: size=442
   0000: 48 47 32 30 00 00 00 0e 43 6f 6d 70 72 65 73 73 |HG20....Compress|
   0010: 69 6f 6e 3d 47 5a 78 9c 63 60 60 d0 e4 76 f6 70 |ion=GZx.c``..v.p|
   0020: f4 73 77 75 0f f2 0f 0d 60 00 02 46 46 76 26 4e |.swu....`..FFv&N|
   0030: c6 b2 d4 a2 e2 cc fc 3c 03 a3 bc a4 e4 8c c4 bc |.......<........|
   0040: f4 d4 62 23 06 06 e6 19 40 f9 4d c1 2a 31 09 cf |..b#....@.M.*1..|
   0050: 9a 3a 52 04 b7 fc db f0 95 e5 a4 f4 97 17 b2 c9 |.:R.............|
   0060: 0c 14 00 02 e6 d9 99 25 1a a7 a4 99 a4 a4 1a 5b |.......%.......[|
   0070: 58 a4 19 27 9b a4 59 a4 1a 59 a4 99 a4 59 26 5a |X..'..Y..Y...Y&Z|
   0080: 18 9a 18 59 5a 26 1a 27 27 25 99 a6 99 1a 70 95 |...YZ&.''%....p.|
   0090: a4 16 97 70 19 28 18 70 a5 e5 e7 73 71 25 a6 a4 |...p.(.p...sq%..|
   00a0: 28 00 19 20 17 af fa df ab ff 7b 3f fb 92 dc 8b |(.. ......{?....|
   00b0: 1f 62 bb 9e b7 d7 d9 87 3d 5a 44 89 2f b0 99 87 |.b......=ZD./...|
   00c0: ec e2 54 63 43 e3 b4 64 43 73 23 33 43 53 0b 63 |..TcC..dCs#3CS.c|
   00d0: d3 14 23 03 a0 fb 2c 2c 0c d3 80 1e 30 49 49 b1 |..#...,,....0II.|
   00e0: 4c 4a 32 48 33 30 b0 34 42 b8 38 29 b1 08 e2 62 |LJ2H30.4B.8)...b|
   00f0: 20 03 6a ca c2 2c db 2f f7 2c fa 6d fc fb 34 be | .j..,./.,.m..4.|
   0100: fc 5c 21 a2 39 cb 66 77 7c 00 0d c3 59 17 14 58 |.\!.9.fw|...Y..X|
   0110: 49 16 06 29 a9 a6 29 86 c6 16 e6 a6 16 a6 26 86 |I..)..).......&.|
   0120: c9 a6 69 06 a6 46 66 a6 89 29 86 26 26 89 49 96 |..i..Ff..).&&.I.|
   0130: 69 89 16 66 29 86 29 49 5c 20 07 3e 16 fe 23 ae |i..f).)I\ .>..#.|
   0140: 26 da 1c ab 10 1f d1 f8 e3 b3 ef cd dd fc 0c 93 |&...............|
   0150: 88 75 34 36 75 04 82 55 17 14 36 a4 38 10 04 d8 |.u46u..U..6.8...|
   0160: 21 01 9a b1 83 f7 e9 45 8b d2 56 c7 a3 1f 82 52 |!......E..V....R|
   0170: d7 8a 78 ed fc d5 76 f1 36 25 81 89 c7 ad ec 90 |..x...v.6%......|
   0180: 54 47 75 2b 89 48 b1 b2 62 c9 89 c9 19 a9 56 45 |TGu+.H..b.....VE|
   0190: a9 65 ba 49 45 89 79 c9 19 ba 60 01 a0 14 23 58 |.e.IE.y...`...#X|
   01a0: 81 35 c8 7d 40 cc 04 e2 a4 a4 a6 25 96 e6 94 60 |.5.}@......%...`|
   01b0: 33 17 5f 54 00 00 d3 1b 0d 4c                   |3._T.....L|
 
   $ echo "http://localhost:$HGPORT1/full.hg" > server/.hg/clonebundles.manifest
   $ hg clone -U http://localhost:$HGPORT full-bundle
   applying clone bundle from http://localhost:$HGPORT1/full.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Feature works over SSH
 
   $ hg clone -U ssh://user@dummy/server ssh-full-clone
   applying clone bundle from http://localhost:$HGPORT1/full.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Inline bundle
 =============
 
 Checking bundle retrieved over the wireprotocol
 
 Feature works over SSH with inline bundle
 -----------------------------------------
 
   $ mkdir server/.hg/bundle-cache/
   $ cp full.hg server/.hg/bundle-cache/
   $ echo "peer-bundle-cache://full.hg" > server/.hg/clonebundles.manifest
   $ hg clone -U ssh://user@dummy/server ssh-inline-clone
   applying clone bundle from peer-bundle-cache://full.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 HTTP Supports
 -------------
 
-Or lack of it actually
-
-Feature does not use inline bundle over HTTP(S) because there is no protocaps support
-(so no way for the client to announce that it supports inline clonebundles)
   $ hg clone -U http://localhost:$HGPORT http-inline-clone
-  requesting all changes
+  applying clone bundle from peer-bundle-cache://full.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
-  new changesets 53245c60e682:aaff8d2ffbbf
+  finished applying clone bundle
+  searching for changes
+  no changes found
+  2 local changesets published
 
 Pre-transmit Hook
 -----------------
 
 Hooks work with inline bundle
 
   $ cp server/.hg/hgrc server/.hg/hgrc-beforeinlinehooks
   $ echo "[hooks]" >> server/.hg/hgrc
   $ echo "pretransmit-inline-clone-bundle=echo foo" >> server/.hg/hgrc
   $ hg clone -U ssh://user@dummy/server ssh-inline-clone-hook
   applying clone bundle from peer-bundle-cache://full.hg
   remote: foo
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Hooks can make an inline bundle fail
 
   $ cp server/.hg/hgrc-beforeinlinehooks server/.hg/hgrc
   $ echo "[hooks]" >> server/.hg/hgrc
   $ echo "pretransmit-inline-clone-bundle=echo bar && false" >> server/.hg/hgrc
   $ hg clone -U ssh://user@dummy/server ssh-inline-clone-hook-fail
   applying clone bundle from peer-bundle-cache://full.hg
   remote: bar
   remote: abort: pretransmit-inline-clone-bundle hook exited with status 1
   abort: stream ended unexpectedly (got 0 bytes, expected 1)
   [255]
   $ cp server/.hg/hgrc-beforeinlinehooks server/.hg/hgrc
 
 Other tests
 ===========
 
 Entry with unknown BUNDLESPEC is filtered and not used
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://bad.entry1 BUNDLESPEC=UNKNOWN
   > http://bad.entry2 BUNDLESPEC=xz-v1
   > http://bad.entry3 BUNDLESPEC=none-v100
   > http://localhost:$HGPORT1/full.hg BUNDLESPEC=gzip-v2
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT filter-unknown-type
   applying clone bundle from http://localhost:$HGPORT1/full.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Automatic fallback when all entries are filtered
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://bad.entry BUNDLESPEC=UNKNOWN
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT filter-all
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
 
 We require a Python version that supports SNI. Therefore, URLs requiring SNI
 are not filtered.
 
   $ cp full.hg sni.hg
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/sni.hg REQUIRESNI=true
   > http://localhost:$HGPORT1/full.hg
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT sni-supported
   applying clone bundle from http://localhost:$HGPORT1/sni.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Stream clone bundles are supported
 
   $ hg -R server debugcreatestreamclonebundle packed.hg
   writing 613 bytes for 4 files
   bundle requirements: generaldelta, revlogv1, sparserevlog (no-rust no-zstd !)
   bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, sparserevlog (no-rust zstd !)
   bundle requirements: generaldelta, revlog-compression-zstd, revlogv1, sparserevlog (rust !)
 
 No bundle spec should work
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/packed.hg
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT stream-clone-no-spec
   applying clone bundle from http://localhost:$HGPORT1/packed.hg
   4 files to transfer, 613 bytes of data
   transferred 613 bytes in *.* seconds (*) (glob)
   finished applying clone bundle
   searching for changes
   no changes found
 
 Bundle spec without parameters should work
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT stream-clone-vanilla-spec
   applying clone bundle from http://localhost:$HGPORT1/packed.hg
   4 files to transfer, 613 bytes of data
   transferred 613 bytes in *.* seconds (*) (glob)
   finished applying clone bundle
   searching for changes
   no changes found
 
 Bundle spec with format requirements should work
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT stream-clone-supported-requirements
   applying clone bundle from http://localhost:$HGPORT1/packed.hg
   4 files to transfer, 613 bytes of data
   transferred 613 bytes in *.* seconds (*) (glob)
   finished applying clone bundle
   searching for changes
   no changes found
 
 Stream bundle spec with unknown requirements should be filtered out
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv42
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT stream-clone-unsupported-requirements
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
 
 Set up manifest for testing preferences
 (Remember, the TYPE does not have to match reality - the URL is
 important)
 
   $ cp full.hg gz-a.hg
   $ cp full.hg gz-b.hg
   $ cp full.hg bz2-a.hg
   $ cp full.hg bz2-b.hg
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 extra=a
   > http://localhost:$HGPORT1/bz2-a.hg BUNDLESPEC=bzip2-v2 extra=a
   > http://localhost:$HGPORT1/gz-b.hg BUNDLESPEC=gzip-v2 extra=b
   > http://localhost:$HGPORT1/bz2-b.hg BUNDLESPEC=bzip2-v2 extra=b
   > EOF
 
 Preferring an undefined attribute will take first entry
 
   $ hg --config ui.clonebundleprefers=foo=bar clone -U http://localhost:$HGPORT prefer-foo
   applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Preferring bz2 type will download first entry of that type
 
   $ hg --config ui.clonebundleprefers=COMPRESSION=bzip2 clone -U http://localhost:$HGPORT prefer-bz
   applying clone bundle from http://localhost:$HGPORT1/bz2-a.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Preferring multiple values of an option works
 
   $ hg --config ui.clonebundleprefers=COMPRESSION=unknown,COMPRESSION=bzip2 clone -U http://localhost:$HGPORT prefer-multiple-bz
   applying clone bundle from http://localhost:$HGPORT1/bz2-a.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Sorting multiple values should get us back to original first entry
 
   $ hg --config ui.clonebundleprefers=BUNDLESPEC=unknown,BUNDLESPEC=gzip-v2,BUNDLESPEC=bzip2-v2 clone -U http://localhost:$HGPORT prefer-multiple-gz
   applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Preferring multiple attributes has correct order
 
   $ hg --config ui.clonebundleprefers=extra=b,BUNDLESPEC=bzip2-v2 clone -U http://localhost:$HGPORT prefer-separate-attributes
   applying clone bundle from http://localhost:$HGPORT1/bz2-b.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Test where attribute is missing from some entries
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
   > http://localhost:$HGPORT1/bz2-a.hg BUNDLESPEC=bzip2-v2
   > http://localhost:$HGPORT1/gz-b.hg BUNDLESPEC=gzip-v2 extra=b
   > http://localhost:$HGPORT1/bz2-b.hg BUNDLESPEC=bzip2-v2 extra=b
   > EOF
 
   $ hg --config ui.clonebundleprefers=extra=b clone -U http://localhost:$HGPORT prefer-partially-defined-attribute
   applying clone bundle from http://localhost:$HGPORT1/gz-b.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
 
 Test a bad attribute list
 
   $ hg --config ui.clonebundleprefers=bad clone -U http://localhost:$HGPORT bad-input
   abort: invalid ui.clonebundleprefers item: bad
   (each comma separated item should be key=value pairs)
   [255]
   $ hg --config ui.clonebundleprefers=key=val,bad,key2=val2 clone \
   >    -U http://localhost:$HGPORT bad-input
   abort: invalid ui.clonebundleprefers item: bad
   (each comma separated item should be key=value pairs)
   [255]
 
 
 Test interaction between clone bundles and --stream
 
 A manifest with just a gzip bundle
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
   > EOF
 
   $ hg clone -U --stream http://localhost:$HGPORT uncompressed-gzip
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   streaming all changes
   9 files to transfer, 816 bytes of data
   transferred 816 bytes in * seconds (*) (glob)
 
 A manifest with a stream clone but no BUNDLESPEC
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/packed.hg
   > EOF
 
   $ hg clone -U --stream http://localhost:$HGPORT uncompressed-no-bundlespec
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   streaming all changes
   9 files to transfer, 816 bytes of data
   transferred 816 bytes in * seconds (*) (glob)
 
 A manifest with a gzip bundle and a stream clone
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1
   > EOF
 
   $ hg clone -U --stream http://localhost:$HGPORT uncompressed-gzip-packed
   applying clone bundle from http://localhost:$HGPORT1/packed.hg
   4 files to transfer, 613 bytes of data
   transferred 613 bytes in * seconds (*) (glob)
   finished applying clone bundle
   searching for changes
   no changes found
 
 A manifest with a gzip bundle and stream clone with supported requirements
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
   > EOF
 
   $ hg clone -U --stream http://localhost:$HGPORT uncompressed-gzip-packed-requirements
   applying clone bundle from http://localhost:$HGPORT1/packed.hg
   4 files to transfer, 613 bytes of data
   transferred 613 bytes in * seconds (*) (glob)
   finished applying clone bundle
   searching for changes
   no changes found
 
 A manifest with a gzip bundle and a stream clone with unsupported requirements
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
   > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv42
   > EOF
 
   $ hg clone -U --stream http://localhost:$HGPORT uncompressed-gzip-packed-unsupported-requirements
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   streaming all changes
   9 files to transfer, 816 bytes of data
   transferred 816 bytes in * seconds (*) (glob)
 
 Test clone bundle retrieved through bundle2
 
   $ cat << EOF >> $HGRCPATH
   > [extensions]
   > largefiles=
   > EOF
   $ killdaemons.py
   $ hg -R server serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
   $ cat hg.pid >> $DAEMON_PIDS
 
   $ hg -R server debuglfput gz-a.hg
   1f74b3d08286b9b3a16fb3fa185dd29219cbc6ae
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > largefile://1f74b3d08286b9b3a16fb3fa185dd29219cbc6ae BUNDLESPEC=gzip-v2
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT largefile-provided --traceback
   applying clone bundle from largefile://1f74b3d08286b9b3a16fb3fa185dd29219cbc6ae
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   searching for changes
   no changes found
   2 local changesets published
   $ killdaemons.py
 
 A manifest with a gzip bundle requiring too much memory for a 16MB system and working
 on a 32MB system.
 
   $ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
   $ cat http.pid >> $DAEMON_PIDS
   $ hg -R server serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
   $ cat hg.pid >> $DAEMON_PIDS
 
   $ cat > server/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 REQUIREDRAM=12MB
   > EOF
 
   $ hg clone -U --debug --config ui.available-memory=16MB http://localhost:$HGPORT gzip-too-large
   using http://localhost:$HGPORT/
   sending capabilities command
-  sending clonebundles command
+  sending clonebundles_manifest command
   filtering http://localhost:$HGPORT1/gz-a.hg as it needs more than 2/3 of system memory
   no compatible clone bundles available on server; falling back to regular clone
   (you may want to report this to the server operator)
   query 1; heads
   sending batch command
   requesting all changes
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
   adding changesets
   add changeset 53245c60e682
   add changeset aaff8d2ffbbf
   adding manifests
   adding file changes
   adding bar revisions
   adding foo revisions
   bundle2-input-part: total payload size 936
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 3 parts total
   checking for updated bookmarks
   updating the branch cache
   added 2 changesets with 2 changes to 2 files
   new changesets 53245c60e682:aaff8d2ffbbf
   calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
   updating the branch cache
   (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
 
   $ hg clone -U --debug --config ui.available-memory=32MB http://localhost:$HGPORT gzip-too-large2
   using http://localhost:$HGPORT/
   sending capabilities command
-  sending clonebundles command
+  sending clonebundles_manifest command
   applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
   bundle2-input-bundle: 1 params with-transaction
   bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
   adding changesets
   add changeset 53245c60e682
   add changeset aaff8d2ffbbf
   adding manifests
   adding file changes
   adding bar revisions
   adding foo revisions
   bundle2-input-part: total payload size 920
   bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
   bundle2-input-part: total payload size 59
   bundle2-input-bundle: 2 parts total
   updating the branch cache
   added 2 changesets with 2 changes to 2 files
   finished applying clone bundle
   query 1; heads
   sending batch command
   searching for changes
   all remote heads known locally
   no changes found
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
   2 local changesets published
   calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
   updating the branch cache
   (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
   $ killdaemons.py
 
 Testing a clone bundles that involves revlog splitting (issue6811)
 ==================================================================
 
   $ cat >> $HGRCPATH << EOF
   > [format]
   > revlog-compression=none
   > use-persistent-nodemap=no
   > EOF
 
   $ hg init server-revlog-split/
   $ cd server-revlog-split
   $ cat >> .hg/hgrc << EOF
   > [extensions]
   > clonebundles =
   > EOF
   $ echo foo > A
   $ hg add A
   $ hg commit -m 'initial commit'
 IMPORTANT: the revlogs must not be split
   $ ls -1 .hg/store/00manifest.*
   .hg/store/00manifest.i
   $ ls -1 .hg/store/data/_a.*
   .hg/store/data/_a.i
 
 do big enough update to split the revlogs
 
   $ $TESTDIR/seq.py 100000 > A
   $ mkdir foo
   $ cd foo
   $ touch `$TESTDIR/seq.py 10000`
   $ cd ..
   $ hg add -q foo
   $ hg commit -m 'split the manifest and one filelog'
 
 IMPORTANT: now the revlogs must be split
   $ ls -1 .hg/store/00manifest.*
   .hg/store/00manifest.d
   .hg/store/00manifest.i
   $ ls -1 .hg/store/data/_a.*
   .hg/store/data/_a.d
   .hg/store/data/_a.i
 
 Add an extra commit on top of that
 
   $ echo foo >> A
   $ hg commit -m 'one extra commit'
 
   $ cd ..
 
 Do a bundle that contains the split, but not the update
 
   $ hg bundle --exact --rev '::(default~1)' -R server-revlog-split/ --type gzip-v2 split-test.hg
   2 changesets found
 
   $ cat > server-revlog-split/.hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/split-test.hg BUNDLESPEC=gzip-v2
   > EOF
 
 start the necessary server
 
   $ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
   $ cat http.pid >> $DAEMON_PIDS
   $ hg -R server-revlog-split serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
   $ cat hg.pid >> $DAEMON_PIDS
 
 Check that clone works fine
 ===========================
 
 Here, the initial clone will trigger a revlog split (which is a bit clowny it
 itself, but whatever). The split revlogs will see additionnal data added to
 them in the subsequent pull. This should not be a problem
 
   $ hg clone http://localhost:$HGPORT revlog-split-in-the-bundle
   applying clone bundle from http://localhost:$HGPORT1/split-test.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 10002 changes to 10001 files
   finished applying clone bundle
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   new changesets e3879eaa1db7
   2 local changesets published
   updating to branch default
   10001 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 check the results
 
   $ cd revlog-split-in-the-bundle
   $ f --size .hg/store/00manifest.*
   .hg/store/00manifest.d: size=499037
   .hg/store/00manifest.i: size=192
   $ f --size .hg/store/data/_a.*
   .hg/store/data/_a.d: size=588917
   .hg/store/data/_a.i: size=192
 
 manifest should work
 
   $ hg  files -r tip | wc -l
   \s*10001 (re)
 
 file content should work
 
   $ hg  cat -r tip A | wc -l
   \s*100001 (re)
 
 

tests/test-stream-bundle-v2.t

@@ -1,204 +1,204 b''
 #require no-reposimplestore
 
 #testcases stream-v2 stream-v3
 
 #if stream-v2
   $ bundle_format="streamv2"
   $ stream_version="v2"
 #endif
 #if stream-v3
   $ bundle_format="streamv3-exp"
   $ stream_version="v3-exp"
   $ cat << EOF >> $HGRCPATH
   > [experimental]
   > stream-v3=yes
   > EOF
 #endif
 
 Test creating a consuming stream bundle v2 and v3
 
   $ getmainid() {
   >    hg -R main log --template '{node}\n' --rev "$1"
   > }
 
   $ cp $HGRCPATH $TESTTMP/hgrc.orig
 
   $ cat >> $HGRCPATH << EOF
   > [experimental]
   > evolution.createmarkers=True
   > evolution.exchange=True
   > bundle2-output-capture=True
   > [ui]
   > logtemplate={rev}:{node|short} {phase} {author} {bookmarks} {desc|firstline}
   > [web]
   > push_ssl = false
   > allow_push = *
   > [phases]
   > publish=False
   > [extensions]
   > drawdag=$TESTDIR/drawdag.py
   > clonebundles=
   > EOF
 
 The extension requires a repo (currently unused)
 
   $ hg init main
   $ cd main
 
   $ hg debugdrawdag <<'EOF'
   > E
   > |
   > D
   > |
   > C
   > |
   > B
   > |
   > A
   > EOF
 
   $ hg bundle -a --type="none-v2;stream=$stream_version" bundle.hg
   $ hg debugbundle bundle.hg
   Stream params: {}
   stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v2 no-zstd !)
   stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v2 zstd no-rust !)
   stream2 -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v2 rust !)
   stream3-exp -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v3 no-zstd !)
   stream3-exp -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v3 zstd no-rust !)
   stream3-exp -- {bytecount: 1693, filecount: 11, requirements: generaldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog} (mandatory: True) (stream-v3 rust !)
   $ hg debugbundle --spec bundle.hg
   none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlogv1%2Csparserevlog (stream-v2 no-zstd !)
   none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (stream-v2 zstd no-rust !)
   none-v2;stream=v2;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (stream-v2 rust !)
   none-v2;stream=v3-exp;requirements%3Dgeneraldelta%2Crevlogv1%2Csparserevlog (stream-v3 no-zstd !)
   none-v2;stream=v3-exp;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (stream-v3 zstd no-rust !)
   none-v2;stream=v3-exp;requirements%3Dgeneraldelta%2Crevlog-compression-zstd%2Crevlogv1%2Csparserevlog (stream-v3 rust !)
 
 Test that we can apply the bundle as a stream clone bundle
 
   $ cat > .hg/clonebundles.manifest << EOF
   > http://localhost:$HGPORT1/bundle.hg BUNDLESPEC=`hg debugbundle --spec bundle.hg`
   > EOF
 
   $ hg serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
   $ cat hg.pid >> $DAEMON_PIDS
 
   $ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
   $ cat http.pid >> $DAEMON_PIDS
 
   $ cd ..
   $ hg clone http://localhost:$HGPORT stream-clone-implicit --debug
   using http://localhost:$HGPORT/
   sending capabilities command
-  sending clonebundles command
+  sending clonebundles_manifest command
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "stream2" (params: 3 mandatory) supported (stream-v2 !)
   bundle2-input-part: "stream3-exp" (params: 3 mandatory) supported (stream-v3 !)
   applying stream bundle
   11 files to transfer, 1.65 KB of data
   starting 4 threads for background file closing (?)
   starting 4 threads for background file closing (?)
   adding [s] data/A.i (66 bytes)
   adding [s] data/B.i (66 bytes)
   adding [s] data/C.i (66 bytes)
   adding [s] data/D.i (66 bytes)
   adding [s] data/E.i (66 bytes)
   adding [s] phaseroots (43 bytes)
   adding [s] 00manifest.i (584 bytes)
   adding [s] 00changelog.i (595 bytes)
   adding [c] branch2-served (94 bytes)
   adding [c] rbc-names-v1 (7 bytes)
   adding [c] rbc-revs-v1 (40 bytes)
   transferred 1.65 KB in * seconds (* */sec) (glob)
   bundle2-input-part: total payload size 1840
   bundle2-input-bundle: 1 parts total
   updating the branch cache
   finished applying clone bundle
   query 1; heads
   sending batch command
   searching for changes
   all remote heads known locally
   no changes found
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
   updating to branch default
   resolving manifests
    branchmerge: False, force: False, partial: False
    ancestor: 000000000000, local: 000000000000+, remote: 9bc730a19041
    A: remote created -> g
   getting A
    B: remote created -> g
   getting B
    C: remote created -> g
   getting C
    D: remote created -> g
   getting D
    E: remote created -> g
   getting E
   5 files updated, 0 files merged, 0 files removed, 0 files unresolved
   updating the branch cache
   (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
 
   $ hg clone --stream http://localhost:$HGPORT stream-clone-explicit --debug
   using http://localhost:$HGPORT/
   sending capabilities command
-  sending clonebundles command
+  sending clonebundles_manifest command
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "stream2" (params: 3 mandatory) supported (stream-v2 !)
   bundle2-input-part: "stream3-exp" (params: 3 mandatory) supported (stream-v3 !)
   applying stream bundle
   11 files to transfer, 1.65 KB of data
   starting 4 threads for background file closing (?)
   starting 4 threads for background file closing (?)
   adding [s] data/A.i (66 bytes)
   adding [s] data/B.i (66 bytes)
   adding [s] data/C.i (66 bytes)
   adding [s] data/D.i (66 bytes)
   adding [s] data/E.i (66 bytes)
   adding [s] phaseroots (43 bytes)
   adding [s] 00manifest.i (584 bytes)
   adding [s] 00changelog.i (595 bytes)
   adding [c] branch2-served (94 bytes)
   adding [c] rbc-names-v1 (7 bytes)
   adding [c] rbc-revs-v1 (40 bytes)
   transferred 1.65 KB in * seconds (* */sec) (glob)
   bundle2-input-part: total payload size 1840
   bundle2-input-bundle: 1 parts total
   updating the branch cache
   finished applying clone bundle
   query 1; heads
   sending batch command
   searching for changes
   all remote heads known locally
   no changes found
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
   updating to branch default
   resolving manifests
    branchmerge: False, force: False, partial: False
    ancestor: 000000000000, local: 000000000000+, remote: 9bc730a19041
    A: remote created -> g
   getting A
    B: remote created -> g
   getting B
    C: remote created -> g
   getting C
    D: remote created -> g
   getting D
    E: remote created -> g
   getting E
   5 files updated, 0 files merged, 0 files removed, 0 files unresolved
   updating the branch cache
   (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)