clone-bundles: add an option to generate bundles in the background

This is what the "on-change" version have been doing from the start, it seems
useful to also offer this option in the command.

hgext/clonebundles.py

@@ -1,996 +1,1022 b''
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 """advertise pre-generated bundles to seed clones
 
 "clonebundles" is a server-side extension used to advertise the existence
 of pre-generated, externally hosted bundle files to clients that are
 cloning so that cloning can be faster, more reliable, and require less
 resources on the server. "pullbundles" is a related feature for sending
 pre-generated bundle files to clients as part of pull operations.
 
 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
 the server, in response to a client's request to clone, dynamically generates
 a bundle containing the entire repository content and sends it to the client.
 There is no caching on the server and the server will have to redundantly
 generate the same outgoing bundle in response to each clone request. For
 servers with large repositories or with high clone volume, the load from
 clones can make scaling the server challenging and costly.
 
 This extension provides server operators the ability to offload
 potentially expensive clone load to an external service. Pre-generated
 bundles also allow using more CPU intensive compression, reducing the
 effective bandwidth requirements.
 
 Here's how clone bundles work:
 
 1. A server operator establishes a mechanism for making bundle files available
    on a hosting service where Mercurial clients can fetch them.
 2. A manifest file listing available bundle URLs and some optional metadata
    is added to the Mercurial repository on the server.
 3. A client initiates a clone against a clone bundles aware server.
 4. The client sees the server is advertising clone bundles and fetches the
    manifest listing available bundles.
 5. The client filters and sorts the available bundles based on what it
    supports and prefers.
 6. The client downloads and applies an available bundle from the
    server-specified URL.
 7. The client reconnects to the original server and performs the equivalent
    of :hg:`pull` to retrieve all repository data not in the bundle. (The
    repository could have been updated between when the bundle was created
    and when the client started the clone.) This may use "pullbundles".
 
 Instead of the server generating full repository bundles for every clone
 request, it generates full bundles once and they are subsequently reused to
 bootstrap new clones. The server may still transfer data at clone time.
 However, this is only data that has been added/changed since the bundle was
 created. For large, established repositories, this can reduce server load for
 clones to less than 1% of original.
 
 Here's how pullbundles work:
 
 1. A manifest file listing available bundles and describing the revisions
    is added to the Mercurial repository on the server.
 2. A new-enough client informs the server that it supports partial pulls
    and initiates a pull.
 3. If the server has pull bundles enabled and sees the client advertising
    partial pulls, it checks for a matching pull bundle in the manifest.
    A bundle matches if the format is supported by the client, the client
    has the required revisions already and needs something from the bundle.
 4. If there is at least one matching bundle, the server sends it to the client.
 5. The client applies the bundle and notices that the server reply was
    incomplete. It initiates another pull.
 
 To work, this extension requires the following of server operators:
 
 * Generating bundle files of repository content (typically periodically,
   such as once per day).
 * Clone bundles: A file server that clients have network access to and that
   Python knows how to talk to through its normal URL handling facility
   (typically an HTTP/HTTPS server).
 * A process for keeping the bundles manifest in sync with available bundle
   files.
 
 Strictly speaking, using a static file hosting server isn't required: a server
 operator could use a dynamic service for retrieving bundle data. However,
 static file hosting services are simple and scalable and should be sufficient
 for most needs.
 
 Bundle files can be generated with the :hg:`bundle` command. Typically
 :hg:`bundle --all` is used to produce a bundle of the entire repository.
 
 :hg:`debugcreatestreamclonebundle` can be used to produce a special
 *streaming clonebundle*. These are bundle files that are extremely efficient
 to produce and consume (read: fast). However, they are larger than
 traditional bundle formats and require that clients support the exact set
 of repository data store formats in use by the repository that created them.
 Typically, a newer server can serve data that is compatible with older clients.
 However, *streaming clone bundles* don't have this guarantee. **Server
 operators need to be aware that newer versions of Mercurial may produce
 streaming clone bundles incompatible with older Mercurial versions.**
 
 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
 file containing the list of available bundle files suitable for seeding
 clones. If this file does not exist, the repository will not advertise the
 existence of clone bundles when clients connect. For pull bundles,
 ``.hg/pullbundles.manifest`` is used.
 
 The manifest file contains a newline (\\n) delimited list of entries.
 
 Each line in this file defines an available bundle. Lines have the format:
 
     <URL> [<key>=<value>[ <key>=<value>]]
 
 That is, a URL followed by an optional, space-delimited list of key=value
 pairs describing additional properties of this bundle. Both keys and values
 are URI encoded.
 
 For pull bundles, the URL is a path under the ``.hg`` directory of the
 repository.
 
 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
 All non-uppercase keys can be used by site installations. An example use
 for custom properties is to use the *datacenter* attribute to define which
 data center a file is hosted in. Clients could then prefer a server in the
 data center closest to them.
 
 The following reserved keys are currently defined:
 
 BUNDLESPEC
    A "bundle specification" string that describes the type of the bundle.
 
    These are string values that are accepted by the "--type" argument of
    :hg:`bundle`.
 
    The values are parsed in strict mode, which means they must be of the
    "<compression>-<type>" form. See
    mercurial.exchange.parsebundlespec() for more details.
 
    :hg:`debugbundle --spec` can be used to print the bundle specification
    string for a bundle file. The output of this command can be used verbatim
    for the value of ``BUNDLESPEC`` (it is already escaped).
 
    Clients will automatically filter out specifications that are unknown or
    unsupported so they won't attempt to download something that likely won't
    apply.
 
    The actual value doesn't impact client behavior beyond filtering:
    clients will still sniff the bundle type from the header of downloaded
    files.
 
    **Use of this key is highly recommended**, as it allows clients to
    easily skip unsupported bundles. If this key is not defined, an old
    client may attempt to apply a bundle that it is incapable of reading.
 
 REQUIRESNI
    Whether Server Name Indication (SNI) is required to connect to the URL.
    SNI allows servers to use multiple certificates on the same IP. It is
    somewhat common in CDNs and other hosting providers. Older Python
    versions do not support SNI. Defining this attribute enables clients
    with older Python versions to filter this entry without experiencing
    an opaque SSL failure at connection time.
 
    If this is defined, it is important to advertise a non-SNI fallback
    URL or clients running old Python releases may not be able to clone
    with the clonebundles facility.
 
    Value should be "true".
 
 REQUIREDRAM
    Value specifies expected memory requirements to decode the payload.
    Values can have suffixes for common bytes sizes. e.g. "64MB".
 
    This key is often used with zstd-compressed bundles using a high
    compression level / window size, which can require 100+ MB of memory
    to decode.
 
 heads
    Used for pull bundles. This contains the ``;`` separated changeset
    hashes of the heads of the bundle content.
 
 bases
    Used for pull bundles. This contains the ``;`` separated changeset
    hashes of the roots of the bundle content. This can be skipped if
    the bundle was created without ``--base``.
 
 Manifests can contain multiple entries. Assuming metadata is defined, clients
 will filter entries from the manifest that they don't support. The remaining
 entries are optionally sorted by client preferences
 (``ui.clonebundleprefers`` config option). The client then attempts
 to fetch the bundle at the first URL in the remaining list.
 
 **Errors when downloading a bundle will fail the entire clone operation:
 clients do not automatically fall back to a traditional clone.** The reason
 for this is that if a server is using clone bundles, it is probably doing so
 because the feature is necessary to help it scale. In other words, there
 is an assumption that clone load will be offloaded to another service and
 that the Mercurial server isn't responsible for serving this clone load.
 If that other service experiences issues and clients start mass falling back to
 the original Mercurial server, the added clone load could overwhelm the server
 due to unexpected load and effectively take it offline. Not having clients
 automatically fall back to cloning from the original server mitigates this
 scenario.
 
 Because there is no automatic Mercurial server fallback on failure of the
 bundle hosting service, it is important for server operators to view the bundle
 hosting service as an extension of the Mercurial server in terms of
 availability and service level agreements: if the bundle hosting service goes
 down, so does the ability for clients to clone. Note: clients will see a
 message informing them how to bypass the clone bundles facility when a failure
 occurs. So server operators should prepare for some people to follow these
 instructions when a failure occurs, thus driving more load to the original
 Mercurial server when the bundle hosting service fails.
 
 
 auto-generation of clone bundles
 --------------------------------
 
 It is possible to set Mercurial to automatically re-generate clone bundles when
 enough new content is available.
 
 Mercurial will take care of the process asynchronously. The defined list of
 bundle-type will be generated, uploaded, and advertised. Older bundles will get
 decommissioned as newer ones replace them.
 
 Bundles Generation:
 ...................
 
 The extension can generate multiple variants of the clone bundle. Each
 different variant will be defined by the "bundle-spec" they use::
 
     [clone-bundles]
     auto-generate.formats= zstd-v2, gzip-v2
 
 See `hg help bundlespec` for details about available options.
 
 By default, new bundles are generated when 5% of the repository contents or at
 least 1000 revisions are not contained in the cached bundles. This option can
 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
 
     [clone-bundles]
     trigger.below-bundled-ratio=0.95
     trigger.revs=1000
 
 This logic can be manually triggered using the `admin::clone-bundles-refresh`
 command, or automatically on each repository change if
 `clone-bundles.auto-generate.on-change` is set to `yes`.
 
     [clone-bundles]
     auto-generate.on-change=yes
     auto-generate.formats= zstd-v2, gzip-v2
 
 Bundles Upload and Serving:
 ...........................
 
 The generated bundles need to be made available to users through a "public" URL.
 This should be donne through `clone-bundles.upload-command` configuration. The
 value of this command should be a shell command. It will have access to the
 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
 basename in the "public" URL is accessible at::
 
   [clone-bundles]
   upload-command=sftp put $HGCB_BUNDLE_PATH \
       sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
 
 If the file was already uploaded, the command must still succeed.
 
 After upload, the file should be available at an url defined by
 `clone-bundles.url-template`.
 
   [clone-bundles]
   url-template=https://bundles.host/cache/clone-bundles/{basename}
 
 Old bundles cleanup:
 ....................
 
 When new bundles are generated, the older ones are no longer necessary and can
 be removed from storage. This is done through the `clone-bundles.delete-command`
 configuration. The command is given the url of the artifact to delete through
 the `$HGCB_BUNDLE_URL` environment variable.
 
   [clone-bundles]
   delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
 
 If the file was already deleted, the command must still succeed.
 """
 
 
 import os
 import weakref
 
 from mercurial.i18n import _
 
 from mercurial import (
     bundlecaches,
     commands,
     error,
     extensions,
     localrepo,
     lock,
     node,
     registrar,
     util,
     wireprotov1server,
 )
 
 
 from mercurial.utils import (
     procutil,
 )
 
 testedwith = b'ships-with-hg-core'
 
 
 def capabilities(orig, repo, proto):
     caps = orig(repo, proto)
 
     # Only advertise if a manifest exists. This does add some I/O to requests.
     # But this should be cheaper than a wasted network round trip due to
     # missing file.
     if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
         caps.append(b'clonebundles')
 
     return caps
 
 
 def extsetup(ui):
     extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities)
 
 
 # logic for bundle auto-generation
 
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
 configitem(b'clone-bundles', b'auto-generate.formats', default=list)
 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
 configitem(b'clone-bundles', b'trigger.revs', default=1000)
 
 configitem(b'clone-bundles', b'upload-command', default=None)
 
 configitem(b'clone-bundles', b'delete-command', default=None)
 
 configitem(b'clone-bundles', b'url-template', default=None)
 
 configitem(b'devel', b'debug.clonebundles', default=False)
 
 
 # category for the post-close transaction hooks
 CAT_POSTCLOSE = b"clonebundles-autobundles"
 
 # template for bundle file names
 BUNDLE_MASK = (
     b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
 )
 
 
 # file in .hg/ use to track clonebundles being auto-generated
 AUTO_GEN_FILE = b'clonebundles.auto-gen'
 
 
 class BundleBase(object):
     """represents the core of properties that matters for us in a bundle
 
     :bundle_type: the bundlespec (see hg help bundlespec)
     :revs:        the number of revisions in the repo at bundle creation time
     :tip_rev:     the rev-num of the tip revision
     :tip_node:    the node id of the tip-most revision in the bundle
 
     :ready:       True if the bundle is ready to be served
     """
 
     ready = False
 
     def __init__(self, bundle_type, revs, tip_rev, tip_node):
         self.bundle_type = bundle_type
         self.revs = revs
         self.tip_rev = tip_rev
         self.tip_node = tip_node
 
     def valid_for(self, repo):
         """is this bundle applicable to the current repository
 
         This is useful for detecting bundles made irrelevant by stripping.
         """
         tip_node = node.bin(self.tip_node)
         return repo.changelog.index.get_rev(tip_node) == self.tip_rev
 
     def __eq__(self, other):
         left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
         right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
         return left == right
 
     def __neq__(self, other):
         return not self == other
 
     def __cmp__(self, other):
         if self == other:
             return 0
         return -1
 
 
 class RequestedBundle(BundleBase):
     """A bundle that should be generated.
 
     Additional attributes compared to BundleBase
     :heads:       list of head revisions (as rev-num)
     :op_id:       a "unique" identifier for the operation triggering the change
     """
 
     def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
         self.head_revs = head_revs
         self.op_id = op_id
         super(RequestedBundle, self).__init__(
             bundle_type,
             revs,
             tip_rev,
             tip_node,
         )
 
     @property
     def suggested_filename(self):
         """A filename that can be used for the generated bundle"""
         data = {
             b'bundle_type': self.bundle_type,
             b'revs': self.revs,
             b'heads': self.head_revs,
             b'tip_rev': self.tip_rev,
             b'tip_node': self.tip_node,
             b'tip_short': self.tip_node[:12],
             b'op_id': self.op_id,
         }
         return BUNDLE_MASK % data
 
     def generate_bundle(self, repo, file_path):
         """generate the bundle at `filepath`"""
         commands.bundle(
             repo.ui,
             repo,
             file_path,
             base=[b"null"],
             rev=self.head_revs,
             type=self.bundle_type,
             quiet=True,
         )
 
     def generating(self, file_path, hostname=None, pid=None):
         """return a GeneratingBundle object from this object"""
         if pid is None:
             pid = os.getpid()
         if hostname is None:
             hostname = lock._getlockprefix()
         return GeneratingBundle(
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             hostname,
             pid,
             file_path,
         )
 
 
 class GeneratingBundle(BundleBase):
     """A bundle being generated
 
     extra attributes compared to BundleBase:
 
     :hostname: the hostname of the machine generating the bundle
     :pid:      the pid of the process generating the bundle
     :filepath: the target filename of the bundle
 
     These attributes exist to help detect stalled generation processes.
     """
 
     ready = False
 
     def __init__(
         self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
     ):
         self.hostname = hostname
         self.pid = pid
         self.filepath = filepath
         super(GeneratingBundle, self).__init__(
             bundle_type, revs, tip_rev, tip_node
         )
 
     @classmethod
     def from_line(cls, line):
         """create an object by deserializing a line from AUTO_GEN_FILE"""
         assert line.startswith(b'PENDING-v1 ')
         (
             __,
             bundle_type,
             revs,
             tip_rev,
             tip_node,
             hostname,
             pid,
             filepath,
         ) = line.split()
         hostname = util.urlreq.unquote(hostname)
         filepath = util.urlreq.unquote(filepath)
         revs = int(revs)
         tip_rev = int(tip_rev)
         pid = int(pid)
         return cls(
             bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
         )
 
     def to_line(self):
         """serialize the object to include as a line in AUTO_GEN_FILE"""
         templ = b"PENDING-v1 %s %d %d %s %s %d %s"
         data = (
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             util.urlreq.quote(self.hostname),
             self.pid,
             util.urlreq.quote(self.filepath),
         )
         return templ % data
 
     def __eq__(self, other):
         if not super(GeneratingBundle, self).__eq__(other):
             return False
         left = (self.hostname, self.pid, self.filepath)
         right = (other.hostname, other.pid, other.filepath)
         return left == right
 
     def uploaded(self, url, basename):
         """return a GeneratedBundle from this object"""
         return GeneratedBundle(
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             url,
             basename,
         )
 
 
 class GeneratedBundle(BundleBase):
     """A bundle that is done being generated and can be served
 
     extra attributes compared to BundleBase:
 
     :file_url: the url where the bundle is available.
     :basename: the "basename" used to upload (useful for deletion)
 
     These attributes exist to generate a bundle manifest
     (.hg/pullbundles.manifest)
     """
 
     ready = True
 
     def __init__(
         self, bundle_type, revs, tip_rev, tip_node, file_url, basename
     ):
         self.file_url = file_url
         self.basename = basename
         super(GeneratedBundle, self).__init__(
             bundle_type, revs, tip_rev, tip_node
         )
 
     @classmethod
     def from_line(cls, line):
         """create an object by deserializing a line from AUTO_GEN_FILE"""
         assert line.startswith(b'DONE-v1 ')
         (
             __,
             bundle_type,
             revs,
             tip_rev,
             tip_node,
             file_url,
             basename,
         ) = line.split()
         revs = int(revs)
         tip_rev = int(tip_rev)
         file_url = util.urlreq.unquote(file_url)
         return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
 
     def to_line(self):
         """serialize the object to include as a line in AUTO_GEN_FILE"""
         templ = b"DONE-v1 %s %d %d %s %s %s"
         data = (
             self.bundle_type,
             self.revs,
             self.tip_rev,
             self.tip_node,
             util.urlreq.quote(self.file_url),
             self.basename,
         )
         return templ % data
 
     def manifest_line(self):
         """serialize the object to include as a line in pullbundles.manifest"""
         templ = b"%s BUNDLESPEC=%s REQUIRESNI=true"
         return templ % (self.file_url, self.bundle_type)
 
     def __eq__(self, other):
         if not super(GeneratedBundle, self).__eq__(other):
             return False
         return self.file_url == other.file_url
 
 
 def parse_auto_gen(content):
     """parse the AUTO_GEN_FILE to return a list of Bundle object"""
     bundles = []
     for line in content.splitlines():
         if line.startswith(b'PENDING-v1 '):
             bundles.append(GeneratingBundle.from_line(line))
         elif line.startswith(b'DONE-v1 '):
             bundles.append(GeneratedBundle.from_line(line))
     return bundles
 
 
 def dumps_auto_gen(bundles):
     """serialize a list of Bundle as a AUTO_GEN_FILE content"""
     lines = []
     for b in bundles:
         lines.append(b"%s\n" % b.to_line())
     lines.sort()
     return b"".join(lines)
 
 
 def read_auto_gen(repo):
     """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
     data = repo.vfs.tryread(AUTO_GEN_FILE)
     if not data:
         return []
     return parse_auto_gen(data)
 
 
 def write_auto_gen(repo, bundles):
     """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
     assert repo._cb_lock_ref is not None
     data = dumps_auto_gen(bundles)
     with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
         f.write(data)
 
 
 def generate_manifest(bundles):
     """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
     bundles = list(bundles)
     bundles.sort(key=lambda b: b.bundle_type)
     lines = []
     for b in bundles:
         lines.append(b"%s\n" % b.manifest_line())
     return b"".join(lines)
 
 
 def update_ondisk_manifest(repo):
     """update the clonebundle manifest with latest url"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
 
         per_types = {}
         for b in bundles:
             if not (b.ready and b.valid_for(repo)):
                 continue
             current = per_types.get(b.bundle_type)
             if current is not None and current.revs >= b.revs:
                 continue
             per_types[b.bundle_type] = b
         manifest = generate_manifest(per_types.values())
         with repo.vfs(
             bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
         ) as f:
             f.write(manifest)
 
 
 def update_bundle_list(repo, new_bundles=(), del_bundles=()):
     """modify the repo's AUTO_GEN_FILE
 
     This method also regenerates the clone bundle manifest when needed"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         if del_bundles:
             bundles = [b for b in bundles if b not in del_bundles]
         new_bundles = [b for b in new_bundles if b not in bundles]
         bundles.extend(new_bundles)
         write_auto_gen(repo, bundles)
         all_changed = []
         all_changed.extend(new_bundles)
         all_changed.extend(del_bundles)
         if any(b.ready for b in all_changed):
             update_ondisk_manifest(repo)
 
 
 def cleanup_tmp_bundle(repo, target):
     """remove a GeneratingBundle file and entry"""
     assert not target.ready
     with repo.clonebundles_lock():
         repo.vfs.tryunlink(target.filepath)
         update_bundle_list(repo, del_bundles=[target])
 
 
 def finalize_one_bundle(repo, target):
     """upload a generated bundle and advertise it in the clonebundles.manifest"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         if target in bundles and target.valid_for(repo):
             result = upload_bundle(repo, target)
             update_bundle_list(repo, new_bundles=[result])
     cleanup_tmp_bundle(repo, target)
 
 
 def find_outdated_bundles(repo, bundles):
     """finds outdated bundles"""
     olds = []
     per_types = {}
     for b in bundles:
         if not b.valid_for(repo):
             olds.append(b)
             continue
         l = per_types.setdefault(b.bundle_type, [])
         l.append(b)
     for key in sorted(per_types):
         all = per_types[key]
         if len(all) > 1:
             all.sort(key=lambda b: b.revs, reverse=True)
             olds.extend(all[1:])
     return olds
 
 
 def collect_garbage(repo):
     """finds outdated bundles and get them deleted"""
     with repo.clonebundles_lock():
         bundles = read_auto_gen(repo)
         olds = find_outdated_bundles(repo, bundles)
         for o in olds:
             delete_bundle(repo, o)
         update_bundle_list(repo, del_bundles=olds)
 
 
 def upload_bundle(repo, bundle):
     """upload the result of a GeneratingBundle and return a GeneratedBundle
 
     The upload is done using the `clone-bundles.upload-command`
     """
     cmd = repo.ui.config(b'clone-bundles', b'upload-command')
     url = repo.ui.config(b'clone-bundles', b'url-template')
     basename = repo.vfs.basename(bundle.filepath)
     filepath = procutil.shellquote(bundle.filepath)
     variables = {
         b'HGCB_BUNDLE_PATH': filepath,
         b'HGCB_BUNDLE_BASENAME': basename,
     }
     env = procutil.shellenviron(environ=variables)
     ret = repo.ui.system(cmd, environ=env)
     if ret:
         raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
     url = (
         url.decode('utf8')
         .format(basename=basename.decode('utf8'))
         .encode('utf8')
     )
     return bundle.uploaded(url, basename)
 
 
 def delete_bundle(repo, bundle):
     """delete a bundle from storage"""
     assert bundle.ready
     msg = b'clone-bundles: deleting bundle %s\n'
     msg %= bundle.basename
     if repo.ui.configbool(b'devel', b'debug.clonebundles'):
         repo.ui.write(msg)
     else:
         repo.ui.debug(msg)
 
     cmd = repo.ui.config(b'clone-bundles', b'delete-command')
     variables = {
         b'HGCB_BUNDLE_URL': bundle.file_url,
         b'HGCB_BASENAME': bundle.basename,
     }
     env = procutil.shellenviron(environ=variables)
     ret = repo.ui.system(cmd, environ=env)
     if ret:
         raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
 
 
 def auto_bundle_needed_actions(repo, bundles, op_id):
     """find the list of bundles that need action
 
     returns a list of RequestedBundle objects that need to be generated and
     uploaded."""
     create_bundles = []
     delete_bundles = []
     repo = repo.filtered(b"immutable")
     targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
     ratio = float(
         repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
     )
     abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
     revs = len(repo.changelog)
     generic_data = {
         'revs': revs,
         'head_revs': repo.changelog.headrevs(),
         'tip_rev': repo.changelog.tiprev(),
         'tip_node': node.hex(repo.changelog.tip()),
         'op_id': op_id,
     }
     for t in targets:
         if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
             data = generic_data.copy()
             data['bundle_type'] = t
             b = RequestedBundle(**data)
             create_bundles.append(b)
     delete_bundles.extend(find_outdated_bundles(repo, bundles))
     return create_bundles, delete_bundles
 
 
 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
     """consider the current cached content and trigger new bundles if needed"""
     threshold = max((revs * ratio), (revs - abs_revs))
     for b in bundles:
         if not b.valid_for(repo) or b.bundle_type != bundle_type:
             continue
         if b.revs > threshold:
             return False
     return True
 
 
 def start_one_bundle(repo, bundle):
     """start the generation of a single bundle file
 
     the `bundle` argument should be a RequestedBundle object.
 
     This data is passed to the `debugmakeclonebundles` "as is".
     """
     data = util.pickle.dumps(bundle)
     cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
     env = procutil.shellenviron()
     msg = b'clone-bundles: starting bundle generation: %s\n'
     stdout = None
     stderr = None
     waits = []
     record_wait = None
     if repo.ui.configbool(b'devel', b'debug.clonebundles'):
         stdout = procutil.stdout
         stderr = procutil.stderr
         repo.ui.write(msg % bundle.bundle_type)
         record_wait = waits.append
     else:
         repo.ui.debug(msg % bundle.bundle_type)
     bg = procutil.runbgcommand
     bg(
         cmd,
         env,
         stdin_bytes=data,
         stdout=stdout,
         stderr=stderr,
         record_wait=record_wait,
     )
     for f in waits:
         f()
 
 
 INTERNAL_CMD = b'debug::internal-make-clone-bundles'
 
 
 @command(INTERNAL_CMD, [], b'')
 def debugmakeclonebundles(ui, repo):
     """Internal command to auto-generate debug bundles"""
     requested_bundle = util.pickle.load(procutil.stdin)
     procutil.stdin.close()
 
     collect_garbage(repo)
 
     fname = requested_bundle.suggested_filename
     fpath = repo.vfs.makedirs(b'tmp-bundles')
     fpath = repo.vfs.join(b'tmp-bundles', fname)
     bundle = requested_bundle.generating(fpath)
     update_bundle_list(repo, new_bundles=[bundle])
 
     requested_bundle.generate_bundle(repo, fpath)
 
     repo.invalidate()
     finalize_one_bundle(repo, bundle)
 
 
 def make_auto_bundler(source_repo):
     reporef = weakref.ref(source_repo)
 
     def autobundle(tr):
         repo = reporef()
         assert repo is not None
         bundles = read_auto_gen(repo)
         new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
         for data in new:
             start_one_bundle(repo, data)
         return None
 
     return autobundle
 
 
 def reposetup(ui, repo):
     """install the two pieces needed for automatic clonebundle generation
 
     - add a "post-close" hook that fires bundling when needed
     - introduce a clone-bundle lock to let multiple processes meddle with the
       state files.
     """
     if not repo.local():
         return
 
     class autobundlesrepo(repo.__class__):
         def transaction(self, *args, **kwargs):
             tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
             enabled = repo.ui.configbool(
                 b'clone-bundles',
                 b'auto-generate.on-change',
             )
             targets = repo.ui.configlist(
                 b'clone-bundles', b'auto-generate.formats'
             )
             if enabled and targets:
                 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
             return tr
 
         @localrepo.unfilteredmethod
         def clonebundles_lock(self, wait=True):
             '''Lock the repository file related to clone bundles'''
             if not util.safehasattr(self, '_cb_lock_ref'):
                 self._cb_lock_ref = None
             l = self._currentlock(self._cb_lock_ref)
             if l is not None:
                 l.lock()
                 return l
 
             l = self._lock(
                 vfs=self.vfs,
                 lockname=b"clonebundleslock",
                 wait=wait,
                 releasefn=None,
                 acquirefn=None,
                 desc=_(b'repository %s') % self.origroot,
             )
             self._cb_lock_ref = weakref.ref(l)
             return l
 
     repo._wlockfreeprefix.add(AUTO_GEN_FILE)
     repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
     repo.__class__ = autobundlesrepo
 
 
-@command(b'admin::clone-bundles-refresh', [], b'')
-def cmd_admin_clone_bundles_refresh(ui, repo: localrepo.localrepository):
+@command(
+    b'admin::clone-bundles-refresh',
+    [
+        (
+            b'',
+            b'background',
+            False,
+            _(b'start bundle generation in the background'),
+        ),
+    ],
+    b'',
+)
+def cmd_admin_clone_bundles_refresh(
+    ui,
+    repo: localrepo.localrepository,
+    background=False,
+):
     """generate clone bundles according to the configuration
 
     This runs the logic for automatic generation, removing outdated bundles and
     generating new ones if necessary. See :hg:`help -e clone-bundles` for
     details about how to configure this feature.
     """
     debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
     bundles = read_auto_gen(repo)
     op_id = b"%d_acbr" % os.getpid()
     create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
 
-    # we clean up outdated bundle before generating new one to keep the last
-    # two version of the bundle around for a while and avoid having to deal
-    # client that just got served a manifest.
-    for o in delete:
-        delete_bundle(repo, o)
-    update_bundle_list(repo, del_bundles=delete)
+    # if some bundles are scheduled for creation in the background, they will
+    # deal with garbage collection too, so no need to synchroniously do it.
+    #
+    # However if no bundles are scheduled for creation, we need to explicitly do
+    # it here.
+    if not (background and create):
+        # we clean up outdated bundles before generating new ones to keep the
+        # last two versions of the bundle around for a while and avoid having to
+        # deal with clients that just got served a manifest.
+        for o in delete:
+            delete_bundle(repo, o)
+        update_bundle_list(repo, del_bundles=delete)
 
     if create:
         fpath = repo.vfs.makedirs(b'tmp-bundles')
-    for requested_bundle in create:
-        if debug:
-            msg = b'clone-bundles: starting bundle generation: %s\n'
-            repo.ui.write(msg % requested_bundle.bundle_type)
-        fname = requested_bundle.suggested_filename
-        fpath = repo.vfs.join(b'tmp-bundles', fname)
-        generating_bundle = requested_bundle.generating(fpath)
-        update_bundle_list(repo, new_bundles=[generating_bundle])
-        requested_bundle.generate_bundle(repo, fpath)
-        result = upload_bundle(repo, generating_bundle)
-        update_bundle_list(repo, new_bundles=[result])
-        update_ondisk_manifest(repo)
-        cleanup_tmp_bundle(repo, generating_bundle)
+
+    if background:
+        for requested_bundle in create:
+            start_one_bundle(repo, requested_bundle)
+    else:
+        for requested_bundle in create:
+            if debug:
+                msg = b'clone-bundles: starting bundle generation: %s\n'
+                repo.ui.write(msg % requested_bundle.bundle_type)
+            fname = requested_bundle.suggested_filename
+            fpath = repo.vfs.join(b'tmp-bundles', fname)
+            generating_bundle = requested_bundle.generating(fpath)
+            update_bundle_list(repo, new_bundles=[generating_bundle])
+            requested_bundle.generate_bundle(repo, fpath)
+            result = upload_bundle(repo, generating_bundle)
+            update_bundle_list(repo, new_bundles=[result])
+            update_ondisk_manifest(repo)
+            cleanup_tmp_bundle(repo, generating_bundle)
 
 
 @command(b'admin::clone-bundles-clear', [], b'')
 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
     """remove existing clone bundle caches
 
     See `hg help admin::clone-bundles-refresh` for details on how to regenerate
     them.
 
     This command will only affect bundles currently available, it will not
     affect bundles being asynchronously generated.
     """
     bundles = read_auto_gen(repo)
     delete = [b for b in bundles if b.ready]
     for o in delete:
         delete_bundle(repo, o)
     update_bundle_list(repo, del_bundles=delete)

tests/test-clonebundles-autogen.t

@@ -1,337 +1,360 b''
 
 #require no-reposimplestore no-chg
 
 initial setup
 
   $ hg init server
   $ cat >> server/.hg/hgrc << EOF
   > [extensions]
   > clonebundles =
   > 
   > [clone-bundles]
   > auto-generate.on-change = yes
   > auto-generate.formats = v2
   > upload-command = cp "\$HGCB_BUNDLE_PATH" "$TESTTMP"/final-upload/
   > delete-command = rm -f "$TESTTMP/final-upload/\$HGCB_BASENAME"
   > url-template = file://$TESTTMP/final-upload/{basename}
   > 
   > [devel]
   > debug.clonebundles=yes
   > EOF
 
   $ mkdir final-upload
   $ hg clone server client
   updating to branch default
   0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ cd client
 
 Test bundles are generated on push
 ==================================
 
   $ touch foo
   $ hg -q commit -A -m 'add foo'
   $ touch bar
   $ hg -q commit -A -m 'add bar'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   2 changesets found
   added 2 changesets with 2 changes to 2 files
   clone-bundles: starting bundle generation: v2
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Newer bundles are generated with more pushes
 --------------------------------------------
 
   $ touch baz
   $ hg -q commit -A -m 'add baz'
   $ touch buz
   $ hg -q commit -A -m 'add buz'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   4 changesets found
   added 2 changesets with 2 changes to 2 files
   clone-bundles: starting bundle generation: v2
 
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-4_revs-6427147b985a_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
   full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Older bundles are cleaned up with more pushes
 ---------------------------------------------
 
   $ touch faz
   $ hg -q commit -A -m 'add faz'
   $ touch fuz
   $ hg -q commit -A -m 'add fuz'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   clone-bundles: deleting bundle full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
   6 changesets found
   added 2 changesets with 2 changes to 2 files
   clone-bundles: starting bundle generation: v2
 
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
   full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Test conditions to get them generated
 =====================================
 
 Check ratio
 
   $ cat >> ../server/.hg/hgrc << EOF
   > [clone-bundles]
   > trigger.below-bundled-ratio = 0.5
   > EOF
   $ touch far
   $ hg -q commit -A -m 'add far'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
   full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Check absolute number of revisions
 
   $ cat >> ../server/.hg/hgrc << EOF
   > [clone-bundles]
   > trigger.revs = 2
   > EOF
   $ touch bur
   $ hg -q commit -A -m 'add bur'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   clone-bundles: deleting bundle full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
   8 changesets found
   added 1 changesets with 1 changes to 1 files
   clone-bundles: starting bundle generation: v2
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
   full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 (that one would not generate new bundles)
 
   $ touch tur
   $ hg -q commit -A -m 'add tur'
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
   full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Test generation through the dedicated command
 =============================================
 
   $ cat >> ../server/.hg/hgrc << EOF
   > [clone-bundles]
   > auto-generate.on-change = no
   > EOF
 
 Check the command can generate content when needed
 --------------------------------------------------
 
 Do a push that makes the condition fulfilled,
 Yet it should not automatically generate a bundle with
 "auto-generate.on-change" not set.
 
   $ touch quoi
   $ hg -q commit -A -m 'add quoi'
 
   $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
   $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
   $ ls -1 ../server/.hg/tmp-bundles
 
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
 
   $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
   $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
   $ ls -1 ../server/.hg/tmp-bundles
   $ test "$pre_push_manifest" = "$post_push_manifest"
   $ test "$pre_push_upload" = "$post_push_upload"
 
 Running the command should detect the stale bundles, and do the full automatic
 generation logic.
 
   $ hg -R ../server/ admin::clone-bundles-refresh
   clone-bundles: deleting bundle full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
   clone-bundles: starting bundle generation: v2
   10 changesets found
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
   full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Check the command cleans up older bundles when possible
 -------------------------------------------------------
 
   $ hg -R ../server/ admin::clone-bundles-refresh
   clone-bundles: deleting bundle full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Nothing is generated when the bundles are sufficiently up to date
 -----------------------------------------------------------------
 
   $ touch feur
   $ hg -q commit -A -m 'add feur'
 
   $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
   $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
   $ ls -1 ../server/.hg/tmp-bundles
 
   $ hg push
   pushing to $TESTTMP/server
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
 
   $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
   $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
   $ ls -1 ../server/.hg/tmp-bundles
   $ test "$pre_push_manifest" = "$post_push_manifest"
   $ test "$pre_push_upload" = "$post_push_upload"
 
   $ hg -R ../server/ admin::clone-bundles-refresh
 
   $ post_refresh_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
   $ post_refresh_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
   $ ls -1 ../server/.hg/tmp-bundles
   $ test "$pre_push_manifest" = "$post_refresh_manifest"
   $ test "$pre_push_upload" = "$post_refresh_upload"
 
 Test modification of configuration
 ==================================
 
 Testing that later runs adapt to configuration changes even if the repository is
 unchanged.
 
 adding more formats
 -------------------
 
 bundle for added formats should be generated
 
 change configuration
 
   $ cat >> ../server/.hg/hgrc << EOF
   > [clone-bundles]
   > auto-generate.formats = v1, v2
   > EOF
 
 refresh the bundles
 
   $ hg -R ../server/ admin::clone-bundles-refresh
   clone-bundles: starting bundle generation: v1
   11 changesets found
 
 the bundle for the "new" format should have been added
 
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
   file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
   full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Changing the ratio
 ------------------
 
 Changing the ratio to something that would have triggered a bundle during the last push.
 
   $ cat >> ../server/.hg/hgrc << EOF
   > [clone-bundles]
   > trigger.below-bundled-ratio = 0.95
   > EOF
 
 refresh the bundles
 
   $ hg -R ../server/ admin::clone-bundles-refresh
   clone-bundles: starting bundle generation: v2
   11 changesets found
 
 
 the "outdated' bundle should be refreshed
 
   $ cat ../server/.hg/clonebundles.manifest
   file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
   file:/*/$TESTTMP/final-upload/full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
   $ ls -1 ../final-upload
   full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
   full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
   full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
   $ ls -1 ../server/.hg/tmp-bundles
 
 Test more command options
 =========================
 
 bundle clearing
 ---------------
 
   $ hg -R ../server/ admin::clone-bundles-clear
   clone-bundles: deleting bundle full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
   clone-bundles: deleting bundle full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
   clone-bundles: deleting bundle full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
 
 Nothing should remain
 
   $ cat ../server/.hg/clonebundles.manifest
   $ ls -1 ../final-upload
   $ ls -1 ../server/.hg/tmp-bundles
+
+background generation
+---------------------
+
+generate bundle using background subprocess
+(since we are in devel mode, the command will still wait for the background
+process to end)
+
+  $ hg -R ../server/ admin::clone-bundles-refresh --background
+  11 changesets found
+  11 changesets found
+  clone-bundles: starting bundle generation: v1
+  clone-bundles: starting bundle generation: v2
+
+bundles should have been generated
+
+  $ cat ../server/.hg/clonebundles.manifest
+  file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
+  file:/*/$TESTTMP/final-upload/full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
+  $ ls -1 ../final-upload
+  full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
+  full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
+  $ ls -1 ../server/.hg/tmp-bundles