stream-clone: introduce the notion of an experimental "v3" version

We introduce a new experimental "v3" stream protocol, disabled by default. In
practice the "v3-exp" protocol introduced in this changeset is identical to v2,
but this changeset, lay the groundwork for having a new protocol:
configuration, capability exchange, test coverage, etc.

The actual protocol work will starts in the coming changesets.

hgext/remotefilelog/__init__.py

@@ -1,1261 +1,1259
 # __init__.py - remotefilelog extension
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """remotefilelog causes Mercurial to lazilly fetch file contents (EXPERIMENTAL)
 
 This extension is HIGHLY EXPERIMENTAL. There are NO BACKWARDS COMPATIBILITY
 GUARANTEES. This means that repositories created with this extension may
 only be usable with the exact version of this extension/Mercurial that was
 used. The extension attempts to enforce this in order to prevent repository
 corruption.
 
 remotefilelog works by fetching file contents lazily and storing them
 in a cache on the client rather than in revlogs. This allows enormous
 histories to be transferred only partially, making them easier to
 operate on.
 
 Configs:
 
     ``packs.maxchainlen`` specifies the maximum delta chain length in pack files
 
     ``packs.maxpacksize`` specifies the maximum pack file size
 
     ``packs.maxpackfilecount`` specifies the maximum number of packs in the
       shared cache (trees only for now)
 
     ``remotefilelog.backgroundprefetch`` runs prefetch in background when True
 
     ``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and
       update, and on other commands that use them. Different from pullprefetch.
 
     ``remotefilelog.gcrepack`` does garbage collection during repack when True
 
     ``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before
       it is garbage collected
 
     ``remotefilelog.repackonhggc`` runs repack on hg gc when True
 
     ``remotefilelog.prefetchdays`` specifies the maximum age of a commit in
       days after which it is no longer prefetched.
 
     ``remotefilelog.prefetchdelay`` specifies delay between background
       prefetches in seconds after operations that change the working copy parent
 
     ``remotefilelog.data.gencountlimit`` constraints the minimum number of data
       pack files required to be considered part of a generation. In particular,
       minimum number of packs files > gencountlimit.
 
     ``remotefilelog.data.generations`` list for specifying the lower bound of
       each generation of the data pack files. For example, list ['100MB','1MB']
       or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [
       1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to
       include in an incremental data repack.
 
     ``remotefilelog.data.repackmaxpacksize`` the maximum size of a pack file for
       it to be considered for an incremental data repack.
 
     ``remotefilelog.data.repacksizelimit`` the maximum total size of pack files
       to include in an incremental data repack.
 
     ``remotefilelog.history.gencountlimit`` constraints the minimum number of
       history pack files required to be considered part of a generation. In
       particular, minimum number of packs files > gencountlimit.
 
     ``remotefilelog.history.generations`` list for specifying the lower bound of
       each generation of the history pack files. For example, list [
       '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [
       0, 1MB), [1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to
       include in an incremental history repack.
 
     ``remotefilelog.history.repackmaxpacksize`` the maximum size of a pack file
       for it to be considered for an incremental history repack.
 
     ``remotefilelog.history.repacksizelimit`` the maximum total size of pack
       files to include in an incremental history repack.
 
     ``remotefilelog.backgroundrepack`` automatically consolidate packs in the
       background
 
     ``remotefilelog.cachepath`` path to cache
 
     ``remotefilelog.cachegroup`` if set, make cache directory sgid to this
       group
 
     ``remotefilelog.cacheprocess`` binary to invoke for fetching file data
 
     ``remotefilelog.debug`` turn on remotefilelog-specific debug output
 
     ``remotefilelog.excludepattern`` pattern of files to exclude from pulls
 
     ``remotefilelog.includepattern`` pattern of files to include in pulls
 
     ``remotefilelog.fetchwarning``: message to print when too many
       single-file fetches occur
 
     ``remotefilelog.getfilesstep`` number of files to request in a single RPC
 
     ``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch
       files, otherwise use optimistic fetching
 
     ``remotefilelog.pullprefetch`` revset for selecting files that should be
       eagerly downloaded rather than lazily
 
     ``remotefilelog.reponame`` name of the repo. If set, used to partition
       data from other repos in a shared store.
 
     ``remotefilelog.server`` if true, enable server-side functionality
 
     ``remotefilelog.servercachepath`` path for caching blobs on the server
 
     ``remotefilelog.serverexpiration`` number of days to keep cached server
       blobs
 
     ``remotefilelog.validatecache`` if set, check cache entries for corruption
       before returning blobs
 
     ``remotefilelog.validatecachelog`` if set, check cache entries for
       corruption before returning metadata
 
 """
 
 import os
 import time
 import traceback
 
 from mercurial.node import (
     hex,
     wdirrev,
 )
 from mercurial.i18n import _
 from mercurial.pycompat import open
 from mercurial import (
     changegroup,
     changelog,
     commands,
     configitems,
     context,
     copies,
     debugcommands as hgdebugcommands,
     dispatch,
     error,
     exchange,
     extensions,
     hg,
     localrepo,
     match as matchmod,
     merge,
     mergestate as mergestatemod,
     patch,
     pycompat,
     registrar,
     repair,
     repoview,
     revset,
     scmutil,
     smartset,
     streamclone,
     util,
 )
 from . import (
     constants,
     debugcommands,
     fileserverclient,
     remotefilectx,
     remotefilelog,
     remotefilelogserver,
     repack as repackmod,
     shallowbundle,
     shallowrepo,
     shallowstore,
     shallowutil,
     shallowverifier,
 )
 
 # ensures debug commands are registered
 hgdebugcommands.command
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 configitem(b'remotefilelog', b'debug', default=False)
 
 configitem(b'remotefilelog', b'reponame', default=b'')
 configitem(b'remotefilelog', b'cachepath', default=None)
 configitem(b'remotefilelog', b'cachegroup', default=None)
 configitem(b'remotefilelog', b'cacheprocess', default=None)
 configitem(b'remotefilelog', b'cacheprocess.includepath', default=None)
 configitem(b"remotefilelog", b"cachelimit", default=b"1000 GB")
 
 configitem(
     b'remotefilelog',
     b'fallbackpath',
     default=configitems.dynamicdefault,
     alias=[(b'remotefilelog', b'fallbackrepo')],
 )
 
 configitem(b'remotefilelog', b'validatecachelog', default=None)
 configitem(b'remotefilelog', b'validatecache', default=b'on')
 configitem(b'remotefilelog', b'server', default=None)
 configitem(b'remotefilelog', b'servercachepath', default=None)
 configitem(b"remotefilelog", b"serverexpiration", default=30)
 configitem(b'remotefilelog', b'backgroundrepack', default=False)
 configitem(b'remotefilelog', b'bgprefetchrevs', default=None)
 configitem(b'remotefilelog', b'pullprefetch', default=None)
 configitem(b'remotefilelog', b'backgroundprefetch', default=False)
 configitem(b'remotefilelog', b'prefetchdelay', default=120)
 configitem(b'remotefilelog', b'prefetchdays', default=14)
 # Other values include 'local' or 'none'. Any unrecognized value is 'all'.
 configitem(b'remotefilelog', b'strip.includefiles', default='all')
 
 configitem(b'remotefilelog', b'getfilesstep', default=10000)
 configitem(b'remotefilelog', b'getfilestype', default=b'optimistic')
 configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault)
 configitem(b'remotefilelog', b'fetchwarning', default=b'')
 
 configitem(b'remotefilelog', b'includepattern', default=None)
 configitem(b'remotefilelog', b'excludepattern', default=None)
 
 configitem(b'remotefilelog', b'gcrepack', default=False)
 configitem(b'remotefilelog', b'repackonhggc', default=False)
 configitem(b'repack', b'chainorphansbysize', default=True, experimental=True)
 
 configitem(b'packs', b'maxpacksize', default=0)
 configitem(b'packs', b'maxchainlen', default=1000)
 
 configitem(b'devel', b'remotefilelog.bg-wait', default=False)
 
 #  default TTL limit is 30 days
 _defaultlimit = 60 * 60 * 24 * 30
 configitem(b'remotefilelog', b'nodettl', default=_defaultlimit)
 
 configitem(b'remotefilelog', b'data.gencountlimit', default=2),
 configitem(
     b'remotefilelog', b'data.generations', default=[b'1GB', b'100MB', b'1MB']
 )
 configitem(b'remotefilelog', b'data.maxrepackpacks', default=50)
 configitem(b'remotefilelog', b'data.repackmaxpacksize', default=b'4GB')
 configitem(b'remotefilelog', b'data.repacksizelimit', default=b'100MB')
 
 configitem(b'remotefilelog', b'history.gencountlimit', default=2),
 configitem(b'remotefilelog', b'history.generations', default=[b'100MB'])
 configitem(b'remotefilelog', b'history.maxrepackpacks', default=50)
 configitem(b'remotefilelog', b'history.repackmaxpacksize', default=b'400MB')
 configitem(b'remotefilelog', b'history.repacksizelimit', default=b'100MB')
 
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
 testedwith = b'ships-with-hg-core'
 
 repoclass = localrepo.localrepository
 repoclass._basesupported.add(constants.SHALLOWREPO_REQUIREMENT)
 
 isenabled = shallowutil.isenabled
 
 
 def uisetup(ui):
     """Wraps user facing Mercurial commands to swap them out with shallow
     versions.
     """
     hg.wirepeersetupfuncs.append(fileserverclient.peersetup)
 
     entry = extensions.wrapcommand(commands.table, b'clone', cloneshallow)
     entry[1].append(
         (
             b'',
             b'shallow',
             None,
             _(b"create a shallow clone which uses remote file history"),
         )
     )
 
     extensions.wrapcommand(
         commands.table, b'debugindex', debugcommands.debugindex
     )
     extensions.wrapcommand(
         commands.table, b'debugindexdot', debugcommands.debugindexdot
     )
     extensions.wrapcommand(commands.table, b'log', log)
     extensions.wrapcommand(commands.table, b'pull', pull)
 
     # Prevent 'hg manifest --all'
     def _manifest(orig, ui, repo, *args, **opts):
         if isenabled(repo) and opts.get('all'):
             raise error.Abort(_(b"--all is not supported in a shallow repo"))
 
         return orig(ui, repo, *args, **opts)
 
     extensions.wrapcommand(commands.table, b"manifest", _manifest)
 
     # Wrap remotefilelog with lfs code
     def _lfsloaded(loaded=False):
         lfsmod = None
         try:
             lfsmod = extensions.find(b'lfs')
         except KeyError:
             pass
         if lfsmod:
             lfsmod.wrapfilelog(remotefilelog.remotefilelog)
             fileserverclient._lfsmod = lfsmod
 
     extensions.afterloaded(b'lfs', _lfsloaded)
 
     # debugdata needs remotefilelog.len to work
     extensions.wrapcommand(commands.table, b'debugdata', debugdatashallow)
 
     changegroup.cgpacker = shallowbundle.shallowcg1packer
 
     extensions.wrapfunction(
         changegroup, b'_addchangegroupfiles', shallowbundle.addchangegroupfiles
     )
     extensions.wrapfunction(
         changegroup, b'makechangegroup', shallowbundle.makechangegroup
     )
     extensions.wrapfunction(localrepo, b'makestore', storewrapper)
     extensions.wrapfunction(exchange, b'pull', exchangepull)
     extensions.wrapfunction(merge, b'applyupdates', applyupdates)
     extensions.wrapfunction(merge, b'_checkunknownfiles', checkunknownfiles)
     extensions.wrapfunction(context.workingctx, b'_checklookup', checklookup)
     extensions.wrapfunction(scmutil, b'_findrenames', findrenames)
     extensions.wrapfunction(
         copies, b'_computeforwardmissing', computeforwardmissing
     )
     extensions.wrapfunction(dispatch, b'runcommand', runcommand)
     extensions.wrapfunction(repair, b'_collectbrokencsets', _collectbrokencsets)
     extensions.wrapfunction(context.changectx, b'filectx', filectx)
     extensions.wrapfunction(context.workingctx, b'filectx', workingfilectx)
     extensions.wrapfunction(patch, b'trydiff', trydiff)
     extensions.wrapfunction(hg, b'verify', _verify)
     scmutil.fileprefetchhooks.add(b'remotefilelog', _fileprefetchhook)
 
     # disappointing hacks below
     extensions.wrapfunction(scmutil, b'getrenamedfn', getrenamedfn)
     extensions.wrapfunction(revset, b'filelog', filelogrevset)
     revset.symbols[b'filelog'] = revset.filelog
 
 
 def cloneshallow(orig, ui, repo, *args, **opts):
     if opts.get('shallow'):
         repos = []
 
         def pull_shallow(orig, self, *args, **kwargs):
             if not isenabled(self):
                 repos.append(self.unfiltered())
                 # set up the client hooks so the post-clone update works
                 setupclient(self.ui, self.unfiltered())
 
                 # setupclient fixed the class on the repo itself
                 # but we also need to fix it on the repoview
                 if isinstance(self, repoview.repoview):
                     self.__class__.__bases__ = (
                         self.__class__.__bases__[0],
                         self.unfiltered().__class__,
                     )
                 self.requirements.add(constants.SHALLOWREPO_REQUIREMENT)
                 with self.lock():
                     # acquire store lock before writing requirements as some
                     # requirements might be written to .hg/store/requires
                     scmutil.writereporequirements(self)
 
                 # Since setupclient hadn't been called, exchange.pull was not
                 # wrapped. So we need to manually invoke our version of it.
                 return exchangepull(orig, self, *args, **kwargs)
             else:
                 return orig(self, *args, **kwargs)
 
         extensions.wrapfunction(exchange, b'pull', pull_shallow)
 
         # Wrap the stream logic to add requirements and to pass include/exclude
         # patterns around.
         def setup_streamout(repo, remote):
             # Replace remote.stream_out with a version that sends file
             # patterns.
             def stream_out_shallow(orig):
                 caps = remote.capabilities()
                 if constants.NETWORK_CAP_LEGACY_SSH_GETFILES in caps:
                     opts = {}
                     if repo.includepattern:
                         opts['includepattern'] = b'\0'.join(repo.includepattern)
                     if repo.excludepattern:
                         opts['excludepattern'] = b'\0'.join(repo.excludepattern)
                     return remote._callstream(b'stream_out_shallow', **opts)
                 else:
                     return orig()
 
             extensions.wrapfunction(remote, b'stream_out', stream_out_shallow)
 
         def stream_wrap(orig, op):
             setup_streamout(op.repo, op.remote)
             return orig(op)
 
         extensions.wrapfunction(
             streamclone, b'maybeperformlegacystreamclone', stream_wrap
         )
 
         def canperformstreamclone(orig, pullop, bundle2=False):
             # remotefilelog is currently incompatible with the
             # bundle2 flavor of streamclones, so force us to use
             # v1 instead.
             if b'v2' in pullop.remotebundle2caps.get(b'stream', []):
-                pullop.remotebundle2caps[b'stream'] = [
-                    c for c in pullop.remotebundle2caps[b'stream'] if c != b'v2'
-                ]
+                pullop.remotebundle2caps[b'stream'] = []
             if bundle2:
                 return False, None
             supported, requirements = orig(pullop, bundle2=bundle2)
             if requirements is not None:
                 requirements.add(constants.SHALLOWREPO_REQUIREMENT)
             return supported, requirements
 
         extensions.wrapfunction(
             streamclone, b'canperformstreamclone', canperformstreamclone
         )
 
     try:
         orig(ui, repo, *args, **opts)
     finally:
         if opts.get('shallow'):
             for r in repos:
                 if util.safehasattr(r, b'fileservice'):
                     r.fileservice.close()
 
 
 def debugdatashallow(orig, *args, **kwds):
     oldlen = remotefilelog.remotefilelog.__len__
     try:
         remotefilelog.remotefilelog.__len__ = lambda x: 1
         return orig(*args, **kwds)
     finally:
         remotefilelog.remotefilelog.__len__ = oldlen
 
 
 def reposetup(ui, repo):
     if not repo.local():
         return
 
     # put here intentionally bc doesnt work in uisetup
     ui.setconfig(b'hooks', b'update.prefetch', wcpprefetch)
     ui.setconfig(b'hooks', b'commit.prefetch', wcpprefetch)
 
     isserverenabled = ui.configbool(b'remotefilelog', b'server')
     isshallowclient = isenabled(repo)
 
     if isserverenabled and isshallowclient:
         raise RuntimeError(b"Cannot be both a server and shallow client.")
 
     if isshallowclient:
         setupclient(ui, repo)
 
     if isserverenabled:
         remotefilelogserver.setupserver(ui, repo)
 
 
 def setupclient(ui, repo):
     if not isinstance(repo, localrepo.localrepository):
         return
 
     # Even clients get the server setup since they need to have the
     # wireprotocol endpoints registered.
     remotefilelogserver.onetimesetup(ui)
     onetimeclientsetup(ui)
 
     shallowrepo.wraprepo(repo)
     repo.store = shallowstore.wrapstore(repo.store)
 
 
 def storewrapper(orig, requirements, path, vfstype):
     s = orig(requirements, path, vfstype)
     if constants.SHALLOWREPO_REQUIREMENT in requirements:
         s = shallowstore.wrapstore(s)
 
     return s
 
 
 # prefetch files before update
 def applyupdates(
     orig, repo, mresult, wctx, mctx, overwrite, wantfiledata, **opts
 ):
     if isenabled(repo):
         manifest = mctx.manifest()
         files = []
         for f, args, msg in mresult.getactions([mergestatemod.ACTION_GET]):
             files.append((f, hex(manifest[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, mresult, wctx, mctx, overwrite, wantfiledata, **opts)
 
 
 # Prefetch merge checkunknownfiles
 def checkunknownfiles(orig, repo, wctx, mctx, force, mresult, *args, **kwargs):
     if isenabled(repo):
         files = []
         sparsematch = repo.maybesparsematch(mctx.rev())
         for f, (m, actionargs, msg) in mresult.filemap():
             if sparsematch and not sparsematch(f):
                 continue
             if m in (
                 mergestatemod.ACTION_CREATED,
                 mergestatemod.ACTION_DELETED_CHANGED,
                 mergestatemod.ACTION_CREATED_MERGE,
             ):
                 files.append((f, hex(mctx.filenode(f))))
             elif m == mergestatemod.ACTION_LOCAL_DIR_RENAME_GET:
                 f2 = actionargs[0]
                 files.append((f2, hex(mctx.filenode(f2))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, wctx, mctx, force, mresult, *args, **kwargs)
 
 
 # Prefetch files before status attempts to look at their size and contents
 def checklookup(orig, self, files, mtime_boundary):
     repo = self._repo
     if isenabled(repo):
         prefetchfiles = []
         for parent in self._parents:
             for f in files:
                 if f in parent:
                     prefetchfiles.append((f, hex(parent.filenode(f))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(prefetchfiles)
     return orig(self, files, mtime_boundary)
 
 
 # Prefetch the logic that compares added and removed files for renames
 def findrenames(orig, repo, matcher, added, removed, *args, **kwargs):
     if isenabled(repo):
         files = []
         pmf = repo[b'.'].manifest()
         for f in removed:
             if f in pmf:
                 files.append((f, hex(pmf[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, matcher, added, removed, *args, **kwargs)
 
 
 # prefetch files before pathcopies check
 def computeforwardmissing(orig, a, b, match=None):
     missing = orig(a, b, match=match)
     repo = a._repo
     if isenabled(repo):
         mb = b.manifest()
 
         files = []
         sparsematch = repo.maybesparsematch(b.rev())
         if sparsematch:
             sparsemissing = set()
             for f in missing:
                 if sparsematch(f):
                     files.append((f, hex(mb[f])))
                     sparsemissing.add(f)
             missing = sparsemissing
 
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return missing
 
 
 # close cache miss server connection after the command has finished
 def runcommand(orig, lui, repo, *args, **kwargs):
     fileservice = None
     # repo can be None when running in chg:
     # - at startup, reposetup was called because serve is not norepo
     # - a norepo command like "help" is called
     if repo and isenabled(repo):
         fileservice = repo.fileservice
     try:
         return orig(lui, repo, *args, **kwargs)
     finally:
         if fileservice:
             fileservice.close()
 
 
 # prevent strip from stripping remotefilelogs
 def _collectbrokencsets(orig, repo, files, striprev):
     if isenabled(repo):
         files = list([f for f in files if not repo.shallowmatch(f)])
     return orig(repo, files, striprev)
 
 
 # changectx wrappers
 def filectx(orig, self, path, fileid=None, filelog=None):
     if fileid is None:
         fileid = self.filenode(path)
     if isenabled(self._repo) and self._repo.shallowmatch(path):
         return remotefilectx.remotefilectx(
             self._repo, path, fileid=fileid, changectx=self, filelog=filelog
         )
     return orig(self, path, fileid=fileid, filelog=filelog)
 
 
 def workingfilectx(orig, self, path, filelog=None):
     if isenabled(self._repo) and self._repo.shallowmatch(path):
         return remotefilectx.remoteworkingfilectx(
             self._repo, path, workingctx=self, filelog=filelog
         )
     return orig(self, path, filelog=filelog)
 
 
 # prefetch required revisions before a diff
 def trydiff(
     orig,
     repo,
     revs,
     ctx1,
     ctx2,
     modified,
     added,
     removed,
     copy,
     getfilectx,
     *args,
     **kwargs
 ):
     if isenabled(repo):
         prefetch = []
         mf1 = ctx1.manifest()
         for fname in modified + added + removed:
             if fname in mf1:
                 fnode = getfilectx(fname, ctx1).filenode()
                 # fnode can be None if it's a edited working ctx file
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
             if fname not in removed:
                 fnode = getfilectx(fname, ctx2).filenode()
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
 
         repo.fileservice.prefetch(prefetch)
 
     return orig(
         repo,
         revs,
         ctx1,
         ctx2,
         modified,
         added,
         removed,
         copy,
         getfilectx,
         *args,
         **kwargs
     )
 
 
 # Prevent verify from processing files
 # a stub for mercurial.hg.verify()
 def _verify(orig, repo, level=None):
     lock = repo.lock()
     try:
         return shallowverifier.shallowverifier(repo).verify()
     finally:
         lock.release()
 
 
 clientonetime = False
 
 
 def onetimeclientsetup(ui):
     global clientonetime
     if clientonetime:
         return
     clientonetime = True
 
     # Don't commit filelogs until we know the commit hash, since the hash
     # is present in the filelog blob.
     # This violates Mercurial's filelog->manifest->changelog write order,
     # but is generally fine for client repos.
     pendingfilecommits = []
 
     def addrawrevision(
         orig,
         self,
         rawtext,
         transaction,
         link,
         p1,
         p2,
         node,
         flags,
         cachedelta=None,
         _metatuple=None,
     ):
         if isinstance(link, int):
             pendingfilecommits.append(
                 (
                     self,
                     rawtext,
                     transaction,
                     link,
                     p1,
                     p2,
                     node,
                     flags,
                     cachedelta,
                     _metatuple,
                 )
             )
             return node
         else:
             return orig(
                 self,
                 rawtext,
                 transaction,
                 link,
                 p1,
                 p2,
                 node,
                 flags,
                 cachedelta,
                 _metatuple=_metatuple,
             )
 
     extensions.wrapfunction(
         remotefilelog.remotefilelog, b'addrawrevision', addrawrevision
     )
 
     def changelogadd(orig, self, *args, **kwargs):
         oldlen = len(self)
         node = orig(self, *args, **kwargs)
         newlen = len(self)
         if oldlen != newlen:
             for oldargs in pendingfilecommits:
                 log, rt, tr, link, p1, p2, n, fl, c, m = oldargs
                 linknode = self.node(link)
                 if linknode == node:
                     log.addrawrevision(rt, tr, linknode, p1, p2, n, fl, c, m)
                 else:
                     raise error.ProgrammingError(
                         b'pending multiple integer revisions are not supported'
                     )
         else:
             # "link" is actually wrong here (it is set to len(changelog))
             # if changelog remains unchanged, skip writing file revisions
             # but still do a sanity check about pending multiple revisions
             if len({x[3] for x in pendingfilecommits}) > 1:
                 raise error.ProgrammingError(
                     b'pending multiple integer revisions are not supported'
                 )
         del pendingfilecommits[:]
         return node
 
     extensions.wrapfunction(changelog.changelog, b'add', changelogadd)
 
 
 def getrenamedfn(orig, repo, endrev=None):
     if not isenabled(repo) or copies.usechangesetcentricalgo(repo):
         return orig(repo, endrev)
 
     rcache = {}
 
     def getrenamed(fn, rev):
         """looks up all renames for a file (up to endrev) the first
         time the file is given. It indexes on the changerev and only
         parses the manifest if linkrev != changerev.
         Returns rename info for fn at changerev rev."""
         if rev in rcache.setdefault(fn, {}):
             return rcache[fn][rev]
 
         try:
             fctx = repo[rev].filectx(fn)
             for ancestor in fctx.ancestors():
                 if ancestor.path() == fn:
                     renamed = ancestor.renamed()
                     rcache[fn][ancestor.rev()] = renamed and renamed[0]
 
             renamed = fctx.renamed()
             return renamed and renamed[0]
         except error.LookupError:
             return None
 
     return getrenamed
 
 
 def filelogrevset(orig, repo, subset, x):
     """``filelog(pattern)``
     Changesets connected to the specified filelog.
 
     For performance reasons, ``filelog()`` does not show every changeset
     that affects the requested file(s). See :hg:`help log` for details. For
     a slower, more accurate result, use ``file()``.
     """
 
     if not isenabled(repo):
         return orig(repo, subset, x)
 
     # i18n: "filelog" is a keyword
     pat = revset.getstring(x, _(b"filelog requires a pattern"))
     m = matchmod.match(
         repo.root, repo.getcwd(), [pat], default=b'relpath', ctx=repo[None]
     )
     s = set()
 
     if not matchmod.patkind(pat):
         # slow
         for r in subset:
             ctx = repo[r]
             cfiles = ctx.files()
             for f in m.files():
                 if f in cfiles:
                     s.add(ctx.rev())
                     break
     else:
         # partial
         files = (f for f in repo[None] if m(f))
         for f in files:
             fctx = repo[None].filectx(f)
             s.add(fctx.linkrev())
             for actx in fctx.ancestors():
                 s.add(actx.linkrev())
 
     return smartset.baseset([r for r in subset if r in s])
 
 
 @command(b'gc', [], _(b'hg gc [REPO...]'), norepo=True)
 def gc(ui, *args, **opts):
     """garbage collect the client and server filelog caches"""
     cachepaths = set()
 
     # get the system client cache
     systemcache = shallowutil.getcachepath(ui, allowempty=True)
     if systemcache:
         cachepaths.add(systemcache)
 
     # get repo client and server cache
     repopaths = []
     pwd = ui.environ.get(b'PWD')
     if pwd:
         repopaths.append(pwd)
 
     repopaths.extend(args)
     repos = []
     for repopath in repopaths:
         try:
             repo = hg.peer(ui, {}, repopath)
             repos.append(repo)
 
             repocache = shallowutil.getcachepath(repo.ui, allowempty=True)
             if repocache:
                 cachepaths.add(repocache)
         except error.RepoError:
             pass
 
     # gc client cache
     for cachepath in cachepaths:
         gcclient(ui, cachepath)
 
     # gc server cache
     for repo in repos:
         remotefilelogserver.gcserver(ui, repo._repo)
 
 
 def gcclient(ui, cachepath):
     # get list of repos that use this cache
     repospath = os.path.join(cachepath, b'repos')
     if not os.path.exists(repospath):
         ui.warn(_(b"no known cache at %s\n") % cachepath)
         return
 
     reposfile = open(repospath, b'rb')
     repos = {r[:-1] for r in reposfile.readlines()}
     reposfile.close()
 
     # build list of useful files
     validrepos = []
     keepkeys = set()
 
     sharedcache = None
     filesrepacked = False
 
     count = 0
     progress = ui.makeprogress(
         _(b"analyzing repositories"), unit=b"repos", total=len(repos)
     )
     for path in repos:
         progress.update(count)
         count += 1
         try:
             path = util.expandpath(os.path.normpath(path))
         except TypeError as e:
             ui.warn(_(b"warning: malformed path: %r:%s\n") % (path, e))
             traceback.print_exc()
             continue
         try:
             peer = hg.peer(ui, {}, path)
             repo = peer._repo
         except error.RepoError:
             continue
 
         validrepos.append(path)
 
         # Protect against any repo or config changes that have happened since
         # this repo was added to the repos file. We'd rather this loop succeed
         # and too much be deleted, than the loop fail and nothing gets deleted.
         if not isenabled(repo):
             continue
 
         if not util.safehasattr(repo, b'name'):
             ui.warn(
                 _(b"repo %s is a misconfigured remotefilelog repo\n") % path
             )
             continue
 
         # If garbage collection on repack and repack on hg gc are enabled
         # then loose files are repacked and garbage collected.
         # Otherwise regular garbage collection is performed.
         repackonhggc = repo.ui.configbool(b'remotefilelog', b'repackonhggc')
         gcrepack = repo.ui.configbool(b'remotefilelog', b'gcrepack')
         if repackonhggc and gcrepack:
             try:
                 repackmod.incrementalrepack(repo)
                 filesrepacked = True
                 continue
             except (IOError, repackmod.RepackAlreadyRunning):
                 # If repack cannot be performed due to not enough disk space
                 # continue doing garbage collection of loose files w/o repack
                 pass
 
         reponame = repo.name
         if not sharedcache:
             sharedcache = repo.sharedstore
 
         # Compute a keepset which is not garbage collected
         def keyfn(fname, fnode):
             return fileserverclient.getcachekey(reponame, fname, hex(fnode))
 
         keepkeys = repackmod.keepset(repo, keyfn=keyfn, lastkeepkeys=keepkeys)
 
     progress.complete()
 
     # write list of valid repos back
     oldumask = os.umask(0o002)
     try:
         reposfile = open(repospath, b'wb')
         reposfile.writelines([(b"%s\n" % r) for r in validrepos])
         reposfile.close()
     finally:
         os.umask(oldumask)
 
     # prune cache
     if sharedcache is not None:
         sharedcache.gc(keepkeys)
     elif not filesrepacked:
         ui.warn(_(b"warning: no valid repos in repofile\n"))
 
 
 def log(orig, ui, repo, *pats, **opts):
     if not isenabled(repo):
         return orig(ui, repo, *pats, **opts)
 
     follow = opts.get('follow')
     revs = opts.get('rev')
     if pats:
         # Force slowpath for non-follow patterns and follows that start from
         # non-working-copy-parent revs.
         if not follow or revs:
             # This forces the slowpath
             opts['removed'] = True
 
         # If this is a non-follow log without any revs specified, recommend that
         # the user add -f to speed it up.
         if not follow and not revs:
             match = scmutil.match(repo[b'.'], pats, pycompat.byteskwargs(opts))
             isfile = not match.anypats()
             if isfile:
                 for file in match.files():
                     if not os.path.isfile(repo.wjoin(file)):
                         isfile = False
                         break
 
             if isfile:
                 ui.warn(
                     _(
                         b"warning: file log can be slow on large repos - "
                         + b"use -f to speed it up\n"
                     )
                 )
 
     return orig(ui, repo, *pats, **opts)
 
 
 def revdatelimit(ui, revset):
     """Update revset so that only changesets no older than 'prefetchdays' days
     are included. The default value is set to 14 days. If 'prefetchdays' is set
     to zero or negative value then date restriction is not applied.
     """
     days = ui.configint(b'remotefilelog', b'prefetchdays')
     if days > 0:
         revset = b'(%s) & date(-%s)' % (revset, days)
     return revset
 
 
 def readytofetch(repo):
     """Check that enough time has passed since the last background prefetch.
     This only relates to prefetches after operations that change the working
     copy parent. Default delay between background prefetches is 2 minutes.
     """
     timeout = repo.ui.configint(b'remotefilelog', b'prefetchdelay')
     fname = repo.vfs.join(b'lastprefetch')
 
     ready = False
     with open(fname, b'a'):
         # the with construct above is used to avoid race conditions
         modtime = os.path.getmtime(fname)
         if (time.time() - modtime) > timeout:
             os.utime(fname, None)
             ready = True
 
     return ready
 
 
 def wcpprefetch(ui, repo, **kwargs):
     """Prefetches in background revisions specified by bgprefetchrevs revset.
     Does background repack if backgroundrepack flag is set in config.
     """
     shallow = isenabled(repo)
     bgprefetchrevs = ui.config(b'remotefilelog', b'bgprefetchrevs')
     isready = readytofetch(repo)
 
     if not (shallow and bgprefetchrevs and isready):
         return
 
     bgrepack = repo.ui.configbool(b'remotefilelog', b'backgroundrepack')
     # update a revset with a date limit
     bgprefetchrevs = revdatelimit(ui, bgprefetchrevs)
 
     def anon(unused_success):
         if util.safehasattr(repo, b'ranprefetch') and repo.ranprefetch:
             return
         repo.ranprefetch = True
         repo.backgroundprefetch(bgprefetchrevs, repack=bgrepack)
 
     repo._afterlock(anon)
 
 
 def pull(orig, ui, repo, *pats, **opts):
     result = orig(ui, repo, *pats, **opts)
 
     if isenabled(repo):
         # prefetch if it's configured
         prefetchrevset = ui.config(b'remotefilelog', b'pullprefetch')
         bgrepack = repo.ui.configbool(b'remotefilelog', b'backgroundrepack')
         bgprefetch = repo.ui.configbool(b'remotefilelog', b'backgroundprefetch')
 
         if prefetchrevset:
             ui.status(_(b"prefetching file contents\n"))
             revs = scmutil.revrange(repo, [prefetchrevset])
             base = repo[b'.'].rev()
             if bgprefetch:
                 repo.backgroundprefetch(prefetchrevset, repack=bgrepack)
             else:
                 repo.prefetch(revs, base=base)
                 if bgrepack:
                     repackmod.backgroundrepack(repo, incremental=True)
         elif bgrepack:
             repackmod.backgroundrepack(repo, incremental=True)
 
     return result
 
 
 def exchangepull(orig, repo, remote, *args, **kwargs):
     # Hook into the callstream/getbundle to insert bundle capabilities
     # during a pull.
     def localgetbundle(
         orig, source, heads=None, common=None, bundlecaps=None, **kwargs
     ):
         if not bundlecaps:
             bundlecaps = set()
         bundlecaps.add(constants.BUNDLE2_CAPABLITY)
         return orig(
             source, heads=heads, common=common, bundlecaps=bundlecaps, **kwargs
         )
 
     if util.safehasattr(remote, b'_callstream'):
         remote._localrepo = repo
     elif util.safehasattr(remote, b'getbundle'):
         extensions.wrapfunction(remote, b'getbundle', localgetbundle)
 
     return orig(repo, remote, *args, **kwargs)
 
 
 def _fileprefetchhook(repo, revmatches):
     if isenabled(repo):
         allfiles = []
         for rev, match in revmatches:
             if rev == wdirrev or rev is None:
                 continue
             ctx = repo[rev]
             mf = ctx.manifest()
             sparsematch = repo.maybesparsematch(ctx.rev())
             for path in ctx.walk(match):
                 if (not sparsematch or sparsematch(path)) and path in mf:
                     allfiles.append((path, hex(mf[path])))
         repo.fileservice.prefetch(allfiles)
 
 
 @command(
     b'debugremotefilelog',
     [
         (b'd', b'decompress', None, _(b'decompress the filelog first')),
     ],
     _(b'hg debugremotefilelog <path>'),
     norepo=True,
 )
 def debugremotefilelog(ui, path, **opts):
     return debugcommands.debugremotefilelog(ui, path, **opts)
 
 
 @command(
     b'verifyremotefilelog',
     [
         (b'd', b'decompress', None, _(b'decompress the filelogs first')),
     ],
     _(b'hg verifyremotefilelogs <directory>'),
     norepo=True,
 )
 def verifyremotefilelog(ui, path, **opts):
     return debugcommands.verifyremotefilelog(ui, path, **opts)
 
 
 @command(
     b'debugdatapack',
     [
         (b'', b'long', None, _(b'print the long hashes')),
         (b'', b'node', b'', _(b'dump the contents of node'), b'NODE'),
     ],
     _(b'hg debugdatapack <paths>'),
     norepo=True,
 )
 def debugdatapack(ui, *paths, **opts):
     return debugcommands.debugdatapack(ui, *paths, **opts)
 
 
 @command(b'debughistorypack', [], _(b'hg debughistorypack <path>'), norepo=True)
 def debughistorypack(ui, path, **opts):
     return debugcommands.debughistorypack(ui, path)
 
 
 @command(b'debugkeepset', [], _(b'hg debugkeepset'))
 def debugkeepset(ui, repo, **opts):
     # The command is used to measure keepset computation time
     def keyfn(fname, fnode):
         return fileserverclient.getcachekey(repo.name, fname, hex(fnode))
 
     repackmod.keepset(repo, keyfn)
     return
 
 
 @command(b'debugwaitonrepack', [], _(b'hg debugwaitonrepack'))
 def debugwaitonrepack(ui, repo, **opts):
     return debugcommands.debugwaitonrepack(repo)
 
 
 @command(b'debugwaitonprefetch', [], _(b'hg debugwaitonprefetch'))
 def debugwaitonprefetch(ui, repo, **opts):
     return debugcommands.debugwaitonprefetch(repo)
 
 
 def resolveprefetchopts(ui, opts):
     if not opts.get(b'rev'):
         revset = [b'.', b'draft()']
 
         prefetchrevset = ui.config(b'remotefilelog', b'pullprefetch', None)
         if prefetchrevset:
             revset.append(b'(%s)' % prefetchrevset)
         bgprefetchrevs = ui.config(b'remotefilelog', b'bgprefetchrevs', None)
         if bgprefetchrevs:
             revset.append(b'(%s)' % bgprefetchrevs)
         revset = b'+'.join(revset)
 
         # update a revset with a date limit
         revset = revdatelimit(ui, revset)
 
         opts[b'rev'] = [revset]
 
     if not opts.get(b'base'):
         opts[b'base'] = None
 
     return opts
 
 
 @command(
     b'prefetch',
     [
         (b'r', b'rev', [], _(b'prefetch the specified revisions'), _(b'REV')),
         (b'', b'repack', False, _(b'run repack after prefetch')),
         (b'b', b'base', b'', _(b"rev that is assumed to already be local")),
     ]
     + commands.walkopts,
     _(b'hg prefetch [OPTIONS] [FILE...]'),
     helpcategory=command.CATEGORY_MAINTENANCE,
 )
 def prefetch(ui, repo, *pats, **opts):
     """prefetch file revisions from the server
 
     Prefetchs file revisions for the specified revs and stores them in the
     local remotefilelog cache.  If no rev is specified, the default rev is
     used which is the union of dot, draft, pullprefetch and bgprefetchrev.
     File names or patterns can be used to limit which files are downloaded.
 
     Return 0 on success.
     """
     opts = pycompat.byteskwargs(opts)
     if not isenabled(repo):
         raise error.Abort(_(b"repo is not shallow"))
 
     opts = resolveprefetchopts(ui, opts)
     revs = scmutil.revrange(repo, opts.get(b'rev'))
     repo.prefetch(revs, opts.get(b'base'), pats, opts)
 
     # Run repack in background
     if opts.get(b'repack'):
         repackmod.backgroundrepack(repo, incremental=True)
 
 
 @command(
     b'repack',
     [
         (b'', b'background', None, _(b'run in a background process'), None),
         (b'', b'incremental', None, _(b'do an incremental repack'), None),
         (
             b'',
             b'packsonly',
             None,
             _(b'only repack packs (skip loose objects)'),
             None,
         ),
     ],
     _(b'hg repack [OPTIONS]'),
 )
 def repack_(ui, repo, *pats, **opts):
     if opts.get('background'):
         repackmod.backgroundrepack(
             repo,
             incremental=opts.get('incremental'),
             packsonly=opts.get('packsonly', False),
         )
         return
 
     options = {b'packsonly': opts.get('packsonly')}
 
     try:
         if opts.get('incremental'):
             repackmod.incrementalrepack(repo, options=options)
         else:
             repackmod.fullrepack(repo, options=options)
     except repackmod.RepackAlreadyRunning as ex:
         # Don't propogate the exception if the repack is already in
         # progress, since we want the command to exit 0.
         repo.ui.warn(b'%s\n' % ex)

mercurial/bundle2.py

@@ -1,2636 +1,2664
 # bundle2.py - generic container format to transmit arbitrary data.
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """Handling of the new bundle2 format
 
 The goal of bundle2 is to act as an atomically packet to transmit a set of
 payloads in an application agnostic way. It consist in a sequence of "parts"
 that will be handed to and processed by the application layer.
 
 
 General format architecture
 ===========================
 
 The format is architectured as follow
 
  - magic string
  - stream level parameters
  - payload parts (any number)
  - end of stream marker.
 
 the Binary format
 ============================
 
 All numbers are unsigned and big-endian.
 
 stream level parameters
 ------------------------
 
 Binary format is as follow
 
 :params size: int32
 
   The total number of Bytes used by the parameters
 
 :params value: arbitrary number of Bytes
 
   A blob of `params size` containing the serialized version of all stream level
   parameters.
 
   The blob contains a space separated list of parameters. Parameters with value
   are stored in the form `<name>=<value>`. Both name and value are urlquoted.
 
   Empty name are obviously forbidden.
 
   Name MUST start with a letter. If this first letter is lower case, the
   parameter is advisory and can be safely ignored. However when the first
   letter is capital, the parameter is mandatory and the bundling process MUST
   stop if he is not able to proceed it.
 
   Stream parameters use a simple textual format for two main reasons:
 
   - Stream level parameters should remain simple and we want to discourage any
     crazy usage.
   - Textual data allow easy human inspection of a bundle2 header in case of
     troubles.
 
   Any Applicative level options MUST go into a bundle2 part instead.
 
 Payload part
 ------------------------
 
 Binary format is as follow
 
 :header size: int32
 
   The total number of Bytes used by the part header. When the header is empty
   (size = 0) this is interpreted as the end of stream marker.
 
 :header:
 
     The header defines how to interpret the part. It contains two piece of
     data: the part type, and the part parameters.
 
     The part type is used to route an application level handler, that can
     interpret payload.
 
     Part parameters are passed to the application level handler.  They are
     meant to convey information that will help the application level object to
     interpret the part payload.
 
     The binary format of the header is has follow
 
     :typesize: (one byte)
 
     :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*)
 
     :partid: A 32bits integer (unique in the bundle) that can be used to refer
              to this part.
 
     :parameters:
 
         Part's parameter may have arbitrary content, the binary structure is::
 
             <mandatory-count><advisory-count><param-sizes><param-data>
 
         :mandatory-count: 1 byte, number of mandatory parameters
 
         :advisory-count:  1 byte, number of advisory parameters
 
         :param-sizes:
 
             N couple of bytes, where N is the total number of parameters. Each
             couple contains (<size-of-key>, <size-of-value) for one parameter.
 
         :param-data:
 
             A blob of bytes from which each parameter key and value can be
             retrieved using the list of size couples stored in the previous
             field.
 
             Mandatory parameters comes first, then the advisory ones.
 
             Each parameter's key MUST be unique within the part.
 
 :payload:
 
     payload is a series of `<chunksize><chunkdata>`.
 
     `chunksize` is an int32, `chunkdata` are plain bytes (as much as
     `chunksize` says)` The payload part is concluded by a zero size chunk.
 
     The current implementation always produces either zero or one chunk.
     This is an implementation limitation that will ultimately be lifted.
 
     `chunksize` can be negative to trigger special case processing. No such
     processing is in place yet.
 
 Bundle processing
 ============================
 
 Each part is processed in order using a "part handler". Handler are registered
 for a certain part type.
 
 The matching of a part to its handler is case insensitive. The case of the
 part type is used to know if a part is mandatory or advisory. If the Part type
 contains any uppercase char it is considered mandatory. When no handler is
 known for a Mandatory part, the process is aborted and an exception is raised.
 If the part is advisory and no handler is known, the part is ignored. When the
 process is aborted, the full bundle is still read from the stream to keep the
 channel usable. But none of the part read from an abort are processed. In the
 future, dropping the stream may become an option for channel we do not care to
 preserve.
 """
 
 
 import collections
 import errno
 import os
 import re
 import string
 import struct
 import sys
 
 from .i18n import _
 from .node import (
     hex,
     short,
 )
 from . import (
     bookmarks,
     changegroup,
     encoding,
     error,
     obsolete,
     phases,
     pushkey,
     pycompat,
     requirements,
     scmutil,
     streamclone,
     tags,
     url,
     util,
 )
 from .utils import (
     stringutil,
     urlutil,
 )
 from .interfaces import repository
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 _pack = struct.pack
 _unpack = struct.unpack
 
 _fstreamparamsize = b'>i'
 _fpartheadersize = b'>i'
 _fparttypesize = b'>B'
 _fpartid = b'>I'
 _fpayloadsize = b'>i'
 _fpartparamcount = b'>BB'
 
 preferedchunksize = 32768
 
 _parttypeforbidden = re.compile(b'[^a-zA-Z0-9_:-]')
 
 
 def outdebug(ui, message):
     """debug regarding output stream (bundling)"""
     if ui.configbool(b'devel', b'bundle2.debug'):
         ui.debug(b'bundle2-output: %s\n' % message)
 
 
 def indebug(ui, message):
     """debug on input stream (unbundling)"""
     if ui.configbool(b'devel', b'bundle2.debug'):
         ui.debug(b'bundle2-input: %s\n' % message)
 
 
 def validateparttype(parttype):
     """raise ValueError if a parttype contains invalid character"""
     if _parttypeforbidden.search(parttype):
         raise ValueError(parttype)
 
 
 def _makefpartparamsizes(nbparams):
     """return a struct format to read part parameter sizes
 
     The number parameters is variable so we need to build that format
     dynamically.
     """
     return b'>' + (b'BB' * nbparams)
 
 
 parthandlermapping = {}
 
 
 def parthandler(parttype, params=()):
     """decorator that register a function as a bundle2 part handler
 
     eg::
 
         @parthandler('myparttype', ('mandatory', 'param', 'handled'))
         def myparttypehandler(...):
             '''process a part of type "my part".'''
             ...
     """
     validateparttype(parttype)
 
     def _decorator(func):
         lparttype = parttype.lower()  # enforce lower case matching.
         assert lparttype not in parthandlermapping
         parthandlermapping[lparttype] = func
         func.params = frozenset(params)
         return func
 
     return _decorator
 
 
 class unbundlerecords:
     """keep record of what happens during and unbundle
 
     New records are added using `records.add('cat', obj)`. Where 'cat' is a
     category of record and obj is an arbitrary object.
 
     `records['cat']` will return all entries of this category 'cat'.
 
     Iterating on the object itself will yield `('category', obj)` tuples
     for all entries.
 
     All iterations happens in chronological order.
     """
 
     def __init__(self):
         self._categories = {}
         self._sequences = []
         self._replies = {}
 
     def add(self, category, entry, inreplyto=None):
         """add a new record of a given category.
 
         The entry can then be retrieved in the list returned by
         self['category']."""
         self._categories.setdefault(category, []).append(entry)
         self._sequences.append((category, entry))
         if inreplyto is not None:
             self.getreplies(inreplyto).add(category, entry)
 
     def getreplies(self, partid):
         """get the records that are replies to a specific part"""
         return self._replies.setdefault(partid, unbundlerecords())
 
     def __getitem__(self, cat):
         return tuple(self._categories.get(cat, ()))
 
     def __iter__(self):
         return iter(self._sequences)
 
     def __len__(self):
         return len(self._sequences)
 
     def __nonzero__(self):
         return bool(self._sequences)
 
     __bool__ = __nonzero__
 
 
 class bundleoperation:
     """an object that represents a single bundling process
 
     Its purpose is to carry unbundle-related objects and states.
 
     A new object should be created at the beginning of each bundle processing.
     The object is to be returned by the processing function.
 
     The object has very little content now it will ultimately contain:
     * an access to the repo the bundle is applied to,
     * a ui object,
     * a way to retrieve a transaction to add changes to the repo,
     * a way to record the result of processing each part,
     * a way to construct a bundle response when applicable.
     """
 
     def __init__(
         self,
         repo,
         transactiongetter,
         captureoutput=True,
         source=b'',
         remote=None,
     ):
         self.repo = repo
         # the peer object who produced this bundle if available
         self.remote = remote
         self.ui = repo.ui
         self.records = unbundlerecords()
         self.reply = None
         self.captureoutput = captureoutput
         self.hookargs = {}
         self._gettransaction = transactiongetter
         # carries value that can modify part behavior
         self.modes = {}
         self.source = source
 
     def gettransaction(self):
         transaction = self._gettransaction()
 
         if self.hookargs:
             # the ones added to the transaction supercede those added
             # to the operation.
             self.hookargs.update(transaction.hookargs)
             transaction.hookargs = self.hookargs
 
         # mark the hookargs as flushed.  further attempts to add to
         # hookargs will result in an abort.
         self.hookargs = None
 
         return transaction
 
     def addhookargs(self, hookargs):
         if self.hookargs is None:
             raise error.ProgrammingError(
                 b'attempted to add hookargs to '
                 b'operation after transaction started'
             )
         self.hookargs.update(hookargs)
 
 
 class TransactionUnavailable(RuntimeError):
     pass
 
 
 def _notransaction():
     """default method to get a transaction while processing a bundle
 
     Raise an exception to highlight the fact that no transaction was expected
     to be created"""
     raise TransactionUnavailable()
 
 
 def applybundle(repo, unbundler, tr, source, url=None, remote=None, **kwargs):
     # transform me into unbundler.apply() as soon as the freeze is lifted
     if isinstance(unbundler, unbundle20):
         tr.hookargs[b'bundle2'] = b'1'
         if source is not None and b'source' not in tr.hookargs:
             tr.hookargs[b'source'] = source
         if url is not None and b'url' not in tr.hookargs:
             tr.hookargs[b'url'] = url
         return processbundle(
             repo, unbundler, lambda: tr, source=source, remote=remote
         )
     else:
         # the transactiongetter won't be used, but we might as well set it
         op = bundleoperation(repo, lambda: tr, source=source, remote=remote)
         _processchangegroup(op, unbundler, tr, source, url, **kwargs)
         return op
 
 
 class partiterator:
     def __init__(self, repo, op, unbundler):
         self.repo = repo
         self.op = op
         self.unbundler = unbundler
         self.iterator = None
         self.count = 0
         self.current = None
 
     def __enter__(self):
         def func():
             itr = enumerate(self.unbundler.iterparts(), 1)
             for count, p in itr:
                 self.count = count
                 self.current = p
                 yield p
                 p.consume()
                 self.current = None
 
         self.iterator = func()
         return self.iterator
 
     def __exit__(self, type, exc, tb):
         if not self.iterator:
             return
 
         # Only gracefully abort in a normal exception situation. User aborts
         # like Ctrl+C throw a KeyboardInterrupt which is not a base Exception,
         # and should not gracefully cleanup.
         if isinstance(exc, Exception):
             # Any exceptions seeking to the end of the bundle at this point are
             # almost certainly related to the underlying stream being bad.
             # And, chances are that the exception we're handling is related to
             # getting in that bad state. So, we swallow the seeking error and
             # re-raise the original error.
             seekerror = False
             try:
                 if self.current:
                     # consume the part content to not corrupt the stream.
                     self.current.consume()
 
                 for part in self.iterator:
                     # consume the bundle content
                     part.consume()
             except Exception:
                 seekerror = True
 
             # Small hack to let caller code distinguish exceptions from bundle2
             # processing from processing the old format. This is mostly needed
             # to handle different return codes to unbundle according to the type
             # of bundle. We should probably clean up or drop this return code
             # craziness in a future version.
             exc.duringunbundle2 = True
             salvaged = []
             replycaps = None
             if self.op.reply is not None:
                 salvaged = self.op.reply.salvageoutput()
                 replycaps = self.op.reply.capabilities
             exc._replycaps = replycaps
             exc._bundle2salvagedoutput = salvaged
 
             # Re-raising from a variable loses the original stack. So only use
             # that form if we need to.
             if seekerror:
                 raise exc
 
         self.repo.ui.debug(
             b'bundle2-input-bundle: %i parts total\n' % self.count
         )
 
 
 def processbundle(
     repo,
     unbundler,
     transactiongetter=None,
     op=None,
     source=b'',
     remote=None,
 ):
     """This function process a bundle, apply effect to/from a repo
 
     It iterates over each part then searches for and uses the proper handling
     code to process the part. Parts are processed in order.
 
     Unknown Mandatory part will abort the process.
 
     It is temporarily possible to provide a prebuilt bundleoperation to the
     function. This is used to ensure output is properly propagated in case of
     an error during the unbundling. This output capturing part will likely be
     reworked and this ability will probably go away in the process.
     """
     if op is None:
         if transactiongetter is None:
             transactiongetter = _notransaction
         op = bundleoperation(
             repo,
             transactiongetter,
             source=source,
             remote=remote,
         )
     # todo:
     # - replace this is a init function soon.
     # - exception catching
     unbundler.params
     if repo.ui.debugflag:
         msg = [b'bundle2-input-bundle:']
         if unbundler.params:
             msg.append(b' %i params' % len(unbundler.params))
         if op._gettransaction is None or op._gettransaction is _notransaction:
             msg.append(b' no-transaction')
         else:
             msg.append(b' with-transaction')
         msg.append(b'\n')
         repo.ui.debug(b''.join(msg))
 
     processparts(repo, op, unbundler)
 
     return op
 
 
 def processparts(repo, op, unbundler):
     with partiterator(repo, op, unbundler) as parts:
         for part in parts:
             _processpart(op, part)
 
 
 def _processchangegroup(op, cg, tr, source, url, **kwargs):
     if op.remote is not None and op.remote.path is not None:
         remote_path = op.remote.path
         kwargs = kwargs.copy()
         kwargs['delta_base_reuse_policy'] = remote_path.delta_reuse_policy
     ret = cg.apply(op.repo, tr, source, url, **kwargs)
     op.records.add(
         b'changegroup',
         {
             b'return': ret,
         },
     )
     return ret
 
 
 def _gethandler(op, part):
     status = b'unknown'  # used by debug output
     try:
         handler = parthandlermapping.get(part.type)
         if handler is None:
             status = b'unsupported-type'
             raise error.BundleUnknownFeatureError(parttype=part.type)
         indebug(op.ui, b'found a handler for part %s' % part.type)
         unknownparams = part.mandatorykeys - handler.params
         if unknownparams:
             unknownparams = list(unknownparams)
             unknownparams.sort()
             status = b'unsupported-params (%s)' % b', '.join(unknownparams)
             raise error.BundleUnknownFeatureError(
                 parttype=part.type, params=unknownparams
             )
         status = b'supported'
     except error.BundleUnknownFeatureError as exc:
         if part.mandatory:  # mandatory parts
             raise
         indebug(op.ui, b'ignoring unsupported advisory part %s' % exc)
         return  # skip to part processing
     finally:
         if op.ui.debugflag:
             msg = [b'bundle2-input-part: "%s"' % part.type]
             if not part.mandatory:
                 msg.append(b' (advisory)')
             nbmp = len(part.mandatorykeys)
             nbap = len(part.params) - nbmp
             if nbmp or nbap:
                 msg.append(b' (params:')
                 if nbmp:
                     msg.append(b' %i mandatory' % nbmp)
                 if nbap:
                     msg.append(b' %i advisory' % nbmp)
                 msg.append(b')')
             msg.append(b' %s\n' % status)
             op.ui.debug(b''.join(msg))
 
     return handler
 
 
 def _processpart(op, part):
     """process a single part from a bundle
 
     The part is guaranteed to have been fully consumed when the function exits
     (even if an exception is raised)."""
     handler = _gethandler(op, part)
     if handler is None:
         return
 
     # handler is called outside the above try block so that we don't
     # risk catching KeyErrors from anything other than the
     # parthandlermapping lookup (any KeyError raised by handler()
     # itself represents a defect of a different variety).
     output = None
     if op.captureoutput and op.reply is not None:
         op.ui.pushbuffer(error=True, subproc=True)
         output = b''
     try:
         handler(op, part)
     finally:
         if output is not None:
             output = op.ui.popbuffer()
         if output:
             outpart = op.reply.newpart(b'output', data=output, mandatory=False)
             outpart.addparam(
                 b'in-reply-to', pycompat.bytestr(part.id), mandatory=False
             )
 
 
 def decodecaps(blob):
     """decode a bundle2 caps bytes blob into a dictionary
 
     The blob is a list of capabilities (one per line)
     Capabilities may have values using a line of the form::
 
         capability=value1,value2,value3
 
     The values are always a list."""
     caps = {}
     for line in blob.splitlines():
         if not line:
             continue
         if b'=' not in line:
             key, vals = line, ()
         else:
             key, vals = line.split(b'=', 1)
             vals = vals.split(b',')
         key = urlreq.unquote(key)
         vals = [urlreq.unquote(v) for v in vals]
         caps[key] = vals
     return caps
 
 
 def encodecaps(caps):
     """encode a bundle2 caps dictionary into a bytes blob"""
     chunks = []
     for ca in sorted(caps):
         vals = caps[ca]
         ca = urlreq.quote(ca)
         vals = [urlreq.quote(v) for v in vals]
         if vals:
             ca = b"%s=%s" % (ca, b','.join(vals))
         chunks.append(ca)
     return b'\n'.join(chunks)
 
 
 bundletypes = {
     b"": (b"", b'UN'),  # only when using unbundle on ssh and old http servers
     # since the unification ssh accepts a header but there
     # is no capability signaling it.
     b"HG20": (),  # special-cased below
     b"HG10UN": (b"HG10UN", b'UN'),
     b"HG10BZ": (b"HG10", b'BZ'),
     b"HG10GZ": (b"HG10GZ", b'GZ'),
 }
 
 # hgweb uses this list to communicate its preferred type
 bundlepriority = [b'HG10GZ', b'HG10BZ', b'HG10UN']
 
 
 class bundle20:
     """represent an outgoing bundle2 container
 
     Use the `addparam` method to add stream level parameter. and `newpart` to
     populate it. Then call `getchunks` to retrieve all the binary chunks of
     data that compose the bundle2 container."""
 
     _magicstring = b'HG20'
 
     def __init__(self, ui, capabilities=()):
         self.ui = ui
         self._params = []
         self._parts = []
         self.capabilities = dict(capabilities)
         self._compengine = util.compengines.forbundletype(b'UN')
         self._compopts = None
         # If compression is being handled by a consumer of the raw
         # data (e.g. the wire protocol), unsetting this flag tells
         # consumers that the bundle is best left uncompressed.
         self.prefercompressed = True
 
     def setcompression(self, alg, compopts=None):
         """setup core part compression to <alg>"""
         if alg in (None, b'UN'):
             return
         assert not any(n.lower() == b'compression' for n, v in self._params)
         self.addparam(b'Compression', alg)
         self._compengine = util.compengines.forbundletype(alg)
         self._compopts = compopts
 
     @property
     def nbparts(self):
         """total number of parts added to the bundler"""
         return len(self._parts)
 
     # methods used to defines the bundle2 content
     def addparam(self, name, value=None):
         """add a stream level parameter"""
         if not name:
             raise error.ProgrammingError(b'empty parameter name')
         if name[0:1] not in pycompat.bytestr(
             string.ascii_letters  # pytype: disable=wrong-arg-types
         ):
             raise error.ProgrammingError(
                 b'non letter first character: %s' % name
             )
         self._params.append((name, value))
 
     def addpart(self, part):
         """add a new part to the bundle2 container
 
         Parts contains the actual applicative payload."""
         assert part.id is None
         part.id = len(self._parts)  # very cheap counter
         self._parts.append(part)
 
     def newpart(self, typeid, *args, **kwargs):
         """create a new part and add it to the containers
 
         As the part is directly added to the containers. For now, this means
         that any failure to properly initialize the part after calling
         ``newpart`` should result in a failure of the whole bundling process.
 
         You can still fall back to manually create and add if you need better
         control."""
         part = bundlepart(typeid, *args, **kwargs)
         self.addpart(part)
         return part
 
     # methods used to generate the bundle2 stream
     def getchunks(self):
         if self.ui.debugflag:
             msg = [b'bundle2-output-bundle: "%s",' % self._magicstring]
             if self._params:
                 msg.append(b' (%i params)' % len(self._params))
             msg.append(b' %i parts total\n' % len(self._parts))
             self.ui.debug(b''.join(msg))
         outdebug(self.ui, b'start emission of %s stream' % self._magicstring)
         yield self._magicstring
         param = self._paramchunk()
         outdebug(self.ui, b'bundle parameter: %s' % param)
         yield _pack(_fstreamparamsize, len(param))
         if param:
             yield param
         for chunk in self._compengine.compressstream(
             self._getcorechunk(), self._compopts
         ):
             yield chunk
 
     def _paramchunk(self):
         """return a encoded version of all stream parameters"""
         blocks = []
         for par, value in self._params:
             par = urlreq.quote(par)
             if value is not None:
                 value = urlreq.quote(value)
                 par = b'%s=%s' % (par, value)
             blocks.append(par)
         return b' '.join(blocks)
 
     def _getcorechunk(self):
         """yield chunk for the core part of the bundle
 
         (all but headers and parameters)"""
         outdebug(self.ui, b'start of parts')
         for part in self._parts:
             outdebug(self.ui, b'bundle part: "%s"' % part.type)
             for chunk in part.getchunks(ui=self.ui):
                 yield chunk
         outdebug(self.ui, b'end of bundle')
         yield _pack(_fpartheadersize, 0)
 
     def salvageoutput(self):
         """return a list with a copy of all output parts in the bundle
 
         This is meant to be used during error handling to make sure we preserve
         server output"""
         salvaged = []
         for part in self._parts:
             if part.type.startswith(b'output'):
                 salvaged.append(part.copy())
         return salvaged
 
 
 class unpackermixin:
     """A mixin to extract bytes and struct data from a stream"""
 
     def __init__(self, fp):
         self._fp = fp
 
     def _unpack(self, format):
         """unpack this struct format from the stream
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         data = self._readexact(struct.calcsize(format))
         return _unpack(format, data)
 
     def _readexact(self, size):
         """read exactly <size> bytes from the stream
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         return changegroup.readexactly(self._fp, size)
 
 
 def getunbundler(ui, fp, magicstring=None):
     """return a valid unbundler object for a given magicstring"""
     if magicstring is None:
         magicstring = changegroup.readexactly(fp, 4)
     magic, version = magicstring[0:2], magicstring[2:4]
     if magic != b'HG':
         ui.debug(
             b"error: invalid magic: %r (version %r), should be 'HG'\n"
             % (magic, version)
         )
         raise error.Abort(_(b'not a Mercurial bundle'))
     unbundlerclass = formatmap.get(version)
     if unbundlerclass is None:
         raise error.Abort(_(b'unknown bundle version %s') % version)
     unbundler = unbundlerclass(ui, fp)
     indebug(ui, b'start processing of %s stream' % magicstring)
     return unbundler
 
 
 class unbundle20(unpackermixin):
     """interpret a bundle2 stream
 
     This class is fed with a binary stream and yields parts through its
     `iterparts` methods."""
 
     _magicstring = b'HG20'
 
     def __init__(self, ui, fp):
         """If header is specified, we do not read it out of the stream."""
         self.ui = ui
         self._compengine = util.compengines.forbundletype(b'UN')
         self._compressed = None
         super(unbundle20, self).__init__(fp)
 
     @util.propertycache
     def params(self):
         """dictionary of stream level parameters"""
         indebug(self.ui, b'reading bundle2 stream parameters')
         params = {}
         paramssize = self._unpack(_fstreamparamsize)[0]
         if paramssize < 0:
             raise error.BundleValueError(
                 b'negative bundle param size: %i' % paramssize
             )
         if paramssize:
             params = self._readexact(paramssize)
             params = self._processallparams(params)
         return params
 
     def _processallparams(self, paramsblock):
         """ """
         params = util.sortdict()
         for p in paramsblock.split(b' '):
             p = p.split(b'=', 1)
             p = [urlreq.unquote(i) for i in p]
             if len(p) < 2:
                 p.append(None)
             self._processparam(*p)
             params[p[0]] = p[1]
         return params
 
     def _processparam(self, name, value):
         """process a parameter, applying its effect if needed
 
         Parameter starting with a lower case letter are advisory and will be
         ignored when unknown.  Those starting with an upper case letter are
         mandatory and will this function will raise a KeyError when unknown.
 
         Note: no option are currently supported. Any input will be either
               ignored or failing.
         """
         if not name:
             raise ValueError('empty parameter name')
         if name[0:1] not in pycompat.bytestr(
             string.ascii_letters  # pytype: disable=wrong-arg-types
         ):
             raise ValueError('non letter first character: %s' % name)
         try:
             handler = b2streamparamsmap[name.lower()]
         except KeyError:
             if name[0:1].islower():
                 indebug(self.ui, b"ignoring unknown parameter %s" % name)
             else:
                 raise error.BundleUnknownFeatureError(params=(name,))
         else:
             handler(self, name, value)
 
     def _forwardchunks(self):
         """utility to transfer a bundle2 as binary
 
         This is made necessary by the fact the 'getbundle' command over 'ssh'
         have no way to know then the reply end, relying on the bundle to be
         interpreted to know its end. This is terrible and we are sorry, but we
         needed to move forward to get general delta enabled.
         """
         yield self._magicstring
         assert 'params' not in vars(self)
         paramssize = self._unpack(_fstreamparamsize)[0]
         if paramssize < 0:
             raise error.BundleValueError(
                 b'negative bundle param size: %i' % paramssize
             )
         if paramssize:
             params = self._readexact(paramssize)
             self._processallparams(params)
             # The payload itself is decompressed below, so drop
             # the compression parameter passed down to compensate.
             outparams = []
             for p in params.split(b' '):
                 k, v = p.split(b'=', 1)
                 if k.lower() != b'compression':
                     outparams.append(p)
             outparams = b' '.join(outparams)
             yield _pack(_fstreamparamsize, len(outparams))
             yield outparams
         else:
             yield _pack(_fstreamparamsize, paramssize)
         # From there, payload might need to be decompressed
         self._fp = self._compengine.decompressorreader(self._fp)
         emptycount = 0
         while emptycount < 2:
             # so we can brainlessly loop
             assert _fpartheadersize == _fpayloadsize
             size = self._unpack(_fpartheadersize)[0]
             yield _pack(_fpartheadersize, size)
             if size:
                 emptycount = 0
             else:
                 emptycount += 1
                 continue
             if size == flaginterrupt:
                 continue
             elif size < 0:
                 raise error.BundleValueError(b'negative chunk size: %i')
             yield self._readexact(size)
 
     def iterparts(self, seekable=False):
         """yield all parts contained in the stream"""
         cls = seekableunbundlepart if seekable else unbundlepart
         # make sure param have been loaded
         self.params
         # From there, payload need to be decompressed
         self._fp = self._compengine.decompressorreader(self._fp)
         indebug(self.ui, b'start extraction of bundle2 parts')
         headerblock = self._readpartheader()
         while headerblock is not None:
             part = cls(self.ui, headerblock, self._fp)
             yield part
             # Ensure part is fully consumed so we can start reading the next
             # part.
             part.consume()
 
             headerblock = self._readpartheader()
         indebug(self.ui, b'end of bundle2 stream')
 
     def _readpartheader(self):
         """reads a part header size and return the bytes blob
 
         returns None if empty"""
         headersize = self._unpack(_fpartheadersize)[0]
         if headersize < 0:
             raise error.BundleValueError(
                 b'negative part header size: %i' % headersize
             )
         indebug(self.ui, b'part header size: %i' % headersize)
         if headersize:
             return self._readexact(headersize)
         return None
 
     def compressed(self):
         self.params  # load params
         return self._compressed
 
     def close(self):
         """close underlying file"""
         if util.safehasattr(self._fp, 'close'):
             return self._fp.close()
 
 
 formatmap = {b'20': unbundle20}
 
 b2streamparamsmap = {}
 
 
 def b2streamparamhandler(name):
     """register a handler for a stream level parameter"""
 
     def decorator(func):
         assert name not in formatmap
         b2streamparamsmap[name] = func
         return func
 
     return decorator
 
 
 @b2streamparamhandler(b'compression')
 def processcompression(unbundler, param, value):
     """read compression parameter and install payload decompression"""
     if value not in util.compengines.supportedbundletypes:
         raise error.BundleUnknownFeatureError(params=(param,), values=(value,))
     unbundler._compengine = util.compengines.forbundletype(value)
     if value is not None:
         unbundler._compressed = True
 
 
 class bundlepart:
     """A bundle2 part contains application level payload
 
     The part `type` is used to route the part to the application level
     handler.
 
     The part payload is contained in ``part.data``. It could be raw bytes or a
     generator of byte chunks.
 
     You can add parameters to the part using the ``addparam`` method.
     Parameters can be either mandatory (default) or advisory. Remote side
     should be able to safely ignore the advisory ones.
 
     Both data and parameters cannot be modified after the generation has begun.
     """
 
     def __init__(
         self,
         parttype,
         mandatoryparams=(),
         advisoryparams=(),
         data=b'',
         mandatory=True,
     ):
         validateparttype(parttype)
         self.id = None
         self.type = parttype
         self._data = data
         self._mandatoryparams = list(mandatoryparams)
         self._advisoryparams = list(advisoryparams)
         # checking for duplicated entries
         self._seenparams = set()
         for pname, __ in self._mandatoryparams + self._advisoryparams:
             if pname in self._seenparams:
                 raise error.ProgrammingError(b'duplicated params: %s' % pname)
             self._seenparams.add(pname)
         # status of the part's generation:
         # - None: not started,
         # - False: currently generated,
         # - True: generation done.
         self._generated = None
         self.mandatory = mandatory
 
     def __repr__(self):
         cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__)
         return '<%s object at %x; id: %s; type: %s; mandatory: %s>' % (
             cls,
             id(self),
             self.id,
             self.type,
             self.mandatory,
         )
 
     def copy(self):
         """return a copy of the part
 
         The new part have the very same content but no partid assigned yet.
         Parts with generated data cannot be copied."""
         assert not util.safehasattr(self.data, 'next')
         return self.__class__(
             self.type,
             self._mandatoryparams,
             self._advisoryparams,
             self._data,
             self.mandatory,
         )
 
     # methods used to defines the part content
     @property
     def data(self):
         return self._data
 
     @data.setter
     def data(self, data):
         if self._generated is not None:
             raise error.ReadOnlyPartError(b'part is being generated')
         self._data = data
 
     @property
     def mandatoryparams(self):
         # make it an immutable tuple to force people through ``addparam``
         return tuple(self._mandatoryparams)
 
     @property
     def advisoryparams(self):
         # make it an immutable tuple to force people through ``addparam``
         return tuple(self._advisoryparams)
 
     def addparam(self, name, value=b'', mandatory=True):
         """add a parameter to the part
 
         If 'mandatory' is set to True, the remote handler must claim support
         for this parameter or the unbundling will be aborted.
 
         The 'name' and 'value' cannot exceed 255 bytes each.
         """
         if self._generated is not None:
             raise error.ReadOnlyPartError(b'part is being generated')
         if name in self._seenparams:
             raise ValueError(b'duplicated params: %s' % name)
         self._seenparams.add(name)
         params = self._advisoryparams
         if mandatory:
             params = self._mandatoryparams
         params.append((name, value))
 
     # methods used to generates the bundle2 stream
     def getchunks(self, ui):
         if self._generated is not None:
             raise error.ProgrammingError(b'part can only be consumed once')
         self._generated = False
 
         if ui.debugflag:
             msg = [b'bundle2-output-part: "%s"' % self.type]
             if not self.mandatory:
                 msg.append(b' (advisory)')
             nbmp = len(self.mandatoryparams)
             nbap = len(self.advisoryparams)
             if nbmp or nbap:
                 msg.append(b' (params:')
                 if nbmp:
                     msg.append(b' %i mandatory' % nbmp)
                 if nbap:
                     msg.append(b' %i advisory' % nbmp)
                 msg.append(b')')
             if not self.data:
                 msg.append(b' empty payload')
             elif util.safehasattr(self.data, 'next') or util.safehasattr(
                 self.data, b'__next__'
             ):
                 msg.append(b' streamed payload')
             else:
                 msg.append(b' %i bytes payload' % len(self.data))
             msg.append(b'\n')
             ui.debug(b''.join(msg))
 
         #### header
         if self.mandatory:
             parttype = self.type.upper()
         else:
             parttype = self.type.lower()
         outdebug(ui, b'part %s: "%s"' % (pycompat.bytestr(self.id), parttype))
         ## parttype
         header = [
             _pack(_fparttypesize, len(parttype)),
             parttype,
             _pack(_fpartid, self.id),
         ]
         ## parameters
         # count
         manpar = self.mandatoryparams
         advpar = self.advisoryparams
         header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
         # size
         parsizes = []
         for key, value in manpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         for key, value in advpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes)
         header.append(paramsizes)
         # key, value
         for key, value in manpar:
             header.append(key)
             header.append(value)
         for key, value in advpar:
             header.append(key)
             header.append(value)
         ## finalize header
         try:
             headerchunk = b''.join(header)
         except TypeError:
             raise TypeError(
                 'Found a non-bytes trying to '
                 'build bundle part header: %r' % header
             )
         outdebug(ui, b'header chunk size: %i' % len(headerchunk))
         yield _pack(_fpartheadersize, len(headerchunk))
         yield headerchunk
         ## payload
         try:
             for chunk in self._payloadchunks():
                 outdebug(ui, b'payload chunk size: %i' % len(chunk))
                 yield _pack(_fpayloadsize, len(chunk))
                 yield chunk
         except GeneratorExit:
             # GeneratorExit means that nobody is listening for our
             # results anyway, so just bail quickly rather than trying
             # to produce an error part.
             ui.debug(b'bundle2-generatorexit\n')
             raise
         except BaseException as exc:
             bexc = stringutil.forcebytestr(exc)
             # backup exception data for later
             ui.debug(
                 b'bundle2-input-stream-interrupt: encoding exception %s' % bexc
             )
             tb = sys.exc_info()[2]
             msg = b'unexpected error: %s' % bexc
             interpart = bundlepart(
                 b'error:abort', [(b'message', msg)], mandatory=False
             )
             interpart.id = 0
             yield _pack(_fpayloadsize, -1)
             for chunk in interpart.getchunks(ui=ui):
                 yield chunk
             outdebug(ui, b'closing payload chunk')
             # abort current part payload
             yield _pack(_fpayloadsize, 0)
             pycompat.raisewithtb(exc, tb)
         # end of payload
         outdebug(ui, b'closing payload chunk')
         yield _pack(_fpayloadsize, 0)
         self._generated = True
 
     def _payloadchunks(self):
         """yield chunks of a the part payload
 
         Exists to handle the different methods to provide data to a part."""
         # we only support fixed size data now.
         # This will be improved in the future.
         if util.safehasattr(self.data, 'next') or util.safehasattr(
             self.data, b'__next__'
         ):
             buff = util.chunkbuffer(self.data)
             chunk = buff.read(preferedchunksize)
             while chunk:
                 yield chunk
                 chunk = buff.read(preferedchunksize)
         elif len(self.data):
             yield self.data
 
 
 flaginterrupt = -1
 
 
 class interrupthandler(unpackermixin):
     """read one part and process it with restricted capability
 
     This allows to transmit exception raised on the producer size during part
     iteration while the consumer is reading a part.
 
     Part processed in this manner only have access to a ui object,"""
 
     def __init__(self, ui, fp):
         super(interrupthandler, self).__init__(fp)
         self.ui = ui
 
     def _readpartheader(self):
         """reads a part header size and return the bytes blob
 
         returns None if empty"""
         headersize = self._unpack(_fpartheadersize)[0]
         if headersize < 0:
             raise error.BundleValueError(
                 b'negative part header size: %i' % headersize
             )
         indebug(self.ui, b'part header size: %i\n' % headersize)
         if headersize:
             return self._readexact(headersize)
         return None
 
     def __call__(self):
 
         self.ui.debug(
             b'bundle2-input-stream-interrupt: opening out of band context\n'
         )
         indebug(self.ui, b'bundle2 stream interruption, looking for a part.')
         headerblock = self._readpartheader()
         if headerblock is None:
             indebug(self.ui, b'no part found during interruption.')
             return
         part = unbundlepart(self.ui, headerblock, self._fp)
         op = interruptoperation(self.ui)
         hardabort = False
         try:
             _processpart(op, part)
         except (SystemExit, KeyboardInterrupt):
             hardabort = True
             raise
         finally:
             if not hardabort:
                 part.consume()
         self.ui.debug(
             b'bundle2-input-stream-interrupt: closing out of band context\n'
         )
 
 
 class interruptoperation:
     """A limited operation to be use by part handler during interruption
 
     It only have access to an ui object.
     """
 
     def __init__(self, ui):
         self.ui = ui
         self.reply = None
         self.captureoutput = False
 
     @property
     def repo(self):
         raise error.ProgrammingError(b'no repo access from stream interruption')
 
     def gettransaction(self):
         raise TransactionUnavailable(b'no repo access from stream interruption')
 
 
 def decodepayloadchunks(ui, fh):
     """Reads bundle2 part payload data into chunks.
 
     Part payload data consists of framed chunks. This function takes
     a file handle and emits those chunks.
     """
     dolog = ui.configbool(b'devel', b'bundle2.debug')
     debug = ui.debug
 
     headerstruct = struct.Struct(_fpayloadsize)
     headersize = headerstruct.size
     unpack = headerstruct.unpack
 
     readexactly = changegroup.readexactly
     read = fh.read
 
     chunksize = unpack(readexactly(fh, headersize))[0]
     indebug(ui, b'payload chunk size: %i' % chunksize)
 
     # changegroup.readexactly() is inlined below for performance.
     while chunksize:
         if chunksize >= 0:
             s = read(chunksize)
             if len(s) < chunksize:
                 raise error.Abort(
                     _(
                         b'stream ended unexpectedly '
                         b' (got %d bytes, expected %d)'
                     )
                     % (len(s), chunksize)
                 )
 
             yield s
         elif chunksize == flaginterrupt:
             # Interrupt "signal" detected. The regular stream is interrupted
             # and a bundle2 part follows. Consume it.
             interrupthandler(ui, fh)()
         else:
             raise error.BundleValueError(
                 b'negative payload chunk size: %s' % chunksize
             )
 
         s = read(headersize)
         if len(s) < headersize:
             raise error.Abort(
                 _(b'stream ended unexpectedly  (got %d bytes, expected %d)')
                 % (len(s), chunksize)
             )
 
         chunksize = unpack(s)[0]
 
         # indebug() inlined for performance.
         if dolog:
             debug(b'bundle2-input: payload chunk size: %i\n' % chunksize)
 
 
 class unbundlepart(unpackermixin):
     """a bundle part read from a bundle"""
 
     def __init__(self, ui, header, fp):
         super(unbundlepart, self).__init__(fp)
         self._seekable = util.safehasattr(fp, 'seek') and util.safehasattr(
             fp, b'tell'
         )
         self.ui = ui
         # unbundle state attr
         self._headerdata = header
         self._headeroffset = 0
         self._initialized = False
         self.consumed = False
         # part data
         self.id = None
         self.type = None
         self.mandatoryparams = None
         self.advisoryparams = None
         self.params = None
         self.mandatorykeys = ()
         self._readheader()
         self._mandatory = None
         self._pos = 0
 
     def _fromheader(self, size):
         """return the next <size> byte from the header"""
         offset = self._headeroffset
         data = self._headerdata[offset : (offset + size)]
         self._headeroffset = offset + size
         return data
 
     def _unpackheader(self, format):
         """read given format from header
 
         This automatically compute the size of the format to read."""
         data = self._fromheader(struct.calcsize(format))
         return _unpack(format, data)
 
     def _initparams(self, mandatoryparams, advisoryparams):
         """internal function to setup all logic related parameters"""
         # make it read only to prevent people touching it by mistake.
         self.mandatoryparams = tuple(mandatoryparams)
         self.advisoryparams = tuple(advisoryparams)
         # user friendly UI
         self.params = util.sortdict(self.mandatoryparams)
         self.params.update(self.advisoryparams)
         self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
 
     def _readheader(self):
         """read the header and setup the object"""
         typesize = self._unpackheader(_fparttypesize)[0]
         self.type = self._fromheader(typesize)
         indebug(self.ui, b'part type: "%s"' % self.type)
         self.id = self._unpackheader(_fpartid)[0]
         indebug(self.ui, b'part id: "%s"' % pycompat.bytestr(self.id))
         # extract mandatory bit from type
         self.mandatory = self.type != self.type.lower()
         self.type = self.type.lower()
         ## reading parameters
         # param count
         mancount, advcount = self._unpackheader(_fpartparamcount)
         indebug(self.ui, b'part parameters: %i' % (mancount + advcount))
         # param size
         fparamsizes = _makefpartparamsizes(mancount + advcount)
         paramsizes = self._unpackheader(fparamsizes)
         # make it a list of couple again
         paramsizes = list(zip(paramsizes[::2], paramsizes[1::2]))
         # split mandatory from advisory
         mansizes = paramsizes[:mancount]
         advsizes = paramsizes[mancount:]
         # retrieve param value
         manparams = []
         for key, value in mansizes:
             manparams.append((self._fromheader(key), self._fromheader(value)))
         advparams = []
         for key, value in advsizes:
             advparams.append((self._fromheader(key), self._fromheader(value)))
         self._initparams(manparams, advparams)
         ## part payload
         self._payloadstream = util.chunkbuffer(self._payloadchunks())
         # we read the data, tell it
         self._initialized = True
 
     def _payloadchunks(self):
         """Generator of decoded chunks in the payload."""
         return decodepayloadchunks(self.ui, self._fp)
 
     def consume(self):
         """Read the part payload until completion.
 
         By consuming the part data, the underlying stream read offset will
         be advanced to the next part (or end of stream).
         """
         if self.consumed:
             return
 
         chunk = self.read(32768)
         while chunk:
             self._pos += len(chunk)
             chunk = self.read(32768)
 
     def read(self, size=None):
         """read payload data"""
         if not self._initialized:
             self._readheader()
         if size is None:
             data = self._payloadstream.read()
         else:
             data = self._payloadstream.read(size)
         self._pos += len(data)
         if size is None or len(data) < size:
             if not self.consumed and self._pos:
                 self.ui.debug(
                     b'bundle2-input-part: total payload size %i\n' % self._pos
                 )
             self.consumed = True
         return data
 
 
 class seekableunbundlepart(unbundlepart):
     """A bundle2 part in a bundle that is seekable.
 
     Regular ``unbundlepart`` instances can only be read once. This class
     extends ``unbundlepart`` to enable bi-directional seeking within the
     part.
 
     Bundle2 part data consists of framed chunks. Offsets when seeking
     refer to the decoded data, not the offsets in the underlying bundle2
     stream.
 
     To facilitate quickly seeking within the decoded data, instances of this
     class maintain a mapping between offsets in the underlying stream and
     the decoded payload. This mapping will consume memory in proportion
     to the number of chunks within the payload (which almost certainly
     increases in proportion with the size of the part).
     """
 
     def __init__(self, ui, header, fp):
         # (payload, file) offsets for chunk starts.
         self._chunkindex = []
 
         super(seekableunbundlepart, self).__init__(ui, header, fp)
 
     def _payloadchunks(self, chunknum=0):
         '''seek to specified chunk and start yielding data'''
         if len(self._chunkindex) == 0:
             assert chunknum == 0, b'Must start with chunk 0'
             self._chunkindex.append((0, self._tellfp()))
         else:
             assert chunknum < len(self._chunkindex), (
                 b'Unknown chunk %d' % chunknum
             )
             self._seekfp(self._chunkindex[chunknum][1])
 
         pos = self._chunkindex[chunknum][0]
 
         for chunk in decodepayloadchunks(self.ui, self._fp):
             chunknum += 1
             pos += len(chunk)
             if chunknum == len(self._chunkindex):
                 self._chunkindex.append((pos, self._tellfp()))
 
             yield chunk
 
     def _findchunk(self, pos):
         '''for a given payload position, return a chunk number and offset'''
         for chunk, (ppos, fpos) in enumerate(self._chunkindex):
             if ppos == pos:
                 return chunk, 0
             elif ppos > pos:
                 return chunk - 1, pos - self._chunkindex[chunk - 1][0]
         raise ValueError(b'Unknown chunk')
 
     def tell(self):
         return self._pos
 
     def seek(self, offset, whence=os.SEEK_SET):
         if whence == os.SEEK_SET:
             newpos = offset
         elif whence == os.SEEK_CUR:
             newpos = self._pos + offset
         elif whence == os.SEEK_END:
             if not self.consumed:
                 # Can't use self.consume() here because it advances self._pos.
                 chunk = self.read(32768)
                 while chunk:
                     chunk = self.read(32768)
             newpos = self._chunkindex[-1][0] - offset
         else:
             raise ValueError(b'Unknown whence value: %r' % (whence,))
 
         if newpos > self._chunkindex[-1][0] and not self.consumed:
             # Can't use self.consume() here because it advances self._pos.
             chunk = self.read(32768)
             while chunk:
                 chunk = self.read(32668)
 
         if not 0 <= newpos <= self._chunkindex[-1][0]:
             raise ValueError(b'Offset out of range')
 
         if self._pos != newpos:
             chunk, internaloffset = self._findchunk(newpos)
             self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk))
             adjust = self.read(internaloffset)
             if len(adjust) != internaloffset:
                 raise error.Abort(_(b'Seek failed\n'))
             self._pos = newpos
 
     def _seekfp(self, offset, whence=0):
         """move the underlying file pointer
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         if self._seekable:
             return self._fp.seek(offset, whence)
         else:
             raise NotImplementedError(_(b'File pointer is not seekable'))
 
     def _tellfp(self):
         """return the file offset, or None if file is not seekable
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         if self._seekable:
             try:
                 return self._fp.tell()
             except IOError as e:
                 if e.errno == errno.ESPIPE:
                     self._seekable = False
                 else:
                     raise
         return None
 
 
 # These are only the static capabilities.
 # Check the 'getrepocaps' function for the rest.
 capabilities = {
     b'HG20': (),
     b'bookmarks': (),
     b'error': (b'abort', b'unsupportedcontent', b'pushraced', b'pushkey'),
     b'listkeys': (),
     b'pushkey': (),
     b'digests': tuple(sorted(util.DIGESTS.keys())),
     b'remote-changegroup': (b'http', b'https'),
     b'hgtagsfnodes': (),
     b'phases': (b'heads',),
     b'stream': (b'v2',),
 }
 
 
 def getrepocaps(repo, allowpushback=False, role=None):
     """return the bundle2 capabilities for a given repo
 
     Exists to allow extensions (like evolution) to mutate the capabilities.
 
     The returned value is used for servers advertising their capabilities as
     well as clients advertising their capabilities to servers as part of
     bundle2 requests. The ``role`` argument specifies which is which.
     """
     if role not in (b'client', b'server'):
         raise error.ProgrammingError(b'role argument must be client or server')
 
     caps = capabilities.copy()
     caps[b'changegroup'] = tuple(
         sorted(changegroup.supportedincomingversions(repo))
     )
     if obsolete.isenabled(repo, obsolete.exchangeopt):
         supportedformat = tuple(b'V%i' % v for v in obsolete.formats)
         caps[b'obsmarkers'] = supportedformat
     if allowpushback:
         caps[b'pushback'] = ()
     cpmode = repo.ui.config(b'server', b'concurrent-push-mode')
     if cpmode == b'check-related':
         caps[b'checkheads'] = (b'related',)
     if b'phases' in repo.ui.configlist(b'devel', b'legacy.exchange'):
         caps.pop(b'phases')
 
     # Don't advertise stream clone support in server mode if not configured.
     if role == b'server':
         streamsupported = repo.ui.configbool(
             b'server', b'uncompressed', untrusted=True
         )
         featuresupported = repo.ui.configbool(b'server', b'bundle2.stream')
 
         if not streamsupported or not featuresupported:
             caps.pop(b'stream')
     # Else always advertise support on client, because payload support
     # should always be advertised.
 
+    if repo.ui.configbool(b'experimental', b'stream-v3'):
+        if b'stream' in caps:
+            caps[b'stream'] += (b'v3-exp',)
+
     # b'rev-branch-cache is no longer advertised, but still supported
     # for legacy clients.
 
     return caps
 
 
 def bundle2caps(remote):
     """return the bundle capabilities of a peer as dict"""
     raw = remote.capable(b'bundle2')
     if not raw and raw != b'':
         return {}
     capsblob = urlreq.unquote(remote.capable(b'bundle2'))
     return decodecaps(capsblob)
 
 
 def obsmarkersversion(caps):
     """extract the list of supported obsmarkers versions from a bundle2caps dict"""
     obscaps = caps.get(b'obsmarkers', ())
     return [int(c[1:]) for c in obscaps if c.startswith(b'V')]
 
 
 def writenewbundle(
     ui,
     repo,
     source,
     filename,
     bundletype,
     outgoing,
     opts,
     vfs=None,
     compression=None,
     compopts=None,
     allow_internal=False,
 ):
     if bundletype.startswith(b'HG10'):
         cg = changegroup.makechangegroup(repo, outgoing, b'01', source)
         return writebundle(
             ui,
             cg,
             filename,
             bundletype,
             vfs=vfs,
             compression=compression,
             compopts=compopts,
         )
     elif not bundletype.startswith(b'HG20'):
         raise error.ProgrammingError(b'unknown bundle type: %s' % bundletype)
 
     # enforce that no internal phase are to be bundled
     bundled_internal = repo.revs(b"%ln and _internal()", outgoing.ancestorsof)
     if bundled_internal and not allow_internal:
         count = len(repo.revs(b'%ln and _internal()', outgoing.missing))
         msg = "backup bundle would contains %d internal changesets"
         msg %= count
         raise error.ProgrammingError(msg)
 
     caps = {}
     if opts.get(b'obsolescence', False):
         caps[b'obsmarkers'] = (b'V1',)
     if opts.get(b'streamv2'):
         caps[b'stream'] = [b'v2']
     bundle = bundle20(ui, caps)
     bundle.setcompression(compression, compopts)
     _addpartsfromopts(ui, repo, bundle, source, outgoing, opts)
     chunkiter = bundle.getchunks()
 
     return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
 
 
 def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts):
     # We should eventually reconcile this logic with the one behind
     # 'exchange.getbundle2partsgenerator'.
     #
     # The type of input from 'getbundle' and 'writenewbundle' are a bit
     # different right now. So we keep them separated for now for the sake of
     # simplicity.
 
     # we might not always want a changegroup in such bundle, for example in
     # stream bundles
     if opts.get(b'changegroup', True):
         cgversion = opts.get(b'cg.version')
         if cgversion is None:
             cgversion = changegroup.safeversion(repo)
         cg = changegroup.makechangegroup(repo, outgoing, cgversion, source)
         part = bundler.newpart(b'changegroup', data=cg.getchunks())
         part.addparam(b'version', cg.version)
         if b'clcount' in cg.extras:
             part.addparam(
                 b'nbchanges', b'%d' % cg.extras[b'clcount'], mandatory=False
             )
         if opts.get(b'phases'):
             target_phase = phases.draft
             for head in outgoing.ancestorsof:
                 target_phase = max(target_phase, repo[head].phase())
             if target_phase > phases.draft:
                 part.addparam(
                     b'targetphase',
                     b'%d' % target_phase,
                     mandatory=False,
                 )
     if repository.REPO_FEATURE_SIDE_DATA in repo.features:
         part.addparam(b'exp-sidedata', b'1')
 
     if opts.get(b'streamv2', False):
         addpartbundlestream2(bundler, repo, stream=True)
 
     if opts.get(b'tagsfnodescache', True):
         addparttagsfnodescache(repo, bundler, outgoing)
 
     if opts.get(b'revbranchcache', True):
         addpartrevbranchcache(repo, bundler, outgoing)
 
     if opts.get(b'obsolescence', False):
         obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
         buildobsmarkerspart(
             bundler,
             obsmarkers,
             mandatory=opts.get(b'obsolescence-mandatory', True),
         )
 
     if opts.get(b'phases', False):
         headsbyphase = phases.subsetphaseheads(repo, outgoing.missing)
         phasedata = phases.binaryencode(headsbyphase)
         bundler.newpart(b'phase-heads', data=phasedata)
 
 
 def addparttagsfnodescache(repo, bundler, outgoing):
     # we include the tags fnode cache for the bundle changeset
     # (as an optional parts)
     cache = tags.hgtagsfnodescache(repo.unfiltered())
     chunks = []
 
     # .hgtags fnodes are only relevant for head changesets. While we could
     # transfer values for all known nodes, there will likely be little to
     # no benefit.
     #
     # We don't bother using a generator to produce output data because
     # a) we only have 40 bytes per head and even esoteric numbers of heads
     # consume little memory (1M heads is 40MB) b) we don't want to send the
     # part if we don't have entries and knowing if we have entries requires
     # cache lookups.
     for node in outgoing.ancestorsof:
         # Don't compute missing, as this may slow down serving.
         fnode = cache.getfnode(node, computemissing=False)
         if fnode:
             chunks.extend([node, fnode])
 
     if chunks:
         bundler.newpart(b'hgtagsfnodes', data=b''.join(chunks))
 
 
 def addpartrevbranchcache(repo, bundler, outgoing):
     # we include the rev branch cache for the bundle changeset
     # (as an optional parts)
     cache = repo.revbranchcache()
     cl = repo.unfiltered().changelog
     branchesdata = collections.defaultdict(lambda: (set(), set()))
     for node in outgoing.missing:
         branch, close = cache.branchinfo(cl.rev(node))
         branchesdata[branch][close].add(node)
 
     def generate():
         for branch, (nodes, closed) in sorted(branchesdata.items()):
             utf8branch = encoding.fromlocal(branch)
             yield rbcstruct.pack(len(utf8branch), len(nodes), len(closed))
             yield utf8branch
             for n in sorted(nodes):
                 yield n
             for n in sorted(closed):
                 yield n
 
     bundler.newpart(b'cache:rev-branch-cache', data=generate(), mandatory=False)
 
 
 def _formatrequirementsspec(requirements):
     requirements = [req for req in requirements if req != b"shared"]
     return urlreq.quote(b','.join(sorted(requirements)))
 
 
 def _formatrequirementsparams(requirements):
     requirements = _formatrequirementsspec(requirements)
     params = b"%s%s" % (urlreq.quote(b"requirements="), requirements)
     return params
 
 
 def format_remote_wanted_sidedata(repo):
     """Formats a repo's wanted sidedata categories into a bytestring for
     capabilities exchange."""
     wanted = b""
     if repo._wanted_sidedata:
         wanted = b','.join(
             pycompat.bytestr(c) for c in sorted(repo._wanted_sidedata)
         )
     return wanted
 
 
 def read_remote_wanted_sidedata(remote):
     sidedata_categories = remote.capable(b'exp-wanted-sidedata')
     return read_wanted_sidedata(sidedata_categories)
 
 
 def read_wanted_sidedata(formatted):
     if formatted:
         return set(formatted.split(b','))
     return set()
 
 
 def addpartbundlestream2(bundler, repo, **kwargs):
     if not kwargs.get('stream', False):
         return
 
     if not streamclone.allowservergeneration(repo):
         msg = _(b'stream data requested but server does not allow this feature')
         hint = _(b'the client seems buggy')
         raise error.Abort(msg, hint=hint)
     if not (b'stream' in bundler.capabilities):
         msg = _(
             b'stream data requested but supported streaming clone versions were not specified'
         )
         hint = _(b'the client seems buggy')
         raise error.Abort(msg, hint=hint)
-    if not (b'v2' in bundler.capabilities[b'stream']):
-        raise error.Abort(_(b'the client does not support streamclone v2'))
+    client_supported = set(bundler.capabilities[b'stream'])
+    server_supported = set(getrepocaps(repo, role=b'client').get(b'stream', []))
+    common_supported = client_supported & server_supported
+    if not common_supported:
+        msg = _(b'no common supported version with the client: %s; %s')
+        str_server = b','.join(sorted(server_supported))
+        str_client = b','.join(sorted(client_supported))
+        msg %= (str_server, str_client)
+        raise error.Abort(msg)
+    version = max(common_supported)
 
     # Stream clones don't compress well. And compression undermines a
     # goal of stream clones, which is to be fast. Communicate the desire
     # to avoid compression to consumers of the bundle.
     bundler.prefercompressed = False
 
     # get the includes and excludes
     includepats = kwargs.get('includepats')
     excludepats = kwargs.get('excludepats')
 
     narrowstream = repo.ui.configbool(
         b'experimental', b'server.stream-narrow-clones'
     )
 
     if (includepats or excludepats) and not narrowstream:
         raise error.Abort(_(b'server does not support narrow stream clones'))
 
     includeobsmarkers = False
     if repo.obsstore:
         remoteversions = obsmarkersversion(bundler.capabilities)
         if not remoteversions:
             raise error.Abort(
                 _(
                     b'server has obsolescence markers, but client '
                     b'cannot receive them via stream clone'
                 )
             )
         elif repo.obsstore._version in remoteversions:
             includeobsmarkers = True
 
-    filecount, bytecount, it = streamclone.generatev2(
-        repo, includepats, excludepats, includeobsmarkers
-    )
-    requirements = streamclone.streamed_requirements(repo)
-    requirements = _formatrequirementsspec(requirements)
-    part = bundler.newpart(b'stream2', data=it)
-    part.addparam(b'bytecount', b'%d' % bytecount, mandatory=True)
-    part.addparam(b'filecount', b'%d' % filecount, mandatory=True)
-    part.addparam(b'requirements', requirements, mandatory=True)
+    if version == b"v2":
+        filecount, bytecount, it = streamclone.generatev2(
+            repo, includepats, excludepats, includeobsmarkers
+        )
+        requirements = streamclone.streamed_requirements(repo)
+        requirements = _formatrequirementsspec(requirements)
+        part = bundler.newpart(b'stream2', data=it)
+        part.addparam(b'bytecount', b'%d' % bytecount, mandatory=True)
+        part.addparam(b'filecount', b'%d' % filecount, mandatory=True)
+        part.addparam(b'requirements', requirements, mandatory=True)
+    elif version == b"v3-exp":
+        filecount, bytecount, it = streamclone.generatev2(
+            repo, includepats, excludepats, includeobsmarkers
+        )
+        requirements = streamclone.streamed_requirements(repo)
+        requirements = _formatrequirementsspec(requirements)
+        part = bundler.newpart(b'stream3', data=it)
+        part.addparam(b'bytecount', b'%d' % bytecount, mandatory=True)
+        part.addparam(b'filecount', b'%d' % filecount, mandatory=True)
+        part.addparam(b'requirements', requirements, mandatory=True)
 
 
 def buildobsmarkerspart(bundler, markers, mandatory=True):
     """add an obsmarker part to the bundler with <markers>
 
     No part is created if markers is empty.
     Raises ValueError if the bundler doesn't support any known obsmarker format.
     """
     if not markers:
         return None
 
     remoteversions = obsmarkersversion(bundler.capabilities)
     version = obsolete.commonversion(remoteversions)
     if version is None:
         raise ValueError(b'bundler does not support common obsmarker format')
     stream = obsolete.encodemarkers(markers, True, version=version)
     return bundler.newpart(b'obsmarkers', data=stream, mandatory=mandatory)
 
 
 def writebundle(
     ui, cg, filename, bundletype, vfs=None, compression=None, compopts=None
 ):
     """Write a bundle file and return its filename.
 
     Existing files will not be overwritten.
     If no filename is specified, a temporary file is created.
     bz2 compression can be turned off.
     The bundle file will be deleted in case of errors.
     """
 
     if bundletype == b"HG20":
         bundle = bundle20(ui)
         bundle.setcompression(compression, compopts)
         part = bundle.newpart(b'changegroup', data=cg.getchunks())
         part.addparam(b'version', cg.version)
         if b'clcount' in cg.extras:
             part.addparam(
                 b'nbchanges', b'%d' % cg.extras[b'clcount'], mandatory=False
             )
         chunkiter = bundle.getchunks()
     else:
         # compression argument is only for the bundle2 case
         assert compression is None
         if cg.version != b'01':
             raise error.Abort(
                 _(b'old bundle types only supports v1 changegroups')
             )
 
         # HG20 is the case without 2 values to unpack, but is handled above.
         # pytype: disable=bad-unpacking
         header, comp = bundletypes[bundletype]
         # pytype: enable=bad-unpacking
 
         if comp not in util.compengines.supportedbundletypes:
             raise error.Abort(_(b'unknown stream compression type: %s') % comp)
         compengine = util.compengines.forbundletype(comp)
 
         def chunkiter():
             yield header
             for chunk in compengine.compressstream(cg.getchunks(), compopts):
                 yield chunk
 
         chunkiter = chunkiter()
 
     # parse the changegroup data, otherwise we will block
     # in case of sshrepo because we don't know the end of the stream
     return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
 
 
 def combinechangegroupresults(op):
     """logic to combine 0 or more addchangegroup results into one"""
     results = [r.get(b'return', 0) for r in op.records[b'changegroup']]
     changedheads = 0
     result = 1
     for ret in results:
         # If any changegroup result is 0, return 0
         if ret == 0:
             result = 0
             break
         if ret < -1:
             changedheads += ret + 1
         elif ret > 1:
             changedheads += ret - 1
     if changedheads > 0:
         result = 1 + changedheads
     elif changedheads < 0:
         result = -1 + changedheads
     return result
 
 
 @parthandler(
     b'changegroup',
     (
         b'version',
         b'nbchanges',
         b'exp-sidedata',
         b'exp-wanted-sidedata',
         b'treemanifest',
         b'targetphase',
     ),
 )
 def handlechangegroup(op, inpart):
     """apply a changegroup part on the repo"""
     from . import localrepo
 
     tr = op.gettransaction()
     unpackerversion = inpart.params.get(b'version', b'01')
     # We should raise an appropriate exception here
     cg = changegroup.getunbundler(unpackerversion, inpart, None)
     # the source and url passed here are overwritten by the one contained in
     # the transaction.hookargs argument. So 'bundle2' is a placeholder
     nbchangesets = None
     if b'nbchanges' in inpart.params:
         nbchangesets = int(inpart.params.get(b'nbchanges'))
     if b'treemanifest' in inpart.params and not scmutil.istreemanifest(op.repo):
         if len(op.repo.changelog) != 0:
             raise error.Abort(
                 _(
                     b"bundle contains tree manifests, but local repo is "
                     b"non-empty and does not use tree manifests"
                 )
             )
         op.repo.requirements.add(requirements.TREEMANIFEST_REQUIREMENT)
         op.repo.svfs.options = localrepo.resolvestorevfsoptions(
             op.repo.ui, op.repo.requirements, op.repo.features
         )
         scmutil.writereporequirements(op.repo)
 
     extrakwargs = {}
     targetphase = inpart.params.get(b'targetphase')
     if targetphase is not None:
         extrakwargs['targetphase'] = int(targetphase)
 
     remote_sidedata = inpart.params.get(b'exp-wanted-sidedata')
     extrakwargs['sidedata_categories'] = read_wanted_sidedata(remote_sidedata)
 
     ret = _processchangegroup(
         op,
         cg,
         tr,
         op.source,
         b'bundle2',
         expectedtotal=nbchangesets,
         **extrakwargs
     )
     if op.reply is not None:
         # This is definitely not the final form of this
         # return. But one need to start somewhere.
         part = op.reply.newpart(b'reply:changegroup', mandatory=False)
         part.addparam(
             b'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False
         )
         part.addparam(b'return', b'%i' % ret, mandatory=False)
     assert not inpart.read()
 
 
 _remotechangegroupparams = tuple(
     [b'url', b'size', b'digests']
     + [b'digest:%s' % k for k in util.DIGESTS.keys()]
 )
 
 
 @parthandler(b'remote-changegroup', _remotechangegroupparams)
 def handleremotechangegroup(op, inpart):
     """apply a bundle10 on the repo, given an url and validation information
 
     All the information about the remote bundle to import are given as
     parameters. The parameters include:
       - url: the url to the bundle10.
       - size: the bundle10 file size. It is used to validate what was
         retrieved by the client matches the server knowledge about the bundle.
       - digests: a space separated list of the digest types provided as
         parameters.
       - digest:<digest-type>: the hexadecimal representation of the digest with
         that name. Like the size, it is used to validate what was retrieved by
         the client matches what the server knows about the bundle.
 
     When multiple digest types are given, all of them are checked.
     """
     try:
         raw_url = inpart.params[b'url']
     except KeyError:
         raise error.Abort(_(b'remote-changegroup: missing "%s" param') % b'url')
     parsed_url = urlutil.url(raw_url)
     if parsed_url.scheme not in capabilities[b'remote-changegroup']:
         raise error.Abort(
             _(b'remote-changegroup does not support %s urls')
             % parsed_url.scheme
         )
 
     try:
         size = int(inpart.params[b'size'])
     except ValueError:
         raise error.Abort(
             _(b'remote-changegroup: invalid value for param "%s"') % b'size'
         )
     except KeyError:
         raise error.Abort(
             _(b'remote-changegroup: missing "%s" param') % b'size'
         )
 
     digests = {}
     for typ in inpart.params.get(b'digests', b'').split():
         param = b'digest:%s' % typ
         try:
             value = inpart.params[param]
         except KeyError:
             raise error.Abort(
                 _(b'remote-changegroup: missing "%s" param') % param
             )
         digests[typ] = value
 
     real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests)
 
     tr = op.gettransaction()
     from . import exchange
 
     cg = exchange.readbundle(op.repo.ui, real_part, raw_url)
     if not isinstance(cg, changegroup.cg1unpacker):
         raise error.Abort(
             _(b'%s: not a bundle version 1.0') % urlutil.hidepassword(raw_url)
         )
     ret = _processchangegroup(op, cg, tr, op.source, b'bundle2')
     if op.reply is not None:
         # This is definitely not the final form of this
         # return. But one need to start somewhere.
         part = op.reply.newpart(b'reply:changegroup')
         part.addparam(
             b'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False
         )
         part.addparam(b'return', b'%i' % ret, mandatory=False)
     try:
         real_part.validate()
     except error.Abort as e:
         raise error.Abort(
             _(b'bundle at %s is corrupted:\n%s')
             % (urlutil.hidepassword(raw_url), e.message)
         )
     assert not inpart.read()
 
 
 @parthandler(b'reply:changegroup', (b'return', b'in-reply-to'))
 def handlereplychangegroup(op, inpart):
     ret = int(inpart.params[b'return'])
     replyto = int(inpart.params[b'in-reply-to'])
     op.records.add(b'changegroup', {b'return': ret}, replyto)
 
 
 @parthandler(b'check:bookmarks')
 def handlecheckbookmarks(op, inpart):
     """check location of bookmarks
 
     This part is to be used to detect push race regarding bookmark, it
     contains binary encoded (bookmark, node) tuple. If the local state does
     not marks the one in the part, a PushRaced exception is raised
     """
     bookdata = bookmarks.binarydecode(op.repo, inpart)
 
     msgstandard = (
         b'remote repository changed while pushing - please try again '
         b'(bookmark "%s" move from %s to %s)'
     )
     msgmissing = (
         b'remote repository changed while pushing - please try again '
         b'(bookmark "%s" is missing, expected %s)'
     )
     msgexist = (
         b'remote repository changed while pushing - please try again '
         b'(bookmark "%s" set on %s, expected missing)'
     )
     for book, node in bookdata:
         currentnode = op.repo._bookmarks.get(book)
         if currentnode != node:
             if node is None:
                 finalmsg = msgexist % (book, short(currentnode))
             elif currentnode is None:
                 finalmsg = msgmissing % (book, short(node))
             else:
                 finalmsg = msgstandard % (
                     book,
                     short(node),
                     short(currentnode),
                 )
             raise error.PushRaced(finalmsg)
 
 
 @parthandler(b'check:heads')
 def handlecheckheads(op, inpart):
     """check that head of the repo did not change
 
     This is used to detect a push race when using unbundle.
     This replaces the "heads" argument of unbundle."""
     h = inpart.read(20)
     heads = []
     while len(h) == 20:
         heads.append(h)
         h = inpart.read(20)
     assert not h
     # Trigger a transaction so that we are guaranteed to have the lock now.
     if op.ui.configbool(b'experimental', b'bundle2lazylocking'):
         op.gettransaction()
     if sorted(heads) != sorted(op.repo.heads()):
         raise error.PushRaced(
             b'remote repository changed while pushing - please try again'
         )
 
 
 @parthandler(b'check:updated-heads')
 def handlecheckupdatedheads(op, inpart):
     """check for race on the heads touched by a push
 
     This is similar to 'check:heads' but focus on the heads actually updated
     during the push. If other activities happen on unrelated heads, it is
     ignored.
 
     This allow server with high traffic to avoid push contention as long as
     unrelated parts of the graph are involved."""
     h = inpart.read(20)
     heads = []
     while len(h) == 20:
         heads.append(h)
         h = inpart.read(20)
     assert not h
     # trigger a transaction so that we are guaranteed to have the lock now.
     if op.ui.configbool(b'experimental', b'bundle2lazylocking'):
         op.gettransaction()
 
     currentheads = set()
     for ls in op.repo.branchmap().iterheads():
         currentheads.update(ls)
 
     for h in heads:
         if h not in currentheads:
             raise error.PushRaced(
                 b'remote repository changed while pushing - '
                 b'please try again'
             )
 
 
 @parthandler(b'check:phases')
 def handlecheckphases(op, inpart):
     """check that phase boundaries of the repository did not change
 
     This is used to detect a push race.
     """
     phasetonodes = phases.binarydecode(inpart)
     unfi = op.repo.unfiltered()
     cl = unfi.changelog
     phasecache = unfi._phasecache
     msg = (
         b'remote repository changed while pushing - please try again '
         b'(%s is %s expected %s)'
     )
     for expectedphase, nodes in phasetonodes.items():
         for n in nodes:
             actualphase = phasecache.phase(unfi, cl.rev(n))
             if actualphase != expectedphase:
                 finalmsg = msg % (
                     short(n),
                     phases.phasenames[actualphase],
                     phases.phasenames[expectedphase],
                 )
                 raise error.PushRaced(finalmsg)
 
 
 @parthandler(b'output')
 def handleoutput(op, inpart):
     """forward output captured on the server to the client"""
     for line in inpart.read().splitlines():
         op.ui.status(_(b'remote: %s\n') % line)
 
 
 @parthandler(b'replycaps')
 def handlereplycaps(op, inpart):
     """Notify that a reply bundle should be created
 
     The payload contains the capabilities information for the reply"""
     caps = decodecaps(inpart.read())
     if op.reply is None:
         op.reply = bundle20(op.ui, caps)
 
 
 class AbortFromPart(error.Abort):
     """Sub-class of Abort that denotes an error from a bundle2 part."""
 
 
 @parthandler(b'error:abort', (b'message', b'hint'))
 def handleerrorabort(op, inpart):
     """Used to transmit abort error over the wire"""
     raise AbortFromPart(
         inpart.params[b'message'], hint=inpart.params.get(b'hint')
     )
 
 
 @parthandler(
     b'error:pushkey',
     (b'namespace', b'key', b'new', b'old', b'ret', b'in-reply-to'),
 )
 def handleerrorpushkey(op, inpart):
     """Used to transmit failure of a mandatory pushkey over the wire"""
     kwargs = {}
     for name in (b'namespace', b'key', b'new', b'old', b'ret'):
         value = inpart.params.get(name)
         if value is not None:
             kwargs[name] = value
     raise error.PushkeyFailed(
         inpart.params[b'in-reply-to'], **pycompat.strkwargs(kwargs)
     )
 
 
 @parthandler(b'error:unsupportedcontent', (b'parttype', b'params'))
 def handleerrorunsupportedcontent(op, inpart):
     """Used to transmit unknown content error over the wire"""
     kwargs = {}
     parttype = inpart.params.get(b'parttype')
     if parttype is not None:
         kwargs[b'parttype'] = parttype
     params = inpart.params.get(b'params')
     if params is not None:
         kwargs[b'params'] = params.split(b'\0')
 
     raise error.BundleUnknownFeatureError(**pycompat.strkwargs(kwargs))
 
 
 @parthandler(b'error:pushraced', (b'message',))
 def handleerrorpushraced(op, inpart):
     """Used to transmit push race error over the wire"""
     raise error.ResponseError(_(b'push failed:'), inpart.params[b'message'])
 
 
 @parthandler(b'listkeys', (b'namespace',))
 def handlelistkeys(op, inpart):
     """retrieve pushkey namespace content stored in a bundle2"""
     namespace = inpart.params[b'namespace']
     r = pushkey.decodekeys(inpart.read())
     op.records.add(b'listkeys', (namespace, r))
 
 
 @parthandler(b'pushkey', (b'namespace', b'key', b'old', b'new'))
 def handlepushkey(op, inpart):
     """process a pushkey request"""
     dec = pushkey.decode
     namespace = dec(inpart.params[b'namespace'])
     key = dec(inpart.params[b'key'])
     old = dec(inpart.params[b'old'])
     new = dec(inpart.params[b'new'])
     # Grab the transaction to ensure that we have the lock before performing the
     # pushkey.
     if op.ui.configbool(b'experimental', b'bundle2lazylocking'):
         op.gettransaction()
     ret = op.repo.pushkey(namespace, key, old, new)
     record = {b'namespace': namespace, b'key': key, b'old': old, b'new': new}
     op.records.add(b'pushkey', record)
     if op.reply is not None:
         rpart = op.reply.newpart(b'reply:pushkey')
         rpart.addparam(
             b'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False
         )
         rpart.addparam(b'return', b'%i' % ret, mandatory=False)
     if inpart.mandatory and not ret:
         kwargs = {}
         for key in (b'namespace', b'key', b'new', b'old', b'ret'):
             if key in inpart.params:
                 kwargs[key] = inpart.params[key]
         raise error.PushkeyFailed(
             partid=b'%d' % inpart.id, **pycompat.strkwargs(kwargs)
         )
 
 
 @parthandler(b'bookmarks')
 def handlebookmark(op, inpart):
     """transmit bookmark information
 
     The part contains binary encoded bookmark information.
 
     The exact behavior of this part can be controlled by the 'bookmarks' mode
     on the bundle operation.
 
     When mode is 'apply' (the default) the bookmark information is applied as
     is to the unbundling repository. Make sure a 'check:bookmarks' part is
     issued earlier to check for push races in such update. This behavior is
     suitable for pushing.
 
     When mode is 'records', the information is recorded into the 'bookmarks'
     records of the bundle operation. This behavior is suitable for pulling.
     """
     changes = bookmarks.binarydecode(op.repo, inpart)
 
     pushkeycompat = op.repo.ui.configbool(
         b'server', b'bookmarks-pushkey-compat'
     )
     bookmarksmode = op.modes.get(b'bookmarks', b'apply')
 
     if bookmarksmode == b'apply':
         tr = op.gettransaction()
         bookstore = op.repo._bookmarks
         if pushkeycompat:
             allhooks = []
             for book, node in changes:
                 hookargs = tr.hookargs.copy()
                 hookargs[b'pushkeycompat'] = b'1'
                 hookargs[b'namespace'] = b'bookmarks'
                 hookargs[b'key'] = book
                 hookargs[b'old'] = hex(bookstore.get(book, b''))
                 hookargs[b'new'] = hex(node if node is not None else b'')
                 allhooks.append(hookargs)
 
             for hookargs in allhooks:
                 op.repo.hook(
                     b'prepushkey', throw=True, **pycompat.strkwargs(hookargs)
                 )
 
         for book, node in changes:
             if bookmarks.isdivergent(book):
                 msg = _(b'cannot accept divergent bookmark %s!') % book
                 raise error.Abort(msg)
 
         bookstore.applychanges(op.repo, op.gettransaction(), changes)
 
         if pushkeycompat:
 
             def runhook(unused_success):
                 for hookargs in allhooks:
                     op.repo.hook(b'pushkey', **pycompat.strkwargs(hookargs))
 
             op.repo._afterlock(runhook)
 
     elif bookmarksmode == b'records':
         for book, node in changes:
             record = {b'bookmark': book, b'node': node}
             op.records.add(b'bookmarks', record)
     else:
         raise error.ProgrammingError(
             b'unknown bookmark mode: %s' % bookmarksmode
         )
 
 
 @parthandler(b'phase-heads')
 def handlephases(op, inpart):
     """apply phases from bundle part to repo"""
     headsbyphase = phases.binarydecode(inpart)
     phases.updatephases(op.repo.unfiltered(), op.gettransaction, headsbyphase)
 
 
 @parthandler(b'reply:pushkey', (b'return', b'in-reply-to'))
 def handlepushkeyreply(op, inpart):
     """retrieve the result of a pushkey request"""
     ret = int(inpart.params[b'return'])
     partid = int(inpart.params[b'in-reply-to'])
     op.records.add(b'pushkey', {b'return': ret}, partid)
 
 
 @parthandler(b'obsmarkers')
 def handleobsmarker(op, inpart):
     """add a stream of obsmarkers to the repo"""
     tr = op.gettransaction()
     markerdata = inpart.read()
     if op.ui.config(b'experimental', b'obsmarkers-exchange-debug'):
         op.ui.writenoi18n(
             b'obsmarker-exchange: %i bytes received\n' % len(markerdata)
         )
     # The mergemarkers call will crash if marker creation is not enabled.
     # we want to avoid this if the part is advisory.
     if not inpart.mandatory and op.repo.obsstore.readonly:
         op.repo.ui.debug(
             b'ignoring obsolescence markers, feature not enabled\n'
         )
         return
     new = op.repo.obsstore.mergemarkers(tr, markerdata)
     op.repo.invalidatevolatilesets()
     op.records.add(b'obsmarkers', {b'new': new})
     if op.reply is not None:
         rpart = op.reply.newpart(b'reply:obsmarkers')
         rpart.addparam(
             b'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False
         )
         rpart.addparam(b'new', b'%i' % new, mandatory=False)
 
 
 @parthandler(b'reply:obsmarkers', (b'new', b'in-reply-to'))
 def handleobsmarkerreply(op, inpart):
     """retrieve the result of a pushkey request"""
     ret = int(inpart.params[b'new'])
     partid = int(inpart.params[b'in-reply-to'])
     op.records.add(b'obsmarkers', {b'new': ret}, partid)
 
 
 @parthandler(b'hgtagsfnodes')
 def handlehgtagsfnodes(op, inpart):
     """Applies .hgtags fnodes cache entries to the local repo.
 
     Payload is pairs of 20 byte changeset nodes and filenodes.
     """
     # Grab the transaction so we ensure that we have the lock at this point.
     if op.ui.configbool(b'experimental', b'bundle2lazylocking'):
         op.gettransaction()
     cache = tags.hgtagsfnodescache(op.repo.unfiltered())
 
     count = 0
     while True:
         node = inpart.read(20)
         fnode = inpart.read(20)
         if len(node) < 20 or len(fnode) < 20:
             op.ui.debug(b'ignoring incomplete received .hgtags fnodes data\n')
             break
         cache.setfnode(node, fnode)
         count += 1
 
     cache.write()
     op.ui.debug(b'applied %i hgtags fnodes cache entries\n' % count)
 
 
 rbcstruct = struct.Struct(b'>III')
 
 
 @parthandler(b'cache:rev-branch-cache')
 def handlerbc(op, inpart):
     """Legacy part, ignored for compatibility with bundles from or
     for Mercurial before 5.7. Newer Mercurial computes the cache
     efficiently enough during unbundling that the additional transfer
     is unnecessary."""
 
 
 @parthandler(b'pushvars')
 def bundle2getvars(op, part):
     '''unbundle a bundle2 containing shellvars on the server'''
     # An option to disable unbundling on server-side for security reasons
     if op.ui.configbool(b'push', b'pushvars.server'):
         hookargs = {}
         for key, value in part.advisoryparams:
             key = key.upper()
             # We want pushed variables to have USERVAR_ prepended so we know
             # they came from the --pushvar flag.
             key = b"USERVAR_" + key
             hookargs[key] = value
         op.addhookargs(hookargs)
 
 
 @parthandler(b'stream2', (b'requirements', b'filecount', b'bytecount'))
 def handlestreamv2bundle(op, part):
 
     requirements = urlreq.unquote(part.params[b'requirements'])
     requirements = requirements.split(b',') if requirements else []
     filecount = int(part.params[b'filecount'])
     bytecount = int(part.params[b'bytecount'])
 
     repo = op.repo
     if len(repo):
         msg = _(b'cannot apply stream clone to non empty repository')
         raise error.Abort(msg)
 
     repo.ui.debug(b'applying stream bundle\n')
     streamclone.applybundlev2(repo, part, filecount, bytecount, requirements)
 
 
+@parthandler(b'stream3', (b'requirements', b'filecount', b'bytecount'))
+def handlestreamv3bundle(op, part):
+    return handlestreamv2bundle(op, part)
+
+
 def widen_bundle(
     bundler, repo, oldmatcher, newmatcher, common, known, cgversion, ellipses
 ):
     """generates bundle2 for widening a narrow clone
 
     bundler is the bundle to which data should be added
     repo is the localrepository instance
     oldmatcher matches what the client already has
     newmatcher matches what the client needs (including what it already has)
     common is set of common heads between server and client
     known is a set of revs known on the client side (used in ellipses)
     cgversion is the changegroup version to send
     ellipses is boolean value telling whether to send ellipses data or not
 
     returns bundle2 of the data required for extending
     """
     commonnodes = set()
     cl = repo.changelog
     for r in repo.revs(b"::%ln", common):
         commonnodes.add(cl.node(r))
     if commonnodes:
         packer = changegroup.getbundler(
             cgversion,
             repo,
             oldmatcher=oldmatcher,
             matcher=newmatcher,
             fullnodes=commonnodes,
         )
         cgdata = packer.generate(
             {repo.nullid},
             list(commonnodes),
             False,
             b'narrow_widen',
             changelog=False,
         )
 
         part = bundler.newpart(b'changegroup', data=cgdata)
         part.addparam(b'version', cgversion)
         if scmutil.istreemanifest(repo):
             part.addparam(b'treemanifest', b'1')
     if repository.REPO_FEATURE_SIDE_DATA in repo.features:
         part.addparam(b'exp-sidedata', b'1')
         wanted = format_remote_wanted_sidedata(repo)
         part.addparam(b'exp-wanted-sidedata', wanted)
 
     return bundler

mercurial/configitems.py

@@ -1,2979 +1,2984
 # configitems.py - centralized declaration of configuration option
 #
 #  Copyright 2017 Pierre-Yves David <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 
 import functools
 import re
 
 from . import (
     encoding,
     error,
 )
 
 
 def loadconfigtable(ui, extname, configtable):
     """update config item known to the ui with the extension ones"""
     for section, items in sorted(configtable.items()):
         knownitems = ui._knownconfig.setdefault(section, itemregister())
         knownkeys = set(knownitems)
         newkeys = set(items)
         for key in sorted(knownkeys & newkeys):
             msg = b"extension '%s' overwrite config item '%s.%s'"
             msg %= (extname, section, key)
             ui.develwarn(msg, config=b'warn-config')
 
         knownitems.update(items)
 
 
 class configitem:
     """represent a known config item
 
     :section: the official config section where to find this item,
        :name: the official name within the section,
     :default: default value for this item,
     :alias: optional list of tuples as alternatives,
     :generic: this is a generic definition, match name using regular expression.
     """
 
     def __init__(
         self,
         section,
         name,
         default=None,
         alias=(),
         generic=False,
         priority=0,
         experimental=False,
     ):
         self.section = section
         self.name = name
         self.default = default
         self.alias = list(alias)
         self.generic = generic
         self.priority = priority
         self.experimental = experimental
         self._re = None
         if generic:
             self._re = re.compile(self.name)
 
 
 class itemregister(dict):
     """A specialized dictionary that can handle wild-card selection"""
 
     def __init__(self):
         super(itemregister, self).__init__()
         self._generics = set()
 
     def update(self, other):
         super(itemregister, self).update(other)
         self._generics.update(other._generics)
 
     def __setitem__(self, key, item):
         super(itemregister, self).__setitem__(key, item)
         if item.generic:
             self._generics.add(item)
 
     def get(self, key):
         baseitem = super(itemregister, self).get(key)
         if baseitem is not None and not baseitem.generic:
             return baseitem
 
         # search for a matching generic item
         generics = sorted(self._generics, key=(lambda x: (x.priority, x.name)))
         for item in generics:
             # we use 'match' instead of 'search' to make the matching simpler
             # for people unfamiliar with regular expression. Having the match
             # rooted to the start of the string will produce less surprising
             # result for user writing simple regex for sub-attribute.
             #
             # For example using "color\..*" match produces an unsurprising
             # result, while using search could suddenly match apparently
             # unrelated configuration that happens to contains "color."
             # anywhere. This is a tradeoff where we favor requiring ".*" on
             # some match to avoid the need to prefix most pattern with "^".
             # The "^" seems more error prone.
             if item._re.match(key):
                 return item
 
         return None
 
 
 coreitems = {}
 
 
 def _register(configtable, *args, **kwargs):
     item = configitem(*args, **kwargs)
     section = configtable.setdefault(item.section, itemregister())
     if item.name in section:
         msg = b"duplicated config item registration for '%s.%s'"
         raise error.ProgrammingError(msg % (item.section, item.name))
     section[item.name] = item
 
 
 # special value for case where the default is derived from other values
 dynamicdefault = object()
 
 # Registering actual config items
 
 
 def getitemregister(configtable):
     f = functools.partial(_register, configtable)
     # export pseudo enum as configitem.*
     f.dynamicdefault = dynamicdefault
     return f
 
 
 coreconfigitem = getitemregister(coreitems)
 
 
 def _registerdiffopts(section, configprefix=b''):
     coreconfigitem(
         section,
         configprefix + b'nodates',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'showfunc',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'unified',
         default=None,
     )
     coreconfigitem(
         section,
         configprefix + b'git',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'ignorews',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'ignorewsamount',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'ignoreblanklines',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'ignorewseol',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'nobinary',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'noprefix',
         default=False,
     )
     coreconfigitem(
         section,
         configprefix + b'word-diff',
         default=False,
     )
 
 
 coreconfigitem(
     b'alias',
     b'.*',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'auth',
     b'cookiefile',
     default=None,
 )
 _registerdiffopts(section=b'annotate')
 # bookmarks.pushing: internal hack for discovery
 coreconfigitem(
     b'bookmarks',
     b'pushing',
     default=list,
 )
 # bundle.mainreporoot: internal hack for bundlerepo
 coreconfigitem(
     b'bundle',
     b'mainreporoot',
     default=b'',
 )
 coreconfigitem(
     b'censor',
     b'policy',
     default=b'abort',
     experimental=True,
 )
 coreconfigitem(
     b'chgserver',
     b'idletimeout',
     default=3600,
 )
 coreconfigitem(
     b'chgserver',
     b'skiphash',
     default=False,
 )
 coreconfigitem(
     b'cmdserver',
     b'log',
     default=None,
 )
 coreconfigitem(
     b'cmdserver',
     b'max-log-files',
     default=7,
 )
 coreconfigitem(
     b'cmdserver',
     b'max-log-size',
     default=b'1 MB',
 )
 coreconfigitem(
     b'cmdserver',
     b'max-repo-cache',
     default=0,
     experimental=True,
 )
 coreconfigitem(
     b'cmdserver',
     b'message-encodings',
     default=list,
 )
 coreconfigitem(
     b'cmdserver',
     b'track-log',
     default=lambda: [b'chgserver', b'cmdserver', b'repocache'],
 )
 coreconfigitem(
     b'cmdserver',
     b'shutdown-on-interrupt',
     default=True,
 )
 coreconfigitem(
     b'color',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'color',
     b'mode',
     default=b'auto',
 )
 coreconfigitem(
     b'color',
     b'pagermode',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'command-templates',
     b'graphnode',
     default=None,
     alias=[(b'ui', b'graphnodetemplate')],
 )
 coreconfigitem(
     b'command-templates',
     b'log',
     default=None,
     alias=[(b'ui', b'logtemplate')],
 )
 coreconfigitem(
     b'command-templates',
     b'mergemarker',
     default=(
         b'{node|short} '
         b'{ifeq(tags, "tip", "", '
         b'ifeq(tags, "", "", "{tags} "))}'
         b'{if(bookmarks, "{bookmarks} ")}'
         b'{ifeq(branch, "default", "", "{branch} ")}'
         b'- {author|user}: {desc|firstline}'
     ),
     alias=[(b'ui', b'mergemarkertemplate')],
 )
 coreconfigitem(
     b'command-templates',
     b'pre-merge-tool-output',
     default=None,
     alias=[(b'ui', b'pre-merge-tool-output-template')],
 )
 coreconfigitem(
     b'command-templates',
     b'oneline-summary',
     default=None,
 )
 coreconfigitem(
     b'command-templates',
     b'oneline-summary.*',
     default=dynamicdefault,
     generic=True,
 )
 _registerdiffopts(section=b'commands', configprefix=b'commit.interactive.')
 coreconfigitem(
     b'commands',
     b'commit.post-status',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'grep.all-files',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'commands',
     b'merge.require-rev',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'push.require-revs',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'resolve.confirm',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'resolve.explicit-re-merge',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'resolve.mark-check',
     default=b'none',
 )
 _registerdiffopts(section=b'commands', configprefix=b'revert.interactive.')
 coreconfigitem(
     b'commands',
     b'show.aliasprefix',
     default=list,
 )
 coreconfigitem(
     b'commands',
     b'status.relative',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'status.skipstates',
     default=[],
     experimental=True,
 )
 coreconfigitem(
     b'commands',
     b'status.terse',
     default=b'',
 )
 coreconfigitem(
     b'commands',
     b'status.verbose',
     default=False,
 )
 coreconfigitem(
     b'commands',
     b'update.check',
     default=None,
 )
 coreconfigitem(
     b'commands',
     b'update.requiredest',
     default=False,
 )
 coreconfigitem(
     b'committemplate',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'convert',
     b'bzr.saverev',
     default=True,
 )
 coreconfigitem(
     b'convert',
     b'cvsps.cache',
     default=True,
 )
 coreconfigitem(
     b'convert',
     b'cvsps.fuzz',
     default=60,
 )
 coreconfigitem(
     b'convert',
     b'cvsps.logencoding',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'cvsps.mergefrom',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'cvsps.mergeto',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'git.committeractions',
     default=lambda: [b'messagedifferent'],
 )
 coreconfigitem(
     b'convert',
     b'git.extrakeys',
     default=list,
 )
 coreconfigitem(
     b'convert',
     b'git.findcopiesharder',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'git.remoteprefix',
     default=b'remote',
 )
 coreconfigitem(
     b'convert',
     b'git.renamelimit',
     default=400,
 )
 coreconfigitem(
     b'convert',
     b'git.saverev',
     default=True,
 )
 coreconfigitem(
     b'convert',
     b'git.similarity',
     default=50,
 )
 coreconfigitem(
     b'convert',
     b'git.skipsubmodules',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'hg.clonebranches',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'hg.ignoreerrors',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'hg.preserve-hash',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'hg.revs',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'hg.saverev',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'hg.sourcename',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'hg.startrev',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'hg.tagsbranch',
     default=b'default',
 )
 coreconfigitem(
     b'convert',
     b'hg.usebranchnames',
     default=True,
 )
 coreconfigitem(
     b'convert',
     b'ignoreancestorcheck',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'convert',
     b'localtimezone',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'p4.encoding',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'convert',
     b'p4.startrev',
     default=0,
 )
 coreconfigitem(
     b'convert',
     b'skiptags',
     default=False,
 )
 coreconfigitem(
     b'convert',
     b'svn.debugsvnlog',
     default=True,
 )
 coreconfigitem(
     b'convert',
     b'svn.trunk',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'svn.tags',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'svn.branches',
     default=None,
 )
 coreconfigitem(
     b'convert',
     b'svn.startrev',
     default=0,
 )
 coreconfigitem(
     b'convert',
     b'svn.dangerous-set-commit-dates',
     default=False,
 )
 coreconfigitem(
     b'debug',
     b'dirstate.delaywrite',
     default=0,
 )
 coreconfigitem(
     b'debug',
     b'revlog.verifyposition.changelog',
     default=b'',
 )
 coreconfigitem(
     b'debug',
     b'revlog.debug-delta',
     default=False,
 )
 # display extra information about the bundling process
 coreconfigitem(
     b'debug',
     b'bundling-stats',
     default=False,
 )
 # display extra information about the unbundling process
 coreconfigitem(
     b'debug',
     b'unbundling-stats',
     default=False,
 )
 coreconfigitem(
     b'defaults',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'devel',
     b'all-warnings',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'bundle2.debug',
     default=False,
 )
 # which kind of delta to put in the bundled changegroup. Possible value
 # - '': use default behavior
 # - p1: force to always use delta against p1
 # - full: force to always use full content
 coreconfigitem(
     b'devel',
     b'bundle.delta',
     default=b'',
 )
 coreconfigitem(
     b'devel',
     b'cache-vfs',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'check-locks',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'check-relroot',
     default=False,
 )
 # Track copy information for all file, not just "added" one (very slow)
 coreconfigitem(
     b'devel',
     b'copy-tracing.trace-all-files',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'default-date',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'deprec-warn',
     default=False,
 )
 # possible values:
 # - auto (the default)
 # - force-append
 # - force-new
 coreconfigitem(
     b'devel',
     b'dirstate.v2.data_update_mode',
     default="auto",
 )
 coreconfigitem(
     b'devel',
     b'disableloaddefaultcerts',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'warn-empty-changegroup',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'legacy.exchange',
     default=list,
 )
 # When True, revlogs use a special reference version of the nodemap, that is not
 # performant but is "known" to behave properly.
 coreconfigitem(
     b'devel',
     b'persistent-nodemap',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'servercafile',
     default=b'',
 )
 # This config option is intended for use in tests only. It is a giant
 # footgun to kill security. Don't define it.
 coreconfigitem(
     b'devel',
     b'server-insecure-exact-protocol',
     default=b'',
 )
 coreconfigitem(
     b'devel',
     b'serverrequirecert',
     default=False,
 )
 # Makes the status algorithm wait for the existence of this file
 # (or until a timeout of `devel.sync.status.pre-dirstate-write-file-timeout`
 # seconds) before taking the lock and writing the dirstate.
 # Status signals that it's ready to wait by creating a file
 # with the same name + `.waiting`.
 # Useful when testing race conditions.
 coreconfigitem(
     b'devel',
     b'sync.status.pre-dirstate-write-file',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'sync.status.pre-dirstate-write-file-timeout',
     default=2,
 )
 coreconfigitem(
     b'devel',
     b'sync.dirstate.post-docket-read-file',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'sync.dirstate.post-docket-read-file-timeout',
     default=2,
 )
 coreconfigitem(
     b'devel',
     b'sync.dirstate.pre-read-file',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'sync.dirstate.pre-read-file-timeout',
     default=2,
 )
 coreconfigitem(
     b'devel',
     b'strip-obsmarkers',
     default=True,
 )
 coreconfigitem(
     b'devel',
     b'warn-config',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'warn-config-default',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'user.obsmarker',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'warn-config-unknown',
     default=None,
 )
 coreconfigitem(
     b'devel',
     b'debug.copies',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'copy-tracing.multi-thread',
     default=True,
 )
 coreconfigitem(
     b'devel',
     b'debug.extensions',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'debug.repo-filters',
     default=False,
 )
 coreconfigitem(
     b'devel',
     b'debug.peer-request',
     default=False,
 )
 # If discovery.exchange-heads is False, the discovery will not start with
 # remote head fetching and local head querying.
 coreconfigitem(
     b'devel',
     b'discovery.exchange-heads',
     default=True,
 )
 # If devel.debug.abort-update is True, then any merge with the working copy,
 # e.g. [hg update], will be aborted after figuring out what needs to be done,
 # but before spawning the parallel worker
 coreconfigitem(
     b'devel',
     b'debug.abort-update',
     default=False,
 )
 # If discovery.grow-sample is False, the sample size used in set discovery will
 # not be increased through the process
 coreconfigitem(
     b'devel',
     b'discovery.grow-sample',
     default=True,
 )
 # When discovery.grow-sample.dynamic is True, the default, the sample size is
 # adapted to the shape of the undecided set (it is set to the max of:
 # <target-size>, len(roots(undecided)), len(heads(undecided)
 coreconfigitem(
     b'devel',
     b'discovery.grow-sample.dynamic',
     default=True,
 )
 # discovery.grow-sample.rate control the rate at which the sample grow
 coreconfigitem(
     b'devel',
     b'discovery.grow-sample.rate',
     default=1.05,
 )
 # If discovery.randomize is False, random sampling during discovery are
 # deterministic. It is meant for integration tests.
 coreconfigitem(
     b'devel',
     b'discovery.randomize',
     default=True,
 )
 # Control the initial size of the discovery sample
 coreconfigitem(
     b'devel',
     b'discovery.sample-size',
     default=200,
 )
 # Control the initial size of the discovery for initial change
 coreconfigitem(
     b'devel',
     b'discovery.sample-size.initial',
     default=100,
 )
 _registerdiffopts(section=b'diff')
 coreconfigitem(
     b'diff',
     b'merge',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'email',
     b'bcc',
     default=None,
 )
 coreconfigitem(
     b'email',
     b'cc',
     default=None,
 )
 coreconfigitem(
     b'email',
     b'charsets',
     default=list,
 )
 coreconfigitem(
     b'email',
     b'from',
     default=None,
 )
 coreconfigitem(
     b'email',
     b'method',
     default=b'smtp',
 )
 coreconfigitem(
     b'email',
     b'reply-to',
     default=None,
 )
 coreconfigitem(
     b'email',
     b'to',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'archivemetatemplate',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'experimental',
     b'auto-publish',
     default=b'publish',
 )
 coreconfigitem(
     b'experimental',
     b'bundle-phases',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'bundle2-advertise',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'bundle2-output-capture',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'bundle2.pushback',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'bundle2lazylocking',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecomplevel',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecomplevel.bzip2',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecomplevel.gzip',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecomplevel.none',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecomplevel.zstd',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecompthreads',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecompthreads.bzip2',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecompthreads.gzip',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecompthreads.none',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'bundlecompthreads.zstd',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'changegroup3',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'changegroup4',
     default=False,
 )
 
 # might remove rank configuration once the computation has no impact
 coreconfigitem(
     b'experimental',
     b'changelog-v2.compute-rank',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'cleanup-as-archived',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'clientcompressionengines',
     default=list,
 )
 coreconfigitem(
     b'experimental',
     b'copytrace',
     default=b'on',
 )
 coreconfigitem(
     b'experimental',
     b'copytrace.movecandidateslimit',
     default=100,
 )
 coreconfigitem(
     b'experimental',
     b'copytrace.sourcecommitlimit',
     default=100,
 )
 coreconfigitem(
     b'experimental',
     b'copies.read-from',
     default=b"filelog-only",
 )
 coreconfigitem(
     b'experimental',
     b'copies.write-to',
     default=b'filelog-only',
 )
 coreconfigitem(
     b'experimental',
     b'crecordtest',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'directaccess',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'directaccess.revnums',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'editortmpinhg',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'evolution',
     default=list,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.allowdivergence',
     default=False,
     alias=[(b'experimental', b'allowdivergence')],
 )
 coreconfigitem(
     b'experimental',
     b'evolution.allowunstable',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.createmarkers',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.effect-flags',
     default=True,
     alias=[(b'experimental', b'effect-flags')],
 )
 coreconfigitem(
     b'experimental',
     b'evolution.exchange',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.bundle-obsmarker',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.bundle-obsmarker:mandatory',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'log.topo',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.report-instabilities',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'evolution.track-operation',
     default=True,
 )
 # repo-level config to exclude a revset visibility
 #
 # The target use case is to use `share` to expose different subset of the same
 # repository, especially server side. See also `server.view`.
 coreconfigitem(
     b'experimental',
     b'extra-filter-revs',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'maxdeltachainspan',
     default=-1,
 )
 # tracks files which were undeleted (merge might delete them but we explicitly
 # kept/undeleted them) and creates new filenodes for them
 coreconfigitem(
     b'experimental',
     b'merge-track-salvaged',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'mmapindexthreshold',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'narrow',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'nonnormalparanoidcheck',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'exportableenviron',
     default=list,
 )
 coreconfigitem(
     b'experimental',
     b'extendedheader.index',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'extendedheader.similarity',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'graphshorten',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'graphstyle.parent',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'experimental',
     b'graphstyle.missing',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'experimental',
     b'graphstyle.grandparent',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'experimental',
     b'hook-track-tags',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'httppostargs',
     default=False,
 )
 coreconfigitem(b'experimental', b'nointerrupt', default=False)
 coreconfigitem(b'experimental', b'nointerrupt-interactiveonly', default=True)
 
 coreconfigitem(
     b'experimental',
     b'obsmarkers-exchange-debug',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'remotenames',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'removeemptydirs',
     default=True,
 )
 coreconfigitem(
     b'experimental',
     b'revert.interactive.select-to-keep',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'revisions.prefixhexnode',
     default=False,
 )
 # "out of experimental" todo list.
 #
 # * include management of a persistent nodemap in the main docket
 # * enforce a "no-truncate" policy for mmap safety
 #      - for censoring operation
 #      - for stripping operation
 #      - for rollback operation
 # * proper streaming (race free) of the docket file
 # * track garbage data to evemtually allow rewriting -existing- sidedata.
 # * Exchange-wise, we will also need to do something more efficient than
 #   keeping references to the affected revlogs, especially memory-wise when
 #   rewriting sidedata.
 # * introduce a proper solution to reduce the number of filelog related files.
 # * use caching for reading sidedata (similar to what we do for data).
 # * no longer set offset=0 if sidedata_size=0 (simplify cutoff computation).
 # * Improvement to consider
 #   - avoid compression header in chunk using the default compression?
 #   - forbid "inline" compression mode entirely?
 #   - split the data offset and flag field (the 2 bytes save are mostly trouble)
 #   - keep track of uncompressed -chunk- size (to preallocate memory better)
 #   - keep track of chain base or size (probably not that useful anymore)
 coreconfigitem(
     b'experimental',
     b'revlogv2',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'revisions.disambiguatewithin',
     default=None,
 )
 coreconfigitem(
     b'experimental',
     b'rust.index',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'server.allow-hidden-access',
     default=list,
 )
 coreconfigitem(
     b'experimental',
     b'server.filesdata.recommended-batch-size',
     default=50000,
 )
 coreconfigitem(
     b'experimental',
     b'server.manifestdata.recommended-batch-size',
     default=100000,
 )
 coreconfigitem(
     b'experimental',
     b'server.stream-narrow-clones',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'single-head-per-branch',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'single-head-per-branch:account-closed-heads',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'single-head-per-branch:public-changes-only',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'sparse-read',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'sparse-read.density-threshold',
     default=0.50,
 )
 coreconfigitem(
     b'experimental',
     b'sparse-read.min-gap-size',
     default=b'65K',
 )
 coreconfigitem(
     b'experimental',
+    b'stream-v3',
+    default=False,
+)
+coreconfigitem(
+    b'experimental',
     b'treemanifest',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'update.atomic-file',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'web.full-garbage-collection-rate',
     default=1,  # still forcing a full collection on each request
 )
 coreconfigitem(
     b'experimental',
     b'worker.wdir-get-thread-safe',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'worker.repository-upgrade',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'xdiff',
     default=False,
 )
 coreconfigitem(
     b'extensions',
     b'[^:]*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'extensions',
     b'[^:]*:required',
     default=False,
     generic=True,
 )
 coreconfigitem(
     b'extdata',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'format',
     b'bookmarks-in-store',
     default=False,
 )
 coreconfigitem(
     b'format',
     b'chunkcachesize',
     default=None,
     experimental=True,
 )
 coreconfigitem(
     # Enable this dirstate format *when creating a new repository*.
     # Which format to use for existing repos is controlled by .hg/requires
     b'format',
     b'use-dirstate-v2',
     default=False,
     experimental=True,
     alias=[(b'format', b'exp-rc-dirstate-v2')],
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-v2.automatic-upgrade-of-mismatching-repositories',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-v2.automatic-upgrade-of-mismatching-repositories:quiet',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-tracked-hint',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-tracked-hint.version',
     default=1,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-tracked-hint.automatic-upgrade-of-mismatching-repositories',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-dirstate-tracked-hint.automatic-upgrade-of-mismatching-repositories:quiet',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'dotencode',
     default=True,
 )
 coreconfigitem(
     b'format',
     b'generaldelta',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'manifestcachesize',
     default=None,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'maxchainlen',
     default=dynamicdefault,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'obsstore-version',
     default=None,
 )
 coreconfigitem(
     b'format',
     b'sparse-revlog',
     default=True,
 )
 coreconfigitem(
     b'format',
     b'revlog-compression',
     default=lambda: [b'zstd', b'zlib'],
     alias=[(b'experimental', b'format.compression')],
 )
 # Experimental TODOs:
 #
 # * Same as for revlogv2 (but for the reduction of the number of files)
 # * Actually computing the rank of changesets
 # * Improvement to investigate
 #   - storing .hgtags fnode
 #   - storing branch related identifier
 
 coreconfigitem(
     b'format',
     b'exp-use-changelog-v2',
     default=None,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'usefncache',
     default=True,
 )
 coreconfigitem(
     b'format',
     b'usegeneraldelta',
     default=True,
 )
 coreconfigitem(
     b'format',
     b'usestore',
     default=True,
 )
 
 
 def _persistent_nodemap_default():
     """compute `use-persistent-nodemap` default value
 
     The feature is disabled unless a fast implementation is available.
     """
     from . import policy
 
     return policy.importrust('revlog') is not None
 
 
 coreconfigitem(
     b'format',
     b'use-persistent-nodemap',
     default=_persistent_nodemap_default,
 )
 coreconfigitem(
     b'format',
     b'exp-use-copies-side-data-changeset',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-share-safe',
     default=True,
 )
 coreconfigitem(
     b'format',
     b'use-share-safe.automatic-upgrade-of-mismatching-repositories',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'format',
     b'use-share-safe.automatic-upgrade-of-mismatching-repositories:quiet',
     default=False,
     experimental=True,
 )
 
 # Moving this on by default means we are confident about the scaling of phases.
 # This is not garanteed to be the case at the time this message is written.
 coreconfigitem(
     b'format',
     b'use-internal-phase',
     default=False,
     experimental=True,
 )
 # The interaction between the archived phase and obsolescence markers needs to
 # be sorted out before wider usage of this are to be considered.
 #
 # At the time this message is written, behavior when archiving obsolete
 # changeset differ significantly from stripping. As part of stripping, we also
 # remove the obsolescence marker associated to the stripped changesets,
 # revealing the precedecessors changesets when applicable. When archiving, we
 # don't touch the obsolescence markers, keeping everything hidden. This can
 # result in quite confusing situation for people combining exchanging draft
 # with the archived phases. As some markers needed by others may be skipped
 # during exchange.
 coreconfigitem(
     b'format',
     b'exp-archived-phase',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'shelve',
     b'store',
     default=b'internal',
     experimental=True,
 )
 coreconfigitem(
     b'fsmonitor',
     b'warn_when_unused',
     default=True,
 )
 coreconfigitem(
     b'fsmonitor',
     b'warn_update_file_count',
     default=50000,
 )
 coreconfigitem(
     b'fsmonitor',
     b'warn_update_file_count_rust',
     default=400000,
 )
 coreconfigitem(
     b'help',
     br'hidden-command\..*',
     default=False,
     generic=True,
 )
 coreconfigitem(
     b'help',
     br'hidden-topic\..*',
     default=False,
     generic=True,
 )
 coreconfigitem(
     b'hooks',
     b'[^:]*',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'hooks',
     b'.*:run-with-plain',
     default=True,
     generic=True,
 )
 coreconfigitem(
     b'hgweb-paths',
     b'.*',
     default=list,
     generic=True,
 )
 coreconfigitem(
     b'hostfingerprints',
     b'.*',
     default=list,
     generic=True,
 )
 coreconfigitem(
     b'hostsecurity',
     b'ciphers',
     default=None,
 )
 coreconfigitem(
     b'hostsecurity',
     b'minimumprotocol',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'hostsecurity',
     b'.*:minimumprotocol$',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'hostsecurity',
     b'.*:ciphers$',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'hostsecurity',
     b'.*:fingerprints$',
     default=list,
     generic=True,
 )
 coreconfigitem(
     b'hostsecurity',
     b'.*:verifycertsfile$',
     default=None,
     generic=True,
 )
 
 coreconfigitem(
     b'http_proxy',
     b'always',
     default=False,
 )
 coreconfigitem(
     b'http_proxy',
     b'host',
     default=None,
 )
 coreconfigitem(
     b'http_proxy',
     b'no',
     default=list,
 )
 coreconfigitem(
     b'http_proxy',
     b'passwd',
     default=None,
 )
 coreconfigitem(
     b'http_proxy',
     b'user',
     default=None,
 )
 
 coreconfigitem(
     b'http',
     b'timeout',
     default=None,
 )
 
 coreconfigitem(
     b'logtoprocess',
     b'commandexception',
     default=None,
 )
 coreconfigitem(
     b'logtoprocess',
     b'commandfinish',
     default=None,
 )
 coreconfigitem(
     b'logtoprocess',
     b'command',
     default=None,
 )
 coreconfigitem(
     b'logtoprocess',
     b'develwarn',
     default=None,
 )
 coreconfigitem(
     b'logtoprocess',
     b'uiblocked',
     default=None,
 )
 coreconfigitem(
     b'merge',
     b'checkunknown',
     default=b'abort',
 )
 coreconfigitem(
     b'merge',
     b'checkignored',
     default=b'abort',
 )
 coreconfigitem(
     b'experimental',
     b'merge.checkpathconflicts',
     default=False,
 )
 coreconfigitem(
     b'merge',
     b'followcopies',
     default=True,
 )
 coreconfigitem(
     b'merge',
     b'on-failure',
     default=b'continue',
 )
 coreconfigitem(
     b'merge',
     b'preferancestor',
     default=lambda: [b'*'],
     experimental=True,
 )
 coreconfigitem(
     b'merge',
     b'strict-capability-check',
     default=False,
 )
 coreconfigitem(
     b'merge',
     b'disable-partial-tools',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     b'.*',
     default=None,
     generic=True,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     br'.*\.patterns',
     default=dynamicdefault,
     generic=True,
     priority=-1,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     br'.*\.executable$',
     default=dynamicdefault,
     generic=True,
     priority=-1,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     br'.*\.order',
     default=0,
     generic=True,
     priority=-1,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     br'.*\.args',
     default=b"$local $base $other",
     generic=True,
     priority=-1,
     experimental=True,
 )
 coreconfigitem(
     b'partial-merge-tools',
     br'.*\.disable',
     default=False,
     generic=True,
     priority=-1,
     experimental=True,
 )
 coreconfigitem(
     b'merge-tools',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.args$',
     default=b"$local $base $other",
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.binary$',
     default=False,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.check$',
     default=list,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.checkchanged$',
     default=False,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.executable$',
     default=dynamicdefault,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.fixeol$',
     default=False,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.gui$',
     default=False,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.mergemarkers$',
     default=b'basic',
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.mergemarkertemplate$',
     default=dynamicdefault,  # take from command-templates.mergemarker
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.priority$',
     default=0,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.premerge$',
     default=dynamicdefault,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.regappend$',
     default=b"",
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'merge-tools',
     br'.*\.symlink$',
     default=False,
     generic=True,
     priority=-1,
 )
 coreconfigitem(
     b'pager',
     b'attend-.*',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'pager',
     b'ignore',
     default=list,
 )
 coreconfigitem(
     b'pager',
     b'pager',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'patch',
     b'eol',
     default=b'strict',
 )
 coreconfigitem(
     b'patch',
     b'fuzz',
     default=2,
 )
 coreconfigitem(
     b'paths',
     b'default',
     default=None,
 )
 coreconfigitem(
     b'paths',
     b'default-push',
     default=None,
 )
 coreconfigitem(
     b'paths',
     b'[^:]*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'paths',
     b'.*:bookmarks.mode',
     default='default',
     generic=True,
 )
 coreconfigitem(
     b'paths',
     b'.*:multi-urls',
     default=False,
     generic=True,
 )
 coreconfigitem(
     b'paths',
     b'.*:pushrev',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'paths',
     b'.*:pushurl',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'paths',
     b'.*:pulled-delta-reuse-policy',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'phases',
     b'checksubrepos',
     default=b'follow',
 )
 coreconfigitem(
     b'phases',
     b'new-commit',
     default=b'draft',
 )
 coreconfigitem(
     b'phases',
     b'publish',
     default=True,
 )
 coreconfigitem(
     b'profiling',
     b'enabled',
     default=False,
 )
 coreconfigitem(
     b'profiling',
     b'format',
     default=b'text',
 )
 coreconfigitem(
     b'profiling',
     b'freq',
     default=1000,
 )
 coreconfigitem(
     b'profiling',
     b'limit',
     default=30,
 )
 coreconfigitem(
     b'profiling',
     b'nested',
     default=0,
 )
 coreconfigitem(
     b'profiling',
     b'output',
     default=None,
 )
 coreconfigitem(
     b'profiling',
     b'showmax',
     default=0.999,
 )
 coreconfigitem(
     b'profiling',
     b'showmin',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'profiling',
     b'showtime',
     default=True,
 )
 coreconfigitem(
     b'profiling',
     b'sort',
     default=b'inlinetime',
 )
 coreconfigitem(
     b'profiling',
     b'statformat',
     default=b'hotpath',
 )
 coreconfigitem(
     b'profiling',
     b'time-track',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'profiling',
     b'type',
     default=b'stat',
 )
 coreconfigitem(
     b'progress',
     b'assume-tty',
     default=False,
 )
 coreconfigitem(
     b'progress',
     b'changedelay',
     default=1,
 )
 coreconfigitem(
     b'progress',
     b'clear-complete',
     default=True,
 )
 coreconfigitem(
     b'progress',
     b'debug',
     default=False,
 )
 coreconfigitem(
     b'progress',
     b'delay',
     default=3,
 )
 coreconfigitem(
     b'progress',
     b'disable',
     default=False,
 )
 coreconfigitem(
     b'progress',
     b'estimateinterval',
     default=60.0,
 )
 coreconfigitem(
     b'progress',
     b'format',
     default=lambda: [b'topic', b'bar', b'number', b'estimate'],
 )
 coreconfigitem(
     b'progress',
     b'refresh',
     default=0.1,
 )
 coreconfigitem(
     b'progress',
     b'width',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'pull',
     b'confirm',
     default=False,
 )
 coreconfigitem(
     b'push',
     b'pushvars.server',
     default=False,
 )
 coreconfigitem(
     b'rewrite',
     b'backup-bundle',
     default=True,
     alias=[(b'ui', b'history-editing-backup')],
 )
 coreconfigitem(
     b'rewrite',
     b'update-timestamp',
     default=False,
 )
 coreconfigitem(
     b'rewrite',
     b'empty-successor',
     default=b'skip',
     experimental=True,
 )
 # experimental as long as format.use-dirstate-v2 is.
 coreconfigitem(
     b'storage',
     b'dirstate-v2.slow-path',
     default=b"abort",
     experimental=True,
 )
 coreconfigitem(
     b'storage',
     b'new-repo-backend',
     default=b'revlogv1',
     experimental=True,
 )
 coreconfigitem(
     b'storage',
     b'revlog.optimize-delta-parent-choice',
     default=True,
     alias=[(b'format', b'aggressivemergedeltas')],
 )
 coreconfigitem(
     b'storage',
     b'revlog.delta-parent-search.candidate-group-chunk-size',
     default=20,
 )
 coreconfigitem(
     b'storage',
     b'revlog.issue6528.fix-incoming',
     default=True,
 )
 # experimental as long as rust is experimental (or a C version is implemented)
 coreconfigitem(
     b'storage',
     b'revlog.persistent-nodemap.mmap',
     default=True,
 )
 # experimental as long as format.use-persistent-nodemap is.
 coreconfigitem(
     b'storage',
     b'revlog.persistent-nodemap.slow-path',
     default=b"abort",
 )
 
 coreconfigitem(
     b'storage',
     b'revlog.reuse-external-delta',
     default=True,
 )
 # This option is True unless `format.generaldelta` is set.
 coreconfigitem(
     b'storage',
     b'revlog.reuse-external-delta-parent',
     default=None,
 )
 coreconfigitem(
     b'storage',
     b'revlog.zlib.level',
     default=None,
 )
 coreconfigitem(
     b'storage',
     b'revlog.zstd.level',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bookmarks-pushkey-compat',
     default=True,
 )
 coreconfigitem(
     b'server',
     b'bundle1',
     default=True,
 )
 coreconfigitem(
     b'server',
     b'bundle1gd',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bundle1.pull',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bundle1gd.pull',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bundle1.push',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bundle1gd.push',
     default=None,
 )
 coreconfigitem(
     b'server',
     b'bundle2.stream',
     default=True,
     alias=[(b'experimental', b'bundle2.stream')],
 )
 coreconfigitem(
     b'server',
     b'compressionengines',
     default=list,
 )
 coreconfigitem(
     b'server',
     b'concurrent-push-mode',
     default=b'check-related',
 )
 coreconfigitem(
     b'server',
     b'disablefullbundle',
     default=False,
 )
 coreconfigitem(
     b'server',
     b'maxhttpheaderlen',
     default=1024,
 )
 coreconfigitem(
     b'server',
     b'pullbundle',
     default=True,
 )
 coreconfigitem(
     b'server',
     b'preferuncompressed',
     default=False,
 )
 coreconfigitem(
     b'server',
     b'streamunbundle',
     default=False,
 )
 coreconfigitem(
     b'server',
     b'uncompressed',
     default=True,
 )
 coreconfigitem(
     b'server',
     b'uncompressedallowsecret',
     default=False,
 )
 coreconfigitem(
     b'server',
     b'view',
     default=b'served',
 )
 coreconfigitem(
     b'server',
     b'validate',
     default=False,
 )
 coreconfigitem(
     b'server',
     b'zliblevel',
     default=-1,
 )
 coreconfigitem(
     b'server',
     b'zstdlevel',
     default=3,
 )
 coreconfigitem(
     b'share',
     b'pool',
     default=None,
 )
 coreconfigitem(
     b'share',
     b'poolnaming',
     default=b'identity',
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-not-safe',
     default=b'abort',
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-safe',
     default=b'abort',
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-not-safe.warn',
     default=True,
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-safe.warn',
     default=True,
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-not-safe:verbose-upgrade',
     default=True,
 )
 coreconfigitem(
     b'share',
     b'safe-mismatch.source-safe:verbose-upgrade',
     default=True,
 )
 coreconfigitem(
     b'shelve',
     b'maxbackups',
     default=10,
 )
 coreconfigitem(
     b'smtp',
     b'host',
     default=None,
 )
 coreconfigitem(
     b'smtp',
     b'local_hostname',
     default=None,
 )
 coreconfigitem(
     b'smtp',
     b'password',
     default=None,
 )
 coreconfigitem(
     b'smtp',
     b'port',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'smtp',
     b'tls',
     default=b'none',
 )
 coreconfigitem(
     b'smtp',
     b'username',
     default=None,
 )
 coreconfigitem(
     b'sparse',
     b'missingwarning',
     default=True,
     experimental=True,
 )
 coreconfigitem(
     b'subrepos',
     b'allowed',
     default=dynamicdefault,  # to make backporting simpler
 )
 coreconfigitem(
     b'subrepos',
     b'hg:allowed',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'subrepos',
     b'git:allowed',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'subrepos',
     b'svn:allowed',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'templates',
     b'.*',
     default=None,
     generic=True,
 )
 coreconfigitem(
     b'templateconfig',
     b'.*',
     default=dynamicdefault,
     generic=True,
 )
 coreconfigitem(
     b'trusted',
     b'groups',
     default=list,
 )
 coreconfigitem(
     b'trusted',
     b'users',
     default=list,
 )
 coreconfigitem(
     b'ui',
     b'_usedassubrepo',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'allowemptycommit',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'archivemeta',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'askusername',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'available-memory',
     default=None,
 )
 
 coreconfigitem(
     b'ui',
     b'clonebundlefallback',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'clonebundleprefers',
     default=list,
 )
 coreconfigitem(
     b'ui',
     b'clonebundles',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'color',
     default=b'auto',
 )
 coreconfigitem(
     b'ui',
     b'commitsubrepos',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'debug',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'debugger',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'editor',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'ui',
     b'detailed-exit-code',
     default=False,
     experimental=True,
 )
 coreconfigitem(
     b'ui',
     b'fallbackencoding',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'forcecwd',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'forcemerge',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'formatdebug',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'formatjson',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'formatted',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'interactive',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'interface',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'interface.chunkselector',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'large-file-limit',
     default=10 * (2 ** 20),
 )
 coreconfigitem(
     b'ui',
     b'logblockedtimes',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'merge',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'mergemarkers',
     default=b'basic',
 )
 coreconfigitem(
     b'ui',
     b'message-output',
     default=b'stdio',
 )
 coreconfigitem(
     b'ui',
     b'nontty',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'origbackuppath',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'paginate',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'patch',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'portablefilenames',
     default=b'warn',
 )
 coreconfigitem(
     b'ui',
     b'promptecho',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'quiet',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'quietbookmarkmove',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'relative-paths',
     default=b'legacy',
 )
 coreconfigitem(
     b'ui',
     b'remotecmd',
     default=b'hg',
 )
 coreconfigitem(
     b'ui',
     b'report_untrusted',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'rollback',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'signal-safe-lock',
     default=True,
 )
 coreconfigitem(
     b'ui',
     b'slash',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'ssh',
     default=b'ssh',
 )
 coreconfigitem(
     b'ui',
     b'ssherrorhint',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'statuscopies',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'strict',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'style',
     default=b'',
 )
 coreconfigitem(
     b'ui',
     b'supportcontact',
     default=None,
 )
 coreconfigitem(
     b'ui',
     b'textwidth',
     default=78,
 )
 coreconfigitem(
     b'ui',
     b'timeout',
     default=b'600',
 )
 coreconfigitem(
     b'ui',
     b'timeout.warn',
     default=0,
 )
 coreconfigitem(
     b'ui',
     b'timestamp-output',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'traceback',
     default=False,
 )
 coreconfigitem(
     b'ui',
     b'tweakdefaults',
     default=False,
 )
 coreconfigitem(b'ui', b'username', alias=[(b'ui', b'user')])
 coreconfigitem(
     b'ui',
     b'verbose',
     default=False,
 )
 coreconfigitem(
     b'verify',
     b'skipflags',
     default=0,
 )
 coreconfigitem(
     b'web',
     b'allowbz2',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'allowgz',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'allow-pull',
     alias=[(b'web', b'allowpull')],
     default=True,
 )
 coreconfigitem(
     b'web',
     b'allow-push',
     alias=[(b'web', b'allow_push')],
     default=list,
 )
 coreconfigitem(
     b'web',
     b'allowzip',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'archivesubrepos',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'cache',
     default=True,
 )
 coreconfigitem(
     b'web',
     b'comparisoncontext',
     default=5,
 )
 coreconfigitem(
     b'web',
     b'contact',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'deny_push',
     default=list,
 )
 coreconfigitem(
     b'web',
     b'guessmime',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'hidden',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'labels',
     default=list,
 )
 coreconfigitem(
     b'web',
     b'logoimg',
     default=b'hglogo.png',
 )
 coreconfigitem(
     b'web',
     b'logourl',
     default=b'https://mercurial-scm.org/',
 )
 coreconfigitem(
     b'web',
     b'accesslog',
     default=b'-',
 )
 coreconfigitem(
     b'web',
     b'address',
     default=b'',
 )
 coreconfigitem(
     b'web',
     b'allow-archive',
     alias=[(b'web', b'allow_archive')],
     default=list,
 )
 coreconfigitem(
     b'web',
     b'allow_read',
     default=list,
 )
 coreconfigitem(
     b'web',
     b'baseurl',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'cacerts',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'certificate',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'collapse',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'csp',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'deny_read',
     default=list,
 )
 coreconfigitem(
     b'web',
     b'descend',
     default=True,
 )
 coreconfigitem(
     b'web',
     b'description',
     default=b"",
 )
 coreconfigitem(
     b'web',
     b'encoding',
     default=lambda: encoding.encoding,
 )
 coreconfigitem(
     b'web',
     b'errorlog',
     default=b'-',
 )
 coreconfigitem(
     b'web',
     b'ipv6',
     default=False,
 )
 coreconfigitem(
     b'web',
     b'maxchanges',
     default=10,
 )
 coreconfigitem(
     b'web',
     b'maxfiles',
     default=10,
 )
 coreconfigitem(
     b'web',
     b'maxshortchanges',
     default=60,
 )
 coreconfigitem(
     b'web',
     b'motd',
     default=b'',
 )
 coreconfigitem(
     b'web',
     b'name',
     default=dynamicdefault,
 )
 coreconfigitem(
     b'web',
     b'port',
     default=8000,
 )
 coreconfigitem(
     b'web',
     b'prefix',
     default=b'',
 )
 coreconfigitem(
     b'web',
     b'push_ssl',
     default=True,
 )
 coreconfigitem(
     b'web',
     b'refreshinterval',
     default=20,
 )
 coreconfigitem(
     b'web',
     b'server-header',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'static',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'staticurl',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'stripes',
     default=1,
 )
 coreconfigitem(
     b'web',
     b'style',
     default=b'paper',
 )
 coreconfigitem(
     b'web',
     b'templates',
     default=None,
 )
 coreconfigitem(
     b'web',
     b'view',
     default=b'served',
     experimental=True,
 )
 coreconfigitem(
     b'worker',
     b'backgroundclose',
     default=dynamicdefault,
 )
 # Windows defaults to a limit of 512 open files. A buffer of 128
 # should give us enough headway.
 coreconfigitem(
     b'worker',
     b'backgroundclosemaxqueue',
     default=384,
 )
 coreconfigitem(
     b'worker',
     b'backgroundcloseminfilecount',
     default=2048,
 )
 coreconfigitem(
     b'worker',
     b'backgroundclosethreadcount',
     default=4,
 )
 coreconfigitem(
     b'worker',
     b'enabled',
     default=True,
 )
 coreconfigitem(
     b'worker',
     b'numcpus',
     default=None,
 )
 
 # Rebase related configuration moved to core because other extension are doing
 # strange things. For example, shelve import the extensions to reuse some bit
 # without formally loading it.
 coreconfigitem(
     b'commands',
     b'rebase.requiredest',
     default=False,
 )
 coreconfigitem(
     b'experimental',
     b'rebaseskipobsolete',
     default=True,
 )
 coreconfigitem(
     b'rebase',
     b'singletransaction',
     default=False,
 )
 coreconfigitem(
     b'rebase',
     b'experimental.inmemory',
     default=False,
 )
 
 # This setting controls creation of a rebase_source extra field
 # during rebase. When False, no such field is created. This is
 # useful eg for incrementally converting changesets and then
 # rebasing them onto an existing repo.
 # WARNING: this is an advanced setting reserved for people who know
 # exactly what they are doing. Misuse of this setting can easily
 # result in obsmarker cycles and a vivid headache.
 coreconfigitem(
     b'rebase',
     b'store-source',
     default=True,
     experimental=True,
 )

mercurial/exchange.py

@@ -1,2875 +1,2875
 # exchange.py - utility to exchange data between repos.
 #
 # Copyright 2005-2007 Olivia Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 
 import collections
 import weakref
 
 from .i18n import _
 from .node import (
     hex,
     nullrev,
 )
 from . import (
     bookmarks as bookmod,
     bundle2,
     bundlecaches,
     changegroup,
     discovery,
     error,
     lock as lockmod,
     logexchange,
     narrowspec,
     obsolete,
     obsutil,
     phases,
     pushkey,
     pycompat,
     requirements,
     scmutil,
     streamclone,
     url as urlmod,
     util,
     wireprototypes,
 )
 from .utils import (
     hashutil,
     stringutil,
     urlutil,
 )
 from .interfaces import repository
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 _NARROWACL_SECTION = b'narrowacl'
 
 
 def readbundle(ui, fh, fname, vfs=None):
     header = changegroup.readexactly(fh, 4)
 
     alg = None
     if not fname:
         fname = b"stream"
         if not header.startswith(b'HG') and header.startswith(b'\0'):
             fh = changegroup.headerlessfixup(fh, header)
             header = b"HG10"
             alg = b'UN'
     elif vfs:
         fname = vfs.join(fname)
 
     magic, version = header[0:2], header[2:4]
 
     if magic != b'HG':
         raise error.Abort(_(b'%s: not a Mercurial bundle') % fname)
     if version == b'10':
         if alg is None:
             alg = changegroup.readexactly(fh, 2)
         return changegroup.cg1unpacker(fh, alg)
     elif version.startswith(b'2'):
         return bundle2.getunbundler(ui, fh, magicstring=magic + version)
     elif version == b'S1':
         return streamclone.streamcloneapplier(fh)
     else:
         raise error.Abort(
             _(b'%s: unknown bundle version %s') % (fname, version)
         )
 
 
 def _format_params(params):
     parts = []
     for key, value in sorted(params.items()):
         value = urlreq.quote(value)
         parts.append(b"%s=%s" % (key, value))
     return b';'.join(parts)
 
 
 def getbundlespec(ui, fh):
     """Infer the bundlespec from a bundle file handle.
 
     The input file handle is seeked and the original seek position is not
     restored.
     """
 
     def speccompression(alg):
         try:
             return util.compengines.forbundletype(alg).bundletype()[0]
         except KeyError:
             return None
 
     params = {}
 
     b = readbundle(ui, fh, None)
     if isinstance(b, changegroup.cg1unpacker):
         alg = b._type
         if alg == b'_truncatedBZ':
             alg = b'BZ'
         comp = speccompression(alg)
         if not comp:
             raise error.Abort(_(b'unknown compression algorithm: %s') % alg)
         return b'%s-v1' % comp
     elif isinstance(b, bundle2.unbundle20):
         if b'Compression' in b.params:
             comp = speccompression(b.params[b'Compression'])
             if not comp:
                 raise error.Abort(
                     _(b'unknown compression algorithm: %s') % comp
                 )
         else:
             comp = b'none'
 
         version = None
         for part in b.iterparts():
             if part.type == b'changegroup':
                 cgversion = part.params[b'version']
                 if cgversion in (b'01', b'02'):
                     version = b'v2'
                 elif cgversion in (b'03',):
                     version = b'v2'
                     params[b'cg.version'] = cgversion
                 else:
                     raise error.Abort(
                         _(
                             b'changegroup version %s does not have '
                             b'a known bundlespec'
                         )
                         % version,
                         hint=_(b'try upgrading your Mercurial client'),
                     )
             elif part.type == b'stream2' and version is None:
                 # A stream2 part requires to be part of a v2 bundle
                 requirements = urlreq.unquote(part.params[b'requirements'])
                 splitted = requirements.split()
                 params = bundle2._formatrequirementsparams(splitted)
                 return b'none-v2;stream=v2;%s' % params
             elif part.type == b'obsmarkers':
                 params[b'obsolescence'] = b'yes'
                 if not part.mandatory:
                     params[b'obsolescence-mandatory'] = b'no'
 
         if not version:
             raise error.Abort(
                 _(b'could not identify changegroup version in bundle')
             )
         spec = b'%s-%s' % (comp, version)
         if params:
             spec += b';'
             spec += _format_params(params)
         return spec
 
     elif isinstance(b, streamclone.streamcloneapplier):
         requirements = streamclone.readbundle1header(fh)[2]
         formatted = bundle2._formatrequirementsparams(requirements)
         return b'none-packed1;%s' % formatted
     else:
         raise error.Abort(_(b'unknown bundle type: %s') % b)
 
 
 def _computeoutgoing(repo, heads, common):
     """Computes which revs are outgoing given a set of common
     and a set of heads.
 
     This is a separate function so extensions can have access to
     the logic.
 
     Returns a discovery.outgoing object.
     """
     cl = repo.changelog
     if common:
         hasnode = cl.hasnode
         common = [n for n in common if hasnode(n)]
     else:
         common = [repo.nullid]
     if not heads:
         heads = cl.heads()
     return discovery.outgoing(repo, common, heads)
 
 
 def _checkpublish(pushop):
     repo = pushop.repo
     ui = repo.ui
     behavior = ui.config(b'experimental', b'auto-publish')
     if pushop.publish or behavior not in (b'warn', b'confirm', b'abort'):
         return
     remotephases = listkeys(pushop.remote, b'phases')
     if not remotephases.get(b'publishing', False):
         return
 
     if pushop.revs is None:
         published = repo.filtered(b'served').revs(b'not public()')
     else:
         published = repo.revs(b'::%ln - public()', pushop.revs)
         # we want to use pushop.revs in the revset even if they themselves are
         # secret, but we don't want to have anything that the server won't see
         # in the result of this expression
         published &= repo.filtered(b'served')
     if published:
         if behavior == b'warn':
             ui.warn(
                 _(b'%i changesets about to be published\n') % len(published)
             )
         elif behavior == b'confirm':
             if ui.promptchoice(
                 _(b'push and publish %i changesets (yn)?$$ &Yes $$ &No')
                 % len(published)
             ):
                 raise error.CanceledError(_(b'user quit'))
         elif behavior == b'abort':
             msg = _(b'push would publish %i changesets') % len(published)
             hint = _(
                 b"use --publish or adjust 'experimental.auto-publish'"
                 b" config"
             )
             raise error.Abort(msg, hint=hint)
 
 
 def _forcebundle1(op):
     """return true if a pull/push must use bundle1
 
     This function is used to allow testing of the older bundle version"""
     ui = op.repo.ui
     # The goal is this config is to allow developer to choose the bundle
     # version used during exchanged. This is especially handy during test.
     # Value is a list of bundle version to be picked from, highest version
     # should be used.
     #
     # developer config: devel.legacy.exchange
     exchange = ui.configlist(b'devel', b'legacy.exchange')
     forcebundle1 = b'bundle2' not in exchange and b'bundle1' in exchange
     return forcebundle1 or not op.remote.capable(b'bundle2')
 
 
 class pushoperation:
     """A object that represent a single push operation
 
     Its purpose is to carry push related state and very common operations.
 
     A new pushoperation should be created at the beginning of each push and
     discarded afterward.
     """
 
     def __init__(
         self,
         repo,
         remote,
         force=False,
         revs=None,
         newbranch=False,
         bookmarks=(),
         publish=False,
         pushvars=None,
     ):
         # repo we push from
         self.repo = repo
         self.ui = repo.ui
         # repo we push to
         self.remote = remote
         # force option provided
         self.force = force
         # revs to be pushed (None is "all")
         self.revs = revs
         # bookmark explicitly pushed
         self.bookmarks = bookmarks
         # allow push of new branch
         self.newbranch = newbranch
         # step already performed
         # (used to check what steps have been already performed through bundle2)
         self.stepsdone = set()
         # Integer version of the changegroup push result
         # - None means nothing to push
         # - 0 means HTTP error
         # - 1 means we pushed and remote head count is unchanged *or*
         #   we have outgoing changesets but refused to push
         # - other values as described by addchangegroup()
         self.cgresult = None
         # Boolean value for the bookmark push
         self.bkresult = None
         # discover.outgoing object (contains common and outgoing data)
         self.outgoing = None
         # all remote topological heads before the push
         self.remoteheads = None
         # Details of the remote branch pre and post push
         #
         # mapping: {'branch': ([remoteheads],
         #                      [newheads],
         #                      [unsyncedheads],
         #                      [discardedheads])}
         # - branch: the branch name
         # - remoteheads: the list of remote heads known locally
         #                None if the branch is new
         # - newheads: the new remote heads (known locally) with outgoing pushed
         # - unsyncedheads: the list of remote heads unknown locally.
         # - discardedheads: the list of remote heads made obsolete by the push
         self.pushbranchmap = None
         # testable as a boolean indicating if any nodes are missing locally.
         self.incoming = None
         # summary of the remote phase situation
         self.remotephases = None
         # phases changes that must be pushed along side the changesets
         self.outdatedphases = None
         # phases changes that must be pushed if changeset push fails
         self.fallbackoutdatedphases = None
         # outgoing obsmarkers
         self.outobsmarkers = set()
         # outgoing bookmarks, list of (bm, oldnode | '', newnode | '')
         self.outbookmarks = []
         # transaction manager
         self.trmanager = None
         # map { pushkey partid -> callback handling failure}
         # used to handle exception from mandatory pushkey part failure
         self.pkfailcb = {}
         # an iterable of pushvars or None
         self.pushvars = pushvars
         # publish pushed changesets
         self.publish = publish
 
     @util.propertycache
     def futureheads(self):
         """future remote heads if the changeset push succeeds"""
         return self.outgoing.ancestorsof
 
     @util.propertycache
     def fallbackheads(self):
         """future remote heads if the changeset push fails"""
         if self.revs is None:
             # not target to push, all common are relevant
             return self.outgoing.commonheads
         unfi = self.repo.unfiltered()
         # I want cheads = heads(::ancestorsof and ::commonheads)
         # (ancestorsof is revs with secret changeset filtered out)
         #
         # This can be expressed as:
         #     cheads = ( (ancestorsof and ::commonheads)
         #              + (commonheads and ::ancestorsof))"
         #              )
         #
         # while trying to push we already computed the following:
         #     common = (::commonheads)
         #     missing = ((commonheads::ancestorsof) - commonheads)
         #
         # We can pick:
         # * ancestorsof part of common (::commonheads)
         common = self.outgoing.common
         rev = self.repo.changelog.index.rev
         cheads = [node for node in self.revs if rev(node) in common]
         # and
         # * commonheads parents on missing
         revset = unfi.set(
             b'%ln and parents(roots(%ln))',
             self.outgoing.commonheads,
             self.outgoing.missing,
         )
         cheads.extend(c.node() for c in revset)
         return cheads
 
     @property
     def commonheads(self):
         """set of all common heads after changeset bundle push"""
         if self.cgresult:
             return self.futureheads
         else:
             return self.fallbackheads
 
 
 # mapping of message used when pushing bookmark
 bookmsgmap = {
     b'update': (
         _(b"updating bookmark %s\n"),
         _(b'updating bookmark %s failed\n'),
     ),
     b'export': (
         _(b"exporting bookmark %s\n"),
         _(b'exporting bookmark %s failed\n'),
     ),
     b'delete': (
         _(b"deleting remote bookmark %s\n"),
         _(b'deleting remote bookmark %s failed\n'),
     ),
 }
 
 
 def push(
     repo,
     remote,
     force=False,
     revs=None,
     newbranch=False,
     bookmarks=(),
     publish=False,
     opargs=None,
 ):
     """Push outgoing changesets (limited by revs) from a local
     repository to remote. Return an integer:
       - None means nothing to push
       - 0 means HTTP error
       - 1 means we pushed and remote head count is unchanged *or*
         we have outgoing changesets but refused to push
       - other values as described by addchangegroup()
     """
     if opargs is None:
         opargs = {}
     pushop = pushoperation(
         repo,
         remote,
         force,
         revs,
         newbranch,
         bookmarks,
         publish,
         **pycompat.strkwargs(opargs)
     )
     if pushop.remote.local():
         missing = (
             set(pushop.repo.requirements) - pushop.remote.local().supported
         )
         if missing:
             msg = _(
                 b"required features are not"
                 b" supported in the destination:"
                 b" %s"
             ) % (b', '.join(sorted(missing)))
             raise error.Abort(msg)
 
     if not pushop.remote.canpush():
         raise error.Abort(_(b"destination does not support push"))
 
     if not pushop.remote.capable(b'unbundle'):
         raise error.Abort(
             _(
                 b'cannot push: destination does not support the '
                 b'unbundle wire protocol command'
             )
         )
     for category in sorted(bundle2.read_remote_wanted_sidedata(pushop.remote)):
         # Check that a computer is registered for that category for at least
         # one revlog kind.
         for kind, computers in repo._sidedata_computers.items():
             if computers.get(category):
                 break
         else:
             raise error.Abort(
                 _(
                     b'cannot push: required sidedata category not supported'
                     b" by this client: '%s'"
                 )
                 % pycompat.bytestr(category)
             )
     # get lock as we might write phase data
     wlock = lock = None
     try:
         # bundle2 push may receive a reply bundle touching bookmarks
         # requiring the wlock. Take it now to ensure proper ordering.
         maypushback = pushop.ui.configbool(b'experimental', b'bundle2.pushback')
         if (
             (not _forcebundle1(pushop))
             and maypushback
             and not bookmod.bookmarksinstore(repo)
         ):
             wlock = pushop.repo.wlock()
         lock = pushop.repo.lock()
         pushop.trmanager = transactionmanager(
             pushop.repo, b'push-response', pushop.remote.url()
         )
     except error.LockUnavailable as err:
         # source repo cannot be locked.
         # We do not abort the push, but just disable the local phase
         # synchronisation.
         msg = b'cannot lock source repository: %s\n' % stringutil.forcebytestr(
             err
         )
         pushop.ui.debug(msg)
 
     with wlock or util.nullcontextmanager():
         with lock or util.nullcontextmanager():
             with pushop.trmanager or util.nullcontextmanager():
                 pushop.repo.checkpush(pushop)
                 _checkpublish(pushop)
                 _pushdiscovery(pushop)
                 if not pushop.force:
                     _checksubrepostate(pushop)
                 if not _forcebundle1(pushop):
                     _pushbundle2(pushop)
                 _pushchangeset(pushop)
                 _pushsyncphase(pushop)
                 _pushobsolete(pushop)
                 _pushbookmark(pushop)
 
     if repo.ui.configbool(b'experimental', b'remotenames'):
         logexchange.pullremotenames(repo, remote)
 
     return pushop
 
 
 # list of steps to perform discovery before push
 pushdiscoveryorder = []
 
 # Mapping between step name and function
 #
 # This exists to help extensions wrap steps if necessary
 pushdiscoverymapping = {}
 
 
 def pushdiscovery(stepname):
     """decorator for function performing discovery before push
 
     The function is added to the step -> function mapping and appended to the
     list of steps.  Beware that decorated function will be added in order (this
     may matter).
 
     You can only use this decorator for a new step, if you want to wrap a step
     from an extension, change the pushdiscovery dictionary directly."""
 
     def dec(func):
         assert stepname not in pushdiscoverymapping
         pushdiscoverymapping[stepname] = func
         pushdiscoveryorder.append(stepname)
         return func
 
     return dec
 
 
 def _pushdiscovery(pushop):
     """Run all discovery steps"""
     for stepname in pushdiscoveryorder:
         step = pushdiscoverymapping[stepname]
         step(pushop)
 
 
 def _checksubrepostate(pushop):
     """Ensure all outgoing referenced subrepo revisions are present locally"""
 
     repo = pushop.repo
 
     # If the repository does not use subrepos, skip the expensive
     # manifest checks.
     if not len(repo.file(b'.hgsub')) or not len(repo.file(b'.hgsubstate')):
         return
 
     for n in pushop.outgoing.missing:
         ctx = repo[n]
 
         if b'.hgsub' in ctx.manifest() and b'.hgsubstate' in ctx.files():
             for subpath in sorted(ctx.substate):
                 sub = ctx.sub(subpath)
                 sub.verify(onpush=True)
 
 
 @pushdiscovery(b'changeset')
 def _pushdiscoverychangeset(pushop):
     """discover the changeset that need to be pushed"""
     fci = discovery.findcommonincoming
     if pushop.revs:
         commoninc = fci(
             pushop.repo,
             pushop.remote,
             force=pushop.force,
             ancestorsof=pushop.revs,
         )
     else:
         commoninc = fci(pushop.repo, pushop.remote, force=pushop.force)
     common, inc, remoteheads = commoninc
     fco = discovery.findcommonoutgoing
     outgoing = fco(
         pushop.repo,
         pushop.remote,
         onlyheads=pushop.revs,
         commoninc=commoninc,
         force=pushop.force,
     )
     pushop.outgoing = outgoing
     pushop.remoteheads = remoteheads
     pushop.incoming = inc
 
 
 @pushdiscovery(b'phase')
 def _pushdiscoveryphase(pushop):
     """discover the phase that needs to be pushed
 
     (computed for both success and failure case for changesets push)"""
     outgoing = pushop.outgoing
     unfi = pushop.repo.unfiltered()
     remotephases = listkeys(pushop.remote, b'phases')
 
     if (
         pushop.ui.configbool(b'ui', b'_usedassubrepo')
         and remotephases  # server supports phases
         and not pushop.outgoing.missing  # no changesets to be pushed
         and remotephases.get(b'publishing', False)
     ):
         # When:
         # - this is a subrepo push
         # - and remote support phase
         # - and no changeset are to be pushed
         # - and remote is publishing
         # We may be in issue 3781 case!
         # We drop the possible phase synchronisation done by
         # courtesy to publish changesets possibly locally draft
         # on the remote.
         pushop.outdatedphases = []
         pushop.fallbackoutdatedphases = []
         return
 
     pushop.remotephases = phases.remotephasessummary(
         pushop.repo, pushop.fallbackheads, remotephases
     )
     droots = pushop.remotephases.draftroots
 
     extracond = b''
     if not pushop.remotephases.publishing:
         extracond = b' and public()'
     revset = b'heads((%%ln::%%ln) %s)' % extracond
     # Get the list of all revs draft on remote by public here.
     # XXX Beware that revset break if droots is not strictly
     # XXX root we may want to ensure it is but it is costly
     fallback = list(unfi.set(revset, droots, pushop.fallbackheads))
     if not pushop.remotephases.publishing and pushop.publish:
         future = list(
             unfi.set(
                 b'%ln and (not public() or %ln::)', pushop.futureheads, droots
             )
         )
     elif not outgoing.missing:
         future = fallback
     else:
         # adds changeset we are going to push as draft
         #
         # should not be necessary for publishing server, but because of an
         # issue fixed in xxxxx we have to do it anyway.
         fdroots = list(
             unfi.set(b'roots(%ln  + %ln::)', outgoing.missing, droots)
         )
         fdroots = [f.node() for f in fdroots]
         future = list(unfi.set(revset, fdroots, pushop.futureheads))
     pushop.outdatedphases = future
     pushop.fallbackoutdatedphases = fallback
 
 
 @pushdiscovery(b'obsmarker')
 def _pushdiscoveryobsmarkers(pushop):
     if not obsolete.isenabled(pushop.repo, obsolete.exchangeopt):
         return
 
     if not pushop.repo.obsstore:
         return
 
     if b'obsolete' not in listkeys(pushop.remote, b'namespaces'):
         return
 
     repo = pushop.repo
     # very naive computation, that can be quite expensive on big repo.
     # However: evolution is currently slow on them anyway.
     nodes = (c.node() for c in repo.set(b'::%ln', pushop.futureheads))
     pushop.outobsmarkers = pushop.repo.obsstore.relevantmarkers(nodes)
 
 
 @pushdiscovery(b'bookmarks')
 def _pushdiscoverybookmarks(pushop):
     ui = pushop.ui
     repo = pushop.repo.unfiltered()
     remote = pushop.remote
     ui.debug(b"checking for updated bookmarks\n")
     ancestors = ()
     if pushop.revs:
         revnums = pycompat.maplist(repo.changelog.rev, pushop.revs)
         ancestors = repo.changelog.ancestors(revnums, inclusive=True)
 
     remotebookmark = bookmod.unhexlifybookmarks(listkeys(remote, b'bookmarks'))
 
     explicit = {
         repo._bookmarks.expandname(bookmark) for bookmark in pushop.bookmarks
     }
 
     comp = bookmod.comparebookmarks(repo, repo._bookmarks, remotebookmark)
     return _processcompared(pushop, ancestors, explicit, remotebookmark, comp)
 
 
 def _processcompared(pushop, pushed, explicit, remotebms, comp):
     """take decision on bookmarks to push to the remote repo
 
     Exists to help extensions alter this behavior.
     """
     addsrc, adddst, advsrc, advdst, diverge, differ, invalid, same = comp
 
     repo = pushop.repo
 
     for b, scid, dcid in advsrc:
         if b in explicit:
             explicit.remove(b)
         if not pushed or repo[scid].rev() in pushed:
             pushop.outbookmarks.append((b, dcid, scid))
     # search added bookmark
     for b, scid, dcid in addsrc:
         if b in explicit:
             explicit.remove(b)
             if bookmod.isdivergent(b):
                 pushop.ui.warn(_(b'cannot push divergent bookmark %s!\n') % b)
                 pushop.bkresult = 2
             else:
                 pushop.outbookmarks.append((b, b'', scid))
     # search for overwritten bookmark
     for b, scid, dcid in list(advdst) + list(diverge) + list(differ):
         if b in explicit:
             explicit.remove(b)
             pushop.outbookmarks.append((b, dcid, scid))
     # search for bookmark to delete
     for b, scid, dcid in adddst:
         if b in explicit:
             explicit.remove(b)
             # treat as "deleted locally"
             pushop.outbookmarks.append((b, dcid, b''))
     # identical bookmarks shouldn't get reported
     for b, scid, dcid in same:
         if b in explicit:
             explicit.remove(b)
 
     if explicit:
         explicit = sorted(explicit)
         # we should probably list all of them
         pushop.ui.warn(
             _(
                 b'bookmark %s does not exist on the local '
                 b'or remote repository!\n'
             )
             % explicit[0]
         )
         pushop.bkresult = 2
 
     pushop.outbookmarks.sort()
 
 
 def _pushcheckoutgoing(pushop):
     outgoing = pushop.outgoing
     unfi = pushop.repo.unfiltered()
     if not outgoing.missing:
         # nothing to push
         scmutil.nochangesfound(unfi.ui, unfi, outgoing.excluded)
         return False
     # something to push
     if not pushop.force:
         # if repo.obsstore == False --> no obsolete
         # then, save the iteration
         if unfi.obsstore:
             # this message are here for 80 char limit reason
             mso = _(b"push includes obsolete changeset: %s!")
             mspd = _(b"push includes phase-divergent changeset: %s!")
             mscd = _(b"push includes content-divergent changeset: %s!")
             mst = {
                 b"orphan": _(b"push includes orphan changeset: %s!"),
                 b"phase-divergent": mspd,
                 b"content-divergent": mscd,
             }
             # If we are to push if there is at least one
             # obsolete or unstable changeset in missing, at
             # least one of the missinghead will be obsolete or
             # unstable. So checking heads only is ok
             for node in outgoing.ancestorsof:
                 ctx = unfi[node]
                 if ctx.obsolete():
                     raise error.Abort(mso % ctx)
                 elif ctx.isunstable():
                     # TODO print more than one instability in the abort
                     # message
                     raise error.Abort(mst[ctx.instabilities()[0]] % ctx)
 
         discovery.checkheads(pushop)
     return True
 
 
 # List of names of steps to perform for an outgoing bundle2, order matters.
 b2partsgenorder = []
 
 # Mapping between step name and function
 #
 # This exists to help extensions wrap steps if necessary
 b2partsgenmapping = {}
 
 
 def b2partsgenerator(stepname, idx=None):
     """decorator for function generating bundle2 part
 
     The function is added to the step -> function mapping and appended to the
     list of steps.  Beware that decorated functions will be added in order
     (this may matter).
 
     You can only use this decorator for new steps, if you want to wrap a step
     from an extension, attack the b2partsgenmapping dictionary directly."""
 
     def dec(func):
         assert stepname not in b2partsgenmapping
         b2partsgenmapping[stepname] = func
         if idx is None:
             b2partsgenorder.append(stepname)
         else:
             b2partsgenorder.insert(idx, stepname)
         return func
 
     return dec
 
 
 def _pushb2ctxcheckheads(pushop, bundler):
     """Generate race condition checking parts
 
     Exists as an independent function to aid extensions
     """
     # * 'force' do not check for push race,
     # * if we don't push anything, there are nothing to check.
     if not pushop.force and pushop.outgoing.ancestorsof:
         allowunrelated = b'related' in bundler.capabilities.get(
             b'checkheads', ()
         )
         emptyremote = pushop.pushbranchmap is None
         if not allowunrelated or emptyremote:
             bundler.newpart(b'check:heads', data=iter(pushop.remoteheads))
         else:
             affected = set()
             for branch, heads in pushop.pushbranchmap.items():
                 remoteheads, newheads, unsyncedheads, discardedheads = heads
                 if remoteheads is not None:
                     remote = set(remoteheads)
                     affected |= set(discardedheads) & remote
                     affected |= remote - set(newheads)
             if affected:
                 data = iter(sorted(affected))
                 bundler.newpart(b'check:updated-heads', data=data)
 
 
 def _pushing(pushop):
     """return True if we are pushing anything"""
     return bool(
         pushop.outgoing.missing
         or pushop.outdatedphases
         or pushop.outobsmarkers
         or pushop.outbookmarks
     )
 
 
 @b2partsgenerator(b'check-bookmarks')
 def _pushb2checkbookmarks(pushop, bundler):
     """insert bookmark move checking"""
     if not _pushing(pushop) or pushop.force:
         return
     b2caps = bundle2.bundle2caps(pushop.remote)
     hasbookmarkcheck = b'bookmarks' in b2caps
     if not (pushop.outbookmarks and hasbookmarkcheck):
         return
     data = []
     for book, old, new in pushop.outbookmarks:
         data.append((book, old))
     checkdata = bookmod.binaryencode(pushop.repo, data)
     bundler.newpart(b'check:bookmarks', data=checkdata)
 
 
 @b2partsgenerator(b'check-phases')
 def _pushb2checkphases(pushop, bundler):
     """insert phase move checking"""
     if not _pushing(pushop) or pushop.force:
         return
     b2caps = bundle2.bundle2caps(pushop.remote)
     hasphaseheads = b'heads' in b2caps.get(b'phases', ())
     if pushop.remotephases is not None and hasphaseheads:
         # check that the remote phase has not changed
         checks = {p: [] for p in phases.allphases}
         checks[phases.public].extend(pushop.remotephases.publicheads)
         checks[phases.draft].extend(pushop.remotephases.draftroots)
         if any(checks.values()):
             for phase in checks:
                 checks[phase].sort()
             checkdata = phases.binaryencode(checks)
             bundler.newpart(b'check:phases', data=checkdata)
 
 
 @b2partsgenerator(b'changeset')
 def _pushb2ctx(pushop, bundler):
     """handle changegroup push through bundle2
 
     addchangegroup result is stored in the ``pushop.cgresult`` attribute.
     """
     if b'changesets' in pushop.stepsdone:
         return
     pushop.stepsdone.add(b'changesets')
     # Send known heads to the server for race detection.
     if not _pushcheckoutgoing(pushop):
         return
     pushop.repo.prepushoutgoinghooks(pushop)
 
     _pushb2ctxcheckheads(pushop, bundler)
 
     b2caps = bundle2.bundle2caps(pushop.remote)
     version = b'01'
     cgversions = b2caps.get(b'changegroup')
     if cgversions:  # 3.1 and 3.2 ship with an empty value
         cgversions = [
             v
             for v in cgversions
             if v in changegroup.supportedoutgoingversions(pushop.repo)
         ]
         if not cgversions:
             raise error.Abort(_(b'no common changegroup version'))
         version = max(cgversions)
 
     remote_sidedata = bundle2.read_remote_wanted_sidedata(pushop.remote)
     cgstream = changegroup.makestream(
         pushop.repo,
         pushop.outgoing,
         version,
         b'push',
         bundlecaps=b2caps,
         remote_sidedata=remote_sidedata,
     )
     cgpart = bundler.newpart(b'changegroup', data=cgstream)
     if cgversions:
         cgpart.addparam(b'version', version)
     if scmutil.istreemanifest(pushop.repo):
         cgpart.addparam(b'treemanifest', b'1')
     if repository.REPO_FEATURE_SIDE_DATA in pushop.repo.features:
         cgpart.addparam(b'exp-sidedata', b'1')
 
     def handlereply(op):
         """extract addchangegroup returns from server reply"""
         cgreplies = op.records.getreplies(cgpart.id)
         assert len(cgreplies[b'changegroup']) == 1
         pushop.cgresult = cgreplies[b'changegroup'][0][b'return']
 
     return handlereply
 
 
 @b2partsgenerator(b'phase')
 def _pushb2phases(pushop, bundler):
     """handle phase push through bundle2"""
     if b'phases' in pushop.stepsdone:
         return
     b2caps = bundle2.bundle2caps(pushop.remote)
     ui = pushop.repo.ui
 
     legacyphase = b'phases' in ui.configlist(b'devel', b'legacy.exchange')
     haspushkey = b'pushkey' in b2caps
     hasphaseheads = b'heads' in b2caps.get(b'phases', ())
 
     if hasphaseheads and not legacyphase:
         return _pushb2phaseheads(pushop, bundler)
     elif haspushkey:
         return _pushb2phasespushkey(pushop, bundler)
 
 
 def _pushb2phaseheads(pushop, bundler):
     """push phase information through a bundle2 - binary part"""
     pushop.stepsdone.add(b'phases')
     if pushop.outdatedphases:
         updates = {p: [] for p in phases.allphases}
         updates[0].extend(h.node() for h in pushop.outdatedphases)
         phasedata = phases.binaryencode(updates)
         bundler.newpart(b'phase-heads', data=phasedata)
 
 
 def _pushb2phasespushkey(pushop, bundler):
     """push phase information through a bundle2 - pushkey part"""
     pushop.stepsdone.add(b'phases')
     part2node = []
 
     def handlefailure(pushop, exc):
         targetid = int(exc.partid)
         for partid, node in part2node:
             if partid == targetid:
                 raise error.Abort(_(b'updating %s to public failed') % node)
 
     enc = pushkey.encode
     for newremotehead in pushop.outdatedphases:
         part = bundler.newpart(b'pushkey')
         part.addparam(b'namespace', enc(b'phases'))
         part.addparam(b'key', enc(newremotehead.hex()))
         part.addparam(b'old', enc(b'%d' % phases.draft))
         part.addparam(b'new', enc(b'%d' % phases.public))
         part2node.append((part.id, newremotehead))
         pushop.pkfailcb[part.id] = handlefailure
 
     def handlereply(op):
         for partid, node in part2node:
             partrep = op.records.getreplies(partid)
             results = partrep[b'pushkey']
             assert len(results) <= 1
             msg = None
             if not results:
                 msg = _(b'server ignored update of %s to public!\n') % node
             elif not int(results[0][b'return']):
                 msg = _(b'updating %s to public failed!\n') % node
             if msg is not None:
                 pushop.ui.warn(msg)
 
     return handlereply
 
 
 @b2partsgenerator(b'obsmarkers')
 def _pushb2obsmarkers(pushop, bundler):
     if b'obsmarkers' in pushop.stepsdone:
         return
     remoteversions = bundle2.obsmarkersversion(bundler.capabilities)
     if obsolete.commonversion(remoteversions) is None:
         return
     pushop.stepsdone.add(b'obsmarkers')
     if pushop.outobsmarkers:
         markers = obsutil.sortedmarkers(pushop.outobsmarkers)
         bundle2.buildobsmarkerspart(bundler, markers)
 
 
 @b2partsgenerator(b'bookmarks')
 def _pushb2bookmarks(pushop, bundler):
     """handle bookmark push through bundle2"""
     if b'bookmarks' in pushop.stepsdone:
         return
     b2caps = bundle2.bundle2caps(pushop.remote)
 
     legacy = pushop.repo.ui.configlist(b'devel', b'legacy.exchange')
     legacybooks = b'bookmarks' in legacy
 
     if not legacybooks and b'bookmarks' in b2caps:
         return _pushb2bookmarkspart(pushop, bundler)
     elif b'pushkey' in b2caps:
         return _pushb2bookmarkspushkey(pushop, bundler)
 
 
 def _bmaction(old, new):
     """small utility for bookmark pushing"""
     if not old:
         return b'export'
     elif not new:
         return b'delete'
     return b'update'
 
 
 def _abortonsecretctx(pushop, node, b):
     """abort if a given bookmark points to a secret changeset"""
     if node and pushop.repo[node].phase() == phases.secret:
         raise error.Abort(
             _(b'cannot push bookmark %s as it points to a secret changeset') % b
         )
 
 
 def _pushb2bookmarkspart(pushop, bundler):
     pushop.stepsdone.add(b'bookmarks')
     if not pushop.outbookmarks:
         return
 
     allactions = []
     data = []
     for book, old, new in pushop.outbookmarks:
         _abortonsecretctx(pushop, new, book)
         data.append((book, new))
         allactions.append((book, _bmaction(old, new)))
     checkdata = bookmod.binaryencode(pushop.repo, data)
     bundler.newpart(b'bookmarks', data=checkdata)
 
     def handlereply(op):
         ui = pushop.ui
         # if success
         for book, action in allactions:
             ui.status(bookmsgmap[action][0] % book)
 
     return handlereply
 
 
 def _pushb2bookmarkspushkey(pushop, bundler):
     pushop.stepsdone.add(b'bookmarks')
     part2book = []
     enc = pushkey.encode
 
     def handlefailure(pushop, exc):
         targetid = int(exc.partid)
         for partid, book, action in part2book:
             if partid == targetid:
                 raise error.Abort(bookmsgmap[action][1].rstrip() % book)
         # we should not be called for part we did not generated
         assert False
 
     for book, old, new in pushop.outbookmarks:
         _abortonsecretctx(pushop, new, book)
         part = bundler.newpart(b'pushkey')
         part.addparam(b'namespace', enc(b'bookmarks'))
         part.addparam(b'key', enc(book))
         part.addparam(b'old', enc(hex(old)))
         part.addparam(b'new', enc(hex(new)))
         action = b'update'
         if not old:
             action = b'export'
         elif not new:
             action = b'delete'
         part2book.append((part.id, book, action))
         pushop.pkfailcb[part.id] = handlefailure
 
     def handlereply(op):
         ui = pushop.ui
         for partid, book, action in part2book:
             partrep = op.records.getreplies(partid)
             results = partrep[b'pushkey']
             assert len(results) <= 1
             if not results:
                 pushop.ui.warn(_(b'server ignored bookmark %s update\n') % book)
             else:
                 ret = int(results[0][b'return'])
                 if ret:
                     ui.status(bookmsgmap[action][0] % book)
                 else:
                     ui.warn(bookmsgmap[action][1] % book)
                     if pushop.bkresult is not None:
                         pushop.bkresult = 1
 
     return handlereply
 
 
 @b2partsgenerator(b'pushvars', idx=0)
 def _getbundlesendvars(pushop, bundler):
     '''send shellvars via bundle2'''
     pushvars = pushop.pushvars
     if pushvars:
         shellvars = {}
         for raw in pushvars:
             if b'=' not in raw:
                 msg = (
                     b"unable to parse variable '%s', should follow "
                     b"'KEY=VALUE' or 'KEY=' format"
                 )
                 raise error.Abort(msg % raw)
             k, v = raw.split(b'=', 1)
             shellvars[k] = v
 
         part = bundler.newpart(b'pushvars')
 
         for key, value in shellvars.items():
             part.addparam(key, value, mandatory=False)
 
 
 def _pushbundle2(pushop):
     """push data to the remote using bundle2
 
     The only currently supported type of data is changegroup but this will
     evolve in the future."""
     bundler = bundle2.bundle20(pushop.ui, bundle2.bundle2caps(pushop.remote))
     pushback = pushop.trmanager and pushop.ui.configbool(
         b'experimental', b'bundle2.pushback'
     )
 
     # create reply capability
     capsblob = bundle2.encodecaps(
         bundle2.getrepocaps(pushop.repo, allowpushback=pushback, role=b'client')
     )
     bundler.newpart(b'replycaps', data=capsblob)
     replyhandlers = []
     for partgenname in b2partsgenorder:
         partgen = b2partsgenmapping[partgenname]
         ret = partgen(pushop, bundler)
         if callable(ret):
             replyhandlers.append(ret)
     # do not push if nothing to push
     if bundler.nbparts <= 1:
         return
     stream = util.chunkbuffer(bundler.getchunks())
     try:
         try:
             with pushop.remote.commandexecutor() as e:
                 reply = e.callcommand(
                     b'unbundle',
                     {
                         b'bundle': stream,
                         b'heads': [b'force'],
                         b'url': pushop.remote.url(),
                     },
                 ).result()
         except error.BundleValueError as exc:
             raise error.RemoteError(_(b'missing support for %s') % exc)
         try:
             trgetter = None
             if pushback:
                 trgetter = pushop.trmanager.transaction
             op = bundle2.processbundle(
                 pushop.repo,
                 reply,
                 trgetter,
                 remote=pushop.remote,
             )
         except error.BundleValueError as exc:
             raise error.RemoteError(_(b'missing support for %s') % exc)
         except bundle2.AbortFromPart as exc:
             pushop.ui.error(_(b'remote: %s\n') % exc)
             if exc.hint is not None:
                 pushop.ui.error(_(b'remote: %s\n') % (b'(%s)' % exc.hint))
             raise error.RemoteError(_(b'push failed on remote'))
     except error.PushkeyFailed as exc:
         partid = int(exc.partid)
         if partid not in pushop.pkfailcb:
             raise
         pushop.pkfailcb[partid](pushop, exc)
     for rephand in replyhandlers:
         rephand(op)
 
 
 def _pushchangeset(pushop):
     """Make the actual push of changeset bundle to remote repo"""
     if b'changesets' in pushop.stepsdone:
         return
     pushop.stepsdone.add(b'changesets')
     if not _pushcheckoutgoing(pushop):
         return
 
     # Should have verified this in push().
     assert pushop.remote.capable(b'unbundle')
 
     pushop.repo.prepushoutgoinghooks(pushop)
     outgoing = pushop.outgoing
     # TODO: get bundlecaps from remote
     bundlecaps = None
     # create a changegroup from local
     if pushop.revs is None and not (
         outgoing.excluded or pushop.repo.changelog.filteredrevs
     ):
         # push everything,
         # use the fast path, no race possible on push
         cg = changegroup.makechangegroup(
             pushop.repo,
             outgoing,
             b'01',
             b'push',
             fastpath=True,
             bundlecaps=bundlecaps,
         )
     else:
         cg = changegroup.makechangegroup(
             pushop.repo, outgoing, b'01', b'push', bundlecaps=bundlecaps
         )
 
     # apply changegroup to remote
     # local repo finds heads on server, finds out what
     # revs it must push. once revs transferred, if server
     # finds it has different heads (someone else won
     # commit/push race), server aborts.
     if pushop.force:
         remoteheads = [b'force']
     else:
         remoteheads = pushop.remoteheads
     # ssh: return remote's addchangegroup()
     # http: return remote's addchangegroup() or 0 for error
     pushop.cgresult = pushop.remote.unbundle(cg, remoteheads, pushop.repo.url())
 
 
 def _pushsyncphase(pushop):
     """synchronise phase information locally and remotely"""
     cheads = pushop.commonheads
     # even when we don't push, exchanging phase data is useful
     remotephases = listkeys(pushop.remote, b'phases')
     if (
         pushop.ui.configbool(b'ui', b'_usedassubrepo')
         and remotephases  # server supports phases
         and pushop.cgresult is None  # nothing was pushed
         and remotephases.get(b'publishing', False)
     ):
         # When:
         # - this is a subrepo push
         # - and remote support phase
         # - and no changeset was pushed
         # - and remote is publishing
         # We may be in issue 3871 case!
         # We drop the possible phase synchronisation done by
         # courtesy to publish changesets possibly locally draft
         # on the remote.
         remotephases = {b'publishing': b'True'}
     if not remotephases:  # old server or public only reply from non-publishing
         _localphasemove(pushop, cheads)
         # don't push any phase data as there is nothing to push
     else:
         ana = phases.analyzeremotephases(pushop.repo, cheads, remotephases)
         pheads, droots = ana
         ### Apply remote phase on local
         if remotephases.get(b'publishing', False):
             _localphasemove(pushop, cheads)
         else:  # publish = False
             _localphasemove(pushop, pheads)
             _localphasemove(pushop, cheads, phases.draft)
         ### Apply local phase on remote
 
         if pushop.cgresult:
             if b'phases' in pushop.stepsdone:
                 # phases already pushed though bundle2
                 return
             outdated = pushop.outdatedphases
         else:
             outdated = pushop.fallbackoutdatedphases
 
         pushop.stepsdone.add(b'phases')
 
         # filter heads already turned public by the push
         outdated = [c for c in outdated if c.node() not in pheads]
         # fallback to independent pushkey command
         for newremotehead in outdated:
             with pushop.remote.commandexecutor() as e:
                 r = e.callcommand(
                     b'pushkey',
                     {
                         b'namespace': b'phases',
                         b'key': newremotehead.hex(),
                         b'old': b'%d' % phases.draft,
                         b'new': b'%d' % phases.public,
                     },
                 ).result()
 
             if not r:
                 pushop.ui.warn(
                     _(b'updating %s to public failed!\n') % newremotehead
                 )
 
 
 def _localphasemove(pushop, nodes, phase=phases.public):
     """move <nodes> to <phase> in the local source repo"""
     if pushop.trmanager:
         phases.advanceboundary(
             pushop.repo, pushop.trmanager.transaction(), phase, nodes
         )
     else:
         # repo is not locked, do not change any phases!
         # Informs the user that phases should have been moved when
         # applicable.
         actualmoves = [n for n in nodes if phase < pushop.repo[n].phase()]
         phasestr = phases.phasenames[phase]
         if actualmoves:
             pushop.ui.status(
                 _(
                     b'cannot lock source repo, skipping '
                     b'local %s phase update\n'
                 )
                 % phasestr
             )
 
 
 def _pushobsolete(pushop):
     """utility function to push obsolete markers to a remote"""
     if b'obsmarkers' in pushop.stepsdone:
         return
     repo = pushop.repo
     remote = pushop.remote
     pushop.stepsdone.add(b'obsmarkers')
     if pushop.outobsmarkers:
         pushop.ui.debug(b'try to push obsolete markers to remote\n')
         rslts = []
         markers = obsutil.sortedmarkers(pushop.outobsmarkers)
         remotedata = obsolete._pushkeyescape(markers)
         for key in sorted(remotedata, reverse=True):
             # reverse sort to ensure we end with dump0
             data = remotedata[key]
             rslts.append(remote.pushkey(b'obsolete', key, b'', data))
         if [r for r in rslts if not r]:
             msg = _(b'failed to push some obsolete markers!\n')
             repo.ui.warn(msg)
 
 
 def _pushbookmark(pushop):
     """Update bookmark position on remote"""
     if pushop.cgresult == 0 or b'bookmarks' in pushop.stepsdone:
         return
     pushop.stepsdone.add(b'bookmarks')
     ui = pushop.ui
     remote = pushop.remote
 
     for b, old, new in pushop.outbookmarks:
         action = b'update'
         if not old:
             action = b'export'
         elif not new:
             action = b'delete'
 
         with remote.commandexecutor() as e:
             r = e.callcommand(
                 b'pushkey',
                 {
                     b'namespace': b'bookmarks',
                     b'key': b,
                     b'old': hex(old),
                     b'new': hex(new),
                 },
             ).result()
 
         if r:
             ui.status(bookmsgmap[action][0] % b)
         else:
             ui.warn(bookmsgmap[action][1] % b)
             # discovery can have set the value form invalid entry
             if pushop.bkresult is not None:
                 pushop.bkresult = 1
 
 
 class pulloperation:
     """A object that represent a single pull operation
 
     It purpose is to carry pull related state and very common operation.
 
     A new should be created at the beginning of each pull and discarded
     afterward.
     """
 
     def __init__(
         self,
         repo,
         remote,
         heads=None,
         force=False,
         bookmarks=(),
         remotebookmarks=None,
         streamclonerequested=None,
         includepats=None,
         excludepats=None,
         depth=None,
         path=None,
     ):
         # repo we pull into
         self.repo = repo
         # repo we pull from
         self.remote = remote
         # path object used to build this remote
         #
         # Ideally, the remote peer would carry that directly.
         self.remote_path = path
         # revision we try to pull (None is "all")
         self.heads = heads
         # bookmark pulled explicitly
         self.explicitbookmarks = [
             repo._bookmarks.expandname(bookmark) for bookmark in bookmarks
         ]
         # do we force pull?
         self.force = force
         # whether a streaming clone was requested
         self.streamclonerequested = streamclonerequested
         # transaction manager
         self.trmanager = None
         # set of common changeset between local and remote before pull
         self.common = None
         # set of pulled head
         self.rheads = None
         # list of missing changeset to fetch remotely
         self.fetch = None
         # remote bookmarks data
         self.remotebookmarks = remotebookmarks
         # result of changegroup pulling (used as return code by pull)
         self.cgresult = None
         # list of step already done
         self.stepsdone = set()
         # Whether we attempted a clone from pre-generated bundles.
         self.clonebundleattempted = False
         # Set of file patterns to include.
         self.includepats = includepats
         # Set of file patterns to exclude.
         self.excludepats = excludepats
         # Number of ancestor changesets to pull from each pulled head.
         self.depth = depth
 
     @util.propertycache
     def pulledsubset(self):
         """heads of the set of changeset target by the pull"""
         # compute target subset
         if self.heads is None:
             # We pulled every thing possible
             # sync on everything common
             c = set(self.common)
             ret = list(self.common)
             for n in self.rheads:
                 if n not in c:
                     ret.append(n)
             return ret
         else:
             # We pulled a specific subset
             # sync on this subset
             return self.heads
 
     @util.propertycache
     def canusebundle2(self):
         return not _forcebundle1(self)
 
     @util.propertycache
     def remotebundle2caps(self):
         return bundle2.bundle2caps(self.remote)
 
     def gettransaction(self):
         # deprecated; talk to trmanager directly
         return self.trmanager.transaction()
 
 
 class transactionmanager(util.transactional):
     """An object to manage the life cycle of a transaction
 
     It creates the transaction on demand and calls the appropriate hooks when
     closing the transaction."""
 
     def __init__(self, repo, source, url):
         self.repo = repo
         self.source = source
         self.url = url
         self._tr = None
 
     def transaction(self):
         """Return an open transaction object, constructing if necessary"""
         if not self._tr:
             trname = b'%s\n%s' % (self.source, urlutil.hidepassword(self.url))
             self._tr = self.repo.transaction(trname)
             self._tr.hookargs[b'source'] = self.source
             self._tr.hookargs[b'url'] = self.url
         return self._tr
 
     def close(self):
         """close transaction if created"""
         if self._tr is not None:
             self._tr.close()
 
     def release(self):
         """release transaction if created"""
         if self._tr is not None:
             self._tr.release()
 
 
 def listkeys(remote, namespace):
     with remote.commandexecutor() as e:
         return e.callcommand(b'listkeys', {b'namespace': namespace}).result()
 
 
 def _fullpullbundle2(repo, pullop):
     # The server may send a partial reply, i.e. when inlining
     # pre-computed bundles. In that case, update the common
     # set based on the results and pull another bundle.
     #
     # There are two indicators that the process is finished:
     # - no changeset has been added, or
     # - all remote heads are known locally.
     # The head check must use the unfiltered view as obsoletion
     # markers can hide heads.
     unfi = repo.unfiltered()
     unficl = unfi.changelog
 
     def headsofdiff(h1, h2):
         """Returns heads(h1 % h2)"""
         res = unfi.set(b'heads(%ln %% %ln)', h1, h2)
         return {ctx.node() for ctx in res}
 
     def headsofunion(h1, h2):
         """Returns heads((h1 + h2) - null)"""
         res = unfi.set(b'heads((%ln + %ln - null))', h1, h2)
         return {ctx.node() for ctx in res}
 
     while True:
         old_heads = unficl.heads()
         clstart = len(unficl)
         _pullbundle2(pullop)
         if requirements.NARROW_REQUIREMENT in repo.requirements:
             # XXX narrow clones filter the heads on the server side during
             # XXX getbundle and result in partial replies as well.
             # XXX Disable pull bundles in this case as band aid to avoid
             # XXX extra round trips.
             break
         if clstart == len(unficl):
             break
         if all(unficl.hasnode(n) for n in pullop.rheads):
             break
         new_heads = headsofdiff(unficl.heads(), old_heads)
         pullop.common = headsofunion(new_heads, pullop.common)
         pullop.rheads = set(pullop.rheads) - pullop.common
 
 
 def add_confirm_callback(repo, pullop):
     """adds a finalize callback to transaction which can be used to show stats
     to user and confirm the pull before committing transaction"""
 
     tr = pullop.trmanager.transaction()
     scmutil.registersummarycallback(
         repo, tr, txnname=b'pull', as_validator=True
     )
     reporef = weakref.ref(repo.unfiltered())
 
     def prompt(tr):
         repo = reporef()
         cm = _(b'accept incoming changes (yn)?$$ &Yes $$ &No')
         if repo.ui.promptchoice(cm):
             raise error.Abort(b"user aborted")
 
     tr.addvalidator(b'900-pull-prompt', prompt)
 
 
 def pull(
     repo,
     remote,
     path=None,
     heads=None,
     force=False,
     bookmarks=(),
     opargs=None,
     streamclonerequested=None,
     includepats=None,
     excludepats=None,
     depth=None,
     confirm=None,
 ):
     """Fetch repository data from a remote.
 
     This is the main function used to retrieve data from a remote repository.
 
     ``repo`` is the local repository to clone into.
     ``remote`` is a peer instance.
     ``heads`` is an iterable of revisions we want to pull. ``None`` (the
     default) means to pull everything from the remote.
     ``bookmarks`` is an iterable of bookmarks requesting to be pulled. By
     default, all remote bookmarks are pulled.
     ``opargs`` are additional keyword arguments to pass to ``pulloperation``
     initialization.
     ``streamclonerequested`` is a boolean indicating whether a "streaming
     clone" is requested. A "streaming clone" is essentially a raw file copy
     of revlogs from the server. This only works when the local repository is
     empty. The default value of ``None`` means to respect the server
     configuration for preferring stream clones.
     ``includepats`` and ``excludepats`` define explicit file patterns to
     include and exclude in storage, respectively. If not defined, narrow
     patterns from the repo instance are used, if available.
     ``depth`` is an integer indicating the DAG depth of history we're
     interested in. If defined, for each revision specified in ``heads``, we
     will fetch up to this many of its ancestors and data associated with them.
     ``confirm`` is a boolean indicating whether the pull should be confirmed
     before committing the transaction. This overrides HGPLAIN.
 
     Returns the ``pulloperation`` created for this pull.
     """
     if opargs is None:
         opargs = {}
 
     # We allow the narrow patterns to be passed in explicitly to provide more
     # flexibility for API consumers.
     if includepats or excludepats:
         includepats = includepats or set()
         excludepats = excludepats or set()
     else:
         includepats, excludepats = repo.narrowpats
 
     narrowspec.validatepatterns(includepats)
     narrowspec.validatepatterns(excludepats)
 
     pullop = pulloperation(
         repo,
         remote,
         path=path,
         heads=heads,
         force=force,
         bookmarks=bookmarks,
         streamclonerequested=streamclonerequested,
         includepats=includepats,
         excludepats=excludepats,
         depth=depth,
         **pycompat.strkwargs(opargs)
     )
 
     peerlocal = pullop.remote.local()
     if peerlocal:
         missing = set(peerlocal.requirements) - pullop.repo.supported
         if missing:
             msg = _(
                 b"required features are not"
                 b" supported in the destination:"
                 b" %s"
             ) % (b', '.join(sorted(missing)))
             raise error.Abort(msg)
 
     for category in repo._wanted_sidedata:
         # Check that a computer is registered for that category for at least
         # one revlog kind.
         for kind, computers in repo._sidedata_computers.items():
             if computers.get(category):
                 break
         else:
             # This should never happen since repos are supposed to be able to
             # generate the sidedata they require.
             raise error.ProgrammingError(
                 _(
                     b'sidedata category requested by local side without local'
                     b"support: '%s'"
                 )
                 % pycompat.bytestr(category)
             )
 
     pullop.trmanager = transactionmanager(repo, b'pull', remote.url())
     wlock = util.nullcontextmanager()
     if not bookmod.bookmarksinstore(repo):
         wlock = repo.wlock()
     with wlock, repo.lock(), pullop.trmanager:
         if confirm or (
             repo.ui.configbool(b"pull", b"confirm") and not repo.ui.plain()
         ):
             add_confirm_callback(repo, pullop)
 
         # This should ideally be in _pullbundle2(). However, it needs to run
         # before discovery to avoid extra work.
         _maybeapplyclonebundle(pullop)
         streamclone.maybeperformlegacystreamclone(pullop)
         _pulldiscovery(pullop)
         if pullop.canusebundle2:
             _fullpullbundle2(repo, pullop)
         _pullchangeset(pullop)
         _pullphase(pullop)
         _pullbookmarks(pullop)
         _pullobsolete(pullop)
 
     # storing remotenames
     if repo.ui.configbool(b'experimental', b'remotenames'):
         logexchange.pullremotenames(repo, remote)
 
     return pullop
 
 
 # list of steps to perform discovery before pull
 pulldiscoveryorder = []
 
 # Mapping between step name and function
 #
 # This exists to help extensions wrap steps if necessary
 pulldiscoverymapping = {}
 
 
 def pulldiscovery(stepname):
     """decorator for function performing discovery before pull
 
     The function is added to the step -> function mapping and appended to the
     list of steps.  Beware that decorated function will be added in order (this
     may matter).
 
     You can only use this decorator for a new step, if you want to wrap a step
     from an extension, change the pulldiscovery dictionary directly."""
 
     def dec(func):
         assert stepname not in pulldiscoverymapping
         pulldiscoverymapping[stepname] = func
         pulldiscoveryorder.append(stepname)
         return func
 
     return dec
 
 
 def _pulldiscovery(pullop):
     """Run all discovery steps"""
     for stepname in pulldiscoveryorder:
         step = pulldiscoverymapping[stepname]
         step(pullop)
 
 
 @pulldiscovery(b'b1:bookmarks')
 def _pullbookmarkbundle1(pullop):
     """fetch bookmark data in bundle1 case
 
     If not using bundle2, we have to fetch bookmarks before changeset
     discovery to reduce the chance and impact of race conditions."""
     if pullop.remotebookmarks is not None:
         return
     if pullop.canusebundle2 and b'listkeys' in pullop.remotebundle2caps:
         # all known bundle2 servers now support listkeys, but lets be nice with
         # new implementation.
         return
     books = listkeys(pullop.remote, b'bookmarks')
     pullop.remotebookmarks = bookmod.unhexlifybookmarks(books)
 
 
 @pulldiscovery(b'changegroup')
 def _pulldiscoverychangegroup(pullop):
     """discovery phase for the pull
 
     Current handle changeset discovery only, will change handle all discovery
     at some point."""
     tmp = discovery.findcommonincoming(
         pullop.repo, pullop.remote, heads=pullop.heads, force=pullop.force
     )
     common, fetch, rheads = tmp
     has_node = pullop.repo.unfiltered().changelog.index.has_node
     if fetch and rheads:
         # If a remote heads is filtered locally, put in back in common.
         #
         # This is a hackish solution to catch most of "common but locally
         # hidden situation".  We do not performs discovery on unfiltered
         # repository because it end up doing a pathological amount of round
         # trip for w huge amount of changeset we do not care about.
         #
         # If a set of such "common but filtered" changeset exist on the server
         # but are not including a remote heads, we'll not be able to detect it,
         scommon = set(common)
         for n in rheads:
             if has_node(n):
                 if n not in scommon:
                     common.append(n)
         if set(rheads).issubset(set(common)):
             fetch = []
     pullop.common = common
     pullop.fetch = fetch
     pullop.rheads = rheads
 
 
 def _pullbundle2(pullop):
     """pull data using bundle2
 
     For now, the only supported data are changegroup."""
     kwargs = {b'bundlecaps': caps20to10(pullop.repo, role=b'client')}
 
     # make ui easier to access
     ui = pullop.repo.ui
 
     # At the moment we don't do stream clones over bundle2. If that is
     # implemented then here's where the check for that will go.
     streaming = streamclone.canperformstreamclone(pullop, bundle2=True)[0]
 
     # declare pull perimeters
     kwargs[b'common'] = pullop.common
     kwargs[b'heads'] = pullop.heads or pullop.rheads
 
     # check server supports narrow and then adding includepats and excludepats
     servernarrow = pullop.remote.capable(wireprototypes.NARROWCAP)
     if servernarrow and pullop.includepats:
         kwargs[b'includepats'] = pullop.includepats
     if servernarrow and pullop.excludepats:
         kwargs[b'excludepats'] = pullop.excludepats
 
     if streaming:
         kwargs[b'cg'] = False
         kwargs[b'stream'] = True
         pullop.stepsdone.add(b'changegroup')
         pullop.stepsdone.add(b'phases')
 
     else:
         # pulling changegroup
         pullop.stepsdone.add(b'changegroup')
 
         kwargs[b'cg'] = pullop.fetch
 
         legacyphase = b'phases' in ui.configlist(b'devel', b'legacy.exchange')
         hasbinaryphase = b'heads' in pullop.remotebundle2caps.get(b'phases', ())
         if not legacyphase and hasbinaryphase:
             kwargs[b'phases'] = True
             pullop.stepsdone.add(b'phases')
 
         if b'listkeys' in pullop.remotebundle2caps:
             if b'phases' not in pullop.stepsdone:
                 kwargs[b'listkeys'] = [b'phases']
 
     bookmarksrequested = False
     legacybookmark = b'bookmarks' in ui.configlist(b'devel', b'legacy.exchange')
     hasbinarybook = b'bookmarks' in pullop.remotebundle2caps
 
     if pullop.remotebookmarks is not None:
         pullop.stepsdone.add(b'request-bookmarks')
 
     if (
         b'request-bookmarks' not in pullop.stepsdone
         and pullop.remotebookmarks is None
         and not legacybookmark
         and hasbinarybook
     ):
         kwargs[b'bookmarks'] = True
         bookmarksrequested = True
 
     if b'listkeys' in pullop.remotebundle2caps:
         if b'request-bookmarks' not in pullop.stepsdone:
             # make sure to always includes bookmark data when migrating
             # `hg incoming --bundle` to using this function.
             pullop.stepsdone.add(b'request-bookmarks')
             kwargs.setdefault(b'listkeys', []).append(b'bookmarks')
 
     # If this is a full pull / clone and the server supports the clone bundles
     # feature, tell the server whether we attempted a clone bundle. The
     # presence of this flag indicates the client supports clone bundles. This
     # will enable the server to treat clients that support clone bundles
     # differently from those that don't.
     if (
         pullop.remote.capable(b'clonebundles')
         and pullop.heads is None
         and list(pullop.common) == [pullop.repo.nullid]
     ):
         kwargs[b'cbattempted'] = pullop.clonebundleattempted
 
     if streaming:
         pullop.repo.ui.status(_(b'streaming all changes\n'))
     elif not pullop.fetch:
         pullop.repo.ui.status(_(b"no changes found\n"))
         pullop.cgresult = 0
     else:
         if pullop.heads is None and list(pullop.common) == [pullop.repo.nullid]:
             pullop.repo.ui.status(_(b"requesting all changes\n"))
     if obsolete.isenabled(pullop.repo, obsolete.exchangeopt):
         remoteversions = bundle2.obsmarkersversion(pullop.remotebundle2caps)
         if obsolete.commonversion(remoteversions) is not None:
             kwargs[b'obsmarkers'] = True
             pullop.stepsdone.add(b'obsmarkers')
     _pullbundle2extraprepare(pullop, kwargs)
 
     remote_sidedata = bundle2.read_remote_wanted_sidedata(pullop.remote)
     if remote_sidedata:
         kwargs[b'remote_sidedata'] = remote_sidedata
 
     with pullop.remote.commandexecutor() as e:
         args = dict(kwargs)
         args[b'source'] = b'pull'
         bundle = e.callcommand(b'getbundle', args).result()
 
         try:
             op = bundle2.bundleoperation(
                 pullop.repo,
                 pullop.gettransaction,
                 source=b'pull',
                 remote=pullop.remote,
             )
             op.modes[b'bookmarks'] = b'records'
             bundle2.processbundle(
                 pullop.repo,
                 bundle,
                 op=op,
                 remote=pullop.remote,
             )
         except bundle2.AbortFromPart as exc:
             pullop.repo.ui.error(_(b'remote: abort: %s\n') % exc)
             raise error.RemoteError(_(b'pull failed on remote'), hint=exc.hint)
         except error.BundleValueError as exc:
             raise error.RemoteError(_(b'missing support for %s') % exc)
 
     if pullop.fetch:
         pullop.cgresult = bundle2.combinechangegroupresults(op)
 
     # processing phases change
     for namespace, value in op.records[b'listkeys']:
         if namespace == b'phases':
             _pullapplyphases(pullop, value)
 
     # processing bookmark update
     if bookmarksrequested:
         books = {}
         for record in op.records[b'bookmarks']:
             books[record[b'bookmark']] = record[b"node"]
         pullop.remotebookmarks = books
     else:
         for namespace, value in op.records[b'listkeys']:
             if namespace == b'bookmarks':
                 pullop.remotebookmarks = bookmod.unhexlifybookmarks(value)
 
     # bookmark data were either already there or pulled in the bundle
     if pullop.remotebookmarks is not None:
         _pullbookmarks(pullop)
 
 
 def _pullbundle2extraprepare(pullop, kwargs):
     """hook function so that extensions can extend the getbundle call"""
 
 
 def _pullchangeset(pullop):
     """pull changeset from unbundle into the local repo"""
     # We delay the open of the transaction as late as possible so we
     # don't open transaction for nothing or you break future useful
     # rollback call
     if b'changegroup' in pullop.stepsdone:
         return
     pullop.stepsdone.add(b'changegroup')
     if not pullop.fetch:
         pullop.repo.ui.status(_(b"no changes found\n"))
         pullop.cgresult = 0
         return
     tr = pullop.gettransaction()
     if pullop.heads is None and list(pullop.common) == [pullop.repo.nullid]:
         pullop.repo.ui.status(_(b"requesting all changes\n"))
     elif pullop.heads is None and pullop.remote.capable(b'changegroupsubset'):
         # issue1320, avoid a race if remote changed after discovery
         pullop.heads = pullop.rheads
 
     if pullop.remote.capable(b'getbundle'):
         # TODO: get bundlecaps from remote
         cg = pullop.remote.getbundle(
             b'pull', common=pullop.common, heads=pullop.heads or pullop.rheads
         )
     elif pullop.heads is None:
         with pullop.remote.commandexecutor() as e:
             cg = e.callcommand(
                 b'changegroup',
                 {
                     b'nodes': pullop.fetch,
                     b'source': b'pull',
                 },
             ).result()
 
     elif not pullop.remote.capable(b'changegroupsubset'):
         raise error.Abort(
             _(
                 b"partial pull cannot be done because "
                 b"other repository doesn't support "
                 b"changegroupsubset."
             )
         )
     else:
         with pullop.remote.commandexecutor() as e:
             cg = e.callcommand(
                 b'changegroupsubset',
                 {
                     b'bases': pullop.fetch,
                     b'heads': pullop.heads,
                     b'source': b'pull',
                 },
             ).result()
 
     bundleop = bundle2.applybundle(
         pullop.repo,
         cg,
         tr,
         b'pull',
         pullop.remote.url(),
         remote=pullop.remote,
     )
     pullop.cgresult = bundle2.combinechangegroupresults(bundleop)
 
 
 def _pullphase(pullop):
     # Get remote phases data from remote
     if b'phases' in pullop.stepsdone:
         return
     remotephases = listkeys(pullop.remote, b'phases')
     _pullapplyphases(pullop, remotephases)
 
 
 def _pullapplyphases(pullop, remotephases):
     """apply phase movement from observed remote state"""
     if b'phases' in pullop.stepsdone:
         return
     pullop.stepsdone.add(b'phases')
     publishing = bool(remotephases.get(b'publishing', False))
     if remotephases and not publishing:
         # remote is new and non-publishing
         pheads, _dr = phases.analyzeremotephases(
             pullop.repo, pullop.pulledsubset, remotephases
         )
         dheads = pullop.pulledsubset
     else:
         # Remote is old or publishing all common changesets
         # should be seen as public
         pheads = pullop.pulledsubset
         dheads = []
     unfi = pullop.repo.unfiltered()
     phase = unfi._phasecache.phase
     rev = unfi.changelog.index.get_rev
     public = phases.public
     draft = phases.draft
 
     # exclude changesets already public locally and update the others
     pheads = [pn for pn in pheads if phase(unfi, rev(pn)) > public]
     if pheads:
         tr = pullop.gettransaction()
         phases.advanceboundary(pullop.repo, tr, public, pheads)
 
     # exclude changesets already draft locally and update the others
     dheads = [pn for pn in dheads if phase(unfi, rev(pn)) > draft]
     if dheads:
         tr = pullop.gettransaction()
         phases.advanceboundary(pullop.repo, tr, draft, dheads)
 
 
 def _pullbookmarks(pullop):
     """process the remote bookmark information to update the local one"""
     if b'bookmarks' in pullop.stepsdone:
         return
     pullop.stepsdone.add(b'bookmarks')
     repo = pullop.repo
     remotebookmarks = pullop.remotebookmarks
     bookmarks_mode = None
     if pullop.remote_path is not None:
         bookmarks_mode = pullop.remote_path.bookmarks_mode
     bookmod.updatefromremote(
         repo.ui,
         repo,
         remotebookmarks,
         pullop.remote.url(),
         pullop.gettransaction,
         explicit=pullop.explicitbookmarks,
         mode=bookmarks_mode,
     )
 
 
 def _pullobsolete(pullop):
     """utility function to pull obsolete markers from a remote
 
     The `gettransaction` is function that return the pull transaction, creating
     one if necessary. We return the transaction to inform the calling code that
     a new transaction have been created (when applicable).
 
     Exists mostly to allow overriding for experimentation purpose"""
     if b'obsmarkers' in pullop.stepsdone:
         return
     pullop.stepsdone.add(b'obsmarkers')
     tr = None
     if obsolete.isenabled(pullop.repo, obsolete.exchangeopt):
         pullop.repo.ui.debug(b'fetching remote obsolete markers\n')
         remoteobs = listkeys(pullop.remote, b'obsolete')
         if b'dump0' in remoteobs:
             tr = pullop.gettransaction()
             markers = []
             for key in sorted(remoteobs, reverse=True):
                 if key.startswith(b'dump'):
                     data = util.b85decode(remoteobs[key])
                     version, newmarks = obsolete._readmarkers(data)
                     markers += newmarks
             if markers:
                 pullop.repo.obsstore.add(tr, markers)
             pullop.repo.invalidatevolatilesets()
     return tr
 
 
 def applynarrowacl(repo, kwargs):
     """Apply narrow fetch access control.
 
     This massages the named arguments for getbundle wire protocol commands
     so requested data is filtered through access control rules.
     """
     ui = repo.ui
     # TODO this assumes existence of HTTP and is a layering violation.
     username = ui.shortuser(ui.environ.get(b'REMOTE_USER') or ui.username())
     user_includes = ui.configlist(
         _NARROWACL_SECTION,
         username + b'.includes',
         ui.configlist(_NARROWACL_SECTION, b'default.includes'),
     )
     user_excludes = ui.configlist(
         _NARROWACL_SECTION,
         username + b'.excludes',
         ui.configlist(_NARROWACL_SECTION, b'default.excludes'),
     )
     if not user_includes:
         raise error.Abort(
             _(b"%s configuration for user %s is empty")
             % (_NARROWACL_SECTION, username)
         )
 
     user_includes = [
         b'path:.' if p == b'*' else b'path:' + p for p in user_includes
     ]
     user_excludes = [
         b'path:.' if p == b'*' else b'path:' + p for p in user_excludes
     ]
 
     req_includes = set(kwargs.get('includepats', []))
     req_excludes = set(kwargs.get('excludepats', []))
 
     req_includes, req_excludes, invalid_includes = narrowspec.restrictpatterns(
         req_includes, req_excludes, user_includes, user_excludes
     )
 
     if invalid_includes:
         raise error.Abort(
             _(b"The following includes are not accessible for %s: %s")
             % (username, stringutil.pprint(invalid_includes))
         )
 
     new_args = {}
     new_args.update(kwargs)
     new_args['narrow'] = True
     new_args['narrow_acl'] = True
     new_args['includepats'] = req_includes
     if req_excludes:
         new_args['excludepats'] = req_excludes
 
     return new_args
 
 
 def _computeellipsis(repo, common, heads, known, match, depth=None):
     """Compute the shape of a narrowed DAG.
 
     Args:
       repo: The repository we're transferring.
       common: The roots of the DAG range we're transferring.
               May be just [nullid], which means all ancestors of heads.
       heads: The heads of the DAG range we're transferring.
       match: The narrowmatcher that allows us to identify relevant changes.
       depth: If not None, only consider nodes to be full nodes if they are at
              most depth changesets away from one of heads.
 
     Returns:
       A tuple of (visitnodes, relevant_nodes, ellipsisroots) where:
 
         visitnodes: The list of nodes (either full or ellipsis) which
                     need to be sent to the client.
         relevant_nodes: The set of changelog nodes which change a file inside
                  the narrowspec. The client needs these as non-ellipsis nodes.
         ellipsisroots: A dict of {rev: parents} that is used in
                        narrowchangegroup to produce ellipsis nodes with the
                        correct parents.
     """
     cl = repo.changelog
     mfl = repo.manifestlog
 
     clrev = cl.rev
 
     commonrevs = {clrev(n) for n in common} | {nullrev}
     headsrevs = {clrev(n) for n in heads}
 
     if depth:
         revdepth = {h: 0 for h in headsrevs}
 
     ellipsisheads = collections.defaultdict(set)
     ellipsisroots = collections.defaultdict(set)
 
     def addroot(head, curchange):
         """Add a root to an ellipsis head, splitting heads with 3 roots."""
         ellipsisroots[head].add(curchange)
         # Recursively split ellipsis heads with 3 roots by finding the
         # roots' youngest common descendant which is an elided merge commit.
         # That descendant takes 2 of the 3 roots as its own, and becomes a
         # root of the head.
         while len(ellipsisroots[head]) > 2:
             child, roots = splithead(head)
             splitroots(head, child, roots)
             head = child  # Recurse in case we just added a 3rd root
 
     def splitroots(head, child, roots):
         ellipsisroots[head].difference_update(roots)
         ellipsisroots[head].add(child)
         ellipsisroots[child].update(roots)
         ellipsisroots[child].discard(child)
 
     def splithead(head):
         r1, r2, r3 = sorted(ellipsisroots[head])
         for nr1, nr2 in ((r2, r3), (r1, r3), (r1, r2)):
             mid = repo.revs(
                 b'sort(merge() & %d::%d & %d::%d, -rev)', nr1, head, nr2, head
             )
             for j in mid:
                 if j == nr2:
                     return nr2, (nr1, nr2)
                 if j not in ellipsisroots or len(ellipsisroots[j]) < 2:
                     return j, (nr1, nr2)
         raise error.Abort(
             _(
                 b'Failed to split up ellipsis node! head: %d, '
                 b'roots: %d %d %d'
             )
             % (head, r1, r2, r3)
         )
 
     missing = list(cl.findmissingrevs(common=commonrevs, heads=headsrevs))
     visit = reversed(missing)
     relevant_nodes = set()
     visitnodes = [cl.node(m) for m in missing]
     required = set(headsrevs) | known
     for rev in visit:
         clrev = cl.changelogrevision(rev)
         ps = [prev for prev in cl.parentrevs(rev) if prev != nullrev]
         if depth is not None:
             curdepth = revdepth[rev]
             for p in ps:
                 revdepth[p] = min(curdepth + 1, revdepth.get(p, depth + 1))
         needed = False
         shallow_enough = depth is None or revdepth[rev] <= depth
         if shallow_enough:
             curmf = mfl[clrev.manifest].read()
             if ps:
                 # We choose to not trust the changed files list in
                 # changesets because it's not always correct. TODO: could
                 # we trust it for the non-merge case?
                 p1mf = mfl[cl.changelogrevision(ps[0]).manifest].read()
                 needed = bool(curmf.diff(p1mf, match))
                 if not needed and len(ps) > 1:
                     # For merge changes, the list of changed files is not
                     # helpful, since we need to emit the merge if a file
                     # in the narrow spec has changed on either side of the
                     # merge. As a result, we do a manifest diff to check.
                     p2mf = mfl[cl.changelogrevision(ps[1]).manifest].read()
                     needed = bool(curmf.diff(p2mf, match))
             else:
                 # For a root node, we need to include the node if any
                 # files in the node match the narrowspec.
                 needed = any(curmf.walk(match))
 
         if needed:
             for head in ellipsisheads[rev]:
                 addroot(head, rev)
             for p in ps:
                 required.add(p)
             relevant_nodes.add(cl.node(rev))
         else:
             if not ps:
                 ps = [nullrev]
             if rev in required:
                 for head in ellipsisheads[rev]:
                     addroot(head, rev)
                 for p in ps:
                     ellipsisheads[p].add(rev)
             else:
                 for p in ps:
                     ellipsisheads[p] |= ellipsisheads[rev]
 
     # add common changesets as roots of their reachable ellipsis heads
     for c in commonrevs:
         for head in ellipsisheads[c]:
             addroot(head, c)
     return visitnodes, relevant_nodes, ellipsisroots
 
 
 def caps20to10(repo, role):
     """return a set with appropriate options to use bundle20 during getbundle"""
     caps = {b'HG20'}
     capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo, role=role))
     caps.add(b'bundle2=' + urlreq.quote(capsblob))
     return caps
 
 
 # List of names of steps to perform for a bundle2 for getbundle, order matters.
 getbundle2partsorder = []
 
 # Mapping between step name and function
 #
 # This exists to help extensions wrap steps if necessary
 getbundle2partsmapping = {}
 
 
 def getbundle2partsgenerator(stepname, idx=None):
     """decorator for function generating bundle2 part for getbundle
 
     The function is added to the step -> function mapping and appended to the
     list of steps.  Beware that decorated functions will be added in order
     (this may matter).
 
     You can only use this decorator for new steps, if you want to wrap a step
     from an extension, attack the getbundle2partsmapping dictionary directly."""
 
     def dec(func):
         assert stepname not in getbundle2partsmapping
         getbundle2partsmapping[stepname] = func
         if idx is None:
             getbundle2partsorder.append(stepname)
         else:
             getbundle2partsorder.insert(idx, stepname)
         return func
 
     return dec
 
 
 def bundle2requested(bundlecaps):
     if bundlecaps is not None:
         return any(cap.startswith(b'HG2') for cap in bundlecaps)
     return False
 
 
 def getbundlechunks(
     repo,
     source,
     heads=None,
     common=None,
     bundlecaps=None,
     remote_sidedata=None,
     **kwargs
 ):
     """Return chunks constituting a bundle's raw data.
 
     Could be a bundle HG10 or a bundle HG20 depending on bundlecaps
     passed.
 
     Returns a 2-tuple of a dict with metadata about the generated bundle
     and an iterator over raw chunks (of varying sizes).
     """
     kwargs = pycompat.byteskwargs(kwargs)
     info = {}
     usebundle2 = bundle2requested(bundlecaps)
     # bundle10 case
     if not usebundle2:
         if bundlecaps and not kwargs.get(b'cg', True):
             raise ValueError(
                 _(b'request for bundle10 must include changegroup')
             )
 
         if kwargs:
             raise ValueError(
                 _(b'unsupported getbundle arguments: %s')
                 % b', '.join(sorted(kwargs.keys()))
             )
         outgoing = _computeoutgoing(repo, heads, common)
         info[b'bundleversion'] = 1
         return (
             info,
             changegroup.makestream(
                 repo,
                 outgoing,
                 b'01',
                 source,
                 bundlecaps=bundlecaps,
                 remote_sidedata=remote_sidedata,
             ),
         )
 
     # bundle20 case
     info[b'bundleversion'] = 2
     b2caps = {}
     for bcaps in bundlecaps:
         if bcaps.startswith(b'bundle2='):
             blob = urlreq.unquote(bcaps[len(b'bundle2=') :])
             b2caps.update(bundle2.decodecaps(blob))
     bundler = bundle2.bundle20(repo.ui, b2caps)
 
     kwargs[b'heads'] = heads
     kwargs[b'common'] = common
 
     for name in getbundle2partsorder:
         func = getbundle2partsmapping[name]
         func(
             bundler,
             repo,
             source,
             bundlecaps=bundlecaps,
             b2caps=b2caps,
             remote_sidedata=remote_sidedata,
             **pycompat.strkwargs(kwargs)
         )
 
     info[b'prefercompressed'] = bundler.prefercompressed
 
     return info, bundler.getchunks()
 
 
-@getbundle2partsgenerator(b'stream2')
+@getbundle2partsgenerator(b'stream')
 def _getbundlestream2(bundler, repo, *args, **kwargs):
     return bundle2.addpartbundlestream2(bundler, repo, **kwargs)
 
 
 @getbundle2partsgenerator(b'changegroup')
 def _getbundlechangegrouppart(
     bundler,
     repo,
     source,
     bundlecaps=None,
     b2caps=None,
     heads=None,
     common=None,
     remote_sidedata=None,
     **kwargs
 ):
     """add a changegroup part to the requested bundle"""
     if not kwargs.get('cg', True) or not b2caps:
         return
 
     version = b'01'
     cgversions = b2caps.get(b'changegroup')
     if cgversions:  # 3.1 and 3.2 ship with an empty value
         cgversions = [
             v
             for v in cgversions
             if v in changegroup.supportedoutgoingversions(repo)
         ]
         if not cgversions:
             raise error.Abort(_(b'no common changegroup version'))
         version = max(cgversions)
 
     outgoing = _computeoutgoing(repo, heads, common)
     if not outgoing.missing:
         return
 
     if kwargs.get('narrow', False):
         include = sorted(filter(bool, kwargs.get('includepats', [])))
         exclude = sorted(filter(bool, kwargs.get('excludepats', [])))
         matcher = narrowspec.match(repo.root, include=include, exclude=exclude)
     else:
         matcher = None
 
     cgstream = changegroup.makestream(
         repo,
         outgoing,
         version,
         source,
         bundlecaps=bundlecaps,
         matcher=matcher,
         remote_sidedata=remote_sidedata,
     )
 
     part = bundler.newpart(b'changegroup', data=cgstream)
     if cgversions:
         part.addparam(b'version', version)
 
     part.addparam(b'nbchanges', b'%d' % len(outgoing.missing), mandatory=False)
 
     if scmutil.istreemanifest(repo):
         part.addparam(b'treemanifest', b'1')
 
     if repository.REPO_FEATURE_SIDE_DATA in repo.features:
         part.addparam(b'exp-sidedata', b'1')
         sidedata = bundle2.format_remote_wanted_sidedata(repo)
         part.addparam(b'exp-wanted-sidedata', sidedata)
 
     if (
         kwargs.get('narrow', False)
         and kwargs.get('narrow_acl', False)
         and (include or exclude)
     ):
         # this is mandatory because otherwise ACL clients won't work
         narrowspecpart = bundler.newpart(b'Narrow:responsespec')
         narrowspecpart.data = b'%s\0%s' % (
             b'\n'.join(include),
             b'\n'.join(exclude),
         )
 
 
 @getbundle2partsgenerator(b'bookmarks')
 def _getbundlebookmarkpart(
     bundler, repo, source, bundlecaps=None, b2caps=None, **kwargs
 ):
     """add a bookmark part to the requested bundle"""
     if not kwargs.get('bookmarks', False):
         return
     if not b2caps or b'bookmarks' not in b2caps:
         raise error.Abort(_(b'no common bookmarks exchange method'))
     books = bookmod.listbinbookmarks(repo)
     data = bookmod.binaryencode(repo, books)
     if data:
         bundler.newpart(b'bookmarks', data=data)
 
 
 @getbundle2partsgenerator(b'listkeys')
 def _getbundlelistkeysparts(
     bundler, repo, source, bundlecaps=None, b2caps=None, **kwargs
 ):
     """add parts containing listkeys namespaces to the requested bundle"""
     listkeys = kwargs.get('listkeys', ())
     for namespace in listkeys:
         part = bundler.newpart(b'listkeys')
         part.addparam(b'namespace', namespace)
         keys = repo.listkeys(namespace).items()
         part.data = pushkey.encodekeys(keys)
 
 
 @getbundle2partsgenerator(b'obsmarkers')
 def _getbundleobsmarkerpart(
     bundler, repo, source, bundlecaps=None, b2caps=None, heads=None, **kwargs
 ):
     """add an obsolescence markers part to the requested bundle"""
     if kwargs.get('obsmarkers', False):
         if heads is None:
             heads = repo.heads()
         subset = [c.node() for c in repo.set(b'::%ln', heads)]
         markers = repo.obsstore.relevantmarkers(subset)
         markers = obsutil.sortedmarkers(markers)
         bundle2.buildobsmarkerspart(bundler, markers)
 
 
 @getbundle2partsgenerator(b'phases')
 def _getbundlephasespart(
     bundler, repo, source, bundlecaps=None, b2caps=None, heads=None, **kwargs
 ):
     """add phase heads part to the requested bundle"""
     if kwargs.get('phases', False):
         if not b2caps or b'heads' not in b2caps.get(b'phases'):
             raise error.Abort(_(b'no common phases exchange method'))
         if heads is None:
             heads = repo.heads()
 
         headsbyphase = collections.defaultdict(set)
         if repo.publishing():
             headsbyphase[phases.public] = heads
         else:
             # find the appropriate heads to move
 
             phase = repo._phasecache.phase
             node = repo.changelog.node
             rev = repo.changelog.rev
             for h in heads:
                 headsbyphase[phase(repo, rev(h))].add(h)
             seenphases = list(headsbyphase.keys())
 
             # We do not handle anything but public and draft phase for now)
             if seenphases:
                 assert max(seenphases) <= phases.draft
 
             # if client is pulling non-public changesets, we need to find
             # intermediate public heads.
             draftheads = headsbyphase.get(phases.draft, set())
             if draftheads:
                 publicheads = headsbyphase.get(phases.public, set())
 
                 revset = b'heads(only(%ln, %ln) and public())'
                 extraheads = repo.revs(revset, draftheads, publicheads)
                 for r in extraheads:
                     headsbyphase[phases.public].add(node(r))
 
         # transform data in a format used by the encoding function
         phasemapping = {
             phase: sorted(headsbyphase[phase]) for phase in phases.allphases
         }
 
         # generate the actual part
         phasedata = phases.binaryencode(phasemapping)
         bundler.newpart(b'phase-heads', data=phasedata)
 
 
 @getbundle2partsgenerator(b'hgtagsfnodes')
 def _getbundletagsfnodes(
     bundler,
     repo,
     source,
     bundlecaps=None,
     b2caps=None,
     heads=None,
     common=None,
     **kwargs
 ):
     """Transfer the .hgtags filenodes mapping.
 
     Only values for heads in this bundle will be transferred.
 
     The part data consists of pairs of 20 byte changeset node and .hgtags
     filenodes raw values.
     """
     # Don't send unless:
     # - changeset are being exchanged,
     # - the client supports it.
     if not b2caps or not (kwargs.get('cg', True) and b'hgtagsfnodes' in b2caps):
         return
 
     outgoing = _computeoutgoing(repo, heads, common)
     bundle2.addparttagsfnodescache(repo, bundler, outgoing)
 
 
 @getbundle2partsgenerator(b'cache:rev-branch-cache')
 def _getbundlerevbranchcache(
     bundler,
     repo,
     source,
     bundlecaps=None,
     b2caps=None,
     heads=None,
     common=None,
     **kwargs
 ):
     """Transfer the rev-branch-cache mapping
 
     The payload is a series of data related to each branch
 
     1) branch name length
     2) number of open heads
     3) number of closed heads
     4) open heads nodes
     5) closed heads nodes
     """
     # Don't send unless:
     # - changeset are being exchanged,
     # - the client supports it.
     # - narrow bundle isn't in play (not currently compatible).
     if (
         not kwargs.get('cg', True)
         or not b2caps
         or b'rev-branch-cache' not in b2caps
         or kwargs.get('narrow', False)
         or repo.ui.has_section(_NARROWACL_SECTION)
     ):
         return
 
     outgoing = _computeoutgoing(repo, heads, common)
     bundle2.addpartrevbranchcache(repo, bundler, outgoing)
 
 
 def check_heads(repo, their_heads, context):
     """check if the heads of a repo have been modified
 
     Used by peer for unbundling.
     """
     heads = repo.heads()
     heads_hash = hashutil.sha1(b''.join(sorted(heads))).digest()
     if not (
         their_heads == [b'force']
         or their_heads == heads
         or their_heads == [b'hashed', heads_hash]
     ):
         # someone else committed/pushed/unbundled while we
         # were transferring data
         raise error.PushRaced(
             b'repository changed while %s - please try again' % context
         )
 
 
 def unbundle(repo, cg, heads, source, url):
     """Apply a bundle to a repo.
 
     this function makes sure the repo is locked during the application and have
     mechanism to check that no push race occurred between the creation of the
     bundle and its application.
 
     If the push was raced as PushRaced exception is raised."""
     r = 0
     # need a transaction when processing a bundle2 stream
     # [wlock, lock, tr] - needs to be an array so nested functions can modify it
     lockandtr = [None, None, None]
     recordout = None
     # quick fix for output mismatch with bundle2 in 3.4
     captureoutput = repo.ui.configbool(
         b'experimental', b'bundle2-output-capture'
     )
     if url.startswith(b'remote:http:') or url.startswith(b'remote:https:'):
         captureoutput = True
     try:
         # note: outside bundle1, 'heads' is expected to be empty and this
         # 'check_heads' call wil be a no-op
         check_heads(repo, heads, b'uploading changes')
         # push can proceed
         if not isinstance(cg, bundle2.unbundle20):
             # legacy case: bundle1 (changegroup 01)
             txnname = b"\n".join([source, urlutil.hidepassword(url)])
             with repo.lock(), repo.transaction(txnname) as tr:
                 op = bundle2.applybundle(repo, cg, tr, source, url)
                 r = bundle2.combinechangegroupresults(op)
         else:
             r = None
             try:
 
                 def gettransaction():
                     if not lockandtr[2]:
                         if not bookmod.bookmarksinstore(repo):
                             lockandtr[0] = repo.wlock()
                         lockandtr[1] = repo.lock()
                         lockandtr[2] = repo.transaction(source)
                         lockandtr[2].hookargs[b'source'] = source
                         lockandtr[2].hookargs[b'url'] = url
                         lockandtr[2].hookargs[b'bundle2'] = b'1'
                     return lockandtr[2]
 
                 # Do greedy locking by default until we're satisfied with lazy
                 # locking.
                 if not repo.ui.configbool(
                     b'experimental', b'bundle2lazylocking'
                 ):
                     gettransaction()
 
                 op = bundle2.bundleoperation(
                     repo,
                     gettransaction,
                     captureoutput=captureoutput,
                     source=b'push',
                 )
                 try:
                     op = bundle2.processbundle(repo, cg, op=op)
                 finally:
                     r = op.reply
                     if captureoutput and r is not None:
                         repo.ui.pushbuffer(error=True, subproc=True)
 
                         def recordout(output):
                             r.newpart(b'output', data=output, mandatory=False)
 
                 if lockandtr[2] is not None:
                     lockandtr[2].close()
             except BaseException as exc:
                 exc.duringunbundle2 = True
                 if captureoutput and r is not None:
                     parts = exc._bundle2salvagedoutput = r.salvageoutput()
 
                     def recordout(output):
                         part = bundle2.bundlepart(
                             b'output', data=output, mandatory=False
                         )
                         parts.append(part)
 
                 raise
     finally:
         lockmod.release(lockandtr[2], lockandtr[1], lockandtr[0])
         if recordout is not None:
             recordout(repo.ui.popbuffer())
     return r
 
 
 def _maybeapplyclonebundle(pullop):
     """Apply a clone bundle from a remote, if possible."""
 
     repo = pullop.repo
     remote = pullop.remote
 
     if not repo.ui.configbool(b'ui', b'clonebundles'):
         return
 
     # Only run if local repo is empty.
     if len(repo):
         return
 
     if pullop.heads:
         return
 
     if not remote.capable(b'clonebundles'):
         return
 
     with remote.commandexecutor() as e:
         res = e.callcommand(b'clonebundles', {}).result()
 
     # If we call the wire protocol command, that's good enough to record the
     # attempt.
     pullop.clonebundleattempted = True
 
     entries = bundlecaches.parseclonebundlesmanifest(repo, res)
     if not entries:
         repo.ui.note(
             _(
                 b'no clone bundles available on remote; '
                 b'falling back to regular clone\n'
             )
         )
         return
 
     entries = bundlecaches.filterclonebundleentries(
         repo, entries, streamclonerequested=pullop.streamclonerequested
     )
 
     if not entries:
         # There is a thundering herd concern here. However, if a server
         # operator doesn't advertise bundles appropriate for its clients,
         # they deserve what's coming. Furthermore, from a client's
         # perspective, no automatic fallback would mean not being able to
         # clone!
         repo.ui.warn(
             _(
                 b'no compatible clone bundles available on server; '
                 b'falling back to regular clone\n'
             )
         )
         repo.ui.warn(
             _(b'(you may want to report this to the server operator)\n')
         )
         return
 
     entries = bundlecaches.sortclonebundleentries(repo.ui, entries)
 
     url = entries[0][b'URL']
     repo.ui.status(_(b'applying clone bundle from %s\n') % url)
     if trypullbundlefromurl(repo.ui, repo, url):
         repo.ui.status(_(b'finished applying clone bundle\n'))
     # Bundle failed.
     #
     # We abort by default to avoid the thundering herd of
     # clients flooding a server that was expecting expensive
     # clone load to be offloaded.
     elif repo.ui.configbool(b'ui', b'clonebundlefallback'):
         repo.ui.warn(_(b'falling back to normal clone\n'))
     else:
         raise error.Abort(
             _(b'error applying bundle'),
             hint=_(
                 b'if this error persists, consider contacting '
                 b'the server operator or disable clone '
                 b'bundles via '
                 b'"--config ui.clonebundles=false"'
             ),
         )
 
 
 def trypullbundlefromurl(ui, repo, url):
     """Attempt to apply a bundle from a URL."""
     with repo.lock(), repo.transaction(b'bundleurl') as tr:
         try:
             fh = urlmod.open(ui, url)
             cg = readbundle(ui, fh, b'stream')
 
             if isinstance(cg, streamclone.streamcloneapplier):
                 cg.apply(repo)
             else:
                 bundle2.applybundle(repo, cg, tr, b'clonebundles', url)
             return True
         except urlerr.httperror as e:
             ui.warn(
                 _(b'HTTP error fetching bundle: %s\n')
                 % stringutil.forcebytestr(e)
             )
         except urlerr.urlerror as e:
             ui.warn(
                 _(b'error fetching bundle: %s\n')
                 % stringutil.forcebytestr(e.reason)
             )
 
         return False

mercurial/streamclone.py

@@ -1,962 +1,966
 # streamclone.py - producing and consuming streaming repository data
 #
 # Copyright 2015 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 
 import contextlib
 import os
 import struct
 
 from .i18n import _
 from .pycompat import open
 from .interfaces import repository
 from . import (
     bookmarks,
     bundle2 as bundle2mod,
     cacheutil,
     error,
     narrowspec,
     phases,
     pycompat,
     requirements as requirementsmod,
     scmutil,
     store,
     transaction,
     util,
 )
 from .revlogutils import (
     nodemap,
 )
 
 
 def new_stream_clone_requirements(default_requirements, streamed_requirements):
     """determine the final set of requirement for a new stream clone
 
     this method combine the "default" requirements that a new repository would
     use with the constaint we get from the stream clone content. We keep local
     configuration choice when possible.
     """
     requirements = set(default_requirements)
     requirements -= requirementsmod.STREAM_FIXED_REQUIREMENTS
     requirements.update(streamed_requirements)
     return requirements
 
 
 def streamed_requirements(repo):
     """the set of requirement the new clone will have to support
 
     This is used for advertising the stream options and to generate the actual
     stream content."""
     requiredformats = (
         repo.requirements & requirementsmod.STREAM_FIXED_REQUIREMENTS
     )
     return requiredformats
 
 
 def canperformstreamclone(pullop, bundle2=False):
     """Whether it is possible to perform a streaming clone as part of pull.
 
     ``bundle2`` will cause the function to consider stream clone through
     bundle2 and only through bundle2.
 
     Returns a tuple of (supported, requirements). ``supported`` is True if
     streaming clone is supported and False otherwise. ``requirements`` is
     a set of repo requirements from the remote, or ``None`` if stream clone
     isn't supported.
     """
     repo = pullop.repo
     remote = pullop.remote
 
     # should we consider streaming clone at all ?
     streamrequested = pullop.streamclonerequested
     # If we don't have a preference, let the server decide for us. This
     # likely only comes into play in LANs.
     if streamrequested is None:
         # The server can advertise whether to prefer streaming clone.
         streamrequested = remote.capable(b'stream-preferred')
     if not streamrequested:
         return False, None
 
     # Streaming clone only works on an empty destination repository
     if len(repo):
         return False, None
 
     # Streaming clone only works if all data is being requested.
     if pullop.heads:
         return False, None
 
     bundle2supported = False
     if pullop.canusebundle2:
         local_caps = bundle2mod.getrepocaps(repo, role=b'client')
         local_supported = set(local_caps.get(b'stream', []))
         remote_supported = set(pullop.remotebundle2caps.get(b'stream', []))
         bundle2supported = bool(local_supported & remote_supported)
         # else
         # Server doesn't support bundle2 stream clone or doesn't support
         # the versions we support. Fall back and possibly allow legacy.
 
     # Ensures legacy code path uses available bundle2.
     if bundle2supported and not bundle2:
         return False, None
     # Ensures bundle2 doesn't try to do a stream clone if it isn't supported.
     elif bundle2 and not bundle2supported:
         return False, None
 
     # In order for stream clone to work, the client has to support all the
     # requirements advertised by the server.
     #
     # The server advertises its requirements via the "stream" and "streamreqs"
     # capability. "stream" (a value-less capability) is advertised if and only
     # if the only requirement is "revlogv1." Else, the "streamreqs" capability
     # is advertised and contains a comma-delimited list of requirements.
     requirements = set()
     if remote.capable(b'stream'):
         requirements.add(requirementsmod.REVLOGV1_REQUIREMENT)
     else:
         streamreqs = remote.capable(b'streamreqs')
         # This is weird and shouldn't happen with modern servers.
         if not streamreqs:
             pullop.repo.ui.warn(
                 _(
                     b'warning: stream clone requested but server has them '
                     b'disabled\n'
                 )
             )
             return False, None
 
         streamreqs = set(streamreqs.split(b','))
         # Server requires something we don't support. Bail.
         missingreqs = streamreqs - repo.supported
         if missingreqs:
             pullop.repo.ui.warn(
                 _(
                     b'warning: stream clone requested but client is missing '
                     b'requirements: %s\n'
                 )
                 % b', '.join(sorted(missingreqs))
             )
             pullop.repo.ui.warn(
                 _(
                     b'(see https://www.mercurial-scm.org/wiki/MissingRequirement '
                     b'for more information)\n'
                 )
             )
             return False, None
         requirements = streamreqs
 
     return True, requirements
 
 
 def maybeperformlegacystreamclone(pullop):
     """Possibly perform a legacy stream clone operation.
 
     Legacy stream clones are performed as part of pull but before all other
     operations.
 
     A legacy stream clone will not be performed if a bundle2 stream clone is
     supported.
     """
     from . import localrepo
 
     supported, requirements = canperformstreamclone(pullop)
 
     if not supported:
         return
 
     repo = pullop.repo
     remote = pullop.remote
 
     # Save remote branchmap. We will use it later to speed up branchcache
     # creation.
     rbranchmap = None
     if remote.capable(b'branchmap'):
         with remote.commandexecutor() as e:
             rbranchmap = e.callcommand(b'branchmap', {}).result()
 
     repo.ui.status(_(b'streaming all changes\n'))
 
     with remote.commandexecutor() as e:
         fp = e.callcommand(b'stream_out', {}).result()
 
     # TODO strictly speaking, this code should all be inside the context
     # manager because the context manager is supposed to ensure all wire state
     # is flushed when exiting. But the legacy peers don't do this, so it
     # doesn't matter.
     l = fp.readline()
     try:
         resp = int(l)
     except ValueError:
         raise error.ResponseError(
             _(b'unexpected response from remote server:'), l
         )
     if resp == 1:
         raise error.Abort(_(b'operation forbidden by server'))
     elif resp == 2:
         raise error.Abort(_(b'locking the remote repository failed'))
     elif resp != 0:
         raise error.Abort(_(b'the server sent an unknown error code'))
 
     l = fp.readline()
     try:
         filecount, bytecount = map(int, l.split(b' ', 1))
     except (ValueError, TypeError):
         raise error.ResponseError(
             _(b'unexpected response from remote server:'), l
         )
 
     with repo.lock():
         consumev1(repo, fp, filecount, bytecount)
         repo.requirements = new_stream_clone_requirements(
             repo.requirements,
             requirements,
         )
         repo.svfs.options = localrepo.resolvestorevfsoptions(
             repo.ui, repo.requirements, repo.features
         )
         scmutil.writereporequirements(repo)
         nodemap.post_stream_cleanup(repo)
 
         if rbranchmap:
             repo._branchcaches.replace(repo, rbranchmap)
 
         repo.invalidate()
 
 
 def allowservergeneration(repo):
     """Whether streaming clones are allowed from the server."""
     if repository.REPO_FEATURE_STREAM_CLONE not in repo.features:
         return False
 
     if not repo.ui.configbool(b'server', b'uncompressed', untrusted=True):
         return False
 
     # The way stream clone works makes it impossible to hide secret changesets.
     # So don't allow this by default.
     secret = phases.hassecret(repo)
     if secret:
         return repo.ui.configbool(b'server', b'uncompressedallowsecret')
 
     return True
 
 
 # This is it's own function so extensions can override it.
 def _walkstreamfiles(repo, matcher=None, phase=False, obsolescence=False):
     return repo.store.walk(matcher, phase=phase, obsolescence=obsolescence)
 
 
 def generatev1(repo):
     """Emit content for version 1 of a streaming clone.
 
     This returns a 3-tuple of (file count, byte size, data iterator).
 
     The data iterator consists of N entries for each file being transferred.
     Each file entry starts as a line with the file name and integer size
     delimited by a null byte.
 
     The raw file data follows. Following the raw file data is the next file
     entry, or EOF.
 
     When used on the wire protocol, an additional line indicating protocol
     success will be prepended to the stream. This function is not responsible
     for adding it.
 
     This function will obtain a repository lock to ensure a consistent view of
     the store is captured. It therefore may raise LockError.
     """
     entries = []
     total_bytes = 0
     # Get consistent snapshot of repo, lock during scan.
     with repo.lock():
         repo.ui.debug(b'scanning\n')
         for entry in _walkstreamfiles(repo):
             for f in entry.files():
                 file_size = f.file_size(repo.store.vfs)
                 if file_size:
                     entries.append((f.unencoded_path, file_size))
                     total_bytes += file_size
         _test_sync_point_walk_1(repo)
     _test_sync_point_walk_2(repo)
 
     repo.ui.debug(
         b'%d files, %d bytes to transfer\n' % (len(entries), total_bytes)
     )
 
     svfs = repo.svfs
     debugflag = repo.ui.debugflag
 
     def emitrevlogdata():
         for name, size in entries:
             if debugflag:
                 repo.ui.debug(b'sending %s (%d bytes)\n' % (name, size))
             # partially encode name over the wire for backwards compat
             yield b'%s\0%d\n' % (store.encodedir(name), size)
             # auditing at this stage is both pointless (paths are already
             # trusted by the local repo) and expensive
             with svfs(name, b'rb', auditpath=False) as fp:
                 if size <= 65536:
                     yield fp.read(size)
                 else:
                     for chunk in util.filechunkiter(fp, limit=size):
                         yield chunk
 
     return len(entries), total_bytes, emitrevlogdata()
 
 
 def generatev1wireproto(repo):
     """Emit content for version 1 of streaming clone suitable for the wire.
 
     This is the data output from ``generatev1()`` with 2 header lines. The
     first line indicates overall success. The 2nd contains the file count and
     byte size of payload.
 
     The success line contains "0" for success, "1" for stream generation not
     allowed, and "2" for error locking the repository (possibly indicating
     a permissions error for the server process).
     """
     if not allowservergeneration(repo):
         yield b'1\n'
         return
 
     try:
         filecount, bytecount, it = generatev1(repo)
     except error.LockError:
         yield b'2\n'
         return
 
     # Indicates successful response.
     yield b'0\n'
     yield b'%d %d\n' % (filecount, bytecount)
     for chunk in it:
         yield chunk
 
 
 def generatebundlev1(repo, compression=b'UN'):
     """Emit content for version 1 of a stream clone bundle.
 
     The first 4 bytes of the output ("HGS1") denote this as stream clone
     bundle version 1.
 
     The next 2 bytes indicate the compression type. Only "UN" is currently
     supported.
 
     The next 16 bytes are two 64-bit big endian unsigned integers indicating
     file count and byte count, respectively.
 
     The next 2 bytes is a 16-bit big endian unsigned short declaring the length
     of the requirements string, including a trailing \0. The following N bytes
     are the requirements string, which is ASCII containing a comma-delimited
     list of repo requirements that are needed to support the data.
 
     The remaining content is the output of ``generatev1()`` (which may be
     compressed in the future).
 
     Returns a tuple of (requirements, data generator).
     """
     if compression != b'UN':
         raise ValueError(b'we do not support the compression argument yet')
 
     requirements = streamed_requirements(repo)
     requires = b','.join(sorted(requirements))
 
     def gen():
         yield b'HGS1'
         yield compression
 
         filecount, bytecount, it = generatev1(repo)
         repo.ui.status(
             _(b'writing %d bytes for %d files\n') % (bytecount, filecount)
         )
 
         yield struct.pack(b'>QQ', filecount, bytecount)
         yield struct.pack(b'>H', len(requires) + 1)
         yield requires + b'\0'
 
         # This is where we'll add compression in the future.
         assert compression == b'UN'
 
         progress = repo.ui.makeprogress(
             _(b'bundle'), total=bytecount, unit=_(b'bytes')
         )
         progress.update(0)
 
         for chunk in it:
             progress.increment(step=len(chunk))
             yield chunk
 
         progress.complete()
 
     return requirements, gen()
 
 
 def consumev1(repo, fp, filecount, bytecount):
     """Apply the contents from version 1 of a streaming clone file handle.
 
     This takes the output from "stream_out" and applies it to the specified
     repository.
 
     Like "stream_out," the status line added by the wire protocol is not
     handled by this function.
     """
     with repo.lock():
         repo.ui.status(
             _(b'%d files to transfer, %s of data\n')
             % (filecount, util.bytecount(bytecount))
         )
         progress = repo.ui.makeprogress(
             _(b'clone'), total=bytecount, unit=_(b'bytes')
         )
         progress.update(0)
         start = util.timer()
 
         # TODO: get rid of (potential) inconsistency
         #
         # If transaction is started and any @filecache property is
         # changed at this point, it causes inconsistency between
         # in-memory cached property and streamclone-ed file on the
         # disk. Nested transaction prevents transaction scope "clone"
         # below from writing in-memory changes out at the end of it,
         # even though in-memory changes are discarded at the end of it
         # regardless of transaction nesting.
         #
         # But transaction nesting can't be simply prohibited, because
         # nesting occurs also in ordinary case (e.g. enabling
         # clonebundles).
 
         with repo.transaction(b'clone'):
             with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
                 for i in range(filecount):
                     # XXX doesn't support '\n' or '\r' in filenames
                     l = fp.readline()
                     try:
                         name, size = l.split(b'\0', 1)
                         size = int(size)
                     except (ValueError, TypeError):
                         raise error.ResponseError(
                             _(b'unexpected response from remote server:'), l
                         )
                     if repo.ui.debugflag:
                         repo.ui.debug(
                             b'adding %s (%s)\n' % (name, util.bytecount(size))
                         )
                     # for backwards compat, name was partially encoded
                     path = store.decodedir(name)
                     with repo.svfs(path, b'w', backgroundclose=True) as ofp:
                         for chunk in util.filechunkiter(fp, limit=size):
                             progress.increment(step=len(chunk))
                             ofp.write(chunk)
 
             # force @filecache properties to be reloaded from
             # streamclone-ed file at next access
             repo.invalidate(clearfilecache=True)
 
         elapsed = util.timer() - start
         if elapsed <= 0:
             elapsed = 0.001
         progress.complete()
         repo.ui.status(
             _(b'transferred %s in %.1f seconds (%s/sec)\n')
             % (
                 util.bytecount(bytecount),
                 elapsed,
                 util.bytecount(bytecount / elapsed),
             )
         )
 
 
 def readbundle1header(fp):
     compression = fp.read(2)
     if compression != b'UN':
         raise error.Abort(
             _(
                 b'only uncompressed stream clone bundles are '
                 b'supported; got %s'
             )
             % compression
         )
 
     filecount, bytecount = struct.unpack(b'>QQ', fp.read(16))
     requireslen = struct.unpack(b'>H', fp.read(2))[0]
     requires = fp.read(requireslen)
 
     if not requires.endswith(b'\0'):
         raise error.Abort(
             _(
                 b'malformed stream clone bundle: '
                 b'requirements not properly encoded'
             )
         )
 
     requirements = set(requires.rstrip(b'\0').split(b','))
 
     return filecount, bytecount, requirements
 
 
 def applybundlev1(repo, fp):
     """Apply the content from a stream clone bundle version 1.
 
     We assume the 4 byte header has been read and validated and the file handle
     is at the 2 byte compression identifier.
     """
     if len(repo):
         raise error.Abort(
             _(b'cannot apply stream clone bundle on non-empty repo')
         )
 
     filecount, bytecount, requirements = readbundle1header(fp)
     missingreqs = requirements - repo.supported
     if missingreqs:
         raise error.Abort(
             _(b'unable to apply stream clone: unsupported format: %s')
             % b', '.join(sorted(missingreqs))
         )
 
     consumev1(repo, fp, filecount, bytecount)
     nodemap.post_stream_cleanup(repo)
 
 
 class streamcloneapplier:
     """Class to manage applying streaming clone bundles.
 
     We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle
     readers to perform bundle type-specific functionality.
     """
 
     def __init__(self, fh):
         self._fh = fh
 
     def apply(self, repo):
         return applybundlev1(repo, self._fh)
 
 
 # type of file to stream
 _fileappend = 0  # append only file
 _filefull = 1  # full snapshot file
 
 # Source of the file
 _srcstore = b's'  # store (svfs)
 _srccache = b'c'  # cache (cache)
 
 # This is it's own function so extensions can override it.
 def _walkstreamfullstorefiles(repo):
     """list snapshot file from the store"""
     fnames = []
     if not repo.publishing():
         fnames.append(b'phaseroots')
     return fnames
 
 
 def _filterfull(entry, copy, vfsmap):
     """actually copy the snapshot files"""
     src, name, ftype, data = entry
     if ftype != _filefull:
         return entry
     return (src, name, ftype, copy(vfsmap[src].join(name)))
 
 
 @contextlib.contextmanager
 def maketempcopies():
     """return a function to temporary copy file"""
 
     files = []
     dst_dir = pycompat.mkdtemp(prefix=b'hg-clone-')
     try:
 
         def copy(src):
             fd, dst = pycompat.mkstemp(
                 prefix=os.path.basename(src), dir=dst_dir
             )
             os.close(fd)
             files.append(dst)
             util.copyfiles(src, dst, hardlink=True)
             return dst
 
         yield copy
     finally:
         for tmp in files:
             util.tryunlink(tmp)
         util.tryrmdir(dst_dir)
 
 
 def _makemap(repo):
     """make a (src -> vfs) map for the repo"""
     vfsmap = {
         _srcstore: repo.svfs,
         _srccache: repo.cachevfs,
     }
     # we keep repo.vfs out of the on purpose, ther are too many danger there
     # (eg: .hg/hgrc)
     assert repo.vfs not in vfsmap.values()
 
     return vfsmap
 
 
 def _emit2(repo, entries, totalfilesize):
     """actually emit the stream bundle"""
     vfsmap = _makemap(repo)
     # we keep repo.vfs out of the on purpose, ther are too many danger there
     # (eg: .hg/hgrc),
     #
     # this assert is duplicated (from _makemap) as author might think this is
     # fine, while this is really not fine.
     if repo.vfs in vfsmap.values():
         raise error.ProgrammingError(
             b'repo.vfs must not be added to vfsmap for security reasons'
         )
 
     progress = repo.ui.makeprogress(
         _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
     )
     progress.update(0)
     with maketempcopies() as copy, progress:
         # copy is delayed until we are in the try
         entries = [_filterfull(e, copy, vfsmap) for e in entries]
         yield None  # this release the lock on the repository
         totalbytecount = 0
 
         for src, name, ftype, data in entries:
             vfs = vfsmap[src]
             yield src
             yield util.uvarintencode(len(name))
             if ftype == _fileappend:
                 fp = vfs(name)
                 size = data
             elif ftype == _filefull:
                 fp = open(data, b'rb')
                 size = util.fstat(fp).st_size
             bytecount = 0
             try:
                 yield util.uvarintencode(size)
                 yield name
                 if size <= 65536:
                     chunks = (fp.read(size),)
                 else:
                     chunks = util.filechunkiter(fp, limit=size)
                 for chunk in chunks:
                     bytecount += len(chunk)
                     totalbytecount += len(chunk)
                     progress.update(totalbytecount)
                     yield chunk
                 if bytecount != size:
                     # Would most likely be caused by a race due to `hg strip` or
                     # a revlog split
                     raise error.Abort(
                         _(
                             b'clone could only read %d bytes from %s, but '
                             b'expected %d bytes'
                         )
                         % (bytecount, name, size)
                     )
             finally:
                 fp.close()
 
 
 def _test_sync_point_walk_1(repo):
     """a function for synchronisation during tests"""
 
 
 def _test_sync_point_walk_2(repo):
     """a function for synchronisation during tests"""
 
 
 def _entries_walk(repo, includes, excludes, includeobsmarkers):
     """emit a seris of files information useful to clone a repo
 
     return (vfs-key, entry) iterator
 
     Where `entry` is StoreEntry. (used even for cache entries)
     """
     assert repo._currentlock(repo._lockref) is not None
 
     matcher = None
     if includes or excludes:
         matcher = narrowspec.match(repo.root, includes, excludes)
 
     phase = not repo.publishing()
     entries = _walkstreamfiles(
         repo,
         matcher,
         phase=phase,
         obsolescence=includeobsmarkers,
     )
     for entry in entries:
         yield (_srcstore, entry)
 
     for name in cacheutil.cachetocopy(repo):
         if repo.cachevfs.exists(name):
             # not really a StoreEntry, but close enough
             entry = store.SimpleStoreEntry(
                 entry_path=name,
                 is_volatile=True,
             )
             yield (_srccache, entry)
 
 
 def _v2_walk(repo, includes, excludes, includeobsmarkers):
     """emit a seris of files information useful to clone a repo
 
     return (entries, totalfilesize)
 
     entries is a list of tuple (vfs-key, file-path, file-type, size)
 
     - `vfs-key`: is a key to the right vfs to write the file (see _makemap)
     - `name`: file path of the file to copy (to be feed to the vfss)
     - `file-type`: do this file need to be copied with the source lock ?
     - `size`: the size of the file (or None)
     """
     assert repo._currentlock(repo._lockref) is not None
     files = []
     totalfilesize = 0
 
     vfsmap = _makemap(repo)
     entries = _entries_walk(repo, includes, excludes, includeobsmarkers)
     for vfs_key, entry in entries:
         vfs = vfsmap[vfs_key]
         for f in entry.files():
             file_size = f.file_size(vfs)
             if file_size:
                 ft = _fileappend
                 if f.is_volatile:
                     ft = _filefull
                 files.append((vfs_key, f.unencoded_path, ft, file_size))
                 totalfilesize += file_size
     return files, totalfilesize
 
 
 def generatev2(repo, includes, excludes, includeobsmarkers):
     """Emit content for version 2 of a streaming clone.
 
     the data stream consists the following entries:
     1) A char representing the file destination (eg: store or cache)
     2) A varint containing the length of the filename
     3) A varint containing the length of file data
     4) N bytes containing the filename (the internal, store-agnostic form)
     5) N bytes containing the file data
 
     Returns a 3-tuple of (file count, file size, data iterator).
     """
 
     with repo.lock():
 
         repo.ui.debug(b'scanning\n')
 
         entries, totalfilesize = _v2_walk(
             repo,
             includes=includes,
             excludes=excludes,
             includeobsmarkers=includeobsmarkers,
         )
 
         chunks = _emit2(repo, entries, totalfilesize)
         first = next(chunks)
         assert first is None
         _test_sync_point_walk_1(repo)
     _test_sync_point_walk_2(repo)
 
     return len(entries), totalfilesize, chunks
 
 
+def generatev3(repo, includes, excludes, includeobsmarkers):
+    return generatev2(repo, includes, excludes, includeobsmarkers)
+
+
 @contextlib.contextmanager
 def nested(*ctxs):
     this = ctxs[0]
     rest = ctxs[1:]
     with this:
         if rest:
             with nested(*rest):
                 yield
         else:
             yield
 
 
 def consumev2(repo, fp, filecount, filesize):
     """Apply the contents from a version 2 streaming clone.
 
     Data is read from an object that only needs to provide a ``read(size)``
     method.
     """
     with repo.lock():
         repo.ui.status(
             _(b'%d files to transfer, %s of data\n')
             % (filecount, util.bytecount(filesize))
         )
 
         start = util.timer()
         progress = repo.ui.makeprogress(
             _(b'clone'), total=filesize, unit=_(b'bytes')
         )
         progress.update(0)
 
         vfsmap = _makemap(repo)
         # we keep repo.vfs out of the on purpose, ther are too many danger
         # there (eg: .hg/hgrc),
         #
         # this assert is duplicated (from _makemap) as author might think this
         # is fine, while this is really not fine.
         if repo.vfs in vfsmap.values():
             raise error.ProgrammingError(
                 b'repo.vfs must not be added to vfsmap for security reasons'
             )
 
         with repo.transaction(b'clone'):
             ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())
             with nested(*ctxs):
                 for i in range(filecount):
                     src = util.readexactly(fp, 1)
                     vfs = vfsmap[src]
                     namelen = util.uvarintdecodestream(fp)
                     datalen = util.uvarintdecodestream(fp)
 
                     name = util.readexactly(fp, namelen)
 
                     if repo.ui.debugflag:
                         repo.ui.debug(
                             b'adding [%s] %s (%s)\n'
                             % (src, name, util.bytecount(datalen))
                         )
 
                     with vfs(name, b'w') as ofp:
                         for chunk in util.filechunkiter(fp, limit=datalen):
                             progress.increment(step=len(chunk))
                             ofp.write(chunk)
 
             # force @filecache properties to be reloaded from
             # streamclone-ed file at next access
             repo.invalidate(clearfilecache=True)
 
         elapsed = util.timer() - start
         if elapsed <= 0:
             elapsed = 0.001
         repo.ui.status(
             _(b'transferred %s in %.1f seconds (%s/sec)\n')
             % (
                 util.bytecount(progress.pos),
                 elapsed,
                 util.bytecount(progress.pos / elapsed),
             )
         )
         progress.complete()
 
 
 def applybundlev2(repo, fp, filecount, filesize, requirements):
     from . import localrepo
 
     missingreqs = [r for r in requirements if r not in repo.supported]
     if missingreqs:
         raise error.Abort(
             _(b'unable to apply stream clone: unsupported format: %s')
             % b', '.join(sorted(missingreqs))
         )
 
     consumev2(repo, fp, filecount, filesize)
 
     repo.requirements = new_stream_clone_requirements(
         repo.requirements,
         requirements,
     )
     repo.svfs.options = localrepo.resolvestorevfsoptions(
         repo.ui, repo.requirements, repo.features
     )
     scmutil.writereporequirements(repo)
     nodemap.post_stream_cleanup(repo)
 
 
 def _copy_files(src_vfs_map, dst_vfs_map, entries, progress):
     hardlink = [True]
 
     def copy_used():
         hardlink[0] = False
         progress.topic = _(b'copying')
 
     for k, path, size in entries:
         src_vfs = src_vfs_map[k]
         dst_vfs = dst_vfs_map[k]
         src_path = src_vfs.join(path)
         dst_path = dst_vfs.join(path)
         # We cannot use dirname and makedirs of dst_vfs here because the store
         # encoding confuses them. See issue 6581 for details.
         dirname = os.path.dirname(dst_path)
         if not os.path.exists(dirname):
             util.makedirs(dirname)
         dst_vfs.register_file(path)
         # XXX we could use the #nb_bytes argument.
         util.copyfile(
             src_path,
             dst_path,
             hardlink=hardlink[0],
             no_hardlink_cb=copy_used,
             check_fs_hardlink=False,
         )
         progress.increment()
     return hardlink[0]
 
 
 def local_copy(src_repo, dest_repo):
     """copy all content from one local repository to another
 
     This is useful for local clone"""
     src_store_requirements = {
         r
         for r in src_repo.requirements
         if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
     }
     dest_store_requirements = {
         r
         for r in dest_repo.requirements
         if r not in requirementsmod.WORKING_DIR_REQUIREMENTS
     }
     assert src_store_requirements == dest_store_requirements
 
     with dest_repo.lock():
         with src_repo.lock():
 
             # bookmark is not integrated to the streaming as it might use the
             # `repo.vfs` and they are too many sentitive data accessible
             # through `repo.vfs` to expose it to streaming clone.
             src_book_vfs = bookmarks.bookmarksvfs(src_repo)
             srcbookmarks = src_book_vfs.join(b'bookmarks')
             bm_count = 0
             if os.path.exists(srcbookmarks):
                 bm_count = 1
 
             entries, totalfilesize = _v2_walk(
                 src_repo,
                 includes=None,
                 excludes=None,
                 includeobsmarkers=True,
             )
             src_vfs_map = _makemap(src_repo)
             dest_vfs_map = _makemap(dest_repo)
             progress = src_repo.ui.makeprogress(
                 topic=_(b'linking'),
                 total=len(entries) + bm_count,
                 unit=_(b'files'),
             )
             # copy  files
             #
             # We could copy the full file while the source repository is locked
             # and the other one without the lock. However, in the linking case,
             # this would also requires checks that nobody is appending any data
             # to the files while we do the clone, so this is not done yet. We
             # could do this blindly when copying files.
             files = ((k, path, size) for k, path, ftype, size in entries)
             hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
 
             # copy bookmarks over
             if bm_count:
                 dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
                 dstbookmarks = dst_book_vfs.join(b'bookmarks')
                 util.copyfile(srcbookmarks, dstbookmarks)
         progress.complete()
         if hardlink:
             msg = b'linked %d files\n'
         else:
             msg = b'copied %d files\n'
         src_repo.ui.debug(msg % (len(entries) + bm_count))
 
         with dest_repo.transaction(b"localclone") as tr:
             dest_repo.store.write(tr)
 
         # clean up transaction file as they do not make sense
         transaction.cleanup_undo_files(dest_repo.ui.warn, dest_repo.vfs_map)

tests/test-clone-stream.t

@@ -1,802 +1,1024
 #require serve no-reposimplestore no-chg
 
-#testcases stream-legacy stream-bundle2
+#testcases stream-legacy stream-bundle2-v2 stream-bundle2-v3
 
 #if stream-legacy
   $ cat << EOF >> $HGRCPATH
   > [server]
   > bundle2.stream = no
   > EOF
 #endif
+#if stream-bundle2-v3
+  $ cat << EOF >> $HGRCPATH
+  > [experimental]
+  > stream-v3 = yes
+  > EOF
+#endif
 
 Initialize repository
 
   $ hg init server
   $ cd server
   $ sh $TESTDIR/testlib/stream_clone_setup.sh
   adding 00changelog-ab349180a0405010.nd
   adding 00changelog.d
   adding 00changelog.i
   adding 00changelog.n
   adding 00manifest.d
   adding 00manifest.i
   adding container/isam-build-centos7/bazel-coverage-generator-sandboxfs-compatibility-0758e3e4f6057904d44399bd666faba9e7f40686.patch
   adding data/foo.d
   adding data/foo.i
   adding data/foo.n
   adding data/undo.babar
   adding data/undo.d
   adding data/undo.foo.d
   adding data/undo.foo.i
   adding data/undo.foo.n
   adding data/undo.i
   adding data/undo.n
   adding data/undo.py
   adding foo.d
   adding foo.i
   adding foo.n
   adding meta/foo.d
   adding meta/foo.i
   adding meta/foo.n
   adding meta/undo.babar
   adding meta/undo.d
   adding meta/undo.foo.d
   adding meta/undo.foo.i
   adding meta/undo.foo.n
   adding meta/undo.i
   adding meta/undo.n
   adding meta/undo.py
   adding savanah/foo.d
   adding savanah/foo.i
   adding savanah/foo.n
   adding savanah/undo.babar
   adding savanah/undo.d
   adding savanah/undo.foo.d
   adding savanah/undo.foo.i
   adding savanah/undo.foo.n
   adding savanah/undo.i
   adding savanah/undo.n
   adding savanah/undo.py
   adding store/C\xc3\xa9lesteVille_is_a_Capital_City (esc)
   adding store/foo.d
   adding store/foo.i
   adding store/foo.n
   adding store/undo.babar
   adding store/undo.d
   adding store/undo.foo.d
   adding store/undo.foo.i
   adding store/undo.foo.n
   adding store/undo.i
   adding store/undo.n
   adding store/undo.py
   adding undo.babar
   adding undo.d
   adding undo.foo.d
   adding undo.foo.i
   adding undo.foo.n
   adding undo.i
   adding undo.n
   adding undo.py
 
   $ hg --config server.uncompressed=false serve -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
 Check local clone
 ==================
 
 The logic is close enough of uncompressed.
 This is present here to reuse the testing around file with "special" names.
 
   $ hg clone server local-clone
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 Check that the clone went well
 
   $ hg verify -R local-clone -q
 
 Check uncompressed
 ==================
 
 Cannot stream clone when server.uncompressed is set
 
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=stream_out'
   200 Script output follows
   
   1
 
 #if stream-legacy
   $ hg debugcapabilities http://localhost:$HGPORT
   Main capabilities:
     batch
     branchmap
     $USUAL_BUNDLE2_CAPS_SERVER$
     changegroupsubset
     compression=$BUNDLE2_COMPRESSIONS$
     getbundle
     httpheader=1024
     httpmediatype=0.1rx,0.1tx,0.2tx
     known
     lookup
     pushkey
     unbundle=HG10GZ,HG10BZ,HG10UN
     unbundlehash
   Bundle2 capabilities:
     HG20
     bookmarks
     changegroup
       01
       02
       03
     checkheads
       related
     digests
       md5
       sha1
       sha512
     error
       abort
       unsupportedcontent
       pushraced
       pushkey
     hgtagsfnodes
     listkeys
     phases
       heads
     pushkey
     remote-changegroup
       http
       https
 
   $ hg clone --stream -U http://localhost:$HGPORT server-disabled
   warning: stream clone requested but server has them disabled
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 3 changesets with 1088 changes to 1088 files
   new changesets 96ee1d7354c4:5223b5e3265f
 
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto 0.2 --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
   200 Script output follows
   content-type: application/mercurial-0.2
   
 
   $ f --size body --hexdump --bytes 100
   body: size=140
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 73 0b 45 52 52 4f 52 3a 41 42 4f 52 54 00 00 00 |s.ERROR:ABORT...|
   0020: 00 01 01 07 3c 04 16 6d 65 73 73 61 67 65 73 74 |....<..messagest|
   0030: 72 65 61 6d 20 64 61 74 61 20 72 65 71 75 65 73 |ream data reques|
   0040: 74 65 64 20 62 75 74 20 73 65 72 76 65 72 20 64 |ted but server d|
   0050: 6f 65 73 20 6e 6f 74 20 61 6c 6c 6f 77 20 74 68 |oes not allow th|
   0060: 69 73 20 66                                     |is f|
 
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg debugcapabilities http://localhost:$HGPORT
+  Main capabilities:
+    batch
+    branchmap
+    $USUAL_BUNDLE2_CAPS_SERVER$
+    changegroupsubset
+    compression=$BUNDLE2_COMPRESSIONS$
+    getbundle
+    httpheader=1024
+    httpmediatype=0.1rx,0.1tx,0.2tx
+    known
+    lookup
+    pushkey
+    unbundle=HG10GZ,HG10BZ,HG10UN
+    unbundlehash
+  Bundle2 capabilities:
+    HG20
+    bookmarks
+    changegroup
+      01
+      02
+      03
+    checkheads
+      related
+    digests
+      md5
+      sha1
+      sha512
+    error
+      abort
+      unsupportedcontent
+      pushraced
+      pushkey
+    hgtagsfnodes
+    listkeys
+    phases
+      heads
+    pushkey
+    remote-changegroup
+      http
+      https
+
+  $ hg clone --stream -U http://localhost:$HGPORT server-disabled
+  warning: stream clone requested but server has them disabled
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 3 changesets with 1088 changes to 1088 files
+  new changesets 96ee1d7354c4:5223b5e3265f
+
+  $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto 0.2 --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
+  200 Script output follows
+  content-type: application/mercurial-0.2
+  
+
+  $ f --size body --hexdump --bytes 100
+  body: size=140
+  0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
+  0010: 73 0b 45 52 52 4f 52 3a 41 42 4f 52 54 00 00 00 |s.ERROR:ABORT...|
+  0020: 00 01 01 07 3c 04 16 6d 65 73 73 61 67 65 73 74 |....<..messagest|
+  0030: 72 65 61 6d 20 64 61 74 61 20 72 65 71 75 65 73 |ream data reques|
+  0040: 74 65 64 20 62 75 74 20 73 65 72 76 65 72 20 64 |ted but server d|
+  0050: 6f 65 73 20 6e 6f 74 20 61 6c 6c 6f 77 20 74 68 |oes not allow th|
+  0060: 69 73 20 66                                     |is f|
+
+#endif
+#if stream-bundle2-v3
   $ hg debugcapabilities http://localhost:$HGPORT
   Main capabilities:
     batch
     branchmap
     $USUAL_BUNDLE2_CAPS_SERVER$
     changegroupsubset
     compression=$BUNDLE2_COMPRESSIONS$
     getbundle
     httpheader=1024
     httpmediatype=0.1rx,0.1tx,0.2tx
     known
     lookup
     pushkey
     unbundle=HG10GZ,HG10BZ,HG10UN
     unbundlehash
   Bundle2 capabilities:
     HG20
     bookmarks
     changegroup
       01
       02
       03
     checkheads
       related
     digests
       md5
       sha1
       sha512
     error
       abort
       unsupportedcontent
       pushraced
       pushkey
     hgtagsfnodes
     listkeys
     phases
       heads
     pushkey
     remote-changegroup
       http
       https
 
   $ hg clone --stream -U http://localhost:$HGPORT server-disabled
   warning: stream clone requested but server has them disabled
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 3 changesets with 1088 changes to 1088 files
   new changesets 96ee1d7354c4:5223b5e3265f
 
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto 0.2 --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
   200 Script output follows
   content-type: application/mercurial-0.2
   
 
   $ f --size body --hexdump --bytes 100
   body: size=140
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 73 0b 45 52 52 4f 52 3a 41 42 4f 52 54 00 00 00 |s.ERROR:ABORT...|
   0020: 00 01 01 07 3c 04 16 6d 65 73 73 61 67 65 73 74 |....<..messagest|
   0030: 72 65 61 6d 20 64 61 74 61 20 72 65 71 75 65 73 |ream data reques|
   0040: 74 65 64 20 62 75 74 20 73 65 72 76 65 72 20 64 |ted but server d|
   0050: 6f 65 73 20 6e 6f 74 20 61 6c 6c 6f 77 20 74 68 |oes not allow th|
   0060: 69 73 20 66                                     |is f|
 
 #endif
 
   $ killdaemons.py
   $ cd server
   $ hg serve -p $HGPORT -d --pid-file=hg.pid --error errors.txt
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
 Basic clone
 
 #if stream-legacy
   $ hg clone --stream -U http://localhost:$HGPORT clone1
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
   $ cat server/errors.txt
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --stream -U http://localhost:$HGPORT clone1
+  streaming all changes
+  1093 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1093 files to transfer, 98.9 KB of data (zstd !)
+  transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
+
+  $ ls -1 clone1/.hg/cache
+  branch2-base
+  branch2-immutable
+  branch2-served
+  branch2-served.hidden
+  branch2-visible
+  branch2-visible-hidden
+  rbc-names-v1
+  rbc-revs-v1
+  tags2
+  tags2-served
+  $ cat server/errors.txt
+#endif
+#if stream-bundle2-v3
   $ hg clone --stream -U http://localhost:$HGPORT clone1
   streaming all changes
   1093 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1093 files to transfer, 98.9 KB of data (zstd !)
   transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
 
   $ ls -1 clone1/.hg/cache
   branch2-base
   branch2-immutable
   branch2-served
   branch2-served.hidden
   branch2-visible
   branch2-visible-hidden
   rbc-names-v1
   rbc-revs-v1
   tags2
   tags2-served
   $ cat server/errors.txt
 #endif
 
 getbundle requests with stream=1 are uncompressed
 
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=getbundle' content-type --bodyfile body --hgproto '0.1 0.2 comp=zlib,none' --requestheader "x-hgarg-1=bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps%250Astream%253Dv2&cg=0&common=0000000000000000000000000000000000000000&heads=c17445101a72edac06facd130d14808dfbd5c7c2&stream=1"
   200 Script output follows
   content-type: application/mercurial-0.2
   
 
 #if no-zstd no-rust
   $ f --size --hex --bytes 256 body
   body: size=119123
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 62 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |b.STREAM2.......|
   0020: 06 09 04 0c 26 62 79 74 65 63 6f 75 6e 74 31 30 |....&bytecount10|
   0030: 34 31 31 35 66 69 6c 65 63 6f 75 6e 74 31 30 39 |4115filecount109|
   0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
   0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 72 65 76 6c |eraldelta%2Crevl|
   0060: 6f 67 76 31 25 32 43 73 70 61 72 73 65 72 65 76 |ogv1%2Csparserev|
   0070: 6c 6f 67 00 00 80 00 73 08 42 64 61 74 61 2f 30 |log....s.Bdata/0|
   0080: 2e 69 00 03 00 01 00 00 00 00 00 00 00 02 00 00 |.i..............|
   0090: 00 01 00 00 00 00 00 00 00 01 ff ff ff ff ff ff |................|
   00a0: ff ff 80 29 63 a0 49 d3 23 87 bf ce fe 56 67 92 |...)c.I.#....Vg.|
   00b0: 67 2c 69 d1 ec 39 00 00 00 00 00 00 00 00 00 00 |g,i..9..........|
   00c0: 00 00 75 30 73 26 45 64 61 74 61 2f 30 30 63 68 |..u0s&Edata/00ch|
   00d0: 61 6e 67 65 6c 6f 67 2d 61 62 33 34 39 31 38 30 |angelog-ab349180|
   00e0: 61 30 34 30 35 30 31 30 2e 6e 64 2e 69 00 03 00 |a0405010.nd.i...|
   00f0: 01 00 00 00 00 00 00 00 05 00 00 00 04 00 00 00 |................|
 #endif
 #if zstd no-rust
   $ f --size --hex --bytes 256 body
   body: size=116310 (no-bigendian !)
   body: size=116305 (bigendian !)
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 7c 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 ||.STREAM2.......|
   0020: 06 09 04 0c 40 62 79 74 65 63 6f 75 6e 74 31 30 |....@bytecount10|
   0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109| (no-bigendian !)
   0030: 31 32 37 31 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1271filecount109| (bigendian !)
   0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
   0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 72 65 76 6c |eraldelta%2Crevl|
   0060: 6f 67 2d 63 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a |og-compression-z|
   0070: 73 74 64 25 32 43 72 65 76 6c 6f 67 76 31 25 32 |std%2Crevlogv1%2|
   0080: 43 73 70 61 72 73 65 72 65 76 6c 6f 67 00 00 80 |Csparserevlog...|
   0090: 00 73 08 42 64 61 74 61 2f 30 2e 69 00 03 00 01 |.s.Bdata/0.i....|
   00a0: 00 00 00 00 00 00 00 02 00 00 00 01 00 00 00 00 |................|
   00b0: 00 00 00 01 ff ff ff ff ff ff ff ff 80 29 63 a0 |.............)c.|
   00c0: 49 d3 23 87 bf ce fe 56 67 92 67 2c 69 d1 ec 39 |I.#....Vg.g,i..9|
   00d0: 00 00 00 00 00 00 00 00 00 00 00 00 75 30 73 26 |............u0s&|
   00e0: 45 64 61 74 61 2f 30 30 63 68 61 6e 67 65 6c 6f |Edata/00changelo|
   00f0: 67 2d 61 62 33 34 39 31 38 30 61 30 34 30 35 30 |g-ab349180a04050|
 #endif
 #if zstd rust no-dirstate-v2
   $ f --size --hex --bytes 256 body
   body: size=116310
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: 7c 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 ||.STREAM2.......|
   0020: 06 09 04 0c 40 62 79 74 65 63 6f 75 6e 74 31 30 |....@bytecount10|
   0030: 31 32 37 36 66 69 6c 65 63 6f 75 6e 74 31 30 39 |1276filecount109|
   0040: 33 72 65 71 75 69 72 65 6d 65 6e 74 73 67 65 6e |3requirementsgen|
   0050: 65 72 61 6c 64 65 6c 74 61 25 32 43 72 65 76 6c |eraldelta%2Crevl|
   0060: 6f 67 2d 63 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a |og-compression-z|
   0070: 73 74 64 25 32 43 72 65 76 6c 6f 67 76 31 25 32 |std%2Crevlogv1%2|
   0080: 43 73 70 61 72 73 65 72 65 76 6c 6f 67 00 00 80 |Csparserevlog...|
   0090: 00 73 08 42 64 61 74 61 2f 30 2e 69 00 03 00 01 |.s.Bdata/0.i....|
   00a0: 00 00 00 00 00 00 00 02 00 00 00 01 00 00 00 00 |................|
   00b0: 00 00 00 01 ff ff ff ff ff ff ff ff 80 29 63 a0 |.............)c.|
   00c0: 49 d3 23 87 bf ce fe 56 67 92 67 2c 69 d1 ec 39 |I.#....Vg.g,i..9|
   00d0: 00 00 00 00 00 00 00 00 00 00 00 00 75 30 73 26 |............u0s&|
   00e0: 45 64 61 74 61 2f 30 30 63 68 61 6e 67 65 6c 6f |Edata/00changelo|
   00f0: 67 2d 61 62 33 34 39 31 38 30 61 30 34 30 35 30 |g-ab349180a04050|
 #endif
 #if zstd dirstate-v2
   $ f --size --hex --bytes 256 body
   body: size=109549
   0000: 04 6e 6f 6e 65 48 47 32 30 00 00 00 00 00 00 00 |.noneHG20.......|
   0010: c0 07 53 54 52 45 41 4d 32 00 00 00 00 03 00 09 |..STREAM2.......|
   0020: 05 09 04 0c 85 62 79 74 65 63 6f 75 6e 74 39 35 |.....bytecount95|
   0030: 38 39 37 66 69 6c 65 63 6f 75 6e 74 31 30 33 30 |897filecount1030|
   0040: 72 65 71 75 69 72 65 6d 65 6e 74 73 64 6f 74 65 |requirementsdote|
   0050: 6e 63 6f 64 65 25 32 43 65 78 70 2d 64 69 72 73 |ncode%2Cexp-dirs|
   0060: 74 61 74 65 2d 76 32 25 32 43 66 6e 63 61 63 68 |tate-v2%2Cfncach|
   0070: 65 25 32 43 67 65 6e 65 72 61 6c 64 65 6c 74 61 |e%2Cgeneraldelta|
   0080: 25 32 43 70 65 72 73 69 73 74 65 6e 74 2d 6e 6f |%2Cpersistent-no|
   0090: 64 65 6d 61 70 25 32 43 72 65 76 6c 6f 67 2d 63 |demap%2Crevlog-c|
   00a0: 6f 6d 70 72 65 73 73 69 6f 6e 2d 7a 73 74 64 25 |ompression-zstd%|
   00b0: 32 43 72 65 76 6c 6f 67 76 31 25 32 43 73 70 61 |2Crevlogv1%2Cspa|
   00c0: 72 73 65 72 65 76 6c 6f 67 25 32 43 73 74 6f 72 |rserevlog%2Cstor|
   00d0: 65 00 00 80 00 73 08 42 64 61 74 61 2f 30 2e 69 |e....s.Bdata/0.i|
   00e0: 00 03 00 01 00 00 00 00 00 00 00 02 00 00 00 01 |................|
   00f0: 00 00 00 00 00 00 00 01 ff ff ff ff ff ff ff ff |................|
 #endif
 
 --uncompressed is an alias to --stream
 
 #if stream-legacy
   $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
+  streaming all changes
+  1093 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1093 files to transfer, 98.9 KB of data (zstd !)
+  transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
+#endif
+#if stream-bundle2-v3
   $ hg clone --uncompressed -U http://localhost:$HGPORT clone1-uncompressed
   streaming all changes
   1093 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1093 files to transfer, 98.9 KB of data (zstd !)
   transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
 #endif
 
 Clone with background file closing enabled
 
 #if stream-legacy
   $ hg --debug --config worker.backgroundclose=true --config worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT clone-background | grep -v adding
   using http://localhost:$HGPORT/
   sending capabilities command
   sending branchmap command
   streaming all changes
   sending stream_out command
   1090 files to transfer, 102 KB of data (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   starting 4 threads for background file closing
   updating the branch cache
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   query 1; heads
   sending batch command
   searching for changes
   all remote heads known locally
   no changes found
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
   updating the branch cache
   (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob)
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
   $ hg --debug --config worker.backgroundclose=true --config worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT clone-background | grep -v adding
   using http://localhost:$HGPORT/
   sending capabilities command
   query 1; heads
   sending batch command
   streaming all changes
   sending getbundle command
   bundle2-input-bundle: with-transaction
   bundle2-input-part: "stream2" (params: 3 mandatory) supported
   applying stream bundle
   1093 files to transfer, 102 KB of data (no-zstd !)
   1093 files to transfer, 98.9 KB of data (zstd !)
   starting 4 threads for background file closing
   starting 4 threads for background file closing
   updating the branch cache
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   bundle2-input-part: total payload size 118984 (no-zstd !)
   transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
   bundle2-input-part: total payload size 116145 (zstd no-bigendian !)
   bundle2-input-part: total payload size 116140 (zstd bigendian !)
   bundle2-input-part: "listkeys" (params: 1 mandatory) supported
   bundle2-input-bundle: 2 parts total
   checking for updated bookmarks
   updating the branch cache
   (sent 3 HTTP requests and * bytes; received * bytes in responses) (glob)
 #endif
+#if stream-bundle2-v3
+  $ hg --debug --config worker.backgroundclose=true --config worker.backgroundcloseminfilecount=1 clone --stream -U http://localhost:$HGPORT clone-background | grep -v adding
+  using http://localhost:$HGPORT/
+  sending capabilities command
+  query 1; heads
+  sending batch command
+  streaming all changes
+  sending getbundle command
+  bundle2-input-bundle: with-transaction
+  bundle2-input-part: "stream3" (params: 3 mandatory) supported
+  applying stream bundle
+  1093 files to transfer, 102 KB of data (no-zstd !)
+  1093 files to transfer, 98.9 KB of data (zstd !)
+  starting 4 threads for background file closing
+  starting 4 threads for background file closing
+  updating the branch cache
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  bundle2-input-part: total payload size 118984 (no-zstd !)
+  transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
+  bundle2-input-part: total payload size 116145 (zstd no-bigendian !)
+  bundle2-input-part: total payload size 116140 (zstd bigendian !)
+  bundle2-input-part: "listkeys" (params: 1 mandatory) supported
+  bundle2-input-bundle: 2 parts total
+  checking for updated bookmarks
+  updating the branch cache
+  (sent 3 HTTP requests and * bytes; received * bytes in responses) (glob)
+#endif
 
 Cannot stream clone when there are secret changesets
 
   $ hg -R server phase --force --secret -r tip
   $ hg clone --stream -U http://localhost:$HGPORT secret-denied
   warning: stream clone requested but server has them disabled
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 1025 changes to 1025 files
   new changesets 96ee1d7354c4:c17445101a72
 
   $ killdaemons.py
 
 Streaming of secrets can be overridden by server config
 
   $ cd server
   $ hg serve --config server.uncompressedallowsecret=true -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
 #if stream-legacy
   $ hg clone --stream -U http://localhost:$HGPORT secret-allowed
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --stream -U http://localhost:$HGPORT secret-allowed
+  streaming all changes
+  1093 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1093 files to transfer, 98.9 KB of data (zstd !)
+  transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
+#endif
+#if stream-bundle2-v3
   $ hg clone --stream -U http://localhost:$HGPORT secret-allowed
   streaming all changes
   1093 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1093 files to transfer, 98.9 KB of data (zstd !)
   transferred 98.9 KB in * seconds (* */sec) (glob) (zstd !)
 #endif
 
   $ killdaemons.py
 
 Verify interaction between preferuncompressed and secret presence
 
   $ cd server
   $ hg serve --config server.preferuncompressed=true -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
   $ hg clone -U http://localhost:$HGPORT preferuncompressed-secret
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 1025 changes to 1025 files
   new changesets 96ee1d7354c4:c17445101a72
 
   $ killdaemons.py
 
 Clone not allowed when full bundles disabled and can't serve secrets
 
   $ cd server
   $ hg serve --config server.disablefullbundle=true -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
   $ hg clone --stream http://localhost:$HGPORT secret-full-disabled
   warning: stream clone requested but server has them disabled
   requesting all changes
   remote: abort: server has pull-based clones disabled
   abort: pull failed on remote
   (remove --pull if specified or upgrade Mercurial)
   [100]
 
 Local stream clone with secrets involved
 (This is just a test over behavior: if you have access to the repo's files,
 there is no security so it isn't important to prevent a clone here.)
 
   $ hg clone -U --stream server local-secret
   warning: stream clone requested but server has them disabled
   requesting all changes
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 1025 changes to 1025 files
   new changesets 96ee1d7354c4:c17445101a72
 
 Stream clone while repo is changing:
 
   $ mkdir changing
   $ cd changing
 
 extension for delaying the server process so we reliably can modify the repo
 while cloning
 
   $ cat > stream_steps.py <<EOF
   > import os
   > import sys
   > from mercurial import (
   >     encoding,
   >     extensions,
   >     streamclone,
   >     testing,
   > )
   > WALKED_FILE_1 = encoding.environ[b'HG_TEST_STREAM_WALKED_FILE_1']
   > WALKED_FILE_2 = encoding.environ[b'HG_TEST_STREAM_WALKED_FILE_2']
   > 
   > def _test_sync_point_walk_1(orig, repo):
   >     testing.write_file(WALKED_FILE_1)
   > 
   > def _test_sync_point_walk_2(orig, repo):
   >     assert repo._currentlock(repo._lockref) is None
   >     testing.wait_file(WALKED_FILE_2)
   > 
   > extensions.wrapfunction(
   >     streamclone,
   >     '_test_sync_point_walk_1',
   >     _test_sync_point_walk_1
   > )
   > extensions.wrapfunction(
   >     streamclone,
   >     '_test_sync_point_walk_2',
   >     _test_sync_point_walk_2
   > )
   > EOF
 
 prepare repo with small and big file to cover both code paths in emitrevlogdata
 
   $ hg init repo
   $ touch repo/f1
   $ $TESTDIR/seq.py 50000 > repo/f2
   $ hg -R repo ci -Aqm "0"
   $ HG_TEST_STREAM_WALKED_FILE_1="$TESTTMP/sync_file_walked_1"
   $ export HG_TEST_STREAM_WALKED_FILE_1
   $ HG_TEST_STREAM_WALKED_FILE_2="$TESTTMP/sync_file_walked_2"
   $ export HG_TEST_STREAM_WALKED_FILE_2
   $ HG_TEST_STREAM_WALKED_FILE_3="$TESTTMP/sync_file_walked_3"
   $ export HG_TEST_STREAM_WALKED_FILE_3
 #   $ cat << EOF >> $HGRCPATH
 #   > [hooks]
 #   > pre-clone=rm -f "$TESTTMP/sync_file_walked_*"
 #   > EOF
   $ hg serve -R repo -p $HGPORT1 -d --error errors.log --pid-file=hg.pid --config extensions.stream_steps="$RUNTESTDIR/testlib/ext-stream-clone-steps.py"
   $ cat hg.pid >> $DAEMON_PIDS
 
 clone while modifying the repo between stating file with write lock and
 actually serving file content
 
   $ (hg clone -q --stream -U http://localhost:$HGPORT1 clone; touch "$HG_TEST_STREAM_WALKED_FILE_3") &
   $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1
   $ echo >> repo/f1
   $ echo >> repo/f2
   $ hg -R repo ci -m "1" --config ui.timeout.warn=-1
   $ touch $HG_TEST_STREAM_WALKED_FILE_2
   $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3
   $ hg -R clone id
   000000000000
   $ cat errors.log
   $ cd ..
 
 Stream repository with bookmarks
 --------------------------------
 
 (revert introduction of secret changeset)
 
   $ hg -R server phase --draft 'secret()'
 
 add a bookmark
 
   $ hg -R server bookmark -r tip some-bookmark
 
 clone it
 
 #if stream-legacy
   $ hg clone --stream http://localhost:$HGPORT with-bookmarks
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --stream http://localhost:$HGPORT with-bookmarks
+  streaming all changes
+  1096 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1096 files to transfer, 99.1 KB of data (zstd !)
+  transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
+  updating to branch default
+  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+#if stream-bundle2-v3
   $ hg clone --stream http://localhost:$HGPORT with-bookmarks
   streaming all changes
   1096 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1096 files to transfer, 99.1 KB of data (zstd !)
   transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
 #endif
   $ hg verify -R with-bookmarks -q
   $ hg -R with-bookmarks bookmarks
      some-bookmark             2:5223b5e3265f
 
 Stream repository with phases
 -----------------------------
 
 Clone as publishing
 
   $ hg -R server phase -r 'all()'
   0: draft
   1: draft
   2: draft
 
 #if stream-legacy
   $ hg clone --stream http://localhost:$HGPORT phase-publish
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --stream http://localhost:$HGPORT phase-publish
+  streaming all changes
+  1096 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1096 files to transfer, 99.1 KB of data (zstd !)
+  transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
+  updating to branch default
+  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+#endif
+#if stream-bundle2-v3
   $ hg clone --stream http://localhost:$HGPORT phase-publish
   streaming all changes
   1096 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1096 files to transfer, 99.1 KB of data (zstd !)
   transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
 #endif
   $ hg verify -R phase-publish -q
   $ hg -R phase-publish phase -r 'all()'
   0: public
   1: public
   2: public
 
 Clone as non publishing
 
   $ cat << EOF >> server/.hg/hgrc
   > [phases]
   > publish = False
   > EOF
   $ killdaemons.py
   $ hg -R server serve -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
 
 #if stream-legacy
 
 With v1 of the stream protocol, changeset are always cloned as public. It make
 stream v1 unsuitable for non-publishing repository.
 
   $ hg clone --stream http://localhost:$HGPORT phase-no-publish
   streaming all changes
   1090 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1090 files to transfer, 98.8 KB of data (zstd !)
   transferred 98.8 KB in * seconds (* */sec) (glob) (zstd !)
   searching for changes
   no changes found
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg -R phase-no-publish phase -r 'all()'
   0: public
   1: public
   2: public
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+  $ hg clone --stream http://localhost:$HGPORT phase-no-publish
+  streaming all changes
+  1097 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1097 files to transfer, 99.1 KB of data (zstd !)
+  transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
+  updating to branch default
+  1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R phase-no-publish phase -r 'all()'
+  0: draft
+  1: draft
+  2: draft
+#endif
+#if stream-bundle2-v3
   $ hg clone --stream http://localhost:$HGPORT phase-no-publish
   streaming all changes
   1097 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1097 files to transfer, 99.1 KB of data (zstd !)
   transferred 99.1 KB in * seconds (* */sec) (glob) (zstd !)
   updating to branch default
   1088 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg -R phase-no-publish phase -r 'all()'
   0: draft
   1: draft
   2: draft
 #endif
   $ hg verify -R phase-no-publish -q
 
   $ killdaemons.py
 
 #if stream-legacy
 
 With v1 of the stream protocol, changeset are always cloned as public. There's
 no obsolescence markers exchange in stream v1.
 
 #endif
-#if stream-bundle2
+#if stream-bundle2-v2
+
+Stream repository with obsolescence
+-----------------------------------
+
+Clone non-publishing with obsolescence
+
+  $ cat >> $HGRCPATH << EOF
+  > [experimental]
+  > evolution=all
+  > EOF
+
+  $ cd server
+  $ echo foo > foo
+  $ hg -q commit -m 'about to be pruned'
+  $ hg debugobsolete `hg log -r . -T '{node}'` -d '0 0' -u test --record-parents
+  1 new obsolescence markers
+  obsoleted 1 changesets
+  $ hg up null -q
+  $ hg log -T '{rev}: {phase}\n'
+  2: draft
+  1: draft
+  0: draft
+  $ hg serve -p $HGPORT -d --pid-file=hg.pid
+  $ cat hg.pid > $DAEMON_PIDS
+  $ cd ..
+
+  $ hg clone -U --stream http://localhost:$HGPORT with-obsolescence
+  streaming all changes
+  1098 files to transfer, 102 KB of data (no-zstd !)
+  transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
+  1098 files to transfer, 99.5 KB of data (zstd !)
+  transferred 99.5 KB in * seconds (* */sec) (glob) (zstd !)
+  $ hg -R with-obsolescence log -T '{rev}: {phase}\n'
+  2: draft
+  1: draft
+  0: draft
+  $ hg debugobsolete -R with-obsolescence
+  8c206a663911c1f97f2f9d7382e417ae55872cfa 0 {5223b5e3265f0df40bb743da62249413d74ac70f} (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
+  $ hg verify -R with-obsolescence -q
+
+  $ hg clone -U --stream --config experimental.evolution=0 http://localhost:$HGPORT with-obsolescence-no-evolution
+  streaming all changes
+  remote: abort: server has obsolescence markers, but client cannot receive them via stream clone
+  abort: pull failed on remote
+  [100]
+
+  $ killdaemons.py
+
+#endif
+#if stream-bundle2-v3
 
 Stream repository with obsolescence
 -----------------------------------
 
 Clone non-publishing with obsolescence
 
   $ cat >> $HGRCPATH << EOF
   > [experimental]
   > evolution=all
   > EOF
 
   $ cd server
   $ echo foo > foo
   $ hg -q commit -m 'about to be pruned'
   $ hg debugobsolete `hg log -r . -T '{node}'` -d '0 0' -u test --record-parents
   1 new obsolescence markers
   obsoleted 1 changesets
   $ hg up null -q
   $ hg log -T '{rev}: {phase}\n'
   2: draft
   1: draft
   0: draft
   $ hg serve -p $HGPORT -d --pid-file=hg.pid
   $ cat hg.pid > $DAEMON_PIDS
   $ cd ..
 
   $ hg clone -U --stream http://localhost:$HGPORT with-obsolescence
   streaming all changes
   1098 files to transfer, 102 KB of data (no-zstd !)
   transferred 102 KB in * seconds (* */sec) (glob) (no-zstd !)
   1098 files to transfer, 99.5 KB of data (zstd !)
   transferred 99.5 KB in * seconds (* */sec) (glob) (zstd !)
   $ hg -R with-obsolescence log -T '{rev}: {phase}\n'
   2: draft
   1: draft
   0: draft
   $ hg debugobsolete -R with-obsolescence
   8c206a663911c1f97f2f9d7382e417ae55872cfa 0 {5223b5e3265f0df40bb743da62249413d74ac70f} (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
   $ hg verify -R with-obsolescence -q
 
   $ hg clone -U --stream --config experimental.evolution=0 http://localhost:$HGPORT with-obsolescence-no-evolution
   streaming all changes
   remote: abort: server has obsolescence markers, but client cannot receive them via stream clone
   abort: pull failed on remote
   [100]
 
   $ killdaemons.py
 
 #endif
 
 Cloning a repo with no requirements doesn't give some obscure error
 
   $ mkdir -p empty-repo/.hg
   $ hg clone -q --stream ssh://user@dummy/empty-repo empty-repo2
   $ hg --cwd empty-repo2 verify -q