remotefilelog: include file contents in bundles produced during strip

`hg strip` and other things that use repair.strip (such as the narrow
extension's `hg tracked --removeinclude`) will "save" some commits that have a
higher revision number than the oldest commit we're stripping, but aren't
actually descended from any of the commits that we're stripping. It saves them
in a bundle, and then reapplies them to the repo.

Remotefilelog doesn't generally participate in strip, it doesn't contribute
files to either the backup bundle or the "saved" bundle, and doesn't adjust
linknodes when commits are stripped. This can break things like push, which
rely on the linknodes.

This change makes it so that remotefilelog includes files in these bundles
during strip operations. During reapplication, the files are reapplied from the
bundle, and the linknode is properly updated.

Differential Revision: https://phab.mercurial-scm.org/D10320

hgext/remotefilelog/__init__.py

@@ -1,1260 +1,1262 b''
 # __init__.py - remotefilelog extension
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """remotefilelog causes Mercurial to lazilly fetch file contents (EXPERIMENTAL)
 
 This extension is HIGHLY EXPERIMENTAL. There are NO BACKWARDS COMPATIBILITY
 GUARANTEES. This means that repositories created with this extension may
 only be usable with the exact version of this extension/Mercurial that was
 used. The extension attempts to enforce this in order to prevent repository
 corruption.
 
 remotefilelog works by fetching file contents lazily and storing them
 in a cache on the client rather than in revlogs. This allows enormous
 histories to be transferred only partially, making them easier to
 operate on.
 
 Configs:
 
     ``packs.maxchainlen`` specifies the maximum delta chain length in pack files
 
     ``packs.maxpacksize`` specifies the maximum pack file size
 
     ``packs.maxpackfilecount`` specifies the maximum number of packs in the
       shared cache (trees only for now)
 
     ``remotefilelog.backgroundprefetch`` runs prefetch in background when True
 
     ``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and
       update, and on other commands that use them. Different from pullprefetch.
 
     ``remotefilelog.gcrepack`` does garbage collection during repack when True
 
     ``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before
       it is garbage collected
 
     ``remotefilelog.repackonhggc`` runs repack on hg gc when True
 
     ``remotefilelog.prefetchdays`` specifies the maximum age of a commit in
       days after which it is no longer prefetched.
 
     ``remotefilelog.prefetchdelay`` specifies delay between background
       prefetches in seconds after operations that change the working copy parent
 
     ``remotefilelog.data.gencountlimit`` constraints the minimum number of data
       pack files required to be considered part of a generation. In particular,
       minimum number of packs files > gencountlimit.
 
     ``remotefilelog.data.generations`` list for specifying the lower bound of
       each generation of the data pack files. For example, list ['100MB','1MB']
       or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [
       1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to
       include in an incremental data repack.
 
     ``remotefilelog.data.repackmaxpacksize`` the maximum size of a pack file for
       it to be considered for an incremental data repack.
 
     ``remotefilelog.data.repacksizelimit`` the maximum total size of pack files
       to include in an incremental data repack.
 
     ``remotefilelog.history.gencountlimit`` constraints the minimum number of
       history pack files required to be considered part of a generation. In
       particular, minimum number of packs files > gencountlimit.
 
     ``remotefilelog.history.generations`` list for specifying the lower bound of
       each generation of the history pack files. For example, list [
       '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [
       0, 1MB), [1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to
       include in an incremental history repack.
 
     ``remotefilelog.history.repackmaxpacksize`` the maximum size of a pack file
       for it to be considered for an incremental history repack.
 
     ``remotefilelog.history.repacksizelimit`` the maximum total size of pack
       files to include in an incremental history repack.
 
     ``remotefilelog.backgroundrepack`` automatically consolidate packs in the
       background
 
     ``remotefilelog.cachepath`` path to cache
 
     ``remotefilelog.cachegroup`` if set, make cache directory sgid to this
       group
 
     ``remotefilelog.cacheprocess`` binary to invoke for fetching file data
 
     ``remotefilelog.debug`` turn on remotefilelog-specific debug output
 
     ``remotefilelog.excludepattern`` pattern of files to exclude from pulls
 
     ``remotefilelog.includepattern`` pattern of files to include in pulls
 
     ``remotefilelog.fetchwarning``: message to print when too many
       single-file fetches occur
 
     ``remotefilelog.getfilesstep`` number of files to request in a single RPC
 
     ``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch
       files, otherwise use optimistic fetching
 
     ``remotefilelog.pullprefetch`` revset for selecting files that should be
       eagerly downloaded rather than lazily
 
     ``remotefilelog.reponame`` name of the repo. If set, used to partition
       data from other repos in a shared store.
 
     ``remotefilelog.server`` if true, enable server-side functionality
 
     ``remotefilelog.servercachepath`` path for caching blobs on the server
 
     ``remotefilelog.serverexpiration`` number of days to keep cached server
       blobs
 
     ``remotefilelog.validatecache`` if set, check cache entries for corruption
       before returning blobs
 
     ``remotefilelog.validatecachelog`` if set, check cache entries for
       corruption before returning metadata
 
 """
 from __future__ import absolute_import
 
 import os
 import time
 import traceback
 
 from mercurial.node import (
     hex,
     wdirrev,
 )
 from mercurial.i18n import _
 from mercurial.pycompat import open
 from mercurial import (
     changegroup,
     changelog,
     commands,
     configitems,
     context,
     copies,
     debugcommands as hgdebugcommands,
     dispatch,
     error,
     exchange,
     extensions,
     hg,
     localrepo,
     match as matchmod,
     merge,
     mergestate as mergestatemod,
     patch,
     pycompat,
     registrar,
     repair,
     repoview,
     revset,
     scmutil,
     smartset,
     streamclone,
     util,
 )
 from . import (
     constants,
     debugcommands,
     fileserverclient,
     remotefilectx,
     remotefilelog,
     remotefilelogserver,
     repack as repackmod,
     shallowbundle,
     shallowrepo,
     shallowstore,
     shallowutil,
     shallowverifier,
 )
 
 # ensures debug commands are registered
 hgdebugcommands.command
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 configitem(b'remotefilelog', b'debug', default=False)
 
 configitem(b'remotefilelog', b'reponame', default=b'')
 configitem(b'remotefilelog', b'cachepath', default=None)
 configitem(b'remotefilelog', b'cachegroup', default=None)
 configitem(b'remotefilelog', b'cacheprocess', default=None)
 configitem(b'remotefilelog', b'cacheprocess.includepath', default=None)
 configitem(b"remotefilelog", b"cachelimit", default=b"1000 GB")
 
 configitem(
     b'remotefilelog',
     b'fallbackpath',
     default=configitems.dynamicdefault,
     alias=[(b'remotefilelog', b'fallbackrepo')],
 )
 
 configitem(b'remotefilelog', b'validatecachelog', default=None)
 configitem(b'remotefilelog', b'validatecache', default=b'on')
 configitem(b'remotefilelog', b'server', default=None)
 configitem(b'remotefilelog', b'servercachepath', default=None)
 configitem(b"remotefilelog", b"serverexpiration", default=30)
 configitem(b'remotefilelog', b'backgroundrepack', default=False)
 configitem(b'remotefilelog', b'bgprefetchrevs', default=None)
 configitem(b'remotefilelog', b'pullprefetch', default=None)
 configitem(b'remotefilelog', b'backgroundprefetch', default=False)
 configitem(b'remotefilelog', b'prefetchdelay', default=120)
 configitem(b'remotefilelog', b'prefetchdays', default=14)
+# Other values include 'local' or 'none'. Any unrecognized value is 'all'.
+configitem(b'remotefilelog', b'strip.includefiles', default='all')
 
 configitem(b'remotefilelog', b'getfilesstep', default=10000)
 configitem(b'remotefilelog', b'getfilestype', default=b'optimistic')
 configitem(b'remotefilelog', b'batchsize', configitems.dynamicdefault)
 configitem(b'remotefilelog', b'fetchwarning', default=b'')
 
 configitem(b'remotefilelog', b'includepattern', default=None)
 configitem(b'remotefilelog', b'excludepattern', default=None)
 
 configitem(b'remotefilelog', b'gcrepack', default=False)
 configitem(b'remotefilelog', b'repackonhggc', default=False)
 configitem(b'repack', b'chainorphansbysize', default=True, experimental=True)
 
 configitem(b'packs', b'maxpacksize', default=0)
 configitem(b'packs', b'maxchainlen', default=1000)
 
 configitem(b'devel', b'remotefilelog.bg-wait', default=False)
 
 #  default TTL limit is 30 days
 _defaultlimit = 60 * 60 * 24 * 30
 configitem(b'remotefilelog', b'nodettl', default=_defaultlimit)
 
 configitem(b'remotefilelog', b'data.gencountlimit', default=2),
 configitem(
     b'remotefilelog', b'data.generations', default=[b'1GB', b'100MB', b'1MB']
 )
 configitem(b'remotefilelog', b'data.maxrepackpacks', default=50)
 configitem(b'remotefilelog', b'data.repackmaxpacksize', default=b'4GB')
 configitem(b'remotefilelog', b'data.repacksizelimit', default=b'100MB')
 
 configitem(b'remotefilelog', b'history.gencountlimit', default=2),
 configitem(b'remotefilelog', b'history.generations', default=[b'100MB'])
 configitem(b'remotefilelog', b'history.maxrepackpacks', default=50)
 configitem(b'remotefilelog', b'history.repackmaxpacksize', default=b'400MB')
 configitem(b'remotefilelog', b'history.repacksizelimit', default=b'100MB')
 
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
 testedwith = b'ships-with-hg-core'
 
 repoclass = localrepo.localrepository
 repoclass._basesupported.add(constants.SHALLOWREPO_REQUIREMENT)
 
 isenabled = shallowutil.isenabled
 
 
 def uisetup(ui):
     """Wraps user facing Mercurial commands to swap them out with shallow
     versions.
     """
     hg.wirepeersetupfuncs.append(fileserverclient.peersetup)
 
     entry = extensions.wrapcommand(commands.table, b'clone', cloneshallow)
     entry[1].append(
         (
             b'',
             b'shallow',
             None,
             _(b"create a shallow clone which uses remote file history"),
         )
     )
 
     extensions.wrapcommand(
         commands.table, b'debugindex', debugcommands.debugindex
     )
     extensions.wrapcommand(
         commands.table, b'debugindexdot', debugcommands.debugindexdot
     )
     extensions.wrapcommand(commands.table, b'log', log)
     extensions.wrapcommand(commands.table, b'pull', pull)
 
     # Prevent 'hg manifest --all'
     def _manifest(orig, ui, repo, *args, **opts):
         if isenabled(repo) and opts.get('all'):
             raise error.Abort(_(b"--all is not supported in a shallow repo"))
 
         return orig(ui, repo, *args, **opts)
 
     extensions.wrapcommand(commands.table, b"manifest", _manifest)
 
     # Wrap remotefilelog with lfs code
     def _lfsloaded(loaded=False):
         lfsmod = None
         try:
             lfsmod = extensions.find(b'lfs')
         except KeyError:
             pass
         if lfsmod:
             lfsmod.wrapfilelog(remotefilelog.remotefilelog)
             fileserverclient._lfsmod = lfsmod
 
     extensions.afterloaded(b'lfs', _lfsloaded)
 
     # debugdata needs remotefilelog.len to work
     extensions.wrapcommand(commands.table, b'debugdata', debugdatashallow)
 
     changegroup.cgpacker = shallowbundle.shallowcg1packer
 
     extensions.wrapfunction(
         changegroup, b'_addchangegroupfiles', shallowbundle.addchangegroupfiles
     )
     extensions.wrapfunction(
         changegroup, b'makechangegroup', shallowbundle.makechangegroup
     )
     extensions.wrapfunction(localrepo, b'makestore', storewrapper)
     extensions.wrapfunction(exchange, b'pull', exchangepull)
     extensions.wrapfunction(merge, b'applyupdates', applyupdates)
     extensions.wrapfunction(merge, b'_checkunknownfiles', checkunknownfiles)
     extensions.wrapfunction(context.workingctx, b'_checklookup', checklookup)
     extensions.wrapfunction(scmutil, b'_findrenames', findrenames)
     extensions.wrapfunction(
         copies, b'_computeforwardmissing', computeforwardmissing
     )
     extensions.wrapfunction(dispatch, b'runcommand', runcommand)
     extensions.wrapfunction(repair, b'_collectbrokencsets', _collectbrokencsets)
     extensions.wrapfunction(context.changectx, b'filectx', filectx)
     extensions.wrapfunction(context.workingctx, b'filectx', workingfilectx)
     extensions.wrapfunction(patch, b'trydiff', trydiff)
     extensions.wrapfunction(hg, b'verify', _verify)
     scmutil.fileprefetchhooks.add(b'remotefilelog', _fileprefetchhook)
 
     # disappointing hacks below
     extensions.wrapfunction(scmutil, b'getrenamedfn', getrenamedfn)
     extensions.wrapfunction(revset, b'filelog', filelogrevset)
     revset.symbols[b'filelog'] = revset.filelog
 
 
 def cloneshallow(orig, ui, repo, *args, **opts):
     if opts.get('shallow'):
         repos = []
 
         def pull_shallow(orig, self, *args, **kwargs):
             if not isenabled(self):
                 repos.append(self.unfiltered())
                 # set up the client hooks so the post-clone update works
                 setupclient(self.ui, self.unfiltered())
 
                 # setupclient fixed the class on the repo itself
                 # but we also need to fix it on the repoview
                 if isinstance(self, repoview.repoview):
                     self.__class__.__bases__ = (
                         self.__class__.__bases__[0],
                         self.unfiltered().__class__,
                     )
                 self.requirements.add(constants.SHALLOWREPO_REQUIREMENT)
                 with self.lock():
                     # acquire store lock before writing requirements as some
                     # requirements might be written to .hg/store/requires
                     scmutil.writereporequirements(self)
 
                 # Since setupclient hadn't been called, exchange.pull was not
                 # wrapped. So we need to manually invoke our version of it.
                 return exchangepull(orig, self, *args, **kwargs)
             else:
                 return orig(self, *args, **kwargs)
 
         extensions.wrapfunction(exchange, b'pull', pull_shallow)
 
         # Wrap the stream logic to add requirements and to pass include/exclude
         # patterns around.
         def setup_streamout(repo, remote):
             # Replace remote.stream_out with a version that sends file
             # patterns.
             def stream_out_shallow(orig):
                 caps = remote.capabilities()
                 if constants.NETWORK_CAP_LEGACY_SSH_GETFILES in caps:
                     opts = {}
                     if repo.includepattern:
                         opts['includepattern'] = b'\0'.join(repo.includepattern)
                     if repo.excludepattern:
                         opts['excludepattern'] = b'\0'.join(repo.excludepattern)
                     return remote._callstream(b'stream_out_shallow', **opts)
                 else:
                     return orig()
 
             extensions.wrapfunction(remote, b'stream_out', stream_out_shallow)
 
         def stream_wrap(orig, op):
             setup_streamout(op.repo, op.remote)
             return orig(op)
 
         extensions.wrapfunction(
             streamclone, b'maybeperformlegacystreamclone', stream_wrap
         )
 
         def canperformstreamclone(orig, pullop, bundle2=False):
             # remotefilelog is currently incompatible with the
             # bundle2 flavor of streamclones, so force us to use
             # v1 instead.
             if b'v2' in pullop.remotebundle2caps.get(b'stream', []):
                 pullop.remotebundle2caps[b'stream'] = [
                     c for c in pullop.remotebundle2caps[b'stream'] if c != b'v2'
                 ]
             if bundle2:
                 return False, None
             supported, requirements = orig(pullop, bundle2=bundle2)
             if requirements is not None:
                 requirements.add(constants.SHALLOWREPO_REQUIREMENT)
             return supported, requirements
 
         extensions.wrapfunction(
             streamclone, b'canperformstreamclone', canperformstreamclone
         )
 
     try:
         orig(ui, repo, *args, **opts)
     finally:
         if opts.get('shallow'):
             for r in repos:
                 if util.safehasattr(r, b'fileservice'):
                     r.fileservice.close()
 
 
 def debugdatashallow(orig, *args, **kwds):
     oldlen = remotefilelog.remotefilelog.__len__
     try:
         remotefilelog.remotefilelog.__len__ = lambda x: 1
         return orig(*args, **kwds)
     finally:
         remotefilelog.remotefilelog.__len__ = oldlen
 
 
 def reposetup(ui, repo):
     if not repo.local():
         return
 
     # put here intentionally bc doesnt work in uisetup
     ui.setconfig(b'hooks', b'update.prefetch', wcpprefetch)
     ui.setconfig(b'hooks', b'commit.prefetch', wcpprefetch)
 
     isserverenabled = ui.configbool(b'remotefilelog', b'server')
     isshallowclient = isenabled(repo)
 
     if isserverenabled and isshallowclient:
         raise RuntimeError(b"Cannot be both a server and shallow client.")
 
     if isshallowclient:
         setupclient(ui, repo)
 
     if isserverenabled:
         remotefilelogserver.setupserver(ui, repo)
 
 
 def setupclient(ui, repo):
     if not isinstance(repo, localrepo.localrepository):
         return
 
     # Even clients get the server setup since they need to have the
     # wireprotocol endpoints registered.
     remotefilelogserver.onetimesetup(ui)
     onetimeclientsetup(ui)
 
     shallowrepo.wraprepo(repo)
     repo.store = shallowstore.wrapstore(repo.store)
 
 
 def storewrapper(orig, requirements, path, vfstype):
     s = orig(requirements, path, vfstype)
     if constants.SHALLOWREPO_REQUIREMENT in requirements:
         s = shallowstore.wrapstore(s)
 
     return s
 
 
 # prefetch files before update
 def applyupdates(
     orig, repo, mresult, wctx, mctx, overwrite, wantfiledata, **opts
 ):
     if isenabled(repo):
         manifest = mctx.manifest()
         files = []
         for f, args, msg in mresult.getactions([mergestatemod.ACTION_GET]):
             files.append((f, hex(manifest[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, mresult, wctx, mctx, overwrite, wantfiledata, **opts)
 
 
 # Prefetch merge checkunknownfiles
 def checkunknownfiles(orig, repo, wctx, mctx, force, mresult, *args, **kwargs):
     if isenabled(repo):
         files = []
         sparsematch = repo.maybesparsematch(mctx.rev())
         for f, (m, actionargs, msg) in mresult.filemap():
             if sparsematch and not sparsematch(f):
                 continue
             if m in (
                 mergestatemod.ACTION_CREATED,
                 mergestatemod.ACTION_DELETED_CHANGED,
                 mergestatemod.ACTION_CREATED_MERGE,
             ):
                 files.append((f, hex(mctx.filenode(f))))
             elif m == mergestatemod.ACTION_LOCAL_DIR_RENAME_GET:
                 f2 = actionargs[0]
                 files.append((f2, hex(mctx.filenode(f2))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, wctx, mctx, force, mresult, *args, **kwargs)
 
 
 # Prefetch files before status attempts to look at their size and contents
 def checklookup(orig, self, files):
     repo = self._repo
     if isenabled(repo):
         prefetchfiles = []
         for parent in self._parents:
             for f in files:
                 if f in parent:
                     prefetchfiles.append((f, hex(parent.filenode(f))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(prefetchfiles)
     return orig(self, files)
 
 
 # Prefetch the logic that compares added and removed files for renames
 def findrenames(orig, repo, matcher, added, removed, *args, **kwargs):
     if isenabled(repo):
         files = []
         pmf = repo[b'.'].manifest()
         for f in removed:
             if f in pmf:
                 files.append((f, hex(pmf[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, matcher, added, removed, *args, **kwargs)
 
 
 # prefetch files before pathcopies check
 def computeforwardmissing(orig, a, b, match=None):
     missing = orig(a, b, match=match)
     repo = a._repo
     if isenabled(repo):
         mb = b.manifest()
 
         files = []
         sparsematch = repo.maybesparsematch(b.rev())
         if sparsematch:
             sparsemissing = set()
             for f in missing:
                 if sparsematch(f):
                     files.append((f, hex(mb[f])))
                     sparsemissing.add(f)
             missing = sparsemissing
 
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return missing
 
 
 # close cache miss server connection after the command has finished
 def runcommand(orig, lui, repo, *args, **kwargs):
     fileservice = None
     # repo can be None when running in chg:
     # - at startup, reposetup was called because serve is not norepo
     # - a norepo command like "help" is called
     if repo and isenabled(repo):
         fileservice = repo.fileservice
     try:
         return orig(lui, repo, *args, **kwargs)
     finally:
         if fileservice:
             fileservice.close()
 
 
 # prevent strip from stripping remotefilelogs
 def _collectbrokencsets(orig, repo, files, striprev):
     if isenabled(repo):
         files = list([f for f in files if not repo.shallowmatch(f)])
     return orig(repo, files, striprev)
 
 
 # changectx wrappers
 def filectx(orig, self, path, fileid=None, filelog=None):
     if fileid is None:
         fileid = self.filenode(path)
     if isenabled(self._repo) and self._repo.shallowmatch(path):
         return remotefilectx.remotefilectx(
             self._repo, path, fileid=fileid, changectx=self, filelog=filelog
         )
     return orig(self, path, fileid=fileid, filelog=filelog)
 
 
 def workingfilectx(orig, self, path, filelog=None):
     if isenabled(self._repo) and self._repo.shallowmatch(path):
         return remotefilectx.remoteworkingfilectx(
             self._repo, path, workingctx=self, filelog=filelog
         )
     return orig(self, path, filelog=filelog)
 
 
 # prefetch required revisions before a diff
 def trydiff(
     orig,
     repo,
     revs,
     ctx1,
     ctx2,
     modified,
     added,
     removed,
     copy,
     getfilectx,
     *args,
     **kwargs
 ):
     if isenabled(repo):
         prefetch = []
         mf1 = ctx1.manifest()
         for fname in modified + added + removed:
             if fname in mf1:
                 fnode = getfilectx(fname, ctx1).filenode()
                 # fnode can be None if it's a edited working ctx file
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
             if fname not in removed:
                 fnode = getfilectx(fname, ctx2).filenode()
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
 
         repo.fileservice.prefetch(prefetch)
 
     return orig(
         repo,
         revs,
         ctx1,
         ctx2,
         modified,
         added,
         removed,
         copy,
         getfilectx,
         *args,
         **kwargs
     )
 
 
 # Prevent verify from processing files
 # a stub for mercurial.hg.verify()
 def _verify(orig, repo, level=None):
     lock = repo.lock()
     try:
         return shallowverifier.shallowverifier(repo).verify()
     finally:
         lock.release()
 
 
 clientonetime = False
 
 
 def onetimeclientsetup(ui):
     global clientonetime
     if clientonetime:
         return
     clientonetime = True
 
     # Don't commit filelogs until we know the commit hash, since the hash
     # is present in the filelog blob.
     # This violates Mercurial's filelog->manifest->changelog write order,
     # but is generally fine for client repos.
     pendingfilecommits = []
 
     def addrawrevision(
         orig,
         self,
         rawtext,
         transaction,
         link,
         p1,
         p2,
         node,
         flags,
         cachedelta=None,
         _metatuple=None,
     ):
         if isinstance(link, int):
             pendingfilecommits.append(
                 (
                     self,
                     rawtext,
                     transaction,
                     link,
                     p1,
                     p2,
                     node,
                     flags,
                     cachedelta,
                     _metatuple,
                 )
             )
             return node
         else:
             return orig(
                 self,
                 rawtext,
                 transaction,
                 link,
                 p1,
                 p2,
                 node,
                 flags,
                 cachedelta,
                 _metatuple=_metatuple,
             )
 
     extensions.wrapfunction(
         remotefilelog.remotefilelog, b'addrawrevision', addrawrevision
     )
 
     def changelogadd(orig, self, *args, **kwargs):
         oldlen = len(self)
         node = orig(self, *args, **kwargs)
         newlen = len(self)
         if oldlen != newlen:
             for oldargs in pendingfilecommits:
                 log, rt, tr, link, p1, p2, n, fl, c, m = oldargs
                 linknode = self.node(link)
                 if linknode == node:
                     log.addrawrevision(rt, tr, linknode, p1, p2, n, fl, c, m)
                 else:
                     raise error.ProgrammingError(
                         b'pending multiple integer revisions are not supported'
                     )
         else:
             # "link" is actually wrong here (it is set to len(changelog))
             # if changelog remains unchanged, skip writing file revisions
             # but still do a sanity check about pending multiple revisions
             if len({x[3] for x in pendingfilecommits}) > 1:
                 raise error.ProgrammingError(
                     b'pending multiple integer revisions are not supported'
                 )
         del pendingfilecommits[:]
         return node
 
     extensions.wrapfunction(changelog.changelog, b'add', changelogadd)
 
 
 def getrenamedfn(orig, repo, endrev=None):
     if not isenabled(repo) or copies.usechangesetcentricalgo(repo):
         return orig(repo, endrev)
 
     rcache = {}
 
     def getrenamed(fn, rev):
         """looks up all renames for a file (up to endrev) the first
         time the file is given. It indexes on the changerev and only
         parses the manifest if linkrev != changerev.
         Returns rename info for fn at changerev rev."""
         if rev in rcache.setdefault(fn, {}):
             return rcache[fn][rev]
 
         try:
             fctx = repo[rev].filectx(fn)
             for ancestor in fctx.ancestors():
                 if ancestor.path() == fn:
                     renamed = ancestor.renamed()
                     rcache[fn][ancestor.rev()] = renamed and renamed[0]
 
             renamed = fctx.renamed()
             return renamed and renamed[0]
         except error.LookupError:
             return None
 
     return getrenamed
 
 
 def filelogrevset(orig, repo, subset, x):
     """``filelog(pattern)``
     Changesets connected to the specified filelog.
 
     For performance reasons, ``filelog()`` does not show every changeset
     that affects the requested file(s). See :hg:`help log` for details. For
     a slower, more accurate result, use ``file()``.
     """
 
     if not isenabled(repo):
         return orig(repo, subset, x)
 
     # i18n: "filelog" is a keyword
     pat = revset.getstring(x, _(b"filelog requires a pattern"))
     m = matchmod.match(
         repo.root, repo.getcwd(), [pat], default=b'relpath', ctx=repo[None]
     )
     s = set()
 
     if not matchmod.patkind(pat):
         # slow
         for r in subset:
             ctx = repo[r]
             cfiles = ctx.files()
             for f in m.files():
                 if f in cfiles:
                     s.add(ctx.rev())
                     break
     else:
         # partial
         files = (f for f in repo[None] if m(f))
         for f in files:
             fctx = repo[None].filectx(f)
             s.add(fctx.linkrev())
             for actx in fctx.ancestors():
                 s.add(actx.linkrev())
 
     return smartset.baseset([r for r in subset if r in s])
 
 
 @command(b'gc', [], _(b'hg gc [REPO...]'), norepo=True)
 def gc(ui, *args, **opts):
     """garbage collect the client and server filelog caches"""
     cachepaths = set()
 
     # get the system client cache
     systemcache = shallowutil.getcachepath(ui, allowempty=True)
     if systemcache:
         cachepaths.add(systemcache)
 
     # get repo client and server cache
     repopaths = []
     pwd = ui.environ.get(b'PWD')
     if pwd:
         repopaths.append(pwd)
 
     repopaths.extend(args)
     repos = []
     for repopath in repopaths:
         try:
             repo = hg.peer(ui, {}, repopath)
             repos.append(repo)
 
             repocache = shallowutil.getcachepath(repo.ui, allowempty=True)
             if repocache:
                 cachepaths.add(repocache)
         except error.RepoError:
             pass
 
     # gc client cache
     for cachepath in cachepaths:
         gcclient(ui, cachepath)
 
     # gc server cache
     for repo in repos:
         remotefilelogserver.gcserver(ui, repo._repo)
 
 
 def gcclient(ui, cachepath):
     # get list of repos that use this cache
     repospath = os.path.join(cachepath, b'repos')
     if not os.path.exists(repospath):
         ui.warn(_(b"no known cache at %s\n") % cachepath)
         return
 
     reposfile = open(repospath, b'rb')
     repos = {r[:-1] for r in reposfile.readlines()}
     reposfile.close()
 
     # build list of useful files
     validrepos = []
     keepkeys = set()
 
     sharedcache = None
     filesrepacked = False
 
     count = 0
     progress = ui.makeprogress(
         _(b"analyzing repositories"), unit=b"repos", total=len(repos)
     )
     for path in repos:
         progress.update(count)
         count += 1
         try:
             path = ui.expandpath(os.path.normpath(path))
         except TypeError as e:
             ui.warn(_(b"warning: malformed path: %r:%s\n") % (path, e))
             traceback.print_exc()
             continue
         try:
             peer = hg.peer(ui, {}, path)
             repo = peer._repo
         except error.RepoError:
             continue
 
         validrepos.append(path)
 
         # Protect against any repo or config changes that have happened since
         # this repo was added to the repos file. We'd rather this loop succeed
         # and too much be deleted, than the loop fail and nothing gets deleted.
         if not isenabled(repo):
             continue
 
         if not util.safehasattr(repo, b'name'):
             ui.warn(
                 _(b"repo %s is a misconfigured remotefilelog repo\n") % path
             )
             continue
 
         # If garbage collection on repack and repack on hg gc are enabled
         # then loose files are repacked and garbage collected.
         # Otherwise regular garbage collection is performed.
         repackonhggc = repo.ui.configbool(b'remotefilelog', b'repackonhggc')
         gcrepack = repo.ui.configbool(b'remotefilelog', b'gcrepack')
         if repackonhggc and gcrepack:
             try:
                 repackmod.incrementalrepack(repo)
                 filesrepacked = True
                 continue
             except (IOError, repackmod.RepackAlreadyRunning):
                 # If repack cannot be performed due to not enough disk space
                 # continue doing garbage collection of loose files w/o repack
                 pass
 
         reponame = repo.name
         if not sharedcache:
             sharedcache = repo.sharedstore
 
         # Compute a keepset which is not garbage collected
         def keyfn(fname, fnode):
             return fileserverclient.getcachekey(reponame, fname, hex(fnode))
 
         keepkeys = repackmod.keepset(repo, keyfn=keyfn, lastkeepkeys=keepkeys)
 
     progress.complete()
 
     # write list of valid repos back
     oldumask = os.umask(0o002)
     try:
         reposfile = open(repospath, b'wb')
         reposfile.writelines([(b"%s\n" % r) for r in validrepos])
         reposfile.close()
     finally:
         os.umask(oldumask)
 
     # prune cache
     if sharedcache is not None:
         sharedcache.gc(keepkeys)
     elif not filesrepacked:
         ui.warn(_(b"warning: no valid repos in repofile\n"))
 
 
 def log(orig, ui, repo, *pats, **opts):
     if not isenabled(repo):
         return orig(ui, repo, *pats, **opts)
 
     follow = opts.get('follow')
     revs = opts.get('rev')
     if pats:
         # Force slowpath for non-follow patterns and follows that start from
         # non-working-copy-parent revs.
         if not follow or revs:
             # This forces the slowpath
             opts['removed'] = True
 
         # If this is a non-follow log without any revs specified, recommend that
         # the user add -f to speed it up.
         if not follow and not revs:
             match = scmutil.match(repo[b'.'], pats, pycompat.byteskwargs(opts))
             isfile = not match.anypats()
             if isfile:
                 for file in match.files():
                     if not os.path.isfile(repo.wjoin(file)):
                         isfile = False
                         break
 
             if isfile:
                 ui.warn(
                     _(
                         b"warning: file log can be slow on large repos - "
                         + b"use -f to speed it up\n"
                     )
                 )
 
     return orig(ui, repo, *pats, **opts)
 
 
 def revdatelimit(ui, revset):
     """Update revset so that only changesets no older than 'prefetchdays' days
     are included. The default value is set to 14 days. If 'prefetchdays' is set
     to zero or negative value then date restriction is not applied.
     """
     days = ui.configint(b'remotefilelog', b'prefetchdays')
     if days > 0:
         revset = b'(%s) & date(-%s)' % (revset, days)
     return revset
 
 
 def readytofetch(repo):
     """Check that enough time has passed since the last background prefetch.
     This only relates to prefetches after operations that change the working
     copy parent. Default delay between background prefetches is 2 minutes.
     """
     timeout = repo.ui.configint(b'remotefilelog', b'prefetchdelay')
     fname = repo.vfs.join(b'lastprefetch')
 
     ready = False
     with open(fname, b'a'):
         # the with construct above is used to avoid race conditions
         modtime = os.path.getmtime(fname)
         if (time.time() - modtime) > timeout:
             os.utime(fname, None)
             ready = True
 
     return ready
 
 
 def wcpprefetch(ui, repo, **kwargs):
     """Prefetches in background revisions specified by bgprefetchrevs revset.
     Does background repack if backgroundrepack flag is set in config.
     """
     shallow = isenabled(repo)
     bgprefetchrevs = ui.config(b'remotefilelog', b'bgprefetchrevs')
     isready = readytofetch(repo)
 
     if not (shallow and bgprefetchrevs and isready):
         return
 
     bgrepack = repo.ui.configbool(b'remotefilelog', b'backgroundrepack')
     # update a revset with a date limit
     bgprefetchrevs = revdatelimit(ui, bgprefetchrevs)
 
     def anon(unused_success):
         if util.safehasattr(repo, b'ranprefetch') and repo.ranprefetch:
             return
         repo.ranprefetch = True
         repo.backgroundprefetch(bgprefetchrevs, repack=bgrepack)
 
     repo._afterlock(anon)
 
 
 def pull(orig, ui, repo, *pats, **opts):
     result = orig(ui, repo, *pats, **opts)
 
     if isenabled(repo):
         # prefetch if it's configured
         prefetchrevset = ui.config(b'remotefilelog', b'pullprefetch')
         bgrepack = repo.ui.configbool(b'remotefilelog', b'backgroundrepack')
         bgprefetch = repo.ui.configbool(b'remotefilelog', b'backgroundprefetch')
 
         if prefetchrevset:
             ui.status(_(b"prefetching file contents\n"))
             revs = scmutil.revrange(repo, [prefetchrevset])
             base = repo[b'.'].rev()
             if bgprefetch:
                 repo.backgroundprefetch(prefetchrevset, repack=bgrepack)
             else:
                 repo.prefetch(revs, base=base)
                 if bgrepack:
                     repackmod.backgroundrepack(repo, incremental=True)
         elif bgrepack:
             repackmod.backgroundrepack(repo, incremental=True)
 
     return result
 
 
 def exchangepull(orig, repo, remote, *args, **kwargs):
     # Hook into the callstream/getbundle to insert bundle capabilities
     # during a pull.
     def localgetbundle(
         orig, source, heads=None, common=None, bundlecaps=None, **kwargs
     ):
         if not bundlecaps:
             bundlecaps = set()
         bundlecaps.add(constants.BUNDLE2_CAPABLITY)
         return orig(
             source, heads=heads, common=common, bundlecaps=bundlecaps, **kwargs
         )
 
     if util.safehasattr(remote, b'_callstream'):
         remote._localrepo = repo
     elif util.safehasattr(remote, b'getbundle'):
         extensions.wrapfunction(remote, b'getbundle', localgetbundle)
 
     return orig(repo, remote, *args, **kwargs)
 
 
 def _fileprefetchhook(repo, revmatches):
     if isenabled(repo):
         allfiles = []
         for rev, match in revmatches:
             if rev == wdirrev or rev is None:
                 continue
             ctx = repo[rev]
             mf = ctx.manifest()
             sparsematch = repo.maybesparsematch(ctx.rev())
             for path in ctx.walk(match):
                 if (not sparsematch or sparsematch(path)) and path in mf:
                     allfiles.append((path, hex(mf[path])))
         repo.fileservice.prefetch(allfiles)
 
 
 @command(
     b'debugremotefilelog',
     [
         (b'd', b'decompress', None, _(b'decompress the filelog first')),
     ],
     _(b'hg debugremotefilelog <path>'),
     norepo=True,
 )
 def debugremotefilelog(ui, path, **opts):
     return debugcommands.debugremotefilelog(ui, path, **opts)
 
 
 @command(
     b'verifyremotefilelog',
     [
         (b'd', b'decompress', None, _(b'decompress the filelogs first')),
     ],
     _(b'hg verifyremotefilelogs <directory>'),
     norepo=True,
 )
 def verifyremotefilelog(ui, path, **opts):
     return debugcommands.verifyremotefilelog(ui, path, **opts)
 
 
 @command(
     b'debugdatapack',
     [
         (b'', b'long', None, _(b'print the long hashes')),
         (b'', b'node', b'', _(b'dump the contents of node'), b'NODE'),
     ],
     _(b'hg debugdatapack <paths>'),
     norepo=True,
 )
 def debugdatapack(ui, *paths, **opts):
     return debugcommands.debugdatapack(ui, *paths, **opts)
 
 
 @command(b'debughistorypack', [], _(b'hg debughistorypack <path>'), norepo=True)
 def debughistorypack(ui, path, **opts):
     return debugcommands.debughistorypack(ui, path)
 
 
 @command(b'debugkeepset', [], _(b'hg debugkeepset'))
 def debugkeepset(ui, repo, **opts):
     # The command is used to measure keepset computation time
     def keyfn(fname, fnode):
         return fileserverclient.getcachekey(repo.name, fname, hex(fnode))
 
     repackmod.keepset(repo, keyfn)
     return
 
 
 @command(b'debugwaitonrepack', [], _(b'hg debugwaitonrepack'))
 def debugwaitonrepack(ui, repo, **opts):
     return debugcommands.debugwaitonrepack(repo)
 
 
 @command(b'debugwaitonprefetch', [], _(b'hg debugwaitonprefetch'))
 def debugwaitonprefetch(ui, repo, **opts):
     return debugcommands.debugwaitonprefetch(repo)
 
 
 def resolveprefetchopts(ui, opts):
     if not opts.get(b'rev'):
         revset = [b'.', b'draft()']
 
         prefetchrevset = ui.config(b'remotefilelog', b'pullprefetch', None)
         if prefetchrevset:
             revset.append(b'(%s)' % prefetchrevset)
         bgprefetchrevs = ui.config(b'remotefilelog', b'bgprefetchrevs', None)
         if bgprefetchrevs:
             revset.append(b'(%s)' % bgprefetchrevs)
         revset = b'+'.join(revset)
 
         # update a revset with a date limit
         revset = revdatelimit(ui, revset)
 
         opts[b'rev'] = [revset]
 
     if not opts.get(b'base'):
         opts[b'base'] = None
 
     return opts
 
 
 @command(
     b'prefetch',
     [
         (b'r', b'rev', [], _(b'prefetch the specified revisions'), _(b'REV')),
         (b'', b'repack', False, _(b'run repack after prefetch')),
         (b'b', b'base', b'', _(b"rev that is assumed to already be local")),
     ]
     + commands.walkopts,
     _(b'hg prefetch [OPTIONS] [FILE...]'),
     helpcategory=command.CATEGORY_MAINTENANCE,
 )
 def prefetch(ui, repo, *pats, **opts):
     """prefetch file revisions from the server
 
     Prefetchs file revisions for the specified revs and stores them in the
     local remotefilelog cache.  If no rev is specified, the default rev is
     used which is the union of dot, draft, pullprefetch and bgprefetchrev.
     File names or patterns can be used to limit which files are downloaded.
 
     Return 0 on success.
     """
     opts = pycompat.byteskwargs(opts)
     if not isenabled(repo):
         raise error.Abort(_(b"repo is not shallow"))
 
     opts = resolveprefetchopts(ui, opts)
     revs = scmutil.revrange(repo, opts.get(b'rev'))
     repo.prefetch(revs, opts.get(b'base'), pats, opts)
 
     # Run repack in background
     if opts.get(b'repack'):
         repackmod.backgroundrepack(repo, incremental=True)
 
 
 @command(
     b'repack',
     [
         (b'', b'background', None, _(b'run in a background process'), None),
         (b'', b'incremental', None, _(b'do an incremental repack'), None),
         (
             b'',
             b'packsonly',
             None,
             _(b'only repack packs (skip loose objects)'),
             None,
         ),
     ],
     _(b'hg repack [OPTIONS]'),
 )
 def repack_(ui, repo, *pats, **opts):
     if opts.get('background'):
         repackmod.backgroundrepack(
             repo,
             incremental=opts.get('incremental'),
             packsonly=opts.get('packsonly', False),
         )
         return
 
     options = {b'packsonly': opts.get('packsonly')}
 
     try:
         if opts.get('incremental'):
             repackmod.incrementalrepack(repo, options=options)
         else:
             repackmod.fullrepack(repo, options=options)
     except repackmod.RepackAlreadyRunning as ex:
         # Don't propogate the exception if the repack is already in
         # progress, since we want the command to exit 0.
         repo.ui.warn(b'%s\n' % ex)

hgext/remotefilelog/shallowbundle.py

@@ -1,307 +1,319 b''
 # shallowbundle.py - bundle10 implementation for use with shallow repositories
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 from __future__ import absolute_import
 
 from mercurial.i18n import _
 from mercurial.node import bin, hex, nullid
 from mercurial import (
     bundlerepo,
     changegroup,
     error,
     match,
     mdiff,
     pycompat,
 )
 from . import (
     constants,
     remotefilelog,
     shallowutil,
 )
 
 NoFiles = 0
 LocalFiles = 1
 AllFiles = 2
 
 
 def shallowgroup(cls, self, nodelist, rlog, lookup, units=None, reorder=None):
     if not isinstance(rlog, remotefilelog.remotefilelog):
         for c in super(cls, self).group(nodelist, rlog, lookup, units=units):
             yield c
         return
 
     if len(nodelist) == 0:
         yield self.close()
         return
 
     nodelist = shallowutil.sortnodes(nodelist, rlog.parents)
 
     # add the parent of the first rev
     p = rlog.parents(nodelist[0])[0]
     nodelist.insert(0, p)
 
     # build deltas
     for i in pycompat.xrange(len(nodelist) - 1):
         prev, curr = nodelist[i], nodelist[i + 1]
         linknode = lookup(curr)
         for c in self.nodechunk(rlog, curr, prev, linknode):
             yield c
 
     yield self.close()
 
 
 class shallowcg1packer(changegroup.cgpacker):
     def generate(self, commonrevs, clnodes, fastpathlinkrev, source, **kwargs):
         if shallowutil.isenabled(self._repo):
             fastpathlinkrev = False
 
         return super(shallowcg1packer, self).generate(
             commonrevs, clnodes, fastpathlinkrev, source, **kwargs
         )
 
     def group(self, nodelist, rlog, lookup, units=None, reorder=None):
         return shallowgroup(
             shallowcg1packer, self, nodelist, rlog, lookup, units=units
         )
 
     def generatefiles(self, changedfiles, *args, **kwargs):
         try:
             linknodes, commonrevs, source = args
         except ValueError:
             commonrevs, source, mfdicts, fastpathlinkrev, fnodes, clrevs = args
         if shallowutil.isenabled(self._repo):
             repo = self._repo
             if isinstance(repo, bundlerepo.bundlerepository):
                 # If the bundle contains filelogs, we can't pull from it, since
                 # bundlerepo is heavily tied to revlogs. Instead require that
                 # the user use unbundle instead.
                 # Force load the filelog data.
                 bundlerepo.bundlerepository.file(repo, b'foo')
                 if repo._cgfilespos:
                     raise error.Abort(
                         b"cannot pull from full bundles",
                         hint=b"use `hg unbundle` instead",
                     )
                 return []
             filestosend = self.shouldaddfilegroups(source)
             if filestosend == NoFiles:
                 changedfiles = list(
                     [f for f in changedfiles if not repo.shallowmatch(f)]
                 )
 
         return super(shallowcg1packer, self).generatefiles(
             changedfiles, *args, **kwargs
         )
 
     def shouldaddfilegroups(self, source):
         repo = self._repo
         if not shallowutil.isenabled(repo):
             return AllFiles
 
         if source == b"push" or source == b"bundle":
             return AllFiles
 
+        # We won't actually strip the files, but we should put them in any
+        # backup bundle generated by strip (especially for cases like narrow's
+        # `hg tracked --removeinclude`, as failing to do so means that the
+        # "saved" changesets during a strip won't have their files reapplied and
+        # thus their linknode adjusted, if necessary).
+        if source == b"strip":
+            cfg = repo.ui.config(b'remotefilelog', b'strip.includefiles')
+            if cfg == b'local':
+                return LocalFiles
+            elif cfg != b'none':
+                return AllFiles
+
         caps = self._bundlecaps or []
         if source == b"serve" or source == b"pull":
             if constants.BUNDLE2_CAPABLITY in caps:
                 return LocalFiles
             else:
                 # Serving to a full repo requires us to serve everything
                 repo.ui.warn(_(b"pulling from a shallow repo\n"))
                 return AllFiles
 
         return NoFiles
 
     def prune(self, rlog, missing, commonrevs):
         if not isinstance(rlog, remotefilelog.remotefilelog):
             return super(shallowcg1packer, self).prune(
                 rlog, missing, commonrevs
             )
 
         repo = self._repo
         results = []
         for fnode in missing:
             fctx = repo.filectx(rlog.filename, fileid=fnode)
             if fctx.linkrev() not in commonrevs:
                 results.append(fnode)
         return results
 
     def nodechunk(self, revlog, node, prevnode, linknode):
         prefix = b''
         if prevnode == nullid:
             delta = revlog.rawdata(node)
             prefix = mdiff.trivialdiffheader(len(delta))
         else:
             # Actually uses remotefilelog.revdiff which works on nodes, not revs
             delta = revlog.revdiff(prevnode, node)
         p1, p2 = revlog.parents(node)
         flags = revlog.flags(node)
         meta = self.builddeltaheader(node, p1, p2, prevnode, linknode, flags)
         meta += prefix
         l = len(meta) + len(delta)
         yield changegroup.chunkheader(l)
         yield meta
         yield delta
 
 
 def makechangegroup(orig, repo, outgoing, version, source, *args, **kwargs):
     if not shallowutil.isenabled(repo):
         return orig(repo, outgoing, version, source, *args, **kwargs)
 
     original = repo.shallowmatch
     try:
         # if serving, only send files the clients has patterns for
         if source == b'serve':
             bundlecaps = kwargs.get('bundlecaps')
             includepattern = None
             excludepattern = None
             for cap in bundlecaps or []:
                 if cap.startswith(b"includepattern="):
                     raw = cap[len(b"includepattern=") :]
                     if raw:
                         includepattern = raw.split(b'\0')
                 elif cap.startswith(b"excludepattern="):
                     raw = cap[len(b"excludepattern=") :]
                     if raw:
                         excludepattern = raw.split(b'\0')
             if includepattern or excludepattern:
                 repo.shallowmatch = match.match(
                     repo.root, b'', None, includepattern, excludepattern
                 )
             else:
                 repo.shallowmatch = match.always()
         return orig(repo, outgoing, version, source, *args, **kwargs)
     finally:
         repo.shallowmatch = original
 
 
 def addchangegroupfiles(
     orig, repo, source, revmap, trp, expectedfiles, *args, **kwargs
 ):
     if not shallowutil.isenabled(repo):
         return orig(repo, source, revmap, trp, expectedfiles, *args, **kwargs)
 
     newfiles = 0
     visited = set()
     revisiondatas = {}
     queue = []
 
     # Normal Mercurial processes each file one at a time, adding all
     # the new revisions for that file at once. In remotefilelog a file
     # revision may depend on a different file's revision (in the case
     # of a rename/copy), so we must lay all revisions down across all
     # files in topological order.
 
     # read all the file chunks but don't add them
     progress = repo.ui.makeprogress(_(b'files'), total=expectedfiles)
     while True:
         chunkdata = source.filelogheader()
         if not chunkdata:
             break
         f = chunkdata[b"filename"]
         repo.ui.debug(b"adding %s revisions\n" % f)
         progress.increment()
 
         if not repo.shallowmatch(f):
             fl = repo.file(f)
             deltas = source.deltaiter()
             fl.addgroup(deltas, revmap, trp)
             continue
 
         chain = None
         while True:
             # returns: (node, p1, p2, cs, deltabase, delta, flags) or None
             revisiondata = source.deltachunk(chain)
             if not revisiondata:
                 break
 
             chain = revisiondata[0]
 
             revisiondatas[(f, chain)] = revisiondata
             queue.append((f, chain))
 
             if f not in visited:
                 newfiles += 1
                 visited.add(f)
 
         if chain is None:
             raise error.Abort(_(b"received file revlog group is empty"))
 
     processed = set()
 
     def available(f, node, depf, depnode):
         if depnode != nullid and (depf, depnode) not in processed:
             if not (depf, depnode) in revisiondatas:
                 # It's not in the changegroup, assume it's already
                 # in the repo
                 return True
             # re-add self to queue
             queue.insert(0, (f, node))
             # add dependency in front
             queue.insert(0, (depf, depnode))
             return False
         return True
 
     skipcount = 0
 
     # Prefetch the non-bundled revisions that we will need
     prefetchfiles = []
     for f, node in queue:
         revisiondata = revisiondatas[(f, node)]
         # revisiondata: (node, p1, p2, cs, deltabase, delta, flags)
         dependents = [revisiondata[1], revisiondata[2], revisiondata[4]]
 
         for dependent in dependents:
             if dependent == nullid or (f, dependent) in revisiondatas:
                 continue
             prefetchfiles.append((f, hex(dependent)))
 
     repo.fileservice.prefetch(prefetchfiles)
 
     # Apply the revisions in topological order such that a revision
     # is only written once it's deltabase and parents have been written.
     while queue:
         f, node = queue.pop(0)
         if (f, node) in processed:
             continue
 
         skipcount += 1
         if skipcount > len(queue) + 1:
             raise error.Abort(_(b"circular node dependency"))
 
         fl = repo.file(f)
 
         revisiondata = revisiondatas[(f, node)]
         # revisiondata: (node, p1, p2, cs, deltabase, delta, flags)
         node, p1, p2, linknode, deltabase, delta, flags, sidedata = revisiondata
 
         if not available(f, node, f, deltabase):
             continue
 
         base = fl.rawdata(deltabase)
         text = mdiff.patch(base, delta)
         if not isinstance(text, bytes):
             text = bytes(text)
 
         meta, text = shallowutil.parsemeta(text)
         if b'copy' in meta:
             copyfrom = meta[b'copy']
             copynode = bin(meta[b'copyrev'])
             if not available(f, node, copyfrom, copynode):
                 continue
 
         for p in [p1, p2]:
             if p != nullid:
                 if not available(f, node, f, p):
                     continue
 
         fl.add(text, meta, trp, linknode, p1, p2)
         processed.add((f, node))
         skipcount = 0
 
     progress.complete()
 
     return len(revisiondatas), newfiles

tests/test-remotefilelog-bgprefetch.t

@@ -1,353 +1,354 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
   $ cat >> $HGRCPATH <<EOF
   > [devel]
   > remotefilelog.bg-wait=True
   > EOF
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ echo z > z
   $ hg commit -qAm x
   $ echo x2 > x
   $ echo y > y
   $ hg commit -qAm y
   $ echo w > w
   $ rm z
   $ hg commit -qAm w
   $ hg bookmark foo
 
   $ cd ..
 
 # clone the repo
 
   $ hgcloneshallow ssh://user@dummy/master shallow --noupdate
   streaming all changes
   2 files to transfer, 776 bytes of data
   transferred 776 bytes in * seconds (*/sec) (glob)
   searching for changes
   no changes found
 
 # Set the prefetchdays config to zero so that all commits are prefetched
 # no matter what their creation date is. Also set prefetchdelay config
 # to zero so that there is no delay between prefetches.
   $ cd shallow
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > prefetchdays=0
   > prefetchdelay=0
   > EOF
   $ cd ..
 
 # prefetch a revision
   $ cd shallow
 
   $ hg prefetch -r 0
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 x
   x
 
 # background prefetch on pull when configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > pullprefetch=bookmark()
   > backgroundprefetch=True
   > EOF
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 6b4b6f66ef8c
   (run 'hg update' to get a working copy)
   prefetching file contents
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/ef95c5376f34698742fe34f315fd82136f8f68c0
   $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
   $TESTTMP/hgcache/master/af/f024fe4ab0fece4091de044c58c9ae4233383a/bb6ccd5dceaa5e9dc220e0dad65e051b94f69a2c
   $TESTTMP/hgcache/repos
 
 # background prefetch with repack on pull when configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > backgroundrepack=True
   > EOF
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 6b4b6f66ef8c
   (run 'hg update' to get a working copy)
   prefetching file contents
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/6e8633deba6e544e5f8edbd7b996d6e31a2c42ae.histidx
   $TESTTMP/hgcache/master/packs/6e8633deba6e544e5f8edbd7b996d6e31a2c42ae.histpack
   $TESTTMP/hgcache/master/packs/8ce5ab3745465ab83bba30a7b9c295e0c8404652.dataidx
   $TESTTMP/hgcache/master/packs/8ce5ab3745465ab83bba30a7b9c295e0c8404652.datapack
   $TESTTMP/hgcache/repos
 
 # background prefetch with repack on update when wcprevset configured
 
   $ clearcache
   $ hg up -r 0
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
   $TESTTMP/hgcache/master/39/5df8f7c51f007019cb30201c49e884b46b92fa/69a1b67522704ec122181c0890bd16e9d3e7516a
   $TESTTMP/hgcache/repos
 
   $ hg up -r 1
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > bgprefetchrevs=.::
   > EOF
 
   $ clearcache
   $ hg up -r 0
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   * files fetched over * fetches - (* misses, 0.00% hit ratio) over *s (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/repos
 
 # Ensure that file 'w' was prefetched - it was not part of the update operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # background prefetch with repack on commit when wcprevset configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > bgprefetchrevs=0::
   > EOF
 
   $ clearcache
   $ find $CACHEDIR -type f | sort
   $ echo b > b
 .. The following output line about files fetches is globed because it is
 .. flaky, the core the test is checked when checking the cache dir, so
 .. hopefully this flakyness is not hiding any actual bug.
   $ hg commit -qAm b
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
   $ hg bookmark temporary
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/repos
 
 # Ensure that file 'w' was prefetched - it was not part of the commit operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # background prefetch with repack on rebase when wcprevset configured
 
   $ hg up -r 2
   3 files updated, 0 files merged, 2 files removed, 0 files unresolved
   (leaving bookmark temporary)
   $ clearcache
   $ find $CACHEDIR -type f | sort
 .. The following output line about files fetches is globed because it is
 .. flaky, the core the test is checked when checking the cache dir, so
 .. hopefully this flakyness is not hiding any actual bug.
   $ hg rebase -s temporary -d foo
   rebasing 3:d9cf06e3b5b6 temporary tip "b"
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/d9cf06e3b5b6-e5c3dc63-rebase.hg
   ? files fetched over ? fetches - (? misses, 0.00% hit ratio) over *s (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/repos
 
 # Ensure that file 'y' was prefetched - it was not part of the rebase operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Check that foregound prefetch with no arguments blocks until background prefetches finish
 
   $ hg up -r 3
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ clearcache
   $ hg prefetch --repack --config ui.timeout.warn=-1
   (running background incremental repack)
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/repos
 
 # Ensure that files were prefetched
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Check that foreground prefetch fetches revs specified by '. + draft() + bgprefetchrevs + pullprefetch'
 
   $ clearcache
   $ hg prefetch --repack --config ui.timeout.warn=-1
   (running background incremental repack)
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/repos
 
 # Ensure that files were prefetched
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Test that if data was prefetched and repacked we dont need to prefetch it again
 # It ensures that Mercurial looks not only in loose files but in packs as well
 
   $ hg prefetch --repack
   (running background incremental repack)

tests/test-remotefilelog-bundles.t

@@ -1,75 +1,76 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ hg commit -qAm x
   $ echo y >> x
   $ hg commit -qAm y
   $ echo z >> x
   $ hg commit -qAm z
 
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow -q
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ cd shallow
 
 Unbundling a shallow bundle
 
   $ hg strip -r 66ee28d0328c
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
-  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
+  2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ hg unbundle .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
   adding changesets
   adding manifests
   adding file changes
-  added 2 changesets with 0 changes to 0 files
+  added 2 changesets with 2 changes to 1 files
   new changesets 66ee28d0328c:16db62c5946f
   (run 'hg update' to get a working copy)
 
 Unbundling a full bundle
 
   $ hg -R ../master bundle -r 66ee28d0328c:: --base "66ee28d0328c^" ../fullbundle.hg
   2 changesets found
   $ hg strip -r 66ee28d0328c
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
   $ hg unbundle ../fullbundle.hg
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 1 files
   new changesets 66ee28d0328c:16db62c5946f (2 drafts)
   (run 'hg update' to get a working copy)
 
 Pulling from a shallow bundle
 
-  $ hg strip -r 66ee28d0328c
+  $ hg strip -r 66ee28d0328c --config remotefilelog.strip.includefiles=none
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg (glob)
   $ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
   pulling from .hg/strip-backup/66ee28d0328c-3d7aafd1-backup.hg
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 0 changes to 0 files
   new changesets 66ee28d0328c (1 drafts)
   (run 'hg update' to get a working copy)
 
-Pulling from a full bundle
+Pulling from a full bundle, also testing that strip produces a full bundle by
+default.
 
   $ hg strip -r 66ee28d0328c
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg (glob)
-  $ hg pull -r 66ee28d0328c ../fullbundle.hg
-  pulling from ../fullbundle.hg
+  $ hg pull -r 66ee28d0328c .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
+  pulling from .hg/strip-backup/66ee28d0328c-b6ee89e7-backup.hg
   searching for changes
   abort: cannot pull from full bundles
   (use `hg unbundle` instead)
   [255]

tests/test-remotefilelog-local.t

@@ -1,209 +1,210 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ echo y > y
   $ echo z > z
   $ hg commit -qAm xy
 
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow -q
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
   $ cd shallow
 
 # status
 
   $ clearcache
   $ echo xx > x
   $ echo yy > y
   $ touch a
   $ hg status
   M x
   M y
   ? a
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ hg add a
   $ hg status
   M x
   M y
   A a
 
 # diff
 
   $ hg debugrebuilddirstate # fixes dirstate non-determinism
   $ hg add a
   $ clearcache
   $ hg diff
   diff -r f3d0bb0d1e48 x
   --- a/x* (glob)
   +++ b/x* (glob)
   @@ -1,1 +1,1 @@
   -x
   +xx
   diff -r f3d0bb0d1e48 y
   --- a/y* (glob)
   +++ b/y* (glob)
   @@ -1,1 +1,1 @@
   -y
   +yy
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
 
 # local commit
 
   $ clearcache
   $ echo a > a
   $ echo xxx > x
   $ echo yyy > y
   $ hg commit -m a
   ? files fetched over 1 fetches - (? misses, 0.00% hit ratio) over *s (glob)
 
 # local commit where the dirstate is clean -- ensure that we do just one fetch
 # (update to a commit on the server first)
 
   $ hg --config debug.dirstate.delaywrite=1 up 0
   2 files updated, 0 files merged, 1 files removed, 0 files unresolved
   $ clearcache
   $ hg debugdirstate
   n 644          2 * x (glob)
   n 644          2 * y (glob)
   n 644          2 * z (glob)
   $ echo xxxx > x
   $ echo yyyy > y
   $ hg commit -m x
   created new head
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
 # restore state for future tests
 
   $ hg -q strip .
   $ hg -q up tip
 
 # rebase
 
   $ clearcache
   $ cd ../master
   $ echo w > w
   $ hg commit -qAm w
 
   $ cd ../shallow
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 0 changes to 0 files (+1 heads)
   new changesets fed61014d323
   (run 'hg heads' to see heads, 'hg merge' to merge)
 
   $ hg rebase -d tip
   rebasing 1:9abfe7bca547 "a"
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/9abfe7bca547-8b11e5ff-rebase.hg (glob)
   3 files fetched over 2 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
 
 # strip
 
   $ clearcache
   $ hg debugrebuilddirstate # fixes dirstate non-determinism
   $ hg strip -r .
   2 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg (glob)
-  4 files fetched over 2 fetches - (4 misses, 0.00% hit ratio) over *s (glob)
+  3 files fetched over 2 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
 
 # unbundle
 
   $ clearcache
   $ ls -A
   .hg
   w
   x
   y
   z
 
   $ hg debugrebuilddirstate # fixes dirstate non-determinism
   $ hg unbundle .hg/strip-backup/19edf50f4de7-df3d0f74-backup.hg
   adding changesets
   adding manifests
   adding file changes
-  added 1 changesets with 0 changes to 0 files
+  added 1 changesets with 3 changes to 3 files
   new changesets 19edf50f4de7 (1 drafts)
   (run 'hg update' to get a working copy)
+  2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg up
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
-  4 files fetched over 1 fetches - (4 misses, 0.00% hit ratio) over *s (glob)
+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ cat a
   a
 
 # revert
 
   $ clearcache
   $ hg revert -r .~2 y z
   no changes needed to z
-  2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ hg checkout -C -r . -q
 
 # explicit bundle should produce full bundle file
 
   $ hg bundle -r 2 --base 1 ../local.bundle
   1 changesets found
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow2 -q
-  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
+  2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ cd shallow2
   $ hg unbundle ../local.bundle
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 3 changes to 3 files
   new changesets 19edf50f4de7 (1 drafts)
   (run 'hg update' to get a working copy)
 
   $ hg log -r 2 --stat
   changeset:   2:19edf50f4de7
   tag:         tip
   user:        test
   date:        Thu Jan 01 00:00:00 1970 +0000
   summary:     a
   
    a |  1 +
    x |  2 +-
    y |  2 +-
    3 files changed, 3 insertions(+), 2 deletions(-)
   
 # Merge
 
   $ echo merge >> w
   $ hg commit -m w
   created new head
   $ hg merge 2
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
   $ hg commit -m merge
   $ hg strip -q -r ".^"
 
 # commit without producing new node
 
   $ cd $TESTTMP
   $ hgcloneshallow ssh://user@dummy/master shallow3 -q
   $ cd shallow3
   $ echo 1 > A
   $ hg commit -m foo -A A
   $ hg log -r . -T '{node}\n'
   383ce605500277f879b7460a16ba620eb6930b7f
   $ hg update -r '.^' -q
   $ echo 1 > A
   $ hg commit -m foo -A A
   warning: commit already existed in the repository!
   $ hg log -r . -T '{node}\n'
   383ce605500277f879b7460a16ba620eb6930b7f

tests/test-remotefilelog-prefetch.t

@@ -1,280 +1,281 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ echo z > z
   $ hg commit -qAm x
   $ echo x2 > x
   $ echo y > y
   $ hg commit -qAm y
   $ hg bookmark foo
 
   $ cd ..
 
 # prefetch a revision
 
   $ hgcloneshallow ssh://user@dummy/master shallow --noupdate
   streaming all changes
   2 files to transfer, 528 bytes of data
   transferred 528 bytes in * seconds (*/sec) (glob)
   searching for changes
   no changes found
   $ cd shallow
 
   $ hg prefetch -r 0
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 x
   x
 
 # prefetch with base
 
   $ clearcache
   $ hg prefetch -r 0::1 -b 0
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 1 x
   x2
   $ hg cat -r 1 y
   y
 
   $ hg cat -r 0 x
   x
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 z
   z
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg prefetch -r 0::1 --base 0
   $ hg prefetch -r 0::1 -b 1
   $ hg prefetch -r 0::1
 
 # prefetch a range of revisions
 
   $ clearcache
   $ hg prefetch -r 0::1
   4 files fetched over 1 fetches - (4 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 x
   x
   $ hg cat -r 1 x
   x2
 
 # prefetch certain files
 
   $ clearcache
   $ hg prefetch -r 1 x
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 1 x
   x2
 
   $ hg cat -r 1 y
   y
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
 # prefetch on pull when configured
 
   $ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob)
+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 109c3a557a73
   (run 'hg update' to get a working copy)
   prefetching file contents
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg up tip
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 # prefetch only fetches changes not in working copy
 
   $ hg strip tip
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/109c3a557a73-3f43405e-backup.hg (glob)
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ clearcache
 
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 109c3a557a73
   (run 'hg update' to get a working copy)
   prefetching file contents
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
 # Make some local commits that produce the same file versions as are on the
 # server. To simulate a situation where we have local commits that were somehow
 # pushed, and we will soon pull.
 
   $ hg prefetch -r 'all()'
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ hg strip -q -r 0
   $ echo x > x
   $ echo z > z
   $ hg commit -qAm x
   $ echo x2 > x
   $ echo y > y
   $ hg commit -qAm y
 
 # prefetch server versions, even if local versions are available
 
   $ clearcache
   $ hg strip -q tip
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 109c3a557a73
   1 local changesets published (?)
   (run 'hg update' to get a working copy)
   prefetching file contents
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ cd ..
 
 # Prefetch unknown files during checkout
 
   $ hgcloneshallow ssh://user@dummy/master shallow2
   streaming all changes
   2 files to transfer, 528 bytes of data
   transferred 528 bytes in * seconds * (glob)
   searching for changes
   no changes found
   updating to branch default
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ cd shallow2
   $ hg up -q null
   $ echo x > x
   $ echo y > y
   $ echo z > z
   $ clearcache
   $ hg up tip
   x: untracked file differs
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob)
   abort: untracked files in working directory differ from files in requested revision
   [20]
   $ hg revert --all
 
 # Test batch fetching of lookup files during hg status
   $ hg up --clean tip
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg debugrebuilddirstate
   $ clearcache
   $ hg status
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob)
 
 # Prefetch during addrename detection
   $ hg up -q --clean tip
   $ hg revert --all
   $ mv x x2
   $ mv y y2
   $ mv z z2
   $ echo a > a
   $ hg add a
   $ rm a
   $ clearcache
   $ hg addremove -s 50 > /dev/null
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob)
   $ hg revert --all
   forgetting x2
   forgetting y2
   forgetting z2
   undeleting x
   undeleting y
   undeleting z
 
 
 # Revert across double renames. Note: the scary "abort", error is because
 # https://bz.mercurial-scm.org/5419 .
 
   $ cd ../master
   $ hg mv z z2
   $ hg commit -m 'move z -> z2'
   $ cd ../shallow2
   $ hg pull -q
   $ clearcache
   $ hg mv y y2
   y2: not overwriting - file exists
   ('hg rename --after' to record the rename)
   [1]
   $ hg mv x x2
   x2: not overwriting - file exists
   ('hg rename --after' to record the rename)
   [1]
   $ hg mv z2 z3
   z2: not copying - file is not managed
   abort: no files to copy
   [10]
   $ find $CACHEDIR -type f | sort
 .. The following output line about files fetches is globed because it is
 .. flaky, the core the test is checked when checking the cache dir, so
 .. hopefully this flakyness is not hiding any actual bug.
   $ hg revert -a -r 1 || true
   ? files fetched over 1 fetches - (? misses, 0.00% hit ratio) over * (glob)
   abort: z2@109c3a557a73: not found in manifest (?)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/ef95c5376f34698742fe34f315fd82136f8f68c0
   $TESTTMP/hgcache/master/39/5df8f7c51f007019cb30201c49e884b46b92fa/69a1b67522704ec122181c0890bd16e9d3e7516a
   $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
   $TESTTMP/hgcache/repos
 
 # warning when we have excess remotefilelog fetching
 
   $ cat > repeated_fetch.py << EOF
   > import binascii
   > from mercurial import extensions, registrar
   > cmdtable = {}
   > command = registrar.command(cmdtable)
   > @command(b'repeated-fetch', [], b'', inferrepo=True)
   > def repeated_fetch(ui, repo, *args, **opts):
   >     for i in range(20):
   >         try:
   >             hexid = (b'%02x' % (i + 1)) * 20
   >             repo.fileservice.prefetch([(b'somefile.txt', hexid)])
   >         except Exception:
   >             pass
   > EOF
 
 We should only output to the user once. We're ignoring most of the output
 because we're not actually fetching anything real here, all the hashes are
 bogus, so it's just going to be errors and a final summary of all the misses.
   $ hg --config extensions.repeated_fetch=repeated_fetch.py \
   >    --config remotefilelog.fetchwarning="fetch warning!" \
   >    --config extensions.blackbox= \
   >    repeated-fetch 2>&1 | grep 'fetch warning'
   fetch warning!
 
 We should output to blackbox three times, with a stack trace on each (though
 that isn't tested here).
   $ grep 'excess remotefilelog fetching' .hg/blackbox.log
   .* excess remotefilelog fetching: (re)
   .* excess remotefilelog fetching: (re)
   .* excess remotefilelog fetching: (re)

tests/test-remotefilelog-sparse.t

@@ -1,102 +1,103 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ echo z > z
   $ hg commit -qAm x1
   $ echo x2 > x
   $ echo z2 > z
   $ hg commit -qAm x2
   $ hg bookmark foo
 
   $ cd ..
 
 # prefetch a revision w/ a sparse checkout
 
   $ hgcloneshallow ssh://user@dummy/master shallow --noupdate
   streaming all changes
   2 files to transfer, 527 bytes of data
   transferred 527 bytes in 0.* seconds (*/sec) (glob)
   searching for changes
   no changes found
   $ cd shallow
   $ printf "[extensions]\nsparse=\n" >> .hg/hgrc
 
   $ hg debugsparse -I x
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 x
   x
 
   $ hg debugsparse -I z
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 z
   z
 
 # prefetch sparse only on pull when configured
 
   $ printf "[remotefilelog]\npullprefetch=bookmark()\n" >> .hg/hgrc
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/876b1317060d-b2e91d8d-backup.hg (glob)
+  2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg debugsparse --delete z
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   updating bookmark foo
   added 1 changesets with 0 changes to 0 files
   new changesets 876b1317060d
   (run 'hg update' to get a working copy)
   prefetching file contents
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
 # Dont consider filtered files when doing copy tracing
 
 ## Push an unrelated commit
   $ cd ../
 
   $ hgcloneshallow ssh://user@dummy/master shallow2
   streaming all changes
   2 files to transfer, 527 bytes of data
   transferred 527 bytes in 0.* seconds (*) (glob)
   searching for changes
   no changes found
   updating to branch default
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ cd shallow2
   $ printf "[extensions]\nsparse=\n" >> .hg/hgrc
 
   $ hg up -q 0
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ touch a
   $ hg ci -Aqm a
   $ hg push -q -f
 
 ## Pull the unrelated commit and rebase onto it - verify unrelated file was not
 pulled
 
   $ cd ../shallow
   $ hg up -q 1
   $ hg pull -q
   $ hg debugsparse -I z
   $ clearcache
   $ hg prefetch -r '. + .^' -I x -I z
   4 files fetched over 1 fetches - (4 misses, 0.00% hit ratio) over * (glob)
   $ hg rebase -d 2 --keep
   rebasing 1:876b1317060d foo "x2"

tests/test-remotefilelog-strip.t

@@ -1,67 +1,68 b''
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ hg commit -qAm x
 
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow -q
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
   $ cd shallow
 
   $ cat >> $TESTTMP/get_file_linknode.py <<EOF
   > from mercurial import node, registrar, scmutil
   > cmdtable = {}
   > command = registrar.command(cmdtable)
   > @command(b'debug-file-linknode', [(b'r', b'rev', b'.', b'rev')], b'hg debug-file-linknode FILE')
   > def debug_file_linknode(ui, repo, file, **opts):
   >   rflctx = scmutil.revsingle(repo.unfiltered(), opts['rev']).filectx(file)
   >   ui.status(b'%s\n' % node.hex(rflctx.ancestormap()[rflctx._filenode][2]))
   > EOF
 
   $ cat >> .hg/hgrc <<EOF
   > [ui]
   > interactive=1
   > [extensions]
   > strip=
   > get_file_linknode=$TESTTMP/get_file_linknode.py
   > [experimental]
   > evolution=createmarkers,allowunstable
   > EOF
   $ echo a > a
   $ hg commit -qAm msg1
   $ hg commit --amend 're:^$' -m msg2
   $ hg commit --amend 're:^$' -m msg3
   $ hg --hidden log -G -T '{rev} {node|short}'
   @  3 df91f74b871e
   |
   | x  2 70494d7ec5ef
   |/
   | x  1 1e423846dde0
   |/
   o  0 b292c1e3311f
   
   $ hg debug-file-linknode -r 70494d a
   df91f74b871e064c89afa1fe9e2f66afa2c125df
   $ hg --hidden strip -r 1 3
   0 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/df91f74b871e-c94d67be-backup.hg
 
   $ hg --hidden log -G -T '{rev} {node|short}'
   o  1 70494d7ec5ef
   |
   @  0 b292c1e3311f
   
-FIXME: This should point to a commit that actually exists in the repo. Otherwise
-remotefilelog has to search every commit in the repository looking for a valid
-linkrev every time it's queried, such as during push.
+Demonstrate that the linknode points to a commit that is actually in the repo
+after the strip operation. Otherwise remotefilelog has to search every commit in
+the repository looking for a valid linkrev every time it's queried, such as
+during push.
   $ hg debug-file-linknode -r 70494d a
-  df91f74b871e064c89afa1fe9e2f66afa2c125df
+  70494d7ec5ef6cd3cd6939a9fd2812f9956bf553