remotefilelog: reduce probability of race-condition in remotefilelog tests

ca1014ad3de4 introduced a new parameter `ensurestart` to speed up
remotefilelog background processes start. Unfortunately it seems to have
increased the possibility of race-conditions in remotefilelog tests testing
those background processes.

With `ensurestart=False`, it seems that it's more probable to enter in a race
condition with `debugwaitonprefetch` and `debugwaitonrepack` in remotefilelog
background tests. Our CI seems to have a high probability of triggering this
race condition so make it configurable to ensure tests are stable.

Differential Revision: https://phab.mercurial-scm.org/D6772

hgext/remotefilelog/__init__.py

@@ -1,1113 +1,1124
 # __init__.py - remotefilelog extension
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """remotefilelog causes Mercurial to lazilly fetch file contents (EXPERIMENTAL)
 
 This extension is HIGHLY EXPERIMENTAL. There are NO BACKWARDS COMPATIBILITY
 GUARANTEES. This means that repositories created with this extension may
 only be usable with the exact version of this extension/Mercurial that was
 used. The extension attempts to enforce this in order to prevent repository
 corruption.
 
 remotefilelog works by fetching file contents lazily and storing them
 in a cache on the client rather than in revlogs. This allows enormous
 histories to be transferred only partially, making them easier to
 operate on.
 
 Configs:
 
     ``packs.maxchainlen`` specifies the maximum delta chain length in pack files
 
     ``packs.maxpacksize`` specifies the maximum pack file size
 
     ``packs.maxpackfilecount`` specifies the maximum number of packs in the
       shared cache (trees only for now)
 
     ``remotefilelog.backgroundprefetch`` runs prefetch in background when True
 
     ``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and
       update, and on other commands that use them. Different from pullprefetch.
 
     ``remotefilelog.gcrepack`` does garbage collection during repack when True
 
     ``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before
       it is garbage collected
 
     ``remotefilelog.repackonhggc`` runs repack on hg gc when True
 
     ``remotefilelog.prefetchdays`` specifies the maximum age of a commit in
       days after which it is no longer prefetched.
 
     ``remotefilelog.prefetchdelay`` specifies delay between background
       prefetches in seconds after operations that change the working copy parent
 
     ``remotefilelog.data.gencountlimit`` constraints the minimum number of data
       pack files required to be considered part of a generation. In particular,
       minimum number of packs files > gencountlimit.
 
     ``remotefilelog.data.generations`` list for specifying the lower bound of
       each generation of the data pack files. For example, list ['100MB','1MB']
       or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [
       1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to
       include in an incremental data repack.
 
     ``remotefilelog.data.repackmaxpacksize`` the maximum size of a pack file for
       it to be considered for an incremental data repack.
 
     ``remotefilelog.data.repacksizelimit`` the maximum total size of pack files
       to include in an incremental data repack.
 
     ``remotefilelog.history.gencountlimit`` constraints the minimum number of
       history pack files required to be considered part of a generation. In
       particular, minimum number of packs files > gencountlimit.
 
     ``remotefilelog.history.generations`` list for specifying the lower bound of
       each generation of the history pack files. For example, list [
       '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [
       0, 1MB), [1MB, 100MB) and [100MB, infinity).
 
     ``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to
       include in an incremental history repack.
 
     ``remotefilelog.history.repackmaxpacksize`` the maximum size of a pack file
       for it to be considered for an incremental history repack.
 
     ``remotefilelog.history.repacksizelimit`` the maximum total size of pack
       files to include in an incremental history repack.
 
     ``remotefilelog.backgroundrepack`` automatically consolidate packs in the
       background
 
     ``remotefilelog.cachepath`` path to cache
 
     ``remotefilelog.cachegroup`` if set, make cache directory sgid to this
       group
 
     ``remotefilelog.cacheprocess`` binary to invoke for fetching file data
 
     ``remotefilelog.debug`` turn on remotefilelog-specific debug output
 
     ``remotefilelog.excludepattern`` pattern of files to exclude from pulls
 
     ``remotefilelog.includepattern`` pattern of files to include in pulls
 
     ``remotefilelog.fetchwarning``: message to print when too many
       single-file fetches occur
 
     ``remotefilelog.getfilesstep`` number of files to request in a single RPC
 
     ``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch
       files, otherwise use optimistic fetching
 
     ``remotefilelog.pullprefetch`` revset for selecting files that should be
       eagerly downloaded rather than lazily
 
     ``remotefilelog.reponame`` name of the repo. If set, used to partition
       data from other repos in a shared store.
 
     ``remotefilelog.server`` if true, enable server-side functionality
 
     ``remotefilelog.servercachepath`` path for caching blobs on the server
 
     ``remotefilelog.serverexpiration`` number of days to keep cached server
       blobs
 
     ``remotefilelog.validatecache`` if set, check cache entries for corruption
       before returning blobs
 
     ``remotefilelog.validatecachelog`` if set, check cache entries for
       corruption before returning metadata
 
 """
 from __future__ import absolute_import
 
 import os
 import time
 import traceback
 
 from mercurial.node import hex
 from mercurial.i18n import _
 from mercurial import (
     changegroup,
     changelog,
     cmdutil,
     commands,
     configitems,
     context,
     copies,
     debugcommands as hgdebugcommands,
     dispatch,
     error,
     exchange,
     extensions,
     hg,
     localrepo,
     match,
     merge,
     node as nodemod,
     patch,
     pycompat,
     registrar,
     repair,
     repoview,
     revset,
     scmutil,
     smartset,
     streamclone,
     util,
 )
 from . import (
     constants,
     debugcommands,
     fileserverclient,
     remotefilectx,
     remotefilelog,
     remotefilelogserver,
     repack as repackmod,
     shallowbundle,
     shallowrepo,
     shallowstore,
     shallowutil,
     shallowverifier,
 )
 
 # ensures debug commands are registered
 hgdebugcommands.command
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 configitem('remotefilelog', 'debug', default=False)
 
 configitem('remotefilelog', 'reponame', default='')
 configitem('remotefilelog', 'cachepath', default=None)
 configitem('remotefilelog', 'cachegroup', default=None)
 configitem('remotefilelog', 'cacheprocess', default=None)
 configitem('remotefilelog', 'cacheprocess.includepath', default=None)
 configitem("remotefilelog", "cachelimit", default="1000 GB")
 
 configitem('remotefilelog', 'fallbackpath', default=configitems.dynamicdefault,
            alias=[('remotefilelog', 'fallbackrepo')])
 
 configitem('remotefilelog', 'validatecachelog', default=None)
 configitem('remotefilelog', 'validatecache', default='on')
 configitem('remotefilelog', 'server', default=None)
 configitem('remotefilelog', 'servercachepath', default=None)
 configitem("remotefilelog", "serverexpiration", default=30)
 configitem('remotefilelog', 'backgroundrepack', default=False)
 configitem('remotefilelog', 'bgprefetchrevs', default=None)
 configitem('remotefilelog', 'pullprefetch', default=None)
 configitem('remotefilelog', 'backgroundprefetch', default=False)
 configitem('remotefilelog', 'prefetchdelay', default=120)
 configitem('remotefilelog', 'prefetchdays', default=14)
 
 configitem('remotefilelog', 'getfilesstep', default=10000)
 configitem('remotefilelog', 'getfilestype', default='optimistic')
 configitem('remotefilelog', 'batchsize', configitems.dynamicdefault)
 configitem('remotefilelog', 'fetchwarning', default='')
 
 configitem('remotefilelog', 'includepattern', default=None)
 configitem('remotefilelog', 'excludepattern', default=None)
 
 configitem('remotefilelog', 'gcrepack', default=False)
 configitem('remotefilelog', 'repackonhggc', default=False)
 configitem('repack', 'chainorphansbysize', default=True, experimental=True)
 
 configitem('packs', 'maxpacksize', default=0)
 configitem('packs', 'maxchainlen', default=1000)
 
+configitem('devel', 'remotefilelog.ensurestart', default=False)
+
 #  default TTL limit is 30 days
 _defaultlimit = 60 * 60 * 24 * 30
 configitem('remotefilelog', 'nodettl', default=_defaultlimit)
 
 configitem('remotefilelog', 'data.gencountlimit', default=2),
 configitem('remotefilelog', 'data.generations',
            default=['1GB', '100MB', '1MB'])
 configitem('remotefilelog', 'data.maxrepackpacks', default=50)
 configitem('remotefilelog', 'data.repackmaxpacksize', default='4GB')
 configitem('remotefilelog', 'data.repacksizelimit', default='100MB')
 
 configitem('remotefilelog', 'history.gencountlimit', default=2),
 configitem('remotefilelog', 'history.generations', default=['100MB'])
 configitem('remotefilelog', 'history.maxrepackpacks', default=50)
 configitem('remotefilelog', 'history.repackmaxpacksize', default='400MB')
 configitem('remotefilelog', 'history.repacksizelimit', default='100MB')
 
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
 testedwith = 'ships-with-hg-core'
 
 repoclass = localrepo.localrepository
 repoclass._basesupported.add(constants.SHALLOWREPO_REQUIREMENT)
 
 isenabled = shallowutil.isenabled
 
 def uisetup(ui):
     """Wraps user facing Mercurial commands to swap them out with shallow
     versions.
     """
     hg.wirepeersetupfuncs.append(fileserverclient.peersetup)
 
     entry = extensions.wrapcommand(commands.table, 'clone', cloneshallow)
     entry[1].append(('', 'shallow', None,
                      _("create a shallow clone which uses remote file "
                        "history")))
 
     extensions.wrapcommand(commands.table, 'debugindex',
         debugcommands.debugindex)
     extensions.wrapcommand(commands.table, 'debugindexdot',
         debugcommands.debugindexdot)
     extensions.wrapcommand(commands.table, 'log', log)
     extensions.wrapcommand(commands.table, 'pull', pull)
 
     # Prevent 'hg manifest --all'
     def _manifest(orig, ui, repo, *args, **opts):
         if (isenabled(repo) and opts.get(r'all')):
             raise error.Abort(_("--all is not supported in a shallow repo"))
 
         return orig(ui, repo, *args, **opts)
     extensions.wrapcommand(commands.table, "manifest", _manifest)
 
     # Wrap remotefilelog with lfs code
     def _lfsloaded(loaded=False):
         lfsmod = None
         try:
             lfsmod = extensions.find('lfs')
         except KeyError:
             pass
         if lfsmod:
             lfsmod.wrapfilelog(remotefilelog.remotefilelog)
             fileserverclient._lfsmod = lfsmod
     extensions.afterloaded('lfs', _lfsloaded)
 
     # debugdata needs remotefilelog.len to work
     extensions.wrapcommand(commands.table, 'debugdata', debugdatashallow)
 
     changegroup.cgpacker = shallowbundle.shallowcg1packer
 
     extensions.wrapfunction(changegroup, '_addchangegroupfiles',
                             shallowbundle.addchangegroupfiles)
     extensions.wrapfunction(
         changegroup, 'makechangegroup', shallowbundle.makechangegroup)
     extensions.wrapfunction(localrepo, 'makestore', storewrapper)
     extensions.wrapfunction(exchange, 'pull', exchangepull)
     extensions.wrapfunction(merge, 'applyupdates', applyupdates)
     extensions.wrapfunction(merge, '_checkunknownfiles', checkunknownfiles)
     extensions.wrapfunction(context.workingctx, '_checklookup', checklookup)
     extensions.wrapfunction(scmutil, '_findrenames', findrenames)
     extensions.wrapfunction(copies, '_computeforwardmissing',
                             computeforwardmissing)
     extensions.wrapfunction(dispatch, 'runcommand', runcommand)
     extensions.wrapfunction(repair, '_collectbrokencsets', _collectbrokencsets)
     extensions.wrapfunction(context.changectx, 'filectx', filectx)
     extensions.wrapfunction(context.workingctx, 'filectx', workingfilectx)
     extensions.wrapfunction(patch, 'trydiff', trydiff)
     extensions.wrapfunction(hg, 'verify', _verify)
     scmutil.fileprefetchhooks.add('remotefilelog', _fileprefetchhook)
 
     # disappointing hacks below
     extensions.wrapfunction(scmutil, 'getrenamedfn', getrenamedfn)
     extensions.wrapfunction(revset, 'filelog', filelogrevset)
     revset.symbols['filelog'] = revset.filelog
     extensions.wrapfunction(cmdutil, 'walkfilerevs', walkfilerevs)
 
 
 def cloneshallow(orig, ui, repo, *args, **opts):
     if opts.get(r'shallow'):
         repos = []
         def pull_shallow(orig, self, *args, **kwargs):
             if not isenabled(self):
                 repos.append(self.unfiltered())
                 # set up the client hooks so the post-clone update works
                 setupclient(self.ui, self.unfiltered())
 
                 # setupclient fixed the class on the repo itself
                 # but we also need to fix it on the repoview
                 if isinstance(self, repoview.repoview):
                     self.__class__.__bases__ = (self.__class__.__bases__[0],
                                                 self.unfiltered().__class__)
                 self.requirements.add(constants.SHALLOWREPO_REQUIREMENT)
                 self._writerequirements()
 
                 # Since setupclient hadn't been called, exchange.pull was not
                 # wrapped. So we need to manually invoke our version of it.
                 return exchangepull(orig, self, *args, **kwargs)
             else:
                 return orig(self, *args, **kwargs)
         extensions.wrapfunction(exchange, 'pull', pull_shallow)
 
         # Wrap the stream logic to add requirements and to pass include/exclude
         # patterns around.
         def setup_streamout(repo, remote):
             # Replace remote.stream_out with a version that sends file
             # patterns.
             def stream_out_shallow(orig):
                 caps = remote.capabilities()
                 if constants.NETWORK_CAP_LEGACY_SSH_GETFILES in caps:
                     opts = {}
                     if repo.includepattern:
                         opts[r'includepattern'] = '\0'.join(repo.includepattern)
                     if repo.excludepattern:
                         opts[r'excludepattern'] = '\0'.join(repo.excludepattern)
                     return remote._callstream('stream_out_shallow', **opts)
                 else:
                     return orig()
             extensions.wrapfunction(remote, 'stream_out', stream_out_shallow)
         def stream_wrap(orig, op):
             setup_streamout(op.repo, op.remote)
             return orig(op)
         extensions.wrapfunction(
             streamclone, 'maybeperformlegacystreamclone', stream_wrap)
 
         def canperformstreamclone(orig, pullop, bundle2=False):
             # remotefilelog is currently incompatible with the
             # bundle2 flavor of streamclones, so force us to use
             # v1 instead.
             if 'v2' in pullop.remotebundle2caps.get('stream', []):
                 pullop.remotebundle2caps['stream'] = [
                     c for c in pullop.remotebundle2caps['stream']
                     if c != 'v2']
             if bundle2:
                 return False, None
             supported, requirements = orig(pullop, bundle2=bundle2)
             if requirements is not None:
                 requirements.add(constants.SHALLOWREPO_REQUIREMENT)
             return supported, requirements
         extensions.wrapfunction(
             streamclone, 'canperformstreamclone', canperformstreamclone)
 
     try:
         orig(ui, repo, *args, **opts)
     finally:
         if opts.get(r'shallow'):
             for r in repos:
                 if util.safehasattr(r, 'fileservice'):
                     r.fileservice.close()
 
 def debugdatashallow(orig, *args, **kwds):
     oldlen = remotefilelog.remotefilelog.__len__
     try:
         remotefilelog.remotefilelog.__len__ = lambda x: 1
         return orig(*args, **kwds)
     finally:
         remotefilelog.remotefilelog.__len__ = oldlen
 
 def reposetup(ui, repo):
     if not repo.local():
         return
 
     # put here intentionally bc doesnt work in uisetup
     ui.setconfig('hooks', 'update.prefetch', wcpprefetch)
     ui.setconfig('hooks', 'commit.prefetch', wcpprefetch)
 
     isserverenabled = ui.configbool('remotefilelog', 'server')
     isshallowclient = isenabled(repo)
 
     if isserverenabled and isshallowclient:
         raise RuntimeError("Cannot be both a server and shallow client.")
 
     if isshallowclient:
         setupclient(ui, repo)
 
     if isserverenabled:
         remotefilelogserver.setupserver(ui, repo)
 
 def setupclient(ui, repo):
     if not isinstance(repo, localrepo.localrepository):
         return
 
     # Even clients get the server setup since they need to have the
     # wireprotocol endpoints registered.
     remotefilelogserver.onetimesetup(ui)
     onetimeclientsetup(ui)
 
     shallowrepo.wraprepo(repo)
     repo.store = shallowstore.wrapstore(repo.store)
 
 def storewrapper(orig, requirements, path, vfstype):
     s = orig(requirements, path, vfstype)
     if constants.SHALLOWREPO_REQUIREMENT in requirements:
         s = shallowstore.wrapstore(s)
 
     return s
 
 # prefetch files before update
 def applyupdates(orig, repo, actions, wctx, mctx, overwrite, wantfiledata,
                  labels=None):
     if isenabled(repo):
         manifest = mctx.manifest()
         files = []
         for f, args, msg in actions['g']:
             files.append((f, hex(manifest[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, actions, wctx, mctx, overwrite, wantfiledata,
                 labels=labels)
 
 # Prefetch merge checkunknownfiles
 def checkunknownfiles(orig, repo, wctx, mctx, force, actions,
     *args, **kwargs):
     if isenabled(repo):
         files = []
         sparsematch = repo.maybesparsematch(mctx.rev())
         for f, (m, actionargs, msg) in actions.iteritems():
             if sparsematch and not sparsematch(f):
                 continue
             if m in ('c', 'dc', 'cm'):
                 files.append((f, hex(mctx.filenode(f))))
             elif m == 'dg':
                 f2 = actionargs[0]
                 files.append((f2, hex(mctx.filenode(f2))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, wctx, mctx, force, actions, *args, **kwargs)
 
 # Prefetch files before status attempts to look at their size and contents
 def checklookup(orig, self, files):
     repo = self._repo
     if isenabled(repo):
         prefetchfiles = []
         for parent in self._parents:
             for f in files:
                 if f in parent:
                     prefetchfiles.append((f, hex(parent.filenode(f))))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(prefetchfiles)
     return orig(self, files)
 
 # Prefetch the logic that compares added and removed files for renames
 def findrenames(orig, repo, matcher, added, removed, *args, **kwargs):
     if isenabled(repo):
         files = []
         pmf = repo['.'].manifest()
         for f in removed:
             if f in pmf:
                 files.append((f, hex(pmf[f])))
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return orig(repo, matcher, added, removed, *args, **kwargs)
 
 # prefetch files before pathcopies check
 def computeforwardmissing(orig, a, b, match=None):
     missing = orig(a, b, match=match)
     repo = a._repo
     if isenabled(repo):
         mb = b.manifest()
 
         files = []
         sparsematch = repo.maybesparsematch(b.rev())
         if sparsematch:
             sparsemissing = set()
             for f in missing:
                 if sparsematch(f):
                     files.append((f, hex(mb[f])))
                     sparsemissing.add(f)
             missing = sparsemissing
 
         # batch fetch the needed files from the server
         repo.fileservice.prefetch(files)
     return missing
 
 # close cache miss server connection after the command has finished
 def runcommand(orig, lui, repo, *args, **kwargs):
     fileservice = None
     # repo can be None when running in chg:
     # - at startup, reposetup was called because serve is not norepo
     # - a norepo command like "help" is called
     if repo and isenabled(repo):
         fileservice = repo.fileservice
     try:
         return orig(lui, repo, *args, **kwargs)
     finally:
         if fileservice:
             fileservice.close()
 
 # prevent strip from stripping remotefilelogs
 def _collectbrokencsets(orig, repo, files, striprev):
     if isenabled(repo):
         files = list([f for f in files if not repo.shallowmatch(f)])
     return orig(repo, files, striprev)
 
 # changectx wrappers
 def filectx(orig, self, path, fileid=None, filelog=None):
     if fileid is None:
         fileid = self.filenode(path)
     if (isenabled(self._repo) and self._repo.shallowmatch(path)):
         return remotefilectx.remotefilectx(self._repo, path, fileid=fileid,
                                            changectx=self, filelog=filelog)
     return orig(self, path, fileid=fileid, filelog=filelog)
 
 def workingfilectx(orig, self, path, filelog=None):
     if (isenabled(self._repo) and self._repo.shallowmatch(path)):
         return remotefilectx.remoteworkingfilectx(self._repo, path,
                                                   workingctx=self,
                                                   filelog=filelog)
     return orig(self, path, filelog=filelog)
 
 # prefetch required revisions before a diff
 def trydiff(orig, repo, revs, ctx1, ctx2, modified, added, removed,
     copy, getfilectx, *args, **kwargs):
     if isenabled(repo):
         prefetch = []
         mf1 = ctx1.manifest()
         for fname in modified + added + removed:
             if fname in mf1:
                 fnode = getfilectx(fname, ctx1).filenode()
                 # fnode can be None if it's a edited working ctx file
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
             if fname not in removed:
                 fnode = getfilectx(fname, ctx2).filenode()
                 if fnode:
                     prefetch.append((fname, hex(fnode)))
 
         repo.fileservice.prefetch(prefetch)
 
     return orig(repo, revs, ctx1, ctx2, modified, added, removed, copy,
                 getfilectx, *args, **kwargs)
 
 # Prevent verify from processing files
 # a stub for mercurial.hg.verify()
 def _verify(orig, repo, level=None):
     lock = repo.lock()
     try:
         return shallowverifier.shallowverifier(repo).verify()
     finally:
         lock.release()
 
 
 clientonetime = False
 def onetimeclientsetup(ui):
     global clientonetime
     if clientonetime:
         return
     clientonetime = True
 
     # Don't commit filelogs until we know the commit hash, since the hash
     # is present in the filelog blob.
     # This violates Mercurial's filelog->manifest->changelog write order,
     # but is generally fine for client repos.
     pendingfilecommits = []
     def addrawrevision(orig, self, rawtext, transaction, link, p1, p2, node,
                        flags, cachedelta=None, _metatuple=None):
         if isinstance(link, int):
             pendingfilecommits.append(
                 (self, rawtext, transaction, link, p1, p2, node, flags,
                  cachedelta, _metatuple))
             return node
         else:
             return orig(self, rawtext, transaction, link, p1, p2, node, flags,
                         cachedelta, _metatuple=_metatuple)
     extensions.wrapfunction(
         remotefilelog.remotefilelog, 'addrawrevision', addrawrevision)
 
     def changelogadd(orig, self, *args):
         oldlen = len(self)
         node = orig(self, *args)
         newlen = len(self)
         if oldlen != newlen:
             for oldargs in pendingfilecommits:
                 log, rt, tr, link, p1, p2, n, fl, c, m = oldargs
                 linknode = self.node(link)
                 if linknode == node:
                     log.addrawrevision(rt, tr, linknode, p1, p2, n, fl, c, m)
                 else:
                     raise error.ProgrammingError(
                         'pending multiple integer revisions are not supported')
         else:
             # "link" is actually wrong here (it is set to len(changelog))
             # if changelog remains unchanged, skip writing file revisions
             # but still do a sanity check about pending multiple revisions
             if len(set(x[3] for x in pendingfilecommits)) > 1:
                 raise error.ProgrammingError(
                     'pending multiple integer revisions are not supported')
         del pendingfilecommits[:]
         return node
     extensions.wrapfunction(changelog.changelog, 'add', changelogadd)
 
 def getrenamedfn(orig, repo, endrev=None):
     if not isenabled(repo) or copies.usechangesetcentricalgo(repo):
         return orig(repo, endrev)
 
     rcache = {}
 
     def getrenamed(fn, rev):
         '''looks up all renames for a file (up to endrev) the first
         time the file is given. It indexes on the changerev and only
         parses the manifest if linkrev != changerev.
         Returns rename info for fn at changerev rev.'''
         if rev in rcache.setdefault(fn, {}):
             return rcache[fn][rev]
 
         try:
             fctx = repo[rev].filectx(fn)
             for ancestor in fctx.ancestors():
                 if ancestor.path() == fn:
                     renamed = ancestor.renamed()
                     rcache[fn][ancestor.rev()] = renamed and renamed[0]
 
             renamed = fctx.renamed()
             return renamed and renamed[0]
         except error.LookupError:
             return None
 
     return getrenamed
 
 def walkfilerevs(orig, repo, match, follow, revs, fncache):
     if not isenabled(repo):
         return orig(repo, match, follow, revs, fncache)
 
     # remotefilelog's can't be walked in rev order, so throw.
     # The caller will see the exception and walk the commit tree instead.
     if not follow:
         raise cmdutil.FileWalkError("Cannot walk via filelog")
 
     wanted = set()
     minrev, maxrev = min(revs), max(revs)
 
     pctx = repo['.']
     for filename in match.files():
         if filename not in pctx:
             raise error.Abort(_('cannot follow file not in parent '
                                'revision: "%s"') % filename)
         fctx = pctx[filename]
 
         linkrev = fctx.linkrev()
         if linkrev >= minrev and linkrev <= maxrev:
             fncache.setdefault(linkrev, []).append(filename)
             wanted.add(linkrev)
 
         for ancestor in fctx.ancestors():
             linkrev = ancestor.linkrev()
             if linkrev >= minrev and linkrev <= maxrev:
                 fncache.setdefault(linkrev, []).append(ancestor.path())
                 wanted.add(linkrev)
 
     return wanted
 
 def filelogrevset(orig, repo, subset, x):
     """``filelog(pattern)``
     Changesets connected to the specified filelog.
 
     For performance reasons, ``filelog()`` does not show every changeset
     that affects the requested file(s). See :hg:`help log` for details. For
     a slower, more accurate result, use ``file()``.
     """
 
     if not isenabled(repo):
         return orig(repo, subset, x)
 
     # i18n: "filelog" is a keyword
     pat = revset.getstring(x, _("filelog requires a pattern"))
     m = match.match(repo.root, repo.getcwd(), [pat], default='relpath',
                        ctx=repo[None])
     s = set()
 
     if not match.patkind(pat):
         # slow
         for r in subset:
             ctx = repo[r]
             cfiles = ctx.files()
             for f in m.files():
                 if f in cfiles:
                     s.add(ctx.rev())
                     break
     else:
         # partial
         files = (f for f in repo[None] if m(f))
         for f in files:
             fctx = repo[None].filectx(f)
             s.add(fctx.linkrev())
             for actx in fctx.ancestors():
                 s.add(actx.linkrev())
 
     return smartset.baseset([r for r in subset if r in s])
 
 @command('gc', [], _('hg gc [REPO...]'), norepo=True)
 def gc(ui, *args, **opts):
     '''garbage collect the client and server filelog caches
     '''
     cachepaths = set()
 
     # get the system client cache
     systemcache = shallowutil.getcachepath(ui, allowempty=True)
     if systemcache:
         cachepaths.add(systemcache)
 
     # get repo client and server cache
     repopaths = []
     pwd = ui.environ.get('PWD')
     if pwd:
         repopaths.append(pwd)
 
     repopaths.extend(args)
     repos = []
     for repopath in repopaths:
         try:
             repo = hg.peer(ui, {}, repopath)
             repos.append(repo)
 
             repocache = shallowutil.getcachepath(repo.ui, allowempty=True)
             if repocache:
                 cachepaths.add(repocache)
         except error.RepoError:
             pass
 
     # gc client cache
     for cachepath in cachepaths:
         gcclient(ui, cachepath)
 
     # gc server cache
     for repo in repos:
         remotefilelogserver.gcserver(ui, repo._repo)
 
 def gcclient(ui, cachepath):
     # get list of repos that use this cache
     repospath = os.path.join(cachepath, 'repos')
     if not os.path.exists(repospath):
         ui.warn(_("no known cache at %s\n") % cachepath)
         return
 
     reposfile = open(repospath, 'rb')
     repos = {r[:-1] for r in reposfile.readlines()}
     reposfile.close()
 
     # build list of useful files
     validrepos = []
     keepkeys = set()
 
     sharedcache = None
     filesrepacked = False
 
     count = 0
     progress = ui.makeprogress(_("analyzing repositories"), unit="repos",
                                total=len(repos))
     for path in repos:
         progress.update(count)
         count += 1
         try:
             path = ui.expandpath(os.path.normpath(path))
         except TypeError as e:
             ui.warn(_("warning: malformed path: %r:%s\n") % (path, e))
             traceback.print_exc()
             continue
         try:
             peer = hg.peer(ui, {}, path)
             repo = peer._repo
         except error.RepoError:
             continue
 
         validrepos.append(path)
 
         # Protect against any repo or config changes that have happened since
         # this repo was added to the repos file. We'd rather this loop succeed
         # and too much be deleted, than the loop fail and nothing gets deleted.
         if not isenabled(repo):
             continue
 
         if not util.safehasattr(repo, 'name'):
             ui.warn(_("repo %s is a misconfigured remotefilelog repo\n") % path)
             continue
 
         # If garbage collection on repack and repack on hg gc are enabled
         # then loose files are repacked and garbage collected.
         # Otherwise regular garbage collection is performed.
         repackonhggc = repo.ui.configbool('remotefilelog', 'repackonhggc')
         gcrepack = repo.ui.configbool('remotefilelog', 'gcrepack')
         if repackonhggc and gcrepack:
             try:
                 repackmod.incrementalrepack(repo)
                 filesrepacked = True
                 continue
             except (IOError, repackmod.RepackAlreadyRunning):
                 # If repack cannot be performed due to not enough disk space
                 # continue doing garbage collection of loose files w/o repack
                 pass
 
         reponame = repo.name
         if not sharedcache:
             sharedcache = repo.sharedstore
 
         # Compute a keepset which is not garbage collected
         def keyfn(fname, fnode):
             return fileserverclient.getcachekey(reponame, fname, hex(fnode))
         keepkeys = repackmod.keepset(repo, keyfn=keyfn, lastkeepkeys=keepkeys)
 
     progress.complete()
 
     # write list of valid repos back
     oldumask = os.umask(0o002)
     try:
         reposfile = open(repospath, 'wb')
         reposfile.writelines([("%s\n" % r) for r in validrepos])
         reposfile.close()
     finally:
         os.umask(oldumask)
 
     # prune cache
     if sharedcache is not None:
         sharedcache.gc(keepkeys)
     elif not filesrepacked:
         ui.warn(_("warning: no valid repos in repofile\n"))
 
 def log(orig, ui, repo, *pats, **opts):
     if not isenabled(repo):
         return orig(ui, repo, *pats, **opts)
 
     follow = opts.get(r'follow')
     revs = opts.get(r'rev')
     if pats:
         # Force slowpath for non-follow patterns and follows that start from
         # non-working-copy-parent revs.
         if not follow or revs:
             # This forces the slowpath
             opts[r'removed'] = True
 
         # If this is a non-follow log without any revs specified, recommend that
         # the user add -f to speed it up.
         if not follow and not revs:
             match = scmutil.match(repo['.'], pats, pycompat.byteskwargs(opts))
             isfile = not match.anypats()
             if isfile:
                 for file in match.files():
                     if not os.path.isfile(repo.wjoin(file)):
                         isfile = False
                         break
 
             if isfile:
                 ui.warn(_("warning: file log can be slow on large repos - " +
                           "use -f to speed it up\n"))
 
     return orig(ui, repo, *pats, **opts)
 
 def revdatelimit(ui, revset):
     """Update revset so that only changesets no older than 'prefetchdays' days
     are included. The default value is set to 14 days. If 'prefetchdays' is set
     to zero or negative value then date restriction is not applied.
     """
     days = ui.configint('remotefilelog', 'prefetchdays')
     if days > 0:
         revset = '(%s) & date(-%s)' % (revset, days)
     return revset
 
 def readytofetch(repo):
     """Check that enough time has passed since the last background prefetch.
     This only relates to prefetches after operations that change the working
     copy parent. Default delay between background prefetches is 2 minutes.
     """
     timeout = repo.ui.configint('remotefilelog', 'prefetchdelay')
     fname = repo.vfs.join('lastprefetch')
 
     ready = False
     with open(fname, 'a'):
         # the with construct above is used to avoid race conditions
         modtime = os.path.getmtime(fname)
         if (time.time() - modtime) > timeout:
             os.utime(fname, None)
             ready = True
 
     return ready
 
 def wcpprefetch(ui, repo, **kwargs):
     """Prefetches in background revisions specified by bgprefetchrevs revset.
     Does background repack if backgroundrepack flag is set in config.
     """
     shallow = isenabled(repo)
     bgprefetchrevs = ui.config('remotefilelog', 'bgprefetchrevs')
     isready = readytofetch(repo)
 
     if not (shallow and bgprefetchrevs and isready):
         return
 
     bgrepack = repo.ui.configbool('remotefilelog', 'backgroundrepack')
     # update a revset with a date limit
     bgprefetchrevs = revdatelimit(ui, bgprefetchrevs)
 
     def anon():
         if util.safehasattr(repo, 'ranprefetch') and repo.ranprefetch:
             return
         repo.ranprefetch = True
         repo.backgroundprefetch(bgprefetchrevs, repack=bgrepack)
 
     repo._afterlock(anon)
 
 def pull(orig, ui, repo, *pats, **opts):
     result = orig(ui, repo, *pats, **opts)
 
     if isenabled(repo):
         # prefetch if it's configured
         prefetchrevset = ui.config('remotefilelog', 'pullprefetch')
         bgrepack = repo.ui.configbool('remotefilelog', 'backgroundrepack')
         bgprefetch = repo.ui.configbool('remotefilelog', 'backgroundprefetch')
+        ensurestart = repo.ui.configbool('devel', 'remotefilelog.ensurestart')
 
         if prefetchrevset:
             ui.status(_("prefetching file contents\n"))
             revs = scmutil.revrange(repo, [prefetchrevset])
             base = repo['.'].rev()
             if bgprefetch:
-                repo.backgroundprefetch(prefetchrevset, repack=bgrepack)
+                repo.backgroundprefetch(prefetchrevset, repack=bgrepack,
+                                        ensurestart=ensurestart)
             else:
                 repo.prefetch(revs, base=base)
                 if bgrepack:
-                    repackmod.backgroundrepack(repo, incremental=True)
+                    repackmod.backgroundrepack(repo, incremental=True,
+                                               ensurestart=ensurestart)
         elif bgrepack:
-            repackmod.backgroundrepack(repo, incremental=True)
+            repackmod.backgroundrepack(repo, incremental=True,
+                                       ensurestart=ensurestart)
 
     return result
 
 def exchangepull(orig, repo, remote, *args, **kwargs):
     # Hook into the callstream/getbundle to insert bundle capabilities
     # during a pull.
     def localgetbundle(orig, source, heads=None, common=None, bundlecaps=None,
                        **kwargs):
         if not bundlecaps:
             bundlecaps = set()
         bundlecaps.add(constants.BUNDLE2_CAPABLITY)
         return orig(source, heads=heads, common=common, bundlecaps=bundlecaps,
                     **kwargs)
 
     if util.safehasattr(remote, '_callstream'):
         remote._localrepo = repo
     elif util.safehasattr(remote, 'getbundle'):
         extensions.wrapfunction(remote, 'getbundle', localgetbundle)
 
     return orig(repo, remote, *args, **kwargs)
 
 def _fileprefetchhook(repo, revs, match):
     if isenabled(repo):
         allfiles = []
         for rev in revs:
             if rev == nodemod.wdirrev or rev is None:
                 continue
             ctx = repo[rev]
             mf = ctx.manifest()
             sparsematch = repo.maybesparsematch(ctx.rev())
             for path in ctx.walk(match):
                 if (not sparsematch or sparsematch(path)) and path in mf:
                     allfiles.append((path, hex(mf[path])))
         repo.fileservice.prefetch(allfiles)
 
 @command('debugremotefilelog', [
     ('d', 'decompress', None, _('decompress the filelog first')),
     ], _('hg debugremotefilelog <path>'), norepo=True)
 def debugremotefilelog(ui, path, **opts):
     return debugcommands.debugremotefilelog(ui, path, **opts)
 
 @command('verifyremotefilelog', [
     ('d', 'decompress', None, _('decompress the filelogs first')),
     ], _('hg verifyremotefilelogs <directory>'), norepo=True)
 def verifyremotefilelog(ui, path, **opts):
     return debugcommands.verifyremotefilelog(ui, path, **opts)
 
 @command('debugdatapack', [
     ('', 'long', None, _('print the long hashes')),
     ('', 'node', '', _('dump the contents of node'), 'NODE'),
     ], _('hg debugdatapack <paths>'), norepo=True)
 def debugdatapack(ui, *paths, **opts):
     return debugcommands.debugdatapack(ui, *paths, **opts)
 
 @command('debughistorypack', [
     ], _('hg debughistorypack <path>'), norepo=True)
 def debughistorypack(ui, path, **opts):
     return debugcommands.debughistorypack(ui, path)
 
 @command('debugkeepset', [
     ], _('hg debugkeepset'))
 def debugkeepset(ui, repo, **opts):
     # The command is used to measure keepset computation time
     def keyfn(fname, fnode):
         return fileserverclient.getcachekey(repo.name, fname, hex(fnode))
     repackmod.keepset(repo, keyfn)
     return
 
 @command('debugwaitonrepack', [
     ], _('hg debugwaitonrepack'))
 def debugwaitonrepack(ui, repo, **opts):
     return debugcommands.debugwaitonrepack(repo)
 
 @command('debugwaitonprefetch', [
     ], _('hg debugwaitonprefetch'))
 def debugwaitonprefetch(ui, repo, **opts):
     return debugcommands.debugwaitonprefetch(repo)
 
 def resolveprefetchopts(ui, opts):
     if not opts.get('rev'):
         revset = ['.', 'draft()']
 
         prefetchrevset = ui.config('remotefilelog', 'pullprefetch', None)
         if prefetchrevset:
             revset.append('(%s)' % prefetchrevset)
         bgprefetchrevs = ui.config('remotefilelog', 'bgprefetchrevs', None)
         if bgprefetchrevs:
             revset.append('(%s)' % bgprefetchrevs)
         revset = '+'.join(revset)
 
         # update a revset with a date limit
         revset = revdatelimit(ui, revset)
 
         opts['rev'] = [revset]
 
     if not opts.get('base'):
         opts['base'] = None
 
     return opts
 
 @command('prefetch', [
     ('r', 'rev', [], _('prefetch the specified revisions'), _('REV')),
     ('', 'repack', False, _('run repack after prefetch')),
     ('b', 'base', '', _("rev that is assumed to already be local")),
     ] + commands.walkopts, _('hg prefetch [OPTIONS] [FILE...]'))
 def prefetch(ui, repo, *pats, **opts):
     """prefetch file revisions from the server
 
     Prefetchs file revisions for the specified revs and stores them in the
     local remotefilelog cache.  If no rev is specified, the default rev is
     used which is the union of dot, draft, pullprefetch and bgprefetchrev.
     File names or patterns can be used to limit which files are downloaded.
 
     Return 0 on success.
     """
     opts = pycompat.byteskwargs(opts)
     if not isenabled(repo):
         raise error.Abort(_("repo is not shallow"))
 
     opts = resolveprefetchopts(ui, opts)
     revs = scmutil.revrange(repo, opts.get('rev'))
     repo.prefetch(revs, opts.get('base'), pats, opts)
 
+    ensurestart = repo.ui.configbool('devel', 'remotefilelog.ensurestart')
+
     # Run repack in background
     if opts.get('repack'):
-        repackmod.backgroundrepack(repo, incremental=True)
+        repackmod.backgroundrepack(repo, incremental=True,
+                                   ensurestart=ensurestart)
 
 @command('repack', [
      ('', 'background', None, _('run in a background process'), None),
      ('', 'incremental', None, _('do an incremental repack'), None),
      ('', 'packsonly', None, _('only repack packs (skip loose objects)'), None),
     ], _('hg repack [OPTIONS]'))
 def repack_(ui, repo, *pats, **opts):
     if opts.get(r'background'):
+        ensurestart = repo.ui.configbool('devel', 'remotefilelog.ensurestart')
         repackmod.backgroundrepack(repo, incremental=opts.get(r'incremental'),
-                                   packsonly=opts.get(r'packsonly', False))
+                                   packsonly=opts.get(r'packsonly', False),
+                                   ensurestart=ensurestart)
         return
 
     options = {'packsonly': opts.get(r'packsonly')}
 
     try:
         if opts.get(r'incremental'):
             repackmod.incrementalrepack(repo, options=options)
         else:
             repackmod.fullrepack(repo, options=options)
     except repackmod.RepackAlreadyRunning as ex:
         # Don't propogate the exception if the repack is already in
         # progress, since we want the command to exit 0.
         repo.ui.warn('%s\n' % ex)

hgext/remotefilelog/repack.py

@@ -1,778 +1,779
 from __future__ import absolute_import
 
 import os
 import time
 
 from mercurial.i18n import _
 from mercurial.node import (
     nullid,
     short,
 )
 from mercurial import (
     encoding,
     error,
     mdiff,
     policy,
     pycompat,
     scmutil,
     util,
     vfs,
 )
 from mercurial.utils import procutil
 from . import (
     constants,
     contentstore,
     datapack,
     extutil,
     historypack,
     metadatastore,
     shallowutil,
 )
 
 osutil = policy.importmod(r'osutil')
 
 class RepackAlreadyRunning(error.Abort):
     pass
 
-def backgroundrepack(repo, incremental=True, packsonly=False):
+def backgroundrepack(repo, incremental=True, packsonly=False,
+                     ensurestart=False):
     cmd = [procutil.hgexecutable(), '-R', repo.origroot, 'repack']
     msg = _("(running background repack)\n")
     if incremental:
         cmd.append('--incremental')
         msg = _("(running background incremental repack)\n")
     if packsonly:
         cmd.append('--packsonly')
     repo.ui.warn(msg)
     # We know this command will find a binary, so don't block on it starting.
-    procutil.runbgcommand(cmd, encoding.environ, ensurestart=False)
+    procutil.runbgcommand(cmd, encoding.environ, ensurestart=ensurestart)
 
 def fullrepack(repo, options=None):
     """If ``packsonly`` is True, stores creating only loose objects are skipped.
     """
     if util.safehasattr(repo, 'shareddatastores'):
         datasource = contentstore.unioncontentstore(
             *repo.shareddatastores)
         historysource = metadatastore.unionmetadatastore(
             *repo.sharedhistorystores,
             allowincomplete=True)
 
         packpath = shallowutil.getcachepackpath(
             repo,
             constants.FILEPACK_CATEGORY)
         _runrepack(repo, datasource, historysource, packpath,
                    constants.FILEPACK_CATEGORY, options=options)
 
     if util.safehasattr(repo.manifestlog, 'datastore'):
         localdata, shareddata = _getmanifeststores(repo)
         lpackpath, ldstores, lhstores = localdata
         spackpath, sdstores, shstores = shareddata
 
         # Repack the shared manifest store
         datasource = contentstore.unioncontentstore(*sdstores)
         historysource = metadatastore.unionmetadatastore(
                         *shstores,
                         allowincomplete=True)
         _runrepack(repo, datasource, historysource, spackpath,
                    constants.TREEPACK_CATEGORY, options=options)
 
         # Repack the local manifest store
         datasource = contentstore.unioncontentstore(
                         *ldstores,
                         allowincomplete=True)
         historysource = metadatastore.unionmetadatastore(
                         *lhstores,
                         allowincomplete=True)
         _runrepack(repo, datasource, historysource, lpackpath,
                    constants.TREEPACK_CATEGORY, options=options)
 
 def incrementalrepack(repo, options=None):
     """This repacks the repo by looking at the distribution of pack files in the
     repo and performing the most minimal repack to keep the repo in good shape.
     """
     if util.safehasattr(repo, 'shareddatastores'):
         packpath = shallowutil.getcachepackpath(
             repo,
             constants.FILEPACK_CATEGORY)
         _incrementalrepack(repo,
                            repo.shareddatastores,
                            repo.sharedhistorystores,
                            packpath,
                            constants.FILEPACK_CATEGORY,
                            options=options)
 
     if util.safehasattr(repo.manifestlog, 'datastore'):
         localdata, shareddata = _getmanifeststores(repo)
         lpackpath, ldstores, lhstores = localdata
         spackpath, sdstores, shstores = shareddata
 
         # Repack the shared manifest store
         _incrementalrepack(repo,
                            sdstores,
                            shstores,
                            spackpath,
                            constants.TREEPACK_CATEGORY,
                            options=options)
 
         # Repack the local manifest store
         _incrementalrepack(repo,
                            ldstores,
                            lhstores,
                            lpackpath,
                            constants.TREEPACK_CATEGORY,
                            allowincompletedata=True,
                            options=options)
 
 def _getmanifeststores(repo):
     shareddatastores = repo.manifestlog.shareddatastores
     localdatastores = repo.manifestlog.localdatastores
     sharedhistorystores = repo.manifestlog.sharedhistorystores
     localhistorystores = repo.manifestlog.localhistorystores
 
     sharedpackpath = shallowutil.getcachepackpath(repo,
                                             constants.TREEPACK_CATEGORY)
     localpackpath = shallowutil.getlocalpackpath(repo.svfs.vfs.base,
                                             constants.TREEPACK_CATEGORY)
 
     return ((localpackpath, localdatastores, localhistorystores),
             (sharedpackpath, shareddatastores, sharedhistorystores))
 
 def _topacks(packpath, files, constructor):
     paths = list(os.path.join(packpath, p) for p in files)
     packs = list(constructor(p) for p in paths)
     return packs
 
 def _deletebigpacks(repo, folder, files):
     """Deletes packfiles that are bigger than ``packs.maxpacksize``.
 
     Returns ``files` with the removed files omitted."""
     maxsize = repo.ui.configbytes("packs", "maxpacksize")
     if maxsize <= 0:
         return files
 
     # This only considers datapacks today, but we could broaden it to include
     # historypacks.
     VALIDEXTS = [".datapack", ".dataidx"]
 
     # Either an oversize index or datapack will trigger cleanup of the whole
     # pack:
     oversized = {os.path.splitext(path)[0] for path, ftype, stat in files
         if (stat.st_size > maxsize and (os.path.splitext(path)[1]
                                         in VALIDEXTS))}
 
     for rootfname in oversized:
         rootpath = os.path.join(folder, rootfname)
         for ext in VALIDEXTS:
             path = rootpath + ext
             repo.ui.debug('removing oversize packfile %s (%s)\n' %
                           (path, util.bytecount(os.stat(path).st_size)))
             os.unlink(path)
     return [row for row in files if os.path.basename(row[0]) not in oversized]
 
 def _incrementalrepack(repo, datastore, historystore, packpath, category,
         allowincompletedata=False, options=None):
     shallowutil.mkstickygroupdir(repo.ui, packpath)
 
     files = osutil.listdir(packpath, stat=True)
     files = _deletebigpacks(repo, packpath, files)
     datapacks = _topacks(packpath,
         _computeincrementaldatapack(repo.ui, files),
         datapack.datapack)
     datapacks.extend(s for s in datastore
                      if not isinstance(s, datapack.datapackstore))
 
     historypacks = _topacks(packpath,
         _computeincrementalhistorypack(repo.ui, files),
         historypack.historypack)
     historypacks.extend(s for s in historystore
                         if not isinstance(s, historypack.historypackstore))
 
     # ``allhistory{files,packs}`` contains all known history packs, even ones we
     # don't plan to repack. They are used during the datapack repack to ensure
     # good ordering of nodes.
     allhistoryfiles = _allpackfileswithsuffix(files, historypack.PACKSUFFIX,
                             historypack.INDEXSUFFIX)
     allhistorypacks = _topacks(packpath,
         (f for f, mode, stat in allhistoryfiles),
         historypack.historypack)
     allhistorypacks.extend(s for s in historystore
                         if not isinstance(s, historypack.historypackstore))
     _runrepack(repo,
                contentstore.unioncontentstore(
                    *datapacks,
                    allowincomplete=allowincompletedata),
                metadatastore.unionmetadatastore(
                    *historypacks,
                    allowincomplete=True),
                packpath, category,
                fullhistory=metadatastore.unionmetadatastore(
                    *allhistorypacks,
                    allowincomplete=True),
                 options=options)
 
 def _computeincrementaldatapack(ui, files):
     opts = {
         'gencountlimit' : ui.configint(
             'remotefilelog', 'data.gencountlimit'),
         'generations' : ui.configlist(
             'remotefilelog', 'data.generations'),
         'maxrepackpacks' : ui.configint(
             'remotefilelog', 'data.maxrepackpacks'),
         'repackmaxpacksize' : ui.configbytes(
             'remotefilelog', 'data.repackmaxpacksize'),
         'repacksizelimit' : ui.configbytes(
             'remotefilelog', 'data.repacksizelimit'),
     }
 
     packfiles = _allpackfileswithsuffix(
         files, datapack.PACKSUFFIX, datapack.INDEXSUFFIX)
     return _computeincrementalpack(packfiles, opts)
 
 def _computeincrementalhistorypack(ui, files):
     opts = {
         'gencountlimit' : ui.configint(
             'remotefilelog', 'history.gencountlimit'),
         'generations' : ui.configlist(
             'remotefilelog', 'history.generations', ['100MB']),
         'maxrepackpacks' : ui.configint(
             'remotefilelog', 'history.maxrepackpacks'),
         'repackmaxpacksize' : ui.configbytes(
             'remotefilelog', 'history.repackmaxpacksize', '400MB'),
         'repacksizelimit' : ui.configbytes(
             'remotefilelog', 'history.repacksizelimit'),
     }
 
     packfiles = _allpackfileswithsuffix(
         files, historypack.PACKSUFFIX, historypack.INDEXSUFFIX)
     return _computeincrementalpack(packfiles, opts)
 
 def _allpackfileswithsuffix(files, packsuffix, indexsuffix):
     result = []
     fileset = set(fn for fn, mode, stat in files)
     for filename, mode, stat in files:
         if not filename.endswith(packsuffix):
             continue
 
         prefix = filename[:-len(packsuffix)]
 
         # Don't process a pack if it doesn't have an index.
         if (prefix + indexsuffix) not in fileset:
             continue
         result.append((prefix, mode, stat))
 
     return result
 
 def _computeincrementalpack(files, opts):
     """Given a set of pack files along with the configuration options, this
     function computes the list of files that should be packed as part of an
     incremental repack.
 
     It tries to strike a balance between keeping incremental repacks cheap (i.e.
     packing small things when possible, and rolling the packs up to the big ones
     over time).
     """
 
     limits = list(sorted((util.sizetoint(s) for s in opts['generations']),
                                 reverse=True))
     limits.append(0)
 
     # Group the packs by generation (i.e. by size)
     generations = []
     for i in pycompat.xrange(len(limits)):
         generations.append([])
 
     sizes = {}
     for prefix, mode, stat in files:
         size = stat.st_size
         if size > opts['repackmaxpacksize']:
             continue
 
         sizes[prefix] = size
         for i, limit in enumerate(limits):
             if size > limit:
                 generations[i].append(prefix)
                 break
 
     # Steps for picking what packs to repack:
     # 1. Pick the largest generation with > gencountlimit pack files.
     # 2. Take the smallest three packs.
     # 3. While total-size-of-packs < repacksizelimit: add another pack
 
     # Find the largest generation with more than gencountlimit packs
     genpacks = []
     for i, limit in enumerate(limits):
         if len(generations[i]) > opts['gencountlimit']:
             # Sort to be smallest last, for easy popping later
             genpacks.extend(sorted(generations[i], reverse=True,
                                    key=lambda x: sizes[x]))
             break
 
     # Take as many packs from the generation as we can
     chosenpacks = genpacks[-3:]
     genpacks = genpacks[:-3]
     repacksize = sum(sizes[n] for n in chosenpacks)
     while (repacksize < opts['repacksizelimit'] and genpacks and
            len(chosenpacks) < opts['maxrepackpacks']):
         chosenpacks.append(genpacks.pop())
         repacksize += sizes[chosenpacks[-1]]
 
     return chosenpacks
 
 def _runrepack(repo, data, history, packpath, category, fullhistory=None,
                options=None):
     shallowutil.mkstickygroupdir(repo.ui, packpath)
 
     def isold(repo, filename, node):
         """Check if the file node is older than a limit.
         Unless a limit is specified in the config the default limit is taken.
         """
         filectx = repo.filectx(filename, fileid=node)
         filetime = repo[filectx.linkrev()].date()
 
         ttl = repo.ui.configint('remotefilelog', 'nodettl')
 
         limit = time.time() - ttl
         return filetime[0] < limit
 
     garbagecollect = repo.ui.configbool('remotefilelog', 'gcrepack')
     if not fullhistory:
         fullhistory = history
     packer = repacker(repo, data, history, fullhistory, category,
                       gc=garbagecollect, isold=isold, options=options)
 
     with datapack.mutabledatapack(repo.ui, packpath) as dpack:
         with historypack.mutablehistorypack(repo.ui, packpath) as hpack:
             try:
                 packer.run(dpack, hpack)
             except error.LockHeld:
                 raise RepackAlreadyRunning(_("skipping repack - another repack "
                                              "is already running"))
 
 def keepset(repo, keyfn, lastkeepkeys=None):
     """Computes a keepset which is not garbage collected.
     'keyfn' is a function that maps filename, node to a unique key.
     'lastkeepkeys' is an optional argument and if provided the keepset
     function updates lastkeepkeys with more keys and returns the result.
     """
     if not lastkeepkeys:
         keepkeys = set()
     else:
         keepkeys = lastkeepkeys
 
     # We want to keep:
     # 1. Working copy parent
     # 2. Draft commits
     # 3. Parents of draft commits
     # 4. Pullprefetch and bgprefetchrevs revsets if specified
     revs = ['.', 'draft()', 'parents(draft())']
     prefetchrevs = repo.ui.config('remotefilelog', 'pullprefetch', None)
     if prefetchrevs:
         revs.append('(%s)' % prefetchrevs)
     prefetchrevs = repo.ui.config('remotefilelog', 'bgprefetchrevs', None)
     if prefetchrevs:
         revs.append('(%s)' % prefetchrevs)
     revs = '+'.join(revs)
 
     revs = ['sort((%s), "topo")' % revs]
     keep = scmutil.revrange(repo, revs)
 
     processed = set()
     lastmanifest = None
 
     # process the commits in toposorted order starting from the oldest
     for r in reversed(keep._list):
         if repo[r].p1().rev() in processed:
             # if the direct parent has already been processed
             # then we only need to process the delta
             m = repo[r].manifestctx().readdelta()
         else:
             # otherwise take the manifest and diff it
             # with the previous manifest if one exists
             if lastmanifest:
                 m = repo[r].manifest().diff(lastmanifest)
             else:
                 m = repo[r].manifest()
         lastmanifest = repo[r].manifest()
         processed.add(r)
 
         # populate keepkeys with keys from the current manifest
         if type(m) is dict:
             # m is a result of diff of two manifests and is a dictionary that
             # maps filename to ((newnode, newflag), (oldnode, oldflag)) tuple
             for filename, diff in m.iteritems():
                 if diff[0][0] is not None:
                     keepkeys.add(keyfn(filename, diff[0][0]))
         else:
             # m is a manifest object
             for filename, filenode in m.iteritems():
                 keepkeys.add(keyfn(filename, filenode))
 
     return keepkeys
 
 class repacker(object):
     """Class for orchestrating the repack of data and history information into a
     new format.
     """
     def __init__(self, repo, data, history, fullhistory, category, gc=False,
                  isold=None, options=None):
         self.repo = repo
         self.data = data
         self.history = history
         self.fullhistory = fullhistory
         self.unit = constants.getunits(category)
         self.garbagecollect = gc
         self.options = options
         if self.garbagecollect:
             if not isold:
                 raise ValueError("Function 'isold' is not properly specified")
             # use (filename, node) tuple as a keepset key
             self.keepkeys = keepset(repo, lambda f, n : (f, n))
             self.isold = isold
 
     def run(self, targetdata, targethistory):
         ledger = repackledger()
 
         with extutil.flock(repacklockvfs(self.repo).join("repacklock"),
                            _('repacking %s') % self.repo.origroot, timeout=0):
             self.repo.hook('prerepack')
 
             # Populate ledger from source
             self.data.markledger(ledger, options=self.options)
             self.history.markledger(ledger, options=self.options)
 
             # Run repack
             self.repackdata(ledger, targetdata)
             self.repackhistory(ledger, targethistory)
 
             # Call cleanup on each source
             for source in ledger.sources:
                 source.cleanup(ledger)
 
     def _chainorphans(self, ui, filename, nodes, orphans, deltabases):
         """Reorderes ``orphans`` into a single chain inside ``nodes`` and
         ``deltabases``.
 
         We often have orphan entries (nodes without a base that aren't
         referenced by other nodes -- i.e., part of a chain) due to gaps in
         history. Rather than store them as individual fulltexts, we prefer to
         insert them as one chain sorted by size.
         """
         if not orphans:
             return nodes
 
         def getsize(node, default=0):
             meta = self.data.getmeta(filename, node)
             if constants.METAKEYSIZE in meta:
                 return meta[constants.METAKEYSIZE]
             else:
                 return default
 
         # Sort orphans by size; biggest first is preferred, since it's more
         # likely to be the newest version assuming files grow over time.
         # (Sort by node first to ensure the sort is stable.)
         orphans = sorted(orphans)
         orphans = list(sorted(orphans, key=getsize, reverse=True))
         if ui.debugflag:
             ui.debug("%s: orphan chain: %s\n" % (filename,
                 ", ".join([short(s) for s in orphans])))
 
         # Create one contiguous chain and reassign deltabases.
         for i, node in enumerate(orphans):
             if i == 0:
                 deltabases[node] = (nullid, 0)
             else:
                 parent = orphans[i - 1]
                 deltabases[node] = (parent, deltabases[parent][1] + 1)
         nodes = [n for n in nodes if n not in orphans]
         nodes += orphans
         return nodes
 
     def repackdata(self, ledger, target):
         ui = self.repo.ui
         maxchainlen = ui.configint('packs', 'maxchainlen', 1000)
 
         byfile = {}
         for entry in ledger.entries.itervalues():
             if entry.datasource:
                 byfile.setdefault(entry.filename, {})[entry.node] = entry
 
         count = 0
         repackprogress = ui.makeprogress(_("repacking data"), unit=self.unit,
                                             total=len(byfile))
         for filename, entries in sorted(byfile.iteritems()):
             repackprogress.update(count)
 
             ancestors = {}
             nodes = list(node for node in entries)
             nohistory = []
             buildprogress = ui.makeprogress(_("building history"), unit='nodes',
                                             total=len(nodes))
             for i, node in enumerate(nodes):
                 if node in ancestors:
                     continue
                 buildprogress.update(i)
                 try:
                     ancestors.update(self.fullhistory.getancestors(filename,
                         node, known=ancestors))
                 except KeyError:
                     # Since we're packing data entries, we may not have the
                     # corresponding history entries for them. It's not a big
                     # deal, but the entries won't be delta'd perfectly.
                     nohistory.append(node)
             buildprogress.complete()
 
             # Order the nodes children first, so we can produce reverse deltas
             orderednodes = list(reversed(self._toposort(ancestors)))
             if len(nohistory) > 0:
                 ui.debug('repackdata: %d nodes without history\n' %
                          len(nohistory))
             orderednodes.extend(sorted(nohistory))
 
             # Filter orderednodes to just the nodes we want to serialize (it
             # currently also has the edge nodes' ancestors).
             orderednodes = list(filter(lambda node: node in nodes,
                                 orderednodes))
 
             # Garbage collect old nodes:
             if self.garbagecollect:
                 neworderednodes = []
                 for node in orderednodes:
                     # If the node is old and is not in the keepset, we skip it,
                     # and mark as garbage collected
                     if ((filename, node) not in self.keepkeys and
                         self.isold(self.repo, filename, node)):
                         entries[node].gced = True
                         continue
                     neworderednodes.append(node)
                 orderednodes = neworderednodes
 
             # Compute delta bases for nodes:
             deltabases = {}
             nobase = set()
             referenced = set()
             nodes = set(nodes)
             processprogress = ui.makeprogress(_("processing nodes"),
                                               unit='nodes',
                                               total=len(orderednodes))
             for i, node in enumerate(orderednodes):
                 processprogress.update(i)
                 # Find delta base
                 # TODO: allow delta'ing against most recent descendant instead
                 # of immediate child
                 deltatuple = deltabases.get(node, None)
                 if deltatuple is None:
                     deltabase, chainlen = nullid, 0
                     deltabases[node] = (nullid, 0)
                     nobase.add(node)
                 else:
                     deltabase, chainlen = deltatuple
                     referenced.add(deltabase)
 
                 # Use available ancestor information to inform our delta choices
                 ancestorinfo = ancestors.get(node)
                 if ancestorinfo:
                     p1, p2, linknode, copyfrom = ancestorinfo
 
                     # The presence of copyfrom means we're at a point where the
                     # file was copied from elsewhere. So don't attempt to do any
                     # deltas with the other file.
                     if copyfrom:
                         p1 = nullid
 
                     if chainlen < maxchainlen:
                         # Record this child as the delta base for its parents.
                         # This may be non optimal, since the parents may have
                         # many children, and this will only choose the last one.
                         # TODO: record all children and try all deltas to find
                         # best
                         if p1 != nullid:
                             deltabases[p1] = (node, chainlen + 1)
                         if p2 != nullid:
                             deltabases[p2] = (node, chainlen + 1)
 
             # experimental config: repack.chainorphansbysize
             if ui.configbool('repack', 'chainorphansbysize'):
                 orphans = nobase - referenced
                 orderednodes = self._chainorphans(ui, filename, orderednodes,
                     orphans, deltabases)
 
             # Compute deltas and write to the pack
             for i, node in enumerate(orderednodes):
                 deltabase, chainlen = deltabases[node]
                 # Compute delta
                 # TODO: Optimize the deltachain fetching. Since we're
                 # iterating over the different version of the file, we may
                 # be fetching the same deltachain over and over again.
                 if deltabase != nullid:
                     deltaentry = self.data.getdelta(filename, node)
                     delta, deltabasename, origdeltabase, meta = deltaentry
                     size = meta.get(constants.METAKEYSIZE)
                     if (deltabasename != filename or origdeltabase != deltabase
                         or size is None):
                         deltabasetext = self.data.get(filename, deltabase)
                         original = self.data.get(filename, node)
                         size = len(original)
                         delta = mdiff.textdiff(deltabasetext, original)
                 else:
                     delta = self.data.get(filename, node)
                     size = len(delta)
                     meta = self.data.getmeta(filename, node)
 
                 # TODO: don't use the delta if it's larger than the fulltext
                 if constants.METAKEYSIZE not in meta:
                     meta[constants.METAKEYSIZE] = size
                 target.add(filename, node, deltabase, delta, meta)
 
                 entries[node].datarepacked = True
 
             processprogress.complete()
             count += 1
 
         repackprogress.complete()
         target.close(ledger=ledger)
 
     def repackhistory(self, ledger, target):
         ui = self.repo.ui
 
         byfile = {}
         for entry in ledger.entries.itervalues():
             if entry.historysource:
                 byfile.setdefault(entry.filename, {})[entry.node] = entry
 
         progress = ui.makeprogress(_("repacking history"), unit=self.unit,
                                    total=len(byfile))
         for filename, entries in sorted(byfile.iteritems()):
             ancestors = {}
             nodes = list(node for node in entries)
 
             for node in nodes:
                 if node in ancestors:
                     continue
                 ancestors.update(self.history.getancestors(filename, node,
                                                            known=ancestors))
 
             # Order the nodes children first
             orderednodes = reversed(self._toposort(ancestors))
 
             # Write to the pack
             dontprocess = set()
             for node in orderednodes:
                 p1, p2, linknode, copyfrom = ancestors[node]
 
                 # If the node is marked dontprocess, but it's also in the
                 # explicit entries set, that means the node exists both in this
                 # file and in another file that was copied to this file.
                 # Usually this happens if the file was copied to another file,
                 # then the copy was deleted, then reintroduced without copy
                 # metadata. The original add and the new add have the same hash
                 # since the content is identical and the parents are null.
                 if node in dontprocess and node not in entries:
                     # If copyfrom == filename, it means the copy history
                     # went to come other file, then came back to this one, so we
                     # should continue processing it.
                     if p1 != nullid and copyfrom != filename:
                         dontprocess.add(p1)
                     if p2 != nullid:
                         dontprocess.add(p2)
                     continue
 
                 if copyfrom:
                     dontprocess.add(p1)
 
                 target.add(filename, node, p1, p2, linknode, copyfrom)
 
                 if node in entries:
                     entries[node].historyrepacked = True
 
             progress.increment()
 
         progress.complete()
         target.close(ledger=ledger)
 
     def _toposort(self, ancestors):
         def parentfunc(node):
             p1, p2, linknode, copyfrom = ancestors[node]
             parents = []
             if p1 != nullid:
                 parents.append(p1)
             if p2 != nullid:
                 parents.append(p2)
             return parents
 
         sortednodes = shallowutil.sortnodes(ancestors.keys(), parentfunc)
         return sortednodes
 
 class repackledger(object):
     """Storage for all the bookkeeping that happens during a repack. It contains
     the list of revisions being repacked, what happened to each revision, and
     which source store contained which revision originally (for later cleanup).
     """
     def __init__(self):
         self.entries = {}
         self.sources = {}
         self.created = set()
 
     def markdataentry(self, source, filename, node):
         """Mark the given filename+node revision as having a data rev in the
         given source.
         """
         entry = self._getorcreateentry(filename, node)
         entry.datasource = True
         entries = self.sources.get(source)
         if not entries:
             entries = set()
             self.sources[source] = entries
         entries.add(entry)
 
     def markhistoryentry(self, source, filename, node):
         """Mark the given filename+node revision as having a history rev in the
         given source.
         """
         entry = self._getorcreateentry(filename, node)
         entry.historysource = True
         entries = self.sources.get(source)
         if not entries:
             entries = set()
             self.sources[source] = entries
         entries.add(entry)
 
     def _getorcreateentry(self, filename, node):
         key = (filename, node)
         value = self.entries.get(key)
         if not value:
             value = repackentry(filename, node)
             self.entries[key] = value
 
         return value
 
     def addcreated(self, value):
         self.created.add(value)
 
 class repackentry(object):
     """Simple class representing a single revision entry in the repackledger.
     """
     __slots__ = (r'filename', r'node', r'datasource', r'historysource',
                  r'datarepacked', r'historyrepacked', r'gced')
     def __init__(self, filename, node):
         self.filename = filename
         self.node = node
         # If the revision has a data entry in the source
         self.datasource = False
         # If the revision has a history entry in the source
         self.historysource = False
         # If the revision's data entry was repacked into the repack target
         self.datarepacked = False
         # If the revision's history entry was repacked into the repack target
         self.historyrepacked = False
         # If garbage collected
         self.gced = False
 
 def repacklockvfs(repo):
     if util.safehasattr(repo, 'name'):
         # Lock in the shared cache so repacks across multiple copies of the same
         # repo are coordinated.
         sharedcachepath = shallowutil.getcachepackpath(
             repo,
             constants.FILEPACK_CATEGORY)
         return vfs.vfs(sharedcachepath)
     else:
         return repo.svfs

hgext/remotefilelog/shallowrepo.py

@@ -1,301 +1,302
 # shallowrepo.py - shallow repository that uses remote filelogs
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 from __future__ import absolute_import
 
 import os
 
 from mercurial.i18n import _
 from mercurial.node import hex, nullid, nullrev
 from mercurial import (
     encoding,
     error,
     localrepo,
     match,
     scmutil,
     sparse,
     util,
 )
 from mercurial.utils import procutil
 from . import (
     connectionpool,
     constants,
     contentstore,
     datapack,
     fileserverclient,
     historypack,
     metadatastore,
     remotefilectx,
     remotefilelog,
     shallowutil,
 )
 
 # These make*stores functions are global so that other extensions can replace
 # them.
 def makelocalstores(repo):
     """In-repo stores, like .hg/store/data; can not be discarded."""
     localpath = os.path.join(repo.svfs.vfs.base, 'data')
     if not os.path.exists(localpath):
         os.makedirs(localpath)
 
     # Instantiate local data stores
     localcontent = contentstore.remotefilelogcontentstore(
         repo, localpath, repo.name, shared=False)
     localmetadata = metadatastore.remotefilelogmetadatastore(
         repo, localpath, repo.name, shared=False)
     return localcontent, localmetadata
 
 def makecachestores(repo):
     """Typically machine-wide, cache of remote data; can be discarded."""
     # Instantiate shared cache stores
     cachepath = shallowutil.getcachepath(repo.ui)
     cachecontent = contentstore.remotefilelogcontentstore(
         repo, cachepath, repo.name, shared=True)
     cachemetadata = metadatastore.remotefilelogmetadatastore(
         repo, cachepath, repo.name, shared=True)
 
     repo.sharedstore = cachecontent
     repo.shareddatastores.append(cachecontent)
     repo.sharedhistorystores.append(cachemetadata)
 
     return cachecontent, cachemetadata
 
 def makeremotestores(repo, cachecontent, cachemetadata):
     """These stores fetch data from a remote server."""
     # Instantiate remote stores
     repo.fileservice = fileserverclient.fileserverclient(repo)
     remotecontent = contentstore.remotecontentstore(
         repo.ui, repo.fileservice, cachecontent)
     remotemetadata = metadatastore.remotemetadatastore(
         repo.ui, repo.fileservice, cachemetadata)
     return remotecontent, remotemetadata
 
 def makepackstores(repo):
     """Packs are more efficient (to read from) cache stores."""
     # Instantiate pack stores
     packpath = shallowutil.getcachepackpath(repo,
                                             constants.FILEPACK_CATEGORY)
     packcontentstore = datapack.datapackstore(repo.ui, packpath)
     packmetadatastore = historypack.historypackstore(repo.ui, packpath)
 
     repo.shareddatastores.append(packcontentstore)
     repo.sharedhistorystores.append(packmetadatastore)
     shallowutil.reportpackmetrics(repo.ui, 'filestore', packcontentstore,
         packmetadatastore)
     return packcontentstore, packmetadatastore
 
 def makeunionstores(repo):
     """Union stores iterate the other stores and return the first result."""
     repo.shareddatastores = []
     repo.sharedhistorystores = []
 
     packcontentstore, packmetadatastore = makepackstores(repo)
     cachecontent, cachemetadata = makecachestores(repo)
     localcontent, localmetadata = makelocalstores(repo)
     remotecontent, remotemetadata = makeremotestores(repo, cachecontent,
                                                      cachemetadata)
 
     # Instantiate union stores
     repo.contentstore = contentstore.unioncontentstore(
         packcontentstore, cachecontent,
         localcontent, remotecontent, writestore=localcontent)
     repo.metadatastore = metadatastore.unionmetadatastore(
         packmetadatastore, cachemetadata, localmetadata, remotemetadata,
         writestore=localmetadata)
 
     fileservicedatawrite = cachecontent
     fileservicehistorywrite = cachemetadata
     repo.fileservice.setstore(repo.contentstore, repo.metadatastore,
                               fileservicedatawrite, fileservicehistorywrite)
     shallowutil.reportpackmetrics(repo.ui, 'filestore',
         packcontentstore, packmetadatastore)
 
 def wraprepo(repo):
     class shallowrepository(repo.__class__):
         @util.propertycache
         def name(self):
             return self.ui.config('remotefilelog', 'reponame')
 
         @util.propertycache
         def fallbackpath(self):
             path = repo.ui.config("remotefilelog", "fallbackpath",
                                   repo.ui.config('paths', 'default'))
             if not path:
                 raise error.Abort("no remotefilelog server "
                     "configured - is your .hg/hgrc trusted?")
 
             return path
 
         def maybesparsematch(self, *revs, **kwargs):
             '''
             A wrapper that allows the remotefilelog to invoke sparsematch() if
             this is a sparse repository, or returns None if this is not a
             sparse repository.
             '''
             if revs:
                 ret = sparse.matcher(repo, revs=revs)
             else:
                 ret = sparse.matcher(repo)
 
             if ret.always():
                 return None
             return ret
 
         def file(self, f):
             if f[0] == '/':
                 f = f[1:]
 
             if self.shallowmatch(f):
                 return remotefilelog.remotefilelog(self.svfs, f, self)
             else:
                 return super(shallowrepository, self).file(f)
 
         def filectx(self, path, *args, **kwargs):
             if self.shallowmatch(path):
                 return remotefilectx.remotefilectx(self, path, *args, **kwargs)
             else:
                 return super(shallowrepository, self).filectx(path, *args,
                                                               **kwargs)
 
         @localrepo.unfilteredmethod
         def commitctx(self, ctx, error=False, origctx=None):
             """Add a new revision to current repository.
             Revision information is passed via the context argument.
             """
 
             # some contexts already have manifest nodes, they don't need any
             # prefetching (for example if we're just editing a commit message
             # we can reuse manifest
             if not ctx.manifestnode():
                 # prefetch files that will likely be compared
                 m1 = ctx.p1().manifest()
                 files = []
                 for f in ctx.modified() + ctx.added():
                     fparent1 = m1.get(f, nullid)
                     if fparent1 != nullid:
                         files.append((f, hex(fparent1)))
                 self.fileservice.prefetch(files)
             return super(shallowrepository, self).commitctx(ctx,
                                                             error=error,
                                                             origctx=origctx)
 
         def backgroundprefetch(self, revs, base=None, repack=False, pats=None,
-                               opts=None):
+                               opts=None, ensurestart=False):
             """Runs prefetch in background with optional repack
             """
             cmd = [procutil.hgexecutable(), '-R', repo.origroot, 'prefetch']
             if repack:
                 cmd.append('--repack')
             if revs:
                 cmd += ['-r', revs]
             # We know this command will find a binary, so don't block
             # on it starting.
-            procutil.runbgcommand(cmd, encoding.environ, ensurestart=False)
+            procutil.runbgcommand(cmd, encoding.environ,
+                                  ensurestart=ensurestart)
 
         def prefetch(self, revs, base=None, pats=None, opts=None):
             """Prefetches all the necessary file revisions for the given revs
             Optionally runs repack in background
             """
             with repo._lock(repo.svfs, 'prefetchlock', True, None, None,
                             _('prefetching in %s') % repo.origroot):
                 self._prefetch(revs, base, pats, opts)
 
         def _prefetch(self, revs, base=None, pats=None, opts=None):
             fallbackpath = self.fallbackpath
             if fallbackpath:
                 # If we know a rev is on the server, we should fetch the server
                 # version of those files, since our local file versions might
                 # become obsolete if the local commits are stripped.
                 localrevs = repo.revs('outgoing(%s)', fallbackpath)
                 if base is not None and base != nullrev:
                     serverbase = list(repo.revs('first(reverse(::%s) - %ld)',
                                                 base, localrevs))
                     if serverbase:
                         base = serverbase[0]
             else:
                 localrevs = repo
 
             mfl = repo.manifestlog
             mfrevlog = mfl.getstorage('')
             if base is not None:
                 mfdict = mfl[repo[base].manifestnode()].read()
                 skip = set(mfdict.iteritems())
             else:
                 skip = set()
 
             # Copy the skip set to start large and avoid constant resizing,
             # and since it's likely to be very similar to the prefetch set.
             files = skip.copy()
             serverfiles = skip.copy()
             visited = set()
             visited.add(nullrev)
             revcount = len(revs)
             progress = self.ui.makeprogress(_('prefetching'), total=revcount)
             progress.update(0)
             for rev in sorted(revs):
                 ctx = repo[rev]
                 if pats:
                     m = scmutil.match(ctx, pats, opts)
                 sparsematch = repo.maybesparsematch(rev)
 
                 mfnode = ctx.manifestnode()
                 mfrev = mfrevlog.rev(mfnode)
 
                 # Decompressing manifests is expensive.
                 # When possible, only read the deltas.
                 p1, p2 = mfrevlog.parentrevs(mfrev)
                 if p1 in visited and p2 in visited:
                     mfdict = mfl[mfnode].readfast()
                 else:
                     mfdict = mfl[mfnode].read()
 
                 diff = mfdict.iteritems()
                 if pats:
                     diff = (pf for pf in diff if m(pf[0]))
                 if sparsematch:
                     diff = (pf for pf in diff if sparsematch(pf[0]))
                 if rev not in localrevs:
                     serverfiles.update(diff)
                 else:
                     files.update(diff)
 
                 visited.add(mfrev)
                 progress.increment()
 
             files.difference_update(skip)
             serverfiles.difference_update(skip)
             progress.complete()
 
             # Fetch files known to be on the server
             if serverfiles:
                 results = [(path, hex(fnode)) for (path, fnode) in serverfiles]
                 repo.fileservice.prefetch(results, force=True)
 
             # Fetch files that may or may not be on the server
             if files:
                 results = [(path, hex(fnode)) for (path, fnode) in files]
                 repo.fileservice.prefetch(results)
 
         def close(self):
             super(shallowrepository, self).close()
             self.connectionpool.close()
 
     repo.__class__ = shallowrepository
 
     repo.shallowmatch = match.always()
 
     makeunionstores(repo)
 
     repo.includepattern = repo.ui.configlist("remotefilelog", "includepattern",
                                              None)
     repo.excludepattern = repo.ui.configlist("remotefilelog", "excludepattern",
                                              None)
     if not util.safehasattr(repo, 'connectionpool'):
         repo.connectionpool = connectionpool.connectionpool(repo)
 
     if repo.includepattern or repo.excludepattern:
         repo.shallowmatch = match.match(repo.root, '', None,
             repo.includepattern, repo.excludepattern)

tests/test-remotefilelog-bgprefetch.t

@@ -1,375 +1,381
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
+# devel.remotefilelog.ensurestart: reduce race condition with
+# waiton{repack/prefetch}
+  $ cat >> $HGRCPATH <<EOF
+  > [devel]
+  > remotefilelog.ensurestart=True
+  > EOF
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > EOF
   $ echo x > x
   $ echo z > z
   $ hg commit -qAm x
   $ echo x2 > x
   $ echo y > y
   $ hg commit -qAm y
   $ echo w > w
   $ rm z
   $ hg commit -qAm w
   $ hg bookmark foo
 
   $ cd ..
 
 # clone the repo
 
   $ hgcloneshallow ssh://user@dummy/master shallow --noupdate
   streaming all changes
   2 files to transfer, 776 bytes of data
   transferred 776 bytes in * seconds (*/sec) (glob)
   searching for changes
   no changes found
 
 # Set the prefetchdays config to zero so that all commits are prefetched
 # no matter what their creation date is. Also set prefetchdelay config
 # to zero so that there is no delay between prefetches.
   $ cd shallow
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > prefetchdays=0
   > prefetchdelay=0
   > EOF
   $ cd ..
 
 # prefetch a revision
   $ cd shallow
 
   $ hg prefetch -r 0
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ hg cat -r 0 x
   x
 
 # background prefetch on pull when configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > pullprefetch=bookmark()
   > backgroundprefetch=True
   > EOF
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 0 changes to 0 files
   updating bookmark foo
   new changesets 6b4b6f66ef8c
   (run 'hg update' to get a working copy)
   prefetching file contents
   $ sleep 0.5
   $ hg debugwaitonprefetch >/dev/null 2>%1
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/ef95c5376f34698742fe34f315fd82136f8f68c0
   $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca
   $TESTTMP/hgcache/master/af/f024fe4ab0fece4091de044c58c9ae4233383a/bb6ccd5dceaa5e9dc220e0dad65e051b94f69a2c
   $TESTTMP/hgcache/repos
 
 # background prefetch with repack on pull when configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > backgroundrepack=True
   > EOF
   $ hg strip tip
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/6b4b6f66ef8c-b4b8bdaf-backup.hg (glob)
 
   $ clearcache
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 0 changes to 0 files
   updating bookmark foo
   new changesets 6b4b6f66ef8c
   (run 'hg update' to get a working copy)
   prefetching file contents
   $ sleep 0.5
   $ hg debugwaitonprefetch >/dev/null 2>%1
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 0.5
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/6e8633deba6e544e5f8edbd7b996d6e31a2c42ae.histidx
   $TESTTMP/hgcache/master/packs/6e8633deba6e544e5f8edbd7b996d6e31a2c42ae.histpack
   $TESTTMP/hgcache/master/packs/8ce5ab3745465ab83bba30a7b9c295e0c8404652.dataidx
   $TESTTMP/hgcache/master/packs/8ce5ab3745465ab83bba30a7b9c295e0c8404652.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # background prefetch with repack on update when wcprevset configured
 
   $ clearcache
   $ hg up -r 0
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0
   $TESTTMP/hgcache/master/39/5df8f7c51f007019cb30201c49e884b46b92fa/69a1b67522704ec122181c0890bd16e9d3e7516a
   $TESTTMP/hgcache/repos
 
   $ hg up -r 1
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > bgprefetchrevs=.::
   > EOF
 
   $ clearcache
   $ hg up -r 0
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   * files fetched over * fetches - (* misses, 0.00% hit ratio) over *s (glob)
   $ sleep 1
   $ hg debugwaitonprefetch >/dev/null 2>%1
   $ sleep 1
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 1
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Ensure that file 'w' was prefetched - it was not part of the update operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # background prefetch with repack on commit when wcprevset configured
 
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > bgprefetchrevs=0::
   > EOF
 
   $ clearcache
   $ find $CACHEDIR -type f | sort
   $ echo b > b
   $ hg commit -qAm b
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob)
   $ hg bookmark temporary
   $ sleep 1
   $ hg debugwaitonprefetch >/dev/null 2>%1
   $ sleep 1
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 1
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Ensure that file 'w' was prefetched - it was not part of the commit operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # background prefetch with repack on rebase when wcprevset configured
 
   $ hg up -r 2
   3 files updated, 0 files merged, 3 files removed, 0 files unresolved
   (leaving bookmark temporary)
   $ clearcache
   $ find $CACHEDIR -type f | sort
   $ hg rebase -s temporary -d foo
   rebasing 3:58147a5b5242 "b" (temporary tip)
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/58147a5b5242-c3678817-rebase.hg (glob)
   3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
   $ sleep 1
   $ hg debugwaitonprefetch >/dev/null 2>%1
   $ sleep 1
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 1
 
 # Ensure that file 'y' was prefetched - it was not part of the rebase operation and therefore
 # could only be downloaded by the background prefetch
 
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Check that foregound prefetch with no arguments blocks until background prefetches finish
 
   $ hg up -r 3
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ clearcache
   $ hg prefetch --repack
   waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?)
   got lock after * seconds (glob) (?)
   (running background incremental repack)
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
 
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 0.5
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Ensure that files were prefetched
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Check that foreground prefetch fetches revs specified by '. + draft() + bgprefetchrevs + pullprefetch'
 
   $ clearcache
   $ hg prefetch --repack
   waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?)
   got lock after * seconds (glob) (?)
   (running background incremental repack)
   * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>%1
   $ sleep 0.5
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx
   $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Ensure that files were prefetched
   $ hg debugdatapack `ls -ct $TESTTMP/hgcache/master/packs/*.datapack | head -n 1`
   $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407:
   w:
   Node          Delta Base    Delta Length  Blob Size
   bb6ccd5dceaa  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   x:
   Node          Delta Base    Delta Length  Blob Size
   ef95c5376f34  000000000000  3             3
   1406e7411862  ef95c5376f34  14            2
   
   Total:                      17            5         (240.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   076f5e2225b3  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
   z:
   Node          Delta Base    Delta Length  Blob Size
   69a1b6752270  000000000000  2             2
   
   Total:                      2             2         (0.0% bigger)
 
 # Test that if data was prefetched and repacked we dont need to prefetch it again
 # It ensures that Mercurial looks not only in loose files but in packs as well
 
   $ hg prefetch --repack
   (running background incremental repack)

tests/test-remotefilelog-repack-fast.t

@@ -1,384 +1,387
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
-
+# devel.remotefilelog.ensurestart: reduce race condition with
+# waiton{repack/prefetch}
   $ cat >> $HGRCPATH <<EOF
   > [remotefilelog]
   > fastdatapack=True
+  > [devel]
+  > remotefilelog.ensurestart=True
   > EOF
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > serverexpiration=-1
   > EOF
   $ echo x > x
   $ hg commit -qAm x
   $ echo x >> x
   $ hg commit -qAm x2
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow -q
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
 # Set the prefetchdays config to zero so that all commits are prefetched
 # no matter what their creation date is.
   $ cd shallow
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > prefetchdays=0
   > EOF
   $ cd ..
 
 # Test that repack cleans up the old files and creates new packs
 
   $ cd shallow
   $ find $CACHEDIR | sort
   $TESTTMP/hgcache
   $TESTTMP/hgcache/master
   $TESTTMP/hgcache/master/11
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/aee31534993a501858fb6dd96a065671922e7d51
   $TESTTMP/hgcache/repos
 
   $ hg repack
 
   $ find $CACHEDIR | sort
   $TESTTMP/hgcache
   $TESTTMP/hgcache/master
   $TESTTMP/hgcache/master/packs
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Test that the packs are readonly
   $ ls_l $CACHEDIR/master/packs
   -r--r--r--    1145 1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   -r--r--r--     172 1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   -r--r--r--    1074 b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   -r--r--r--      72 b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   -rw-r--r--       0 repacklock
 
 # Test that the data in the new packs is accessible
   $ hg cat -r . x
   x
   x
 
 # Test that adding new data and repacking it results in the loose data and the
 # old packs being combined.
 
   $ cd ../master
   $ echo x >> x
   $ hg commit -m x3
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
   $ hg repack --traceback
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Verify all the file data is still available
   $ hg cat -r . x
   x
   x
   x
   $ hg cat -r '.^' x
   x
   x
 
 # Test that repacking again without new data does not delete the pack files
 # and did not change the pack names
   $ hg repack
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Run two repacks at once
   $ hg repack --config "hooks.prerepack=sleep 3" &
   $ sleep 1
   $ hg repack
   skipping repack - another repack is already running
   $ hg debugwaitonrepack >/dev/null 2>&1
 
 # Run repack in the background
   $ cd ../master
   $ echo x >> x
   $ hg commit -m x4
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1bb2e6237e035c8f8ef508e281f1ce075bc6db72
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
   $ hg repack --background
   (running background repack)
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>&1
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0.dataidx
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0.datapack
   $TESTTMP/hgcache/master/packs/604552d403a1381749faf656feca0ca265a6d52c.histidx
   $TESTTMP/hgcache/master/packs/604552d403a1381749faf656feca0ca265a6d52c.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Test debug commands
 
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  d4a3ed9310e5  12            4
   
   Total:                      32            18        (77.8% bigger)
   $ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   x:
   Node                                      Delta Base                                Delta Length  Blob Size
   1bb2e6237e035c8f8ef508e281f1ce075bc6db72  0000000000000000000000000000000000000000  8             8
   d4a3ed9310e5bd9887e3bf779da5077efab28216  1bb2e6237e035c8f8ef508e281f1ce075bc6db72  12            6
   aee31534993a501858fb6dd96a065671922e7d51  d4a3ed9310e5bd9887e3bf779da5077efab28216  12            4
   
   Total:                                                                              32            18        (77.8% bigger)
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   
   x
   Node                                      Delta Base                                Delta SHA1                                Delta Length
   d4a3ed9310e5bd9887e3bf779da5077efab28216  1bb2e6237e035c8f8ef508e281f1ce075bc6db72  77029ab56e83ea2115dd53ff87483682abe5d7ca  12
   Node                                      Delta Base                                Delta SHA1                                Delta Length
   1bb2e6237e035c8f8ef508e281f1ce075bc6db72  0000000000000000000000000000000000000000  7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5  8
   $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
   
   x
   Node          P1 Node       P2 Node       Link Node     Copy From
   1bb2e6237e03  d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5  aee31534993a  000000000000  421535db10b6  
   aee31534993a  1406e7411862  000000000000  a89d614e2364  
   1406e7411862  000000000000  000000000000  b292c1e3311f  
 
 # Test copy tracing from a pack
   $ cd ../master
   $ hg mv x y
   $ hg commit -m 'move x to y'
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ hg log -f y -T '{desc}\n'
   move x to y
   x4
   x3
   x2
   x
 
 # Test copy trace across rename and back
   $ cp -R $TESTTMP/hgcache/master/packs $TESTTMP/backuppacks
   $ cd ../master
   $ hg mv y x
   $ hg commit -m 'move y back to x'
   $ hg revert -r 0 x
   $ mv x y
   $ hg add y
   $ echo >> y
   $ hg revert x
   $ hg commit -m 'add y back without metadata'
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ ls $TESTTMP/hgcache/master/packs
   bfd60adb76018bb952e27cd23fc151bf94865d7d.histidx
   bfd60adb76018bb952e27cd23fc151bf94865d7d.histpack
   fb3aa57b22789ebcc45706c352e2d6af099c5816.dataidx
   fb3aa57b22789ebcc45706c352e2d6af099c5816.datapack
   repacklock
   $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
   
   x
   Node          P1 Node       P2 Node       Link Node     Copy From
   cd410a44d584  577959738234  000000000000  609547eda446  y
   1bb2e6237e03  d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5  aee31534993a  000000000000  421535db10b6  
   aee31534993a  1406e7411862  000000000000  a89d614e2364  
   1406e7411862  000000000000  000000000000  b292c1e3311f  
   
   y
   Node          P1 Node       P2 Node       Link Node     Copy From
   577959738234  1bb2e6237e03  000000000000  c7faf2fc439a  x
   21f46f2721e7  000000000000  000000000000  d6868642b790  
   $ hg strip -r '.^'
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/609547eda446-b26b56a8-backup.hg (glob)
   $ hg -R ../master strip -r '.^'
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/master/.hg/strip-backup/609547eda446-b26b56a8-backup.hg (glob)
 
   $ rm -rf $TESTTMP/hgcache/master/packs
   $ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs
 
 # Test repacking datapack without history
   $ rm -rf $CACHEDIR/master/packs/*hist*
   $ hg repack
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/922aca43dbbeda4d250565372e8892ec7b08da6a:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  d4a3ed9310e5  12            4
   
   Total:                      32            18        (77.8% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   577959738234  000000000000  70            8
   
   Total:                      70            8         (775.0% bigger)
 
   $ hg cat -r ".^" x
   x
   x
   x
   x
 
 Incremental repack
   $ rm -rf $CACHEDIR/master/packs/*
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > data.generations=60
   >   150
   > EOF
 
 Single pack - repack does nothing
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   [1]
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   [1]
   $ hg repack --incremental
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1
   $ hg prefetch -r 1
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 2
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 3
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
   $ hg repack --incremental
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   -r--r--r--     226 39443fa1064182e93d968b5cba292eb5283260d0.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     336 604552d403a1381749faf656feca0ca265a6d52c.histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 1 gen3 pack, 1 gen0 pack - does nothing
   $ hg repack --incremental
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   -r--r--r--     226 39443fa1064182e93d968b5cba292eb5283260d0.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     336 604552d403a1381749faf656feca0ca265a6d52c.histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 Pull should run background repack
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > backgroundrepack=True
   > EOF
   $ clearcache
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 1
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 2
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 3
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
 
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   no changes found
   (running background incremental repack)
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>&1
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--     303 156a6c1c83aeb69422d7936e0a46ba9bc06a71c0.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     336 604552d403a1381749faf656feca0ca265a6d52c.histpack
 
 Test environment variable resolution
   $ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ find $TESTTMP/envcache | sort
   $TESTTMP/envcache
   $TESTTMP/envcache/master
   $TESTTMP/envcache/master/95
   $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a
   $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/577959738234a1eb241ed3ed4b22a575833f56e0
   $TESTTMP/envcache/repos
 
 Test local remotefilelog blob is correct when based on a pack
   $ hg prefetch -r .
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ echo >> y
   $ hg commit -m y2
   $ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
   size: 9 bytes
   path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 
   key: b70860edba4f 
   
           node =>           p1            p2      linknode     copyfrom
   b70860edba4f => 577959738234  000000000000  08d3fbc98c48  
   577959738234 => 1bb2e6237e03  000000000000  c7faf2fc439a  x
   1bb2e6237e03 => d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5 => aee31534993a  000000000000  421535db10b6  
   aee31534993a => 1406e7411862  000000000000  a89d614e2364  
   1406e7411862 => 000000000000  000000000000  b292c1e3311f  

tests/test-remotefilelog-repack.t

@@ -1,462 +1,468
 #require no-windows
 
   $ . "$TESTDIR/remotefilelog-library.sh"
+# devel.remotefilelog.ensurestart: reduce race condition with
+# waiton{repack/prefetch}
+  $ cat >> $HGRCPATH <<EOF
+  > [devel]
+  > remotefilelog.ensurestart=True
+  > EOF
 
   $ hg init master
   $ cd master
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > server=True
   > serverexpiration=-1
   > EOF
   $ echo x > x
   $ hg commit -qAm x
   $ echo x >> x
   $ hg commit -qAm x2
   $ cd ..
 
   $ hgcloneshallow ssh://user@dummy/master shallow -q
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
 
 # Set the prefetchdays config to zero so that all commits are prefetched
 # no matter what their creation date is.
   $ cd shallow
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > prefetchdays=0
   > EOF
   $ cd ..
 
 # Test that repack cleans up the old files and creates new packs
 
   $ cd shallow
   $ find $CACHEDIR | sort
   $TESTTMP/hgcache
   $TESTTMP/hgcache/master
   $TESTTMP/hgcache/master/11
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/aee31534993a501858fb6dd96a065671922e7d51
   $TESTTMP/hgcache/repos
 
   $ hg repack
 
   $ find $CACHEDIR | sort
   $TESTTMP/hgcache
   $TESTTMP/hgcache/master
   $TESTTMP/hgcache/master/packs
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Test that the packs are readonly
   $ ls_l $CACHEDIR/master/packs
   -r--r--r--    1145 1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   -r--r--r--     172 1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   -r--r--r--    1074 b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   -r--r--r--      72 b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   -rw-r--r--       0 repacklock
 
 # Test that the data in the new packs is accessible
   $ hg cat -r . x
   x
   x
 
 # Test that adding new data and repacking it results in the loose data and the
 # old packs being combined.
 
   $ cd ../master
   $ echo x >> x
   $ hg commit -m x3
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # First assert that with --packsonly, the loose object will be ignored:
 
   $ hg repack --packsonly
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histidx
   $TESTTMP/hgcache/master/packs/1e91b207daf5d7b48f1be9c587d6b5ae654ce78c.histpack
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.dataidx
   $TESTTMP/hgcache/master/packs/b1e0cfc7f345e408a7825e3081501959488d59ce.datapack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
   $ hg repack --traceback
 
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Verify all the file data is still available
   $ hg cat -r . x
   x
   x
   x
   $ hg cat -r '.^' x
   x
   x
 
 # Test that repacking again without new data does not delete the pack files
 # and did not change the pack names
   $ hg repack
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Run two repacks at once
   $ hg repack --config "hooks.prerepack=sleep 3" &
   $ sleep 1
   $ hg repack
   skipping repack - another repack is already running
   $ hg debugwaitonrepack >/dev/null 2>&1
 
 # Run repack in the background
   $ cd ../master
   $ echo x >> x
   $ hg commit -m x4
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1bb2e6237e035c8f8ef508e281f1ce075bc6db72
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.dataidx
   $TESTTMP/hgcache/master/packs/78840d69389c7404327f7477e3931c89945c37d1.datapack
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histidx
   $TESTTMP/hgcache/master/packs/8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
   $ hg repack --background
   (running background repack)
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>&1
   $ find $CACHEDIR -type f | sort
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0.dataidx
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0.datapack
   $TESTTMP/hgcache/master/packs/604552d403a1381749faf656feca0ca265a6d52c.histidx
   $TESTTMP/hgcache/master/packs/604552d403a1381749faf656feca0ca265a6d52c.histpack
   $TESTTMP/hgcache/master/packs/repacklock
   $TESTTMP/hgcache/repos
 
 # Test debug commands
 
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  d4a3ed9310e5  12            4
   
   Total:                      32            18        (77.8% bigger)
   $ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   x:
   Node                                      Delta Base                                Delta Length  Blob Size
   1bb2e6237e035c8f8ef508e281f1ce075bc6db72  0000000000000000000000000000000000000000  8             8
   d4a3ed9310e5bd9887e3bf779da5077efab28216  1bb2e6237e035c8f8ef508e281f1ce075bc6db72  12            6
   aee31534993a501858fb6dd96a065671922e7d51  d4a3ed9310e5bd9887e3bf779da5077efab28216  12            4
   
   Total:                                                                              32            18        (77.8% bigger)
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216
   $TESTTMP/hgcache/master/packs/39443fa1064182e93d968b5cba292eb5283260d0:
   
   x
   Node                                      Delta Base                                Delta SHA1                                Delta Length
   d4a3ed9310e5bd9887e3bf779da5077efab28216  1bb2e6237e035c8f8ef508e281f1ce075bc6db72  77029ab56e83ea2115dd53ff87483682abe5d7ca  12
   Node                                      Delta Base                                Delta SHA1                                Delta Length
   1bb2e6237e035c8f8ef508e281f1ce075bc6db72  0000000000000000000000000000000000000000  7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5  8
   $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
   
   x
   Node          P1 Node       P2 Node       Link Node     Copy From
   1bb2e6237e03  d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5  aee31534993a  000000000000  421535db10b6  
   aee31534993a  1406e7411862  000000000000  a89d614e2364  
   1406e7411862  000000000000  000000000000  b292c1e3311f  
 
 # Test copy tracing from a pack
   $ cd ../master
   $ hg mv x y
   $ hg commit -m 'move x to y'
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ hg log -f y -T '{desc}\n'
   move x to y
   x4
   x3
   x2
   x
 
 # Test copy trace across rename and back
   $ cp -R $TESTTMP/hgcache/master/packs $TESTTMP/backuppacks
   $ cd ../master
   $ hg mv y x
   $ hg commit -m 'move y back to x'
   $ hg revert -r 0 x
   $ mv x y
   $ hg add y
   $ echo >> y
   $ hg revert x
   $ hg commit -m 'add y back without metadata'
   $ cd ../shallow
   $ hg pull -q
   $ hg up -q tip
   2 files fetched over 2 fetches - (2 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ ls $TESTTMP/hgcache/master/packs
   bfd60adb76018bb952e27cd23fc151bf94865d7d.histidx
   bfd60adb76018bb952e27cd23fc151bf94865d7d.histpack
   fb3aa57b22789ebcc45706c352e2d6af099c5816.dataidx
   fb3aa57b22789ebcc45706c352e2d6af099c5816.datapack
   repacklock
   $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
   
   x
   Node          P1 Node       P2 Node       Link Node     Copy From
   cd410a44d584  577959738234  000000000000  609547eda446  y
   1bb2e6237e03  d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5  aee31534993a  000000000000  421535db10b6  
   aee31534993a  1406e7411862  000000000000  a89d614e2364  
   1406e7411862  000000000000  000000000000  b292c1e3311f  
   
   y
   Node          P1 Node       P2 Node       Link Node     Copy From
   577959738234  1bb2e6237e03  000000000000  c7faf2fc439a  x
   21f46f2721e7  000000000000  000000000000  d6868642b790  
   $ hg strip -r '.^'
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/609547eda446-b26b56a8-backup.hg (glob)
   $ hg -R ../master strip -r '.^'
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   saved backup bundle to $TESTTMP/master/.hg/strip-backup/609547eda446-b26b56a8-backup.hg (glob)
 
   $ rm -rf $TESTTMP/hgcache/master/packs
   $ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs
 
 # Test repacking datapack without history
   $ rm -rf $CACHEDIR/master/packs/*hist*
   $ hg repack
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/922aca43dbbeda4d250565372e8892ec7b08da6a:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  d4a3ed9310e5  12            4
   
   Total:                      32            18        (77.8% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   577959738234  000000000000  70            8
   
   Total:                      70            8         (775.0% bigger)
 
   $ hg cat -r ".^" x
   x
   x
   x
   x
 
 Incremental repack
   $ rm -rf $CACHEDIR/master/packs/*
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > data.generations=60
   >   150
   > EOF
 
 Single pack - repack does nothing
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   [1]
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   [1]
   $ hg repack --incremental
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1
   $ hg prefetch -r 1
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 2
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 38
   abort: unknown revision '38'!
   [255]
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 For the data packs, setting the limit for the repackmaxpacksize to be 64 such
 that data pack with size 65 is more than the limit. This effectively ensures
 that no generation has 3 packs and therefore, no packs are chosen for the
 incremental repacking. As for the history packs, setting repackmaxpacksize to be
 0 which should always result in no repacking.
   $ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=64 \
   > --config remotefilelog.history.repackmaxpacksize=0
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   -r--r--r--     149 78840d69389c7404327f7477e3931c89945c37d1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     254 8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 Setting limit for the repackmaxpacksize to be the size of the biggest pack file
 which ensures that it is effectively ignored in the incremental repacking.
   $ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=65 \
   > --config remotefilelog.history.repackmaxpacksize=336
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   -r--r--r--     149 78840d69389c7404327f7477e3931c89945c37d1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     254 8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 1 gen3 pack, 1 gen0 pack - does nothing
   $ hg repack --incremental
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--      70 052643fdcdebbd42d7c180a651a30d46098e6fe1.datapack
   -r--r--r--     149 78840d69389c7404327f7477e3931c89945c37d1.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     254 8abe7889aae389337d12ebe6085d4ee13854c7c9.histpack
   -r--r--r--      90 955a622173324b2d8b53e1147f209f1cf125302e.histpack
 
 Pull should run background repack
   $ cat >> .hg/hgrc <<EOF
   > [remotefilelog]
   > backgroundrepack=True
   > EOF
   $ clearcache
   $ hg prefetch -r 0
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 1
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 2
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ hg prefetch -r 3
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
 
   $ hg pull
   pulling from ssh://user@dummy/master
   searching for changes
   no changes found
   (running background incremental repack)
   $ sleep 0.5
   $ hg debugwaitonrepack >/dev/null 2>&1
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
   -r--r--r--     303 156a6c1c83aeb69422d7936e0a46ba9bc06a71c0.datapack
   $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
   -r--r--r--     336 604552d403a1381749faf656feca0ca265a6d52c.histpack
 
 Test environment variable resolution
   $ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ find $TESTTMP/envcache | sort
   $TESTTMP/envcache
   $TESTTMP/envcache/master
   $TESTTMP/envcache/master/95
   $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a
   $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/577959738234a1eb241ed3ed4b22a575833f56e0
   $TESTTMP/envcache/repos
 
 Test local remotefilelog blob is correct when based on a pack
   $ hg prefetch -r .
   1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
   $ echo >> y
   $ hg commit -m y2
   $ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
   size: 9 bytes
   path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 
   key: b70860edba4f 
   
           node =>           p1            p2      linknode     copyfrom
   b70860edba4f => 577959738234  000000000000  08d3fbc98c48  
   577959738234 => 1bb2e6237e03  000000000000  c7faf2fc439a  x
   1bb2e6237e03 => d4a3ed9310e5  000000000000  0b03bbc9e1e7  
   d4a3ed9310e5 => aee31534993a  000000000000  421535db10b6  
   aee31534993a => 1406e7411862  000000000000  a89d614e2364  
   1406e7411862 => 000000000000  000000000000  b292c1e3311f  
 
 Test limiting the max delta chain length
   $ hg repack --config packs.maxchainlen=1
   $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx
   $TESTTMP/hgcache/master/packs/f258af4c033dd5cd32b4dbc42a1efcd8e4c7d909:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  000000000000  4             4
   1406e7411862  aee31534993a  12            2
   
   Total:                      36            20        (80.0% bigger)
   y:
   Node          Delta Base    Delta Length  Blob Size
   577959738234  000000000000  70            8
   
   Total:                      70            8         (775.0% bigger)
 
 Test huge pack cleanup using different values of packs.maxpacksize:
   $ hg repack --incremental --debug
   $ hg repack --incremental --debug --config packs.maxpacksize=512
   removing oversize packfile $TESTTMP/hgcache/master/packs/f258af4c033dd5cd32b4dbc42a1efcd8e4c7d909.datapack (425 bytes)
   removing oversize packfile $TESTTMP/hgcache/master/packs/f258af4c033dd5cd32b4dbc42a1efcd8e4c7d909.dataidx (1.21 KB)
 
 Do a repack where the new pack reuses a delta from the old pack
   $ clearcache
   $ hg prefetch -r '2::3'
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ hg debugdatapack $CACHEDIR/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/9ec6b30891bd851320acb7c66b69a2bdf41c8df3:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   
   Total:                      20            14        (42.9% bigger)
   $ hg prefetch -r '0::1'
   2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob)
   $ hg repack
   $ hg debugdatapack $CACHEDIR/master/packs/*.datapack
   $TESTTMP/hgcache/master/packs/156a6c1c83aeb69422d7936e0a46ba9bc06a71c0:
   x:
   Node          Delta Base    Delta Length  Blob Size
   1bb2e6237e03  000000000000  8             8
   d4a3ed9310e5  1bb2e6237e03  12            6
   aee31534993a  d4a3ed9310e5  12            4
   1406e7411862  aee31534993a  12            2
   
   Total:                      44            20        (120.0% bigger)