lfs: add repository feature denoting the use of LFS

Whether LFS is enabled seems like a useful feature to expose. This
will also facilitate some future work around LFS feature compatibility.

Differential Revision: https://phab.mercurial-scm.org/D4710

hgext/lfs/__init__.py

@@ -1,401 +1,405
 # lfs - hash-preserving large file support using Git-LFS protocol
 #
 # Copyright 2017 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 """lfs - large file support (EXPERIMENTAL)
 
 This extension allows large files to be tracked outside of the normal
 repository storage and stored on a centralized server, similar to the
 ``largefiles`` extension.  The ``git-lfs`` protocol is used when
 communicating with the server, so existing git infrastructure can be
 harnessed.  Even though the files are stored outside of the repository,
 they are still integrity checked in the same manner as normal files.
 
 The files stored outside of the repository are downloaded on demand,
 which reduces the time to clone, and possibly the local disk usage.
 This changes fundamental workflows in a DVCS, so careful thought
 should be given before deploying it.  :hg:`convert` can be used to
 convert LFS repositories to normal repositories that no longer
 require this extension, and do so without changing the commit hashes.
 This allows the extension to be disabled if the centralized workflow
 becomes burdensome.  However, the pre and post convert clones will
 not be able to communicate with each other unless the extension is
 enabled on both.
 
 To start a new repository, or to add LFS files to an existing one, just
 create an ``.hglfs`` file as described below in the root directory of
 the repository.  Typically, this file should be put under version
 control, so that the settings will propagate to other repositories with
 push and pull.  During any commit, Mercurial will consult this file to
 determine if an added or modified file should be stored externally.  The
 type of storage depends on the characteristics of the file at each
 commit.  A file that is near a size threshold may switch back and forth
 between LFS and normal storage, as needed.
 
 Alternately, both normal repositories and largefile controlled
 repositories can be converted to LFS by using :hg:`convert` and the
 ``lfs.track`` config option described below.  The ``.hglfs`` file
 should then be created and added, to control subsequent LFS selection.
 The hashes are also unchanged in this case.  The LFS and non-LFS
 repositories can be distinguished because the LFS repository will
 abort any command if this extension is disabled.
 
 Committed LFS files are held locally, until the repository is pushed.
 Prior to pushing the normal repository data, the LFS files that are
 tracked by the outgoing commits are automatically uploaded to the
 configured central server.  No LFS files are transferred on
 :hg:`pull` or :hg:`clone`.  Instead, the files are downloaded on
 demand as they need to be read, if a cached copy cannot be found
 locally.  Both committing and downloading an LFS file will link the
 file to a usercache, to speed up future access.  See the `usercache`
 config setting described below.
 
 .hglfs::
 
     The extension reads its configuration from a versioned ``.hglfs``
     configuration file found in the root of the working directory. The
     ``.hglfs`` file uses the same syntax as all other Mercurial
     configuration files. It uses a single section, ``[track]``.
 
     The ``[track]`` section specifies which files are stored as LFS (or
     not). Each line is keyed by a file pattern, with a predicate value.
     The first file pattern match is used, so put more specific patterns
     first.  The available predicates are ``all()``, ``none()``, and
     ``size()``. See "hg help filesets.size" for the latter.
 
     Example versioned ``.hglfs`` file::
 
       [track]
       # No Makefile or python file, anywhere, will be LFS
       **Makefile = none()
       **.py = none()
 
       **.zip = all()
       **.exe = size(">1MB")
 
       # Catchall for everything not matched above
       ** = size(">10MB")
 
 Configs::
 
     [lfs]
     # Remote endpoint. Multiple protocols are supported:
     # - http(s)://user:[e-mail]/path
     #   git-lfs endpoint
     # - file:///tmp/path
     #   local filesystem, usually for testing
     # if unset, lfs will assume the remote repository also handles blob storage
     # for http(s) URLs.  Otherwise, lfs will prompt to set this when it must
     # use this value.
     # (default: unset)
     url = https://example.com/repo.git/info/lfs
 
     # Which files to track in LFS.  Path tests are "**.extname" for file
     # extensions, and "path:under/some/directory" for path prefix.  Both
     # are relative to the repository root.
     # File size can be tested with the "size()" fileset, and tests can be
     # joined with fileset operators.  (See "hg help filesets.operators".)
     #
     # Some examples:
     # - all()                       # everything
     # - none()                      # nothing
     # - size(">20MB")               # larger than 20MB
     # - !**.txt                     # anything not a *.txt file
     # - **.zip | **.tar.gz | **.7z  # some types of compressed files
     # - path:bin                    # files under "bin" in the project root
     # - (**.php & size(">2MB")) | (**.js & size(">5MB")) | **.tar.gz
     #     | (path:bin & !path:/bin/README) | size(">1GB")
     # (default: none())
     #
     # This is ignored if there is a tracked '.hglfs' file, and this setting
     # will eventually be deprecated and removed.
     track = size(">10M")
 
     # how many times to retry before giving up on transferring an object
     retry = 5
 
     # the local directory to store lfs files for sharing across local clones.
     # If not set, the cache is located in an OS specific cache location.
     usercache = /path/to/global/cache
 """
 
 from __future__ import absolute_import
 
 from mercurial.i18n import _
 
 from mercurial import (
     bundle2,
     changegroup,
     cmdutil,
     config,
     context,
     error,
     exchange,
     extensions,
     filelog,
     filesetlang,
     hg,
     localrepo,
     minifileset,
     node,
     pycompat,
     registrar,
+    repository,
     revlog,
     scmutil,
     templateutil,
     upgrade,
     util,
     vfs as vfsmod,
     wireprotoserver,
     wireprotov1server,
 )
 
 from . import (
     blobstore,
     wireprotolfsserver,
     wrapper,
 )
 
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
 testedwith = 'ships-with-hg-core'
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 configitem('experimental', 'lfs.serve',
     default=True,
 )
 configitem('experimental', 'lfs.user-agent',
     default=None,
 )
 configitem('experimental', 'lfs.disableusercache',
     default=False,
 )
 configitem('experimental', 'lfs.worker-enable',
     default=False,
 )
 
 configitem('lfs', 'url',
     default=None,
 )
 configitem('lfs', 'usercache',
     default=None,
 )
 # Deprecated
 configitem('lfs', 'threshold',
     default=None,
 )
 configitem('lfs', 'track',
     default='none()',
 )
 configitem('lfs', 'retry',
     default=5,
 )
 
 cmdtable = {}
 command = registrar.command(cmdtable)
 
 templatekeyword = registrar.templatekeyword()
 filesetpredicate = registrar.filesetpredicate()
 
 def featuresetup(ui, supported):
     # don't die on seeing a repo with the lfs requirement
     supported |= {'lfs'}
 
 def uisetup(ui):
     localrepo.featuresetupfuncs.add(featuresetup)
 
 def reposetup(ui, repo):
     # Nothing to do with a remote repo
     if not repo.local():
         return
 
     repo.svfs.lfslocalblobstore = blobstore.local(repo)
     repo.svfs.lfsremoteblobstore = blobstore.remote(repo)
 
     class lfsrepo(repo.__class__):
         @localrepo.unfilteredmethod
         def commitctx(self, ctx, error=False):
             repo.svfs.options['lfstrack'] = _trackedmatcher(self)
             return super(lfsrepo, self).commitctx(ctx, error)
 
     repo.__class__ = lfsrepo
 
     if 'lfs' not in repo.requirements:
         def checkrequireslfs(ui, repo, **kwargs):
             if 'lfs' not in repo.requirements:
                 last = kwargs.get(r'node_last')
                 _bin = node.bin
                 if last:
                     s = repo.set('%n:%n', _bin(kwargs[r'node']), _bin(last))
                 else:
                     s = repo.set('%n', _bin(kwargs[r'node']))
             match = repo.narrowmatch()
             for ctx in s:
                 # TODO: is there a way to just walk the files in the commit?
                 if any(ctx[f].islfs() for f in ctx.files()
                        if f in ctx and match(f)):
                     repo.requirements.add('lfs')
+                    repo.features.add(repository.REPO_FEATURE_LFS)
                     repo._writerequirements()
                     repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
                     break
 
         ui.setconfig('hooks', 'commit.lfs', checkrequireslfs, 'lfs')
         ui.setconfig('hooks', 'pretxnchangegroup.lfs', checkrequireslfs, 'lfs')
     else:
         repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
 
 def _trackedmatcher(repo):
     """Return a function (path, size) -> bool indicating whether or not to
     track a given file with lfs."""
     if not repo.wvfs.exists('.hglfs'):
         # No '.hglfs' in wdir.  Fallback to config for now.
         trackspec = repo.ui.config('lfs', 'track')
 
         # deprecated config: lfs.threshold
         threshold = repo.ui.configbytes('lfs', 'threshold')
         if threshold:
             filesetlang.parse(trackspec)  # make sure syntax errors are confined
             trackspec = "(%s) | size('>%d')" % (trackspec, threshold)
 
         return minifileset.compile(trackspec)
 
     data = repo.wvfs.tryread('.hglfs')
     if not data:
         return lambda p, s: False
 
     # Parse errors here will abort with a message that points to the .hglfs file
     # and line number.
     cfg = config.config()
     cfg.parse('.hglfs', data)
 
     try:
         rules = [(minifileset.compile(pattern), minifileset.compile(rule))
                  for pattern, rule in cfg.items('track')]
     except error.ParseError as e:
         # The original exception gives no indicator that the error is in the
         # .hglfs file, so add that.
 
         # TODO: See if the line number of the file can be made available.
         raise error.Abort(_('parse error in .hglfs: %s') % e)
 
     def _match(path, size):
         for pat, rule in rules:
             if pat(path, size):
                 return rule(path, size)
 
         return False
 
     return _match
 
 def wrapfilelog(filelog):
     wrapfunction = extensions.wrapfunction
 
     wrapfunction(filelog, 'addrevision', wrapper.filelogaddrevision)
     wrapfunction(filelog, 'renamed', wrapper.filelogrenamed)
     wrapfunction(filelog, 'size', wrapper.filelogsize)
 
 def extsetup(ui):
     wrapfilelog(filelog.filelog)
 
     wrapfunction = extensions.wrapfunction
 
+    wrapfunction(localrepo, 'makefilestorage', wrapper.localrepomakefilestorage)
+
     wrapfunction(cmdutil, '_updatecatformatter', wrapper._updatecatformatter)
     wrapfunction(scmutil, 'wrapconvertsink', wrapper.convertsink)
 
     wrapfunction(upgrade, '_finishdatamigration',
                  wrapper.upgradefinishdatamigration)
 
     wrapfunction(upgrade, 'preservedrequirements',
                  wrapper.upgraderequirements)
 
     wrapfunction(upgrade, 'supporteddestrequirements',
                  wrapper.upgraderequirements)
 
     wrapfunction(changegroup,
                  'allsupportedversions',
                  wrapper.allsupportedversions)
 
     wrapfunction(exchange, 'push', wrapper.push)
     wrapfunction(wireprotov1server, '_capabilities', wrapper._capabilities)
     wrapfunction(wireprotoserver, 'handlewsgirequest',
                  wireprotolfsserver.handlewsgirequest)
 
     wrapfunction(context.basefilectx, 'cmp', wrapper.filectxcmp)
     wrapfunction(context.basefilectx, 'isbinary', wrapper.filectxisbinary)
     context.basefilectx.islfs = wrapper.filectxislfs
 
     revlog.addflagprocessor(
         revlog.REVIDX_EXTSTORED,
         (
             wrapper.readfromstore,
             wrapper.writetostore,
             wrapper.bypasscheckhash,
         ),
     )
 
     wrapfunction(hg, 'clone', wrapper.hgclone)
     wrapfunction(hg, 'postshare', wrapper.hgpostshare)
 
     scmutil.fileprefetchhooks.add('lfs', wrapper._prefetchfiles)
 
     # Make bundle choose changegroup3 instead of changegroup2. This affects
     # "hg bundle" command. Note: it does not cover all bundle formats like
     # "packed1". Using "packed1" with lfs will likely cause trouble.
     exchange._bundlespeccontentopts["v2"]["cg.version"] = "03"
 
     # bundlerepo uses "vfsmod.readonlyvfs(othervfs)", we need to make sure lfs
     # options and blob stores are passed from othervfs to the new readonlyvfs.
     wrapfunction(vfsmod.readonlyvfs, '__init__', wrapper.vfsinit)
 
     # when writing a bundle via "hg bundle" command, upload related LFS blobs
     wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)
 
 @filesetpredicate('lfs()')
 def lfsfileset(mctx, x):
     """File that uses LFS storage."""
     # i18n: "lfs" is a keyword
     filesetlang.getargs(x, 0, 0, _("lfs takes no arguments"))
     ctx = mctx.ctx
     def lfsfilep(f):
         return wrapper.pointerfromctx(ctx, f, removed=True) is not None
     return mctx.predicate(lfsfilep, predrepr='<lfs>')
 
 @templatekeyword('lfs_files', requires={'ctx'})
 def lfsfiles(context, mapping):
     """List of strings. All files modified, added, or removed by this
     changeset."""
     ctx = context.resource(mapping, 'ctx')
 
     pointers = wrapper.pointersfromctx(ctx, removed=True) # {path: pointer}
     files = sorted(pointers.keys())
 
     def pointer(v):
         # In the file spec, version is first and the other keys are sorted.
         sortkeyfunc = lambda x: (x[0] != 'version', x)
         items = sorted(pointers[v].iteritems(), key=sortkeyfunc)
         return util.sortdict(items)
 
     makemap = lambda v: {
         'file': v,
         'lfsoid': pointers[v].oid() if pointers[v] else None,
         'lfspointer': templateutil.hybriddict(pointer(v)),
     }
 
     # TODO: make the separator ', '?
     f = templateutil._showcompatlist(context, mapping, 'lfs_file', files)
     return templateutil.hybrid(f, files, makemap, pycompat.identity)
 
 @command('debuglfsupload',
          [('r', 'rev', [], _('upload large files introduced by REV'))])
 def debuglfsupload(ui, repo, **opts):
     """upload lfs blobs added by the working copy parent or given revisions"""
     revs = opts.get(r'rev', [])
     pointers = wrapper.extractpointers(repo, scmutil.revrange(repo, revs))
     wrapper.uploadblobs(repo, pointers)

hgext/lfs/wrapper.py

@@ -1,425 +1,432
 # wrapper.py - methods wrapping core mercurial logic
 #
 # Copyright 2017 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 import hashlib
 
 from mercurial.i18n import _
 from mercurial.node import bin, hex, nullid, short
 
 from mercurial import (
     error,
+    repository,
     revlog,
     util,
 )
 
 from mercurial.utils import (
     stringutil,
 )
 
 from ..largefiles import lfutil
 
 from . import (
     blobstore,
     pointer,
 )
 
+def localrepomakefilestorage(orig, requirements, features, **kwargs):
+    if b'lfs' in requirements:
+        features.add(repository.REPO_FEATURE_LFS)
+
+    return orig(requirements=requirements, features=features, **kwargs)
+
 def allsupportedversions(orig, ui):
     versions = orig(ui)
     versions.add('03')
     return versions
 
 def _capabilities(orig, repo, proto):
     '''Wrap server command to announce lfs server capability'''
     caps = orig(repo, proto)
     if util.safehasattr(repo.svfs, 'lfslocalblobstore'):
         # XXX: change to 'lfs=serve' when separate git server isn't required?
         caps.append('lfs')
     return caps
 
 def bypasscheckhash(self, text):
     return False
 
 def readfromstore(self, text):
     """Read filelog content from local blobstore transform for flagprocessor.
 
     Default tranform for flagprocessor, returning contents from blobstore.
     Returns a 2-typle (text, validatehash) where validatehash is True as the
     contents of the blobstore should be checked using checkhash.
     """
     p = pointer.deserialize(text)
     oid = p.oid()
     store = self.opener.lfslocalblobstore
     if not store.has(oid):
         p.filename = self.filename
         self.opener.lfsremoteblobstore.readbatch([p], store)
 
     # The caller will validate the content
     text = store.read(oid, verify=False)
 
     # pack hg filelog metadata
     hgmeta = {}
     for k in p.keys():
         if k.startswith('x-hg-'):
             name = k[len('x-hg-'):]
             hgmeta[name] = p[k]
     if hgmeta or text.startswith('\1\n'):
         text = revlog.packmeta(hgmeta, text)
 
     return (text, True)
 
 def writetostore(self, text):
     # hg filelog metadata (includes rename, etc)
     hgmeta, offset = revlog.parsemeta(text)
     if offset and offset > 0:
         # lfs blob does not contain hg filelog metadata
         text = text[offset:]
 
     # git-lfs only supports sha256
     oid = hex(hashlib.sha256(text).digest())
     self.opener.lfslocalblobstore.write(oid, text)
 
     # replace contents with metadata
     longoid = 'sha256:%s' % oid
     metadata = pointer.gitlfspointer(oid=longoid, size='%d' % len(text))
 
     # by default, we expect the content to be binary. however, LFS could also
     # be used for non-binary content. add a special entry for non-binary data.
     # this will be used by filectx.isbinary().
     if not stringutil.binary(text):
         # not hg filelog metadata (affecting commit hash), no "x-hg-" prefix
         metadata['x-is-binary'] = '0'
 
     # translate hg filelog metadata to lfs metadata with "x-hg-" prefix
     if hgmeta is not None:
         for k, v in hgmeta.iteritems():
             metadata['x-hg-%s' % k] = v
 
     rawtext = metadata.serialize()
     return (rawtext, False)
 
 def _islfs(rlog, node=None, rev=None):
     if rev is None:
         if node is None:
             # both None - likely working copy content where node is not ready
             return False
         rev = rlog.rev(node)
     else:
         node = rlog.node(rev)
     if node == nullid:
         return False
     flags = rlog.flags(rev)
     return bool(flags & revlog.REVIDX_EXTSTORED)
 
 def filelogaddrevision(orig, self, text, transaction, link, p1, p2,
                        cachedelta=None, node=None,
                        flags=revlog.REVIDX_DEFAULT_FLAGS, **kwds):
     # The matcher isn't available if reposetup() wasn't called.
     lfstrack = self.opener.options.get('lfstrack')
 
     if lfstrack:
         textlen = len(text)
         # exclude hg rename meta from file size
         meta, offset = revlog.parsemeta(text)
         if offset:
             textlen -= offset
 
         if lfstrack(self.filename, textlen):
             flags |= revlog.REVIDX_EXTSTORED
 
     return orig(self, text, transaction, link, p1, p2, cachedelta=cachedelta,
                 node=node, flags=flags, **kwds)
 
 def filelogrenamed(orig, self, node):
     if _islfs(self, node):
         rawtext = self.revision(node, raw=True)
         if not rawtext:
             return False
         metadata = pointer.deserialize(rawtext)
         if 'x-hg-copy' in metadata and 'x-hg-copyrev' in metadata:
             return metadata['x-hg-copy'], bin(metadata['x-hg-copyrev'])
         else:
             return False
     return orig(self, node)
 
 def filelogsize(orig, self, rev):
     if _islfs(self, rev=rev):
         # fast path: use lfs metadata to answer size
         rawtext = self.revision(rev, raw=True)
         metadata = pointer.deserialize(rawtext)
         return int(metadata['size'])
     return orig(self, rev)
 
 def filectxcmp(orig, self, fctx):
     """returns True if text is different than fctx"""
     # some fctx (ex. hg-git) is not based on basefilectx and do not have islfs
     if self.islfs() and getattr(fctx, 'islfs', lambda: False)():
         # fast path: check LFS oid
         p1 = pointer.deserialize(self.rawdata())
         p2 = pointer.deserialize(fctx.rawdata())
         return p1.oid() != p2.oid()
     return orig(self, fctx)
 
 def filectxisbinary(orig, self):
     if self.islfs():
         # fast path: use lfs metadata to answer isbinary
         metadata = pointer.deserialize(self.rawdata())
         # if lfs metadata says nothing, assume it's binary by default
         return bool(int(metadata.get('x-is-binary', 1)))
     return orig(self)
 
 def filectxislfs(self):
     return _islfs(self.filelog(), self.filenode())
 
 def _updatecatformatter(orig, fm, ctx, matcher, path, decode):
     orig(fm, ctx, matcher, path, decode)
     fm.data(rawdata=ctx[path].rawdata())
 
 def convertsink(orig, sink):
     sink = orig(sink)
     if sink.repotype == 'hg':
         class lfssink(sink.__class__):
             def putcommit(self, files, copies, parents, commit, source, revmap,
                           full, cleanp2):
                 pc = super(lfssink, self).putcommit
                 node = pc(files, copies, parents, commit, source, revmap, full,
                           cleanp2)
 
                 if 'lfs' not in self.repo.requirements:
                     ctx = self.repo[node]
 
                     # The file list may contain removed files, so check for
                     # membership before assuming it is in the context.
                     if any(f in ctx and ctx[f].islfs() for f, n in files):
                         self.repo.requirements.add('lfs')
                         self.repo._writerequirements()
 
                         # Permanently enable lfs locally
                         self.repo.vfs.append(
                             'hgrc', util.tonativeeol('\n[extensions]\nlfs=\n'))
 
                 return node
 
         sink.__class__ = lfssink
 
     return sink
 
 def vfsinit(orig, self, othervfs):
     orig(self, othervfs)
     # copy lfs related options
     for k, v in othervfs.options.items():
         if k.startswith('lfs'):
             self.options[k] = v
     # also copy lfs blobstores. note: this can run before reposetup, so lfs
     # blobstore attributes are not always ready at this time.
     for name in ['lfslocalblobstore', 'lfsremoteblobstore']:
         if util.safehasattr(othervfs, name):
             setattr(self, name, getattr(othervfs, name))
 
 def hgclone(orig, ui, opts, *args, **kwargs):
     result = orig(ui, opts, *args, **kwargs)
 
     if result is not None:
         sourcerepo, destrepo = result
         repo = destrepo.local()
 
         # When cloning to a remote repo (like through SSH), no repo is available
         # from the peer.  Therefore the hgrc can't be updated.
         if not repo:
             return result
 
         # If lfs is required for this repo, permanently enable it locally
         if 'lfs' in repo.requirements:
             repo.vfs.append('hgrc',
                             util.tonativeeol('\n[extensions]\nlfs=\n'))
 
     return result
 
 def hgpostshare(orig, sourcerepo, destrepo, defaultpath=None):
     orig(sourcerepo, destrepo, defaultpath=defaultpath)
 
     # If lfs is required for this repo, permanently enable it locally
     if 'lfs' in destrepo.requirements:
         destrepo.vfs.append('hgrc', util.tonativeeol('\n[extensions]\nlfs=\n'))
 
 def _prefetchfiles(repo, revs, match):
     """Ensure that required LFS blobs are present, fetching them as a group if
     needed."""
     if not util.safehasattr(repo.svfs, 'lfslocalblobstore'):
         return
 
     pointers = []
     oids = set()
     localstore = repo.svfs.lfslocalblobstore
 
     for rev in revs:
         ctx = repo[rev]
         for f in ctx.walk(match):
             p = pointerfromctx(ctx, f)
             if p and p.oid() not in oids and not localstore.has(p.oid()):
                 p.filename = f
                 pointers.append(p)
                 oids.add(p.oid())
 
     if pointers:
         # Recalculating the repo store here allows 'paths.default' that is set
         # on the repo by a clone command to be used for the update.
         blobstore.remote(repo).readbatch(pointers, localstore)
 
 def _canskipupload(repo):
     # Skip if this hasn't been passed to reposetup()
     if not util.safehasattr(repo.svfs, 'lfsremoteblobstore'):
         return True
 
     # if remotestore is a null store, upload is a no-op and can be skipped
     return isinstance(repo.svfs.lfsremoteblobstore, blobstore._nullremote)
 
 def candownload(repo):
     # Skip if this hasn't been passed to reposetup()
     if not util.safehasattr(repo.svfs, 'lfsremoteblobstore'):
         return False
 
     # if remotestore is a null store, downloads will lead to nothing
     return not isinstance(repo.svfs.lfsremoteblobstore, blobstore._nullremote)
 
 def uploadblobsfromrevs(repo, revs):
     '''upload lfs blobs introduced by revs
 
     Note: also used by other extensions e. g. infinitepush. avoid renaming.
     '''
     if _canskipupload(repo):
         return
     pointers = extractpointers(repo, revs)
     uploadblobs(repo, pointers)
 
 def prepush(pushop):
     """Prepush hook.
 
     Read through the revisions to push, looking for filelog entries that can be
     deserialized into metadata so that we can block the push on their upload to
     the remote blobstore.
     """
     return uploadblobsfromrevs(pushop.repo, pushop.outgoing.missing)
 
 def push(orig, repo, remote, *args, **kwargs):
     """bail on push if the extension isn't enabled on remote when needed, and
     update the remote store based on the destination path."""
     if 'lfs' in repo.requirements:
         # If the remote peer is for a local repo, the requirement tests in the
         # base class method enforce lfs support.  Otherwise, some revisions in
         # this repo use lfs, and the remote repo needs the extension loaded.
         if not remote.local() and not remote.capable('lfs'):
             # This is a copy of the message in exchange.push() when requirements
             # are missing between local repos.
             m = _("required features are not supported in the destination: %s")
             raise error.Abort(m % 'lfs',
                               hint=_('enable the lfs extension on the server'))
 
         # Repositories where this extension is disabled won't have the field.
         # But if there's a requirement, then the extension must be loaded AND
         # there may be blobs to push.
         remotestore = repo.svfs.lfsremoteblobstore
         try:
             repo.svfs.lfsremoteblobstore = blobstore.remote(repo, remote.url())
             return orig(repo, remote, *args, **kwargs)
         finally:
             repo.svfs.lfsremoteblobstore = remotestore
     else:
         return orig(repo, remote, *args, **kwargs)
 
 def writenewbundle(orig, ui, repo, source, filename, bundletype, outgoing,
                    *args, **kwargs):
     """upload LFS blobs added by outgoing revisions on 'hg bundle'"""
     uploadblobsfromrevs(repo, outgoing.missing)
     return orig(ui, repo, source, filename, bundletype, outgoing, *args,
                 **kwargs)
 
 def extractpointers(repo, revs):
     """return a list of lfs pointers added by given revs"""
     repo.ui.debug('lfs: computing set of blobs to upload\n')
     pointers = {}
 
     makeprogress = repo.ui.makeprogress
     with makeprogress(_('lfs search'), _('changesets'), len(revs)) as progress:
         for r in revs:
             ctx = repo[r]
             for p in pointersfromctx(ctx).values():
                 pointers[p.oid()] = p
             progress.increment()
         return sorted(pointers.values())
 
 def pointerfromctx(ctx, f, removed=False):
     """return a pointer for the named file from the given changectx, or None if
     the file isn't LFS.
 
     Optionally, the pointer for a file deleted from the context can be returned.
     Since no such pointer is actually stored, and to distinguish from a non LFS
     file, this pointer is represented by an empty dict.
     """
     _ctx = ctx
     if f not in ctx:
         if not removed:
             return None
         if f in ctx.p1():
             _ctx = ctx.p1()
         elif f in ctx.p2():
             _ctx = ctx.p2()
         else:
             return None
     fctx = _ctx[f]
     if not _islfs(fctx.filelog(), fctx.filenode()):
         return None
     try:
         p = pointer.deserialize(fctx.rawdata())
         if ctx == _ctx:
             return p
         return {}
     except pointer.InvalidPointer as ex:
         raise error.Abort(_('lfs: corrupted pointer (%s@%s): %s\n')
                           % (f, short(_ctx.node()), ex))
 
 def pointersfromctx(ctx, removed=False):
     """return a dict {path: pointer} for given single changectx.
 
     If ``removed`` == True and the LFS file was removed from ``ctx``, the value
     stored for the path is an empty dict.
     """
     result = {}
     for f in ctx.files():
         p = pointerfromctx(ctx, f, removed=removed)
         if p is not None:
             result[f] = p
     return result
 
 def uploadblobs(repo, pointers):
     """upload given pointers from local blobstore"""
     if not pointers:
         return
 
     remoteblob = repo.svfs.lfsremoteblobstore
     remoteblob.writebatch(pointers, repo.svfs.lfslocalblobstore)
 
 def upgradefinishdatamigration(orig, ui, srcrepo, dstrepo, requirements):
     orig(ui, srcrepo, dstrepo, requirements)
 
     # Skip if this hasn't been passed to reposetup()
     if (util.safehasattr(srcrepo.svfs, 'lfslocalblobstore') and
         util.safehasattr(dstrepo.svfs, 'lfslocalblobstore')):
         srclfsvfs = srcrepo.svfs.lfslocalblobstore.vfs
         dstlfsvfs = dstrepo.svfs.lfslocalblobstore.vfs
 
         for dirpath, dirs, files in srclfsvfs.walk():
             for oid in files:
                 ui.write(_('copying lfs blob %s\n') % oid)
                 lfutil.link(srclfsvfs.join(oid), dstlfsvfs.join(oid))
 
 def upgraderequirements(orig, repo):
     reqs = orig(repo)
     if 'lfs' in repo.requirements:
         reqs.add('lfs')
     return reqs

mercurial/repository.py

@@ -1,1632 +1,1634
 # repository.py - Interfaces and base classes for repositories and peers.
 #
 # Copyright 2017 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 from .i18n import _
 from . import (
     error,
 )
 from .utils import (
     interfaceutil,
 )
 
 # When narrowing is finalized and no longer subject to format changes,
 # we should move this to just "narrow" or similar.
 NARROW_REQUIREMENT = 'narrowhg-experimental'
 
 # Local repository feature string.
 
 # Revlogs are being used for file storage.
 REPO_FEATURE_REVLOG_FILE_STORAGE = b'revlogfilestorage'
 # The storage part of the repository is shared from an external source.
 REPO_FEATURE_SHARED_STORAGE = b'sharedstore'
+# LFS supported for backing file storage.
+REPO_FEATURE_LFS = b'lfs'
 
 class ipeerconnection(interfaceutil.Interface):
     """Represents a "connection" to a repository.
 
     This is the base interface for representing a connection to a repository.
     It holds basic properties and methods applicable to all peer types.
 
     This is not a complete interface definition and should not be used
     outside of this module.
     """
     ui = interfaceutil.Attribute("""ui.ui instance""")
 
     def url():
         """Returns a URL string representing this peer.
 
         Currently, implementations expose the raw URL used to construct the
         instance. It may contain credentials as part of the URL. The
         expectations of the value aren't well-defined and this could lead to
         data leakage.
 
         TODO audit/clean consumers and more clearly define the contents of this
         value.
         """
 
     def local():
         """Returns a local repository instance.
 
         If the peer represents a local repository, returns an object that
         can be used to interface with it. Otherwise returns ``None``.
         """
 
     def peer():
         """Returns an object conforming to this interface.
 
         Most implementations will ``return self``.
         """
 
     def canpush():
         """Returns a boolean indicating if this peer can be pushed to."""
 
     def close():
         """Close the connection to this peer.
 
         This is called when the peer will no longer be used. Resources
         associated with the peer should be cleaned up.
         """
 
 class ipeercapabilities(interfaceutil.Interface):
     """Peer sub-interface related to capabilities."""
 
     def capable(name):
         """Determine support for a named capability.
 
         Returns ``False`` if capability not supported.
 
         Returns ``True`` if boolean capability is supported. Returns a string
         if capability support is non-boolean.
 
         Capability strings may or may not map to wire protocol capabilities.
         """
 
     def requirecap(name, purpose):
         """Require a capability to be present.
 
         Raises a ``CapabilityError`` if the capability isn't present.
         """
 
 class ipeercommands(interfaceutil.Interface):
     """Client-side interface for communicating over the wire protocol.
 
     This interface is used as a gateway to the Mercurial wire protocol.
     methods commonly call wire protocol commands of the same name.
     """
 
     def branchmap():
         """Obtain heads in named branches.
 
         Returns a dict mapping branch name to an iterable of nodes that are
         heads on that branch.
         """
 
     def capabilities():
         """Obtain capabilities of the peer.
 
         Returns a set of string capabilities.
         """
 
     def clonebundles():
         """Obtains the clone bundles manifest for the repo.
 
         Returns the manifest as unparsed bytes.
         """
 
     def debugwireargs(one, two, three=None, four=None, five=None):
         """Used to facilitate debugging of arguments passed over the wire."""
 
     def getbundle(source, **kwargs):
         """Obtain remote repository data as a bundle.
 
         This command is how the bulk of repository data is transferred from
         the peer to the local repository
 
         Returns a generator of bundle data.
         """
 
     def heads():
         """Determine all known head revisions in the peer.
 
         Returns an iterable of binary nodes.
         """
 
     def known(nodes):
         """Determine whether multiple nodes are known.
 
         Accepts an iterable of nodes whose presence to check for.
 
         Returns an iterable of booleans indicating of the corresponding node
         at that index is known to the peer.
         """
 
     def listkeys(namespace):
         """Obtain all keys in a pushkey namespace.
 
         Returns an iterable of key names.
         """
 
     def lookup(key):
         """Resolve a value to a known revision.
 
         Returns a binary node of the resolved revision on success.
         """
 
     def pushkey(namespace, key, old, new):
         """Set a value using the ``pushkey`` protocol.
 
         Arguments correspond to the pushkey namespace and key to operate on and
         the old and new values for that key.
 
         Returns a string with the peer result. The value inside varies by the
         namespace.
         """
 
     def stream_out():
         """Obtain streaming clone data.
 
         Successful result should be a generator of data chunks.
         """
 
     def unbundle(bundle, heads, url):
         """Transfer repository data to the peer.
 
         This is how the bulk of data during a push is transferred.
 
         Returns the integer number of heads added to the peer.
         """
 
 class ipeerlegacycommands(interfaceutil.Interface):
     """Interface for implementing support for legacy wire protocol commands.
 
     Wire protocol commands transition to legacy status when they are no longer
     used by modern clients. To facilitate identifying which commands are
     legacy, the interfaces are split.
     """
 
     def between(pairs):
         """Obtain nodes between pairs of nodes.
 
         ``pairs`` is an iterable of node pairs.
 
         Returns an iterable of iterables of nodes corresponding to each
         requested pair.
         """
 
     def branches(nodes):
         """Obtain ancestor changesets of specific nodes back to a branch point.
 
         For each requested node, the peer finds the first ancestor node that is
         a DAG root or is a merge.
 
         Returns an iterable of iterables with the resolved values for each node.
         """
 
     def changegroup(nodes, source):
         """Obtain a changegroup with data for descendants of specified nodes."""
 
     def changegroupsubset(bases, heads, source):
         pass
 
 class ipeercommandexecutor(interfaceutil.Interface):
     """Represents a mechanism to execute remote commands.
 
     This is the primary interface for requesting that wire protocol commands
     be executed. Instances of this interface are active in a context manager
     and have a well-defined lifetime. When the context manager exits, all
     outstanding requests are waited on.
     """
 
     def callcommand(name, args):
         """Request that a named command be executed.
 
         Receives the command name and a dictionary of command arguments.
 
         Returns a ``concurrent.futures.Future`` that will resolve to the
         result of that command request. That exact value is left up to
         the implementation and possibly varies by command.
 
         Not all commands can coexist with other commands in an executor
         instance: it depends on the underlying wire protocol transport being
         used and the command itself.
 
         Implementations MAY call ``sendcommands()`` automatically if the
         requested command can not coexist with other commands in this executor.
 
         Implementations MAY call ``sendcommands()`` automatically when the
         future's ``result()`` is called. So, consumers using multiple
         commands with an executor MUST ensure that ``result()`` is not called
         until all command requests have been issued.
         """
 
     def sendcommands():
         """Trigger submission of queued command requests.
 
         Not all transports submit commands as soon as they are requested to
         run. When called, this method forces queued command requests to be
         issued. It will no-op if all commands have already been sent.
 
         When called, no more new commands may be issued with this executor.
         """
 
     def close():
         """Signal that this command request is finished.
 
         When called, no more new commands may be issued. All outstanding
         commands that have previously been issued are waited on before
         returning. This not only includes waiting for the futures to resolve,
         but also waiting for all response data to arrive. In other words,
         calling this waits for all on-wire state for issued command requests
         to finish.
 
         When used as a context manager, this method is called when exiting the
         context manager.
 
         This method may call ``sendcommands()`` if there are buffered commands.
         """
 
 class ipeerrequests(interfaceutil.Interface):
     """Interface for executing commands on a peer."""
 
     def commandexecutor():
         """A context manager that resolves to an ipeercommandexecutor.
 
         The object this resolves to can be used to issue command requests
         to the peer.
 
         Callers should call its ``callcommand`` method to issue command
         requests.
 
         A new executor should be obtained for each distinct set of commands
         (possibly just a single command) that the consumer wants to execute
         as part of a single operation or round trip. This is because some
         peers are half-duplex and/or don't support persistent connections.
         e.g. in the case of HTTP peers, commands sent to an executor represent
         a single HTTP request. While some peers may support multiple command
         sends over the wire per executor, consumers need to code to the least
         capable peer. So it should be assumed that command executors buffer
         called commands until they are told to send them and that each
         command executor could result in a new connection or wire-level request
         being issued.
         """
 
 class ipeerbase(ipeerconnection, ipeercapabilities, ipeerrequests):
     """Unified interface for peer repositories.
 
     All peer instances must conform to this interface.
     """
 
 @interfaceutil.implementer(ipeerbase)
 class peer(object):
     """Base class for peer repositories."""
 
     def capable(self, name):
         caps = self.capabilities()
         if name in caps:
             return True
 
         name = '%s=' % name
         for cap in caps:
             if cap.startswith(name):
                 return cap[len(name):]
 
         return False
 
     def requirecap(self, name, purpose):
         if self.capable(name):
             return
 
         raise error.CapabilityError(
             _('cannot %s; remote repository does not support the %r '
               'capability') % (purpose, name))
 
 class iverifyproblem(interfaceutil.Interface):
     """Represents a problem with the integrity of the repository.
 
     Instances of this interface are emitted to describe an integrity issue
     with a repository (e.g. corrupt storage, missing data, etc).
 
     Instances are essentially messages associated with severity.
     """
     warning = interfaceutil.Attribute(
         """Message indicating a non-fatal problem.""")
 
     error = interfaceutil.Attribute(
         """Message indicating a fatal problem.""")
 
 class irevisiondelta(interfaceutil.Interface):
     """Represents a delta between one revision and another.
 
     Instances convey enough information to allow a revision to be exchanged
     with another repository.
 
     Instances represent the fulltext revision data or a delta against
     another revision. Therefore the ``revision`` and ``delta`` attributes
     are mutually exclusive.
 
     Typically used for changegroup generation.
     """
 
     node = interfaceutil.Attribute(
         """20 byte node of this revision.""")
 
     p1node = interfaceutil.Attribute(
         """20 byte node of 1st parent of this revision.""")
 
     p2node = interfaceutil.Attribute(
         """20 byte node of 2nd parent of this revision.""")
 
     linknode = interfaceutil.Attribute(
         """20 byte node of the changelog revision this node is linked to.""")
 
     flags = interfaceutil.Attribute(
         """2 bytes of integer flags that apply to this revision.""")
 
     basenode = interfaceutil.Attribute(
         """20 byte node of the revision this data is a delta against.
 
         ``nullid`` indicates that the revision is a full revision and not
         a delta.
         """)
 
     baserevisionsize = interfaceutil.Attribute(
         """Size of base revision this delta is against.
 
         May be ``None`` if ``basenode`` is ``nullid``.
         """)
 
     revision = interfaceutil.Attribute(
         """Raw fulltext of revision data for this node.""")
 
     delta = interfaceutil.Attribute(
         """Delta between ``basenode`` and ``node``.
 
         Stored in the bdiff delta format.
         """)
 
 class irevisiondeltarequest(interfaceutil.Interface):
     """Represents a request to generate an ``irevisiondelta``."""
 
     node = interfaceutil.Attribute(
         """20 byte node of revision being requested.""")
 
     p1node = interfaceutil.Attribute(
         """20 byte node of 1st parent of revision.""")
 
     p2node = interfaceutil.Attribute(
         """20 byte node of 2nd parent of revision.""")
 
     linknode = interfaceutil.Attribute(
         """20 byte node to store in ``linknode`` attribute.""")
 
     basenode = interfaceutil.Attribute(
         """Base revision that delta should be generated against.
 
         If ``nullid``, the derived ``irevisiondelta`` should have its
         ``revision`` field populated and no delta should be generated.
 
         If ``None``, the delta may be generated against any revision that
         is an ancestor of this revision. Or a full revision may be used.
 
         If any other value, the delta should be produced against that
         revision.
         """)
 
     ellipsis = interfaceutil.Attribute(
         """Boolean on whether the ellipsis flag should be set.""")
 
 class ifilerevisionssequence(interfaceutil.Interface):
     """Contains index data for all revisions of a file.
 
     Types implementing this behave like lists of tuples. The index
     in the list corresponds to the revision number. The values contain
     index metadata.
 
     The *null* revision (revision number -1) is always the last item
     in the index.
     """
 
     def __len__():
         """The total number of revisions."""
 
     def __getitem__(rev):
         """Returns the object having a specific revision number.
 
         Returns an 8-tuple with the following fields:
 
         offset+flags
            Contains the offset and flags for the revision. 64-bit unsigned
            integer where first 6 bytes are the offset and the next 2 bytes
            are flags. The offset can be 0 if it is not used by the store.
         compressed size
             Size of the revision data in the store. It can be 0 if it isn't
             needed by the store.
         uncompressed size
             Fulltext size. It can be 0 if it isn't needed by the store.
         base revision
             Revision number of revision the delta for storage is encoded
             against. -1 indicates not encoded against a base revision.
         link revision
             Revision number of changelog revision this entry is related to.
         p1 revision
             Revision number of 1st parent. -1 if no 1st parent.
         p2 revision
             Revision number of 2nd parent. -1 if no 1st parent.
         node
             Binary node value for this revision number.
 
         Negative values should index off the end of the sequence. ``-1``
         should return the null revision. ``-2`` should return the most
         recent revision.
         """
 
     def __contains__(rev):
         """Whether a revision number exists."""
 
     def insert(self, i, entry):
         """Add an item to the index at specific revision."""
 
 class ifileindex(interfaceutil.Interface):
     """Storage interface for index data of a single file.
 
     File storage data is divided into index metadata and data storage.
     This interface defines the index portion of the interface.
 
     The index logically consists of:
 
     * A mapping between revision numbers and nodes.
     * DAG data (storing and querying the relationship between nodes).
     * Metadata to facilitate storage.
     """
     index = interfaceutil.Attribute(
         """An ``ifilerevisionssequence`` instance.""")
 
     def __len__():
         """Obtain the number of revisions stored for this file."""
 
     def __iter__():
         """Iterate over revision numbers for this file."""
 
     def revs(start=0, stop=None):
         """Iterate over revision numbers for this file, with control."""
 
     def parents(node):
         """Returns a 2-tuple of parent nodes for a revision.
 
         Values will be ``nullid`` if the parent is empty.
         """
 
     def parentrevs(rev):
         """Like parents() but operates on revision numbers."""
 
     def rev(node):
         """Obtain the revision number given a node.
 
         Raises ``error.LookupError`` if the node is not known.
         """
 
     def node(rev):
         """Obtain the node value given a revision number.
 
         Raises ``IndexError`` if the node is not known.
         """
 
     def lookup(node):
         """Attempt to resolve a value to a node.
 
         Value can be a binary node, hex node, revision number, or a string
         that can be converted to an integer.
 
         Raises ``error.LookupError`` if a node could not be resolved.
         """
 
     def linkrev(rev):
         """Obtain the changeset revision number a revision is linked to."""
 
     def flags(rev):
         """Obtain flags used to affect storage of a revision."""
 
     def iscensored(rev):
         """Return whether a revision's content has been censored."""
 
     def commonancestorsheads(node1, node2):
         """Obtain an iterable of nodes containing heads of common ancestors.
 
         See ``ancestor.commonancestorsheads()``.
         """
 
     def descendants(revs):
         """Obtain descendant revision numbers for a set of revision numbers.
 
         If ``nullrev`` is in the set, this is equivalent to ``revs()``.
         """
 
     def heads(start=None, stop=None):
         """Obtain a list of nodes that are DAG heads, with control.
 
         The set of revisions examined can be limited by specifying
         ``start`` and ``stop``. ``start`` is a node. ``stop`` is an
         iterable of nodes. DAG traversal starts at earlier revision
         ``start`` and iterates forward until any node in ``stop`` is
         encountered.
         """
 
     def children(node):
         """Obtain nodes that are children of a node.
 
         Returns a list of nodes.
         """
 
     def deltaparent(rev):
         """"Return the revision that is a suitable parent to delta against."""
 
 class ifiledata(interfaceutil.Interface):
     """Storage interface for data storage of a specific file.
 
     This complements ``ifileindex`` and provides an interface for accessing
     data for a tracked file.
     """
     def rawsize(rev):
         """The size of the fulltext data for a revision as stored."""
 
     def size(rev):
         """Obtain the fulltext size of file data.
 
         Any metadata is excluded from size measurements. Use ``rawsize()`` if
         metadata size is important.
         """
 
     def checkhash(fulltext, node, p1=None, p2=None, rev=None):
         """Validate the stored hash of a given fulltext and node.
 
         Raises ``error.StorageError`` is hash validation fails.
         """
 
     def revision(node, raw=False):
         """"Obtain fulltext data for a node.
 
         By default, any storage transformations are applied before the data
         is returned. If ``raw`` is True, non-raw storage transformations
         are not applied.
 
         The fulltext data may contain a header containing metadata. Most
         consumers should use ``read()`` to obtain the actual file data.
         """
 
     def read(node):
         """Resolve file fulltext data.
 
         This is similar to ``revision()`` except any metadata in the data
         headers is stripped.
         """
 
     def renamed(node):
         """Obtain copy metadata for a node.
 
         Returns ``False`` if no copy metadata is stored or a 2-tuple of
         (path, node) from which this revision was copied.
         """
 
     def cmp(node, fulltext):
         """Compare fulltext to another revision.
 
         Returns True if the fulltext is different from what is stored.
 
         This takes copy metadata into account.
 
         TODO better document the copy metadata and censoring logic.
         """
 
     def revdiff(rev1, rev2):
         """Obtain a delta between two revision numbers.
 
         Operates on raw data in the store (``revision(node, raw=True)``).
 
         The returned data is the result of ``bdiff.bdiff`` on the raw
         revision data.
         """
 
     def emitrevisiondeltas(requests):
         """Produce ``irevisiondelta`` from ``irevisiondeltarequest``s.
 
         Given an iterable of objects conforming to the ``irevisiondeltarequest``
         interface, emits objects conforming to the ``irevisiondelta``
         interface.
 
         This method is a generator.
 
         ``irevisiondelta`` should be emitted in the same order of
         ``irevisiondeltarequest`` that was passed in.
 
         The emitted objects MUST conform by the results of
         ``irevisiondeltarequest``. Namely, they must respect any requests
         for building a delta from a specific ``basenode`` if defined.
 
         When sending deltas, implementations must take into account whether
         the client has the base delta before encoding a delta against that
         revision. A revision encountered previously in ``requests`` is
         always a suitable base revision. An example of a bad delta is a delta
         against a non-ancestor revision. Another example of a bad delta is a
         delta against a censored revision.
         """
 
 class ifilemutation(interfaceutil.Interface):
     """Storage interface for mutation events of a tracked file."""
 
     def add(filedata, meta, transaction, linkrev, p1, p2):
         """Add a new revision to the store.
 
         Takes file data, dictionary of metadata, a transaction, linkrev,
         and parent nodes.
 
         Returns the node that was added.
 
         May no-op if a revision matching the supplied data is already stored.
         """
 
     def addrevision(revisiondata, transaction, linkrev, p1, p2, node=None,
                     flags=0, cachedelta=None):
         """Add a new revision to the store.
 
         This is similar to ``add()`` except it operates at a lower level.
 
         The data passed in already contains a metadata header, if any.
 
         ``node`` and ``flags`` can be used to define the expected node and
         the flags to use with storage.
 
         ``add()`` is usually called when adding files from e.g. the working
         directory. ``addrevision()`` is often called by ``add()`` and for
         scenarios where revision data has already been computed, such as when
         applying raw data from a peer repo.
         """
 
     def addgroup(deltas, linkmapper, transaction, addrevisioncb=None):
         """Process a series of deltas for storage.
 
         ``deltas`` is an iterable of 7-tuples of
         (node, p1, p2, linknode, deltabase, delta, flags) defining revisions
         to add.
 
         The ``delta`` field contains ``mpatch`` data to apply to a base
         revision, identified by ``deltabase``. The base node can be
         ``nullid``, in which case the header from the delta can be ignored
         and the delta used as the fulltext.
 
         ``addrevisioncb`` should be called for each node as it is committed.
 
         Returns a list of nodes that were processed. A node will be in the list
         even if it existed in the store previously.
         """
 
     def censorrevision(tr, node, tombstone=b''):
         """Remove the content of a single revision.
 
         The specified ``node`` will have its content purged from storage.
         Future attempts to access the revision data for this node will
         result in failure.
 
         A ``tombstone`` message can optionally be stored. This message may be
         displayed to users when they attempt to access the missing revision
         data.
 
         Storage backends may have stored deltas against the previous content
         in this revision. As part of censoring a revision, these storage
         backends are expected to rewrite any internally stored deltas such
         that they no longer reference the deleted content.
         """
 
     def getstrippoint(minlink):
         """Find the minimum revision that must be stripped to strip a linkrev.
 
         Returns a 2-tuple containing the minimum revision number and a set
         of all revisions numbers that would be broken by this strip.
 
         TODO this is highly revlog centric and should be abstracted into
         a higher-level deletion API. ``repair.strip()`` relies on this.
         """
 
     def strip(minlink, transaction):
         """Remove storage of items starting at a linkrev.
 
         This uses ``getstrippoint()`` to determine the first node to remove.
         Then it effectively truncates storage for all revisions after that.
 
         TODO this is highly revlog centric and should be abstracted into a
         higher-level deletion API.
         """
 
 class ifilestorage(ifileindex, ifiledata, ifilemutation):
     """Complete storage interface for a single tracked file."""
 
     _generaldelta = interfaceutil.Attribute(
         """Whether deltas can be against any parent revision.
 
         TODO this is used by changegroup code and it could probably be
         folded into another API.
         """)
 
     def files():
         """Obtain paths that are backing storage for this file.
 
         TODO this is used heavily by verify code and there should probably
         be a better API for that.
         """
 
     def verifyintegrity(state):
         """Verifies the integrity of file storage.
 
         ``state`` is a dict holding state of the verifier process. It can be
         used to communicate data between invocations of multiple storage
         primitives.
 
         The method yields objects conforming to the ``iverifyproblem``
         interface.
         """
 
 class idirs(interfaceutil.Interface):
     """Interface representing a collection of directories from paths.
 
     This interface is essentially a derived data structure representing
     directories from a collection of paths.
     """
 
     def addpath(path):
         """Add a path to the collection.
 
         All directories in the path will be added to the collection.
         """
 
     def delpath(path):
         """Remove a path from the collection.
 
         If the removal was the last path in a particular directory, the
         directory is removed from the collection.
         """
 
     def __iter__():
         """Iterate over the directories in this collection of paths."""
 
     def __contains__(path):
         """Whether a specific directory is in this collection."""
 
 class imanifestdict(interfaceutil.Interface):
     """Interface representing a manifest data structure.
 
     A manifest is effectively a dict mapping paths to entries. Each entry
     consists of a binary node and extra flags affecting that entry.
     """
 
     def __getitem__(path):
         """Returns the binary node value for a path in the manifest.
 
         Raises ``KeyError`` if the path does not exist in the manifest.
 
         Equivalent to ``self.find(path)[0]``.
         """
 
     def find(path):
         """Returns the entry for a path in the manifest.
 
         Returns a 2-tuple of (node, flags).
 
         Raises ``KeyError`` if the path does not exist in the manifest.
         """
 
     def __len__():
         """Return the number of entries in the manifest."""
 
     def __nonzero__():
         """Returns True if the manifest has entries, False otherwise."""
 
     __bool__ = __nonzero__
 
     def __setitem__(path, node):
         """Define the node value for a path in the manifest.
 
         If the path is already in the manifest, its flags will be copied to
         the new entry.
         """
 
     def __contains__(path):
         """Whether a path exists in the manifest."""
 
     def __delitem__(path):
         """Remove a path from the manifest.
 
         Raises ``KeyError`` if the path is not in the manifest.
         """
 
     def __iter__():
         """Iterate over paths in the manifest."""
 
     def iterkeys():
         """Iterate over paths in the manifest."""
 
     def keys():
         """Obtain a list of paths in the manifest."""
 
     def filesnotin(other, match=None):
         """Obtain the set of paths in this manifest but not in another.
 
         ``match`` is an optional matcher function to be applied to both
         manifests.
 
         Returns a set of paths.
         """
 
     def dirs():
         """Returns an object implementing the ``idirs`` interface."""
 
     def hasdir(dir):
         """Returns a bool indicating if a directory is in this manifest."""
 
     def matches(match):
         """Generate a new manifest filtered through a matcher.
 
         Returns an object conforming to the ``imanifestdict`` interface.
         """
 
     def walk(match):
         """Generator of paths in manifest satisfying a matcher.
 
         This is equivalent to ``self.matches(match).iterkeys()`` except a new
         manifest object is not created.
 
         If the matcher has explicit files listed and they don't exist in
         the manifest, ``match.bad()`` is called for each missing file.
         """
 
     def diff(other, match=None, clean=False):
         """Find differences between this manifest and another.
 
         This manifest is compared to ``other``.
 
         If ``match`` is provided, the two manifests are filtered against this
         matcher and only entries satisfying the matcher are compared.
 
         If ``clean`` is True, unchanged files are included in the returned
         object.
 
         Returns a dict with paths as keys and values of 2-tuples of 2-tuples of
         the form ``((node1, flag1), (node2, flag2))`` where ``(node1, flag1)``
         represents the node and flags for this manifest and ``(node2, flag2)``
         are the same for the other manifest.
         """
 
     def setflag(path, flag):
         """Set the flag value for a given path.
 
         Raises ``KeyError`` if the path is not already in the manifest.
         """
 
     def get(path, default=None):
         """Obtain the node value for a path or a default value if missing."""
 
     def flags(path, default=''):
         """Return the flags value for a path or a default value if missing."""
 
     def copy():
         """Return a copy of this manifest."""
 
     def items():
         """Returns an iterable of (path, node) for items in this manifest."""
 
     def iteritems():
         """Identical to items()."""
 
     def iterentries():
         """Returns an iterable of (path, node, flags) for this manifest.
 
         Similar to ``iteritems()`` except items are a 3-tuple and include
         flags.
         """
 
     def text():
         """Obtain the raw data representation for this manifest.
 
         Result is used to create a manifest revision.
         """
 
     def fastdelta(base, changes):
         """Obtain a delta between this manifest and another given changes.
 
         ``base`` in the raw data representation for another manifest.
 
         ``changes`` is an iterable of ``(path, to_delete)``.
 
         Returns a 2-tuple containing ``bytearray(self.text())`` and the
         delta between ``base`` and this manifest.
         """
 
 class imanifestrevisionbase(interfaceutil.Interface):
     """Base interface representing a single revision of a manifest.
 
     Should not be used as a primary interface: should always be inherited
     as part of a larger interface.
     """
 
     def new():
         """Obtain a new manifest instance.
 
         Returns an object conforming to the ``imanifestrevisionwritable``
         interface. The instance will be associated with the same
         ``imanifestlog`` collection as this instance.
         """
 
     def copy():
         """Obtain a copy of this manifest instance.
 
         Returns an object conforming to the ``imanifestrevisionwritable``
         interface. The instance will be associated with the same
         ``imanifestlog`` collection as this instance.
         """
 
     def read():
         """Obtain the parsed manifest data structure.
 
         The returned object conforms to the ``imanifestdict`` interface.
         """
 
 class imanifestrevisionstored(imanifestrevisionbase):
     """Interface representing a manifest revision committed to storage."""
 
     def node():
         """The binary node for this manifest."""
 
     parents = interfaceutil.Attribute(
         """List of binary nodes that are parents for this manifest revision."""
     )
 
     def readdelta(shallow=False):
         """Obtain the manifest data structure representing changes from parent.
 
         This manifest is compared to its 1st parent. A new manifest representing
         those differences is constructed.
 
         The returned object conforms to the ``imanifestdict`` interface.
         """
 
     def readfast(shallow=False):
         """Calls either ``read()`` or ``readdelta()``.
 
         The faster of the two options is called.
         """
 
     def find(key):
         """Calls self.read().find(key)``.
 
         Returns a 2-tuple of ``(node, flags)`` or raises ``KeyError``.
         """
 
 class imanifestrevisionwritable(imanifestrevisionbase):
     """Interface representing a manifest revision that can be committed."""
 
     def write(transaction, linkrev, p1node, p2node, added, removed, match=None):
         """Add this revision to storage.
 
         Takes a transaction object, the changeset revision number it will
         be associated with, its parent nodes, and lists of added and
         removed paths.
 
         If match is provided, storage can choose not to inspect or write out
         items that do not match. Storage is still required to be able to provide
         the full manifest in the future for any directories written (these
         manifests should not be "narrowed on disk").
 
         Returns the binary node of the created revision.
         """
 
 class imanifeststorage(interfaceutil.Interface):
     """Storage interface for manifest data."""
 
     tree = interfaceutil.Attribute(
         """The path to the directory this manifest tracks.
 
         The empty bytestring represents the root manifest.
         """)
 
     index = interfaceutil.Attribute(
         """An ``ifilerevisionssequence`` instance.""")
 
     indexfile = interfaceutil.Attribute(
         """Path of revlog index file.
 
         TODO this is revlog specific and should not be exposed.
         """)
 
     opener = interfaceutil.Attribute(
         """VFS opener to use to access underlying files used for storage.
 
         TODO this is revlog specific and should not be exposed.
         """)
 
     version = interfaceutil.Attribute(
         """Revlog version number.
 
         TODO this is revlog specific and should not be exposed.
         """)
 
     _generaldelta = interfaceutil.Attribute(
         """Whether generaldelta storage is being used.
 
         TODO this is revlog specific and should not be exposed.
         """)
 
     fulltextcache = interfaceutil.Attribute(
         """Dict with cache of fulltexts.
 
         TODO this doesn't feel appropriate for the storage interface.
         """)
 
     def __len__():
         """Obtain the number of revisions stored for this manifest."""
 
     def __iter__():
         """Iterate over revision numbers for this manifest."""
 
     def rev(node):
         """Obtain the revision number given a binary node.
 
         Raises ``error.LookupError`` if the node is not known.
         """
 
     def node(rev):
         """Obtain the node value given a revision number.
 
         Raises ``error.LookupError`` if the revision is not known.
         """
 
     def lookup(value):
         """Attempt to resolve a value to a node.
 
         Value can be a binary node, hex node, revision number, or a bytes
         that can be converted to an integer.
 
         Raises ``error.LookupError`` if a ndoe could not be resolved.
 
         TODO this is only used by debug* commands and can probably be deleted
         easily.
         """
 
     def parents(node):
         """Returns a 2-tuple of parent nodes for a node.
 
         Values will be ``nullid`` if the parent is empty.
         """
 
     def parentrevs(rev):
         """Like parents() but operates on revision numbers."""
 
     def linkrev(rev):
         """Obtain the changeset revision number a revision is linked to."""
 
     def revision(node, _df=None, raw=False):
         """Obtain fulltext data for a node."""
 
     def revdiff(rev1, rev2):
         """Obtain a delta between two revision numbers.
 
         The returned data is the result of ``bdiff.bdiff()`` on the raw
         revision data.
         """
 
     def cmp(node, fulltext):
         """Compare fulltext to another revision.
 
         Returns True if the fulltext is different from what is stored.
         """
 
     def emitrevisiondeltas(requests):
         """Produce ``irevisiondelta`` from ``irevisiondeltarequest``s.
 
         See the documentation for ``ifiledata`` for more.
         """
 
     def addgroup(deltas, linkmapper, transaction, addrevisioncb=None):
         """Process a series of deltas for storage.
 
         See the documentation in ``ifilemutation`` for more.
         """
 
     def getstrippoint(minlink):
         """Find minimum revision that must be stripped to strip a linkrev.
 
         See the documentation in ``ifilemutation`` for more.
         """
 
     def strip(minlink, transaction):
         """Remove storage of items starting at a linkrev.
 
         See the documentation in ``ifilemutation`` for more.
         """
 
     def checksize():
         """Obtain the expected sizes of backing files.
 
         TODO this is used by verify and it should not be part of the interface.
         """
 
     def files():
         """Obtain paths that are backing storage for this manifest.
 
         TODO this is used by verify and there should probably be a better API
         for this functionality.
         """
 
     def deltaparent(rev):
         """Obtain the revision that a revision is delta'd against.
 
         TODO delta encoding is an implementation detail of storage and should
         not be exposed to the storage interface.
         """
 
     def clone(tr, dest, **kwargs):
         """Clone this instance to another."""
 
     def clearcaches(clear_persisted_data=False):
         """Clear any caches associated with this instance."""
 
     def dirlog(d):
         """Obtain a manifest storage instance for a tree."""
 
     def add(m, transaction, link, p1, p2, added, removed, readtree=None,
             match=None):
         """Add a revision to storage.
 
         ``m`` is an object conforming to ``imanifestdict``.
 
         ``link`` is the linkrev revision number.
 
         ``p1`` and ``p2`` are the parent revision numbers.
 
         ``added`` and ``removed`` are iterables of added and removed paths,
         respectively.
 
         ``readtree`` is a function that can be used to read the child tree(s)
         when recursively writing the full tree structure when using
         treemanifets.
 
         ``match`` is a matcher that can be used to hint to storage that not all
         paths must be inspected; this is an optimization and can be safely
         ignored. Note that the storage must still be able to reproduce a full
         manifest including files that did not match.
         """
 
 class imanifestlog(interfaceutil.Interface):
     """Interface representing a collection of manifest snapshots.
 
     Represents the root manifest in a repository.
 
     Also serves as a means to access nested tree manifests and to cache
     tree manifests.
     """
 
     def __getitem__(node):
         """Obtain a manifest instance for a given binary node.
 
         Equivalent to calling ``self.get('', node)``.
 
         The returned object conforms to the ``imanifestrevisionstored``
         interface.
         """
 
     def get(tree, node, verify=True):
         """Retrieve the manifest instance for a given directory and binary node.
 
         ``node`` always refers to the node of the root manifest (which will be
         the only manifest if flat manifests are being used).
 
         If ``tree`` is the empty string, the root manifest is returned.
         Otherwise the manifest for the specified directory will be returned
         (requires tree manifests).
 
         If ``verify`` is True, ``LookupError`` is raised if the node is not
         known.
 
         The returned object conforms to the ``imanifestrevisionstored``
         interface.
         """
 
     def getstorage(tree):
         """Retrieve an interface to storage for a particular tree.
 
         If ``tree`` is the empty bytestring, storage for the root manifest will
         be returned. Otherwise storage for a tree manifest is returned.
 
         TODO formalize interface for returned object.
         """
 
     def clearcaches():
         """Clear caches associated with this collection."""
 
     def rev(node):
         """Obtain the revision number for a binary node.
 
         Raises ``error.LookupError`` if the node is not known.
         """
 
 class ilocalrepositoryfilestorage(interfaceutil.Interface):
     """Local repository sub-interface providing access to tracked file storage.
 
     This interface defines how a repository accesses storage for a single
     tracked file path.
     """
 
     def file(f):
         """Obtain a filelog for a tracked path.
 
         The returned type conforms to the ``ifilestorage`` interface.
         """
 
 class ilocalrepositorymain(interfaceutil.Interface):
     """Main interface for local repositories.
 
     This currently captures the reality of things - not how things should be.
     """
 
     supportedformats = interfaceutil.Attribute(
         """Set of requirements that apply to stream clone.
 
         This is actually a class attribute and is shared among all instances.
         """)
 
     supported = interfaceutil.Attribute(
         """Set of requirements that this repo is capable of opening.""")
 
     requirements = interfaceutil.Attribute(
         """Set of requirements this repo uses.""")
 
     features = interfaceutil.Attribute(
         """Set of "features" this repository supports.
 
         A "feature" is a loosely-defined term. It can refer to a feature
         in the classical sense or can describe an implementation detail
         of the repository. For example, a ``readonly`` feature may denote
         the repository as read-only. Or a ``revlogfilestore`` feature may
         denote that the repository is using revlogs for file storage.
 
         The intent of features is to provide a machine-queryable mechanism
         for repo consumers to test for various repository characteristics.
 
         Features are similar to ``requirements``. The main difference is that
         requirements are stored on-disk and represent requirements to open the
         repository. Features are more run-time capabilities of the repository
         and more granular capabilities (which may be derived from requirements).
         """)
 
     filtername = interfaceutil.Attribute(
         """Name of the repoview that is active on this repo.""")
 
     wvfs = interfaceutil.Attribute(
         """VFS used to access the working directory.""")
 
     vfs = interfaceutil.Attribute(
         """VFS rooted at the .hg directory.
 
         Used to access repository data not in the store.
         """)
 
     svfs = interfaceutil.Attribute(
         """VFS rooted at the store.
 
         Used to access repository data in the store. Typically .hg/store.
         But can point elsewhere if the store is shared.
         """)
 
     root = interfaceutil.Attribute(
         """Path to the root of the working directory.""")
 
     path = interfaceutil.Attribute(
         """Path to the .hg directory.""")
 
     origroot = interfaceutil.Attribute(
         """The filesystem path that was used to construct the repo.""")
 
     auditor = interfaceutil.Attribute(
         """A pathauditor for the working directory.
 
         This checks if a path refers to a nested repository.
 
         Operates on the filesystem.
         """)
 
     nofsauditor = interfaceutil.Attribute(
         """A pathauditor for the working directory.
 
         This is like ``auditor`` except it doesn't do filesystem checks.
         """)
 
     baseui = interfaceutil.Attribute(
         """Original ui instance passed into constructor.""")
 
     ui = interfaceutil.Attribute(
         """Main ui instance for this instance.""")
 
     sharedpath = interfaceutil.Attribute(
         """Path to the .hg directory of the repo this repo was shared from.""")
 
     store = interfaceutil.Attribute(
         """A store instance.""")
 
     spath = interfaceutil.Attribute(
         """Path to the store.""")
 
     sjoin = interfaceutil.Attribute(
         """Alias to self.store.join.""")
 
     cachevfs = interfaceutil.Attribute(
         """A VFS used to access the cache directory.
 
         Typically .hg/cache.
         """)
 
     filteredrevcache = interfaceutil.Attribute(
         """Holds sets of revisions to be filtered.""")
 
     names = interfaceutil.Attribute(
         """A ``namespaces`` instance.""")
 
     def close():
         """Close the handle on this repository."""
 
     def peer():
         """Obtain an object conforming to the ``peer`` interface."""
 
     def unfiltered():
         """Obtain an unfiltered/raw view of this repo."""
 
     def filtered(name, visibilityexceptions=None):
         """Obtain a named view of this repository."""
 
     obsstore = interfaceutil.Attribute(
         """A store of obsolescence data.""")
 
     changelog = interfaceutil.Attribute(
         """A handle on the changelog revlog.""")
 
     manifestlog = interfaceutil.Attribute(
         """An instance conforming to the ``imanifestlog`` interface.
 
         Provides access to manifests for the repository.
         """)
 
     dirstate = interfaceutil.Attribute(
         """Working directory state.""")
 
     narrowpats = interfaceutil.Attribute(
         """Matcher patterns for this repository's narrowspec.""")
 
     def narrowmatch():
         """Obtain a matcher for the narrowspec."""
 
     def setnarrowpats(newincludes, newexcludes):
         """Define the narrowspec for this repository."""
 
     def __getitem__(changeid):
         """Try to resolve a changectx."""
 
     def __contains__(changeid):
         """Whether a changeset exists."""
 
     def __nonzero__():
         """Always returns True."""
         return True
 
     __bool__ = __nonzero__
 
     def __len__():
         """Returns the number of changesets in the repo."""
 
     def __iter__():
         """Iterate over revisions in the changelog."""
 
     def revs(expr, *args):
         """Evaluate a revset.
 
         Emits revisions.
         """
 
     def set(expr, *args):
         """Evaluate a revset.
 
         Emits changectx instances.
         """
 
     def anyrevs(specs, user=False, localalias=None):
         """Find revisions matching one of the given revsets."""
 
     def url():
         """Returns a string representing the location of this repo."""
 
     def hook(name, throw=False, **args):
         """Call a hook."""
 
     def tags():
         """Return a mapping of tag to node."""
 
     def tagtype(tagname):
         """Return the type of a given tag."""
 
     def tagslist():
         """Return a list of tags ordered by revision."""
 
     def nodetags(node):
         """Return the tags associated with a node."""
 
     def nodebookmarks(node):
         """Return the list of bookmarks pointing to the specified node."""
 
     def branchmap():
         """Return a mapping of branch to heads in that branch."""
 
     def revbranchcache():
         pass
 
     def branchtip(branchtip, ignoremissing=False):
         """Return the tip node for a given branch."""
 
     def lookup(key):
         """Resolve the node for a revision."""
 
     def lookupbranch(key):
         """Look up the branch name of the given revision or branch name."""
 
     def known(nodes):
         """Determine whether a series of nodes is known.
 
         Returns a list of bools.
         """
 
     def local():
         """Whether the repository is local."""
         return True
 
     def publishing():
         """Whether the repository is a publishing repository."""
 
     def cancopy():
         pass
 
     def shared():
         """The type of shared repository or None."""
 
     def wjoin(f, *insidef):
         """Calls self.vfs.reljoin(self.root, f, *insidef)"""
 
     def setparents(p1, p2):
         """Set the parent nodes of the working directory."""
 
     def filectx(path, changeid=None, fileid=None):
         """Obtain a filectx for the given file revision."""
 
     def getcwd():
         """Obtain the current working directory from the dirstate."""
 
     def pathto(f, cwd=None):
         """Obtain the relative path to a file."""
 
     def adddatafilter(name, fltr):
         pass
 
     def wread(filename):
         """Read a file from wvfs, using data filters."""
 
     def wwrite(filename, data, flags, backgroundclose=False, **kwargs):
         """Write data to a file in the wvfs, using data filters."""
 
     def wwritedata(filename, data):
         """Resolve data for writing to the wvfs, using data filters."""
 
     def currenttransaction():
         """Obtain the current transaction instance or None."""
 
     def transaction(desc, report=None):
         """Open a new transaction to write to the repository."""
 
     def undofiles():
         """Returns a list of (vfs, path) for files to undo transactions."""
 
     def recover():
         """Roll back an interrupted transaction."""
 
     def rollback(dryrun=False, force=False):
         """Undo the last transaction.
 
         DANGEROUS.
         """
 
     def updatecaches(tr=None, full=False):
         """Warm repo caches."""
 
     def invalidatecaches():
         """Invalidate cached data due to the repository mutating."""
 
     def invalidatevolatilesets():
         pass
 
     def invalidatedirstate():
         """Invalidate the dirstate."""
 
     def invalidate(clearfilecache=False):
         pass
 
     def invalidateall():
         pass
 
     def lock(wait=True):
         """Lock the repository store and return a lock instance."""
 
     def wlock(wait=True):
         """Lock the non-store parts of the repository."""
 
     def currentwlock():
         """Return the wlock if it's held or None."""
 
     def checkcommitpatterns(wctx, vdirs, match, status, fail):
         pass
 
     def commit(text='', user=None, date=None, match=None, force=False,
                editor=False, extra=None):
         """Add a new revision to the repository."""
 
     def commitctx(ctx, error=False):
         """Commit a commitctx instance to the repository."""
 
     def destroying():
         """Inform the repository that nodes are about to be destroyed."""
 
     def destroyed():
         """Inform the repository that nodes have been destroyed."""
 
     def status(node1='.', node2=None, match=None, ignored=False,
                clean=False, unknown=False, listsubrepos=False):
         """Convenience method to call repo[x].status()."""
 
     def addpostdsstatus(ps):
         pass
 
     def postdsstatus():
         pass
 
     def clearpostdsstatus():
         pass
 
     def heads(start=None):
         """Obtain list of nodes that are DAG heads."""
 
     def branchheads(branch=None, start=None, closed=False):
         pass
 
     def branches(nodes):
         pass
 
     def between(pairs):
         pass
 
     def checkpush(pushop):
         pass
 
     prepushoutgoinghooks = interfaceutil.Attribute(
         """util.hooks instance.""")
 
     def pushkey(namespace, key, old, new):
         pass
 
     def listkeys(namespace):
         pass
 
     def debugwireargs(one, two, three=None, four=None, five=None):
         pass
 
     def savecommitmessage(text):
         pass
 
 class completelocalrepository(ilocalrepositorymain,
                               ilocalrepositoryfilestorage):
     """Complete interface for a local repository."""