bundle2: don't use seekable bundle2 parts by default (issue5691)

The last commit removed the last use of the bundle2 part seek() API
in the generic bundle2 part iteration code. This means we can now
switch to using unseekable bundle2 parts by default and have the
special consumers that actually need the behavior request it.

This commit changes unbundle20.iterparts() to expose non-seekable
unbundlepart instances by default. If seekable parts are needed,
callers can pass "seekable=True." The bundlerepo class needs
seekable parts, so it does this.

The interrupt handler is also changed to use a regular unbundlepart.
So, by default, all consumers except bundlerepo will see unseekable
parts.

Because the behavior of the iterparts() benchmark changed, we add
a variation to test seekable parts vs unseekable parts. And because
parts no longer have seek() unless "seekable=True," we update the
"part seek" benchmark.

Speaking of benchmarks, this change has the following impact to
`hg perfbundleread` on an uncompressed bundle of the Firefox repo
(6,070,036,163 bytes):

! read(8k)
! wall 0.722709 comb 0.720000 user 0.150000 sys 0.570000 (best of 14)
! read(16k)
! wall 0.602208 comb 0.590000 user 0.080000 sys 0.510000 (best of 17)
! read(32k)
! wall 0.554018 comb 0.560000 user 0.050000 sys 0.510000 (best of 18)
! read(128k)
! wall 0.520086 comb 0.530000 user 0.020000 sys 0.510000 (best of 20)
! bundle2 forwardchunks()
! wall 2.996329 comb 3.000000 user 2.300000 sys 0.700000 (best of 4)
! bundle2 iterparts()
! wall 8.070791 comb 8.060000 user 7.180000 sys 0.880000 (best of 3)
! wall 6.983756 comb 6.980000 user 6.220000 sys 0.760000 (best of 3)
! bundle2 iterparts() seekable
! wall 8.132131 comb 8.110000 user 7.160000 sys 0.950000 (best of 3)
! bundle2 part seek()
! wall 10.370142 comb 10.350000 user 7.430000 sys 2.920000 (best of 3)
! wall 10.860942 comb 10.840000 user 7.790000 sys 3.050000 (best of 3)
! bundle2 part read(8k)
! wall 8.599892 comb 8.580000 user 7.720000 sys 0.860000 (best of 3)
! wall 7.258035 comb 7.260000 user 6.470000 sys 0.790000 (best of 3)
! bundle2 part read(16k)
! wall 8.265361 comb 8.250000 user 7.360000 sys 0.890000 (best of 3)
! wall 7.099891 comb 7.080000 user 6.310000 sys 0.770000 (best of 3)
! bundle2 part read(32k)
! wall 8.290308 comb 8.280000 user 7.330000 sys 0.950000 (best of 3)
! wall 6.964685 comb 6.950000 user 6.130000 sys 0.820000 (best of 3)
! bundle2 part read(128k)
! wall 8.204900 comb 8.150000 user 7.210000 sys 0.940000 (best of 3)
! wall 6.852867 comb 6.850000 user 6.060000 sys 0.790000 (best of 3)

The significant speedup is due to not incurring the overhead to track
payload offset data. Of course, this overhead is proportional to
bundle2 part size. So a multiple gigabyte changegroup part is on the
extreme side of the spectrum for real-world impact.

In addition to the CPU efficiency wins, not tracking offset data
also means not using memory to hold that data. Using a bundle based on
the example BSD repository in issue 5691, this change has a drastic
impact to memory usage during `hg unbundle` (`hg clone` would behave
similarly). Before, memory usage incrementally increased for the
duration of bundle processing. In other words, as we advanced through
the changegroup and bundle2 part, we kept allocating more memory to
hold offset data. After this change, we still increase memory during
changegroup application. But the rate of increase is significantly
slower. (A bulk of the remaining gradual increase appears to be the
storing of revlog sizes in the transaction object to facilitate
rollback.)

The RSS at the end of filelog application is as follows:

Before: ~752 MB
After:  ~567 MB

So, we were storing ~185 MB of offset data that we never even used.
Talk about wasteful!

.. api::

   bundle2 parts are no longer seekable by default.

.. perf::

   bundle2 read I/O throughput significantly increased.

.. perf::

   Significant memory use reductions when reading from bundle2 bundles.

   On the BSD repository, peak RSS during changegroup application
   decreased by ~185 MB from ~752 MB to ~567 MB.

Differential Revision: https://phab.mercurial-scm.org/D1390

contrib/perf.py

@@ -1,1620 +1,1625
 # perf.py - performance test routines
 '''helper extension to measure performance'''
 
 # "historical portability" policy of perf.py:
 #
 # We have to do:
 # - make perf.py "loadable" with as wide Mercurial version as possible
 #   This doesn't mean that perf commands work correctly with that Mercurial.
 #   BTW, perf.py itself has been available since 1.1 (or eb240755386d).
 # - make historical perf command work correctly with as wide Mercurial
 #   version as possible
 #
 # We have to do, if possible with reasonable cost:
 # - make recent perf command for historical feature work correctly
 #   with early Mercurial
 #
 # We don't have to do:
 # - make perf command for recent feature work correctly with early
 #   Mercurial
 
 from __future__ import absolute_import
 import functools
 import gc
 import os
 import random
 import struct
 import sys
 import time
 from mercurial import (
     changegroup,
     cmdutil,
     commands,
     copies,
     error,
     extensions,
     mdiff,
     merge,
     revlog,
     util,
 )
 
 # for "historical portability":
 # try to import modules separately (in dict order), and ignore
 # failure, because these aren't available with early Mercurial
 try:
     from mercurial import branchmap # since 2.5 (or bcee63733aad)
 except ImportError:
     pass
 try:
     from mercurial import obsolete # since 2.3 (or ad0d6c2b3279)
 except ImportError:
     pass
 try:
     from mercurial import registrar # since 3.7 (or 37d50250b696)
     dir(registrar) # forcibly load it
 except ImportError:
     registrar = None
 try:
     from mercurial import repoview # since 2.5 (or 3a6ddacb7198)
 except ImportError:
     pass
 try:
     from mercurial import scmutil # since 1.9 (or 8b252e826c68)
 except ImportError:
     pass
 
 # for "historical portability":
 # define util.safehasattr forcibly, because util.safehasattr has been
 # available since 1.9.3 (or 94b200a11cf7)
 _undefined = object()
 def safehasattr(thing, attr):
     return getattr(thing, attr, _undefined) is not _undefined
 setattr(util, 'safehasattr', safehasattr)
 
 # for "historical portability":
 # define util.timer forcibly, because util.timer has been available
 # since ae5d60bb70c9
 if safehasattr(time, 'perf_counter'):
     util.timer = time.perf_counter
 elif os.name == 'nt':
     util.timer = time.clock
 else:
     util.timer = time.time
 
 # for "historical portability":
 # use locally defined empty option list, if formatteropts isn't
 # available, because commands.formatteropts has been available since
 # 3.2 (or 7a7eed5176a4), even though formatting itself has been
 # available since 2.2 (or ae5f92e154d3)
 formatteropts = getattr(cmdutil, "formatteropts",
                         getattr(commands, "formatteropts", []))
 
 # for "historical portability":
 # use locally defined option list, if debugrevlogopts isn't available,
 # because commands.debugrevlogopts has been available since 3.7 (or
 # 5606f7d0d063), even though cmdutil.openrevlog() has been available
 # since 1.9 (or a79fea6b3e77).
 revlogopts = getattr(cmdutil, "debugrevlogopts",
                      getattr(commands, "debugrevlogopts", [
         ('c', 'changelog', False, ('open changelog')),
         ('m', 'manifest', False, ('open manifest')),
         ('', 'dir', False, ('open directory manifest')),
         ]))
 
 cmdtable = {}
 
 # for "historical portability":
 # define parsealiases locally, because cmdutil.parsealiases has been
 # available since 1.5 (or 6252852b4332)
 def parsealiases(cmd):
     return cmd.lstrip("^").split("|")
 
 if safehasattr(registrar, 'command'):
     command = registrar.command(cmdtable)
 elif safehasattr(cmdutil, 'command'):
     import inspect
     command = cmdutil.command(cmdtable)
     if 'norepo' not in inspect.getargspec(command)[0]:
         # for "historical portability":
         # wrap original cmdutil.command, because "norepo" option has
         # been available since 3.1 (or 75a96326cecb)
         _command = command
         def command(name, options=(), synopsis=None, norepo=False):
             if norepo:
                 commands.norepo += ' %s' % ' '.join(parsealiases(name))
             return _command(name, list(options), synopsis)
 else:
     # for "historical portability":
     # define "@command" annotation locally, because cmdutil.command
     # has been available since 1.9 (or 2daa5179e73f)
     def command(name, options=(), synopsis=None, norepo=False):
         def decorator(func):
             if synopsis:
                 cmdtable[name] = func, list(options), synopsis
             else:
                 cmdtable[name] = func, list(options)
             if norepo:
                 commands.norepo += ' %s' % ' '.join(parsealiases(name))
             return func
         return decorator
 
 try:
     import mercurial.registrar
     import mercurial.configitems
     configtable = {}
     configitem = mercurial.registrar.configitem(configtable)
     configitem('perf', 'presleep',
         default=mercurial.configitems.dynamicdefault,
     )
     configitem('perf', 'stub',
         default=mercurial.configitems.dynamicdefault,
     )
     configitem('perf', 'parentscount',
         default=mercurial.configitems.dynamicdefault,
     )
 except (ImportError, AttributeError):
     pass
 
 def getlen(ui):
     if ui.configbool("perf", "stub", False):
         return lambda x: 1
     return len
 
 def gettimer(ui, opts=None):
     """return a timer function and formatter: (timer, formatter)
 
     This function exists to gather the creation of formatter in a single
     place instead of duplicating it in all performance commands."""
 
     # enforce an idle period before execution to counteract power management
     # experimental config: perf.presleep
     time.sleep(getint(ui, "perf", "presleep", 1))
 
     if opts is None:
         opts = {}
     # redirect all to stderr unless buffer api is in use
     if not ui._buffers:
         ui = ui.copy()
         uifout = safeattrsetter(ui, 'fout', ignoremissing=True)
         if uifout:
             # for "historical portability":
             # ui.fout/ferr have been available since 1.9 (or 4e1ccd4c2b6d)
             uifout.set(ui.ferr)
 
     # get a formatter
     uiformatter = getattr(ui, 'formatter', None)
     if uiformatter:
         fm = uiformatter('perf', opts)
     else:
         # for "historical portability":
         # define formatter locally, because ui.formatter has been
         # available since 2.2 (or ae5f92e154d3)
         from mercurial import node
         class defaultformatter(object):
             """Minimized composition of baseformatter and plainformatter
             """
             def __init__(self, ui, topic, opts):
                 self._ui = ui
                 if ui.debugflag:
                     self.hexfunc = node.hex
                 else:
                     self.hexfunc = node.short
             def __nonzero__(self):
                 return False
             __bool__ = __nonzero__
             def startitem(self):
                 pass
             def data(self, **data):
                 pass
             def write(self, fields, deftext, *fielddata, **opts):
                 self._ui.write(deftext % fielddata, **opts)
             def condwrite(self, cond, fields, deftext, *fielddata, **opts):
                 if cond:
                     self._ui.write(deftext % fielddata, **opts)
             def plain(self, text, **opts):
                 self._ui.write(text, **opts)
             def end(self):
                 pass
         fm = defaultformatter(ui, 'perf', opts)
 
     # stub function, runs code only once instead of in a loop
     # experimental config: perf.stub
     if ui.configbool("perf", "stub", False):
         return functools.partial(stub_timer, fm), fm
     return functools.partial(_timer, fm), fm
 
 def stub_timer(fm, func, title=None):
     func()
 
 def _timer(fm, func, title=None):
     gc.collect()
     results = []
     begin = util.timer()
     count = 0
     while True:
         ostart = os.times()
         cstart = util.timer()
         r = func()
         cstop = util.timer()
         ostop = os.times()
         count += 1
         a, b = ostart, ostop
         results.append((cstop - cstart, b[0] - a[0], b[1]-a[1]))
         if cstop - begin > 3 and count >= 100:
             break
         if cstop - begin > 10 and count >= 3:
             break
 
     fm.startitem()
 
     if title:
         fm.write('title', '! %s\n', title)
     if r:
         fm.write('result', '! result: %s\n', r)
     m = min(results)
     fm.plain('!')
     fm.write('wall', ' wall %f', m[0])
     fm.write('comb', ' comb %f', m[1] + m[2])
     fm.write('user', ' user %f', m[1])
     fm.write('sys',  ' sys %f', m[2])
     fm.write('count',  ' (best of %d)', count)
     fm.plain('\n')
 
 # utilities for historical portability
 
 def getint(ui, section, name, default):
     # for "historical portability":
     # ui.configint has been available since 1.9 (or fa2b596db182)
     v = ui.config(section, name, None)
     if v is None:
         return default
     try:
         return int(v)
     except ValueError:
         raise error.ConfigError(("%s.%s is not an integer ('%s')")
                                 % (section, name, v))
 
 def safeattrsetter(obj, name, ignoremissing=False):
     """Ensure that 'obj' has 'name' attribute before subsequent setattr
 
     This function is aborted, if 'obj' doesn't have 'name' attribute
     at runtime. This avoids overlooking removal of an attribute, which
     breaks assumption of performance measurement, in the future.
 
     This function returns the object to (1) assign a new value, and
     (2) restore an original value to the attribute.
 
     If 'ignoremissing' is true, missing 'name' attribute doesn't cause
     abortion, and this function returns None. This is useful to
     examine an attribute, which isn't ensured in all Mercurial
     versions.
     """
     if not util.safehasattr(obj, name):
         if ignoremissing:
             return None
         raise error.Abort(("missing attribute %s of %s might break assumption"
                            " of performance measurement") % (name, obj))
 
     origvalue = getattr(obj, name)
     class attrutil(object):
         def set(self, newvalue):
             setattr(obj, name, newvalue)
         def restore(self):
             setattr(obj, name, origvalue)
 
     return attrutil()
 
 # utilities to examine each internal API changes
 
 def getbranchmapsubsettable():
     # for "historical portability":
     # subsettable is defined in:
     # - branchmap since 2.9 (or 175c6fd8cacc)
     # - repoview since 2.5 (or 59a9f18d4587)
     for mod in (branchmap, repoview):
         subsettable = getattr(mod, 'subsettable', None)
         if subsettable:
             return subsettable
 
     # bisecting in bcee63733aad::59a9f18d4587 can reach here (both
     # branchmap and repoview modules exist, but subsettable attribute
     # doesn't)
     raise error.Abort(("perfbranchmap not available with this Mercurial"),
                       hint="use 2.5 or later")
 
 def getsvfs(repo):
     """Return appropriate object to access files under .hg/store
     """
     # for "historical portability":
     # repo.svfs has been available since 2.3 (or 7034365089bf)
     svfs = getattr(repo, 'svfs', None)
     if svfs:
         return svfs
     else:
         return getattr(repo, 'sopener')
 
 def getvfs(repo):
     """Return appropriate object to access files under .hg
     """
     # for "historical portability":
     # repo.vfs has been available since 2.3 (or 7034365089bf)
     vfs = getattr(repo, 'vfs', None)
     if vfs:
         return vfs
     else:
         return getattr(repo, 'opener')
 
 def repocleartagscachefunc(repo):
     """Return the function to clear tags cache according to repo internal API
     """
     if util.safehasattr(repo, '_tagscache'): # since 2.0 (or 9dca7653b525)
         # in this case, setattr(repo, '_tagscache', None) or so isn't
         # correct way to clear tags cache, because existing code paths
         # expect _tagscache to be a structured object.
         def clearcache():
             # _tagscache has been filteredpropertycache since 2.5 (or
             # 98c867ac1330), and delattr() can't work in such case
             if '_tagscache' in vars(repo):
                 del repo.__dict__['_tagscache']
         return clearcache
 
     repotags = safeattrsetter(repo, '_tags', ignoremissing=True)
     if repotags: # since 1.4 (or 5614a628d173)
         return lambda : repotags.set(None)
 
     repotagscache = safeattrsetter(repo, 'tagscache', ignoremissing=True)
     if repotagscache: # since 0.6 (or d7df759d0e97)
         return lambda : repotagscache.set(None)
 
     # Mercurial earlier than 0.6 (or d7df759d0e97) logically reaches
     # this point, but it isn't so problematic, because:
     # - repo.tags of such Mercurial isn't "callable", and repo.tags()
     #   in perftags() causes failure soon
     # - perf.py itself has been available since 1.1 (or eb240755386d)
     raise error.Abort(("tags API of this hg command is unknown"))
 
 # utilities to clear cache
 
 def clearfilecache(repo, attrname):
     unfi = repo.unfiltered()
     if attrname in vars(unfi):
         delattr(unfi, attrname)
     unfi._filecache.pop(attrname, None)
 
 # perf commands
 
 @command('perfwalk', formatteropts)
 def perfwalk(ui, repo, *pats, **opts):
     timer, fm = gettimer(ui, opts)
     m = scmutil.match(repo[None], pats, {})
     timer(lambda: len(list(repo.dirstate.walk(m, subrepos=[], unknown=True,
                                               ignored=False))))
     fm.end()
 
 @command('perfannotate', formatteropts)
 def perfannotate(ui, repo, f, **opts):
     timer, fm = gettimer(ui, opts)
     fc = repo['.'][f]
     timer(lambda: len(fc.annotate(True)))
     fm.end()
 
 @command('perfstatus',
          [('u', 'unknown', False,
            'ask status to look for unknown files')] + formatteropts)
 def perfstatus(ui, repo, **opts):
     #m = match.always(repo.root, repo.getcwd())
     #timer(lambda: sum(map(len, repo.dirstate.status(m, [], False, False,
     #                                                False))))
     timer, fm = gettimer(ui, opts)
     timer(lambda: sum(map(len, repo.status(unknown=opts['unknown']))))
     fm.end()
 
 @command('perfaddremove', formatteropts)
 def perfaddremove(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     try:
         oldquiet = repo.ui.quiet
         repo.ui.quiet = True
         matcher = scmutil.match(repo[None])
         timer(lambda: scmutil.addremove(repo, matcher, "", dry_run=True))
     finally:
         repo.ui.quiet = oldquiet
         fm.end()
 
 def clearcaches(cl):
     # behave somewhat consistently across internal API changes
     if util.safehasattr(cl, 'clearcaches'):
         cl.clearcaches()
     elif util.safehasattr(cl, '_nodecache'):
         from mercurial.node import nullid, nullrev
         cl._nodecache = {nullid: nullrev}
         cl._nodepos = None
 
 @command('perfheads', formatteropts)
 def perfheads(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     cl = repo.changelog
     def d():
         len(cl.headrevs())
         clearcaches(cl)
     timer(d)
     fm.end()
 
 @command('perftags', formatteropts)
 def perftags(ui, repo, **opts):
     import mercurial.changelog
     import mercurial.manifest
     timer, fm = gettimer(ui, opts)
     svfs = getsvfs(repo)
     repocleartagscache = repocleartagscachefunc(repo)
     def t():
         repo.changelog = mercurial.changelog.changelog(svfs)
         repo.manifestlog = mercurial.manifest.manifestlog(svfs, repo)
         repocleartagscache()
         return len(repo.tags())
     timer(t)
     fm.end()
 
 @command('perfancestors', formatteropts)
 def perfancestors(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     heads = repo.changelog.headrevs()
     def d():
         for a in repo.changelog.ancestors(heads):
             pass
     timer(d)
     fm.end()
 
 @command('perfancestorset', formatteropts)
 def perfancestorset(ui, repo, revset, **opts):
     timer, fm = gettimer(ui, opts)
     revs = repo.revs(revset)
     heads = repo.changelog.headrevs()
     def d():
         s = repo.changelog.ancestors(heads)
         for rev in revs:
             rev in s
     timer(d)
     fm.end()
 
 @command('perfbookmarks', formatteropts)
 def perfbookmarks(ui, repo, **opts):
     """benchmark parsing bookmarks from disk to memory"""
     timer, fm = gettimer(ui, opts)
     def d():
         clearfilecache(repo, '_bookmarks')
         repo._bookmarks
     timer(d)
     fm.end()
 
 @command('perfbundleread', formatteropts, 'BUNDLE')
 def perfbundleread(ui, repo, bundlepath, **opts):
     """Benchmark reading of bundle files.
 
     This command is meant to isolate the I/O part of bundle reading as
     much as possible.
     """
     from mercurial import (
         bundle2,
         exchange,
         streamclone,
     )
 
     def makebench(fn):
         def run():
             with open(bundlepath, 'rb') as fh:
                 bundle = exchange.readbundle(ui, fh, bundlepath)
                 fn(bundle)
 
         return run
 
     def makereadnbytes(size):
         def run():
             with open(bundlepath, 'rb') as fh:
                 bundle = exchange.readbundle(ui, fh, bundlepath)
                 while bundle.read(size):
                     pass
 
         return run
 
     def makestdioread(size):
         def run():
             with open(bundlepath, 'rb') as fh:
                 while fh.read(size):
                     pass
 
         return run
 
     # bundle1
 
     def deltaiter(bundle):
         for delta in bundle.deltaiter():
             pass
 
     def iterchunks(bundle):
         for chunk in bundle.getchunks():
             pass
 
     # bundle2
 
     def forwardchunks(bundle):
         for chunk in bundle._forwardchunks():
             pass
 
     def iterparts(bundle):
         for part in bundle.iterparts():
             pass
 
+    def iterpartsseekable(bundle):
+        for part in bundle.iterparts(seekable=True):
+            pass
+
     def seek(bundle):
-        for part in bundle.iterparts():
+        for part in bundle.iterparts(seekable=True):
             part.seek(0, os.SEEK_END)
 
     def makepartreadnbytes(size):
         def run():
             with open(bundlepath, 'rb') as fh:
                 bundle = exchange.readbundle(ui, fh, bundlepath)
                 for part in bundle.iterparts():
                     while part.read(size):
                         pass
 
         return run
 
     benches = [
         (makestdioread(8192), 'read(8k)'),
         (makestdioread(16384), 'read(16k)'),
         (makestdioread(32768), 'read(32k)'),
         (makestdioread(131072), 'read(128k)'),
     ]
 
     with open(bundlepath, 'rb') as fh:
         bundle = exchange.readbundle(ui, fh, bundlepath)
 
         if isinstance(bundle, changegroup.cg1unpacker):
             benches.extend([
                 (makebench(deltaiter), 'cg1 deltaiter()'),
                 (makebench(iterchunks), 'cg1 getchunks()'),
                 (makereadnbytes(8192), 'cg1 read(8k)'),
                 (makereadnbytes(16384), 'cg1 read(16k)'),
                 (makereadnbytes(32768), 'cg1 read(32k)'),
                 (makereadnbytes(131072), 'cg1 read(128k)'),
             ])
         elif isinstance(bundle, bundle2.unbundle20):
             benches.extend([
                 (makebench(forwardchunks), 'bundle2 forwardchunks()'),
                 (makebench(iterparts), 'bundle2 iterparts()'),
+                (makebench(iterpartsseekable), 'bundle2 iterparts() seekable'),
                 (makebench(seek), 'bundle2 part seek()'),
                 (makepartreadnbytes(8192), 'bundle2 part read(8k)'),
                 (makepartreadnbytes(16384), 'bundle2 part read(16k)'),
                 (makepartreadnbytes(32768), 'bundle2 part read(32k)'),
                 (makepartreadnbytes(131072), 'bundle2 part read(128k)'),
             ])
         elif isinstance(bundle, streamclone.streamcloneapplier):
             raise error.Abort('stream clone bundles not supported')
         else:
             raise error.Abort('unhandled bundle type: %s' % type(bundle))
 
     for fn, title in benches:
         timer, fm = gettimer(ui, opts)
         timer(fn, title=title)
         fm.end()
 
 @command('perfchangegroupchangelog', formatteropts +
          [('', 'version', '02', 'changegroup version'),
           ('r', 'rev', '', 'revisions to add to changegroup')])
 def perfchangegroupchangelog(ui, repo, version='02', rev=None, **opts):
     """Benchmark producing a changelog group for a changegroup.
 
     This measures the time spent processing the changelog during a
     bundle operation. This occurs during `hg bundle` and on a server
     processing a `getbundle` wire protocol request (handles clones
     and pull requests).
 
     By default, all revisions are added to the changegroup.
     """
     cl = repo.changelog
     revs = [cl.lookup(r) for r in repo.revs(rev or 'all()')]
     bundler = changegroup.getbundler(version, repo)
 
     def lookup(node):
         # The real bundler reads the revision in order to access the
         # manifest node and files list. Do that here.
         cl.read(node)
         return node
 
     def d():
         for chunk in bundler.group(revs, cl, lookup):
             pass
 
     timer, fm = gettimer(ui, opts)
     timer(d)
     fm.end()
 
 @command('perfdirs', formatteropts)
 def perfdirs(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     dirstate = repo.dirstate
     'a' in dirstate
     def d():
         dirstate.hasdir('a')
         del dirstate._map._dirs
     timer(d)
     fm.end()
 
 @command('perfdirstate', formatteropts)
 def perfdirstate(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     "a" in repo.dirstate
     def d():
         repo.dirstate.invalidate()
         "a" in repo.dirstate
     timer(d)
     fm.end()
 
 @command('perfdirstatedirs', formatteropts)
 def perfdirstatedirs(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     "a" in repo.dirstate
     def d():
         repo.dirstate.hasdir("a")
         del repo.dirstate._map._dirs
     timer(d)
     fm.end()
 
 @command('perfdirstatefoldmap', formatteropts)
 def perfdirstatefoldmap(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     dirstate = repo.dirstate
     'a' in dirstate
     def d():
         dirstate._map.filefoldmap.get('a')
         del dirstate._map.filefoldmap
     timer(d)
     fm.end()
 
 @command('perfdirfoldmap', formatteropts)
 def perfdirfoldmap(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     dirstate = repo.dirstate
     'a' in dirstate
     def d():
         dirstate._map.dirfoldmap.get('a')
         del dirstate._map.dirfoldmap
         del dirstate._map._dirs
     timer(d)
     fm.end()
 
 @command('perfdirstatewrite', formatteropts)
 def perfdirstatewrite(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     ds = repo.dirstate
     "a" in ds
     def d():
         ds._dirty = True
         ds.write(repo.currenttransaction())
     timer(d)
     fm.end()
 
 @command('perfmergecalculate',
          [('r', 'rev', '.', 'rev to merge against')] + formatteropts)
 def perfmergecalculate(ui, repo, rev, **opts):
     timer, fm = gettimer(ui, opts)
     wctx = repo[None]
     rctx = scmutil.revsingle(repo, rev, rev)
     ancestor = wctx.ancestor(rctx)
     # we don't want working dir files to be stat'd in the benchmark, so prime
     # that cache
     wctx.dirty()
     def d():
         # acceptremote is True because we don't want prompts in the middle of
         # our benchmark
         merge.calculateupdates(repo, wctx, rctx, [ancestor], False, False,
                                acceptremote=True, followcopies=True)
     timer(d)
     fm.end()
 
 @command('perfpathcopies', [], "REV REV")
 def perfpathcopies(ui, repo, rev1, rev2, **opts):
     timer, fm = gettimer(ui, opts)
     ctx1 = scmutil.revsingle(repo, rev1, rev1)
     ctx2 = scmutil.revsingle(repo, rev2, rev2)
     def d():
         copies.pathcopies(ctx1, ctx2)
     timer(d)
     fm.end()
 
 @command('perfphases',
          [('', 'full', False, 'include file reading time too'),
          ], "")
 def perfphases(ui, repo, **opts):
     """benchmark phasesets computation"""
     timer, fm = gettimer(ui, opts)
     _phases = repo._phasecache
     full = opts.get('full')
     def d():
         phases = _phases
         if full:
             clearfilecache(repo, '_phasecache')
             phases = repo._phasecache
         phases.invalidate()
         phases.loadphaserevs(repo)
     timer(d)
     fm.end()
 
 @command('perfmanifest', [], 'REV')
 def perfmanifest(ui, repo, rev, **opts):
     timer, fm = gettimer(ui, opts)
     ctx = scmutil.revsingle(repo, rev, rev)
     t = ctx.manifestnode()
     def d():
         repo.manifestlog.clearcaches()
         repo.manifestlog[t].read()
     timer(d)
     fm.end()
 
 @command('perfchangeset', formatteropts)
 def perfchangeset(ui, repo, rev, **opts):
     timer, fm = gettimer(ui, opts)
     n = repo[rev].node()
     def d():
         repo.changelog.read(n)
         #repo.changelog._cache = None
     timer(d)
     fm.end()
 
 @command('perfindex', formatteropts)
 def perfindex(ui, repo, **opts):
     import mercurial.revlog
     timer, fm = gettimer(ui, opts)
     mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
     n = repo["tip"].node()
     svfs = getsvfs(repo)
     def d():
         cl = mercurial.revlog.revlog(svfs, "00changelog.i")
         cl.rev(n)
     timer(d)
     fm.end()
 
 @command('perfstartup', formatteropts)
 def perfstartup(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     cmd = sys.argv[0]
     def d():
         if os.name != 'nt':
             os.system("HGRCPATH= %s version -q > /dev/null" % cmd)
         else:
             os.environ['HGRCPATH'] = ' '
             os.system("%s version -q > NUL" % cmd)
     timer(d)
     fm.end()
 
 @command('perfparents', formatteropts)
 def perfparents(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     # control the number of commits perfparents iterates over
     # experimental config: perf.parentscount
     count = getint(ui, "perf", "parentscount", 1000)
     if len(repo.changelog) < count:
         raise error.Abort("repo needs %d commits for this test" % count)
     repo = repo.unfiltered()
     nl = [repo.changelog.node(i) for i in xrange(count)]
     def d():
         for n in nl:
             repo.changelog.parents(n)
     timer(d)
     fm.end()
 
 @command('perfctxfiles', formatteropts)
 def perfctxfiles(ui, repo, x, **opts):
     x = int(x)
     timer, fm = gettimer(ui, opts)
     def d():
         len(repo[x].files())
     timer(d)
     fm.end()
 
 @command('perfrawfiles', formatteropts)
 def perfrawfiles(ui, repo, x, **opts):
     x = int(x)
     timer, fm = gettimer(ui, opts)
     cl = repo.changelog
     def d():
         len(cl.read(x)[3])
     timer(d)
     fm.end()
 
 @command('perflookup', formatteropts)
 def perflookup(ui, repo, rev, **opts):
     timer, fm = gettimer(ui, opts)
     timer(lambda: len(repo.lookup(rev)))
     fm.end()
 
 @command('perfrevrange', formatteropts)
 def perfrevrange(ui, repo, *specs, **opts):
     timer, fm = gettimer(ui, opts)
     revrange = scmutil.revrange
     timer(lambda: len(revrange(repo, specs)))
     fm.end()
 
 @command('perfnodelookup', formatteropts)
 def perfnodelookup(ui, repo, rev, **opts):
     timer, fm = gettimer(ui, opts)
     import mercurial.revlog
     mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
     n = repo[rev].node()
     cl = mercurial.revlog.revlog(getsvfs(repo), "00changelog.i")
     def d():
         cl.rev(n)
         clearcaches(cl)
     timer(d)
     fm.end()
 
 @command('perflog',
          [('', 'rename', False, 'ask log to follow renames')] + formatteropts)
 def perflog(ui, repo, rev=None, **opts):
     if rev is None:
         rev=[]
     timer, fm = gettimer(ui, opts)
     ui.pushbuffer()
     timer(lambda: commands.log(ui, repo, rev=rev, date='', user='',
                                copies=opts.get('rename')))
     ui.popbuffer()
     fm.end()
 
 @command('perfmoonwalk', formatteropts)
 def perfmoonwalk(ui, repo, **opts):
     """benchmark walking the changelog backwards
 
     This also loads the changelog data for each revision in the changelog.
     """
     timer, fm = gettimer(ui, opts)
     def moonwalk():
         for i in xrange(len(repo), -1, -1):
             ctx = repo[i]
             ctx.branch() # read changelog data (in addition to the index)
     timer(moonwalk)
     fm.end()
 
 @command('perftemplating', formatteropts)
 def perftemplating(ui, repo, rev=None, **opts):
     if rev is None:
         rev=[]
     timer, fm = gettimer(ui, opts)
     ui.pushbuffer()
     timer(lambda: commands.log(ui, repo, rev=rev, date='', user='',
                                template='{date|shortdate} [{rev}:{node|short}]'
                                ' {author|person}: {desc|firstline}\n'))
     ui.popbuffer()
     fm.end()
 
 @command('perfcca', formatteropts)
 def perfcca(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     timer(lambda: scmutil.casecollisionauditor(ui, False, repo.dirstate))
     fm.end()
 
 @command('perffncacheload', formatteropts)
 def perffncacheload(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     s = repo.store
     def d():
         s.fncache._load()
     timer(d)
     fm.end()
 
 @command('perffncachewrite', formatteropts)
 def perffncachewrite(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     s = repo.store
     s.fncache._load()
     lock = repo.lock()
     tr = repo.transaction('perffncachewrite')
     def d():
         s.fncache._dirty = True
         s.fncache.write(tr)
     timer(d)
     tr.close()
     lock.release()
     fm.end()
 
 @command('perffncacheencode', formatteropts)
 def perffncacheencode(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
     s = repo.store
     s.fncache._load()
     def d():
         for p in s.fncache.entries:
             s.encode(p)
     timer(d)
     fm.end()
 
 @command('perfbdiff', revlogopts + formatteropts + [
     ('', 'count', 1, 'number of revisions to test (when using --startrev)'),
     ('', 'alldata', False, 'test bdiffs for all associated revisions')],
     '-c|-m|FILE REV')
 def perfbdiff(ui, repo, file_, rev=None, count=None, **opts):
     """benchmark a bdiff between revisions
 
     By default, benchmark a bdiff between its delta parent and itself.
 
     With ``--count``, benchmark bdiffs between delta parents and self for N
     revisions starting at the specified revision.
 
     With ``--alldata``, assume the requested revision is a changeset and
     measure bdiffs for all changes related to that changeset (manifest
     and filelogs).
     """
     if opts['alldata']:
         opts['changelog'] = True
 
     if opts.get('changelog') or opts.get('manifest'):
         file_, rev = None, file_
     elif rev is None:
         raise error.CommandError('perfbdiff', 'invalid arguments')
 
     textpairs = []
 
     r = cmdutil.openrevlog(repo, 'perfbdiff', file_, opts)
 
     startrev = r.rev(r.lookup(rev))
     for rev in range(startrev, min(startrev + count, len(r) - 1)):
         if opts['alldata']:
             # Load revisions associated with changeset.
             ctx = repo[rev]
             mtext = repo.manifestlog._revlog.revision(ctx.manifestnode())
             for pctx in ctx.parents():
                 pman = repo.manifestlog._revlog.revision(pctx.manifestnode())
                 textpairs.append((pman, mtext))
 
             # Load filelog revisions by iterating manifest delta.
             man = ctx.manifest()
             pman = ctx.p1().manifest()
             for filename, change in pman.diff(man).items():
                 fctx = repo.file(filename)
                 f1 = fctx.revision(change[0][0] or -1)
                 f2 = fctx.revision(change[1][0] or -1)
                 textpairs.append((f1, f2))
         else:
             dp = r.deltaparent(rev)
             textpairs.append((r.revision(dp), r.revision(rev)))
 
     def d():
         for pair in textpairs:
             mdiff.textdiff(*pair)
 
     timer, fm = gettimer(ui, opts)
     timer(d)
     fm.end()
 
 @command('perfdiffwd', formatteropts)
 def perfdiffwd(ui, repo, **opts):
     """Profile diff of working directory changes"""
     timer, fm = gettimer(ui, opts)
     options = {
         'w': 'ignore_all_space',
         'b': 'ignore_space_change',
         'B': 'ignore_blank_lines',
         }
 
     for diffopt in ('', 'w', 'b', 'B', 'wB'):
         opts = dict((options[c], '1') for c in diffopt)
         def d():
             ui.pushbuffer()
             commands.diff(ui, repo, **opts)
             ui.popbuffer()
         title = 'diffopts: %s' % (diffopt and ('-' + diffopt) or 'none')
         timer(d, title)
     fm.end()
 
 @command('perfrevlogindex', revlogopts + formatteropts,
          '-c|-m|FILE')
 def perfrevlogindex(ui, repo, file_=None, **opts):
     """Benchmark operations against a revlog index.
 
     This tests constructing a revlog instance, reading index data,
     parsing index data, and performing various operations related to
     index data.
     """
 
     rl = cmdutil.openrevlog(repo, 'perfrevlogindex', file_, opts)
 
     opener = getattr(rl, 'opener')  # trick linter
     indexfile = rl.indexfile
     data = opener.read(indexfile)
 
     header = struct.unpack('>I', data[0:4])[0]
     version = header & 0xFFFF
     if version == 1:
         revlogio = revlog.revlogio()
         inline = header & (1 << 16)
     else:
         raise error.Abort(('unsupported revlog version: %d') % version)
 
     rllen = len(rl)
 
     node0 = rl.node(0)
     node25 = rl.node(rllen // 4)
     node50 = rl.node(rllen // 2)
     node75 = rl.node(rllen // 4 * 3)
     node100 = rl.node(rllen - 1)
 
     allrevs = range(rllen)
     allrevsrev = list(reversed(allrevs))
     allnodes = [rl.node(rev) for rev in range(rllen)]
     allnodesrev = list(reversed(allnodes))
 
     def constructor():
         revlog.revlog(opener, indexfile)
 
     def read():
         with opener(indexfile) as fh:
             fh.read()
 
     def parseindex():
         revlogio.parseindex(data, inline)
 
     def getentry(revornode):
         index = revlogio.parseindex(data, inline)[0]
         index[revornode]
 
     def getentries(revs, count=1):
         index = revlogio.parseindex(data, inline)[0]
 
         for i in range(count):
             for rev in revs:
                 index[rev]
 
     def resolvenode(node):
         nodemap = revlogio.parseindex(data, inline)[1]
         # This only works for the C code.
         if nodemap is None:
             return
 
         try:
             nodemap[node]
         except error.RevlogError:
             pass
 
     def resolvenodes(nodes, count=1):
         nodemap = revlogio.parseindex(data, inline)[1]
         if nodemap is None:
             return
 
         for i in range(count):
             for node in nodes:
                 try:
                     nodemap[node]
                 except error.RevlogError:
                     pass
 
     benches = [
         (constructor, 'revlog constructor'),
         (read, 'read'),
         (parseindex, 'create index object'),
         (lambda: getentry(0), 'retrieve index entry for rev 0'),
         (lambda: resolvenode('a' * 20), 'look up missing node'),
         (lambda: resolvenode(node0), 'look up node at rev 0'),
         (lambda: resolvenode(node25), 'look up node at 1/4 len'),
         (lambda: resolvenode(node50), 'look up node at 1/2 len'),
         (lambda: resolvenode(node75), 'look up node at 3/4 len'),
         (lambda: resolvenode(node100), 'look up node at tip'),
         # 2x variation is to measure caching impact.
         (lambda: resolvenodes(allnodes),
          'look up all nodes (forward)'),
         (lambda: resolvenodes(allnodes, 2),
          'look up all nodes 2x (forward)'),
         (lambda: resolvenodes(allnodesrev),
          'look up all nodes (reverse)'),
         (lambda: resolvenodes(allnodesrev, 2),
          'look up all nodes 2x (reverse)'),
         (lambda: getentries(allrevs),
          'retrieve all index entries (forward)'),
         (lambda: getentries(allrevs, 2),
          'retrieve all index entries 2x (forward)'),
         (lambda: getentries(allrevsrev),
          'retrieve all index entries (reverse)'),
         (lambda: getentries(allrevsrev, 2),
          'retrieve all index entries 2x (reverse)'),
     ]
 
     for fn, title in benches:
         timer, fm = gettimer(ui, opts)
         timer(fn, title=title)
         fm.end()
 
 @command('perfrevlogrevisions', revlogopts + formatteropts +
          [('d', 'dist', 100, 'distance between the revisions'),
           ('s', 'startrev', 0, 'revision to start reading at'),
           ('', 'reverse', False, 'read in reverse')],
          '-c|-m|FILE')
 def perfrevlogrevisions(ui, repo, file_=None, startrev=0, reverse=False,
                         **opts):
     """Benchmark reading a series of revisions from a revlog.
 
     By default, we read every ``-d/--dist`` revision from 0 to tip of
     the specified revlog.
 
     The start revision can be defined via ``-s/--startrev``.
     """
     rl = cmdutil.openrevlog(repo, 'perfrevlogrevisions', file_, opts)
     rllen = getlen(ui)(rl)
 
     def d():
         rl.clearcaches()
 
         beginrev = startrev
         endrev = rllen
         dist = opts['dist']
 
         if reverse:
             beginrev, endrev = endrev, beginrev
             dist = -1 * dist
 
         for x in xrange(beginrev, endrev, dist):
             # Old revisions don't support passing int.
             n = rl.node(x)
             rl.revision(n)
 
     timer, fm = gettimer(ui, opts)
     timer(d)
     fm.end()
 
 @command('perfrevlogchunks', revlogopts + formatteropts +
          [('e', 'engines', '', 'compression engines to use'),
           ('s', 'startrev', 0, 'revision to start at')],
          '-c|-m|FILE')
 def perfrevlogchunks(ui, repo, file_=None, engines=None, startrev=0, **opts):
     """Benchmark operations on revlog chunks.
 
     Logically, each revlog is a collection of fulltext revisions. However,
     stored within each revlog are "chunks" of possibly compressed data. This
     data needs to be read and decompressed or compressed and written.
 
     This command measures the time it takes to read+decompress and recompress
     chunks in a revlog. It effectively isolates I/O and compression performance.
     For measurements of higher-level operations like resolving revisions,
     see ``perfrevlogrevisions`` and ``perfrevlogrevision``.
     """
     rl = cmdutil.openrevlog(repo, 'perfrevlogchunks', file_, opts)
 
     # _chunkraw was renamed to _getsegmentforrevs.
     try:
         segmentforrevs = rl._getsegmentforrevs
     except AttributeError:
         segmentforrevs = rl._chunkraw
 
     # Verify engines argument.
     if engines:
         engines = set(e.strip() for e in engines.split(','))
         for engine in engines:
             try:
                 util.compressionengines[engine]
             except KeyError:
                 raise error.Abort('unknown compression engine: %s' % engine)
     else:
         engines = []
         for e in util.compengines:
             engine = util.compengines[e]
             try:
                 if engine.available():
                     engine.revlogcompressor().compress('dummy')
                     engines.append(e)
             except NotImplementedError:
                 pass
 
     revs = list(rl.revs(startrev, len(rl) - 1))
 
     def rlfh(rl):
         if rl._inline:
             return getsvfs(repo)(rl.indexfile)
         else:
             return getsvfs(repo)(rl.datafile)
 
     def doread():
         rl.clearcaches()
         for rev in revs:
             segmentforrevs(rev, rev)
 
     def doreadcachedfh():
         rl.clearcaches()
         fh = rlfh(rl)
         for rev in revs:
             segmentforrevs(rev, rev, df=fh)
 
     def doreadbatch():
         rl.clearcaches()
         segmentforrevs(revs[0], revs[-1])
 
     def doreadbatchcachedfh():
         rl.clearcaches()
         fh = rlfh(rl)
         segmentforrevs(revs[0], revs[-1], df=fh)
 
     def dochunk():
         rl.clearcaches()
         fh = rlfh(rl)
         for rev in revs:
             rl._chunk(rev, df=fh)
 
     chunks = [None]
 
     def dochunkbatch():
         rl.clearcaches()
         fh = rlfh(rl)
         # Save chunks as a side-effect.
         chunks[0] = rl._chunks(revs, df=fh)
 
     def docompress(compressor):
         rl.clearcaches()
 
         try:
             # Swap in the requested compression engine.
             oldcompressor = rl._compressor
             rl._compressor = compressor
             for chunk in chunks[0]:
                 rl.compress(chunk)
         finally:
             rl._compressor = oldcompressor
 
     benches = [
         (lambda: doread(), 'read'),
         (lambda: doreadcachedfh(), 'read w/ reused fd'),
         (lambda: doreadbatch(), 'read batch'),
         (lambda: doreadbatchcachedfh(), 'read batch w/ reused fd'),
         (lambda: dochunk(), 'chunk'),
         (lambda: dochunkbatch(), 'chunk batch'),
     ]
 
     for engine in sorted(engines):
         compressor = util.compengines[engine].revlogcompressor()
         benches.append((functools.partial(docompress, compressor),
                         'compress w/ %s' % engine))
 
     for fn, title in benches:
         timer, fm = gettimer(ui, opts)
         timer(fn, title=title)
         fm.end()
 
 @command('perfrevlogrevision', revlogopts + formatteropts +
          [('', 'cache', False, 'use caches instead of clearing')],
          '-c|-m|FILE REV')
 def perfrevlogrevision(ui, repo, file_, rev=None, cache=None, **opts):
     """Benchmark obtaining a revlog revision.
 
     Obtaining a revlog revision consists of roughly the following steps:
 
     1. Compute the delta chain
     2. Obtain the raw chunks for that delta chain
     3. Decompress each raw chunk
     4. Apply binary patches to obtain fulltext
     5. Verify hash of fulltext
 
     This command measures the time spent in each of these phases.
     """
     if opts.get('changelog') or opts.get('manifest'):
         file_, rev = None, file_
     elif rev is None:
         raise error.CommandError('perfrevlogrevision', 'invalid arguments')
 
     r = cmdutil.openrevlog(repo, 'perfrevlogrevision', file_, opts)
 
     # _chunkraw was renamed to _getsegmentforrevs.
     try:
         segmentforrevs = r._getsegmentforrevs
     except AttributeError:
         segmentforrevs = r._chunkraw
 
     node = r.lookup(rev)
     rev = r.rev(node)
 
     def getrawchunks(data, chain):
         start = r.start
         length = r.length
         inline = r._inline
         iosize = r._io.size
         buffer = util.buffer
         offset = start(chain[0])
 
         chunks = []
         ladd = chunks.append
 
         for rev in chain:
             chunkstart = start(rev)
             if inline:
                 chunkstart += (rev + 1) * iosize
             chunklength = length(rev)
             ladd(buffer(data, chunkstart - offset, chunklength))
 
         return chunks
 
     def dodeltachain(rev):
         if not cache:
             r.clearcaches()
         r._deltachain(rev)
 
     def doread(chain):
         if not cache:
             r.clearcaches()
         segmentforrevs(chain[0], chain[-1])
 
     def dorawchunks(data, chain):
         if not cache:
             r.clearcaches()
         getrawchunks(data, chain)
 
     def dodecompress(chunks):
         decomp = r.decompress
         for chunk in chunks:
             decomp(chunk)
 
     def dopatch(text, bins):
         if not cache:
             r.clearcaches()
         mdiff.patches(text, bins)
 
     def dohash(text):
         if not cache:
             r.clearcaches()
         r.checkhash(text, node, rev=rev)
 
     def dorevision():
         if not cache:
             r.clearcaches()
         r.revision(node)
 
     chain = r._deltachain(rev)[0]
     data = segmentforrevs(chain[0], chain[-1])[1]
     rawchunks = getrawchunks(data, chain)
     bins = r._chunks(chain)
     text = str(bins[0])
     bins = bins[1:]
     text = mdiff.patches(text, bins)
 
     benches = [
         (lambda: dorevision(), 'full'),
         (lambda: dodeltachain(rev), 'deltachain'),
         (lambda: doread(chain), 'read'),
         (lambda: dorawchunks(data, chain), 'rawchunks'),
         (lambda: dodecompress(rawchunks), 'decompress'),
         (lambda: dopatch(text, bins), 'patch'),
         (lambda: dohash(text), 'hash'),
     ]
 
     for fn, title in benches:
         timer, fm = gettimer(ui, opts)
         timer(fn, title=title)
         fm.end()
 
 @command('perfrevset',
          [('C', 'clear', False, 'clear volatile cache between each call.'),
           ('', 'contexts', False, 'obtain changectx for each revision')]
          + formatteropts, "REVSET")
 def perfrevset(ui, repo, expr, clear=False, contexts=False, **opts):
     """benchmark the execution time of a revset
 
     Use the --clean option if need to evaluate the impact of build volatile
     revisions set cache on the revset execution. Volatile cache hold filtered
     and obsolete related cache."""
     timer, fm = gettimer(ui, opts)
     def d():
         if clear:
             repo.invalidatevolatilesets()
         if contexts:
             for ctx in repo.set(expr): pass
         else:
             for r in repo.revs(expr): pass
     timer(d)
     fm.end()
 
 @command('perfvolatilesets',
          [('', 'clear-obsstore', False, 'drop obsstore between each call.'),
          ] + formatteropts)
 def perfvolatilesets(ui, repo, *names, **opts):
     """benchmark the computation of various volatile set
 
     Volatile set computes element related to filtering and obsolescence."""
     timer, fm = gettimer(ui, opts)
     repo = repo.unfiltered()
 
     def getobs(name):
         def d():
             repo.invalidatevolatilesets()
             if opts['clear_obsstore']:
                 clearfilecache(repo, 'obsstore')
             obsolete.getrevs(repo, name)
         return d
 
     allobs = sorted(obsolete.cachefuncs)
     if names:
         allobs = [n for n in allobs if n in names]
 
     for name in allobs:
         timer(getobs(name), title=name)
 
     def getfiltered(name):
         def d():
             repo.invalidatevolatilesets()
             if opts['clear_obsstore']:
                 clearfilecache(repo, 'obsstore')
             repoview.filterrevs(repo, name)
         return d
 
     allfilter = sorted(repoview.filtertable)
     if names:
         allfilter = [n for n in allfilter if n in names]
 
     for name in allfilter:
         timer(getfiltered(name), title=name)
     fm.end()
 
 @command('perfbranchmap',
          [('f', 'full', False,
            'Includes build time of subset'),
           ('', 'clear-revbranch', False,
            'purge the revbranch cache between computation'),
          ] + formatteropts)
 def perfbranchmap(ui, repo, full=False, clear_revbranch=False, **opts):
     """benchmark the update of a branchmap
 
     This benchmarks the full repo.branchmap() call with read and write disabled
     """
     timer, fm = gettimer(ui, opts)
     def getbranchmap(filtername):
         """generate a benchmark function for the filtername"""
         if filtername is None:
             view = repo
         else:
             view = repo.filtered(filtername)
         def d():
             if clear_revbranch:
                 repo.revbranchcache()._clear()
             if full:
                 view._branchcaches.clear()
             else:
                 view._branchcaches.pop(filtername, None)
             view.branchmap()
         return d
     # add filter in smaller subset to bigger subset
     possiblefilters = set(repoview.filtertable)
     subsettable = getbranchmapsubsettable()
     allfilters = []
     while possiblefilters:
         for name in possiblefilters:
             subset = subsettable.get(name)
             if subset not in possiblefilters:
                 break
         else:
             assert False, 'subset cycle %s!' % possiblefilters
         allfilters.append(name)
         possiblefilters.remove(name)
 
     # warm the cache
     if not full:
         for name in allfilters:
             repo.filtered(name).branchmap()
     # add unfiltered
     allfilters.append(None)
 
     branchcacheread = safeattrsetter(branchmap, 'read')
     branchcachewrite = safeattrsetter(branchmap.branchcache, 'write')
     branchcacheread.set(lambda repo: None)
     branchcachewrite.set(lambda bc, repo: None)
     try:
         for name in allfilters:
             timer(getbranchmap(name), title=str(name))
     finally:
         branchcacheread.restore()
         branchcachewrite.restore()
     fm.end()
 
 @command('perfloadmarkers')
 def perfloadmarkers(ui, repo):
     """benchmark the time to parse the on-disk markers for a repo
 
     Result is the number of markers in the repo."""
     timer, fm = gettimer(ui)
     svfs = getsvfs(repo)
     timer(lambda: len(obsolete.obsstore(svfs)))
     fm.end()
 
 @command('perflrucachedict', formatteropts +
     [('', 'size', 4, 'size of cache'),
      ('', 'gets', 10000, 'number of key lookups'),
      ('', 'sets', 10000, 'number of key sets'),
      ('', 'mixed', 10000, 'number of mixed mode operations'),
      ('', 'mixedgetfreq', 50, 'frequency of get vs set ops in mixed mode')],
     norepo=True)
 def perflrucache(ui, size=4, gets=10000, sets=10000, mixed=10000,
                  mixedgetfreq=50, **opts):
     def doinit():
         for i in xrange(10000):
             util.lrucachedict(size)
 
     values = []
     for i in xrange(size):
         values.append(random.randint(0, sys.maxint))
 
     # Get mode fills the cache and tests raw lookup performance with no
     # eviction.
     getseq = []
     for i in xrange(gets):
         getseq.append(random.choice(values))
 
     def dogets():
         d = util.lrucachedict(size)
         for v in values:
             d[v] = v
         for key in getseq:
             value = d[key]
             value # silence pyflakes warning
 
     # Set mode tests insertion speed with cache eviction.
     setseq = []
     for i in xrange(sets):
         setseq.append(random.randint(0, sys.maxint))
 
     def dosets():
         d = util.lrucachedict(size)
         for v in setseq:
             d[v] = v
 
     # Mixed mode randomly performs gets and sets with eviction.
     mixedops = []
     for i in xrange(mixed):
         r = random.randint(0, 100)
         if r < mixedgetfreq:
             op = 0
         else:
             op = 1
 
         mixedops.append((op, random.randint(0, size * 2)))
 
     def domixed():
         d = util.lrucachedict(size)
 
         for op, v in mixedops:
             if op == 0:
                 try:
                     d[v]
                 except KeyError:
                     pass
             else:
                 d[v] = v
 
     benches = [
         (doinit, 'init'),
         (dogets, 'gets'),
         (dosets, 'sets'),
         (domixed, 'mixed')
     ]
 
     for fn, title in benches:
         timer, fm = gettimer(ui, opts)
         timer(fn, title=title)
         fm.end()
 
 @command('perfwrite', formatteropts)
 def perfwrite(ui, repo, **opts):
     """microbenchmark ui.write
     """
     timer, fm = gettimer(ui, opts)
     def write():
         for i in range(100000):
             ui.write(('Testing write performance\n'))
     timer(write)
     fm.end()
 
 def uisetup(ui):
     if (util.safehasattr(cmdutil, 'openrevlog') and
         not util.safehasattr(commands, 'debugrevlogopts')):
         # for "historical portability":
         # In this case, Mercurial should be 1.9 (or a79fea6b3e77) -
         # 3.7 (or 5606f7d0d063). Therefore, '--dir' option for
         # openrevlog() should cause failure, because it has been
         # available since 3.5 (or 49c583ca48c4).
         def openrevlog(orig, repo, cmd, file_, opts):
             if opts.get('dir') and not util.safehasattr(repo, 'dirlog'):
                 raise error.Abort("This version doesn't support --dir option",
                                   hint="use 3.5 or later")
             return orig(repo, cmd, file_, opts)
         extensions.wrapfunction(cmdutil, 'openrevlog', openrevlog)

mercurial/bundle2.py

@@ -1,1999 +1,2000
 # bundle2.py - generic container format to transmit arbitrary data.
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """Handling of the new bundle2 format
 
 The goal of bundle2 is to act as an atomically packet to transmit a set of
 payloads in an application agnostic way. It consist in a sequence of "parts"
 that will be handed to and processed by the application layer.
 
 
 General format architecture
 ===========================
 
 The format is architectured as follow
 
  - magic string
  - stream level parameters
  - payload parts (any number)
  - end of stream marker.
 
 the Binary format
 ============================
 
 All numbers are unsigned and big-endian.
 
 stream level parameters
 ------------------------
 
 Binary format is as follow
 
 :params size: int32
 
   The total number of Bytes used by the parameters
 
 :params value: arbitrary number of Bytes
 
   A blob of `params size` containing the serialized version of all stream level
   parameters.
 
   The blob contains a space separated list of parameters. Parameters with value
   are stored in the form `<name>=<value>`. Both name and value are urlquoted.
 
   Empty name are obviously forbidden.
 
   Name MUST start with a letter. If this first letter is lower case, the
   parameter is advisory and can be safely ignored. However when the first
   letter is capital, the parameter is mandatory and the bundling process MUST
   stop if he is not able to proceed it.
 
   Stream parameters use a simple textual format for two main reasons:
 
   - Stream level parameters should remain simple and we want to discourage any
     crazy usage.
   - Textual data allow easy human inspection of a bundle2 header in case of
     troubles.
 
   Any Applicative level options MUST go into a bundle2 part instead.
 
 Payload part
 ------------------------
 
 Binary format is as follow
 
 :header size: int32
 
   The total number of Bytes used by the part header. When the header is empty
   (size = 0) this is interpreted as the end of stream marker.
 
 :header:
 
     The header defines how to interpret the part. It contains two piece of
     data: the part type, and the part parameters.
 
     The part type is used to route an application level handler, that can
     interpret payload.
 
     Part parameters are passed to the application level handler.  They are
     meant to convey information that will help the application level object to
     interpret the part payload.
 
     The binary format of the header is has follow
 
     :typesize: (one byte)
 
     :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*)
 
     :partid: A 32bits integer (unique in the bundle) that can be used to refer
              to this part.
 
     :parameters:
 
         Part's parameter may have arbitrary content, the binary structure is::
 
             <mandatory-count><advisory-count><param-sizes><param-data>
 
         :mandatory-count: 1 byte, number of mandatory parameters
 
         :advisory-count:  1 byte, number of advisory parameters
 
         :param-sizes:
 
             N couple of bytes, where N is the total number of parameters. Each
             couple contains (<size-of-key>, <size-of-value) for one parameter.
 
         :param-data:
 
             A blob of bytes from which each parameter key and value can be
             retrieved using the list of size couples stored in the previous
             field.
 
             Mandatory parameters comes first, then the advisory ones.
 
             Each parameter's key MUST be unique within the part.
 
 :payload:
 
     payload is a series of `<chunksize><chunkdata>`.
 
     `chunksize` is an int32, `chunkdata` are plain bytes (as much as
     `chunksize` says)` The payload part is concluded by a zero size chunk.
 
     The current implementation always produces either zero or one chunk.
     This is an implementation limitation that will ultimately be lifted.
 
     `chunksize` can be negative to trigger special case processing. No such
     processing is in place yet.
 
 Bundle processing
 ============================
 
 Each part is processed in order using a "part handler". Handler are registered
 for a certain part type.
 
 The matching of a part to its handler is case insensitive. The case of the
 part type is used to know if a part is mandatory or advisory. If the Part type
 contains any uppercase char it is considered mandatory. When no handler is
 known for a Mandatory part, the process is aborted and an exception is raised.
 If the part is advisory and no handler is known, the part is ignored. When the
 process is aborted, the full bundle is still read from the stream to keep the
 channel usable. But none of the part read from an abort are processed. In the
 future, dropping the stream may become an option for channel we do not care to
 preserve.
 """
 
 from __future__ import absolute_import, division
 
 import errno
 import os
 import re
 import string
 import struct
 import sys
 
 from .i18n import _
 from . import (
     changegroup,
     error,
     node as nodemod,
     obsolete,
     phases,
     pushkey,
     pycompat,
     tags,
     url,
     util,
 )
 
 urlerr = util.urlerr
 urlreq = util.urlreq
 
 _pack = struct.pack
 _unpack = struct.unpack
 
 _fstreamparamsize = '>i'
 _fpartheadersize = '>i'
 _fparttypesize = '>B'
 _fpartid = '>I'
 _fpayloadsize = '>i'
 _fpartparamcount = '>BB'
 
 preferedchunksize = 4096
 
 _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
 
 def outdebug(ui, message):
     """debug regarding output stream (bundling)"""
     if ui.configbool('devel', 'bundle2.debug'):
         ui.debug('bundle2-output: %s\n' % message)
 
 def indebug(ui, message):
     """debug on input stream (unbundling)"""
     if ui.configbool('devel', 'bundle2.debug'):
         ui.debug('bundle2-input: %s\n' % message)
 
 def validateparttype(parttype):
     """raise ValueError if a parttype contains invalid character"""
     if _parttypeforbidden.search(parttype):
         raise ValueError(parttype)
 
 def _makefpartparamsizes(nbparams):
     """return a struct format to read part parameter sizes
 
     The number parameters is variable so we need to build that format
     dynamically.
     """
     return '>'+('BB'*nbparams)
 
 parthandlermapping = {}
 
 def parthandler(parttype, params=()):
     """decorator that register a function as a bundle2 part handler
 
     eg::
 
         @parthandler('myparttype', ('mandatory', 'param', 'handled'))
         def myparttypehandler(...):
             '''process a part of type "my part".'''
             ...
     """
     validateparttype(parttype)
     def _decorator(func):
         lparttype = parttype.lower() # enforce lower case matching.
         assert lparttype not in parthandlermapping
         parthandlermapping[lparttype] = func
         func.params = frozenset(params)
         return func
     return _decorator
 
 class unbundlerecords(object):
     """keep record of what happens during and unbundle
 
     New records are added using `records.add('cat', obj)`. Where 'cat' is a
     category of record and obj is an arbitrary object.
 
     `records['cat']` will return all entries of this category 'cat'.
 
     Iterating on the object itself will yield `('category', obj)` tuples
     for all entries.
 
     All iterations happens in chronological order.
     """
 
     def __init__(self):
         self._categories = {}
         self._sequences = []
         self._replies = {}
 
     def add(self, category, entry, inreplyto=None):
         """add a new record of a given category.
 
         The entry can then be retrieved in the list returned by
         self['category']."""
         self._categories.setdefault(category, []).append(entry)
         self._sequences.append((category, entry))
         if inreplyto is not None:
             self.getreplies(inreplyto).add(category, entry)
 
     def getreplies(self, partid):
         """get the records that are replies to a specific part"""
         return self._replies.setdefault(partid, unbundlerecords())
 
     def __getitem__(self, cat):
         return tuple(self._categories.get(cat, ()))
 
     def __iter__(self):
         return iter(self._sequences)
 
     def __len__(self):
         return len(self._sequences)
 
     def __nonzero__(self):
         return bool(self._sequences)
 
     __bool__ = __nonzero__
 
 class bundleoperation(object):
     """an object that represents a single bundling process
 
     Its purpose is to carry unbundle-related objects and states.
 
     A new object should be created at the beginning of each bundle processing.
     The object is to be returned by the processing function.
 
     The object has very little content now it will ultimately contain:
     * an access to the repo the bundle is applied to,
     * a ui object,
     * a way to retrieve a transaction to add changes to the repo,
     * a way to record the result of processing each part,
     * a way to construct a bundle response when applicable.
     """
 
     def __init__(self, repo, transactiongetter, captureoutput=True):
         self.repo = repo
         self.ui = repo.ui
         self.records = unbundlerecords()
         self.reply = None
         self.captureoutput = captureoutput
         self.hookargs = {}
         self._gettransaction = transactiongetter
 
     def gettransaction(self):
         transaction = self._gettransaction()
 
         if self.hookargs:
             # the ones added to the transaction supercede those added
             # to the operation.
             self.hookargs.update(transaction.hookargs)
             transaction.hookargs = self.hookargs
 
         # mark the hookargs as flushed.  further attempts to add to
         # hookargs will result in an abort.
         self.hookargs = None
 
         return transaction
 
     def addhookargs(self, hookargs):
         if self.hookargs is None:
             raise error.ProgrammingError('attempted to add hookargs to '
                                          'operation after transaction started')
         self.hookargs.update(hookargs)
 
 class TransactionUnavailable(RuntimeError):
     pass
 
 def _notransaction():
     """default method to get a transaction while processing a bundle
 
     Raise an exception to highlight the fact that no transaction was expected
     to be created"""
     raise TransactionUnavailable()
 
 def applybundle(repo, unbundler, tr, source=None, url=None, **kwargs):
     # transform me into unbundler.apply() as soon as the freeze is lifted
     if isinstance(unbundler, unbundle20):
         tr.hookargs['bundle2'] = '1'
         if source is not None and 'source' not in tr.hookargs:
             tr.hookargs['source'] = source
         if url is not None and 'url' not in tr.hookargs:
             tr.hookargs['url'] = url
         return processbundle(repo, unbundler, lambda: tr)
     else:
         # the transactiongetter won't be used, but we might as well set it
         op = bundleoperation(repo, lambda: tr)
         _processchangegroup(op, unbundler, tr, source, url, **kwargs)
         return op
 
 class partiterator(object):
     def __init__(self, repo, op, unbundler):
         self.repo = repo
         self.op = op
         self.unbundler = unbundler
         self.iterator = None
         self.count = 0
         self.current = None
 
     def __enter__(self):
         def func():
             itr = enumerate(self.unbundler.iterparts())
             for count, p in itr:
                 self.count = count
                 self.current = p
                 yield p
                 p.consume()
                 self.current = None
         self.iterator = func()
         return self.iterator
 
     def __exit__(self, type, exc, tb):
         if not self.iterator:
             return
 
         # Only gracefully abort in a normal exception situation. User aborts
         # like Ctrl+C throw a KeyboardInterrupt which is not a base Exception,
         # and should not gracefully cleanup.
         if isinstance(exc, Exception):
             # Any exceptions seeking to the end of the bundle at this point are
             # almost certainly related to the underlying stream being bad.
             # And, chances are that the exception we're handling is related to
             # getting in that bad state. So, we swallow the seeking error and
             # re-raise the original error.
             seekerror = False
             try:
                 if self.current:
                     # consume the part content to not corrupt the stream.
                     self.current.consume()
 
                 for part in self.iterator:
                     # consume the bundle content
                     part.consume()
             except Exception:
                 seekerror = True
 
             # Small hack to let caller code distinguish exceptions from bundle2
             # processing from processing the old format. This is mostly needed
             # to handle different return codes to unbundle according to the type
             # of bundle. We should probably clean up or drop this return code
             # craziness in a future version.
             exc.duringunbundle2 = True
             salvaged = []
             replycaps = None
             if self.op.reply is not None:
                 salvaged = self.op.reply.salvageoutput()
                 replycaps = self.op.reply.capabilities
             exc._replycaps = replycaps
             exc._bundle2salvagedoutput = salvaged
 
             # Re-raising from a variable loses the original stack. So only use
             # that form if we need to.
             if seekerror:
                 raise exc
 
         self.repo.ui.debug('bundle2-input-bundle: %i parts total\n' %
                            self.count)
 
 def processbundle(repo, unbundler, transactiongetter=None, op=None):
     """This function process a bundle, apply effect to/from a repo
 
     It iterates over each part then searches for and uses the proper handling
     code to process the part. Parts are processed in order.
 
     Unknown Mandatory part will abort the process.
 
     It is temporarily possible to provide a prebuilt bundleoperation to the
     function. This is used to ensure output is properly propagated in case of
     an error during the unbundling. This output capturing part will likely be
     reworked and this ability will probably go away in the process.
     """
     if op is None:
         if transactiongetter is None:
             transactiongetter = _notransaction
         op = bundleoperation(repo, transactiongetter)
     # todo:
     # - replace this is a init function soon.
     # - exception catching
     unbundler.params
     if repo.ui.debugflag:
         msg = ['bundle2-input-bundle:']
         if unbundler.params:
             msg.append(' %i params' % len(unbundler.params))
         if op._gettransaction is None or op._gettransaction is _notransaction:
             msg.append(' no-transaction')
         else:
             msg.append(' with-transaction')
         msg.append('\n')
         repo.ui.debug(''.join(msg))
 
     processparts(repo, op, unbundler)
 
     return op
 
 def processparts(repo, op, unbundler):
     with partiterator(repo, op, unbundler) as parts:
         for part in parts:
             _processpart(op, part)
 
 def _processchangegroup(op, cg, tr, source, url, **kwargs):
     ret = cg.apply(op.repo, tr, source, url, **kwargs)
     op.records.add('changegroup', {
         'return': ret,
     })
     return ret
 
 def _gethandler(op, part):
     status = 'unknown' # used by debug output
     try:
         handler = parthandlermapping.get(part.type)
         if handler is None:
             status = 'unsupported-type'
             raise error.BundleUnknownFeatureError(parttype=part.type)
         indebug(op.ui, 'found a handler for part %s' % part.type)
         unknownparams = part.mandatorykeys - handler.params
         if unknownparams:
             unknownparams = list(unknownparams)
             unknownparams.sort()
             status = 'unsupported-params (%s)' % ', '.join(unknownparams)
             raise error.BundleUnknownFeatureError(parttype=part.type,
                                                   params=unknownparams)
         status = 'supported'
     except error.BundleUnknownFeatureError as exc:
         if part.mandatory: # mandatory parts
             raise
         indebug(op.ui, 'ignoring unsupported advisory part %s' % exc)
         return # skip to part processing
     finally:
         if op.ui.debugflag:
             msg = ['bundle2-input-part: "%s"' % part.type]
             if not part.mandatory:
                 msg.append(' (advisory)')
             nbmp = len(part.mandatorykeys)
             nbap = len(part.params) - nbmp
             if nbmp or nbap:
                 msg.append(' (params:')
                 if nbmp:
                     msg.append(' %i mandatory' % nbmp)
                 if nbap:
                     msg.append(' %i advisory' % nbmp)
                 msg.append(')')
             msg.append(' %s\n' % status)
             op.ui.debug(''.join(msg))
 
     return handler
 
 def _processpart(op, part):
     """process a single part from a bundle
 
     The part is guaranteed to have been fully consumed when the function exits
     (even if an exception is raised)."""
     handler = _gethandler(op, part)
     if handler is None:
         return
 
     # handler is called outside the above try block so that we don't
     # risk catching KeyErrors from anything other than the
     # parthandlermapping lookup (any KeyError raised by handler()
     # itself represents a defect of a different variety).
     output = None
     if op.captureoutput and op.reply is not None:
         op.ui.pushbuffer(error=True, subproc=True)
         output = ''
     try:
         handler(op, part)
     finally:
         if output is not None:
             output = op.ui.popbuffer()
         if output:
             outpart = op.reply.newpart('output', data=output,
                                        mandatory=False)
             outpart.addparam(
                 'in-reply-to', pycompat.bytestr(part.id), mandatory=False)
 
 def decodecaps(blob):
     """decode a bundle2 caps bytes blob into a dictionary
 
     The blob is a list of capabilities (one per line)
     Capabilities may have values using a line of the form::
 
         capability=value1,value2,value3
 
     The values are always a list."""
     caps = {}
     for line in blob.splitlines():
         if not line:
             continue
         if '=' not in line:
             key, vals = line, ()
         else:
             key, vals = line.split('=', 1)
             vals = vals.split(',')
         key = urlreq.unquote(key)
         vals = [urlreq.unquote(v) for v in vals]
         caps[key] = vals
     return caps
 
 def encodecaps(caps):
     """encode a bundle2 caps dictionary into a bytes blob"""
     chunks = []
     for ca in sorted(caps):
         vals = caps[ca]
         ca = urlreq.quote(ca)
         vals = [urlreq.quote(v) for v in vals]
         if vals:
             ca = "%s=%s" % (ca, ','.join(vals))
         chunks.append(ca)
     return '\n'.join(chunks)
 
 bundletypes = {
     "": ("", 'UN'),       # only when using unbundle on ssh and old http servers
                           # since the unification ssh accepts a header but there
                           # is no capability signaling it.
     "HG20": (), # special-cased below
     "HG10UN": ("HG10UN", 'UN'),
     "HG10BZ": ("HG10", 'BZ'),
     "HG10GZ": ("HG10GZ", 'GZ'),
 }
 
 # hgweb uses this list to communicate its preferred type
 bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN']
 
 class bundle20(object):
     """represent an outgoing bundle2 container
 
     Use the `addparam` method to add stream level parameter. and `newpart` to
     populate it. Then call `getchunks` to retrieve all the binary chunks of
     data that compose the bundle2 container."""
 
     _magicstring = 'HG20'
 
     def __init__(self, ui, capabilities=()):
         self.ui = ui
         self._params = []
         self._parts = []
         self.capabilities = dict(capabilities)
         self._compengine = util.compengines.forbundletype('UN')
         self._compopts = None
 
     def setcompression(self, alg, compopts=None):
         """setup core part compression to <alg>"""
         if alg in (None, 'UN'):
             return
         assert not any(n.lower() == 'compression' for n, v in self._params)
         self.addparam('Compression', alg)
         self._compengine = util.compengines.forbundletype(alg)
         self._compopts = compopts
 
     @property
     def nbparts(self):
         """total number of parts added to the bundler"""
         return len(self._parts)
 
     # methods used to defines the bundle2 content
     def addparam(self, name, value=None):
         """add a stream level parameter"""
         if not name:
             raise ValueError(r'empty parameter name')
         if name[0:1] not in pycompat.bytestr(string.ascii_letters):
             raise ValueError(r'non letter first character: %s' % name)
         self._params.append((name, value))
 
     def addpart(self, part):
         """add a new part to the bundle2 container
 
         Parts contains the actual applicative payload."""
         assert part.id is None
         part.id = len(self._parts) # very cheap counter
         self._parts.append(part)
 
     def newpart(self, typeid, *args, **kwargs):
         """create a new part and add it to the containers
 
         As the part is directly added to the containers. For now, this means
         that any failure to properly initialize the part after calling
         ``newpart`` should result in a failure of the whole bundling process.
 
         You can still fall back to manually create and add if you need better
         control."""
         part = bundlepart(typeid, *args, **kwargs)
         self.addpart(part)
         return part
 
     # methods used to generate the bundle2 stream
     def getchunks(self):
         if self.ui.debugflag:
             msg = ['bundle2-output-bundle: "%s",' % self._magicstring]
             if self._params:
                 msg.append(' (%i params)' % len(self._params))
             msg.append(' %i parts total\n' % len(self._parts))
             self.ui.debug(''.join(msg))
         outdebug(self.ui, 'start emission of %s stream' % self._magicstring)
         yield self._magicstring
         param = self._paramchunk()
         outdebug(self.ui, 'bundle parameter: %s' % param)
         yield _pack(_fstreamparamsize, len(param))
         if param:
             yield param
         for chunk in self._compengine.compressstream(self._getcorechunk(),
                                                      self._compopts):
             yield chunk
 
     def _paramchunk(self):
         """return a encoded version of all stream parameters"""
         blocks = []
         for par, value in self._params:
             par = urlreq.quote(par)
             if value is not None:
                 value = urlreq.quote(value)
                 par = '%s=%s' % (par, value)
             blocks.append(par)
         return ' '.join(blocks)
 
     def _getcorechunk(self):
         """yield chunk for the core part of the bundle
 
         (all but headers and parameters)"""
         outdebug(self.ui, 'start of parts')
         for part in self._parts:
             outdebug(self.ui, 'bundle part: "%s"' % part.type)
             for chunk in part.getchunks(ui=self.ui):
                 yield chunk
         outdebug(self.ui, 'end of bundle')
         yield _pack(_fpartheadersize, 0)
 
 
     def salvageoutput(self):
         """return a list with a copy of all output parts in the bundle
 
         This is meant to be used during error handling to make sure we preserve
         server output"""
         salvaged = []
         for part in self._parts:
             if part.type.startswith('output'):
                 salvaged.append(part.copy())
         return salvaged
 
 
 class unpackermixin(object):
     """A mixin to extract bytes and struct data from a stream"""
 
     def __init__(self, fp):
         self._fp = fp
 
     def _unpack(self, format):
         """unpack this struct format from the stream
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         data = self._readexact(struct.calcsize(format))
         return _unpack(format, data)
 
     def _readexact(self, size):
         """read exactly <size> bytes from the stream
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         return changegroup.readexactly(self._fp, size)
 
 def getunbundler(ui, fp, magicstring=None):
     """return a valid unbundler object for a given magicstring"""
     if magicstring is None:
         magicstring = changegroup.readexactly(fp, 4)
     magic, version = magicstring[0:2], magicstring[2:4]
     if magic != 'HG':
         ui.debug(
             "error: invalid magic: %r (version %r), should be 'HG'\n"
             % (magic, version))
         raise error.Abort(_('not a Mercurial bundle'))
     unbundlerclass = formatmap.get(version)
     if unbundlerclass is None:
         raise error.Abort(_('unknown bundle version %s') % version)
     unbundler = unbundlerclass(ui, fp)
     indebug(ui, 'start processing of %s stream' % magicstring)
     return unbundler
 
 class unbundle20(unpackermixin):
     """interpret a bundle2 stream
 
     This class is fed with a binary stream and yields parts through its
     `iterparts` methods."""
 
     _magicstring = 'HG20'
 
     def __init__(self, ui, fp):
         """If header is specified, we do not read it out of the stream."""
         self.ui = ui
         self._compengine = util.compengines.forbundletype('UN')
         self._compressed = None
         super(unbundle20, self).__init__(fp)
 
     @util.propertycache
     def params(self):
         """dictionary of stream level parameters"""
         indebug(self.ui, 'reading bundle2 stream parameters')
         params = {}
         paramssize = self._unpack(_fstreamparamsize)[0]
         if paramssize < 0:
             raise error.BundleValueError('negative bundle param size: %i'
                                          % paramssize)
         if paramssize:
             params = self._readexact(paramssize)
             params = self._processallparams(params)
         return params
 
     def _processallparams(self, paramsblock):
         """"""
         params = util.sortdict()
         for p in paramsblock.split(' '):
             p = p.split('=', 1)
             p = [urlreq.unquote(i) for i in p]
             if len(p) < 2:
                 p.append(None)
             self._processparam(*p)
             params[p[0]] = p[1]
         return params
 
 
     def _processparam(self, name, value):
         """process a parameter, applying its effect if needed
 
         Parameter starting with a lower case letter are advisory and will be
         ignored when unknown.  Those starting with an upper case letter are
         mandatory and will this function will raise a KeyError when unknown.
 
         Note: no option are currently supported. Any input will be either
               ignored or failing.
         """
         if not name:
             raise ValueError(r'empty parameter name')
         if name[0:1] not in pycompat.bytestr(string.ascii_letters):
             raise ValueError(r'non letter first character: %s' % name)
         try:
             handler = b2streamparamsmap[name.lower()]
         except KeyError:
             if name[0:1].islower():
                 indebug(self.ui, "ignoring unknown parameter %s" % name)
             else:
                 raise error.BundleUnknownFeatureError(params=(name,))
         else:
             handler(self, name, value)
 
     def _forwardchunks(self):
         """utility to transfer a bundle2 as binary
 
         This is made necessary by the fact the 'getbundle' command over 'ssh'
         have no way to know then the reply end, relying on the bundle to be
         interpreted to know its end. This is terrible and we are sorry, but we
         needed to move forward to get general delta enabled.
         """
         yield self._magicstring
         assert 'params' not in vars(self)
         paramssize = self._unpack(_fstreamparamsize)[0]
         if paramssize < 0:
             raise error.BundleValueError('negative bundle param size: %i'
                                          % paramssize)
         yield _pack(_fstreamparamsize, paramssize)
         if paramssize:
             params = self._readexact(paramssize)
             self._processallparams(params)
             yield params
             assert self._compengine.bundletype == 'UN'
         # From there, payload might need to be decompressed
         self._fp = self._compengine.decompressorreader(self._fp)
         emptycount = 0
         while emptycount < 2:
             # so we can brainlessly loop
             assert _fpartheadersize == _fpayloadsize
             size = self._unpack(_fpartheadersize)[0]
             yield _pack(_fpartheadersize, size)
             if size:
                 emptycount = 0
             else:
                 emptycount += 1
                 continue
             if size == flaginterrupt:
                 continue
             elif size < 0:
                 raise error.BundleValueError('negative chunk size: %i')
             yield self._readexact(size)
 
 
-    def iterparts(self):
+    def iterparts(self, seekable=False):
         """yield all parts contained in the stream"""
+        cls = seekableunbundlepart if seekable else unbundlepart
         # make sure param have been loaded
         self.params
         # From there, payload need to be decompressed
         self._fp = self._compengine.decompressorreader(self._fp)
         indebug(self.ui, 'start extraction of bundle2 parts')
         headerblock = self._readpartheader()
         while headerblock is not None:
-            part = seekableunbundlepart(self.ui, headerblock, self._fp)
+            part = cls(self.ui, headerblock, self._fp)
             yield part
             # Ensure part is fully consumed so we can start reading the next
             # part.
             part.consume()
 
             headerblock = self._readpartheader()
         indebug(self.ui, 'end of bundle2 stream')
 
     def _readpartheader(self):
         """reads a part header size and return the bytes blob
 
         returns None if empty"""
         headersize = self._unpack(_fpartheadersize)[0]
         if headersize < 0:
             raise error.BundleValueError('negative part header size: %i'
                                          % headersize)
         indebug(self.ui, 'part header size: %i' % headersize)
         if headersize:
             return self._readexact(headersize)
         return None
 
     def compressed(self):
         self.params # load params
         return self._compressed
 
     def close(self):
         """close underlying file"""
         if util.safehasattr(self._fp, 'close'):
             return self._fp.close()
 
 formatmap = {'20': unbundle20}
 
 b2streamparamsmap = {}
 
 def b2streamparamhandler(name):
     """register a handler for a stream level parameter"""
     def decorator(func):
         assert name not in formatmap
         b2streamparamsmap[name] = func
         return func
     return decorator
 
 @b2streamparamhandler('compression')
 def processcompression(unbundler, param, value):
     """read compression parameter and install payload decompression"""
     if value not in util.compengines.supportedbundletypes:
         raise error.BundleUnknownFeatureError(params=(param,),
                                               values=(value,))
     unbundler._compengine = util.compengines.forbundletype(value)
     if value is not None:
         unbundler._compressed = True
 
 class bundlepart(object):
     """A bundle2 part contains application level payload
 
     The part `type` is used to route the part to the application level
     handler.
 
     The part payload is contained in ``part.data``. It could be raw bytes or a
     generator of byte chunks.
 
     You can add parameters to the part using the ``addparam`` method.
     Parameters can be either mandatory (default) or advisory. Remote side
     should be able to safely ignore the advisory ones.
 
     Both data and parameters cannot be modified after the generation has begun.
     """
 
     def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
                  data='', mandatory=True):
         validateparttype(parttype)
         self.id = None
         self.type = parttype
         self._data = data
         self._mandatoryparams = list(mandatoryparams)
         self._advisoryparams = list(advisoryparams)
         # checking for duplicated entries
         self._seenparams = set()
         for pname, __ in self._mandatoryparams + self._advisoryparams:
             if pname in self._seenparams:
                 raise error.ProgrammingError('duplicated params: %s' % pname)
             self._seenparams.add(pname)
         # status of the part's generation:
         # - None: not started,
         # - False: currently generated,
         # - True: generation done.
         self._generated = None
         self.mandatory = mandatory
 
     def __repr__(self):
         cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__)
         return ('<%s object at %x; id: %s; type: %s; mandatory: %s>'
                 % (cls, id(self), self.id, self.type, self.mandatory))
 
     def copy(self):
         """return a copy of the part
 
         The new part have the very same content but no partid assigned yet.
         Parts with generated data cannot be copied."""
         assert not util.safehasattr(self.data, 'next')
         return self.__class__(self.type, self._mandatoryparams,
                               self._advisoryparams, self._data, self.mandatory)
 
     # methods used to defines the part content
     @property
     def data(self):
         return self._data
 
     @data.setter
     def data(self, data):
         if self._generated is not None:
             raise error.ReadOnlyPartError('part is being generated')
         self._data = data
 
     @property
     def mandatoryparams(self):
         # make it an immutable tuple to force people through ``addparam``
         return tuple(self._mandatoryparams)
 
     @property
     def advisoryparams(self):
         # make it an immutable tuple to force people through ``addparam``
         return tuple(self._advisoryparams)
 
     def addparam(self, name, value='', mandatory=True):
         """add a parameter to the part
 
         If 'mandatory' is set to True, the remote handler must claim support
         for this parameter or the unbundling will be aborted.
 
         The 'name' and 'value' cannot exceed 255 bytes each.
         """
         if self._generated is not None:
             raise error.ReadOnlyPartError('part is being generated')
         if name in self._seenparams:
             raise ValueError('duplicated params: %s' % name)
         self._seenparams.add(name)
         params = self._advisoryparams
         if mandatory:
             params = self._mandatoryparams
         params.append((name, value))
 
     # methods used to generates the bundle2 stream
     def getchunks(self, ui):
         if self._generated is not None:
             raise error.ProgrammingError('part can only be consumed once')
         self._generated = False
 
         if ui.debugflag:
             msg = ['bundle2-output-part: "%s"' % self.type]
             if not self.mandatory:
                 msg.append(' (advisory)')
             nbmp = len(self.mandatoryparams)
             nbap = len(self.advisoryparams)
             if nbmp or nbap:
                 msg.append(' (params:')
                 if nbmp:
                     msg.append(' %i mandatory' % nbmp)
                 if nbap:
                     msg.append(' %i advisory' % nbmp)
                 msg.append(')')
             if not self.data:
                 msg.append(' empty payload')
             elif (util.safehasattr(self.data, 'next')
                   or util.safehasattr(self.data, '__next__')):
                 msg.append(' streamed payload')
             else:
                 msg.append(' %i bytes payload' % len(self.data))
             msg.append('\n')
             ui.debug(''.join(msg))
 
         #### header
         if self.mandatory:
             parttype = self.type.upper()
         else:
             parttype = self.type.lower()
         outdebug(ui, 'part %s: "%s"' % (pycompat.bytestr(self.id), parttype))
         ## parttype
         header = [_pack(_fparttypesize, len(parttype)),
                   parttype, _pack(_fpartid, self.id),
                  ]
         ## parameters
         # count
         manpar = self.mandatoryparams
         advpar = self.advisoryparams
         header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
         # size
         parsizes = []
         for key, value in manpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         for key, value in advpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes)
         header.append(paramsizes)
         # key, value
         for key, value in manpar:
             header.append(key)
             header.append(value)
         for key, value in advpar:
             header.append(key)
             header.append(value)
         ## finalize header
         try:
             headerchunk = ''.join(header)
         except TypeError:
             raise TypeError(r'Found a non-bytes trying to '
                             r'build bundle part header: %r' % header)
         outdebug(ui, 'header chunk size: %i' % len(headerchunk))
         yield _pack(_fpartheadersize, len(headerchunk))
         yield headerchunk
         ## payload
         try:
             for chunk in self._payloadchunks():
                 outdebug(ui, 'payload chunk size: %i' % len(chunk))
                 yield _pack(_fpayloadsize, len(chunk))
                 yield chunk
         except GeneratorExit:
             # GeneratorExit means that nobody is listening for our
             # results anyway, so just bail quickly rather than trying
             # to produce an error part.
             ui.debug('bundle2-generatorexit\n')
             raise
         except BaseException as exc:
             bexc = util.forcebytestr(exc)
             # backup exception data for later
             ui.debug('bundle2-input-stream-interrupt: encoding exception %s'
                      % bexc)
             tb = sys.exc_info()[2]
             msg = 'unexpected error: %s' % bexc
             interpart = bundlepart('error:abort', [('message', msg)],
                                    mandatory=False)
             interpart.id = 0
             yield _pack(_fpayloadsize, -1)
             for chunk in interpart.getchunks(ui=ui):
                 yield chunk
             outdebug(ui, 'closing payload chunk')
             # abort current part payload
             yield _pack(_fpayloadsize, 0)
             pycompat.raisewithtb(exc, tb)
         # end of payload
         outdebug(ui, 'closing payload chunk')
         yield _pack(_fpayloadsize, 0)
         self._generated = True
 
     def _payloadchunks(self):
         """yield chunks of a the part payload
 
         Exists to handle the different methods to provide data to a part."""
         # we only support fixed size data now.
         # This will be improved in the future.
         if (util.safehasattr(self.data, 'next')
             or util.safehasattr(self.data, '__next__')):
             buff = util.chunkbuffer(self.data)
             chunk = buff.read(preferedchunksize)
             while chunk:
                 yield chunk
                 chunk = buff.read(preferedchunksize)
         elif len(self.data):
             yield self.data
 
 
 flaginterrupt = -1
 
 class interrupthandler(unpackermixin):
     """read one part and process it with restricted capability
 
     This allows to transmit exception raised on the producer size during part
     iteration while the consumer is reading a part.
 
     Part processed in this manner only have access to a ui object,"""
 
     def __init__(self, ui, fp):
         super(interrupthandler, self).__init__(fp)
         self.ui = ui
 
     def _readpartheader(self):
         """reads a part header size and return the bytes blob
 
         returns None if empty"""
         headersize = self._unpack(_fpartheadersize)[0]
         if headersize < 0:
             raise error.BundleValueError('negative part header size: %i'
                                          % headersize)
         indebug(self.ui, 'part header size: %i\n' % headersize)
         if headersize:
             return self._readexact(headersize)
         return None
 
     def __call__(self):
 
         self.ui.debug('bundle2-input-stream-interrupt:'
                       ' opening out of band context\n')
         indebug(self.ui, 'bundle2 stream interruption, looking for a part.')
         headerblock = self._readpartheader()
         if headerblock is None:
             indebug(self.ui, 'no part found during interruption.')
             return
-        part = seekableunbundlepart(self.ui, headerblock, self._fp)
+        part = unbundlepart(self.ui, headerblock, self._fp)
         op = interruptoperation(self.ui)
         hardabort = False
         try:
             _processpart(op, part)
         except (SystemExit, KeyboardInterrupt):
             hardabort = True
             raise
         finally:
             if not hardabort:
                 part.consume()
         self.ui.debug('bundle2-input-stream-interrupt:'
                       ' closing out of band context\n')
 
 class interruptoperation(object):
     """A limited operation to be use by part handler during interruption
 
     It only have access to an ui object.
     """
 
     def __init__(self, ui):
         self.ui = ui
         self.reply = None
         self.captureoutput = False
 
     @property
     def repo(self):
         raise error.ProgrammingError('no repo access from stream interruption')
 
     def gettransaction(self):
         raise TransactionUnavailable('no repo access from stream interruption')
 
 def decodepayloadchunks(ui, fh):
     """Reads bundle2 part payload data into chunks.
 
     Part payload data consists of framed chunks. This function takes
     a file handle and emits those chunks.
     """
     headersize = struct.calcsize(_fpayloadsize)
     readexactly = changegroup.readexactly
 
     chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
     indebug(ui, 'payload chunk size: %i' % chunksize)
 
     while chunksize:
         if chunksize >= 0:
             yield readexactly(fh, chunksize)
         elif chunksize == flaginterrupt:
             # Interrupt "signal" detected. The regular stream is interrupted
             # and a bundle2 part follows. Consume it.
             interrupthandler(ui, fh)()
         else:
             raise error.BundleValueError(
                 'negative payload chunk size: %s' % chunksize)
 
         chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
         indebug(ui, 'payload chunk size: %i' % chunksize)
 
 class unbundlepart(unpackermixin):
     """a bundle part read from a bundle"""
 
     def __init__(self, ui, header, fp):
         super(unbundlepart, self).__init__(fp)
         self._seekable = (util.safehasattr(fp, 'seek') and
                           util.safehasattr(fp, 'tell'))
         self.ui = ui
         # unbundle state attr
         self._headerdata = header
         self._headeroffset = 0
         self._initialized = False
         self.consumed = False
         # part data
         self.id = None
         self.type = None
         self.mandatoryparams = None
         self.advisoryparams = None
         self.params = None
         self.mandatorykeys = ()
         self._readheader()
         self._mandatory = None
         self._pos = 0
 
     def _fromheader(self, size):
         """return the next <size> byte from the header"""
         offset = self._headeroffset
         data = self._headerdata[offset:(offset + size)]
         self._headeroffset = offset + size
         return data
 
     def _unpackheader(self, format):
         """read given format from header
 
         This automatically compute the size of the format to read."""
         data = self._fromheader(struct.calcsize(format))
         return _unpack(format, data)
 
     def _initparams(self, mandatoryparams, advisoryparams):
         """internal function to setup all logic related parameters"""
         # make it read only to prevent people touching it by mistake.
         self.mandatoryparams = tuple(mandatoryparams)
         self.advisoryparams  = tuple(advisoryparams)
         # user friendly UI
         self.params = util.sortdict(self.mandatoryparams)
         self.params.update(self.advisoryparams)
         self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
 
     def _readheader(self):
         """read the header and setup the object"""
         typesize = self._unpackheader(_fparttypesize)[0]
         self.type = self._fromheader(typesize)
         indebug(self.ui, 'part type: "%s"' % self.type)
         self.id = self._unpackheader(_fpartid)[0]
         indebug(self.ui, 'part id: "%s"' % pycompat.bytestr(self.id))
         # extract mandatory bit from type
         self.mandatory = (self.type != self.type.lower())
         self.type = self.type.lower()
         ## reading parameters
         # param count
         mancount, advcount = self._unpackheader(_fpartparamcount)
         indebug(self.ui, 'part parameters: %i' % (mancount + advcount))
         # param size
         fparamsizes = _makefpartparamsizes(mancount + advcount)
         paramsizes = self._unpackheader(fparamsizes)
         # make it a list of couple again
         paramsizes = list(zip(paramsizes[::2], paramsizes[1::2]))
         # split mandatory from advisory
         mansizes = paramsizes[:mancount]
         advsizes = paramsizes[mancount:]
         # retrieve param value
         manparams = []
         for key, value in mansizes:
             manparams.append((self._fromheader(key), self._fromheader(value)))
         advparams = []
         for key, value in advsizes:
             advparams.append((self._fromheader(key), self._fromheader(value)))
         self._initparams(manparams, advparams)
         ## part payload
         self._payloadstream = util.chunkbuffer(self._payloadchunks())
         # we read the data, tell it
         self._initialized = True
 
     def _payloadchunks(self):
         """Generator of decoded chunks in the payload."""
         return decodepayloadchunks(self.ui, self._fp)
 
     def consume(self):
         """Read the part payload until completion.
 
         By consuming the part data, the underlying stream read offset will
         be advanced to the next part (or end of stream).
         """
         if self.consumed:
             return
 
         chunk = self.read(32768)
         while chunk:
             self._pos += len(chunk)
             chunk = self.read(32768)
 
     def read(self, size=None):
         """read payload data"""
         if not self._initialized:
             self._readheader()
         if size is None:
             data = self._payloadstream.read()
         else:
             data = self._payloadstream.read(size)
         self._pos += len(data)
         if size is None or len(data) < size:
             if not self.consumed and self._pos:
                 self.ui.debug('bundle2-input-part: total payload size %i\n'
                               % self._pos)
             self.consumed = True
         return data
 
 class seekableunbundlepart(unbundlepart):
     """A bundle2 part in a bundle that is seekable.
 
     Regular ``unbundlepart`` instances can only be read once. This class
     extends ``unbundlepart`` to enable bi-directional seeking within the
     part.
 
     Bundle2 part data consists of framed chunks. Offsets when seeking
     refer to the decoded data, not the offsets in the underlying bundle2
     stream.
 
     To facilitate quickly seeking within the decoded data, instances of this
     class maintain a mapping between offsets in the underlying stream and
     the decoded payload. This mapping will consume memory in proportion
     to the number of chunks within the payload (which almost certainly
     increases in proportion with the size of the part).
     """
     def __init__(self, ui, header, fp):
         # (payload, file) offsets for chunk starts.
         self._chunkindex = []
 
         super(seekableunbundlepart, self).__init__(ui, header, fp)
 
     def _payloadchunks(self, chunknum=0):
         '''seek to specified chunk and start yielding data'''
         if len(self._chunkindex) == 0:
             assert chunknum == 0, 'Must start with chunk 0'
             self._chunkindex.append((0, self._tellfp()))
         else:
             assert chunknum < len(self._chunkindex), \
                    'Unknown chunk %d' % chunknum
             self._seekfp(self._chunkindex[chunknum][1])
 
         pos = self._chunkindex[chunknum][0]
 
         for chunk in decodepayloadchunks(self.ui, self._fp):
             chunknum += 1
             pos += len(chunk)
             if chunknum == len(self._chunkindex):
                 self._chunkindex.append((pos, self._tellfp()))
 
             yield chunk
 
     def _findchunk(self, pos):
         '''for a given payload position, return a chunk number and offset'''
         for chunk, (ppos, fpos) in enumerate(self._chunkindex):
             if ppos == pos:
                 return chunk, 0
             elif ppos > pos:
                 return chunk - 1, pos - self._chunkindex[chunk - 1][0]
         raise ValueError('Unknown chunk')
 
     def tell(self):
         return self._pos
 
     def seek(self, offset, whence=os.SEEK_SET):
         if whence == os.SEEK_SET:
             newpos = offset
         elif whence == os.SEEK_CUR:
             newpos = self._pos + offset
         elif whence == os.SEEK_END:
             if not self.consumed:
                 self.read()
             newpos = self._chunkindex[-1][0] - offset
         else:
             raise ValueError('Unknown whence value: %r' % (whence,))
 
         if newpos > self._chunkindex[-1][0] and not self.consumed:
             self.read()
         if not 0 <= newpos <= self._chunkindex[-1][0]:
             raise ValueError('Offset out of range')
 
         if self._pos != newpos:
             chunk, internaloffset = self._findchunk(newpos)
             self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk))
             adjust = self.read(internaloffset)
             if len(adjust) != internaloffset:
                 raise error.Abort(_('Seek failed\n'))
             self._pos = newpos
 
     def _seekfp(self, offset, whence=0):
         """move the underlying file pointer
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         if self._seekable:
             return self._fp.seek(offset, whence)
         else:
             raise NotImplementedError(_('File pointer is not seekable'))
 
     def _tellfp(self):
         """return the file offset, or None if file is not seekable
 
         This method is meant for internal usage by the bundle2 protocol only.
         They directly manipulate the low level stream including bundle2 level
         instruction.
 
         Do not use it to implement higher-level logic or methods."""
         if self._seekable:
             try:
                 return self._fp.tell()
             except IOError as e:
                 if e.errno == errno.ESPIPE:
                     self._seekable = False
                 else:
                     raise
         return None
 
 # These are only the static capabilities.
 # Check the 'getrepocaps' function for the rest.
 capabilities = {'HG20': (),
                 'error': ('abort', 'unsupportedcontent', 'pushraced',
                           'pushkey'),
                 'listkeys': (),
                 'pushkey': (),
                 'digests': tuple(sorted(util.DIGESTS.keys())),
                 'remote-changegroup': ('http', 'https'),
                 'hgtagsfnodes': (),
                 'phases': ('heads',),
                }
 
 def getrepocaps(repo, allowpushback=False):
     """return the bundle2 capabilities for a given repo
 
     Exists to allow extensions (like evolution) to mutate the capabilities.
     """
     caps = capabilities.copy()
     caps['changegroup'] = tuple(sorted(
         changegroup.supportedincomingversions(repo)))
     if obsolete.isenabled(repo, obsolete.exchangeopt):
         supportedformat = tuple('V%i' % v for v in obsolete.formats)
         caps['obsmarkers'] = supportedformat
     if allowpushback:
         caps['pushback'] = ()
     cpmode = repo.ui.config('server', 'concurrent-push-mode')
     if cpmode == 'check-related':
         caps['checkheads'] = ('related',)
     if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'):
         caps.pop('phases')
     return caps
 
 def bundle2caps(remote):
     """return the bundle capabilities of a peer as dict"""
     raw = remote.capable('bundle2')
     if not raw and raw != '':
         return {}
     capsblob = urlreq.unquote(remote.capable('bundle2'))
     return decodecaps(capsblob)
 
 def obsmarkersversion(caps):
     """extract the list of supported obsmarkers versions from a bundle2caps dict
     """
     obscaps = caps.get('obsmarkers', ())
     return [int(c[1:]) for c in obscaps if c.startswith('V')]
 
 def writenewbundle(ui, repo, source, filename, bundletype, outgoing, opts,
                    vfs=None, compression=None, compopts=None):
     if bundletype.startswith('HG10'):
         cg = changegroup.makechangegroup(repo, outgoing, '01', source)
         return writebundle(ui, cg, filename, bundletype, vfs=vfs,
                            compression=compression, compopts=compopts)
     elif not bundletype.startswith('HG20'):
         raise error.ProgrammingError('unknown bundle type: %s' % bundletype)
 
     caps = {}
     if 'obsolescence' in opts:
         caps['obsmarkers'] = ('V1',)
     bundle = bundle20(ui, caps)
     bundle.setcompression(compression, compopts)
     _addpartsfromopts(ui, repo, bundle, source, outgoing, opts)
     chunkiter = bundle.getchunks()
 
     return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
 
 def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts):
     # We should eventually reconcile this logic with the one behind
     # 'exchange.getbundle2partsgenerator'.
     #
     # The type of input from 'getbundle' and 'writenewbundle' are a bit
     # different right now. So we keep them separated for now for the sake of
     # simplicity.
 
     # we always want a changegroup in such bundle
     cgversion = opts.get('cg.version')
     if cgversion is None:
         cgversion = changegroup.safeversion(repo)
     cg = changegroup.makechangegroup(repo, outgoing, cgversion, source)
     part = bundler.newpart('changegroup', data=cg.getchunks())
     part.addparam('version', cg.version)
     if 'clcount' in cg.extras:
         part.addparam('nbchanges', '%d' % cg.extras['clcount'],
                       mandatory=False)
     if opts.get('phases') and repo.revs('%ln and secret()',
                                         outgoing.missingheads):
         part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
 
     addparttagsfnodescache(repo, bundler, outgoing)
 
     if opts.get('obsolescence', False):
         obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
         buildobsmarkerspart(bundler, obsmarkers)
 
     if opts.get('phases', False):
         headsbyphase = phases.subsetphaseheads(repo, outgoing.missing)
         phasedata = phases.binaryencode(headsbyphase)
         bundler.newpart('phase-heads', data=phasedata)
 
 def addparttagsfnodescache(repo, bundler, outgoing):
     # we include the tags fnode cache for the bundle changeset
     # (as an optional parts)
     cache = tags.hgtagsfnodescache(repo.unfiltered())
     chunks = []
 
     # .hgtags fnodes are only relevant for head changesets. While we could
     # transfer values for all known nodes, there will likely be little to
     # no benefit.
     #
     # We don't bother using a generator to produce output data because
     # a) we only have 40 bytes per head and even esoteric numbers of heads
     # consume little memory (1M heads is 40MB) b) we don't want to send the
     # part if we don't have entries and knowing if we have entries requires
     # cache lookups.
     for node in outgoing.missingheads:
         # Don't compute missing, as this may slow down serving.
         fnode = cache.getfnode(node, computemissing=False)
         if fnode is not None:
             chunks.extend([node, fnode])
 
     if chunks:
         bundler.newpart('hgtagsfnodes', data=''.join(chunks))
 
 def buildobsmarkerspart(bundler, markers):
     """add an obsmarker part to the bundler with <markers>
 
     No part is created if markers is empty.
     Raises ValueError if the bundler doesn't support any known obsmarker format.
     """
     if not markers:
         return None
 
     remoteversions = obsmarkersversion(bundler.capabilities)
     version = obsolete.commonversion(remoteversions)
     if version is None:
         raise ValueError('bundler does not support common obsmarker format')
     stream = obsolete.encodemarkers(markers, True, version=version)
     return bundler.newpart('obsmarkers', data=stream)
 
 def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None,
                 compopts=None):
     """Write a bundle file and return its filename.
 
     Existing files will not be overwritten.
     If no filename is specified, a temporary file is created.
     bz2 compression can be turned off.
     The bundle file will be deleted in case of errors.
     """
 
     if bundletype == "HG20":
         bundle = bundle20(ui)
         bundle.setcompression(compression, compopts)
         part = bundle.newpart('changegroup', data=cg.getchunks())
         part.addparam('version', cg.version)
         if 'clcount' in cg.extras:
             part.addparam('nbchanges', '%d' % cg.extras['clcount'],
                           mandatory=False)
         chunkiter = bundle.getchunks()
     else:
         # compression argument is only for the bundle2 case
         assert compression is None
         if cg.version != '01':
             raise error.Abort(_('old bundle types only supports v1 '
                                 'changegroups'))
         header, comp = bundletypes[bundletype]
         if comp not in util.compengines.supportedbundletypes:
             raise error.Abort(_('unknown stream compression type: %s')
                               % comp)
         compengine = util.compengines.forbundletype(comp)
         def chunkiter():
             yield header
             for chunk in compengine.compressstream(cg.getchunks(), compopts):
                 yield chunk
         chunkiter = chunkiter()
 
     # parse the changegroup data, otherwise we will block
     # in case of sshrepo because we don't know the end of the stream
     return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
 
 def combinechangegroupresults(op):
     """logic to combine 0 or more addchangegroup results into one"""
     results = [r.get('return', 0)
                for r in op.records['changegroup']]
     changedheads = 0
     result = 1
     for ret in results:
         # If any changegroup result is 0, return 0
         if ret == 0:
             result = 0
             break
         if ret < -1:
             changedheads += ret + 1
         elif ret > 1:
             changedheads += ret - 1
     if changedheads > 0:
         result = 1 + changedheads
     elif changedheads < 0:
         result = -1 + changedheads
     return result
 
 @parthandler('changegroup', ('version', 'nbchanges', 'treemanifest',
                              'targetphase'))
 def handlechangegroup(op, inpart):
     """apply a changegroup part on the repo
 
     This is a very early implementation that will massive rework before being
     inflicted to any end-user.
     """
     tr = op.gettransaction()
     unpackerversion = inpart.params.get('version', '01')
     # We should raise an appropriate exception here
     cg = changegroup.getunbundler(unpackerversion, inpart, None)
     # the source and url passed here are overwritten by the one contained in
     # the transaction.hookargs argument. So 'bundle2' is a placeholder
     nbchangesets = None
     if 'nbchanges' in inpart.params:
         nbchangesets = int(inpart.params.get('nbchanges'))
     if ('treemanifest' in inpart.params and
         'treemanifest' not in op.repo.requirements):
         if len(op.repo.changelog) != 0:
             raise error.Abort(_(
                 "bundle contains tree manifests, but local repo is "
                 "non-empty and does not use tree manifests"))
         op.repo.requirements.add('treemanifest')
         op.repo._applyopenerreqs()
         op.repo._writerequirements()
     extrakwargs = {}
     targetphase = inpart.params.get('targetphase')
     if targetphase is not None:
         extrakwargs['targetphase'] = int(targetphase)
     ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2',
                               expectedtotal=nbchangesets, **extrakwargs)
     if op.reply is not None:
         # This is definitely not the final form of this
         # return. But one need to start somewhere.
         part = op.reply.newpart('reply:changegroup', mandatory=False)
         part.addparam(
             'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
         part.addparam('return', '%i' % ret, mandatory=False)
     assert not inpart.read()
 
 _remotechangegroupparams = tuple(['url', 'size', 'digests'] +
     ['digest:%s' % k for k in util.DIGESTS.keys()])
 @parthandler('remote-changegroup', _remotechangegroupparams)
 def handleremotechangegroup(op, inpart):
     """apply a bundle10 on the repo, given an url and validation information
 
     All the information about the remote bundle to import are given as
     parameters. The parameters include:
       - url: the url to the bundle10.
       - size: the bundle10 file size. It is used to validate what was
         retrieved by the client matches the server knowledge about the bundle.
       - digests: a space separated list of the digest types provided as
         parameters.
       - digest:<digest-type>: the hexadecimal representation of the digest with
         that name. Like the size, it is used to validate what was retrieved by
         the client matches what the server knows about the bundle.
 
     When multiple digest types are given, all of them are checked.
     """
     try:
         raw_url = inpart.params['url']
     except KeyError:
         raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url')
     parsed_url = util.url(raw_url)
     if parsed_url.scheme not in capabilities['remote-changegroup']:
         raise error.Abort(_('remote-changegroup does not support %s urls') %
             parsed_url.scheme)
 
     try:
         size = int(inpart.params['size'])
     except ValueError:
         raise error.Abort(_('remote-changegroup: invalid value for param "%s"')
             % 'size')
     except KeyError:
         raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size')
 
     digests = {}
     for typ in inpart.params.get('digests', '').split():
         param = 'digest:%s' % typ
         try:
             value = inpart.params[param]
         except KeyError:
             raise error.Abort(_('remote-changegroup: missing "%s" param') %
                 param)
         digests[typ] = value
 
     real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests)
 
     tr = op.gettransaction()
     from . import exchange
     cg = exchange.readbundle(op.repo.ui, real_part, raw_url)
     if not isinstance(cg, changegroup.cg1unpacker):
         raise error.Abort(_('%s: not a bundle version 1.0') %
             util.hidepassword(raw_url))
     ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2')
     if op.reply is not None:
         # This is definitely not the final form of this
         # return. But one need to start somewhere.
         part = op.reply.newpart('reply:changegroup')
         part.addparam(
             'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
         part.addparam('return', '%i' % ret, mandatory=False)
     try:
         real_part.validate()
     except error.Abort as e:
         raise error.Abort(_('bundle at %s is corrupted:\n%s') %
             (util.hidepassword(raw_url), str(e)))
     assert not inpart.read()
 
 @parthandler('reply:changegroup', ('return', 'in-reply-to'))
 def handlereplychangegroup(op, inpart):
     ret = int(inpart.params['return'])
     replyto = int(inpart.params['in-reply-to'])
     op.records.add('changegroup', {'return': ret}, replyto)
 
 @parthandler('check:heads')
 def handlecheckheads(op, inpart):
     """check that head of the repo did not change
 
     This is used to detect a push race when using unbundle.
     This replaces the "heads" argument of unbundle."""
     h = inpart.read(20)
     heads = []
     while len(h) == 20:
         heads.append(h)
         h = inpart.read(20)
     assert not h
     # Trigger a transaction so that we are guaranteed to have the lock now.
     if op.ui.configbool('experimental', 'bundle2lazylocking'):
         op.gettransaction()
     if sorted(heads) != sorted(op.repo.heads()):
         raise error.PushRaced('repository changed while pushing - '
                               'please try again')
 
 @parthandler('check:updated-heads')
 def handlecheckupdatedheads(op, inpart):
     """check for race on the heads touched by a push
 
     This is similar to 'check:heads' but focus on the heads actually updated
     during the push. If other activities happen on unrelated heads, it is
     ignored.
 
     This allow server with high traffic to avoid push contention as long as
     unrelated parts of the graph are involved."""
     h = inpart.read(20)
     heads = []
     while len(h) == 20:
         heads.append(h)
         h = inpart.read(20)
     assert not h
     # trigger a transaction so that we are guaranteed to have the lock now.
     if op.ui.configbool('experimental', 'bundle2lazylocking'):
         op.gettransaction()
 
     currentheads = set()
     for ls in op.repo.branchmap().itervalues():
         currentheads.update(ls)
 
     for h in heads:
         if h not in currentheads:
             raise error.PushRaced('repository changed while pushing - '
                                   'please try again')
 
 @parthandler('check:phases')
 def handlecheckphases(op, inpart):
     """check that phase boundaries of the repository did not change
 
     This is used to detect a push race.
     """
     phasetonodes = phases.binarydecode(inpart)
     unfi = op.repo.unfiltered()
     cl = unfi.changelog
     phasecache = unfi._phasecache
     msg = ('repository changed while pushing - please try again '
            '(%s is %s expected %s)')
     for expectedphase, nodes in enumerate(phasetonodes):
         for n in nodes:
             actualphase = phasecache.phase(unfi, cl.rev(n))
             if actualphase != expectedphase:
                 finalmsg = msg % (nodemod.short(n),
                                   phases.phasenames[actualphase],
                                   phases.phasenames[expectedphase])
                 raise error.PushRaced(finalmsg)
 
 @parthandler('output')
 def handleoutput(op, inpart):
     """forward output captured on the server to the client"""
     for line in inpart.read().splitlines():
         op.ui.status(_('remote: %s\n') % line)
 
 @parthandler('replycaps')
 def handlereplycaps(op, inpart):
     """Notify that a reply bundle should be created
 
     The payload contains the capabilities information for the reply"""
     caps = decodecaps(inpart.read())
     if op.reply is None:
         op.reply = bundle20(op.ui, caps)
 
 class AbortFromPart(error.Abort):
     """Sub-class of Abort that denotes an error from a bundle2 part."""
 
 @parthandler('error:abort', ('message', 'hint'))
 def handleerrorabort(op, inpart):
     """Used to transmit abort error over the wire"""
     raise AbortFromPart(inpart.params['message'],
                         hint=inpart.params.get('hint'))
 
 @parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret',
                                'in-reply-to'))
 def handleerrorpushkey(op, inpart):
     """Used to transmit failure of a mandatory pushkey over the wire"""
     kwargs = {}
     for name in ('namespace', 'key', 'new', 'old', 'ret'):
         value = inpart.params.get(name)
         if value is not None:
             kwargs[name] = value
     raise error.PushkeyFailed(inpart.params['in-reply-to'], **kwargs)
 
 @parthandler('error:unsupportedcontent', ('parttype', 'params'))
 def handleerrorunsupportedcontent(op, inpart):
     """Used to transmit unknown content error over the wire"""
     kwargs = {}
     parttype = inpart.params.get('parttype')
     if parttype is not None:
         kwargs['parttype'] = parttype
     params = inpart.params.get('params')
     if params is not None:
         kwargs['params'] = params.split('\0')
 
     raise error.BundleUnknownFeatureError(**kwargs)
 
 @parthandler('error:pushraced', ('message',))
 def handleerrorpushraced(op, inpart):
     """Used to transmit push race error over the wire"""
     raise error.ResponseError(_('push failed:'), inpart.params['message'])
 
 @parthandler('listkeys', ('namespace',))
 def handlelistkeys(op, inpart):
     """retrieve pushkey namespace content stored in a bundle2"""
     namespace = inpart.params['namespace']
     r = pushkey.decodekeys(inpart.read())
     op.records.add('listkeys', (namespace, r))
 
 @parthandler('pushkey', ('namespace', 'key', 'old', 'new'))
 def handlepushkey(op, inpart):
     """process a pushkey request"""
     dec = pushkey.decode
     namespace = dec(inpart.params['namespace'])
     key = dec(inpart.params['key'])
     old = dec(inpart.params['old'])
     new = dec(inpart.params['new'])
     # Grab the transaction to ensure that we have the lock before performing the
     # pushkey.
     if op.ui.configbool('experimental', 'bundle2lazylocking'):
         op.gettransaction()
     ret = op.repo.pushkey(namespace, key, old, new)
     record = {'namespace': namespace,
               'key': key,
               'old': old,
               'new': new}
     op.records.add('pushkey', record)
     if op.reply is not None:
         rpart = op.reply.newpart('reply:pushkey')
         rpart.addparam(
             'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
         rpart.addparam('return', '%i' % ret, mandatory=False)
     if inpart.mandatory and not ret:
         kwargs = {}
         for key in ('namespace', 'key', 'new', 'old', 'ret'):
             if key in inpart.params:
                 kwargs[key] = inpart.params[key]
         raise error.PushkeyFailed(partid=str(inpart.id), **kwargs)
 
 @parthandler('phase-heads')
 def handlephases(op, inpart):
     """apply phases from bundle part to repo"""
     headsbyphase = phases.binarydecode(inpart)
     phases.updatephases(op.repo.unfiltered(), op.gettransaction, headsbyphase)
 
 @parthandler('reply:pushkey', ('return', 'in-reply-to'))
 def handlepushkeyreply(op, inpart):
     """retrieve the result of a pushkey request"""
     ret = int(inpart.params['return'])
     partid = int(inpart.params['in-reply-to'])
     op.records.add('pushkey', {'return': ret}, partid)
 
 @parthandler('obsmarkers')
 def handleobsmarker(op, inpart):
     """add a stream of obsmarkers to the repo"""
     tr = op.gettransaction()
     markerdata = inpart.read()
     if op.ui.config('experimental', 'obsmarkers-exchange-debug'):
         op.ui.write(('obsmarker-exchange: %i bytes received\n')
                     % len(markerdata))
     # The mergemarkers call will crash if marker creation is not enabled.
     # we want to avoid this if the part is advisory.
     if not inpart.mandatory and op.repo.obsstore.readonly:
         op.repo.ui.debug('ignoring obsolescence markers, feature not enabled')
         return
     new = op.repo.obsstore.mergemarkers(tr, markerdata)
     op.repo.invalidatevolatilesets()
     if new:
         op.repo.ui.status(_('%i new obsolescence markers\n') % new)
     op.records.add('obsmarkers', {'new': new})
     if op.reply is not None:
         rpart = op.reply.newpart('reply:obsmarkers')
         rpart.addparam(
             'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
         rpart.addparam('new', '%i' % new, mandatory=False)
 
 
 @parthandler('reply:obsmarkers', ('new', 'in-reply-to'))
 def handleobsmarkerreply(op, inpart):
     """retrieve the result of a pushkey request"""
     ret = int(inpart.params['new'])
     partid = int(inpart.params['in-reply-to'])
     op.records.add('obsmarkers', {'new': ret}, partid)
 
 @parthandler('hgtagsfnodes')
 def handlehgtagsfnodes(op, inpart):
     """Applies .hgtags fnodes cache entries to the local repo.
 
     Payload is pairs of 20 byte changeset nodes and filenodes.
     """
     # Grab the transaction so we ensure that we have the lock at this point.
     if op.ui.configbool('experimental', 'bundle2lazylocking'):
         op.gettransaction()
     cache = tags.hgtagsfnodescache(op.repo.unfiltered())
 
     count = 0
     while True:
         node = inpart.read(20)
         fnode = inpart.read(20)
         if len(node) < 20 or len(fnode) < 20:
             op.ui.debug('ignoring incomplete received .hgtags fnodes data\n')
             break
         cache.setfnode(node, fnode)
         count += 1
 
     cache.write()
     op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
 
 @parthandler('pushvars')
 def bundle2getvars(op, part):
     '''unbundle a bundle2 containing shellvars on the server'''
     # An option to disable unbundling on server-side for security reasons
     if op.ui.configbool('push', 'pushvars.server'):
         hookargs = {}
         for key, value in part.advisoryparams:
             key = key.upper()
             # We want pushed variables to have USERVAR_ prepended so we know
             # they came from the --pushvar flag.
             key = "USERVAR_" + key
             hookargs[key] = value
         op.addhookargs(hookargs)

mercurial/bundlerepo.py

@@ -1,598 +1,598
 # bundlerepo.py - repository class for viewing uncompressed bundles
 #
 # Copyright 2006, 2007 Benoit Boissinot <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 """Repository class for viewing uncompressed bundles.
 
 This provides a read-only repository interface to bundles as if they
 were part of the actual repository.
 """
 
 from __future__ import absolute_import
 
 import os
 import shutil
 import tempfile
 
 from .i18n import _
 from .node import nullid
 
 from . import (
     bundle2,
     changegroup,
     changelog,
     cmdutil,
     discovery,
     error,
     exchange,
     filelog,
     localrepo,
     manifest,
     mdiff,
     node as nodemod,
     pathutil,
     phases,
     pycompat,
     revlog,
     util,
     vfs as vfsmod,
 )
 
 class bundlerevlog(revlog.revlog):
     def __init__(self, opener, indexfile, cgunpacker, linkmapper):
         # How it works:
         # To retrieve a revision, we need to know the offset of the revision in
         # the bundle (an unbundle object). We store this offset in the index
         # (start). The base of the delta is stored in the base field.
         #
         # To differentiate a rev in the bundle from a rev in the revlog, we
         # check revision against repotiprev.
         opener = vfsmod.readonlyvfs(opener)
         revlog.revlog.__init__(self, opener, indexfile)
         self.bundle = cgunpacker
         n = len(self)
         self.repotiprev = n - 1
         self.bundlerevs = set() # used by 'bundle()' revset expression
         for deltadata in cgunpacker.deltaiter():
             node, p1, p2, cs, deltabase, delta, flags = deltadata
 
             size = len(delta)
             start = cgunpacker.tell() - size
 
             link = linkmapper(cs)
             if node in self.nodemap:
                 # this can happen if two branches make the same change
                 self.bundlerevs.add(self.nodemap[node])
                 continue
 
             for p in (p1, p2):
                 if p not in self.nodemap:
                     raise error.LookupError(p, self.indexfile,
                                             _("unknown parent"))
 
             if deltabase not in self.nodemap:
                 raise LookupError(deltabase, self.indexfile,
                                   _('unknown delta base'))
 
             baserev = self.rev(deltabase)
             # start, size, full unc. size, base (unused), link, p1, p2, node
             e = (revlog.offset_type(start, flags), size, -1, baserev, link,
                  self.rev(p1), self.rev(p2), node)
             self.index.insert(-1, e)
             self.nodemap[node] = n
             self.bundlerevs.add(n)
             n += 1
 
     def _chunk(self, rev, df=None):
         # Warning: in case of bundle, the diff is against what we stored as
         # delta base, not against rev - 1
         # XXX: could use some caching
         if rev <= self.repotiprev:
             return revlog.revlog._chunk(self, rev)
         self.bundle.seek(self.start(rev))
         return self.bundle.read(self.length(rev))
 
     def revdiff(self, rev1, rev2):
         """return or calculate a delta between two revisions"""
         if rev1 > self.repotiprev and rev2 > self.repotiprev:
             # hot path for bundle
             revb = self.index[rev2][3]
             if revb == rev1:
                 return self._chunk(rev2)
         elif rev1 <= self.repotiprev and rev2 <= self.repotiprev:
             return revlog.revlog.revdiff(self, rev1, rev2)
 
         return mdiff.textdiff(self.revision(rev1, raw=True),
                               self.revision(rev2, raw=True))
 
     def revision(self, nodeorrev, _df=None, raw=False):
         """return an uncompressed revision of a given node or revision
         number.
         """
         if isinstance(nodeorrev, int):
             rev = nodeorrev
             node = self.node(rev)
         else:
             node = nodeorrev
             rev = self.rev(node)
 
         if node == nullid:
             return ""
 
         rawtext = None
         chain = []
         iterrev = rev
         # reconstruct the revision if it is from a changegroup
         while iterrev > self.repotiprev:
             if self._cache and self._cache[1] == iterrev:
                 rawtext = self._cache[2]
                 break
             chain.append(iterrev)
             iterrev = self.index[iterrev][3]
         if rawtext is None:
             rawtext = self.baserevision(iterrev)
 
         while chain:
             delta = self._chunk(chain.pop())
             rawtext = mdiff.patches(rawtext, [delta])
 
         text, validatehash = self._processflags(rawtext, self.flags(rev),
                                                 'read', raw=raw)
         if validatehash:
             self.checkhash(text, node, rev=rev)
         self._cache = (node, rev, rawtext)
         return text
 
     def baserevision(self, nodeorrev):
         # Revlog subclasses may override 'revision' method to modify format of
         # content retrieved from revlog. To use bundlerevlog with such class one
         # needs to override 'baserevision' and make more specific call here.
         return revlog.revlog.revision(self, nodeorrev, raw=True)
 
     def addrevision(self, *args, **kwargs):
         raise NotImplementedError
 
     def addgroup(self, *args, **kwargs):
         raise NotImplementedError
 
     def strip(self, *args, **kwargs):
         raise NotImplementedError
 
     def checksize(self):
         raise NotImplementedError
 
 class bundlechangelog(bundlerevlog, changelog.changelog):
     def __init__(self, opener, cgunpacker):
         changelog.changelog.__init__(self, opener)
         linkmapper = lambda x: x
         bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
                               linkmapper)
 
     def baserevision(self, nodeorrev):
         # Although changelog doesn't override 'revision' method, some extensions
         # may replace this class with another that does. Same story with
         # manifest and filelog classes.
 
         # This bypasses filtering on changelog.node() and rev() because we need
         # revision text of the bundle base even if it is hidden.
         oldfilter = self.filteredrevs
         try:
             self.filteredrevs = ()
             return changelog.changelog.revision(self, nodeorrev, raw=True)
         finally:
             self.filteredrevs = oldfilter
 
 class bundlemanifest(bundlerevlog, manifest.manifestrevlog):
     def __init__(self, opener, cgunpacker, linkmapper, dirlogstarts=None,
                  dir=''):
         manifest.manifestrevlog.__init__(self, opener, dir=dir)
         bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
                               linkmapper)
         if dirlogstarts is None:
             dirlogstarts = {}
             if self.bundle.version == "03":
                 dirlogstarts = _getfilestarts(self.bundle)
         self._dirlogstarts = dirlogstarts
         self._linkmapper = linkmapper
 
     def baserevision(self, nodeorrev):
         node = nodeorrev
         if isinstance(node, int):
             node = self.node(node)
 
         if node in self.fulltextcache:
             result = '%s' % self.fulltextcache[node]
         else:
             result = manifest.manifestrevlog.revision(self, nodeorrev, raw=True)
         return result
 
     def dirlog(self, d):
         if d in self._dirlogstarts:
             self.bundle.seek(self._dirlogstarts[d])
             return bundlemanifest(
                 self.opener, self.bundle, self._linkmapper,
                 self._dirlogstarts, dir=d)
         return super(bundlemanifest, self).dirlog(d)
 
 class bundlefilelog(bundlerevlog, filelog.filelog):
     def __init__(self, opener, path, cgunpacker, linkmapper):
         filelog.filelog.__init__(self, opener, path)
         bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
                               linkmapper)
 
     def baserevision(self, nodeorrev):
         return filelog.filelog.revision(self, nodeorrev, raw=True)
 
 class bundlepeer(localrepo.localpeer):
     def canpush(self):
         return False
 
 class bundlephasecache(phases.phasecache):
     def __init__(self, *args, **kwargs):
         super(bundlephasecache, self).__init__(*args, **kwargs)
         if util.safehasattr(self, 'opener'):
             self.opener = vfsmod.readonlyvfs(self.opener)
 
     def write(self):
         raise NotImplementedError
 
     def _write(self, fp):
         raise NotImplementedError
 
     def _updateroots(self, phase, newroots, tr):
         self.phaseroots[phase] = newroots
         self.invalidate()
         self.dirty = True
 
 def _getfilestarts(cgunpacker):
     filespos = {}
     for chunkdata in iter(cgunpacker.filelogheader, {}):
         fname = chunkdata['filename']
         filespos[fname] = cgunpacker.tell()
         for chunk in iter(lambda: cgunpacker.deltachunk(None), {}):
             pass
     return filespos
 
 class bundlerepository(localrepo.localrepository):
     """A repository instance that is a union of a local repo and a bundle.
 
     Instances represent a read-only repository composed of a local repository
     with the contents of a bundle file applied. The repository instance is
     conceptually similar to the state of a repository after an
     ``hg unbundle`` operation. However, the contents of the bundle are never
     applied to the actual base repository.
     """
     def __init__(self, ui, repopath, bundlepath):
         self._tempparent = None
         try:
             localrepo.localrepository.__init__(self, ui, repopath)
         except error.RepoError:
             self._tempparent = tempfile.mkdtemp()
             localrepo.instance(ui, self._tempparent, 1)
             localrepo.localrepository.__init__(self, ui, self._tempparent)
         self.ui.setconfig('phases', 'publish', False, 'bundlerepo')
 
         if repopath:
             self._url = 'bundle:' + util.expandpath(repopath) + '+' + bundlepath
         else:
             self._url = 'bundle:' + bundlepath
 
         self.tempfile = None
         f = util.posixfile(bundlepath, "rb")
         bundle = exchange.readbundle(ui, f, bundlepath)
 
         if isinstance(bundle, bundle2.unbundle20):
             self._bundlefile = bundle
             self._cgunpacker = None
 
             cgpart = None
-            for part in bundle.iterparts():
+            for part in bundle.iterparts(seekable=True):
                 if part.type == 'changegroup':
                     if cgpart:
                         raise NotImplementedError("can't process "
                                                   "multiple changegroups")
                     cgpart = part
 
                 self._handlebundle2part(bundle, part)
 
             if not cgpart:
                 raise error.Abort(_("No changegroups found"))
 
             # This is required to placate a later consumer, which expects
             # the payload offset to be at the beginning of the changegroup.
             # We need to do this after the iterparts() generator advances
             # because iterparts() will seek to end of payload after the
             # generator returns control to iterparts().
             cgpart.seek(0, os.SEEK_SET)
 
         elif isinstance(bundle, changegroup.cg1unpacker):
             if bundle.compressed():
                 f = self._writetempbundle(bundle.read, '.hg10un',
                                           header='HG10UN')
                 bundle = exchange.readbundle(ui, f, bundlepath, self.vfs)
 
             self._bundlefile = bundle
             self._cgunpacker = bundle
         else:
             raise error.Abort(_('bundle type %s cannot be read') %
                               type(bundle))
 
         # dict with the mapping 'filename' -> position in the changegroup.
         self._cgfilespos = {}
 
         self.firstnewrev = self.changelog.repotiprev + 1
         phases.retractboundary(self, None, phases.draft,
                                [ctx.node() for ctx in self[self.firstnewrev:]])
 
     def _handlebundle2part(self, bundle, part):
         if part.type != 'changegroup':
             return
 
         cgstream = part
         version = part.params.get('version', '01')
         legalcgvers = changegroup.supportedincomingversions(self)
         if version not in legalcgvers:
             msg = _('Unsupported changegroup version: %s')
             raise error.Abort(msg % version)
         if bundle.compressed():
             cgstream = self._writetempbundle(part.read, '.cg%sun' % version)
 
         self._cgunpacker = changegroup.getunbundler(version, cgstream, 'UN')
 
     def _writetempbundle(self, readfn, suffix, header=''):
         """Write a temporary file to disk
         """
         fdtemp, temp = self.vfs.mkstemp(prefix="hg-bundle-",
                                         suffix=suffix)
         self.tempfile = temp
 
         with os.fdopen(fdtemp, pycompat.sysstr('wb')) as fptemp:
             fptemp.write(header)
             while True:
                 chunk = readfn(2**18)
                 if not chunk:
                     break
                 fptemp.write(chunk)
 
         return self.vfs.open(self.tempfile, mode="rb")
 
     @localrepo.unfilteredpropertycache
     def _phasecache(self):
         return bundlephasecache(self, self._phasedefaults)
 
     @localrepo.unfilteredpropertycache
     def changelog(self):
         # consume the header if it exists
         self._cgunpacker.changelogheader()
         c = bundlechangelog(self.svfs, self._cgunpacker)
         self.manstart = self._cgunpacker.tell()
         return c
 
     def _constructmanifest(self):
         self._cgunpacker.seek(self.manstart)
         # consume the header if it exists
         self._cgunpacker.manifestheader()
         linkmapper = self.unfiltered().changelog.rev
         m = bundlemanifest(self.svfs, self._cgunpacker, linkmapper)
         self.filestart = self._cgunpacker.tell()
         return m
 
     def _consumemanifest(self):
         """Consumes the manifest portion of the bundle, setting filestart so the
         file portion can be read."""
         self._cgunpacker.seek(self.manstart)
         self._cgunpacker.manifestheader()
         for delta in self._cgunpacker.deltaiter():
             pass
         self.filestart = self._cgunpacker.tell()
 
     @localrepo.unfilteredpropertycache
     def manstart(self):
         self.changelog
         return self.manstart
 
     @localrepo.unfilteredpropertycache
     def filestart(self):
         self.manifestlog
 
         # If filestart was not set by self.manifestlog, that means the
         # manifestlog implementation did not consume the manifests from the
         # changegroup (ex: it might be consuming trees from a separate bundle2
         # part instead). So we need to manually consume it.
         if 'filestart' not in self.__dict__:
             self._consumemanifest()
 
         return self.filestart
 
     def url(self):
         return self._url
 
     def file(self, f):
         if not self._cgfilespos:
             self._cgunpacker.seek(self.filestart)
             self._cgfilespos = _getfilestarts(self._cgunpacker)
 
         if f in self._cgfilespos:
             self._cgunpacker.seek(self._cgfilespos[f])
             linkmapper = self.unfiltered().changelog.rev
             return bundlefilelog(self.svfs, f, self._cgunpacker, linkmapper)
         else:
             return filelog.filelog(self.svfs, f)
 
     def close(self):
         """Close assigned bundle file immediately."""
         self._bundlefile.close()
         if self.tempfile is not None:
             self.vfs.unlink(self.tempfile)
         if self._tempparent:
             shutil.rmtree(self._tempparent, True)
 
     def cancopy(self):
         return False
 
     def peer(self):
         return bundlepeer(self)
 
     def getcwd(self):
         return pycompat.getcwd() # always outside the repo
 
     # Check if parents exist in localrepo before setting
     def setparents(self, p1, p2=nullid):
         p1rev = self.changelog.rev(p1)
         p2rev = self.changelog.rev(p2)
         msg = _("setting parent to node %s that only exists in the bundle\n")
         if self.changelog.repotiprev < p1rev:
             self.ui.warn(msg % nodemod.hex(p1))
         if self.changelog.repotiprev < p2rev:
             self.ui.warn(msg % nodemod.hex(p2))
         return super(bundlerepository, self).setparents(p1, p2)
 
 def instance(ui, path, create):
     if create:
         raise error.Abort(_('cannot create new bundle repository'))
     # internal config: bundle.mainreporoot
     parentpath = ui.config("bundle", "mainreporoot")
     if not parentpath:
         # try to find the correct path to the working directory repo
         parentpath = cmdutil.findrepo(pycompat.getcwd())
         if parentpath is None:
             parentpath = ''
     if parentpath:
         # Try to make the full path relative so we get a nice, short URL.
         # In particular, we don't want temp dir names in test outputs.
         cwd = pycompat.getcwd()
         if parentpath == cwd:
             parentpath = ''
         else:
             cwd = pathutil.normasprefix(cwd)
             if parentpath.startswith(cwd):
                 parentpath = parentpath[len(cwd):]
     u = util.url(path)
     path = u.localpath()
     if u.scheme == 'bundle':
         s = path.split("+", 1)
         if len(s) == 1:
             repopath, bundlename = parentpath, s[0]
         else:
             repopath, bundlename = s
     else:
         repopath, bundlename = parentpath, path
     return bundlerepository(ui, repopath, bundlename)
 
 class bundletransactionmanager(object):
     def transaction(self):
         return None
 
     def close(self):
         raise NotImplementedError
 
     def release(self):
         raise NotImplementedError
 
 def getremotechanges(ui, repo, other, onlyheads=None, bundlename=None,
                      force=False):
     '''obtains a bundle of changes incoming from other
 
     "onlyheads" restricts the returned changes to those reachable from the
       specified heads.
     "bundlename", if given, stores the bundle to this file path permanently;
       otherwise it's stored to a temp file and gets deleted again when you call
       the returned "cleanupfn".
     "force" indicates whether to proceed on unrelated repos.
 
     Returns a tuple (local, csets, cleanupfn):
 
     "local" is a local repo from which to obtain the actual incoming
       changesets; it is a bundlerepo for the obtained bundle when the
       original "other" is remote.
     "csets" lists the incoming changeset node ids.
     "cleanupfn" must be called without arguments when you're done processing
       the changes; it closes both the original "other" and the one returned
       here.
     '''
     tmp = discovery.findcommonincoming(repo, other, heads=onlyheads,
                                        force=force)
     common, incoming, rheads = tmp
     if not incoming:
         try:
             if bundlename:
                 os.unlink(bundlename)
         except OSError:
             pass
         return repo, [], other.close
 
     commonset = set(common)
     rheads = [x for x in rheads if x not in commonset]
 
     bundle = None
     bundlerepo = None
     localrepo = other.local()
     if bundlename or not localrepo:
         # create a bundle (uncompressed if other repo is not local)
 
         # developer config: devel.legacy.exchange
         legexc = ui.configlist('devel', 'legacy.exchange')
         forcebundle1 = 'bundle2' not in legexc and 'bundle1' in legexc
         canbundle2 = (not forcebundle1
                       and other.capable('getbundle')
                       and other.capable('bundle2'))
         if canbundle2:
             kwargs = {}
             kwargs['common'] = common
             kwargs['heads'] = rheads
             kwargs['bundlecaps'] = exchange.caps20to10(repo)
             kwargs['cg'] = True
             b2 = other.getbundle('incoming', **kwargs)
             fname = bundle = changegroup.writechunks(ui, b2._forwardchunks(),
                                                      bundlename)
         else:
             if other.capable('getbundle'):
                 cg = other.getbundle('incoming', common=common, heads=rheads)
             elif onlyheads is None and not other.capable('changegroupsubset'):
                 # compat with older servers when pulling all remote heads
                 cg = other.changegroup(incoming, "incoming")
                 rheads = None
             else:
                 cg = other.changegroupsubset(incoming, rheads, 'incoming')
             if localrepo:
                 bundletype = "HG10BZ"
             else:
                 bundletype = "HG10UN"
             fname = bundle = bundle2.writebundle(ui, cg, bundlename,
                                                      bundletype)
         # keep written bundle?
         if bundlename:
             bundle = None
         if not localrepo:
             # use the created uncompressed bundlerepo
             localrepo = bundlerepo = bundlerepository(repo.baseui, repo.root,
                                                       fname)
             # this repo contains local and other now, so filter out local again
             common = repo.heads()
     if localrepo:
         # Part of common may be remotely filtered
         # So use an unfiltered version
         # The discovery process probably need cleanup to avoid that
         localrepo = localrepo.unfiltered()
 
     csets = localrepo.changelog.findmissing(common, rheads)
 
     if bundlerepo:
         reponodes = [ctx.node() for ctx in bundlerepo[bundlerepo.firstnewrev:]]
         remotephases = other.listkeys('phases')
 
         pullop = exchange.pulloperation(bundlerepo, other, heads=reponodes)
         pullop.trmanager = bundletransactionmanager()
         exchange._pullapplyphases(pullop, remotephases)
 
     def cleanup():
         if bundlerepo:
             bundlerepo.close()
         if bundle:
             os.unlink(bundle)
         other.close()
 
     return (localrepo, csets, cleanup)