##// END OF EJS Templates
revset: remove "small" argument from "_optimize"...
revset: remove "small" argument from "_optimize" `_optimize` calculates weights of subtrees. "small" affects some weight calculation (either 1 or 0.5). The weights are now only useful in `and` optimization where we might swap two arguments and use `andsmally`. In the real world, it seems unlikely that revsets with weight of 0.5 or 1 matters the `and` order optimization. I think the important thing is to get weights of expensive revsets right (ex. `contains`). This patch removes the `small` argument to simplify the interface. As for choosing between 0.5 vs 1, things returning a single revision (`ancestor`, `string`) has a weight of 0.5. Things returning multiple revisions returns 1. This could be sometimes useful in the `andsmally` optimization, ex. (((:)-2) & expensive()) & ((1-2) & expensive()) ^^^ ^ ^^^^^^^ ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ weight=1 weight=0.5 would have an `andsmally` optimization so `1-2` gets executed first, which seems to be desirable. Differential Revision: https://phab.mercurial-scm.org/D656

File last commit:

r34258:2844c4bd merge default
r34273:53fb09c7 default
Show More
changegroup.py
1015 lines | 38.8 KiB | text/x-python | PythonLexer
Martin Geisler
put license and copyright info into comment blocks
r8226 # changegroup.py - Mercurial changegroup manipulation functions
#
# Copyright 2006 Matt Mackall <mpm@selenic.com>
#
# This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Matt Mackall
Replace demandload with new demandimport
r3877
Gregory Szorc
changegroup: use absolute_import
r25921 from __future__ import absolute_import
import os
import struct
import tempfile
Pierre-Yves David
localrepo: move the addchangegroup method in changegroup module...
r20933 import weakref
Gregory Szorc
changegroup: use absolute_import
r25921
from .i18n import _
from .node import (
hex,
nullrev,
short,
)
from . import (
dagutil,
error,
mdiff,
phases,
Pulkit Goyal
py3: convert the mode argument of os.fdopen to unicodes (2 of 2)
r30925 pycompat,
Gregory Szorc
changegroup: use absolute_import
r25921 util,
)
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981
Sune Foldager
changegroup: rename bundle-related functions and classes...
r22390 _CHANGEGROUPV1_DELTA_HEADER = "20s20s20s20s"
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 _CHANGEGROUPV2_DELTA_HEADER = "20s20s20s20s20s"
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 _CHANGEGROUPV3_DELTA_HEADER = ">20s20s20s20s20sH"
Benoit Boissinot
bundler: make parsechunk return the base revision of the delta
r14141
Mads Kiilerich
changegroup: verify all stream reads...
r13457 def readexactly(stream, n):
'''read n bytes from stream.read and abort if less was available'''
s = stream.read(n)
if len(s) < n:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("stream ended unexpectedly"
Mads Kiilerich
changegroup: verify all stream reads...
r13457 " (got %d bytes, expected %d)")
% (len(s), n))
return s
def getchunk(stream):
"""return the next chunk from stream as a string"""
d = readexactly(stream, 4)
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981 l = struct.unpack(">l", d)[0]
if l <= 4:
Mads Kiilerich
changegroup: don't accept odd chunk headers
r13458 if l:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("invalid chunk length %d") % l)
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981 return ""
Mads Kiilerich
changegroup: verify all stream reads...
r13457 return readexactly(stream, l - 4)
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981
Matt Mackall
changegroup: avoid large copies...
r5368 def chunkheader(length):
Greg Ward
Improve some docstrings relating to changegroups and prepush().
r9437 """return a changegroup chunk header (string)"""
Matt Mackall
changegroup: avoid large copies...
r5368 return struct.pack(">l", length + 4)
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981
def closechunk():
Greg Ward
Improve some docstrings relating to changegroups and prepush().
r9437 """return a changegroup chunk header (string) for a zero-length chunk"""
Thomas Arendsen Hein
make incoming work via ssh (issue139); move chunk code into separate module....
r1981 return struct.pack(">l", 0)
Pierre-Yves David
changegroup: extract the file management part in its own function...
r26540 def writechunks(ui, chunks, filename, vfs=None):
"""Write chunks to a file and return its filename.
Matt Mackall
move write_bundle to changegroup.py
r3659
Pierre-Yves David
changegroup: extract the file management part in its own function...
r26540 The stream is assumed to be a bundle file.
Matt Mackall
move write_bundle to changegroup.py
r3659 Existing files will not be overwritten.
If no filename is specified, a temporary file is created.
"""
fh = None
cleanup = None
try:
if filename:
FUJIWARA Katsunori
changegroup: add "vfs" argument to "writebundle()" for relative access via vfs...
r20976 if vfs:
fh = vfs.open(filename, "wb")
else:
Gregory Szorc
changegroup: increase write buffer size to 128k...
r30212 # Increase default buffer size because default is usually
# small (4k is common on Linux).
fh = open(filename, "wb", 131072)
Matt Mackall
move write_bundle to changegroup.py
r3659 else:
fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
Pulkit Goyal
py3: convert the mode argument of os.fdopen to unicodes (2 of 2)
r30925 fh = os.fdopen(fd, pycompat.sysstr("wb"))
Matt Mackall
move write_bundle to changegroup.py
r3659 cleanup = filename
Pierre-Yves David
changegroup: extract the file management part in its own function...
r26540 for c in chunks:
fh.write(c)
Matt Mackall
move write_bundle to changegroup.py
r3659 cleanup = None
return filename
finally:
if fh is not None:
fh.close()
if cleanup is not None:
FUJIWARA Katsunori
changegroup: add "vfs" argument to "writebundle()" for relative access via vfs...
r20976 if filename and vfs:
vfs.unlink(cleanup)
else:
os.unlink(cleanup)
Matt Mackall
create a readbundle function
r3660
Sune Foldager
changegroup: rename bundle-related functions and classes...
r22390 class cg1unpacker(object):
Augie Fackler
changegroup: document the public surface area of cg?unpackers...
r26708 """Unpacker for cg1 changegroup streams.
A changegroup unpacker handles the framing of the revision data in
the wire format. Most consumers will want to use the apply()
method to add the changes from the changegroup to a repository.
If you're forwarding a changegroup unmodified to another consumer,
use getchunks(), which returns an iterator of changegroup
chunks. This is mostly useful for cases where you need to know the
data stream has ended by observing the end of the changegroup.
deltachunk() is useful only if you're applying delta data. Most
consumers should prefer apply() instead.
A few other public methods exist. Those are used only for
bundlerepo and some debug commands - their use is discouraged.
"""
Sune Foldager
changegroup: rename bundle-related functions and classes...
r22390 deltaheader = _CHANGEGROUPV1_DELTA_HEADER
Benoit Boissinot
bundler: make parsechunk return the base revision of the delta
r14141 deltaheadersize = struct.calcsize(deltaheader)
Eric Sumner
changegroup.writebundle: HG2Y support...
r23896 version = '01'
Martin von Zweigbergk
changegroup: cg3 has two empty groups *after* manifests...
r27920 _grouplistcount = 1 # One list of files after the manifests
Gregory Szorc
bundle2: store changeset count when creating file bundles...
r29593 def __init__(self, fh, alg, extras=None):
Gregory Szorc
changegroup: use compression engines API...
r30354 if alg is None:
alg = 'UN'
if alg not in util.compengines.supportedbundletypes:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('unknown stream compression type: %s')
Pierre-Yves David
changegroup: move all compressions utilities in util...
r26266 % alg)
Pierre-Yves David
changegroup: use a different compression key for BZ in HG10...
r26392 if alg == 'BZ':
alg = '_truncatedBZ'
Gregory Szorc
changegroup: use compression engines API...
r30354
compengine = util.compengines.forbundletype(alg)
self._stream = compengine.decompressorreader(fh)
Matt Mackall
bundlerepo: remove duplication of bundle decompressors
r12044 self._type = alg
Gregory Szorc
bundle2: store changeset count when creating file bundles...
r29593 self.extras = extras or {}
Matt Mackall
bundle: refactor progress callback...
r12334 self.callback = None
Augie Fackler
changegroup: note why a few methods on cg1unpacker exist...
r26706
# These methods (compressed, read, seek, tell) all appear to only
# be used by bundlerepo, but it's a little hard to tell.
Matt Mackall
bundlerepo: remove duplication of bundle decompressors
r12044 def compressed(self):
Stanislau Hlebik
cg1packer: fix `compressed` method...
r30589 return self._type is not None and self._type != 'UN'
Matt Mackall
bundle: introduce bundle class
r12043 def read(self, l):
return self._stream.read(l)
Matt Mackall
bundle: make unbundle object seekable...
r12330 def seek(self, pos):
return self._stream.seek(pos)
def tell(self):
Matt Mackall
bundlerepo: use bundle objects everywhere
r12332 return self._stream.tell()
Matt Mackall
bundlerepo: restore close() method
r12347 def close(self):
return self._stream.close()
Matt Mackall
bundle: refactor progress callback...
r12334
Augie Fackler
changegroup: mark cg1unpacker.chunklength as private
r26707 def _chunklength(self):
Jim Hague
changegroup: fix typo introduced in 9f2c407caf34
r13459 d = readexactly(self._stream, 4)
Mads Kiilerich
changegroup: don't accept odd chunk headers
r13458 l = struct.unpack(">l", d)[0]
if l <= 4:
if l:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("invalid chunk length %d") % l)
Mads Kiilerich
changegroup: don't accept odd chunk headers
r13458 return 0
if self.callback:
Matt Mackall
bundle: refactor progress callback...
r12334 self.callback()
Mads Kiilerich
changegroup: don't accept odd chunk headers
r13458 return l - 4
Matt Mackall
bundle: refactor progress callback...
r12334
Benoit Boissinot
unbundler: separate delta and header parsing...
r14144 def changelogheader(self):
"""v10 does not have a changelog header chunk"""
return {}
def manifestheader(self):
"""v10 does not have a manifest header chunk"""
return {}
def filelogheader(self):
"""return the header of the filelogs chunk, v10 only has the filename"""
Augie Fackler
changegroup: mark cg1unpacker.chunklength as private
r26707 l = self._chunklength()
Benoit Boissinot
unbundler: separate delta and header parsing...
r14144 if not l:
return {}
fname = readexactly(self._stream, l)
Augie Fackler
changegroup: move from dict() construction to {} literals...
r20675 return {'filename': fname}
Matt Mackall
bundle: refactor progress callback...
r12334
Benoit Boissinot
bundler: make parsechunk return the base revision of the delta
r14141 def _deltaheader(self, headertuple, prevnode):
node, p1, p2, cs = headertuple
if prevnode is None:
deltabase = p1
else:
deltabase = prevnode
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 flags = 0
return node, p1, p2, deltabase, cs, flags
Benoit Boissinot
bundler: make parsechunk return the base revision of the delta
r14141
Benoit Boissinot
unbundler: separate delta and header parsing...
r14144 def deltachunk(self, prevnode):
Augie Fackler
changegroup: mark cg1unpacker.chunklength as private
r26707 l = self._chunklength()
Matt Mackall
bundle: move chunk parsing into unbundle class
r12336 if not l:
return {}
Benoit Boissinot
bundler: make parsechunk return the base revision of the delta
r14141 headerdata = readexactly(self._stream, self.deltaheadersize)
header = struct.unpack(self.deltaheader, headerdata)
delta = readexactly(self._stream, l - self.deltaheadersize)
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 node, p1, p2, deltabase, cs, flags = self._deltaheader(header, prevnode)
Augie Fackler
changegroup: move from dict() construction to {} literals...
r20675 return {'node': node, 'p1': p1, 'p2': p2, 'cs': cs,
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 'deltabase': deltabase, 'delta': delta, 'flags': flags}
Matt Mackall
bundle: move chunk parsing into unbundle class
r12336
Pierre-Yves David
changegroup: move chunk extraction into a getchunks method of unbundle10...
r20999 def getchunks(self):
"""returns all the chunks contains in the bundle
Used when you need to forward the binary stream to a file or another
network API. To do so, it parse the changegroup data, otherwise it will
block in case of sshrepo because it don't know the end of the stream.
"""
Durham Goode
changegroup: fix to allow empty manifest parts...
r34093 # For changegroup 1 and 2, we expect 3 parts: changelog, manifestlog,
# and a list of filelogs. For changegroup 3, we expect 4 parts:
# changelog, manifestlog, a list of tree manifestlogs, and a list of
# filelogs.
#
# Changelog and manifestlog parts are terminated with empty chunks. The
# tree and file parts are a list of entry sections. Each entry section
# is a series of chunks terminating in an empty chunk. The list of these
# entry sections is terminated in yet another empty chunk, so we know
# we've reached the end of the tree/file list when we reach an empty
# chunk that was proceeded by no non-empty chunks.
parts = 0
while parts < 2 + self._grouplistcount:
noentries = True
Pierre-Yves David
changegroup: move chunk extraction into a getchunks method of unbundle10...
r20999 while True:
chunk = getchunk(self)
if not chunk:
Durham Goode
changegroup: fix to allow empty manifest parts...
r34093 # The first two empty chunks represent the end of the
# changelog and the manifestlog portions. The remaining
# empty chunks represent either A) the end of individual
# tree or file entries in the file list, or B) the end of
# the entire list. It's the end of the entire list if there
# were no entries (i.e. noentries is True).
if parts < 2:
parts += 1
elif noentries:
parts += 1
Pierre-Yves David
changegroup: move chunk extraction into a getchunks method of unbundle10...
r20999 break
Durham Goode
changegroup: fix to allow empty manifest parts...
r34093 noentries = False
Pierre-Yves David
changegroup: move chunk extraction into a getchunks method of unbundle10...
r20999 yield chunkheader(len(chunk))
pos = 0
while pos < len(chunk):
next = pos + 2**20
yield chunk[pos:next]
pos = next
yield closechunk()
Augie Fackler
changegroup: move manifest unpacking into its own method...
r26712 def _unpackmanifests(self, repo, revmap, trp, prog, numchanges):
# We know that we'll never have more manifests than we had
# changesets.
self.callback = prog(_('manifests'), numchanges)
# no need to check for empty manifest group here:
# if the result of the merge of 1 and 2 is the same in 3 and 4,
# no new manifest will be created and the manifest group will
# be empty during the pull
self.manifestheader()
Durham Goode
changegroup: remove changegroup dependency from revlog.addgroup...
r34147 deltas = self.deltaiter(revmap)
repo.manifestlog._revlog.addgroup(deltas, trp)
Augie Fackler
changegroup: move manifest unpacking into its own method...
r26712 repo.ui.progress(_('manifests'), None)
Martin von Zweigbergk
changegroup: exclude submanifests from manifest progress...
r28360 self.callback = None
Augie Fackler
changegroup: move manifest unpacking into its own method...
r26712
Martin von Zweigbergk
changegroup: remove option to allow empty changegroup (API)...
r33308 def apply(self, repo, tr, srctype, url, targetphase=phases.draft,
expectedtotal=None):
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695 """Add the changegroup returned by source.read() to this repo.
srctype is a string like 'push', 'pull', or 'unbundle'. url is
the URL of the repo where this changegroup is coming from.
Return an integer summarizing the change to this repo:
- nothing changed or no source: 0
- more heads than before: 1+added heads (2..n)
- fewer heads than before: -1-removed heads (-2..-n)
- number of heads stays the same: 1
"""
repo = repo.unfiltered()
def csmap(x):
repo.ui.debug("add changeset %s\n" % short(x))
return len(cl)
def revmap(x):
return cl.rev(x)
changesets = files = revisions = 0
Pierre-Yves David
changegroup: fix the scope of a try finally...
r26880 try:
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 # The transaction may already carry source information. In this
# case we use the top level data. We overwrite the argument
# because we need to use the top level value (if they exist)
# in this function.
srctype = tr.hookargs.setdefault('source', srctype)
url = tr.hookargs.setdefault('url', url)
Augie Fackler
changegroup: wrap some ** expansions in strkwargs
r33634 repo.hook('prechangegroup',
throw=True, **pycompat.strkwargs(tr.hookargs))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 # write changelog data to temp files so concurrent readers
# will not see an inconsistent view
cl = repo.changelog
cl.delayupdate(tr)
oldheads = set(cl.heads())
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 trp = weakref.proxy(tr)
# pull off the changeset group
repo.ui.status(_("adding changesets\n"))
clstart = len(cl)
class prog(object):
def __init__(self, step, total):
self._step = step
self._total = total
self._count = 1
def __call__(self):
repo.ui.progress(self._step, self._count, unit=_('chunks'),
total=self._total)
self._count += 1
self.callback = prog(_('changesets'), expectedtotal)
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 efiles = set()
def onchangelog(cl, node):
efiles.update(cl.readfiles(node))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 self.changelogheader()
Durham Goode
changegroup: remove changegroup dependency from revlog.addgroup...
r34147 deltas = self.deltaiter(csmap)
cgnodes = cl.addgroup(deltas, trp, addrevisioncb=onchangelog)
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 efiles = len(efiles)
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: remove option to allow empty changegroup (API)...
r33308 if not cgnodes:
Martin von Zweigbergk
changegroup: don't fail on empty changegroup (API)...
r33309 repo.ui.develwarn('applied empty changegroup',
config='empty-changegroup')
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 clend = len(cl)
changesets = clend - clstart
repo.ui.progress(_('changesets'), None)
self.callback = None
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 # pull off the manifest group
repo.ui.status(_("adding manifests\n"))
self._unpackmanifests(repo, revmap, trp, prog, changesets)
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 needfiles = {}
configitems: register the 'server.validate' config
r33224 if repo.ui.configbool('server', 'validate'):
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 cl = repo.changelog
ml = repo.manifestlog
# validate incoming csets have their manifests
for cset in xrange(clstart, clend):
mfnode = cl.changelogrevision(cset).manifest
mfest = ml[mfnode].readdelta()
# store file cgnodes we must see
for f, n in mfest.iteritems():
needfiles.setdefault(f, set()).add(n)
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 # process the files
repo.ui.status(_("adding file changes\n"))
newrevs, newfiles = _addchangegroupfiles(
repo, self, revmap, trp, efiles, needfiles)
revisions += newrevs
files += newfiles
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 deltaheads = 0
if oldheads:
heads = cl.heads()
deltaheads = len(heads) - len(oldheads)
for h in heads:
if h not in oldheads and repo[h].closesbranch():
deltaheads -= 1
htext = ""
if deltaheads:
htext = _(" (%+d heads)") % deltaheads
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 repo.ui.status(_("added %d changesets"
" with %d changes to %d files%s\n")
% (changesets, revisions, files, htext))
repo.invalidatevolatilesets()
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 if changesets > 0:
if 'node' not in tr.hookargs:
tr.hookargs['node'] = hex(cl.node(clstart))
tr.hookargs['node_last'] = hex(cl.node(clend - 1))
hookargs = dict(tr.hookargs)
else:
hookargs = dict(tr.hookargs)
hookargs['node'] = hex(cl.node(clstart))
hookargs['node_last'] = hex(cl.node(clend - 1))
Augie Fackler
changegroup: wrap some ** expansions in strkwargs
r33634 repo.hook('pretxnchangegroup',
throw=True, **pycompat.strkwargs(hookargs))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 added = [cl.node(r) for r in xrange(clstart, clend)]
Boris Feld
phases: rework phase movement code in 'cg.apply' to use 'registernew'...
r33456 phaseall = None
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 if srctype in ('push', 'serve'):
# Old servers can not push the boundary themselves.
# New servers won't push the boundary if changeset already
# exists locally as secret
#
# We should not use added here but the list of all change in
# the bundle
if repo.publishing():
Boris Feld
phases: rework phase movement code in 'cg.apply' to use 'registernew'...
r33456 targetphase = phaseall = phases.public
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 else:
Boris Feld
phases: rework phase movement code in 'cg.apply' to use 'registernew'...
r33456 # closer target phase computation
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 # Those changesets have been pushed from the
# outside, their phases are going to be pushed
# alongside. Therefor `targetphase` is
# ignored.
Boris Feld
phases: rework phase movement code in 'cg.apply' to use 'registernew'...
r33456 targetphase = phaseall = phases.draft
if added:
phases.registernew(repo, tr, targetphase, added)
if phaseall is not None:
phases.advanceboundary(repo, tr, phaseall, cgnodes)
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 if changesets > 0:
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 def runhooks():
# These hooks run when the lock releases, not when the
# transaction closes. So it's possible for the changelog
# to have changed since we last saw it.
if clstart >= len(repo):
return
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Augie Fackler
changegroup: more **kwargs...
r33678 repo.hook("changegroup", **pycompat.strkwargs(hookargs))
Bryan O'Sullivan
with: use context manager for transaction in changegroup apply...
r27867
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 for n in added:
args = hookargs.copy()
args['node'] = hex(n)
del args['node_last']
Augie Fackler
changegroup: more **kwargs...
r33678 repo.hook("incoming", **pycompat.strkwargs(args))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 newheads = [h for h in repo.heads()
if h not in oldheads]
repo.ui.log("incoming",
"%s incoming changes - new heads: %s\n",
len(added),
', '.join([hex(c[:6]) for c in newheads]))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Martin von Zweigbergk
changegroup: delete "if True" and reflow
r32931 tr.addpostclose('changegroup-runhooks-%020i' % clstart,
lambda tr: repo._afterlock(runhooks))
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695 finally:
repo.ui.flush()
# never return 0 here:
Martin von Zweigbergk
changegroup: rename "dh" to the clearer "deltaheads"...
r32870 if deltaheads < 0:
Martin von Zweigbergk
bundle2: record changegroup data in 'op.records' (API)...
r33030 ret = deltaheads - 1
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695 else:
Martin von Zweigbergk
bundle2: record changegroup data in 'op.records' (API)...
r33030 ret = deltaheads + 1
Boris Feld
changegroup: stop returning and recording added nodes in 'cg.apply'...
r33461 return ret
Augie Fackler
changegroup: migrate addchangegroup() to forward to cg?unpacker.apply()...
r26695
Durham Goode
changegroup: remove changegroup dependency from revlog.addgroup...
r34147 def deltaiter(self, linkmapper):
"""
returns an iterator of the deltas in this changegroup
Useful for passing to the underlying storage system to be stored.
"""
chain = None
for chunkdata in iter(lambda: self.deltachunk(chain), {}):
node = chunkdata['node']
p1 = chunkdata['p1']
p2 = chunkdata['p2']
cs = chunkdata['cs']
deltabase = chunkdata['deltabase']
delta = chunkdata['delta']
flags = chunkdata['flags']
link = linkmapper(cs)
chain = node
yield (node, p1, p2, link, deltabase, delta, flags)
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 class cg2unpacker(cg1unpacker):
Augie Fackler
changegroup: document the public surface area of cg?unpackers...
r26708 """Unpacker for cg2 streams.
cg2 streams add support for generaldelta, so the delta header
format is slightly different. All other features about the data
remain the same.
"""
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 deltaheader = _CHANGEGROUPV2_DELTA_HEADER
deltaheadersize = struct.calcsize(deltaheader)
Eric Sumner
changegroup.writebundle: HG2Y support...
r23896 version = '02'
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181
def _deltaheader(self, headertuple, prevnode):
node, p1, p2, deltabase, cs = headertuple
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 flags = 0
return node, p1, p2, deltabase, cs, flags
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 class cg3unpacker(cg2unpacker):
"""Unpacker for cg3 streams.
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 cg3 streams add support for exchanging treemanifests and revlog
Martin von Zweigbergk
changegroup3: add empty chunk separating directories and files...
r27753 flags. It adds the revlog flags to the delta header and an empty chunk
separating manifests and files.
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 """
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 deltaheader = _CHANGEGROUPV3_DELTA_HEADER
deltaheadersize = struct.calcsize(deltaheader)
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 version = '03'
Martin von Zweigbergk
changegroup: cg3 has two empty groups *after* manifests...
r27920 _grouplistcount = 2 # One list of manifests and one list of files
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 def _deltaheader(self, headertuple, prevnode):
node, p1, p2, deltabase, cs, flags = headertuple
return node, p1, p2, deltabase, cs, flags
Martin von Zweigbergk
changegroup3: move treemanifest support into _unpackmanifests()...
r27754 def _unpackmanifests(self, repo, revmap, trp, prog, numchanges):
super(cg3unpacker, self)._unpackmanifests(repo, revmap, trp, prog,
numchanges)
Augie Fackler
changegroup: use `iter(callable, sentinel)` instead of while True...
r29724 for chunkdata in iter(self.filelogheader, {}):
Martin von Zweigbergk
changegroup3: move treemanifest support into _unpackmanifests()...
r27754 # If we get here, there are directory manifests in the changegroup
d = chunkdata["filename"]
repo.ui.debug("adding %s revisions\n" % d)
Durham Goode
changegroup: remove remaining uses of repo.manifest...
r30339 dirlog = repo.manifestlog._revlog.dirlog(d)
Durham Goode
changegroup: remove changegroup dependency from revlog.addgroup...
r34147 deltas = self.deltaiter(revmap)
if not dirlog.addgroup(deltas, trp):
Martin von Zweigbergk
changegroup3: move treemanifest support into _unpackmanifests()...
r27754 raise error.Abort(_("received dir revlog group is empty"))
Matt Mackall
bundle: push chunkbuffer down into decompress...
r12329 class headerlessfixup(object):
def __init__(self, fh, h):
self._h = h
self._fh = fh
def read(self, n):
if self._h:
d, self._h = self._h[:n], self._h[n:]
if len(d) < n:
Mads Kiilerich
changegroup: verify all stream reads...
r13457 d += readexactly(self._fh, n - len(d))
Matt Mackall
bundle: push chunkbuffer down into decompress...
r12329 return d
Mads Kiilerich
changegroup: verify all stream reads...
r13457 return readexactly(self._fh, n)
Matt Mackall
bundle: push chunkbuffer down into decompress...
r12329
Sune Foldager
changegroup: rename bundle-related functions and classes...
r22390 class cg1packer(object):
deltaheader = _CHANGEGROUPV1_DELTA_HEADER
Eric Sumner
changegroup.writebundle: HG2Y support...
r23896 version = '01'
Durham Goode
changegroup: add bundlecaps back...
r32287 def __init__(self, repo, bundlecaps=None):
Sune Foldager
bundle-ng: move gengroup into bundler, pass repo object to bundler...
r19202 """Given a source repo, construct a bundler.
Durham Goode
changegroup: add bundlecaps back...
r32287
bundlecaps is optional and can be used to specify the set of
capabilities which can be used to build the bundle. While bundlecaps is
unused in core Mercurial, extensions rely on this feature to communicate
capabilities to customize the changegroup packer.
Sune Foldager
bundle-ng: move gengroup into bundler, pass repo object to bundler...
r19202 """
Durham Goode
changegroup: add bundlecaps back...
r32287 # Set of capabilities we can use to build the bundle.
if bundlecaps is None:
bundlecaps = set()
self._bundlecaps = bundlecaps
Matt Mackall
generaldelta: mark experimental reordering option
r25831 # experimental config: bundle.reorder
configitems: register the 'bundle.reorder' config
r33180 reorder = repo.ui.config('bundle', 'reorder')
Sune Foldager
bundle-ng: move gengroup into bundler, pass repo object to bundler...
r19202 if reorder == 'auto':
reorder = None
else:
reorder = util.parsebool(reorder)
self._repo = repo
self._reorder = reorder
Benoit Boissinot
bundle-ng: move progress handling out of the linkrev callback
r19208 self._progress = repo.ui.progress
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 if self._repo.ui.verbose and not self._repo.ui.debugflag:
self._verbosenote = self._repo.ui.note
else:
self._verbosenote = lambda s: None
Matt Mackall
changegroup: introduce bundler objects...
r13831 def close(self):
return closechunk()
Sune Foldager
bundle-ng: move group into the bundler...
r19200
Matt Mackall
changegroup: introduce bundler objects...
r13831 def fileheader(self, fname):
return chunkheader(len(fname)) + fname
Sune Foldager
bundle-ng: move group into the bundler...
r19200
Augie Fackler
changegroup: extract method that sorts nodes to send...
r29236 # Extracted both for clarity and for overriding in extensions.
def _sortgroup(self, revlog, nodelist, lookup):
"""Sort nodes for change group and turn them into revnums."""
# for generaldelta revlogs, we linearize the revs; this will both be
# much quicker and generate a much smaller bundle
if (revlog._generaldelta and self._reorder is None) or self._reorder:
dag = dagutil.revlogdag(revlog)
return dag.linearize(set(revlog.rev(n) for n in nodelist))
else:
return sorted([revlog.rev(n) for n in nodelist])
Martin von Zweigbergk
changegroup.group: drop 'reorder' parameter...
r24912 def group(self, nodelist, revlog, lookup, units=None):
Sune Foldager
bundle-ng: move group into the bundler...
r19200 """Calculate a delta group, yielding a sequence of changegroup chunks
(strings).
Given a list of changeset revs, return a set of deltas and
metadata corresponding to nodes. The first delta is
first parent(nodelist[0]) -> nodelist[0], the receiver is
guaranteed to have this parent as it has all history before
these changesets. In the case firstparent is nullrev the
changegroup starts with a full revision.
Benoit Boissinot
bundle-ng: move progress handling out of the linkrev callback
r19208
If units is not None, progress detail will be generated, units specifies
the type of revlog that is touched (changelog, manifest, etc.).
Sune Foldager
bundle-ng: move group into the bundler...
r19200 """
# if we don't have any revisions touched by these changesets, bail
if len(nodelist) == 0:
yield self.close()
return
Augie Fackler
changegroup: extract method that sorts nodes to send...
r29236 revs = self._sortgroup(revlog, nodelist, lookup)
Sune Foldager
bundle-ng: move group into the bundler...
r19200
# add the parent of the first rev
p = revlog.parentrevs(revs[0])[0]
revs.insert(0, p)
# build deltas
Benoit Boissinot
bundle-ng: move progress handling out of the linkrev callback
r19208 total = len(revs) - 1
msgbundling = _('bundling')
Sune Foldager
bundle-ng: move group into the bundler...
r19200 for r in xrange(len(revs) - 1):
Benoit Boissinot
bundle-ng: move progress handling out of the linkrev callback
r19208 if units is not None:
self._progress(msgbundling, r + 1, unit=units, total=total)
Sune Foldager
bundle-ng: move group into the bundler...
r19200 prev, curr = revs[r], revs[r + 1]
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 linknode = lookup(revlog.node(curr))
for c in self.revchunk(revlog, curr, prev, linknode):
Sune Foldager
bundle-ng: move group into the bundler...
r19200 yield c
Martin von Zweigbergk
changegroup: close progress in same function as it's started...
r24901 if units is not None:
self._progress(msgbundling, None)
Sune Foldager
bundle-ng: move group into the bundler...
r19200 yield self.close()
Durham Goode
bundle: refactor changegroup prune to be its own function...
r19289 # filter any nodes that claim to be part of the known set
Martin von Zweigbergk
changegroup: removed unused 'source' parameter from prune()...
r24896 def prune(self, revlog, missing, commonrevs):
Durham Goode
bundle: refactor changegroup prune to be its own function...
r19289 rr, rl = revlog.rev, revlog.linkrev
return [n for n in missing if rl(rr(n)) not in commonrevs]
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 def _packmanifests(self, dir, mfnodes, lookuplinknode):
Augie Fackler
changegroup: move manifest packing into a separate function...
r26711 """Pack flat manifests into a changegroup stream."""
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 assert not dir
Durham Goode
changegroup: remove remaining uses of repo.manifest...
r30339 for chunk in self.group(mfnodes, self._repo.manifestlog._revlog,
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 lookuplinknode, units=_('manifests')):
Augie Fackler
changegroup: move manifest packing into a separate function...
r26711 yield chunk
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228
def _manifestsdone(self):
return ''
Augie Fackler
changegroup: move manifest packing into a separate function...
r26711
Benoit Boissinot
bundle-ng: move bundle generation to changegroup.py
r19204 def generate(self, commonrevs, clnodes, fastpathlinkrev, source):
Sune Foldager
bundle-ng: move gengroup into bundler, pass repo object to bundler...
r19202 '''yield a sequence of changegroup chunks (strings)'''
repo = self._repo
Martin von Zweigbergk
changegroup: drop _changelog and _manifest properties...
r24978 cl = repo.changelog
Benoit Boissinot
bundle-ng: move bundle generation to changegroup.py
r19204
Durham Goode
changegroup: fix file linkrevs during reorders (issue4462)...
r23381 clrevorder = {}
Benoit Boissinot
bundle-ng: move bundle generation to changegroup.py
r19204 mfs = {} # needed manifests
fnodes = {} # needed file nodes
Martin von Zweigbergk
changegroup: drop special-casing of flat manifests...
r28241 changedfiles = set()
Benoit Boissinot
bundle-ng: move bundle generation to changegroup.py
r19204
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 # Callback for the changelog, used to collect changed files and manifest
# nodes.
# Returns the linkrev node (identity in the changelog case).
def lookupcl(x):
c = cl.read(x)
Durham Goode
changegroup: fix file linkrevs during reorders (issue4462)...
r23381 clrevorder[x] = len(clrevorder)
Augie Fackler
changegroup: avoid iterating the whole manifest...
r27237 n = c[0]
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 # record the first changeset introducing this manifest version
Augie Fackler
changegroup: avoid iterating the whole manifest...
r27237 mfs.setdefault(n, x)
# Record a complete list of potentially-changed files in
# this manifest.
Martin von Zweigbergk
changegroup: drop special-casing of flat manifests...
r28241 changedfiles.update(c[3])
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 return x
Benoit Boissinot
bundle-ng: move bundle generation to changegroup.py
r19204
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 self._verbosenote(_('uncompressed size of bundle content:\n'))
size = 0
Martin von Zweigbergk
changegroup.group: drop 'reorder' parameter...
r24912 for chunk in self.group(clnodes, cl, lookupcl, units=_('changesets')):
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 size += len(chunk)
Gregory Szorc
changegroup: don't define lookupmf() until it is needed...
r23224 yield chunk
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 self._verbosenote(_('%8.i (changelog)\n') % size)
Gregory Szorc
changegroup: don't define lookupmf() until it is needed...
r23224
Martin von Zweigbergk
changegroup: document the cases where reordering complicates linkrevs
r24977 # We need to make sure that the linkrev in the changegroup refers to
# the first changeset that introduced the manifest or file revision.
# The fastpath is usually safer than the slowpath, because the filelogs
# are walked in revlog order.
#
# When taking the slowpath with reorder=None and the manifest revlog
# uses generaldelta, the manifest may be walked in the "wrong" order.
# Without 'clrevorder', we would get an incorrect linkrev (see fix in
# cc0ff93d0c0c).
#
# When taking the fastpath, we are only vulnerable to reordering
# of the changelog itself. The changelog never uses generaldelta, so
# it is only reordered when reorder=True. To handle this case, we
# simply take the slowpath, which already has the 'clrevorder' logic.
# This was also fixed in cc0ff93d0c0c.
Martin von Zweigbergk
changegroup: extract condition for linkrev fastpath...
r24976 fastpathlinkrev = fastpathlinkrev and not self._reorder
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 # Treemanifests don't work correctly with fastpathlinkrev
# either, because we don't discover which directory nodes to
# send along with files. This could probably be fixed.
fastpathlinkrev = fastpathlinkrev and (
'treemanifest' not in repo.requirements)
Martin von Zweigbergk
changegroup: extract generatemanifests()...
r28227
for chunk in self.generatemanifests(commonrevs, clrevorder,
Durham Goode
changegroup: add source parameter to generatemanifests...
r34148 fastpathlinkrev, mfs, fnodes, source):
Martin von Zweigbergk
changegroup: extract generatemanifests()...
r28227 yield chunk
mfs.clear()
clrevs = set(cl.rev(x) for x in clnodes)
if not fastpathlinkrev:
def linknodes(unused, fname):
return fnodes.get(fname, {})
else:
cln = cl.node
def linknodes(filerevlog, fname):
llr = filerevlog.linkrev
fln = filerevlog.node
revs = ((r, llr(r)) for r in filerevlog)
return dict((fln(r), cln(lr)) for r, lr in revs if lr in clrevs)
for chunk in self.generatefiles(changedfiles, linknodes, commonrevs,
source):
yield chunk
yield self.close()
if clnodes:
repo.hook('outgoing', node=hex(clnodes[0]), source=source)
def generatemanifests(self, commonrevs, clrevorder, fastpathlinkrev, mfs,
Durham Goode
changegroup: add source parameter to generatemanifests...
r34148 fnodes, source):
"""Returns an iterator of changegroup chunks containing manifests.
`source` is unused here, but is used by extensions like remotefilelog to
change what is sent based in pulls vs pushes, etc.
"""
Martin von Zweigbergk
changegroup: extract generatemanifests()...
r28227 repo = self._repo
Durham Goode
manifest: get rid of manifest.readshallowfast...
r30294 mfl = repo.manifestlog
dirlog = mfl._revlog.dirlog
Martin von Zweigbergk
changegroup: write root manifests and subdir manifests in a single loop...
r28232 tmfnodes = {'': mfs}
Martin von Zweigbergk
changegroup: extract generatemanifests()...
r28227
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 # Callback for the manifest, used to collect linkrevs for filelog
# revisions.
# Returns the linkrev node (collected in lookupcl).
Martin von Zweigbergk
changegroup: introduce makelookupmflinknode(dir)...
r28231 def makelookupmflinknode(dir):
if fastpathlinkrev:
assert not dir
return mfs.__getitem__
Augie Fackler
changegroup: remove one special case from lookupmflinknode...
r27239 def lookupmflinknode(x):
"""Callback for looking up the linknode for manifests.
Augie Fackler
changegroup: document manifest linkrev callback some more...
r27219
Augie Fackler
changegroup: remove one special case from lookupmflinknode...
r27239 Returns the linkrev node for the specified manifest.
Augie Fackler
changegroup: document manifest linkrev callback some more...
r27219
Augie Fackler
changegroup: remove one special case from lookupmflinknode...
r27239 SIDE EFFECT:
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 1) fclnodes gets populated with the list of relevant
file nodes if we're not using fastpathlinkrev
2) When treemanifests are in use, collects treemanifest nodes
to send
Augie Fackler
changegroup: document manifest linkrev callback some more...
r27219
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 Note that this means manifests must be completely sent to
the client before you can trust the list of files and
treemanifests to send.
Augie Fackler
changegroup: remove one special case from lookupmflinknode...
r27239 """
Martin von Zweigbergk
changegroup: fix treemanifests on merges...
r28240 clnode = tmfnodes[dir][x]
Durham Goode
manifest: get rid of manifest.readshallowfast...
r30294 mdata = mfl.get(dir, x).readfast(shallow=True)
Martin von Zweigbergk
changegroup: drop special-casing of flat manifests...
r28241 for p, n, fl in mdata.iterentries():
if fl == 't': # subdirectory manifest
subdir = dir + p + '/'
tmfclnodes = tmfnodes.setdefault(subdir, {})
tmfclnode = tmfclnodes.setdefault(n, clnode)
if clrevorder[clnode] < clrevorder[tmfclnode]:
tmfclnodes[n] = clnode
else:
f = dir + p
Martin von Zweigbergk
changegroup: fix treemanifests on merges...
r28240 fclnodes = fnodes.setdefault(f, {})
fclnode = fclnodes.setdefault(n, clnode)
if clrevorder[clnode] < clrevorder[fclnode]:
fclnodes[n] = clnode
Augie Fackler
changegroup: remove one special case from lookupmflinknode...
r27239 return clnode
Martin von Zweigbergk
changegroup: introduce makelookupmflinknode(dir)...
r28231 return lookupmflinknode
Sune Foldager
bundle-ng: simplify bundle10.generate...
r19206
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 size = 0
Martin von Zweigbergk
changegroup: write root manifests and subdir manifests in a single loop...
r28232 while tmfnodes:
dir = min(tmfnodes)
nodes = tmfnodes[dir]
Martin von Zweigbergk
changegroup: fix treemanifests on merges...
r28240 prunednodes = self.prune(dirlog(dir), nodes, commonrevs)
Martin von Zweigbergk
changegroup: don't send empty subdirectory manifest groups...
r29371 if not dir or prunednodes:
for x in self._packmanifests(dir, prunednodes,
makelookupmflinknode(dir)):
size += len(x)
yield x
Martin von Zweigbergk
changegroup: write root manifests and subdir manifests in a single loop...
r28232 del tmfnodes[dir]
Martin von Zweigbergk
changegroup: include subdirectory manifests in verbose size...
r28229 self._verbosenote(_('%8.i (manifests)\n') % size)
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 yield self._manifestsdone()
Sune Foldager
bundle-ng: simplify bundle10.generate...
r19206
Martin von Zweigbergk
changegroup: document that 'source' parameter exists for extensions...
r24897 # The 'source' parameter is useful for extensions
Durham Goode
bundle: move file chunk generation to it's own function...
r19334 def generatefiles(self, changedfiles, linknodes, commonrevs, source):
repo = self._repo
progress = self._progress
msgbundling = _('bundling')
total = len(changedfiles)
# for progress output
msgfiles = _('files')
for i, fname in enumerate(sorted(changedfiles)):
filerevlog = repo.file(fname)
if not filerevlog:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("empty or missing revlog for %s") % fname)
Durham Goode
bundle: move file chunk generation to it's own function...
r19334
linkrevnodes = linknodes(filerevlog, fname)
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 # Lookup for filenodes, we collected the linkrev nodes above in the
# fastpath case and with lookupmf in the slowpath case.
def lookupfilelog(x):
return linkrevnodes[x]
Martin von Zweigbergk
changegroup: removed unused 'source' parameter from prune()...
r24896 filenodes = self.prune(filerevlog, linkrevnodes, commonrevs)
Sune Foldager
bundle-ng: simplify bundle10.generate...
r19206 if filenodes:
Benoit Boissinot
bundle-ng: move progress handling out of the linkrev callback
r19208 progress(msgbundling, i + 1, item=fname, unit=msgfiles,
total=total)
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 h = self.fileheader(fname)
size = len(h)
yield h
Martin von Zweigbergk
changegroup.group: drop 'reorder' parameter...
r24912 for chunk in self.group(filenodes, filerevlog, lookupfilelog):
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 size += len(chunk)
Sune Foldager
bundle-ng: move gengroup into bundler, pass repo object to bundler...
r19202 yield chunk
Mads Kiilerich
bundle: when verbose, show what takes up the space in the generated bundle...
r23748 self._verbosenote(_('%8.i %s\n') % (size, fname))
Martin von Zweigbergk
changegroup: close progress in same function as it's started...
r24901 progress(msgbundling, None)
Sune Foldager
bundle-ng: move group into the bundler...
r19200
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 def deltaparent(self, revlog, rev, p1, p2, prev):
return prev
Benoit Boissinot
bundle-ng: simplify lookup and state handling...
r19207 def revchunk(self, revlog, rev, prev, linknode):
Benoit Boissinot
changegroup: new bundler API
r14143 node = revlog.node(rev)
p1, p2 = revlog.parentrevs(rev)
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 base = self.deltaparent(revlog, rev, p1, p2, prev)
Benoit Boissinot
changegroup: new bundler API
r14143
prefix = ''
Mike Edgar
changegroup: emit full-replacement deltas if either revision is censored...
r24190 if revlog.iscensored(base) or revlog.iscensored(rev):
try:
Remi Chaintron
revlog: add 'raw' argument to revision and _addrevision...
r30743 delta = revlog.revision(node, raw=True)
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except error.CensoredNodeError as e:
Mike Edgar
changegroup: emit full-replacement deltas if either revision is censored...
r24190 delta = e.tombstone
if base == nullrev:
prefix = mdiff.trivialdiffheader(len(delta))
else:
baselen = revlog.rawsize(base)
prefix = mdiff.replacediffheader(baselen, len(delta))
elif base == nullrev:
Remi Chaintron
revlog: add 'raw' argument to revision and _addrevision...
r30743 delta = revlog.revision(node, raw=True)
Benoit Boissinot
changegroup: new bundler API
r14143 prefix = mdiff.trivialdiffheader(len(delta))
else:
delta = revlog.revdiff(base, rev)
p1n, p2n = revlog.parents(node)
basenode = revlog.node(base)
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 flags = revlog.flags(rev)
meta = self.builddeltaheader(node, p1n, p2n, basenode, linknode, flags)
Benoit Boissinot
changegroup: new bundler API
r14143 meta += prefix
l = len(meta) + len(delta)
Matt Mackall
changegroup: introduce bundler objects...
r13831 yield chunkheader(l)
yield meta
Benoit Boissinot
changegroup: new bundler API
r14143 yield delta
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
Benoit Boissinot
changegroup: new bundler API
r14143 # do nothing with basenode, it is implicitly the previous one in HG10
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 # do nothing with flags, it is implicitly 0 for cg1 and cg2
Benoit Boissinot
changegroup: new bundler API
r14143 return struct.pack(self.deltaheader, node, p1n, p2n, linknode)
Pierre-Yves David
localrepo: move the _changegroupsubset method in changegroup module...
r20925
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 class cg2packer(cg1packer):
Eric Sumner
changegroup.writebundle: HG2Y support...
r23896 version = '02'
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 deltaheader = _CHANGEGROUPV2_DELTA_HEADER
Durham Goode
changegroup: add bundlecaps back...
r32287 def __init__(self, repo, bundlecaps=None):
super(cg2packer, self).__init__(repo, bundlecaps)
Martin von Zweigbergk
cg2packer: set reorder=False in __init__ instead of in group()...
r24911 if self._reorder is None:
# Since generaldelta is directly supported by cg2, reordering
# generally doesn't help, so we disable it by default (treating
# bundle.reorder=auto just like bundle.reorder=False).
self._reorder = False
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181
def deltaparent(self, revlog, rev, p1, p2, prev):
dp = revlog.deltaparent(rev)
Pierre-Yves David
changegroup: skip delta when the underlying revlog do not use them...
r30211 if dp == nullrev and revlog.storedeltachains:
# Avoid sending full revisions when delta parent is null. Pick prev
# in that case. It's tempting to pick p1 in this case, as p1 will
# be smaller in the common case. However, computing a delta against
# p1 may require resolving the raw text of p1, which could be
# expensive. The revlog caches should have prev cached, meaning
# less CPU for changegroup generation. There is likely room to add
# a flag and/or config option to control this behavior.
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 return prev
Pierre-Yves David
changegroup: skip delta when the underlying revlog do not use them...
r30211 elif dp == nullrev:
# revlog is configured to use full snapshot for a reason,
# stick to full snapshot.
return nullrev
elif dp not in (p1, p2, prev):
# Pick prev when we can't be sure remote has the base revision.
return prev
else:
return dp
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
# Do nothing with flags, it is implicitly 0 in cg1 and cg2
Sune Foldager
changegroup: introduce cg2packer/unpacker...
r23181 return struct.pack(self.deltaheader, node, p1n, p2n, basenode, linknode)
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 class cg3packer(cg2packer):
version = '03'
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 deltaheader = _CHANGEGROUPV3_DELTA_HEADER
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 def _packmanifests(self, dir, mfnodes, lookuplinknode):
if dir:
yield self.fileheader(dir)
Durham Goode
changegroup: remove remaining uses of repo.manifest...
r30339
dirlog = self._repo.manifestlog._revlog.dirlog(dir)
for chunk in self.group(mfnodes, dirlog, lookuplinknode,
units=_('manifests')):
Martin von Zweigbergk
changegroup: make _packmanifests() dumber...
r28228 yield chunk
def _manifestsdone(self):
return self.close()
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
return struct.pack(
self.deltaheader, node, p1n, p2n, basenode, linknode, flags)
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432
Martin von Zweigbergk
changegroup: hide packermap behind methods...
r27751 _packermap = {'01': (cg1packer, cg1unpacker),
Augie Fackler
changegroup: reformat packermap and add comment...
r26709 # cg2 adds support for exchanging generaldelta
'02': (cg2packer, cg2unpacker),
Martin von Zweigbergk
changegroup3: add empty chunk separating directories and files...
r27753 # cg3 adds support for exchanging revlog flags and treemanifests
Augie Fackler
changegroup: introduce cg3, which has support for exchanging treemanifests...
r27432 '03': (cg3packer, cg3unpacker),
Augie Fackler
changegroup: reformat packermap and add comment...
r26709 }
Pierre-Yves David
changegroup: add a "packermap" dictionary to track different packer versions...
r23168
Pierre-Yves David
changegroup: pass 'repo' to allsupportedversions...
r30627 def allsupportedversions(repo):
Martin von Zweigbergk
changegroup: don't support versions 01 and 02 with treemanifests...
r27928 versions = set(_packermap.keys())
Pierre-Yves David
changegroup: pass 'repo' to allsupportedversions...
r30627 if not (repo.ui.configbool('experimental', 'changegroup3') or
Pierre-Yves David
changegroup: simplify logic around enabling changegroup 03...
r30628 repo.ui.configbool('experimental', 'treemanifest') or
'treemanifest' in repo.requirements):
Pierre-Yves David
changegroup: simplify 'allsupportedversions' logic...
r30626 versions.discard('03')
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953 return versions
# Changegroup versions that can be applied to the repo
def supportedincomingversions(repo):
Pierre-Yves David
changegroup: simplify logic around enabling changegroup 03...
r30628 return allsupportedversions(repo)
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953
# Changegroup versions that can be created from the repo
def supportedoutgoingversions(repo):
Pierre-Yves David
changegroup: pass 'repo' to allsupportedversions...
r30627 versions = allsupportedversions(repo)
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953 if 'treemanifest' in repo.requirements:
Martin von Zweigbergk
changegroup: don't support versions 01 and 02 with treemanifests...
r27928 # Versions 01 and 02 support only flat manifests and it's just too
# expensive to convert between the flat manifest and tree manifest on
# the fly. Since tree manifests are hashed differently, all of history
# would have to be converted. Instead, we simply don't even pretend to
# support versions 01 and 02.
versions.discard('01')
versions.discard('02')
Martin von Zweigbergk
changegroup3: introduce experimental.changegroup3 boolean config...
r27752 return versions
Martin von Zweigbergk
changegroup: hide packermap behind methods...
r27751
Martin von Zweigbergk
repair: preserve phase also when not using generaldelta (issue5678)...
r34179 def localversion(repo):
# Finds the best version to use for bundles that are meant to be used
# locally, such as those from strip and shelve, and temporary bundles.
return max(supportedoutgoingversions(repo))
Martin von Zweigbergk
changegroup: introduce safeversion()...
r27929 def safeversion(repo):
# Finds the smallest version that it's safe to assume clients of the repo
Martin von Zweigbergk
shelve: use cg3 for treemanifests...
r27931 # will support. For example, all hg versions that support generaldelta also
# support changegroup 02.
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953 versions = supportedoutgoingversions(repo)
Martin von Zweigbergk
changegroup: introduce safeversion()...
r27929 if 'generaldelta' in repo.requirements:
versions.discard('01')
assert versions
return min(versions)
Durham Goode
changegroup: add bundlecaps back...
r32287 def getbundler(version, repo, bundlecaps=None):
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953 assert version in supportedoutgoingversions(repo)
Durham Goode
changegroup: add bundlecaps back...
r32287 return _packermap[version][0](repo, bundlecaps)
Martin von Zweigbergk
changegroup: hide packermap behind methods...
r27751
Gregory Szorc
bundle2: store changeset count when creating file bundles...
r29593 def getunbundler(version, fh, alg, extras=None):
return _packermap[version][1](fh, alg, extras=extras)
Martin von Zweigbergk
changegroup: hide packermap behind methods...
r27751
Pierre-Yves David
localrepo: move the changegroupinfo method in changegroup module...
r20926 def _changegroupinfo(repo, nodes, source):
if repo.ui.verbose or source == 'bundle':
repo.ui.status(_("%d changesets found\n") % len(nodes))
if repo.ui.debugflag:
repo.ui.debug("list of changesets:\n")
for node in nodes:
repo.ui.debug("%s\n" % hex(node))
Durham Goode
changegroup: replace getsubset with makechangegroup...
r34098 def makechangegroup(repo, outgoing, version, source, fastpath=False,
bundlecaps=None):
cgstream = makestream(repo, outgoing, version, source,
fastpath=fastpath, bundlecaps=bundlecaps)
return getunbundler(version, util.chunkbuffer(cgstream), None,
{'clcount': len(outgoing.missing) })
Durham Goode
changegroup: rename getsubsetraw to makestream...
r34105 def makestream(repo, outgoing, version, source, fastpath=False,
bundlecaps=None):
bundler = getbundler(version, repo, bundlecaps=bundlecaps)
Pierre-Yves David
localrepo: move the _changegroupsubset method in changegroup module...
r20925 repo = repo.unfiltered()
commonrevs = outgoing.common
csets = outgoing.missing
heads = outgoing.missingheads
# We go through the fast path if we get told to, or if all (unfiltered
# heads have been requested (since we then know there all linkrevs will
# be pulled by the client).
heads.sort()
fastpathlinkrev = fastpath or (
repo.filtername is None and heads == sorted(repo.heads()))
repo.hook('preoutgoing', throw=True, source=source)
Pierre-Yves David
localrepo: move the changegroupinfo method in changegroup module...
r20926 _changegroupinfo(repo, csets, source)
Sune Foldager
changegroup: introduce "raw" versions of some commands...
r23177 return bundler.generate(commonrevs, csets, fastpathlinkrev, source)
Martin von Zweigbergk
changegroup: progress for added files is not measured in "chunks"...
r28361 def _addchangegroupfiles(repo, source, revmap, trp, expectedfiles, needfiles):
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 revisions = 0
files = 0
Augie Fackler
changegroup: use `iter(callable, sentinel)` instead of while True...
r29724 for chunkdata in iter(source.filelogheader, {}):
Martin von Zweigbergk
changegroup: progress for added files is not measured in "chunks"...
r28361 files += 1
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 f = chunkdata["filename"]
repo.ui.debug("adding %s revisions\n" % f)
Martin von Zweigbergk
changegroup: progress for added files is not measured in "chunks"...
r28361 repo.ui.progress(_('files'), files, unit=_('files'),
total=expectedfiles)
Martin von Zweigbergk
changegroup3: move treemanifest support into _unpackmanifests()...
r27754 fl = repo.file(f)
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 o = len(fl)
Mike Edgar
revlog: in addgroup, reject ill-formed deltas based on censored nodes...
r24120 try:
Durham Goode
changegroup: remove changegroup dependency from revlog.addgroup...
r34147 deltas = source.deltaiter(revmap)
if not fl.addgroup(deltas, trp):
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("received file revlog group is empty"))
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except error.CensoredBaseError as e:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_("received delta base is censored: %s") % e)
Martin von Zweigbergk
changegroup3: move treemanifest support into _unpackmanifests()...
r27754 revisions += len(fl) - o
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 if f in needfiles:
needs = needfiles[f]
for new in xrange(o, len(fl)):
n = fl.node(new)
if n in needs:
needs.remove(n)
else:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 _("received spurious file revlog entry"))
if not needs:
del needfiles[f]
repo.ui.progress(_('files'), None)
for f, needs in needfiles.iteritems():
fl = repo.file(f)
for n in needs:
try:
fl.rev(n)
except error.LookupError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(
Pierre-Yves David
localrepo: move the addchangegroupfiles method in changegroup module...
r20932 _('missing file data for %s:%s - run hg verify') %
(f, hex(n)))
return revisions, files