bundle2.py
1923 lines
| 69.6 KiB
| text/x-python
|
PythonLexer
/ mercurial / bundle2.py
|
r20801 | # bundle2.py - generic container format to transmit arbitrary data. | ||
# | ||||
# Copyright 2013 Facebook, Inc. | ||||
# | ||||
# This software may be used and distributed according to the terms of the | ||||
# GNU General Public License version 2 or any later version. | ||||
"""Handling of the new bundle2 format | ||||
The goal of bundle2 is to act as an atomically packet to transmit a set of | ||||
payloads in an application agnostic way. It consist in a sequence of "parts" | ||||
that will be handed to and processed by the application layer. | ||||
General format architecture | ||||
=========================== | ||||
The format is architectured as follow | ||||
- magic string | ||||
- stream level parameters | ||||
- payload parts (any number) | ||||
- end of stream marker. | ||||
|
r20856 | the Binary format | ||
|
r20801 | ============================ | ||
|
r21024 | All numbers are unsigned and big-endian. | ||
|
r20801 | |||
stream level parameters | ||||
------------------------ | ||||
Binary format is as follow | ||||
|
r23009 | :params size: int32 | ||
|
r20801 | |||
The total number of Bytes used by the parameters | ||||
:params value: arbitrary number of Bytes | ||||
A blob of `params size` containing the serialized version of all stream level | ||||
parameters. | ||||
|
r21024 | The blob contains a space separated list of parameters. Parameters with value | ||
|
r20811 | are stored in the form `<name>=<value>`. Both name and value are urlquoted. | ||
|
r20804 | |||
|
r20813 | Empty name are obviously forbidden. | ||
|
r20844 | Name MUST start with a letter. If this first letter is lower case, the | ||
|
r21024 | parameter is advisory and can be safely ignored. However when the first | ||
|
r20844 | letter is capital, the parameter is mandatory and the bundling process MUST | ||
stop if he is not able to proceed it. | ||||
|
r20814 | |||
|
r20808 | Stream parameters use a simple textual format for two main reasons: | ||
|
r20804 | |||
|
r21024 | - Stream level parameters should remain simple and we want to discourage any | ||
|
r20808 | crazy usage. | ||
|
r21024 | - Textual data allow easy human inspection of a bundle2 header in case of | ||
|
r20808 | troubles. | ||
Any Applicative level options MUST go into a bundle2 part instead. | ||||
|
r20801 | |||
Payload part | ||||
------------------------ | ||||
Binary format is as follow | ||||
|
r23009 | :header size: int32 | ||
|
r20801 | |||
|
r25507 | The total number of Bytes used by the part header. When the header is empty | ||
|
r20801 | (size = 0) this is interpreted as the end of stream marker. | ||
|
r20856 | :header: | ||
The header defines how to interpret the part. It contains two piece of | ||||
data: the part type, and the part parameters. | ||||
The part type is used to route an application level handler, that can | ||||
interpret payload. | ||||
Part parameters are passed to the application level handler. They are | ||||
meant to convey information that will help the application level object to | ||||
interpret the part payload. | ||||
The binary format of the header is has follow | ||||
:typesize: (one byte) | ||||
|
r20877 | |||
|
r23916 | :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*) | ||
|
r20877 | |||
|
r20995 | :partid: A 32bits integer (unique in the bundle) that can be used to refer | ||
to this part. | ||||
|
r20877 | :parameters: | ||
|
r21024 | Part's parameter may have arbitrary content, the binary structure is:: | ||
|
r20877 | |||
<mandatory-count><advisory-count><param-sizes><param-data> | ||||
:mandatory-count: 1 byte, number of mandatory parameters | ||||
:advisory-count: 1 byte, number of advisory parameters | ||||
:param-sizes: | ||||
N couple of bytes, where N is the total number of parameters. Each | ||||
couple contains (<size-of-key>, <size-of-value) for one parameter. | ||||
:param-data: | ||||
A blob of bytes from which each parameter key and value can be | ||||
retrieved using the list of size couples stored in the previous | ||||
field. | ||||
Mandatory parameters comes first, then the advisory ones. | ||||
|
r20856 | |||
|
r21607 | Each parameter's key MUST be unique within the part. | ||
|
r20856 | :payload: | ||
|
r20876 | payload is a series of `<chunksize><chunkdata>`. | ||
|
r23009 | `chunksize` is an int32, `chunkdata` are plain bytes (as much as | ||
|
r20876 | `chunksize` says)` The payload part is concluded by a zero size chunk. | ||
The current implementation always produces either zero or one chunk. | ||||
|
r21024 | This is an implementation limitation that will ultimately be lifted. | ||
|
r20891 | |||
|
r23009 | `chunksize` can be negative to trigger special case processing. No such | ||
processing is in place yet. | ||||
|
r20891 | Bundle processing | ||
============================ | ||||
Each part is processed in order using a "part handler". Handler are registered | ||||
for a certain part type. | ||||
The matching of a part to its handler is case insensitive. The case of the | ||||
part type is used to know if a part is mandatory or advisory. If the Part type | ||||
contains any uppercase char it is considered mandatory. When no handler is | ||||
known for a Mandatory part, the process is aborted and an exception is raised. | ||||
|
r20892 | If the part is advisory and no handler is known, the part is ignored. When the | ||
process is aborted, the full bundle is still read from the stream to keep the | ||||
channel usable. But none of the part read from an abort are processed. In the | ||||
future, dropping the stream may become an option for channel we do not care to | ||||
preserve. | ||||
|
r20801 | """ | ||
|
r33635 | from __future__ import absolute_import, division | ||
|
r25919 | |||
|
r24026 | import errno | ||
|
r25919 | import re | ||
|
r20814 | import string | ||
|
r25919 | import struct | ||
import sys | ||||
|
r20804 | |||
|
r25919 | from .i18n import _ | ||
from . import ( | ||||
changegroup, | ||||
error, | ||||
obsolete, | ||||
|
r33031 | phases, | ||
|
r25919 | pushkey, | ||
|
r30030 | pycompat, | ||
|
r25919 | tags, | ||
url, | ||||
util, | ||||
) | ||||
|
r20802 | |||
|
r28883 | urlerr = util.urlerr | ||
urlreq = util.urlreq | ||||
|
r20804 | _pack = struct.pack | ||
_unpack = struct.unpack | ||||
|
r23009 | _fstreamparamsize = '>i' | ||
_fpartheadersize = '>i' | ||||
|
r20856 | _fparttypesize = '>B' | ||
|
r20995 | _fpartid = '>I' | ||
|
r23009 | _fpayloadsize = '>i' | ||
|
r20877 | _fpartparamcount = '>BB' | ||
|
r21001 | preferedchunksize = 4096 | ||
|
r23868 | _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]') | ||
|
r25313 | def outdebug(ui, message): | ||
"""debug regarding output stream (bundling)""" | ||||
r33160 | if ui.configbool('devel', 'bundle2.debug'): | |||
|
r25336 | ui.debug('bundle2-output: %s\n' % message) | ||
|
r25313 | |||
|
r25318 | def indebug(ui, message): | ||
"""debug on input stream (unbundling)""" | ||||
r33160 | if ui.configbool('devel', 'bundle2.debug'): | |||
|
r25336 | ui.debug('bundle2-input: %s\n' % message) | ||
|
r25318 | |||
|
r23868 | def validateparttype(parttype): | ||
"""raise ValueError if a parttype contains invalid character""" | ||||
|
r23916 | if _parttypeforbidden.search(parttype): | ||
|
r23868 | raise ValueError(parttype) | ||
|
r20877 | def _makefpartparamsizes(nbparams): | ||
"""return a struct format to read part parameter sizes | ||||
The number parameters is variable so we need to build that format | ||||
dynamically. | ||||
""" | ||||
return '>'+('BB'*nbparams) | ||||
|
r20804 | |||
|
r20890 | parthandlermapping = {} | ||
|
r20889 | |||
|
r21623 | def parthandler(parttype, params=()): | ||
|
r20890 | """decorator that register a function as a bundle2 part handler | ||
eg:: | ||||
|
r21624 | @parthandler('myparttype', ('mandatory', 'param', 'handled')) | ||
|
r20890 | def myparttypehandler(...): | ||
'''process a part of type "my part".''' | ||||
... | ||||
""" | ||||
|
r23868 | validateparttype(parttype) | ||
|
r20890 | def _decorator(func): | ||
|
r20891 | lparttype = parttype.lower() # enforce lower case matching. | ||
assert lparttype not in parthandlermapping | ||||
parthandlermapping[lparttype] = func | ||||
|
r21623 | func.params = frozenset(params) | ||
|
r20890 | return func | ||
return _decorator | ||||
|
r20889 | |||
|
r20949 | class unbundlerecords(object): | ||
"""keep record of what happens during and unbundle | ||||
New records are added using `records.add('cat', obj)`. Where 'cat' is a | ||||
|
r21024 | category of record and obj is an arbitrary object. | ||
|
r20949 | |||
`records['cat']` will return all entries of this category 'cat'. | ||||
Iterating on the object itself will yield `('category', obj)` tuples | ||||
for all entries. | ||||
All iterations happens in chronological order. | ||||
""" | ||||
def __init__(self): | ||||
self._categories = {} | ||||
self._sequences = [] | ||||
|
r20996 | self._replies = {} | ||
|
r20949 | |||
|
r20996 | def add(self, category, entry, inreplyto=None): | ||
|
r20949 | """add a new record of a given category. | ||
The entry can then be retrieved in the list returned by | ||||
self['category'].""" | ||||
self._categories.setdefault(category, []).append(entry) | ||||
self._sequences.append((category, entry)) | ||||
|
r20996 | if inreplyto is not None: | ||
self.getreplies(inreplyto).add(category, entry) | ||||
def getreplies(self, partid): | ||||
|
r23139 | """get the records that are replies to a specific part""" | ||
|
r20996 | return self._replies.setdefault(partid, unbundlerecords()) | ||
|
r20949 | |||
def __getitem__(self, cat): | ||||
return tuple(self._categories.get(cat, ())) | ||||
def __iter__(self): | ||||
return iter(self._sequences) | ||||
def __len__(self): | ||||
return len(self._sequences) | ||||
def __nonzero__(self): | ||||
return bool(self._sequences) | ||||
|
r31476 | __bool__ = __nonzero__ | ||
|
r20948 | class bundleoperation(object): | ||
"""an object that represents a single bundling process | ||||
Its purpose is to carry unbundle-related objects and states. | ||||
A new object should be created at the beginning of each bundle processing. | ||||
The object is to be returned by the processing function. | ||||
The object has very little content now it will ultimately contain: | ||||
* an access to the repo the bundle is applied to, | ||||
* a ui object, | ||||
* a way to retrieve a transaction to add changes to the repo, | ||||
* a way to record the result of processing each part, | ||||
* a way to construct a bundle response when applicable. | ||||
""" | ||||
|
r24878 | def __init__(self, repo, transactiongetter, captureoutput=True): | ||
|
r20948 | self.repo = repo | ||
self.ui = repo.ui | ||||
|
r20949 | self.records = unbundlerecords() | ||
|
r20997 | self.reply = None | ||
|
r24878 | self.captureoutput = captureoutput | ||
|
r33629 | self.hookargs = {} | ||
|
r33630 | self._gettransaction = transactiongetter | ||
def gettransaction(self): | ||||
transaction = self._gettransaction() | ||||
|
r33809 | if self.hookargs: | ||
|
r33630 | # the ones added to the transaction supercede those added | ||
# to the operation. | ||||
self.hookargs.update(transaction.hookargs) | ||||
transaction.hookargs = self.hookargs | ||||
|
r33809 | # mark the hookargs as flushed. further attempts to add to | ||
# hookargs will result in an abort. | ||||
self.hookargs = None | ||||
|
r33630 | |||
return transaction | ||||
|
r33629 | |||
def addhookargs(self, hookargs): | ||||
|
r33630 | if self.hookargs is None: | ||
|
r33808 | raise error.ProgrammingError('attempted to add hookargs to ' | ||
'operation after transaction started') | ||||
|
r33629 | self.hookargs.update(hookargs) | ||
|
r20948 | |||
|
r20952 | class TransactionUnavailable(RuntimeError): | ||
pass | ||||
def _notransaction(): | ||||
"""default method to get a transaction while processing a bundle | ||||
Raise an exception to highlight the fact that no transaction was expected | ||||
to be created""" | ||||
raise TransactionUnavailable() | ||||
|
r33043 | def applybundle(repo, unbundler, tr, source=None, url=None, **kwargs): | ||
|
r26790 | # transform me into unbundler.apply() as soon as the freeze is lifted | ||
|
r33043 | if isinstance(unbundler, unbundle20): | ||
tr.hookargs['bundle2'] = '1' | ||||
if source is not None and 'source' not in tr.hookargs: | ||||
tr.hookargs['source'] = source | ||||
if url is not None and 'url' not in tr.hookargs: | ||||
tr.hookargs['url'] = url | ||||
return processbundle(repo, unbundler, lambda: tr) | ||||
else: | ||||
|
r33044 | # the transactiongetter won't be used, but we might as well set it | ||
op = bundleoperation(repo, lambda: tr) | ||||
_processchangegroup(op, unbundler, tr, source, url, **kwargs) | ||||
return op | ||||
|
r26790 | |||
|
r34149 | class partiterator(object): | ||
|
r34151 | def __init__(self, repo, op, unbundler): | ||
|
r34150 | self.repo = repo | ||
|
r34151 | self.op = op | ||
|
r34149 | self.unbundler = unbundler | ||
|
r34150 | self.iterator = None | ||
self.count = 0 | ||||
|
r34259 | self.current = None | ||
|
r34149 | |||
def __enter__(self): | ||||
|
r34150 | def func(): | ||
itr = enumerate(self.unbundler.iterparts()) | ||||
for count, p in itr: | ||||
self.count = count | ||||
|
r34259 | self.current = p | ||
|
r34150 | yield p | ||
|
r34259 | p.seek(0, 2) | ||
self.current = None | ||||
|
r34150 | self.iterator = func() | ||
return self.iterator | ||||
|
r34149 | |||
|
r34151 | def __exit__(self, type, exc, tb): | ||
|
r34150 | if not self.iterator: | ||
return | ||||
|
r34638 | # Only gracefully abort in a normal exception situation. User aborts | ||
# like Ctrl+C throw a KeyboardInterrupt which is not a base Exception, | ||||
# and should not gracefully cleanup. | ||||
if isinstance(exc, Exception): | ||||
|
r34151 | # Any exceptions seeking to the end of the bundle at this point are | ||
# almost certainly related to the underlying stream being bad. | ||||
# And, chances are that the exception we're handling is related to | ||||
# getting in that bad state. So, we swallow the seeking error and | ||||
# re-raise the original error. | ||||
seekerror = False | ||||
try: | ||||
|
r34638 | if self.current: | ||
# consume the part content to not corrupt the stream. | ||||
self.current.seek(0, 2) | ||||
|
r34151 | for part in self.iterator: | ||
# consume the bundle content | ||||
part.seek(0, 2) | ||||
except Exception: | ||||
seekerror = True | ||||
# Small hack to let caller code distinguish exceptions from bundle2 | ||||
# processing from processing the old format. This is mostly needed | ||||
# to handle different return codes to unbundle according to the type | ||||
# of bundle. We should probably clean up or drop this return code | ||||
# craziness in a future version. | ||||
exc.duringunbundle2 = True | ||||
salvaged = [] | ||||
replycaps = None | ||||
if self.op.reply is not None: | ||||
salvaged = self.op.reply.salvageoutput() | ||||
replycaps = self.op.reply.capabilities | ||||
exc._replycaps = replycaps | ||||
exc._bundle2salvagedoutput = salvaged | ||||
# Re-raising from a variable loses the original stack. So only use | ||||
# that form if we need to. | ||||
if seekerror: | ||||
raise exc | ||||
|
r34150 | self.repo.ui.debug('bundle2-input-bundle: %i parts total\n' % | ||
self.count) | ||||
|
r34149 | |||
|
r24851 | def processbundle(repo, unbundler, transactiongetter=None, op=None): | ||
|
r20889 | """This function process a bundle, apply effect to/from a repo | ||
|
r20947 | It iterates over each part then searches for and uses the proper handling | ||
code to process the part. Parts are processed in order. | ||||
|
r20889 | |||
|
r20891 | Unknown Mandatory part will abort the process. | ||
|
r24851 | |||
It is temporarily possible to provide a prebuilt bundleoperation to the | ||||
function. This is used to ensure output is properly propagated in case of | ||||
an error during the unbundling. This output capturing part will likely be | ||||
reworked and this ability will probably go away in the process. | ||||
|
r20889 | """ | ||
|
r24851 | if op is None: | ||
if transactiongetter is None: | ||||
transactiongetter = _notransaction | ||||
op = bundleoperation(repo, transactiongetter) | ||||
|
r20889 | # todo: | ||
# - replace this is a init function soon. | ||||
# - exception catching | ||||
unbundler.params | ||||
|
r25331 | if repo.ui.debugflag: | ||
msg = ['bundle2-input-bundle:'] | ||||
if unbundler.params: | ||||
|
r33123 | msg.append(' %i params' % len(unbundler.params)) | ||
|
r33765 | if op._gettransaction is None or op._gettransaction is _notransaction: | ||
|
r25331 | msg.append(' no-transaction') | ||
else: | ||||
msg.append(' with-transaction') | ||||
msg.append('\n') | ||||
repo.ui.debug(''.join(msg)) | ||||
|
r34149 | |||
|
r34262 | processparts(repo, op, unbundler) | ||
return op | ||||
def processparts(repo, op, unbundler): | ||||
|
r34151 | with partiterator(repo, op, unbundler) as parts: | ||
for part in parts: | ||||
_processpart(op, part) | ||||
|
r25332 | |||
|
r33038 | def _processchangegroup(op, cg, tr, source, url, **kwargs): | ||
|
r33461 | ret = cg.apply(op.repo, tr, source, url, **kwargs) | ||
|
r33038 | op.records.add('changegroup', { | ||
'return': ret, | ||||
}) | ||||
return ret | ||||
|
r34260 | def _gethandler(op, part): | ||
status = 'unknown' # used by debug output | ||||
try: | ||||
handler = parthandlermapping.get(part.type) | ||||
if handler is None: | ||||
status = 'unsupported-type' | ||||
raise error.BundleUnknownFeatureError(parttype=part.type) | ||||
|
r34278 | indebug(op.ui, 'found a handler for part %s' % part.type) | ||
|
r34260 | unknownparams = part.mandatorykeys - handler.params | ||
if unknownparams: | ||||
unknownparams = list(unknownparams) | ||||
unknownparams.sort() | ||||
|
r34278 | status = 'unsupported-params (%s)' % ', '.join(unknownparams) | ||
|
r34260 | raise error.BundleUnknownFeatureError(parttype=part.type, | ||
params=unknownparams) | ||||
status = 'supported' | ||||
except error.BundleUnknownFeatureError as exc: | ||||
if part.mandatory: # mandatory parts | ||||
raise | ||||
indebug(op.ui, 'ignoring unsupported advisory part %s' % exc) | ||||
return # skip to part processing | ||||
finally: | ||||
if op.ui.debugflag: | ||||
msg = ['bundle2-input-part: "%s"' % part.type] | ||||
if not part.mandatory: | ||||
msg.append(' (advisory)') | ||||
nbmp = len(part.mandatorykeys) | ||||
nbap = len(part.params) - nbmp | ||||
if nbmp or nbap: | ||||
msg.append(' (params:') | ||||
if nbmp: | ||||
msg.append(' %i mandatory' % nbmp) | ||||
if nbap: | ||||
msg.append(' %i advisory' % nbmp) | ||||
msg.append(')') | ||||
msg.append(' %s\n' % status) | ||||
op.ui.debug(''.join(msg)) | ||||
return handler | ||||
|
r23008 | def _processpart(op, part): | ||
"""process a single part from a bundle | ||||
The part is guaranteed to have been fully consumed when the function exits | ||||
(even if an exception is raised).""" | ||||
|
r34261 | handler = _gethandler(op, part) | ||
if handler is None: | ||||
return | ||||
|
r23008 | |||
|
r34261 | # handler is called outside the above try block so that we don't | ||
# risk catching KeyErrors from anything other than the | ||||
# parthandlermapping lookup (any KeyError raised by handler() | ||||
# itself represents a defect of a different variety). | ||||
output = None | ||||
if op.captureoutput and op.reply is not None: | ||||
op.ui.pushbuffer(error=True, subproc=True) | ||||
output = '' | ||||
try: | ||||
handler(op, part) | ||||
|
r23008 | finally: | ||
|
r34261 | if output is not None: | ||
output = op.ui.popbuffer() | ||||
if output: | ||||
outpart = op.reply.newpart('output', data=output, | ||||
mandatory=False) | ||||
outpart.addparam( | ||||
'in-reply-to', pycompat.bytestr(part.id), mandatory=False) | ||||
|
r23008 | |||
|
r21138 | def decodecaps(blob): | ||
|
r23139 | """decode a bundle2 caps bytes blob into a dictionary | ||
|
r21138 | |||
The blob is a list of capabilities (one per line) | ||||
Capabilities may have values using a line of the form:: | ||||
capability=value1,value2,value3 | ||||
The values are always a list.""" | ||||
caps = {} | ||||
for line in blob.splitlines(): | ||||
if not line: | ||||
continue | ||||
if '=' not in line: | ||||
key, vals = line, () | ||||
else: | ||||
key, vals = line.split('=', 1) | ||||
vals = vals.split(',') | ||||
|
r28883 | key = urlreq.unquote(key) | ||
vals = [urlreq.unquote(v) for v in vals] | ||||
|
r21138 | caps[key] = vals | ||
return caps | ||||
|
r21139 | def encodecaps(caps): | ||
"""encode a bundle2 caps dictionary into a bytes blob""" | ||||
chunks = [] | ||||
for ca in sorted(caps): | ||||
vals = caps[ca] | ||||
|
r28883 | ca = urlreq.quote(ca) | ||
vals = [urlreq.quote(v) for v in vals] | ||||
|
r21139 | if vals: | ||
ca = "%s=%s" % (ca, ','.join(vals)) | ||||
chunks.append(ca) | ||||
return '\n'.join(chunks) | ||||
|
r28666 | bundletypes = { | ||
|
r30351 | "": ("", 'UN'), # only when using unbundle on ssh and old http servers | ||
|
r28666 | # since the unification ssh accepts a header but there | ||
# is no capability signaling it. | ||||
"HG20": (), # special-cased below | ||||
|
r30351 | "HG10UN": ("HG10UN", 'UN'), | ||
|
r28666 | "HG10BZ": ("HG10", 'BZ'), | ||
"HG10GZ": ("HG10GZ", 'GZ'), | ||||
} | ||||
# hgweb uses this list to communicate its preferred type | ||||
bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN'] | ||||
|
r20801 | class bundle20(object): | ||
"""represent an outgoing bundle2 container | ||||
|
r21599 | Use the `addparam` method to add stream level parameter. and `newpart` to | ||
|
r20856 | populate it. Then call `getchunks` to retrieve all the binary chunks of | ||
|
r21024 | data that compose the bundle2 container.""" | ||
|
r20801 | |||
|
r24686 | _magicstring = 'HG20' | ||
|
r24640 | |||
|
r21134 | def __init__(self, ui, capabilities=()): | ||
|
r20842 | self.ui = ui | ||
|
r20801 | self._params = [] | ||
self._parts = [] | ||||
|
r21136 | self.capabilities = dict(capabilities) | ||
|
r30351 | self._compengine = util.compengines.forbundletype('UN') | ||
|
r30757 | self._compopts = None | ||
|
r26404 | |||
|
r30757 | def setcompression(self, alg, compopts=None): | ||
|
r26404 | """setup core part compression to <alg>""" | ||
|
r30439 | if alg in (None, 'UN'): | ||
|
r26404 | return | ||
|
r30915 | assert not any(n.lower() == 'compression' for n, v in self._params) | ||
|
r26404 | self.addparam('Compression', alg) | ||
|
r30351 | self._compengine = util.compengines.forbundletype(alg) | ||
|
r30757 | self._compopts = compopts | ||
|
r20801 | |||
|
r21900 | @property | ||
def nbparts(self): | ||||
"""total number of parts added to the bundler""" | ||||
return len(self._parts) | ||||
|
r21597 | # methods used to defines the bundle2 content | ||
|
r20804 | def addparam(self, name, value=None): | ||
"""add a stream level parameter""" | ||||
|
r20813 | if not name: | ||
|
r34280 | raise ValueError(r'empty parameter name') | ||
|
r34281 | if name[0:1] not in pycompat.bytestr(string.ascii_letters): | ||
|
r34280 | raise ValueError(r'non letter first character: %s' % name) | ||
|
r20804 | self._params.append((name, value)) | ||
|
r20856 | def addpart(self, part): | ||
"""add a new part to the bundle2 container | ||||
|
r21024 | Parts contains the actual applicative payload.""" | ||
|
r20995 | assert part.id is None | ||
part.id = len(self._parts) # very cheap counter | ||||
|
r20856 | self._parts.append(part) | ||
|
r21598 | def newpart(self, typeid, *args, **kwargs): | ||
|
r21602 | """create a new part and add it to the containers | ||
As the part is directly added to the containers. For now, this means | ||||
that any failure to properly initialize the part after calling | ||||
``newpart`` should result in a failure of the whole bundling process. | ||||
You can still fall back to manually create and add if you need better | ||||
control.""" | ||||
|
r21598 | part = bundlepart(typeid, *args, **kwargs) | ||
|
r21599 | self.addpart(part) | ||
|
r21598 | return part | ||
|
r21597 | # methods used to generate the bundle2 stream | ||
|
r20801 | def getchunks(self): | ||
|
r25322 | if self.ui.debugflag: | ||
msg = ['bundle2-output-bundle: "%s",' % self._magicstring] | ||||
if self._params: | ||||
msg.append(' (%i params)' % len(self._params)) | ||||
msg.append(' %i parts total\n' % len(self._parts)) | ||||
self.ui.debug(''.join(msg)) | ||||
|
r25315 | outdebug(self.ui, 'start emission of %s stream' % self._magicstring) | ||
|
r24640 | yield self._magicstring | ||
|
r20804 | param = self._paramchunk() | ||
|
r25315 | outdebug(self.ui, 'bundle parameter: %s' % param) | ||
|
r20804 | yield _pack(_fstreamparamsize, len(param)) | ||
if param: | ||||
yield param | ||||
|
r30757 | for chunk in self._compengine.compressstream(self._getcorechunk(), | ||
self._compopts): | ||||
|
r30357 | yield chunk | ||
|
r20802 | |||
|
r20804 | def _paramchunk(self): | ||
"""return a encoded version of all stream parameters""" | ||||
blocks = [] | ||||
|
r20809 | for par, value in self._params: | ||
|
r28883 | par = urlreq.quote(par) | ||
|
r20809 | if value is not None: | ||
|
r28883 | value = urlreq.quote(value) | ||
|
r20809 | par = '%s=%s' % (par, value) | ||
blocks.append(par) | ||||
|
r20804 | return ' '.join(blocks) | ||
|
r26396 | def _getcorechunk(self): | ||
"""yield chunk for the core part of the bundle | ||||
(all but headers and parameters)""" | ||||
outdebug(self.ui, 'start of parts') | ||||
for part in self._parts: | ||||
outdebug(self.ui, 'bundle part: "%s"' % part.type) | ||||
for chunk in part.getchunks(ui=self.ui): | ||||
yield chunk | ||||
outdebug(self.ui, 'end of bundle') | ||||
yield _pack(_fpartheadersize, 0) | ||||
|
r24794 | def salvageoutput(self): | ||
"""return a list with a copy of all output parts in the bundle | ||||
This is meant to be used during error handling to make sure we preserve | ||||
server output""" | ||||
salvaged = [] | ||||
for part in self._parts: | ||||
if part.type.startswith('output'): | ||||
salvaged.append(part.copy()) | ||||
return salvaged | ||||
|
r21013 | class unpackermixin(object): | ||
"""A mixin to extract bytes and struct data from a stream""" | ||||
|
r20802 | |||
|
r21013 | def __init__(self, fp): | ||
|
r20802 | self._fp = fp | ||
def _unpack(self, format): | ||||
|
r31862 | """unpack this struct format from the stream | ||
This method is meant for internal usage by the bundle2 protocol only. | ||||
They directly manipulate the low level stream including bundle2 level | ||||
instruction. | ||||
Do not use it to implement higher-level logic or methods.""" | ||||
|
r20802 | data = self._readexact(struct.calcsize(format)) | ||
return _unpack(format, data) | ||||
def _readexact(self, size): | ||||
|
r31862 | """read exactly <size> bytes from the stream | ||
This method is meant for internal usage by the bundle2 protocol only. | ||||
They directly manipulate the low level stream including bundle2 level | ||||
instruction. | ||||
Do not use it to implement higher-level logic or methods.""" | ||||
|
r20802 | return changegroup.readexactly(self._fp, size) | ||
|
r25640 | def getunbundler(ui, fp, magicstring=None): | ||
"""return a valid unbundler object for a given magicstring""" | ||||
if magicstring is None: | ||||
magicstring = changegroup.readexactly(fp, 4) | ||||
magic, version = magicstring[0:2], magicstring[2:4] | ||||
|
r24648 | if magic != 'HG': | ||
|
r33122 | ui.debug( | ||
"error: invalid magic: %r (version %r), should be 'HG'\n" | ||||
% (magic, version)) | ||||
|
r26587 | raise error.Abort(_('not a Mercurial bundle')) | ||
|
r24648 | unbundlerclass = formatmap.get(version) | ||
if unbundlerclass is None: | ||||
|
r26587 | raise error.Abort(_('unknown bundle version %s') % version) | ||
|
r24648 | unbundler = unbundlerclass(ui, fp) | ||
|
r25640 | indebug(ui, 'start processing of %s stream' % magicstring) | ||
|
r24642 | return unbundler | ||
|
r24641 | |||
|
r21013 | class unbundle20(unpackermixin): | ||
"""interpret a bundle2 stream | ||||
|
r21129 | This class is fed with a binary stream and yields parts through its | ||
`iterparts` methods.""" | ||||
|
r21013 | |||
|
r26542 | _magicstring = 'HG20' | ||
|
r24642 | def __init__(self, ui, fp): | ||
|
r21066 | """If header is specified, we do not read it out of the stream.""" | ||
|
r21013 | self.ui = ui | ||
|
r30353 | self._compengine = util.compengines.forbundletype('UN') | ||
|
r26802 | self._compressed = None | ||
|
r21013 | super(unbundle20, self).__init__(fp) | ||
|
r20802 | @util.propertycache | ||
def params(self): | ||||
|
r21024 | """dictionary of stream level parameters""" | ||
|
r25320 | indebug(self.ui, 'reading bundle2 stream parameters') | ||
|
r20805 | params = {} | ||
paramssize = self._unpack(_fstreamparamsize)[0] | ||||
|
r23011 | if paramssize < 0: | ||
raise error.BundleValueError('negative bundle param size: %i' | ||||
% paramssize) | ||||
|
r20805 | if paramssize: | ||
|
r26541 | params = self._readexact(paramssize) | ||
params = self._processallparams(params) | ||||
|
r20805 | return params | ||
|
r20802 | |||
|
r26541 | def _processallparams(self, paramsblock): | ||
"""""" | ||||
|
r29591 | params = util.sortdict() | ||
|
r26541 | for p in paramsblock.split(' '): | ||
p = p.split('=', 1) | ||||
|
r28883 | p = [urlreq.unquote(i) for i in p] | ||
|
r26541 | if len(p) < 2: | ||
p.append(None) | ||||
self._processparam(*p) | ||||
params[p[0]] = p[1] | ||||
return params | ||||
|
r20844 | def _processparam(self, name, value): | ||
"""process a parameter, applying its effect if needed | ||||
Parameter starting with a lower case letter are advisory and will be | ||||
ignored when unknown. Those starting with an upper case letter are | ||||
mandatory and will this function will raise a KeyError when unknown. | ||||
Note: no option are currently supported. Any input will be either | ||||
ignored or failing. | ||||
""" | ||||
if not name: | ||||
|
r34278 | raise ValueError(r'empty parameter name') | ||
if name[0:1] not in pycompat.bytestr(string.ascii_letters): | ||||
raise ValueError(r'non letter first character: %s' % name) | ||||
|
r26395 | try: | ||
handler = b2streamparamsmap[name.lower()] | ||||
except KeyError: | ||||
|
r34278 | if name[0:1].islower(): | ||
indebug(self.ui, "ignoring unknown parameter %s" % name) | ||||
|
r26395 | else: | ||
raise error.BundleUnknownFeatureError(params=(name,)) | ||||
|
r20844 | else: | ||
|
r26395 | handler(self, name, value) | ||
|
r20844 | |||
|
r26542 | def _forwardchunks(self): | ||
"""utility to transfer a bundle2 as binary | ||||
This is made necessary by the fact the 'getbundle' command over 'ssh' | ||||
have no way to know then the reply end, relying on the bundle to be | ||||
interpreted to know its end. This is terrible and we are sorry, but we | ||||
needed to move forward to get general delta enabled. | ||||
""" | ||||
yield self._magicstring | ||||
assert 'params' not in vars(self) | ||||
paramssize = self._unpack(_fstreamparamsize)[0] | ||||
if paramssize < 0: | ||||
raise error.BundleValueError('negative bundle param size: %i' | ||||
% paramssize) | ||||
yield _pack(_fstreamparamsize, paramssize) | ||||
if paramssize: | ||||
params = self._readexact(paramssize) | ||||
self._processallparams(params) | ||||
yield params | ||||
|
r30353 | assert self._compengine.bundletype == 'UN' | ||
|
r26542 | # From there, payload might need to be decompressed | ||
|
r30353 | self._fp = self._compengine.decompressorreader(self._fp) | ||
|
r26542 | emptycount = 0 | ||
while emptycount < 2: | ||||
# so we can brainlessly loop | ||||
assert _fpartheadersize == _fpayloadsize | ||||
size = self._unpack(_fpartheadersize)[0] | ||||
yield _pack(_fpartheadersize, size) | ||||
if size: | ||||
emptycount = 0 | ||||
else: | ||||
emptycount += 1 | ||||
continue | ||||
if size == flaginterrupt: | ||||
continue | ||||
elif size < 0: | ||||
raise error.BundleValueError('negative chunk size: %i') | ||||
yield self._readexact(size) | ||||
|
r21129 | def iterparts(self): | ||
|
r20802 | """yield all parts contained in the stream""" | ||
# make sure param have been loaded | ||||
self.params | ||||
|
r26404 | # From there, payload need to be decompressed | ||
|
r30353 | self._fp = self._compengine.decompressorreader(self._fp) | ||
|
r25320 | indebug(self.ui, 'start extraction of bundle2 parts') | ||
|
r21014 | headerblock = self._readpartheader() | ||
while headerblock is not None: | ||||
part = unbundlepart(self.ui, headerblock, self._fp) | ||||
|
r20802 | yield part | ||
|
r33889 | # Seek to the end of the part to force it's consumption so the next | ||
# part can be read. But then seek back to the beginning so the | ||||
# code consuming this generator has a part that starts at 0. | ||||
|
r24048 | part.seek(0, 2) | ||
|
r33889 | part.seek(0) | ||
|
r21014 | headerblock = self._readpartheader() | ||
|
r25320 | indebug(self.ui, 'end of bundle2 stream') | ||
|
r20802 | |||
|
r21014 | def _readpartheader(self): | ||
"""reads a part header size and return the bytes blob | ||||
|
r20864 | |||
|
r21014 | returns None if empty""" | ||
|
r20864 | headersize = self._unpack(_fpartheadersize)[0] | ||
|
r23011 | if headersize < 0: | ||
raise error.BundleValueError('negative part header size: %i' | ||||
% headersize) | ||||
|
r25320 | indebug(self.ui, 'part header size: %i' % headersize) | ||
|
r21014 | if headersize: | ||
return self._readexact(headersize) | ||||
return None | ||||
|
r20864 | |||
|
r24071 | def compressed(self): | ||
|
r26802 | self.params # load params | ||
return self._compressed | ||||
|
r20802 | |||
|
r31863 | def close(self): | ||
"""close underlying file""" | ||||
if util.safehasattr(self._fp, 'close'): | ||||
return self._fp.close() | ||||
|
r24686 | formatmap = {'20': unbundle20} | ||
|
r24648 | |||
|
r26395 | b2streamparamsmap = {} | ||
def b2streamparamhandler(name): | ||||
"""register a handler for a stream level parameter""" | ||||
def decorator(func): | ||||
assert name not in formatmap | ||||
b2streamparamsmap[name] = func | ||||
return func | ||||
return decorator | ||||
|
r26404 | @b2streamparamhandler('compression') | ||
def processcompression(unbundler, param, value): | ||||
"""read compression parameter and install payload decompression""" | ||||
|
r30353 | if value not in util.compengines.supportedbundletypes: | ||
|
r26404 | raise error.BundleUnknownFeatureError(params=(param,), | ||
values=(value,)) | ||||
|
r30353 | unbundler._compengine = util.compengines.forbundletype(value) | ||
|
r26802 | if value is not None: | ||
unbundler._compressed = True | ||||
|
r26404 | |||
|
r21005 | class bundlepart(object): | ||
|
r20856 | """A bundle2 part contains application level payload | ||
The part `type` is used to route the part to the application level | ||||
handler. | ||||
|
r21604 | |||
The part payload is contained in ``part.data``. It could be raw bytes or a | ||||
|
r21605 | generator of byte chunks. | ||
You can add parameters to the part using the ``addparam`` method. | ||||
Parameters can be either mandatory (default) or advisory. Remote side | ||||
should be able to safely ignore the advisory ones. | ||||
Both data and parameters cannot be modified after the generation has begun. | ||||
|
r20856 | """ | ||
|
r20877 | def __init__(self, parttype, mandatoryparams=(), advisoryparams=(), | ||
|
r23590 | data='', mandatory=True): | ||
|
r23868 | validateparttype(parttype) | ||
|
r20995 | self.id = None | ||
|
r20856 | self.type = parttype | ||
|
r21604 | self._data = data | ||
|
r21605 | self._mandatoryparams = list(mandatoryparams) | ||
self._advisoryparams = list(advisoryparams) | ||||
|
r21607 | # checking for duplicated entries | ||
self._seenparams = set() | ||||
for pname, __ in self._mandatoryparams + self._advisoryparams: | ||||
if pname in self._seenparams: | ||||
|
r31647 | raise error.ProgrammingError('duplicated params: %s' % pname) | ||
|
r21607 | self._seenparams.add(pname) | ||
|
r21601 | # status of the part's generation: | ||
# - None: not started, | ||||
# - False: currently generated, | ||||
# - True: generation done. | ||||
self._generated = None | ||||
|
r23590 | self.mandatory = mandatory | ||
|
r20856 | |||
|
r30872 | def __repr__(self): | ||
cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__) | ||||
return ('<%s object at %x; id: %s; type: %s; mandatory: %s>' | ||||
% (cls, id(self), self.id, self.type, self.mandatory)) | ||||
|
r24793 | def copy(self): | ||
"""return a copy of the part | ||||
The new part have the very same content but no partid assigned yet. | ||||
Parts with generated data cannot be copied.""" | ||||
assert not util.safehasattr(self.data, 'next') | ||||
return self.__class__(self.type, self._mandatoryparams, | ||||
self._advisoryparams, self._data, self.mandatory) | ||||
|
r21604 | # methods used to defines the part content | ||
|
r27879 | @property | ||
def data(self): | ||||
return self._data | ||||
@data.setter | ||||
def data(self, data): | ||||
|
r21604 | if self._generated is not None: | ||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21604 | self._data = data | ||
|
r21605 | @property | ||
def mandatoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._mandatoryparams) | ||||
@property | ||||
def advisoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._advisoryparams) | ||||
def addparam(self, name, value='', mandatory=True): | ||||
|
r31861 | """add a parameter to the part | ||
If 'mandatory' is set to True, the remote handler must claim support | ||||
for this parameter or the unbundling will be aborted. | ||||
The 'name' and 'value' cannot exceed 255 bytes each. | ||||
""" | ||||
|
r21605 | if self._generated is not None: | ||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21607 | if name in self._seenparams: | ||
raise ValueError('duplicated params: %s' % name) | ||||
self._seenparams.add(name) | ||||
|
r21605 | params = self._advisoryparams | ||
if mandatory: | ||||
params = self._mandatoryparams | ||||
params.append((name, value)) | ||||
|
r21601 | # methods used to generates the bundle2 stream | ||
|
r25321 | def getchunks(self, ui): | ||
|
r21601 | if self._generated is not None: | ||
|
r31647 | raise error.ProgrammingError('part can only be consumed once') | ||
|
r21601 | self._generated = False | ||
|
r25323 | |||
if ui.debugflag: | ||||
msg = ['bundle2-output-part: "%s"' % self.type] | ||||
if not self.mandatory: | ||||
msg.append(' (advisory)') | ||||
nbmp = len(self.mandatoryparams) | ||||
nbap = len(self.advisoryparams) | ||||
if nbmp or nbap: | ||||
msg.append(' (params:') | ||||
if nbmp: | ||||
msg.append(' %i mandatory' % nbmp) | ||||
if nbap: | ||||
msg.append(' %i advisory' % nbmp) | ||||
msg.append(')') | ||||
if not self.data: | ||||
msg.append(' empty payload') | ||||
|
r34279 | elif (util.safehasattr(self.data, 'next') | ||
or util.safehasattr(self.data, '__next__')): | ||||
|
r25323 | msg.append(' streamed payload') | ||
else: | ||||
msg.append(' %i bytes payload' % len(self.data)) | ||||
msg.append('\n') | ||||
ui.debug(''.join(msg)) | ||||
|
r20877 | #### header | ||
|
r23590 | if self.mandatory: | ||
parttype = self.type.upper() | ||||
else: | ||||
parttype = self.type.lower() | ||||
|
r33677 | outdebug(ui, 'part %s: "%s"' % (pycompat.bytestr(self.id), parttype)) | ||
|
r20877 | ## parttype | ||
|
r23590 | header = [_pack(_fparttypesize, len(parttype)), | ||
parttype, _pack(_fpartid, self.id), | ||||
|
r20856 | ] | ||
|
r20877 | ## parameters | ||
# count | ||||
manpar = self.mandatoryparams | ||||
advpar = self.advisoryparams | ||||
header.append(_pack(_fpartparamcount, len(manpar), len(advpar))) | ||||
# size | ||||
parsizes = [] | ||||
for key, value in manpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
for key, value in advpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
|
r33635 | paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes) | ||
|
r20877 | header.append(paramsizes) | ||
# key, value | ||||
for key, value in manpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
for key, value in advpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
## finalize header | ||||
|
r34246 | try: | ||
headerchunk = ''.join(header) | ||||
except TypeError: | ||||
raise TypeError(r'Found a non-bytes trying to ' | ||||
r'build bundle part header: %r' % header) | ||||
|
r25321 | outdebug(ui, 'header chunk size: %i' % len(headerchunk)) | ||
|
r20856 | yield _pack(_fpartheadersize, len(headerchunk)) | ||
yield headerchunk | ||||
|
r20877 | ## payload | ||
|
r23067 | try: | ||
for chunk in self._payloadchunks(): | ||||
|
r25321 | outdebug(ui, 'payload chunk size: %i' % len(chunk)) | ||
|
r23067 | yield _pack(_fpayloadsize, len(chunk)) | ||
yield chunk | ||||
|
r26144 | except GeneratorExit: | ||
# GeneratorExit means that nobody is listening for our | ||||
# results anyway, so just bail quickly rather than trying | ||||
# to produce an error part. | ||||
ui.debug('bundle2-generatorexit\n') | ||||
raise | ||||
|
r25660 | except BaseException as exc: | ||
|
r33645 | bexc = util.forcebytestr(exc) | ||
|
r23067 | # backup exception data for later | ||
|
r25324 | ui.debug('bundle2-input-stream-interrupt: encoding exception %s' | ||
|
r33639 | % bexc) | ||
|
r32186 | tb = sys.exc_info()[2] | ||
|
r33639 | msg = 'unexpected error: %s' % bexc | ||
|
r24686 | interpart = bundlepart('error:abort', [('message', msg)], | ||
|
r23590 | mandatory=False) | ||
|
r23067 | interpart.id = 0 | ||
yield _pack(_fpayloadsize, -1) | ||||
|
r25321 | for chunk in interpart.getchunks(ui=ui): | ||
|
r23067 | yield chunk | ||
|
r25321 | outdebug(ui, 'closing payload chunk') | ||
|
r23067 | # abort current part payload | ||
yield _pack(_fpayloadsize, 0) | ||||
|
r32186 | pycompat.raisewithtb(exc, tb) | ||
|
r21000 | # end of payload | ||
|
r25321 | outdebug(ui, 'closing payload chunk') | ||
|
r21000 | yield _pack(_fpayloadsize, 0) | ||
|
r21601 | self._generated = True | ||
|
r21000 | |||
def _payloadchunks(self): | ||||
"""yield chunks of a the part payload | ||||
Exists to handle the different methods to provide data to a part.""" | ||||
|
r20876 | # we only support fixed size data now. | ||
# This will be improved in the future. | ||||
|
r33636 | if (util.safehasattr(self.data, 'next') | ||
or util.safehasattr(self.data, '__next__')): | ||||
|
r21001 | buff = util.chunkbuffer(self.data) | ||
chunk = buff.read(preferedchunksize) | ||||
while chunk: | ||||
yield chunk | ||||
chunk = buff.read(preferedchunksize) | ||||
elif len(self.data): | ||||
|
r20876 | yield self.data | ||
|
r20802 | |||
|
r23066 | |||
flaginterrupt = -1 | ||||
class interrupthandler(unpackermixin): | ||||
"""read one part and process it with restricted capability | ||||
This allows to transmit exception raised on the producer size during part | ||||
iteration while the consumer is reading a part. | ||||
Part processed in this manner only have access to a ui object,""" | ||||
def __init__(self, ui, fp): | ||||
super(interrupthandler, self).__init__(fp) | ||||
self.ui = ui | ||||
def _readpartheader(self): | ||||
"""reads a part header size and return the bytes blob | ||||
returns None if empty""" | ||||
headersize = self._unpack(_fpartheadersize)[0] | ||||
if headersize < 0: | ||||
raise error.BundleValueError('negative part header size: %i' | ||||
% headersize) | ||||
|
r25318 | indebug(self.ui, 'part header size: %i\n' % headersize) | ||
|
r23066 | if headersize: | ||
return self._readexact(headersize) | ||||
return None | ||||
def __call__(self): | ||||
|
r25335 | |||
self.ui.debug('bundle2-input-stream-interrupt:' | ||||
' opening out of band context\n') | ||||
|
r25320 | indebug(self.ui, 'bundle2 stream interruption, looking for a part.') | ||
|
r23066 | headerblock = self._readpartheader() | ||
if headerblock is None: | ||||
|
r25320 | indebug(self.ui, 'no part found during interruption.') | ||
|
r23066 | return | ||
part = unbundlepart(self.ui, headerblock, self._fp) | ||||
op = interruptoperation(self.ui) | ||||
|
r34259 | hardabort = False | ||
try: | ||||
_processpart(op, part) | ||||
except (SystemExit, KeyboardInterrupt): | ||||
hardabort = True | ||||
raise | ||||
finally: | ||||
if not hardabort: | ||||
part.seek(0, 2) | ||||
|
r25335 | self.ui.debug('bundle2-input-stream-interrupt:' | ||
' closing out of band context\n') | ||||
|
r23066 | |||
class interruptoperation(object): | ||||
"""A limited operation to be use by part handler during interruption | ||||
It only have access to an ui object. | ||||
""" | ||||
def __init__(self, ui): | ||||
self.ui = ui | ||||
self.reply = None | ||||
|
r24878 | self.captureoutput = False | ||
|
r23066 | |||
@property | ||||
def repo(self): | ||||
|
r31647 | raise error.ProgrammingError('no repo access from stream interruption') | ||
|
r23066 | |||
def gettransaction(self): | ||||
raise TransactionUnavailable('no repo access from stream interruption') | ||||
|
r21014 | class unbundlepart(unpackermixin): | ||
"""a bundle part read from a bundle""" | ||||
def __init__(self, ui, header, fp): | ||||
super(unbundlepart, self).__init__(fp) | ||||
|
r31889 | self._seekable = (util.safehasattr(fp, 'seek') and | ||
util.safehasattr(fp, 'tell')) | ||||
|
r21014 | self.ui = ui | ||
# unbundle state attr | ||||
self._headerdata = header | ||||
|
r21015 | self._headeroffset = 0 | ||
|
r21019 | self._initialized = False | ||
self.consumed = False | ||||
|
r21014 | # part data | ||
self.id = None | ||||
self.type = None | ||||
self.mandatoryparams = None | ||||
self.advisoryparams = None | ||||
|
r21610 | self.params = None | ||
|
r21612 | self.mandatorykeys = () | ||
|
r21019 | self._payloadstream = None | ||
self._readheader() | ||||
|
r23585 | self._mandatory = None | ||
|
r24035 | self._chunkindex = [] #(payload, file) position tuples for chunk starts | ||
|
r24036 | self._pos = 0 | ||
|
r21014 | |||
|
r21015 | def _fromheader(self, size): | ||
"""return the next <size> byte from the header""" | ||||
offset = self._headeroffset | ||||
data = self._headerdata[offset:(offset + size)] | ||||
|
r21019 | self._headeroffset = offset + size | ||
|
r21015 | return data | ||
|
r21016 | def _unpackheader(self, format): | ||
"""read given format from header | ||||
This automatically compute the size of the format to read.""" | ||||
data = self._fromheader(struct.calcsize(format)) | ||||
return _unpack(format, data) | ||||
|
r21608 | def _initparams(self, mandatoryparams, advisoryparams): | ||
"""internal function to setup all logic related parameters""" | ||||
|
r21609 | # make it read only to prevent people touching it by mistake. | ||
self.mandatoryparams = tuple(mandatoryparams) | ||||
self.advisoryparams = tuple(advisoryparams) | ||||
|
r21610 | # user friendly UI | ||
|
r29591 | self.params = util.sortdict(self.mandatoryparams) | ||
self.params.update(self.advisoryparams) | ||||
|
r21612 | self.mandatorykeys = frozenset(p[0] for p in mandatoryparams) | ||
|
r21608 | |||
|
r24035 | def _payloadchunks(self, chunknum=0): | ||
'''seek to specified chunk and start yielding data''' | ||||
if len(self._chunkindex) == 0: | ||||
assert chunknum == 0, 'Must start with chunk 0' | ||||
|
r31889 | self._chunkindex.append((0, self._tellfp())) | ||
|
r24035 | else: | ||
assert chunknum < len(self._chunkindex), \ | ||||
'Unknown chunk %d' % chunknum | ||||
|
r31889 | self._seekfp(self._chunkindex[chunknum][1]) | ||
|
r24035 | |||
pos = self._chunkindex[chunknum][0] | ||||
|
r24034 | payloadsize = self._unpack(_fpayloadsize)[0] | ||
|
r25320 | indebug(self.ui, 'payload chunk size: %i' % payloadsize) | ||
|
r24034 | while payloadsize: | ||
if payloadsize == flaginterrupt: | ||||
# interruption detection, the handler will now read a | ||||
# single part and process it. | ||||
interrupthandler(self.ui, self._fp)() | ||||
elif payloadsize < 0: | ||||
msg = 'negative payload chunk size: %i' % payloadsize | ||||
raise error.BundleValueError(msg) | ||||
else: | ||||
|
r24035 | result = self._readexact(payloadsize) | ||
chunknum += 1 | ||||
pos += payloadsize | ||||
if chunknum == len(self._chunkindex): | ||||
|
r31889 | self._chunkindex.append((pos, self._tellfp())) | ||
|
r24035 | yield result | ||
|
r24034 | payloadsize = self._unpack(_fpayloadsize)[0] | ||
|
r25320 | indebug(self.ui, 'payload chunk size: %i' % payloadsize) | ||
|
r24034 | |||
|
r24037 | def _findchunk(self, pos): | ||
'''for a given payload position, return a chunk number and offset''' | ||||
for chunk, (ppos, fpos) in enumerate(self._chunkindex): | ||||
if ppos == pos: | ||||
return chunk, 0 | ||||
elif ppos > pos: | ||||
return chunk - 1, pos - self._chunkindex[chunk - 1][0] | ||||
raise ValueError('Unknown chunk') | ||||
|
r21019 | def _readheader(self): | ||
|
r21014 | """read the header and setup the object""" | ||
|
r21016 | typesize = self._unpackheader(_fparttypesize)[0] | ||
|
r21015 | self.type = self._fromheader(typesize) | ||
|
r25320 | indebug(self.ui, 'part type: "%s"' % self.type) | ||
|
r21016 | self.id = self._unpackheader(_fpartid)[0] | ||
|
r33677 | indebug(self.ui, 'part id: "%s"' % pycompat.bytestr(self.id)) | ||
|
r23585 | # extract mandatory bit from type | ||
self.mandatory = (self.type != self.type.lower()) | ||||
self.type = self.type.lower() | ||||
|
r21014 | ## reading parameters | ||
# param count | ||||
|
r21016 | mancount, advcount = self._unpackheader(_fpartparamcount) | ||
|
r25320 | indebug(self.ui, 'part parameters: %i' % (mancount + advcount)) | ||
|
r21014 | # param size | ||
|
r21016 | fparamsizes = _makefpartparamsizes(mancount + advcount) | ||
paramsizes = self._unpackheader(fparamsizes) | ||||
|
r21014 | # make it a list of couple again | ||
|
r33637 | paramsizes = list(zip(paramsizes[::2], paramsizes[1::2])) | ||
|
r21014 | # split mandatory from advisory | ||
mansizes = paramsizes[:mancount] | ||||
advsizes = paramsizes[mancount:] | ||||
|
r23139 | # retrieve param value | ||
|
r21014 | manparams = [] | ||
for key, value in mansizes: | ||||
|
r21015 | manparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21014 | advparams = [] | ||
for key, value in advsizes: | ||||
|
r21015 | advparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21608 | self._initparams(manparams, advparams) | ||
|
r21014 | ## part payload | ||
|
r24034 | self._payloadstream = util.chunkbuffer(self._payloadchunks()) | ||
|
r21019 | # we read the data, tell it | ||
self._initialized = True | ||||
def read(self, size=None): | ||||
"""read payload data""" | ||||
if not self._initialized: | ||||
self._readheader() | ||||
if size is None: | ||||
data = self._payloadstream.read() | ||||
else: | ||||
data = self._payloadstream.read(size) | ||||
|
r25334 | self._pos += len(data) | ||
|
r21019 | if size is None or len(data) < size: | ||
|
r25334 | if not self.consumed and self._pos: | ||
self.ui.debug('bundle2-input-part: total payload size %i\n' | ||||
% self._pos) | ||||
|
r21019 | self.consumed = True | ||
return data | ||||
|
r24036 | def tell(self): | ||
return self._pos | ||||
|
r24037 | def seek(self, offset, whence=0): | ||
if whence == 0: | ||||
newpos = offset | ||||
elif whence == 1: | ||||
newpos = self._pos + offset | ||||
elif whence == 2: | ||||
if not self.consumed: | ||||
self.read() | ||||
newpos = self._chunkindex[-1][0] - offset | ||||
else: | ||||
raise ValueError('Unknown whence value: %r' % (whence,)) | ||||
if newpos > self._chunkindex[-1][0] and not self.consumed: | ||||
self.read() | ||||
if not 0 <= newpos <= self._chunkindex[-1][0]: | ||||
raise ValueError('Offset out of range') | ||||
if self._pos != newpos: | ||||
chunk, internaloffset = self._findchunk(newpos) | ||||
self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk)) | ||||
adjust = self.read(internaloffset) | ||||
if len(adjust) != internaloffset: | ||||
|
r26587 | raise error.Abort(_('Seek failed\n')) | ||
|
r24037 | self._pos = newpos | ||
|
r31889 | def _seekfp(self, offset, whence=0): | ||
"""move the underlying file pointer | ||||
This method is meant for internal usage by the bundle2 protocol only. | ||||
They directly manipulate the low level stream including bundle2 level | ||||
instruction. | ||||
Do not use it to implement higher-level logic or methods.""" | ||||
if self._seekable: | ||||
return self._fp.seek(offset, whence) | ||||
else: | ||||
raise NotImplementedError(_('File pointer is not seekable')) | ||||
def _tellfp(self): | ||||
"""return the file offset, or None if file is not seekable | ||||
This method is meant for internal usage by the bundle2 protocol only. | ||||
They directly manipulate the low level stream including bundle2 level | ||||
instruction. | ||||
Do not use it to implement higher-level logic or methods.""" | ||||
if self._seekable: | ||||
try: | ||||
return self._fp.tell() | ||||
except IOError as e: | ||||
if e.errno == errno.ESPIPE: | ||||
self._seekable = False | ||||
else: | ||||
raise | ||||
return None | ||||
|
r25317 | # These are only the static capabilities. | ||
# Check the 'getrepocaps' function for the rest. | ||||
|
r24686 | capabilities = {'HG20': (), | ||
|
r25493 | 'error': ('abort', 'unsupportedcontent', 'pushraced', | ||
'pushkey'), | ||||
|
r24686 | 'listkeys': (), | ||
'pushkey': (), | ||||
|
r23029 | 'digests': tuple(sorted(util.DIGESTS.keys())), | ||
|
r24686 | 'remote-changegroup': ('http', 'https'), | ||
|
r25401 | 'hgtagsfnodes': (), | ||
|
r34323 | 'phases': ('heads',), | ||
|
r22341 | } | ||
|
r23439 | def getrepocaps(repo, allowpushback=False): | ||
|
r22342 | """return the bundle2 capabilities for a given repo | ||
|
r22343 | Exists to allow extensions (like evolution) to mutate the capabilities. | ||
|
r22342 | """ | ||
|
r22343 | caps = capabilities.copy() | ||
|
r27953 | caps['changegroup'] = tuple(sorted( | ||
changegroup.supportedincomingversions(repo))) | ||||
|
r22953 | if obsolete.isenabled(repo, obsolete.exchangeopt): | ||
|
r22343 | supportedformat = tuple('V%i' % v for v in obsolete.formats) | ||
|
r24686 | caps['obsmarkers'] = supportedformat | ||
|
r23439 | if allowpushback: | ||
|
r24686 | caps['pushback'] = () | ||
r33219 | cpmode = repo.ui.config('server', 'concurrent-push-mode') | |||
r32892 | if cpmode == 'check-related': | |||
r32709 | caps['checkheads'] = ('related',) | |||
|
r34323 | if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'): | ||
caps.pop('phases') | ||||
|
r22343 | return caps | ||
|
r22342 | |||
|
r21644 | def bundle2caps(remote): | ||
|
r23139 | """return the bundle capabilities of a peer as dict""" | ||
|
r24686 | raw = remote.capable('bundle2') | ||
|
r21644 | if not raw and raw != '': | ||
return {} | ||||
|
r28883 | capsblob = urlreq.unquote(remote.capable('bundle2')) | ||
|
r21644 | return decodecaps(capsblob) | ||
|
r21014 | |||
|
r22344 | def obsmarkersversion(caps): | ||
"""extract the list of supported obsmarkers versions from a bundle2caps dict | ||||
""" | ||||
|
r24686 | obscaps = caps.get('obsmarkers', ()) | ||
|
r22344 | return [int(c[1:]) for c in obscaps if c.startswith('V')] | ||
r32216 | def writenewbundle(ui, repo, source, filename, bundletype, outgoing, opts, | |||
vfs=None, compression=None, compopts=None): | ||||
if bundletype.startswith('HG10'): | ||||
|
r34103 | cg = changegroup.makechangegroup(repo, outgoing, '01', source) | ||
r32216 | return writebundle(ui, cg, filename, bundletype, vfs=vfs, | |||
compression=compression, compopts=compopts) | ||||
elif not bundletype.startswith('HG20'): | ||||
raise error.ProgrammingError('unknown bundle type: %s' % bundletype) | ||||
r32516 | caps = {} | |||
if 'obsolescence' in opts: | ||||
caps['obsmarkers'] = ('V1',) | ||||
bundle = bundle20(ui, caps) | ||||
r32216 | bundle.setcompression(compression, compopts) | |||
_addpartsfromopts(ui, repo, bundle, source, outgoing, opts) | ||||
chunkiter = bundle.getchunks() | ||||
return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs) | ||||
def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts): | ||||
# We should eventually reconcile this logic with the one behind | ||||
# 'exchange.getbundle2partsgenerator'. | ||||
# | ||||
# The type of input from 'getbundle' and 'writenewbundle' are a bit | ||||
# different right now. So we keep them separated for now for the sake of | ||||
# simplicity. | ||||
# we always want a changegroup in such bundle | ||||
cgversion = opts.get('cg.version') | ||||
if cgversion is None: | ||||
cgversion = changegroup.safeversion(repo) | ||||
|
r34103 | cg = changegroup.makechangegroup(repo, outgoing, cgversion, source) | ||
r32216 | part = bundler.newpart('changegroup', data=cg.getchunks()) | |||
part.addparam('version', cg.version) | ||||
if 'clcount' in cg.extras: | ||||
|
r34218 | part.addparam('nbchanges', '%d' % cg.extras['clcount'], | ||
r32216 | mandatory=False) | |||
|
r33408 | if opts.get('phases') and repo.revs('%ln and secret()', | ||
outgoing.missingheads): | ||||
part.addparam('targetphase', '%d' % phases.secret, mandatory=False) | ||||
r32216 | ||||
r32218 | addparttagsfnodescache(repo, bundler, outgoing) | |||
r32516 | if opts.get('obsolescence', False): | |||
obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing) | ||||
buildobsmarkerspart(bundler, obsmarkers) | ||||
|
r33031 | if opts.get('phases', False): | ||
headsbyphase = phases.subsetphaseheads(repo, outgoing.missing) | ||||
|
r34320 | phasedata = phases.binaryencode(headsbyphase) | ||
bundler.newpart('phase-heads', data=phasedata) | ||||
|
r33031 | |||
r32217 | def addparttagsfnodescache(repo, bundler, outgoing): | |||
# we include the tags fnode cache for the bundle changeset | ||||
# (as an optional parts) | ||||
cache = tags.hgtagsfnodescache(repo.unfiltered()) | ||||
chunks = [] | ||||
# .hgtags fnodes are only relevant for head changesets. While we could | ||||
# transfer values for all known nodes, there will likely be little to | ||||
# no benefit. | ||||
# | ||||
# We don't bother using a generator to produce output data because | ||||
# a) we only have 40 bytes per head and even esoteric numbers of heads | ||||
# consume little memory (1M heads is 40MB) b) we don't want to send the | ||||
# part if we don't have entries and knowing if we have entries requires | ||||
# cache lookups. | ||||
for node in outgoing.missingheads: | ||||
# Don't compute missing, as this may slow down serving. | ||||
fnode = cache.getfnode(node, computemissing=False) | ||||
if fnode is not None: | ||||
chunks.extend([node, fnode]) | ||||
if chunks: | ||||
bundler.newpart('hgtagsfnodes', data=''.join(chunks)) | ||||
r32515 | def buildobsmarkerspart(bundler, markers): | |||
"""add an obsmarker part to the bundler with <markers> | ||||
No part is created if markers is empty. | ||||
Raises ValueError if the bundler doesn't support any known obsmarker format. | ||||
""" | ||||
if not markers: | ||||
return None | ||||
remoteversions = obsmarkersversion(bundler.capabilities) | ||||
version = obsolete.commonversion(remoteversions) | ||||
if version is None: | ||||
raise ValueError('bundler does not support common obsmarker format') | ||||
stream = obsolete.encodemarkers(markers, True, version=version) | ||||
return bundler.newpart('obsmarkers', data=stream) | ||||
|
r30757 | def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None, | ||
compopts=None): | ||||
|
r28666 | """Write a bundle file and return its filename. | ||
Existing files will not be overwritten. | ||||
If no filename is specified, a temporary file is created. | ||||
bz2 compression can be turned off. | ||||
The bundle file will be deleted in case of errors. | ||||
""" | ||||
if bundletype == "HG20": | ||||
bundle = bundle20(ui) | ||||
|
r30757 | bundle.setcompression(compression, compopts) | ||
|
r28666 | part = bundle.newpart('changegroup', data=cg.getchunks()) | ||
part.addparam('version', cg.version) | ||||
|
r29593 | if 'clcount' in cg.extras: | ||
|
r34218 | part.addparam('nbchanges', '%d' % cg.extras['clcount'], | ||
|
r29593 | mandatory=False) | ||
|
r28666 | chunkiter = bundle.getchunks() | ||
else: | ||||
# compression argument is only for the bundle2 case | ||||
assert compression is None | ||||
if cg.version != '01': | ||||
raise error.Abort(_('old bundle types only supports v1 ' | ||||
'changegroups')) | ||||
header, comp = bundletypes[bundletype] | ||||
|
r30351 | if comp not in util.compengines.supportedbundletypes: | ||
|
r28666 | raise error.Abort(_('unknown stream compression type: %s') | ||
% comp) | ||||
|
r30351 | compengine = util.compengines.forbundletype(comp) | ||
|
r28666 | def chunkiter(): | ||
yield header | ||||
|
r30757 | for chunk in compengine.compressstream(cg.getchunks(), compopts): | ||
|
r30357 | yield chunk | ||
|
r28666 | chunkiter = chunkiter() | ||
# parse the changegroup data, otherwise we will block | ||||
# in case of sshrepo because we don't know the end of the stream | ||||
return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs) | ||||
|
r33037 | def combinechangegroupresults(op): | ||
|
r33036 | """logic to combine 0 or more addchangegroup results into one""" | ||
|
r33037 | results = [r.get('return', 0) | ||
for r in op.records['changegroup']] | ||||
|
r33036 | changedheads = 0 | ||
result = 1 | ||||
for ret in results: | ||||
# If any changegroup result is 0, return 0 | ||||
if ret == 0: | ||||
result = 0 | ||||
break | ||||
if ret < -1: | ||||
changedheads += ret + 1 | ||||
elif ret > 1: | ||||
changedheads += ret - 1 | ||||
if changedheads > 0: | ||||
result = 1 + changedheads | ||||
elif changedheads < 0: | ||||
result = -1 + changedheads | ||||
return result | ||||
|
r33407 | @parthandler('changegroup', ('version', 'nbchanges', 'treemanifest', | ||
'targetphase')) | ||||
|
r20998 | def handlechangegroup(op, inpart): | ||
|
r20950 | """apply a changegroup part on the repo | ||
This is a very early implementation that will massive rework before being | ||||
inflicted to any end-user. | ||||
""" | ||||
|
r32930 | tr = op.gettransaction() | ||
|
r23170 | unpackerversion = inpart.params.get('version', '01') | ||
# We should raise an appropriate exception here | ||||
|
r27751 | cg = changegroup.getunbundler(unpackerversion, inpart, None) | ||
|
r23001 | # the source and url passed here are overwritten by the one contained in | ||
# the transaction.hookargs argument. So 'bundle2' is a placeholder | ||||
|
r25518 | nbchangesets = None | ||
if 'nbchanges' in inpart.params: | ||||
nbchangesets = int(inpart.params.get('nbchanges')) | ||||
|
r27734 | if ('treemanifest' in inpart.params and | ||
'treemanifest' not in op.repo.requirements): | ||||
if len(op.repo.changelog) != 0: | ||||
raise error.Abort(_( | ||||
"bundle contains tree manifests, but local repo is " | ||||
"non-empty and does not use tree manifests")) | ||||
op.repo.requirements.add('treemanifest') | ||||
op.repo._applyopenerreqs() | ||||
op.repo._writerequirements() | ||||
|
r33407 | extrakwargs = {} | ||
targetphase = inpart.params.get('targetphase') | ||||
if targetphase is not None: | ||||
extrakwargs['targetphase'] = int(targetphase) | ||||
|
r33038 | ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2', | ||
|
r33407 | expectedtotal=nbchangesets, **extrakwargs) | ||
|
r20998 | if op.reply is not None: | ||
|
r23139 | # This is definitely not the final form of this | ||
|
r20998 | # return. But one need to start somewhere. | ||
|
r24686 | part = op.reply.newpart('reply:changegroup', mandatory=False) | ||
|
r33638 | part.addparam( | ||
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False) | ||||
|
r21606 | part.addparam('return', '%i' % ret, mandatory=False) | ||
|
r21019 | assert not inpart.read() | ||
|
r20950 | |||
|
r23029 | _remotechangegroupparams = tuple(['url', 'size', 'digests'] + | ||
['digest:%s' % k for k in util.DIGESTS.keys()]) | ||||
|
r24686 | @parthandler('remote-changegroup', _remotechangegroupparams) | ||
|
r23029 | def handleremotechangegroup(op, inpart): | ||
"""apply a bundle10 on the repo, given an url and validation information | ||||
All the information about the remote bundle to import are given as | ||||
parameters. The parameters include: | ||||
- url: the url to the bundle10. | ||||
- size: the bundle10 file size. It is used to validate what was | ||||
retrieved by the client matches the server knowledge about the bundle. | ||||
- digests: a space separated list of the digest types provided as | ||||
parameters. | ||||
- digest:<digest-type>: the hexadecimal representation of the digest with | ||||
that name. Like the size, it is used to validate what was retrieved by | ||||
the client matches what the server knows about the bundle. | ||||
When multiple digest types are given, all of them are checked. | ||||
""" | ||||
try: | ||||
raw_url = inpart.params['url'] | ||||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url') | ||
|
r23029 | parsed_url = util.url(raw_url) | ||
|
r24686 | if parsed_url.scheme not in capabilities['remote-changegroup']: | ||
|
r26587 | raise error.Abort(_('remote-changegroup does not support %s urls') % | ||
|
r23029 | parsed_url.scheme) | ||
try: | ||||
size = int(inpart.params['size']) | ||||
except ValueError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: invalid value for param "%s"') | ||
|
r23029 | % 'size') | ||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size') | ||
|
r23029 | |||
digests = {} | ||||
for typ in inpart.params.get('digests', '').split(): | ||||
param = 'digest:%s' % typ | ||||
try: | ||||
value = inpart.params[param] | ||||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % | ||
|
r23029 | param) | ||
digests[typ] = value | ||||
real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests) | ||||
|
r32930 | tr = op.gettransaction() | ||
|
r25919 | from . import exchange | ||
|
r23029 | cg = exchange.readbundle(op.repo.ui, real_part, raw_url) | ||
if not isinstance(cg, changegroup.cg1unpacker): | ||||
|
r26587 | raise error.Abort(_('%s: not a bundle version 1.0') % | ||
|
r23029 | util.hidepassword(raw_url)) | ||
|
r33038 | ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2') | ||
|
r23029 | if op.reply is not None: | ||
|
r23139 | # This is definitely not the final form of this | ||
|
r23029 | # return. But one need to start somewhere. | ||
|
r24686 | part = op.reply.newpart('reply:changegroup') | ||
|
r33638 | part.addparam( | ||
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False) | ||||
|
r23029 | part.addparam('return', '%i' % ret, mandatory=False) | ||
try: | ||||
real_part.validate() | ||||
|
r26587 | except error.Abort as e: | ||
raise error.Abort(_('bundle at %s is corrupted:\n%s') % | ||||
|
r23029 | (util.hidepassword(raw_url), str(e))) | ||
assert not inpart.read() | ||||
|
r24686 | @parthandler('reply:changegroup', ('return', 'in-reply-to')) | ||
|
r22548 | def handlereplychangegroup(op, inpart): | ||
|
r21611 | ret = int(inpart.params['return']) | ||
replyto = int(inpart.params['in-reply-to']) | ||||
op.records.add('changegroup', {'return': ret}, replyto) | ||||
|
r20950 | |||
|
r24686 | @parthandler('check:heads') | ||
|
r22548 | def handlecheckheads(op, inpart): | ||
|
r21060 | """check that head of the repo did not change | ||
This is used to detect a push race when using unbundle. | ||||
This replaces the "heads" argument of unbundle.""" | ||||
h = inpart.read(20) | ||||
heads = [] | ||||
while len(h) == 20: | ||||
heads.append(h) | ||||
h = inpart.read(20) | ||||
assert not h | ||||
|
r26565 | # Trigger a transaction so that we are guaranteed to have the lock now. | ||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r29294 | if sorted(heads) != sorted(op.repo.heads()): | ||
|
r21185 | raise error.PushRaced('repository changed while pushing - ' | ||
'please try again') | ||||
|
r21130 | |||
r32709 | @parthandler('check:updated-heads') | |||
def handlecheckupdatedheads(op, inpart): | ||||
"""check for race on the heads touched by a push | ||||
This is similar to 'check:heads' but focus on the heads actually updated | ||||
during the push. If other activities happen on unrelated heads, it is | ||||
ignored. | ||||
This allow server with high traffic to avoid push contention as long as | ||||
unrelated parts of the graph are involved.""" | ||||
h = inpart.read(20) | ||||
heads = [] | ||||
while len(h) == 20: | ||||
heads.append(h) | ||||
h = inpart.read(20) | ||||
assert not h | ||||
# trigger a transaction so that we are guaranteed to have the lock now. | ||||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
currentheads = set() | ||||
for ls in op.repo.branchmap().itervalues(): | ||||
currentheads.update(ls) | ||||
for h in heads: | ||||
if h not in currentheads: | ||||
raise error.PushRaced('repository changed while pushing - ' | ||||
'please try again') | ||||
|
r24686 | @parthandler('output') | ||
|
r21131 | def handleoutput(op, inpart): | ||
"""forward output captured on the server to the client""" | ||||
for line in inpart.read().splitlines(): | ||||
|
r29851 | op.ui.status(_('remote: %s\n') % line) | ||
|
r21131 | |||
|
r24686 | @parthandler('replycaps') | ||
|
r21130 | def handlereplycaps(op, inpart): | ||
"""Notify that a reply bundle should be created | ||||
|
r21138 | The payload contains the capabilities information for the reply""" | ||
caps = decodecaps(inpart.read()) | ||||
|
r21130 | if op.reply is None: | ||
|
r21135 | op.reply = bundle20(op.ui, caps) | ||
|
r21131 | |||
|
r26829 | class AbortFromPart(error.Abort): | ||
"""Sub-class of Abort that denotes an error from a bundle2 part.""" | ||||
|
r24686 | @parthandler('error:abort', ('message', 'hint')) | ||
|
r24741 | def handleerrorabort(op, inpart): | ||
|
r21177 | """Used to transmit abort error over the wire""" | ||
|
r26829 | raise AbortFromPart(inpart.params['message'], | ||
hint=inpart.params.get('hint')) | ||||
|
r21183 | |||
|
r25493 | @parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret', | ||
'in-reply-to')) | ||||
def handleerrorpushkey(op, inpart): | ||||
"""Used to transmit failure of a mandatory pushkey over the wire""" | ||||
kwargs = {} | ||||
for name in ('namespace', 'key', 'new', 'old', 'ret'): | ||||
value = inpart.params.get(name) | ||||
if value is not None: | ||||
kwargs[name] = value | ||||
raise error.PushkeyFailed(inpart.params['in-reply-to'], **kwargs) | ||||
|
r24686 | @parthandler('error:unsupportedcontent', ('parttype', 'params')) | ||
|
r24741 | def handleerrorunsupportedcontent(op, inpart): | ||
|
r21619 | """Used to transmit unknown content error over the wire""" | ||
|
r21622 | kwargs = {} | ||
|
r21627 | parttype = inpart.params.get('parttype') | ||
if parttype is not None: | ||||
kwargs['parttype'] = parttype | ||||
|
r21622 | params = inpart.params.get('params') | ||
if params is not None: | ||||
kwargs['params'] = params.split('\0') | ||||
|
r26393 | raise error.BundleUnknownFeatureError(**kwargs) | ||
|
r21186 | |||
|
r24686 | @parthandler('error:pushraced', ('message',)) | ||
|
r24741 | def handleerrorpushraced(op, inpart): | ||
|
r21186 | """Used to transmit push race error over the wire""" | ||
|
r21611 | raise error.ResponseError(_('push failed:'), inpart.params['message']) | ||
|
r21655 | |||
|
r24686 | @parthandler('listkeys', ('namespace',)) | ||
|
r21655 | def handlelistkeys(op, inpart): | ||
"""retrieve pushkey namespace content stored in a bundle2""" | ||||
namespace = inpart.params['namespace'] | ||||
r = pushkey.decodekeys(inpart.read()) | ||||
op.records.add('listkeys', (namespace, r)) | ||||
|
r21660 | |||
|
r24686 | @parthandler('pushkey', ('namespace', 'key', 'old', 'new')) | ||
|
r21660 | def handlepushkey(op, inpart): | ||
"""process a pushkey request""" | ||||
dec = pushkey.decode | ||||
namespace = dec(inpart.params['namespace']) | ||||
key = dec(inpart.params['key']) | ||||
old = dec(inpart.params['old']) | ||||
new = dec(inpart.params['new']) | ||||
|
r26565 | # Grab the transaction to ensure that we have the lock before performing the | ||
# pushkey. | ||||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r21660 | ret = op.repo.pushkey(namespace, key, old, new) | ||
record = {'namespace': namespace, | ||||
'key': key, | ||||
'old': old, | ||||
'new': new} | ||||
op.records.add('pushkey', record) | ||||
if op.reply is not None: | ||||
|
r24686 | rpart = op.reply.newpart('reply:pushkey') | ||
|
r33638 | rpart.addparam( | ||
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False) | ||||
|
r21660 | rpart.addparam('return', '%i' % ret, mandatory=False) | ||
|
r25481 | if inpart.mandatory and not ret: | ||
|
r25484 | kwargs = {} | ||
for key in ('namespace', 'key', 'new', 'old', 'ret'): | ||||
if key in inpart.params: | ||||
kwargs[key] = inpart.params[key] | ||||
raise error.PushkeyFailed(partid=str(inpart.id), **kwargs) | ||||
|
r21660 | |||
|
r33031 | @parthandler('phase-heads') | ||
def handlephases(op, inpart): | ||||
"""apply phases from bundle part to repo""" | ||||
|
r34321 | headsbyphase = phases.binarydecode(inpart) | ||
|
r34322 | phases.updatephases(op.repo.unfiltered(), op.gettransaction, headsbyphase) | ||
|
r33031 | |||
|
r24686 | @parthandler('reply:pushkey', ('return', 'in-reply-to')) | ||
|
r21660 | def handlepushkeyreply(op, inpart): | ||
"""retrieve the result of a pushkey request""" | ||||
ret = int(inpart.params['return']) | ||||
partid = int(inpart.params['in-reply-to']) | ||||
op.records.add('pushkey', {'return': ret}, partid) | ||||
|
r22336 | |||
|
r24686 | @parthandler('obsmarkers') | ||
|
r22336 | def handleobsmarker(op, inpart): | ||
"""add a stream of obsmarkers to the repo""" | ||||
tr = op.gettransaction() | ||||
|
r24733 | markerdata = inpart.read() | ||
|
r33499 | if op.ui.config('experimental', 'obsmarkers-exchange-debug'): | ||
|
r24733 | op.ui.write(('obsmarker-exchange: %i bytes received\n') | ||
% len(markerdata)) | ||||
|
r26685 | # The mergemarkers call will crash if marker creation is not enabled. | ||
# we want to avoid this if the part is advisory. | ||||
if not inpart.mandatory and op.repo.obsstore.readonly: | ||||
op.repo.ui.debug('ignoring obsolescence markers, feature not enabled') | ||||
return | ||||
|
r24733 | new = op.repo.obsstore.mergemarkers(tr, markerdata) | ||
r32314 | op.repo.invalidatevolatilesets() | |||
|
r22337 | if new: | ||
op.repo.ui.status(_('%i new obsolescence markers\n') % new) | ||||
|
r22338 | op.records.add('obsmarkers', {'new': new}) | ||
|
r22340 | if op.reply is not None: | ||
|
r24686 | rpart = op.reply.newpart('reply:obsmarkers') | ||
|
r33638 | rpart.addparam( | ||
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False) | ||||
|
r22340 | rpart.addparam('new', '%i' % new, mandatory=False) | ||
|
r24686 | @parthandler('reply:obsmarkers', ('new', 'in-reply-to')) | ||
|
r25506 | def handleobsmarkerreply(op, inpart): | ||
|
r22340 | """retrieve the result of a pushkey request""" | ||
ret = int(inpart.params['new']) | ||||
partid = int(inpart.params['in-reply-to']) | ||||
op.records.add('obsmarkers', {'new': ret}, partid) | ||||
|
r25401 | |||
@parthandler('hgtagsfnodes') | ||||
def handlehgtagsfnodes(op, inpart): | ||||
"""Applies .hgtags fnodes cache entries to the local repo. | ||||
Payload is pairs of 20 byte changeset nodes and filenodes. | ||||
""" | ||||
|
r26565 | # Grab the transaction so we ensure that we have the lock at this point. | ||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r25401 | cache = tags.hgtagsfnodescache(op.repo.unfiltered()) | ||
count = 0 | ||||
while True: | ||||
node = inpart.read(20) | ||||
fnode = inpart.read(20) | ||||
if len(node) < 20 or len(fnode) < 20: | ||||
|
r25641 | op.ui.debug('ignoring incomplete received .hgtags fnodes data\n') | ||
|
r25401 | break | ||
cache.setfnode(node, fnode) | ||||
count += 1 | ||||
cache.write() | ||||
op.ui.debug('applied %i hgtags fnodes cache entries\n' % count) | ||||
|
r33656 | |||
@parthandler('pushvars') | ||||
def bundle2getvars(op, part): | ||||
'''unbundle a bundle2 containing shellvars on the server''' | ||||
# An option to disable unbundling on server-side for security reasons | ||||
|
r33834 | if op.ui.configbool('push', 'pushvars.server'): | ||
|
r33656 | hookargs = {} | ||
for key, value in part.advisoryparams: | ||||
key = key.upper() | ||||
# We want pushed variables to have USERVAR_ prepended so we know | ||||
# they came from the --pushvar flag. | ||||
key = "USERVAR_" + key | ||||
hookargs[key] = value | ||||
op.addhookargs(hookargs) | ||||