bundle2.py
1527 lines
| 54.7 KiB
| text/x-python
|
PythonLexer
/ mercurial / bundle2.py
|
r20801 | # bundle2.py - generic container format to transmit arbitrary data. | ||
# | ||||
# Copyright 2013 Facebook, Inc. | ||||
# | ||||
# This software may be used and distributed according to the terms of the | ||||
# GNU General Public License version 2 or any later version. | ||||
"""Handling of the new bundle2 format | ||||
The goal of bundle2 is to act as an atomically packet to transmit a set of | ||||
payloads in an application agnostic way. It consist in a sequence of "parts" | ||||
that will be handed to and processed by the application layer. | ||||
General format architecture | ||||
=========================== | ||||
The format is architectured as follow | ||||
- magic string | ||||
- stream level parameters | ||||
- payload parts (any number) | ||||
- end of stream marker. | ||||
|
r20856 | the Binary format | ||
|
r20801 | ============================ | ||
|
r21024 | All numbers are unsigned and big-endian. | ||
|
r20801 | |||
stream level parameters | ||||
------------------------ | ||||
Binary format is as follow | ||||
|
r23009 | :params size: int32 | ||
|
r20801 | |||
The total number of Bytes used by the parameters | ||||
:params value: arbitrary number of Bytes | ||||
A blob of `params size` containing the serialized version of all stream level | ||||
parameters. | ||||
|
r21024 | The blob contains a space separated list of parameters. Parameters with value | ||
|
r20811 | are stored in the form `<name>=<value>`. Both name and value are urlquoted. | ||
|
r20804 | |||
|
r20813 | Empty name are obviously forbidden. | ||
|
r20844 | Name MUST start with a letter. If this first letter is lower case, the | ||
|
r21024 | parameter is advisory and can be safely ignored. However when the first | ||
|
r20844 | letter is capital, the parameter is mandatory and the bundling process MUST | ||
stop if he is not able to proceed it. | ||||
|
r20814 | |||
|
r20808 | Stream parameters use a simple textual format for two main reasons: | ||
|
r20804 | |||
|
r21024 | - Stream level parameters should remain simple and we want to discourage any | ||
|
r20808 | crazy usage. | ||
|
r21024 | - Textual data allow easy human inspection of a bundle2 header in case of | ||
|
r20808 | troubles. | ||
Any Applicative level options MUST go into a bundle2 part instead. | ||||
|
r20801 | |||
Payload part | ||||
------------------------ | ||||
Binary format is as follow | ||||
|
r23009 | :header size: int32 | ||
|
r20801 | |||
|
r25507 | The total number of Bytes used by the part header. When the header is empty | ||
|
r20801 | (size = 0) this is interpreted as the end of stream marker. | ||
|
r20856 | :header: | ||
The header defines how to interpret the part. It contains two piece of | ||||
data: the part type, and the part parameters. | ||||
The part type is used to route an application level handler, that can | ||||
interpret payload. | ||||
Part parameters are passed to the application level handler. They are | ||||
meant to convey information that will help the application level object to | ||||
interpret the part payload. | ||||
The binary format of the header is has follow | ||||
:typesize: (one byte) | ||||
|
r20877 | |||
|
r23916 | :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*) | ||
|
r20877 | |||
|
r20995 | :partid: A 32bits integer (unique in the bundle) that can be used to refer | ||
to this part. | ||||
|
r20877 | :parameters: | ||
|
r21024 | Part's parameter may have arbitrary content, the binary structure is:: | ||
|
r20877 | |||
<mandatory-count><advisory-count><param-sizes><param-data> | ||||
:mandatory-count: 1 byte, number of mandatory parameters | ||||
:advisory-count: 1 byte, number of advisory parameters | ||||
:param-sizes: | ||||
N couple of bytes, where N is the total number of parameters. Each | ||||
couple contains (<size-of-key>, <size-of-value) for one parameter. | ||||
:param-data: | ||||
A blob of bytes from which each parameter key and value can be | ||||
retrieved using the list of size couples stored in the previous | ||||
field. | ||||
Mandatory parameters comes first, then the advisory ones. | ||||
|
r20856 | |||
|
r21607 | Each parameter's key MUST be unique within the part. | ||
|
r20856 | :payload: | ||
|
r20876 | payload is a series of `<chunksize><chunkdata>`. | ||
|
r23009 | `chunksize` is an int32, `chunkdata` are plain bytes (as much as | ||
|
r20876 | `chunksize` says)` The payload part is concluded by a zero size chunk. | ||
The current implementation always produces either zero or one chunk. | ||||
|
r21024 | This is an implementation limitation that will ultimately be lifted. | ||
|
r20891 | |||
|
r23009 | `chunksize` can be negative to trigger special case processing. No such | ||
processing is in place yet. | ||||
|
r20891 | Bundle processing | ||
============================ | ||||
Each part is processed in order using a "part handler". Handler are registered | ||||
for a certain part type. | ||||
The matching of a part to its handler is case insensitive. The case of the | ||||
part type is used to know if a part is mandatory or advisory. If the Part type | ||||
contains any uppercase char it is considered mandatory. When no handler is | ||||
known for a Mandatory part, the process is aborted and an exception is raised. | ||||
|
r20892 | If the part is advisory and no handler is known, the part is ignored. When the | ||
process is aborted, the full bundle is still read from the stream to keep the | ||||
channel usable. But none of the part read from an abort are processed. In the | ||||
future, dropping the stream may become an option for channel we do not care to | ||||
preserve. | ||||
|
r20801 | """ | ||
|
r25919 | from __future__ import absolute_import | ||
|
r24026 | import errno | ||
|
r25919 | import re | ||
|
r20814 | import string | ||
|
r25919 | import struct | ||
import sys | ||||
import urllib | ||||
|
r20804 | |||
|
r25919 | from .i18n import _ | ||
from . import ( | ||||
changegroup, | ||||
error, | ||||
obsolete, | ||||
pushkey, | ||||
tags, | ||||
url, | ||||
util, | ||||
) | ||||
|
r20802 | |||
|
r20804 | _pack = struct.pack | ||
_unpack = struct.unpack | ||||
|
r23009 | _fstreamparamsize = '>i' | ||
_fpartheadersize = '>i' | ||||
|
r20856 | _fparttypesize = '>B' | ||
|
r20995 | _fpartid = '>I' | ||
|
r23009 | _fpayloadsize = '>i' | ||
|
r20877 | _fpartparamcount = '>BB' | ||
|
r21001 | preferedchunksize = 4096 | ||
|
r23868 | _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]') | ||
|
r25313 | def outdebug(ui, message): | ||
"""debug regarding output stream (bundling)""" | ||||
|
r25336 | if ui.configbool('devel', 'bundle2.debug', False): | ||
ui.debug('bundle2-output: %s\n' % message) | ||||
|
r25313 | |||
|
r25318 | def indebug(ui, message): | ||
"""debug on input stream (unbundling)""" | ||||
|
r25336 | if ui.configbool('devel', 'bundle2.debug', False): | ||
ui.debug('bundle2-input: %s\n' % message) | ||||
|
r25318 | |||
|
r23868 | def validateparttype(parttype): | ||
"""raise ValueError if a parttype contains invalid character""" | ||||
|
r23916 | if _parttypeforbidden.search(parttype): | ||
|
r23868 | raise ValueError(parttype) | ||
|
r20877 | def _makefpartparamsizes(nbparams): | ||
"""return a struct format to read part parameter sizes | ||||
The number parameters is variable so we need to build that format | ||||
dynamically. | ||||
""" | ||||
return '>'+('BB'*nbparams) | ||||
|
r20804 | |||
|
r20890 | parthandlermapping = {} | ||
|
r20889 | |||
|
r21623 | def parthandler(parttype, params=()): | ||
|
r20890 | """decorator that register a function as a bundle2 part handler | ||
eg:: | ||||
|
r21624 | @parthandler('myparttype', ('mandatory', 'param', 'handled')) | ||
|
r20890 | def myparttypehandler(...): | ||
'''process a part of type "my part".''' | ||||
... | ||||
""" | ||||
|
r23868 | validateparttype(parttype) | ||
|
r20890 | def _decorator(func): | ||
|
r20891 | lparttype = parttype.lower() # enforce lower case matching. | ||
assert lparttype not in parthandlermapping | ||||
parthandlermapping[lparttype] = func | ||||
|
r21623 | func.params = frozenset(params) | ||
|
r20890 | return func | ||
return _decorator | ||||
|
r20889 | |||
|
r20949 | class unbundlerecords(object): | ||
"""keep record of what happens during and unbundle | ||||
New records are added using `records.add('cat', obj)`. Where 'cat' is a | ||||
|
r21024 | category of record and obj is an arbitrary object. | ||
|
r20949 | |||
`records['cat']` will return all entries of this category 'cat'. | ||||
Iterating on the object itself will yield `('category', obj)` tuples | ||||
for all entries. | ||||
All iterations happens in chronological order. | ||||
""" | ||||
def __init__(self): | ||||
self._categories = {} | ||||
self._sequences = [] | ||||
|
r20996 | self._replies = {} | ||
|
r20949 | |||
|
r20996 | def add(self, category, entry, inreplyto=None): | ||
|
r20949 | """add a new record of a given category. | ||
The entry can then be retrieved in the list returned by | ||||
self['category'].""" | ||||
self._categories.setdefault(category, []).append(entry) | ||||
self._sequences.append((category, entry)) | ||||
|
r20996 | if inreplyto is not None: | ||
self.getreplies(inreplyto).add(category, entry) | ||||
def getreplies(self, partid): | ||||
|
r23139 | """get the records that are replies to a specific part""" | ||
|
r20996 | return self._replies.setdefault(partid, unbundlerecords()) | ||
|
r20949 | |||
def __getitem__(self, cat): | ||||
return tuple(self._categories.get(cat, ())) | ||||
def __iter__(self): | ||||
return iter(self._sequences) | ||||
def __len__(self): | ||||
return len(self._sequences) | ||||
def __nonzero__(self): | ||||
return bool(self._sequences) | ||||
|
r20948 | class bundleoperation(object): | ||
"""an object that represents a single bundling process | ||||
Its purpose is to carry unbundle-related objects and states. | ||||
A new object should be created at the beginning of each bundle processing. | ||||
The object is to be returned by the processing function. | ||||
The object has very little content now it will ultimately contain: | ||||
* an access to the repo the bundle is applied to, | ||||
* a ui object, | ||||
* a way to retrieve a transaction to add changes to the repo, | ||||
* a way to record the result of processing each part, | ||||
* a way to construct a bundle response when applicable. | ||||
""" | ||||
|
r24878 | def __init__(self, repo, transactiongetter, captureoutput=True): | ||
|
r20948 | self.repo = repo | ||
self.ui = repo.ui | ||||
|
r20949 | self.records = unbundlerecords() | ||
|
r20952 | self.gettransaction = transactiongetter | ||
|
r20997 | self.reply = None | ||
|
r24878 | self.captureoutput = captureoutput | ||
|
r20948 | |||
|
r20952 | class TransactionUnavailable(RuntimeError): | ||
pass | ||||
def _notransaction(): | ||||
"""default method to get a transaction while processing a bundle | ||||
Raise an exception to highlight the fact that no transaction was expected | ||||
to be created""" | ||||
raise TransactionUnavailable() | ||||
|
r24851 | def processbundle(repo, unbundler, transactiongetter=None, op=None): | ||
|
r20889 | """This function process a bundle, apply effect to/from a repo | ||
|
r20947 | It iterates over each part then searches for and uses the proper handling | ||
code to process the part. Parts are processed in order. | ||||
|
r20889 | |||
This is very early version of this function that will be strongly reworked | ||||
before final usage. | ||||
|
r20891 | Unknown Mandatory part will abort the process. | ||
|
r24851 | |||
It is temporarily possible to provide a prebuilt bundleoperation to the | ||||
function. This is used to ensure output is properly propagated in case of | ||||
an error during the unbundling. This output capturing part will likely be | ||||
reworked and this ability will probably go away in the process. | ||||
|
r20889 | """ | ||
|
r24851 | if op is None: | ||
if transactiongetter is None: | ||||
transactiongetter = _notransaction | ||||
op = bundleoperation(repo, transactiongetter) | ||||
|
r20889 | # todo: | ||
# - replace this is a init function soon. | ||||
# - exception catching | ||||
unbundler.params | ||||
|
r25331 | if repo.ui.debugflag: | ||
msg = ['bundle2-input-bundle:'] | ||||
if unbundler.params: | ||||
msg.append(' %i params') | ||||
if op.gettransaction is None: | ||||
msg.append(' no-transaction') | ||||
else: | ||||
msg.append(' with-transaction') | ||||
msg.append('\n') | ||||
repo.ui.debug(''.join(msg)) | ||||
|
r25332 | iterparts = enumerate(unbundler.iterparts()) | ||
|
r21019 | part = None | ||
|
r25332 | nbpart = 0 | ||
|
r20892 | try: | ||
|
r25332 | for nbpart, part in iterparts: | ||
|
r23008 | _processpart(op, part) | ||
|
r25660 | except BaseException as exc: | ||
|
r25332 | for nbpart, part in iterparts: | ||
|
r21019 | # consume the bundle content | ||
|
r24047 | part.seek(0, 2) | ||
|
r21176 | # Small hack to let caller code distinguish exceptions from bundle2 | ||
|
r23139 | # processing from processing the old format. This is mostly | ||
|
r21176 | # needed to handle different return codes to unbundle according to the | ||
# type of bundle. We should probably clean up or drop this return code | ||||
# craziness in a future version. | ||||
exc.duringunbundle2 = True | ||||
|
r24795 | salvaged = [] | ||
|
r25492 | replycaps = None | ||
|
r24795 | if op.reply is not None: | ||
salvaged = op.reply.salvageoutput() | ||||
|
r25492 | replycaps = op.reply.capabilities | ||
exc._replycaps = replycaps | ||||
|
r24795 | exc._bundle2salvagedoutput = salvaged | ||
|
r20892 | raise | ||
|
r25332 | finally: | ||
repo.ui.debug('bundle2-input-bundle: %i parts total\n' % nbpart) | ||||
|
r20949 | return op | ||
|
r20889 | |||
|
r23008 | def _processpart(op, part): | ||
"""process a single part from a bundle | ||||
The part is guaranteed to have been fully consumed when the function exits | ||||
(even if an exception is raised).""" | ||||
|
r25333 | status = 'unknown' # used by debug output | ||
|
r23008 | try: | ||
try: | ||||
|
r23586 | handler = parthandlermapping.get(part.type) | ||
|
r23008 | if handler is None: | ||
|
r25333 | status = 'unsupported-type' | ||
|
r26393 | raise error.BundleUnknownFeatureError(parttype=part.type) | ||
|
r25320 | indebug(op.ui, 'found a handler for part %r' % part.type) | ||
|
r23008 | unknownparams = part.mandatorykeys - handler.params | ||
if unknownparams: | ||||
unknownparams = list(unknownparams) | ||||
unknownparams.sort() | ||||
|
r25333 | status = 'unsupported-params (%s)' % unknownparams | ||
|
r26393 | raise error.BundleUnknownFeatureError(parttype=part.type, | ||
params=unknownparams) | ||||
|
r25333 | status = 'supported' | ||
|
r26393 | except error.BundleUnknownFeatureError as exc: | ||
|
r23585 | if part.mandatory: # mandatory parts | ||
|
r23008 | raise | ||
|
r25320 | indebug(op.ui, 'ignoring unsupported advisory part %s' % exc) | ||
|
r23008 | return # skip to part processing | ||
|
r25333 | finally: | ||
if op.ui.debugflag: | ||||
msg = ['bundle2-input-part: "%s"' % part.type] | ||||
if not part.mandatory: | ||||
msg.append(' (advisory)') | ||||
nbmp = len(part.mandatorykeys) | ||||
nbap = len(part.params) - nbmp | ||||
if nbmp or nbap: | ||||
msg.append(' (params:') | ||||
if nbmp: | ||||
msg.append(' %i mandatory' % nbmp) | ||||
if nbap: | ||||
msg.append(' %i advisory' % nbmp) | ||||
msg.append(')') | ||||
msg.append(' %s\n' % status) | ||||
op.ui.debug(''.join(msg)) | ||||
|
r23008 | |||
# handler is called outside the above try block so that we don't | ||||
# risk catching KeyErrors from anything other than the | ||||
# parthandlermapping lookup (any KeyError raised by handler() | ||||
# itself represents a defect of a different variety). | ||||
output = None | ||||
|
r24878 | if op.captureoutput and op.reply is not None: | ||
|
r24849 | op.ui.pushbuffer(error=True, subproc=True) | ||
|
r23008 | output = '' | ||
try: | ||||
handler(op, part) | ||||
finally: | ||||
if output is not None: | ||||
output = op.ui.popbuffer() | ||||
|
r24743 | if output: | ||
outpart = op.reply.newpart('output', data=output, | ||||
mandatory=False) | ||||
outpart.addparam('in-reply-to', str(part.id), mandatory=False) | ||||
|
r23008 | finally: | ||
# consume the part content to not corrupt the stream. | ||||
|
r24047 | part.seek(0, 2) | ||
|
r23008 | |||
|
r21138 | def decodecaps(blob): | ||
|
r23139 | """decode a bundle2 caps bytes blob into a dictionary | ||
|
r21138 | |||
The blob is a list of capabilities (one per line) | ||||
Capabilities may have values using a line of the form:: | ||||
capability=value1,value2,value3 | ||||
The values are always a list.""" | ||||
caps = {} | ||||
for line in blob.splitlines(): | ||||
if not line: | ||||
continue | ||||
if '=' not in line: | ||||
key, vals = line, () | ||||
else: | ||||
key, vals = line.split('=', 1) | ||||
vals = vals.split(',') | ||||
key = urllib.unquote(key) | ||||
vals = [urllib.unquote(v) for v in vals] | ||||
caps[key] = vals | ||||
return caps | ||||
|
r21139 | def encodecaps(caps): | ||
"""encode a bundle2 caps dictionary into a bytes blob""" | ||||
chunks = [] | ||||
for ca in sorted(caps): | ||||
vals = caps[ca] | ||||
ca = urllib.quote(ca) | ||||
vals = [urllib.quote(v) for v in vals] | ||||
if vals: | ||||
ca = "%s=%s" % (ca, ','.join(vals)) | ||||
chunks.append(ca) | ||||
return '\n'.join(chunks) | ||||
|
r20801 | class bundle20(object): | ||
"""represent an outgoing bundle2 container | ||||
|
r21599 | Use the `addparam` method to add stream level parameter. and `newpart` to | ||
|
r20856 | populate it. Then call `getchunks` to retrieve all the binary chunks of | ||
|
r21024 | data that compose the bundle2 container.""" | ||
|
r20801 | |||
|
r24686 | _magicstring = 'HG20' | ||
|
r24640 | |||
|
r21134 | def __init__(self, ui, capabilities=()): | ||
|
r20842 | self.ui = ui | ||
|
r20801 | self._params = [] | ||
self._parts = [] | ||||
|
r21136 | self.capabilities = dict(capabilities) | ||
|
r26404 | self._compressor = util.compressors[None]() | ||
def setcompression(self, alg): | ||||
"""setup core part compression to <alg>""" | ||||
if alg is None: | ||||
return | ||||
assert not any(n.lower() == 'Compression' for n, v in self._params) | ||||
self.addparam('Compression', alg) | ||||
self._compressor = util.compressors[alg]() | ||||
|
r20801 | |||
|
r21900 | @property | ||
def nbparts(self): | ||||
"""total number of parts added to the bundler""" | ||||
return len(self._parts) | ||||
|
r21597 | # methods used to defines the bundle2 content | ||
|
r20804 | def addparam(self, name, value=None): | ||
"""add a stream level parameter""" | ||||
|
r20813 | if not name: | ||
raise ValueError('empty parameter name') | ||||
|
r20814 | if name[0] not in string.letters: | ||
raise ValueError('non letter first character: %r' % name) | ||||
|
r20804 | self._params.append((name, value)) | ||
|
r20856 | def addpart(self, part): | ||
"""add a new part to the bundle2 container | ||||
|
r21024 | Parts contains the actual applicative payload.""" | ||
|
r20995 | assert part.id is None | ||
part.id = len(self._parts) # very cheap counter | ||||
|
r20856 | self._parts.append(part) | ||
|
r21598 | def newpart(self, typeid, *args, **kwargs): | ||
|
r21602 | """create a new part and add it to the containers | ||
As the part is directly added to the containers. For now, this means | ||||
that any failure to properly initialize the part after calling | ||||
``newpart`` should result in a failure of the whole bundling process. | ||||
You can still fall back to manually create and add if you need better | ||||
control.""" | ||||
|
r21598 | part = bundlepart(typeid, *args, **kwargs) | ||
|
r21599 | self.addpart(part) | ||
|
r21598 | return part | ||
|
r21597 | # methods used to generate the bundle2 stream | ||
|
r20801 | def getchunks(self): | ||
|
r25322 | if self.ui.debugflag: | ||
msg = ['bundle2-output-bundle: "%s",' % self._magicstring] | ||||
if self._params: | ||||
msg.append(' (%i params)' % len(self._params)) | ||||
msg.append(' %i parts total\n' % len(self._parts)) | ||||
self.ui.debug(''.join(msg)) | ||||
|
r25315 | outdebug(self.ui, 'start emission of %s stream' % self._magicstring) | ||
|
r24640 | yield self._magicstring | ||
|
r20804 | param = self._paramchunk() | ||
|
r25315 | outdebug(self.ui, 'bundle parameter: %s' % param) | ||
|
r20804 | yield _pack(_fstreamparamsize, len(param)) | ||
if param: | ||||
yield param | ||||
|
r26404 | # starting compression | ||
|
r26396 | for chunk in self._getcorechunk(): | ||
|
r26404 | yield self._compressor.compress(chunk) | ||
yield self._compressor.flush() | ||||
|
r20802 | |||
|
r20804 | def _paramchunk(self): | ||
"""return a encoded version of all stream parameters""" | ||||
blocks = [] | ||||
|
r20809 | for par, value in self._params: | ||
|
r20811 | par = urllib.quote(par) | ||
|
r20809 | if value is not None: | ||
|
r20811 | value = urllib.quote(value) | ||
|
r20809 | par = '%s=%s' % (par, value) | ||
blocks.append(par) | ||||
|
r20804 | return ' '.join(blocks) | ||
|
r26396 | def _getcorechunk(self): | ||
"""yield chunk for the core part of the bundle | ||||
(all but headers and parameters)""" | ||||
outdebug(self.ui, 'start of parts') | ||||
for part in self._parts: | ||||
outdebug(self.ui, 'bundle part: "%s"' % part.type) | ||||
for chunk in part.getchunks(ui=self.ui): | ||||
yield chunk | ||||
outdebug(self.ui, 'end of bundle') | ||||
yield _pack(_fpartheadersize, 0) | ||||
|
r24794 | def salvageoutput(self): | ||
"""return a list with a copy of all output parts in the bundle | ||||
This is meant to be used during error handling to make sure we preserve | ||||
server output""" | ||||
salvaged = [] | ||||
for part in self._parts: | ||||
if part.type.startswith('output'): | ||||
salvaged.append(part.copy()) | ||||
return salvaged | ||||
|
r21013 | class unpackermixin(object): | ||
"""A mixin to extract bytes and struct data from a stream""" | ||||
|
r20802 | |||
|
r21013 | def __init__(self, fp): | ||
|
r20802 | self._fp = fp | ||
|
r24026 | self._seekable = (util.safehasattr(fp, 'seek') and | ||
util.safehasattr(fp, 'tell')) | ||||
|
r20802 | |||
def _unpack(self, format): | ||||
"""unpack this struct format from the stream""" | ||||
data = self._readexact(struct.calcsize(format)) | ||||
return _unpack(format, data) | ||||
def _readexact(self, size): | ||||
"""read exactly <size> bytes from the stream""" | ||||
return changegroup.readexactly(self._fp, size) | ||||
|
r24070 | def seek(self, offset, whence=0): | ||
|
r24026 | """move the underlying file pointer""" | ||
if self._seekable: | ||||
return self._fp.seek(offset, whence) | ||||
else: | ||||
raise NotImplementedError(_('File pointer is not seekable')) | ||||
def tell(self): | ||||
"""return the file offset, or None if file is not seekable""" | ||||
if self._seekable: | ||||
try: | ||||
return self._fp.tell() | ||||
|
r25660 | except IOError as e: | ||
|
r24026 | if e.errno == errno.ESPIPE: | ||
self._seekable = False | ||||
else: | ||||
raise | ||||
return None | ||||
def close(self): | ||||
"""close underlying file""" | ||||
if util.safehasattr(self._fp, 'close'): | ||||
return self._fp.close() | ||||
|
r21013 | |||
|
r25640 | def getunbundler(ui, fp, magicstring=None): | ||
"""return a valid unbundler object for a given magicstring""" | ||||
if magicstring is None: | ||||
magicstring = changegroup.readexactly(fp, 4) | ||||
magic, version = magicstring[0:2], magicstring[2:4] | ||||
|
r24648 | if magic != 'HG': | ||
|
r26587 | raise error.Abort(_('not a Mercurial bundle')) | ||
|
r24648 | unbundlerclass = formatmap.get(version) | ||
if unbundlerclass is None: | ||||
|
r26587 | raise error.Abort(_('unknown bundle version %s') % version) | ||
|
r24648 | unbundler = unbundlerclass(ui, fp) | ||
|
r25640 | indebug(ui, 'start processing of %s stream' % magicstring) | ||
|
r24642 | return unbundler | ||
|
r24641 | |||
|
r21013 | class unbundle20(unpackermixin): | ||
"""interpret a bundle2 stream | ||||
|
r21129 | This class is fed with a binary stream and yields parts through its | ||
`iterparts` methods.""" | ||||
|
r21013 | |||
|
r26542 | _magicstring = 'HG20' | ||
|
r24642 | def __init__(self, ui, fp): | ||
|
r21066 | """If header is specified, we do not read it out of the stream.""" | ||
|
r21013 | self.ui = ui | ||
|
r26404 | self._decompressor = util.decompressors[None] | ||
|
r21013 | super(unbundle20, self).__init__(fp) | ||
|
r20802 | @util.propertycache | ||
def params(self): | ||||
|
r21024 | """dictionary of stream level parameters""" | ||
|
r25320 | indebug(self.ui, 'reading bundle2 stream parameters') | ||
|
r20805 | params = {} | ||
paramssize = self._unpack(_fstreamparamsize)[0] | ||||
|
r23011 | if paramssize < 0: | ||
raise error.BundleValueError('negative bundle param size: %i' | ||||
% paramssize) | ||||
|
r20805 | if paramssize: | ||
|
r26541 | params = self._readexact(paramssize) | ||
params = self._processallparams(params) | ||||
|
r20805 | return params | ||
|
r20802 | |||
|
r26541 | def _processallparams(self, paramsblock): | ||
"""""" | ||||
params = {} | ||||
for p in paramsblock.split(' '): | ||||
p = p.split('=', 1) | ||||
p = [urllib.unquote(i) for i in p] | ||||
if len(p) < 2: | ||||
p.append(None) | ||||
self._processparam(*p) | ||||
params[p[0]] = p[1] | ||||
return params | ||||
|
r20844 | def _processparam(self, name, value): | ||
"""process a parameter, applying its effect if needed | ||||
Parameter starting with a lower case letter are advisory and will be | ||||
ignored when unknown. Those starting with an upper case letter are | ||||
mandatory and will this function will raise a KeyError when unknown. | ||||
Note: no option are currently supported. Any input will be either | ||||
ignored or failing. | ||||
""" | ||||
if not name: | ||||
raise ValueError('empty parameter name') | ||||
if name[0] not in string.letters: | ||||
raise ValueError('non letter first character: %r' % name) | ||||
|
r26395 | try: | ||
handler = b2streamparamsmap[name.lower()] | ||||
except KeyError: | ||||
if name[0].islower(): | ||||
indebug(self.ui, "ignoring unknown parameter %r" % name) | ||||
else: | ||||
raise error.BundleUnknownFeatureError(params=(name,)) | ||||
|
r20844 | else: | ||
|
r26395 | handler(self, name, value) | ||
|
r20844 | |||
|
r26542 | def _forwardchunks(self): | ||
"""utility to transfer a bundle2 as binary | ||||
This is made necessary by the fact the 'getbundle' command over 'ssh' | ||||
have no way to know then the reply end, relying on the bundle to be | ||||
interpreted to know its end. This is terrible and we are sorry, but we | ||||
needed to move forward to get general delta enabled. | ||||
""" | ||||
yield self._magicstring | ||||
assert 'params' not in vars(self) | ||||
paramssize = self._unpack(_fstreamparamsize)[0] | ||||
if paramssize < 0: | ||||
raise error.BundleValueError('negative bundle param size: %i' | ||||
% paramssize) | ||||
yield _pack(_fstreamparamsize, paramssize) | ||||
if paramssize: | ||||
params = self._readexact(paramssize) | ||||
self._processallparams(params) | ||||
yield params | ||||
assert self._decompressor is util.decompressors[None] | ||||
# From there, payload might need to be decompressed | ||||
self._fp = self._decompressor(self._fp) | ||||
emptycount = 0 | ||||
while emptycount < 2: | ||||
# so we can brainlessly loop | ||||
assert _fpartheadersize == _fpayloadsize | ||||
size = self._unpack(_fpartheadersize)[0] | ||||
yield _pack(_fpartheadersize, size) | ||||
if size: | ||||
emptycount = 0 | ||||
else: | ||||
emptycount += 1 | ||||
continue | ||||
if size == flaginterrupt: | ||||
continue | ||||
elif size < 0: | ||||
raise error.BundleValueError('negative chunk size: %i') | ||||
yield self._readexact(size) | ||||
|
r21129 | def iterparts(self): | ||
|
r20802 | """yield all parts contained in the stream""" | ||
# make sure param have been loaded | ||||
self.params | ||||
|
r26404 | # From there, payload need to be decompressed | ||
self._fp = self._decompressor(self._fp) | ||||
|
r25320 | indebug(self.ui, 'start extraction of bundle2 parts') | ||
|
r21014 | headerblock = self._readpartheader() | ||
while headerblock is not None: | ||||
part = unbundlepart(self.ui, headerblock, self._fp) | ||||
|
r20802 | yield part | ||
|
r24048 | part.seek(0, 2) | ||
|
r21014 | headerblock = self._readpartheader() | ||
|
r25320 | indebug(self.ui, 'end of bundle2 stream') | ||
|
r20802 | |||
|
r21014 | def _readpartheader(self): | ||
"""reads a part header size and return the bytes blob | ||||
|
r20864 | |||
|
r21014 | returns None if empty""" | ||
|
r20864 | headersize = self._unpack(_fpartheadersize)[0] | ||
|
r23011 | if headersize < 0: | ||
raise error.BundleValueError('negative part header size: %i' | ||||
% headersize) | ||||
|
r25320 | indebug(self.ui, 'part header size: %i' % headersize) | ||
|
r21014 | if headersize: | ||
return self._readexact(headersize) | ||||
return None | ||||
|
r20864 | |||
|
r24071 | def compressed(self): | ||
return False | ||||
|
r20802 | |||
|
r24686 | formatmap = {'20': unbundle20} | ||
|
r24648 | |||
|
r26395 | b2streamparamsmap = {} | ||
def b2streamparamhandler(name): | ||||
"""register a handler for a stream level parameter""" | ||||
def decorator(func): | ||||
assert name not in formatmap | ||||
b2streamparamsmap[name] = func | ||||
return func | ||||
return decorator | ||||
|
r26404 | @b2streamparamhandler('compression') | ||
def processcompression(unbundler, param, value): | ||||
"""read compression parameter and install payload decompression""" | ||||
if value not in util.decompressors: | ||||
raise error.BundleUnknownFeatureError(params=(param,), | ||||
values=(value,)) | ||||
unbundler._decompressor = util.decompressors[value] | ||||
|
r21005 | class bundlepart(object): | ||
|
r20856 | """A bundle2 part contains application level payload | ||
The part `type` is used to route the part to the application level | ||||
handler. | ||||
|
r21604 | |||
The part payload is contained in ``part.data``. It could be raw bytes or a | ||||
|
r21605 | generator of byte chunks. | ||
You can add parameters to the part using the ``addparam`` method. | ||||
Parameters can be either mandatory (default) or advisory. Remote side | ||||
should be able to safely ignore the advisory ones. | ||||
Both data and parameters cannot be modified after the generation has begun. | ||||
|
r20856 | """ | ||
|
r20877 | def __init__(self, parttype, mandatoryparams=(), advisoryparams=(), | ||
|
r23590 | data='', mandatory=True): | ||
|
r23868 | validateparttype(parttype) | ||
|
r20995 | self.id = None | ||
|
r20856 | self.type = parttype | ||
|
r21604 | self._data = data | ||
|
r21605 | self._mandatoryparams = list(mandatoryparams) | ||
self._advisoryparams = list(advisoryparams) | ||||
|
r21607 | # checking for duplicated entries | ||
self._seenparams = set() | ||||
for pname, __ in self._mandatoryparams + self._advisoryparams: | ||||
if pname in self._seenparams: | ||||
raise RuntimeError('duplicated params: %s' % pname) | ||||
self._seenparams.add(pname) | ||||
|
r21601 | # status of the part's generation: | ||
# - None: not started, | ||||
# - False: currently generated, | ||||
# - True: generation done. | ||||
self._generated = None | ||||
|
r23590 | self.mandatory = mandatory | ||
|
r20856 | |||
|
r24793 | def copy(self): | ||
"""return a copy of the part | ||||
The new part have the very same content but no partid assigned yet. | ||||
Parts with generated data cannot be copied.""" | ||||
assert not util.safehasattr(self.data, 'next') | ||||
return self.__class__(self.type, self._mandatoryparams, | ||||
self._advisoryparams, self._data, self.mandatory) | ||||
|
r21604 | # methods used to defines the part content | ||
def __setdata(self, data): | ||||
if self._generated is not None: | ||||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21604 | self._data = data | ||
def __getdata(self): | ||||
return self._data | ||||
data = property(__getdata, __setdata) | ||||
|
r21605 | @property | ||
def mandatoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._mandatoryparams) | ||||
@property | ||||
def advisoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._advisoryparams) | ||||
def addparam(self, name, value='', mandatory=True): | ||||
if self._generated is not None: | ||||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21607 | if name in self._seenparams: | ||
raise ValueError('duplicated params: %s' % name) | ||||
self._seenparams.add(name) | ||||
|
r21605 | params = self._advisoryparams | ||
if mandatory: | ||||
params = self._mandatoryparams | ||||
params.append((name, value)) | ||||
|
r21601 | # methods used to generates the bundle2 stream | ||
|
r25321 | def getchunks(self, ui): | ||
|
r21601 | if self._generated is not None: | ||
raise RuntimeError('part can only be consumed once') | ||||
self._generated = False | ||||
|
r25323 | |||
if ui.debugflag: | ||||
msg = ['bundle2-output-part: "%s"' % self.type] | ||||
if not self.mandatory: | ||||
msg.append(' (advisory)') | ||||
nbmp = len(self.mandatoryparams) | ||||
nbap = len(self.advisoryparams) | ||||
if nbmp or nbap: | ||||
msg.append(' (params:') | ||||
if nbmp: | ||||
msg.append(' %i mandatory' % nbmp) | ||||
if nbap: | ||||
msg.append(' %i advisory' % nbmp) | ||||
msg.append(')') | ||||
if not self.data: | ||||
msg.append(' empty payload') | ||||
elif util.safehasattr(self.data, 'next'): | ||||
msg.append(' streamed payload') | ||||
else: | ||||
msg.append(' %i bytes payload' % len(self.data)) | ||||
msg.append('\n') | ||||
ui.debug(''.join(msg)) | ||||
|
r20877 | #### header | ||
|
r23590 | if self.mandatory: | ||
parttype = self.type.upper() | ||||
else: | ||||
parttype = self.type.lower() | ||||
|
r25321 | outdebug(ui, 'part %s: "%s"' % (self.id, parttype)) | ||
|
r20877 | ## parttype | ||
|
r23590 | header = [_pack(_fparttypesize, len(parttype)), | ||
parttype, _pack(_fpartid, self.id), | ||||
|
r20856 | ] | ||
|
r20877 | ## parameters | ||
# count | ||||
manpar = self.mandatoryparams | ||||
advpar = self.advisoryparams | ||||
header.append(_pack(_fpartparamcount, len(manpar), len(advpar))) | ||||
# size | ||||
parsizes = [] | ||||
for key, value in manpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
for key, value in advpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
paramsizes = _pack(_makefpartparamsizes(len(parsizes) / 2), *parsizes) | ||||
header.append(paramsizes) | ||||
# key, value | ||||
for key, value in manpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
for key, value in advpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
## finalize header | ||||
|
r20856 | headerchunk = ''.join(header) | ||
|
r25321 | outdebug(ui, 'header chunk size: %i' % len(headerchunk)) | ||
|
r20856 | yield _pack(_fpartheadersize, len(headerchunk)) | ||
yield headerchunk | ||||
|
r20877 | ## payload | ||
|
r23067 | try: | ||
for chunk in self._payloadchunks(): | ||||
|
r25321 | outdebug(ui, 'payload chunk size: %i' % len(chunk)) | ||
|
r23067 | yield _pack(_fpayloadsize, len(chunk)) | ||
yield chunk | ||||
|
r26144 | except GeneratorExit: | ||
# GeneratorExit means that nobody is listening for our | ||||
# results anyway, so just bail quickly rather than trying | ||||
# to produce an error part. | ||||
ui.debug('bundle2-generatorexit\n') | ||||
raise | ||||
|
r25660 | except BaseException as exc: | ||
|
r23067 | # backup exception data for later | ||
|
r25324 | ui.debug('bundle2-input-stream-interrupt: encoding exception %s' | ||
% exc) | ||||
|
r23067 | exc_info = sys.exc_info() | ||
msg = 'unexpected error: %s' % exc | ||||
|
r24686 | interpart = bundlepart('error:abort', [('message', msg)], | ||
|
r23590 | mandatory=False) | ||
|
r23067 | interpart.id = 0 | ||
yield _pack(_fpayloadsize, -1) | ||||
|
r25321 | for chunk in interpart.getchunks(ui=ui): | ||
|
r23067 | yield chunk | ||
|
r25321 | outdebug(ui, 'closing payload chunk') | ||
|
r23067 | # abort current part payload | ||
yield _pack(_fpayloadsize, 0) | ||||
raise exc_info[0], exc_info[1], exc_info[2] | ||||
|
r21000 | # end of payload | ||
|
r25321 | outdebug(ui, 'closing payload chunk') | ||
|
r21000 | yield _pack(_fpayloadsize, 0) | ||
|
r21601 | self._generated = True | ||
|
r21000 | |||
def _payloadchunks(self): | ||||
"""yield chunks of a the part payload | ||||
Exists to handle the different methods to provide data to a part.""" | ||||
|
r20876 | # we only support fixed size data now. | ||
# This will be improved in the future. | ||||
|
r21001 | if util.safehasattr(self.data, 'next'): | ||
buff = util.chunkbuffer(self.data) | ||||
chunk = buff.read(preferedchunksize) | ||||
while chunk: | ||||
yield chunk | ||||
chunk = buff.read(preferedchunksize) | ||||
elif len(self.data): | ||||
|
r20876 | yield self.data | ||
|
r20802 | |||
|
r23066 | |||
flaginterrupt = -1 | ||||
class interrupthandler(unpackermixin): | ||||
"""read one part and process it with restricted capability | ||||
This allows to transmit exception raised on the producer size during part | ||||
iteration while the consumer is reading a part. | ||||
Part processed in this manner only have access to a ui object,""" | ||||
def __init__(self, ui, fp): | ||||
super(interrupthandler, self).__init__(fp) | ||||
self.ui = ui | ||||
def _readpartheader(self): | ||||
"""reads a part header size and return the bytes blob | ||||
returns None if empty""" | ||||
headersize = self._unpack(_fpartheadersize)[0] | ||||
if headersize < 0: | ||||
raise error.BundleValueError('negative part header size: %i' | ||||
% headersize) | ||||
|
r25318 | indebug(self.ui, 'part header size: %i\n' % headersize) | ||
|
r23066 | if headersize: | ||
return self._readexact(headersize) | ||||
return None | ||||
def __call__(self): | ||||
|
r25335 | |||
self.ui.debug('bundle2-input-stream-interrupt:' | ||||
' opening out of band context\n') | ||||
|
r25320 | indebug(self.ui, 'bundle2 stream interruption, looking for a part.') | ||
|
r23066 | headerblock = self._readpartheader() | ||
if headerblock is None: | ||||
|
r25320 | indebug(self.ui, 'no part found during interruption.') | ||
|
r23066 | return | ||
part = unbundlepart(self.ui, headerblock, self._fp) | ||||
op = interruptoperation(self.ui) | ||||
_processpart(op, part) | ||||
|
r25335 | self.ui.debug('bundle2-input-stream-interrupt:' | ||
' closing out of band context\n') | ||||
|
r23066 | |||
class interruptoperation(object): | ||||
"""A limited operation to be use by part handler during interruption | ||||
It only have access to an ui object. | ||||
""" | ||||
def __init__(self, ui): | ||||
self.ui = ui | ||||
self.reply = None | ||||
|
r24878 | self.captureoutput = False | ||
|
r23066 | |||
@property | ||||
def repo(self): | ||||
raise RuntimeError('no repo access from stream interruption') | ||||
def gettransaction(self): | ||||
raise TransactionUnavailable('no repo access from stream interruption') | ||||
|
r21014 | class unbundlepart(unpackermixin): | ||
"""a bundle part read from a bundle""" | ||||
def __init__(self, ui, header, fp): | ||||
super(unbundlepart, self).__init__(fp) | ||||
self.ui = ui | ||||
# unbundle state attr | ||||
self._headerdata = header | ||||
|
r21015 | self._headeroffset = 0 | ||
|
r21019 | self._initialized = False | ||
self.consumed = False | ||||
|
r21014 | # part data | ||
self.id = None | ||||
self.type = None | ||||
self.mandatoryparams = None | ||||
self.advisoryparams = None | ||||
|
r21610 | self.params = None | ||
|
r21612 | self.mandatorykeys = () | ||
|
r21019 | self._payloadstream = None | ||
self._readheader() | ||||
|
r23585 | self._mandatory = None | ||
|
r24035 | self._chunkindex = [] #(payload, file) position tuples for chunk starts | ||
|
r24036 | self._pos = 0 | ||
|
r21014 | |||
|
r21015 | def _fromheader(self, size): | ||
"""return the next <size> byte from the header""" | ||||
offset = self._headeroffset | ||||
data = self._headerdata[offset:(offset + size)] | ||||
|
r21019 | self._headeroffset = offset + size | ||
|
r21015 | return data | ||
|
r21016 | def _unpackheader(self, format): | ||
"""read given format from header | ||||
This automatically compute the size of the format to read.""" | ||||
data = self._fromheader(struct.calcsize(format)) | ||||
return _unpack(format, data) | ||||
|
r21608 | def _initparams(self, mandatoryparams, advisoryparams): | ||
"""internal function to setup all logic related parameters""" | ||||
|
r21609 | # make it read only to prevent people touching it by mistake. | ||
self.mandatoryparams = tuple(mandatoryparams) | ||||
self.advisoryparams = tuple(advisoryparams) | ||||
|
r21610 | # user friendly UI | ||
self.params = dict(self.mandatoryparams) | ||||
self.params.update(dict(self.advisoryparams)) | ||||
|
r21612 | self.mandatorykeys = frozenset(p[0] for p in mandatoryparams) | ||
|
r21608 | |||
|
r24035 | def _payloadchunks(self, chunknum=0): | ||
'''seek to specified chunk and start yielding data''' | ||||
if len(self._chunkindex) == 0: | ||||
assert chunknum == 0, 'Must start with chunk 0' | ||||
self._chunkindex.append((0, super(unbundlepart, self).tell())) | ||||
else: | ||||
assert chunknum < len(self._chunkindex), \ | ||||
'Unknown chunk %d' % chunknum | ||||
super(unbundlepart, self).seek(self._chunkindex[chunknum][1]) | ||||
pos = self._chunkindex[chunknum][0] | ||||
|
r24034 | payloadsize = self._unpack(_fpayloadsize)[0] | ||
|
r25320 | indebug(self.ui, 'payload chunk size: %i' % payloadsize) | ||
|
r24034 | while payloadsize: | ||
if payloadsize == flaginterrupt: | ||||
# interruption detection, the handler will now read a | ||||
# single part and process it. | ||||
interrupthandler(self.ui, self._fp)() | ||||
elif payloadsize < 0: | ||||
msg = 'negative payload chunk size: %i' % payloadsize | ||||
raise error.BundleValueError(msg) | ||||
else: | ||||
|
r24035 | result = self._readexact(payloadsize) | ||
chunknum += 1 | ||||
pos += payloadsize | ||||
if chunknum == len(self._chunkindex): | ||||
self._chunkindex.append((pos, | ||||
super(unbundlepart, self).tell())) | ||||
yield result | ||||
|
r24034 | payloadsize = self._unpack(_fpayloadsize)[0] | ||
|
r25320 | indebug(self.ui, 'payload chunk size: %i' % payloadsize) | ||
|
r24034 | |||
|
r24037 | def _findchunk(self, pos): | ||
'''for a given payload position, return a chunk number and offset''' | ||||
for chunk, (ppos, fpos) in enumerate(self._chunkindex): | ||||
if ppos == pos: | ||||
return chunk, 0 | ||||
elif ppos > pos: | ||||
return chunk - 1, pos - self._chunkindex[chunk - 1][0] | ||||
raise ValueError('Unknown chunk') | ||||
|
r21019 | def _readheader(self): | ||
|
r21014 | """read the header and setup the object""" | ||
|
r21016 | typesize = self._unpackheader(_fparttypesize)[0] | ||
|
r21015 | self.type = self._fromheader(typesize) | ||
|
r25320 | indebug(self.ui, 'part type: "%s"' % self.type) | ||
|
r21016 | self.id = self._unpackheader(_fpartid)[0] | ||
|
r25320 | indebug(self.ui, 'part id: "%s"' % self.id) | ||
|
r23585 | # extract mandatory bit from type | ||
self.mandatory = (self.type != self.type.lower()) | ||||
self.type = self.type.lower() | ||||
|
r21014 | ## reading parameters | ||
# param count | ||||
|
r21016 | mancount, advcount = self._unpackheader(_fpartparamcount) | ||
|
r25320 | indebug(self.ui, 'part parameters: %i' % (mancount + advcount)) | ||
|
r21014 | # param size | ||
|
r21016 | fparamsizes = _makefpartparamsizes(mancount + advcount) | ||
paramsizes = self._unpackheader(fparamsizes) | ||||
|
r21014 | # make it a list of couple again | ||
paramsizes = zip(paramsizes[::2], paramsizes[1::2]) | ||||
# split mandatory from advisory | ||||
mansizes = paramsizes[:mancount] | ||||
advsizes = paramsizes[mancount:] | ||||
|
r23139 | # retrieve param value | ||
|
r21014 | manparams = [] | ||
for key, value in mansizes: | ||||
|
r21015 | manparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21014 | advparams = [] | ||
for key, value in advsizes: | ||||
|
r21015 | advparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21608 | self._initparams(manparams, advparams) | ||
|
r21014 | ## part payload | ||
|
r24034 | self._payloadstream = util.chunkbuffer(self._payloadchunks()) | ||
|
r21019 | # we read the data, tell it | ||
self._initialized = True | ||||
def read(self, size=None): | ||||
"""read payload data""" | ||||
if not self._initialized: | ||||
self._readheader() | ||||
if size is None: | ||||
data = self._payloadstream.read() | ||||
else: | ||||
data = self._payloadstream.read(size) | ||||
|
r25334 | self._pos += len(data) | ||
|
r21019 | if size is None or len(data) < size: | ||
|
r25334 | if not self.consumed and self._pos: | ||
self.ui.debug('bundle2-input-part: total payload size %i\n' | ||||
% self._pos) | ||||
|
r21019 | self.consumed = True | ||
return data | ||||
|
r24036 | def tell(self): | ||
return self._pos | ||||
|
r24037 | def seek(self, offset, whence=0): | ||
if whence == 0: | ||||
newpos = offset | ||||
elif whence == 1: | ||||
newpos = self._pos + offset | ||||
elif whence == 2: | ||||
if not self.consumed: | ||||
self.read() | ||||
newpos = self._chunkindex[-1][0] - offset | ||||
else: | ||||
raise ValueError('Unknown whence value: %r' % (whence,)) | ||||
if newpos > self._chunkindex[-1][0] and not self.consumed: | ||||
self.read() | ||||
if not 0 <= newpos <= self._chunkindex[-1][0]: | ||||
raise ValueError('Offset out of range') | ||||
if self._pos != newpos: | ||||
chunk, internaloffset = self._findchunk(newpos) | ||||
self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk)) | ||||
adjust = self.read(internaloffset) | ||||
if len(adjust) != internaloffset: | ||||
|
r26587 | raise error.Abort(_('Seek failed\n')) | ||
|
r24037 | self._pos = newpos | ||
|
r25317 | # These are only the static capabilities. | ||
# Check the 'getrepocaps' function for the rest. | ||||
|
r24686 | capabilities = {'HG20': (), | ||
|
r25493 | 'error': ('abort', 'unsupportedcontent', 'pushraced', | ||
'pushkey'), | ||||
|
r24686 | 'listkeys': (), | ||
'pushkey': (), | ||||
|
r23029 | 'digests': tuple(sorted(util.DIGESTS.keys())), | ||
|
r24686 | 'remote-changegroup': ('http', 'https'), | ||
|
r25401 | 'hgtagsfnodes': (), | ||
|
r22341 | } | ||
|
r23439 | def getrepocaps(repo, allowpushback=False): | ||
|
r22342 | """return the bundle2 capabilities for a given repo | ||
|
r22343 | Exists to allow extensions (like evolution) to mutate the capabilities. | ||
|
r22342 | """ | ||
|
r22343 | caps = capabilities.copy() | ||
|
r24686 | caps['changegroup'] = tuple(sorted(changegroup.packermap.keys())) | ||
|
r22953 | if obsolete.isenabled(repo, obsolete.exchangeopt): | ||
|
r22343 | supportedformat = tuple('V%i' % v for v in obsolete.formats) | ||
|
r24686 | caps['obsmarkers'] = supportedformat | ||
|
r23439 | if allowpushback: | ||
|
r24686 | caps['pushback'] = () | ||
|
r22343 | return caps | ||
|
r22342 | |||
|
r21644 | def bundle2caps(remote): | ||
|
r23139 | """return the bundle capabilities of a peer as dict""" | ||
|
r24686 | raw = remote.capable('bundle2') | ||
|
r21644 | if not raw and raw != '': | ||
return {} | ||||
|
r24686 | capsblob = urllib.unquote(remote.capable('bundle2')) | ||
|
r21644 | return decodecaps(capsblob) | ||
|
r21014 | |||
|
r22344 | def obsmarkersversion(caps): | ||
"""extract the list of supported obsmarkers versions from a bundle2caps dict | ||||
""" | ||||
|
r24686 | obscaps = caps.get('obsmarkers', ()) | ||
|
r22344 | return [int(c[1:]) for c in obscaps if c.startswith('V')] | ||
|
r25518 | @parthandler('changegroup', ('version', 'nbchanges')) | ||
|
r20998 | def handlechangegroup(op, inpart): | ||
|
r20950 | """apply a changegroup part on the repo | ||
This is a very early implementation that will massive rework before being | ||||
inflicted to any end-user. | ||||
""" | ||||
|
r20952 | # Make sure we trigger a transaction creation | ||
# | ||||
# The addchangegroup function will get a transaction object by itself, but | ||||
# we need to make sure we trigger the creation of a transaction object used | ||||
# for the whole processing scope. | ||||
op.gettransaction() | ||||
|
r23170 | unpackerversion = inpart.params.get('version', '01') | ||
# We should raise an appropriate exception here | ||||
unpacker = changegroup.packermap[unpackerversion][1] | ||||
|
r26269 | cg = unpacker(inpart, None) | ||
|
r23001 | # the source and url passed here are overwritten by the one contained in | ||
# the transaction.hookargs argument. So 'bundle2' is a placeholder | ||||
|
r25518 | nbchangesets = None | ||
if 'nbchanges' in inpart.params: | ||||
nbchangesets = int(inpart.params.get('nbchanges')) | ||||
|
r26698 | ret = cg.apply(op.repo, 'bundle2', 'bundle2', expectedtotal=nbchangesets) | ||
|
r20950 | op.records.add('changegroup', {'return': ret}) | ||
|
r20998 | if op.reply is not None: | ||
|
r23139 | # This is definitely not the final form of this | ||
|
r20998 | # return. But one need to start somewhere. | ||
|
r24686 | part = op.reply.newpart('reply:changegroup', mandatory=False) | ||
|
r21606 | part.addparam('in-reply-to', str(inpart.id), mandatory=False) | ||
part.addparam('return', '%i' % ret, mandatory=False) | ||||
|
r21019 | assert not inpart.read() | ||
|
r20950 | |||
|
r23029 | _remotechangegroupparams = tuple(['url', 'size', 'digests'] + | ||
['digest:%s' % k for k in util.DIGESTS.keys()]) | ||||
|
r24686 | @parthandler('remote-changegroup', _remotechangegroupparams) | ||
|
r23029 | def handleremotechangegroup(op, inpart): | ||
"""apply a bundle10 on the repo, given an url and validation information | ||||
All the information about the remote bundle to import are given as | ||||
parameters. The parameters include: | ||||
- url: the url to the bundle10. | ||||
- size: the bundle10 file size. It is used to validate what was | ||||
retrieved by the client matches the server knowledge about the bundle. | ||||
- digests: a space separated list of the digest types provided as | ||||
parameters. | ||||
- digest:<digest-type>: the hexadecimal representation of the digest with | ||||
that name. Like the size, it is used to validate what was retrieved by | ||||
the client matches what the server knows about the bundle. | ||||
When multiple digest types are given, all of them are checked. | ||||
""" | ||||
try: | ||||
raw_url = inpart.params['url'] | ||||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url') | ||
|
r23029 | parsed_url = util.url(raw_url) | ||
|
r24686 | if parsed_url.scheme not in capabilities['remote-changegroup']: | ||
|
r26587 | raise error.Abort(_('remote-changegroup does not support %s urls') % | ||
|
r23029 | parsed_url.scheme) | ||
try: | ||||
size = int(inpart.params['size']) | ||||
except ValueError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: invalid value for param "%s"') | ||
|
r23029 | % 'size') | ||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size') | ||
|
r23029 | |||
digests = {} | ||||
for typ in inpart.params.get('digests', '').split(): | ||||
param = 'digest:%s' % typ | ||||
try: | ||||
value = inpart.params[param] | ||||
except KeyError: | ||||
|
r26587 | raise error.Abort(_('remote-changegroup: missing "%s" param') % | ||
|
r23029 | param) | ||
digests[typ] = value | ||||
real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests) | ||||
# Make sure we trigger a transaction creation | ||||
# | ||||
# The addchangegroup function will get a transaction object by itself, but | ||||
# we need to make sure we trigger the creation of a transaction object used | ||||
# for the whole processing scope. | ||||
op.gettransaction() | ||||
|
r25919 | from . import exchange | ||
|
r23029 | cg = exchange.readbundle(op.repo.ui, real_part, raw_url) | ||
if not isinstance(cg, changegroup.cg1unpacker): | ||||
|
r26587 | raise error.Abort(_('%s: not a bundle version 1.0') % | ||
|
r23029 | util.hidepassword(raw_url)) | ||
|
r26698 | ret = cg.apply(op.repo, 'bundle2', 'bundle2') | ||
|
r23029 | op.records.add('changegroup', {'return': ret}) | ||
if op.reply is not None: | ||||
|
r23139 | # This is definitely not the final form of this | ||
|
r23029 | # return. But one need to start somewhere. | ||
|
r24686 | part = op.reply.newpart('reply:changegroup') | ||
|
r23029 | part.addparam('in-reply-to', str(inpart.id), mandatory=False) | ||
part.addparam('return', '%i' % ret, mandatory=False) | ||||
try: | ||||
real_part.validate() | ||||
|
r26587 | except error.Abort as e: | ||
raise error.Abort(_('bundle at %s is corrupted:\n%s') % | ||||
|
r23029 | (util.hidepassword(raw_url), str(e))) | ||
assert not inpart.read() | ||||
|
r24686 | @parthandler('reply:changegroup', ('return', 'in-reply-to')) | ||
|
r22548 | def handlereplychangegroup(op, inpart): | ||
|
r21611 | ret = int(inpart.params['return']) | ||
replyto = int(inpart.params['in-reply-to']) | ||||
op.records.add('changegroup', {'return': ret}, replyto) | ||||
|
r20950 | |||
|
r24686 | @parthandler('check:heads') | ||
|
r22548 | def handlecheckheads(op, inpart): | ||
|
r21060 | """check that head of the repo did not change | ||
This is used to detect a push race when using unbundle. | ||||
This replaces the "heads" argument of unbundle.""" | ||||
h = inpart.read(20) | ||||
heads = [] | ||||
while len(h) == 20: | ||||
heads.append(h) | ||||
h = inpart.read(20) | ||||
assert not h | ||||
|
r26565 | # Trigger a transaction so that we are guaranteed to have the lock now. | ||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r21060 | if heads != op.repo.heads(): | ||
|
r21185 | raise error.PushRaced('repository changed while pushing - ' | ||
'please try again') | ||||
|
r21130 | |||
|
r24686 | @parthandler('output') | ||
|
r21131 | def handleoutput(op, inpart): | ||
"""forward output captured on the server to the client""" | ||||
for line in inpart.read().splitlines(): | ||||
|
r24846 | op.ui.status(('remote: %s\n' % line)) | ||
|
r21131 | |||
|
r24686 | @parthandler('replycaps') | ||
|
r21130 | def handlereplycaps(op, inpart): | ||
"""Notify that a reply bundle should be created | ||||
|
r21138 | The payload contains the capabilities information for the reply""" | ||
caps = decodecaps(inpart.read()) | ||||
|
r21130 | if op.reply is None: | ||
|
r21135 | op.reply = bundle20(op.ui, caps) | ||
|
r21131 | |||
|
r24686 | @parthandler('error:abort', ('message', 'hint')) | ||
|
r24741 | def handleerrorabort(op, inpart): | ||
|
r21177 | """Used to transmit abort error over the wire""" | ||
|
r26587 | raise error.Abort(inpart.params['message'], hint=inpart.params.get('hint')) | ||
|
r21183 | |||
|
r25493 | @parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret', | ||
'in-reply-to')) | ||||
def handleerrorpushkey(op, inpart): | ||||
"""Used to transmit failure of a mandatory pushkey over the wire""" | ||||
kwargs = {} | ||||
for name in ('namespace', 'key', 'new', 'old', 'ret'): | ||||
value = inpart.params.get(name) | ||||
if value is not None: | ||||
kwargs[name] = value | ||||
raise error.PushkeyFailed(inpart.params['in-reply-to'], **kwargs) | ||||
|
r24686 | @parthandler('error:unsupportedcontent', ('parttype', 'params')) | ||
|
r24741 | def handleerrorunsupportedcontent(op, inpart): | ||
|
r21619 | """Used to transmit unknown content error over the wire""" | ||
|
r21622 | kwargs = {} | ||
|
r21627 | parttype = inpart.params.get('parttype') | ||
if parttype is not None: | ||||
kwargs['parttype'] = parttype | ||||
|
r21622 | params = inpart.params.get('params') | ||
if params is not None: | ||||
kwargs['params'] = params.split('\0') | ||||
|
r26393 | raise error.BundleUnknownFeatureError(**kwargs) | ||
|
r21186 | |||
|
r24686 | @parthandler('error:pushraced', ('message',)) | ||
|
r24741 | def handleerrorpushraced(op, inpart): | ||
|
r21186 | """Used to transmit push race error over the wire""" | ||
|
r21611 | raise error.ResponseError(_('push failed:'), inpart.params['message']) | ||
|
r21655 | |||
|
r24686 | @parthandler('listkeys', ('namespace',)) | ||
|
r21655 | def handlelistkeys(op, inpart): | ||
"""retrieve pushkey namespace content stored in a bundle2""" | ||||
namespace = inpart.params['namespace'] | ||||
r = pushkey.decodekeys(inpart.read()) | ||||
op.records.add('listkeys', (namespace, r)) | ||||
|
r21660 | |||
|
r24686 | @parthandler('pushkey', ('namespace', 'key', 'old', 'new')) | ||
|
r21660 | def handlepushkey(op, inpart): | ||
"""process a pushkey request""" | ||||
dec = pushkey.decode | ||||
namespace = dec(inpart.params['namespace']) | ||||
key = dec(inpart.params['key']) | ||||
old = dec(inpart.params['old']) | ||||
new = dec(inpart.params['new']) | ||||
|
r26565 | # Grab the transaction to ensure that we have the lock before performing the | ||
# pushkey. | ||||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r21660 | ret = op.repo.pushkey(namespace, key, old, new) | ||
record = {'namespace': namespace, | ||||
'key': key, | ||||
'old': old, | ||||
'new': new} | ||||
op.records.add('pushkey', record) | ||||
if op.reply is not None: | ||||
|
r24686 | rpart = op.reply.newpart('reply:pushkey') | ||
|
r21660 | rpart.addparam('in-reply-to', str(inpart.id), mandatory=False) | ||
rpart.addparam('return', '%i' % ret, mandatory=False) | ||||
|
r25481 | if inpart.mandatory and not ret: | ||
|
r25484 | kwargs = {} | ||
for key in ('namespace', 'key', 'new', 'old', 'ret'): | ||||
if key in inpart.params: | ||||
kwargs[key] = inpart.params[key] | ||||
raise error.PushkeyFailed(partid=str(inpart.id), **kwargs) | ||||
|
r21660 | |||
|
r24686 | @parthandler('reply:pushkey', ('return', 'in-reply-to')) | ||
|
r21660 | def handlepushkeyreply(op, inpart): | ||
"""retrieve the result of a pushkey request""" | ||||
ret = int(inpart.params['return']) | ||||
partid = int(inpart.params['in-reply-to']) | ||||
op.records.add('pushkey', {'return': ret}, partid) | ||||
|
r22336 | |||
|
r24686 | @parthandler('obsmarkers') | ||
|
r22336 | def handleobsmarker(op, inpart): | ||
"""add a stream of obsmarkers to the repo""" | ||||
tr = op.gettransaction() | ||||
|
r24733 | markerdata = inpart.read() | ||
if op.ui.config('experimental', 'obsmarkers-exchange-debug', False): | ||||
op.ui.write(('obsmarker-exchange: %i bytes received\n') | ||||
% len(markerdata)) | ||||
|
r26685 | # The mergemarkers call will crash if marker creation is not enabled. | ||
# we want to avoid this if the part is advisory. | ||||
if not inpart.mandatory and op.repo.obsstore.readonly: | ||||
op.repo.ui.debug('ignoring obsolescence markers, feature not enabled') | ||||
return | ||||
|
r24733 | new = op.repo.obsstore.mergemarkers(tr, markerdata) | ||
|
r22337 | if new: | ||
op.repo.ui.status(_('%i new obsolescence markers\n') % new) | ||||
|
r22338 | op.records.add('obsmarkers', {'new': new}) | ||
|
r22340 | if op.reply is not None: | ||
|
r24686 | rpart = op.reply.newpart('reply:obsmarkers') | ||
|
r22340 | rpart.addparam('in-reply-to', str(inpart.id), mandatory=False) | ||
rpart.addparam('new', '%i' % new, mandatory=False) | ||||
|
r24686 | @parthandler('reply:obsmarkers', ('new', 'in-reply-to')) | ||
|
r25506 | def handleobsmarkerreply(op, inpart): | ||
|
r22340 | """retrieve the result of a pushkey request""" | ||
ret = int(inpart.params['new']) | ||||
partid = int(inpart.params['in-reply-to']) | ||||
op.records.add('obsmarkers', {'new': ret}, partid) | ||||
|
r25401 | |||
@parthandler('hgtagsfnodes') | ||||
def handlehgtagsfnodes(op, inpart): | ||||
"""Applies .hgtags fnodes cache entries to the local repo. | ||||
Payload is pairs of 20 byte changeset nodes and filenodes. | ||||
""" | ||||
|
r26565 | # Grab the transaction so we ensure that we have the lock at this point. | ||
if op.ui.configbool('experimental', 'bundle2lazylocking'): | ||||
op.gettransaction() | ||||
|
r25401 | cache = tags.hgtagsfnodescache(op.repo.unfiltered()) | ||
count = 0 | ||||
while True: | ||||
node = inpart.read(20) | ||||
fnode = inpart.read(20) | ||||
if len(node) < 20 or len(fnode) < 20: | ||||
|
r25641 | op.ui.debug('ignoring incomplete received .hgtags fnodes data\n') | ||
|
r25401 | break | ||
cache.setfnode(node, fnode) | ||||
count += 1 | ||||
cache.write() | ||||
op.ui.debug('applied %i hgtags fnodes cache entries\n' % count) | ||||