bundle2.py
853 lines
| 29.2 KiB
| text/x-python
|
PythonLexer
/ mercurial / bundle2.py
|
r20801 | # bundle2.py - generic container format to transmit arbitrary data. | ||
# | ||||
# Copyright 2013 Facebook, Inc. | ||||
# | ||||
# This software may be used and distributed according to the terms of the | ||||
# GNU General Public License version 2 or any later version. | ||||
"""Handling of the new bundle2 format | ||||
The goal of bundle2 is to act as an atomically packet to transmit a set of | ||||
payloads in an application agnostic way. It consist in a sequence of "parts" | ||||
that will be handed to and processed by the application layer. | ||||
General format architecture | ||||
=========================== | ||||
The format is architectured as follow | ||||
- magic string | ||||
- stream level parameters | ||||
- payload parts (any number) | ||||
- end of stream marker. | ||||
|
r20856 | the Binary format | ||
|
r20801 | ============================ | ||
|
r21024 | All numbers are unsigned and big-endian. | ||
|
r20801 | |||
stream level parameters | ||||
------------------------ | ||||
Binary format is as follow | ||||
:params size: (16 bits integer) | ||||
The total number of Bytes used by the parameters | ||||
:params value: arbitrary number of Bytes | ||||
A blob of `params size` containing the serialized version of all stream level | ||||
parameters. | ||||
|
r21024 | The blob contains a space separated list of parameters. Parameters with value | ||
|
r20811 | are stored in the form `<name>=<value>`. Both name and value are urlquoted. | ||
|
r20804 | |||
|
r20813 | Empty name are obviously forbidden. | ||
|
r20844 | Name MUST start with a letter. If this first letter is lower case, the | ||
|
r21024 | parameter is advisory and can be safely ignored. However when the first | ||
|
r20844 | letter is capital, the parameter is mandatory and the bundling process MUST | ||
stop if he is not able to proceed it. | ||||
|
r20814 | |||
|
r20808 | Stream parameters use a simple textual format for two main reasons: | ||
|
r20804 | |||
|
r21024 | - Stream level parameters should remain simple and we want to discourage any | ||
|
r20808 | crazy usage. | ||
|
r21024 | - Textual data allow easy human inspection of a bundle2 header in case of | ||
|
r20808 | troubles. | ||
Any Applicative level options MUST go into a bundle2 part instead. | ||||
|
r20801 | |||
Payload part | ||||
------------------------ | ||||
Binary format is as follow | ||||
:header size: (16 bits inter) | ||||
The total number of Bytes used by the part headers. When the header is empty | ||||
(size = 0) this is interpreted as the end of stream marker. | ||||
|
r20856 | :header: | ||
The header defines how to interpret the part. It contains two piece of | ||||
data: the part type, and the part parameters. | ||||
The part type is used to route an application level handler, that can | ||||
interpret payload. | ||||
Part parameters are passed to the application level handler. They are | ||||
meant to convey information that will help the application level object to | ||||
interpret the part payload. | ||||
The binary format of the header is has follow | ||||
:typesize: (one byte) | ||||
|
r20877 | |||
|
r21024 | :parttype: alphanumerical part name | ||
|
r20877 | |||
|
r20995 | :partid: A 32bits integer (unique in the bundle) that can be used to refer | ||
to this part. | ||||
|
r20877 | :parameters: | ||
|
r21024 | Part's parameter may have arbitrary content, the binary structure is:: | ||
|
r20877 | |||
<mandatory-count><advisory-count><param-sizes><param-data> | ||||
:mandatory-count: 1 byte, number of mandatory parameters | ||||
:advisory-count: 1 byte, number of advisory parameters | ||||
:param-sizes: | ||||
N couple of bytes, where N is the total number of parameters. Each | ||||
couple contains (<size-of-key>, <size-of-value) for one parameter. | ||||
:param-data: | ||||
A blob of bytes from which each parameter key and value can be | ||||
retrieved using the list of size couples stored in the previous | ||||
field. | ||||
Mandatory parameters comes first, then the advisory ones. | ||||
|
r20856 | |||
|
r21607 | Each parameter's key MUST be unique within the part. | ||
|
r20856 | :payload: | ||
|
r20876 | payload is a series of `<chunksize><chunkdata>`. | ||
`chunksize` is a 32 bits integer, `chunkdata` are plain bytes (as much as | ||||
`chunksize` says)` The payload part is concluded by a zero size chunk. | ||||
The current implementation always produces either zero or one chunk. | ||||
|
r21024 | This is an implementation limitation that will ultimately be lifted. | ||
|
r20891 | |||
Bundle processing | ||||
============================ | ||||
Each part is processed in order using a "part handler". Handler are registered | ||||
for a certain part type. | ||||
The matching of a part to its handler is case insensitive. The case of the | ||||
part type is used to know if a part is mandatory or advisory. If the Part type | ||||
contains any uppercase char it is considered mandatory. When no handler is | ||||
known for a Mandatory part, the process is aborted and an exception is raised. | ||||
|
r20892 | If the part is advisory and no handler is known, the part is ignored. When the | ||
process is aborted, the full bundle is still read from the stream to keep the | ||||
channel usable. But none of the part read from an abort are processed. In the | ||||
future, dropping the stream may become an option for channel we do not care to | ||||
preserve. | ||||
|
r20801 | """ | ||
|
r20802 | import util | ||
|
r20804 | import struct | ||
|
r20811 | import urllib | ||
|
r20814 | import string | ||
|
r20804 | |||
|
r21184 | import changegroup, error | ||
|
r20803 | from i18n import _ | ||
|
r20802 | |||
|
r20804 | _pack = struct.pack | ||
_unpack = struct.unpack | ||||
|
r21144 | _magicstring = 'HG2X' | ||
|
r20801 | |||
|
r20804 | _fstreamparamsize = '>H' | ||
|
r20856 | _fpartheadersize = '>H' | ||
_fparttypesize = '>B' | ||||
|
r20995 | _fpartid = '>I' | ||
|
r20876 | _fpayloadsize = '>I' | ||
|
r20877 | _fpartparamcount = '>BB' | ||
|
r21001 | preferedchunksize = 4096 | ||
|
r20877 | def _makefpartparamsizes(nbparams): | ||
"""return a struct format to read part parameter sizes | ||||
The number parameters is variable so we need to build that format | ||||
dynamically. | ||||
""" | ||||
return '>'+('BB'*nbparams) | ||||
|
r20804 | |||
|
r20890 | parthandlermapping = {} | ||
|
r20889 | |||
|
r21623 | def parthandler(parttype, params=()): | ||
|
r20890 | """decorator that register a function as a bundle2 part handler | ||
eg:: | ||||
|
r21624 | @parthandler('myparttype', ('mandatory', 'param', 'handled')) | ||
|
r20890 | def myparttypehandler(...): | ||
'''process a part of type "my part".''' | ||||
... | ||||
""" | ||||
def _decorator(func): | ||||
|
r20891 | lparttype = parttype.lower() # enforce lower case matching. | ||
assert lparttype not in parthandlermapping | ||||
parthandlermapping[lparttype] = func | ||||
|
r21623 | func.params = frozenset(params) | ||
|
r20890 | return func | ||
return _decorator | ||||
|
r20889 | |||
|
r20949 | class unbundlerecords(object): | ||
"""keep record of what happens during and unbundle | ||||
New records are added using `records.add('cat', obj)`. Where 'cat' is a | ||||
|
r21024 | category of record and obj is an arbitrary object. | ||
|
r20949 | |||
`records['cat']` will return all entries of this category 'cat'. | ||||
Iterating on the object itself will yield `('category', obj)` tuples | ||||
for all entries. | ||||
All iterations happens in chronological order. | ||||
""" | ||||
def __init__(self): | ||||
self._categories = {} | ||||
self._sequences = [] | ||||
|
r20996 | self._replies = {} | ||
|
r20949 | |||
|
r20996 | def add(self, category, entry, inreplyto=None): | ||
|
r20949 | """add a new record of a given category. | ||
The entry can then be retrieved in the list returned by | ||||
self['category'].""" | ||||
self._categories.setdefault(category, []).append(entry) | ||||
self._sequences.append((category, entry)) | ||||
|
r20996 | if inreplyto is not None: | ||
self.getreplies(inreplyto).add(category, entry) | ||||
def getreplies(self, partid): | ||||
"""get the subrecords that replies to a specific part""" | ||||
return self._replies.setdefault(partid, unbundlerecords()) | ||||
|
r20949 | |||
def __getitem__(self, cat): | ||||
return tuple(self._categories.get(cat, ())) | ||||
def __iter__(self): | ||||
return iter(self._sequences) | ||||
def __len__(self): | ||||
return len(self._sequences) | ||||
def __nonzero__(self): | ||||
return bool(self._sequences) | ||||
|
r20948 | class bundleoperation(object): | ||
"""an object that represents a single bundling process | ||||
Its purpose is to carry unbundle-related objects and states. | ||||
A new object should be created at the beginning of each bundle processing. | ||||
The object is to be returned by the processing function. | ||||
The object has very little content now it will ultimately contain: | ||||
* an access to the repo the bundle is applied to, | ||||
* a ui object, | ||||
* a way to retrieve a transaction to add changes to the repo, | ||||
* a way to record the result of processing each part, | ||||
* a way to construct a bundle response when applicable. | ||||
""" | ||||
|
r20952 | def __init__(self, repo, transactiongetter): | ||
|
r20948 | self.repo = repo | ||
self.ui = repo.ui | ||||
|
r20949 | self.records = unbundlerecords() | ||
|
r20952 | self.gettransaction = transactiongetter | ||
|
r20997 | self.reply = None | ||
|
r20948 | |||
|
r20952 | class TransactionUnavailable(RuntimeError): | ||
pass | ||||
def _notransaction(): | ||||
"""default method to get a transaction while processing a bundle | ||||
Raise an exception to highlight the fact that no transaction was expected | ||||
to be created""" | ||||
raise TransactionUnavailable() | ||||
def processbundle(repo, unbundler, transactiongetter=_notransaction): | ||||
|
r20889 | """This function process a bundle, apply effect to/from a repo | ||
|
r20947 | It iterates over each part then searches for and uses the proper handling | ||
code to process the part. Parts are processed in order. | ||||
|
r20889 | |||
This is very early version of this function that will be strongly reworked | ||||
before final usage. | ||||
|
r20891 | Unknown Mandatory part will abort the process. | ||
|
r20889 | """ | ||
|
r20952 | op = bundleoperation(repo, transactiongetter) | ||
|
r20889 | # todo: | ||
# - replace this is a init function soon. | ||||
# - exception catching | ||||
unbundler.params | ||||
|
r21129 | iterparts = unbundler.iterparts() | ||
|
r21019 | part = None | ||
|
r20892 | try: | ||
for part in iterparts: | ||||
parttype = part.type | ||||
# part key are matched lower case | ||||
key = parttype.lower() | ||||
try: | ||||
|
r21626 | handler = parthandlermapping.get(key) | ||
if handler is None: | ||||
raise error.BundleValueError(parttype=key) | ||||
|
r20948 | op.ui.debug('found a handler for part %r\n' % parttype) | ||
|
r21626 | unknownparams = part.mandatorykeys - handler.params | ||
if unknownparams: | ||||
unknownparams = list(unknownparams) | ||||
unknownparams.sort() | ||||
raise error.BundleValueError(parttype=key, | ||||
params=unknownparams) | ||||
except error.BundleValueError, exc: | ||||
|
r20892 | if key != parttype: # mandatory parts | ||
|
r21626 | raise | ||
op.ui.debug('ignoring unsupported advisory part %s\n' % exc) | ||||
|
r21019 | # consuming the part | ||
part.read() | ||||
|
r20892 | continue | ||
|
r21004 | |||
|
r21625 | |||
|
r21004 | # handler is called outside the above try block so that we don't | ||
# risk catching KeyErrors from anything other than the | ||||
# parthandlermapping lookup (any KeyError raised by handler() | ||||
# itself represents a defect of a different variety). | ||||
|
r21131 | output = None | ||
if op.reply is not None: | ||||
|
r21133 | op.ui.pushbuffer(error=True) | ||
|
r21131 | output = '' | ||
try: | ||||
handler(op, part) | ||||
finally: | ||||
if output is not None: | ||||
output = op.ui.popbuffer() | ||||
if output: | ||||
|
r21606 | outpart = op.reply.newpart('b2x:output', data=output) | ||
outpart.addparam('in-reply-to', str(part.id), mandatory=False) | ||||
|
r21019 | part.read() | ||
|
r21176 | except Exception, exc: | ||
|
r21019 | if part is not None: | ||
# consume the bundle content | ||||
part.read() | ||||
|
r20892 | for part in iterparts: | ||
|
r21019 | # consume the bundle content | ||
part.read() | ||||
|
r21176 | # Small hack to let caller code distinguish exceptions from bundle2 | ||
# processing fron the ones from bundle1 processing. This is mostly | ||||
# needed to handle different return codes to unbundle according to the | ||||
# type of bundle. We should probably clean up or drop this return code | ||||
# craziness in a future version. | ||||
exc.duringunbundle2 = True | ||||
|
r20892 | raise | ||
|
r20949 | return op | ||
|
r20889 | |||
|
r21138 | def decodecaps(blob): | ||
"""decode a bundle2 caps bytes blob into a dictionnary | ||||
The blob is a list of capabilities (one per line) | ||||
Capabilities may have values using a line of the form:: | ||||
capability=value1,value2,value3 | ||||
The values are always a list.""" | ||||
caps = {} | ||||
for line in blob.splitlines(): | ||||
if not line: | ||||
continue | ||||
if '=' not in line: | ||||
key, vals = line, () | ||||
else: | ||||
key, vals = line.split('=', 1) | ||||
vals = vals.split(',') | ||||
key = urllib.unquote(key) | ||||
vals = [urllib.unquote(v) for v in vals] | ||||
caps[key] = vals | ||||
return caps | ||||
|
r21139 | def encodecaps(caps): | ||
"""encode a bundle2 caps dictionary into a bytes blob""" | ||||
chunks = [] | ||||
for ca in sorted(caps): | ||||
vals = caps[ca] | ||||
ca = urllib.quote(ca) | ||||
vals = [urllib.quote(v) for v in vals] | ||||
if vals: | ||||
ca = "%s=%s" % (ca, ','.join(vals)) | ||||
chunks.append(ca) | ||||
return '\n'.join(chunks) | ||||
|
r20801 | class bundle20(object): | ||
"""represent an outgoing bundle2 container | ||||
|
r21599 | Use the `addparam` method to add stream level parameter. and `newpart` to | ||
|
r20856 | populate it. Then call `getchunks` to retrieve all the binary chunks of | ||
|
r21024 | data that compose the bundle2 container.""" | ||
|
r20801 | |||
|
r21134 | def __init__(self, ui, capabilities=()): | ||
|
r20842 | self.ui = ui | ||
|
r20801 | self._params = [] | ||
self._parts = [] | ||||
|
r21136 | self.capabilities = dict(capabilities) | ||
|
r20801 | |||
|
r21597 | # methods used to defines the bundle2 content | ||
|
r20804 | def addparam(self, name, value=None): | ||
"""add a stream level parameter""" | ||||
|
r20813 | if not name: | ||
raise ValueError('empty parameter name') | ||||
|
r20814 | if name[0] not in string.letters: | ||
raise ValueError('non letter first character: %r' % name) | ||||
|
r20804 | self._params.append((name, value)) | ||
|
r20856 | def addpart(self, part): | ||
"""add a new part to the bundle2 container | ||||
|
r21024 | Parts contains the actual applicative payload.""" | ||
|
r20995 | assert part.id is None | ||
part.id = len(self._parts) # very cheap counter | ||||
|
r20856 | self._parts.append(part) | ||
|
r21598 | def newpart(self, typeid, *args, **kwargs): | ||
|
r21602 | """create a new part and add it to the containers | ||
As the part is directly added to the containers. For now, this means | ||||
that any failure to properly initialize the part after calling | ||||
``newpart`` should result in a failure of the whole bundling process. | ||||
You can still fall back to manually create and add if you need better | ||||
control.""" | ||||
|
r21598 | part = bundlepart(typeid, *args, **kwargs) | ||
|
r21599 | self.addpart(part) | ||
|
r21598 | return part | ||
|
r21597 | # methods used to generate the bundle2 stream | ||
|
r20801 | def getchunks(self): | ||
|
r20842 | self.ui.debug('start emission of %s stream\n' % _magicstring) | ||
|
r20801 | yield _magicstring | ||
|
r20804 | param = self._paramchunk() | ||
|
r20842 | self.ui.debug('bundle parameter: %s\n' % param) | ||
|
r20804 | yield _pack(_fstreamparamsize, len(param)) | ||
if param: | ||||
yield param | ||||
|
r20856 | self.ui.debug('start of parts\n') | ||
for part in self._parts: | ||||
self.ui.debug('bundle part: "%s"\n' % part.type) | ||||
for chunk in part.getchunks(): | ||||
yield chunk | ||||
|
r20842 | self.ui.debug('end of bundle\n') | ||
|
r20801 | yield '\0\0' | ||
|
r20802 | |||
|
r20804 | def _paramchunk(self): | ||
"""return a encoded version of all stream parameters""" | ||||
blocks = [] | ||||
|
r20809 | for par, value in self._params: | ||
|
r20811 | par = urllib.quote(par) | ||
|
r20809 | if value is not None: | ||
|
r20811 | value = urllib.quote(value) | ||
|
r20809 | par = '%s=%s' % (par, value) | ||
blocks.append(par) | ||||
|
r20804 | return ' '.join(blocks) | ||
|
r21013 | class unpackermixin(object): | ||
"""A mixin to extract bytes and struct data from a stream""" | ||||
|
r20802 | |||
|
r21013 | def __init__(self, fp): | ||
|
r20802 | self._fp = fp | ||
def _unpack(self, format): | ||||
"""unpack this struct format from the stream""" | ||||
data = self._readexact(struct.calcsize(format)) | ||||
return _unpack(format, data) | ||||
def _readexact(self, size): | ||||
"""read exactly <size> bytes from the stream""" | ||||
return changegroup.readexactly(self._fp, size) | ||||
|
r21013 | |||
class unbundle20(unpackermixin): | ||||
"""interpret a bundle2 stream | ||||
|
r21129 | This class is fed with a binary stream and yields parts through its | ||
`iterparts` methods.""" | ||||
|
r21013 | |||
|
r21066 | def __init__(self, ui, fp, header=None): | ||
"""If header is specified, we do not read it out of the stream.""" | ||||
|
r21013 | self.ui = ui | ||
super(unbundle20, self).__init__(fp) | ||||
|
r21066 | if header is None: | ||
header = self._readexact(4) | ||||
magic, version = header[0:2], header[2:4] | ||||
if magic != 'HG': | ||||
raise util.Abort(_('not a Mercurial bundle')) | ||||
|
r21144 | if version != '2X': | ||
|
r21066 | raise util.Abort(_('unknown bundle version %s') % version) | ||
|
r21013 | self.ui.debug('start processing of %s stream\n' % header) | ||
|
r20802 | @util.propertycache | ||
def params(self): | ||||
|
r21024 | """dictionary of stream level parameters""" | ||
|
r20843 | self.ui.debug('reading bundle2 stream parameters\n') | ||
|
r20805 | params = {} | ||
paramssize = self._unpack(_fstreamparamsize)[0] | ||||
if paramssize: | ||||
for p in self._readexact(paramssize).split(' '): | ||||
|
r20810 | p = p.split('=', 1) | ||
|
r20812 | p = [urllib.unquote(i) for i in p] | ||
|
r20810 | if len(p) < 2: | ||
p.append(None) | ||||
|
r20844 | self._processparam(*p) | ||
|
r20810 | params[p[0]] = p[1] | ||
|
r20805 | return params | ||
|
r20802 | |||
|
r20844 | def _processparam(self, name, value): | ||
"""process a parameter, applying its effect if needed | ||||
Parameter starting with a lower case letter are advisory and will be | ||||
ignored when unknown. Those starting with an upper case letter are | ||||
mandatory and will this function will raise a KeyError when unknown. | ||||
Note: no option are currently supported. Any input will be either | ||||
ignored or failing. | ||||
""" | ||||
if not name: | ||||
raise ValueError('empty parameter name') | ||||
if name[0] not in string.letters: | ||||
raise ValueError('non letter first character: %r' % name) | ||||
# Some logic will be later added here to try to process the option for | ||||
# a dict of known parameter. | ||||
if name[0].islower(): | ||||
self.ui.debug("ignoring unknown parameter %r\n" % name) | ||||
else: | ||||
raise KeyError(name) | ||||
|
r21129 | def iterparts(self): | ||
|
r20802 | """yield all parts contained in the stream""" | ||
# make sure param have been loaded | ||||
self.params | ||||
|
r20843 | self.ui.debug('start extraction of bundle2 parts\n') | ||
|
r21014 | headerblock = self._readpartheader() | ||
while headerblock is not None: | ||||
part = unbundlepart(self.ui, headerblock, self._fp) | ||||
|
r20802 | yield part | ||
|
r21014 | headerblock = self._readpartheader() | ||
|
r20843 | self.ui.debug('end of bundle2 stream\n') | ||
|
r20802 | |||
|
r21014 | def _readpartheader(self): | ||
"""reads a part header size and return the bytes blob | ||||
|
r20864 | |||
|
r21014 | returns None if empty""" | ||
|
r20864 | headersize = self._unpack(_fpartheadersize)[0] | ||
self.ui.debug('part header size: %i\n' % headersize) | ||||
|
r21014 | if headersize: | ||
return self._readexact(headersize) | ||||
return None | ||||
|
r20864 | |||
|
r20802 | |||
|
r21005 | class bundlepart(object): | ||
|
r20856 | """A bundle2 part contains application level payload | ||
The part `type` is used to route the part to the application level | ||||
handler. | ||||
|
r21604 | |||
The part payload is contained in ``part.data``. It could be raw bytes or a | ||||
|
r21605 | generator of byte chunks. | ||
You can add parameters to the part using the ``addparam`` method. | ||||
Parameters can be either mandatory (default) or advisory. Remote side | ||||
should be able to safely ignore the advisory ones. | ||||
Both data and parameters cannot be modified after the generation has begun. | ||||
|
r20856 | """ | ||
|
r20877 | def __init__(self, parttype, mandatoryparams=(), advisoryparams=(), | ||
data=''): | ||||
|
r20995 | self.id = None | ||
|
r20856 | self.type = parttype | ||
|
r21604 | self._data = data | ||
|
r21605 | self._mandatoryparams = list(mandatoryparams) | ||
self._advisoryparams = list(advisoryparams) | ||||
|
r21607 | # checking for duplicated entries | ||
self._seenparams = set() | ||||
for pname, __ in self._mandatoryparams + self._advisoryparams: | ||||
if pname in self._seenparams: | ||||
raise RuntimeError('duplicated params: %s' % pname) | ||||
self._seenparams.add(pname) | ||||
|
r21601 | # status of the part's generation: | ||
# - None: not started, | ||||
# - False: currently generated, | ||||
# - True: generation done. | ||||
self._generated = None | ||||
|
r20856 | |||
|
r21604 | # methods used to defines the part content | ||
def __setdata(self, data): | ||||
if self._generated is not None: | ||||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21604 | self._data = data | ||
def __getdata(self): | ||||
return self._data | ||||
data = property(__getdata, __setdata) | ||||
|
r21605 | @property | ||
def mandatoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._mandatoryparams) | ||||
@property | ||||
def advisoryparams(self): | ||||
# make it an immutable tuple to force people through ``addparam`` | ||||
return tuple(self._advisoryparams) | ||||
def addparam(self, name, value='', mandatory=True): | ||||
if self._generated is not None: | ||||
|
r21618 | raise error.ReadOnlyPartError('part is being generated') | ||
|
r21607 | if name in self._seenparams: | ||
raise ValueError('duplicated params: %s' % name) | ||||
self._seenparams.add(name) | ||||
|
r21605 | params = self._advisoryparams | ||
if mandatory: | ||||
params = self._mandatoryparams | ||||
params.append((name, value)) | ||||
|
r21601 | # methods used to generates the bundle2 stream | ||
|
r20856 | def getchunks(self): | ||
|
r21601 | if self._generated is not None: | ||
raise RuntimeError('part can only be consumed once') | ||||
self._generated = False | ||||
|
r20877 | #### header | ||
## parttype | ||||
|
r20856 | header = [_pack(_fparttypesize, len(self.type)), | ||
|
r20995 | self.type, _pack(_fpartid, self.id), | ||
|
r20856 | ] | ||
|
r20877 | ## parameters | ||
# count | ||||
manpar = self.mandatoryparams | ||||
advpar = self.advisoryparams | ||||
header.append(_pack(_fpartparamcount, len(manpar), len(advpar))) | ||||
# size | ||||
parsizes = [] | ||||
for key, value in manpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
for key, value in advpar: | ||||
parsizes.append(len(key)) | ||||
parsizes.append(len(value)) | ||||
paramsizes = _pack(_makefpartparamsizes(len(parsizes) / 2), *parsizes) | ||||
header.append(paramsizes) | ||||
# key, value | ||||
for key, value in manpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
for key, value in advpar: | ||||
header.append(key) | ||||
header.append(value) | ||||
## finalize header | ||||
|
r20856 | headerchunk = ''.join(header) | ||
yield _pack(_fpartheadersize, len(headerchunk)) | ||||
yield headerchunk | ||||
|
r20877 | ## payload | ||
|
r21000 | for chunk in self._payloadchunks(): | ||
yield _pack(_fpayloadsize, len(chunk)) | ||||
yield chunk | ||||
# end of payload | ||||
yield _pack(_fpayloadsize, 0) | ||||
|
r21601 | self._generated = True | ||
|
r21000 | |||
def _payloadchunks(self): | ||||
"""yield chunks of a the part payload | ||||
Exists to handle the different methods to provide data to a part.""" | ||||
|
r20876 | # we only support fixed size data now. | ||
# This will be improved in the future. | ||||
|
r21001 | if util.safehasattr(self.data, 'next'): | ||
buff = util.chunkbuffer(self.data) | ||||
chunk = buff.read(preferedchunksize) | ||||
while chunk: | ||||
yield chunk | ||||
chunk = buff.read(preferedchunksize) | ||||
elif len(self.data): | ||||
|
r20876 | yield self.data | ||
|
r20802 | |||
|
r21014 | class unbundlepart(unpackermixin): | ||
"""a bundle part read from a bundle""" | ||||
def __init__(self, ui, header, fp): | ||||
super(unbundlepart, self).__init__(fp) | ||||
self.ui = ui | ||||
# unbundle state attr | ||||
self._headerdata = header | ||||
|
r21015 | self._headeroffset = 0 | ||
|
r21019 | self._initialized = False | ||
self.consumed = False | ||||
|
r21014 | # part data | ||
self.id = None | ||||
self.type = None | ||||
self.mandatoryparams = None | ||||
self.advisoryparams = None | ||||
|
r21610 | self.params = None | ||
|
r21612 | self.mandatorykeys = () | ||
|
r21019 | self._payloadstream = None | ||
self._readheader() | ||||
|
r21014 | |||
|
r21015 | def _fromheader(self, size): | ||
"""return the next <size> byte from the header""" | ||||
offset = self._headeroffset | ||||
data = self._headerdata[offset:(offset + size)] | ||||
|
r21019 | self._headeroffset = offset + size | ||
|
r21015 | return data | ||
|
r21016 | def _unpackheader(self, format): | ||
"""read given format from header | ||||
This automatically compute the size of the format to read.""" | ||||
data = self._fromheader(struct.calcsize(format)) | ||||
return _unpack(format, data) | ||||
|
r21608 | def _initparams(self, mandatoryparams, advisoryparams): | ||
"""internal function to setup all logic related parameters""" | ||||
|
r21609 | # make it read only to prevent people touching it by mistake. | ||
self.mandatoryparams = tuple(mandatoryparams) | ||||
self.advisoryparams = tuple(advisoryparams) | ||||
|
r21610 | # user friendly UI | ||
self.params = dict(self.mandatoryparams) | ||||
self.params.update(dict(self.advisoryparams)) | ||||
|
r21612 | self.mandatorykeys = frozenset(p[0] for p in mandatoryparams) | ||
|
r21608 | |||
|
r21019 | def _readheader(self): | ||
|
r21014 | """read the header and setup the object""" | ||
|
r21016 | typesize = self._unpackheader(_fparttypesize)[0] | ||
|
r21015 | self.type = self._fromheader(typesize) | ||
|
r21014 | self.ui.debug('part type: "%s"\n' % self.type) | ||
|
r21016 | self.id = self._unpackheader(_fpartid)[0] | ||
|
r21014 | self.ui.debug('part id: "%s"\n' % self.id) | ||
## reading parameters | ||||
# param count | ||||
|
r21016 | mancount, advcount = self._unpackheader(_fpartparamcount) | ||
|
r21014 | self.ui.debug('part parameters: %i\n' % (mancount + advcount)) | ||
# param size | ||||
|
r21016 | fparamsizes = _makefpartparamsizes(mancount + advcount) | ||
paramsizes = self._unpackheader(fparamsizes) | ||||
|
r21014 | # make it a list of couple again | ||
paramsizes = zip(paramsizes[::2], paramsizes[1::2]) | ||||
# split mandatory from advisory | ||||
mansizes = paramsizes[:mancount] | ||||
advsizes = paramsizes[mancount:] | ||||
# retrive param value | ||||
manparams = [] | ||||
for key, value in mansizes: | ||||
|
r21015 | manparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21014 | advparams = [] | ||
for key, value in advsizes: | ||||
|
r21015 | advparams.append((self._fromheader(key), self._fromheader(value))) | ||
|
r21608 | self._initparams(manparams, advparams) | ||
|
r21014 | ## part payload | ||
|
r21019 | def payloadchunks(): | ||
|
r21014 | payloadsize = self._unpack(_fpayloadsize)[0] | ||
self.ui.debug('payload chunk size: %i\n' % payloadsize) | ||||
|
r21019 | while payloadsize: | ||
yield self._readexact(payloadsize) | ||||
payloadsize = self._unpack(_fpayloadsize)[0] | ||||
self.ui.debug('payload chunk size: %i\n' % payloadsize) | ||||
self._payloadstream = util.chunkbuffer(payloadchunks()) | ||||
# we read the data, tell it | ||||
self._initialized = True | ||||
def read(self, size=None): | ||||
"""read payload data""" | ||||
if not self._initialized: | ||||
self._readheader() | ||||
if size is None: | ||||
data = self._payloadstream.read() | ||||
else: | ||||
data = self._payloadstream.read(size) | ||||
if size is None or len(data) < size: | ||||
self.consumed = True | ||||
return data | ||||
|
r21014 | |||
|
r21146 | @parthandler('b2x:changegroup') | ||
|
r20998 | def handlechangegroup(op, inpart): | ||
|
r20950 | """apply a changegroup part on the repo | ||
This is a very early implementation that will massive rework before being | ||||
inflicted to any end-user. | ||||
""" | ||||
|
r20952 | # Make sure we trigger a transaction creation | ||
# | ||||
# The addchangegroup function will get a transaction object by itself, but | ||||
# we need to make sure we trigger the creation of a transaction object used | ||||
# for the whole processing scope. | ||||
op.gettransaction() | ||||
|
r21062 | cg = changegroup.unbundle10(inpart, 'UN') | ||
|
r20950 | ret = changegroup.addchangegroup(op.repo, cg, 'bundle2', 'bundle2') | ||
op.records.add('changegroup', {'return': ret}) | ||||
|
r20998 | if op.reply is not None: | ||
# This is definitly not the final form of this | ||||
# return. But one need to start somewhere. | ||||
|
r21606 | part = op.reply.newpart('b2x:reply:changegroup') | ||
part.addparam('in-reply-to', str(inpart.id), mandatory=False) | ||||
part.addparam('return', '%i' % ret, mandatory=False) | ||||
|
r21019 | assert not inpart.read() | ||
|
r20950 | |||
|
r21624 | @parthandler('b2x:reply:changegroup', ('return', 'in-reply-to')) | ||
|
r20998 | def handlechangegroup(op, inpart): | ||
|
r21611 | ret = int(inpart.params['return']) | ||
replyto = int(inpart.params['in-reply-to']) | ||||
op.records.add('changegroup', {'return': ret}, replyto) | ||||
|
r20950 | |||
|
r21146 | @parthandler('b2x:check:heads') | ||
|
r21060 | def handlechangegroup(op, inpart): | ||
"""check that head of the repo did not change | ||||
This is used to detect a push race when using unbundle. | ||||
This replaces the "heads" argument of unbundle.""" | ||||
h = inpart.read(20) | ||||
heads = [] | ||||
while len(h) == 20: | ||||
heads.append(h) | ||||
h = inpart.read(20) | ||||
assert not h | ||||
if heads != op.repo.heads(): | ||||
|
r21185 | raise error.PushRaced('repository changed while pushing - ' | ||
'please try again') | ||||
|
r21130 | |||
|
r21146 | @parthandler('b2x:output') | ||
|
r21131 | def handleoutput(op, inpart): | ||
"""forward output captured on the server to the client""" | ||||
for line in inpart.read().splitlines(): | ||||
op.ui.write(('remote: %s\n' % line)) | ||||
|
r21146 | @parthandler('b2x:replycaps') | ||
|
r21130 | def handlereplycaps(op, inpart): | ||
"""Notify that a reply bundle should be created | ||||
|
r21138 | The payload contains the capabilities information for the reply""" | ||
caps = decodecaps(inpart.read()) | ||||
|
r21130 | if op.reply is None: | ||
|
r21135 | op.reply = bundle20(op.ui, caps) | ||
|
r21131 | |||
|
r21624 | @parthandler('b2x:error:abort', ('message', 'hint')) | ||
|
r21177 | def handlereplycaps(op, inpart): | ||
"""Used to transmit abort error over the wire""" | ||||
|
r21611 | raise util.Abort(inpart.params['message'], hint=inpart.params.get('hint')) | ||
|
r21183 | |||
|
r21624 | @parthandler('b2x:error:unsupportedcontent', ('parttype', 'params')) | ||
|
r21183 | def handlereplycaps(op, inpart): | ||
|
r21619 | """Used to transmit unknown content error over the wire""" | ||
|
r21622 | kwargs = {} | ||
kwargs['parttype'] = inpart.params['parttype'] | ||||
params = inpart.params.get('params') | ||||
if params is not None: | ||||
kwargs['params'] = params.split('\0') | ||||
raise error.BundleValueError(**kwargs) | ||||
|
r21186 | |||
|
r21623 | @parthandler('b2x:error:pushraced', ('message',)) | ||
|
r21186 | def handlereplycaps(op, inpart): | ||
"""Used to transmit push race error over the wire""" | ||||
|
r21611 | raise error.ResponseError(_('push failed:'), inpart.params['message']) | ||