##// END OF EJS Templates
dirstate: move dropping of folded filenames into the dirstate map...
dirstate: move dropping of folded filenames into the dirstate map When dropping files from the dirstate, the corresponding entry in the filefoldmap is also dropped. Move this into the dirstate map object. A future implementation of the dirstate will maintain the filefoldmap differently. Differential Revision: https://phab.mercurial-scm.org/D1343

File last commit:

r35037:241d9cac default
r35081:e6c64744 default
Show More
bundle2.py
1946 lines | 70.6 KiB | text/x-python | PythonLexer
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 # bundle2.py - generic container format to transmit arbitrary data.
#
# Copyright 2013 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""Handling of the new bundle2 format
The goal of bundle2 is to act as an atomically packet to transmit a set of
payloads in an application agnostic way. It consist in a sequence of "parts"
that will be handed to and processed by the application layer.
General format architecture
===========================
The format is architectured as follow
- magic string
- stream level parameters
- payload parts (any number)
- end of stream marker.
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 the Binary format
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 ============================
Mads Kiilerich
spelling: fixes from spell checker
r21024 All numbers are unsigned and big-endian.
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
stream level parameters
------------------------
Binary format is as follow
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 :params size: int32
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
The total number of Bytes used by the parameters
:params value: arbitrary number of Bytes
A blob of `params size` containing the serialized version of all stream level
parameters.
Mads Kiilerich
spelling: fixes from spell checker
r21024 The blob contains a space separated list of parameters. Parameters with value
Pierre-Yves David
bundle2: urlquote stream parameter name and value...
r20811 are stored in the form `<name>=<value>`. Both name and value are urlquoted.
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804
Pierre-Yves David
bundle2: refuse empty parameter name...
r20813 Empty name are obviously forbidden.
Pierre-Yves David
bundle2: implement the mandatory/advisory logic for parameter...
r20844 Name MUST start with a letter. If this first letter is lower case, the
Mads Kiilerich
spelling: fixes from spell checker
r21024 parameter is advisory and can be safely ignored. However when the first
Pierre-Yves David
bundle2: implement the mandatory/advisory logic for parameter...
r20844 letter is capital, the parameter is mandatory and the bundling process MUST
stop if he is not able to proceed it.
Pierre-Yves David
bundle2: force the first char of parameter to be an letter....
r20814
Pierre-Yves David
bundle2: clarify stream parameter design in the documentation...
r20808 Stream parameters use a simple textual format for two main reasons:
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804
Mads Kiilerich
spelling: fixes from spell checker
r21024 - Stream level parameters should remain simple and we want to discourage any
Pierre-Yves David
bundle2: clarify stream parameter design in the documentation...
r20808 crazy usage.
Mads Kiilerich
spelling: fixes from spell checker
r21024 - Textual data allow easy human inspection of a bundle2 header in case of
Pierre-Yves David
bundle2: clarify stream parameter design in the documentation...
r20808 troubles.
Any Applicative level options MUST go into a bundle2 part instead.
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
Payload part
------------------------
Binary format is as follow
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 :header size: int32
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
Martin von Zweigbergk
bundle2: clarify in docstring that header size is for a single header...
r25507 The total number of Bytes used by the part header. When the header is empty
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 (size = 0) this is interpreted as the end of stream marker.
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 :header:
The header defines how to interpret the part. It contains two piece of
data: the part type, and the part parameters.
The part type is used to route an application level handler, that can
interpret payload.
Part parameters are passed to the application level handler. They are
meant to convey information that will help the application level object to
interpret the part payload.
The binary format of the header is has follow
:typesize: (one byte)
Pierre-Yves David
bundle2: part params
r20877
Matt Mackall
bundle2: fix parttype enforcement...
r23916 :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*)
Pierre-Yves David
bundle2: part params
r20877
Pierre-Yves David
bundle2: add an integer id to part...
r20995 :partid: A 32bits integer (unique in the bundle) that can be used to refer
to this part.
Pierre-Yves David
bundle2: part params
r20877 :parameters:
Mads Kiilerich
spelling: fixes from spell checker
r21024 Part's parameter may have arbitrary content, the binary structure is::
Pierre-Yves David
bundle2: part params
r20877
<mandatory-count><advisory-count><param-sizes><param-data>
:mandatory-count: 1 byte, number of mandatory parameters
:advisory-count: 1 byte, number of advisory parameters
:param-sizes:
N couple of bytes, where N is the total number of parameters. Each
couple contains (<size-of-key>, <size-of-value) for one parameter.
:param-data:
A blob of bytes from which each parameter key and value can be
retrieved using the list of size couples stored in the previous
field.
Mandatory parameters comes first, then the advisory ones.
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856
Pierre-Yves David
bundle2: forbid duplicate parameter keys...
r21607 Each parameter's key MUST be unique within the part.
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 :payload:
Pierre-Yves David
bundle2: support for bundling and unbundling payload...
r20876 payload is a series of `<chunksize><chunkdata>`.
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 `chunksize` is an int32, `chunkdata` are plain bytes (as much as
Pierre-Yves David
bundle2: support for bundling and unbundling payload...
r20876 `chunksize` says)` The payload part is concluded by a zero size chunk.
The current implementation always produces either zero or one chunk.
Mads Kiilerich
spelling: fixes from spell checker
r21024 This is an implementation limitation that will ultimately be lifted.
Pierre-Yves David
bundle2: add some distinction between mandatory and advisory part...
r20891
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 `chunksize` can be negative to trigger special case processing. No such
processing is in place yet.
Pierre-Yves David
bundle2: add some distinction between mandatory and advisory part...
r20891 Bundle processing
============================
Each part is processed in order using a "part handler". Handler are registered
for a certain part type.
The matching of a part to its handler is case insensitive. The case of the
part type is used to know if a part is mandatory or advisory. If the Part type
contains any uppercase char it is considered mandatory. When no handler is
known for a Mandatory part, the process is aborted and an exception is raised.
Pierre-Yves David
bundle2: read the whole bundle from stream on abort...
r20892 If the part is advisory and no handler is known, the part is ignored. When the
process is aborted, the full bundle is still read from the stream to keep the
channel usable. But none of the part read from an abort are processed. In the
future, dropping the stream may become an option for channel we do not care to
preserve.
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 """
Augie Fackler
bundle2: use modern Python division...
r33635 from __future__ import absolute_import, division
Gregory Szorc
bundle2: use absolute_import
r25919
Eric Sumner
bundle2.unpackermixin: control for underlying file descriptor...
r24026 import errno
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 import os
Gregory Szorc
bundle2: use absolute_import
r25919 import re
Pierre-Yves David
bundle2: force the first char of parameter to be an letter....
r20814 import string
Gregory Szorc
bundle2: use absolute_import
r25919 import struct
import sys
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804
Gregory Szorc
bundle2: use absolute_import
r25919 from .i18n import _
from . import (
changegroup,
error,
Boris Feld
phase: introduce a new 'check:phases' part...
r34821 node as nodemod,
Gregory Szorc
bundle2: use absolute_import
r25919 obsolete,
Martin von Zweigbergk
bundle: add config option to include phases...
r33031 phases,
Gregory Szorc
bundle2: use absolute_import
r25919 pushkey,
Yuya Nishihara
pycompat: provide 'ispy3' constant...
r30030 pycompat,
Gregory Szorc
bundle2: use absolute_import
r25919 tags,
url,
util,
)
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 urlerr = util.urlerr
urlreq = util.urlreq
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 _pack = struct.pack
_unpack = struct.unpack
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 _fstreamparamsize = '>i'
_fpartheadersize = '>i'
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 _fparttypesize = '>B'
Pierre-Yves David
bundle2: add an integer id to part...
r20995 _fpartid = '>I'
Pierre-Yves David
bundle2: change header size and make them signed (new format)...
r23009 _fpayloadsize = '>i'
Pierre-Yves David
bundle2: part params
r20877 _fpartparamcount = '>BB'
Pierre-Yves David
bundle2: support chunk iterator as part data...
r21001 preferedchunksize = 4096
Pierre-Yves David
bundle2: enforce parttype as alphanumerical...
r23868 _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
Pierre-Yves David
bundle2: introduce a specific function for bundling debug message...
r25313 def outdebug(ui, message):
"""debug regarding output stream (bundling)"""
config: register the 'devel.bundle2.debug' config...
r33160 if ui.configbool('devel', 'bundle2.debug'):
Pierre-Yves David
bundle2: hide bundle2 stream debug under a config flag...
r25336 ui.debug('bundle2-output: %s\n' % message)
Pierre-Yves David
bundle2: introduce a specific function for bundling debug message...
r25313
Pierre-Yves David
bundle2: introduce a specific function for debug messages while unbundling...
r25318 def indebug(ui, message):
"""debug on input stream (unbundling)"""
config: register the 'devel.bundle2.debug' config...
r33160 if ui.configbool('devel', 'bundle2.debug'):
Pierre-Yves David
bundle2: hide bundle2 stream debug under a config flag...
r25336 ui.debug('bundle2-input: %s\n' % message)
Pierre-Yves David
bundle2: introduce a specific function for debug messages while unbundling...
r25318
Pierre-Yves David
bundle2: enforce parttype as alphanumerical...
r23868 def validateparttype(parttype):
"""raise ValueError if a parttype contains invalid character"""
Matt Mackall
bundle2: fix parttype enforcement...
r23916 if _parttypeforbidden.search(parttype):
Pierre-Yves David
bundle2: enforce parttype as alphanumerical...
r23868 raise ValueError(parttype)
Pierre-Yves David
bundle2: part params
r20877 def _makefpartparamsizes(nbparams):
"""return a struct format to read part parameter sizes
The number parameters is variable so we need to build that format
dynamically.
"""
return '>'+('BB'*nbparams)
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804
Pierre-Yves David
bundle2: introduce a `parthandler` decorator...
r20890 parthandlermapping = {}
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889
Pierre-Yves David
bundle2: make it possible to declare params handled by a part handler...
r21623 def parthandler(parttype, params=()):
Pierre-Yves David
bundle2: introduce a `parthandler` decorator...
r20890 """decorator that register a function as a bundle2 part handler
eg::
Pierre-Yves David
bundle2: declare supported parameters for all handlers...
r21624 @parthandler('myparttype', ('mandatory', 'param', 'handled'))
Pierre-Yves David
bundle2: introduce a `parthandler` decorator...
r20890 def myparttypehandler(...):
'''process a part of type "my part".'''
...
"""
Pierre-Yves David
bundle2: enforce parttype as alphanumerical...
r23868 validateparttype(parttype)
Pierre-Yves David
bundle2: introduce a `parthandler` decorator...
r20890 def _decorator(func):
Pierre-Yves David
bundle2: add some distinction between mandatory and advisory part...
r20891 lparttype = parttype.lower() # enforce lower case matching.
assert lparttype not in parthandlermapping
parthandlermapping[lparttype] = func
Pierre-Yves David
bundle2: make it possible to declare params handled by a part handler...
r21623 func.params = frozenset(params)
Pierre-Yves David
bundle2: introduce a `parthandler` decorator...
r20890 return func
return _decorator
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949 class unbundlerecords(object):
"""keep record of what happens during and unbundle
New records are added using `records.add('cat', obj)`. Where 'cat' is a
Mads Kiilerich
spelling: fixes from spell checker
r21024 category of record and obj is an arbitrary object.
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949
`records['cat']` will return all entries of this category 'cat'.
Iterating on the object itself will yield `('category', obj)` tuples
for all entries.
All iterations happens in chronological order.
"""
def __init__(self):
self._categories = {}
self._sequences = []
Pierre-Yves David
bundle2: add reply awareness to unbundlerecords...
r20996 self._replies = {}
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949
Pierre-Yves David
bundle2: add reply awareness to unbundlerecords...
r20996 def add(self, category, entry, inreplyto=None):
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949 """add a new record of a given category.
The entry can then be retrieved in the list returned by
self['category']."""
self._categories.setdefault(category, []).append(entry)
self._sequences.append((category, entry))
Pierre-Yves David
bundle2: add reply awareness to unbundlerecords...
r20996 if inreplyto is not None:
self.getreplies(inreplyto).add(category, entry)
def getreplies(self, partid):
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 """get the records that are replies to a specific part"""
Pierre-Yves David
bundle2: add reply awareness to unbundlerecords...
r20996 return self._replies.setdefault(partid, unbundlerecords())
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949
def __getitem__(self, cat):
return tuple(self._categories.get(cat, ()))
def __iter__(self):
return iter(self._sequences)
def __len__(self):
return len(self._sequences)
def __nonzero__(self):
return bool(self._sequences)
Gregory Szorc
py3: add __bool__ to every class defining __nonzero__...
r31476 __bool__ = __nonzero__
Pierre-Yves David
bundle2: introduce a bundleoperation object...
r20948 class bundleoperation(object):
"""an object that represents a single bundling process
Its purpose is to carry unbundle-related objects and states.
A new object should be created at the beginning of each bundle processing.
The object is to be returned by the processing function.
The object has very little content now it will ultimately contain:
* an access to the repo the bundle is applied to,
* a ui object,
* a way to retrieve a transaction to add changes to the repo,
* a way to record the result of processing each part,
* a way to construct a bundle response when applicable.
"""
Pierre-Yves David
bundle2: disable ouput capture unless we use http (issue4613 issue4615)...
r24878 def __init__(self, repo, transactiongetter, captureoutput=True):
Pierre-Yves David
bundle2: introduce a bundleoperation object...
r20948 self.repo = repo
self.ui = repo.ui
Pierre-Yves David
bundle2: record processing results in the bundleoperation object...
r20949 self.records = unbundlerecords()
Pierre-Yves David
bundle2: produce a bundle2 reply...
r20997 self.reply = None
Pierre-Yves David
bundle2: disable ouput capture unless we use http (issue4613 issue4615)...
r24878 self.captureoutput = captureoutput
Pulkit Goyal
bundle2: add the capability to store hookargs on bundle operation object...
r33629 self.hookargs = {}
Pulkit Goyal
bundle2: load hookargs from bundleoperation into transaction when started one...
r33630 self._gettransaction = transactiongetter
def gettransaction(self):
transaction = self._gettransaction()
Yuya Nishihara
bundle2: relax the condition to update transaction.hookargs...
r33809 if self.hookargs:
Pulkit Goyal
bundle2: load hookargs from bundleoperation into transaction when started one...
r33630 # the ones added to the transaction supercede those added
# to the operation.
self.hookargs.update(transaction.hookargs)
transaction.hookargs = self.hookargs
Yuya Nishihara
bundle2: relax the condition to update transaction.hookargs...
r33809 # mark the hookargs as flushed. further attempts to add to
# hookargs will result in an abort.
self.hookargs = None
Pulkit Goyal
bundle2: load hookargs from bundleoperation into transaction when started one...
r33630
return transaction
Pulkit Goyal
bundle2: add the capability to store hookargs on bundle operation object...
r33629
def addhookargs(self, hookargs):
Pulkit Goyal
bundle2: load hookargs from bundleoperation into transaction when started one...
r33630 if self.hookargs is None:
Yuya Nishihara
bundle2: raise ProgrammingError for invalid call of addhookargs()...
r33808 raise error.ProgrammingError('attempted to add hookargs to '
'operation after transaction started')
Pulkit Goyal
bundle2: add the capability to store hookargs on bundle operation object...
r33629 self.hookargs.update(hookargs)
Pierre-Yves David
bundle2: introduce a bundleoperation object...
r20948
Pierre-Yves David
bundle2: make it possible have a global transaction for the unbundling...
r20952 class TransactionUnavailable(RuntimeError):
pass
def _notransaction():
"""default method to get a transaction while processing a bundle
Raise an exception to highlight the fact that no transaction was expected
to be created"""
raise TransactionUnavailable()
Martin von Zweigbergk
bundle: make applybundle() delegate v1 bundles to applybundle1()
r33043 def applybundle(repo, unbundler, tr, source=None, url=None, **kwargs):
Pierre-Yves David
bundle2: introduce an "applybundle" function...
r26790 # transform me into unbundler.apply() as soon as the freeze is lifted
Martin von Zweigbergk
bundle: make applybundle() delegate v1 bundles to applybundle1()
r33043 if isinstance(unbundler, unbundle20):
tr.hookargs['bundle2'] = '1'
if source is not None and 'source' not in tr.hookargs:
tr.hookargs['source'] = source
if url is not None and 'url' not in tr.hookargs:
tr.hookargs['url'] = url
return processbundle(repo, unbundler, lambda: tr)
else:
Martin von Zweigbergk
bundle: inline applybundle1()...
r33044 # the transactiongetter won't be used, but we might as well set it
op = bundleoperation(repo, lambda: tr)
_processchangegroup(op, unbundler, tr, source, url, **kwargs)
return op
Pierre-Yves David
bundle2: introduce an "applybundle" function...
r26790
Durham Goode
bundle2: move part iterator a separate class...
r34149 class partiterator(object):
Durham Goode
bundle2: move exception handling into part iterator...
r34151 def __init__(self, repo, op, unbundler):
Durham Goode
bundle2: move part counter to partiterator...
r34150 self.repo = repo
Durham Goode
bundle2: move exception handling into part iterator...
r34151 self.op = op
Durham Goode
bundle2: move part iterator a separate class...
r34149 self.unbundler = unbundler
Durham Goode
bundle2: move part counter to partiterator...
r34150 self.iterator = None
self.count = 0
Durham Goode
bundle2: move processpart stream maintenance into part iterator...
r34259 self.current = None
Durham Goode
bundle2: move part iterator a separate class...
r34149
def __enter__(self):
Durham Goode
bundle2: move part counter to partiterator...
r34150 def func():
itr = enumerate(self.unbundler.iterparts())
for count, p in itr:
self.count = count
Durham Goode
bundle2: move processpart stream maintenance into part iterator...
r34259 self.current = p
Durham Goode
bundle2: move part counter to partiterator...
r34150 yield p
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 p.seek(0, os.SEEK_END)
Durham Goode
bundle2: move processpart stream maintenance into part iterator...
r34259 self.current = None
Durham Goode
bundle2: move part counter to partiterator...
r34150 self.iterator = func()
return self.iterator
Durham Goode
bundle2: move part iterator a separate class...
r34149
Durham Goode
bundle2: move exception handling into part iterator...
r34151 def __exit__(self, type, exc, tb):
Durham Goode
bundle2: move part counter to partiterator...
r34150 if not self.iterator:
return
Durham Goode
bundle2: immediate exit for ctrl+c (issue5692)...
r34638 # Only gracefully abort in a normal exception situation. User aborts
# like Ctrl+C throw a KeyboardInterrupt which is not a base Exception,
# and should not gracefully cleanup.
if isinstance(exc, Exception):
Durham Goode
bundle2: move exception handling into part iterator...
r34151 # Any exceptions seeking to the end of the bundle at this point are
# almost certainly related to the underlying stream being bad.
# And, chances are that the exception we're handling is related to
# getting in that bad state. So, we swallow the seeking error and
# re-raise the original error.
seekerror = False
try:
Durham Goode
bundle2: immediate exit for ctrl+c (issue5692)...
r34638 if self.current:
# consume the part content to not corrupt the stream.
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 self.current.seek(0, os.SEEK_END)
Durham Goode
bundle2: immediate exit for ctrl+c (issue5692)...
r34638
Durham Goode
bundle2: move exception handling into part iterator...
r34151 for part in self.iterator:
# consume the bundle content
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 part.seek(0, os.SEEK_END)
Durham Goode
bundle2: move exception handling into part iterator...
r34151 except Exception:
seekerror = True
# Small hack to let caller code distinguish exceptions from bundle2
# processing from processing the old format. This is mostly needed
# to handle different return codes to unbundle according to the type
# of bundle. We should probably clean up or drop this return code
# craziness in a future version.
exc.duringunbundle2 = True
salvaged = []
replycaps = None
if self.op.reply is not None:
salvaged = self.op.reply.salvageoutput()
replycaps = self.op.reply.capabilities
exc._replycaps = replycaps
exc._bundle2salvagedoutput = salvaged
# Re-raising from a variable loses the original stack. So only use
# that form if we need to.
if seekerror:
raise exc
Durham Goode
bundle2: move part counter to partiterator...
r34150 self.repo.ui.debug('bundle2-input-bundle: %i parts total\n' %
self.count)
Durham Goode
bundle2: move part iterator a separate class...
r34149
Pierre-Yves David
bundle2: also save output when error happens during part processing...
r24851 def processbundle(repo, unbundler, transactiongetter=None, op=None):
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889 """This function process a bundle, apply effect to/from a repo
Pierre-Yves David
bundle2: feed a unbundle20 to the `processbundle` function...
r20947 It iterates over each part then searches for and uses the proper handling
code to process the part. Parts are processed in order.
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889
Pierre-Yves David
bundle2: add some distinction between mandatory and advisory part...
r20891 Unknown Mandatory part will abort the process.
Pierre-Yves David
bundle2: also save output when error happens during part processing...
r24851
It is temporarily possible to provide a prebuilt bundleoperation to the
function. This is used to ensure output is properly propagated in case of
an error during the unbundling. This output capturing part will likely be
reworked and this ability will probably go away in the process.
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889 """
Pierre-Yves David
bundle2: also save output when error happens during part processing...
r24851 if op is None:
if transactiongetter is None:
transactiongetter = _notransaction
op = bundleoperation(repo, transactiongetter)
Pierre-Yves David
bundle2: first version of a bundle processing...
r20889 # todo:
# - replace this is a init function soon.
# - exception catching
unbundler.params
Pierre-Yves David
bundle2: add generic debug output regarding processed bundle...
r25331 if repo.ui.debugflag:
msg = ['bundle2-input-bundle:']
if unbundler.params:
Siddharth Agarwal
bundle2: add debug info about the number of stream params...
r33123 msg.append(' %i params' % len(unbundler.params))
Boris Feld
bundle2: fix transaction availability detection...
r33765 if op._gettransaction is None or op._gettransaction is _notransaction:
Pierre-Yves David
bundle2: add generic debug output regarding processed bundle...
r25331 msg.append(' no-transaction')
else:
msg.append(' with-transaction')
msg.append('\n')
repo.ui.debug(''.join(msg))
Durham Goode
bundle2: move part iterator a separate class...
r34149
Durham Goode
bundle2: move part processing to a separate function...
r34262 processparts(repo, op, unbundler)
return op
def processparts(repo, op, unbundler):
Durham Goode
bundle2: move exception handling into part iterator...
r34151 with partiterator(repo, op, unbundler) as parts:
for part in parts:
_processpart(op, part)
Pierre-Yves David
bundle2: add generic debug output at the end of bundle processing...
r25332
Martin von Zweigbergk
bundle: extract _processchangegroup() method...
r33038 def _processchangegroup(op, cg, tr, source, url, **kwargs):
Boris Feld
changegroup: stop returning and recording added nodes in 'cg.apply'...
r33461 ret = cg.apply(op.repo, tr, source, url, **kwargs)
Martin von Zweigbergk
bundle: extract _processchangegroup() method...
r33038 op.records.add('changegroup', {
'return': ret,
})
return ret
Durham Goode
bundle2: move handler validation out of processpart...
r34260 def _gethandler(op, part):
status = 'unknown' # used by debug output
try:
handler = parthandlermapping.get(part.type)
if handler is None:
status = 'unsupported-type'
raise error.BundleUnknownFeatureError(parttype=part.type)
Augie Fackler
bundle2: stop using %r to quote part names...
r34278 indebug(op.ui, 'found a handler for part %s' % part.type)
Durham Goode
bundle2: move handler validation out of processpart...
r34260 unknownparams = part.mandatorykeys - handler.params
if unknownparams:
unknownparams = list(unknownparams)
unknownparams.sort()
Augie Fackler
bundle2: stop using %r to quote part names...
r34278 status = 'unsupported-params (%s)' % ', '.join(unknownparams)
Durham Goode
bundle2: move handler validation out of processpart...
r34260 raise error.BundleUnknownFeatureError(parttype=part.type,
params=unknownparams)
status = 'supported'
except error.BundleUnknownFeatureError as exc:
if part.mandatory: # mandatory parts
raise
indebug(op.ui, 'ignoring unsupported advisory part %s' % exc)
return # skip to part processing
finally:
if op.ui.debugflag:
msg = ['bundle2-input-part: "%s"' % part.type]
if not part.mandatory:
msg.append(' (advisory)')
nbmp = len(part.mandatorykeys)
nbap = len(part.params) - nbmp
if nbmp or nbap:
msg.append(' (params:')
if nbmp:
msg.append(' %i mandatory' % nbmp)
if nbap:
msg.append(' %i advisory' % nbmp)
msg.append(')')
msg.append(' %s\n' % status)
op.ui.debug(''.join(msg))
return handler
Pierre-Yves David
bundle2: extract processing of part into its own function...
r23008 def _processpart(op, part):
"""process a single part from a bundle
The part is guaranteed to have been fully consumed when the function exits
(even if an exception is raised)."""
Durham Goode
bundle2: remove unnecessary try finally...
r34261 handler = _gethandler(op, part)
if handler is None:
return
Pierre-Yves David
bundle2: extract processing of part into its own function...
r23008
Durham Goode
bundle2: remove unnecessary try finally...
r34261 # handler is called outside the above try block so that we don't
# risk catching KeyErrors from anything other than the
# parthandlermapping lookup (any KeyError raised by handler()
# itself represents a defect of a different variety).
output = None
if op.captureoutput and op.reply is not None:
op.ui.pushbuffer(error=True, subproc=True)
output = ''
try:
handler(op, part)
Pierre-Yves David
bundle2: extract processing of part into its own function...
r23008 finally:
Durham Goode
bundle2: remove unnecessary try finally...
r34261 if output is not None:
output = op.ui.popbuffer()
if output:
outpart = op.reply.newpart('output', data=output,
mandatory=False)
outpart.addparam(
'in-reply-to', pycompat.bytestr(part.id), mandatory=False)
Pierre-Yves David
bundle2: extract processing of part into its own function...
r23008
Pierre-Yves David
bundle2: extract capabilities decoding...
r21138 def decodecaps(blob):
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 """decode a bundle2 caps bytes blob into a dictionary
Pierre-Yves David
bundle2: extract capabilities decoding...
r21138
The blob is a list of capabilities (one per line)
Capabilities may have values using a line of the form::
capability=value1,value2,value3
The values are always a list."""
caps = {}
for line in blob.splitlines():
if not line:
continue
if '=' not in line:
key, vals = line, ()
else:
key, vals = line.split('=', 1)
vals = vals.split(',')
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 key = urlreq.unquote(key)
vals = [urlreq.unquote(v) for v in vals]
Pierre-Yves David
bundle2: extract capabilities decoding...
r21138 caps[key] = vals
return caps
Pierre-Yves David
bundle2: capabilities encoding
r21139 def encodecaps(caps):
"""encode a bundle2 caps dictionary into a bytes blob"""
chunks = []
for ca in sorted(caps):
vals = caps[ca]
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 ca = urlreq.quote(ca)
vals = [urlreq.quote(v) for v in vals]
Pierre-Yves David
bundle2: capabilities encoding
r21139 if vals:
ca = "%s=%s" % (ca, ','.join(vals))
chunks.append(ca)
return '\n'.join(chunks)
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 bundletypes = {
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 "": ("", 'UN'), # only when using unbundle on ssh and old http servers
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 # since the unification ssh accepts a header but there
# is no capability signaling it.
"HG20": (), # special-cased below
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 "HG10UN": ("HG10UN", 'UN'),
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 "HG10BZ": ("HG10", 'BZ'),
"HG10GZ": ("HG10GZ", 'GZ'),
}
# hgweb uses this list to communicate its preferred type
bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN']
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 class bundle20(object):
"""represent an outgoing bundle2 container
Pierre-Yves David
bundle2: have ``newpart`` automatically add the part to the bundle...
r21599 Use the `addparam` method to add stream level parameter. and `newpart` to
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 populate it. Then call `getchunks` to retrieve all the binary chunks of
Mads Kiilerich
spelling: fixes from spell checker
r21024 data that compose the bundle2 container."""
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 _magicstring = 'HG20'
Pierre-Yves David
bundle20: move magic string into the class...
r24640
Pierre-Yves David
bundle2: adds a capabilities attribute on bundler20...
r21134 def __init__(self, ui, capabilities=()):
Pierre-Yves David
bundle2: print debug information during bundling...
r20842 self.ui = ui
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 self._params = []
self._parts = []
Pierre-Yves David
bundle2: support for capabilities with values...
r21136 self.capabilities = dict(capabilities)
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 self._compengine = util.compengines.forbundletype('UN')
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 self._compopts = None
Pierre-Yves David
bundle2: allow compressed bundle...
r26404
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 def setcompression(self, alg, compopts=None):
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 """setup core part compression to <alg>"""
Gregory Szorc
bundle2: equate 'UN' with no compression...
r30439 if alg in (None, 'UN'):
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 return
Siddharth Agarwal
bundle2: fix assertion that 'compression' hasn't been set...
r30915 assert not any(n.lower() == 'compression' for n, v in self._params)
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 self.addparam('Compression', alg)
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 self._compengine = util.compengines.forbundletype(alg)
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 self._compopts = compopts
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801
Pierre-Yves David
bundle2: add a ``bundle20.nbparts`` property...
r21900 @property
def nbparts(self):
"""total number of parts added to the bundler"""
return len(self._parts)
Pierre-Yves David
bundle2: small doc update on the bundler...
r21597 # methods used to defines the bundle2 content
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 def addparam(self, name, value=None):
"""add a stream level parameter"""
Pierre-Yves David
bundle2: refuse empty parameter name...
r20813 if not name:
Augie Fackler
bundle2: make ValueError messages native strings
r34280 raise ValueError(r'empty parameter name')
Augie Fackler
bundle2: portably grab first byte of part name for letter check
r34281 if name[0:1] not in pycompat.bytestr(string.ascii_letters):
Augie Fackler
bundle2: make ValueError messages native strings
r34280 raise ValueError(r'non letter first character: %s' % name)
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 self._params.append((name, value))
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 def addpart(self, part):
"""add a new part to the bundle2 container
Mads Kiilerich
spelling: fixes from spell checker
r21024 Parts contains the actual applicative payload."""
Pierre-Yves David
bundle2: add an integer id to part...
r20995 assert part.id is None
part.id = len(self._parts) # very cheap counter
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 self._parts.append(part)
Pierre-Yves David
bundle2: add a ``newpart`` method to ``bundle20``...
r21598 def newpart(self, typeid, *args, **kwargs):
Pierre-Yves David
bundle2: warn about error during initialization in ``newpart`` docstring...
r21602 """create a new part and add it to the containers
As the part is directly added to the containers. For now, this means
that any failure to properly initialize the part after calling
``newpart`` should result in a failure of the whole bundling process.
You can still fall back to manually create and add if you need better
control."""
Pierre-Yves David
bundle2: add a ``newpart`` method to ``bundle20``...
r21598 part = bundlepart(typeid, *args, **kwargs)
Pierre-Yves David
bundle2: have ``newpart`` automatically add the part to the bundle...
r21599 self.addpart(part)
Pierre-Yves David
bundle2: add a ``newpart`` method to ``bundle20``...
r21598 return part
Pierre-Yves David
bundle2: small doc update on the bundler...
r21597 # methods used to generate the bundle2 stream
Pierre-Yves David
bundle2: very first version of a bundle2 bundler...
r20801 def getchunks(self):
Pierre-Yves David
bundle2: add generic debug output regarding generated bundle...
r25322 if self.ui.debugflag:
msg = ['bundle2-output-bundle: "%s",' % self._magicstring]
if self._params:
msg.append(' (%i params)' % len(self._params))
msg.append(' %i parts total\n' % len(self._parts))
self.ui.debug(''.join(msg))
Pierre-Yves David
bundle2: handle new line in 'outdebug' function...
r25315 outdebug(self.ui, 'start emission of %s stream' % self._magicstring)
Pierre-Yves David
bundle20: move magic string into the class...
r24640 yield self._magicstring
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 param = self._paramchunk()
Pierre-Yves David
bundle2: handle new line in 'outdebug' function...
r25315 outdebug(self.ui, 'bundle parameter: %s' % param)
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 yield _pack(_fstreamparamsize, len(param))
if param:
yield param
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 for chunk in self._compengine.compressstream(self._getcorechunk(),
self._compopts):
Gregory Szorc
bundle2: use compressstream compression engine API...
r30357 yield chunk
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 def _paramchunk(self):
"""return a encoded version of all stream parameters"""
blocks = []
Pierre-Yves David
bundle2: support for bundling parameter value...
r20809 for par, value in self._params:
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 par = urlreq.quote(par)
Pierre-Yves David
bundle2: support for bundling parameter value...
r20809 if value is not None:
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 value = urlreq.quote(value)
Pierre-Yves David
bundle2: support for bundling parameter value...
r20809 par = '%s=%s' % (par, value)
blocks.append(par)
Pierre-Yves David
bundle2: support bundling simple parameter...
r20804 return ' '.join(blocks)
Pierre-Yves David
bundle20: extract core payload generation in its own function...
r26396 def _getcorechunk(self):
"""yield chunk for the core part of the bundle
(all but headers and parameters)"""
outdebug(self.ui, 'start of parts')
for part in self._parts:
outdebug(self.ui, 'bundle part: "%s"' % part.type)
for chunk in part.getchunks(ui=self.ui):
yield chunk
outdebug(self.ui, 'end of bundle')
yield _pack(_fpartheadersize, 0)
Pierre-Yves David
bundle2: add a 'salvageoutput' method on bundle20...
r24794 def salvageoutput(self):
"""return a list with a copy of all output parts in the bundle
This is meant to be used during error handling to make sure we preserve
server output"""
salvaged = []
for part in self._parts:
if part.type.startswith('output'):
salvaged.append(part.copy())
return salvaged
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013 class unpackermixin(object):
"""A mixin to extract bytes and struct data from a stream"""
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013 def __init__(self, fp):
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 self._fp = fp
def _unpack(self, format):
Pierre-Yves David
bundle2: clarify the docstring of unpackermixin methods...
r31862 """unpack this struct format from the stream
This method is meant for internal usage by the bundle2 protocol only.
They directly manipulate the low level stream including bundle2 level
instruction.
Do not use it to implement higher-level logic or methods."""
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 data = self._readexact(struct.calcsize(format))
return _unpack(format, data)
def _readexact(self, size):
Pierre-Yves David
bundle2: clarify the docstring of unpackermixin methods...
r31862 """read exactly <size> bytes from the stream
This method is meant for internal usage by the bundle2 protocol only.
They directly manipulate the low level stream including bundle2 level
instruction.
Do not use it to implement higher-level logic or methods."""
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 return changegroup.readexactly(self._fp, size)
Pierre-Yves David
bundle2.getunbundler: rename "header" to "magicstring"...
r25640 def getunbundler(ui, fp, magicstring=None):
"""return a valid unbundler object for a given magicstring"""
if magicstring is None:
magicstring = changegroup.readexactly(fp, 4)
magic, version = magicstring[0:2], magicstring[2:4]
Pierre-Yves David
unbundle20: allow generic dispatch between unbundlers...
r24648 if magic != 'HG':
Siddharth Agarwal
bundle2: add some debugging information to the not-a-bundle error...
r33122 ui.debug(
"error: invalid magic: %r (version %r), should be 'HG'\n"
% (magic, version))
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('not a Mercurial bundle'))
Pierre-Yves David
unbundle20: allow generic dispatch between unbundlers...
r24648 unbundlerclass = formatmap.get(version)
if unbundlerclass is None:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('unknown bundle version %s') % version)
Pierre-Yves David
unbundle20: allow generic dispatch between unbundlers...
r24648 unbundler = unbundlerclass(ui, fp)
Pierre-Yves David
bundle2.getunbundler: rename "header" to "magicstring"...
r25640 indebug(ui, 'start processing of %s stream' % magicstring)
Pierre-Yves David
unbundle20: move header parsing into the 'getunbundler' function...
r24642 return unbundler
Pierre-Yves David
unbundle20: retrieve unbundler instances through a factory function...
r24641
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013 class unbundle20(unpackermixin):
"""interpret a bundle2 stream
Pierre-Yves David
bundle2: use an official iterparts method to unbundle parts...
r21129 This class is fed with a binary stream and yields parts through its
`iterparts` methods."""
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013
Pierre-Yves David
bundle2: add a way to just forward the bundle2 stream to another user...
r26542 _magicstring = 'HG20'
Pierre-Yves David
unbundle20: move header parsing into the 'getunbundler' function...
r24642 def __init__(self, ui, fp):
Pierre-Yves David
bundle2: make header reading optional...
r21066 """If header is specified, we do not read it out of the stream."""
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013 self.ui = ui
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 self._compengine = util.compengines.forbundletype('UN')
Pierre-Yves David
bundle2: make unbundle.compressed return True when compressed...
r26802 self._compressed = None
Pierre-Yves David
bundle2: extract stream/unpack logic in an unpackermixin...
r21013 super(unbundle20, self).__init__(fp)
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 @util.propertycache
def params(self):
Mads Kiilerich
spelling: fixes from spell checker
r21024 """dictionary of stream level parameters"""
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'reading bundle2 stream parameters')
Pierre-Yves David
bundle2: support for unbundling simple parameter...
r20805 params = {}
paramssize = self._unpack(_fstreamparamsize)[0]
Pierre-Yves David
bundle2: detect and disallow a negative chunk size...
r23011 if paramssize < 0:
raise error.BundleValueError('negative bundle param size: %i'
% paramssize)
Pierre-Yves David
bundle2: support for unbundling simple parameter...
r20805 if paramssize:
Pierre-Yves David
bundle2: split parameter retrieval and processing...
r26541 params = self._readexact(paramssize)
params = self._processallparams(params)
Pierre-Yves David
bundle2: support for unbundling simple parameter...
r20805 return params
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: split parameter retrieval and processing...
r26541 def _processallparams(self, paramsblock):
""""""
Gregory Szorc
bundle2: use a sorted dict for holding parameters...
r29591 params = util.sortdict()
Pierre-Yves David
bundle2: split parameter retrieval and processing...
r26541 for p in paramsblock.split(' '):
p = p.split('=', 1)
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 p = [urlreq.unquote(i) for i in p]
Pierre-Yves David
bundle2: split parameter retrieval and processing...
r26541 if len(p) < 2:
p.append(None)
self._processparam(*p)
params[p[0]] = p[1]
return params
Pierre-Yves David
bundle2: implement the mandatory/advisory logic for parameter...
r20844 def _processparam(self, name, value):
"""process a parameter, applying its effect if needed
Parameter starting with a lower case letter are advisory and will be
ignored when unknown. Those starting with an upper case letter are
mandatory and will this function will raise a KeyError when unknown.
Note: no option are currently supported. Any input will be either
ignored or failing.
"""
if not name:
Augie Fackler
bundle2: stop using %r to quote part names...
r34278 raise ValueError(r'empty parameter name')
if name[0:1] not in pycompat.bytestr(string.ascii_letters):
raise ValueError(r'non letter first character: %s' % name)
Pierre-Yves David
unbundle20: allow registering handlers for stream level parameters...
r26395 try:
handler = b2streamparamsmap[name.lower()]
except KeyError:
Augie Fackler
bundle2: stop using %r to quote part names...
r34278 if name[0:1].islower():
indebug(self.ui, "ignoring unknown parameter %s" % name)
Pierre-Yves David
unbundle20: allow registering handlers for stream level parameters...
r26395 else:
raise error.BundleUnknownFeatureError(params=(name,))
Pierre-Yves David
bundle2: implement the mandatory/advisory logic for parameter...
r20844 else:
Pierre-Yves David
unbundle20: allow registering handlers for stream level parameters...
r26395 handler(self, name, value)
Pierre-Yves David
bundle2: implement the mandatory/advisory logic for parameter...
r20844
Pierre-Yves David
bundle2: add a way to just forward the bundle2 stream to another user...
r26542 def _forwardchunks(self):
"""utility to transfer a bundle2 as binary
This is made necessary by the fact the 'getbundle' command over 'ssh'
have no way to know then the reply end, relying on the bundle to be
interpreted to know its end. This is terrible and we are sorry, but we
needed to move forward to get general delta enabled.
"""
yield self._magicstring
assert 'params' not in vars(self)
paramssize = self._unpack(_fstreamparamsize)[0]
if paramssize < 0:
raise error.BundleValueError('negative bundle param size: %i'
% paramssize)
yield _pack(_fstreamparamsize, paramssize)
if paramssize:
params = self._readexact(paramssize)
self._processallparams(params)
yield params
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 assert self._compengine.bundletype == 'UN'
Pierre-Yves David
bundle2: add a way to just forward the bundle2 stream to another user...
r26542 # From there, payload might need to be decompressed
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 self._fp = self._compengine.decompressorreader(self._fp)
Pierre-Yves David
bundle2: add a way to just forward the bundle2 stream to another user...
r26542 emptycount = 0
while emptycount < 2:
# so we can brainlessly loop
assert _fpartheadersize == _fpayloadsize
size = self._unpack(_fpartheadersize)[0]
yield _pack(_fpartheadersize, size)
if size:
emptycount = 0
else:
emptycount += 1
continue
if size == flaginterrupt:
continue
elif size < 0:
raise error.BundleValueError('negative chunk size: %i')
yield self._readexact(size)
Pierre-Yves David
bundle2: use an official iterparts method to unbundle parts...
r21129 def iterparts(self):
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 """yield all parts contained in the stream"""
# make sure param have been loaded
self.params
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 # From there, payload need to be decompressed
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 self._fp = self._compengine.decompressorreader(self._fp)
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'start extraction of bundle2 parts')
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 headerblock = self._readpartheader()
while headerblock is not None:
part = unbundlepart(self.ui, headerblock, self._fp)
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802 yield part
Durham Goode
bundle2: seek part back during iteration...
r33889 # Seek to the end of the part to force it's consumption so the next
# part can be read. But then seek back to the beginning so the
# code consuming this generator has a part that starts at 0.
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 part.seek(0, os.SEEK_END)
part.seek(0, os.SEEK_SET)
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 headerblock = self._readpartheader()
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'end of bundle2 stream')
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 def _readpartheader(self):
"""reads a part header size and return the bytes blob
Pierre-Yves David
bundle2: support unbundling empty part...
r20864
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 returns None if empty"""
Pierre-Yves David
bundle2: support unbundling empty part...
r20864 headersize = self._unpack(_fpartheadersize)[0]
Pierre-Yves David
bundle2: detect and disallow a negative chunk size...
r23011 if headersize < 0:
raise error.BundleValueError('negative part header size: %i'
% headersize)
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'part header size: %i' % headersize)
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 if headersize:
return self._readexact(headersize)
return None
Pierre-Yves David
bundle2: support unbundling empty part...
r20864
Eric Sumner
bundle2.unbundle20: add compressed() method...
r24071 def compressed(self):
Pierre-Yves David
bundle2: make unbundle.compressed return True when compressed...
r26802 self.params # load params
return self._compressed
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: move the 'close' method off the unpackermixin...
r31863 def close(self):
"""close underlying file"""
if util.safehasattr(self._fp, 'close'):
return self._fp.close()
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 formatmap = {'20': unbundle20}
Pierre-Yves David
unbundle20: allow generic dispatch between unbundlers...
r24648
Pierre-Yves David
unbundle20: allow registering handlers for stream level parameters...
r26395 b2streamparamsmap = {}
def b2streamparamhandler(name):
"""register a handler for a stream level parameter"""
def decorator(func):
assert name not in formatmap
b2streamparamsmap[name] = func
return func
return decorator
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 @b2streamparamhandler('compression')
def processcompression(unbundler, param, value):
"""read compression parameter and install payload decompression"""
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 if value not in util.compengines.supportedbundletypes:
Pierre-Yves David
bundle2: allow compressed bundle...
r26404 raise error.BundleUnknownFeatureError(params=(param,),
values=(value,))
Gregory Szorc
bundle2: use compression engines API to obtain decompressor...
r30353 unbundler._compengine = util.compengines.forbundletype(value)
Pierre-Yves David
bundle2: make unbundle.compressed return True when compressed...
r26802 if value is not None:
unbundler._compressed = True
Pierre-Yves David
bundle2: allow compressed bundle...
r26404
Pierre-Yves David
bundle2: rename part to bundlepart...
r21005 class bundlepart(object):
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 """A bundle2 part contains application level payload
The part `type` is used to route the part to the application level
handler.
Pierre-Yves David
bundle2: the ability to set ``data`` attribute of the part is now official...
r21604
The part payload is contained in ``part.data``. It could be raw bytes or a
Pierre-Yves David
bundle2: introduce a ``addparam`` method on part...
r21605 generator of byte chunks.
You can add parameters to the part using the ``addparam`` method.
Parameters can be either mandatory (default) or advisory. Remote side
should be able to safely ignore the advisory ones.
Both data and parameters cannot be modified after the generation has begun.
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 """
Pierre-Yves David
bundle2: part params
r20877 def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
Eric Sumner
bundle2.bundlepart: make mandatory part flag explicit in API...
r23590 data='', mandatory=True):
Pierre-Yves David
bundle2: enforce parttype as alphanumerical...
r23868 validateparttype(parttype)
Pierre-Yves David
bundle2: add an integer id to part...
r20995 self.id = None
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 self.type = parttype
Pierre-Yves David
bundle2: the ability to set ``data`` attribute of the part is now official...
r21604 self._data = data
Pierre-Yves David
bundle2: introduce a ``addparam`` method on part...
r21605 self._mandatoryparams = list(mandatoryparams)
self._advisoryparams = list(advisoryparams)
Pierre-Yves David
bundle2: forbid duplicate parameter keys...
r21607 # checking for duplicated entries
self._seenparams = set()
for pname, __ in self._mandatoryparams + self._advisoryparams:
if pname in self._seenparams:
Jun Wu
bundle2: use ProgrammingError
r31647 raise error.ProgrammingError('duplicated params: %s' % pname)
Pierre-Yves David
bundle2: forbid duplicate parameter keys...
r21607 self._seenparams.add(pname)
Pierre-Yves David
bundle2: track life cycle of parts...
r21601 # status of the part's generation:
# - None: not started,
# - False: currently generated,
# - True: generation done.
self._generated = None
Eric Sumner
bundle2.bundlepart: make mandatory part flag explicit in API...
r23590 self.mandatory = mandatory
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856
Pierre-Yves David
bundle2: implement a basic __repr__ for bundle2 part...
r30872 def __repr__(self):
cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__)
return ('<%s object at %x; id: %s; type: %s; mandatory: %s>'
% (cls, id(self), self.id, self.type, self.mandatory))
Pierre-Yves David
bundle2: add a 'copy' method on parts...
r24793 def copy(self):
"""return a copy of the part
The new part have the very same content but no partid assigned yet.
Parts with generated data cannot be copied."""
assert not util.safehasattr(self.data, 'next')
return self.__class__(self.type, self._mandatoryparams,
self._advisoryparams, self._data, self.mandatory)
Pierre-Yves David
bundle2: the ability to set ``data`` attribute of the part is now official...
r21604 # methods used to defines the part content
Augie Fackler
cleanup: use modern @property/@foo.setter property specification...
r27879 @property
def data(self):
return self._data
@data.setter
def data(self, data):
Pierre-Yves David
bundle2: the ability to set ``data`` attribute of the part is now official...
r21604 if self._generated is not None:
Pierre-Yves David
bundle2: move exception classes into the error module...
r21618 raise error.ReadOnlyPartError('part is being generated')
Pierre-Yves David
bundle2: the ability to set ``data`` attribute of the part is now official...
r21604 self._data = data
Pierre-Yves David
bundle2: introduce a ``addparam`` method on part...
r21605 @property
def mandatoryparams(self):
# make it an immutable tuple to force people through ``addparam``
return tuple(self._mandatoryparams)
@property
def advisoryparams(self):
# make it an immutable tuple to force people through ``addparam``
return tuple(self._advisoryparams)
def addparam(self, name, value='', mandatory=True):
Pierre-Yves David
bundle2: add documention to 'part.addparams'...
r31861 """add a parameter to the part
If 'mandatory' is set to True, the remote handler must claim support
for this parameter or the unbundling will be aborted.
The 'name' and 'value' cannot exceed 255 bytes each.
"""
Pierre-Yves David
bundle2: introduce a ``addparam`` method on part...
r21605 if self._generated is not None:
Pierre-Yves David
bundle2: move exception classes into the error module...
r21618 raise error.ReadOnlyPartError('part is being generated')
Pierre-Yves David
bundle2: forbid duplicate parameter keys...
r21607 if name in self._seenparams:
raise ValueError('duplicated params: %s' % name)
self._seenparams.add(name)
Pierre-Yves David
bundle2: introduce a ``addparam`` method on part...
r21605 params = self._advisoryparams
if mandatory:
params = self._mandatoryparams
params.append((name, value))
Pierre-Yves David
bundle2: track life cycle of parts...
r21601 # methods used to generates the bundle2 stream
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 def getchunks(self, ui):
Pierre-Yves David
bundle2: track life cycle of parts...
r21601 if self._generated is not None:
Jun Wu
bundle2: use ProgrammingError
r31647 raise error.ProgrammingError('part can only be consumed once')
Pierre-Yves David
bundle2: track life cycle of parts...
r21601 self._generated = False
Pierre-Yves David
bundle2: add generic debug output regarding generated parts...
r25323
if ui.debugflag:
msg = ['bundle2-output-part: "%s"' % self.type]
if not self.mandatory:
msg.append(' (advisory)')
nbmp = len(self.mandatoryparams)
nbap = len(self.advisoryparams)
if nbmp or nbap:
msg.append(' (params:')
if nbmp:
msg.append(' %i mandatory' % nbmp)
if nbap:
msg.append(' %i advisory' % nbmp)
msg.append(')')
if not self.data:
msg.append(' empty payload')
Augie Fackler
bundle2: update check for a generator to work on Python 3
r34279 elif (util.safehasattr(self.data, 'next')
or util.safehasattr(self.data, '__next__')):
Pierre-Yves David
bundle2: add generic debug output regarding generated parts...
r25323 msg.append(' streamed payload')
else:
msg.append(' %i bytes payload' % len(self.data))
msg.append('\n')
ui.debug(''.join(msg))
Pierre-Yves David
bundle2: part params
r20877 #### header
Eric Sumner
bundle2.bundlepart: make mandatory part flag explicit in API...
r23590 if self.mandatory:
parttype = self.type.upper()
else:
parttype = self.type.lower()
Augie Fackler
bundle2: convert ints to strings using pycompat.bytestring()...
r33677 outdebug(ui, 'part %s: "%s"' % (pycompat.bytestr(self.id), parttype))
Pierre-Yves David
bundle2: part params
r20877 ## parttype
Eric Sumner
bundle2.bundlepart: make mandatory part flag explicit in API...
r23590 header = [_pack(_fparttypesize, len(parttype)),
parttype, _pack(_fpartid, self.id),
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 ]
Pierre-Yves David
bundle2: part params
r20877 ## parameters
# count
manpar = self.mandatoryparams
advpar = self.advisoryparams
header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
# size
parsizes = []
for key, value in manpar:
parsizes.append(len(key))
parsizes.append(len(value))
for key, value in advpar:
parsizes.append(len(key))
parsizes.append(len(value))
Augie Fackler
bundle2: use modern Python division...
r33635 paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes)
Pierre-Yves David
bundle2: part params
r20877 header.append(paramsizes)
# key, value
for key, value in manpar:
header.append(key)
header.append(value)
for key, value in advpar:
header.append(key)
header.append(value)
## finalize header
Augie Fackler
bundle2: raise a more helpful error if building a bundle part header fails...
r34246 try:
headerchunk = ''.join(header)
except TypeError:
raise TypeError(r'Found a non-bytes trying to '
r'build bundle part header: %r' % header)
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 outdebug(ui, 'header chunk size: %i' % len(headerchunk))
Pierre-Yves David
bundle2: support bundling of empty part (with a type)...
r20856 yield _pack(_fpartheadersize, len(headerchunk))
yield headerchunk
Pierre-Yves David
bundle2: part params
r20877 ## payload
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 try:
for chunk in self._payloadchunks():
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 outdebug(ui, 'payload chunk size: %i' % len(chunk))
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 yield _pack(_fpayloadsize, len(chunk))
yield chunk
Augie Fackler
bundle2: don't try to recover from a GeneratorExit (issue4785)...
r26144 except GeneratorExit:
# GeneratorExit means that nobody is listening for our
# results anyway, so just bail quickly rather than trying
# to produce an error part.
ui.debug('bundle2-generatorexit\n')
raise
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except BaseException as exc:
Yuya Nishihara
py3: convert arbitrary exception object to byte string more reliably...
r33645 bexc = util.forcebytestr(exc)
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 # backup exception data for later
Pierre-Yves David
bundle2: add generic debug output regarding generated interruption...
r25324 ui.debug('bundle2-input-stream-interrupt: encoding exception %s'
Augie Fackler
bundle2: obtain repr() of exception in a python3-safe way...
r33639 % bexc)
Yuya Nishihara
pycompat: extract helper to raise exception with traceback...
r32186 tb = sys.exc_info()[2]
Augie Fackler
bundle2: obtain repr() of exception in a python3-safe way...
r33639 msg = 'unexpected error: %s' % bexc
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 interpart = bundlepart('error:abort', [('message', msg)],
Eric Sumner
bundle2.bundlepart: make mandatory part flag explicit in API...
r23590 mandatory=False)
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 interpart.id = 0
yield _pack(_fpayloadsize, -1)
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 for chunk in interpart.getchunks(ui=ui):
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 yield chunk
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 outdebug(ui, 'closing payload chunk')
Pierre-Yves David
bundle2: transmit exception during part generation...
r23067 # abort current part payload
yield _pack(_fpayloadsize, 0)
Yuya Nishihara
pycompat: extract helper to raise exception with traceback...
r32186 pycompat.raisewithtb(exc, tb)
Pierre-Yves David
bundle2: extract a _payloadchunks method for part...
r21000 # end of payload
Pierre-Yves David
bundle2: add debug output for part generation...
r25321 outdebug(ui, 'closing payload chunk')
Pierre-Yves David
bundle2: extract a _payloadchunks method for part...
r21000 yield _pack(_fpayloadsize, 0)
Pierre-Yves David
bundle2: track life cycle of parts...
r21601 self._generated = True
Pierre-Yves David
bundle2: extract a _payloadchunks method for part...
r21000
def _payloadchunks(self):
"""yield chunks of a the part payload
Exists to handle the different methods to provide data to a part."""
Pierre-Yves David
bundle2: support for bundling and unbundling payload...
r20876 # we only support fixed size data now.
# This will be improved in the future.
Augie Fackler
bundle2: look for __next__ as well as next to identify iterators...
r33636 if (util.safehasattr(self.data, 'next')
or util.safehasattr(self.data, '__next__')):
Pierre-Yves David
bundle2: support chunk iterator as part data...
r21001 buff = util.chunkbuffer(self.data)
chunk = buff.read(preferedchunksize)
while chunk:
yield chunk
chunk = buff.read(preferedchunksize)
elif len(self.data):
Pierre-Yves David
bundle2: support for bundling and unbundling payload...
r20876 yield self.data
Pierre-Yves David
bundle2: a very first version of bundle2 unbundler...
r20802
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066
flaginterrupt = -1
class interrupthandler(unpackermixin):
"""read one part and process it with restricted capability
This allows to transmit exception raised on the producer size during part
iteration while the consumer is reading a part.
Part processed in this manner only have access to a ui object,"""
def __init__(self, ui, fp):
super(interrupthandler, self).__init__(fp)
self.ui = ui
def _readpartheader(self):
"""reads a part header size and return the bytes blob
returns None if empty"""
headersize = self._unpack(_fpartheadersize)[0]
if headersize < 0:
raise error.BundleValueError('negative part header size: %i'
% headersize)
Pierre-Yves David
bundle2: introduce a specific function for debug messages while unbundling...
r25318 indebug(self.ui, 'part header size: %i\n' % headersize)
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066 if headersize:
return self._readexact(headersize)
return None
def __call__(self):
Pierre-Yves David
bundle2: add generic debug output regarding processed interruption...
r25335
self.ui.debug('bundle2-input-stream-interrupt:'
' opening out of band context\n')
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'bundle2 stream interruption, looking for a part.')
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066 headerblock = self._readpartheader()
if headerblock is None:
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'no part found during interruption.')
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066 return
part = unbundlepart(self.ui, headerblock, self._fp)
op = interruptoperation(self.ui)
Durham Goode
bundle2: move processpart stream maintenance into part iterator...
r34259 hardabort = False
try:
_processpart(op, part)
except (SystemExit, KeyboardInterrupt):
hardabort = True
raise
finally:
if not hardabort:
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 part.seek(0, os.SEEK_END)
Pierre-Yves David
bundle2: add generic debug output regarding processed interruption...
r25335 self.ui.debug('bundle2-input-stream-interrupt:'
' closing out of band context\n')
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066
class interruptoperation(object):
"""A limited operation to be use by part handler during interruption
It only have access to an ui object.
"""
def __init__(self, ui):
self.ui = ui
self.reply = None
Pierre-Yves David
bundle2: disable ouput capture unless we use http (issue4613 issue4615)...
r24878 self.captureoutput = False
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066
@property
def repo(self):
Jun Wu
bundle2: use ProgrammingError
r31647 raise error.ProgrammingError('no repo access from stream interruption')
Pierre-Yves David
bundle2: add a interrupt mechanism...
r23066
def gettransaction(self):
raise TransactionUnavailable('no repo access from stream interruption')
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 class unbundlepart(unpackermixin):
"""a bundle part read from a bundle"""
def __init__(self, ui, header, fp):
super(unbundlepart, self).__init__(fp)
Pierre-Yves David
bundle2: move 'seek' and 'tell' methods off the unpackermixin class...
r31889 self._seekable = (util.safehasattr(fp, 'seek') and
util.safehasattr(fp, 'tell'))
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 self.ui = ui
# unbundle state attr
self._headerdata = header
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 self._headeroffset = 0
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 self._initialized = False
self.consumed = False
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 # part data
self.id = None
self.type = None
self.mandatoryparams = None
self.advisoryparams = None
Pierre-Yves David
bundle2: introduce a ``params`` dictionary on unbundled parts...
r21610 self.params = None
Pierre-Yves David
bundle2: expose mandatory params in a mandatorykeys attribute...
r21612 self.mandatorykeys = ()
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 self._payloadstream = None
self._readheader()
Eric Sumner
bundle2.unbundlepart: decouple mandatory from parttype...
r23585 self._mandatory = None
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035 self._chunkindex = [] #(payload, file) position tuples for chunk starts
Eric Sumner
bundle2.unbundlepart: tell() implementation...
r24036 self._pos = 0
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 def _fromheader(self, size):
"""return the next <size> byte from the header"""
offset = self._headeroffset
data = self._headerdata[offset:(offset + size)]
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 self._headeroffset = offset + size
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 return data
Pierre-Yves David
bundle2: move unpackheader closure into the class...
r21016 def _unpackheader(self, format):
"""read given format from header
This automatically compute the size of the format to read."""
data = self._fromheader(struct.calcsize(format))
return _unpack(format, data)
Pierre-Yves David
bundle2: introduce an ``_initparams`` method...
r21608 def _initparams(self, mandatoryparams, advisoryparams):
"""internal function to setup all logic related parameters"""
Pierre-Yves David
bundle2: make sure unbundled part param are read-only...
r21609 # make it read only to prevent people touching it by mistake.
self.mandatoryparams = tuple(mandatoryparams)
self.advisoryparams = tuple(advisoryparams)
Pierre-Yves David
bundle2: introduce a ``params`` dictionary on unbundled parts...
r21610 # user friendly UI
Gregory Szorc
bundle2: use a sorted dict for holding parameters...
r29591 self.params = util.sortdict(self.mandatoryparams)
self.params.update(self.advisoryparams)
Pierre-Yves David
bundle2: expose mandatory params in a mandatorykeys attribute...
r21612 self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
Pierre-Yves David
bundle2: introduce an ``_initparams`` method...
r21608
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035 def _payloadchunks(self, chunknum=0):
'''seek to specified chunk and start yielding data'''
if len(self._chunkindex) == 0:
assert chunknum == 0, 'Must start with chunk 0'
Pierre-Yves David
bundle2: move 'seek' and 'tell' methods off the unpackermixin class...
r31889 self._chunkindex.append((0, self._tellfp()))
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035 else:
assert chunknum < len(self._chunkindex), \
'Unknown chunk %d' % chunknum
Pierre-Yves David
bundle2: move 'seek' and 'tell' methods off the unpackermixin class...
r31889 self._seekfp(self._chunkindex[chunknum][1])
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035
pos = self._chunkindex[chunknum][0]
Eric Sumner
bundle2.unbundlepart: raise payloadchunks from a closure to a method...
r24034 payloadsize = self._unpack(_fpayloadsize)[0]
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'payload chunk size: %i' % payloadsize)
Eric Sumner
bundle2.unbundlepart: raise payloadchunks from a closure to a method...
r24034 while payloadsize:
if payloadsize == flaginterrupt:
# interruption detection, the handler will now read a
# single part and process it.
interrupthandler(self.ui, self._fp)()
elif payloadsize < 0:
msg = 'negative payload chunk size: %i' % payloadsize
raise error.BundleValueError(msg)
else:
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035 result = self._readexact(payloadsize)
chunknum += 1
pos += payloadsize
if chunknum == len(self._chunkindex):
Pierre-Yves David
bundle2: move 'seek' and 'tell' methods off the unpackermixin class...
r31889 self._chunkindex.append((pos, self._tellfp()))
Eric Sumner
bundle2.unbundlepart: keep an index of chunks and their locations...
r24035 yield result
Eric Sumner
bundle2.unbundlepart: raise payloadchunks from a closure to a method...
r24034 payloadsize = self._unpack(_fpayloadsize)[0]
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'payload chunk size: %i' % payloadsize)
Eric Sumner
bundle2.unbundlepart: raise payloadchunks from a closure to a method...
r24034
Eric Sumner
bundle2.unbundlepart: implement seek()...
r24037 def _findchunk(self, pos):
'''for a given payload position, return a chunk number and offset'''
for chunk, (ppos, fpos) in enumerate(self._chunkindex):
if ppos == pos:
return chunk, 0
elif ppos > pos:
return chunk - 1, pos - self._chunkindex[chunk - 1][0]
raise ValueError('Unknown chunk')
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 def _readheader(self):
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 """read the header and setup the object"""
Pierre-Yves David
bundle2: move unpackheader closure into the class...
r21016 typesize = self._unpackheader(_fparttypesize)[0]
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 self.type = self._fromheader(typesize)
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'part type: "%s"' % self.type)
Pierre-Yves David
bundle2: move unpackheader closure into the class...
r21016 self.id = self._unpackheader(_fpartid)[0]
Augie Fackler
bundle2: convert ints to strings using pycompat.bytestring()...
r33677 indebug(self.ui, 'part id: "%s"' % pycompat.bytestr(self.id))
Eric Sumner
bundle2.unbundlepart: decouple mandatory from parttype...
r23585 # extract mandatory bit from type
self.mandatory = (self.type != self.type.lower())
self.type = self.type.lower()
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 ## reading parameters
# param count
Pierre-Yves David
bundle2: move unpackheader closure into the class...
r21016 mancount, advcount = self._unpackheader(_fpartparamcount)
Pierre-Yves David
bundle2: handle new line in 'indebug' function...
r25320 indebug(self.ui, 'part parameters: %i' % (mancount + advcount))
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 # param size
Pierre-Yves David
bundle2: move unpackheader closure into the class...
r21016 fparamsizes = _makefpartparamsizes(mancount + advcount)
paramsizes = self._unpackheader(fparamsizes)
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 # make it a list of couple again
Augie Fackler
bundle2: work around zip() being lazy in Python 3
r33637 paramsizes = list(zip(paramsizes[::2], paramsizes[1::2]))
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 # split mandatory from advisory
mansizes = paramsizes[:mancount]
advsizes = paramsizes[mancount:]
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 # retrieve param value
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 manparams = []
for key, value in mansizes:
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 manparams.append((self._fromheader(key), self._fromheader(value)))
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 advparams = []
for key, value in advsizes:
Pierre-Yves David
bundle2: move the fromheader closure into the class itself...
r21015 advparams.append((self._fromheader(key), self._fromheader(value)))
Pierre-Yves David
bundle2: introduce an ``_initparams`` method...
r21608 self._initparams(manparams, advparams)
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014 ## part payload
Eric Sumner
bundle2.unbundlepart: raise payloadchunks from a closure to a method...
r24034 self._payloadstream = util.chunkbuffer(self._payloadchunks())
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 # we read the data, tell it
self._initialized = True
def read(self, size=None):
"""read payload data"""
if not self._initialized:
self._readheader()
if size is None:
data = self._payloadstream.read()
else:
data = self._payloadstream.read(size)
Pierre-Yves David
bundle2: add generic debug output regarding processed part payload...
r25334 self._pos += len(data)
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 if size is None or len(data) < size:
Pierre-Yves David
bundle2: add generic debug output regarding processed part payload...
r25334 if not self.consumed and self._pos:
self.ui.debug('bundle2-input-part: total payload size %i\n'
% self._pos)
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 self.consumed = True
return data
Eric Sumner
bundle2.unbundlepart: tell() implementation...
r24036 def tell(self):
return self._pos
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 def seek(self, offset, whence=os.SEEK_SET):
if whence == os.SEEK_SET:
Eric Sumner
bundle2.unbundlepart: implement seek()...
r24037 newpos = offset
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 elif whence == os.SEEK_CUR:
Eric Sumner
bundle2.unbundlepart: implement seek()...
r24037 newpos = self._pos + offset
Gregory Szorc
bundle2: use os.SEEK_* constants...
r35037 elif whence == os.SEEK_END:
Eric Sumner
bundle2.unbundlepart: implement seek()...
r24037 if not self.consumed:
self.read()
newpos = self._chunkindex[-1][0] - offset
else:
raise ValueError('Unknown whence value: %r' % (whence,))
if newpos > self._chunkindex[-1][0] and not self.consumed:
self.read()
if not 0 <= newpos <= self._chunkindex[-1][0]:
raise ValueError('Offset out of range')
if self._pos != newpos:
chunk, internaloffset = self._findchunk(newpos)
self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk))
adjust = self.read(internaloffset)
if len(adjust) != internaloffset:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('Seek failed\n'))
Eric Sumner
bundle2.unbundlepart: implement seek()...
r24037 self._pos = newpos
Pierre-Yves David
bundle2: move 'seek' and 'tell' methods off the unpackermixin class...
r31889 def _seekfp(self, offset, whence=0):
"""move the underlying file pointer
This method is meant for internal usage by the bundle2 protocol only.
They directly manipulate the low level stream including bundle2 level
instruction.
Do not use it to implement higher-level logic or methods."""
if self._seekable:
return self._fp.seek(offset, whence)
else:
raise NotImplementedError(_('File pointer is not seekable'))
def _tellfp(self):
"""return the file offset, or None if file is not seekable
This method is meant for internal usage by the bundle2 protocol only.
They directly manipulate the low level stream including bundle2 level
instruction.
Do not use it to implement higher-level logic or methods."""
if self._seekable:
try:
return self._fp.tell()
except IOError as e:
if e.errno == errno.ESPIPE:
self._seekable = False
else:
raise
return None
Pierre-Yves David
bundle2: add an informative comment to the capability dict...
r25317 # These are only the static capabilities.
# Check the 'getrepocaps' function for the rest.
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 capabilities = {'HG20': (),
Pierre-Yves David
bundle2: convey PushkeyFailed error over the wire...
r25493 'error': ('abort', 'unsupportedcontent', 'pushraced',
'pushkey'),
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 'listkeys': (),
'pushkey': (),
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 'digests': tuple(sorted(util.DIGESTS.keys())),
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 'remote-changegroup': ('http', 'https'),
Gregory Szorc
bundle2: part handler for processing .hgtags fnodes mappings...
r25401 'hgtagsfnodes': (),
Boris Feld
pull: use 'phase-heads' to retrieve phase information...
r34323 'phases': ('heads',),
Pierre-Yves David
obsmarker: move bundle2caps from the localrepo class to the bundle2 module...
r22341 }
Eric Sumner
bundle2-push: provide transaction to reply unbundler...
r23439 def getrepocaps(repo, allowpushback=False):
Pierre-Yves David
bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo...
r22342 """return the bundle2 capabilities for a given repo
Pierre-Yves David
bundle2: advertise the obsmarker part in bundle2 capabilities
r22343 Exists to allow extensions (like evolution) to mutate the capabilities.
Pierre-Yves David
bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo...
r22342 """
Pierre-Yves David
bundle2: advertise the obsmarker part in bundle2 capabilities
r22343 caps = capabilities.copy()
Martin von Zweigbergk
changegroup: fix pulling to treemanifest repo from flat repo (issue5066)...
r27953 caps['changegroup'] = tuple(sorted(
changegroup.supportedincomingversions(repo)))
Durham Goode
obsolete: add exchange option...
r22953 if obsolete.isenabled(repo, obsolete.exchangeopt):
Pierre-Yves David
bundle2: advertise the obsmarker part in bundle2 capabilities
r22343 supportedformat = tuple('V%i' % v for v in obsolete.formats)
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 caps['obsmarkers'] = supportedformat
Eric Sumner
bundle2-push: provide transaction to reply unbundler...
r23439 if allowpushback:
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 caps['pushback'] = ()
configitems: register the 'server.concurrent-push-mode' config
r33219 cpmode = repo.ui.config('server', 'concurrent-push-mode')
check-concurrency: expose the feature as 'concurrent-push-mode'...
r32892 if cpmode == 'check-related':
push: add a way to allow concurrent pushes on unrelated heads...
r32709 caps['checkheads'] = ('related',)
Boris Feld
pull: use 'phase-heads' to retrieve phase information...
r34323 if 'phases' in repo.ui.configlist('devel', 'legacy.exchange'):
caps.pop('phases')
Pierre-Yves David
bundle2: advertise the obsmarker part in bundle2 capabilities
r22343 return caps
Pierre-Yves David
bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo...
r22342
Pierre-Yves David
bundle2: introduce a bundle2caps function...
r21644 def bundle2caps(remote):
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 """return the bundle capabilities of a peer as dict"""
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 raw = remote.capable('bundle2')
Pierre-Yves David
bundle2: introduce a bundle2caps function...
r21644 if not raw and raw != '':
return {}
timeless
pycompat: switch to util.urlreq/util.urlerr for py3 compat
r28883 capsblob = urlreq.unquote(remote.capable('bundle2'))
Pierre-Yves David
bundle2: introduce a bundle2caps function...
r21644 return decodecaps(capsblob)
Pierre-Yves David
bundle2: add an unbundle part responsible from unbundling part...
r21014
Pierre-Yves David
bundle2: add a `obsmarkersversion` function to extract supported version...
r22344 def obsmarkersversion(caps):
"""extract the list of supported obsmarkers versions from a bundle2caps dict
"""
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 obscaps = caps.get('obsmarkers', ())
Pierre-Yves David
bundle2: add a `obsmarkersversion` function to extract supported version...
r22344 return [int(c[1:]) for c in obscaps if c.startswith('V')]
bundle: introduce an higher level function to write bundle on disk...
r32216 def writenewbundle(ui, repo, source, filename, bundletype, outgoing, opts,
vfs=None, compression=None, compopts=None):
if bundletype.startswith('HG10'):
Durham Goode
changegroup: replace getchangegroup with makechangegroup...
r34103 cg = changegroup.makechangegroup(repo, outgoing, '01', source)
bundle: introduce an higher level function to write bundle on disk...
r32216 return writebundle(ui, cg, filename, bundletype, vfs=vfs,
compression=compression, compopts=compopts)
elif not bundletype.startswith('HG20'):
raise error.ProgrammingError('unknown bundle type: %s' % bundletype)
bundle: add an experimental knob to include obsmarkers in bundle...
r32516 caps = {}
if 'obsolescence' in opts:
caps['obsmarkers'] = ('V1',)
bundle = bundle20(ui, caps)
bundle: introduce an higher level function to write bundle on disk...
r32216 bundle.setcompression(compression, compopts)
_addpartsfromopts(ui, repo, bundle, source, outgoing, opts)
chunkiter = bundle.getchunks()
return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts):
# We should eventually reconcile this logic with the one behind
# 'exchange.getbundle2partsgenerator'.
#
# The type of input from 'getbundle' and 'writenewbundle' are a bit
# different right now. So we keep them separated for now for the sake of
# simplicity.
# we always want a changegroup in such bundle
cgversion = opts.get('cg.version')
if cgversion is None:
cgversion = changegroup.safeversion(repo)
Durham Goode
changegroup: replace getchangegroup with makechangegroup...
r34103 cg = changegroup.makechangegroup(repo, outgoing, cgversion, source)
bundle: introduce an higher level function to write bundle on disk...
r32216 part = bundler.newpart('changegroup', data=cg.getchunks())
part.addparam('version', cg.version)
if 'clcount' in cg.extras:
Augie Fackler
bundles: turn nbchanges int into a bytestr using pycompat.bytestr...
r34218 part.addparam('nbchanges', '%d' % cg.extras['clcount'],
bundle: introduce an higher level function to write bundle on disk...
r32216 mandatory=False)
Boris Feld
bundle2: automatically add 'targetphase' parameter in writenewbundle...
r33408 if opts.get('phases') and repo.revs('%ln and secret()',
outgoing.missingheads):
part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
bundle: introduce an higher level function to write bundle on disk...
r32216
bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543)...
r32218 addparttagsfnodescache(repo, bundler, outgoing)
bundle: add an experimental knob to include obsmarkers in bundle...
r32516 if opts.get('obsolescence', False):
obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
buildobsmarkerspart(bundler, obsmarkers)
Martin von Zweigbergk
bundle: add config option to include phases...
r33031 if opts.get('phases', False):
headsbyphase = phases.subsetphaseheads(repo, outgoing.missing)
Boris Feld
phases: move binary encoding into a reusable function...
r34320 phasedata = phases.binaryencode(headsbyphase)
bundler.newpart('phase-heads', data=phasedata)
Martin von Zweigbergk
bundle: add config option to include phases...
r33031
bundle2: move tagsfnodecache generation in a generic function...
r32217 def addparttagsfnodescache(repo, bundler, outgoing):
# we include the tags fnode cache for the bundle changeset
# (as an optional parts)
cache = tags.hgtagsfnodescache(repo.unfiltered())
chunks = []
# .hgtags fnodes are only relevant for head changesets. While we could
# transfer values for all known nodes, there will likely be little to
# no benefit.
#
# We don't bother using a generator to produce output data because
# a) we only have 40 bytes per head and even esoteric numbers of heads
# consume little memory (1M heads is 40MB) b) we don't want to send the
# part if we don't have entries and knowing if we have entries requires
# cache lookups.
for node in outgoing.missingheads:
# Don't compute missing, as this may slow down serving.
fnode = cache.getfnode(node, computemissing=False)
if fnode is not None:
chunks.extend([node, fnode])
if chunks:
bundler.newpart('hgtagsfnodes', data=''.join(chunks))
bundle2: move function building obsmarker-part in the bundle2 module...
r32515 def buildobsmarkerspart(bundler, markers):
"""add an obsmarker part to the bundler with <markers>
No part is created if markers is empty.
Raises ValueError if the bundler doesn't support any known obsmarker format.
"""
if not markers:
return None
remoteversions = obsmarkersversion(bundler.capabilities)
version = obsolete.commonversion(remoteversions)
if version is None:
raise ValueError('bundler does not support common obsmarker format')
stream = obsolete.encodemarkers(markers, True, version=version)
return bundler.newpart('obsmarkers', data=stream)
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None,
compopts=None):
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 """Write a bundle file and return its filename.
Existing files will not be overwritten.
If no filename is specified, a temporary file is created.
bz2 compression can be turned off.
The bundle file will be deleted in case of errors.
"""
if bundletype == "HG20":
bundle = bundle20(ui)
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 bundle.setcompression(compression, compopts)
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 part = bundle.newpart('changegroup', data=cg.getchunks())
part.addparam('version', cg.version)
Gregory Szorc
bundle2: store changeset count when creating file bundles...
r29593 if 'clcount' in cg.extras:
Augie Fackler
bundles: turn nbchanges int into a bytestr using pycompat.bytestr...
r34218 part.addparam('nbchanges', '%d' % cg.extras['clcount'],
Gregory Szorc
bundle2: store changeset count when creating file bundles...
r29593 mandatory=False)
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 chunkiter = bundle.getchunks()
else:
# compression argument is only for the bundle2 case
assert compression is None
if cg.version != '01':
raise error.Abort(_('old bundle types only supports v1 '
'changegroups'))
header, comp = bundletypes[bundletype]
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 if comp not in util.compengines.supportedbundletypes:
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 raise error.Abort(_('unknown stream compression type: %s')
% comp)
Gregory Szorc
bundle2: use new compression engine API for compression...
r30351 compengine = util.compengines.forbundletype(comp)
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 def chunkiter():
yield header
Gregory Szorc
bundle2: allow compression options to be passed to compressor...
r30757 for chunk in compengine.compressstream(cg.getchunks(), compopts):
Gregory Szorc
bundle2: use compressstream compression engine API...
r30357 yield chunk
Martin von Zweigbergk
bundle: move writebundle() from changegroup.py to bundle2.py (API)...
r28666 chunkiter = chunkiter()
# parse the changegroup data, otherwise we will block
# in case of sshrepo because we don't know the end of the stream
return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
Martin von Zweigbergk
bundle: make combinechangegroupresults() take a bundleoperation...
r33037 def combinechangegroupresults(op):
Martin von Zweigbergk
bundle: move combineresults() from changegroup to bundle2...
r33036 """logic to combine 0 or more addchangegroup results into one"""
Martin von Zweigbergk
bundle: make combinechangegroupresults() take a bundleoperation...
r33037 results = [r.get('return', 0)
for r in op.records['changegroup']]
Martin von Zweigbergk
bundle: move combineresults() from changegroup to bundle2...
r33036 changedheads = 0
result = 1
for ret in results:
# If any changegroup result is 0, return 0
if ret == 0:
result = 0
break
if ret < -1:
changedheads += ret + 1
elif ret > 1:
changedheads += ret - 1
if changedheads > 0:
result = 1 + changedheads
elif changedheads < 0:
result = -1 + changedheads
return result
Boris Feld
bundle2: support the 'targetphase' parameter for the changegroup part...
r33407 @parthandler('changegroup', ('version', 'nbchanges', 'treemanifest',
'targetphase'))
Pierre-Yves David
bundle2: use reply part to return result of addchangegroup...
r20998 def handlechangegroup(op, inpart):
Pierre-Yves David
bundle2: first crude version of bundling changeset with bundle2...
r20950 """apply a changegroup part on the repo
This is a very early implementation that will massive rework before being
inflicted to any end-user.
"""
Martin von Zweigbergk
changegroup: let callers pass in transaction to apply() (API)...
r32930 tr = op.gettransaction()
Pierre-Yves David
bundle2: support a "version" argument in `changegroup` part...
r23170 unpackerversion = inpart.params.get('version', '01')
# We should raise an appropriate exception here
Martin von Zweigbergk
changegroup: hide packermap behind methods...
r27751 cg = changegroup.getunbundler(unpackerversion, inpart, None)
Pierre-Yves David
bundle2: add a comment about addchangegroup source and url
r23001 # the source and url passed here are overwritten by the one contained in
# the transaction.hookargs argument. So 'bundle2' is a placeholder
Pierre-Yves David
bundle2: provide number of changesets information to 'addchangegroup'...
r25518 nbchangesets = None
if 'nbchanges' in inpart.params:
nbchangesets = int(inpart.params.get('nbchanges'))
Martin von Zweigbergk
treemanifests: set bundle2 part parameter indicating treemanifest...
r27734 if ('treemanifest' in inpart.params and
'treemanifest' not in op.repo.requirements):
if len(op.repo.changelog) != 0:
raise error.Abort(_(
"bundle contains tree manifests, but local repo is "
"non-empty and does not use tree manifests"))
op.repo.requirements.add('treemanifest')
op.repo._applyopenerreqs()
op.repo._writerequirements()
Boris Feld
bundle2: support the 'targetphase' parameter for the changegroup part...
r33407 extrakwargs = {}
targetphase = inpart.params.get('targetphase')
if targetphase is not None:
extrakwargs['targetphase'] = int(targetphase)
Martin von Zweigbergk
bundle: extract _processchangegroup() method...
r33038 ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2',
Boris Feld
bundle2: support the 'targetphase' parameter for the changegroup part...
r33407 expectedtotal=nbchangesets, **extrakwargs)
Pierre-Yves David
bundle2: use reply part to return result of addchangegroup...
r20998 if op.reply is not None:
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 # This is definitely not the final form of this
Pierre-Yves David
bundle2: use reply part to return result of addchangegroup...
r20998 # return. But one need to start somewhere.
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 part = op.reply.newpart('reply:changegroup', mandatory=False)
Augie Fackler
bundle2: use bytestr() instead of str() to convert part id to bytes...
r33638 part.addparam(
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
Pierre-Yves David
bundle2: update part creators to ``addparam`` when relevant...
r21606 part.addparam('return', '%i' % ret, mandatory=False)
Pierre-Yves David
bundle2: lazy unbundle of part payload...
r21019 assert not inpart.read()
Pierre-Yves David
bundle2: first crude version of bundling changeset with bundle2...
r20950
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 _remotechangegroupparams = tuple(['url', 'size', 'digests'] +
['digest:%s' % k for k in util.DIGESTS.keys()])
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('remote-changegroup', _remotechangegroupparams)
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 def handleremotechangegroup(op, inpart):
"""apply a bundle10 on the repo, given an url and validation information
All the information about the remote bundle to import are given as
parameters. The parameters include:
- url: the url to the bundle10.
- size: the bundle10 file size. It is used to validate what was
retrieved by the client matches the server knowledge about the bundle.
- digests: a space separated list of the digest types provided as
parameters.
- digest:<digest-type>: the hexadecimal representation of the digest with
that name. Like the size, it is used to validate what was retrieved by
the client matches what the server knows about the bundle.
When multiple digest types are given, all of them are checked.
"""
try:
raw_url = inpart.params['url']
except KeyError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url')
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 parsed_url = util.url(raw_url)
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 if parsed_url.scheme not in capabilities['remote-changegroup']:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('remote-changegroup does not support %s urls') %
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 parsed_url.scheme)
try:
size = int(inpart.params['size'])
except ValueError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('remote-changegroup: invalid value for param "%s"')
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 % 'size')
except KeyError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size')
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029
digests = {}
for typ in inpart.params.get('digests', '').split():
param = 'digest:%s' % typ
try:
value = inpart.params[param]
except KeyError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('remote-changegroup: missing "%s" param') %
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 param)
digests[typ] = value
real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests)
Martin von Zweigbergk
changegroup: let callers pass in transaction to apply() (API)...
r32930 tr = op.gettransaction()
Gregory Szorc
bundle2: use absolute_import
r25919 from . import exchange
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 cg = exchange.readbundle(op.repo.ui, real_part, raw_url)
if not isinstance(cg, changegroup.cg1unpacker):
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('%s: not a bundle version 1.0') %
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 util.hidepassword(raw_url))
Martin von Zweigbergk
bundle: extract _processchangegroup() method...
r33038 ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2')
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 if op.reply is not None:
Mads Kiilerich
spelling: fixes from proofreading of spell checker issues
r23139 # This is definitely not the final form of this
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 # return. But one need to start somewhere.
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 part = op.reply.newpart('reply:changegroup')
Augie Fackler
bundle2: use bytestr() instead of str() to convert part id to bytes...
r33638 part.addparam(
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 part.addparam('return', '%i' % ret, mandatory=False)
try:
real_part.validate()
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 except error.Abort as e:
raise error.Abort(_('bundle at %s is corrupted:\n%s') %
Mike Hommey
bundle2: client side support for a part to import external bundles...
r23029 (util.hidepassword(raw_url), str(e)))
assert not inpart.read()
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('reply:changegroup', ('return', 'in-reply-to'))
Mike Hommey
bundle2: rename functions that have the same name
r22548 def handlereplychangegroup(op, inpart):
Pierre-Yves David
bundle2: use the new ``part.params`` dictionary...
r21611 ret = int(inpart.params['return'])
replyto = int(inpart.params['in-reply-to'])
op.records.add('changegroup', {'return': ret}, replyto)
Pierre-Yves David
bundle2: first crude version of bundling changeset with bundle2...
r20950
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('check:heads')
Mike Hommey
bundle2: rename functions that have the same name
r22548 def handlecheckheads(op, inpart):
Pierre-Yves David
bundle2: add a "check:heads" handler...
r21060 """check that head of the repo did not change
This is used to detect a push race when using unbundle.
This replaces the "heads" argument of unbundle."""
h = inpart.read(20)
heads = []
while len(h) == 20:
heads.append(h)
h = inpart.read(20)
assert not h
Durham Goode
bundle2: add op.gettransaction() to handlers that need the lock...
r26565 # Trigger a transaction so that we are guaranteed to have the lock now.
if op.ui.configbool('experimental', 'bundle2lazylocking'):
op.gettransaction()
Mads Kiilerich
bundle2: don't assume ordering of heads checked after push...
r29294 if sorted(heads) != sorted(op.repo.heads()):
Pierre-Yves David
bundle2: add an error message to push race error...
r21185 raise error.PushRaced('repository changed while pushing - '
'please try again')
Pierre-Yves David
bundle2: introduce `replycaps` part for on-demand reply...
r21130
push: add a way to allow concurrent pushes on unrelated heads...
r32709 @parthandler('check:updated-heads')
def handlecheckupdatedheads(op, inpart):
"""check for race on the heads touched by a push
This is similar to 'check:heads' but focus on the heads actually updated
during the push. If other activities happen on unrelated heads, it is
ignored.
This allow server with high traffic to avoid push contention as long as
unrelated parts of the graph are involved."""
h = inpart.read(20)
heads = []
while len(h) == 20:
heads.append(h)
h = inpart.read(20)
assert not h
# trigger a transaction so that we are guaranteed to have the lock now.
if op.ui.configbool('experimental', 'bundle2lazylocking'):
op.gettransaction()
currentheads = set()
for ls in op.repo.branchmap().itervalues():
currentheads.update(ls)
for h in heads:
if h not in currentheads:
raise error.PushRaced('repository changed while pushing - '
'please try again')
Boris Feld
phase: introduce a new 'check:phases' part...
r34821 @parthandler('check:phases')
def handlecheckphases(op, inpart):
"""check that phase boundaries of the repository did not change
This is used to detect a push race.
"""
phasetonodes = phases.binarydecode(inpart)
unfi = op.repo.unfiltered()
cl = unfi.changelog
phasecache = unfi._phasecache
msg = ('repository changed while pushing - please try again '
'(%s is %s expected %s)')
for expectedphase, nodes in enumerate(phasetonodes):
for n in nodes:
actualphase = phasecache.phase(unfi, cl.rev(n))
if actualphase != expectedphase:
finalmsg = msg % (nodemod.short(n),
phases.phasenames[actualphase],
phases.phasenames[expectedphase])
raise error.PushRaced(finalmsg)
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('output')
Pierre-Yves David
bundle2: capture remote stdout while unbundling...
r21131 def handleoutput(op, inpart):
"""forward output captured on the server to the client"""
for line in inpart.read().splitlines():
Akihiko Odaki
bundle2: localize handleoutput remote prompts...
r29851 op.ui.status(_('remote: %s\n') % line)
Pierre-Yves David
bundle2: capture remote stdout while unbundling...
r21131
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('replycaps')
Pierre-Yves David
bundle2: introduce `replycaps` part for on-demand reply...
r21130 def handlereplycaps(op, inpart):
"""Notify that a reply bundle should be created
Pierre-Yves David
bundle2: extract capabilities decoding...
r21138 The payload contains the capabilities information for the reply"""
caps = decodecaps(inpart.read())
Pierre-Yves David
bundle2: introduce `replycaps` part for on-demand reply...
r21130 if op.reply is None:
Pierre-Yves David
bundle2: add capabilities support in `replycaps` part...
r21135 op.reply = bundle20(op.ui, caps)
Pierre-Yves David
bundle2: capture remote stdout while unbundling...
r21131
Gregory Szorc
bundle2: attribute remote failures to remote (issue4788)...
r26829 class AbortFromPart(error.Abort):
"""Sub-class of Abort that denotes an error from a bundle2 part."""
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('error:abort', ('message', 'hint'))
Pierre-Yves David
bundle2: fix names for error part handler...
r24741 def handleerrorabort(op, inpart):
Pierre-Yves David
bundle2: gracefully handle abort during unbundle...
r21177 """Used to transmit abort error over the wire"""
Gregory Szorc
bundle2: attribute remote failures to remote (issue4788)...
r26829 raise AbortFromPart(inpart.params['message'],
hint=inpart.params.get('hint'))
Pierre-Yves David
bundle2: gracefully handle UnknownPartError during unbundle...
r21183
Pierre-Yves David
bundle2: convey PushkeyFailed error over the wire...
r25493 @parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret',
'in-reply-to'))
def handleerrorpushkey(op, inpart):
"""Used to transmit failure of a mandatory pushkey over the wire"""
kwargs = {}
for name in ('namespace', 'key', 'new', 'old', 'ret'):
value = inpart.params.get(name)
if value is not None:
kwargs[name] = value
raise error.PushkeyFailed(inpart.params['in-reply-to'], **kwargs)
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('error:unsupportedcontent', ('parttype', 'params'))
Pierre-Yves David
bundle2: fix names for error part handler...
r24741 def handleerrorunsupportedcontent(op, inpart):
Pierre-Yves David
bundle2: rename b2x:error:unknownpart to b2x:error:unsupportedcontent...
r21619 """Used to transmit unknown content error over the wire"""
Pierre-Yves David
bundle2: support transmission of params error over the wire...
r21622 kwargs = {}
Pierre-Yves David
bundle2: support None parttype in BundleValueError...
r21627 parttype = inpart.params.get('parttype')
if parttype is not None:
kwargs['parttype'] = parttype
Pierre-Yves David
bundle2: support transmission of params error over the wire...
r21622 params = inpart.params.get('params')
if params is not None:
kwargs['params'] = params.split('\0')
Pierre-Yves David
bundle2: rename error exception class for unsupported feature...
r26393 raise error.BundleUnknownFeatureError(**kwargs)
Pierre-Yves David
bundle2: gracefully handle PushRaced error during unbundle...
r21186
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('error:pushraced', ('message',))
Pierre-Yves David
bundle2: fix names for error part handler...
r24741 def handleerrorpushraced(op, inpart):
Pierre-Yves David
bundle2: gracefully handle PushRaced error during unbundle...
r21186 """Used to transmit push race error over the wire"""
Pierre-Yves David
bundle2: use the new ``part.params`` dictionary...
r21611 raise error.ResponseError(_('push failed:'), inpart.params['message'])
Pierre-Yves David
bundle: introduce a listkey handler...
r21655
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('listkeys', ('namespace',))
Pierre-Yves David
bundle: introduce a listkey handler...
r21655 def handlelistkeys(op, inpart):
"""retrieve pushkey namespace content stored in a bundle2"""
namespace = inpart.params['namespace']
r = pushkey.decodekeys(inpart.read())
op.records.add('listkeys', (namespace, r))
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('pushkey', ('namespace', 'key', 'old', 'new'))
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660 def handlepushkey(op, inpart):
"""process a pushkey request"""
dec = pushkey.decode
namespace = dec(inpart.params['namespace'])
key = dec(inpart.params['key'])
old = dec(inpart.params['old'])
new = dec(inpart.params['new'])
Durham Goode
bundle2: add op.gettransaction() to handlers that need the lock...
r26565 # Grab the transaction to ensure that we have the lock before performing the
# pushkey.
if op.ui.configbool('experimental', 'bundle2lazylocking'):
op.gettransaction()
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660 ret = op.repo.pushkey(namespace, key, old, new)
record = {'namespace': namespace,
'key': key,
'old': old,
'new': new}
op.records.add('pushkey', record)
if op.reply is not None:
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 rpart = op.reply.newpart('reply:pushkey')
Augie Fackler
bundle2: use bytestr() instead of str() to convert part id to bytes...
r33638 rpart.addparam(
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660 rpart.addparam('return', '%i' % ret, mandatory=False)
Pierre-Yves David
bundle2: abort when a mandatory pushkey part fails...
r25481 if inpart.mandatory and not ret:
Pierre-Yves David
bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error...
r25484 kwargs = {}
for key in ('namespace', 'key', 'new', 'old', 'ret'):
if key in inpart.params:
kwargs[key] = inpart.params[key]
raise error.PushkeyFailed(partid=str(inpart.id), **kwargs)
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660
Martin von Zweigbergk
bundle: add config option to include phases...
r33031 @parthandler('phase-heads')
def handlephases(op, inpart):
"""apply phases from bundle part to repo"""
Boris Feld
phases: move the binary decoding function in the phases module...
r34321 headsbyphase = phases.binarydecode(inpart)
Boris Feld
bundle2: only grab a transaction when 'phase-heads' affect the repository...
r34322 phases.updatephases(op.repo.unfiltered(), op.gettransaction, headsbyphase)
Martin von Zweigbergk
bundle: add config option to include phases...
r33031
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('reply:pushkey', ('return', 'in-reply-to'))
Pierre-Yves David
bundle2: add ``pushkey`` support...
r21660 def handlepushkeyreply(op, inpart):
"""retrieve the result of a pushkey request"""
ret = int(inpart.params['return'])
partid = int(inpart.params['in-reply-to'])
op.records.add('pushkey', {'return': ret}, partid)
Pierre-Yves David
bundle2: add an obsmarkers part handler...
r22336
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('obsmarkers')
Pierre-Yves David
bundle2: add an obsmarkers part handler...
r22336 def handleobsmarker(op, inpart):
"""add a stream of obsmarkers to the repo"""
tr = op.gettransaction()
Pierre-Yves David
obsolete: experimental flag to get debug about obsmarkers exchange...
r24733 markerdata = inpart.read()
Jun Wu
codemod: register core configitems using a script...
r33499 if op.ui.config('experimental', 'obsmarkers-exchange-debug'):
Pierre-Yves David
obsolete: experimental flag to get debug about obsmarkers exchange...
r24733 op.ui.write(('obsmarker-exchange: %i bytes received\n')
% len(markerdata))
Pierre-Yves David
bundle2: gracefully skip 'obsmarkers' part if evolution is disabled...
r26685 # The mergemarkers call will crash if marker creation is not enabled.
# we want to avoid this if the part is advisory.
if not inpart.mandatory and op.repo.obsstore.readonly:
op.repo.ui.debug('ignoring obsolescence markers, feature not enabled')
return
Pierre-Yves David
obsolete: experimental flag to get debug about obsmarkers exchange...
r24733 new = op.repo.obsstore.mergemarkers(tr, markerdata)
obsolete: invalidate "volatile" set cache after merging marker...
r32314 op.repo.invalidatevolatilesets()
Pierre-Yves David
obsmarker: write a message with the number of markers added through bundle2
r22337 if new:
op.repo.ui.status(_('%i new obsolescence markers\n') % new)
Pierre-Yves David
obssmarker: add a bundle2 record with the number of markers added
r22338 op.records.add('obsmarkers', {'new': new})
Pierre-Yves David
obsmarker: produce a reply part for markers received through bundle2
r22340 if op.reply is not None:
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 rpart = op.reply.newpart('reply:obsmarkers')
Augie Fackler
bundle2: use bytestr() instead of str() to convert part id to bytes...
r33638 rpart.addparam(
'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
Pierre-Yves David
obsmarker: produce a reply part for markers received through bundle2
r22340 rpart.addparam('new', '%i' % new, mandatory=False)
Pierre-Yves David
bundle2: rename format, parts and config to final names...
r24686 @parthandler('reply:obsmarkers', ('new', 'in-reply-to'))
Martin von Zweigbergk
bundle2: rename duplicate handlepushkeyreply to handleobsmarkerreply...
r25506 def handleobsmarkerreply(op, inpart):
Pierre-Yves David
obsmarker: produce a reply part for markers received through bundle2
r22340 """retrieve the result of a pushkey request"""
ret = int(inpart.params['new'])
partid = int(inpart.params['in-reply-to'])
op.records.add('obsmarkers', {'new': ret}, partid)
Gregory Szorc
bundle2: part handler for processing .hgtags fnodes mappings...
r25401
@parthandler('hgtagsfnodes')
def handlehgtagsfnodes(op, inpart):
"""Applies .hgtags fnodes cache entries to the local repo.
Payload is pairs of 20 byte changeset nodes and filenodes.
"""
Durham Goode
bundle2: add op.gettransaction() to handlers that need the lock...
r26565 # Grab the transaction so we ensure that we have the lock at this point.
if op.ui.configbool('experimental', 'bundle2lazylocking'):
op.gettransaction()
Gregory Szorc
bundle2: part handler for processing .hgtags fnodes mappings...
r25401 cache = tags.hgtagsfnodescache(op.repo.unfiltered())
count = 0
while True:
node = inpart.read(20)
fnode = inpart.read(20)
if len(node) < 20 or len(fnode) < 20:
Gregory Szorc
bundle2: reword debug message for invalid .hgtags data...
r25641 op.ui.debug('ignoring incomplete received .hgtags fnodes data\n')
Gregory Szorc
bundle2: part handler for processing .hgtags fnodes mappings...
r25401 break
cache.setfnode(node, fnode)
count += 1
cache.write()
op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
Pulkit Goyal
pushvars: move fb extension pushvars to core...
r33656
@parthandler('pushvars')
def bundle2getvars(op, part):
'''unbundle a bundle2 containing shellvars on the server'''
# An option to disable unbundling on server-side for security reasons
Pulkit Goyal
pushvars: add a coreconfigitem for push.pushvars.server...
r33834 if op.ui.configbool('push', 'pushvars.server'):
Pulkit Goyal
pushvars: move fb extension pushvars to core...
r33656 hookargs = {}
for key, value in part.advisoryparams:
key = key.upper()
# We want pushed variables to have USERVAR_ prepended so we know
# they came from the --pushvar flag.
key = "USERVAR_" + key
hookargs[key] = value
op.addhookargs(hookargs)