bundle2: produce a bundle2 reply

We do not know yet what kind of data future features and extensions will need to
exchange. To handle that, bundle2 allows to send arbitrary content to the
server. As a consequence, we need to be able to reply arbitrary content to the
client. And, we can use bundle2 to transmit those arbitrary data.

When a client will push a bundle2 to the server, the server will reply with a
bundle2 itself.

This changeset installs the first stone of this logic and test it.

mercurial/bundle2.py

@@ -1,575 +1,579
 # bundle2.py - generic container format to transmit arbitrary data.
 #
 # Copyright 2013 Facebook, Inc.
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 """Handling of the new bundle2 format
 
 The goal of bundle2 is to act as an atomically packet to transmit a set of
 payloads in an application agnostic way. It consist in a sequence of "parts"
 that will be handed to and processed by the application layer.
 
 
 General format architecture
 ===========================
 
 The format is architectured as follow
 
  - magic string
  - stream level parameters
  - payload parts (any number)
  - end of stream marker.
 
 the Binary format
 ============================
 
 All numbers are unsigned and big endian.
 
 stream level parameters
 ------------------------
 
 Binary format is as follow
 
 :params size: (16 bits integer)
 
   The total number of Bytes used by the parameters
 
 :params value: arbitrary number of Bytes
 
   A blob of `params size` containing the serialized version of all stream level
   parameters.
 
   The blob contains a space separated list of parameters. parameter with value
   are stored in the form `<name>=<value>`. Both name and value are urlquoted.
 
   Empty name are obviously forbidden.
 
   Name MUST start with a letter. If this first letter is lower case, the
   parameter is advisory and can be safefly ignored. However when the first
   letter is capital, the parameter is mandatory and the bundling process MUST
   stop if he is not able to proceed it.
 
   Stream parameters use a simple textual format for two main reasons:
 
   - Stream level parameters should remains simple and we want to discourage any
     crazy usage.
   - Textual data allow easy human inspection of a the bundle2 header in case of
     troubles.
 
   Any Applicative level options MUST go into a bundle2 part instead.
 
 Payload part
 ------------------------
 
 Binary format is as follow
 
 :header size: (16 bits inter)
 
   The total number of Bytes used by the part headers. When the header is empty
   (size = 0) this is interpreted as the end of stream marker.
 
 :header:
 
     The header defines how to interpret the part. It contains two piece of
     data: the part type, and the part parameters.
 
     The part type is used to route an application level handler, that can
     interpret payload.
 
     Part parameters are passed to the application level handler.  They are
     meant to convey information that will help the application level object to
     interpret the part payload.
 
     The binary format of the header is has follow
 
     :typesize: (one byte)
 
     :typename: alphanumerical part name
 
     :partid: A 32bits integer (unique in the bundle) that can be used to refer
              to this part.
 
     :parameters:
 
         Part's parameter may have arbitraty content, the binary structure is::
 
             <mandatory-count><advisory-count><param-sizes><param-data>
 
         :mandatory-count: 1 byte, number of mandatory parameters
 
         :advisory-count:  1 byte, number of advisory parameters
 
         :param-sizes:
 
             N couple of bytes, where N is the total number of parameters. Each
             couple contains (<size-of-key>, <size-of-value) for one parameter.
 
         :param-data:
 
             A blob of bytes from which each parameter key and value can be
             retrieved using the list of size couples stored in the previous
             field.
 
             Mandatory parameters comes first, then the advisory ones.
 
 :payload:
 
     payload is a series of `<chunksize><chunkdata>`.
 
     `chunksize` is a 32 bits integer, `chunkdata` are plain bytes (as much as
     `chunksize` says)` The payload part is concluded by a zero size chunk.
 
     The current implementation always produces either zero or one chunk.
     This is an implementation limitation that will ultimatly be lifted.
 
 Bundle processing
 ============================
 
 Each part is processed in order using a "part handler". Handler are registered
 for a certain part type.
 
 The matching of a part to its handler is case insensitive. The case of the
 part type is used to know if a part is mandatory or advisory. If the Part type
 contains any uppercase char it is considered mandatory. When no handler is
 known for a Mandatory part, the process is aborted and an exception is raised.
 If the part is advisory and no handler is known, the part is ignored. When the
 process is aborted, the full bundle is still read from the stream to keep the
 channel usable. But none of the part read from an abort are processed. In the
 future, dropping the stream may become an option for channel we do not care to
 preserve.
 """
 
 import util
 import struct
 import urllib
 import string
 import StringIO
 
 import changegroup
 from i18n import _
 
 _pack = struct.pack
 _unpack = struct.unpack
 
 _magicstring = 'HG20'
 
 _fstreamparamsize = '>H'
 _fpartheadersize = '>H'
 _fparttypesize = '>B'
 _fpartid = '>I'
 _fpayloadsize = '>I'
 _fpartparamcount = '>BB'
 
 def _makefpartparamsizes(nbparams):
     """return a struct format to read part parameter sizes
 
     The number parameters is variable so we need to build that format
     dynamically.
     """
     return '>'+('BB'*nbparams)
 
 parthandlermapping = {}
 
 def parthandler(parttype):
     """decorator that register a function as a bundle2 part handler
 
     eg::
 
         @parthandler('myparttype')
         def myparttypehandler(...):
             '''process a part of type "my part".'''
             ...
     """
     def _decorator(func):
         lparttype = parttype.lower() # enforce lower case matching.
         assert lparttype not in parthandlermapping
         parthandlermapping[lparttype] = func
         return func
     return _decorator
 
 class unbundlerecords(object):
     """keep record of what happens during and unbundle
 
     New records are added using `records.add('cat', obj)`. Where 'cat' is a
     category of record and obj is an arbitraty object.
 
     `records['cat']` will return all entries of this category 'cat'.
 
     Iterating on the object itself will yield `('category', obj)` tuples
     for all entries.
 
     All iterations happens in chronological order.
     """
 
     def __init__(self):
         self._categories = {}
         self._sequences = []
         self._replies = {}
 
     def add(self, category, entry, inreplyto=None):
         """add a new record of a given category.
 
         The entry can then be retrieved in the list returned by
         self['category']."""
         self._categories.setdefault(category, []).append(entry)
         self._sequences.append((category, entry))
         if inreplyto is not None:
             self.getreplies(inreplyto).add(category, entry)
 
     def getreplies(self, partid):
         """get the subrecords that replies to a specific part"""
         return self._replies.setdefault(partid, unbundlerecords())
 
     def __getitem__(self, cat):
         return tuple(self._categories.get(cat, ()))
 
     def __iter__(self):
         return iter(self._sequences)
 
     def __len__(self):
         return len(self._sequences)
 
     def __nonzero__(self):
         return bool(self._sequences)
 
 class bundleoperation(object):
     """an object that represents a single bundling process
 
     Its purpose is to carry unbundle-related objects and states.
 
     A new object should be created at the beginning of each bundle processing.
     The object is to be returned by the processing function.
 
     The object has very little content now it will ultimately contain:
     * an access to the repo the bundle is applied to,
     * a ui object,
     * a way to retrieve a transaction to add changes to the repo,
     * a way to record the result of processing each part,
     * a way to construct a bundle response when applicable.
     """
 
     def __init__(self, repo, transactiongetter):
         self.repo = repo
         self.ui = repo.ui
         self.records = unbundlerecords()
         self.gettransaction = transactiongetter
+        self.reply = None
 
 class TransactionUnavailable(RuntimeError):
     pass
 
 def _notransaction():
     """default method to get a transaction while processing a bundle
 
     Raise an exception to highlight the fact that no transaction was expected
     to be created"""
     raise TransactionUnavailable()
 
 def processbundle(repo, unbundler, transactiongetter=_notransaction):
     """This function process a bundle, apply effect to/from a repo
 
     It iterates over each part then searches for and uses the proper handling
     code to process the part. Parts are processed in order.
 
     This is very early version of this function that will be strongly reworked
     before final usage.
 
     Unknown Mandatory part will abort the process.
     """
     op = bundleoperation(repo, transactiongetter)
     # todo:
+    # - only create reply bundle if requested.
+    op.reply = bundle20(op.ui)
+    # todo:
     # - replace this is a init function soon.
     # - exception catching
     unbundler.params
     iterparts = iter(unbundler)
     try:
         for part in iterparts:
             parttype = part.type
             # part key are matched lower case
             key = parttype.lower()
             try:
                 handler = parthandlermapping[key]
                 op.ui.debug('found a handler for part %r\n' % parttype)
             except KeyError:
                 if key != parttype: # mandatory parts
                     # todo:
                     # - use a more precise exception
                     raise
                 op.ui.debug('ignoring unknown advisory part %r\n' % key)
                 # todo:
                 # - consume the part once we use streaming
                 continue
             handler(op, part)
     except Exception:
         for part in iterparts:
             pass # consume the bundle content
         raise
     return op
 
 class bundle20(object):
     """represent an outgoing bundle2 container
 
     Use the `addparam` method to add stream level parameter. and `addpart` to
     populate it. Then call `getchunks` to retrieve all the binary chunks of
     datathat compose the bundle2 container."""
 
     def __init__(self, ui):
         self.ui = ui
         self._params = []
         self._parts = []
 
     def addparam(self, name, value=None):
         """add a stream level parameter"""
         if not name:
             raise ValueError('empty parameter name')
         if name[0] not in string.letters:
             raise ValueError('non letter first character: %r' % name)
         self._params.append((name, value))
 
     def addpart(self, part):
         """add a new part to the bundle2 container
 
         Parts contains the actuall applicative payload."""
         assert part.id is None
         part.id = len(self._parts) # very cheap counter
         self._parts.append(part)
 
     def getchunks(self):
         self.ui.debug('start emission of %s stream\n' % _magicstring)
         yield _magicstring
         param = self._paramchunk()
         self.ui.debug('bundle parameter: %s\n' % param)
         yield _pack(_fstreamparamsize, len(param))
         if param:
             yield param
 
         self.ui.debug('start of parts\n')
         for part in self._parts:
             self.ui.debug('bundle part: "%s"\n' % part.type)
             for chunk in part.getchunks():
                 yield chunk
         self.ui.debug('end of bundle\n')
         yield '\0\0'
 
     def _paramchunk(self):
         """return a encoded version of all stream parameters"""
         blocks = []
         for par, value in self._params:
             par = urllib.quote(par)
             if value is not None:
                 value = urllib.quote(value)
                 par = '%s=%s' % (par, value)
             blocks.append(par)
         return ' '.join(blocks)
 
 class unbundle20(object):
     """interpret a bundle2 stream
 
     (this will eventually yield parts)"""
 
     def __init__(self, ui, fp):
         self.ui = ui
         self._fp = fp
         header = self._readexact(4)
         magic, version = header[0:2], header[2:4]
         if magic != 'HG':
             raise util.Abort(_('not a Mercurial bundle'))
         if version != '20':
             raise util.Abort(_('unknown bundle version %s') % version)
         self.ui.debug('start processing of %s stream\n' % header)
 
     def _unpack(self, format):
         """unpack this struct format from the stream"""
         data = self._readexact(struct.calcsize(format))
         return _unpack(format, data)
 
     def _readexact(self, size):
         """read exactly <size> bytes from the stream"""
         return changegroup.readexactly(self._fp, size)
 
     @util.propertycache
     def params(self):
         """dictionnary of stream level parameters"""
         self.ui.debug('reading bundle2 stream parameters\n')
         params = {}
         paramssize = self._unpack(_fstreamparamsize)[0]
         if paramssize:
             for p in self._readexact(paramssize).split(' '):
                 p = p.split('=', 1)
                 p = [urllib.unquote(i) for i in p]
                 if len(p) < 2:
                     p.append(None)
                 self._processparam(*p)
                 params[p[0]] = p[1]
         return params
 
     def _processparam(self, name, value):
         """process a parameter, applying its effect if needed
 
         Parameter starting with a lower case letter are advisory and will be
         ignored when unknown.  Those starting with an upper case letter are
         mandatory and will this function will raise a KeyError when unknown.
 
         Note: no option are currently supported. Any input will be either
               ignored or failing.
         """
         if not name:
             raise ValueError('empty parameter name')
         if name[0] not in string.letters:
             raise ValueError('non letter first character: %r' % name)
         # Some logic will be later added here to try to process the option for
         # a dict of known parameter.
         if name[0].islower():
             self.ui.debug("ignoring unknown parameter %r\n" % name)
         else:
             raise KeyError(name)
 
 
     def __iter__(self):
         """yield all parts contained in the stream"""
         # make sure param have been loaded
         self.params
         self.ui.debug('start extraction of bundle2 parts\n')
         part = self._readpart()
         while part is not None:
             yield part
             part = self._readpart()
         self.ui.debug('end of bundle2 stream\n')
 
     def _readpart(self):
         """return None when an end of stream markers is reach"""
 
         headersize = self._unpack(_fpartheadersize)[0]
         self.ui.debug('part header size: %i\n' % headersize)
         if not headersize:
             return None
         headerblock = self._readexact(headersize)
         # some utility to help reading from the header block
         self._offset = 0 # layer violation to have something easy to understand
         def fromheader(size):
             """return the next <size> byte from the header"""
             offset = self._offset
             data = headerblock[offset:(offset + size)]
             self._offset = offset + size
             return data
         def unpackheader(format):
             """read given format from header
 
             This automatically compute the size of the format to read."""
             data = fromheader(struct.calcsize(format))
             return _unpack(format, data)
 
         typesize = unpackheader(_fparttypesize)[0]
         parttype = fromheader(typesize)
         self.ui.debug('part type: "%s"\n' % parttype)
         partid = unpackheader(_fpartid)[0]
         self.ui.debug('part id: "%s"\n' % partid)
         ## reading parameters
         # param count
         mancount, advcount = unpackheader(_fpartparamcount)
         self.ui.debug('part parameters: %i\n' % (mancount + advcount))
         # param size
         paramsizes = unpackheader(_makefpartparamsizes(mancount + advcount))
         # make it a list of couple again
         paramsizes = zip(paramsizes[::2], paramsizes[1::2])
         # split mandatory from advisory
         mansizes = paramsizes[:mancount]
         advsizes = paramsizes[mancount:]
         # retrive param value
         manparams = []
         for key, value in mansizes:
             manparams.append((fromheader(key), fromheader(value)))
         advparams = []
         for key, value in advsizes:
             advparams.append((fromheader(key), fromheader(value)))
         del self._offset # clean up layer, nobody saw anything.
         ## part payload
         payload = []
         payloadsize = self._unpack(_fpayloadsize)[0]
         self.ui.debug('payload chunk size: %i\n' % payloadsize)
         while payloadsize:
             payload.append(self._readexact(payloadsize))
             payloadsize = self._unpack(_fpayloadsize)[0]
             self.ui.debug('payload chunk size: %i\n' % payloadsize)
         payload = ''.join(payload)
         current = part(parttype, manparams, advparams, data=payload)
         current.id = partid
         return current
 
 
 class part(object):
     """A bundle2 part contains application level payload
 
     The part `type` is used to route the part to the application level
     handler.
     """
 
     def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
                  data=''):
         self.id = None
         self.type = parttype
         self.data = data
         self.mandatoryparams = mandatoryparams
         self.advisoryparams = advisoryparams
 
     def getchunks(self):
         #### header
         ## parttype
         header = [_pack(_fparttypesize, len(self.type)),
                   self.type, _pack(_fpartid, self.id),
                  ]
         ## parameters
         # count
         manpar = self.mandatoryparams
         advpar = self.advisoryparams
         header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
         # size
         parsizes = []
         for key, value in manpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         for key, value in advpar:
             parsizes.append(len(key))
             parsizes.append(len(value))
         paramsizes = _pack(_makefpartparamsizes(len(parsizes) / 2), *parsizes)
         header.append(paramsizes)
         # key, value
         for key, value in manpar:
             header.append(key)
             header.append(value)
         for key, value in advpar:
             header.append(key)
             header.append(value)
         ## finalize header
         headerchunk = ''.join(header)
         yield _pack(_fpartheadersize, len(headerchunk))
         yield headerchunk
         ## payload
         # we only support fixed size data now.
         # This will be improved in the future.
         if len(self.data):
             yield _pack(_fpayloadsize, len(self.data))
             yield self.data
         # end of payload
         yield _pack(_fpayloadsize, 0)
 
 @parthandler('changegroup')
 def handlechangegroup(op, part):
     """apply a changegroup part on the repo
 
     This is a very early implementation that will massive rework before being
     inflicted to any end-user.
     """
     # Make sure we trigger a transaction creation
     #
     # The addchangegroup function will get a transaction object by itself, but
     # we need to make sure we trigger the creation of a transaction object used
     # for the whole processing scope.
     op.gettransaction()
     data = StringIO.StringIO(part.data)
     data.seek(0)
     cg = changegroup.readbundle(data, 'bundle2part')
     ret = changegroup.addchangegroup(op.repo, cg, 'bundle2', 'bundle2')
     op.records.add('changegroup', {'return': ret})
 
 

tests/test-bundle2.t

@@ -1,610 +1,673
 
 Create an extension to test bundle2 API
 
   $ cat > bundle2.py << EOF
   > """A small extension to test bundle2 implementation
   > 
   > Current bundle2 implementation is far too limited to be used in any core
   > code. We still need to be able to test it while it grow up.
   > """
   > 
   > import sys
   > from mercurial import cmdutil
   > from mercurial import util
   > from mercurial import bundle2
   > from mercurial import scmutil
   > from mercurial import discovery
   > from mercurial import changegroup
   > cmdtable = {}
   > command = cmdutil.command(cmdtable)
   > 
   > ELEPHANTSSONG = """Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko
   > Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
   > Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko."""
   > assert len(ELEPHANTSSONG) == 178 # future test say 178 bytes, trust it.
   > 
   > @bundle2.parthandler('test:song')
   > def songhandler(op, part):
   >     """handle a "test:song" bundle2 part, printing the lyrics on stdin"""
   >     op.ui.write('The choir starts singing:\n')
   >     verses = 0
   >     for line in part.data.split('\n'):
   >         op.ui.write('    %s\n' % line)
   >         verses += 1
   >     op.records.add('song', {'verses': verses})
   > 
+  > @bundle2.parthandler('test:ping')
+  > def pinghandler(op, part):
+  >     op.ui.write('received ping request (id %i)\n' % part.id)
+  >     if op.reply is not None:
+  >         op.reply.addpart(bundle2.part('test:pong',
+  >                                       [('in-reply-to', str(part.id))]))
+  > 
   > @command('bundle2',
   >          [('', 'param', [], 'stream level parameter'),
   >           ('', 'unknown', False, 'include an unknown mandatory part in the bundle'),
   >           ('', 'parts', False, 'include some arbitrary parts to the bundle'),
   >           ('r', 'rev', [], 'includes those changeset in the bundle'),],
   >          '[OUTPUTFILE]')
   > def cmdbundle2(ui, repo, path=None, **opts):
   >     """write a bundle2 container on standard ouput"""
   >     bundler = bundle2.bundle20(ui)
   >     for p in opts['param']:
   >         p = p.split('=', 1)
   >         try:
   >             bundler.addparam(*p)
   >         except ValueError, exc:
   >             raise util.Abort('%s' % exc)
   > 
   >     revs = opts['rev']
   >     if 'rev' in opts:
   >         revs = scmutil.revrange(repo, opts['rev'])
   >         if revs:
   >             # very crude version of a changegroup part creation
   >             bundled = repo.revs('%ld::%ld', revs, revs)
   >             headmissing = [c.node() for c in repo.set('heads(%ld)', revs)]
   >             headcommon  = [c.node() for c in repo.set('parents(%ld) - %ld', revs, revs)]
   >             outgoing = discovery.outgoing(repo.changelog, headcommon, headmissing)
   >             cg = changegroup.getlocalbundle(repo, 'test:bundle2', outgoing, None)
   >             assert cg is not None
   >             # make me lazy later
   >             tempname = changegroup.writebundle(cg, None, 'HG10UN')
   >             data = open(tempname).read()
   >             part = bundle2.part('changegroup', data=data)
   >             bundler.addpart(part)
   > 
   >     if opts['parts']:
   >        part = bundle2.part('test:empty')
   >        bundler.addpart(part)
   >        # add a second one to make sure we handle multiple parts
   >        part = bundle2.part('test:empty')
   >        bundler.addpart(part)
   >        part = bundle2.part('test:song', data=ELEPHANTSSONG)
   >        bundler.addpart(part)
   >        part = bundle2.part('test:math',
   >                            [('pi', '3.14'), ('e', '2.72')],
   >                            [('cooking', 'raw')],
   >                            '42')
   >        bundler.addpart(part)
   >     if opts['unknown']:
   >        part = bundle2.part('test:UNKNOWN',
   >                            data='some random content')
   >        bundler.addpart(part)
+  >     if opts['parts']:
+  >        part = bundle2.part('test:ping')
+  >        bundler.addpart(part)
   > 
   >     if path is None:
   >        file = sys.stdout
   >     else:
   >         file = open(path, 'w')
   > 
   >     for chunk in bundler.getchunks():
   >         file.write(chunk)
   > 
   > @command('unbundle2', [], '')
-  > def cmdunbundle2(ui, repo):
+  > def cmdunbundle2(ui, repo, replypath=None):
   >     """process a bundle2 stream from stdin on the current repo"""
   >     try:
   >         tr = None
   >         lock = repo.lock()
   >         tr = repo.transaction('processbundle')
   >         try:
   >             unbundler = bundle2.unbundle20(ui, sys.stdin)
   >             op = bundle2.processbundle(repo, unbundler, lambda: tr)
   >             tr.close()
   >         except KeyError, exc:
   >             raise util.Abort('missing support for %s' % exc)
   >     finally:
   >         if tr is not None:
   >             tr.release()
   >         lock.release()
   >         remains = sys.stdin.read()
   >         ui.write('%i unread bytes\n' % len(remains))
   >     if op.records['song']:
   >         totalverses = sum(r['verses'] for r in op.records['song'])
   >         ui.write('%i total verses sung\n' % totalverses)
   >     for rec in op.records['changegroup']:
   >         ui.write('addchangegroup return: %i\n' % rec['return'])
+  >     if op.reply is not None and replypath is not None:
+  >         file = open(replypath, 'w')
+  >         for chunk in op.reply.getchunks():
+  >             file.write(chunk)
   > 
   > @command('statbundle2', [], '')
   > def cmdstatbundle2(ui, repo):
   >     """print statistic on the bundle2 container read from stdin"""
   >     unbundler = bundle2.unbundle20(ui, sys.stdin)
   >     try:
   >         params = unbundler.params
   >     except KeyError, exc:
   >        raise util.Abort('unknown parameters: %s' % exc)
   >     ui.write('options count: %i\n' % len(params))
   >     for key in sorted(params):
   >         ui.write('- %s\n' % key)
   >         value = params[key]
   >         if value is not None:
   >             ui.write('    %s\n' % value)
   >     parts = list(unbundler)
   >     ui.write('parts count:   %i\n' % len(parts))
   >     for p in parts:
   >         ui.write('  :%s:\n' % p.type)
   >         ui.write('    mandatory: %i\n' % len(p.mandatoryparams))
   >         ui.write('    advisory: %i\n' % len(p.advisoryparams))
   >         ui.write('    payload: %i bytes\n' % len(p.data))
   > EOF
   $ cat >> $HGRCPATH << EOF
   > [extensions]
   > bundle2=$TESTTMP/bundle2.py
   > [server]
   > bundle2=True
   > EOF
 
 The extension requires a repo (currently unused)
 
   $ hg init main
   $ cd main
   $ touch a
   $ hg add a
   $ hg commit -m 'a'
 
 
 Empty bundle
 =================
 
 - no option
 - no parts
 
 Test bundling
 
   $ hg bundle2
   HG20\x00\x00\x00\x00 (no-eol) (esc)
 
 Test unbundling
 
   $ hg bundle2 | hg statbundle2
   options count: 0
   parts count:   0
 
 Test old style bundle are detected and refused
 
   $ hg bundle --all ../bundle.hg
   1 changesets found
   $ hg statbundle2 < ../bundle.hg
   abort: unknown bundle version 10
   [255]
 
 Test parameters
 =================
 
 - some options
 - no parts
 
 advisory parameters, no value
 -------------------------------
 
 Simplest possible parameters form
 
 Test generation simple option
 
   $ hg bundle2 --param 'caution'
   HG20\x00\x07caution\x00\x00 (no-eol) (esc)
 
 Test unbundling
 
   $ hg bundle2 --param 'caution' | hg statbundle2
   options count: 1
   - caution
   parts count:   0
 
 Test generation multiple option
 
   $ hg bundle2 --param 'caution' --param 'meal'
   HG20\x00\x0ccaution meal\x00\x00 (no-eol) (esc)
 
 Test unbundling
 
   $ hg bundle2 --param 'caution' --param 'meal' | hg statbundle2
   options count: 2
   - caution
   - meal
   parts count:   0
 
 advisory parameters, with value
 -------------------------------
 
 Test generation
 
   $ hg bundle2 --param 'caution' --param 'meal=vegan' --param 'elephants'
   HG20\x00\x1ccaution meal=vegan elephants\x00\x00 (no-eol) (esc)
 
 Test unbundling
 
   $ hg bundle2 --param 'caution' --param 'meal=vegan' --param 'elephants' | hg statbundle2
   options count: 3
   - caution
   - elephants
   - meal
       vegan
   parts count:   0
 
 parameter with special char in value
 ---------------------------------------------------
 
 Test generation
 
   $ hg bundle2 --param 'e|! 7/=babar%#==tutu' --param simple
   HG20\x00)e%7C%21%207/=babar%25%23%3D%3Dtutu simple\x00\x00 (no-eol) (esc)
 
 Test unbundling
 
   $ hg bundle2 --param 'e|! 7/=babar%#==tutu' --param simple | hg statbundle2
   options count: 2
   - e|! 7/
       babar%#==tutu
   - simple
   parts count:   0
 
 Test unknown mandatory option
 ---------------------------------------------------
 
   $ hg bundle2 --param 'Gravity' | hg statbundle2
   abort: unknown parameters: 'Gravity'
   [255]
 
 Test debug output
 ---------------------------------------------------
 
 bundling debug
 
   $ hg bundle2 --debug --param 'e|! 7/=babar%#==tutu' --param simple ../out.hg2
   start emission of HG20 stream
   bundle parameter: e%7C%21%207/=babar%25%23%3D%3Dtutu simple
   start of parts
   end of bundle
 
 file content is ok
 
   $ cat ../out.hg2
   HG20\x00)e%7C%21%207/=babar%25%23%3D%3Dtutu simple\x00\x00 (no-eol) (esc)
 
 unbundling debug
 
   $ hg statbundle2 --debug < ../out.hg2
   start processing of HG20 stream
   reading bundle2 stream parameters
   ignoring unknown parameter 'e|! 7/'
   ignoring unknown parameter 'simple'
   options count: 2
   - e|! 7/
       babar%#==tutu
   - simple
   start extraction of bundle2 parts
   part header size: 0
   end of bundle2 stream
   parts count:   0
 
 
 Test buggy input
 ---------------------------------------------------
 
 empty parameter name
 
   $ hg bundle2 --param '' --quiet
   abort: empty parameter name
   [255]
 
 bad parameter name
 
   $ hg bundle2 --param 42babar
   abort: non letter first character: '42babar'
   [255]
 
 
 Test part
 =================
 
   $ hg bundle2 --parts ../parts.hg2 --debug
   start emission of HG20 stream
   bundle parameter: 
   start of parts
   bundle part: "test:empty"
   bundle part: "test:empty"
   bundle part: "test:song"
   bundle part: "test:math"
+  bundle part: "test:ping"
   end of bundle
 
   $ cat ../parts.hg2
   HG20\x00\x00\x00\x11 (esc)
   test:empty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x11 (esc)
   test:empty\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x10	test:song\x00\x00\x00\x02\x00\x00\x00\x00\x00\xb2Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko (esc)
   Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
-  Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko.\x00\x00\x00\x00\x00+	test:math\x00\x00\x00\x03\x02\x01\x02\x04\x01\x04\x07\x03pi3.14e2.72cookingraw\x00\x00\x00\x0242\x00\x00\x00\x00\x00\x00 (no-eol) (esc)
+  Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko.\x00\x00\x00\x00\x00+	test:math\x00\x00\x00\x03\x02\x01\x02\x04\x01\x04\x07\x03pi3.14e2.72cookingraw\x00\x00\x00\x0242\x00\x00\x00\x00\x00\x10	test:ping\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00 (no-eol) (esc)
 
 
   $ hg statbundle2 < ../parts.hg2
   options count: 0
-  parts count:   4
+  parts count:   5
     :test:empty:
       mandatory: 0
       advisory: 0
       payload: 0 bytes
     :test:empty:
       mandatory: 0
       advisory: 0
       payload: 0 bytes
     :test:song:
       mandatory: 0
       advisory: 0
       payload: 178 bytes
     :test:math:
       mandatory: 2
       advisory: 1
       payload: 2 bytes
+    :test:ping:
+      mandatory: 0
+      advisory: 0
+      payload: 0 bytes
 
   $ hg statbundle2 --debug < ../parts.hg2
   start processing of HG20 stream
   reading bundle2 stream parameters
   options count: 0
   start extraction of bundle2 parts
   part header size: 17
   part type: "test:empty"
   part id: "0"
   part parameters: 0
   payload chunk size: 0
   part header size: 17
   part type: "test:empty"
   part id: "1"
   part parameters: 0
   payload chunk size: 0
   part header size: 16
   part type: "test:song"
   part id: "2"
   part parameters: 0
   payload chunk size: 178
   payload chunk size: 0
   part header size: 43
   part type: "test:math"
   part id: "3"
   part parameters: 3
   payload chunk size: 2
   payload chunk size: 0
+  part header size: 16
+  part type: "test:ping"
+  part id: "4"
+  part parameters: 0
+  payload chunk size: 0
   part header size: 0
   end of bundle2 stream
-  parts count:   4
+  parts count:   5
     :test:empty:
       mandatory: 0
       advisory: 0
       payload: 0 bytes
     :test:empty:
       mandatory: 0
       advisory: 0
       payload: 0 bytes
     :test:song:
       mandatory: 0
       advisory: 0
       payload: 178 bytes
     :test:math:
       mandatory: 2
       advisory: 1
       payload: 2 bytes
+    :test:ping:
+      mandatory: 0
+      advisory: 0
+      payload: 0 bytes
 
 Test actual unbundling of test part
 =======================================
 
 Process the bundle
 
   $ hg unbundle2 --debug < ../parts.hg2
   start processing of HG20 stream
   reading bundle2 stream parameters
   start extraction of bundle2 parts
   part header size: 17
   part type: "test:empty"
   part id: "0"
   part parameters: 0
   payload chunk size: 0
   ignoring unknown advisory part 'test:empty'
   part header size: 17
   part type: "test:empty"
   part id: "1"
   part parameters: 0
   payload chunk size: 0
   ignoring unknown advisory part 'test:empty'
   part header size: 16
   part type: "test:song"
   part id: "2"
   part parameters: 0
   payload chunk size: 178
   payload chunk size: 0
   found a handler for part 'test:song'
   The choir starts singing:
       Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko
       Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
       Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko.
   part header size: 43
   part type: "test:math"
   part id: "3"
   part parameters: 3
   payload chunk size: 2
   payload chunk size: 0
   ignoring unknown advisory part 'test:math'
+  part header size: 16
+  part type: "test:ping"
+  part id: "4"
+  part parameters: 0
+  payload chunk size: 0
+  found a handler for part 'test:ping'
+  received ping request (id 4)
   part header size: 0
   end of bundle2 stream
   0 unread bytes
   3 total verses sung
 
+Unbundle with an unknown mandatory part
+(should abort)
 
   $ hg bundle2 --parts --unknown ../unknown.hg2
 
   $ hg unbundle2 < ../unknown.hg2
   The choir starts singing:
       Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko
       Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
       Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko.
   0 unread bytes
   abort: missing support for 'test:unknown'
   [255]
 
+unbundle with a reply
+
+  $ hg unbundle2 ../reply.hg2 < ../parts.hg2
+  The choir starts singing:
+      Patali Dirapata, Cromda Cromda Ripalo, Pata Pata, Ko Ko Ko
+      Bokoro Dipoulito, Rondi Rondi Pepino, Pata Pata, Ko Ko Ko
+      Emana Karassoli, Loucra Loucra Ponponto, Pata Pata, Ko Ko Ko.
+  received ping request (id 4)
+  0 unread bytes
+  3 total verses sung
+
+The reply is a bundle
+
+  $ cat ../reply.hg2
+  HG20\x00\x00\x00\x1e	test:pong\x00\x00\x00\x00\x01\x00\x0b\x01in-reply-to4\x00\x00\x00\x00\x00\x00 (no-eol) (esc)
+
+The reply is valid
+
+  $ hg statbundle2 < ../reply.hg2
+  options count: 0
+  parts count:   1
+    :test:pong:
+      mandatory: 1
+      advisory: 0
+      payload: 0 bytes
+
 Support for changegroup
 ===================================
 
   $ hg unbundle $TESTDIR/bundles/rebase.hg
   adding changesets
   adding manifests
   adding file changes
   added 8 changesets with 7 changes to 7 files (+3 heads)
   (run 'hg heads' to see heads, 'hg merge' to merge)
 
   $ hg log -G
   o  changeset:   8:02de42196ebe
   |  tag:         tip
   |  parent:      6:24b6387c8c8c
   |  user:        Nicolas Dumazet <[e-mail]>
   |  date:        Sat Apr 30 15:24:48 2011 +0200
   |  summary:     H
   |
   | o  changeset:   7:eea13746799a
   |/|  parent:      6:24b6387c8c8c
   | |  parent:      5:9520eea781bc
   | |  user:        Nicolas Dumazet <[e-mail]>
   | |  date:        Sat Apr 30 15:24:48 2011 +0200
   | |  summary:     G
   | |
   o |  changeset:   6:24b6387c8c8c
   | |  parent:      1:cd010b8cd998
   | |  user:        Nicolas Dumazet <[e-mail]>
   | |  date:        Sat Apr 30 15:24:48 2011 +0200
   | |  summary:     F
   | |
   | o  changeset:   5:9520eea781bc
   |/   parent:      1:cd010b8cd998
   |    user:        Nicolas Dumazet <[e-mail]>
   |    date:        Sat Apr 30 15:24:48 2011 +0200
   |    summary:     E
   |
   | o  changeset:   4:32af7686d403
   | |  user:        Nicolas Dumazet <[e-mail]>
   | |  date:        Sat Apr 30 15:24:48 2011 +0200
   | |  summary:     D
   | |
   | o  changeset:   3:5fddd98957c8
   | |  user:        Nicolas Dumazet <[e-mail]>
   | |  date:        Sat Apr 30 15:24:48 2011 +0200
   | |  summary:     C
   | |
   | o  changeset:   2:42ccdea3bb16
   |/   user:        Nicolas Dumazet <[e-mail]>
   |    date:        Sat Apr 30 15:24:48 2011 +0200
   |    summary:     B
   |
   o  changeset:   1:cd010b8cd998
      parent:      -1:000000000000
      user:        Nicolas Dumazet <[e-mail]>
      date:        Sat Apr 30 15:24:48 2011 +0200
      summary:     A
   
   @  changeset:   0:3903775176ed
      user:        test
      date:        Thu Jan 01 00:00:00 1970 +0000
      summary:     a
   
 
   $ hg bundle2 --debug --rev '8+7+5+4' ../rev.hg2
   4 changesets found
   list of changesets:
   32af7686d403cf45b5d95f2d70cebea587ac806a
   9520eea781bcca16c1e15acc0ba14335a0e8e5ba
   eea13746799a9e0bfd88f29d3c2e9dc9389f524f
   02de42196ebee42ef284b6780a87cdc96e8eaab6
   bundling: 1/4 changesets (25.00%)
   bundling: 2/4 changesets (50.00%)
   bundling: 3/4 changesets (75.00%)
   bundling: 4/4 changesets (100.00%)
   bundling: 1/4 manifests (25.00%)
   bundling: 2/4 manifests (50.00%)
   bundling: 3/4 manifests (75.00%)
   bundling: 4/4 manifests (100.00%)
   bundling: D 1/3 files (33.33%)
   bundling: E 2/3 files (66.67%)
   bundling: H 3/3 files (100.00%)
   start emission of HG20 stream
   bundle parameter: 
   start of parts
   bundle part: "changegroup"
   end of bundle
 
   $ cat ../rev.hg2
   HG20\x00\x00\x00\x12\x0bchangegroup\x00\x00\x00\x00\x00\x00\x00\x00\x06\x19HG10UN\x00\x00\x00\xa42\xafv\x86\xd4\x03\xcfE\xb5\xd9_-p\xce\xbe\xa5\x87\xac\x80j_\xdd\xd9\x89W\xc8\xa5JMCm\xfe\x1d\xa9\xd8\x7f!\xa1\xb9{\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002\xafv\x86\xd4\x03\xcfE\xb5\xd9_-p\xce\xbe\xa5\x87\xac\x80j\x00\x00\x00\x00\x00\x00\x00)\x00\x00\x00)6e1f4c47ecb533ffd0c8e52cdc88afb6cd39e20c (esc)
   \x00\x00\x00f\x00\x00\x00h\x00\x00\x00\x02D (esc)
   \x00\x00\x00i\x00\x00\x00j\x00\x00\x00\x01D\x00\x00\x00\xa4\x95 \xee\xa7\x81\xbc\xca\x16\xc1\xe1Z\xcc\x0b\xa1C5\xa0\xe8\xe5\xba\xcd\x01\x0b\x8c\xd9\x98\xf3\x98\x1aZ\x81\x15\xf9O\x8d\xa4\xabP`\x89\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x95 \xee\xa7\x81\xbc\xca\x16\xc1\xe1Z\xcc\x0b\xa1C5\xa0\xe8\xe5\xba\x00\x00\x00\x00\x00\x00\x00)\x00\x00\x00)4dece9c826f69490507b98c6383a3009b295837d (esc)
   \x00\x00\x00f\x00\x00\x00h\x00\x00\x00\x02E (esc)
   \x00\x00\x00i\x00\x00\x00j\x00\x00\x00\x01E\x00\x00\x00\xa2\xee\xa17Fy\x9a\x9e\x0b\xfd\x88\xf2\x9d<.\x9d\xc98\x9fRO$\xb68|\x8c\x8c\xae7\x17\x88\x80\xf3\xfa\x95\xde\xd3\xcb\x1c\xf7\x85\x95 \xee\xa7\x81\xbc\xca\x16\xc1\xe1Z\xcc\x0b\xa1C5\xa0\xe8\xe5\xba\xee\xa17Fy\x9a\x9e\x0b\xfd\x88\xf2\x9d<.\x9d\xc98\x9fRO\x00\x00\x00\x00\x00\x00\x00)\x00\x00\x00)365b93d57fdf4814e2b5911d6bacff2b12014441 (esc)
   \x00\x00\x00f\x00\x00\x00h\x00\x00\x00\x00\x00\x00\x00i\x00\x00\x00j\x00\x00\x00\x01G\x00\x00\x00\xa4\x02\xdeB\x19n\xbe\xe4.\xf2\x84\xb6x (esc)
   \x87\xcd\xc9n\x8e\xaa\xb6$\xb68|\x8c\x8c\xae7\x17\x88\x80\xf3\xfa\x95\xde\xd3\xcb\x1c\xf7\x85\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xdeB\x19n\xbe\xe4.\xf2\x84\xb6x (esc)
   \x87\xcd\xc9n\x8e\xaa\xb6\x00\x00\x00\x00\x00\x00\x00)\x00\x00\x00)8bee48edc7318541fc0013ee41b089276a8c24bf (esc)
   \x00\x00\x00f\x00\x00\x00f\x00\x00\x00\x02H (esc)
   \x00\x00\x00g\x00\x00\x00h\x00\x00\x00\x01H\x00\x00\x00\x00\x00\x00\x00\x8bn\x1fLG\xec\xb53\xff\xd0\xc8\xe5,\xdc\x88\xaf\xb6\xcd9\xe2\x0cf\xa5\xa0\x18\x17\xfd\xf5#\x9c'8\x02\xb5\xb7a\x8d\x05\x1c\x89\xe4\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002\xafv\x86\xd4\x03\xcfE\xb5\xd9_-p\xce\xbe\xa5\x87\xac\x80j\x00\x00\x00\x81\x00\x00\x00\x81\x00\x00\x00+D\x00c3f1ca2924c16a19b0656a84900e504e5b0aec2d (esc)
   \x00\x00\x00\x8bM\xec\xe9\xc8&\xf6\x94\x90P{\x98\xc68:0	\xb2\x95\x83}\x00}\x8c\x9d\x88\x84\x13%\xf5\xc6\xb0cq\xb3[N\x8a+\x1a\x83\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x95 \xee\xa7\x81\xbc\xca\x16\xc1\xe1Z\xcc\x0b\xa1C5\xa0\xe8\xe5\xba\x00\x00\x00+\x00\x00\x00\xac\x00\x00\x00+E\x009c6fd0350a6c0d0c49d4a9c5017cf07043f54e58 (esc)
   \x00\x00\x00\x8b6[\x93\xd5\x7f\xdfH\x14\xe2\xb5\x91\x1dk\xac\xff+\x12\x01DA(\xa5\x84\xc6^\xf1!\xf8\x9e\xb6j\xb7\xd0\xbc\x15=\x80\x99\xe7\xceM\xec\xe9\xc8&\xf6\x94\x90P{\x98\xc68:0	\xb2\x95\x83}\xee\xa17Fy\x9a\x9e\x0b\xfd\x88\xf2\x9d<.\x9d\xc98\x9fRO\x00\x00\x00V\x00\x00\x00V\x00\x00\x00+F\x0022bfcfd62a21a3287edbd4d656218d0f525ed76a (esc)
   \x00\x00\x00\x97\x8b\xeeH\xed\xc71\x85A\xfc\x00\x13\xeeA\xb0\x89'j\x8c$\xbf(\xa5\x84\xc6^\xf1!\xf8\x9e\xb6j\xb7\xd0\xbc\x15=\x80\x99\xe7\xce\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xdeB\x19n\xbe\xe4.\xf2\x84\xb6x (esc)
   \x87\xcd\xc9n\x8e\xaa\xb6\x00\x00\x00+\x00\x00\x00V\x00\x00\x00\x00\x00\x00\x00\x81\x00\x00\x00\x81\x00\x00\x00+H\x008500189e74a9e0475e822093bc7db0d631aeb0b4 (esc)
   \x00\x00\x00\x00\x00\x00\x00\x05D\x00\x00\x00b\xc3\xf1\xca)$\xc1j\x19\xb0ej\x84\x90\x0ePN[ (esc)
   \xec-\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002\xafv\x86\xd4\x03\xcfE\xb5\xd9_-p\xce\xbe\xa5\x87\xac\x80j\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02D (esc)
   \x00\x00\x00\x00\x00\x00\x00\x05E\x00\x00\x00b\x9co\xd05 (esc)
   l\r (no-eol) (esc)
   \x0cI\xd4\xa9\xc5\x01|\xf0pC\xf5NX\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x95 \xee\xa7\x81\xbc\xca\x16\xc1\xe1Z\xcc\x0b\xa1C5\xa0\xe8\xe5\xba\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02E (esc)
   \x00\x00\x00\x00\x00\x00\x00\x05H\x00\x00\x00b\x85\x00\x18\x9et\xa9\xe0G^\x82 \x93\xbc}\xb0\xd61\xae\xb0\xb4\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\xdeB\x19n\xbe\xe4.\xf2\x84\xb6x (esc)
   \x87\xcd\xc9n\x8e\xaa\xb6\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02H (esc)
   \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 (no-eol) (esc)
 
   $ hg unbundle2 < ../rev.hg2
   adding changesets
   adding manifests
   adding file changes
   added 0 changesets with 0 changes to 3 files
   0 unread bytes
   addchangegroup return: 1
 
 Real world exchange
 =====================
 
 
 clone --pull
 
   $ cd ..
   $ hg clone main other --pull --rev 9520eea781bc
   adding changesets
   adding manifests
   adding file changes
   added 2 changesets with 2 changes to 2 files
   updating to branch default
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg -R other log -G
   @  changeset:   1:9520eea781bc
   |  tag:         tip
   |  user:        Nicolas Dumazet <[e-mail]>
   |  date:        Sat Apr 30 15:24:48 2011 +0200
   |  summary:     E
   |
   o  changeset:   0:cd010b8cd998
      user:        Nicolas Dumazet <[e-mail]>
      date:        Sat Apr 30 15:24:48 2011 +0200
      summary:     A
   
 
 pull
 
   $ hg -R other pull
   pulling from $TESTTMP/main (glob)
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 7 changesets with 6 changes to 6 files (+3 heads)
   (run 'hg heads' to see heads, 'hg merge' to merge)