##// END OF EJS Templates
upstream » mercurial-mirror
RSS

Clone from https://www.mercurial-scm.org/repo/hg-all

streamclone: define first iteration of version 2 of stream format...
streamclone: define first iteration of version 2 of stream format (This patch is based on a first draft from Gregory Szorc, with deeper rework) Version 1 of the stream clone format was invented many years ago and suffers from a few deficiencies: 1) Filenames are stored in store-encoded (on filesystem) form rather than in their internal form. This makes future compatibility with new store filename encodings more difficult. 2) File entry "headers" consist of a newline of the file name followed by the string file size. Converting strings to integers is avoidable overhead. We can't store filenames with newlines (manifests have this limitation as well, so it isn't a major concern). But the big concern here is the necessity for readline(). Scanning for newlines means reading ahead and that means extra buffer allocations and slicing (in Python) and this makes performance suffer. 3) Filenames aren't compressed optimally. Filenames should be compressed well since there is a lot of repeated data. However, since they are scattered all over the stream (with revlog data in between), they typically fall outside the window size of the compressor and don't compress. 4) It can only exchange stored based content, being able to exchange caches too would be nice. 5) It is limited to a stream-based protocol and isn't suitable for an on-disk format for general repository reading because the offset of individual file entries requires scanning the entire file to find file records. As part of enabling streaming clones to work in bundle2, #2 proved to have a significant negative impact on performance. Since bundle2 provides the opportunity to start fresh, Gregory Szorc figured he would take the opportunity to invent a new streaming clone data format. The new format devised in this series addresses #1, #2, and #4. It punts on #3 because it was complex without yielding a significant gain and on #5 because devising a new store format that "packs" multiple revlogs into a single "packed revlog" is massive scope bloat. However, this v2 format might be suitable for streaming into a "packed revlog" with minimal processing. If it works, great. If not, we can always invent stream format when it is needed. This patch only introduces the bases of the format. We'll get it usable through bundle2 first, then we'll extend the format in future patches to bring it to its full potential (especially #4).
Gregory Szorc - - Load All Authors

File last commit:

r29747:aba2bb2a default
r35774:cfdccd56 default
Show More
bundles.txt
94 lines | 3.2 KiB | text/plain | TextLexer
/ mercurial / help / internals / bundles.txt
History | Annotation | Raw |Copy content |Copy permalink
A bundle is a container for repository data.
Bundles are used as standalone files as well as the interchange format
over the wire protocol used when two Mercurial peers communicate with
each other.
Headers
=======
Bundles produced since Mercurial 0.7 (September 2005) have a 4 byte
header identifying the major bundle type. The header always begins with
``HG`` and the follow 2 bytes indicate the bundle type/version. Some
bundle types have additional data after this 4 byte header.
The following sections describe each bundle header/type.
HG10
----
``HG10`` headers indicate a *changegroup bundle*. This is the original
bundle format, so it is sometimes referred to as *bundle1*. It has been
present since version 0.7 (released September 2005).
This header is followed by 2 bytes indicating the compression algorithm
used for data that follows. All subsequent data following this
compression identifier is compressed according to the algorithm/method
specified.
Supported algorithms include the following.
``BZ``
*bzip2* compression.
Bzip2 compressors emit a leading ``BZ`` header. Mercurial uses this
leading ``BZ`` as part of the bundle header. Therefore consumers
of bzip2 bundles need to *seed* the bzip2 decompressor with ``BZ`` or
seek the input stream back to the beginning of the algorithm component
of the bundle header so that decompressor input is valid. This behavior
is unique among supported compression algorithms.
Supported since version 0.7 (released December 2006).
``GZ``
*zlib* compression.
Supported since version 0.9.2 (released December 2006).
``UN``
*Uncompressed* or no compression. Unmodified changegroup data follows.
Supported since version 0.9.2 (released December 2006).
3rd party extensions may implement their own compression. However, no
authority reserves values for their compression algorithm identifiers.
HG2X
----
``HG2X`` headers (where ``X`` is any value) denote a *bundle2* bundle.
Bundle2 bundles are a container format for various kinds of repository
data and capabilities, beyond changegroup data (which was the only data
supported by ``HG10`` bundles.
``HG20`` is currently the only defined bundle2 version.
The ``HG20`` format is not yet documented here. See the inline comments
in ``mercurial/exchange.py`` for now.
Initial ``HG20`` support was added in Mercurial 3.0 (released May
2014). However, bundle2 bundles were hidden behind an experimental flag
until version 3.5 (released August 2015), when they were enabled in the
wire protocol. Various commands (including ``hg bundle``) did not
support generating bundle2 files until Mercurial 3.6 (released November
2015).
HGS1
----
*Experimental*
A ``HGS1`` header indicates a *streaming clone bundle*. This is a bundle
that contains raw revlog data from a repository store. (Typically revlog
data is exchanged in the form of changegroups.)
The purpose of *streaming clone bundles* are to *clone* repository data
very efficiently.
The ``HGS1`` header is always followed by 2 bytes indicating a
compression algorithm of the data that follows. Only ``UN``
(uncompressed data) is currently allowed.
``HGS1UN`` support was added as an experimental feature in version 3.6
(released November 2015) as part of the initial offering of the *clone
bundles* feature.