##// END OF EJS Templates
upstream » mercurial-mirror
RSS

Clone from https://www.mercurial-scm.org/repo/hg-all

revlog: use compression engine APIs for decompression...
revlog: use compression engine APIs for decompression Now that compression engines declare their header in revlog chunks and can decompress revlog chunks, we refactor revlog.decompress() to use them. Making full use of the property that revlog compressor objects are reusable, revlog instances now maintain a dict mapping an engine's revlog header to a compressor object. This is not only a performance optimization for engines where compressor object reuse can result in better performance, but it also serves as a cache of header values so we don't need to perform redundant lookups against the compression engine manager. (Yes, I measured and the overhead of a function call versus a dict lookup was observed.) Replacing the previous inline lookup table with a dict lookup was measured to make chunk reading ~2.5% slower on changelogs and ~4.5% slower on manifests. So, the inline lookup table has been mostly preserved so we don't lose performance. This is unfortunate. But many decompression operations complete in microseconds, so Python attribute lookup, dict lookup, and function calls do matter. The impact of this change on mozilla-unified is as follows: $ hg perfrevlogchunks -c ! chunk ! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6) ! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6) ! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4) ! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4) ! chunk batch ! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5) ! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4) Changelog chunk reading is slightly faster but manifest reading is slower. What gives? On this repo, 99.85% of changelog entries are zlib compressed (the 'x' header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch swapped the test order of 'x' and '\0' so now 'x' is tested first. This makes changelogs faster since they almost always hit the first branch. This makes a significant percentage of manifest '\0' chunks slower because that code path now performs an extra test. Yes, I too can't believe we're able to measure the impact of an if..elif with simple string compares. I reckon this code would benefit from being written in C...
Gregory Szorc - - Load All Authors

File last commit:

r29747:aba2bb2a default
r30817:2b279126 default
Show More
changegroups.txt
153 lines | 6.1 KiB | text/plain | TextLexer
/ mercurial / help / internals / changegroups.txt
History | Annotation | Raw |Copy content |Copy permalink
Changegroups are representations of repository revlog data, specifically
the changelog, manifest, and filelogs.
There are 3 versions of changegroups: ``1``, ``2``, and ``3``. From a
high-level, versions ``1`` and ``2`` are almost exactly the same, with
the only difference being a header on entries in the changeset
segment. Version ``3`` adds support for exchanging treemanifests and
includes revlog flags in the delta header.
Changegroups consists of 3 logical segments::
+---------------------------------+
| | | |
| changeset | manifest | filelogs |
| | | |
+---------------------------------+
The principle building block of each segment is a *chunk*. A *chunk*
is a framed piece of data::
+---------------------------------------+
| | |
| length | data |
| (32 bits) | <length> bytes |
| | |
+---------------------------------------+
Each chunk starts with a 32-bit big-endian signed integer indicating
the length of the raw data that follows.
There is a special case chunk that has 0 length (``0x00000000``). We
call this an *empty chunk*.
Delta Groups
============
A *delta group* expresses the content of a revlog as a series of deltas,
or patches against previous revisions.
Delta groups consist of 0 or more *chunks* followed by the *empty chunk*
to signal the end of the delta group::
+------------------------------------------------------------------------+
| | | | | |
| chunk0 length | chunk0 data | chunk1 length | chunk1 data | 0x0 |
| (32 bits) | (various) | (32 bits) | (various) | (32 bits) |
| | | | | |
+------------------------------------------------------------+-----------+
Each *chunk*'s data consists of the following::
+-----------------------------------------+
| | | |
| delta header | mdiff header | delta |
| (various) | (12 bytes) | (various) |
| | | |
+-----------------------------------------+
The *length* field is the byte length of the remaining 3 logical pieces
of data. The *delta* is a diff from an existing entry in the changelog.
The *delta header* is different between versions ``1``, ``2``, and
``3`` of the changegroup format.
Version 1::
+------------------------------------------------------+
| | | | |
| node | p1 node | p2 node | link node |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) |
| | | | |
+------------------------------------------------------+
Version 2::
+------------------------------------------------------------------+
| | | | | |
| node | p1 node | p2 node | base node | link node |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) |
| | | | | |
+------------------------------------------------------------------+
Version 3::
+------------------------------------------------------------------------------+
| | | | | | |
| node | p1 node | p2 node | base node | link node | flags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) |
| | | | | | |
+------------------------------------------------------------------------------+
The *mdiff header* consists of 3 32-bit big-endian signed integers
describing offsets at which to apply the following delta content::
+-------------------------------------+
| | | |
| offset | old length | new length |
| (32 bits) | (32 bits) | (32 bits) |
| | | |
+-------------------------------------+
In version 1, the delta is always applied against the previous node from
the changegroup or the first parent if this is the first entry in the
changegroup.
In version 2, the delta base node is encoded in the entry in the
changegroup. This allows the delta to be expressed against any parent,
which can result in smaller deltas and more efficient encoding of data.
Changeset Segment
=================
The *changeset segment* consists of a single *delta group* holding
changelog data. It is followed by an *empty chunk* to denote the
boundary to the *manifests segment*.
Manifest Segment
================
The *manifest segment* consists of a single *delta group* holding
manifest data. It is followed by an *empty chunk* to denote the boundary
to the *filelogs segment*.
Filelogs Segment
================
The *filelogs* segment consists of multiple sub-segments, each
corresponding to an individual file whose data is being described::
+--------------------------------------+
| | | | |
| filelog0 | filelog1 | filelog2 | ... |
| | | | |
+--------------------------------------+
In version ``3`` of the changegroup format, filelogs may include
directory logs when treemanifests are in use. directory logs are
identified by having a trailing '/' on their filename (see below).
The final filelog sub-segment is followed by an *empty chunk* to denote
the end of the segment and the overall changegroup.
Each filelog sub-segment consists of the following::
+------------------------------------------+
| | | |
| filename size | filename | delta group |
| (32 bits) | (various) | (various) |
| | | |
+------------------------------------------+
That is, a *chunk* consisting of the filename (not terminated or padded)
followed by N chunks constituting the *delta group* for this file.