##// END OF EJS Templates
upstream » mercurial-mirror
RSS

Clone from https://www.mercurial-scm.org/repo/hg-all

mmap: populate the mapping by default...
mmap: populate the mapping by default Without pre-population, accessing all data through a mmap can result in many pagefault, reducing performance significantly. If the mmap is prepopulated, the performance can no longer get slower than a full read. (See benchmark number below) In some cases were very few data is read, prepopulating can be overkill and slower than populating on access (through page fault). So that behavior can be controlled when the caller can pre-determine the best behavior. (See benchmark number below) In addition, testing with populating in a secondary thread yield great result combining the best of each approach. This might be implemented in later changesets. In all cases, using mmap has a great effect on memory usage when many processes run in parallel on the same machine. ### Benchmarks # What did I run A couple of month back I ran a large benchmark campaign to assess the impact of various approach for using mmap with the revlog (and other files), it highlighted a few benchmarks that capture the impact of the changes well. So to validate this change I checked the following: - log command displaying various revisions (read the changelog index) - log command displaying the patch of listed revisions (read the changelog index, the manifest index and a few files indexes) - unbundling a few revisions (read and write changelog, manifest and few files indexes, and walk the graph to update some cache) - pushing a few revisions (read and write changelog, manifest and few files indexes, walk the graph to update some cache, performs various accesses locally and remotely during discovery) Benchmarks were run using the default module policy (c+py) and the rust one. No significant difference were found between the two implementation, so we will present result using the default policy (unless otherwise specified). I ran them on a few repositories : - mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using zstd compression and sparse-revlog - pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog - netbeans: a copy of netbeans from 2018-08-01 using zstd compression and sparse-revlog - mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and sparse-revlog - mozilla-try persistent-nodemap: Same as the above but with a persistent nodemap. Used for the log --patch benchmark only # Results For the smaller repositories (mercurial, pypy), the impact of mmap is almost imperceptible, other cost dominating the operation. The impact of prepopulating is undiscernible in the benchmark we ran. For larger repositories the benchmark support explanation given above: On netbeans, the log can be about 1% faster without repopulation (for a difference < 100ms) but unbundle becomes a bit slower, even when small. ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet with-populate: 0.240157 no-populate: 0.265087 (+10.38%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) ## benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 0.771919 no-populate: 0.792025 (+2.60%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) For mozilla-try, the "slow down" from pre-populate for small `hg log` is more visible, but still small in absolute time. (using rust value for the persistent nodemap value to be relevant). ### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # benchmark.variants.patch = yes # benchmark.variants.limit-rev = 1 with-populate: 0.237813 no-populate: 0.229452 (-3.52%, -0.01) # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes with-populate: 1.213578 no-populate: 1.205189 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = tip with-populate: 0.198607 no-populate: 0.195038 (-1.80%, -0.00) However pre-populating provide a significant boost on more complex operations like unbundle or push: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 4.798632 no-populate: 4.953295 (+3.22%, +0.15) # benchmark.variants.revs = any-100-extra-rev with-populate: 4.903618 no-populate: 5.014963 (+2.27%, +0.11) ## benchmark.name = hg.command.unbundle # benchmark.variants.revs = any-1-extra-rev with-populate: 1.423411 no-populate: 1.585365 (+11.38%, +0.16) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.537909 no-populate: 1.688489 (+9.79%, +0.15)
Mathias De Mare - - Load All Authors

File last commit:

r51538:bf7404f2 stable
r52574:522b4d72 default
Show More
bundlespec.txt
121 lines | 3.7 KiB | text/plain | TextLexer
/ mercurial / helptext / bundlespec.txt
History | Annotation | Raw |Copy content |Copy permalink
Mercurial supports generating standalone "bundle" files that hold repository
data. These "bundles" are typically saved locally and used later or exchanged
between different repositories, possibly on different machines. Example
commands using bundles are :hg:`bundle` and :hg:`unbundle`.
Generation of bundle files is controlled by a "bundle specification"
("bundlespec") string. This string tells the bundle generation process how
to create the bundle.
A "bundlespec" string is composed of the following elements:
type
A string denoting the bundle format to use.
compression
Denotes the compression engine to use compressing the raw bundle data.
parameters
Arbitrary key-value parameters to further control bundle generation.
A "bundlespec" string has the following formats:
<type>
The literal bundle format string is used.
<compression>-<type>
The compression engine and format are delimited by a hyphen (``-``).
Optional parameters follow the ``<type>``. Parameters are URI escaped
``key=value`` pairs. Each pair is delimited by a semicolon (``;``). The
first parameter begins after a ``;`` immediately following the ``<type>``
value.
Available Types
===============
The following bundle <type> strings are available:
v1
Produces a legacy "changegroup" version 1 bundle.
This format is compatible with nearly all Mercurial clients because it is
the oldest. However, it has some limitations, which is why it is no longer
the default for new repositories.
``v1`` bundles can be used with modern repositories using the "generaldelta"
storage format. However, it may take longer to produce the bundle and the
resulting bundle may be significantly larger than a ``v2`` bundle.
``v1`` bundles can only use the ``gzip``, ``bzip2``, and ``none`` compression
formats.
v2
Produces a version 2 bundle.
Version 2 bundles are an extensible format that can store additional
repository data (such as bookmarks and phases information) and they can
store data more efficiently, resulting in smaller bundles.
Version 2 bundles can also use modern compression engines, such as
``zstd``, making them faster to compress and often smaller.
Available Compression Engines
=============================
The following bundle <compression> engines can be used:
.. bundlecompressionmarker
The compression engines can be prepended with ``stream`` to create a streaming bundle.
These are bundles that are extremely efficient to produce and consume,
but do not have guaranteed compatibility with older clients.
Available Options
=================
The following options exist:
changegroup
Include the changegroup data in the bundle (default to True).
cg.version
Select the version of the changegroup to use. Available options are : 01, 02
or 03. By default it will be automatically selected according to the current
repository format.
obsolescence
Include obsolescence-markers relevant to the bundled changesets.
phases
Include phase information relevant to the bundled changesets.
revbranchcache
Include the "tags-fnodes" cache inside the bundle.
tagsfnodescache
Include the "tags-fnodes" cache inside the bundle.
Examples
========
``v2``
Produce a ``v2`` bundle using default options, including compression.
``none-v1``
Produce a ``v1`` bundle with no compression.
``zstd-v2``
Produce a ``v2`` bundle with zstandard compression using default
settings.
``zstd-v1``
This errors because ``zstd`` is not supported for ``v1`` types.
``none-streamv2``
Produce a ``v2`` streaming bundle with no compression.
``zstd-v2;obsolescence=true;phases=true``
Produce a ``v2`` bundle with zstandard compression which includes
obsolescence markers and phases.