##// END OF EJS Templates
mmap: populate the mapping by default...
mmap: populate the mapping by default Without pre-population, accessing all data through a mmap can result in many pagefault, reducing performance significantly. If the mmap is prepopulated, the performance can no longer get slower than a full read. (See benchmark number below) In some cases were very few data is read, prepopulating can be overkill and slower than populating on access (through page fault). So that behavior can be controlled when the caller can pre-determine the best behavior. (See benchmark number below) In addition, testing with populating in a secondary thread yield great result combining the best of each approach. This might be implemented in later changesets. In all cases, using mmap has a great effect on memory usage when many processes run in parallel on the same machine. ### Benchmarks # What did I run A couple of month back I ran a large benchmark campaign to assess the impact of various approach for using mmap with the revlog (and other files), it highlighted a few benchmarks that capture the impact of the changes well. So to validate this change I checked the following: - log command displaying various revisions (read the changelog index) - log command displaying the patch of listed revisions (read the changelog index, the manifest index and a few files indexes) - unbundling a few revisions (read and write changelog, manifest and few files indexes, and walk the graph to update some cache) - pushing a few revisions (read and write changelog, manifest and few files indexes, walk the graph to update some cache, performs various accesses locally and remotely during discovery) Benchmarks were run using the default module policy (c+py) and the rust one. No significant difference were found between the two implementation, so we will present result using the default policy (unless otherwise specified). I ran them on a few repositories : - mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using zstd compression and sparse-revlog - pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog - netbeans: a copy of netbeans from 2018-08-01 using zstd compression and sparse-revlog - mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and sparse-revlog - mozilla-try persistent-nodemap: Same as the above but with a persistent nodemap. Used for the log --patch benchmark only # Results For the smaller repositories (mercurial, pypy), the impact of mmap is almost imperceptible, other cost dominating the operation. The impact of prepopulating is undiscernible in the benchmark we ran. For larger repositories the benchmark support explanation given above: On netbeans, the log can be about 1% faster without repopulation (for a difference < 100ms) but unbundle becomes a bit slower, even when small. ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet with-populate: 0.240157 no-populate: 0.265087 (+10.38%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) ## benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 0.771919 no-populate: 0.792025 (+2.60%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) For mozilla-try, the "slow down" from pre-populate for small `hg log` is more visible, but still small in absolute time. (using rust value for the persistent nodemap value to be relevant). ### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # benchmark.variants.patch = yes # benchmark.variants.limit-rev = 1 with-populate: 0.237813 no-populate: 0.229452 (-3.52%, -0.01) # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes with-populate: 1.213578 no-populate: 1.205189 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = tip with-populate: 0.198607 no-populate: 0.195038 (-1.80%, -0.00) However pre-populating provide a significant boost on more complex operations like unbundle or push: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 4.798632 no-populate: 4.953295 (+3.22%, +0.15) # benchmark.variants.revs = any-100-extra-rev with-populate: 4.903618 no-populate: 5.014963 (+2.27%, +0.11) ## benchmark.name = hg.command.unbundle # benchmark.variants.revs = any-1-extra-rev with-populate: 1.423411 no-populate: 1.585365 (+11.38%, +0.16) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.537909 no-populate: 1.688489 (+9.79%, +0.15)

File last commit:

r51421:8fb3e942 default
r52574:522b4d72 default
Show More
stabletailsort.py
172 lines | 5.4 KiB | text/x-python | PythonLexer
# stabletailsort.py - stable ordering of revisions
#
# Copyright 2021-2023 Pacien TRAN-GIRARD <pacien.trangirard@pacien.net>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""
Stable-tail sort computation.
The "stable-tail sort", or STS, is a reverse topological ordering of the
ancestors of a node, which tends to share large suffixes with the stable-tail
sort of ancestors and other nodes, giving it its name.
Its properties should make it suitable for making chunks of ancestors with high
reuse and incrementality for example.
This module and implementation are experimental. Most functions are not yet
optimised to operate on large production graphs.
"""
import itertools
from ..node import nullrev
from .. import ancestor
def _sorted_parents(cl, p1, p2):
"""
Chooses and returns the pair (px, pt) from (p1, p2).
Where
"px" denotes the parent starting the "exclusive" part, and
"pt" denotes the parent starting the "Tail" part.
"px" is chosen as the parent with the lowest rank with the goal of
minimising the size of the exclusive part and maximise the size of the
tail part, hopefully reducing the overall complexity of the stable-tail
sort.
In case of equal ranks, the stable node ID is used as a tie-breaker.
"""
r1, r2 = cl.fast_rank(p1), cl.fast_rank(p2)
if r1 < r2:
return (p1, p2)
elif r1 > r2:
return (p2, p1)
elif cl.node(p1) < cl.node(p2):
return (p1, p2)
else:
return (p2, p1)
def _nonoedipal_parent_revs(cl, rev):
"""
Returns the non-Å“dipal parent pair of the given revision.
An Å“dipal merge is a merge with parents p1, p2 with either
p1 in ancestors(p2) or p2 in ancestors(p1).
In the first case, p1 is the Å“dipal parent.
In the second case, p2 is the Å“dipal parent.
Å’dipal edges start empty exclusive parts. They do not bring new ancestors.
As such, they can be skipped when computing any topological sort or any
iteration over the ancestors of a node.
The Å“dipal edges are eliminated here using the rank information.
"""
p1, p2 = cl.parentrevs(rev)
if p1 == nullrev or cl.fast_rank(p2) == cl.fast_rank(rev) - 1:
return p2, nullrev
elif p2 == nullrev or cl.fast_rank(p1) == cl.fast_rank(rev) - 1:
return p1, nullrev
else:
return p1, p2
def _parents(cl, rev):
p1, p2 = _nonoedipal_parent_revs(cl, rev)
if p2 == nullrev:
return p1, p2
return _sorted_parents(cl, p1, p2)
def _stable_tail_sort_naive(cl, head_rev):
"""
Naive topological iterator of the ancestors given by the stable-tail sort.
The stable-tail sort of a node "h" is defined as the sequence:
sts(h) := [h] + excl(h) + sts(pt(h))
where excl(h) := u for u in sts(px(h)) if u not in ancestors(pt(h))
This implementation uses a call-stack whose size is
O(number of open merges).
As such, this implementation exists mainly as a defining reference.
"""
cursor_rev = head_rev
while cursor_rev != nullrev:
yield cursor_rev
px, pt = _parents(cl, cursor_rev)
if pt == nullrev:
cursor_rev = px
else:
tail_ancestors = ancestor.lazyancestors(
cl.parentrevs, (pt,), inclusive=True
)
exclusive_ancestors = (
a
for a in _stable_tail_sort_naive(cl, px)
if a not in tail_ancestors
)
# Notice that excl(cur) is disjoint from ancestors(pt),
# so there is no double-counting:
# rank(cur) = len([cur]) + len(excl(cur)) + rank(pt)
excl_part_size = cl.fast_rank(cursor_rev) - cl.fast_rank(pt) - 1
yield from itertools.islice(exclusive_ancestors, excl_part_size)
cursor_rev = pt
def _find_all_leaps_naive(cl, head_rev):
"""
Yields the leaps in the stable-tail sort of the given revision.
A leap is a pair of revisions (source, target) consecutive in the
stable-tail sort of a head, for which target != px(source).
Leaps are yielded in the same order as encountered in the stable-tail sort,
from head to root.
"""
sts = _stable_tail_sort_naive(cl, head_rev)
prev = next(sts)
for current in sts:
if current != _parents(cl, prev)[0]:
yield (prev, current)
prev = current
def _find_specific_leaps_naive(cl, head_rev):
"""
Returns the specific leaps in the stable-tail sort of the given revision.
Specific leaps are leaps appear in the stable-tail sort of a given
revision, but not in the stable-tail sort of any of its ancestors.
The final leaps (leading to the pt of the considered merge) are omitted.
Only merge nodes can have associated specific leaps.
This implementations uses the whole leap sets of the given revision and
of its parents.
"""
px, pt = _parents(cl, head_rev)
if px == nullrev or pt == nullrev:
return # linear nodes cannot have specific leaps
parents_leaps = set(_find_all_leaps_naive(cl, px))
sts = _stable_tail_sort_naive(cl, head_rev)
prev = next(sts)
for current in sts:
if current == pt:
break
if current != _parents(cl, prev)[0]:
leap = (prev, current)
if leap not in parents_leaps:
yield leap
prev = current