##// END OF EJS Templates
exchange: improve computation of relevant markers for large repos...
exchange: improve computation of relevant markers for large repos Compute the candidate nodes with relevant markers directly from keys of the predecessors/successors/children dictionaries of obsstore. This is faster than iterating over all nodes directly. This test could be further improved for repositories with relative few markers compared to the repository size, but this is no longer hot already. With the current loop structure, the obshashrange use works as well as before as it passes lists with a single node. Adjust the interface by allowing revision lists as well as node lists. This helps cases that computes ancestors as it reduces the materialisation cost. Use this in _pushdiscoveryobsmarker and _getbundleobsmarkerpart. Improve the latter further by directly using ancestors(). Performance benchmarks show notable and welcome improvement to no-op push and pull (that would also apply to other push/pull). This apply to push and pull done without evolve. ### push/pull Benchmark parameter # bin-env-vars.hg.flavor = default # benchmark.variants.explicit-rev = none # benchmark.variants.protocol = ssh # benchmark.variants.revs = none ## benchmark.name = hg.command.pull # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 5.968537 seconds after: 5.668507 seconds (-5.03%, -0.30) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.446232 seconds after: 0.835553 seconds (-42.23%, -0.61) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.777412 seconds after: 2.523454 seconds (-56.32%, -3.25) ## benchmark.name = hg.command.push # data-env-vars.name = mercurial-devel-2024-03-22-zstd-sparse-revlog before: 6.155501 seconds after: 5.885072 seconds (-4.39%, -0.27) # data-env-vars.name = tryton-devel-2024-03-22-zstd-sparse-revlog before: 1.491054 seconds after: 0.934882 seconds (-37.30%, -0.56) # data-env-vars.name = netbsd-src-draft-2024-09-19-zstd-sparse-revlog before: 5.902494 seconds after: 2.957644 seconds (-49.89%, -2.94) There is not notable different in these result using the "rust" flavor instead of the "default". The performance impact on the same operation when using evolve were also tested and no impact was noted.

File last commit:

r52757:1c5810ce default
r52789:8583d138 default
Show More
stabletailsort.py
174 lines | 5.4 KiB | text/x-python | PythonLexer
# stabletailsort.py - stable ordering of revisions
#
# Copyright 2021-2023 Pacien TRAN-GIRARD <pacien.trangirard@pacien.net>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""
Stable-tail sort computation.
The "stable-tail sort", or STS, is a reverse topological ordering of the
ancestors of a node, which tends to share large suffixes with the stable-tail
sort of ancestors and other nodes, giving it its name.
Its properties should make it suitable for making chunks of ancestors with high
reuse and incrementality for example.
This module and implementation are experimental. Most functions are not yet
optimised to operate on large production graphs.
"""
from __future__ import annotations
import itertools
from ..node import nullrev
from .. import ancestor
def _sorted_parents(cl, p1, p2):
"""
Chooses and returns the pair (px, pt) from (p1, p2).
Where
"px" denotes the parent starting the "exclusive" part, and
"pt" denotes the parent starting the "Tail" part.
"px" is chosen as the parent with the lowest rank with the goal of
minimising the size of the exclusive part and maximise the size of the
tail part, hopefully reducing the overall complexity of the stable-tail
sort.
In case of equal ranks, the stable node ID is used as a tie-breaker.
"""
r1, r2 = cl.fast_rank(p1), cl.fast_rank(p2)
if r1 < r2:
return (p1, p2)
elif r1 > r2:
return (p2, p1)
elif cl.node(p1) < cl.node(p2):
return (p1, p2)
else:
return (p2, p1)
def _nonoedipal_parent_revs(cl, rev):
"""
Returns the non-Å“dipal parent pair of the given revision.
An Å“dipal merge is a merge with parents p1, p2 with either
p1 in ancestors(p2) or p2 in ancestors(p1).
In the first case, p1 is the Å“dipal parent.
In the second case, p2 is the Å“dipal parent.
Å’dipal edges start empty exclusive parts. They do not bring new ancestors.
As such, they can be skipped when computing any topological sort or any
iteration over the ancestors of a node.
The Å“dipal edges are eliminated here using the rank information.
"""
p1, p2 = cl.parentrevs(rev)
if p1 == nullrev or cl.fast_rank(p2) == cl.fast_rank(rev) - 1:
return p2, nullrev
elif p2 == nullrev or cl.fast_rank(p1) == cl.fast_rank(rev) - 1:
return p1, nullrev
else:
return p1, p2
def _parents(cl, rev):
p1, p2 = _nonoedipal_parent_revs(cl, rev)
if p2 == nullrev:
return p1, p2
return _sorted_parents(cl, p1, p2)
def _stable_tail_sort_naive(cl, head_rev):
"""
Naive topological iterator of the ancestors given by the stable-tail sort.
The stable-tail sort of a node "h" is defined as the sequence:
sts(h) := [h] + excl(h) + sts(pt(h))
where excl(h) := u for u in sts(px(h)) if u not in ancestors(pt(h))
This implementation uses a call-stack whose size is
O(number of open merges).
As such, this implementation exists mainly as a defining reference.
"""
cursor_rev = head_rev
while cursor_rev != nullrev:
yield cursor_rev
px, pt = _parents(cl, cursor_rev)
if pt == nullrev:
cursor_rev = px
else:
tail_ancestors = ancestor.lazyancestors(
cl.parentrevs, (pt,), inclusive=True
)
exclusive_ancestors = (
a
for a in _stable_tail_sort_naive(cl, px)
if a not in tail_ancestors
)
# Notice that excl(cur) is disjoint from ancestors(pt),
# so there is no double-counting:
# rank(cur) = len([cur]) + len(excl(cur)) + rank(pt)
excl_part_size = cl.fast_rank(cursor_rev) - cl.fast_rank(pt) - 1
yield from itertools.islice(exclusive_ancestors, excl_part_size)
cursor_rev = pt
def _find_all_leaps_naive(cl, head_rev):
"""
Yields the leaps in the stable-tail sort of the given revision.
A leap is a pair of revisions (source, target) consecutive in the
stable-tail sort of a head, for which target != px(source).
Leaps are yielded in the same order as encountered in the stable-tail sort,
from head to root.
"""
sts = _stable_tail_sort_naive(cl, head_rev)
prev = next(sts)
for current in sts:
if current != _parents(cl, prev)[0]:
yield (prev, current)
prev = current
def _find_specific_leaps_naive(cl, head_rev):
"""
Returns the specific leaps in the stable-tail sort of the given revision.
Specific leaps are leaps appear in the stable-tail sort of a given
revision, but not in the stable-tail sort of any of its ancestors.
The final leaps (leading to the pt of the considered merge) are omitted.
Only merge nodes can have associated specific leaps.
This implementations uses the whole leap sets of the given revision and
of its parents.
"""
px, pt = _parents(cl, head_rev)
if px == nullrev or pt == nullrev:
return # linear nodes cannot have specific leaps
parents_leaps = set(_find_all_leaps_naive(cl, px))
sts = _stable_tail_sort_naive(cl, head_rev)
prev = next(sts)
for current in sts:
if current == pt:
break
if current != _parents(cl, prev)[0]:
leap = (prev, current)
if leap not in parents_leaps:
yield leap
prev = current