##// END OF EJS Templates
mmap: populate the mapping by default...
mmap: populate the mapping by default Without pre-population, accessing all data through a mmap can result in many pagefault, reducing performance significantly. If the mmap is prepopulated, the performance can no longer get slower than a full read. (See benchmark number below) In some cases were very few data is read, prepopulating can be overkill and slower than populating on access (through page fault). So that behavior can be controlled when the caller can pre-determine the best behavior. (See benchmark number below) In addition, testing with populating in a secondary thread yield great result combining the best of each approach. This might be implemented in later changesets. In all cases, using mmap has a great effect on memory usage when many processes run in parallel on the same machine. ### Benchmarks # What did I run A couple of month back I ran a large benchmark campaign to assess the impact of various approach for using mmap with the revlog (and other files), it highlighted a few benchmarks that capture the impact of the changes well. So to validate this change I checked the following: - log command displaying various revisions (read the changelog index) - log command displaying the patch of listed revisions (read the changelog index, the manifest index and a few files indexes) - unbundling a few revisions (read and write changelog, manifest and few files indexes, and walk the graph to update some cache) - pushing a few revisions (read and write changelog, manifest and few files indexes, walk the graph to update some cache, performs various accesses locally and remotely during discovery) Benchmarks were run using the default module policy (c+py) and the rust one. No significant difference were found between the two implementation, so we will present result using the default policy (unless otherwise specified). I ran them on a few repositories : - mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using zstd compression and sparse-revlog - pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog - netbeans: a copy of netbeans from 2018-08-01 using zstd compression and sparse-revlog - mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and sparse-revlog - mozilla-try persistent-nodemap: Same as the above but with a persistent nodemap. Used for the log --patch benchmark only # Results For the smaller repositories (mercurial, pypy), the impact of mmap is almost imperceptible, other cost dominating the operation. The impact of prepopulating is undiscernible in the benchmark we ran. For larger repositories the benchmark support explanation given above: On netbeans, the log can be about 1% faster without repopulation (for a difference < 100ms) but unbundle becomes a bit slower, even when small. ### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.command.unbundle # benchmark.variants.issue6528 = disabled # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev # benchmark.variants.source = unbundle # benchmark.variants.verbosity = quiet with-populate: 0.240157 no-populate: 0.265087 (+10.38%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) ## benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 0.771919 no-populate: 0.792025 (+2.60%, +0.02) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.459518 no-populate: 1.481290 (+1.49%, +0.02) For mozilla-try, the "slow down" from pre-populate for small `hg log` is more visible, but still small in absolute time. (using rust value for the persistent nodemap value to be relevant). ### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm # benchmark.name = hg.command.log # bin-env-vars.hg.flavor = rust # benchmark.variants.patch = yes # benchmark.variants.limit-rev = 1 with-populate: 0.237813 no-populate: 0.229452 (-3.52%, -0.01) # benchmark.variants.limit-rev = 10 # benchmark.variants.patch = yes with-populate: 1.213578 no-populate: 1.205189 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.variants.limit-rev = 1000 # benchmark.variants.patch = no # benchmark.variants.rev = tip with-populate: 0.198607 no-populate: 0.195038 (-1.80%, -0.00) However pre-populating provide a significant boost on more complex operations like unbundle or push: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = hg.command.push # benchmark.variants.explicit-rev = none # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = yes # benchmark.variants.revs = any-1-extra-rev with-populate: 4.798632 no-populate: 4.953295 (+3.22%, +0.15) # benchmark.variants.revs = any-100-extra-rev with-populate: 4.903618 no-populate: 5.014963 (+2.27%, +0.11) ## benchmark.name = hg.command.unbundle # benchmark.variants.revs = any-1-extra-rev with-populate: 1.423411 no-populate: 1.585365 (+11.38%, +0.16) # benchmark.variants.revs = any-100-extra-rev with-populate: 1.537909 no-populate: 1.688489 (+9.79%, +0.15)

File last commit:

r50237:67b210bb default
r52574:522b4d72 default
Show More
auto_upgrade.py
254 lines | 8.5 KiB | text/x-python | PythonLexer
# upgrade.py - functions for automatic upgrade of Mercurial repository
#
# Copyright (c) 2022-present, Pierre-Yves David
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from ..i18n import _
from .. import (
error,
requirements as requirementsmod,
scmutil,
)
from . import (
actions,
engine,
)
class AutoUpgradeOperation(actions.BaseOperation):
"""A limited Upgrade Operation used to run simple auto upgrade task
(Expand it as needed in the future)
"""
def __init__(self, req):
super().__init__(
new_requirements=req,
backup_store=False,
)
def get_share_safe_action(repo):
"""return an automatic-upgrade action for `share-safe` if applicable
If no action is needed, return None, otherwise return a callback to upgrade
or downgrade the repository according the configuration and repository
format.
"""
ui = repo.ui
requirements = repo.requirements
auto_upgrade_share_source = ui.configbool(
b'format',
b'use-share-safe.automatic-upgrade-of-mismatching-repositories',
)
auto_upgrade_quiet = ui.configbool(
b'format',
b'use-share-safe.automatic-upgrade-of-mismatching-repositories:quiet',
)
action = None
if (
auto_upgrade_share_source
and requirementsmod.SHARED_REQUIREMENT not in requirements
):
sf_config = ui.configbool(b'format', b'use-share-safe')
sf_local = requirementsmod.SHARESAFE_REQUIREMENT in requirements
if sf_config and not sf_local:
msg = _(
b"automatically upgrading repository to the `share-safe`"
b" feature\n"
)
hint = b"(see `hg help config.format.use-share-safe` for details)\n"
def action():
if not (ui.quiet or auto_upgrade_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.add(requirementsmod.SHARESAFE_REQUIREMENT)
scmutil.writereporequirements(repo, requirements)
elif sf_local and not sf_config:
msg = _(
b"automatically downgrading repository from the `share-safe`"
b" feature\n"
)
hint = b"(see `hg help config.format.use-share-safe` for details)\n"
def action():
if not (ui.quiet or auto_upgrade_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.discard(requirementsmod.SHARESAFE_REQUIREMENT)
scmutil.writereporequirements(repo, requirements)
return action
def get_tracked_hint_action(repo):
"""return an automatic-upgrade action for `tracked-hint` if applicable
If no action is needed, return None, otherwise return a callback to upgrade
or downgrade the repository according the configuration and repository
format.
"""
ui = repo.ui
requirements = set(repo.requirements)
auto_upgrade_tracked_hint = ui.configbool(
b'format',
b'use-dirstate-tracked-hint.automatic-upgrade-of-mismatching-repositories',
)
auto_upgrade_quiet = ui.configbool(
b'format',
b'use-dirstate-tracked-hint.automatic-upgrade-of-mismatching-repositories:quiet',
)
action = None
if auto_upgrade_tracked_hint:
th_config = ui.configbool(b'format', b'use-dirstate-tracked-hint')
th_local = requirementsmod.DIRSTATE_TRACKED_HINT_V1 in requirements
if th_config and not th_local:
msg = _(
b"automatically upgrading repository to the `tracked-hint`"
b" feature\n"
)
hint = b"(see `hg help config.format.use-dirstate-tracked-hint` for details)\n"
def action():
if not (ui.quiet or auto_upgrade_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.add(requirementsmod.DIRSTATE_TRACKED_HINT_V1)
op = AutoUpgradeOperation(requirements)
engine.upgrade_tracked_hint(ui, repo, op, add=True)
elif th_local and not th_config:
msg = _(
b"automatically downgrading repository from the `tracked-hint`"
b" feature\n"
)
hint = b"(see `hg help config.format.use-dirstate-tracked-hint` for details)\n"
def action():
if not (ui.quiet or auto_upgrade_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.discard(requirementsmod.DIRSTATE_TRACKED_HINT_V1)
op = AutoUpgradeOperation(requirements)
engine.upgrade_tracked_hint(ui, repo, op, add=False)
return action
def get_dirstate_v2_action(repo):
"""return an automatic-upgrade action for `dirstate-v2` if applicable
If no action is needed, return None, otherwise return a callback to upgrade
or downgrade the repository according the configuration and repository
format.
"""
ui = repo.ui
requirements = set(repo.requirements)
auto_upgrade_dv2 = ui.configbool(
b'format',
b'use-dirstate-v2.automatic-upgrade-of-mismatching-repositories',
)
auto_upgrade_dv2_quiet = ui.configbool(
b'format',
b'use-dirstate-v2.automatic-upgrade-of-mismatching-repositories:quiet',
)
action = None
if auto_upgrade_dv2:
d2_config = ui.configbool(b'format', b'use-dirstate-v2')
d2_local = requirementsmod.DIRSTATE_V2_REQUIREMENT in requirements
if d2_config and not d2_local:
msg = _(
b"automatically upgrading repository to the `dirstate-v2`"
b" feature\n"
)
hint = (
b"(see `hg help config.format.use-dirstate-v2` for details)\n"
)
def action():
if not (ui.quiet or auto_upgrade_dv2_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.add(requirementsmod.DIRSTATE_V2_REQUIREMENT)
fake_op = AutoUpgradeOperation(requirements)
engine.upgrade_dirstate(repo.ui, repo, fake_op, b'v1', b'v2')
elif d2_local and not d2_config:
msg = _(
b"automatically downgrading repository from the `dirstate-v2`"
b" feature\n"
)
hint = (
b"(see `hg help config.format.use-dirstate-v2` for details)\n"
)
def action():
if not (ui.quiet or auto_upgrade_dv2_quiet):
ui.write_err(msg)
ui.write_err(hint)
requirements.discard(requirementsmod.DIRSTATE_V2_REQUIREMENT)
fake_op = AutoUpgradeOperation(requirements)
engine.upgrade_dirstate(repo.ui, repo, fake_op, b'v2', b'v1')
return action
AUTO_UPGRADE_ACTIONS = [
get_dirstate_v2_action,
get_share_safe_action,
get_tracked_hint_action,
]
def may_auto_upgrade(repo, maker_func):
"""potentially perform auto-upgrade and return the final repository to use
Auto-upgrade are "quick" repository upgrade that might automatically be run
by "any" repository access. See `hg help config.format` for automatic
upgrade documentation.
note: each relevant upgrades are done one after the other for simplicity.
This avoid having repository is partially inconsistent state while
upgrading.
repo: the current repository instance
maker_func: a factory function that can recreate a repository after an upgrade
"""
clear = False
loop = 0
try:
while not clear:
loop += 1
if loop > 100:
# XXX basic protection against infinite loop, make it better.
raise error.ProgrammingError("Too many auto upgrade loops")
clear = True
for get_action in AUTO_UPGRADE_ACTIONS:
action = get_action(repo)
if action is not None:
clear = False
with repo.wlock(wait=False), repo.lock(wait=False):
action = get_action(repo)
if action is not None:
action()
repo = maker_func()
except error.LockError:
# if we cannot get the lock, ignore the auto-upgrade attemps and
# proceed. We might want to make this behavior configurable in the
# future.
pass
return repo