##// END OF EJS Templates
procutil: avoid using os.fork() to implement runbgcommand...
procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

File last commit:

r47575:d4ba4d51 default
r47651:8759e22f default
Show More
requirements.py
82 lines | 3.3 KiB | text/x-python | PythonLexer
Pulkit Goyal
requirements: introduce new requirements related module...
r45932 # requirements.py - objects and functions related to repository requirements
#
Raphaël Gomès
contributor: change mentions of mpm to olivia...
r47575 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
Pulkit Goyal
requirements: introduce new requirements related module...
r45932 #
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
Raphaël Gomès
requirements: also add a generaldelta constant...
r47372 GENERALDELTA_REQUIREMENT = b'generaldelta'
Raphaël Gomès
requirements: also add a dotencode constant...
r47381 DOTENCODE_REQUIREMENT = b'dotencode'
Raphaël Gomès
requirements: also add a store constant...
r47382 STORE_REQUIREMENT = b'store'
Raphaël Gomès
requirements: also add a fncache constant...
r47383 FNCACHE_REQUIREMENT = b'fncache'
Raphaël Gomès
requirements: also add a generaldelta constant...
r47372
Pulkit Goyal
requirements: introduce new requirements related module...
r45932 # When narrowing is finalized and no longer subject to format changes,
# we should move this to just "narrow" or similar.
NARROW_REQUIREMENT = b'narrowhg-experimental'
# Enables sparse working directory usage
SPARSE_REQUIREMENT = b'exp-sparse'
# Enables the internal phase which is used to hide changesets instead
# of stripping them
INTERNAL_PHASE_REQUIREMENT = b'internal-phase'
# Stores manifest in Tree structure
TREEMANIFEST_REQUIREMENT = b'treemanifest'
Pulkit Goyal
localrepo: move requirements constant to requirements module...
r45933
Raphaël Gomès
requirements: add constant for revlog v1 requirement...
r47371 REVLOGV1_REQUIREMENT = b'revlogv1'
Pulkit Goyal
localrepo: move requirements constant to requirements module...
r45933 # Increment the sub-version when the revlog v2 format changes to lock out old
# clients.
Raphaël Gomès
revlog: introduce v2 format...
r47438 REVLOGV2_REQUIREMENT = b'exp-revlogv2.2'
Pulkit Goyal
localrepo: move requirements constant to requirements module...
r45933
# A repository with the sparserevlog feature will have delta chains that
# can spread over a larger span. Sparse reading cuts these large spans into
# pieces, so that each piece isn't too big.
# Without the sparserevlog capability, reading from the repository could use
# huge amounts of memory, because the whole span would be read at once,
# including all the intermediate revisions that aren't pertinent for the chain.
# This is why once a repository has enabled sparse-read, it becomes required.
SPARSEREVLOG_REQUIREMENT = b'sparserevlog'
# A repository with the sidedataflag requirement will allow to store extra
# information for revision without altering their original hashes.
SIDEDATA_REQUIREMENT = b'exp-sidedata-flag'
# A repository with the the copies-sidedata-changeset requirement will store
# copies related information in changeset's sidedata.
COPIESSDC_REQUIREMENT = b'exp-copies-sidedata-changeset'
# The repository use persistent nodemap for the changelog and the manifest.
NODEMAP_REQUIREMENT = b'persistent-nodemap'
Pulkit Goyal
requirements: introduce a set of working directory specific requirements...
r45934
Pulkit Goyal
requirements: introduce constants for `shared` and `relshared` requirements...
r45946 # Denotes that the current repository is a share
SHARED_REQUIREMENT = b'shared'
# Denotes that current repository is a share and the shared source path is
# relative to the current repository root path
RELATIVE_SHARED_REQUIREMENT = b'relshared'
Pulkit Goyal
share: introduce config option to store requires in .hg/store...
r46055 # A repository with share implemented safely. The repository has different
# store and working copy requirements i.e. both `.hg/requires` and
# `.hg/store/requires` are present.
Pulkit Goyal
share: move share safe functionality out of experimental...
r47052 SHARESAFE_REQUIREMENT = b'share-safe'
Pulkit Goyal
share: introduce config option to store requires in .hg/store...
r46055
Pulkit Goyal
requirements: introduce a set of working directory specific requirements...
r45934 # List of requirements which are working directory specific
# These requirements cannot be shared between repositories if they
# share the same store
Pulkit Goyal
requirements: introduce constants for `shared` and `relshared` requirements...
r45946 # * sparse is a working directory specific functionality and hence working
# directory specific requirement
# * SHARED_REQUIREMENT and RELATIVE_SHARED_REQUIREMENT are requirements which
# represents that the current working copy/repository shares store of another
# repo. Hence both of them should be stored in working copy
Pulkit Goyal
share: introduce config option to store requires in .hg/store...
r46055 # * SHARESAFE_REQUIREMENT needs to be stored in working dir to mark that rest of
# the requirements are stored in store's requires
Pulkit Goyal
requirements: introduce constants for `shared` and `relshared` requirements...
r45946 WORKING_DIR_REQUIREMENTS = {
SPARSE_REQUIREMENT,
SHARED_REQUIREMENT,
RELATIVE_SHARED_REQUIREMENT,
Pulkit Goyal
share: introduce config option to store requires in .hg/store...
r46055 SHARESAFE_REQUIREMENT,
Pulkit Goyal
requirements: introduce constants for `shared` and `relshared` requirements...
r45946 }