##// END OF EJS Templates
procutil: avoid using os.fork() to implement runbgcommand...
procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

File last commit:

r47575:d4ba4d51 default
r47651:8759e22f default
Show More
pushkey.py
71 lines | 1.7 KiB | text/x-python | PythonLexer
Matt Mackall
pushkey: add pushkey core
r11367 # pushkey.py - dispatching for pushing and pulling keys
#
Raphaël Gomès
contributor: change mentions of mpm to olivia...
r47575 # Copyright 2010 Olivia Mackall <olivia@selenic.com>
Matt Mackall
pushkey: add pushkey core
r11367 #
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
Gregory Szorc
pushkey: use absolute_import
r25969 from __future__ import absolute_import
from . import (
bookmarks,
encoding,
obsolete,
phases,
)
Matt Mackall
bookmarks: move pushkey functions into core
r13353
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Matt Mackall
pushkey: add pushkey core
r11367 def _nslist(repo):
n = {}
for k in _namespaces:
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 n[k] = b""
Durham Goode
obsolete: add exchange option...
r22953 if not obsolete.isenabled(repo, obsolete.exchangeopt):
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 n.pop(b'obsolete')
Matt Mackall
pushkey: add pushkey core
r11367 return n
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
_namespaces = {
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 b"namespaces": (lambda *x: False, _nslist),
b"bookmarks": (bookmarks.pushbookmark, bookmarks.listbookmarks),
b"phases": (phases.pushphase, phases.listphases),
b"obsolete": (obsolete.pushmarker, obsolete.listmarkers),
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345 }
Matt Mackall
pushkey: add pushkey core
r11367
def register(namespace, pushkey, listkeys):
_namespaces[namespace] = (pushkey, listkeys)
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Matt Mackall
pushkey: add pushkey core
r11367 def _get(namespace):
return _namespaces.get(namespace, (lambda *x: False, lambda *x: {}))
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Matt Mackall
pushkey: add pushkey core
r11367 def push(repo, namespace, key, old, new):
'''should succeed iff value was old'''
pk = _get(namespace)[0]
return pk(repo, key, old, new)
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Matt Mackall
pushkey: add pushkey core
r11367 def list(repo, namespace):
'''return a dict'''
lk = _get(namespace)[1]
return lk(repo)
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Pierre-Yves David
pushkey: add an ``encode`` function...
r21661 encode = encoding.fromlocal
Pierre-Yves David
pushkey: add a ``decode`` function...
r21659 decode = encoding.tolocal
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Pierre-Yves David
pushkey: introduce an ``encodekeys`` function...
r21650 def encodekeys(keys):
"""encode the content of a pushkey namespace for exchange over the wire"""
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 return b'\n'.join([b'%s\t%s' % (encode(k), encode(v)) for k, v in keys])
Pierre-Yves David
pushkey: introduce an ``decodekeys`` function...
r21652
Augie Fackler
style: run a patched black on a subset of mercurial...
r43345
Pierre-Yves David
pushkey: introduce an ``decodekeys`` function...
r21652 def decodekeys(data):
"""decode the content of a pushkey namespace from exchange over the wire"""
result = {}
for l in data.splitlines():
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 k, v = l.split(b'\t')
Pierre-Yves David
pushkey: add a ``decode`` function...
r21659 result[decode(k)] = decode(v)
Pierre-Yves David
pushkey: introduce an ``decodekeys`` function...
r21652 return result