##// END OF EJS Templates
procutil: avoid using os.fork() to implement runbgcommand...
procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

File last commit:

r47575:d4ba4d51 default
r47651:8759e22f default
Show More
resourceutil.py
104 lines | 3.2 KiB | text/x-python | PythonLexer
Martin von Zweigbergk
procutil: move mainfrozen() to new resourceutil.py...
r44067 # resourceutil.py - utility for looking up resources
#
# Copyright 2005 K. Thananchayan <thananck@yahoo.com>
Raphaël Gomès
contributor: change mentions of mpm to olivia...
r47575 # Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
Martin von Zweigbergk
procutil: move mainfrozen() to new resourceutil.py...
r44067 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
import imp
Martin von Zweigbergk
util: move definition of datapath to resourceutil...
r44068 import os
Martin von Zweigbergk
procutil: move mainfrozen() to new resourceutil.py...
r44067 import sys
from .. import pycompat
def mainfrozen():
"""return True if we are a frozen executable.
The code supports py2exe (most common, Windows only) and tools/freeze
(portable, not much used).
"""
return (
Manuel Jacob
resourceutil: fix location of line comments...
r45476 pycompat.safehasattr(sys, "frozen") # new py2exe
or pycompat.safehasattr(sys, "importers") # old py2exe
or imp.is_frozen("__main__") # tools/freeze
)
Martin von Zweigbergk
util: move definition of datapath to resourceutil...
r44068
# the location of data files matching the source code
Augie Fackler
resourceutil: blacken
r44723 if mainfrozen() and getattr(sys, "frozen", None) != "macosx_app":
Martin von Zweigbergk
util: move definition of datapath to resourceutil...
r44068 # executable version (py2exe) doesn't support __file__
datapath = os.path.dirname(pycompat.sysexecutable)
Matt Harbison
resourceutil: correct the root path for file based lookup under py2exe...
r44678 _rootpath = datapath
Matt Harbison
resourceutil: account for the non-resource-like file hierarchy under py2exe...
r44706
# The installers store the files outside of library.zip, like
# C:\Program Files\Mercurial\defaultrc\*.rc. This strips the
# leading "mercurial." off of the package name, so that these
# pseudo resources are found in their directory next to the
# executable.
def _package_path(package):
Augie Fackler
resourceutil: blacken
r44723 dirs = package.split(b".")
assert dirs[0] == b"mercurial"
Matt Harbison
resourceutil: account for the non-resource-like file hierarchy under py2exe...
r44706 return os.path.join(_rootpath, *dirs[1:])
Augie Fackler
resourceutil: blacken
r44723
Martin von Zweigbergk
util: move definition of datapath to resourceutil...
r44068 else:
datapath = os.path.dirname(os.path.dirname(pycompat.fsencode(__file__)))
Matt Harbison
resourceutil: don't limit resources to the `mercurial` package...
r44479 _rootpath = os.path.dirname(datapath)
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323
Matt Harbison
resourceutil: account for the non-resource-like file hierarchy under py2exe...
r44706 def _package_path(package):
Augie Fackler
resourceutil: blacken
r44723 return os.path.join(_rootpath, *package.split(b"."))
Matt Harbison
resourceutil: account for the non-resource-like file hierarchy under py2exe...
r44706
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323 try:
Martin von Zweigbergk
resourceutil: document when we expect to take the importlib.resouces code path...
r46024 # importlib.resources exists from Python 3.7; see fallback in except clause
# further down
Martin von Zweigbergk
resourceutil: use `from importlib import resources`...
r44407 from importlib import resources
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323
Matt Harbison
resourceutil: implement `is_resource()`...
r44480 from .. import encoding
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323 # Force loading of the resources module
Martin von Zweigbergk
resourceutil: use `from importlib import resources`...
r44407 resources.open_binary # pytype: disable=module-attr
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323
def open_resource(package, name):
Martin von Zweigbergk
resourceutil: use `from importlib import resources`...
r44407 return resources.open_binary( # pytype: disable=module-attr
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323 pycompat.sysstr(package), pycompat.sysstr(name)
)
Matt Harbison
resourceutil: implement `is_resource()`...
r44480 def is_resource(package, name):
Matt Harbison
typing: disable a few errors calling py3.7+ functions in resourceutil.py...
r47541 return resources.is_resource( # pytype: disable=module-attr
Matt Harbison
resourceutil: implement `is_resource()`...
r44480 pycompat.sysstr(package), encoding.strfromlocal(name)
)
Matt Harbison
resourceutil: implement `contents()` to iterate over resources in a package...
r44481 def contents(package):
Matt Harbison
typing: disable a few errors calling py3.7+ functions in resourceutil.py...
r47541 # pytype: disable=module-attr
Matt Harbison
resourceutil: implement `contents()` to iterate over resources in a package...
r44481 for r in resources.contents(pycompat.sysstr(package)):
Matt Harbison
typing: disable a few errors calling py3.7+ functions in resourceutil.py...
r47541 # pytype: enable=module-attr
Matt Harbison
resourceutil: implement `contents()` to iterate over resources in a package...
r44481 yield encoding.strtolocal(r)
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323
Martin von Zweigbergk
resourceutil: use `from importlib import resources`...
r44407 except (ImportError, AttributeError):
Martin von Zweigbergk
resourceutil: document when we expect to take the importlib.resouces code path...
r46024 # importlib.resources was not found (almost definitely because we're on a
# Python version before 3.7)
Martin von Zweigbergk
help: get helptext/ data from `resources` module if available...
r44323
def open_resource(package, name):
path = os.path.join(_package_path(package), name)
Augie Fackler
resourceutil: blacken
r44723 return open(path, "rb")
Matt Harbison
resourceutil: implement `is_resource()`...
r44480
def is_resource(package, name):
path = os.path.join(_package_path(package), name)
try:
return os.path.isfile(pycompat.fsdecode(path))
except (IOError, OSError):
return False
Matt Harbison
resourceutil: implement `contents()` to iterate over resources in a package...
r44481
def contents(package):
path = pycompat.fsdecode(_package_path(package))
for p in os.listdir(path):
yield pycompat.fsencode(p)