##// END OF EJS Templates
procutil: avoid using os.fork() to implement runbgcommand...
procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

File last commit:

r47538:6266d195 default
r47651:8759e22f default
Show More
unionrepo.py
291 lines | 9.0 KiB | text/x-python | PythonLexer
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 # unionrepo.py - repository class for viewing union of repository changesets
#
# Derived from bundlerepo.py
# Copyright 2006, 2007 Benoit Boissinot <bboissin@gmail.com>
# Copyright 2013 Unity Technologies, Mads Kiilerich <madski@unity3d.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""Repository class for "in-memory pull" of one local repository to another,
allowing operations like diff and log with revsets.
"""
Gregory Szorc
unionrepo: use absolute_import
r25988 from __future__ import absolute_import
from .i18n import _
Gregory Szorc
py3: manually import getattr where it is needed...
r43359 from .pycompat import getattr
Gregory Szorc
unionrepo: use absolute_import
r25988
from . import (
changelog,
cmdutil,
Matt Harbison
py3: rename pycompat.getcwd() to encoding.getcwd() (API)...
r39843 encoding,
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 error,
Gregory Szorc
unionrepo: use absolute_import
r25988 filelog,
localrepo,
manifest,
mdiff,
pathutil,
revlog,
util,
Pierre-Yves David
vfs: use 'vfs' module directly in 'mercurial.unionrepo'...
r31242 vfs as vfsmod,
Gregory Szorc
unionrepo: use absolute_import
r25988 )
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Augie Fackler
formatting: blacken the codebase...
r43346
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 class unionrevlog(revlog.revlog):
def __init__(self, opener, indexfile, revlog2, linkmapper):
# How it works:
# To retrieve a revision, we just need to know the node id so we can
# look it up in revlog2.
#
# To differentiate a rev in the second revlog from a rev in the revlog,
# we check revision against repotiprev.
Pierre-Yves David
vfs: use 'vfs' module directly in 'mercurial.unionrepo'...
r31242 opener = vfsmod.readonlyvfs(opener)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 revlog.revlog.__init__(self, opener, indexfile)
self.revlog2 = revlog2
n = len(self)
self.repotiprev = n - 1
Augie Fackler
formatting: blacken the codebase...
r43346 self.bundlerevs = set() # used by 'bundle()' revset expression
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 for rev2 in self.revlog2:
rev = self.revlog2.index[rev2]
# rev numbers - in revlog2, very different from self.rev
Yuya Nishihara
unionrepo: fill in uncompressed length of revlog entry...
r38194 _start, _csize, rsize, base, linkrev, p1rev, p2rev, node = rev
Mike Edgar
changegroup: add flags field to cg3 delta header...
r27433 flags = _start & 0xFFFF
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Augie Fackler
formatting: blacken the codebase...
r43346 if linkmapper is None: # link is to same revlog
assert linkrev == rev2 # we never link back
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 link = n
Augie Fackler
formatting: blacken the codebase...
r43346 else: # rev must be mapped from repo2 cl to unified cl by linkmapper
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 link = linkmapper(linkrev)
Augie Fackler
formatting: blacken the codebase...
r43346 if linkmapper is not None: # link is to same revlog
Pierre-Yves David
unionrepo: take delta base in account with building unified revlog...
r26230 base = linkmapper(base)
index: use `index.get_rev` in `unionrepo.unionrevlog`...
r43965 this_rev = self.index.get_rev(node)
if this_rev is not None:
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 # this happens for the common revlog revisions
index: use `index.get_rev` in `unionrepo.unionrevlog`...
r43965 self.bundlerevs.add(this_rev)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 continue
p1node = self.revlog2.node(p1rev)
p2node = self.revlog2.node(p2rev)
Joerg Sonnenberger
unionrepo: don't insert index tuples with None as int field...
r46419 # TODO: it's probably wrong to set compressed length to -1, but
Yuya Nishihara
unionrepo: fill in uncompressed length of revlog entry...
r38194 # I have no idea if csize is valid in the base revlog context.
Augie Fackler
formatting: blacken the codebase...
r43346 e = (
flags,
Joerg Sonnenberger
unionrepo: don't insert index tuples with None as int field...
r46419 -1,
Augie Fackler
formatting: blacken the codebase...
r43346 rsize,
base,
link,
self.rev(p1node),
self.rev(p2node),
node,
)
Martin von Zweigbergk
index: replace insert(-1, e) method by append(e) method...
r38886 self.index.append(e)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 self.bundlerevs.add(n)
n += 1
def _chunk(self, rev):
if rev <= self.repotiprev:
return revlog.revlog._chunk(self, rev)
return self.revlog2._chunk(self.node(rev))
def revdiff(self, rev1, rev2):
"""return or calculate a delta between two revisions"""
if rev1 > self.repotiprev and rev2 > self.repotiprev:
return self.revlog2.revdiff(
self.revlog2.rev(self.node(rev1)),
Augie Fackler
formatting: blacken the codebase...
r43346 self.revlog2.rev(self.node(rev2)),
)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 elif rev1 <= self.repotiprev and rev2 <= self.repotiprev:
unionrepo: use normal inheritance scheme to call revdiff
r43095 return super(unionrevlog, self).revdiff(rev1, rev2)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
unionrepo: fix `revdiff` implementation to use `rawdata`...
r43094 return mdiff.textdiff(self.rawdata(rev1), self.rawdata(rev2))
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
unionrepo: use a lower level overide in unionrepo too...
r43092 def _revisiondata(self, nodeorrev, _df=None, raw=False):
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if isinstance(nodeorrev, int):
rev = nodeorrev
node = self.node(rev)
else:
node = nodeorrev
rev = self.rev(node)
if rev > self.repotiprev:
unionrepo: use a lower level overide in unionrepo too...
r43092 # work around manifestrevlog NOT being a revlog
revlog2 = getattr(self.revlog2, '_revlog', self.revlog2)
func = revlog2._revisiondata
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 else:
unionrepo: use a lower level overide in unionrepo too...
r43092 func = super(unionrevlog, self)._revisiondata
return func(node, _df=_df, raw=raw)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
def addrevision(self, text, transaction, link, p1=None, p2=None, d=None):
raise NotImplementedError
Augie Fackler
formatting: blacken the codebase...
r43346
def addgroup(
self,
deltas,
linkmapper,
transaction,
Joerg Sonnenberger
revlog: decouple caching from addrevision callback for addgroup...
r47085 alwayscache=False,
Augie Fackler
formatting: blacken the codebase...
r43346 addrevisioncb=None,
Joerg Sonnenberger
revlog: extend addgroup() with callback for duplicates...
r46373 duplicaterevisioncb=None,
Augie Fackler
formatting: blacken the codebase...
r43346 maybemissingparents=False,
):
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 raise NotImplementedError
Augie Fackler
formatting: blacken the codebase...
r43346
Joerg Sonnenberger
unionrepo: sync with repository API...
r42382 def strip(self, minlink, transaction):
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 raise NotImplementedError
Augie Fackler
formatting: blacken the codebase...
r43346
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 def checksize(self):
raise NotImplementedError
Augie Fackler
formatting: blacken the codebase...
r43346
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 class unionchangelog(unionrevlog, changelog.changelog):
def __init__(self, opener, opener2):
changelog.changelog.__init__(self, opener)
linkmapper = None
changelog2 = changelog.changelog(opener2)
Augie Fackler
formatting: blacken the codebase...
r43346 unionrevlog.__init__(
self, opener, self.indexfile, changelog2, linkmapper
)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Durham Goode
manifest: add unionmanifestlog support...
r30374 class unionmanifest(unionrevlog, manifest.manifestrevlog):
Joerg Sonnenberger
node: introduce nodeconstants class...
r47538 def __init__(self, nodeconstants, opener, opener2, linkmapper):
manifest.manifestrevlog.__init__(self, nodeconstants, opener)
manifest2 = manifest.manifestrevlog(nodeconstants, opener2)
Augie Fackler
formatting: blacken the codebase...
r43346 unionrevlog.__init__(
self, opener, self.indexfile, manifest2, linkmapper
)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Gregory Szorc
filelog: wrap revlog instead of inheriting it (API)...
r37515 class unionfilelog(filelog.filelog):
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 def __init__(self, opener, path, opener2, linkmapper, repo):
filelog.filelog.__init__(self, opener, path)
filelog2 = filelog.filelog(opener2, path)
Augie Fackler
formatting: blacken the codebase...
r43346 self._revlog = unionrevlog(
opener, self.indexfile, filelog2._revlog, linkmapper
)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 self._repo = repo
Gregory Szorc
filelog: wrap revlog instead of inheriting it (API)...
r37515 self.repotiprev = self._revlog.repotiprev
self.revlog2 = self._revlog.revlog2
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Mike Edgar
revlog: add "iscensored()" to revlog public API...
r24118 def iscensored(self, rev):
"""Check if a revision is censored."""
if rev <= self.repotiprev:
return filelog.filelog.iscensored(self, rev)
Sean Farley
unionrepo: fix wrong rev being checked in iscensored (issue5024)
r27723 node = self.node(rev)
return self.revlog2.iscensored(self.revlog2.rev(node))
Mike Edgar
revlog: add "iscensored()" to revlog public API...
r24118
Augie Fackler
formatting: blacken the codebase...
r43346
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 class unionpeer(localrepo.localpeer):
def canpush(self):
return False
Augie Fackler
formatting: blacken the codebase...
r43346
Gregory Szorc
unionrepo: dynamically create repository type from base repository...
r39641 class unionrepository(object):
"""Represents the union of data in 2 repositories.
Instances are not usable if constructed directly. Use ``instance()``
or ``makeunionrepository()`` to create a usable instance.
"""
Augie Fackler
formatting: blacken the codebase...
r43346
Gregory Szorc
unionrepo: dynamically create repository type from base repository...
r39641 def __init__(self, repo2, url):
self.repo2 = repo2
self._url = url
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 self.ui.setconfig(b'phases', b'publish', False, b'unionrepo')
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
@localrepo.unfilteredpropertycache
def changelog(self):
Angel Ezquerra
localrepo: remove all external users of localrepo.sopener...
r23878 return unionchangelog(self.svfs, self.repo2.svfs)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Gregory Szorc
localrepo: pass root manifest into manifestlog.__init__...
r39799 @localrepo.unfilteredpropertycache
def manifestlog(self):
Augie Fackler
formatting: blacken the codebase...
r43346 rootstore = unionmanifest(
Joerg Sonnenberger
node: introduce nodeconstants class...
r47538 self.nodeconstants,
self.svfs,
self.repo2.svfs,
self.unfiltered()._clrev,
Augie Fackler
formatting: blacken the codebase...
r43346 )
return manifest.manifestlog(
self.svfs, self, rootstore, self.narrowmatch()
)
Gregory Szorc
localrepo: pass root manifest into manifestlog.__init__...
r39799
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 def _clrev(self, rev2):
"""map from repo2 changelog rev to temporary rev in self.changelog"""
node = self.repo2.changelog.node(rev2)
return self.changelog.rev(node)
def url(self):
return self._url
def file(self, f):
Augie Fackler
formatting: blacken the codebase...
r43346 return unionfilelog(
self.svfs, f, self.repo2.svfs, self.unfiltered()._clrev, self
)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
def close(self):
self.repo2.close()
def cancopy(self):
return False
def peer(self):
return unionpeer(self)
def getcwd(self):
Augie Fackler
formatting: blacken the codebase...
r43346 return encoding.getcwd() # always outside the repo
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944
Gregory Szorc
hg: allow extra arguments to be passed to repo creation (API)...
r39585 def instance(ui, path, create, intents=None, createopts=None):
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if create:
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 raise error.Abort(_(b'cannot create new union repository'))
parentpath = ui.config(b"bundle", b"mainreporoot")
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if not parentpath:
# try to find the correct path to the working directory repo
Matt Harbison
py3: rename pycompat.getcwd() to encoding.getcwd() (API)...
r39843 parentpath = cmdutil.findrepo(encoding.getcwd())
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if parentpath is None:
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 parentpath = b''
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if parentpath:
# Try to make the full path relative so we get a nice, short URL.
# In particular, we don't want temp dir names in test outputs.
Matt Harbison
py3: rename pycompat.getcwd() to encoding.getcwd() (API)...
r39843 cwd = encoding.getcwd()
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if parentpath == cwd:
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 parentpath = b''
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 else:
FUJIWARA Katsunori
unionrepo: use pathutil.normasprefix to ensure os.sep at the end of cwd...
r24835 cwd = pathutil.normasprefix(cwd)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if parentpath.startswith(cwd):
Augie Fackler
formatting: blacken the codebase...
r43346 parentpath = parentpath[len(cwd) :]
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 if path.startswith(b'union:'):
s = path.split(b":", 1)[1].split(b"+", 1)
Mads Kiilerich
unionrepo: read-only operations on a union of two localrepos...
r18944 if len(s) == 1:
repopath, repopath2 = parentpath, s[0]
else:
repopath, repopath2 = s
else:
repopath, repopath2 = parentpath, path
Gregory Szorc
unionrepo: dynamically create repository type from base repository...
r39641
return makeunionrepository(ui, repopath, repopath2)
Augie Fackler
formatting: blacken the codebase...
r43346
Gregory Szorc
unionrepo: dynamically create repository type from base repository...
r39641 def makeunionrepository(ui, repopath1, repopath2):
"""Make a union repository object from 2 local repo paths."""
repo1 = localrepo.instance(ui, repopath1, create=False)
repo2 = localrepo.instance(ui, repopath2, create=False)
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 url = b'union:%s+%s' % (
Augie Fackler
formatting: blacken the codebase...
r43346 util.expandpath(repopath1),
util.expandpath(repopath2),
)
Gregory Szorc
unionrepo: dynamically create repository type from base repository...
r39641
class derivedunionrepository(unionrepository, repo1.__class__):
pass
repo = repo1
repo.__class__ = derivedunionrepository
unionrepository.__init__(repo1, repo2, url)
return repo