##// END OF EJS Templates
merge: mark file gets as not thread safe (issue5933)...
merge: mark file gets as not thread safe (issue5933) In default installs, this has the effect of disabling the thread-based worker on Windows when manifesting files in the working directory. My measurements have shown that with revlog-based repositories, Mercurial spends a lot of CPU time in revlog code resolving file data. This ends up incurring a lot of context switching across threads and slows down `hg update` operations when going from an empty working directory to the tip of the repo. On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs): before: 487s wall after: 360s wall (equivalent to worker.enabled=false) cpus=2: 379s wall Even with only 2 threads, the thread pool is still slower. The introduction of the thread-based worker (02b36e860e0b) states that it resulted in a "~50%" speedup for `hg sparse --enable-profile` and `hg sparse --disable-profile`. This disagrees with my measurement above. I theorize a few reasons for this: 1) Removal of files from the working directory is I/O - not CPU - bound and should benefit from a thread pool (unless I/O is insanely fast and the GIL release is near instantaneous). So tests like `hg sparse --enable-profile` may exercise deletion throughput and aren't good benchmarks for worker tasks that are CPU heavy. 2) The patch was authored by someone at Facebook. The results were likely measured against a repository using remotefilelog. And I believe that revision retrieval during working directory updates with remotefilelog will often use a remote store, thus being I/O and not CPU bound. This probably resulted in an overstated performance gain. Since there appears to be a need to enable the thread-based worker with some stores, I've made the flagging of file gets as thread safe configurable. I've made it experimental because I don't want to formalize a boolean flag for this option and because this attribute is best captured against the store implementation. But we don't have a proper store API for this yet. I'd rather cross this bridge later. It is possible there are revlog-based repositories that do benefit from a thread-based worker. I didn't do very comprehensive testing. If there are, we may want to devise a more proper algorithm for whether to use the thread-based worker, including possibly config options to limit the number of threads to use. But until I see evidence that justifies complexity, simplicity wins. Differential Revision: https://phab.mercurial-scm.org/D3963

File last commit:

r33368:ffb30661 default
r38755:be498426 default
Show More
transport.py
132 lines | 4.9 KiB | text/x-python | PythonLexer
# -*- coding: utf-8 -*-
# Copyright (C) 2007 Daniel Holth <dholth@fastmail.fm>
# This is a stripped-down version of the original bzr-svn transport.py,
# Copyright (C) 2006 Jelmer Vernooij <jelmer@samba.org>
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
from __future__ import absolute_import
import svn.client
import svn.core
import svn.ra
Pool = svn.core.Pool
SubversionException = svn.core.SubversionException
from mercurial import (
util,
)
# Some older versions of the Python bindings need to be
# explicitly initialized. But what we want to do probably
# won't work worth a darn against those libraries anyway!
svn.ra.initialize()
svn_config = None
def _create_auth_baton(pool):
"""Create a Subversion authentication baton. """
import svn.client
# Give the client context baton a suite of authentication
# providers.h
providers = [
svn.client.get_simple_provider(pool),
svn.client.get_username_provider(pool),
svn.client.get_ssl_client_cert_file_provider(pool),
svn.client.get_ssl_client_cert_pw_file_provider(pool),
svn.client.get_ssl_server_trust_file_provider(pool),
]
# Platform-dependent authentication methods
getprovider = getattr(svn.core, 'svn_auth_get_platform_specific_provider',
None)
if getprovider:
# Available in svn >= 1.6
for name in ('gnome_keyring', 'keychain', 'kwallet', 'windows'):
for type in ('simple', 'ssl_client_cert_pw', 'ssl_server_trust'):
p = getprovider(name, type, pool)
if p:
providers.append(p)
else:
if util.safehasattr(svn.client, 'get_windows_simple_provider'):
providers.append(svn.client.get_windows_simple_provider(pool))
return svn.core.svn_auth_open(providers, pool)
class NotBranchError(SubversionException):
pass
class SvnRaTransport(object):
"""
Open an ra connection to a Subversion repository.
"""
def __init__(self, url="", ra=None):
self.pool = Pool()
self.svn_url = url
self.username = ''
self.password = ''
# Only Subversion 1.4 has reparent()
if ra is None or not util.safehasattr(svn.ra, 'reparent'):
self.client = svn.client.create_context(self.pool)
ab = _create_auth_baton(self.pool)
self.client.auth_baton = ab
global svn_config
if svn_config is None:
svn_config = svn.core.svn_config_get_config(None)
self.client.config = svn_config
try:
self.ra = svn.client.open_ra_session(
self.svn_url,
self.client, self.pool)
except SubversionException as xxx_todo_changeme:
(inst, num) = xxx_todo_changeme.args
if num in (svn.core.SVN_ERR_RA_ILLEGAL_URL,
svn.core.SVN_ERR_RA_LOCAL_REPOS_OPEN_FAILED,
svn.core.SVN_ERR_BAD_URL):
raise NotBranchError(url)
raise
else:
self.ra = ra
svn.ra.reparent(self.ra, self.svn_url.encode('utf8'))
class Reporter(object):
def __init__(self, reporter_data):
self._reporter, self._baton = reporter_data
def set_path(self, path, revnum, start_empty, lock_token, pool=None):
svn.ra.reporter2_invoke_set_path(self._reporter, self._baton,
path, revnum, start_empty, lock_token, pool)
def delete_path(self, path, pool=None):
svn.ra.reporter2_invoke_delete_path(self._reporter, self._baton,
path, pool)
def link_path(self, path, url, revision, start_empty, lock_token,
pool=None):
svn.ra.reporter2_invoke_link_path(self._reporter, self._baton,
path, url, revision, start_empty, lock_token,
pool)
def finish_report(self, pool=None):
svn.ra.reporter2_invoke_finish_report(self._reporter,
self._baton, pool)
def abort_report(self, pool=None):
svn.ra.reporter2_invoke_abort_report(self._reporter,
self._baton, pool)
def do_update(self, revnum, path, *args, **kwargs):
return self.Reporter(svn.ra.do_update(self.ra, revnum, path,
*args, **kwargs))