##// END OF EJS Templates
merge: mark file gets as not thread safe (issue5933)...
merge: mark file gets as not thread safe (issue5933) In default installs, this has the effect of disabling the thread-based worker on Windows when manifesting files in the working directory. My measurements have shown that with revlog-based repositories, Mercurial spends a lot of CPU time in revlog code resolving file data. This ends up incurring a lot of context switching across threads and slows down `hg update` operations when going from an empty working directory to the tip of the repo. On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs): before: 487s wall after: 360s wall (equivalent to worker.enabled=false) cpus=2: 379s wall Even with only 2 threads, the thread pool is still slower. The introduction of the thread-based worker (02b36e860e0b) states that it resulted in a "~50%" speedup for `hg sparse --enable-profile` and `hg sparse --disable-profile`. This disagrees with my measurement above. I theorize a few reasons for this: 1) Removal of files from the working directory is I/O - not CPU - bound and should benefit from a thread pool (unless I/O is insanely fast and the GIL release is near instantaneous). So tests like `hg sparse --enable-profile` may exercise deletion throughput and aren't good benchmarks for worker tasks that are CPU heavy. 2) The patch was authored by someone at Facebook. The results were likely measured against a repository using remotefilelog. And I believe that revision retrieval during working directory updates with remotefilelog will often use a remote store, thus being I/O and not CPU bound. This probably resulted in an overstated performance gain. Since there appears to be a need to enable the thread-based worker with some stores, I've made the flagging of file gets as thread safe configurable. I've made it experimental because I don't want to formalize a boolean flag for this option and because this attribute is best captured against the store implementation. But we don't have a proper store API for this yet. I'd rather cross this bridge later. It is possible there are revlog-based repositories that do benefit from a thread-based worker. I didn't do very comprehensive testing. If there are, we may want to devise a more proper algorithm for whether to use the thread-based worker, including possibly config options to limit the number of threads to use. But until I see evidence that justifies complexity, simplicity wins. Differential Revision: https://phab.mercurial-scm.org/D3963

File last commit:

r38710:2570dca0 default
r38755:be498426 default
Show More
__init__.py
96 lines | 3.4 KiB | text/x-python | PythonLexer
Martin Geisler
highlight: add copyright and license header
r8251 # highlight - syntax highlighting in hgweb, based on Pygments
#
# Copyright 2008, 2009 Patrick Mezard <pmezard@gmail.com> and others
#
# This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Martin Geisler
highlight: add copyright and license header
r8251 #
# The original module was split in an interface and an implementation
# file to defer pygments loading and speedup extension setup.
Dirkjan Ochtman
extensions: fix up description lines some more
r8932 """syntax highlighting for hgweb (requires Pygments)
Patrick Mezard
highlight: split code to improve startup times
r6938
Martin Geisler
highlight: wrap docstrings at 70 characters
r9262 It depends on the Pygments syntax highlighting library:
http://pygments.org/
Patrick Mezard
highlight: split code to improve startup times
r6938
Gregory Szorc
highlight: add option to prevent content-only based fallback...
r26680 There are the following configuration options::
Patrick Mezard
highlight: split code to improve startup times
r6938
Martin Geisler
highlight: use reST syntax for literal block
r9210 [web]
av6
highlight: add highlightfiles config option which takes a fileset (issue3005)...
r26249 pygments_style = <style> (default: colorful)
highlightfiles = <fileset> (default: size('<5M'))
Gregory Szorc
highlight: add option to prevent content-only based fallback...
r26680 highlightonlymatchfilename = <bool> (default False)
``highlightonlymatchfilename`` will only highlight files if their type could
be identified by their filename. When this is not enabled (the default),
Pygments will try very hard to identify the file type from content and any
match (even matches with a low confidence score) will be used.
Patrick Mezard
highlight: split code to improve startup times
r6938 """
Pulkit Goyal
py3: make files use absolute_import and print_function...
r29485 from __future__ import absolute_import
from . import highlight
from mercurial.hgweb import (
webcommands,
webutil,
)
from mercurial import (
extensions,
)
Augie Fackler
extensions: change magic "shipped with hg" string...
r29841 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
Augie Fackler
extensions: document that `testedwith = 'internal'` is special...
r25186 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
Augie Fackler
extensions: change magic "shipped with hg" string...
r29841 testedwith = 'ships-with-hg-core'
Patrick Mezard
highlight: split code to improve startup times
r6938
Gregory Szorc
highlight: inline checkfctx()...
r26679 def pygmentize(web, field, fctx, tmpl):
style = web.config('web', 'pygments_style', 'colorful')
expr = web.config('web', 'highlightfiles', "size('<5M')")
Gregory Szorc
highlight: add option to prevent content-only based fallback...
r26680 filenameonly = web.configbool('web', 'highlightonlymatchfilename', False)
Gregory Szorc
highlight: inline checkfctx()...
r26679
av6
highlight: add highlightfiles config option which takes a fileset (issue3005)...
r26249 ctx = fctx.changectx()
Yuya Nishihara
highlight: use matcher API to test if file should be pygmentized...
r38710 m = ctx.matchfileset(expr)
if m(fctx.path()):
Gregory Szorc
highlight: add option to prevent content-only based fallback...
r26680 highlight.pygmentize(field, fctx, style, tmpl,
guessfilenameonly=filenameonly)
Gregory Szorc
highlight: consolidate duplicate code...
r26678
Gregory Szorc
hgweb: stop passing req and tmpl into @webcommand functions (API)...
r36903 def filerevision_highlight(orig, web, fctx):
Yuya Nishihara
highlight: peek Content-Type header set by hgweb...
r37036 mt = web.res.headers['Content-Type']
Rocco Rutte
highlight: only pygmentize for HTML mimetypes...
r6987 # only pygmentize for mimetype containing 'html' so we both match
# 'text/html' and possibly 'application/xhtml+xml' in the future
# so that we don't have to touch the extension when the mimetype
# for a template changes; also hgweb optimizes the case that a
# raw file is sent using rawfile() and doesn't call us, so we
# can't clash with the file's content-type here in case we
# pygmentize a html file
if 'html' in mt:
Gregory Szorc
hgweb: use templater on requestcontext instance...
r36900 pygmentize(web, 'fileline', fctx, web.tmpl)
Gregory Szorc
highlight: consolidate duplicate code...
r26678
Gregory Szorc
hgweb: stop passing req and tmpl into @webcommand functions (API)...
r36903 return orig(web, fctx)
Patrick Mezard
highlight: split code to improve startup times
r6938
Gregory Szorc
hgweb: stop passing req and tmpl into @webcommand functions (API)...
r36903 def annotate_highlight(orig, web):
Yuya Nishihara
highlight: peek Content-Type header set by hgweb...
r37036 mt = web.res.headers['Content-Type']
Rocco Rutte
highlight: only pygmentize for HTML mimetypes...
r6987 if 'html' in mt:
Gregory Szorc
hgweb: pass modern request type into various webutil functions (API)...
r36902 fctx = webutil.filectx(web.repo, web.req)
Gregory Szorc
hgweb: use templater on requestcontext instance...
r36900 pygmentize(web, 'annotateline', fctx, web.tmpl)
Gregory Szorc
highlight: consolidate duplicate code...
r26678
Gregory Szorc
hgweb: stop passing req and tmpl into @webcommand functions (API)...
r36903 return orig(web)
Patrick Mezard
highlight: split code to improve startup times
r6938
Gregory Szorc
hgweb: stop passing req and tmpl into @webcommand functions (API)...
r36903 def generate_css(web):
Patrick Mezard
highlight: split code to improve startup times
r6938 pg_style = web.config('web', 'pygments_style', 'colorful')
Mads Kiilerich
check-code: check for spaces around = for named parameters
r19872 fmter = highlight.HtmlFormatter(style=pg_style)
Gregory Szorc
hgweb: port most @webcommand to use modern response type...
r36887 web.res.headers['Content-Type'] = 'text/css'
web.res.setbodybytes(''.join([
'/* pygments_style = %s */\n\n' % pg_style,
fmter.get_style_defs(''),
]))
Gregory Szorc
hgweb: always return iterable from @webcommand functions (API)...
r36896 return web.res.sendresponse()
Patrick Mezard
highlight: split code to improve startup times
r6938
Martin Geisler
highlight: move code from module top-level into extsetup
r9409 def extsetup():
# monkeypatch in the new version
Brodie Rao
cleanup: eradicate long lines
r16683 extensions.wrapfunction(webcommands, '_filerevision',
filerevision_highlight)
Martin Geisler
highlight: move code from module top-level into extsetup
r9409 extensions.wrapfunction(webcommands, 'annotate', annotate_highlight)
webcommands.highlightcss = generate_css
webcommands.__all__.append('highlightcss')