##// END OF EJS Templates
pathcopies: give up any optimization based on `introrev`...
pathcopies: give up any optimization based on `introrev` Between 8a0136f69027 and d98fb3f42f33, we sped up the search for the introduction revision during path copies. However, further checking show that finding the introduction revision is still expensive and that we are better off without it. So we simply drop it and only rely on the linkrev optimisation. I ran `perfpathcopies` on 6989 pair of revision in the pypy repository (`hg perfhelper-pathcopies`. The result is massively in favor of dropping this condition. The result of the copy tracing are unchanged. Attempt to use a smaller changes preserving linkrev usage were unsuccessful, it can return wrong result. The following changesets broke test-mv-cp-st-diff.t - if not f.isintroducedafter(limit): + if limit >= 0 and f.linkrev() < limit: return None Here are various numbers (before this changeset/after this changesets) source destination before after saved-time ratio worth cases e66f24650daf 695dfb0f493b 1.062843 1.246369 -0.183526 1.172675 c979853a3b6a 8d60fe293e79 1.036985 1.196414 -0.159429 1.153743 22349fa2fc33 fbb1c9fd86c0 0.879926 1.038682 -0.158756 1.180420 682b98f3e672 a4878080a536 0.909952 1.063801 -0.153849 1.169074 5adabc9b9848 920958a93997 0.993622 1.147452 -0.153830 1.154817 worse 1% dbfbfcf077e9 aea8f2fd3593 1.016595 1.082999 -0.066404 1.065320 worse 5% c95f1ced15f2 7d29d5e39734 0.453694 0.471156 -0.017462 1.038488 worse 10% 3e144ed1d5b7 2aef0e942480 0.035140 0.037535 -0.002395 1.068156 worse 25% 321fc60db035 801748ba582a 0.009267 0.009325 -0.000058 1.006259 median 2088ce763fc2 e6991321d78b 0.000665 0.000651 0.000014 0.978947 best 25% 915631a97de6 385b31354be6 0.040743 0.040363 0.000380 0.990673 best 10% ad495c36a765 19c10384d3e7 0.431658 0.411490 0.020168 0.953278 best 5% d13ae7d283ae 813c99f810ac 1.141404 1.075346 0.066058 0.942126 best 1% 81593cb4a496 99ae11866969 1.833297 0.063823 1.769474 0.034813 best cases c3b14617fbd7 743a0fcaa4eb 1101.811740 2.735970 1099.075770 0.002483 c3b14617fbd7 9ba6ab77fd29 1116.753953 2.800729 1113.953224 0.002508 058b99d6e81f 57e249b7a3ea 1246.128485 3.042762 1243.085723 0.002442 9a8c361aab49 0354a250d371 1253.111894 3.085796 1250.026098 0.002463 442dbbc53c68 3ec1002a818c 1261.786294 3.138607 1258.647687 0.002487 As one can see, the average case is not really impacted. However, the worth case we get after this changeset are much better than the one we had before it. We have 30 pairs where improvements are above 10 minutes. This reflect in the combined time for all pairs before: 26256s after: 1300s (-95%) If we remove these pathological 30 cases, we still see a significant improvements: before: 1631s after: 1245s (-24%)

File last commit:

r43346:2372284d default
r43469:c16fe77e default
Show More
check-translation.py
272 lines | 7.4 KiB | text/x-python | PythonLexer
/ i18n / check-translation.py
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 #!/usr/bin/env python
#
# check-translation.py - check Mercurial specific translation problems
Augie Fackler
i18n: update check-translation script to pass import checker
r33900 from __future__ import absolute_import
import re
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152
import polib
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 scanners = []
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 checkers = []
Augie Fackler
formatting: blacken the codebase...
r43346
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 def scanner():
def decorator(func):
scanners.append(func)
return func
Augie Fackler
formatting: blacken the codebase...
r43346
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 return decorator
Augie Fackler
formatting: blacken the codebase...
r43346
Mads Kiilerich
cleanup: rename check-translation.py checker function - don't hide global var
r22203 def levelchecker(level, msgidpat):
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 def decorator(func):
if msgidpat:
match = re.compile(msgidpat).search
else:
match = lambda msgid: True
checkers.append((func, level))
func.match = match
return func
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 return decorator
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 def match(checker, pe):
"""Examine whether POEntry "pe" is target of specified checker or not
"""
if not checker.match(pe.msgid):
return
# examine suppression by translator comment
nochecker = 'no-%s-check' % checker.__name__
for tc in pe.tcomment.split():
if nochecker == tc:
return
return True
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 ####################
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 def fatalchecker(msgidpat=None):
Mads Kiilerich
cleanup: rename check-translation.py checker function - don't hide global var
r22203 return levelchecker('fatal', msgidpat)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 @fatalchecker(r'\$\$')
def promptchoice(pe):
"""Check translation of the string given to "ui.promptchoice()"
>>> pe = polib.POEntry(
... msgid ='prompt$$missing &sep$$missing &amp$$followed by &none',
... msgstr='prompt missing &sep$$missing amp$$followed by none&')
>>> match(promptchoice, pe)
True
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in promptchoice(pe): print(e)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 number of choices differs between msgid and msgstr
msgstr has invalid choice missing '&'
msgstr has invalid '&' followed by none
"""
idchoices = [c.rstrip(' ') for c in pe.msgid.split('$$')[1:]]
strchoices = [c.rstrip(' ') for c in pe.msgstr.split('$$')[1:]]
if len(idchoices) != len(strchoices):
yield "number of choices differs between msgid and msgstr"
indices = [(c, c.find('&')) for c in strchoices]
if [c for c, i in indices if i == -1]:
yield "msgstr has invalid choice missing '&'"
if [c for c, i in indices if len(c) == i + 1]:
yield "msgstr has invalid '&' followed by none"
Augie Fackler
formatting: blacken the codebase...
r43346
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 deprecatedpe = None
Augie Fackler
formatting: blacken the codebase...
r43346
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 @scanner()
def deprecatedsetup(pofile):
Yuya Nishihara
i18n: do not abuse msgstr of "DEPRECATED" to check for bad translation...
r26852 pes = [p for p in pofile if p.msgid == '(DEPRECATED)' and p.msgstr]
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 if len(pes):
global deprecatedpe
deprecatedpe = pes[0]
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: fix regexp pattern to detect translation for DEPRECATED...
r26837 @fatalchecker(r'\(DEPRECATED\)')
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 def deprecated(pe):
"""Check for DEPRECATED
>>> ped = polib.POEntry(
Yuya Nishihara
i18n: do not abuse msgstr of "DEPRECATED" to check for bad translation...
r26852 ... msgid = '(DEPRECATED)',
... msgstr= '(DETACERPED)')
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 >>> deprecatedsetup([ped])
>>> pe = polib.POEntry(
... msgid = 'Something (DEPRECATED)',
timeless@mozdev.org
tests: add more doctests for check-translation deprecated
r26277 ... msgstr= 'something (DEPRECATED)')
>>> match(deprecated, pe)
True
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in deprecated(pe): print(e)
timeless@mozdev.org
tests: add more doctests for check-translation deprecated
r26277 >>> pe = polib.POEntry(
... msgid = 'Something (DEPRECATED)',
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 ... msgstr= 'something (DETACERPED)')
>>> match(deprecated, pe)
True
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in deprecated(pe): print(e)
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 >>> pe = polib.POEntry(
... msgid = 'Something (DEPRECATED)',
... msgstr= 'something')
>>> match(deprecated, pe)
True
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in deprecated(pe): print(e)
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 msgstr inconsistently translated (DEPRECATED)
FUJIWARA Katsunori
i18n: fix regexp pattern to detect translation for DEPRECATED...
r26837 >>> pe = polib.POEntry(
... msgid = 'Something (DEPRECATED, foo bar)',
... msgstr= 'something (DETACERPED, foo bar)')
>>> match(deprecated, pe)
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 """
Augie Fackler
formatting: blacken the codebase...
r43346 if not (
'(DEPRECATED)' in pe.msgstr
or (deprecatedpe and deprecatedpe.msgstr in pe.msgstr)
):
timeless@mozdev.org
tests: cleanup check-translation deprecated
r26276 yield "msgstr inconsistently translated (DEPRECATED)"
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 ####################
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 def warningchecker(msgidpat=None):
Mads Kiilerich
cleanup: rename check-translation.py checker function - don't hide global var
r22203 return levelchecker('warning', msgidpat)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: check equality of tail '::'-ness between msgid and msgstr...
r20514 @warningchecker()
def taildoublecolons(pe):
"""Check equality of tail '::'-ness between msgid and msgstr
>>> pe = polib.POEntry(
... msgid ='ends with ::',
... msgstr='ends with ::')
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in taildoublecolons(pe): print(e)
FUJIWARA Katsunori
i18n: check equality of tail '::'-ness between msgid and msgstr...
r20514 >>> pe = polib.POEntry(
... msgid ='ends with ::',
... msgstr='ends without double-colons')
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in taildoublecolons(pe): print(e)
FUJIWARA Katsunori
i18n: check equality of tail '::'-ness between msgid and msgstr...
r20514 tail '::'-ness differs between msgid and msgstr
>>> pe = polib.POEntry(
... msgid ='ends without double-colons',
... msgstr='ends with ::')
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in taildoublecolons(pe): print(e)
FUJIWARA Katsunori
i18n: check equality of tail '::'-ness between msgid and msgstr...
r20514 tail '::'-ness differs between msgid and msgstr
"""
if pe.msgid.endswith('::') != pe.msgstr.endswith('::'):
yield "tail '::'-ness differs between msgid and msgstr"
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: check equality of initial indentation between msgid and msgstr...
r20515 @warningchecker()
def indentation(pe):
"""Check equality of initial indentation between msgid and msgstr
This may report unexpected warning, because this doesn't aware
the syntax of rst document and the context of msgstr.
>>> pe = polib.POEntry(
... msgid =' indented text',
... msgstr=' narrowed indentation')
Augie Fackler
i18n: fix check-translation.py to be less broken on Python 3...
r33681 >>> for e in indentation(pe): print(e)
FUJIWARA Katsunori
i18n: check equality of initial indentation between msgid and msgstr...
r20515 initial indentation width differs betweeen msgid and msgstr
"""
idindent = len(pe.msgid) - len(pe.msgid.lstrip())
strindent = len(pe.msgstr) - len(pe.msgstr.lstrip())
if idindent != strindent:
yield "initial indentation width differs betweeen msgid and msgstr"
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 ####################
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 def check(pofile, fatal=True, warning=False):
Augie Fackler
formatting: blacken the codebase...
r43346 targetlevel = {'fatal': fatal, 'warning': warning}
targetcheckers = [
(checker, level) for checker, level in checkers if targetlevel[level]
]
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 if not targetcheckers:
return []
detected = []
timeless@mozdev.org
tests: check for inconsistently translated DEPRECATED...
r26261 for checker in scanners:
checker(pofile)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 for pe in pofile.translated_entries():
errors = []
for checker, level in targetcheckers:
if match(checker, pe):
Augie Fackler
formatting: blacken the codebase...
r43346 errors.extend(
(level, checker.__name__, error) for error in checker(pe)
)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 if errors:
detected.append((pe, errors))
return detected
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 ########################################
if __name__ == "__main__":
import sys
import optparse
Augie Fackler
formatting: blacken the codebase...
r43346 optparser = optparse.OptionParser(
"""%prog [options] pofile ...
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152
This checks Mercurial specific translation problems in specified
'*.po' files.
Each detected problems are shown in the format below::
filename:linenum:type(checker): problem detail .....
"type" is "fatal" or "warning". "checker" is the name of the function
detecting corresponded error.
Checking by checker "foo" on the specific msgstr can be suppressed by
the "translator comment" like below. Multiple "no-xxxx-check" should
be separated by whitespaces::
# no-foo-check
msgid = "....."
msgstr = "....."
Augie Fackler
formatting: blacken the codebase...
r43346 """
)
optparser.add_option(
"",
"--warning",
help="show also warning level problems",
action="store_true",
)
optparser.add_option(
"",
"--doctest",
help="run doctest of this tool, instead of check",
action="store_true",
)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 (options, args) = optparser.parse_args()
if options.doctest:
Matt Mackall
tests: fix missing import in check-translations
r20164 import os
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
tests: fix Mac doctest escape code garbage for check-translations
r20158 if 'TERM' in os.environ:
del os.environ['TERM']
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 import doctest
Augie Fackler
formatting: blacken the codebase...
r43346
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 failures, tests = doctest.testmod()
sys.exit(failures and 1 or 0)
detected = []
warning = options.warning
for f in args:
Augie Fackler
formatting: blacken the codebase...
r43346 detected.extend(
(f, pe, errors)
for pe, errors in check(polib.pofile(f), warning=warning)
)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 if detected:
for f, pe, errors in detected:
for level, checker, error in errors:
Augie Fackler
formatting: blacken the codebase...
r43346 sys.stderr.write(
'%s:%d:%s(%s): %s\n'
% (f, pe.linenum, level, checker, error)
)
FUJIWARA Katsunori
i18n: add the tool to check Mercurial specific translation problems in *.po...
r20152 sys.exit(1)