##// END OF EJS Templates
parsers: inline fields of dirstate values in C version...
parsers: inline fields of dirstate values in C version Previously, while unpacking the dirstate we'd create 3-4 new CPython objects for most dirstate values: - the state is a single character string, which is pooled by CPython - the mode is a new object if it isn't 0 due to being in the lookup set - the size is a new object if it is greater than 255 - the mtime is a new object if it isn't -1 due to being in the lookup set - the tuple to contain them all In some cases such as regular hg status, we actually look at all the objects. In other cases like hg add, hg status for a subdirectory, or hg status with the third-party hgwatchman enabled, we look at almost none of the objects. This patch eliminates most object creation in these cases by defining a custom C struct that is exposed to Python with an interface similar to a tuple. Only when tuple elements are actually requested are the respective objects created. The gains, where they're expected, are significant. The following tests are run against a working copy with over 270,000 files. parse_dirstate becomes significantly faster: $ hg perfdirstate before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35) after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95) and as a result, several commands benefit: $ time hg status # with hgwatchman enabled before: 0.42s user 0.14s system 99% cpu 0.563 total after: 0.34s user 0.12s system 99% cpu 0.471 total $ time hg add new-file before: 0.85s user 0.18s system 99% cpu 1.033 total after: 0.76s user 0.17s system 99% cpu 0.931 total There is a slight regression in regular status performance, but this is fixed in an upcoming patch.

File last commit:

r15530:eeac5e17 default
r21809:e250b830 default
Show More
bdiff.py
87 lines | 2.3 KiB | text/x-python | PythonLexer
Martin Geisler
pure Python implementation of bdiff.c
r7703 # bdiff.py - Python implementation of bdiff.c
#
# Copyright 2009 Matt Mackall <mpm@selenic.com> and others
#
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Martin Geisler
pure Python implementation of bdiff.c
r7703
Patrick Mezard
mdiff: replace wscleanup() regexps with C loops...
r15530 import struct, difflib, re
Matt Mackall
pure/bdiff: fix circular import
r7944
def splitnewlines(text):
'''like str.splitlines, but only split on newlines.'''
lines = [l + '\n' for l in text.split('\n')]
if lines:
if lines[-1] == '\n':
lines.pop()
else:
lines[-1] = lines[-1][:-1]
return lines
Martin Geisler
pure Python implementation of bdiff.c
r7703
def _normalizeblocks(a, b, blocks):
prev = None
Dan Villiom Podlaski Christiansen
pure bdiff: don't use a generator...
r14066 r = []
Martin Geisler
pure Python implementation of bdiff.c
r7703 for curr in blocks:
if prev is None:
prev = curr
continue
shift = 0
a1, b1, l1 = prev
a1end = a1 + l1
b1end = b1 + l1
a2, b2, l2 = curr
a2end = a2 + l2
b2end = b2 + l2
if a1end == a2:
Matt Mackall
many, many trivial check-code fixups
r10282 while (a1end + shift < a2end and
a[a1end + shift] == b[b1end + shift]):
Martin Geisler
pure Python implementation of bdiff.c
r7703 shift += 1
elif b1end == b2:
Matt Mackall
many, many trivial check-code fixups
r10282 while (b1end + shift < b2end and
a[a1end + shift] == b[b1end + shift]):
Martin Geisler
pure Python implementation of bdiff.c
r7703 shift += 1
Dan Villiom Podlaski Christiansen
pure bdiff: don't use a generator...
r14066 r.append((a1, b1, l1 + shift))
Matt Mackall
many, many trivial check-code fixups
r10282 prev = a2 + shift, b2 + shift, l2 - shift
Dan Villiom Podlaski Christiansen
pure bdiff: don't use a generator...
r14066 r.append(prev)
return r
Martin Geisler
pure Python implementation of bdiff.c
r7703
def bdiff(a, b):
a = str(a).splitlines(True)
b = str(b).splitlines(True)
if not a:
s = "".join(b)
return s and (struct.pack(">lll", 0, 0, len(s)) + s)
bin = []
p = [0]
for i in a: p.append(p[-1] + len(i))
d = difflib.SequenceMatcher(None, a, b).get_matching_blocks()
d = _normalizeblocks(a, b, d)
la = 0
lb = 0
for am, bm, size in d:
s = "".join(b[lb:bm])
if am > la or s:
bin.append(struct.pack(">lll", p[la], p[am], len(s)) + s)
la = am + size
lb = bm + size
return "".join(bin)
def blocks(a, b):
Matt Mackall
pure/bdiff: fix circular import
r7944 an = splitnewlines(a)
bn = splitnewlines(b)
Martin Geisler
pure Python implementation of bdiff.c
r7703 d = difflib.SequenceMatcher(None, an, bn).get_matching_blocks()
d = _normalizeblocks(an, bn, d)
return [(i, i + n, j, j + n) for (i, j, n) in d]
Patrick Mezard
mdiff: replace wscleanup() regexps with C loops...
r15530 def fixws(text, allws):
if allws:
text = re.sub('[ \t\r]+', '', text)
else:
text = re.sub('[ \t\r]+', ' ', text)
text = text.replace(' \n', '\n')
return text