##// END OF EJS Templates
parsers: inline fields of dirstate values in C version...
parsers: inline fields of dirstate values in C version Previously, while unpacking the dirstate we'd create 3-4 new CPython objects for most dirstate values: - the state is a single character string, which is pooled by CPython - the mode is a new object if it isn't 0 due to being in the lookup set - the size is a new object if it is greater than 255 - the mtime is a new object if it isn't -1 due to being in the lookup set - the tuple to contain them all In some cases such as regular hg status, we actually look at all the objects. In other cases like hg add, hg status for a subdirectory, or hg status with the third-party hgwatchman enabled, we look at almost none of the objects. This patch eliminates most object creation in these cases by defining a custom C struct that is exposed to Python with an interface similar to a tuple. Only when tuple elements are actually requested are the respective objects created. The gains, where they're expected, are significant. The following tests are run against a working copy with over 270,000 files. parse_dirstate becomes significantly faster: $ hg perfdirstate before: wall 0.186437 comb 0.180000 user 0.160000 sys 0.020000 (best of 35) after: wall 0.093158 comb 0.100000 user 0.090000 sys 0.010000 (best of 95) and as a result, several commands benefit: $ time hg status # with hgwatchman enabled before: 0.42s user 0.14s system 99% cpu 0.563 total after: 0.34s user 0.12s system 99% cpu 0.471 total $ time hg add new-file before: 0.85s user 0.18s system 99% cpu 1.033 total after: 0.76s user 0.17s system 99% cpu 0.931 total There is a slight regression in regular status performance, but this is fixed in an upcoming patch.

File last commit:

r16598:20a9d823 stable
r21809:e250b830 default
Show More
base85.py
75 lines | 1.9 KiB | text/x-python | PythonLexer
# base85.py: pure python base85 codec
#
# Copyright (C) 2009 Brendan Cully <brendan@kublai.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
import struct
_b85chars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" \
"abcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~"
_b85chars2 = [(a + b) for a in _b85chars for b in _b85chars]
_b85dec = {}
def _mkb85dec():
for i, c in enumerate(_b85chars):
_b85dec[c] = i
def b85encode(text, pad=False):
"""encode text in base85 format"""
l = len(text)
r = l % 4
if r:
text += '\0' * (4 - r)
longs = len(text) >> 2
words = struct.unpack('>%dL' % (longs), text)
out = ''.join(_b85chars[(word // 52200625) % 85] +
_b85chars2[(word // 7225) % 7225] +
_b85chars2[word % 7225]
for word in words)
if pad:
return out
# Trim padding
olen = l % 4
if olen:
olen += 1
olen += l // 4 * 5
return out[:olen]
def b85decode(text):
"""decode base85-encoded text"""
if not _b85dec:
_mkb85dec()
l = len(text)
out = []
for i in range(0, len(text), 5):
chunk = text[i:i + 5]
acc = 0
for j, c in enumerate(chunk):
try:
acc = acc * 85 + _b85dec[c]
except KeyError:
raise ValueError('bad base85 character at position %d'
% (i + j))
if acc > 4294967295:
raise ValueError('Base85 overflow in hunk starting at byte %d' % i)
out.append(acc)
# Pad final chunk if necessary
cl = l % 5
if cl:
acc *= 85 ** (5 - cl)
if cl > 1:
acc += 0xffffff >> (cl - 2) * 8
out[-1] = acc
out = struct.pack('>%dL' % (len(out)), *out)
if cl:
out = out[:-(5 - cl)]
return out