##// END OF EJS Templates
compression: introduce a `storage.revlog.zlib.level` configuration...
compression: introduce a `storage.revlog.zlib.level` configuration This option control the zlib compression level used when compression revlog chunk. This is also a good excuse to pave the way for a similar configuration option for the zstd compression engine. Having a dedicated option for each compression algorithm is useful because they don't support the same range of values. Using a higher zlib compression impact CPU consumption at compression time, but does not directly affected decompression time. However dealing with small compressed chunk can directly help decompression and indirectly help other revlog logic. I ran some basic test on repositories using different level. I am using the mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite. All tested repository use sparse-revlog and got all their delta recomputed. The different compression level has a small effect on the repository size (about 10% variation in the total range). My quick analysis is that revlog mostly store small delta, that are not affected by the compression level much. So the variation probably mostly comes from better compression of the snapshots revisions, and snapshot revision only represent a small portion of the repository content. I also made some basic timings measurements. The "read" timings are gathered using simple run of `hg perfrevlogrevisions`, the "write" timings using `hg perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and mozilla central). The timings are gathered on a generic machine, (not one of our performance locked machine), so small variation might not be meaningful. However large trend remains relevant. Keep in mind that these numbers are not pure compression/decompression time. They also involve the full revlog logic. In particular the difference in chunk size has an impact on the delta chain structure, affecting performance when writing or reading them. On read/write performance, the compression level has a bigger impact. Counter-intuitively, the higher compression levels improve "write" performance for the large repositories in our tested setting. Maybe because the last 5000 delta chain end up having a very different shape in this specific spot? Or maybe because of a more general trend of better delta chains thanks to the smaller chunk and snapshot. This series does not intend to change the default compression level. However, these result call for a deeper analysis of this performance difference in the future. Full data ========= repo level .hg/store size 00manifest.d read write ---------------------------------------------------------------- mercurial 1 49,402,813 5,963,475 0.170159 53.250304 mercurial 6 47,197,397 5,875,730 0.182820 56.264320 mercurial 9 47,121,596 5,849,781 0.189219 56.293612 pypy 1 370,830,572 28,462,425 2.679217 460.721984 pypy 6 340,112,317 27,648,747 2.768691 467.537158 pypy 9 338,360,736 27,639,003 2.763495 476.589918 netbeans 1 1,281,847,810 165,495,457 122.477027 520.560316 netbeans 6 1,205,284,353 159,161,207 139.876147 715.930400 netbeans 9 1,197,135,671 155,034,586 141.620281 678.297064 mozilla 1 2,775,497,186 298,527,987 147.867662 751.263721 mozilla 6 2,596,856,420 286,597,671 170.572118 987.056093 mozilla 9 2,587,542,494 287,018,264 163.622338 739.803002

File last commit:

r40329:c303d65d default
r42210:1fac9b93 default
Show More
censor.py
100 lines | 3.6 KiB | text/x-python | PythonLexer
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 # Copyright (C) 2015 - Mike Edgar <adgar@google.com>
#
# This extension enables removal of file content at a given revision,
# rewriting the data/metadata of successive revisions to preserve revision log
# integrity.
"""erase file content at a given revision
The censor command instructs Mercurial to erase all content of a file at a given
revision *without updating the changeset hash.* This allows existing history to
remain valid while preventing future clones/pulls from receiving the erased
data.
Typical uses for censor are due to security or legal requirements, including::
Mads Kiilerich
spelling: trivial spell checking
r26781 * Passwords, private keys, cryptographic material
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 * Licensed data/code/libraries for which the license has expired
* Personally Identifiable Information or other private data
Censored nodes can interrupt mercurial's typical operation whenever the excised
data needs to be materialized. Some commands, like ``hg cat``/``hg revert``,
simply fail when asked to produce censored data. Others, like ``hg verify`` and
``hg update``, must be capable of tolerating censored data to continue to
function in a meaningful way. Such commands only tolerate censored file
FUJIWARA Katsunori
censor: fix incorrect configuration name for ignoring error at censored file...
r24890 revisions if they are allowed by the "censor.policy=ignore" config option.
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 """
Gregory Szorc
censor: use absolute_import
r28092 from __future__ import absolute_import
from mercurial.i18n import _
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 from mercurial.node import short
Gregory Szorc
censor: use absolute_import
r28092
from mercurial import (
error,
Yuya Nishihara
registrar: move cmdutil.command to registrar module (API)...
r32337 registrar,
Gregory Szorc
censor: use absolute_import
r28092 scmutil,
)
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
cmdtable = {}
Yuya Nishihara
registrar: move cmdutil.command to registrar module (API)...
r32337 command = registrar.command(cmdtable)
Augie Fackler
extensions: change magic "shipped with hg" string...
r29841 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
Augie Fackler
extensions: document that `testedwith = 'internal'` is special...
r25186 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
Augie Fackler
extensions: change magic "shipped with hg" string...
r29841 testedwith = 'ships-with-hg-core'
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
@command('censor',
[('r', 'rev', '', _('censor file from specified revision'), _('REV')),
('t', 'tombstone', '', _('replacement tombstone data'), _('TEXT'))],
rdamazio@google.com
help: assigning categories to existing commands...
r40329 _('-r REV [-t TEXT] [FILE]'),
helpcategory=command.CATEGORY_MAINTENANCE)
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 def censor(ui, repo, path, rev='', tombstone='', **opts):
Matt Harbison
censor: use context manager for lock management
r38460 with repo.wlock(), repo.lock():
FUJIWARA Katsunori
censor: make censor acquire locks before processing...
r27290 return _docensor(ui, repo, path, rev, tombstone, **opts)
def _docensor(ui, repo, path, rev='', tombstone='', **opts):
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 if not path:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('must specify file path to censor'))
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 if not rev:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('must specify revision to censor'))
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
FUJIWARA Katsunori
censor: make various path forms available like other Mercurial commands...
r25806 wctx = repo[None]
m = scmutil.match(wctx, (path,))
if m.anypats() or len(m.files()) != 1:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('can only specify an explicit filename'))
FUJIWARA Katsunori
censor: make various path forms available like other Mercurial commands...
r25806 path = m.files()[0]
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 flog = repo.file(path)
if not len(flog):
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('cannot censor file with no history'))
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
rev = scmutil.revsingle(repo, rev, rev).rev()
try:
ctx = repo[rev]
except KeyError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('invalid revision identifier %s') % rev)
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
try:
fctx = ctx.filectx(path)
except error.LookupError:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('file does not exist at revision %s') % rev)
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347
fnode = fctx.filenode()
Valentin Gatien-Baron
censor: use a reasonable amount of memory...
r39651 heads = []
for headnode in repo.heads():
Yuya Nishihara
censor: rename loop variable to silence pyflakes warning...
r39697 hc = repo[headnode]
if path in hc and hc.filenode(path) == fnode:
heads.append(hc)
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 if heads:
headlist = ', '.join([short(c.node()) for c in heads])
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('cannot censor file in heads (%s)') % headlist,
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 hint=_('clean/delete and commit first'))
wp = wctx.parents()
if ctx.node() in [p.node() for p in wp]:
Pierre-Yves David
error: get Abort from 'error' instead of 'util'...
r26587 raise error.Abort(_('cannot censor working directory'),
Mike Edgar
censor: add censor command to hgext with basic client-side tests...
r24347 hint=_('clean/delete/update first'))
Gregory Szorc
revlog: move censor logic out of censor extension...
r39814 with repo.transaction(b'censor') as tr:
flog.censorrevision(tr, fnode, tombstone=tombstone)