##// END OF EJS Templates
tests: add test extension implementing custom filelog storage...
tests: add test extension implementing custom filelog storage In order to better support partial clones, we'll need alternate repository storage mechanisms that aren't based on revlogs. Today, the interface for repository storage isn't very well defined. And there are various layering violations and assumptions made throughout the code that storage is backed by revlogs. In order to support alternate storage mechanisms, we'll need to formally declare and adhere to interfaces for storage. This will be a long, arduous process. This commit creates an extension that implements non-revlog storage for files. It defines a custom type that quacks like the existing revlog/filelog API but isn't backed by a revlog. The backing storage is - for simplicity reasons - a CBOR index and per-node files representing fulltext data. The localrepository class is modified so file(f) returns instances of this class instead of filelog instances. The purpose of this extension is to tease out what the actual filelog interface is - based on running the test harness - so we can formalize that interface and then implement a *real* alternate storage backend. Using `run-tests.py --extra-config-opt` to run the test harness with this extension enabled yields 83 failures out of 634 ran tests. The most common test failures are due to: * Issues with `hg verify` * LFS and largefiles (probably flags processing related) * Narrow. * Any test touching or inspecting individual filelog paths. * help and error output that is confused by the presence of an extension. * `hg debug*` commands doing low-level, revlog-y things. An 88% pass rate is pretty good for an initial implementation if you ask me! There is a bit of duplicate code in the new extension. That's by design: a point of this code is to tease out dependencies on revlog. That being said, there is opportunity to consolidate code by moving things out of the revlog API. For example, DAG traversal operations don't necessarily need to be implemented at the storage level. (Although for performance reasons they probably do.) Once we have a more well-defined interface, we could probably define the default implementations in terms of the base interface, pull those in via class inheritance, and have implementations override with faster versions if they so choose. (Or something like that.) But for now, the duplicate code should be acceptable. Differential Revision: https://phab.mercurial-scm.org/D3029

File last commit:

r30180:736f92c4 default
r37355:d257c5f2 default
Show More
localstore.py
68 lines | 2.4 KiB | text/x-python | PythonLexer
various
hgext: add largefiles extension...
r15168 # Copyright 2009-2010 Gregory P. Ward
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
# Copyright 2010-2011 Fog Creek Software
# Copyright 2010-2011 Unity Technologies
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
Greg Ward
largefiles: improve comments, internal docstrings...
r15252 '''store class for local filesystem'''
liscju
py3: make largefiles/localstore.py use absolute_import
r29310 from __future__ import absolute_import
various
hgext: add largefiles extension...
r15168
from mercurial.i18n import _
Mads Kiilerich
largefiles: always use filechunkiter when iterating files...
r30180 from mercurial import util
various
hgext: add largefiles extension...
r15168
liscju
py3: make largefiles/localstore.py use absolute_import
r29310 from . import (
basestore,
lfutil,
)
various
hgext: add largefiles extension...
r15168
class localstore(basestore.basestore):
Benjamin Pollack
largefiles: make the store primary, and the user cache secondary...
r15317 '''localstore first attempts to grab files out of the store in the remote
Mads Kiilerich
fix trivial spelling errors
r17424 Mercurial repository. Failing that, it attempts to grab the files from
Benjamin Pollack
largefiles: make the store primary, and the user cache secondary...
r15317 the user cache.'''
various
hgext: add largefiles extension...
r15168
def __init__(self, ui, repo, remote):
Sune Foldager
peer: introduce peer methods to prepare for peer classes...
r17191 self.remote = remote.local()
Mads Kiilerich
largefiles: cleanup of warnings on errors getting largefiles...
r18155 super(localstore, self).__init__(ui, repo, self.remote.url())
various
hgext: add largefiles extension...
r15168
Benjamin Pollack
largefiles: make the store primary, and the user cache secondary...
r15317 def put(self, source, hash):
if lfutil.instore(self.remote, hash):
return
Mads Kiilerich
largefiles: 'put' should store 'source' file in under 'hash', also in localstore
r19007 lfutil.link(source, lfutil.storepath(self.remote, hash))
various
hgext: add largefiles extension...
r15168
Matt Harbison
largefiles: adjust localstore to handle batch statlfile requests (issue3583)...
r17411 def exists(self, hashes):
retval = {}
for hash in hashes:
retval[hash] = lfutil.instore(self.remote, hash)
return retval
various
hgext: add largefiles extension...
r15168 def _getfile(self, tmpfile, filename, hash):
Mads Kiilerich
largefiles: refactoring - use findfile in localstore._getfile
r19000 path = lfutil.findfile(self.remote, hash)
if not path:
Mads Kiilerich
largefiles: cleanup of warnings on errors getting largefiles...
r18155 raise basestore.StoreError(filename, hash, self.url,
Martin Geisler
largefiles: lowercase messages
r16928 _("can't get file locally"))
Bryan O'Sullivan
largefiles: use a context manager in _getfile
r27769 with open(path, 'rb') as fd:
Mads Kiilerich
largefiles: always use filechunkiter when iterating files...
r30180 return lfutil.copyandhash(
util.filechunkiter(fd), tmpfile)
various
hgext: add largefiles extension...
r15168
liscju
largefiles: change basestore._verifyfile to take list of files to check...
r29067 def _verifyfiles(self, contents, filestocheck):
failed = False
for cset, filename, expectedhash in filestocheck:
liscju
largefiles: check file in the repo store before checking remotely (issue5257)...
r29421 storepath, exists = lfutil.findstorepath(self.repo, expectedhash)
if not exists:
storepath, exists = lfutil.findstorepath(
self.remote, expectedhash)
liscju
largefiles: change basestore._verifyfile to take list of files to check...
r29067 if not exists:
various
hgext: add largefiles extension...
r15168 self.ui.warn(
liscju
largefiles: change basestore._verifyfile to take list of files to check...
r29067 _('changeset %s: %s references missing %s\n')
Mads Kiilerich
largefiles: report localstore errors with single line warnings messages...
r18545 % (cset, filename, storepath))
liscju
largefiles: change basestore._verifyfile to take list of files to check...
r29067 failed = True
elif contents:
actualhash = lfutil.hashfile(storepath)
if actualhash != expectedhash:
self.ui.warn(
_('changeset %s: %s references corrupted %s\n')
% (cset, filename, storepath))
failed = True
return failed