##// END OF EJS Templates
exchangev2: fetch manifest revisions...
exchangev2: fetch manifest revisions Now that the server has support for retrieving manifest data, we can implement the client bits to call it. We teach the changeset fetching code to capture the manifest revisions that are encountered on incoming changesets. We then feed this into a new function which filters out known manifests and then batches up manifest data requests to the server. This is different from the previous wire protocol in a few notable ways. First, the client fetches manifest data separately and explicitly. Before, we'd ask the server for data pertaining to some changesets (via a "getbundle" command) and manifests (and files) would be sent automatically. Providing an API for looking up just manifest data separately gives clients much more flexibility for manifest management. For example, a client may choose to only fetch manifest data on demand instead of prefetching it (i.e. partial clone). Second, we send N commands to the server for manifest retrieval instead of 1. This property has a few nice side-effects. One is that the deterministic nature of the requests lends itself to server-side caching. For example, say the remote has 50,000 manifests. If the server is configured to cache responses, each time a new commit arrives, you will have a cache miss and need to regenerate all outgoing data. But if you makes N requests requesting 10,000 manifests each, a new commit will still yield cache hits on the initial, unchanged manifest batches/requests. A derived benefit from these properties is that resumable clone is conceptually simpler to implement. When making a monolithic request for all of the repository data, recovering from an interrupted clone is hard because the server was in the driver's seat and was maintaining state about all the data that needed transferred. With the client driving fetching, the client can persist the set of unfetched entities and retry/resume a fetch if something goes wrong. Or we can fetch all data N changesets at a time and slowly build up a repository. This approach is drastically easier to implement when we have server APIs exposing low-level repository primitives (such as manifests and files). We don't yet support tree manifests. But it should be possible to implement that with the existing wire protocol command. Differential Revision: https://phab.mercurial-scm.org/D4489

File last commit:

r39488:481db51c merge default
r39674:d292328e default
Show More
policy.py
109 lines | 3.6 KiB | text/x-python | PythonLexer
timeless
debuginstall: expose modulepolicy...
r29266 # policy.py - module policy logic for Mercurial.
#
# Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
import os
import sys
# Rules for how modules can be loaded. Values are:
#
# c - require C extensions
# allow - allow pure Python implementation when C loading fails
Maciej Fijalkowski
policy: add cffi policy for PyPy...
r29490 # cffi - required cffi versions (implemented within pure module)
# cffi-allow - allow pure Python implementation if cffi version is missing
timeless
debuginstall: expose modulepolicy...
r29266 # py - only load pure Python modules
#
Yuya Nishihara
policy: relax the default for in-place build...
r32251 # By default, fall back to the pure modules so the in-place build can
# run without recompiling the C extensions. This will be overridden by
# __modulepolicy__ generated by setup.py.
policy = b'allow'
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 _packageprefs = {
# policy: (versioned package, pure package)
b'c': (r'cext', None),
b'allow': (r'cext', r'pure'),
Yuya Nishihara
cffi: split modules from pure...
r32512 b'cffi': (r'cffi', None),
b'cffi-allow': (r'cffi', r'pure'),
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 b'py': (None, r'pure'),
}
Maciej Fijalkowski
policy: add cffi policy for PyPy...
r29490
timeless
debuginstall: expose modulepolicy...
r29266 try:
from . import __modulepolicy__
policy = __modulepolicy__.modulepolicy
except ImportError:
pass
# PyPy doesn't load C extensions.
#
# The canonical way to do this is to test platform.python_implementation().
# But we don't import platform and don't bloat for it here.
Yuya Nishihara
policy: mark all string literals as sysstr or bytes...
r32205 if r'__pypy__' in sys.builtin_module_names:
policy = b'cffi'
timeless
debuginstall: expose modulepolicy...
r29266
# Environment variable can always force settings.
FUJIWARA Katsunori
py3: add "b" prefix to string literals related to module policy...
r31361 if sys.version_info[0] >= 3:
Yuya Nishihara
policy: mark all string literals as sysstr or bytes...
r32205 if r'HGMODULEPOLICY' in os.environ:
policy = os.environ[r'HGMODULEPOLICY'].encode(r'utf-8')
FUJIWARA Katsunori
py3: add "b" prefix to string literals related to module policy...
r31361 else:
Yuya Nishihara
policy: mark all string literals as sysstr or bytes...
r32205 policy = os.environ.get(r'HGMODULEPOLICY', policy)
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366
def _importfrom(pkgname, modname):
# from .<pkgname> import <modname> (where . is looked through this module)
fakelocals = {}
pkg = __import__(pkgname, globals(), fakelocals, [modname], level=1)
try:
fakelocals[modname] = mod = getattr(pkg, modname)
except AttributeError:
raise ImportError(r'cannot import name %s' % modname)
# force import; fakelocals[modname] may be replaced with the real module
getattr(mod, r'__doc__', None)
return fakelocals[modname]
Jun Wu
policy: define C module versions individually...
r32428 # keep in sync with "version" in C modules
_cextversions = {
Yuya Nishihara
policy: extend API version checks for cffi...
r32511 (r'cext', r'base85'): 1,
Jun Wu
bdiff: add a xdiffblocks method...
r36693 (r'cext', r'bdiff'): 3,
Yuya Nishihara
policy: extend API version checks for cffi...
r32511 (r'cext', r'mpatch'): 1,
Augie Fackler
osutil: implement minimal __getitem__ compatibility on our custom listdir type...
r36798 (r'cext', r'osutil'): 4,
Gregory Szorc
merge with stable...
r39488 (r'cext', r'parsers'): 11,
Jun Wu
policy: define C module versions individually...
r32428 }
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 # map import request to other package or module
_modredirects = {
Yuya Nishihara
encoding: drop circular import by proxying through '<policy>.charencode'...
r33756 (r'cext', r'charencode'): (r'cext', r'parsers'),
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 (r'cffi', r'base85'): (r'pure', r'base85'),
Yuya Nishihara
encoding: drop circular import by proxying through '<policy>.charencode'...
r33756 (r'cffi', r'charencode'): (r'pure', r'charencode'),
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 (r'cffi', r'parsers'): (r'pure', r'parsers'),
}
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 def _checkmod(pkgname, modname, mod):
Yuya Nishihara
policy: extend API version checks for cffi...
r32511 expected = _cextversions.get((pkgname, modname))
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 actual = getattr(mod, r'version', None)
if actual != expected:
raise ImportError(r'cannot import module %s.%s '
r'(expected version: %d, actual: %r)'
% (pkgname, modname, expected, actual))
def importmod(modname):
"""Import module according to policy and check API version"""
try:
verpkg, purepkg = _packageprefs[policy]
except KeyError:
raise ImportError(r'invalid HGMODULEPOLICY %r' % policy)
assert verpkg or purepkg
if verpkg:
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 pn, mn = _modredirects.get((verpkg, modname), (verpkg, modname))
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 try:
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 mod = _importfrom(pn, mn)
if pn == verpkg:
_checkmod(pn, mn, mod)
Yuya Nishihara
policy: add helper to import cext/pure module...
r32366 return mod
except ImportError:
if not purepkg:
raise
Yuya Nishihara
policy: reroute proxy modules internally...
r33755 pn, mn = _modredirects.get((purepkg, modname), (purepkg, modname))
return _importfrom(pn, mn)