##// END OF EJS Templates
repair: migrate revlogs during upgrade...
repair: migrate revlogs during upgrade Our next step for in-place upgrade is to migrate store data. Revlogs are the biggest source of data within the store and a store is useless without them, so we implement their migration first. Our strategy for migrating revlogs is to walk the store and call `revlog.clone()` on each revlog. There are some minor complications. Because revlogs have different storage options (e.g. changelog has generaldelta and delta chains disabled), we need to obtain the correct class of revlog so inserted data is encoded properly for its type. Various attempts at implementing progress indicators that didn't lead to frustration from false "it's almost done" indicators were made. I initially used a single progress bar based on number of revlogs. However, this quickly churned through all filelogs, got to 99% then effectively froze at 99.99% when it got to the manifest. So I converted the progress bar to total revision count. This was a little bit better. But the manifest was still significantly slower than filelogs and it took forever to process the last few percent. I then tried both revision/chunk bytes and raw bytes as the denominator. This had the opposite effect: because so much data is in manifests, it would churn through filelogs without showing much progress. When it got to manifests, it would fill in 90+% of the progress bar. I finally gave up having a unified progress bar and instead implemented 3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and 1 for changelog revisions. I added extra messages indicating the total number of revisions of each so users know there are more progress bars coming. I also added extra messages before and after each stage to give extra details about what is happening. Strictly speaking, this isn't necessary. But the numbers are impressive. For example, when converting a non-generaldelta mozilla-central repository, the messages you see are: migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog) migrating 1.67 GB in store; 2508 GB tracked data migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data) finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data) That "2508 GB" figure really blew me away. I had no clue that the raw tracked data in mozilla-central was that large. Granted, 2451 GB is in the manifest and "only" 57.3 GB is in filelogs. But still. It's worth noting that gratuitous loading of source revlogs in order to display numbers and progress bars does serve a purpose: it ensures we can open all source revlogs. We don't want to spend several minutes copying revlogs only to encounter a permissions error or similar later. As part of this commit, we also add swapping of the store directory to the upgrade function. After revlogs are converted, we move the old store into the backup directory then move the temporary repo's store into the old store's location. On well-behaved systems, this should be 2 atomic operations and the window of inconsistency show be very narrow. There are still a few improvements to be made to store copying and upgrading. But this commit gets the bulk of the work out of the way.

File last commit:

r30608:b52e8a4f default
r30779:38aa1ca9 default
Show More
registrar.py
254 lines | 7.5 KiB | text/x-python | PythonLexer
# registrar.py - utilities to register function for specific purpose
#
# Copyright FUJIWARA Katsunori <foozy@lares.dti.ne.jp> and others
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
from . import (
error,
pycompat,
util,
)
class _funcregistrarbase(object):
"""Base of decorator to register a function for specific purpose
This decorator stores decorated functions into own dict 'table'.
The least derived class can be defined by overriding 'formatdoc',
for example::
class keyword(_funcregistrarbase):
_docformat = ":%s: %s"
This should be used as below:
keyword = registrar.keyword()
@keyword('bar')
def barfunc(*args, **kwargs):
'''Explanation of bar keyword ....
'''
pass
In this case:
- 'barfunc' is stored as 'bar' in '_table' of an instance 'keyword' above
- 'barfunc.__doc__' becomes ":bar: Explanation of bar keyword"
"""
def __init__(self, table=None):
if table is None:
self._table = {}
else:
self._table = table
def __call__(self, decl, *args, **kwargs):
return lambda func: self._doregister(func, decl, *args, **kwargs)
def _doregister(self, func, decl, *args, **kwargs):
name = self._getname(decl)
if name in self._table:
msg = 'duplicate registration for name: "%s"' % name
raise error.ProgrammingError(msg)
if func.__doc__ and not util.safehasattr(func, '_origdoc'):
doc = func.__doc__.strip()
func._origdoc = doc
func.__doc__ = self._formatdoc(decl, doc)
self._table[name] = func
self._extrasetup(name, func, *args, **kwargs)
return func
def _parsefuncdecl(self, decl):
"""Parse function declaration and return the name of function in it
"""
i = decl.find('(')
if i >= 0:
return decl[:i]
else:
return decl
def _getname(self, decl):
"""Return the name of the registered function from decl
Derived class should override this, if it allows more
descriptive 'decl' string than just a name.
"""
return decl
_docformat = None
def _formatdoc(self, decl, doc):
"""Return formatted document of the registered function for help
'doc' is '__doc__.strip()' of the registered function.
"""
return self._docformat % (decl, doc)
def _extrasetup(self, name, func):
"""Execute exra setup for registered function, if needed
"""
pass
class revsetpredicate(_funcregistrarbase):
"""Decorator to register revset predicate
Usage::
revsetpredicate = registrar.revsetpredicate()
@revsetpredicate('mypredicate(arg1, arg2[, arg3])')
def mypredicatefunc(repo, subset, x):
'''Explanation of this revset predicate ....
'''
pass
The first string argument is used also in online help.
Optional argument 'safe' indicates whether a predicate is safe for
DoS attack (False by default).
Optional argument 'takeorder' indicates whether a predicate function
takes ordering policy as the last argument.
'revsetpredicate' instance in example above can be used to
decorate multiple functions.
Decorated functions are registered automatically at loading
extension, if an instance named as 'revsetpredicate' is used for
decorating in extension.
Otherwise, explicit 'revset.loadpredicate()' is needed.
"""
_getname = _funcregistrarbase._parsefuncdecl
_docformat = pycompat.sysstr("``%s``\n %s")
def _extrasetup(self, name, func, safe=False, takeorder=False):
func._safe = safe
func._takeorder = takeorder
class filesetpredicate(_funcregistrarbase):
"""Decorator to register fileset predicate
Usage::
filesetpredicate = registrar.filesetpredicate()
@filesetpredicate('mypredicate()')
def mypredicatefunc(mctx, x):
'''Explanation of this fileset predicate ....
'''
pass
The first string argument is used also in online help.
Optional argument 'callstatus' indicates whether a predicate
implies 'matchctx.status()' at runtime or not (False, by
default).
Optional argument 'callexisting' indicates whether a predicate
implies 'matchctx.existing()' at runtime or not (False, by
default).
'filesetpredicate' instance in example above can be used to
decorate multiple functions.
Decorated functions are registered automatically at loading
extension, if an instance named as 'filesetpredicate' is used for
decorating in extension.
Otherwise, explicit 'fileset.loadpredicate()' is needed.
"""
_getname = _funcregistrarbase._parsefuncdecl
_docformat = pycompat.sysstr("``%s``\n %s")
def _extrasetup(self, name, func, callstatus=False, callexisting=False):
func._callstatus = callstatus
func._callexisting = callexisting
class _templateregistrarbase(_funcregistrarbase):
"""Base of decorator to register functions as template specific one
"""
_docformat = pycompat.sysstr(":%s: %s")
class templatekeyword(_templateregistrarbase):
"""Decorator to register template keyword
Usage::
templatekeyword = registrar.templatekeyword()
@templatekeyword('mykeyword')
def mykeywordfunc(repo, ctx, templ, cache, revcache, **args):
'''Explanation of this template keyword ....
'''
pass
The first string argument is used also in online help.
'templatekeyword' instance in example above can be used to
decorate multiple functions.
Decorated functions are registered automatically at loading
extension, if an instance named as 'templatekeyword' is used for
decorating in extension.
Otherwise, explicit 'templatekw.loadkeyword()' is needed.
"""
class templatefilter(_templateregistrarbase):
"""Decorator to register template filer
Usage::
templatefilter = registrar.templatefilter()
@templatefilter('myfilter')
def myfilterfunc(text):
'''Explanation of this template filter ....
'''
pass
The first string argument is used also in online help.
'templatefilter' instance in example above can be used to
decorate multiple functions.
Decorated functions are registered automatically at loading
extension, if an instance named as 'templatefilter' is used for
decorating in extension.
Otherwise, explicit 'templatefilters.loadkeyword()' is needed.
"""
class templatefunc(_templateregistrarbase):
"""Decorator to register template function
Usage::
templatefunc = registrar.templatefunc()
@templatefunc('myfunc(arg1, arg2[, arg3])')
def myfuncfunc(context, mapping, args):
'''Explanation of this template function ....
'''
pass
The first string argument is used also in online help.
'templatefunc' instance in example above can be used to
decorate multiple functions.
Decorated functions are registered automatically at loading
extension, if an instance named as 'templatefunc' is used for
decorating in extension.
Otherwise, explicit 'templater.loadfunction()' is needed.
"""
_getname = _funcregistrarbase._parsefuncdecl