##// END OF EJS Templates
repair: migrate revlogs during upgrade...
repair: migrate revlogs during upgrade Our next step for in-place upgrade is to migrate store data. Revlogs are the biggest source of data within the store and a store is useless without them, so we implement their migration first. Our strategy for migrating revlogs is to walk the store and call `revlog.clone()` on each revlog. There are some minor complications. Because revlogs have different storage options (e.g. changelog has generaldelta and delta chains disabled), we need to obtain the correct class of revlog so inserted data is encoded properly for its type. Various attempts at implementing progress indicators that didn't lead to frustration from false "it's almost done" indicators were made. I initially used a single progress bar based on number of revlogs. However, this quickly churned through all filelogs, got to 99% then effectively froze at 99.99% when it got to the manifest. So I converted the progress bar to total revision count. This was a little bit better. But the manifest was still significantly slower than filelogs and it took forever to process the last few percent. I then tried both revision/chunk bytes and raw bytes as the denominator. This had the opposite effect: because so much data is in manifests, it would churn through filelogs without showing much progress. When it got to manifests, it would fill in 90+% of the progress bar. I finally gave up having a unified progress bar and instead implemented 3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and 1 for changelog revisions. I added extra messages indicating the total number of revisions of each so users know there are more progress bars coming. I also added extra messages before and after each stage to give extra details about what is happening. Strictly speaking, this isn't necessary. But the numbers are impressive. For example, when converting a non-generaldelta mozilla-central repository, the messages you see are: migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog) migrating 1.67 GB in store; 2508 GB tracked data migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data) finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data) That "2508 GB" figure really blew me away. I had no clue that the raw tracked data in mozilla-central was that large. Granted, 2451 GB is in the manifest and "only" 57.3 GB is in filelogs. But still. It's worth noting that gratuitous loading of source revlogs in order to display numbers and progress bars does serve a purpose: it ensures we can open all source revlogs. We don't want to spend several minutes copying revlogs only to encounter a permissions error or similar later. As part of this commit, we also add swapping of the store directory to the upgrade function. After revlogs are converted, we move the old store into the backup directory then move the temporary repo's store into the old store's location. On well-behaved systems, this should be 2 atomic operations and the window of inconsistency show be very narrow. There are still a few improvements to be made to store copying and upgrading. But this commit gets the bulk of the work out of the way.

File last commit:

r30509:add7bcad default
r30779:38aa1ca9 default
Show More
__init__.py
95 lines | 2.6 KiB | text/x-python | PythonLexer
Eric Hopper
Fixing up comment headers for split up code.
r2391 # hgweb/__init__.py - web interface to a mercurial repository
Eric Hopper
Moving hgweb.py into it's own module in preparation for breaking it up.
r2311 #
# Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net>
# Copyright 2005 Matt Mackall <mpm@selenic.com>
#
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Eric Hopper
Moving hgweb.py into it's own module in preparation for breaking it up.
r2311
Yuya Nishihara
hgweb: use absolute_import
r27046 from __future__ import absolute_import
Matt Mackall
hgweb: make hgweb.hgweb a unified interface to hgweb/hgwebdir
r10996 import os
Yuya Nishihara
hgweb: use absolute_import
r27046
Yuya Nishihara
hgweb: move httpservice object from commands module...
r27138 from ..i18n import _
from .. import (
Yuya Nishihara
hgweb: extract factory function of httpservice object...
r27139 error,
Yuya Nishihara
hgweb: move httpservice object from commands module...
r27138 util,
)
Yuya Nishihara
hgweb: use absolute_import
r27046 from . import (
hgweb_mod,
hgwebdir_mod,
Yuya Nishihara
hgweb: move httpservice object from commands module...
r27138 server,
Yuya Nishihara
hgweb: use absolute_import
r27046 )
Matt Mackall
Replace demandload with new demandimport
r3877
Matt Mackall
hgweb: make hgweb.hgweb a unified interface to hgweb/hgwebdir
r10996 def hgweb(config, name=None, baseui=None):
'''create an hgweb wsgi object
config can be one of:
- repo object (single repo view)
- path to repo (single repo view)
- path to config file (multi-repo view)
- dict of virtual:real pairs (multi-repo view)
- list of virtual:real tuples (multi-repo view)
'''
if ((isinstance(config, str) and not os.path.isdir(config)) or
isinstance(config, dict) or isinstance(config, list)):
# create a multi-dir interface
return hgwebdir_mod.hgwebdir(config, baseui=baseui)
return hgweb_mod.hgweb(config, name=name, baseui=baseui)
Matt Mackall
Replace demandload with new demandimport
r3877
Matt Mackall
hgweb: make top-level prototypes mirror their callees
r10992 def hgwebdir(config, baseui=None):
return hgwebdir_mod.hgwebdir(config, baseui=baseui)
Matt Mackall
Replace demandload with new demandimport
r3877
Yuya Nishihara
hgweb: move httpservice object from commands module...
r27138 class httpservice(object):
def __init__(self, ui, app, opts):
self.ui = ui
self.app = app
self.opts = opts
def init(self):
util.setsignalhandler()
self.httpd = server.create_server(self.ui, self.app)
if self.opts['port'] and not self.ui.verbose:
return
if self.httpd.prefix:
prefix = self.httpd.prefix.strip('/') + '/'
else:
prefix = ''
port = ':%d' % self.httpd.port
if port == ':80':
port = ''
bindaddr = self.httpd.addr
if bindaddr == '0.0.0.0':
bindaddr = '*'
elif ':' in bindaddr: # IPv6
bindaddr = '[%s]' % bindaddr
fqaddr = self.httpd.fqaddr
if ':' in fqaddr:
fqaddr = '[%s]' % fqaddr
if self.opts['port']:
write = self.ui.status
else:
write = self.ui.write
write(_('listening at http://%s%s/%s (bound to %s:%d)\n') %
(fqaddr, port, prefix, bindaddr, self.httpd.port))
self.ui.flush() # avoid buffering of status message
def run(self):
self.httpd.serve_forever()
Yuya Nishihara
hgweb: extract factory function of httpservice object...
r27139
Yuya Nishihara
hgweb: extract app factory...
r30508 def createapp(baseui, repo, webconf):
Yuya Nishihara
hgweb: eliminate duck-typing to select hgweb or hgwebdir by command option...
r27181 if webconf:
Yuya Nishihara
hgweb: extract app factory...
r30508 return hgwebdir_mod.hgwebdir(webconf, baseui=baseui)
Yuya Nishihara
hgweb: eliminate duck-typing to select hgweb or hgwebdir by command option...
r27181 else:
Yuya Nishihara
hgweb: extract factory function of httpservice object...
r27139 if not repo:
raise error.RepoError(_("there is no Mercurial repository"
" here (.hg not found)"))
Yuya Nishihara
hgweb: extract app factory...
r30508 return hgweb_mod.hgweb(repo, baseui=baseui)