##// END OF EJS Templates
changegroup: remove reordering control (BC)...
changegroup: remove reordering control (BC) This logic - including the experimental bundle.reorder option - was originally added in a8e3931e3fb5 in 2011 and then later ported to changegroup.py. The intent of this option and associated logic is to control the ordering of revisions in deltagroups in changegroups. At the time it was implemented, only changegroup version 1 existed and generaldelta revlogs were just coming into the world. Changegroup version 1 requires that deltas be made against the last revision sent over the wire. Used with generaldelta, this created an impedance mismatch of sorts and resulted in changegroup producers spending a lot of time recomputing deltas. Revision reordering was introduced so outgoing revisions would be sent in "generaldelta order" and producers would be able to reuse internal deltas from storage. Later on, we introduced changegroup version 2. It supported denoting which revision a delta was against. So we no longer needed to sort outgoing revisions to ensure optimal delta generation from the producer. So, subsequent changegroup versions disabled reordering. We also later made the changelog not store deltas by default. And we also made the changelog send out deltas in storage order. Why we do this for changelog, I'm not sure. Maybe we want to preserve revision order across clones? It doesn't really matter for this commit. Fast forward to 2018. We want to abstract storage backends. And having changegroup code require knowledge about how deltas are stored internally interferes with that goal. This commit removes reordering control from changegroup generation. After this commit, the reordering behavior is: * The changelog is always sent out in storage order (no behavior change). * Non-changelog generaldelta revlogs are reordered to always be in DAG topological order (previously, generaldelta revlogs would be emitted in storage order for version 2 and 3 changegroups). * Non-changelog non-generaldelta revlogs are sent in storage order (no behavior change). * There exists no config option to override behavior. The big difference here is that generaldelta revlogs now *always* have their revisions sorted in DAG order before going out over the wire. This behavior was previously only done for changegroup version 1. Version 2 and version 3 changegroups disabled reordering because the interchange format supported encoding arbitrary delta parents, so reordering wasn't strictly necessary. I can think of a few significant implications for this change. Because changegroup receivers will now see non-changelog revisions in DAG order instead of storage order, the internal storage order of manifests and files may differ substantially between producer and consumer. I don't think this matters that much, since the storage order of manifests and files is largely hidden from users. Only the storage order of changelog matters (because `hg log` shows the changelog in storage order). I don't think there should be any controversy here. The reordering of revisions has implications for changegroup producers. Previously, generaldelta revlogs would be emitted in storage order. And in the common case, the internally-stored delta could effectively be copied from disk into the deltagroup delta. This meant that emitting delta groups for generaldelta revlogs would be mostly linear read I/O. This is desirable for performance. With us now reordering generaldelta revlog revisions in DAG order, the read operations may use more random I/O instead of sequential I/O. This could result in performance loss. But with the prevalence of SSDs and fast random I/O, I'm not too worried. (Note: the optimal emission order for revlogs is actually delta encoding order. But the changegroup code wasn't doing that before or after this change. We could potentially implement that in a later commit.) Changegroups in DAG order will have implications for receivers. Previously, receiving storage order might mean seeing a number of interleaved branches. This would mean long delta chains, sparse I/O, and possibly more fulltext revisions instead of deltas, blowing up storage storage. (This is the same set of problems that sparse revlogs aims to address.) With the producer now sending revisions in DAG order, the receiver also stores revisions in DAG order. That means revisions for the same DAG branch are all grouped together. And this should yield better storage outcomes. In other words, sending the reordered changegroup allows the receiver to have better storage order and for the producer to not propagate its (possibly sub-optimal) internal storage order. On the mozilla-unified repository, this change influences bundle generation: $ hg bundle -t none-v2 -a before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000) after: time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000) before: 7,150,228,967 bytes (uncompressed) after: 7,041,556,273 bytes (uncompressed) before: 1,669,063,234 bytes (zstd l=3) after: 1,628,598,830 bytes (zstd l=3) $ hg unbundle before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000) after: time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000) 00manifest.d size: source: 274,924,292 bytes before: 304,741,626 bytes after: 245,252,087 bytes .hg/store total file size: source: 2,649,133,490 before: 2,680,888,130 after: 2,627,875,673 We see the bundle size drop. That's probably because if a revlog internally isn't storing a delta, it will choose to delta against the last emitted revision. And on repos with interleaved branches (like mozilla-unified), the previous revision could be an unrelated branch and therefore be a large delta. But with this patch, the previous revision is likely p1 or p2 and a delta should be small. We also see the manifest size drop by ~50 MB. It's worth noting that the manifest actually *increased* in size by ~25 MB in the old strategy and decreased ~25 MB from its source in the new strategy. Again, my explanation for this is that the DAG ordering in the changegroup is resulting in better grouping of revisions in the receiver, which results in more compact delta chains and higher storage efficiency. Unbundle time also dropped. I suspect this is due to the revlog having to work less to compute deltas since the incoming deltas are more optimal. i.e. the receiver spends less time resolving fulltext revisions as incoming deltas bounce around between DAG branches and delta chains. We also see bundle generation time increase. This is not desirable. However, the regression is only significant on the original repository: if we generate a bundle from the repository created from the new, always reordered bundles, we're close to baseline (if not at it with expected noise): $ hg bundle -t none-v2 -a before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000) after (original): time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000) after (new repo): time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000) This regression is a bit worrying because it will impact serving canonical repositories (that don't have optimal internal storage unless they are reordered - possibly as part of running `hg debugupgraderepo`). However, this regression will only be noticed by very large changegroups. And I'm guessing/hoping that any repository that large is using clonebundles to mitigate server load. Again, sending DAG order isn't the optimal send order for servers: sending in storage-delta order is. But in order to enable storage-optimal send order, we'll need a storage API that handles sorting. Future commits will introduce such an API. Differential Revision: https://phab.mercurial-scm.org/D4721

File last commit:

r37752:a1110db1 default
r39897:db5501d9 default
Show More
common.py
261 lines | 8.7 KiB | text/x-python | PythonLexer
Eric Hopper
Fixing up comment headers for split up code.
r2391 # hgweb/common.py - Utility functions needed by hgweb_mod and hgwebdir_mod
Eric Hopper
Final stage of the hgweb split up....
r2356 #
# Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net>
Vadim Gelfer
update copyrights.
r2859 # Copyright 2005, 2006 Matt Mackall <mpm@selenic.com>
Eric Hopper
Final stage of the hgweb split up....
r2356 #
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Eric Hopper
Final stage of the hgweb split up....
r2356
Yuya Nishihara
hgweb: use absolute_import
r27046 from __future__ import absolute_import
Gregory Szorc
hgweb: support Content Security Policy...
r30766 import base64
Yuya Nishihara
hgweb: use absolute_import
r27046 import errno
import mimetypes
import os
Augie Fackler
cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime...
r36799 import stat
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561
Pulkit Goyal
py3: replace os.sep with pycompat.ossep (part 3 of 4)
r30615 from .. import (
Pulkit Goyal
py3: replace os.environ with encoding.environ (part 3 of 5)
r30636 encoding,
Pulkit Goyal
py3: replace os.sep with pycompat.ossep (part 3 of 4)
r30615 pycompat,
util,
)
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566
httpserver = util.httpserver
Dirkjan Ochtman
hgweb: explicit response status
r5993 HTTP_OK = 200
Matt Harbison
lfs: add support for serving blob files
r37167 HTTP_CREATED = 201
Dirkjan Ochtman
hgweb: support very simple caching model (issue1845)
r12183 HTTP_NOT_MODIFIED = 304
Dirkjan Ochtman
hgweb: explicit response status
r5993 HTTP_BAD_REQUEST = 400
Dirkjan Ochtman
hgweb: raise ErrorResponses to communicate protocol errors
r6926 HTTP_UNAUTHORIZED = 401
Rocco Rutte
hgweb: Respond with HTTP 403 for disabled archive types instead of 404...
r7029 HTTP_FORBIDDEN = 403
Dirkjan Ochtman
hgweb: explicit response status
r5993 HTTP_NOT_FOUND = 404
Dirkjan Ochtman
hgweb: raise ErrorResponses to communicate protocol errors
r6926 HTTP_METHOD_NOT_ALLOWED = 405
Matt Harbison
lfs: update the HTTP status codes in error cases...
r37711 HTTP_NOT_ACCEPTABLE = 406
HTTP_UNSUPPORTED_MEDIA_TYPE = 415
Dirkjan Ochtman
hgweb: explicit response status
r5993 HTTP_SERVER_ERROR = 500
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 def ismember(ui, username, userlist):
"""Check if username is a member of userlist.
If userlist has a single '*' member, all users are considered members.
Mads Kiilerich
spelling: random spell checker fixes
r19951 Can be overridden by extensions to provide more complex authorization
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 schemes.
"""
return userlist == ['*'] or username in userlist
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 def checkauthz(hgweb, req, op):
'''Check permission for operation based on request data (including
authentication info). Return if op allowed, else raise an ErrorResponse
exception.'''
Gregory Szorc
hgweb: transition permissions hooks to modern request type (API)...
r36893 user = req.remoteuser
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910
deny_read = hgweb.configlist('web', 'deny_read')
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 if deny_read and (not user or ismember(hgweb.repo.ui, user, deny_read)):
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 raise ErrorResponse(HTTP_UNAUTHORIZED, 'read not authorized')
allow_read = hgweb.configlist('web', 'allow_read')
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 if allow_read and (not ismember(hgweb.repo.ui, user, allow_read)):
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 raise ErrorResponse(HTTP_UNAUTHORIZED, 'read not authorized')
if op == 'pull' and not hgweb.allowpull:
raise ErrorResponse(HTTP_UNAUTHORIZED, 'pull not authorized')
elif op == 'pull' or op is None: # op is None for interface requests
return
Matt Harbison
lfs: add basic routing for the server side wire protocol processing...
r37165 # Allow LFS uploading via PUT requests
if op == 'upload':
if req.method != 'PUT':
msg = 'upload requires PUT request'
raise ErrorResponse(HTTP_METHOD_NOT_ALLOWED, msg)
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 # enforce that you can only push using POST requests
Matt Harbison
lfs: add basic routing for the server side wire protocol processing...
r37165 elif req.method != 'POST':
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 msg = 'push requires POST request'
raise ErrorResponse(HTTP_METHOD_NOT_ALLOWED, msg)
# require ssl by default for pushing, auth info cannot be sniffed
# and replayed
Gregory Szorc
hgweb: transition permissions hooks to modern request type (API)...
r36893 if hgweb.configbool('web', 'push_ssl') and req.urlscheme != 'https':
Yuya Nishihara
hgweb: respond 403 forbidden for ssl required error...
r17456 raise ErrorResponse(HTTP_FORBIDDEN, 'ssl required')
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910
deny = hgweb.configlist('web', 'deny_push')
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 if deny and (not user or ismember(hgweb.repo.ui, user, deny)):
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 raise ErrorResponse(HTTP_UNAUTHORIZED, 'push not authorized')
David Demelier
config: rename allow_push to allow-push...
r35029 allow = hgweb.configlist('web', 'allow-push')
Wagner Bruna
hgweb: refactor checks for granting and revoking user permissions...
r19032 if not (allow and ismember(hgweb.repo.ui, user, allow)):
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910 raise ErrorResponse(HTTP_UNAUTHORIZED, 'push not authorized')
Martin Geisler
hgweb: initialize permhooks at definition time...
r14058 # Hooks for hgweb permission checks; extensions can add hooks here.
# Each hook is invoked like this: hook(hgweb, request, operation),
Matt Harbison
lfs: add basic routing for the server side wire protocol processing...
r37165 # where operation is either read, pull, push or upload. Hooks should either
Martin Geisler
hgweb: initialize permhooks at definition time...
r14058 # raise an ErrorResponse exception, or just return.
#
# It is possible to do both authentication and authorization through
# this.
permhooks = [checkauthz]
Sune Foldager
hgweb: add support for extension-provided permission hooks...
r9910
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561 class ErrorResponse(Exception):
Gregory Szorc
hgweb: don't use mutable default argument value
r31390 def __init__(self, code, message=None, headers=None):
Mads Kiilerich
hgweb: give ErrorResponse a descriptive string/Exception representation...
r13444 if message is None:
message = _statusmessage(code)
Augie Fackler
hgweb: pass exception message to builtin Exception ctor as sysstr...
r36447 Exception.__init__(self, pycompat.sysstr(message))
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561 self.code = code
Pierre-Yves David
hgweb: explicitly tests for None...
r31435 if headers is None:
headers = []
self.headers = headers
Bryan O'Sullivan
hgweb: fix breaking tests on Python < 2.5
r5563
Augie Fackler
hgweb: add support for 100-continue as recommended by PEP 333.
r13570 class continuereader(object):
Gregory Szorc
hgweb: document continuereader...
r36869 """File object wrapper to handle HTTP 100-continue.
This is used by servers so they automatically handle Expect: 100-continue
request headers. On first read of the request body, the 100 Continue
response is sent. This should trigger the client into actually sending
the request body.
"""
Augie Fackler
hgweb: add support for 100-continue as recommended by PEP 333.
r13570 def __init__(self, f, write):
self.f = f
self._write = write
self.continued = False
def read(self, amt=-1):
if not self.continued:
self.continued = True
self._write('HTTP/1.1 100 Continue\r\n\r\n')
return self.f.read(amt)
def __getattr__(self, attr):
if attr in ('close', 'readline', 'readlines', '__iter__'):
return getattr(self.f, attr)
Brodie Rao
cleanup: "raise SomeException()" -> "raise SomeException"
r16687 raise AttributeError
Bryan O'Sullivan
hgweb: fix breaking tests on Python < 2.5
r5563
def _statusmessage(code):
Pulkit Goyal
py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import...
r29566 responses = httpserver.basehttprequesthandler.responses
Augie Fackler
hgweb: these strings should be sysstrs, not bytes...
r37752 return pycompat.bytesurl(
responses.get(code, (r'Error', r'Unknown error'))[0])
Thomas Arendsen Hein
Removed tabs and trailing whitespace in python files
r5760
Sune Foldager
hgweb: send proper error messages to the client...
r9694 def statusmessage(code, message=None):
return '%d %s' % (code, message or _statusmessage(code))
Eric Hopper
Final stage of the hgweb split up....
r2356
Pierre-Yves David
hgweb: drop the default argument for get_stat...
r25717 def get_stat(spath, fn):
"""stat fn if it exists, spath otherwise"""
Anton Shestakov
hgweb: refresh hgweb.repo on phase change (issue4061)...
r22577 cl_path = os.path.join(spath, fn)
Benoit Boissinot
switch to the .hg/store layout, fix the tests
r3853 if os.path.exists(cl_path):
Martin Geisler
hgweb: detect change based on changelog size too...
r13958 return os.stat(cl_path)
Eric Hopper
Final stage of the hgweb split up....
r2356 else:
Martin Geisler
hgweb: detect change based on changelog size too...
r13958 return os.stat(spath)
def get_mtime(spath):
Augie Fackler
cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime...
r36799 return get_stat(spath, "00changelog.i")[stat.ST_MTIME]
Eric Hopper
Final stage of the hgweb split up....
r2356
Gregory Szorc
hgweb: extract path traversal checking into standalone function...
r31790 def ispathsafe(path):
"""Determine if a path is safe to use for filesystem access."""
parts = path.split('/')
for part in parts:
Yuya Nishihara
py3: make os.curdir a bytes
r36666 if (part in ('', pycompat.oscurdir, pycompat.ospardir) or
Gregory Szorc
hgweb: extract path traversal checking into standalone function...
r31790 pycompat.ossep in part or
pycompat.osaltsep is not None and pycompat.osaltsep in part):
return False
return True
Gregory Szorc
hgweb: port static file handling to new response API...
r36889 def staticfile(directory, fname, res):
Dirkjan Ochtman
send conservatively capitalized HTTP headers
r5930 """return a file inside directory with guessed Content-Type header
Eric Hopper
Final stage of the hgweb split up....
r2356
fname always uses '/' as directory separator and isn't allowed to
contain unusual path components.
Dirkjan Ochtman
send conservatively capitalized HTTP headers
r5930 Content-Type is guessed using the mimetypes module.
Eric Hopper
Final stage of the hgweb split up....
r2356 Return an empty string if fname is illegal or file not found.
"""
Gregory Szorc
hgweb: extract path traversal checking into standalone function...
r31790 if not ispathsafe(fname):
return
fpath = os.path.join(*fname.split('/'))
Brendan Cully
Allow per-file shadowing of static directory in templatepath
r7288 if isinstance(directory, str):
directory = [directory]
for d in directory:
path = os.path.join(d, fpath)
if os.path.exists(path):
break
Eric Hopper
Final stage of the hgweb split up....
r2356 try:
os.stat(path)
Augie Fackler
hgweb: mimetype guessing needs a unicode path...
r34722 ct = mimetypes.guess_type(pycompat.fsdecode(path))[0] or "text/plain"
Gregory Szorc
hgweb: use context manager for file I/O
r31789 with open(path, 'rb') as fh:
data = fh.read()
Gregory Szorc
hgweb: port static file handling to new response API...
r36889 res.headers['Content-Type'] = ct
res.setbodybytes(data)
return res
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561 except TypeError:
timeless
Generally replace "file name" with "filename" in help and comments.
r8761 raise ErrorResponse(HTTP_SERVER_ERROR, 'illegal filename')
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except OSError as err:
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561 if err.errno == errno.ENOENT:
Dirkjan Ochtman
hgweb: explicit response status
r5993 raise ErrorResponse(HTTP_NOT_FOUND)
Bryan O'Sullivan
hgweb: return meaningful HTTP status codes instead of nonsense
r5561 else:
Augie Fackler
python3: wrap all uses of <exception>.strerror with strtolocal...
r34024 raise ErrorResponse(HTTP_SERVER_ERROR,
encoding.strtolocal(err.strerror))
Thomas Arendsen Hein
hgweb: Search templates in templatepath/style/map, too, using a common function....
r3276
Thomas Arendsen Hein
hgweb: use generator to count parity of horizontal stripes for easier reading....
r4462 def paritygen(stripecount, offset=0):
"""count parity of horizontal stripes for easier reading"""
if stripecount and offset:
# account for offset, e.g. due to building the list in reverse
count = (stripecount + offset) % stripecount
Pulkit Goyal
py3: use '//' for integer division in hgweb/common.py...
r36415 parity = (stripecount + offset) // stripecount & 1
Thomas Arendsen Hein
hgweb: use generator to count parity of horizontal stripes for easier reading....
r4462 else:
count = 0
parity = 0
while True:
yield parity
count += 1
if stripecount and count >= stripecount:
parity = 1 - parity
count = 0
Thomas Arendsen Hein
Don't let ui.username override web.contact (issue900)...
r5779 def get_contact(config):
"""Return repo contact information or empty string.
web.contact is the primary source, but if that is not set, try
ui.username or $EMAIL as a fallback to display something useful.
"""
return (config("web", "contact") or
config("ui", "username") or
Pulkit Goyal
py3: replace os.environ with encoding.environ (part 3 of 5)
r30636 encoding.environ.get("EMAIL") or "")
Dirkjan Ochtman
hgweb: support very simple caching model (issue1845)
r12183
Gregory Szorc
hgweb: support Content Security Policy...
r30766 def cspvalues(ui):
"""Obtain the Content-Security-Policy header and nonce value.
Returns a 2-tuple of the CSP header value and the nonce value.
First value is ``None`` if CSP isn't enabled. Second value is ``None``
if CSP isn't enabled or if the CSP header doesn't need a nonce.
"""
Jun Wu
hgweb: do not import uuid immediately to avoid its side effect...
r34644 # Without demandimport, "import uuid" could have an immediate side-effect
# running "ldconfig" on Linux trying to find libuuid.
# With Python <= 2.7.12, that "ldconfig" is run via a shell and the shell
# may pollute the terminal with:
#
# shell-init: error retrieving current directory: getcwd: cannot access
# parent directories: No such file or directory
#
# Python >= 2.7.13 has fixed it by running "ldconfig" directly without a
# shell (hg changeset a09ae70f3489).
#
# Moved "import uuid" from here so it's executed after we know we have
# a sane cwd (i.e. after dispatch.py cwd check).
#
# We can move it back once we no longer need Python <= 2.7.12 support.
import uuid
Gregory Szorc
hgweb: support Content Security Policy...
r30766 # Don't allow untrusted CSP setting since it be disable protections
# from a trusted/global source.
csp = ui.config('web', 'csp', untrusted=False)
nonce = None
if csp and '%nonce%' in csp:
nonce = base64.urlsafe_b64encode(uuid.uuid4().bytes).rstrip('=')
csp = csp.replace('%nonce%', nonce)
return csp, nonce