##// END OF EJS Templates
revlog: add a mechanism to verify expected file position before appending...
revlog: add a mechanism to verify expected file position before appending If someone uses `hg debuglocks`, or some non-hg process writes to the .hg directory without respecting the locks, or if the repo's on a networked filesystem, it's possible for the revlog code to write out corrupted data. The form of this corruption can vary depending on what data was written and how that happened. We are in the "networked filesystem" case (though I've had users also do this to themselves with the "`hg debuglocks`" scenario), and most often see this with the changelog. What ends up happening is we produce two items (let's call them rev1 and rev2) in the .i file that have the same linkrev, baserev, and offset into the .d file, while the data in the .d file is appended properly. rev2's compressed_size is accurate for rev2, but when we go to decompress the data in the .d file, we use the offset that's recorded in the index file, which is the same as rev1, and attempt to decompress rev2.compressed_size bytes of rev1's data. This usually does not succeed. :) When using inline data, this also fails, though I haven't investigated why too closely. This shows up as a "patch decode" error. I believe what's happening there is that we're basically ignoring the offset field, getting the data properly, but since baserev != rev, it thinks this is a delta based on rev (instead of a full text) and can't actually apply it as such. For now, I'm going to make this an optional component and default it to entirely off. I may increase the default severity of this in the future, once I've enabled it for my users and we gain more experience with it. Luckily, most of my users have a versioned filesystem and can roll back to before the corruption has been written, it's just a hassle to do so and not everyone knows how (so it's a support burden). Users on other filesystems will not have that luxury, and this can cause them to have a corrupted repository that they are unlikely to know how to resolve, and they'll see this as a data-loss event. Refusing to create the corruption is a much better user experience. This mechanism is not perfect. There may be false-negatives (racy writes that are not detected). There should not be any false-positives (non-racy writes that are detected as such). This is not a mechanism that makes putting a repo on a networked filesystem "safe" or "supported", just *less* likely to cause corruption. Differential Revision: https://phab.mercurial-scm.org/D9952

File last commit:

r44730:eecc0052 stable
r47349:e9901d01 default
Show More
__init__.py
243 lines | 6.9 KiB | text/x-python | PythonLexer
Matt Mackall
zeroconf: initial implementation...
r7071 # zeroconf.py - zeroconf support for Mercurial
#
# Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
#
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Cédric Duval
extensions: improve the consistency of synopses...
r8894 '''discover and advertise repositories on the local network
David Soria Parra
zeroconf: add extension documentation
r7606
Vernon Tang
zeroconf: improve the extension's documentation...
r43806 The zeroconf extension will advertise :hg:`serve` instances over
DNS-SD so that they can be discovered using the :hg:`paths` command
without knowing the server's IP address.
David Soria Parra
zeroconf: add extension documentation
r7606
Martin Geisler
zeroconf: small fixes in docstring...
r11504 To allow other people to discover your repository using run
:hg:`serve` in your repository::
David Soria Parra
zeroconf: add extension documentation
r7606
Martin Geisler
zeroconf: use reST syntax for literal blocks
r9218 $ cd test
$ hg serve
David Soria Parra
zeroconf: add extension documentation
r7606
Martin Geisler
zeroconf: small fixes in docstring...
r11504 You can discover Zeroconf-enabled repositories by running
:hg:`paths`::
David Soria Parra
zeroconf: add extension documentation
r7606
Martin Geisler
zeroconf: use reST syntax for literal blocks
r9218 $ hg paths
zc-test = http://example.com:8000/test
David Soria Parra
zeroconf: add extension documentation
r7606 '''
timeless
zeroconf: use absolute_import
r28296 from __future__ import absolute_import
David Soria Parra
zeroconf: add extension documentation
r7606
timeless
zeroconf: use absolute_import
r28296 import os
import socket
import time
Renato Cunha
hgext/zeroconf/__init__.py: Separate relative and absolute imports....
r11340
timeless
zeroconf: use absolute_import
r28296 from . import Zeroconf
from mercurial import (
dispatch,
encoding,
extensions,
hg,
Gregory Szorc
zeroconf: port to Python 3...
r42752 pycompat,
config: also respect HGRCSKIPREPO in the zeroconf extension...
r44730 rcutil,
Augie Fackler
zeroconf: import ui as uimod per test-check-module-imports
r28308 ui as uimod,
timeless
zeroconf: use absolute_import
r28296 )
Augie Fackler
formatting: blacken the codebase...
r43346 from mercurial.hgweb import server as servermod
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
extensions: change magic "shipped with hg" string...
r29841 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
Augie Fackler
extensions: document that `testedwith = 'internal'` is special...
r25186 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 testedwith = b'ships-with-hg-core'
Augie Fackler
hgext: mark all first-party extensions as such
r16743
Matt Mackall
zeroconf: initial implementation...
r7071 # publish
server = None
localip = None
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 def getip():
# finds external-facing interface without sending any packets (Linux)
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 s.connect(('1.0.0.1', 0))
Matt Mackall
zeroconf: initial implementation...
r7071 ip = s.getsockname()[0]
return ip
Brodie Rao
cleanup: replace naked excepts with more specific ones
r16688 except socket.error:
Matt Mackall
zeroconf: initial implementation...
r7071 pass
# Generic method, sometimes gives useless results
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 try:
dumbip = socket.gethostbyaddr(socket.gethostname())[2][0]
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 if ':' in dumbip:
dumbip = '127.0.0.1'
if not dumbip.startswith('127.'):
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 return dumbip
Augie Fackler
zeroconf: gethostbyaddr may also fail with socket.herror
r10317 except (socket.gaierror, socket.herror):
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 dumbip = '127.0.0.1'
Matt Mackall
zeroconf: initial implementation...
r7071
# works elsewhere, but actually sends a packet
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 s.connect(('1.0.0.1', 1))
Matt Mackall
zeroconf: initial implementation...
r7071 ip = s.getsockname()[0]
return ip
Brodie Rao
cleanup: replace naked excepts with more specific ones
r16688 except socket.error:
Matt Mackall
zeroconf: initial implementation...
r7071 pass
return dumbip
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 def publish(name, desc, path, port):
global server, localip
if not server:
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 ip = getip()
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 if ip.startswith('127.'):
Augie Fackler
zeroconf: Don't break serve if no internet connection is present.
r7295 # if we have no internet connection, this can happen.
return
Matt Mackall
zeroconf: initial implementation...
r7071 localip = socket.inet_aton(ip)
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 server = Zeroconf.Zeroconf(ip)
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 hostname = socket.gethostname().split('.')[0]
host = hostname + ".local"
Augie Fackler
cleanup: remove pointless r-prefixes on double-quoted strings...
r43809 name = "%s-%s" % (hostname, name)
Matt Mackall
zeroconf: initial implementation...
r7071
# advertise to browsers
Augie Fackler
formatting: blacken the codebase...
r43346 svc = Zeroconf.ServiceInfo(
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 b'_http._tcp.local.',
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 pycompat.bytestr(name + '._http._tcp.local.'),
Augie Fackler
formatting: blacken the codebase...
r43346 server=host,
port=port,
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 properties={b'description': desc, b'path': b"/" + path},
Augie Fackler
formatting: blacken the codebase...
r43346 address=localip,
weight=0,
priority=0,
)
Matt Mackall
zeroconf: initial implementation...
r7071 server.registerService(svc)
# advertise to Mercurial clients
Augie Fackler
formatting: blacken the codebase...
r43346 svc = Zeroconf.ServiceInfo(
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 b'_hg._tcp.local.',
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 pycompat.bytestr(name + '._hg._tcp.local.'),
Augie Fackler
formatting: blacken the codebase...
r43346 server=host,
port=port,
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 properties={b'description': desc, b'path': b"/" + path},
Augie Fackler
formatting: blacken the codebase...
r43346 address=localip,
weight=0,
priority=0,
)
Matt Mackall
zeroconf: initial implementation...
r7071 server.registerService(svc)
Augie Fackler
formatting: blacken the codebase...
r43346
Benoit Boissinot
zeroconf: use port from server instead of picking port from config (issue3746)...
r18190 def zc_create_server(create_server, ui, app):
httpd = create_server(ui, app)
port = httpd.port
Matt Mackall
zeroconf: initial implementation...
r7071
Benoit Boissinot
zeroconf: use port from server instead of picking port from config (issue3746)...
r18190 try:
repos = app.repos
except AttributeError:
# single repo
Gregory Szorc
zeroconf: access repo on hgweb_mod properly (issue5036)...
r27910 with app._obtainrepo() as repo:
name = app.reponame or os.path.basename(repo.root)
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 path = repo.ui.config(b"web", b"prefix", b"").strip(b'/')
desc = repo.ui.config(b"web", b"description")
Boris Feld
configitems: register the 'web.description' config
r34235 if not desc:
desc = name
Benoit Boissinot
zeroconf: use port from server instead of picking port from config (issue3746)...
r18190 publish(name, desc, path, port)
else:
# webdir
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 prefix = app.ui.config(b"web", b"prefix", b"").strip(b'/') + b'/'
Benoit Boissinot
zeroconf: use port from server instead of picking port from config (issue3746)...
r18190 for repo, path in repos:
u = app.ui.copy()
config: also respect HGRCSKIPREPO in the zeroconf extension...
r44730 if rcutil.use_repo_hgrc():
u.readconfig(os.path.join(path, b'.hg', b'hgrc'))
Alexander Solovyov
zeroconf: code cleanup, fixing variable names to be meaningful
r9488 name = os.path.basename(repo)
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 path = (prefix + repo).strip(b'/')
desc = u.config(b'web', b'description')
Boris Feld
configitems: register the 'web.description' config
r34235 if not desc:
desc = name
Benoit Boissinot
zeroconf: use port from server instead of picking port from config (issue3746)...
r18190 publish(name, desc, path, port)
return httpd
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 # listen
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 class listener(object):
def __init__(self):
self.found = {}
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 def removeService(self, server, type, name):
if repr(name) in self.found:
del self.found[repr(name)]
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 def addService(self, server, type, name):
self.found[repr(name)] = server.getServiceInfo(type, name)
Augie Fackler
formatting: blacken the codebase...
r43346
Matt Mackall
zeroconf: initial implementation...
r7071 def getzcpaths():
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 ip = getip()
Augie Fackler
cleanup: remove pointless r-prefixes on single-quoted strings...
r43906 if ip.startswith('127.'):
Alexander Solovyov
zeroconf: guess ip for Zeroconf...
r8264 return
server = Zeroconf.Zeroconf(ip)
Matt Mackall
zeroconf: initial implementation...
r7071 l = listener()
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 Zeroconf.ServiceBrowser(server, b"_hg._tcp.local.", l)
Matt Mackall
zeroconf: initial implementation...
r7071 time.sleep(1)
server.close()
Alexander Solovyov
zeroconf: code cleanup, fixing variable names to be meaningful
r9488 for value in l.found.values():
Augie Fackler
formatting: blacken the codebase...
r43346 name = value.name[: value.name.index(b'.')]
Augie Fackler
cleanup: remove pointless r-prefixes on double-quoted strings...
r43809 url = "http://%s:%s%s" % (
Augie Fackler
formatting: blacken the codebase...
r43346 socket.inet_ntoa(value.address),
value.port,
Augie Fackler
cleanup: remove pointless r-prefixes on double-quoted strings...
r43809 value.properties.get("path", "/"),
Augie Fackler
formatting: blacken the codebase...
r43346 )
Gregory Szorc
zeroconf: port to Python 3...
r42752 yield b"zc-" + name, pycompat.bytestr(url)
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
formatting: blacken the codebase...
r43346
zeroconf: blindly forward extra argument to the core config method...
r33175 def config(orig, self, section, key, *args, **kwargs):
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 if section == b"paths" and key.startswith(b"zc-"):
Alexander Solovyov
zeroconf: code cleanup, fixing variable names to be meaningful
r9488 for name, path in getzcpaths():
if name == key:
return path
zeroconf: blindly forward extra argument to the core config method...
r33175 return orig(self, section, key, *args, **kwargs)
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
formatting: blacken the codebase...
r43346
Yuya Nishihara
zeroconf: forward all arguments passed to ui.configitems() wrapper...
r28038 def configitems(orig, self, section, *args, **kwargs):
repos = orig(self, section, *args, **kwargs)
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 if section == b"paths":
Alexander Solovyov
zeroconf: code cleanup, fixing variable names to be meaningful
r9488 repos += getzcpaths()
return repos
Matt Mackall
zeroconf: initial implementation...
r7071
Augie Fackler
formatting: blacken the codebase...
r43346
Danek Duvall
zeroconf: fix crash in "hg paths" when zeroconf server is up...
r28250 def configsuboptions(orig, self, section, name, *args, **kwargs):
opt, sub = orig(self, section, name, *args, **kwargs)
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 if section == b"paths" and name.startswith(b"zc-"):
Danek Duvall
zeroconf: fix crash in "hg paths" when zeroconf server is up...
r28250 # We have to find the URL in the zeroconf paths. We can't cons up any
# suboptions, so we use any that we found in the original config.
for zcname, zcurl in getzcpaths():
if zcname == name:
return zcurl, sub
return opt, sub
Augie Fackler
formatting: blacken the codebase...
r43346
Henrik Stuart
zeroconf: override default destination folder on clone
r10342 def defaultdest(orig, source):
for name, path in getzcpaths():
if path == source:
return name.encode(encoding.encoding)
return orig(source)
Augie Fackler
formatting: blacken the codebase...
r43346
Nicolas Dumazet
zeroconf: notify the Zeroconf threads when hg exits...
r14104 def cleanupafterdispatch(orig, ui, options, cmd, cmdfunc):
try:
return orig(ui, options, cmd, cmdfunc)
finally:
# we need to call close() on the server to notify() the various
# threading Conditions and allow the background threads to exit
global server
if server:
server.close()
Augie Fackler
formatting: blacken the codebase...
r43346
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 extensions.wrapfunction(dispatch, b'_runcommand', cleanupafterdispatch)
Nicolas Dumazet
zeroconf: notify the Zeroconf threads when hg exits...
r14104
Augie Fackler
formatting: byteify all mercurial/ and hgext/ string literals...
r43347 extensions.wrapfunction(uimod.ui, b'config', config)
extensions.wrapfunction(uimod.ui, b'configitems', configitems)
extensions.wrapfunction(uimod.ui, b'configsuboptions', configsuboptions)
extensions.wrapfunction(hg, b'defaultdest', defaultdest)
extensions.wrapfunction(servermod, b'create_server', zc_create_server)