##// END OF EJS Templates
revlog: linearize created changegroups in generaldelta revlogs...
revlog: linearize created changegroups in generaldelta revlogs This greatly improves the speed of the bundling process, and often reduces the bundle size considerably. (Although if the repository is already ordered, this has little effect on both time and bundle size.) For non-generaldelta clients, the reduced bundle size translates to a reduced repository size, similar to shrinking the revlogs (which uses the exact same algorithm). For generaldelta clients the difference is minor. When the new bundle format comes, reordering will not be necessary since we can then store the deltaparent relationsships directly. The eventual default behavior for clients and servers is presented in the table below, where "new" implies support for GD as well as the new bundle format: old client new client old server old bundle, no reorder old bundle, no reorder new server, non-GD old bundle, no reorder[1] old bundle, no reorder[2] new server, GD old bundle, reorder[3] new bundle, no reorder[4] [1] reordering is expensive on the server in this case, skip it [2] client can choose to do its own redelta here [3] reordering is needed because otherwise the pull does a lot of extra work on the server [4] reordering isn't needed because client can get deltabase in bundle format Currently, the default is to reorder on GD-servers, and not otherwise. A new setting, bundle.reorder, has been added to override the default reordering behavior. It can be set to either 'auto' (the default), or any true or false value as a standard boolean setting, to either force the reordering on or off regardless of generaldelta. Some timing data from a relatively branch test repository follows. All bundling is done with --all --type none options. Non-generaldelta, non-shrunk repo: ----------------------------------- Size: 276M Without reorder (default): Bundle time: 14.4 seconds Bundle size: 939M With reorder: Bundle time: 1 minute, 29.3 seconds Bundle size: 381M Generaldelta, non-shrunk repo: ----------------------------------- Size: 87M Without reorder: Bundle time: 2 minutes, 1.4 seconds Bundle size: 939M With reorder (default): Bundle time: 25.5 seconds Bundle size: 381M

File last commit:

r14233:659f34b8 default
r14365:a8e3931e default
Show More
sshserver.py
144 lines | 4.0 KiB | text/x-python | PythonLexer
Vadim Gelfer
fix comment.
r2399 # sshserver.py - ssh protocol server support for mercurial
Vadim Gelfer
refactor ssh server.
r2396 #
Thomas Arendsen Hein
Updated copyright notices and add "and others" to "hg version"
r4635 # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
Vadim Gelfer
update copyrights.
r2859 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
Vadim Gelfer
refactor ssh server.
r2396 #
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Vadim Gelfer
refactor ssh server.
r2396
Matt Mackall
bundle: encapsulate all bundle streams in unbundle class
r12337 import util, hook, wireproto, changegroup
Matt Mackall
ssh: drop some old imports
r11596 import os, sys
Vadim Gelfer
refactor ssh server.
r2396
class sshserver(object):
def __init__(self, ui, repo):
self.ui = ui
self.repo = repo
self.lock = None
self.fin = sys.stdin
self.fout = sys.stdout
Matt Mackall
hook: redirect stdout to stderr for ssh and http servers
r5833 hook.redirect(True)
Vadim Gelfer
refactor ssh server.
r2396 sys.stdout = sys.stderr
# Prevent insertion/deletion of CRs
Adrian Buehlmann
rename util.set_binary to setbinary
r14233 util.setbinary(self.fin)
util.setbinary(self.fout)
Vadim Gelfer
refactor ssh server.
r2396
Matt Mackall
protocol: add ssh getargs...
r11579 def getargs(self, args):
data = {}
keys = args.split()
for n in xrange(len(keys)):
argline = self.fin.readline()[:-1]
arg, l = argline.split()
if arg not in keys:
raise util.Abort("unexpected parameter %r" % arg)
if arg == '*':
star = {}
Peter Arrenbrecht
wireproto: fix handling of '*' args for HTTP and SSH
r13721 for k in xrange(int(l)):
argline = self.fin.readline()[:-1]
Matt Mackall
protocol: add ssh getargs...
r11579 arg, l = argline.split()
val = self.fin.read(int(l))
star[arg] = val
data['*'] = star
else:
Peter Arrenbrecht
wireproto: fix handling of '*' args for HTTP and SSH
r13721 val = self.fin.read(int(l))
Matt Mackall
protocol: add ssh getargs...
r11579 data[arg] = val
return [data[k] for k in keys]
def getarg(self, name):
return self.getargs(name)[0]
Vadim Gelfer
refactor ssh server.
r2396
Dirkjan Ochtman
protocol: shuffle server methods to group send methods
r11621 def getfile(self, fpout):
Dirkjan Ochtman
protocol: rename send methods to get grouping by prefix
r11622 self.sendresponse('')
Dirkjan Ochtman
protocol: shuffle server methods to group send methods
r11621 count = int(self.fin.readline())
while count:
fpout.write(self.fin.read(count))
count = int(self.fin.readline())
def redirect(self):
pass
Dirkjan Ochtman
protocol: extract compression from streaming mechanics
r11623 def groupchunks(self, changegroup):
Matt Mackall
protocol: unify changegroup commands...
r11584 while True:
d = changegroup.read(4096)
if not d:
break
Dirkjan Ochtman
protocol: extract compression from streaming mechanics
r11623 yield d
Matt Mackall
protocol: unify changegroup commands...
r11584
Dirkjan Ochtman
protocol: extract compression from streaming mechanics
r11623 def sendresponse(self, v):
self.fout.write("%d\n" % len(v))
self.fout.write(v)
Matt Mackall
protocol: unify changegroup commands...
r11584 self.fout.flush()
Matt Mackall
protocol: unify stream_out command
r11585 def sendstream(self, source):
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 for chunk in source.gen:
Matt Mackall
protocol: unify stream_out command
r11585 self.fout.write(chunk)
self.fout.flush()
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 def sendpushresponse(self, rsp):
Dirkjan Ochtman
protocol: rename send methods to get grouping by prefix
r11622 self.sendresponse('')
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 self.sendresponse(str(rsp.res))
Matt Mackall
protocol: unify unbundle on the server side
r11593
Benoit Boissinot
wireproto: introduce pusherr() to deal with "unsynced changes" error...
r12703 def sendpusherror(self, rsp):
self.sendresponse(rsp.res)
Vadim Gelfer
refactor ssh server.
r2396 def serve_forever(self):
Ronny Pfannschmidt
switch lock releasing in the core from gc to explicit
r8109 try:
Matt Mackall
many, many trivial check-code fixups
r10282 while self.serve_one():
pass
Ronny Pfannschmidt
switch lock releasing in the core from gc to explicit
r8109 finally:
if self.lock is not None:
self.lock.release()
Vadim Gelfer
refactor ssh server.
r2396 sys.exit(0)
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 handlers = {
str: sendresponse,
wireproto.streamres: sendstream,
wireproto.pushres: sendpushresponse,
Benoit Boissinot
wireproto: introduce pusherr() to deal with "unsynced changes" error...
r12703 wireproto.pusherr: sendpusherror,
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 }
Vadim Gelfer
refactor ssh server.
r2396 def serve_one(self):
cmd = self.fin.readline()[:-1]
Dirkjan Ochtman
protocol: command must be checked before passing in
r11618 if cmd and cmd in wireproto.commands:
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 rsp = wireproto.dispatch(self.repo, self, cmd)
self.handlers[rsp.__class__](self, rsp)
Dirkjan Ochtman
protocol: command must be checked before passing in
r11618 elif cmd:
Vadim Gelfer
refactor ssh server.
r2396 impl = getattr(self, 'do_' + cmd, None)
Matt Mackall
many, many trivial check-code fixups
r10282 if impl:
Matt Mackall
protocol: move most ssh responses to returns
r11580 r = impl()
if r is not None:
Dirkjan Ochtman
protocol: rename send methods to get grouping by prefix
r11622 self.sendresponse(r)
else: self.sendresponse("")
Vadim Gelfer
refactor ssh server.
r2396 return cmd != ''
def do_lock(self):
Vadim Gelfer
extend network protocol to stop clients from locking servers...
r2439 '''DEPRECATED - allowing remote client to lock repo is not safe'''
Vadim Gelfer
refactor ssh server.
r2396 self.lock = self.repo.lock()
Matt Mackall
protocol: move most ssh responses to returns
r11580 return ""
Vadim Gelfer
refactor ssh server.
r2396
def do_unlock(self):
Vadim Gelfer
extend network protocol to stop clients from locking servers...
r2439 '''DEPRECATED'''
Vadim Gelfer
refactor ssh server.
r2396 if self.lock:
self.lock.release()
self.lock = None
Matt Mackall
protocol: move most ssh responses to returns
r11580 return ""
Vadim Gelfer
refactor ssh server.
r2396
def do_addchangegroup(self):
Vadim Gelfer
extend network protocol to stop clients from locking servers...
r2439 '''DEPRECATED'''
Vadim Gelfer
refactor ssh server.
r2396 if not self.lock:
Dirkjan Ochtman
protocol: rename send methods to get grouping by prefix
r11622 self.sendresponse("not locked")
Vadim Gelfer
refactor ssh server.
r2396 return
Dirkjan Ochtman
protocol: rename send methods to get grouping by prefix
r11622 self.sendresponse("")
Matt Mackall
bundle: encapsulate all bundle streams in unbundle class
r12337 cg = changegroup.unbundle10(self.fin, "UN")
r = self.repo.addchangegroup(cg, 'serve', self._client(),
Matt Mackall
addchangegroup: pass in lock to release it before changegroup hook is called...
r11442 lock=self.lock)
Matt Mackall
protocol: move most ssh responses to returns
r11580 return str(r)
Vadim Gelfer
extend network protocol to stop clients from locking servers...
r2439
Matt Mackall
protocol: unify unbundle on the server side
r11593 def _client(self):
Vadim Gelfer
hooks: add url to changegroup, incoming, prechangegroup, pretxnchangegroup hooks...
r2673 client = os.environ.get('SSH_CLIENT', '').split(' ', 1)[0]
return 'remote:ssh:' + client