##// END OF EJS Templates
demandimport: replace more references to _demandmod instances...
demandimport: replace more references to _demandmod instances _demandmod instances may be referenced by multiple importing modules. Before this patch, the _demandmod instance only maintained a reference to its first consumer when using the "from X import Y" syntax. This is because we only created a single _demandmod instance (attached to the parent X module). If multiple modules A and B performed "from X import Y", we'd produce a single _demandmod instance "demandmod" with the following references: X.Y = <demandmod> A.Y = <demandmod> B.Y = <demandmod> The locals from the first consumer (A) would be stored in <demandmod1>. When <demandmod1> was loaded, we'd look at the locals for the first consumer and replace the symbol, if necessary. This resulted in state: X.Y = <module> A.Y = <module> B.Y = <demandmod> B's reference to Y wasn't updated and was still using the proxy object because we just didn't record that B had a reference to <demandmod> that needed updating! With this patch, we add support for tracking which modules in addition to the initial importer have a reference to the _demandmod instance and we replace those references at module load time. In the case of posix.py, this fixes an issue where the "encoding" module was being proxied, resulting in hundreds of thousands of __getattribute__ lookups on the _demandmod instance during dirstate operations on mozilla-central, speeding up execution by many milliseconds. There are likely several other operation that benefit from this change as well. The new mechanism isn't perfect: references in locals (not globals) may likely linger. So, if there is an import inside a function and a symbol from that module is used in a hot loop, we could have unwanted overhead from proxying through _demandmod. Non-global imports are discouraged anyway. So hopefully this isn't a big deal in practice. We could potentially deploy a code checker that bans use of attribute lookups of function-level-imported modules inside loops. This deficiency in theory could be avoided by storing the set of globals and locals dicts to update in the _demandmod instance. However, I tried this and it didn't work. One reason is that some globals are _demandmod instances. We could work around this, but it's a bit more work. There also might be other module import foo at play. The solution as implemented is better than what we had and IMO is good enough for the time being. It's worth noting that this sub-optimal behavior was made worse by the introduction of absolute_import and its recommended "from . import X" syntax for importing modules from the "mercurial" package. If we ever wrote performance tests, measuring the amount of module imports and __getattribute__ proxy calls through _demandmod instances would be something I'd have it check.

File last commit:

r20903:8d477543 default
r26457:7e813050 default
Show More
protocol.py
98 lines | 3.1 KiB | text/x-python | PythonLexer
Dirkjan Ochtman
separate the wire protocol commands from the user interface commands
r5598 #
# Copyright 21 May 2005 - (c) 2005 Jake Edge <jake@edge2.net>
# Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
#
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
Dirkjan Ochtman
separate the wire protocol commands from the user interface commands
r5598
Idan Kamara
ui: use I/O descriptors internally...
r14614 import cgi, cStringIO, zlib, urllib
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 from mercurial import util, wireproto
from common import HTTP_OK
Dirkjan Ochtman
hgweb: explicitly check if requested command exists
r5963
Dirkjan Ochtman
hgweb: explicit response status
r5993 HGTYPE = 'application/mercurial-0.1'
Andrew Pritchard
wireproto: add out-of-band error class to allow remote repo to report errors...
r15017 HGERRTYPE = 'application/hg-error'
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595
Pierre-Yves David
wireproto: introduce an abstractserverproto class...
r20903 class webproto(wireproto.abstractserverproto):
Idan Kamara
ui: use I/O descriptors internally...
r14614 def __init__(self, req, ui):
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 self.req = req
self.response = ''
Idan Kamara
ui: use I/O descriptors internally...
r14614 self.ui = ui
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 def getargs(self, args):
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 knownargs = self._args()
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 data = {}
keys = args.split()
for k in keys:
if k == '*':
star = {}
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 for key in knownargs.keys():
Peter Arrenbrecht
wireproto: fix handling of '*' args for HTTP and SSH
r13721 if key != 'cmd' and key not in keys:
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 star[key] = knownargs[key][0]
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 data['*'] = star
else:
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 data[k] = knownargs[k][0]
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 return [data[k] for k in keys]
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 def _args(self):
args = self.req.form.copy()
chunks = []
Matt Mackall
http: minor tweaks to long arg handling...
r14094 i = 1
Martin Geisler
check-code: flag 0/1 used as constant Boolean expression
r14494 while True:
Matt Mackall
http: minor tweaks to long arg handling...
r14094 h = self.req.env.get('HTTP_X_HGARG_' + str(i))
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 if h is None:
break
chunks += [h]
Matt Mackall
http: minor tweaks to long arg handling...
r14094 i += 1
Steven Brown
httprepo: long arguments support (issue2126)...
r14093 args.update(cgi.parse_qs(''.join(chunks), keep_blank_values=True))
return args
Dirkjan Ochtman
protocol: shuffle server methods to group send methods
r11621 def getfile(self, fp):
length = int(self.req.env['CONTENT_LENGTH'])
for s in util.filechunkiter(self.req, limit=length):
fp.write(s)
def redirect(self):
Idan Kamara
ui: use I/O descriptors internally...
r14614 self.oldio = self.ui.fout, self.ui.ferr
self.ui.ferr = self.ui.fout = cStringIO.StringIO()
def restore(self):
val = self.ui.fout.getvalue()
self.ui.ferr, self.ui.fout = self.oldio
return val
Dirkjan Ochtman
protocol: extract compression from streaming mechanics
r11623 def groupchunks(self, cg):
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 z = zlib.compressobj()
Martin Geisler
check-code: flag 0/1 used as constant Boolean expression
r14494 while True:
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 chunk = cg.read(4096)
if not chunk:
break
Dirkjan Ochtman
protocol: extract compression from streaming mechanics
r11623 yield z.compress(chunk)
yield z.flush()
Matt Mackall
protocol: move hgweb protocol support back into protocol.py...
r11595 def _client(self):
return 'remote:%s:%s:%s' % (
self.req.env.get('wsgi.url_scheme') or 'http',
urllib.quote(self.req.env.get('REMOTE_HOST', '')),
urllib.quote(self.req.env.get('REMOTE_USER', '')))
def iscmd(cmd):
return cmd in wireproto.commands
def call(repo, req, cmd):
Idan Kamara
ui: use I/O descriptors internally...
r14614 p = webproto(req, repo.ui)
Dirkjan Ochtman
protocol: wrap non-string protocol responses in classes
r11625 rsp = wireproto.dispatch(repo, p, cmd)
Dirkjan Ochtman
protocol: use generators instead of req.write() for hgweb stream responses
r11626 if isinstance(rsp, str):
Mads Kiilerich
hgweb: pass the actual response body to request.response, not just the length...
r18352 req.respond(HTTP_OK, HGTYPE, body=rsp)
return []
Dirkjan Ochtman
protocol: use generators instead of req.write() for hgweb stream responses
r11626 elif isinstance(rsp, wireproto.streamres):
req.respond(HTTP_OK, HGTYPE)
return rsp.gen
elif isinstance(rsp, wireproto.pushres):
Idan Kamara
ui: use I/O descriptors internally...
r14614 val = p.restore()
Mads Kiilerich
hgweb: use Content-Length for pushres...
r18346 rsp = '%d\n%s' % (rsp.res, val)
Mads Kiilerich
hgweb: pass the actual response body to request.response, not just the length...
r18352 req.respond(HTTP_OK, HGTYPE, body=rsp)
return []
Benoit Boissinot
wireproto: introduce pusherr() to deal with "unsynced changes" error...
r12703 elif isinstance(rsp, wireproto.pusherr):
Benoit Boissinot
wireproto/http: drain the incoming bundle in case of errors
r12704 # drain the incoming bundle
req.drain()
Idan Kamara
ui: use I/O descriptors internally...
r14614 p.restore()
Benoit Boissinot
wireproto: introduce pusherr() to deal with "unsynced changes" error...
r12703 rsp = '0\n%s\n' % rsp.res
Mads Kiilerich
hgweb: pass the actual response body to request.response, not just the length...
r18352 req.respond(HTTP_OK, HGTYPE, body=rsp)
return []
Andrew Pritchard
wireproto: add out-of-band error class to allow remote repo to report errors...
r15017 elif isinstance(rsp, wireproto.ooberror):
rsp = rsp.message
Mads Kiilerich
hgweb: pass the actual response body to request.response, not just the length...
r18352 req.respond(HTTP_OK, HGERRTYPE, body=rsp)
return []