##// END OF EJS Templates
demandimport: replace more references to _demandmod instances...
demandimport: replace more references to _demandmod instances _demandmod instances may be referenced by multiple importing modules. Before this patch, the _demandmod instance only maintained a reference to its first consumer when using the "from X import Y" syntax. This is because we only created a single _demandmod instance (attached to the parent X module). If multiple modules A and B performed "from X import Y", we'd produce a single _demandmod instance "demandmod" with the following references: X.Y = <demandmod> A.Y = <demandmod> B.Y = <demandmod> The locals from the first consumer (A) would be stored in <demandmod1>. When <demandmod1> was loaded, we'd look at the locals for the first consumer and replace the symbol, if necessary. This resulted in state: X.Y = <module> A.Y = <module> B.Y = <demandmod> B's reference to Y wasn't updated and was still using the proxy object because we just didn't record that B had a reference to <demandmod> that needed updating! With this patch, we add support for tracking which modules in addition to the initial importer have a reference to the _demandmod instance and we replace those references at module load time. In the case of posix.py, this fixes an issue where the "encoding" module was being proxied, resulting in hundreds of thousands of __getattribute__ lookups on the _demandmod instance during dirstate operations on mozilla-central, speeding up execution by many milliseconds. There are likely several other operation that benefit from this change as well. The new mechanism isn't perfect: references in locals (not globals) may likely linger. So, if there is an import inside a function and a symbol from that module is used in a hot loop, we could have unwanted overhead from proxying through _demandmod. Non-global imports are discouraged anyway. So hopefully this isn't a big deal in practice. We could potentially deploy a code checker that bans use of attribute lookups of function-level-imported modules inside loops. This deficiency in theory could be avoided by storing the set of globals and locals dicts to update in the _demandmod instance. However, I tried this and it didn't work. One reason is that some globals are _demandmod instances. We could work around this, but it's a bit more work. There also might be other module import foo at play. The solution as implemented is better than what we had and IMO is good enough for the time being. It's worth noting that this sub-optimal behavior was made worse by the introduction of absolute_import and its recommended "from . import X" syntax for importing modules from the "mercurial" package. If we ever wrote performance tests, measuring the amount of module imports and __getattribute__ proxy calls through _demandmod instances would be something I'd have it check.

File last commit:

r26387:e16f80f8 default
r26457:7e813050 default
Show More
lock.py
231 lines | 7.5 KiB | text/x-python | PythonLexer
Greg Ward
localrepo: document the locking scheme a little better...
r9309 # lock.py - simple advisory locking scheme for mercurial
mpm@selenic.com
Simply repository locking...
r161 #
Vadim Gelfer
update copyrights.
r2859 # Copyright 2005, 2006 Matt Mackall <mpm@selenic.com>
mpm@selenic.com
Simply repository locking...
r161 #
Martin Geisler
updated license to be explicit about GPL version 2
r8225 # This software may be used and distributed according to the terms of the
Matt Mackall
Update license to GPLv2+
r10263 # GNU General Public License version 2 or any later version.
mpm@selenic.com
Simply repository locking...
r161
Gregory Szorc
lock: use absolute_import
r25956 from __future__ import absolute_import
import errno
import os
import socket
import time
Ronny Pfannschmidt
add a deprecation warning for gc based lock releasing
r8113 import warnings
mpm@selenic.com
Simply repository locking...
r161
Gregory Szorc
lock: use absolute_import
r25956 from . import (
error,
util,
)
Eric Hopper
Convert all classes to new-style classes by deriving them from object.
r1559 class lock(object):
Greg Ward
localrepo: document the locking scheme a little better...
r9309 '''An advisory lock held by one process to control access to a set
of files. Non-cooperating processes or incorrectly written scripts
can ignore Mercurial's locking scheme and stomp all over the
repository, so don't do that.
Typically used via localrepository.lock() to lock the repository
store (.hg/store/) or localrepository.wlock() to lock everything
else under .hg/.'''
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 # lock is symlink on platforms that support it, file on others.
# symlink is used because create of directory entry and contents
# are atomic even over nfs.
# old-style lock: symlink to pid
# new-style lock: symlink to hostname:pid
Bryan O'Sullivan
lock.py: cache hostname, but not pid, in case we fork
r4947 _host = None
Siddharth Agarwal
lock: move acquirefn call to inside the lock...
r26321 def __init__(self, vfs, file, timeout=-1, releasefn=None, acquirefn=None,
Siddharth Agarwal
lock: introduce state to keep track of inheritance...
r26356 desc=None, parentlock=None):
FUJIWARA Katsunori
lock: take both vfs and lock file path relative to vfs to access via vfs...
r20091 self.vfs = vfs
mpm@selenic.com
Simply repository locking...
r161 self.f = file
self.held = 0
Benoit Boissinot
add a timeout when a lock is held (default 1024 sec)...
r1787 self.timeout = timeout
Benoit Boissinot
add a releasefn keyword to lock.lock...
r1530 self.releasefn = releasefn
Siddharth Agarwal
lock: move acquirefn call to inside the lock...
r26321 self.acquirefn = acquirefn
Vadim Gelfer
fix backtrace printed when cannot get lock....
r2016 self.desc = desc
Siddharth Agarwal
lock: introduce state to keep track of inheritance...
r26356 self.parentlock = parentlock
self._parentheld = False
self._inherited = False
Matt Mackall
lock: change name of release chain
r15589 self.postrelease = []
Siddharth Agarwal
lock: add a wrapper to os.getpid() to make testing easier...
r26383 self.pid = self._getpid()
Mads Kiilerich
localrepo: give a sigh of relief when getting lock after waiting for it...
r20380 self.delay = self.lock()
Siddharth Agarwal
lock: move acquirefn call to inside the lock...
r26321 if self.acquirefn:
self.acquirefn()
mpm@selenic.com
Simply repository locking...
r161
def __del__(self):
Ronny Pfannschmidt
made repo locks recursive and deprecate refcounting based lock releasing...
r8108 if self.held:
Ronny Pfannschmidt
add a deprecation warning for gc based lock releasing
r8113 warnings.warn("use lock.release instead of del lock",
category=DeprecationWarning,
stacklevel=2)
Ronny Pfannschmidt
made repo locks recursive and deprecate refcounting based lock releasing...
r8108 # ensure the lock will be removed
# even if recursive locking did occur
self.held = 1
mpm@selenic.com
Simply repository locking...
r161 self.release()
Siddharth Agarwal
lock: add a wrapper to os.getpid() to make testing easier...
r26383 def _getpid(self):
# wrapper around os.getpid() to make testing easier
return os.getpid()
mpm@selenic.com
Simply repository locking...
r161 def lock(self):
Benoit Boissinot
add a timeout when a lock is held (default 1024 sec)...
r1787 timeout = self.timeout
Martin Geisler
check-code: flag 0/1 used as constant Boolean expression
r14494 while True:
mpm@selenic.com
Simply repository locking...
r161 try:
Matt Mackall
lock: make trylock private
r26082 self._trylock()
Mads Kiilerich
localrepo: give a sigh of relief when getting lock after waiting for it...
r20380 return self.timeout - timeout
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except error.LockHeld as inst:
Benoit Boissinot
add a timeout when a lock is held (default 1024 sec)...
r1787 if timeout != 0:
mpm@selenic.com
Simply repository locking...
r161 time.sleep(1)
Benoit Boissinot
add a timeout when a lock is held (default 1024 sec)...
r1787 if timeout > 0:
timeout -= 1
mpm@selenic.com
Simply repository locking...
r161 continue
Matt Mackall
error: move lock errors...
r7640 raise error.LockHeld(errno.ETIMEDOUT, inst.filename, self.desc,
inst.locker)
mpm@selenic.com
Whitespace cleanups...
r515
Matt Mackall
lock: make trylock private
r26082 def _trylock(self):
Ronny Pfannschmidt
made repo locks recursive and deprecate refcounting based lock releasing...
r8108 if self.held:
self.held += 1
return
Bryan O'Sullivan
lock.py: cache hostname, but not pid, in case we fork
r4947 if lock._host is None:
lock._host = socket.gethostname()
Bryan O'Sullivan
lock: if we fork, ensure that only the parent releases...
r18907 lockname = '%s:%s' % (lock._host, self.pid)
Matt Mackall
lock: loop a finite number of times in trylock (issue4787)...
r26081 retry = 5
while not self.held and retry:
retry -= 1
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 try:
FUJIWARA Katsunori
lock: take both vfs and lock file path relative to vfs to access via vfs...
r20091 self.vfs.makelock(lockname, self.f)
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 self.held = 1
Gregory Szorc
global: mass rewrite to use modern exception syntax...
r25660 except (OSError, IOError) as why:
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 if why.errno == errno.EEXIST:
Siddharth Agarwal
lock: recognize parent locks while acquiring...
r26387 locker = self._readlock()
# special case where a parent process holds the lock -- this
# is different from the pid being different because we do
# want the unlock and postrelease functions to be called,
# but the lockfile to not be removed.
if locker == self.parentlock:
self._parentheld = True
self.held = 1
return
locker = self._testlock(locker)
Thomas Arendsen Hein
Don't step into an endless loop when lock file is empty.
r3686 if locker is not None:
FUJIWARA Katsunori
lock: take both vfs and lock file path relative to vfs to access via vfs...
r20091 raise error.LockHeld(errno.EAGAIN,
self.vfs.join(self.f), self.desc,
Matt Mackall
error: move lock errors...
r7640 locker)
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 else:
Matt Mackall
error: move lock errors...
r7640 raise error.LockUnavailable(why.errno, why.strerror,
why.filename, self.desc)
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877
Siddharth Agarwal
lock: factor code to read lock into a separate function...
r26290 def _readlock(self):
"""read lock and return its value
Returns None if no lock exists, pid for old-style locks, and host:pid
for new-style locks.
"""
try:
return self.vfs.readlock(self.f)
except (OSError, IOError) as why:
if why.errno == errno.ENOENT:
return None
raise
Siddharth Agarwal
lock: factor out lock testing into a separate function...
r26291 def _testlock(self, locker):
Siddharth Agarwal
lock: factor code to read lock into a separate function...
r26290 if locker is None:
return None
Benoit Boissinot
use __contains__, index or split instead of str.find...
r2579 try:
host, pid = locker.split(":", 1)
except ValueError:
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 return locker
Bryan O'Sullivan
lock.py: cache hostname, but not pid, in case we fork
r4947 if host != lock._host:
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 return locker
mpm@selenic.com
Simply repository locking...
r161 try:
Benoit Boissinot
use __contains__, index or split instead of str.find...
r2579 pid = int(pid)
Benoit Boissinot
lock: catch specific exceptions
r9685 except ValueError:
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 return locker
if util.testpid(pid):
return locker
# if locker dead, break lock. must do this with another lock
# held, or can race and break valid lock.
try:
FUJIWARA Katsunori
lock: take both vfs and lock file path relative to vfs to access via vfs...
r20091 l = lock(self.vfs, self.f + '.break', timeout=0)
self.vfs.unlink(self.f)
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 l.release()
Matt Mackall
error: move lock errors...
r7640 except error.LockError:
Vadim Gelfer
change lock format to let us detect and break stale locks....
r1877 return locker
mpm@selenic.com
Simply repository locking...
r161
Siddharth Agarwal
lock: factor out lock testing into a separate function...
r26291 def testlock(self):
"""return id of locker if lock is valid, else None.
If old-style lock, we cannot tell what machine locker is on.
with new-style lock, if locker is on this machine, we can
see if locker is alive. If locker is on this machine but
not alive, we can safely break lock.
The lock file is only deleted when None is returned.
"""
locker = self._readlock()
return self._testlock(locker)
Siddharth Agarwal
lock: add a method to prepare the lock for inheritance...
r26357 def prepinherit(self):
"""prepare for the lock to be inherited by a Mercurial subprocess
Returns a string that will be recognized by the lock in the
subprocess. Communicating this string to the subprocess needs to be done
separately -- typically by an environment variable.
"""
if not self.held:
raise error.LockInheritanceContractViolation(
'prepinherit can only be called while lock is held')
if self._inherited:
raise error.LockInheritanceContractViolation(
'prepinherit cannot be called while lock is already inherited')
if self.releasefn:
self.releasefn()
if self._parentheld:
lockname = self.parentlock
else:
lockname = '%s:%s' % (lock._host, self.pid)
self._inherited = True
return lockname
Siddharth Agarwal
lock: add a method to reacquire the lock after subprocesses exit...
r26358 def reacquire(self):
if not self._inherited:
raise error.LockInheritanceContractViolation(
'reacquire can only be called after prepinherit')
if self.acquirefn:
self.acquirefn()
self._inherited = False
mpm@selenic.com
Simply repository locking...
r161 def release(self):
Pierre-Yves David
lock: add mechanism to register post release callback
r15583 """release the lock and execute callback function if any
Bryan O'Sullivan
Merge spelling fixes
r17537 If the lock has been acquired multiple times, the actual release is
timeless@mozdev.org
spelling: release
r17510 delayed to the last release call."""
Ronny Pfannschmidt
made repo locks recursive and deprecate refcounting based lock releasing...
r8108 if self.held > 1:
self.held -= 1
Benoit Boissinot
lock: use '==' instead of 'is' for integer equality ('is' may not work)
r9680 elif self.held == 1:
mpm@selenic.com
Simply repository locking...
r161 self.held = 0
Siddharth Agarwal
lock: add a wrapper to os.getpid() to make testing easier...
r26383 if self._getpid() != self.pid:
Bryan O'Sullivan
lock: if we fork, ensure that only the parent releases...
r18907 # we forked, and are not the parent
return
mpm@selenic.com
Fix troubles with clone and exception handling...
r503 try:
Siddharth Agarwal
lock: while releasing, unlink lockfile even if the release function throws...
r23032 if self.releasefn:
self.releasefn()
finally:
Siddharth Agarwal
lock.release: do not unlink inherited locks...
r26359 if not self._parentheld:
try:
self.vfs.unlink(self.f)
except OSError:
pass
Matt Mackall
lock: change name of release chain
r15589 for callback in self.postrelease:
Pierre-Yves David
lock: add mechanism to register post release callback
r15583 callback()
mpm@selenic.com
Simply repository locking...
r161
Ronny Pfannschmidt
made repo locks recursive and deprecate refcounting based lock releasing...
r8108 def release(*locks):
for lock in locks:
if lock is not None:
lock.release()