upstream/mercurial-mirror Files · contrib/casesmash.py

merge: mark file gets as not thread safe (issue5933)...

merge: mark file gets as not thread safe (issue5933) In default installs, this has the effect of disabling the thread-based worker on Windows when manifesting files in the working directory. My measurements have shown that with revlog-based repositories, Mercurial spends a lot of CPU time in revlog code resolving file data. This ends up incurring a lot of context switching across threads and slows down `hg update` operations when going from an empty working directory to the tip of the repo. On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs): before: 487s wall after: 360s wall (equivalent to worker.enabled=false) cpus=2: 379s wall Even with only 2 threads, the thread pool is still slower. The introduction of the thread-based worker (02b36e860e0b) states that it resulted in a "~50%" speedup for `hg sparse --enable-profile` and `hg sparse --disable-profile`. This disagrees with my measurement above. I theorize a few reasons for this: 1) Removal of files from the working directory is I/O - not CPU - bound and should benefit from a thread pool (unless I/O is insanely fast and the GIL release is near instantaneous). So tests like `hg sparse --enable-profile` may exercise deletion throughput and aren't good benchmarks for worker tasks that are CPU heavy. 2) The patch was authored by someone at Facebook. The results were likely measured against a repository using remotefilelog. And I believe that revision retrieval during working directory updates with remotefilelog will often use a remote store, thus being I/O and not CPU bound. This probably resulted in an overstated performance gain. Since there appears to be a need to enable the thread-based worker with some stores, I've made the flagging of file gets as thread safe configurable. I've made it experimental because I don't want to formalize a boolean flag for this option and because this attribute is best captured against the store implementation. But we don't have a proper store API for this yet. I'd rather cross this bridge later. It is possible there are revlog-based repositories that do benefit from a thread-based worker. I didn't do very comprehensive testing. If there are, we may want to devise a more proper algorithm for whether to use the thread-based worker, including possibly config options to limit the number of threads to use. But until I see evidence that justifies complexity, simplicity wins. Differential Revision: https://phab.mercurial-scm.org/D3963

Pulkit Goyal - - Load All Authors

File last commit:

r28351:42a7301f default


                r38755:be498426

default

Download file

             casesmash.py
        
                    38 lines
            
             | 938 B
            
                | text/x-python
            
             |
                PythonLexer
            
             / contrib / casesmash.py
          
                    History
                
                 |
                  Annotation
                 | Raw
                 |Copy content
                 |Copy permalink

      from __future__ import absolute_import

      import __builtin__

      import os

      from mercurial import (

          util,

      )

      def lowerwrap(scope, funcname):

          f = getattr(scope, funcname)

          def wrap(fname, *args, **kwargs):

              d, base = os.path.split(fname)

              try:

                  files = os.listdir(d or '.')

              except OSError:

                  files = []

              if base in files:

                  return f(fname, *args, **kwargs)

              for fn in files:

                  if fn.lower() == base.lower():

                      return f(os.path.join(d, fn), *args, **kwargs)

              return f(fname, *args, **kwargs)

          scope.__dict__[funcname] = wrap

      def normcase(path):

          return path.lower()

      os.path.normcase = normcase

      for f in 'file open'.split():

          lowerwrap(__builtin__, f)

      for f in "chmod chown open lstat stat remove unlink".split():

          lowerwrap(os, f)

      for f in "exists lexists".split():

          lowerwrap(os.path, f)

      lowerwrap(util, 'posixfile')

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

				from __future__ import absolute_import
				import __builtin__
				import os
				from mercurial import (
				util,
				)

				def lowerwrap(scope, funcname):
				f = getattr(scope, funcname)
				def wrap(fname, args, *kwargs):
				d, base = os.path.split(fname)
				try:
				files = os.listdir(d or '.')
				except OSError:
				files = []
				if base in files:
				return f(fname, args, *kwargs)
				for fn in files:
				if fn.lower() == base.lower():
				return f(os.path.join(d, fn), args, *kwargs)
				return f(fname, args, *kwargs)
				scope.__dict__[funcname] = wrap

				def normcase(path):
				return path.lower()

				os.path.normcase = normcase

				for f in 'file open'.split():
				lowerwrap(__builtin__, f)

				for f in "chmod chown open lstat stat remove unlink".split():
				lowerwrap(os, f)

				for f in "exists lexists".split():
				lowerwrap(os.path, f)

				lowerwrap(util, 'posixfile')