upstream/mercurial-mirror Files · contrib/memory.py

merge: perform background file closing in batchget...

merge: perform background file closing in batchget As demonstrated with stream clones, closing files on background threads on Windows can yield a significant speedup because closing files that have been created/appended to is slow on Windows/NTFS. Working directory updates can write thousands of files. Therefore it is susceptible to excessive slowness on Windows due to slow file closes. This patch enables background file closing when performing working directory file writes. The impact when performing an `hg up tip` on mozilla-central (136,357 files) from an empty working directory is significant: Before: 535s (8:55) After: 133s (2:13) Delta: -402s (6:42) That's a 4x speedup! By comparison, that same machine can perform the same operation in ~15s on Linux. So Windows went from ~35x to ~9x slower. Not bad but there's still work to do. As a reminder, background file closing is only activated on Windows because it is only beneficial on that platform. So this patch shouldn't change non-Windows behavior at all. It's worth noting that non-Windows systems perform working directory updates with multiple processes. Unfortunately, worker.py doesn't yet support Windows. So, there is still plenty of room for making working directory updates faster on Windows. Even if multiple processes are used on Windows, I believe background file closing will still provide a benefit, as individual processes will still be slowed down by the file close bottleneck (assuming the I/O system isn't saturated).

Bryan O'Sullivan - - Load All Authors

File last commit:

r27795:3e0d27d2 default


                r28200:588695cc

default

Download file

             memory.py
        
                    31 lines
            
             | 1016 B
            
                | text/x-python
            
             |
                PythonLexer
            
             / contrib / memory.py
          
                    History
                
                 |
                  Annotation
                 | Raw
                 |Copy content
                 |Copy permalink

      # memory.py - track memory usage

      #

      # Copyright 2009 Matt Mackall <mpm@selenic.com> and others

      #

      # This software may be used and distributed according to the terms of the

      # GNU General Public License version 2 or any later version.

      '''helper extension to measure memory usage

      Reads current and peak memory usage from ``/proc/self/status`` and

      prints it to ``stderr`` on exit.

      '''

      import atexit

      def memusage(ui):

          """Report memory usage of the current process."""

          result = {'peak': 0, 'rss': 0}

          with open('/proc/self/status', 'r') as status:

              # This will only work on systems with a /proc file system

              # (like Linux).

              for line in status:

                  parts = line.split()

                  key = parts[0][2:-1].lower()

                  if key in result:

                      result[key] = int(parts[1])

          ui.write_err(", ".join(["%s: %.1f MiB" % (key, value / 1024.0)

                                  for key, value in result.iteritems()]) + "\n")

      def extsetup(ui):

          atexit.register(memusage, ui)

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

				# memory.py - track memory usage
				#
				# Copyright 2009 Matt Mackall <mpm@selenic.com> and others
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				'''helper extension to measure memory usage

				Reads current and peak memory usage from ``/proc/self/status`` and
				prints it to ``stderr`` on exit.
				'''

				import atexit

				def memusage(ui):
				"""Report memory usage of the current process."""
				result = {'peak': 0, 'rss': 0}
				with open('/proc/self/status', 'r') as status:
				# This will only work on systems with a /proc file system
				# (like Linux).
				for line in status:
				parts = line.split()
				key = parts[0][2:-1].lower()
				if key in result:
				result[key] = int(parts[1])
				ui.write_err(", ".join(["%s: %.1f MiB" % (key, value / 1024.0)
				for key, value in result.iteritems()]) + "\n")

				def extsetup(ui):
				atexit.register(memusage, ui)