upstream/mercurial-mirror Files · mercurial/pure/diffhelpers.py

hgweb: tweak zlib chunking behavior...

hgweb: tweak zlib chunking behavior When doing streaming compression with zlib, zlib appears to emit chunks with data after ~20-30kb on average is available. In other words, most calls to compress() return an empty string. On the mozilla-unified repo, only 48,433 of 921,167 (5.26%) of calls to compress() returned data. In other words, we were sending hundreds of thousands of empty chunks via a generator where they touched who knows how many frames (my guess is millions). Filtering out the empty chunks from the generator cuts down on overhead. In addition, we were previously feeding 8kb chunks into zlib compression. Since this function tends to emit *compressed* data after 20-30kb is available, it would take several calls before data was produced. We increase the amount of data fed in at a time to 32kb. This reduces the number of calls to compress() from 921,167 to 115,146. It also reduces the number of output chunks from 48,433 to 31,377. This does increase the average output chunk size by a little. But I don't think this will matter in most scenarios. The combination of these 2 changes appears to shave ~6s CPU time or ~3% from a server serving the mozilla-unified repo.

Gregory Szorc - - Load All Authors

File last commit:

r27336:80214358 default


                r29792:58467204

default

Download file

             diffhelpers.py
        
                    62 lines
            
             | 1.6 KiB
            
                | text/x-python
            
             |
                PythonLexer
            
             / mercurial / pure / diffhelpers.py
          
                    History
                
                 |
                  Annotation
                 | Raw
                 |Copy content
                 |Copy permalink

      # diffhelpers.py - pure Python implementation of diffhelpers.c

      #

      # Copyright 2009 Matt Mackall <mpm@selenic.com> and others

      #

      # This software may be used and distributed according to the terms of the

      # GNU General Public License version 2 or any later version.

      from __future__ import absolute_import

      def addlines(fp, hunk, lena, lenb, a, b):

          while True:

              todoa = lena - len(a)

              todob = lenb - len(b)

              num = max(todoa, todob)

              if num == 0:

                  break

              for i in xrange(num):

                  s = fp.readline()

                  c = s[0]

                  if s == "\\ No newline at end of file\n":

                      fix_newline(hunk, a, b)

                      continue

                  if c == "\n":

                      # Some patches may be missing the control char

                      # on empty lines. Supply a leading space.

                      s = " \n"

                  hunk.append(s)

                  if c == "+":

                      b.append(s[1:])

                  elif c == "-":

                      a.append(s)

                  else:

                      b.append(s[1:])

                      a.append(s)

          return 0

      def fix_newline(hunk, a, b):

          l = hunk[-1]

          # tolerate CRLF in last line

          if l.endswith('\r\n'):

              hline = l[:-2]

          else:

              hline = l[:-1]

          c = hline[0]

          if c in " +":

              b[-1] = hline[1:]

          if c in " -":

              a[-1] = hline

          hunk[-1] = hline

          return 0

      def testhunk(a, b, bstart):

          alen = len(a)

          blen = len(b)

          if alen > blen - bstart:

              return -1

          for i in xrange(alen):

              if a[i][1:] != b[i + bstart]:

                  return -1

          return 0

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

				# diffhelpers.py - pure Python implementation of diffhelpers.c
				#
				# Copyright 2009 Matt Mackall <mpm@selenic.com> and others
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				from __future__ import absolute_import

				def addlines(fp, hunk, lena, lenb, a, b):
				while True:
				todoa = lena - len(a)
				todob = lenb - len(b)
				num = max(todoa, todob)
				if num == 0:
				break
				for i in xrange(num):
				s = fp.readline()
				c = s[0]
				if s == "\\ No newline at end of file\n":
				fix_newline(hunk, a, b)
				continue
				if c == "\n":
				# Some patches may be missing the control char
				# on empty lines. Supply a leading space.
				s = " \n"
				hunk.append(s)
				if c == "+":
				b.append(s[1:])
				elif c == "-":
				a.append(s)
				else:
				b.append(s[1:])
				a.append(s)
				return 0

				def fix_newline(hunk, a, b):
				l = hunk[-1]
				# tolerate CRLF in last line
				if l.endswith('\r\n'):
				hline = l[:-2]
				else:
				hline = l[:-1]
				c = hline[0]

				if c in " +":
				b[-1] = hline[1:]
				if c in " -":
				a[-1] = hline
				hunk[-1] = hline
				return 0


				def testhunk(a, b, bstart):
				alen = len(a)
				blen = len(b)
				if alen > blen - bstart:
				return -1
				for i in xrange(alen):
				if a[i][1:] != b[i + bstart]:
				return -1
				return 0