upstream/mercurial-mirror Files · contrib/fuzz/revlog_corpus.py

worker: Use buffered input from the pickle stream...

worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

Kyle Lippincott - - Load All Authors

File last commit:

r44288:ba84a1ae default


                r44718:cb52e619

stable

Download file

             revlog_corpus.py
        
                    25 lines
            
             | 852 B
            
                | text/x-python
            
             |
                PythonLexer
            
             / contrib / fuzz / revlog_corpus.py
          
                    History
                
                 |
                  Source
                 | Raw
                 |Copy content
                 |Copy permalink

        Augie Fackler
    
fuzz: remove debug prints from revlog_corpus.py...

              r44260
            
      from __future__ import absolute_import

        Augie Fackler
    
fuzz: new fuzzer for revlog's parse_index2 method...

              r41050
            
      import argparse

      import os

      import zipfile

      ap = argparse.ArgumentParser()

      ap.add_argument("out", metavar="some.zip", type=str, nargs=1)

      args = ap.parse_args()

        Augie Fackler
    
formatting: blacken the codebase...

              r43346
            
      reporoot = os.path.normpath(os.path.join(os.path.dirname(__file__), '..', '..'))

        Augie Fackler
    
fuzz: new fuzzer for revlog's parse_index2 method...

              r41050
            
      # typically a standalone index

      changelog = os.path.join(reporoot, '.hg', 'store', '00changelog.i')

      # an inline revlog with only a few revisions

      contributing = os.path.join(

        Augie Fackler
    
formatting: blacken the codebase...

              r43346
            
          reporoot, '.hg', 'store', 'data', 'contrib', 'fuzz', 'mpatch.cc.i'

      )

        Augie Fackler
    
fuzz: new fuzzer for revlog's parse_index2 method...

              r41050
            
      with zipfile.ZipFile(args.out[0], "w", zipfile.ZIP_STORED) as zf:

          if os.path.exists(changelog):

        Kyle Lippincott
    
fuzz: fix test-fuzz-targets.t to run with python3...

              r44288
            
              with open(changelog, 'rb') as f:

        Augie Fackler
    
fuzz: new fuzzer for revlog's parse_index2 method...

              r41050
            
                  zf.writestr("00changelog.i", f.read())

          if os.path.exists(contributing):

        Kyle Lippincott
    
fuzz: fix test-fuzz-targets.t to run with python3...

              r44288
            
              with open(contributing, 'rb') as f:

        Augie Fackler
    
fuzz: new fuzzer for revlog's parse_index2 method...

              r41050
            
                  zf.writestr("contributing.i", f.read())

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

Augie Fackler fuzz: remove debug prints from revlog_corpus.py...	r44260	from __future__ import absolute_import
Augie Fackler fuzz: new fuzzer for revlog's parse_index2 method...	r41050
		import argparse
		import os
		import zipfile

		ap = argparse.ArgumentParser()
		ap.add_argument("out", metavar="some.zip", type=str, nargs=1)
		args = ap.parse_args()

Augie Fackler formatting: blacken the codebase...	r43346	reporoot = os.path.normpath(os.path.join(os.path.dirname(__file__), '..', '..'))
Augie Fackler fuzz: new fuzzer for revlog's parse_index2 method...	r41050	# typically a standalone index
		changelog = os.path.join(reporoot, '.hg', 'store', '00changelog.i')
		# an inline revlog with only a few revisions
		contributing = os.path.join(
Augie Fackler formatting: blacken the codebase...	r43346	reporoot, '.hg', 'store', 'data', 'contrib', 'fuzz', 'mpatch.cc.i'
		)
Augie Fackler fuzz: new fuzzer for revlog's parse_index2 method...	r41050
		with zipfile.ZipFile(args.out[0], "w", zipfile.ZIP_STORED) as zf:
		if os.path.exists(changelog):
Kyle Lippincott fuzz: fix test-fuzz-targets.t to run with python3...	r44288	with open(changelog, 'rb') as f:
Augie Fackler fuzz: new fuzzer for revlog's parse_index2 method...	r41050	zf.writestr("00changelog.i", f.read())
		if os.path.exists(contributing):
Kyle Lippincott fuzz: fix test-fuzz-targets.t to run with python3...	r44288	with open(contributing, 'rb') as f:
Augie Fackler fuzz: new fuzzer for revlog's parse_index2 method...	r41050	zf.writestr("contributing.i", f.read())