upstream/mercurial-mirror Files · contrib/fuzz/fuzzutil.h

worker: Use buffered input from the pickle stream...

worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

Augie Fackler - - Load All Authors

File last commit:

r44013:6f5c352f default


                r44718:cb52e619

stable

Download file

             fuzzutil.h
        
                    37 lines
            
             | 1002 B
            
                | text/x-c
            
             |
                CLexer
            
             / contrib / fuzz / fuzzutil.h
          
                    History
                
                 |
                  Source
                 | Raw
                 |Copy content
                 |Copy permalink

        Augie Fackler
    
fuzz: extract some common utilities and use modern C++ idioms...

              r38191
            
      #ifndef CONTRIB_FUZZ_FUZZUTIL_H

      #define CONTRIB_FUZZ_FUZZUTIL_H

      #include <iostream>

      #include <memory>

        Augie Fackler
    
fuzzutil: make it possible to use absl when C++17 isn't supported...

              r38192
            
      #include <stdint.h>

      /* Try and use std::optional, but failing that assume we'll have a

       * workable https://abseil.io/ install on the include path to get

       * their backport of std::optional. */

      #ifdef __has_include

      #if __has_include(<optional>) && __cplusplus >= 201703L

        Augie Fackler
    
fuzz: extract some common utilities and use modern C++ idioms...

              r38191
            
      #include <optional>

        Augie Fackler
    
fuzzutil: make it possible to use absl when C++17 isn't supported...

              r38192
            
      #define CONTRIB_FUZZ_HAVE_STD_OPTIONAL

      #endif

      #endif

      #ifdef CONTRIB_FUZZ_HAVE_STD_OPTIONAL

      namespace contrib

      {

      using std::nullopt;

      using std::optional;

      } /* namespace contrib */

      #else

      #include "third_party/absl/types/optional.h"

      namespace contrib

      {

      using absl::nullopt;

      using absl::optional;

      } /* namespace contrib */

      #endif

        Augie Fackler
    
fuzz: extract some common utilities and use modern C++ idioms...

              r38191
            
      /* set DEBUG to 1 for a few debugging prints, or 2 for a lot */

      #define DEBUG 0

      #define LOG(level)                                                             \

      	if (level <= DEBUG)                                                    \

      	std::cout

      #endif /* CONTRIB_FUZZ_FUZZUTIL_H */

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

Augie Fackler fuzz: extract some common utilities and use modern C++ idioms...	r38191	#ifndef CONTRIB_FUZZ_FUZZUTIL_H
		#define CONTRIB_FUZZ_FUZZUTIL_H
		#include <iostream>
		#include <memory>
Augie Fackler fuzzutil: make it possible to use absl when C++17 isn't supported...	r38192	#include <stdint.h>

		/* Try and use std::optional, but failing that assume we'll have a
		* workable https://abseil.io/ install on the include path to get
		* their backport of std::optional. */
		#ifdef __has_include
		#if __has_include(<optional>) && __cplusplus >= 201703L
Augie Fackler fuzz: extract some common utilities and use modern C++ idioms...	r38191	#include <optional>
Augie Fackler fuzzutil: make it possible to use absl when C++17 isn't supported...	r38192	#define CONTRIB_FUZZ_HAVE_STD_OPTIONAL
		#endif
		#endif
		#ifdef CONTRIB_FUZZ_HAVE_STD_OPTIONAL
		namespace contrib
		{
		using std::nullopt;
		using std::optional;
		} /* namespace contrib */
		#else
		#include "third_party/absl/types/optional.h"
		namespace contrib
		{
		using absl::nullopt;
		using absl::optional;
		} /* namespace contrib */
		#endif
Augie Fackler fuzz: extract some common utilities and use modern C++ idioms...	r38191
		/* set DEBUG to 1 for a few debugging prints, or 2 for a lot */
		#define DEBUG 0
		#define LOG(level) \
		if (level <= DEBUG) \
		std::cout

		#endif /* CONTRIB_FUZZ_FUZZUTIL_H */