##// END OF EJS Templates
worker: Use buffered input from the pickle stream...
worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

File last commit:

r44253:01ec70a8 default
r44718:cb52e619 stable
Show More
standalone_fuzz_target_runner.cc
45 lines | 1.5 KiB | text/x-c | CppLexer
/ contrib / fuzz / standalone_fuzz_target_runner.cc
Augie Fackler
fuzz: use a more standard approach to allow local builds of fuzzers...
r44265 // Copyright 2017 Google Inc. All Rights Reserved.
// Licensed under the Apache License, Version 2.0 (the "License");
// Example of a standalone runner for "fuzz targets".
// It reads all files passed as parameters and feeds their contents
// one by one into the fuzz target (LLVMFuzzerTestOneInput).
// This runner does not do any fuzzing, but allows us to run the fuzz target
// on the test corpus (e.g. "do_stuff_test_data") or on a single file,
// e.g. the one that comes from a bug report.
#include <cassert>
#include <fstream>
#include <iostream>
#include <vector>
// Forward declare the "fuzz target" interface.
// We deliberately keep this inteface simple and header-free.
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);
Augie Fackler
fuzz: make standalone_fuzz_target_runner call LLVMFuzzerInitialize...
r44266 extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv);
Augie Fackler
fuzz: use a more standard approach to allow local builds of fuzzers...
r44265 int main(int argc, char **argv)
{
Augie Fackler
fuzz: make standalone_fuzz_target_runner call LLVMFuzzerInitialize...
r44266 LLVMFuzzerInitialize(&argc, &argv);
Augie Fackler
fuzz: use a more standard approach to allow local builds of fuzzers...
r44265 for (int i = 1; i < argc; i++) {
std::ifstream in(argv[i]);
in.seekg(0, in.end);
size_t length = in.tellg();
in.seekg(0, in.beg);
std::cout << "Reading " << length << " bytes from " << argv[i]
<< std::endl;
// Allocate exactly length bytes so that we reliably catch
// buffer overflows.
std::vector<char> bytes(length);
in.read(bytes.data(), bytes.size());
assert(in);
LLVMFuzzerTestOneInput(
reinterpret_cast<const uint8_t *>(bytes.data()),
bytes.size());
std::cout << "Execution successful" << std::endl;
}
return 0;
}
// no-check-code since this is from a third party