##// END OF EJS Templates
worker: Use buffered input from the pickle stream...
worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

File last commit:

r44311:8766728d default
r44718:cb52e619 stable
Show More
revlog.cc
61 lines | 1.6 KiB | text/x-c | CppLexer
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050 #include <Python.h>
#include <assert.h>
#include <stdlib.h>
#include <unistd.h>
#include <string>
#include "pyutil.h"
extern "C" {
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 static PYCODETYPE *code;
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050
extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv)
{
contrib::initpy(*argv[0]);
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 code = (PYCODETYPE *)Py_CompileString(R"py(
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050 for inline in (True, False):
try:
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 index, cache = parsers.parse_index2(data, inline)
Augie Fackler
fuzz: exercise more of the revlog API...
r41340 index.slicechunktodensity(list(range(len(index))), 0.5, 262144)
Augie Fackler
fuzz: exercise a little more revlog code...
r43421 index.stats()
index.findsnapshots({}, 0)
10 in index
Augie Fackler
fuzz: exercise more of the revlog API...
r41340 for rev in range(len(index)):
Augie Fackler
fuzz: exercise a little more revlog code...
r43421 index.reachableroots(0, [len(index)-1], [rev])
Augie Fackler
fuzz: exercise more of the revlog API...
r41340 node = index[rev][7]
partial = index.shortest(node)
index.partialmatch(node[:partial])
Augie Fackler
fuzz: exercise a little more revlog code...
r43421 index.deltachain(rev, None, True)
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050 except Exception as e:
pass
# uncomment this print if you're editing this Python code
# to debug failures.
# print e
)py",
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 "fuzzer", Py_file_input);
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050 return 0;
}
int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size)
{
Augie Fackler
fuzz: don't allow enormous revlog inputs either...
r41339 // Don't allow fuzzer inputs larger than 60k, since we'll just bog
// down and not accomplish much.
if (Size > 60000) {
return 0;
}
Augie Fackler
fuzz: new fuzzer for revlog's parse_index2 method...
r41050 PyObject *text =
PyBytes_FromStringAndSize((const char *)Data, (Py_ssize_t)Size);
PyObject *locals = PyDict_New();
PyDict_SetItemString(locals, "data", text);
PyObject *res = PyEval_EvalCode(code, contrib::pyglobals(), locals);
if (!res) {
PyErr_Print();
}
Py_XDECREF(res);
Py_DECREF(locals);
Py_DECREF(text);
return 0; // Non-zero return values are reserved for future use.
}
}