##// END OF EJS Templates
worker: Use buffered input from the pickle stream...
worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

File last commit:

r44058:99e231af default
r44718:cb52e619 stable
Show More
dumprevlog
47 lines | 1.1 KiB | text/plain | TextLexer
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 #!/usr/bin/env python
# Dump revlogs as raw data stream
# $ find .hg/store/ -name "*.i" | xargs dumprevlog > repo.dump
Pulkit Goyal
py3: make contrib/dumprevlog use print_function
r29166 from __future__ import absolute_import, print_function
Pulkit Goyal
py3: make contrib/dumprevlog use absolute_import
r29165
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 import sys
Pulkit Goyal
py3: make contrib/dumprevlog use absolute_import
r29165 from mercurial import (
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 encoding,
Pulkit Goyal
py3: make contrib/dumprevlog use absolute_import
r29165 node,
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 pycompat,
Pulkit Goyal
py3: make contrib/dumprevlog use absolute_import
r29165 revlog,
Yuya Nishihara
procutil: bulk-replace function calls to point to new module
r37138 )
Gregory Szorc
black: blacken scripts...
r44058 from mercurial.utils import procutil
Adrian Buehlmann
contrib: fix binary file issues with dumprevlog on Windows...
r6466
for fp in (sys.stdin, sys.stdout, sys.stderr):
Yuya Nishihara
procutil: bulk-replace function calls to point to new module
r37138 procutil.setbinary(fp)
Matt Mackall
add simple dump and undump scripts to contrib/
r6433
Gregory Szorc
black: blacken scripts...
r44058
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 def binopen(path, mode=b'rb'):
if b'b' not in mode:
mode = mode + b'b'
return open(path, pycompat.sysstr(mode))
Gregory Szorc
black: blacken scripts...
r44058
vfs: give all vfs an options attribute by default...
r43295 binopen.options = {}
Matt Harbison
py3: byteify contrib/dumprevlog
r39983
Gregory Szorc
black: blacken scripts...
r44058
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 def printb(data, end=b'\n'):
sys.stdout.flush()
pycompat.stdout.write(data + end)
Boris Feld
dumprevlog: handle being passed a mode parameter...
r35982
Gregory Szorc
black: blacken scripts...
r44058
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 for f in sys.argv[1:]:
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 r = revlog.revlog(binopen, encoding.strtolocal(f))
Pulkit Goyal
py3: make contrib/dumprevlog use print_function
r29166 print("file:", f)
Matt Mackall
add __len__ and __iter__ methods to repo and revlog
r6750 for i in r:
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 n = r.node(i)
p = r.parents(n)
d = r.revision(n)
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 printb(b"node: %s" % node.hex(n))
printb(b"linkrev: %d" % r.linkrev(i))
printb(b"parents: %s %s" % (node.hex(p[0]), node.hex(p[1])))
printb(b"length: %d" % len(d))
printb(b"-start-")
printb(d)
printb(b"-end-")