##// END OF EJS Templates
worker: Use buffered input from the pickle stream...
worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

File last commit:

r44058:99e231af default
r44718:cb52e619 stable
Show More
undumprevlog
49 lines | 1.2 KiB | text/plain | TextLexer
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 #!/usr/bin/env python
# Undump a dump from dumprevlog
# $ hg init
# $ undumprevlog < repo.dump
Augie Fackler
undumprevlog: update to valid Python 3 syntax...
r33874 from __future__ import absolute_import, print_function
Pulkit Goyal
py3: make contrib/undumprevlog use absolute_import
r29167
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 import sys
Pulkit Goyal
py3: make contrib/undumprevlog use absolute_import
r29167 from mercurial import (
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 encoding,
Pulkit Goyal
py3: make contrib/undumprevlog use absolute_import
r29167 node,
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 pycompat,
Pulkit Goyal
py3: make contrib/undumprevlog use absolute_import
r29167 revlog,
transaction,
Pierre-Yves David
vfs: use 'vfs' module directly in 'contrib/undumprevlog'...
r31248 vfs as vfsmod,
Pulkit Goyal
py3: make contrib/undumprevlog use absolute_import
r29167 )
Gregory Szorc
black: blacken scripts...
r44058 from mercurial.utils import procutil
Matt Mackall
add simple dump and undump scripts to contrib/
r6433
Adrian Buehlmann
contrib: fix binary file issues with dumprevlog on Windows...
r6466 for fp in (sys.stdin, sys.stdout, sys.stderr):
Yuya Nishihara
procutil: bulk-replace function calls to point to new module
r37138 procutil.setbinary(fp)
Adrian Buehlmann
contrib: fix binary file issues with dumprevlog on Windows...
r6466
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 opener = vfsmod.vfs(b'.', False)
Gregory Szorc
black: blacken scripts...
r44058 tr = transaction.transaction(
sys.stderr.write, opener, {b'store': opener}, b"undump.journal"
)
Mads Kiilerich
tests: run check-code on Python files without .py extension
r19022 while True:
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 l = sys.stdin.readline()
if not l:
break
if l.startswith("file:"):
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 f = encoding.strtolocal(l[6:-1])
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 r = revlog.revlog(opener, f)
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 pycompat.stdout.write(b'%s\n' % f)
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 elif l.startswith("node:"):
n = node.bin(l[6:-1])
elif l.startswith("linkrev:"):
lr = int(l[9:-1])
elif l.startswith("parents:"):
p = l[9:-1].split()
p1 = node.bin(p[0])
p2 = node.bin(p[1])
elif l.startswith("length:"):
length = int(l[8:-1])
Gregory Szorc
black: blacken scripts...
r44058 sys.stdin.readline() # start marker
Matt Harbison
py3: byteify contrib/dumprevlog
r39983 d = encoding.strtolocal(sys.stdin.read(length))
Gregory Szorc
black: blacken scripts...
r44058 sys.stdin.readline() # end marker
Matt Mackall
add simple dump and undump scripts to contrib/
r6433 r.addrevision(d, tr, lr, p1, p2)
tr.close()