##// END OF EJS Templates
worker: Use buffered input from the pickle stream...
worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

File last commit:

r43346:2372284d default
r44718:cb52e619 stable
Show More
__init__.py
57 lines | 1.5 KiB | text/x-python | PythonLexer
Gregory Szorc
automation: perform tasks on remote machines...
r42191 # __init__.py - High-level automation interfaces
#
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native.
import pathlib
import secrets
Augie Fackler
formatting: blacken the codebase...
r43346 from .aws import AWSConnection
Gregory Szorc
automation: perform tasks on remote machines...
r42191
class HGAutomation:
"""High-level interface for Mercurial automation.
Holds global state, provides access to other primitives, etc.
"""
def __init__(self, state_path: pathlib.Path):
self.state_path = state_path
state_path.mkdir(exist_ok=True)
def default_password(self):
"""Obtain the default password to use for remote machines.
A new password will be generated if one is not stored.
"""
p = self.state_path / 'default-password'
try:
with p.open('r', encoding='ascii') as fh:
data = fh.read().strip()
if data:
return data
except FileNotFoundError:
pass
password = secrets.token_urlsafe(24)
with p.open('w', encoding='ascii') as fh:
fh.write(password)
fh.write('\n')
p.chmod(0o0600)
return password
Augie Fackler
formatting: blacken the codebase...
r43346 def aws_connection(self, region: str, ensure_ec2_state: bool = True):
Gregory Szorc
automation: perform tasks on remote machines...
r42191 """Obtain an AWSConnection instance bound to a specific region."""
Gregory Szorc
automation: don't create resources when deleting things...
r42463 return AWSConnection(self, region, ensure_ec2_state=ensure_ec2_state)