##// END OF EJS Templates
procutil: avoid using os.fork() to implement runbgcommand...
procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

File last commit:

r44085:53607fd3 stable
r47651:8759e22f default
Show More
osutilbuild.py
108 lines | 2.5 KiB | text/x-python | PythonLexer
from __future__ import absolute_import
import cffi
ffi = cffi.FFI()
ffi.set_source(
"mercurial.cffi._osutil",
"""
#include <sys/attr.h>
#include <sys/vnode.h>
#include <unistd.h>
#include <fcntl.h>
#include <time.h>
typedef struct val_attrs {
uint32_t length;
attribute_set_t returned;
attrreference_t name_info;
fsobj_type_t obj_type;
struct timespec mtime;
uint32_t accessmask;
off_t datalength;
} __attribute__((aligned(4), packed)) val_attrs_t;
""",
include_dirs=['mercurial'],
)
ffi.cdef(
'''
typedef uint32_t attrgroup_t;
typedef struct attrlist {
uint16_t bitmapcount; /* number of attr. bit sets in list */
uint16_t reserved; /* (to maintain 4-byte alignment) */
attrgroup_t commonattr; /* common attribute group */
attrgroup_t volattr; /* volume attribute group */
attrgroup_t dirattr; /* directory attribute group */
attrgroup_t fileattr; /* file attribute group */
attrgroup_t forkattr; /* fork attribute group */
...;
};
typedef struct attribute_set {
...;
} attribute_set_t;
typedef struct attrreference {
int attr_dataoffset;
int attr_length;
...;
} attrreference_t;
typedef int ... off_t;
typedef struct val_attrs {
uint32_t length;
attribute_set_t returned;
attrreference_t name_info;
uint32_t obj_type;
struct timespec mtime;
uint32_t accessmask;
off_t datalength;
...;
} val_attrs_t;
/* the exact layout of the above struct will be figured out during build time */
typedef int ... time_t;
typedef struct timespec {
time_t tv_sec;
...;
};
int getattrlist(const char* path, struct attrlist * attrList, void * attrBuf,
size_t attrBufSize, unsigned int options);
int getattrlistbulk(int dirfd, struct attrlist * attrList, void * attrBuf,
size_t attrBufSize, uint64_t options);
#define ATTR_BIT_MAP_COUNT ...
#define ATTR_CMN_NAME ...
#define ATTR_CMN_OBJTYPE ...
#define ATTR_CMN_MODTIME ...
#define ATTR_CMN_ACCESSMASK ...
#define ATTR_CMN_ERROR ...
#define ATTR_CMN_RETURNED_ATTRS ...
#define ATTR_FILE_DATALENGTH ...
#define VREG ...
#define VDIR ...
#define VLNK ...
#define VBLK ...
#define VCHR ...
#define VFIFO ...
#define VSOCK ...
#define S_IFMT ...
int open(const char *path, int oflag, int perm);
int close(int);
#define O_RDONLY ...
'''
)
if __name__ == '__main__':
ffi.compile()