##// END OF EJS Templates
memory-usage: fix `hg log --follow --rev R F` space complexity...
memory-usage: fix `hg log --follow --rev R F` space complexity When running `hg log --follow --rev REVS FILES`, the log code will walk the history of all FILES starting from the file revisions that exists in each REVS. Before doing so, it looks if the files actually exists in the target revisions. To do so, it opens the manifest of each revision in REVS to look up if we find the associated items in FILES. Before this changeset this was done in a way that created a changectx for each target revision, keeping them in memory while we look into each file. If the set of REVS is large, this means keeping the manifest for each entry in REVS in memory. That can be largeā€¦ if REV is in the form `::X`, this can quickly become huge and saturate the memory. We have seen usage allocating 2GB per second until memory runs out. So this changeset invert the two loop so that only one revision is kept in memory during the operation. This solve the memory explosion issue.

File last commit:

r44311:8766728d default
r50517:dcb2581e stable
Show More
fncache.cc
69 lines | 1.8 KiB | text/x-c | CppLexer
Augie Fackler
fuzz: new fuzzer for fncache-related functions...
r43422 #include <Python.h>
#include <assert.h>
#include <stdlib.h>
#include <unistd.h>
#include "pyutil.h"
#include <iostream>
#include <string>
extern "C" {
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 static PYCODETYPE *code;
Augie Fackler
fuzz: new fuzzer for fncache-related functions...
r43422
extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv)
{
contrib::initpy(*argv[0]);
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 code = (PYCODETYPE *)Py_CompileString(R"py(
Augie Fackler
fuzz: new fuzzer for fncache-related functions...
r43422 try:
for fn in (
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 parsers.isasciistr,
parsers.asciilower,
parsers.asciiupper,
parsers.encodedir,
parsers.pathencode,
parsers.lowerencode,
Augie Fackler
fuzz: new fuzzer for fncache-related functions...
r43422 ):
try:
fn(data)
except UnicodeDecodeError:
pass # some functions emit this exception
except AttributeError:
# pathencode needs hashlib, which fails to import because the time
# module fails to import. We should try and fix that some day, but
# for now we at least get coverage on non-hashencoded codepaths.
if fn != pathencode:
raise
# uncomment this for debugging exceptions
# except Exception as e:
# raise Exception('%r: %r' % (fn, e))
except Exception as e:
pass
# uncomment this print if you're editing this Python code
# to debug failures.
# print(e)
)py",
Augie Fackler
fuzz: add support for fuzzing under either Python 2 or 3...
r44311 "fuzzer", Py_file_input);
Augie Fackler
fuzz: new fuzzer for fncache-related functions...
r43422 if (!code) {
std::cerr << "failed to compile Python code!" << std::endl;
}
return 0;
}
int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size)
{
PyObject *mtext =
PyBytes_FromStringAndSize((const char *)Data, (Py_ssize_t)Size);
PyObject *locals = PyDict_New();
PyDict_SetItemString(locals, "data", mtext);
PyObject *res = PyEval_EvalCode(code, contrib::pyglobals(), locals);
if (!res) {
PyErr_Print();
}
Py_XDECREF(res);
Py_DECREF(locals);
Py_DECREF(mtext);
return 0; // Non-zero return values are reserved for future use.
}
}