##// END OF EJS Templates
memory-usage: fix `hg log --follow --rev R F` space complexity...
memory-usage: fix `hg log --follow --rev R F` space complexity When running `hg log --follow --rev REVS FILES`, the log code will walk the history of all FILES starting from the file revisions that exists in each REVS. Before doing so, it looks if the files actually exists in the target revisions. To do so, it opens the manifest of each revision in REVS to look up if we find the associated items in FILES. Before this changeset this was done in a way that created a changectx for each target revision, keeping them in memory while we look into each file. If the set of REVS is large, this means keeping the manifest for each entry in REVS in memory. That can be largeā€¦ if REV is in the form `::X`, this can quickly become huge and saturate the memory. We have seen usage allocating 2GB per second until memory runs out. So this changeset invert the two loop so that only one revision is kept in memory during the operation. This solve the memory explosion issue.

File last commit:

r45488:ee5f27d7 default
r50517:dcb2581e stable
Show More
pyutil.cc
76 lines | 1.7 KiB | text/x-c | CppLexer
#include "pyutil.h"
#include <iostream>
#include <string>
namespace contrib
{
#if PY_MAJOR_VERSION >= 3
#define HG_FUZZER_PY3 1
PyMODINIT_FUNC PyInit_parsers(void);
#else
PyMODINIT_FUNC initparsers(void);
#endif
static char cpypath[8192] = "\0";
static PyObject *mainmod;
static PyObject *globals;
void initpy(const char *cselfpath)
{
#ifdef HG_FUZZER_PY3
const std::string subdir = "/sanpy/lib/python3.8";
#else
const std::string subdir = "/sanpy/lib/python2.7";
#endif
/* HACK ALERT: we need a full Python installation built without
pymalloc and with ASAN, so we dump one in
$OUT/sanpy/lib/python2.7. This helps us wire that up. */
std::string selfpath(cselfpath);
std::string pypath;
auto pos = selfpath.rfind("/");
if (pos == std::string::npos) {
char wd[8192];
if (!getcwd(wd, 8192)) {
std::cerr << "Failed to call getcwd: errno " << errno
<< std::endl;
exit(1);
}
pypath = std::string(wd) + subdir;
} else {
pypath = selfpath.substr(0, pos) + subdir;
}
strncpy(cpypath, pypath.c_str(), pypath.size());
setenv("PYTHONPATH", cpypath, 1);
setenv("PYTHONNOUSERSITE", "1", 1);
/* prevent Python from looking up users in the fuzz environment */
setenv("PYTHONUSERBASE", cpypath, 1);
#ifdef HG_FUZZER_PY3
std::wstring wcpypath(pypath.begin(), pypath.end());
Py_SetPythonHome(wcpypath.c_str());
#else
Py_SetPythonHome(cpypath);
#endif
Py_InitializeEx(0);
mainmod = PyImport_AddModule("__main__");
globals = PyModule_GetDict(mainmod);
#ifdef HG_FUZZER_PY3
PyObject *mod = PyInit_parsers();
#else
initparsers();
PyObject *mod = PyImport_ImportModule("parsers");
#endif
PyDict_SetItemString(globals, "parsers", mod);
}
PyObject *pyglobals()
{
return globals;
}
} // namespace contrib