##// END OF EJS Templates
mercurial: implement a source transforming module loader on Python 3...
mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

File last commit:

r13302:a4e0908c default
r29550:1c22400d default
Show More
hgsh.c
440 lines | 8.6 KiB | text/x-c | CLexer
/*
* hgsh.c - restricted login shell for mercurial
*
* Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
*
* This software may be used and distributed according to the terms of the
* GNU General Public License, incorporated herein by reference.
*
* this program is login shell for dedicated mercurial user account. it
* only allows few actions:
*
* 1. run hg in server mode on specific repository. no other hg commands
* are allowed. we try to verify that repo to be accessed exists under
* given top-level directory.
*
* 2. (optional) forward ssh connection from firewall/gateway machine to
* "real" mercurial host, to let users outside intranet pull and push
* changes through firewall.
*
* 3. (optional) run normal shell, to allow to "su" to mercurial user, use
* "sudo" to run programs as that user, or run cron jobs as that user.
*
* only tested on linux yet. patches for non-linux systems welcome.
*/
#ifndef _GNU_SOURCE
#define _GNU_SOURCE /* for asprintf */
#endif
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sysexits.h>
#include <unistd.h>
/*
* user config.
*
* if you see a hostname below, just use first part of hostname. example,
* if you have host named foo.bar.com, use "foo".
*/
/*
* HG_GATEWAY: hostname of gateway/firewall machine that people outside your
* intranet ssh into if they need to ssh to other machines. if you do not
* have such machine, set to NULL.
*/
#ifndef HG_GATEWAY
#define HG_GATEWAY "gateway"
#endif
/*
* HG_HOST: hostname of mercurial server. if any machine is allowed, set to
* NULL.
*/
#ifndef HG_HOST
#define HG_HOST "mercurial"
#endif
/*
* HG_USER: username to log in from HG_GATEWAY to HG_HOST. if gateway and
* host username are same, set to NULL.
*/
#ifndef HG_USER
#define HG_USER "hg"
#endif
/*
* HG_ROOT: root of tree full of mercurial repos. if you do not want to
* validate location of repo when someone is try to access, set to NULL.
*/
#ifndef HG_ROOT
#define HG_ROOT "/home/hg/repos"
#endif
/*
* HG: path to the mercurial executable to run.
*/
#ifndef HG
#define HG "/home/hg/bin/hg"
#endif
/*
* HG_SHELL: shell to use for actions like "sudo" and "su" access to
* mercurial user, and cron jobs. if you want to make these things
* impossible, set to NULL.
*/
#ifndef HG_SHELL
#define HG_SHELL NULL
/* #define HG_SHELL "/bin/bash" */
#endif
/*
* HG_HELP: some way for users to get support if they have problem. if they
* should not get helpful message, set to NULL.
*/
#ifndef HG_HELP
#define HG_HELP "please contact support@example.com for help."
#endif
/*
* SSH: path to ssh executable to run, if forwarding from HG_GATEWAY to
* HG_HOST. if you want to use rsh instead (why?), you need to modify
* arguments it is called with. see forward_through_gateway.
*/
#ifndef SSH
#define SSH "/usr/bin/ssh"
#endif
/*
* tell whether to print command that is to be executed. useful for
* debugging. should not interfere with mercurial operation, since
* mercurial only cares about stdin and stdout, and this prints to stderr.
*/
static const int debug = 0;
static void print_cmdline(int argc, char **argv)
{
FILE *fp = stderr;
int i;
fputs("command: ", fp);
for (i = 0; i < argc; i++) {
char *spc = strpbrk(argv[i], " \t\r\n");
if (spc) {
fputc('\'', fp);
}
fputs(argv[i], fp);
if (spc) {
fputc('\'', fp);
}
if (i < argc - 1) {
fputc(' ', fp);
}
}
fputc('\n', fp);
fflush(fp);
}
static void usage(const char *reason, int exitcode)
{
char *hg_help = HG_HELP;
if (reason) {
fprintf(stderr, "*** Error: %s.\n", reason);
}
fprintf(stderr, "*** This program has been invoked incorrectly.\n");
if (hg_help) {
fprintf(stderr, "*** %s\n", hg_help);
}
exit(exitcode ? exitcode : EX_USAGE);
}
/*
* run on gateway host to make another ssh connection, to "real" mercurial
* server. it sends its command line unmodified to far end.
*
* never called if HG_GATEWAY is NULL.
*/
static void forward_through_gateway(int argc, char **argv)
{
char *ssh = SSH;
char *hg_host = HG_HOST;
char *hg_user = HG_USER;
char **nargv = alloca((10 + argc) * sizeof(char *));
int i = 0, j;
nargv[i++] = ssh;
nargv[i++] = "-q";
nargv[i++] = "-T";
nargv[i++] = "-x";
if (hg_user) {
nargv[i++] = "-l";
nargv[i++] = hg_user;
}
nargv[i++] = hg_host;
/*
* sshd called us with added "-c", because it thinks we are a shell.
* drop it if we find it.
*/
j = 1;
if (j < argc && strcmp(argv[j], "-c") == 0) {
j++;
}
for (; j < argc; i++, j++) {
nargv[i] = argv[j];
}
nargv[i] = NULL;
if (debug) {
print_cmdline(i, nargv);
}
execv(ssh, nargv);
perror(ssh);
exit(EX_UNAVAILABLE);
}
/*
* run shell. let administrator "su" to mercurial user's account to do
* administrative works.
*
* never called if HG_SHELL is NULL.
*/
static void run_shell(int argc, char **argv)
{
char *hg_shell = HG_SHELL;
char **nargv;
char *c;
int i;
nargv = alloca((argc + 3) * sizeof(char *));
c = strrchr(hg_shell, '/');
/* tell "real" shell it is login shell, if needed. */
if (argv[0][0] == '-' && c) {
nargv[0] = strdup(c);
if (nargv[0] == NULL) {
perror("malloc");
exit(EX_OSERR);
}
nargv[0][0] = '-';
} else {
nargv[0] = hg_shell;
}
for (i = 1; i < argc; i++) {
nargv[i] = argv[i];
}
nargv[i] = NULL;
if (debug) {
print_cmdline(i, nargv);
}
execv(hg_shell, nargv);
perror(hg_shell);
exit(EX_OSFILE);
}
enum cmdline {
hg_init,
hg_serve,
};
/*
* attempt to verify that a directory is really a hg repo, by testing
* for the existence of a subdirectory.
*/
static int validate_repo(const char *repo_root, const char *subdir)
{
char *abs_path;
struct stat st;
int ret;
if (asprintf(&abs_path, "%s.hg/%s", repo_root, subdir) == -1) {
ret = -1;
goto bail;
}
/* verify that we really are looking at valid repo. */
if (stat(abs_path, &st) == -1) {
ret = 0;
} else {
ret = 1;
}
bail:
return ret;
}
/*
* paranoid wrapper, runs hg executable in server mode.
*/
static void serve_data(int argc, char **argv)
{
char *hg_root = HG_ROOT;
char *repo, *repo_root;
enum cmdline cmd;
char *nargv[6];
size_t repolen;
int i;
/*
* check argv for looking okay. we should be invoked with argv
* resembling like this:
*
* hgsh
* -c
* hg -R some/path serve --stdio
*
* the "-c" is added by sshd, because it thinks we are login shell.
*/
if (argc != 3) {
goto badargs;
}
if (strcmp(argv[1], "-c") != 0) {
goto badargs;
}
if (sscanf(argv[2], "hg init %as", &repo) == 1) {
cmd = hg_init;
}
else if (sscanf(argv[2], "hg -R %as serve --stdio", &repo) == 1) {
cmd = hg_serve;
} else {
goto badargs;
}
repolen = repo ? strlen(repo) : 0;
if (repolen == 0) {
goto badargs;
}
if (hg_root) {
if (asprintf(&repo_root, "%s/%s/", hg_root, repo) == -1) {
goto badargs;
}
/*
* attempt to stop break out from inside the
* repository tree. could do something more clever
* here, because e.g. we could traverse a symlink that
* looks safe, but really breaks us out of tree.
*/
if (strstr(repo_root, "/../") != NULL) {
goto badargs;
}
/* only hg init expects no repo. */
if (cmd != hg_init) {
int valid;
valid = validate_repo(repo_root, "data");
if (valid == -1) {
goto badargs;
}
if (valid == 0) {
valid = validate_repo(repo_root, "store");
if (valid == -1) {
goto badargs;
}
}
if (valid == 0) {
perror(repo);
exit(EX_DATAERR);
}
}
if (chdir(hg_root) == -1) {
perror(hg_root);
exit(EX_SOFTWARE);
}
}
i = 0;
switch (cmd) {
case hg_serve:
nargv[i++] = HG;
nargv[i++] = "-R";
nargv[i++] = repo;
nargv[i++] = "serve";
nargv[i++] = "--stdio";
break;
case hg_init:
nargv[i++] = HG;
nargv[i++] = "init";
nargv[i++] = repo;
break;
}
nargv[i] = NULL;
if (debug) {
print_cmdline(i, nargv);
}
execv(HG, nargv);
perror(HG);
exit(EX_UNAVAILABLE);
badargs:
/* print useless error message. */
usage("invalid arguments", EX_DATAERR);
}
int main(int argc, char **argv)
{
char host[1024];
char *c;
if (gethostname(host, sizeof(host)) == -1) {
perror("gethostname");
exit(EX_OSERR);
}
if ((c = strchr(host, '.')) != NULL) {
*c = '\0';
}
if (getenv("SSH_CLIENT")) {
char *hg_gateway = HG_GATEWAY;
char *hg_host = HG_HOST;
if (hg_gateway && strcmp(host, hg_gateway) == 0) {
forward_through_gateway(argc, argv);
}
if (hg_host && strcmp(host, hg_host) != 0) {
usage("invoked on unexpected host", EX_USAGE);
}
serve_data(argc, argv);
} else if (HG_SHELL) {
run_shell(argc, argv);
} else {
usage("invalid arguments", EX_DATAERR);
}
return 0;
}