##// END OF EJS Templates
mercurial: implement a source transforming module loader on Python 3...
mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

File last commit:

r27878:e7bd55db default
r29550:1c22400d default
Show More
buildrpm
161 lines | 4.0 KiB | text/plain | TextLexer
Mathias De Maré
buildrpm: use bash shebang, since we use bash features in the script...
r27878 #!/bin/bash -e
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564 #
Mads Kiilerich
buildrpm: various minor cleanup
r21638 # Build a Mercurial RPM from the current repo
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564 #
Mads Kiilerich
contrib/buildrpm: Support python 2.4 and 2.6
r8867 # Tested on
Mads Kiilerich
buildrpm: various minor cleanup
r21638 # - Fedora 20
# - CentOS 5
# - centOS 6
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564
Augie Fackler
packaging: extract packagelib for common code from builddeb and buildrpm
r24972 . $(dirname $0)/packagelib.sh
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 BUILD=1
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 RPMBUILDDIR="$PWD/rpmbuild"
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 while [ "$1" ]; do
case "$1" in
--prepare )
shift
BUILD=
;;
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 --withpython | --with-python)
shift
Mads Kiilerich
contrib: offer Python 2.7.10
r26735 PYTHONVER=2.7.10
PYTHONMD5=d7547558fd673bd9d38e2108c6b42521
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 ;;
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 --rpmbuilddir )
shift
RPMBUILDDIR="$1"
shift
;;
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 * )
echo "Invalid parameter $1!" 1>&2
exit 1
;;
esac
done
Gilles Moris
buildrpm: enable to start the script from anywhere...
r9811 cd "`dirname $0`/.."
Mads Kiilerich
buildrpm: complain when hg command isn't available...
r7431
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 specfile=$PWD/contrib/mercurial.spec
Gilles Moris
buildrpm: cleanup script
r9812 if [ ! -f $specfile ]; then
echo "Cannot find $specfile!" 1>&2
exit 1
fi
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564
Gilles Moris
buildrpm: cleanup script
r9812 if [ ! -d .hg ]; then
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564 echo 'You are not inside a Mercurial repository!' 1>&2
exit 1
fi
Augie Fackler
packaging: extract packagelib for common code from builddeb and buildrpm
r24972 gethgversion
Gilles Moris
buildrpm: build from working dir parent and use hg version for RPM versioning...
r9809
Augie Fackler
packaging: rework version detection and declaration (issue4912)...
r26833 # TODO: handle distance/node set, and type set
if [ -z "$type" ] ; then
release=1
else
release=0.9_$type
fi
if [ -n "$distance" ] ; then
release=$release+$distance_$node
fi
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 if [ "$PYTHONVER" ]; then
release=$release+$PYTHONVER
RPMPYTHONVER=$PYTHONVER
else
RPMPYTHONVER=%{nil}
fi
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564
Mathias De Maré
buildrpm: move creation of RPM directories from dockerrpm...
r27788 mkdir -p $RPMBUILDDIR/{SOURCES,BUILD,SRPMS,RPMS}
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 $HG archive -t tgz $RPMBUILDDIR/SOURCES/mercurial-$version-$release.tar.gz
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 if [ "$PYTHONVER" ]; then
(
Mads Kiilerich
rpms: create missing builds dir if it doesn't exist
r24730 mkdir -p build
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 cd build
PYTHON_SRCFILE=Python-$PYTHONVER.tgz
[ -f $PYTHON_SRCFILE ] || curl -Lo $PYTHON_SRCFILE http://www.python.org/ftp/python/$PYTHONVER/$PYTHON_SRCFILE
Mads Kiilerich
contrib: buildrpm checking of md5 checksums of downloaded Python and Docutils
r23141 if [ "$PYTHONMD5" ]; then
echo "$PYTHONMD5 $PYTHON_SRCFILE" | md5sum -w -c
fi
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 ln -f $PYTHON_SRCFILE $RPMBUILDDIR/SOURCES/$PYTHON_SRCFILE
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436
DOCUTILSVER=`sed -ne "s/^%global docutilsname docutils-//p" $specfile`
DOCUTILS_SRCFILE=docutils-$DOCUTILSVER.tar.gz
[ -f $DOCUTILS_SRCFILE ] || curl -Lo $DOCUTILS_SRCFILE http://downloads.sourceforge.net/project/docutils/docutils/$DOCUTILSVER/$DOCUTILS_SRCFILE
Mads Kiilerich
contrib: buildrpm checking of md5 checksums of downloaded Python and Docutils
r23141 DOCUTILSMD5=`sed -ne "s/^%global docutilsmd5 //p" $specfile`
if [ "$DOCUTILSMD5" ]; then
echo "$DOCUTILSMD5 $DOCUTILS_SRCFILE" | md5sum -w -c
fi
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 ln -f $DOCUTILS_SRCFILE $RPMBUILDDIR/SOURCES/$DOCUTILS_SRCFILE
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 )
fi
Augie Fackler
buildrpm: mkdir -p two needed directories (issue4779)...
r26139 mkdir -p $RPMBUILDDIR/SPECS
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 rpmspec=$RPMBUILDDIR/SPECS/mercurial.spec
Gilles Moris
buildrpm: build full RPM package including sources
r9813
Gilles Moris
buildrpm: cleanup script
r9812 sed -e "s,^Version:.*,Version: $version," \
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564 -e "s,^Release:.*,Release: $release," \
Gilles Moris
buildrpm: build full RPM package including sources
r9813 $specfile > $rpmspec
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564
Gilles Moris
buildrpm: enhance changelog of the RPM file...
r9814 echo >> $rpmspec
echo "%changelog" >> $rpmspec
if echo $version | grep '+' > /dev/null 2>&1; then
latesttag="`echo $version | sed -e 's/+.*//'`"
$HG log -r .:"$latesttag" -fM \
--template '{date|hgdate}\t{author}\t{desc|firstline}\n' | python -c '
import sys, time
def datestr(date, format):
return time.strftime(format, time.gmtime(float(date[0]) - date[1]))
Adam Spiers
buildrpm: auto-generate %changelog in .spec file...
r4754
Gilles Moris
buildrpm: enhance changelog of the RPM file...
r9814 changelog = []
for l in sys.stdin.readlines():
tok = l.split("\t")
hgdate = tuple(int(v) for v in tok[0].split())
changelog.append((datestr(hgdate, "%F"), tok[1], hgdate, tok[2]))
prevtitle = ""
for l in sorted(changelog, reverse=True):
title = "* %s %s" % (datestr(l[2], "%a %b %d %Y"), l[1])
if prevtitle != title:
prevtitle = title
print
print title
print "- %s" % l[3].strip()
' >> $rpmspec
else
$HG log \
--template '{date|hgdate}\t{author}\t{desc|firstline}\n' \
.hgtags | python -c '
import sys, time
def datestr(date, format):
return time.strftime(format, time.gmtime(float(date[0]) - date[1]))
for l in sys.stdin.readlines():
tok = l.split("\t")
hgdate = tuple(int(v) for v in tok[0].split())
print "* %s %s\n- %s" % (datestr(hgdate, "%a %b %d %Y"), tok[1], tok[2])
' >> $rpmspec
fi
Adam Spiers
buildrpm: auto-generate %changelog in .spec file...
r4754
Mads Kiilerich
buildrpm: introduce --withpython for building rpms that includes Python 2.7
r22436 sed -i \
-e "s/^%define withpython.*$/%define withpython $RPMPYTHONVER/" \
$rpmspec
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 if [ "$BUILD" ]; then
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 rpmbuild --define "_topdir $RPMBUILDDIR" -ba $rpmspec --clean
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 if [ $? = 0 ]; then
echo
echo "Built packages for $version-$release:"
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 find $RPMBUILDDIR/*RPMS/ -type f -newer $rpmspec
Mads Kiilerich
buildrpm: introduce --prepare for preparing without actually building rpms
r22435 fi
else
Mads Kiilerich
buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir...
r22437 echo "Prepared sources for $version-$release $rpmspec are in $RPMBUILDDIR/SOURCES/ - use like:"
echo "rpmbuild --define '_topdir $RPMBUILDDIR' -ba $rpmspec --clean"
mpm@selenic.com
[PATCH] Add contrib/buildrpm script...
r564 fi