##// END OF EJS Templates
mercurial: implement a source transforming module loader on Python 3...
mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

File last commit:

r19023:86531a70 default
r29550:1c22400d default
Show More
pylintrc
313 lines | 9.0 KiB | text/plain | TextLexer
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 # lint Python modules using external checkers.
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 # This is the main checker controlling the other ones and the reports
# generation. It is itself both a raw checker and an astng checker in order
# to:
# * handle message activation / deactivation at the module level
# * handle some basic but necessary stats'data (number of classes, methods...)
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [MASTER]
# Specify a configuration file.
#rcfile=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Profiled execution.
profile=no
# Add <file or directory> to the black list. It should be a base name, not a
# path. You may set this option multiple times.
ignore=CVS
# Pickle collected data for later comparisons.
persistent=yes
# Set the cache size for astng objects.
cache-size=500
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=
[MESSAGES CONTROL]
# Enable only checker(s) with the given id(s). This option conflicts with the
# disable-checker option
#enable-checker=
# Enable all checker(s) except those with the given id(s). This option
# conflicts with the enable-checker option
#disable-checker=
# Enable all messages in the listed categories (IRCWEF).
#enable-msg-cat=
# Disable all messages in the listed categories (IRCWEF).
disable-msg-cat=I
# Enable the message(s) with the given id(s).
#enable-msg=
# Disable the message(s) with the given id(s).
# W0704: except: pass
# C0111: missing docstring
# W0403: for the time being absolute imports don't play nice with demandimport
disable-msg=W0704,C0111,W0403
[REPORTS]
# Set the output format. Available formats are text, parseable, colorized, msvs
# (visual studio) and html
output-format=text
# Include message's id in output
include-ids=yes
# Put messages in a separate file for each module / package specified on the
# command line instead of printing them on stdout. Reports (if any) will be
# written in a file name "pylint_global.[txt|html]".
files-output=no
# Tells whether to display a full report or only the messages
reports=yes
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (R0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
# Add a comment according to your evaluation note. This is used by the global
# evaluation report (R0004).
comment=no
# Enable the report(s) with the given id(s).
#enable-report=
# Disable the report(s) with the given id(s).
#disable-report=
# try to find bugs in the code using type inference
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [TYPECHECK]
# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes
# List of classes names for which member attributes should not be checked
# (useful for classes with attributes dynamically set).
ignored-classes=SQLObject
# When zope mode is activated, add a predefined set of Zope acquired attributes
# to generated-members.
zope=no
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E0201 when accessed.
generated-members=REQUEST,acl_users,aq_parent
# checks for
# * unused variables / imports
# * undefined variables
# * redefinition of variable from builtins or from an outer scope
# * use of variable before assignment
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [VARIABLES]
# Tells whether we should check for unused import in __init__ files.
init-import=yes
# A regular expression matching names used for dummy variables (i.e. not used).
dummy-variables-rgx=dummy
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid to define new builtins when possible.
additional-builtins=
# checks for :
# * doc strings
# * modules / classes / functions / methods / arguments / variables name
# * number of arguments, local variables, branches, returns and statements in
# functions, methods
# * required module attributes
# * dangerous default values as arguments
# * redefinition of function / method / class
# * uses of the global statement
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [BASIC]
# Required attributes for module, separated by a comma
required-attributes=
# Regular expression which should only match functions or classes name which do
# not require a docstring
no-docstring-rgx=__.*__
# Regular expression which should only match correct module names
module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
# Regular expression which should only match correct module level names
const-rgx=(([a-zA-Z_][a-zA-Z0-9_]*)|(__.*__))$
# Regular expression which should only match correct class names
class-rgx=[a-zA-Z_][a-zA-Z0-9]+$
# Regular expression which should only match correct function names
function-rgx=[a-z_][a-z0-9_]{2,30}$
# Regular expression which should only match correct method names
method-rgx=[a-z_][a-z0-9_]{2,30}$
# Regular expression which should only match correct instance attribute names
attr-rgx=[a-z_][a-z0-9_]{1,30}$
# Regular expression which should only match correct argument names
argument-rgx=[a-z_][a-z0-9_]{0,30}$
# Regular expression which should only match correct variable names
variable-rgx=[a-z_][a-z0-9_]{0,30}$
# Regular expression which should only match correct list comprehension /
# generator expression variable names
inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
# Good variable names which should always be accepted, separated by a comma
good-names=i,j,k,ex,Run,_,ui,c,fn,f,fd,l
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,bar,baz,toto,tutu,tata
# List of builtins function names that should not be used, separated by a comma
#bad-functions=map,filter,apply,input
bad-functions=map,filter,apply,input
# checks for
# * external modules dependencies
# * relative / wildcard imports
# * cyclic imports
# * uses of deprecated modules
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [IMPORTS]
# Deprecated modules which should not be used, separated by a comma
Benoit Boissinot
pylintrc: string isn't deprecated, only some of its methods
r10403 deprecated-modules=regsub,TERMIOS,Bastion,rexec
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report R0402 must not be disabled)
import-graph=
# Create a graph of external dependencies in the given file (report R0402 must
# not be disabled)
ext-import-graph=
# Create a graph of internal dependencies in the given file (report R0402 must
# not be disabled)
int-import-graph=
# checks for sign of poor/misdesign:
# * number of methods, attributes, local variables...
# * size, complexity of functions, methods
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [DESIGN]
# Maximum number of arguments for function / method
max-args=5
# Maximum number of locals for function / method body
max-locals=15
# Maximum number of return / yield for function / method body
max-returns=6
# Maximum number of branch for function / method body
max-branchs=12
# Maximum number of statements in function / method body
max-statements=50
# Maximum number of parents for a class (see R0901).
max-parents=7
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# checks for :
# * methods without self as first argument
# * overridden methods signature
# * access only to existent members via self
# * attributes not defined in the __init__ method
# * supported interfaces implementation
# * unreachable code
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [CLASSES]
# List of interface methods to ignore, separated by a comma. This is used for
# instance to not check methods defines in Zope's Interface base class.
ignore-iface-methods=isImplementedBy,deferred,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,__new__,setUp
# checks for :
# * unauthorized constructions
# * strict indentation
# * line length
# * use of <> instead of !=
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [FORMAT]
# Maximum number of characters on a single line.
max-line-length=80
# Maximum number of lines in a module
max-module-lines=1000
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# checks for:
# * warning notes in the code like FIXME, XXX
# * PEP 263: source code with non ascii character but no encoding declaration
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,XXX,TODO
# checks for similarities and duplicated code. This computation may be
# memory / CPU intensive, so you should disable it if you experiments some
# problems.
Mads Kiilerich
codingstyle: remove trailing spaces in various text files...
r19023 #
Benoit Boissinot
add pylintrc with some mercurial specific configuration
r10393 [SIMILARITIES]
# Minimum lines number of a similarity.
min-similarity-lines=4
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes