##// END OF EJS Templates
highlight: fixes garbled text in non-UTF-8 environment...
highlight: fixes garbled text in non-UTF-8 environment This patch treats all files inside repository as encoded by locale's encoding when pygmentize. We can assume that most files are written in locale's encoding, but current implementation treats them as UTF-8. So there's no way to specify the encoding of files. Current implementation, db7557359636 (issue1341): 1. Convert original `text`, which is treated as UTF-8, to locale's encoding. `encoding.tolocal()` is the method to convert from internal UTF-8 to local. If original `text` is not UTF-8, e.g. Japanese EUC-JP, some characters become garbled here. 2. pygmentize, with no UnicodeDecodeError. This patch: 1. Convert original `text`, which is treated as locale's encoding, to unicode. Pygments prefers unicode object than raw str. [1]_ If original `text` is not encoded by locale's encoding, some characters become garbled here. 2. pygmentize, also with no UnicodeDecodeError :) 3. Convert unicode back to raw str, which is encoded by locale's. .. [1] http://pygments.org/docs/unicode/

File last commit:

r9424:799373ff default
r9424:799373ff default
Show More
highlight.py
61 lines | 2.1 KiB | text/x-python | PythonLexer
# highlight.py - highlight extension implementation file
#
# Copyright 2007-2009 Adam Hupp <adam@hupp.org> and others
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2, incorporated herein by reference.
#
# The original module was split in an interface and an implementation
# file to defer pygments loading and speedup extension setup.
from mercurial import demandimport
demandimport.ignore.extend(['pkgutil', 'pkg_resources', '__main__',])
from mercurial import util, encoding
from pygments import highlight
from pygments.util import ClassNotFound
from pygments.lexers import guess_lexer, guess_lexer_for_filename, TextLexer
from pygments.formatters import HtmlFormatter
SYNTAX_CSS = ('\n<link rel="stylesheet" href="{url}highlightcss" '
'type="text/css" />')
def pygmentize(field, fctx, style, tmpl):
# append a <link ...> to the syntax highlighting css
old_header = ''.join(tmpl('header'))
if SYNTAX_CSS not in old_header:
new_header = old_header + SYNTAX_CSS
tmpl.cache['header'] = new_header
text = fctx.data()
if util.binary(text):
return
# Pygments is best used with Unicode strings:
# <http://pygments.org/docs/unicode/>
text = text.decode(encoding.encoding, 'replace')
# To get multi-line strings right, we can't format line-by-line
try:
lexer = guess_lexer_for_filename(fctx.path(), text[:1024])
except (ClassNotFound, ValueError):
try:
lexer = guess_lexer(text[:1024])
except (ClassNotFound, ValueError):
lexer = TextLexer()
formatter = HtmlFormatter(style=style)
colorized = highlight(text, lexer, formatter)
# strip wrapping div
colorized = colorized[:colorized.find('\n</pre>')]
colorized = colorized[colorized.find('<pre>')+5:]
coloriter = (s.encode(encoding.encoding, 'replace')
for s in colorized.splitlines())
tmpl.filters['colorize'] = lambda x: coloriter.next()
oldl = tmpl.cache[field]
newl = oldl.replace('line|escape', 'line|colorize')
tmpl.cache[field] = newl