procutil: always popen() in binary mode

On Python 3, non-binary stream is useless. Let's convert line ending by
ourselves.

Note that we don't need fromnativeeol() in patch._externalpatch() since
any whitespace characters are rstrip()-ed.

hgext/bugzilla.py

@@ -1,1129 +1,1129
 # bugzilla.py - bugzilla integration for mercurial
 #
 # Copyright 2006 Vadim Gelfer <[e-mail]>
 # Copyright 2011-4 Jim Hague <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 '''hooks for integrating with the Bugzilla bug tracker
 
 This hook extension adds comments on bugs in Bugzilla when changesets
 that refer to bugs by Bugzilla ID are seen. The comment is formatted using
 the Mercurial template mechanism.
 
 The bug references can optionally include an update for Bugzilla of the
 hours spent working on the bug. Bugs can also be marked fixed.
 
 Four basic modes of access to Bugzilla are provided:
 
 1. Access via the Bugzilla REST-API. Requires bugzilla 5.0 or later.
 
 2. Access via the Bugzilla XMLRPC interface. Requires Bugzilla 3.4 or later.
 
 3. Check data via the Bugzilla XMLRPC interface and submit bug change
    via email to Bugzilla email interface. Requires Bugzilla 3.4 or later.
 
 4. Writing directly to the Bugzilla database. Only Bugzilla installations
    using MySQL are supported. Requires Python MySQLdb.
 
 Writing directly to the database is susceptible to schema changes, and
 relies on a Bugzilla contrib script to send out bug change
 notification emails. This script runs as the user running Mercurial,
 must be run on the host with the Bugzilla install, and requires
 permission to read Bugzilla configuration details and the necessary
 MySQL user and password to have full access rights to the Bugzilla
 database. For these reasons this access mode is now considered
 deprecated, and will not be updated for new Bugzilla versions going
 forward. Only adding comments is supported in this access mode.
 
 Access via XMLRPC needs a Bugzilla username and password to be specified
 in the configuration. Comments are added under that username. Since the
 configuration must be readable by all Mercurial users, it is recommended
 that the rights of that user are restricted in Bugzilla to the minimum
 necessary to add comments. Marking bugs fixed requires Bugzilla 4.0 and later.
 
 Access via XMLRPC/email uses XMLRPC to query Bugzilla, but sends
 email to the Bugzilla email interface to submit comments to bugs.
 The From: address in the email is set to the email address of the Mercurial
 user, so the comment appears to come from the Mercurial user. In the event
 that the Mercurial user email is not recognized by Bugzilla as a Bugzilla
 user, the email associated with the Bugzilla username used to log into
 Bugzilla is used instead as the source of the comment. Marking bugs fixed
 works on all supported Bugzilla versions.
 
 Access via the REST-API needs either a Bugzilla username and password
 or an apikey specified in the configuration. Comments are made under
 the given username or the user associated with the apikey in Bugzilla.
 
 Configuration items common to all access modes:
 
 bugzilla.version
   The access type to use. Values recognized are:
 
   :``restapi``:      Bugzilla REST-API, Bugzilla 5.0 and later.
   :``xmlrpc``:       Bugzilla XMLRPC interface.
   :``xmlrpc+email``: Bugzilla XMLRPC and email interfaces.
   :``3.0``:          MySQL access, Bugzilla 3.0 and later.
   :``2.18``:         MySQL access, Bugzilla 2.18 and up to but not
                      including 3.0.
   :``2.16``:         MySQL access, Bugzilla 2.16 and up to but not
                      including 2.18.
 
 bugzilla.regexp
   Regular expression to match bug IDs for update in changeset commit message.
   It must contain one "()" named group ``<ids>`` containing the bug
   IDs separated by non-digit characters. It may also contain
   a named group ``<hours>`` with a floating-point number giving the
   hours worked on the bug. If no named groups are present, the first
   "()" group is assumed to contain the bug IDs, and work time is not
   updated. The default expression matches ``Bug 1234``, ``Bug no. 1234``,
   ``Bug number 1234``, ``Bugs 1234,5678``, ``Bug 1234 and 5678`` and
   variations thereof, followed by an hours number prefixed by ``h`` or
   ``hours``, e.g. ``hours 1.5``. Matching is case insensitive.
 
 bugzilla.fixregexp
   Regular expression to match bug IDs for marking fixed in changeset
   commit message. This must contain a "()" named group ``<ids>` containing
   the bug IDs separated by non-digit characters. It may also contain
   a named group ``<hours>`` with a floating-point number giving the
   hours worked on the bug. If no named groups are present, the first
   "()" group is assumed to contain the bug IDs, and work time is not
   updated. The default expression matches ``Fixes 1234``, ``Fixes bug 1234``,
   ``Fixes bugs 1234,5678``, ``Fixes 1234 and 5678`` and
   variations thereof, followed by an hours number prefixed by ``h`` or
   ``hours``, e.g. ``hours 1.5``. Matching is case insensitive.
 
 bugzilla.fixstatus
   The status to set a bug to when marking fixed. Default ``RESOLVED``.
 
 bugzilla.fixresolution
   The resolution to set a bug to when marking fixed. Default ``FIXED``.
 
 bugzilla.style
   The style file to use when formatting comments.
 
 bugzilla.template
   Template to use when formatting comments. Overrides style if
   specified. In addition to the usual Mercurial keywords, the
   extension specifies:
 
   :``{bug}``:     The Bugzilla bug ID.
   :``{root}``:    The full pathname of the Mercurial repository.
   :``{webroot}``: Stripped pathname of the Mercurial repository.
   :``{hgweb}``:   Base URL for browsing Mercurial repositories.
 
   Default ``changeset {node|short} in repo {root} refers to bug
   {bug}.\\ndetails:\\n\\t{desc|tabindent}``
 
 bugzilla.strip
   The number of path separator characters to strip from the front of
   the Mercurial repository path (``{root}`` in templates) to produce
   ``{webroot}``. For example, a repository with ``{root}``
   ``/var/local/my-project`` with a strip of 2 gives a value for
   ``{webroot}`` of ``my-project``. Default 0.
 
 web.baseurl
   Base URL for browsing Mercurial repositories. Referenced from
   templates as ``{hgweb}``.
 
 Configuration items common to XMLRPC+email and MySQL access modes:
 
 bugzilla.usermap
   Path of file containing Mercurial committer email to Bugzilla user email
   mappings. If specified, the file should contain one mapping per
   line::
 
     committer = Bugzilla user
 
   See also the ``[usermap]`` section.
 
 The ``[usermap]`` section is used to specify mappings of Mercurial
 committer email to Bugzilla user email. See also ``bugzilla.usermap``.
 Contains entries of the form ``committer = Bugzilla user``.
 
 XMLRPC and REST-API access mode configuration:
 
 bugzilla.bzurl
   The base URL for the Bugzilla installation.
   Default ``http://localhost/bugzilla``.
 
 bugzilla.user
   The username to use to log into Bugzilla via XMLRPC. Default
   ``bugs``.
 
 bugzilla.password
   The password for Bugzilla login.
 
 REST-API access mode uses the options listed above as well as:
 
 bugzilla.apikey
   An apikey generated on the Bugzilla instance for api access.
   Using an apikey removes the need to store the user and password
   options.
 
 XMLRPC+email access mode uses the XMLRPC access mode configuration items,
 and also:
 
 bugzilla.bzemail
   The Bugzilla email address.
 
 In addition, the Mercurial email settings must be configured. See the
 documentation in hgrc(5), sections ``[email]`` and ``[smtp]``.
 
 MySQL access mode configuration:
 
 bugzilla.host
   Hostname of the MySQL server holding the Bugzilla database.
   Default ``localhost``.
 
 bugzilla.db
   Name of the Bugzilla database in MySQL. Default ``bugs``.
 
 bugzilla.user
   Username to use to access MySQL server. Default ``bugs``.
 
 bugzilla.password
   Password to use to access MySQL server.
 
 bugzilla.timeout
   Database connection timeout (seconds). Default 5.
 
 bugzilla.bzuser
   Fallback Bugzilla user name to record comments with, if changeset
   committer cannot be found as a Bugzilla user.
 
 bugzilla.bzdir
    Bugzilla install directory. Used by default notify. Default
    ``/var/www/html/bugzilla``.
 
 bugzilla.notify
   The command to run to get Bugzilla to send bug change notification
   emails. Substitutes from a map with 3 keys, ``bzdir``, ``id`` (bug
   id) and ``user`` (committer bugzilla email). Default depends on
   version; from 2.18 it is "cd %(bzdir)s && perl -T
   contrib/sendbugmail.pl %(id)s %(user)s".
 
 Activating the extension::
 
     [extensions]
     bugzilla =
 
     [hooks]
     # run bugzilla hook on every change pulled or pushed in here
     incoming.bugzilla = python:hgext.bugzilla.hook
 
 Example configurations:
 
 XMLRPC example configuration. This uses the Bugzilla at
 ``http://my-project.org/bugzilla``, logging in as user
 ``[e-mail]`` with password ``plugh``. It is used with a
 collection of Mercurial repositories in ``/var/local/hg/repos/``,
 with a web interface at ``http://my-project.org/hg``. ::
 
     [bugzilla]
     bzurl=http://my-project.org/bugzilla
     user=[e-mail]
     password=plugh
     version=xmlrpc
     template=Changeset {node|short} in {root|basename}.
              {hgweb}/{webroot}/rev/{node|short}\\n
              {desc}\\n
     strip=5
 
     [web]
     baseurl=http://my-project.org/hg
 
 XMLRPC+email example configuration. This uses the Bugzilla at
 ``http://my-project.org/bugzilla``, logging in as user
 ``[e-mail]`` with password ``plugh``. It is used with a
 collection of Mercurial repositories in ``/var/local/hg/repos/``,
 with a web interface at ``http://my-project.org/hg``. Bug comments
 are sent to the Bugzilla email address
 ``[e-mail]``. ::
 
     [bugzilla]
     bzurl=http://my-project.org/bugzilla
     user=[e-mail]
     password=plugh
     version=xmlrpc+email
     bzemail=[e-mail]
     template=Changeset {node|short} in {root|basename}.
              {hgweb}/{webroot}/rev/{node|short}\\n
              {desc}\\n
     strip=5
 
     [web]
     baseurl=http://my-project.org/hg
 
     [usermap]
     [e-mail]=[e-mail]
 
 MySQL example configuration. This has a local Bugzilla 3.2 installation
 in ``/opt/bugzilla-3.2``. The MySQL database is on ``localhost``,
 the Bugzilla database name is ``bugs`` and MySQL is
 accessed with MySQL username ``bugs`` password ``XYZZY``. It is used
 with a collection of Mercurial repositories in ``/var/local/hg/repos/``,
 with a web interface at ``http://my-project.org/hg``. ::
 
     [bugzilla]
     host=localhost
     password=XYZZY
     version=3.0
     bzuser=[e-mail]
     bzdir=/opt/bugzilla-3.2
     template=Changeset {node|short} in {root|basename}.
              {hgweb}/{webroot}/rev/{node|short}\\n
              {desc}\\n
     strip=5
 
     [web]
     baseurl=http://my-project.org/hg
 
     [usermap]
     [e-mail]=[e-mail]
 
 All the above add a comment to the Bugzilla bug record of the form::
 
     Changeset 3b16791d6642 in repository-name.
     http://my-project.org/hg/repository-name/rev/3b16791d6642
 
     Changeset commit comment. Bug 1234.
 '''
 
 from __future__ import absolute_import
 
 import json
 import re
 import time
 
 from mercurial.i18n import _
 from mercurial.node import short
 from mercurial import (
     error,
     logcmdutil,
     mail,
     registrar,
     url,
     util,
 )
 from mercurial.utils import (
     procutil,
     stringutil,
 )
 
 xmlrpclib = util.xmlrpclib
 
 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
 testedwith = 'ships-with-hg-core'
 
 configtable = {}
 configitem = registrar.configitem(configtable)
 
 configitem('bugzilla', 'apikey',
     default='',
 )
 configitem('bugzilla', 'bzdir',
     default='/var/www/html/bugzilla',
 )
 configitem('bugzilla', 'bzemail',
     default=None,
 )
 configitem('bugzilla', 'bzurl',
     default='http://localhost/bugzilla/',
 )
 configitem('bugzilla', 'bzuser',
     default=None,
 )
 configitem('bugzilla', 'db',
     default='bugs',
 )
 configitem('bugzilla', 'fixregexp',
     default=(r'fix(?:es)?\s*(?:bugs?\s*)?,?\s*'
              r'(?:nos?\.?|num(?:ber)?s?)?\s*'
              r'(?P<ids>(?:#?\d+\s*(?:,?\s*(?:and)?)?\s*)+)'
              r'\.?\s*(?:h(?:ours?)?\s*(?P<hours>\d*(?:\.\d+)?))?')
 )
 configitem('bugzilla', 'fixresolution',
     default='FIXED',
 )
 configitem('bugzilla', 'fixstatus',
     default='RESOLVED',
 )
 configitem('bugzilla', 'host',
     default='localhost',
 )
 configitem('bugzilla', 'notify',
     default=configitem.dynamicdefault,
 )
 configitem('bugzilla', 'password',
     default=None,
 )
 configitem('bugzilla', 'regexp',
     default=(r'bugs?\s*,?\s*(?:#|nos?\.?|num(?:ber)?s?)?\s*'
              r'(?P<ids>(?:\d+\s*(?:,?\s*(?:and)?)?\s*)+)'
              r'\.?\s*(?:h(?:ours?)?\s*(?P<hours>\d*(?:\.\d+)?))?')
 )
 configitem('bugzilla', 'strip',
     default=0,
 )
 configitem('bugzilla', 'style',
     default=None,
 )
 configitem('bugzilla', 'template',
     default=None,
 )
 configitem('bugzilla', 'timeout',
     default=5,
 )
 configitem('bugzilla', 'user',
     default='bugs',
 )
 configitem('bugzilla', 'usermap',
     default=None,
 )
 configitem('bugzilla', 'version',
     default=None,
 )
 
 class bzaccess(object):
     '''Base class for access to Bugzilla.'''
 
     def __init__(self, ui):
         self.ui = ui
         usermap = self.ui.config('bugzilla', 'usermap')
         if usermap:
             self.ui.readconfig(usermap, sections=['usermap'])
 
     def map_committer(self, user):
         '''map name of committer to Bugzilla user name.'''
         for committer, bzuser in self.ui.configitems('usermap'):
             if committer.lower() == user.lower():
                 return bzuser
         return user
 
     # Methods to be implemented by access classes.
     #
     # 'bugs' is a dict keyed on bug id, where values are a dict holding
     # updates to bug state. Recognized dict keys are:
     #
     # 'hours': Value, float containing work hours to be updated.
     # 'fix':   If key present, bug is to be marked fixed. Value ignored.
 
     def filter_real_bug_ids(self, bugs):
         '''remove bug IDs that do not exist in Bugzilla from bugs.'''
 
     def filter_cset_known_bug_ids(self, node, bugs):
         '''remove bug IDs where node occurs in comment text from bugs.'''
 
     def updatebug(self, bugid, newstate, text, committer):
         '''update the specified bug. Add comment text and set new states.
 
         If possible add the comment as being from the committer of
         the changeset. Otherwise use the default Bugzilla user.
         '''
 
     def notify(self, bugs, committer):
         '''Force sending of Bugzilla notification emails.
 
         Only required if the access method does not trigger notification
         emails automatically.
         '''
 
 # Bugzilla via direct access to MySQL database.
 class bzmysql(bzaccess):
     '''Support for direct MySQL access to Bugzilla.
 
     The earliest Bugzilla version this is tested with is version 2.16.
 
     If your Bugzilla is version 3.4 or above, you are strongly
     recommended to use the XMLRPC access method instead.
     '''
 
     @staticmethod
     def sql_buglist(ids):
         '''return SQL-friendly list of bug ids'''
         return '(' + ','.join(map(str, ids)) + ')'
 
     _MySQLdb = None
 
     def __init__(self, ui):
         try:
             import MySQLdb as mysql
             bzmysql._MySQLdb = mysql
         except ImportError as err:
             raise error.Abort(_('python mysql support not available: %s') % err)
 
         bzaccess.__init__(self, ui)
 
         host = self.ui.config('bugzilla', 'host')
         user = self.ui.config('bugzilla', 'user')
         passwd = self.ui.config('bugzilla', 'password')
         db = self.ui.config('bugzilla', 'db')
         timeout = int(self.ui.config('bugzilla', 'timeout'))
         self.ui.note(_('connecting to %s:%s as %s, password %s\n') %
                      (host, db, user, '*' * len(passwd)))
         self.conn = bzmysql._MySQLdb.connect(host=host,
                                                    user=user, passwd=passwd,
                                                    db=db,
                                                    connect_timeout=timeout)
         self.cursor = self.conn.cursor()
         self.longdesc_id = self.get_longdesc_id()
         self.user_ids = {}
         self.default_notify = "cd %(bzdir)s && ./processmail %(id)s %(user)s"
 
     def run(self, *args, **kwargs):
         '''run a query.'''
         self.ui.note(_('query: %s %s\n') % (args, kwargs))
         try:
             self.cursor.execute(*args, **kwargs)
         except bzmysql._MySQLdb.MySQLError:
             self.ui.note(_('failed query: %s %s\n') % (args, kwargs))
             raise
 
     def get_longdesc_id(self):
         '''get identity of longdesc field'''
         self.run('select fieldid from fielddefs where name = "longdesc"')
         ids = self.cursor.fetchall()
         if len(ids) != 1:
             raise error.Abort(_('unknown database schema'))
         return ids[0][0]
 
     def filter_real_bug_ids(self, bugs):
         '''filter not-existing bugs from set.'''
         self.run('select bug_id from bugs where bug_id in %s' %
                  bzmysql.sql_buglist(bugs.keys()))
         existing = [id for (id,) in self.cursor.fetchall()]
         for id in bugs.keys():
             if id not in existing:
                 self.ui.status(_('bug %d does not exist\n') % id)
                 del bugs[id]
 
     def filter_cset_known_bug_ids(self, node, bugs):
         '''filter bug ids that already refer to this changeset from set.'''
         self.run('''select bug_id from longdescs where
                     bug_id in %s and thetext like "%%%s%%"''' %
                  (bzmysql.sql_buglist(bugs.keys()), short(node)))
         for (id,) in self.cursor.fetchall():
             self.ui.status(_('bug %d already knows about changeset %s\n') %
                            (id, short(node)))
             del bugs[id]
 
     def notify(self, bugs, committer):
         '''tell bugzilla to send mail.'''
         self.ui.status(_('telling bugzilla to send mail:\n'))
         (user, userid) = self.get_bugzilla_user(committer)
         for id in bugs.keys():
             self.ui.status(_('  bug %s\n') % id)
             cmdfmt = self.ui.config('bugzilla', 'notify', self.default_notify)
             bzdir = self.ui.config('bugzilla', 'bzdir')
             try:
                 # Backwards-compatible with old notify string, which
                 # took one string. This will throw with a new format
                 # string.
                 cmd = cmdfmt % id
             except TypeError:
                 cmd = cmdfmt % {'bzdir': bzdir, 'id': id, 'user': user}
             self.ui.note(_('running notify command %s\n') % cmd)
-            fp = procutil.popen('(%s) 2>&1' % cmd)
-            out = fp.read()
+            fp = procutil.popen('(%s) 2>&1' % cmd, 'rb')
+            out = util.fromnativeeol(fp.read())
             ret = fp.close()
             if ret:
                 self.ui.warn(out)
                 raise error.Abort(_('bugzilla notify command %s') %
                                   procutil.explainexit(ret)[0])
         self.ui.status(_('done\n'))
 
     def get_user_id(self, user):
         '''look up numeric bugzilla user id.'''
         try:
             return self.user_ids[user]
         except KeyError:
             try:
                 userid = int(user)
             except ValueError:
                 self.ui.note(_('looking up user %s\n') % user)
                 self.run('''select userid from profiles
                             where login_name like %s''', user)
                 all = self.cursor.fetchall()
                 if len(all) != 1:
                     raise KeyError(user)
                 userid = int(all[0][0])
             self.user_ids[user] = userid
             return userid
 
     def get_bugzilla_user(self, committer):
         '''See if committer is a registered bugzilla user. Return
         bugzilla username and userid if so. If not, return default
         bugzilla username and userid.'''
         user = self.map_committer(committer)
         try:
             userid = self.get_user_id(user)
         except KeyError:
             try:
                 defaultuser = self.ui.config('bugzilla', 'bzuser')
                 if not defaultuser:
                     raise error.Abort(_('cannot find bugzilla user id for %s') %
                                      user)
                 userid = self.get_user_id(defaultuser)
                 user = defaultuser
             except KeyError:
                 raise error.Abort(_('cannot find bugzilla user id for %s or %s')
                                  % (user, defaultuser))
         return (user, userid)
 
     def updatebug(self, bugid, newstate, text, committer):
         '''update bug state with comment text.
 
         Try adding comment as committer of changeset, otherwise as
         default bugzilla user.'''
         if len(newstate) > 0:
             self.ui.warn(_("Bugzilla/MySQL cannot update bug state\n"))
 
         (user, userid) = self.get_bugzilla_user(committer)
         now = time.strftime(r'%Y-%m-%d %H:%M:%S')
         self.run('''insert into longdescs
                     (bug_id, who, bug_when, thetext)
                     values (%s, %s, %s, %s)''',
                  (bugid, userid, now, text))
         self.run('''insert into bugs_activity (bug_id, who, bug_when, fieldid)
                     values (%s, %s, %s, %s)''',
                  (bugid, userid, now, self.longdesc_id))
         self.conn.commit()
 
 class bzmysql_2_18(bzmysql):
     '''support for bugzilla 2.18 series.'''
 
     def __init__(self, ui):
         bzmysql.__init__(self, ui)
         self.default_notify = \
             "cd %(bzdir)s && perl -T contrib/sendbugmail.pl %(id)s %(user)s"
 
 class bzmysql_3_0(bzmysql_2_18):
     '''support for bugzilla 3.0 series.'''
 
     def __init__(self, ui):
         bzmysql_2_18.__init__(self, ui)
 
     def get_longdesc_id(self):
         '''get identity of longdesc field'''
         self.run('select id from fielddefs where name = "longdesc"')
         ids = self.cursor.fetchall()
         if len(ids) != 1:
             raise error.Abort(_('unknown database schema'))
         return ids[0][0]
 
 # Bugzilla via XMLRPC interface.
 
 class cookietransportrequest(object):
     """A Transport request method that retains cookies over its lifetime.
 
     The regular xmlrpclib transports ignore cookies. Which causes
     a bit of a problem when you need a cookie-based login, as with
     the Bugzilla XMLRPC interface prior to 4.4.3.
 
     So this is a helper for defining a Transport which looks for
     cookies being set in responses and saves them to add to all future
     requests.
     """
 
     # Inspiration drawn from
     # http://blog.godson.in/2010/09/how-to-make-python-xmlrpclib-client.html
     # http://www.itkovian.net/base/transport-class-for-pythons-xml-rpc-lib/
 
     cookies = []
     def send_cookies(self, connection):
         if self.cookies:
             for cookie in self.cookies:
                 connection.putheader("Cookie", cookie)
 
     def request(self, host, handler, request_body, verbose=0):
         self.verbose = verbose
         self.accept_gzip_encoding = False
 
         # issue XML-RPC request
         h = self.make_connection(host)
         if verbose:
             h.set_debuglevel(1)
 
         self.send_request(h, handler, request_body)
         self.send_host(h, host)
         self.send_cookies(h)
         self.send_user_agent(h)
         self.send_content(h, request_body)
 
         # Deal with differences between Python 2.6 and 2.7.
         # In the former h is a HTTP(S). In the latter it's a
         # HTTP(S)Connection. Luckily, the 2.6 implementation of
         # HTTP(S) has an underlying HTTP(S)Connection, so extract
         # that and use it.
         try:
             response = h.getresponse()
         except AttributeError:
             response = h._conn.getresponse()
 
         # Add any cookie definitions to our list.
         for header in response.msg.getallmatchingheaders("Set-Cookie"):
             val = header.split(": ", 1)[1]
             cookie = val.split(";", 1)[0]
             self.cookies.append(cookie)
 
         if response.status != 200:
             raise xmlrpclib.ProtocolError(host + handler, response.status,
                                           response.reason, response.msg.headers)
 
         payload = response.read()
         parser, unmarshaller = self.getparser()
         parser.feed(payload)
         parser.close()
 
         return unmarshaller.close()
 
 # The explicit calls to the underlying xmlrpclib __init__() methods are
 # necessary. The xmlrpclib.Transport classes are old-style classes, and
 # it turns out their __init__() doesn't get called when doing multiple
 # inheritance with a new-style class.
 class cookietransport(cookietransportrequest, xmlrpclib.Transport):
     def __init__(self, use_datetime=0):
         if util.safehasattr(xmlrpclib.Transport, "__init__"):
             xmlrpclib.Transport.__init__(self, use_datetime)
 
 class cookiesafetransport(cookietransportrequest, xmlrpclib.SafeTransport):
     def __init__(self, use_datetime=0):
         if util.safehasattr(xmlrpclib.Transport, "__init__"):
             xmlrpclib.SafeTransport.__init__(self, use_datetime)
 
 class bzxmlrpc(bzaccess):
     """Support for access to Bugzilla via the Bugzilla XMLRPC API.
 
     Requires a minimum Bugzilla version 3.4.
     """
 
     def __init__(self, ui):
         bzaccess.__init__(self, ui)
 
         bzweb = self.ui.config('bugzilla', 'bzurl')
         bzweb = bzweb.rstrip("/") + "/xmlrpc.cgi"
 
         user = self.ui.config('bugzilla', 'user')
         passwd = self.ui.config('bugzilla', 'password')
 
         self.fixstatus = self.ui.config('bugzilla', 'fixstatus')
         self.fixresolution = self.ui.config('bugzilla', 'fixresolution')
 
         self.bzproxy = xmlrpclib.ServerProxy(bzweb, self.transport(bzweb))
         ver = self.bzproxy.Bugzilla.version()['version'].split('.')
         self.bzvermajor = int(ver[0])
         self.bzverminor = int(ver[1])
         login = self.bzproxy.User.login({'login': user, 'password': passwd,
                                          'restrict_login': True})
         self.bztoken = login.get('token', '')
 
     def transport(self, uri):
         if util.urlreq.urlparse(uri, "http")[0] == "https":
             return cookiesafetransport()
         else:
             return cookietransport()
 
     def get_bug_comments(self, id):
         """Return a string with all comment text for a bug."""
         c = self.bzproxy.Bug.comments({'ids': [id],
                                        'include_fields': ['text'],
                                        'token': self.bztoken})
         return ''.join([t['text'] for t in c['bugs'][str(id)]['comments']])
 
     def filter_real_bug_ids(self, bugs):
         probe = self.bzproxy.Bug.get({'ids': sorted(bugs.keys()),
                                       'include_fields': [],
                                       'permissive': True,
                                       'token': self.bztoken,
                                       })
         for badbug in probe['faults']:
             id = badbug['id']
             self.ui.status(_('bug %d does not exist\n') % id)
             del bugs[id]
 
     def filter_cset_known_bug_ids(self, node, bugs):
         for id in sorted(bugs.keys()):
             if self.get_bug_comments(id).find(short(node)) != -1:
                 self.ui.status(_('bug %d already knows about changeset %s\n') %
                                (id, short(node)))
                 del bugs[id]
 
     def updatebug(self, bugid, newstate, text, committer):
         args = {}
         if 'hours' in newstate:
             args['work_time'] = newstate['hours']
 
         if self.bzvermajor >= 4:
             args['ids'] = [bugid]
             args['comment'] = {'body' : text}
             if 'fix' in newstate:
                 args['status'] = self.fixstatus
                 args['resolution'] = self.fixresolution
             args['token'] = self.bztoken
             self.bzproxy.Bug.update(args)
         else:
             if 'fix' in newstate:
                 self.ui.warn(_("Bugzilla/XMLRPC needs Bugzilla 4.0 or later "
                                "to mark bugs fixed\n"))
             args['id'] = bugid
             args['comment'] = text
             self.bzproxy.Bug.add_comment(args)
 
 class bzxmlrpcemail(bzxmlrpc):
     """Read data from Bugzilla via XMLRPC, send updates via email.
 
     Advantages of sending updates via email:
       1. Comments can be added as any user, not just logged in user.
       2. Bug statuses or other fields not accessible via XMLRPC can
          potentially be updated.
 
     There is no XMLRPC function to change bug status before Bugzilla
     4.0, so bugs cannot be marked fixed via XMLRPC before Bugzilla 4.0.
     But bugs can be marked fixed via email from 3.4 onwards.
     """
 
     # The email interface changes subtly between 3.4 and 3.6. In 3.4,
     # in-email fields are specified as '@<fieldname> = <value>'. In
     # 3.6 this becomes '@<fieldname> <value>'. And fieldname @bug_id
     # in 3.4 becomes @id in 3.6. 3.6 and 4.0 both maintain backwards
     # compatibility, but rather than rely on this use the new format for
     # 4.0 onwards.
 
     def __init__(self, ui):
         bzxmlrpc.__init__(self, ui)
 
         self.bzemail = self.ui.config('bugzilla', 'bzemail')
         if not self.bzemail:
             raise error.Abort(_("configuration 'bzemail' missing"))
         mail.validateconfig(self.ui)
 
     def makecommandline(self, fieldname, value):
         if self.bzvermajor >= 4:
             return "@%s %s" % (fieldname, str(value))
         else:
             if fieldname == "id":
                 fieldname = "bug_id"
             return "@%s = %s" % (fieldname, str(value))
 
     def send_bug_modify_email(self, bugid, commands, comment, committer):
         '''send modification message to Bugzilla bug via email.
 
         The message format is documented in the Bugzilla email_in.pl
         specification. commands is a list of command lines, comment is the
         comment text.
 
         To stop users from crafting commit comments with
         Bugzilla commands, specify the bug ID via the message body, rather
         than the subject line, and leave a blank line after it.
         '''
         user = self.map_committer(committer)
         matches = self.bzproxy.User.get({'match': [user],
                                          'token': self.bztoken})
         if not matches['users']:
             user = self.ui.config('bugzilla', 'user')
             matches = self.bzproxy.User.get({'match': [user],
                                              'token': self.bztoken})
             if not matches['users']:
                 raise error.Abort(_("default bugzilla user %s email not found")
                                   % user)
         user = matches['users'][0]['email']
         commands.append(self.makecommandline("id", bugid))
 
         text = "\n".join(commands) + "\n\n" + comment
 
         _charsets = mail._charsets(self.ui)
         user = mail.addressencode(self.ui, user, _charsets)
         bzemail = mail.addressencode(self.ui, self.bzemail, _charsets)
         msg = mail.mimeencode(self.ui, text, _charsets)
         msg['From'] = user
         msg['To'] = bzemail
         msg['Subject'] = mail.headencode(self.ui, "Bug modification", _charsets)
         sendmail = mail.connect(self.ui)
         sendmail(user, bzemail, msg.as_string())
 
     def updatebug(self, bugid, newstate, text, committer):
         cmds = []
         if 'hours' in newstate:
             cmds.append(self.makecommandline("work_time", newstate['hours']))
         if 'fix' in newstate:
             cmds.append(self.makecommandline("bug_status", self.fixstatus))
             cmds.append(self.makecommandline("resolution", self.fixresolution))
         self.send_bug_modify_email(bugid, cmds, text, committer)
 
 class NotFound(LookupError):
     pass
 
 class bzrestapi(bzaccess):
     """Read and write bugzilla data using the REST API available since
     Bugzilla 5.0.
     """
     def __init__(self, ui):
         bzaccess.__init__(self, ui)
         bz = self.ui.config('bugzilla', 'bzurl')
         self.bzroot = '/'.join([bz, 'rest'])
         self.apikey = self.ui.config('bugzilla', 'apikey')
         self.user = self.ui.config('bugzilla', 'user')
         self.passwd = self.ui.config('bugzilla', 'password')
         self.fixstatus = self.ui.config('bugzilla', 'fixstatus')
         self.fixresolution = self.ui.config('bugzilla', 'fixresolution')
 
     def apiurl(self, targets, include_fields=None):
         url = '/'.join([self.bzroot] + [str(t) for t in targets])
         qv = {}
         if self.apikey:
             qv['api_key'] = self.apikey
         elif self.user and self.passwd:
             qv['login'] = self.user
             qv['password'] = self.passwd
         if include_fields:
             qv['include_fields'] = include_fields
         if qv:
             url = '%s?%s' % (url, util.urlreq.urlencode(qv))
         return url
 
     def _fetch(self, burl):
         try:
             resp = url.open(self.ui, burl)
             return json.loads(resp.read())
         except util.urlerr.httperror as inst:
             if inst.code == 401:
                 raise error.Abort(_('authorization failed'))
             if inst.code == 404:
                 raise NotFound()
             else:
                 raise
 
     def _submit(self, burl, data, method='POST'):
         data = json.dumps(data)
         if method == 'PUT':
             class putrequest(util.urlreq.request):
                 def get_method(self):
                     return 'PUT'
             request_type = putrequest
         else:
             request_type = util.urlreq.request
         req = request_type(burl, data,
                            {'Content-Type': 'application/json'})
         try:
             resp = url.opener(self.ui).open(req)
             return json.loads(resp.read())
         except util.urlerr.httperror as inst:
             if inst.code == 401:
                 raise error.Abort(_('authorization failed'))
             if inst.code == 404:
                 raise NotFound()
             else:
                 raise
 
     def filter_real_bug_ids(self, bugs):
         '''remove bug IDs that do not exist in Bugzilla from bugs.'''
         badbugs = set()
         for bugid in bugs:
             burl = self.apiurl(('bug', bugid), include_fields='status')
             try:
                 self._fetch(burl)
             except NotFound:
                 badbugs.add(bugid)
         for bugid in badbugs:
             del bugs[bugid]
 
     def filter_cset_known_bug_ids(self, node, bugs):
         '''remove bug IDs where node occurs in comment text from bugs.'''
         sn = short(node)
         for bugid in bugs.keys():
             burl = self.apiurl(('bug', bugid, 'comment'), include_fields='text')
             result = self._fetch(burl)
             comments = result['bugs'][str(bugid)]['comments']
             if any(sn in c['text'] for c in comments):
                 self.ui.status(_('bug %d already knows about changeset %s\n') %
                                (bugid, sn))
                 del bugs[bugid]
 
     def updatebug(self, bugid, newstate, text, committer):
         '''update the specified bug. Add comment text and set new states.
 
         If possible add the comment as being from the committer of
         the changeset. Otherwise use the default Bugzilla user.
         '''
         bugmod = {}
         if 'hours' in newstate:
             bugmod['work_time'] = newstate['hours']
         if 'fix' in newstate:
             bugmod['status'] = self.fixstatus
             bugmod['resolution'] = self.fixresolution
         if bugmod:
             # if we have to change the bugs state do it here
             bugmod['comment'] = {
                 'comment': text,
                 'is_private': False,
                 'is_markdown': False,
             }
             burl = self.apiurl(('bug', bugid))
             self._submit(burl, bugmod, method='PUT')
             self.ui.debug('updated bug %s\n' % bugid)
         else:
             burl = self.apiurl(('bug', bugid, 'comment'))
             self._submit(burl, {
                 'comment': text,
                 'is_private': False,
                 'is_markdown': False,
             })
             self.ui.debug('added comment to bug %s\n' % bugid)
 
     def notify(self, bugs, committer):
         '''Force sending of Bugzilla notification emails.
 
         Only required if the access method does not trigger notification
         emails automatically.
         '''
         pass
 
 class bugzilla(object):
     # supported versions of bugzilla. different versions have
     # different schemas.
     _versions = {
         '2.16': bzmysql,
         '2.18': bzmysql_2_18,
         '3.0':  bzmysql_3_0,
         'xmlrpc': bzxmlrpc,
         'xmlrpc+email': bzxmlrpcemail,
         'restapi': bzrestapi,
         }
 
     def __init__(self, ui, repo):
         self.ui = ui
         self.repo = repo
 
         bzversion = self.ui.config('bugzilla', 'version')
         try:
             bzclass = bugzilla._versions[bzversion]
         except KeyError:
             raise error.Abort(_('bugzilla version %s not supported') %
                              bzversion)
         self.bzdriver = bzclass(self.ui)
 
         self.bug_re = re.compile(
             self.ui.config('bugzilla', 'regexp'), re.IGNORECASE)
         self.fix_re = re.compile(
             self.ui.config('bugzilla', 'fixregexp'), re.IGNORECASE)
         self.split_re = re.compile(r'\D+')
 
     def find_bugs(self, ctx):
         '''return bugs dictionary created from commit comment.
 
         Extract bug info from changeset comments. Filter out any that are
         not known to Bugzilla, and any that already have a reference to
         the given changeset in their comments.
         '''
         start = 0
         hours = 0.0
         bugs = {}
         bugmatch = self.bug_re.search(ctx.description(), start)
         fixmatch = self.fix_re.search(ctx.description(), start)
         while True:
             bugattribs = {}
             if not bugmatch and not fixmatch:
                 break
             if not bugmatch:
                 m = fixmatch
             elif not fixmatch:
                 m = bugmatch
             else:
                 if bugmatch.start() < fixmatch.start():
                     m = bugmatch
                 else:
                     m = fixmatch
             start = m.end()
             if m is bugmatch:
                 bugmatch = self.bug_re.search(ctx.description(), start)
                 if 'fix' in bugattribs:
                     del bugattribs['fix']
             else:
                 fixmatch = self.fix_re.search(ctx.description(), start)
                 bugattribs['fix'] = None
 
             try:
                 ids = m.group('ids')
             except IndexError:
                 ids = m.group(1)
             try:
                 hours = float(m.group('hours'))
                 bugattribs['hours'] = hours
             except IndexError:
                 pass
             except TypeError:
                 pass
             except ValueError:
                 self.ui.status(_("%s: invalid hours\n") % m.group('hours'))
 
             for id in self.split_re.split(ids):
                 if not id:
                     continue
                 bugs[int(id)] = bugattribs
         if bugs:
             self.bzdriver.filter_real_bug_ids(bugs)
         if bugs:
             self.bzdriver.filter_cset_known_bug_ids(ctx.node(), bugs)
         return bugs
 
     def update(self, bugid, newstate, ctx):
         '''update bugzilla bug with reference to changeset.'''
 
         def webroot(root):
             '''strip leading prefix of repo root and turn into
             url-safe path.'''
             count = int(self.ui.config('bugzilla', 'strip'))
             root = util.pconvert(root)
             while count > 0:
                 c = root.find('/')
                 if c == -1:
                     break
                 root = root[c + 1:]
                 count -= 1
             return root
 
         mapfile = None
         tmpl = self.ui.config('bugzilla', 'template')
         if not tmpl:
             mapfile = self.ui.config('bugzilla', 'style')
         if not mapfile and not tmpl:
             tmpl = _('changeset {node|short} in repo {root} refers '
                      'to bug {bug}.\ndetails:\n\t{desc|tabindent}')
         spec = logcmdutil.templatespec(tmpl, mapfile)
         t = logcmdutil.changesettemplater(self.ui, self.repo, spec)
         self.ui.pushbuffer()
         t.show(ctx, changes=ctx.changeset(),
                bug=str(bugid),
                hgweb=self.ui.config('web', 'baseurl'),
                root=self.repo.root,
                webroot=webroot(self.repo.root))
         data = self.ui.popbuffer()
         self.bzdriver.updatebug(bugid, newstate, data,
                                 stringutil.email(ctx.user()))
 
     def notify(self, bugs, committer):
         '''ensure Bugzilla users are notified of bug change.'''
         self.bzdriver.notify(bugs, committer)
 
 def hook(ui, repo, hooktype, node=None, **kwargs):
     '''add comment to bugzilla for each changeset that refers to a
     bugzilla bug id. only add a comment once per bug, so same change
     seen multiple times does not fill bug with duplicate data.'''
     if node is None:
         raise error.Abort(_('hook type %s does not pass a changeset id') %
                          hooktype)
     try:
         bz = bugzilla(ui, repo)
         ctx = repo[node]
         bugs = bz.find_bugs(ctx)
         if bugs:
             for bug in bugs:
                 bz.update(bug, bugs[bug], ctx)
             bz.notify(bugs, stringutil.email(ctx.user()))
     except Exception as e:
         raise error.Abort(_('Bugzilla error: %s') % e)

hgext/convert/cvsps.py

@@ -1,958 +1,958
 # Mercurial built-in replacement for cvsps.
 #
 # Copyright 2008, Frank Kingswood <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 from __future__ import absolute_import
 
 import os
 import re
 
 from mercurial.i18n import _
 from mercurial import (
     encoding,
     error,
     hook,
     pycompat,
     util,
 )
 from mercurial.utils import (
     dateutil,
     procutil,
     stringutil,
 )
 
 pickle = util.pickle
 
 class logentry(object):
     '''Class logentry has the following attributes:
         .author    - author name as CVS knows it
         .branch    - name of branch this revision is on
         .branches  - revision tuple of branches starting at this revision
         .comment   - commit message
         .commitid  - CVS commitid or None
         .date      - the commit date as a (time, tz) tuple
         .dead      - true if file revision is dead
         .file      - Name of file
         .lines     - a tuple (+lines, -lines) or None
         .parent    - Previous revision of this entry
         .rcs       - name of file as returned from CVS
         .revision  - revision number as tuple
         .tags      - list of tags on the file
         .synthetic - is this a synthetic "file ... added on ..." revision?
         .mergepoint - the branch that has been merged from (if present in
                       rlog output) or None
         .branchpoints - the branches that start at the current entry or empty
     '''
     def __init__(self, **entries):
         self.synthetic = False
         self.__dict__.update(entries)
 
     def __repr__(self):
         items = ("%s=%r"%(k, self.__dict__[k]) for k in sorted(self.__dict__))
         return "%s(%s)"%(type(self).__name__, ", ".join(items))
 
 class logerror(Exception):
     pass
 
 def getrepopath(cvspath):
     """Return the repository path from a CVS path.
 
     >>> getrepopath(b'/foo/bar')
     '/foo/bar'
     >>> getrepopath(b'c:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:10/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:10c:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:c:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:[e-mail]:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b':pserver:[e-mail]:c:/foo/bar')
     '/foo/bar'
     >>> getrepopath(b'user@server/path/to/repository')
     '/path/to/repository'
     """
     # According to CVS manual, CVS paths are expressed like:
     # [:method:][[user][:password]@]hostname[:[port]]/path/to/repository
     #
     # CVSpath is splitted into parts and then position of the first occurrence
     # of the '/' char after the '@' is located. The solution is the rest of the
     # string after that '/' sign including it
 
     parts = cvspath.split(':')
     atposition = parts[-1].find('@')
     start = 0
 
     if atposition != -1:
         start = atposition
 
     repopath = parts[-1][parts[-1].find('/', start):]
     return repopath
 
 def createlog(ui, directory=None, root="", rlog=True, cache=None):
     '''Collect the CVS rlog'''
 
     # Because we store many duplicate commit log messages, reusing strings
     # saves a lot of memory and pickle storage space.
     _scache = {}
     def scache(s):
         "return a shared version of a string"
         return _scache.setdefault(s, s)
 
     ui.status(_('collecting CVS rlog\n'))
 
     log = []      # list of logentry objects containing the CVS state
 
     # patterns to match in CVS (r)log output, by state of use
     re_00 = re.compile('RCS file: (.+)$')
     re_01 = re.compile('cvs \\[r?log aborted\\]: (.+)$')
     re_02 = re.compile('cvs (r?log|server): (.+)\n$')
     re_03 = re.compile("(Cannot access.+CVSROOT)|"
                        "(can't create temporary directory.+)$")
     re_10 = re.compile('Working file: (.+)$')
     re_20 = re.compile('symbolic names:')
     re_30 = re.compile('\t(.+): ([\\d.]+)$')
     re_31 = re.compile('----------------------------$')
     re_32 = re.compile('======================================='
                        '======================================$')
     re_50 = re.compile('revision ([\\d.]+)(\s+locked by:\s+.+;)?$')
     re_60 = re.compile(r'date:\s+(.+);\s+author:\s+(.+);\s+state:\s+(.+?);'
                        r'(\s+lines:\s+(\+\d+)?\s+(-\d+)?;)?'
                        r'(\s+commitid:\s+([^;]+);)?'
                        r'(.*mergepoint:\s+([^;]+);)?')
     re_70 = re.compile('branches: (.+);$')
 
     file_added_re = re.compile(r'file [^/]+ was (initially )?added on branch')
 
     prefix = ''   # leading path to strip of what we get from CVS
 
     if directory is None:
         # Current working directory
 
         # Get the real directory in the repository
         try:
             prefix = open(os.path.join('CVS','Repository'), 'rb').read().strip()
             directory = prefix
             if prefix == ".":
                 prefix = ""
         except IOError:
             raise logerror(_('not a CVS sandbox'))
 
         if prefix and not prefix.endswith(pycompat.ossep):
             prefix += pycompat.ossep
 
         # Use the Root file in the sandbox, if it exists
         try:
             root = open(os.path.join('CVS','Root'), 'rb').read().strip()
         except IOError:
             pass
 
     if not root:
         root = encoding.environ.get('CVSROOT', '')
 
     # read log cache if one exists
     oldlog = []
     date = None
 
     if cache:
         cachedir = os.path.expanduser('~/.hg.cvsps')
         if not os.path.exists(cachedir):
             os.mkdir(cachedir)
 
         # The cvsps cache pickle needs a uniquified name, based on the
         # repository location. The address may have all sort of nasties
         # in it, slashes, colons and such. So here we take just the
         # alphanumeric characters, concatenated in a way that does not
         # mix up the various components, so that
         #    :pserver:user@server:/path
         # and
         #    /pserver/user/server/path
         # are mapped to different cache file names.
         cachefile = root.split(":") + [directory, "cache"]
         cachefile = ['-'.join(re.findall(br'\w+', s)) for s in cachefile if s]
         cachefile = os.path.join(cachedir,
                                  '.'.join([s for s in cachefile if s]))
 
     if cache == 'update':
         try:
             ui.note(_('reading cvs log cache %s\n') % cachefile)
             oldlog = pickle.load(open(cachefile, 'rb'))
             for e in oldlog:
                 if not (util.safehasattr(e, 'branchpoints') and
                         util.safehasattr(e, 'commitid') and
                         util.safehasattr(e, 'mergepoint')):
                     ui.status(_('ignoring old cache\n'))
                     oldlog = []
                     break
 
             ui.note(_('cache has %d log entries\n') % len(oldlog))
         except Exception as e:
             ui.note(_('error reading cache: %r\n') % e)
 
         if oldlog:
             date = oldlog[-1].date    # last commit date as a (time,tz) tuple
             date = dateutil.datestr(date, '%Y/%m/%d %H:%M:%S %1%2')
 
     # build the CVS commandline
     cmd = ['cvs', '-q']
     if root:
         cmd.append('-d%s' % root)
         p = util.normpath(getrepopath(root))
         if not p.endswith('/'):
             p += '/'
         if prefix:
             # looks like normpath replaces "" by "."
             prefix = p + util.normpath(prefix)
         else:
             prefix = p
     cmd.append(['log', 'rlog'][rlog])
     if date:
         # no space between option and date string
         cmd.append('-d>%s' % date)
     cmd.append(directory)
 
     # state machine begins here
     tags = {}     # dictionary of revisions on current file with their tags
     branchmap = {} # mapping between branch names and revision numbers
     rcsmap = {}
     state = 0
     store = False # set when a new record can be appended
 
     cmd = [procutil.shellquote(arg) for arg in cmd]
     ui.note(_("running %s\n") % (' '.join(cmd)))
     ui.debug("prefix=%r directory=%r root=%r\n" % (prefix, directory, root))
 
-    pfp = procutil.popen(' '.join(cmd))
-    peek = pfp.readline()
+    pfp = procutil.popen(' '.join(cmd), 'rb')
+    peek = util.fromnativeeol(pfp.readline())
     while True:
         line = peek
         if line == '':
             break
-        peek = pfp.readline()
+        peek = util.fromnativeeol(pfp.readline())
         if line.endswith('\n'):
             line = line[:-1]
         #ui.debug('state=%d line=%r\n' % (state, line))
 
         if state == 0:
             # initial state, consume input until we see 'RCS file'
             match = re_00.match(line)
             if match:
                 rcs = match.group(1)
                 tags = {}
                 if rlog:
                     filename = util.normpath(rcs[:-2])
                     if filename.startswith(prefix):
                         filename = filename[len(prefix):]
                     if filename.startswith('/'):
                         filename = filename[1:]
                     if filename.startswith('Attic/'):
                         filename = filename[6:]
                     else:
                         filename = filename.replace('/Attic/', '/')
                     state = 2
                     continue
                 state = 1
                 continue
             match = re_01.match(line)
             if match:
                 raise logerror(match.group(1))
             match = re_02.match(line)
             if match:
                 raise logerror(match.group(2))
             if re_03.match(line):
                 raise logerror(line)
 
         elif state == 1:
             # expect 'Working file' (only when using log instead of rlog)
             match = re_10.match(line)
             assert match, _('RCS file must be followed by working file')
             filename = util.normpath(match.group(1))
             state = 2
 
         elif state == 2:
             # expect 'symbolic names'
             if re_20.match(line):
                 branchmap = {}
                 state = 3
 
         elif state == 3:
             # read the symbolic names and store as tags
             match = re_30.match(line)
             if match:
                 rev = [int(x) for x in match.group(2).split('.')]
 
                 # Convert magic branch number to an odd-numbered one
                 revn = len(rev)
                 if revn > 3 and (revn % 2) == 0 and rev[-2] == 0:
                     rev = rev[:-2] + rev[-1:]
                 rev = tuple(rev)
 
                 if rev not in tags:
                     tags[rev] = []
                 tags[rev].append(match.group(1))
                 branchmap[match.group(1)] = match.group(2)
 
             elif re_31.match(line):
                 state = 5
             elif re_32.match(line):
                 state = 0
 
         elif state == 4:
             # expecting '------' separator before first revision
             if re_31.match(line):
                 state = 5
             else:
                 assert not re_32.match(line), _('must have at least '
                                                 'some revisions')
 
         elif state == 5:
             # expecting revision number and possibly (ignored) lock indication
             # we create the logentry here from values stored in states 0 to 4,
             # as this state is re-entered for subsequent revisions of a file.
             match = re_50.match(line)
             assert match, _('expected revision number')
             e = logentry(rcs=scache(rcs),
                          file=scache(filename),
                          revision=tuple([int(x) for x in
                                          match.group(1).split('.')]),
                          branches=[],
                          parent=None,
                          commitid=None,
                          mergepoint=None,
                          branchpoints=set())
 
             state = 6
 
         elif state == 6:
             # expecting date, author, state, lines changed
             match = re_60.match(line)
             assert match, _('revision must be followed by date line')
             d = match.group(1)
             if d[2] == '/':
                 # Y2K
                 d = '19' + d
 
             if len(d.split()) != 3:
                 # cvs log dates always in GMT
                 d = d + ' UTC'
             e.date = dateutil.parsedate(d, ['%y/%m/%d %H:%M:%S',
                                         '%Y/%m/%d %H:%M:%S',
                                         '%Y-%m-%d %H:%M:%S'])
             e.author = scache(match.group(2))
             e.dead = match.group(3).lower() == 'dead'
 
             if match.group(5):
                 if match.group(6):
                     e.lines = (int(match.group(5)), int(match.group(6)))
                 else:
                     e.lines = (int(match.group(5)), 0)
             elif match.group(6):
                 e.lines = (0, int(match.group(6)))
             else:
                 e.lines = None
 
             if match.group(7): # cvs 1.12 commitid
                 e.commitid = match.group(8)
 
             if match.group(9): # cvsnt mergepoint
                 myrev = match.group(10).split('.')
                 if len(myrev) == 2: # head
                     e.mergepoint = 'HEAD'
                 else:
                     myrev = '.'.join(myrev[:-2] + ['0', myrev[-2]])
                     branches = [b for b in branchmap if branchmap[b] == myrev]
                     assert len(branches) == 1, ('unknown branch: %s'
                                                 % e.mergepoint)
                     e.mergepoint = branches[0]
 
             e.comment = []
             state = 7
 
         elif state == 7:
             # read the revision numbers of branches that start at this revision
             # or store the commit log message otherwise
             m = re_70.match(line)
             if m:
                 e.branches = [tuple([int(y) for y in x.strip().split('.')])
                                 for x in m.group(1).split(';')]
                 state = 8
             elif re_31.match(line) and re_50.match(peek):
                 state = 5
                 store = True
             elif re_32.match(line):
                 state = 0
                 store = True
             else:
                 e.comment.append(line)
 
         elif state == 8:
             # store commit log message
             if re_31.match(line):
                 cpeek = peek
                 if cpeek.endswith('\n'):
                     cpeek = cpeek[:-1]
                 if re_50.match(cpeek):
                     state = 5
                     store = True
                 else:
                     e.comment.append(line)
             elif re_32.match(line):
                 state = 0
                 store = True
             else:
                 e.comment.append(line)
 
         # When a file is added on a branch B1, CVS creates a synthetic
         # dead trunk revision 1.1 so that the branch has a root.
         # Likewise, if you merge such a file to a later branch B2 (one
         # that already existed when the file was added on B1), CVS
         # creates a synthetic dead revision 1.1.x.1 on B2.  Don't drop
         # these revisions now, but mark them synthetic so
         # createchangeset() can take care of them.
         if (store and
               e.dead and
               e.revision[-1] == 1 and      # 1.1 or 1.1.x.1
               len(e.comment) == 1 and
               file_added_re.match(e.comment[0])):
             ui.debug('found synthetic revision in %s: %r\n'
                      % (e.rcs, e.comment[0]))
             e.synthetic = True
 
         if store:
             # clean up the results and save in the log.
             store = False
             e.tags = sorted([scache(x) for x in tags.get(e.revision, [])])
             e.comment = scache('\n'.join(e.comment))
 
             revn = len(e.revision)
             if revn > 3 and (revn % 2) == 0:
                 e.branch = tags.get(e.revision[:-1], [None])[0]
             else:
                 e.branch = None
 
             # find the branches starting from this revision
             branchpoints = set()
             for branch, revision in branchmap.iteritems():
                 revparts = tuple([int(i) for i in revision.split('.')])
                 if len(revparts) < 2: # bad tags
                     continue
                 if revparts[-2] == 0 and revparts[-1] % 2 == 0:
                     # normal branch
                     if revparts[:-2] == e.revision:
                         branchpoints.add(branch)
                 elif revparts == (1, 1, 1): # vendor branch
                     if revparts in e.branches:
                         branchpoints.add(branch)
             e.branchpoints = branchpoints
 
             log.append(e)
 
             rcsmap[e.rcs.replace('/Attic/', '/')] = e.rcs
 
             if len(log) % 100 == 0:
                 ui.status(stringutil.ellipsis('%d %s' % (len(log), e.file), 80)
                           + '\n')
 
     log.sort(key=lambda x: (x.rcs, x.revision))
 
     # find parent revisions of individual files
     versions = {}
     for e in sorted(oldlog, key=lambda x: (x.rcs, x.revision)):
         rcs = e.rcs.replace('/Attic/', '/')
         if rcs in rcsmap:
             e.rcs = rcsmap[rcs]
         branch = e.revision[:-1]
         versions[(e.rcs, branch)] = e.revision
 
     for e in log:
         branch = e.revision[:-1]
         p = versions.get((e.rcs, branch), None)
         if p is None:
             p = e.revision[:-2]
         e.parent = p
         versions[(e.rcs, branch)] = e.revision
 
     # update the log cache
     if cache:
         if log:
             # join up the old and new logs
             log.sort(key=lambda x: x.date)
 
             if oldlog and oldlog[-1].date >= log[0].date:
                 raise logerror(_('log cache overlaps with new log entries,'
                                  ' re-run without cache.'))
 
             log = oldlog + log
 
             # write the new cachefile
             ui.note(_('writing cvs log cache %s\n') % cachefile)
             pickle.dump(log, open(cachefile, 'wb'))
         else:
             log = oldlog
 
     ui.status(_('%d log entries\n') % len(log))
 
     encodings = ui.configlist('convert', 'cvsps.logencoding')
     if encodings:
         def revstr(r):
             # this is needed, because logentry.revision is a tuple of "int"
             # (e.g. (1, 2) for "1.2")
             return '.'.join(pycompat.maplist(pycompat.bytestr, r))
 
         for entry in log:
             comment = entry.comment
             for e in encodings:
                 try:
                     entry.comment = comment.decode(e).encode('utf-8')
                     if ui.debugflag:
                         ui.debug("transcoding by %s: %s of %s\n" %
                                  (e, revstr(entry.revision), entry.file))
                     break
                 except UnicodeDecodeError:
                     pass # try next encoding
                 except LookupError as inst: # unknown encoding, maybe
                     raise error.Abort(inst,
                                       hint=_('check convert.cvsps.logencoding'
                                              ' configuration'))
             else:
                 raise error.Abort(_("no encoding can transcode"
                                     " CVS log message for %s of %s")
                                   % (revstr(entry.revision), entry.file),
                                   hint=_('check convert.cvsps.logencoding'
                                          ' configuration'))
 
     hook.hook(ui, None, "cvslog", True, log=log)
 
     return log
 
 
 class changeset(object):
     '''Class changeset has the following attributes:
         .id        - integer identifying this changeset (list index)
         .author    - author name as CVS knows it
         .branch    - name of branch this changeset is on, or None
         .comment   - commit message
         .commitid  - CVS commitid or None
         .date      - the commit date as a (time,tz) tuple
         .entries   - list of logentry objects in this changeset
         .parents   - list of one or two parent changesets
         .tags      - list of tags on this changeset
         .synthetic - from synthetic revision "file ... added on branch ..."
         .mergepoint- the branch that has been merged from or None
         .branchpoints- the branches that start at the current entry or empty
     '''
     def __init__(self, **entries):
         self.id = None
         self.synthetic = False
         self.__dict__.update(entries)
 
     def __repr__(self):
         items = ("%s=%r"%(k, self.__dict__[k]) for k in sorted(self.__dict__))
         return "%s(%s)"%(type(self).__name__, ", ".join(items))
 
 def createchangeset(ui, log, fuzz=60, mergefrom=None, mergeto=None):
     '''Convert log into changesets.'''
 
     ui.status(_('creating changesets\n'))
 
     # try to order commitids by date
     mindate = {}
     for e in log:
         if e.commitid:
             mindate[e.commitid] = min(e.date, mindate.get(e.commitid))
 
     # Merge changesets
     log.sort(key=lambda x: (mindate.get(x.commitid), x.commitid, x.comment,
                             x.author, x.branch, x.date, x.branchpoints))
 
     changesets = []
     files = set()
     c = None
     for i, e in enumerate(log):
 
         # Check if log entry belongs to the current changeset or not.
 
         # Since CVS is file-centric, two different file revisions with
         # different branchpoints should be treated as belonging to two
         # different changesets (and the ordering is important and not
         # honoured by cvsps at this point).
         #
         # Consider the following case:
         # foo 1.1 branchpoints: [MYBRANCH]
         # bar 1.1 branchpoints: [MYBRANCH, MYBRANCH2]
         #
         # Here foo is part only of MYBRANCH, but not MYBRANCH2, e.g. a
         # later version of foo may be in MYBRANCH2, so foo should be the
         # first changeset and bar the next and MYBRANCH and MYBRANCH2
         # should both start off of the bar changeset. No provisions are
         # made to ensure that this is, in fact, what happens.
         if not (c and e.branchpoints == c.branchpoints and
                 (# cvs commitids
                  (e.commitid is not None and e.commitid == c.commitid) or
                  (# no commitids, use fuzzy commit detection
                   (e.commitid is None or c.commitid is None) and
                    e.comment == c.comment and
                    e.author == c.author and
                    e.branch == c.branch and
                    ((c.date[0] + c.date[1]) <=
                     (e.date[0] + e.date[1]) <=
                     (c.date[0] + c.date[1]) + fuzz) and
                    e.file not in files))):
             c = changeset(comment=e.comment, author=e.author,
                           branch=e.branch, date=e.date,
                           entries=[], mergepoint=e.mergepoint,
                           branchpoints=e.branchpoints, commitid=e.commitid)
             changesets.append(c)
 
             files = set()
             if len(changesets) % 100 == 0:
                 t = '%d %s' % (len(changesets), repr(e.comment)[1:-1])
                 ui.status(stringutil.ellipsis(t, 80) + '\n')
 
         c.entries.append(e)
         files.add(e.file)
         c.date = e.date       # changeset date is date of latest commit in it
 
     # Mark synthetic changesets
 
     for c in changesets:
         # Synthetic revisions always get their own changeset, because
         # the log message includes the filename.  E.g. if you add file3
         # and file4 on a branch, you get four log entries and three
         # changesets:
         #   "File file3 was added on branch ..." (synthetic, 1 entry)
         #   "File file4 was added on branch ..." (synthetic, 1 entry)
         #   "Add file3 and file4 to fix ..."     (real, 2 entries)
         # Hence the check for 1 entry here.
         c.synthetic = len(c.entries) == 1 and c.entries[0].synthetic
 
     # Sort files in each changeset
 
     def entitycompare(l, r):
         'Mimic cvsps sorting order'
         l = l.file.split('/')
         r = r.file.split('/')
         nl = len(l)
         nr = len(r)
         n = min(nl, nr)
         for i in range(n):
             if i + 1 == nl and nl < nr:
                 return -1
             elif i + 1 == nr and nl > nr:
                 return +1
             elif l[i] < r[i]:
                 return -1
             elif l[i] > r[i]:
                 return +1
         return 0
 
     for c in changesets:
         c.entries.sort(entitycompare)
 
     # Sort changesets by date
 
     odd = set()
     def cscmp(l, r):
         d = sum(l.date) - sum(r.date)
         if d:
             return d
 
         # detect vendor branches and initial commits on a branch
         le = {}
         for e in l.entries:
             le[e.rcs] = e.revision
         re = {}
         for e in r.entries:
             re[e.rcs] = e.revision
 
         d = 0
         for e in l.entries:
             if re.get(e.rcs, None) == e.parent:
                 assert not d
                 d = 1
                 break
 
         for e in r.entries:
             if le.get(e.rcs, None) == e.parent:
                 if d:
                     odd.add((l, r))
                 d = -1
                 break
         # By this point, the changesets are sufficiently compared that
         # we don't really care about ordering. However, this leaves
         # some race conditions in the tests, so we compare on the
         # number of files modified, the files contained in each
         # changeset, and the branchpoints in the change to ensure test
         # output remains stable.
 
         # recommended replacement for cmp from
         # https://docs.python.org/3.0/whatsnew/3.0.html
         c = lambda x, y: (x > y) - (x < y)
         # Sort bigger changes first.
         if not d:
             d = c(len(l.entries), len(r.entries))
         # Try sorting by filename in the change.
         if not d:
             d = c([e.file for e in l.entries], [e.file for e in r.entries])
         # Try and put changes without a branch point before ones with
         # a branch point.
         if not d:
             d = c(len(l.branchpoints), len(r.branchpoints))
         return d
 
     changesets.sort(cscmp)
 
     # Collect tags
 
     globaltags = {}
     for c in changesets:
         for e in c.entries:
             for tag in e.tags:
                 # remember which is the latest changeset to have this tag
                 globaltags[tag] = c
 
     for c in changesets:
         tags = set()
         for e in c.entries:
             tags.update(e.tags)
         # remember tags only if this is the latest changeset to have it
         c.tags = sorted(tag for tag in tags if globaltags[tag] is c)
 
     # Find parent changesets, handle {{mergetobranch BRANCHNAME}}
     # by inserting dummy changesets with two parents, and handle
     # {{mergefrombranch BRANCHNAME}} by setting two parents.
 
     if mergeto is None:
         mergeto = r'{{mergetobranch ([-\w]+)}}'
     if mergeto:
         mergeto = re.compile(mergeto)
 
     if mergefrom is None:
         mergefrom = r'{{mergefrombranch ([-\w]+)}}'
     if mergefrom:
         mergefrom = re.compile(mergefrom)
 
     versions = {}    # changeset index where we saw any particular file version
     branches = {}    # changeset index where we saw a branch
     n = len(changesets)
     i = 0
     while i < n:
         c = changesets[i]
 
         for f in c.entries:
             versions[(f.rcs, f.revision)] = i
 
         p = None
         if c.branch in branches:
             p = branches[c.branch]
         else:
             # first changeset on a new branch
             # the parent is a changeset with the branch in its
             # branchpoints such that it is the latest possible
             # commit without any intervening, unrelated commits.
 
             for candidate in xrange(i):
                 if c.branch not in changesets[candidate].branchpoints:
                     if p is not None:
                         break
                     continue
                 p = candidate
 
         c.parents = []
         if p is not None:
             p = changesets[p]
 
             # Ensure no changeset has a synthetic changeset as a parent.
             while p.synthetic:
                 assert len(p.parents) <= 1, \
                        _('synthetic changeset cannot have multiple parents')
                 if p.parents:
                     p = p.parents[0]
                 else:
                     p = None
                     break
 
             if p is not None:
                 c.parents.append(p)
 
         if c.mergepoint:
             if c.mergepoint == 'HEAD':
                 c.mergepoint = None
             c.parents.append(changesets[branches[c.mergepoint]])
 
         if mergefrom:
             m = mergefrom.search(c.comment)
             if m:
                 m = m.group(1)
                 if m == 'HEAD':
                     m = None
                 try:
                     candidate = changesets[branches[m]]
                 except KeyError:
                     ui.warn(_("warning: CVS commit message references "
                               "non-existent branch %r:\n%s\n")
                             % (m, c.comment))
                 if m in branches and c.branch != m and not candidate.synthetic:
                     c.parents.append(candidate)
 
         if mergeto:
             m = mergeto.search(c.comment)
             if m:
                 if m.groups():
                     m = m.group(1)
                     if m == 'HEAD':
                         m = None
                 else:
                     m = None   # if no group found then merge to HEAD
                 if m in branches and c.branch != m:
                     # insert empty changeset for merge
                     cc = changeset(
                         author=c.author, branch=m, date=c.date,
                         comment='convert-repo: CVS merge from branch %s'
                         % c.branch,
                         entries=[], tags=[],
                         parents=[changesets[branches[m]], c])
                     changesets.insert(i + 1, cc)
                     branches[m] = i + 1
 
                     # adjust our loop counters now we have inserted a new entry
                     n += 1
                     i += 2
                     continue
 
         branches[c.branch] = i
         i += 1
 
     # Drop synthetic changesets (safe now that we have ensured no other
     # changesets can have them as parents).
     i = 0
     while i < len(changesets):
         if changesets[i].synthetic:
             del changesets[i]
         else:
             i += 1
 
     # Number changesets
 
     for i, c in enumerate(changesets):
         c.id = i + 1
 
     if odd:
         for l, r in odd:
             if l.id is not None and r.id is not None:
                 ui.warn(_('changeset %d is both before and after %d\n')
                         % (l.id, r.id))
 
     ui.status(_('%d changeset entries\n') % len(changesets))
 
     hook.hook(ui, None, "cvschangesets", True, changesets=changesets)
 
     return changesets
 
 
 def debugcvsps(ui, *args, **opts):
     '''Read CVS rlog for current directory or named path in
     repository, and convert the log to changesets based on matching
     commit log entries and dates.
     '''
     opts = pycompat.byteskwargs(opts)
     if opts["new_cache"]:
         cache = "write"
     elif opts["update_cache"]:
         cache = "update"
     else:
         cache = None
 
     revisions = opts["revisions"]
 
     try:
         if args:
             log = []
             for d in args:
                 log += createlog(ui, d, root=opts["root"], cache=cache)
         else:
             log = createlog(ui, root=opts["root"], cache=cache)
     except logerror as e:
         ui.write("%r\n"%e)
         return
 
     changesets = createchangeset(ui, log, opts["fuzz"])
     del log
 
     # Print changesets (optionally filtered)
 
     off = len(revisions)
     branches = {}    # latest version number in each branch
     ancestors = {}   # parent branch
     for cs in changesets:
 
         if opts["ancestors"]:
             if cs.branch not in branches and cs.parents and cs.parents[0].id:
                 ancestors[cs.branch] = (changesets[cs.parents[0].id - 1].branch,
                                         cs.parents[0].id)
             branches[cs.branch] = cs.id
 
         # limit by branches
         if opts["branches"] and (cs.branch or 'HEAD') not in opts["branches"]:
             continue
 
         if not off:
             # Note: trailing spaces on several lines here are needed to have
             #       bug-for-bug compatibility with cvsps.
             ui.write('---------------------\n')
             ui.write(('PatchSet %d \n' % cs.id))
             ui.write(('Date: %s\n' % dateutil.datestr(cs.date,
                                                  '%Y/%m/%d %H:%M:%S %1%2')))
             ui.write(('Author: %s\n' % cs.author))
             ui.write(('Branch: %s\n' % (cs.branch or 'HEAD')))
             ui.write(('Tag%s: %s \n' % (['', 's'][len(cs.tags) > 1],
                                   ','.join(cs.tags) or '(none)')))
             if cs.branchpoints:
                 ui.write(('Branchpoints: %s \n') %
                          ', '.join(sorted(cs.branchpoints)))
             if opts["parents"] and cs.parents:
                 if len(cs.parents) > 1:
                     ui.write(('Parents: %s\n' %
                              (','.join([str(p.id) for p in cs.parents]))))
                 else:
                     ui.write(('Parent: %d\n' % cs.parents[0].id))
 
             if opts["ancestors"]:
                 b = cs.branch
                 r = []
                 while b:
                     b, c = ancestors[b]
                     r.append('%s:%d:%d' % (b or "HEAD", c, branches[b]))
                 if r:
                     ui.write(('Ancestors: %s\n' % (','.join(r))))
 
             ui.write(('Log:\n'))
             ui.write('%s\n\n' % cs.comment)
             ui.write(('Members: \n'))
             for f in cs.entries:
                 fn = f.file
                 if fn.startswith(opts["prefix"]):
                     fn = fn[len(opts["prefix"]):]
                 ui.write('\t%s:%s->%s%s \n' % (
                         fn, '.'.join([str(x) for x in f.parent]) or 'INITIAL',
                         '.'.join([str(x) for x in f.revision]),
                         ['', '(DEAD)'][f.dead]))
             ui.write('\n')
 
         # have we seen the start tag?
         if revisions and off:
             if revisions[0] == str(cs.id) or \
                 revisions[0] in cs.tags:
                 off = False
 
         # see if we reached the end tag
         if len(revisions) > 1 and not off:
             if revisions[1] == str(cs.id) or \
                 revisions[1] in cs.tags:
                 break

mercurial/mail.py

@@ -1,341 +1,341
 # mail.py - mail sending bits for mercurial
 #
 # Copyright 2006 Matt Mackall <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import
 
 import email
 import email.charset
 import email.header
 import email.message
 import os
 import smtplib
 import socket
 import time
 
 from .i18n import _
 from . import (
     encoding,
     error,
     pycompat,
     sslutil,
     util,
 )
 from .utils import (
     procutil,
     stringutil,
 )
 
 class STARTTLS(smtplib.SMTP):
     '''Derived class to verify the peer certificate for STARTTLS.
 
     This class allows to pass any keyword arguments to SSL socket creation.
     '''
     def __init__(self, ui, host=None, **kwargs):
         smtplib.SMTP.__init__(self, **kwargs)
         self._ui = ui
         self._host = host
 
     def starttls(self, keyfile=None, certfile=None):
         if not self.has_extn("starttls"):
             msg = "STARTTLS extension not supported by server"
             raise smtplib.SMTPException(msg)
         (resp, reply) = self.docmd("STARTTLS")
         if resp == 220:
             self.sock = sslutil.wrapsocket(self.sock, keyfile, certfile,
                                            ui=self._ui,
                                            serverhostname=self._host)
             self.file = smtplib.SSLFakeFile(self.sock)
             self.helo_resp = None
             self.ehlo_resp = None
             self.esmtp_features = {}
             self.does_esmtp = 0
         return (resp, reply)
 
 class SMTPS(smtplib.SMTP):
     '''Derived class to verify the peer certificate for SMTPS.
 
     This class allows to pass any keyword arguments to SSL socket creation.
     '''
     def __init__(self, ui, keyfile=None, certfile=None, host=None,
                  **kwargs):
         self.keyfile = keyfile
         self.certfile = certfile
         smtplib.SMTP.__init__(self, **kwargs)
         self._host = host
         self.default_port = smtplib.SMTP_SSL_PORT
         self._ui = ui
 
     def _get_socket(self, host, port, timeout):
         if self.debuglevel > 0:
             self._ui.debug('connect: %r\n' % (host, port))
         new_socket = socket.create_connection((host, port), timeout)
         new_socket = sslutil.wrapsocket(new_socket,
                                         self.keyfile, self.certfile,
                                         ui=self._ui,
                                         serverhostname=self._host)
         self.file = smtplib.SSLFakeFile(new_socket)
         return new_socket
 
 def _smtp(ui):
     '''build an smtp connection and return a function to send mail'''
     local_hostname = ui.config('smtp', 'local_hostname')
     tls = ui.config('smtp', 'tls')
     # backward compatible: when tls = true, we use starttls.
     starttls = tls == 'starttls' or stringutil.parsebool(tls)
     smtps = tls == 'smtps'
     if (starttls or smtps) and not util.safehasattr(socket, 'ssl'):
         raise error.Abort(_("can't use TLS: Python SSL support not installed"))
     mailhost = ui.config('smtp', 'host')
     if not mailhost:
         raise error.Abort(_('smtp.host not configured - cannot send mail'))
     if smtps:
         ui.note(_('(using smtps)\n'))
         s = SMTPS(ui, local_hostname=local_hostname, host=mailhost)
     elif starttls:
         s = STARTTLS(ui, local_hostname=local_hostname, host=mailhost)
     else:
         s = smtplib.SMTP(local_hostname=local_hostname)
     if smtps:
         defaultport = 465
     else:
         defaultport = 25
     mailport = util.getport(ui.config('smtp', 'port', defaultport))
     ui.note(_('sending mail: smtp host %s, port %d\n') %
             (mailhost, mailport))
     s.connect(host=mailhost, port=mailport)
     if starttls:
         ui.note(_('(using starttls)\n'))
         s.ehlo()
         s.starttls()
         s.ehlo()
     if starttls or smtps:
         ui.note(_('(verifying remote certificate)\n'))
         sslutil.validatesocket(s.sock)
     username = ui.config('smtp', 'username')
     password = ui.config('smtp', 'password')
     if username and not password:
         password = ui.getpass()
     if username and password:
         ui.note(_('(authenticating to mail server as %s)\n') %
                   (username))
         try:
             s.login(username, password)
         except smtplib.SMTPException as inst:
             raise error.Abort(inst)
 
     def send(sender, recipients, msg):
         try:
             return s.sendmail(sender, recipients, msg)
         except smtplib.SMTPRecipientsRefused as inst:
             recipients = [r[1] for r in inst.recipients.values()]
             raise error.Abort('\n' + '\n'.join(recipients))
         except smtplib.SMTPException as inst:
             raise error.Abort(inst)
 
     return send
 
 def _sendmail(ui, sender, recipients, msg):
     '''send mail using sendmail.'''
     program = ui.config('email', 'method')
     cmdline = '%s -f %s %s' % (program, stringutil.email(sender),
                                ' '.join(map(stringutil.email, recipients)))
     ui.note(_('sending mail: %s\n') % cmdline)
-    fp = procutil.popen(cmdline, 'w')
-    fp.write(msg)
+    fp = procutil.popen(cmdline, 'wb')
+    fp.write(util.tonativeeol(msg))
     ret = fp.close()
     if ret:
         raise error.Abort('%s %s' % (
             os.path.basename(program.split(None, 1)[0]),
             procutil.explainexit(ret)[0]))
 
 def _mbox(mbox, sender, recipients, msg):
     '''write mails to mbox'''
     fp = open(mbox, 'ab+')
     # Should be time.asctime(), but Windows prints 2-characters day
     # of month instead of one. Make them print the same thing.
     date = time.strftime(r'%a %b %d %H:%M:%S %Y', time.localtime())
     fp.write('From %s %s\n' % (sender, date))
     fp.write(msg)
     fp.write('\n\n')
     fp.close()
 
 def connect(ui, mbox=None):
     '''make a mail connection. return a function to send mail.
     call as sendmail(sender, list-of-recipients, msg).'''
     if mbox:
         open(mbox, 'wb').close()
         return lambda s, r, m: _mbox(mbox, s, r, m)
     if ui.config('email', 'method') == 'smtp':
         return _smtp(ui)
     return lambda s, r, m: _sendmail(ui, s, r, m)
 
 def sendmail(ui, sender, recipients, msg, mbox=None):
     send = connect(ui, mbox=mbox)
     return send(sender, recipients, msg)
 
 def validateconfig(ui):
     '''determine if we have enough config data to try sending email.'''
     method = ui.config('email', 'method')
     if method == 'smtp':
         if not ui.config('smtp', 'host'):
             raise error.Abort(_('smtp specified as email transport, '
                                'but no smtp host configured'))
     else:
         if not procutil.findexe(method):
             raise error.Abort(_('%r specified as email transport, '
                                'but not in PATH') % method)
 
 def codec2iana(cs):
     ''''''
     cs = pycompat.sysbytes(email.charset.Charset(cs).input_charset.lower())
 
     # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1"
     if cs.startswith("iso") and not cs.startswith("iso-"):
         return "iso-" + cs[3:]
     return cs
 
 def mimetextpatch(s, subtype='plain', display=False):
     '''Return MIME message suitable for a patch.
     Charset will be detected by first trying to decode as us-ascii, then utf-8,
     and finally the global encodings. If all those fail, fall back to
     ISO-8859-1, an encoding with that allows all byte sequences.
     Transfer encodings will be used if necessary.'''
 
     cs = ['us-ascii', 'utf-8', encoding.encoding, encoding.fallbackencoding]
     if display:
         return mimetextqp(s, subtype, 'us-ascii')
     for charset in cs:
         try:
             s.decode(pycompat.sysstr(charset))
             return mimetextqp(s, subtype, codec2iana(charset))
         except UnicodeDecodeError:
             pass
 
     return mimetextqp(s, subtype, "iso-8859-1")
 
 def mimetextqp(body, subtype, charset):
     '''Return MIME message.
     Quoted-printable transfer encoding will be used if necessary.
     '''
     cs = email.charset.Charset(charset)
     msg = email.message.Message()
     msg.set_type(pycompat.sysstr('text/' + subtype))
 
     for line in body.splitlines():
         if len(line) > 950:
             cs.body_encoding = email.charset.QP
             break
 
     msg.set_payload(body, cs)
 
     return msg
 
 def _charsets(ui):
     '''Obtains charsets to send mail parts not containing patches.'''
     charsets = [cs.lower() for cs in ui.configlist('email', 'charsets')]
     fallbacks = [encoding.fallbackencoding.lower(),
                  encoding.encoding.lower(), 'utf-8']
     for cs in fallbacks: # find unique charsets while keeping order
         if cs not in charsets:
             charsets.append(cs)
     return [cs for cs in charsets if not cs.endswith('ascii')]
 
 def _encode(ui, s, charsets):
     '''Returns (converted) string, charset tuple.
     Finds out best charset by cycling through sendcharsets in descending
     order. Tries both encoding and fallbackencoding for input. Only as
     last resort send as is in fake ascii.
     Caveat: Do not use for mail parts containing patches!'''
     try:
         s.decode('ascii')
     except UnicodeDecodeError:
         sendcharsets = charsets or _charsets(ui)
         for ics in (encoding.encoding, encoding.fallbackencoding):
             try:
                 u = s.decode(ics)
             except UnicodeDecodeError:
                 continue
             for ocs in sendcharsets:
                 try:
                     return u.encode(ocs), ocs
                 except UnicodeEncodeError:
                     pass
                 except LookupError:
                     ui.warn(_('ignoring invalid sendcharset: %s\n') % ocs)
     # if ascii, or all conversion attempts fail, send (broken) ascii
     return s, 'us-ascii'
 
 def headencode(ui, s, charsets=None, display=False):
     '''Returns RFC-2047 compliant header from given string.'''
     if not display:
         # split into words?
         s, cs = _encode(ui, s, charsets)
         return str(email.header.Header(s, cs))
     return s
 
 def _addressencode(ui, name, addr, charsets=None):
     name = headencode(ui, name, charsets)
     try:
         acc, dom = addr.split('@')
         acc = acc.encode('ascii')
         dom = dom.decode(encoding.encoding).encode('idna')
         addr = '%s@%s' % (acc, dom)
     except UnicodeDecodeError:
         raise error.Abort(_('invalid email address: %s') % addr)
     except ValueError:
         try:
             # too strict?
             addr = addr.encode('ascii')
         except UnicodeDecodeError:
             raise error.Abort(_('invalid local address: %s') % addr)
     return email.utils.formataddr((name, addr))
 
 def addressencode(ui, address, charsets=None, display=False):
     '''Turns address into RFC-2047 compliant header.'''
     if display or not address:
         return address or ''
     name, addr = email.utils.parseaddr(address)
     return _addressencode(ui, name, addr, charsets)
 
 def addrlistencode(ui, addrs, charsets=None, display=False):
     '''Turns a list of addresses into a list of RFC-2047 compliant headers.
     A single element of input list may contain multiple addresses, but output
     always has one address per item'''
     if display:
         return [a.strip() for a in addrs if a.strip()]
 
     result = []
     for name, addr in email.utils.getaddresses(addrs):
         if name or addr:
             result.append(_addressencode(ui, name, addr, charsets))
     return result
 
 def mimeencode(ui, s, charsets=None, display=False):
     '''creates mime text object, encodes it if needed, and sets
     charset and transfer-encoding accordingly.'''
     cs = 'us-ascii'
     if not display:
         s, cs = _encode(ui, s, charsets)
     return mimetextqp(s, 'plain', cs)
 
 def headdecode(s):
     '''Decodes RFC-2047 header'''
     uparts = []
     for part, charset in email.header.decode_header(s):
         if charset is not None:
             try:
                 uparts.append(part.decode(charset))
                 continue
             except UnicodeDecodeError:
                 pass
         try:
             uparts.append(part.decode('UTF-8'))
             continue
         except UnicodeDecodeError:
             pass
         uparts.append(part.decode('ISO-8859-1'))
     return encoding.unitolocal(u' '.join(uparts))

mercurial/patch.py

@@ -1,2911 +1,2912
 # patch.py - patch file parsing routines
 #
 # Copyright 2006 Brendan Cully <[e-mail]>
 # Copyright 2007 Chris Mason <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 from __future__ import absolute_import, print_function
 
 import collections
 import copy
 import difflib
 import email
 import errno
 import hashlib
 import os
 import posixpath
 import re
 import shutil
 import tempfile
 import zlib
 
 from .i18n import _
 from .node import (
     hex,
     short,
 )
 from . import (
     copies,
     encoding,
     error,
     mail,
     mdiff,
     pathutil,
     policy,
     pycompat,
     scmutil,
     similar,
     util,
     vfs as vfsmod,
 )
 from .utils import (
     dateutil,
     procutil,
     stringutil,
 )
 
 diffhelpers = policy.importmod(r'diffhelpers')
 stringio = util.stringio
 
 gitre = re.compile(br'diff --git a/(.*) b/(.*)')
 tabsplitter = re.compile(br'(\t+|[^\t]+)')
 _nonwordre = re.compile(br'([^a-zA-Z0-9_\x80-\xff])')
 
 PatchError = error.PatchError
 
 # public functions
 
 def split(stream):
     '''return an iterator of individual patches from a stream'''
     def isheader(line, inheader):
         if inheader and line[0] in (' ', '\t'):
             # continuation
             return True
         if line[0] in (' ', '-', '+'):
             # diff line - don't check for header pattern in there
             return False
         l = line.split(': ', 1)
         return len(l) == 2 and ' ' not in l[0]
 
     def chunk(lines):
         return stringio(''.join(lines))
 
     def hgsplit(stream, cur):
         inheader = True
 
         for line in stream:
             if not line.strip():
                 inheader = False
             if not inheader and line.startswith('# HG changeset patch'):
                 yield chunk(cur)
                 cur = []
                 inheader = True
 
             cur.append(line)
 
         if cur:
             yield chunk(cur)
 
     def mboxsplit(stream, cur):
         for line in stream:
             if line.startswith('From '):
                 for c in split(chunk(cur[1:])):
                     yield c
                 cur = []
 
             cur.append(line)
 
         if cur:
             for c in split(chunk(cur[1:])):
                 yield c
 
     def mimesplit(stream, cur):
         def msgfp(m):
             fp = stringio()
             g = email.Generator.Generator(fp, mangle_from_=False)
             g.flatten(m)
             fp.seek(0)
             return fp
 
         for line in stream:
             cur.append(line)
         c = chunk(cur)
 
         m = pycompat.emailparser().parse(c)
         if not m.is_multipart():
             yield msgfp(m)
         else:
             ok_types = ('text/plain', 'text/x-diff', 'text/x-patch')
             for part in m.walk():
                 ct = part.get_content_type()
                 if ct not in ok_types:
                     continue
                 yield msgfp(part)
 
     def headersplit(stream, cur):
         inheader = False
 
         for line in stream:
             if not inheader and isheader(line, inheader):
                 yield chunk(cur)
                 cur = []
                 inheader = True
             if inheader and not isheader(line, inheader):
                 inheader = False
 
             cur.append(line)
 
         if cur:
             yield chunk(cur)
 
     def remainder(cur):
         yield chunk(cur)
 
     class fiter(object):
         def __init__(self, fp):
             self.fp = fp
 
         def __iter__(self):
             return self
 
         def next(self):
             l = self.fp.readline()
             if not l:
                 raise StopIteration
             return l
 
         __next__ = next
 
     inheader = False
     cur = []
 
     mimeheaders = ['content-type']
 
     if not util.safehasattr(stream, 'next'):
         # http responses, for example, have readline but not next
         stream = fiter(stream)
 
     for line in stream:
         cur.append(line)
         if line.startswith('# HG changeset patch'):
             return hgsplit(stream, cur)
         elif line.startswith('From '):
             return mboxsplit(stream, cur)
         elif isheader(line, inheader):
             inheader = True
             if line.split(':', 1)[0].lower() in mimeheaders:
                 # let email parser handle this
                 return mimesplit(stream, cur)
         elif line.startswith('--- ') and inheader:
             # No evil headers seen by diff start, split by hand
             return headersplit(stream, cur)
         # Not enough info, keep reading
 
     # if we are here, we have a very plain patch
     return remainder(cur)
 
 ## Some facility for extensible patch parsing:
 # list of pairs ("header to match", "data key")
 patchheadermap = [('Date', 'date'),
                   ('Branch', 'branch'),
                   ('Node ID', 'nodeid'),
                  ]
 
 def extract(ui, fileobj):
     '''extract patch from data read from fileobj.
 
     patch can be a normal patch or contained in an email message.
 
     return a dictionary. Standard keys are:
       - filename,
       - message,
       - user,
       - date,
       - branch,
       - node,
       - p1,
       - p2.
     Any item can be missing from the dictionary. If filename is missing,
     fileobj did not contain a patch. Caller must unlink filename when done.'''
 
     # attempt to detect the start of a patch
     # (this heuristic is borrowed from quilt)
     diffre = re.compile(br'^(?:Index:[ \t]|diff[ \t]-|RCS file: |'
                         br'retrieving revision [0-9]+(\.[0-9]+)*$|'
                         br'---[ \t].*?^\+\+\+[ \t]|'
                         br'\*\*\*[ \t].*?^---[ \t])',
                         re.MULTILINE | re.DOTALL)
 
     data = {}
     fd, tmpname = tempfile.mkstemp(prefix='hg-patch-')
     tmpfp = os.fdopen(fd, r'wb')
     try:
         msg = pycompat.emailparser().parse(fileobj)
 
         subject = msg['Subject'] and mail.headdecode(msg['Subject'])
         data['user'] = msg['From'] and mail.headdecode(msg['From'])
         if not subject and not data['user']:
             # Not an email, restore parsed headers if any
             subject = '\n'.join(': '.join(map(encoding.strtolocal, h))
                                 for h in msg.items()) + '\n'
 
         # should try to parse msg['Date']
         parents = []
 
         if subject:
             if subject.startswith('[PATCH'):
                 pend = subject.find(']')
                 if pend >= 0:
                     subject = subject[pend + 1:].lstrip()
             subject = re.sub(br'\n[ \t]+', ' ', subject)
             ui.debug('Subject: %s\n' % subject)
         if data['user']:
             ui.debug('From: %s\n' % data['user'])
         diffs_seen = 0
         ok_types = ('text/plain', 'text/x-diff', 'text/x-patch')
         message = ''
         for part in msg.walk():
             content_type = pycompat.bytestr(part.get_content_type())
             ui.debug('Content-Type: %s\n' % content_type)
             if content_type not in ok_types:
                 continue
             payload = part.get_payload(decode=True)
             m = diffre.search(payload)
             if m:
                 hgpatch = False
                 hgpatchheader = False
                 ignoretext = False
 
                 ui.debug('found patch at byte %d\n' % m.start(0))
                 diffs_seen += 1
                 cfp = stringio()
                 for line in payload[:m.start(0)].splitlines():
                     if line.startswith('# HG changeset patch') and not hgpatch:
                         ui.debug('patch generated by hg export\n')
                         hgpatch = True
                         hgpatchheader = True
                         # drop earlier commit message content
                         cfp.seek(0)
                         cfp.truncate()
                         subject = None
                     elif hgpatchheader:
                         if line.startswith('# User '):
                             data['user'] = line[7:]
                             ui.debug('From: %s\n' % data['user'])
                         elif line.startswith("# Parent "):
                             parents.append(line[9:].lstrip())
                         elif line.startswith("# "):
                             for header, key in patchheadermap:
                                 prefix = '# %s ' % header
                                 if line.startswith(prefix):
                                     data[key] = line[len(prefix):]
                         else:
                             hgpatchheader = False
                     elif line == '---':
                         ignoretext = True
                     if not hgpatchheader and not ignoretext:
                         cfp.write(line)
                         cfp.write('\n')
                 message = cfp.getvalue()
                 if tmpfp:
                     tmpfp.write(payload)
                     if not payload.endswith('\n'):
                         tmpfp.write('\n')
             elif not diffs_seen and message and content_type == 'text/plain':
                 message += '\n' + payload
     except: # re-raises
         tmpfp.close()
         os.unlink(tmpname)
         raise
 
     if subject and not message.startswith(subject):
         message = '%s\n%s' % (subject, message)
     data['message'] = message
     tmpfp.close()
     if parents:
         data['p1'] = parents.pop(0)
         if parents:
             data['p2'] = parents.pop(0)
 
     if diffs_seen:
         data['filename'] = tmpname
     else:
         os.unlink(tmpname)
     return data
 
 class patchmeta(object):
     """Patched file metadata
 
     'op' is the performed operation within ADD, DELETE, RENAME, MODIFY
     or COPY.  'path' is patched file path. 'oldpath' is set to the
     origin file when 'op' is either COPY or RENAME, None otherwise. If
     file mode is changed, 'mode' is a tuple (islink, isexec) where
     'islink' is True if the file is a symlink and 'isexec' is True if
     the file is executable. Otherwise, 'mode' is None.
     """
     def __init__(self, path):
         self.path = path
         self.oldpath = None
         self.mode = None
         self.op = 'MODIFY'
         self.binary = False
 
     def setmode(self, mode):
         islink = mode & 0o20000
         isexec = mode & 0o100
         self.mode = (islink, isexec)
 
     def copy(self):
         other = patchmeta(self.path)
         other.oldpath = self.oldpath
         other.mode = self.mode
         other.op = self.op
         other.binary = self.binary
         return other
 
     def _ispatchinga(self, afile):
         if afile == '/dev/null':
             return self.op == 'ADD'
         return afile == 'a/' + (self.oldpath or self.path)
 
     def _ispatchingb(self, bfile):
         if bfile == '/dev/null':
             return self.op == 'DELETE'
         return bfile == 'b/' + self.path
 
     def ispatching(self, afile, bfile):
         return self._ispatchinga(afile) and self._ispatchingb(bfile)
 
     def __repr__(self):
         return "<patchmeta %s %r>" % (self.op, self.path)
 
 def readgitpatch(lr):
     """extract git-style metadata about patches from <patchname>"""
 
     # Filter patch for git information
     gp = None
     gitpatches = []
     for line in lr:
         line = line.rstrip(' \r\n')
         if line.startswith('diff --git a/'):
             m = gitre.match(line)
             if m:
                 if gp:
                     gitpatches.append(gp)
                 dst = m.group(2)
                 gp = patchmeta(dst)
         elif gp:
             if line.startswith('--- '):
                 gitpatches.append(gp)
                 gp = None
                 continue
             if line.startswith('rename from '):
                 gp.op = 'RENAME'
                 gp.oldpath = line[12:]
             elif line.startswith('rename to '):
                 gp.path = line[10:]
             elif line.startswith('copy from '):
                 gp.op = 'COPY'
                 gp.oldpath = line[10:]
             elif line.startswith('copy to '):
                 gp.path = line[8:]
             elif line.startswith('deleted file'):
                 gp.op = 'DELETE'
             elif line.startswith('new file mode '):
                 gp.op = 'ADD'
                 gp.setmode(int(line[-6:], 8))
             elif line.startswith('new mode '):
                 gp.setmode(int(line[-6:], 8))
             elif line.startswith('GIT binary patch'):
                 gp.binary = True
     if gp:
         gitpatches.append(gp)
 
     return gitpatches
 
 class linereader(object):
     # simple class to allow pushing lines back into the input stream
     def __init__(self, fp):
         self.fp = fp
         self.buf = []
 
     def push(self, line):
         if line is not None:
             self.buf.append(line)
 
     def readline(self):
         if self.buf:
             l = self.buf[0]
             del self.buf[0]
             return l
         return self.fp.readline()
 
     def __iter__(self):
         return iter(self.readline, '')
 
 class abstractbackend(object):
     def __init__(self, ui):
         self.ui = ui
 
     def getfile(self, fname):
         """Return target file data and flags as a (data, (islink,
         isexec)) tuple. Data is None if file is missing/deleted.
         """
         raise NotImplementedError
 
     def setfile(self, fname, data, mode, copysource):
         """Write data to target file fname and set its mode. mode is a
         (islink, isexec) tuple. If data is None, the file content should
         be left unchanged. If the file is modified after being copied,
         copysource is set to the original file name.
         """
         raise NotImplementedError
 
     def unlink(self, fname):
         """Unlink target file."""
         raise NotImplementedError
 
     def writerej(self, fname, failed, total, lines):
         """Write rejected lines for fname. total is the number of hunks
         which failed to apply and total the total number of hunks for this
         files.
         """
 
     def exists(self, fname):
         raise NotImplementedError
 
     def close(self):
         raise NotImplementedError
 
 class fsbackend(abstractbackend):
     def __init__(self, ui, basedir):
         super(fsbackend, self).__init__(ui)
         self.opener = vfsmod.vfs(basedir)
 
     def getfile(self, fname):
         if self.opener.islink(fname):
             return (self.opener.readlink(fname), (True, False))
 
         isexec = False
         try:
             isexec = self.opener.lstat(fname).st_mode & 0o100 != 0
         except OSError as e:
             if e.errno != errno.ENOENT:
                 raise
         try:
             return (self.opener.read(fname), (False, isexec))
         except IOError as e:
             if e.errno != errno.ENOENT:
                 raise
             return None, None
 
     def setfile(self, fname, data, mode, copysource):
         islink, isexec = mode
         if data is None:
             self.opener.setflags(fname, islink, isexec)
             return
         if islink:
             self.opener.symlink(data, fname)
         else:
             self.opener.write(fname, data)
             if isexec:
                 self.opener.setflags(fname, False, True)
 
     def unlink(self, fname):
         self.opener.unlinkpath(fname, ignoremissing=True)
 
     def writerej(self, fname, failed, total, lines):
         fname = fname + ".rej"
         self.ui.warn(
             _("%d out of %d hunks FAILED -- saving rejects to file %s\n") %
             (failed, total, fname))
         fp = self.opener(fname, 'w')
         fp.writelines(lines)
         fp.close()
 
     def exists(self, fname):
         return self.opener.lexists(fname)
 
 class workingbackend(fsbackend):
     def __init__(self, ui, repo, similarity):
         super(workingbackend, self).__init__(ui, repo.root)
         self.repo = repo
         self.similarity = similarity
         self.removed = set()
         self.changed = set()
         self.copied = []
 
     def _checkknown(self, fname):
         if self.repo.dirstate[fname] == '?' and self.exists(fname):
             raise PatchError(_('cannot patch %s: file is not tracked') % fname)
 
     def setfile(self, fname, data, mode, copysource):
         self._checkknown(fname)
         super(workingbackend, self).setfile(fname, data, mode, copysource)
         if copysource is not None:
             self.copied.append((copysource, fname))
         self.changed.add(fname)
 
     def unlink(self, fname):
         self._checkknown(fname)
         super(workingbackend, self).unlink(fname)
         self.removed.add(fname)
         self.changed.add(fname)
 
     def close(self):
         wctx = self.repo[None]
         changed = set(self.changed)
         for src, dst in self.copied:
             scmutil.dirstatecopy(self.ui, self.repo, wctx, src, dst)
         if self.removed:
             wctx.forget(sorted(self.removed))
             for f in self.removed:
                 if f not in self.repo.dirstate:
                     # File was deleted and no longer belongs to the
                     # dirstate, it was probably marked added then
                     # deleted, and should not be considered by
                     # marktouched().
                     changed.discard(f)
         if changed:
             scmutil.marktouched(self.repo, changed, self.similarity)
         return sorted(self.changed)
 
 class filestore(object):
     def __init__(self, maxsize=None):
         self.opener = None
         self.files = {}
         self.created = 0
         self.maxsize = maxsize
         if self.maxsize is None:
             self.maxsize = 4*(2**20)
         self.size = 0
         self.data = {}
 
     def setfile(self, fname, data, mode, copied=None):
         if self.maxsize < 0 or (len(data) + self.size) <= self.maxsize:
             self.data[fname] = (data, mode, copied)
             self.size += len(data)
         else:
             if self.opener is None:
                 root = tempfile.mkdtemp(prefix='hg-patch-')
                 self.opener = vfsmod.vfs(root)
             # Avoid filename issues with these simple names
             fn = '%d' % self.created
             self.opener.write(fn, data)
             self.created += 1
             self.files[fname] = (fn, mode, copied)
 
     def getfile(self, fname):
         if fname in self.data:
             return self.data[fname]
         if not self.opener or fname not in self.files:
             return None, None, None
         fn, mode, copied = self.files[fname]
         return self.opener.read(fn), mode, copied
 
     def close(self):
         if self.opener:
             shutil.rmtree(self.opener.base)
 
 class repobackend(abstractbackend):
     def __init__(self, ui, repo, ctx, store):
         super(repobackend, self).__init__(ui)
         self.repo = repo
         self.ctx = ctx
         self.store = store
         self.changed = set()
         self.removed = set()
         self.copied = {}
 
     def _checkknown(self, fname):
         if fname not in self.ctx:
             raise PatchError(_('cannot patch %s: file is not tracked') % fname)
 
     def getfile(self, fname):
         try:
             fctx = self.ctx[fname]
         except error.LookupError:
             return None, None
         flags = fctx.flags()
         return fctx.data(), ('l' in flags, 'x' in flags)
 
     def setfile(self, fname, data, mode, copysource):
         if copysource:
             self._checkknown(copysource)
         if data is None:
             data = self.ctx[fname].data()
         self.store.setfile(fname, data, mode, copysource)
         self.changed.add(fname)
         if copysource:
             self.copied[fname] = copysource
 
     def unlink(self, fname):
         self._checkknown(fname)
         self.removed.add(fname)
 
     def exists(self, fname):
         return fname in self.ctx
 
     def close(self):
         return self.changed | self.removed
 
 # @@ -start,len +start,len @@ or @@ -start +start @@ if len is 1
 unidesc = re.compile('@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@')
 contextdesc = re.compile('(?:---|\*\*\*) (\d+)(?:,(\d+))? (?:---|\*\*\*)')
 eolmodes = ['strict', 'crlf', 'lf', 'auto']
 
 class patchfile(object):
     def __init__(self, ui, gp, backend, store, eolmode='strict'):
         self.fname = gp.path
         self.eolmode = eolmode
         self.eol = None
         self.backend = backend
         self.ui = ui
         self.lines = []
         self.exists = False
         self.missing = True
         self.mode = gp.mode
         self.copysource = gp.oldpath
         self.create = gp.op in ('ADD', 'COPY', 'RENAME')
         self.remove = gp.op == 'DELETE'
         if self.copysource is None:
             data, mode = backend.getfile(self.fname)
         else:
             data, mode = store.getfile(self.copysource)[:2]
         if data is not None:
             self.exists = self.copysource is None or backend.exists(self.fname)
             self.missing = False
             if data:
                 self.lines = mdiff.splitnewlines(data)
             if self.mode is None:
                 self.mode = mode
             if self.lines:
                 # Normalize line endings
                 if self.lines[0].endswith('\r\n'):
                     self.eol = '\r\n'
                 elif self.lines[0].endswith('\n'):
                     self.eol = '\n'
                 if eolmode != 'strict':
                     nlines = []
                     for l in self.lines:
                         if l.endswith('\r\n'):
                             l = l[:-2] + '\n'
                         nlines.append(l)
                     self.lines = nlines
         else:
             if self.create:
                 self.missing = False
             if self.mode is None:
                 self.mode = (False, False)
         if self.missing:
             self.ui.warn(_("unable to find '%s' for patching\n") % self.fname)
             self.ui.warn(_("(use '--prefix' to apply patch relative to the "
                            "current directory)\n"))
 
         self.hash = {}
         self.dirty = 0
         self.offset = 0
         self.skew = 0
         self.rej = []
         self.fileprinted = False
         self.printfile(False)
         self.hunks = 0
 
     def writelines(self, fname, lines, mode):
         if self.eolmode == 'auto':
             eol = self.eol
         elif self.eolmode == 'crlf':
             eol = '\r\n'
         else:
             eol = '\n'
 
         if self.eolmode != 'strict' and eol and eol != '\n':
             rawlines = []
             for l in lines:
                 if l and l[-1] == '\n':
                     l = l[:-1] + eol
                 rawlines.append(l)
             lines = rawlines
 
         self.backend.setfile(fname, ''.join(lines), mode, self.copysource)
 
     def printfile(self, warn):
         if self.fileprinted:
             return
         if warn or self.ui.verbose:
             self.fileprinted = True
         s = _("patching file %s\n") % self.fname
         if warn:
             self.ui.warn(s)
         else:
             self.ui.note(s)
 
 
     def findlines(self, l, linenum):
         # looks through the hash and finds candidate lines.  The
         # result is a list of line numbers sorted based on distance
         # from linenum
 
         cand = self.hash.get(l, [])
         if len(cand) > 1:
             # resort our list of potentials forward then back.
             cand.sort(key=lambda x: abs(x - linenum))
         return cand
 
     def write_rej(self):
         # our rejects are a little different from patch(1).  This always
         # creates rejects in the same form as the original patch.  A file
         # header is inserted so that you can run the reject through patch again
         # without having to type the filename.
         if not self.rej:
             return
         base = os.path.basename(self.fname)
         lines = ["--- %s\n+++ %s\n" % (base, base)]
         for x in self.rej:
             for l in x.hunk:
                 lines.append(l)
                 if l[-1:] != '\n':
                     lines.append("\n\ No newline at end of file\n")
         self.backend.writerej(self.fname, len(self.rej), self.hunks, lines)
 
     def apply(self, h):
         if not h.complete():
             raise PatchError(_("bad hunk #%d %s (%d %d %d %d)") %
                             (h.number, h.desc, len(h.a), h.lena, len(h.b),
                             h.lenb))
 
         self.hunks += 1
 
         if self.missing:
             self.rej.append(h)
             return -1
 
         if self.exists and self.create:
             if self.copysource:
                 self.ui.warn(_("cannot create %s: destination already "
                                "exists\n") % self.fname)
             else:
                 self.ui.warn(_("file %s already exists\n") % self.fname)
             self.rej.append(h)
             return -1
 
         if isinstance(h, binhunk):
             if self.remove:
                 self.backend.unlink(self.fname)
             else:
                 l = h.new(self.lines)
                 self.lines[:] = l
                 self.offset += len(l)
                 self.dirty = True
             return 0
 
         horig = h
         if (self.eolmode in ('crlf', 'lf')
             or self.eolmode == 'auto' and self.eol):
             # If new eols are going to be normalized, then normalize
             # hunk data before patching. Otherwise, preserve input
             # line-endings.
             h = h.getnormalized()
 
         # fast case first, no offsets, no fuzz
         old, oldstart, new, newstart = h.fuzzit(0, False)
         oldstart += self.offset
         orig_start = oldstart
         # if there's skew we want to emit the "(offset %d lines)" even
         # when the hunk cleanly applies at start + skew, so skip the
         # fast case code
         if (self.skew == 0 and
             diffhelpers.testhunk(old, self.lines, oldstart) == 0):
             if self.remove:
                 self.backend.unlink(self.fname)
             else:
                 self.lines[oldstart:oldstart + len(old)] = new
                 self.offset += len(new) - len(old)
                 self.dirty = True
             return 0
 
         # ok, we couldn't match the hunk. Lets look for offsets and fuzz it
         self.hash = {}
         for x, s in enumerate(self.lines):
             self.hash.setdefault(s, []).append(x)
 
         for fuzzlen in xrange(self.ui.configint("patch", "fuzz") + 1):
             for toponly in [True, False]:
                 old, oldstart, new, newstart = h.fuzzit(fuzzlen, toponly)
                 oldstart = oldstart + self.offset + self.skew
                 oldstart = min(oldstart, len(self.lines))
                 if old:
                     cand = self.findlines(old[0][1:], oldstart)
                 else:
                     # Only adding lines with no or fuzzed context, just
                     # take the skew in account
                     cand = [oldstart]
 
                 for l in cand:
                     if not old or diffhelpers.testhunk(old, self.lines, l) == 0:
                         self.lines[l : l + len(old)] = new
                         self.offset += len(new) - len(old)
                         self.skew = l - orig_start
                         self.dirty = True
                         offset = l - orig_start - fuzzlen
                         if fuzzlen:
                             msg = _("Hunk #%d succeeded at %d "
                                     "with fuzz %d "
                                     "(offset %d lines).\n")
                             self.printfile(True)
                             self.ui.warn(msg %
                                 (h.number, l + 1, fuzzlen, offset))
                         else:
                             msg = _("Hunk #%d succeeded at %d "
                                     "(offset %d lines).\n")
                             self.ui.note(msg % (h.number, l + 1, offset))
                         return fuzzlen
         self.printfile(True)
         self.ui.warn(_("Hunk #%d FAILED at %d\n") % (h.number, orig_start))
         self.rej.append(horig)
         return -1
 
     def close(self):
         if self.dirty:
             self.writelines(self.fname, self.lines, self.mode)
         self.write_rej()
         return len(self.rej)
 
 class header(object):
     """patch header
     """
     diffgit_re = re.compile('diff --git a/(.*) b/(.*)$')
     diff_re = re.compile('diff -r .* (.*)$')
     allhunks_re = re.compile('(?:index|deleted file) ')
     pretty_re = re.compile('(?:new file|deleted file) ')
     special_re = re.compile('(?:index|deleted|copy|rename) ')
     newfile_re = re.compile('(?:new file)')
 
     def __init__(self, header):
         self.header = header
         self.hunks = []
 
     def binary(self):
         return any(h.startswith('index ') for h in self.header)
 
     def pretty(self, fp):
         for h in self.header:
             if h.startswith('index '):
                 fp.write(_('this modifies a binary file (all or nothing)\n'))
                 break
             if self.pretty_re.match(h):
                 fp.write(h)
                 if self.binary():
                     fp.write(_('this is a binary file\n'))
                 break
             if h.startswith('---'):
                 fp.write(_('%d hunks, %d lines changed\n') %
                          (len(self.hunks),
                           sum([max(h.added, h.removed) for h in self.hunks])))
                 break
             fp.write(h)
 
     def write(self, fp):
         fp.write(''.join(self.header))
 
     def allhunks(self):
         return any(self.allhunks_re.match(h) for h in self.header)
 
     def files(self):
         match = self.diffgit_re.match(self.header[0])
         if match:
             fromfile, tofile = match.groups()
             if fromfile == tofile:
                 return [fromfile]
             return [fromfile, tofile]
         else:
             return self.diff_re.match(self.header[0]).groups()
 
     def filename(self):
         return self.files()[-1]
 
     def __repr__(self):
         return '<header %s>' % (' '.join(map(repr, self.files())))
 
     def isnewfile(self):
         return any(self.newfile_re.match(h) for h in self.header)
 
     def special(self):
         # Special files are shown only at the header level and not at the hunk
         # level for example a file that has been deleted is a special file.
         # The user cannot change the content of the operation, in the case of
         # the deleted file he has to take the deletion or not take it, he
         # cannot take some of it.
         # Newly added files are special if they are empty, they are not special
         # if they have some content as we want to be able to change it
         nocontent = len(self.header) == 2
         emptynewfile = self.isnewfile() and nocontent
         return emptynewfile or \
                 any(self.special_re.match(h) for h in self.header)
 
 class recordhunk(object):
     """patch hunk
 
     XXX shouldn't we merge this with the other hunk class?
     """
 
     def __init__(self, header, fromline, toline, proc, before, hunk, after,
                  maxcontext=None):
         def trimcontext(lines, reverse=False):
             if maxcontext is not None:
                 delta = len(lines) - maxcontext
                 if delta > 0:
                     if reverse:
                         return delta, lines[delta:]
                     else:
                         return delta, lines[:maxcontext]
             return 0, lines
 
         self.header = header
         trimedbefore, self.before = trimcontext(before, True)
         self.fromline = fromline + trimedbefore
         self.toline = toline + trimedbefore
         _trimedafter, self.after = trimcontext(after, False)
         self.proc = proc
         self.hunk = hunk
         self.added, self.removed = self.countchanges(self.hunk)
 
     def __eq__(self, v):
         if not isinstance(v, recordhunk):
             return False
 
         return ((v.hunk == self.hunk) and
                 (v.proc == self.proc) and
                 (self.fromline == v.fromline) and
                 (self.header.files() == v.header.files()))
 
     def __hash__(self):
         return hash((tuple(self.hunk),
             tuple(self.header.files()),
             self.fromline,
             self.proc))
 
     def countchanges(self, hunk):
         """hunk -> (n+,n-)"""
         add = len([h for h in hunk if h.startswith('+')])
         rem = len([h for h in hunk if h.startswith('-')])
         return add, rem
 
     def reversehunk(self):
         """return another recordhunk which is the reverse of the hunk
 
         If this hunk is diff(A, B), the returned hunk is diff(B, A). To do
         that, swap fromline/toline and +/- signs while keep other things
         unchanged.
         """
         m = {'+': '-', '-': '+', '\\': '\\'}
         hunk = ['%s%s' % (m[l[0:1]], l[1:]) for l in self.hunk]
         return recordhunk(self.header, self.toline, self.fromline, self.proc,
                           self.before, hunk, self.after)
 
     def write(self, fp):
         delta = len(self.before) + len(self.after)
         if self.after and self.after[-1] == '\\ No newline at end of file\n':
             delta -= 1
         fromlen = delta + self.removed
         tolen = delta + self.added
         fp.write('@@ -%d,%d +%d,%d @@%s\n' %
                  (self.fromline, fromlen, self.toline, tolen,
                   self.proc and (' ' + self.proc)))
         fp.write(''.join(self.before + self.hunk + self.after))
 
     pretty = write
 
     def filename(self):
         return self.header.filename()
 
     def __repr__(self):
         return '<hunk %r@%d>' % (self.filename(), self.fromline)
 
 def getmessages():
     return {
         'multiple': {
             'apply': _("apply change %d/%d to '%s'?"),
             'discard': _("discard change %d/%d to '%s'?"),
             'record': _("record change %d/%d to '%s'?"),
         },
         'single': {
             'apply': _("apply this change to '%s'?"),
             'discard': _("discard this change to '%s'?"),
             'record': _("record this change to '%s'?"),
         },
         'help': {
             'apply': _('[Ynesfdaq?]'
                          '$$ &Yes, apply this change'
                          '$$ &No, skip this change'
                          '$$ &Edit this change manually'
                          '$$ &Skip remaining changes to this file'
                          '$$ Apply remaining changes to this &file'
                          '$$ &Done, skip remaining changes and files'
                          '$$ Apply &all changes to all remaining files'
                          '$$ &Quit, applying no changes'
                          '$$ &? (display help)'),
             'discard': _('[Ynesfdaq?]'
                          '$$ &Yes, discard this change'
                          '$$ &No, skip this change'
                          '$$ &Edit this change manually'
                          '$$ &Skip remaining changes to this file'
                          '$$ Discard remaining changes to this &file'
                          '$$ &Done, skip remaining changes and files'
                          '$$ Discard &all changes to all remaining files'
                          '$$ &Quit, discarding no changes'
                          '$$ &? (display help)'),
             'record': _('[Ynesfdaq?]'
                         '$$ &Yes, record this change'
                         '$$ &No, skip this change'
                         '$$ &Edit this change manually'
                         '$$ &Skip remaining changes to this file'
                         '$$ Record remaining changes to this &file'
                         '$$ &Done, skip remaining changes and files'
                         '$$ Record &all changes to all remaining files'
                         '$$ &Quit, recording no changes'
                         '$$ &? (display help)'),
         }
     }
 
 def filterpatch(ui, headers, operation=None):
     """Interactively filter patch chunks into applied-only chunks"""
     messages = getmessages()
 
     if operation is None:
         operation = 'record'
 
     def prompt(skipfile, skipall, query, chunk):
         """prompt query, and process base inputs
 
         - y/n for the rest of file
         - y/n for the rest
         - ? (help)
         - q (quit)
 
         Return True/False and possibly updated skipfile and skipall.
         """
         newpatches = None
         if skipall is not None:
             return skipall, skipfile, skipall, newpatches
         if skipfile is not None:
             return skipfile, skipfile, skipall, newpatches
         while True:
             resps = messages['help'][operation]
             r = ui.promptchoice("%s %s" % (query, resps))
             ui.write("\n")
             if r == 8: # ?
                 for c, t in ui.extractchoices(resps)[1]:
                     ui.write('%s - %s\n' % (c, encoding.lower(t)))
                 continue
             elif r == 0: # yes
                 ret = True
             elif r == 1: # no
                 ret = False
             elif r == 2: # Edit patch
                 if chunk is None:
                     ui.write(_('cannot edit patch for whole file'))
                     ui.write("\n")
                     continue
                 if chunk.header.binary():
                     ui.write(_('cannot edit patch for binary file'))
                     ui.write("\n")
                     continue
                 # Patch comment based on the Git one (based on comment at end of
                 # https://mercurial-scm.org/wiki/RecordExtension)
                 phelp = '---' + _("""
 To remove '-' lines, make them ' ' lines (context).
 To remove '+' lines, delete them.
 Lines starting with # will be removed from the patch.
 
 If the patch applies cleanly, the edited hunk will immediately be
 added to the record list. If it does not apply cleanly, a rejects
 file will be generated: you can use that when you try again. If
 all lines of the hunk are removed, then the edit is aborted and
 the hunk is left unchanged.
 """)
                 (patchfd, patchfn) = tempfile.mkstemp(prefix="hg-editor-",
                                                       suffix=".diff")
                 ncpatchfp = None
                 try:
                     # Write the initial patch
                     f = util.nativeeolwriter(os.fdopen(patchfd, r'wb'))
                     chunk.header.write(f)
                     chunk.write(f)
                     f.write('\n'.join(['# ' + i for i in phelp.splitlines()]))
                     f.close()
                     # Start the editor and wait for it to complete
                     editor = ui.geteditor()
                     ret = ui.system("%s \"%s\"" % (editor, patchfn),
                                     environ={'HGUSER': ui.username()},
                                     blockedtag='filterpatch')
                     if ret != 0:
                         ui.warn(_("editor exited with exit code %d\n") % ret)
                         continue
                     # Remove comment lines
                     patchfp = open(patchfn, r'rb')
                     ncpatchfp = stringio()
                     for line in util.iterfile(patchfp):
                         line = util.fromnativeeol(line)
                         if not line.startswith('#'):
                             ncpatchfp.write(line)
                     patchfp.close()
                     ncpatchfp.seek(0)
                     newpatches = parsepatch(ncpatchfp)
                 finally:
                     os.unlink(patchfn)
                     del ncpatchfp
                 # Signal that the chunk shouldn't be applied as-is, but
                 # provide the new patch to be used instead.
                 ret = False
             elif r == 3: # Skip
                 ret = skipfile = False
             elif r == 4: # file (Record remaining)
                 ret = skipfile = True
             elif r == 5: # done, skip remaining
                 ret = skipall = False
             elif r == 6: # all
                 ret = skipall = True
             elif r == 7: # quit
                 raise error.Abort(_('user quit'))
             return ret, skipfile, skipall, newpatches
 
     seen = set()
     applied = {}        # 'filename' -> [] of chunks
     skipfile, skipall = None, None
     pos, total = 1, sum(len(h.hunks) for h in headers)
     for h in headers:
         pos += len(h.hunks)
         skipfile = None
         fixoffset = 0
         hdr = ''.join(h.header)
         if hdr in seen:
             continue
         seen.add(hdr)
         if skipall is None:
             h.pretty(ui)
         msg = (_('examine changes to %s?') %
                _(' and ').join("'%s'" % f for f in h.files()))
         r, skipfile, skipall, np = prompt(skipfile, skipall, msg, None)
         if not r:
             continue
         applied[h.filename()] = [h]
         if h.allhunks():
             applied[h.filename()] += h.hunks
             continue
         for i, chunk in enumerate(h.hunks):
             if skipfile is None and skipall is None:
                 chunk.pretty(ui)
             if total == 1:
                 msg = messages['single'][operation] % chunk.filename()
             else:
                 idx = pos - len(h.hunks) + i
                 msg = messages['multiple'][operation] % (idx, total,
                                                          chunk.filename())
             r, skipfile, skipall, newpatches = prompt(skipfile,
                     skipall, msg, chunk)
             if r:
                 if fixoffset:
                     chunk = copy.copy(chunk)
                     chunk.toline += fixoffset
                 applied[chunk.filename()].append(chunk)
             elif newpatches is not None:
                 for newpatch in newpatches:
                     for newhunk in newpatch.hunks:
                         if fixoffset:
                             newhunk.toline += fixoffset
                         applied[newhunk.filename()].append(newhunk)
             else:
                 fixoffset += chunk.removed - chunk.added
     return (sum([h for h in applied.itervalues()
                if h[0].special() or len(h) > 1], []), {})
 class hunk(object):
     def __init__(self, desc, num, lr, context):
         self.number = num
         self.desc = desc
         self.hunk = [desc]
         self.a = []
         self.b = []
         self.starta = self.lena = None
         self.startb = self.lenb = None
         if lr is not None:
             if context:
                 self.read_context_hunk(lr)
             else:
                 self.read_unified_hunk(lr)
 
     def getnormalized(self):
         """Return a copy with line endings normalized to LF."""
 
         def normalize(lines):
             nlines = []
             for line in lines:
                 if line.endswith('\r\n'):
                     line = line[:-2] + '\n'
                 nlines.append(line)
             return nlines
 
         # Dummy object, it is rebuilt manually
         nh = hunk(self.desc, self.number, None, None)
         nh.number = self.number
         nh.desc = self.desc
         nh.hunk = self.hunk
         nh.a = normalize(self.a)
         nh.b = normalize(self.b)
         nh.starta = self.starta
         nh.startb = self.startb
         nh.lena = self.lena
         nh.lenb = self.lenb
         return nh
 
     def read_unified_hunk(self, lr):
         m = unidesc.match(self.desc)
         if not m:
             raise PatchError(_("bad hunk #%d") % self.number)
         self.starta, self.lena, self.startb, self.lenb = m.groups()
         if self.lena is None:
             self.lena = 1
         else:
             self.lena = int(self.lena)
         if self.lenb is None:
             self.lenb = 1
         else:
             self.lenb = int(self.lenb)
         self.starta = int(self.starta)
         self.startb = int(self.startb)
         diffhelpers.addlines(lr, self.hunk, self.lena, self.lenb, self.a,
                              self.b)
         # if we hit eof before finishing out the hunk, the last line will
         # be zero length.  Lets try to fix it up.
         while len(self.hunk[-1]) == 0:
             del self.hunk[-1]
             del self.a[-1]
             del self.b[-1]
             self.lena -= 1
             self.lenb -= 1
         self._fixnewline(lr)
 
     def read_context_hunk(self, lr):
         self.desc = lr.readline()
         m = contextdesc.match(self.desc)
         if not m:
             raise PatchError(_("bad hunk #%d") % self.number)
         self.starta, aend = m.groups()
         self.starta = int(self.starta)
         if aend is None:
             aend = self.starta
         self.lena = int(aend) - self.starta
         if self.starta:
             self.lena += 1
         for x in xrange(self.lena):
             l = lr.readline()
             if l.startswith('---'):
                 # lines addition, old block is empty
                 lr.push(l)
                 break
             s = l[2:]
             if l.startswith('- ') or l.startswith('! '):
                 u = '-' + s
             elif l.startswith('  '):
                 u = ' ' + s
             else:
                 raise PatchError(_("bad hunk #%d old text line %d") %
                                  (self.number, x))
             self.a.append(u)
             self.hunk.append(u)
 
         l = lr.readline()
         if l.startswith('\ '):
             s = self.a[-1][:-1]
             self.a[-1] = s
             self.hunk[-1] = s
             l = lr.readline()
         m = contextdesc.match(l)
         if not m:
             raise PatchError(_("bad hunk #%d") % self.number)
         self.startb, bend = m.groups()
         self.startb = int(self.startb)
         if bend is None:
             bend = self.startb
         self.lenb = int(bend) - self.startb
         if self.startb:
             self.lenb += 1
         hunki = 1
         for x in xrange(self.lenb):
             l = lr.readline()
             if l.startswith('\ '):
                 # XXX: the only way to hit this is with an invalid line range.
                 # The no-eol marker is not counted in the line range, but I
                 # guess there are diff(1) out there which behave differently.
                 s = self.b[-1][:-1]
                 self.b[-1] = s
                 self.hunk[hunki - 1] = s
                 continue
             if not l:
                 # line deletions, new block is empty and we hit EOF
                 lr.push(l)
                 break
             s = l[2:]
             if l.startswith('+ ') or l.startswith('! '):
                 u = '+' + s
             elif l.startswith('  '):
                 u = ' ' + s
             elif len(self.b) == 0:
                 # line deletions, new block is empty
                 lr.push(l)
                 break
             else:
                 raise PatchError(_("bad hunk #%d old text line %d") %
                                  (self.number, x))
             self.b.append(s)
             while True:
                 if hunki >= len(self.hunk):
                     h = ""
                 else:
                     h = self.hunk[hunki]
                 hunki += 1
                 if h == u:
                     break
                 elif h.startswith('-'):
                     continue
                 else:
                     self.hunk.insert(hunki - 1, u)
                     break
 
         if not self.a:
             # this happens when lines were only added to the hunk
             for x in self.hunk:
                 if x.startswith('-') or x.startswith(' '):
                     self.a.append(x)
         if not self.b:
             # this happens when lines were only deleted from the hunk
             for x in self.hunk:
                 if x.startswith('+') or x.startswith(' '):
                     self.b.append(x[1:])
         # @@ -start,len +start,len @@
         self.desc = "@@ -%d,%d +%d,%d @@\n" % (self.starta, self.lena,
                                              self.startb, self.lenb)
         self.hunk[0] = self.desc
         self._fixnewline(lr)
 
     def _fixnewline(self, lr):
         l = lr.readline()
         if l.startswith('\ '):
             diffhelpers.fix_newline(self.hunk, self.a, self.b)
         else:
             lr.push(l)
 
     def complete(self):
         return len(self.a) == self.lena and len(self.b) == self.lenb
 
     def _fuzzit(self, old, new, fuzz, toponly):
         # this removes context lines from the top and bottom of list 'l'.  It
         # checks the hunk to make sure only context lines are removed, and then
         # returns a new shortened list of lines.
         fuzz = min(fuzz, len(old))
         if fuzz:
             top = 0
             bot = 0
             hlen = len(self.hunk)
             for x in xrange(hlen - 1):
                 # the hunk starts with the @@ line, so use x+1
                 if self.hunk[x + 1][0] == ' ':
                     top += 1
                 else:
                     break
             if not toponly:
                 for x in xrange(hlen - 1):
                     if self.hunk[hlen - bot - 1][0] == ' ':
                         bot += 1
                     else:
                         break
 
             bot = min(fuzz, bot)
             top = min(fuzz, top)
             return old[top:len(old) - bot], new[top:len(new) - bot], top
         return old, new, 0
 
     def fuzzit(self, fuzz, toponly):
         old, new, top = self._fuzzit(self.a, self.b, fuzz, toponly)
         oldstart = self.starta + top
         newstart = self.startb + top
         # zero length hunk ranges already have their start decremented
         if self.lena and oldstart > 0:
             oldstart -= 1
         if self.lenb and newstart > 0:
             newstart -= 1
         return old, oldstart, new, newstart
 
 class binhunk(object):
     'A binary patch file.'
     def __init__(self, lr, fname):
         self.text = None
         self.delta = False
         self.hunk = ['GIT binary patch\n']
         self._fname = fname
         self._read(lr)
 
     def complete(self):
         return self.text is not None
 
     def new(self, lines):
         if self.delta:
             return [applybindelta(self.text, ''.join(lines))]
         return [self.text]
 
     def _read(self, lr):
         def getline(lr, hunk):
             l = lr.readline()
             hunk.append(l)
             return l.rstrip('\r\n')
 
         size = 0
         while True:
             line = getline(lr, self.hunk)
             if not line:
                 raise PatchError(_('could not extract "%s" binary data')
                                  % self._fname)
             if line.startswith('literal '):
                 size = int(line[8:].rstrip())
                 break
             if line.startswith('delta '):
                 size = int(line[6:].rstrip())
                 self.delta = True
                 break
         dec = []
         line = getline(lr, self.hunk)
         while len(line) > 1:
             l = line[0:1]
             if l <= 'Z' and l >= 'A':
                 l = ord(l) - ord('A') + 1
             else:
                 l = ord(l) - ord('a') + 27
             try:
                 dec.append(util.b85decode(line[1:])[:l])
             except ValueError as e:
                 raise PatchError(_('could not decode "%s" binary patch: %s')
                                  % (self._fname, stringutil.forcebytestr(e)))
             line = getline(lr, self.hunk)
         text = zlib.decompress(''.join(dec))
         if len(text) != size:
             raise PatchError(_('"%s" length is %d bytes, should be %d')
                              % (self._fname, len(text), size))
         self.text = text
 
 def parsefilename(str):
     # --- filename \t|space stuff
     s = str[4:].rstrip('\r\n')
     i = s.find('\t')
     if i < 0:
         i = s.find(' ')
         if i < 0:
             return s
     return s[:i]
 
 def reversehunks(hunks):
     '''reverse the signs in the hunks given as argument
 
     This function operates on hunks coming out of patch.filterpatch, that is
     a list of the form: [header1, hunk1, hunk2, header2...]. Example usage:
 
     >>> rawpatch = b"""diff --git a/folder1/g b/folder1/g
     ... --- a/folder1/g
     ... +++ b/folder1/g
     ... @@ -1,7 +1,7 @@
     ... +firstline
     ...  c
     ...  1
     ...  2
     ... + 3
     ... -4
     ...  5
     ...  d
     ... +lastline"""
     >>> hunks = parsepatch([rawpatch])
     >>> hunkscomingfromfilterpatch = []
     >>> for h in hunks:
     ...     hunkscomingfromfilterpatch.append(h)
     ...     hunkscomingfromfilterpatch.extend(h.hunks)
 
     >>> reversedhunks = reversehunks(hunkscomingfromfilterpatch)
     >>> from . import util
     >>> fp = util.stringio()
     >>> for c in reversedhunks:
     ...      c.write(fp)
     >>> fp.seek(0) or None
     >>> reversedpatch = fp.read()
     >>> print(pycompat.sysstr(reversedpatch))
     diff --git a/folder1/g b/folder1/g
     --- a/folder1/g
     +++ b/folder1/g
     @@ -1,4 +1,3 @@
     -firstline
      c
      1
      2
     @@ -2,6 +1,6 @@
      c
      1
      2
     - 3
     +4
      5
      d
     @@ -6,3 +5,2 @@
      5
      d
     -lastline
 
     '''
 
     newhunks = []
     for c in hunks:
         if util.safehasattr(c, 'reversehunk'):
             c = c.reversehunk()
         newhunks.append(c)
     return newhunks
 
 def parsepatch(originalchunks, maxcontext=None):
     """patch -> [] of headers -> [] of hunks
 
     If maxcontext is not None, trim context lines if necessary.
 
     >>> rawpatch = b'''diff --git a/folder1/g b/folder1/g
     ... --- a/folder1/g
     ... +++ b/folder1/g
     ... @@ -1,8 +1,10 @@
     ...  1
     ...  2
     ... -3
     ...  4
     ...  5
     ...  6
     ... +6.1
     ... +6.2
     ...  7
     ...  8
     ... +9'''
     >>> out = util.stringio()
     >>> headers = parsepatch([rawpatch], maxcontext=1)
     >>> for header in headers:
     ...     header.write(out)
     ...     for hunk in header.hunks:
     ...         hunk.write(out)
     >>> print(pycompat.sysstr(out.getvalue()))
     diff --git a/folder1/g b/folder1/g
     --- a/folder1/g
     +++ b/folder1/g
     @@ -2,3 +2,2 @@
      2
     -3
      4
     @@ -6,2 +5,4 @@
      6
     +6.1
     +6.2
      7
     @@ -8,1 +9,2 @@
      8
     +9
     """
     class parser(object):
         """patch parsing state machine"""
         def __init__(self):
             self.fromline = 0
             self.toline = 0
             self.proc = ''
             self.header = None
             self.context = []
             self.before = []
             self.hunk = []
             self.headers = []
 
         def addrange(self, limits):
             fromstart, fromend, tostart, toend, proc = limits
             self.fromline = int(fromstart)
             self.toline = int(tostart)
             self.proc = proc
 
         def addcontext(self, context):
             if self.hunk:
                 h = recordhunk(self.header, self.fromline, self.toline,
                         self.proc, self.before, self.hunk, context, maxcontext)
                 self.header.hunks.append(h)
                 self.fromline += len(self.before) + h.removed
                 self.toline += len(self.before) + h.added
                 self.before = []
                 self.hunk = []
             self.context = context
 
         def addhunk(self, hunk):
             if self.context:
                 self.before = self.context
                 self.context = []
             self.hunk = hunk
 
         def newfile(self, hdr):
             self.addcontext([])
             h = header(hdr)
             self.headers.append(h)
             self.header = h
 
         def addother(self, line):
             pass # 'other' lines are ignored
 
         def finished(self):
             self.addcontext([])
             return self.headers
 
         transitions = {
             'file': {'context': addcontext,
                      'file': newfile,
                      'hunk': addhunk,
                      'range': addrange},
             'context': {'file': newfile,
                         'hunk': addhunk,
                         'range': addrange,
                         'other': addother},
             'hunk': {'context': addcontext,
                      'file': newfile,
                      'range': addrange},
             'range': {'context': addcontext,
                       'hunk': addhunk},
             'other': {'other': addother},
             }
 
     p = parser()
     fp = stringio()
     fp.write(''.join(originalchunks))
     fp.seek(0)
 
     state = 'context'
     for newstate, data in scanpatch(fp):
         try:
             p.transitions[state][newstate](p, data)
         except KeyError:
             raise PatchError('unhandled transition: %s -> %s' %
                                    (state, newstate))
         state = newstate
     del fp
     return p.finished()
 
 def pathtransform(path, strip, prefix):
     '''turn a path from a patch into a path suitable for the repository
 
     prefix, if not empty, is expected to be normalized with a / at the end.
 
     Returns (stripped components, path in repository).
 
     >>> pathtransform(b'a/b/c', 0, b'')
     ('', 'a/b/c')
     >>> pathtransform(b'   a/b/c   ', 0, b'')
     ('', '   a/b/c')
     >>> pathtransform(b'   a/b/c   ', 2, b'')
     ('a/b/', 'c')
     >>> pathtransform(b'a/b/c', 0, b'd/e/')
     ('', 'd/e/a/b/c')
     >>> pathtransform(b'   a//b/c   ', 2, b'd/e/')
     ('a//b/', 'd/e/c')
     >>> pathtransform(b'a/b/c', 3, b'')
     Traceback (most recent call last):
     PatchError: unable to strip away 1 of 3 dirs from a/b/c
     '''
     pathlen = len(path)
     i = 0
     if strip == 0:
         return '', prefix + path.rstrip()
     count = strip
     while count > 0:
         i = path.find('/', i)
         if i == -1:
             raise PatchError(_("unable to strip away %d of %d dirs from %s") %
                              (count, strip, path))
         i += 1
         # consume '//' in the path
         while i < pathlen - 1 and path[i:i + 1] == '/':
             i += 1
         count -= 1
     return path[:i].lstrip(), prefix + path[i:].rstrip()
 
 def makepatchmeta(backend, afile_orig, bfile_orig, hunk, strip, prefix):
     nulla = afile_orig == "/dev/null"
     nullb = bfile_orig == "/dev/null"
     create = nulla and hunk.starta == 0 and hunk.lena == 0
     remove = nullb and hunk.startb == 0 and hunk.lenb == 0
     abase, afile = pathtransform(afile_orig, strip, prefix)
     gooda = not nulla and backend.exists(afile)
     bbase, bfile = pathtransform(bfile_orig, strip, prefix)
     if afile == bfile:
         goodb = gooda
     else:
         goodb = not nullb and backend.exists(bfile)
     missing = not goodb and not gooda and not create
 
     # some diff programs apparently produce patches where the afile is
     # not /dev/null, but afile starts with bfile
     abasedir = afile[:afile.rfind('/') + 1]
     bbasedir = bfile[:bfile.rfind('/') + 1]
     if (missing and abasedir == bbasedir and afile.startswith(bfile)
         and hunk.starta == 0 and hunk.lena == 0):
         create = True
         missing = False
 
     # If afile is "a/b/foo" and bfile is "a/b/foo.orig" we assume the
     # diff is between a file and its backup. In this case, the original
     # file should be patched (see original mpatch code).
     isbackup = (abase == bbase and bfile.startswith(afile))
     fname = None
     if not missing:
         if gooda and goodb:
             if isbackup:
                 fname = afile
             else:
                 fname = bfile
         elif gooda:
             fname = afile
 
     if not fname:
         if not nullb:
             if isbackup:
                 fname = afile
             else:
                 fname = bfile
         elif not nulla:
             fname = afile
         else:
             raise PatchError(_("undefined source and destination files"))
 
     gp = patchmeta(fname)
     if create:
         gp.op = 'ADD'
     elif remove:
         gp.op = 'DELETE'
     return gp
 
 def scanpatch(fp):
     """like patch.iterhunks, but yield different events
 
     - ('file',    [header_lines + fromfile + tofile])
     - ('context', [context_lines])
     - ('hunk',    [hunk_lines])
     - ('range',   (-start,len, +start,len, proc))
     """
     lines_re = re.compile(br'@@ -(\d+),(\d+) \+(\d+),(\d+) @@\s*(.*)')
     lr = linereader(fp)
 
     def scanwhile(first, p):
         """scan lr while predicate holds"""
         lines = [first]
         for line in iter(lr.readline, ''):
             if p(line):
                 lines.append(line)
             else:
                 lr.push(line)
                 break
         return lines
 
     for line in iter(lr.readline, ''):
         if line.startswith('diff --git a/') or line.startswith('diff -r '):
             def notheader(line):
                 s = line.split(None, 1)
                 return not s or s[0] not in ('---', 'diff')
             header = scanwhile(line, notheader)
             fromfile = lr.readline()
             if fromfile.startswith('---'):
                 tofile = lr.readline()
                 header += [fromfile, tofile]
             else:
                 lr.push(fromfile)
             yield 'file', header
         elif line[0:1] == ' ':
             yield 'context', scanwhile(line, lambda l: l[0] in ' \\')
         elif line[0] in '-+':
             yield 'hunk', scanwhile(line, lambda l: l[0] in '-+\\')
         else:
             m = lines_re.match(line)
             if m:
                 yield 'range', m.groups()
             else:
                 yield 'other', line
 
 def scangitpatch(lr, firstline):
     """
     Git patches can emit:
     - rename a to b
     - change b
     - copy a to c
     - change c
 
     We cannot apply this sequence as-is, the renamed 'a' could not be
     found for it would have been renamed already. And we cannot copy
     from 'b' instead because 'b' would have been changed already. So
     we scan the git patch for copy and rename commands so we can
     perform the copies ahead of time.
     """
     pos = 0
     try:
         pos = lr.fp.tell()
         fp = lr.fp
     except IOError:
         fp = stringio(lr.fp.read())
     gitlr = linereader(fp)
     gitlr.push(firstline)
     gitpatches = readgitpatch(gitlr)
     fp.seek(pos)
     return gitpatches
 
 def iterhunks(fp):
     """Read a patch and yield the following events:
     - ("file", afile, bfile, firsthunk): select a new target file.
     - ("hunk", hunk): a new hunk is ready to be applied, follows a
     "file" event.
     - ("git", gitchanges): current diff is in git format, gitchanges
     maps filenames to gitpatch records. Unique event.
     """
     afile = ""
     bfile = ""
     state = None
     hunknum = 0
     emitfile = newfile = False
     gitpatches = None
 
     # our states
     BFILE = 1
     context = None
     lr = linereader(fp)
 
     for x in iter(lr.readline, ''):
         if state == BFILE and (
             (not context and x.startswith('@'))
             or (context is not False and x.startswith('***************'))
             or x.startswith('GIT binary patch')):
             gp = None
             if (gitpatches and
                 gitpatches[-1].ispatching(afile, bfile)):
                 gp = gitpatches.pop()
             if x.startswith('GIT binary patch'):
                 h = binhunk(lr, gp.path)
             else:
                 if context is None and x.startswith('***************'):
                     context = True
                 h = hunk(x, hunknum + 1, lr, context)
             hunknum += 1
             if emitfile:
                 emitfile = False
                 yield 'file', (afile, bfile, h, gp and gp.copy() or None)
             yield 'hunk', h
         elif x.startswith('diff --git a/'):
             m = gitre.match(x.rstrip(' \r\n'))
             if not m:
                 continue
             if gitpatches is None:
                 # scan whole input for git metadata
                 gitpatches = scangitpatch(lr, x)
                 yield 'git', [g.copy() for g in gitpatches
                               if g.op in ('COPY', 'RENAME')]
                 gitpatches.reverse()
             afile = 'a/' + m.group(1)
             bfile = 'b/' + m.group(2)
             while gitpatches and not gitpatches[-1].ispatching(afile, bfile):
                 gp = gitpatches.pop()
                 yield 'file', ('a/' + gp.path, 'b/' + gp.path, None, gp.copy())
             if not gitpatches:
                 raise PatchError(_('failed to synchronize metadata for "%s"')
                                  % afile[2:])
             gp = gitpatches[-1]
             newfile = True
         elif x.startswith('---'):
             # check for a unified diff
             l2 = lr.readline()
             if not l2.startswith('+++'):
                 lr.push(l2)
                 continue
             newfile = True
             context = False
             afile = parsefilename(x)
             bfile = parsefilename(l2)
         elif x.startswith('***'):
             # check for a context diff
             l2 = lr.readline()
             if not l2.startswith('---'):
                 lr.push(l2)
                 continue
             l3 = lr.readline()
             lr.push(l3)
             if not l3.startswith("***************"):
                 lr.push(l2)
                 continue
             newfile = True
             context = True
             afile = parsefilename(x)
             bfile = parsefilename(l2)
 
         if newfile:
             newfile = False
             emitfile = True
             state = BFILE
             hunknum = 0
 
     while gitpatches:
         gp = gitpatches.pop()
         yield 'file', ('a/' + gp.path, 'b/' + gp.path, None, gp.copy())
 
 def applybindelta(binchunk, data):
     """Apply a binary delta hunk
     The algorithm used is the algorithm from git's patch-delta.c
     """
     def deltahead(binchunk):
         i = 0
         for c in binchunk:
             i += 1
             if not (ord(c) & 0x80):
                 return i
         return i
     out = ""
     s = deltahead(binchunk)
     binchunk = binchunk[s:]
     s = deltahead(binchunk)
     binchunk = binchunk[s:]
     i = 0
     while i < len(binchunk):
         cmd = ord(binchunk[i])
         i += 1
         if (cmd & 0x80):
             offset = 0
             size = 0
             if (cmd & 0x01):
                 offset = ord(binchunk[i])
                 i += 1
             if (cmd & 0x02):
                 offset |= ord(binchunk[i]) << 8
                 i += 1
             if (cmd & 0x04):
                 offset |= ord(binchunk[i]) << 16
                 i += 1
             if (cmd & 0x08):
                 offset |= ord(binchunk[i]) << 24
                 i += 1
             if (cmd & 0x10):
                 size = ord(binchunk[i])
                 i += 1
             if (cmd & 0x20):
                 size |= ord(binchunk[i]) << 8
                 i += 1
             if (cmd & 0x40):
                 size |= ord(binchunk[i]) << 16
                 i += 1
             if size == 0:
                 size = 0x10000
             offset_end = offset + size
             out += data[offset:offset_end]
         elif cmd != 0:
             offset_end = i + cmd
             out += binchunk[i:offset_end]
             i += cmd
         else:
             raise PatchError(_('unexpected delta opcode 0'))
     return out
 
 def applydiff(ui, fp, backend, store, strip=1, prefix='', eolmode='strict'):
     """Reads a patch from fp and tries to apply it.
 
     Returns 0 for a clean patch, -1 if any rejects were found and 1 if
     there was any fuzz.
 
     If 'eolmode' is 'strict', the patch content and patched file are
     read in binary mode. Otherwise, line endings are ignored when
     patching then normalized according to 'eolmode'.
     """
     return _applydiff(ui, fp, patchfile, backend, store, strip=strip,
                       prefix=prefix, eolmode=eolmode)
 
 def _canonprefix(repo, prefix):
     if prefix:
         prefix = pathutil.canonpath(repo.root, repo.getcwd(), prefix)
         if prefix != '':
             prefix += '/'
     return prefix
 
 def _applydiff(ui, fp, patcher, backend, store, strip=1, prefix='',
                eolmode='strict'):
     prefix = _canonprefix(backend.repo, prefix)
     def pstrip(p):
         return pathtransform(p, strip - 1, prefix)[1]
 
     rejects = 0
     err = 0
     current_file = None
 
     for state, values in iterhunks(fp):
         if state == 'hunk':
             if not current_file:
                 continue
             ret = current_file.apply(values)
             if ret > 0:
                 err = 1
         elif state == 'file':
             if current_file:
                 rejects += current_file.close()
                 current_file = None
             afile, bfile, first_hunk, gp = values
             if gp:
                 gp.path = pstrip(gp.path)
                 if gp.oldpath:
                     gp.oldpath = pstrip(gp.oldpath)
             else:
                 gp = makepatchmeta(backend, afile, bfile, first_hunk, strip,
                                    prefix)
             if gp.op == 'RENAME':
                 backend.unlink(gp.oldpath)
             if not first_hunk:
                 if gp.op == 'DELETE':
                     backend.unlink(gp.path)
                     continue
                 data, mode = None, None
                 if gp.op in ('RENAME', 'COPY'):
                     data, mode = store.getfile(gp.oldpath)[:2]
                     if data is None:
                         # This means that the old path does not exist
                         raise PatchError(_("source file '%s' does not exist")
                                            % gp.oldpath)
                 if gp.mode:
                     mode = gp.mode
                     if gp.op == 'ADD':
                         # Added files without content have no hunk and
                         # must be created
                         data = ''
                 if data or mode:
                     if (gp.op in ('ADD', 'RENAME', 'COPY')
                         and backend.exists(gp.path)):
                         raise PatchError(_("cannot create %s: destination "
                                            "already exists") % gp.path)
                     backend.setfile(gp.path, data, mode, gp.oldpath)
                 continue
             try:
                 current_file = patcher(ui, gp, backend, store,
                                        eolmode=eolmode)
             except PatchError as inst:
                 ui.warn(str(inst) + '\n')
                 current_file = None
                 rejects += 1
                 continue
         elif state == 'git':
             for gp in values:
                 path = pstrip(gp.oldpath)
                 data, mode = backend.getfile(path)
                 if data is None:
                     # The error ignored here will trigger a getfile()
                     # error in a place more appropriate for error
                     # handling, and will not interrupt the patching
                     # process.
                     pass
                 else:
                     store.setfile(path, data, mode)
         else:
             raise error.Abort(_('unsupported parser state: %s') % state)
 
     if current_file:
         rejects += current_file.close()
 
     if rejects:
         return -1
     return err
 
 def _externalpatch(ui, repo, patcher, patchname, strip, files,
                    similarity):
     """use <patcher> to apply <patchname> to the working directory.
     returns whether patch was applied with fuzz factor."""
 
     fuzz = False
     args = []
     cwd = repo.root
     if cwd:
         args.append('-d %s' % procutil.shellquote(cwd))
-    fp = procutil.popen('%s %s -p%d < %s' % (patcher, ' '.join(args), strip,
-                                             procutil.shellquote(patchname)))
+    cmd = ('%s %s -p%d < %s'
+           % (patcher, ' '.join(args), strip, procutil.shellquote(patchname)))
+    fp = procutil.popen(cmd, 'rb')
     try:
         for line in util.iterfile(fp):
             line = line.rstrip()
             ui.note(line + '\n')
             if line.startswith('patching file '):
                 pf = util.parsepatchoutput(line)
                 printed_file = False
                 files.add(pf)
             elif line.find('with fuzz') >= 0:
                 fuzz = True
                 if not printed_file:
                     ui.warn(pf + '\n')
                     printed_file = True
                 ui.warn(line + '\n')
             elif line.find('saving rejects to file') >= 0:
                 ui.warn(line + '\n')
             elif line.find('FAILED') >= 0:
                 if not printed_file:
                     ui.warn(pf + '\n')
                     printed_file = True
                 ui.warn(line + '\n')
     finally:
         if files:
             scmutil.marktouched(repo, files, similarity)
     code = fp.close()
     if code:
         raise PatchError(_("patch command failed: %s") %
                          procutil.explainexit(code)[0])
     return fuzz
 
 def patchbackend(ui, backend, patchobj, strip, prefix, files=None,
                  eolmode='strict'):
     if files is None:
         files = set()
     if eolmode is None:
         eolmode = ui.config('patch', 'eol')
     if eolmode.lower() not in eolmodes:
         raise error.Abort(_('unsupported line endings type: %s') % eolmode)
     eolmode = eolmode.lower()
 
     store = filestore()
     try:
         fp = open(patchobj, 'rb')
     except TypeError:
         fp = patchobj
     try:
         ret = applydiff(ui, fp, backend, store, strip=strip, prefix=prefix,
                         eolmode=eolmode)
     finally:
         if fp != patchobj:
             fp.close()
         files.update(backend.close())
         store.close()
     if ret < 0:
         raise PatchError(_('patch failed to apply'))
     return ret > 0
 
 def internalpatch(ui, repo, patchobj, strip, prefix='', files=None,
                   eolmode='strict', similarity=0):
     """use builtin patch to apply <patchobj> to the working directory.
     returns whether patch was applied with fuzz factor."""
     backend = workingbackend(ui, repo, similarity)
     return patchbackend(ui, backend, patchobj, strip, prefix, files, eolmode)
 
 def patchrepo(ui, repo, ctx, store, patchobj, strip, prefix, files=None,
               eolmode='strict'):
     backend = repobackend(ui, repo, ctx, store)
     return patchbackend(ui, backend, patchobj, strip, prefix, files, eolmode)
 
 def patch(ui, repo, patchname, strip=1, prefix='', files=None, eolmode='strict',
           similarity=0):
     """Apply <patchname> to the working directory.
 
     'eolmode' specifies how end of lines should be handled. It can be:
     - 'strict': inputs are read in binary mode, EOLs are preserved
     - 'crlf': EOLs are ignored when patching and reset to CRLF
     - 'lf': EOLs are ignored when patching and reset to LF
     - None: get it from user settings, default to 'strict'
     'eolmode' is ignored when using an external patcher program.
 
     Returns whether patch was applied with fuzz factor.
     """
     patcher = ui.config('ui', 'patch')
     if files is None:
         files = set()
     if patcher:
         return _externalpatch(ui, repo, patcher, patchname, strip,
                               files, similarity)
     return internalpatch(ui, repo, patchname, strip, prefix, files, eolmode,
                          similarity)
 
 def changedfiles(ui, repo, patchpath, strip=1, prefix=''):
     backend = fsbackend(ui, repo.root)
     prefix = _canonprefix(repo, prefix)
     with open(patchpath, 'rb') as fp:
         changed = set()
         for state, values in iterhunks(fp):
             if state == 'file':
                 afile, bfile, first_hunk, gp = values
                 if gp:
                     gp.path = pathtransform(gp.path, strip - 1, prefix)[1]
                     if gp.oldpath:
                         gp.oldpath = pathtransform(gp.oldpath, strip - 1,
                                                    prefix)[1]
                 else:
                     gp = makepatchmeta(backend, afile, bfile, first_hunk, strip,
                                        prefix)
                 changed.add(gp.path)
                 if gp.op == 'RENAME':
                     changed.add(gp.oldpath)
             elif state not in ('hunk', 'git'):
                 raise error.Abort(_('unsupported parser state: %s') % state)
         return changed
 
 class GitDiffRequired(Exception):
     pass
 
 def diffallopts(ui, opts=None, untrusted=False, section='diff'):
     '''return diffopts with all features supported and parsed'''
     return difffeatureopts(ui, opts=opts, untrusted=untrusted, section=section,
                            git=True, whitespace=True, formatchanging=True)
 
 diffopts = diffallopts
 
 def difffeatureopts(ui, opts=None, untrusted=False, section='diff', git=False,
                     whitespace=False, formatchanging=False):
     '''return diffopts with only opted-in features parsed
 
     Features:
     - git: git-style diffs
     - whitespace: whitespace options like ignoreblanklines and ignorews
     - formatchanging: options that will likely break or cause correctness issues
       with most diff parsers
     '''
     def get(key, name=None, getter=ui.configbool, forceplain=None):
         if opts:
             v = opts.get(key)
             # diffopts flags are either None-default (which is passed
             # through unchanged, so we can identify unset values), or
             # some other falsey default (eg --unified, which defaults
             # to an empty string). We only want to override the config
             # entries from hgrc with command line values if they
             # appear to have been set, which is any truthy value,
             # True, or False.
             if v or isinstance(v, bool):
                 return v
         if forceplain is not None and ui.plain():
             return forceplain
         return getter(section, name or key, untrusted=untrusted)
 
     # core options, expected to be understood by every diff parser
     buildopts = {
         'nodates': get('nodates'),
         'showfunc': get('show_function', 'showfunc'),
         'context': get('unified', getter=ui.config),
     }
     buildopts['worddiff'] = ui.configbool('experimental', 'worddiff')
     buildopts['xdiff'] = ui.configbool('experimental', 'xdiff')
 
     if git:
         buildopts['git'] = get('git')
 
         # since this is in the experimental section, we need to call
         # ui.configbool directory
         buildopts['showsimilarity'] = ui.configbool('experimental',
                                                     'extendedheader.similarity')
 
         # need to inspect the ui object instead of using get() since we want to
         # test for an int
         hconf = ui.config('experimental', 'extendedheader.index')
         if hconf is not None:
             hlen = None
             try:
                 # the hash config could be an integer (for length of hash) or a
                 # word (e.g. short, full, none)
                 hlen = int(hconf)
                 if hlen < 0 or hlen > 40:
                     msg = _("invalid length for extendedheader.index: '%d'\n")
                     ui.warn(msg % hlen)
             except ValueError:
                 # default value
                 if hconf == 'short' or hconf == '':
                     hlen = 12
                 elif hconf == 'full':
                     hlen = 40
                 elif hconf != 'none':
                     msg = _("invalid value for extendedheader.index: '%s'\n")
                     ui.warn(msg % hconf)
             finally:
                 buildopts['index'] = hlen
 
     if whitespace:
         buildopts['ignorews'] = get('ignore_all_space', 'ignorews')
         buildopts['ignorewsamount'] = get('ignore_space_change',
                                           'ignorewsamount')
         buildopts['ignoreblanklines'] = get('ignore_blank_lines',
                                             'ignoreblanklines')
         buildopts['ignorewseol'] = get('ignore_space_at_eol', 'ignorewseol')
     if formatchanging:
         buildopts['text'] = opts and opts.get('text')
         binary = None if opts is None else opts.get('binary')
         buildopts['nobinary'] = (not binary if binary is not None
                                  else get('nobinary', forceplain=False))
         buildopts['noprefix'] = get('noprefix', forceplain=False)
 
     return mdiff.diffopts(**pycompat.strkwargs(buildopts))
 
 def diff(repo, node1=None, node2=None, match=None, changes=None,
          opts=None, losedatafn=None, prefix='', relroot='', copy=None,
          hunksfilterfn=None):
     '''yields diff of changes to files between two nodes, or node and
     working directory.
 
     if node1 is None, use first dirstate parent instead.
     if node2 is None, compare node1 with working directory.
 
     losedatafn(**kwarg) is a callable run when opts.upgrade=True and
     every time some change cannot be represented with the current
     patch format. Return False to upgrade to git patch format, True to
     accept the loss or raise an exception to abort the diff. It is
     called with the name of current file being diffed as 'fn'. If set
     to None, patches will always be upgraded to git format when
     necessary.
 
     prefix is a filename prefix that is prepended to all filenames on
     display (used for subrepos).
 
     relroot, if not empty, must be normalized with a trailing /. Any match
     patterns that fall outside it will be ignored.
 
     copy, if not empty, should contain mappings {dst@y: src@x} of copy
     information.
 
     hunksfilterfn, if not None, should be a function taking a filectx and
     hunks generator that may yield filtered hunks.
     '''
     for fctx1, fctx2, hdr, hunks in diffhunks(
             repo, node1=node1, node2=node2,
             match=match, changes=changes, opts=opts,
             losedatafn=losedatafn, prefix=prefix, relroot=relroot, copy=copy,
     ):
         if hunksfilterfn is not None:
             # If the file has been removed, fctx2 is None; but this should
             # not occur here since we catch removed files early in
             # logcmdutil.getlinerangerevs() for 'hg log -L'.
             assert fctx2 is not None, \
                 'fctx2 unexpectly None in diff hunks filtering'
             hunks = hunksfilterfn(fctx2, hunks)
         text = ''.join(sum((list(hlines) for hrange, hlines in hunks), []))
         if hdr and (text or len(hdr) > 1):
             yield '\n'.join(hdr) + '\n'
         if text:
             yield text
 
 def diffhunks(repo, node1=None, node2=None, match=None, changes=None,
               opts=None, losedatafn=None, prefix='', relroot='', copy=None):
     """Yield diff of changes to files in the form of (`header`, `hunks`) tuples
     where `header` is a list of diff headers and `hunks` is an iterable of
     (`hunkrange`, `hunklines`) tuples.
 
     See diff() for the meaning of parameters.
     """
 
     if opts is None:
         opts = mdiff.defaultopts
 
     if not node1 and not node2:
         node1 = repo.dirstate.p1()
 
     def lrugetfilectx():
         cache = {}
         order = collections.deque()
         def getfilectx(f, ctx):
             fctx = ctx.filectx(f, filelog=cache.get(f))
             if f not in cache:
                 if len(cache) > 20:
                     del cache[order.popleft()]
                 cache[f] = fctx.filelog()
             else:
                 order.remove(f)
             order.append(f)
             return fctx
         return getfilectx
     getfilectx = lrugetfilectx()
 
     ctx1 = repo[node1]
     ctx2 = repo[node2]
 
     relfiltered = False
     if relroot != '' and match.always():
         # as a special case, create a new matcher with just the relroot
         pats = [relroot]
         match = scmutil.match(ctx2, pats, default='path')
         relfiltered = True
 
     if not changes:
         changes = repo.status(ctx1, ctx2, match=match)
     modified, added, removed = changes[:3]
 
     if not modified and not added and not removed:
         return []
 
     if repo.ui.debugflag:
         hexfunc = hex
     else:
         hexfunc = short
     revs = [hexfunc(node) for node in [ctx1.node(), ctx2.node()] if node]
 
     if copy is None:
         copy = {}
         if opts.git or opts.upgrade:
             copy = copies.pathcopies(ctx1, ctx2, match=match)
 
     if relroot is not None:
         if not relfiltered:
             # XXX this would ideally be done in the matcher, but that is
             # generally meant to 'or' patterns, not 'and' them. In this case we
             # need to 'and' all the patterns from the matcher with relroot.
             def filterrel(l):
                 return [f for f in l if f.startswith(relroot)]
             modified = filterrel(modified)
             added = filterrel(added)
             removed = filterrel(removed)
             relfiltered = True
         # filter out copies where either side isn't inside the relative root
         copy = dict(((dst, src) for (dst, src) in copy.iteritems()
                      if dst.startswith(relroot)
                      and src.startswith(relroot)))
 
     modifiedset = set(modified)
     addedset = set(added)
     removedset = set(removed)
     for f in modified:
         if f not in ctx1:
             # Fix up added, since merged-in additions appear as
             # modifications during merges
             modifiedset.remove(f)
             addedset.add(f)
     for f in removed:
         if f not in ctx1:
             # Merged-in additions that are then removed are reported as removed.
             # They are not in ctx1, so We don't want to show them in the diff.
             removedset.remove(f)
     modified = sorted(modifiedset)
     added = sorted(addedset)
     removed = sorted(removedset)
     for dst, src in list(copy.items()):
         if src not in ctx1:
             # Files merged in during a merge and then copied/renamed are
             # reported as copies. We want to show them in the diff as additions.
             del copy[dst]
 
     def difffn(opts, losedata):
         return trydiff(repo, revs, ctx1, ctx2, modified, added, removed,
                        copy, getfilectx, opts, losedata, prefix, relroot)
     if opts.upgrade and not opts.git:
         try:
             def losedata(fn):
                 if not losedatafn or not losedatafn(fn=fn):
                     raise GitDiffRequired
             # Buffer the whole output until we are sure it can be generated
             return list(difffn(opts.copy(git=False), losedata))
         except GitDiffRequired:
             return difffn(opts.copy(git=True), None)
     else:
         return difffn(opts, None)
 
 def difflabel(func, *args, **kw):
     '''yields 2-tuples of (output, label) based on the output of func()'''
     inlinecolor = False
     if kw.get(r'opts'):
         inlinecolor = kw[r'opts'].worddiff
     headprefixes = [('diff', 'diff.diffline'),
                     ('copy', 'diff.extended'),
                     ('rename', 'diff.extended'),
                     ('old', 'diff.extended'),
                     ('new', 'diff.extended'),
                     ('deleted', 'diff.extended'),
                     ('index', 'diff.extended'),
                     ('similarity', 'diff.extended'),
                     ('---', 'diff.file_a'),
                     ('+++', 'diff.file_b')]
     textprefixes = [('@', 'diff.hunk'),
                     ('-', 'diff.deleted'),
                     ('+', 'diff.inserted')]
     head = False
     for chunk in func(*args, **kw):
         lines = chunk.split('\n')
         matches = {}
         if inlinecolor:
             matches = _findmatches(lines)
         for i, line in enumerate(lines):
             if i != 0:
                 yield ('\n', '')
             if head:
                 if line.startswith('@'):
                     head = False
             else:
                 if line and line[0] not in ' +-@\\':
                     head = True
             stripline = line
             diffline = False
             if not head and line and line[0] in '+-':
                 # highlight tabs and trailing whitespace, but only in
                 # changed lines
                 stripline = line.rstrip()
                 diffline = True
 
             prefixes = textprefixes
             if head:
                 prefixes = headprefixes
             for prefix, label in prefixes:
                 if stripline.startswith(prefix):
                     if diffline:
                         if i in matches:
                             for t, l in _inlinediff(lines[i].rstrip(),
                                                     lines[matches[i]].rstrip(),
                                                     label):
                                 yield (t, l)
                         else:
                             for token in tabsplitter.findall(stripline):
                                 if token.startswith('\t'):
                                     yield (token, 'diff.tab')
                                 else:
                                     yield (token, label)
                     else:
                         yield (stripline, label)
                     break
             else:
                 yield (line, '')
             if line != stripline:
                 yield (line[len(stripline):], 'diff.trailingwhitespace')
 
 def _findmatches(slist):
     '''Look for insertion matches to deletion and returns a dict of
     correspondences.
     '''
     lastmatch = 0
     matches = {}
     for i, line in enumerate(slist):
         if line == '':
             continue
         if line[0] == '-':
             lastmatch = max(lastmatch, i)
             newgroup = False
             for j, newline in enumerate(slist[lastmatch + 1:]):
                 if newline == '':
                     continue
                 if newline[0] == '-' and newgroup: # too far, no match
                     break
                 if newline[0] == '+': # potential match
                     newgroup = True
                     sim = difflib.SequenceMatcher(None, line, newline).ratio()
                     if sim > 0.7:
                         lastmatch = lastmatch + 1 + j
                         matches[i] = lastmatch
                         matches[lastmatch] = i
                         break
     return matches
 
 def _inlinediff(s1, s2, operation):
     '''Perform string diff to highlight specific changes.'''
     operation_skip = '+?' if operation == 'diff.deleted' else '-?'
     if operation == 'diff.deleted':
         s2, s1 = s1, s2
 
     buff = []
     # we never want to higlight the leading +-
     if operation == 'diff.deleted' and s2.startswith('-'):
         label = operation
         token = '-'
         s2 = s2[1:]
         s1 = s1[1:]
     elif operation == 'diff.inserted' and s1.startswith('+'):
         label = operation
         token = '+'
         s2 = s2[1:]
         s1 = s1[1:]
     else:
         raise error.ProgrammingError("Case not expected, operation = %s" %
                                      operation)
 
     s = difflib.ndiff(_nonwordre.split(s2), _nonwordre.split(s1))
     for part in s:
         if part[0] in operation_skip or len(part) == 2:
             continue
         l = operation + '.highlight'
         if part[0] in ' ':
             l = operation
         if part[2:] == '\t':
             l = 'diff.tab'
         if l == label: # contiguous token with same label
             token += part[2:]
             continue
         else:
             buff.append((token, label))
             label = l
             token = part[2:]
     buff.append((token, label))
 
     return buff
 
 def diffui(*args, **kw):
     '''like diff(), but yields 2-tuples of (output, label) for ui.write()'''
     return difflabel(diff, *args, **kw)
 
 def _filepairs(modified, added, removed, copy, opts):
     '''generates tuples (f1, f2, copyop), where f1 is the name of the file
     before and f2 is the the name after. For added files, f1 will be None,
     and for removed files, f2 will be None. copyop may be set to None, 'copy'
     or 'rename' (the latter two only if opts.git is set).'''
     gone = set()
 
     copyto = dict([(v, k) for k, v in copy.items()])
 
     addedset, removedset = set(added), set(removed)
 
     for f in sorted(modified + added + removed):
         copyop = None
         f1, f2 = f, f
         if f in addedset:
             f1 = None
             if f in copy:
                 if opts.git:
                     f1 = copy[f]
                     if f1 in removedset and f1 not in gone:
                         copyop = 'rename'
                         gone.add(f1)
                     else:
                         copyop = 'copy'
         elif f in removedset:
             f2 = None
             if opts.git:
                 # have we already reported a copy above?
                 if (f in copyto and copyto[f] in addedset
                     and copy[copyto[f]] == f):
                     continue
         yield f1, f2, copyop
 
 def trydiff(repo, revs, ctx1, ctx2, modified, added, removed,
             copy, getfilectx, opts, losedatafn, prefix, relroot):
     '''given input data, generate a diff and yield it in blocks
 
     If generating a diff would lose data like flags or binary data and
     losedatafn is not None, it will be called.
 
     relroot is removed and prefix is added to every path in the diff output.
 
     If relroot is not empty, this function expects every path in modified,
     added, removed and copy to start with it.'''
 
     def gitindex(text):
         if not text:
             text = ""
         l = len(text)
         s = hashlib.sha1('blob %d\0' % l)
         s.update(text)
         return hex(s.digest())
 
     if opts.noprefix:
         aprefix = bprefix = ''
     else:
         aprefix = 'a/'
         bprefix = 'b/'
 
     def diffline(f, revs):
         revinfo = ' '.join(["-r %s" % rev for rev in revs])
         return 'diff %s %s' % (revinfo, f)
 
     def isempty(fctx):
         return fctx is None or fctx.size() == 0
 
     date1 = dateutil.datestr(ctx1.date())
     date2 = dateutil.datestr(ctx2.date())
 
     gitmode = {'l': '120000', 'x': '100755', '': '100644'}
 
     if relroot != '' and (repo.ui.configbool('devel', 'all-warnings')
                           or repo.ui.configbool('devel', 'check-relroot')):
         for f in modified + added + removed + list(copy) + list(copy.values()):
             if f is not None and not f.startswith(relroot):
                 raise AssertionError(
                     "file %s doesn't start with relroot %s" % (f, relroot))
 
     for f1, f2, copyop in _filepairs(modified, added, removed, copy, opts):
         content1 = None
         content2 = None
         fctx1 = None
         fctx2 = None
         flag1 = None
         flag2 = None
         if f1:
             fctx1 = getfilectx(f1, ctx1)
             if opts.git or losedatafn:
                 flag1 = ctx1.flags(f1)
         if f2:
             fctx2 = getfilectx(f2, ctx2)
             if opts.git or losedatafn:
                 flag2 = ctx2.flags(f2)
         # if binary is True, output "summary" or "base85", but not "text diff"
         if opts.text:
             binary = False
         else:
             binary = any(f.isbinary() for f in [fctx1, fctx2] if f is not None)
 
         if losedatafn and not opts.git:
             if (binary or
                 # copy/rename
                 f2 in copy or
                 # empty file creation
                 (not f1 and isempty(fctx2)) or
                 # empty file deletion
                 (isempty(fctx1) and not f2) or
                 # create with flags
                 (not f1 and flag2) or
                 # change flags
                 (f1 and f2 and flag1 != flag2)):
                 losedatafn(f2 or f1)
 
         path1 = f1 or f2
         path2 = f2 or f1
         path1 = posixpath.join(prefix, path1[len(relroot):])
         path2 = posixpath.join(prefix, path2[len(relroot):])
         header = []
         if opts.git:
             header.append('diff --git %s%s %s%s' %
                           (aprefix, path1, bprefix, path2))
             if not f1: # added
                 header.append('new file mode %s' % gitmode[flag2])
             elif not f2: # removed
                 header.append('deleted file mode %s' % gitmode[flag1])
             else:  # modified/copied/renamed
                 mode1, mode2 = gitmode[flag1], gitmode[flag2]
                 if mode1 != mode2:
                     header.append('old mode %s' % mode1)
                     header.append('new mode %s' % mode2)
                 if copyop is not None:
                     if opts.showsimilarity:
                         sim = similar.score(ctx1[path1], ctx2[path2]) * 100
                         header.append('similarity index %d%%' % sim)
                     header.append('%s from %s' % (copyop, path1))
                     header.append('%s to %s' % (copyop, path2))
         elif revs and not repo.ui.quiet:
             header.append(diffline(path1, revs))
 
         #  fctx.is  | diffopts                | what to   | is fctx.data()
         #  binary() | text nobinary git index | output?   | outputted?
         # ------------------------------------|----------------------------
         #  yes      | no   no       no  *     | summary   | no
         #  yes      | no   no       yes *     | base85    | yes
         #  yes      | no   yes      no  *     | summary   | no
         #  yes      | no   yes      yes 0     | summary   | no
         #  yes      | no   yes      yes >0    | summary   | semi [1]
         #  yes      | yes  *        *   *     | text diff | yes
         #  no       | *    *        *   *     | text diff | yes
         # [1]: hash(fctx.data()) is outputted. so fctx.data() cannot be faked
         if binary and (not opts.git or (opts.git and opts.nobinary and not
                                         opts.index)):
             # fast path: no binary content will be displayed, content1 and
             # content2 are only used for equivalent test. cmp() could have a
             # fast path.
             if fctx1 is not None:
                 content1 = b'\0'
             if fctx2 is not None:
                 if fctx1 is not None and not fctx1.cmp(fctx2):
                     content2 = b'\0' # not different
                 else:
                     content2 = b'\0\0'
         else:
             # normal path: load contents
             if fctx1 is not None:
                 content1 = fctx1.data()
             if fctx2 is not None:
                 content2 = fctx2.data()
 
         if binary and opts.git and not opts.nobinary:
             text = mdiff.b85diff(content1, content2)
             if text:
                 header.append('index %s..%s' %
                               (gitindex(content1), gitindex(content2)))
             hunks = (None, [text]),
         else:
             if opts.git and opts.index > 0:
                 flag = flag1
                 if flag is None:
                     flag = flag2
                 header.append('index %s..%s %s' %
                               (gitindex(content1)[0:opts.index],
                                gitindex(content2)[0:opts.index],
                                gitmode[flag]))
 
             uheaders, hunks = mdiff.unidiff(content1, date1,
                                             content2, date2,
                                             path1, path2,
                                             binary=binary, opts=opts)
             header.extend(uheaders)
         yield fctx1, fctx2, header, hunks
 
 def diffstatsum(stats):
     maxfile, maxtotal, addtotal, removetotal, binary = 0, 0, 0, 0, False
     for f, a, r, b in stats:
         maxfile = max(maxfile, encoding.colwidth(f))
         maxtotal = max(maxtotal, a + r)
         addtotal += a
         removetotal += r
         binary = binary or b
 
     return maxfile, maxtotal, addtotal, removetotal, binary
 
 def diffstatdata(lines):
     diffre = re.compile('^diff .*-r [a-z0-9]+\s(.*)$')
 
     results = []
     filename, adds, removes, isbinary = None, 0, 0, False
 
     def addresult():
         if filename:
             results.append((filename, adds, removes, isbinary))
 
     # inheader is used to track if a line is in the
     # header portion of the diff.  This helps properly account
     # for lines that start with '--' or '++'
     inheader = False
 
     for line in lines:
         if line.startswith('diff'):
             addresult()
             # starting a new file diff
             # set numbers to 0 and reset inheader
             inheader = True
             adds, removes, isbinary = 0, 0, False
             if line.startswith('diff --git a/'):
                 filename = gitre.search(line).group(2)
             elif line.startswith('diff -r'):
                 # format: "diff -r ... -r ... filename"
                 filename = diffre.search(line).group(1)
         elif line.startswith('@@'):
             inheader = False
         elif line.startswith('+') and not inheader:
             adds += 1
         elif line.startswith('-') and not inheader:
             removes += 1
         elif (line.startswith('GIT binary patch') or
               line.startswith('Binary file')):
             isbinary = True
     addresult()
     return results
 
 def diffstat(lines, width=80):
     output = []
     stats = diffstatdata(lines)
     maxname, maxtotal, totaladds, totalremoves, hasbinary = diffstatsum(stats)
 
     countwidth = len(str(maxtotal))
     if hasbinary and countwidth < 3:
         countwidth = 3
     graphwidth = width - countwidth - maxname - 6
     if graphwidth < 10:
         graphwidth = 10
 
     def scale(i):
         if maxtotal <= graphwidth:
             return i
         # If diffstat runs out of room it doesn't print anything,
         # which isn't very useful, so always print at least one + or -
         # if there were at least some changes.
         return max(i * graphwidth // maxtotal, int(bool(i)))
 
     for filename, adds, removes, isbinary in stats:
         if isbinary:
             count = 'Bin'
         else:
             count = '%d' % (adds + removes)
         pluses = '+' * scale(adds)
         minuses = '-' * scale(removes)
         output.append(' %s%s |  %*s %s%s\n' %
                       (filename, ' ' * (maxname - encoding.colwidth(filename)),
                        countwidth, count, pluses, minuses))
 
     if stats:
         output.append(_(' %d files changed, %d insertions(+), '
                         '%d deletions(-)\n')
                       % (len(stats), totaladds, totalremoves))
 
     return ''.join(output)
 
 def diffstatui(*args, **kw):
     '''like diffstat(), but yields 2-tuples of (output, label) for
     ui.write()
     '''
 
     for line in diffstat(*args, **kw).splitlines():
         if line and line[-1] in '+-':
             name, graph = line.rsplit(' ', 1)
             yield (name + ' ', '')
             m = re.search(br'\++', graph)
             if m:
                 yield (m.group(0), 'diffstat.inserted')
             m = re.search(br'-+', graph)
             if m:
                 yield (m.group(0), 'diffstat.deleted')
         else:
             yield (line, '')
         yield ('\n', '')