merge default into stable for 4.0 code freeze

CONTRIBUTING

@@ -0,0 +1,13 b''
+Our full contribution guidelines are in our wiki, please see:
+
+https://www.mercurial-scm.org/wiki/ContributingChanges
+
+If you just want a checklist to follow, you can go straight to
+
+https://www.mercurial-scm.org/wiki/ContributingChanges#Submission_checklist
+
+If you can't run the entire testsuite for some reason (it can be
+difficult on Windows), please at least run `contrib/check-code.py` on
+any files you've modified and run `python contrib/check-commit` on any
+commits you've made (for example, `python contrib/check-commit
+273ce12ad8f1` will report some style violations on a very old commit).

mercurial/help/internals/wireprotocol.txt

@@ -0,0 +1,773 b''
+The Mercurial wire protocol is a request-response based protocol
+with multiple wire representations.
+
+Each request is modeled as a command name, a dictionary of arguments, and
+optional raw input. Command arguments and their types are intrinsic
+properties of commands. So is the response type of the command. This means
+clients can't always send arbitrary arguments to servers and servers can't
+return multiple response types.
+
+The protocol is synchronous and does not support multiplexing (concurrent
+commands).
+
+Transport Protocols
+===================
+
+HTTP Transport
+--------------
+
+Commands are issued as HTTP/1.0 or HTTP/1.1 requests. Commands are
+sent to the base URL of the repository with the command name sent in
+the ``cmd`` query string parameter. e.g.
+``https://example.com/repo?cmd=capabilities``. The HTTP method is ``GET``
+or ``POST`` depending on the command and whether there is a request
+body.
+
+Command arguments can be sent multiple ways.
+
+The simplest is part of the URL query string using ``x-www-form-urlencoded``
+encoding (see Python's ``urllib.urlencode()``. However, many servers impose
+length limitations on the URL. So this mechanism is typically only used if
+the server doesn't support other mechanisms.
+
+If the server supports the ``httpheader`` capability, command arguments can
+be sent in HTTP request headers named ``X-HgArg-<N>`` where ``<N>`` is an
+integer starting at 1. A ``x-www-form-urlencoded`` representation of the
+arguments is obtained. This full string is then split into chunks and sent
+in numbered ``X-HgArg-<N>`` headers. The maximum length of each HTTP header
+is defined by the server in the ``httpheader`` capability value, which defaults
+to ``1024``. The server reassembles the encoded arguments string by
+concatenating the ``X-HgArg-<N>`` headers then URL decodes them into a
+dictionary.
+
+The list of ``X-HgArg-<N>`` headers should be added to the ``Vary`` request
+header to instruct caches to take these headers into consideration when caching
+requests.
+
+If the server supports the ``httppostargs`` capability, the client
+may send command arguments in the HTTP request body as part of an
+HTTP POST request. The command arguments will be URL encoded just like
+they would for sending them via HTTP headers. However, no splitting is
+performed: the raw arguments are included in the HTTP request body.
+
+The client sends a ``X-HgArgs-Post`` header with the string length of the
+encoded arguments data. Additional data may be included in the HTTP
+request body immediately following the argument data. The offset of the
+non-argument data is defined by the ``X-HgArgs-Post`` header. The
+``X-HgArgs-Post`` header is not required if there is no argument data.
+
+Additional command data can be sent as part of the HTTP request body. The
+default ``Content-Type`` when sending data is ``application/mercurial-0.1``.
+A ``Content-Length`` header is currently always sent.
+
+Example HTTP requests::
+
+    GET /repo?cmd=capabilities
+    X-HgArg-1: foo=bar&baz=hello%20world
+
+The ``Content-Type`` HTTP response header identifies the response as coming
+from Mercurial and can also be used to signal an error has occurred.
+
+The ``application/mercurial-0.1`` media type indicates a generic Mercurial
+response. It matches the media type sent by the client.
+
+The ``application/hg-error`` media type indicates a generic error occurred.
+The content of the HTTP response body typically holds text describing the
+error.
+
+The ``application/hg-changegroup`` media type indicates a changegroup response
+type.
+
+Clients also accept the ``text/plain`` media type. All other media
+types should cause the client to error.
+
+Clients should issue a ``User-Agent`` request header that identifies the client.
+The server should not use the ``User-Agent`` for feature detection.
+
+A command returning a ``string`` response issues the
+``application/mercurial-0.1`` media type and the HTTP response body contains
+the raw string value. A ``Content-Length`` header is typically issued.
+
+A command returning a ``stream`` response issues the
+``application/mercurial-0.1`` media type and the HTTP response is typically
+using *chunked transfer* (``Transfer-Encoding: chunked``).
+
+SSH Transport
+=============
+
+The SSH transport is a custom text-based protocol suitable for use over any
+bi-directional stream transport. It is most commonly used with SSH.
+
+A SSH transport server can be started with ``hg serve --stdio``. The stdin,
+stderr, and stdout file descriptors of the started process are used to exchange
+data. When Mercurial connects to a remote server over SSH, it actually starts
+a ``hg serve --stdio`` process on the remote server.
+
+Commands are issued by sending the command name followed by a trailing newline
+``\n`` to the server. e.g. ``capabilities\n``.
+
+Command arguments are sent in the following format::
+
+    <argument> <length>\n<value>
+
+That is, the argument string name followed by a space followed by the
+integer length of the value (expressed as a string) followed by a newline
+(``\n``) followed by the raw argument value.
+
+Dictionary arguments are encoded differently::
+
+    <argument> <# elements>\n
+    <key1> <length1>\n<value1>
+    <key2> <length2>\n<value2>
+    ...
+
+Non-argument data is sent immediately after the final argument value. It is
+encoded in chunks::
+
+    <length>\n<data>
+
+Each command declares a list of supported arguments and their types. If a
+client sends an unknown argument to the server, the server should abort
+immediately. The special argument ``*`` in a command's definition indicates
+that all argument names are allowed.
+
+The definition of supported arguments and types is initially made when a
+new command is implemented. The client and server must initially independently
+agree on the arguments and their types. This initial set of arguments can be
+supplemented through the presence of *capabilities* advertised by the server.
+
+Each command has a defined expected response type.
+
+A ``string`` response type is a length framed value. The response consists of
+the string encoded integer length of a value followed by a newline (``\n``)
+followed by the value. Empty values are allowed (and are represented as
+``0\n``).
+
+A ``stream`` response type consists of raw bytes of data. There is no framing.
+
+A generic error response type is also supported. It consists of a an error
+message written to ``stderr`` followed by ``\n-\n``. In addition, ``\n`` is
+written to ``stdout``.
+
+If the server receives an unknown command, it will send an empty ``string``
+response.
+
+The server terminates if it receives an empty command (a ``\n`` character).
+
+Capabilities
+============
+
+Servers advertise supported wire protocol features. This allows clients to
+probe for server features before blindly calling a command or passing a
+specific argument.
+
+The server's features are exposed via a *capabilities* string. This is a
+space-delimited string of tokens/features. Some features are single words
+like ``lookup`` or ``batch``. Others are complicated key-value pairs
+advertising sub-features. e.g. ``httpheader=2048``. When complex, non-word
+values are used, each feature name can define its own encoding of sub-values.
+Comma-delimited and ``x-www-form-urlencoded`` values are common.
+
+The following document capabilities defined by the canonical Mercurial server
+implementation.
+
+batch
+-----
+
+Whether the server supports the ``batch`` command.
+
+This capability/command was introduced in Mercurial 1.9 (released July 2011).
+
+branchmap
+---------
+
+Whether the server supports the ``branchmap`` command.
+
+This capability/command was introduced in Mercurial 1.3 (released July 2009).
+
+bundle2-exp
+-----------
+
+Precursor to ``bundle2`` capability that was used before bundle2 was a
+stable feature.
+
+This capability was introduced in Mercurial 3.0 behind an experimental
+flag. This capability should not be observed in the wild.
+
+bundle2
+-------
+
+Indicates whether the server supports the ``bundle2`` data exchange format.
+
+The value of the capability is a URL quoted, newline (``\n``) delimited
+list of keys or key-value pairs.
+
+A key is simply a URL encoded string.
+
+A key-value pair is a URL encoded key separated from a URL encoded value by
+an ``=``. If the value is a list, elements are delimited by a ``,`` after
+URL encoding.
+
+For example, say we have the values::
+
+  {'HG20': [], 'changegroup': ['01', '02'], 'digests': ['sha1', 'sha512']}
+
+We would first construct a string::
+
+  HG20\nchangegroup=01,02\ndigests=sha1,sha512
+
+We would then URL quote this string::
+
+  HG20%0Achangegroup%3D01%2C02%0Adigests%3Dsha1%2Csha512
+
+This capability was introduced in Mercurial 3.4 (released May 2015).
+
+changegroupsubset
+-----------------
+
+Whether the server supports the ``changegroupsubset`` command.
+
+This capability was introduced in Mercurial 0.9.2 (released December
+2006).
+
+This capability was introduced at the same time as the ``lookup``
+capability/command.
+
+getbundle
+---------
+
+Whether the server supports the ``getbundle`` command.
+
+This capability was introduced in Mercurial 1.9 (released July 2011).
+
+httpheader
+----------
+
+Whether the server supports receiving command arguments via HTTP request
+headers.
+
+The value of the capability is an integer describing the max header
+length that clients should send. Clients should ignore any content after a
+comma in the value, as this is reserved for future use.
+
+This capability was introduced in Mercurial 1.9 (released July 2011).
+
+httppostargs
+------------
+
+**Experimental**
+
+Indicates that the server supports and prefers clients send command arguments
+via a HTTP POST request as part of the request body.
+
+This capability was introduced in Mercurial 3.8 (released May 2016).
+
+known
+-----
+
+Whether the server supports the ``known`` command.
+
+This capability/command was introduced in Mercurial 1.9 (released July 2011).
+
+lookup
+------
+
+Whether the server supports the ``lookup`` command.
+
+This capability was introduced in Mercurial 0.9.2 (released December
+2006).
+
+This capability was introduced at the same time as the ``changegroupsubset``
+capability/command.
+
+pushkey
+-------
+
+Whether the server supports the ``pushkey`` and ``listkeys`` commands.
+
+This capability was introduced in Mercurial 1.6 (released July 2010).
+
+standardbundle
+--------------
+
+**Unsupported**
+
+This capability was introduced during the Mercurial 0.9.2 development cycle in
+2006. It was never present in a release, as it was replaced by the ``unbundle``
+capability. This capability should not be encountered in the wild.
+
+stream-preferred
+----------------
+
+If present the server prefers that clients clone using the streaming clone
+protocol (``hg clone --uncompressed``) rather than the standard
+changegroup/bundle based protocol.
+
+This capability was introduced in Mercurial 2.2 (released May 2012).
+
+streamreqs
+----------
+
+Indicates whether the server supports *streaming clones* and the *requirements*
+that clients must support to receive it.
+
+If present, the server supports the ``stream_out`` command, which transmits
+raw revlogs from the repository instead of changegroups. This provides a faster
+cloning mechanism at the expense of more bandwidth used.
+
+The value of this capability is a comma-delimited list of repo format
+*requirements*. These are requirements that impact the reading of data in
+the ``.hg/store`` directory. An example value is
+``streamreqs=generaldelta,revlogv1`` indicating the server repo requires
+the ``revlogv1`` and ``generaldelta`` requirements.
+
+If the only format requirement is ``revlogv1``, the server may expose the
+``stream`` capability instead of the ``streamreqs`` capability.
+
+This capability was introduced in Mercurial 1.7 (released November 2010).
+
+stream
+------
+
+Whether the server supports *streaming clones* from ``revlogv1`` repos.
+
+If present, the server supports the ``stream_out`` command, which transmits
+raw revlogs from the repository instead of changegroups. This provides a faster
+cloning mechanism at the expense of more bandwidth used.
+
+This capability was introduced in Mercurial 0.9.1 (released July 2006).
+
+When initially introduced, the value of the capability was the numeric
+revlog revision. e.g. ``stream=1``. This indicates the changegroup is using
+``revlogv1``. This simple integer value wasn't powerful enough, so the
+``streamreqs`` capability was invented to handle cases where the repo
+requirements have more than just ``revlogv1``. Newer servers omit the
+``=1`` since it was the only value supported and the value of ``1`` can
+be implied by clients.
+
+unbundlehash
+------------
+
+Whether the ``unbundle`` commands supports receiving a hash of all the
+heads instead of a list.
+
+For more, see the documentation for the ``unbundle`` command.
+
+This capability was introduced in Mercurial 1.9 (released July 2011).
+
+unbundle
+--------
+
+Whether the server supports pushing via the ``unbundle`` command.
+
+This capability/command has been present since Mercurial 0.9.1 (released
+July 2006).
+
+Mercurial 0.9.2 (released December 2006) added values to the capability
+indicating which bundle types the server supports receiving. This value is a
+comma-delimited list. e.g. ``HG10GZ,HG10BZ,HG10UN``. The order of values
+reflects the priority/preference of that type, where the first value is the
+most preferred type.
+
+Handshake Protocol
+==================
+
+While not explicitly required, it is common for clients to perform a
+*handshake* when connecting to a server. The handshake accomplishes 2 things:
+
+* Obtaining capabilities and other server features
+* Flushing extra server output (e.g. SSH servers may print extra text
+  when connecting that may confuse the wire protocol)
+
+This isn't a traditional *handshake* as far as network protocols go because
+there is no persistent state as a result of the handshake: the handshake is
+simply the issuing of commands and commands are stateless.
+
+The canonical clients perform a capabilities lookup at connection establishment
+time. This is because clients must assume a server only supports the features
+of the original Mercurial server implementation until proven otherwise (from
+advertised capabilities). Nearly every server running today supports features
+that weren't present in the original Mercurial server implementation. Rather
+than wait for a client to perform functionality that needs to consult
+capabilities, it issues the lookup at connection start to avoid any delay later.
+
+For HTTP servers, the client sends a ``capabilities`` command request as
+soon as the connection is established. The server responds with a capabilities
+string, which the client parses.
+
+For SSH servers, the client sends the ``hello`` command (no arguments)
+and a ``between`` command with the ``pairs`` argument having the value
+``0000000000000000000000000000000000000000-0000000000000000000000000000000000000000``.
+
+The ``between`` command has been supported since the original Mercurial
+server. Requesting the empty range will return a ``\n`` string response,
+which will be encoded as ``1\n\n`` (value length of ``1`` followed by a newline
+followed by the value, which happens to  be a newline).
+
+The ``hello`` command was later introduced. Servers supporting it will issue
+a response to that command before sending the ``1\n\n`` response to the
+``between`` command. Servers not supporting ``hello`` will send an empty
+response (``0\n``).
+
+In addition to the expected output from the ``hello`` and ``between`` commands,
+servers may also send other output, such as *message of the day (MOTD)*
+announcements. Clients assume servers will send this output before the
+Mercurial server replies to the client-issued commands. So any server output
+not conforming to the expected command responses is assumed to be not related
+to Mercurial and can be ignored.
+
+Commands
+========
+
+This section contains a list of all wire protocol commands implemented by
+the canonical Mercurial server.
+
+batch
+-----
+
+Issue multiple commands while sending a single command request. The purpose
+of this command is to allow a client to issue multiple commands while avoiding
+multiple round trips to the server therefore enabling commands to complete
+quicker.
+
+The command accepts a ``cmds`` argument that contains a list of commands to
+execute.
+
+The value of ``cmds`` is a ``;`` delimited list of strings. Each string has the
+form ``<command> <arguments>``. That is, the command name followed by a space
+followed by an argument string.
+
+The argument string is a ``,`` delimited list of ``<key>=<value>`` values
+corresponding to command arguments. Both the argument name and value are
+escaped using a special substitution map::
+
+   : -> :c
+   , -> :o
+   ; -> :s
+   = -> :e
+
+The response type for this command is ``string``. The value contains a
+``;`` delimited list of responses for each requested command. Each value
+in this list is escaped using the same substitution map used for arguments.
+
+If an error occurs, the generic error response may be sent.
+
+between
+-------
+
+(Legacy command used for discovery in old clients)
+
+Obtain nodes between pairs of nodes.
+
+The ``pairs`` arguments contains a space-delimited list of ``-`` delimited
+hex node pairs. e.g.::
+
+   a072279d3f7fd3a4aa7ffa1a5af8efc573e1c896-6dc58916e7c070f678682bfe404d2e2d68291a18
+
+Return type is a ``string``. Value consists of lines corresponding to each
+requested range. Each line contains a space-delimited list of hex nodes.
+A newline ``\n`` terminates each line, including the last one.
+
+branchmap
+---------
+
+Obtain heads in named branches.
+
+Accepts no arguments. Return type is a ``string``.
+
+Return value contains lines with URL encoded branch names followed by a space
+followed by a space-delimited list of hex nodes of heads on that branch.
+e.g.::
+
+    default a072279d3f7fd3a4aa7ffa1a5af8efc573e1c896 6dc58916e7c070f678682bfe404d2e2d68291a18
+    stable baae3bf31522f41dd5e6d7377d0edd8d1cf3fccc
+
+There is no trailing newline.
+
+branches
+--------
+
+Obtain ancestor changesets of specific nodes back to a branch point.
+
+Despite the name, this command has nothing to do with Mercurial named branches.
+Instead, it is related to DAG branches.
+
+The command accepts a ``nodes`` argument, which is a string of space-delimited
+hex nodes.
+
+For each node requested, the server will find the first ancestor node that is
+a DAG root or is a merge.
+
+Return type is a ``string``. Return value contains lines with result data for
+each requested node. Each line contains space-delimited nodes followed by a
+newline (``\n``). The 4 nodes reported on each line correspond to the requested
+node, the ancestor node found, and its 2 parent nodes (which may be the null
+node).
+
+capabilities
+------------
+
+Obtain the capabilities string for the repo.
+
+Unlike the ``hello`` command, the capabilities string is not prefixed.
+There is no trailing newline.
+
+This command does not accept any arguments. Return type is a ``string``.
+
+changegroup
+-----------
+
+(Legacy command: use ``getbundle`` instead)
+
+Obtain a changegroup version 1 with data for changesets that are
+descendants of client-specified changesets.
+
+The ``roots`` arguments contains a list of space-delimited hex nodes.
+
+The server responds with a changegroup version 1 containing all
+changesets between the requested root/base nodes and the repo's head nodes
+at the time of the request.
+
+The return type is a ``stream``.
+
+changegroupsubset
+-----------------
+
+(Legacy command: use ``getbundle`` instead)
+
+Obtain a changegroup version 1 with data for changesetsets between
+client specified base and head nodes.
+
+The ``bases`` argument contains a list of space-delimited hex nodes.
+The ``heads`` argument contains a list of space-delimited hex nodes.
+
+The server responds with a changegroup version 1 containing all
+changesets between the requested base and head nodes at the time of the
+request.
+
+The return type is a ``stream``.
+
+clonebundles
+------------
+
+Obtains a manifest of bundle URLs available to seed clones.
+
+Each returned line contains a URL followed by metadata. See the
+documentation in the ``clonebundles`` extension for more.
+
+The return type is a ``string``.
+
+getbundle
+---------
+
+Obtain a bundle containing repository data.
+
+This command accepts the following arguments:
+
+heads
+   List of space-delimited hex nodes of heads to retrieve.
+common
+   List of space-delimited hex nodes that the client has in common with the
+   server.
+obsmarkers
+   Boolean indicating whether to include obsolescence markers as part
+   of the response. Only works with bundle2.
+bundlecaps
+   Comma-delimited set of strings defining client bundle capabilities.
+listkeys
+   Comma-delimited list of strings of ``pushkey`` namespaces. For each
+   namespace listed, a bundle2 part will be included with the content of
+   that namespace.
+cg
+   Boolean indicating whether changegroup data is requested.
+cbattempted
+   Boolean indicating whether the client attempted to use the *clone bundles*
+   feature before performing this request.
+
+The return type on success is a ``stream`` where the value is bundle.
+On the HTTP transport, the response is zlib compressed.
+
+If an error occurs, a generic error response can be sent.
+
+Unless the client sends a false value for the ``cg`` argument, the returned
+bundle contains a changegroup with the nodes between the specified ``common``
+and ``heads`` nodes. Depending on the command arguments, the type and content
+of the returned bundle can vary significantly.
+
+The default behavior is for the server to send a raw changegroup version
+``01`` response.
+
+If the ``bundlecaps`` provided by the client contain a value beginning
+with ``HG2``, a bundle2 will be returned. The bundle2 data may contain
+additional repository data, such as ``pushkey`` namespace values.
+
+heads
+-----
+
+Returns a list of space-delimited hex nodes of repository heads followed
+by a newline. e.g.
+``a9eeb3adc7ddb5006c088e9eda61791c777cbf7c 31f91a3da534dc849f0d6bfc00a395a97cf218a1\n``
+
+This command does not accept any arguments. The return type is a ``string``.
+
+hello
+-----
+
+Returns lines describing interesting things about the server in an RFC-822
+like format.
+
+Currently, the only line defines the server capabilities. It has the form::
+
+    capabilities: <value>
+
+See above for more about the capabilities string.
+
+SSH clients typically issue this command as soon as a connection is
+established.
+
+This command does not accept any arguments. The return type is a ``string``.
+
+listkeys
+--------
+
+List values in a specified ``pushkey`` namespace.
+
+The ``namespace`` argument defines the pushkey namespace to operate on.
+
+The return type is a ``string``. The value is an encoded dictionary of keys.
+
+Key-value pairs are delimited by newlines (``\n``). Within each line, keys and
+values are separated by a tab (``\t``). Keys and values are both strings.
+
+lookup
+------
+
+Try to resolve a value to a known repository revision.
+
+The ``key`` argument is converted from bytes to an
+``encoding.localstr`` instance then passed into
+``localrepository.__getitem__`` in an attempt to resolve it.
+
+The return type is a ``string``.
+
+Upon successful resolution, returns ``1 <hex node>\n``. On failure,
+returns ``0 <error string>\n``. e.g.::
+
+   1 273ce12ad8f155317b2c078ec75a4eba507f1fba\n
+
+   0 unknown revision 'foo'\n
+
+known
+-----
+
+Determine whether multiple nodes are known.
+
+The ``nodes`` argument is a list of space-delimited hex nodes to check
+for existence.
+
+The return type is ``string``.
+
+Returns a string consisting of ``0``s and ``1``s indicating whether nodes
+are known. If the Nth node specified in the ``nodes`` argument is known,
+a ``1`` will be returned at byte offset N. If the node isn't known, ``0``
+will be present at byte offset N.
+
+There is no trailing newline.
+
+pushkey
+-------
+
+Set a value using the ``pushkey`` protocol.
+
+Accepts arguments ``namespace``, ``key``, ``old``, and ``new``, which
+correspond to the pushkey namespace to operate on, the key within that
+namespace to change, the old value (which may be empty), and the new value.
+All arguments are string types.
+
+The return type is a ``string``. The value depends on the transport protocol.
+
+The SSH transport sends a string encoded integer followed by a newline
+(``\n``) which indicates operation result. The server may send additional
+output on the ``stderr`` stream that should be displayed to the user.
+
+The HTTP transport sends a string encoded integer followed by a newline
+followed by additional server output that should be displayed to the user.
+This may include output from hooks, etc.
+
+The integer result varies by namespace. ``0`` means an error has occurred
+and there should be additional output to display to the user.
+
+stream_out
+----------
+
+Obtain *streaming clone* data.
+
+The return type is either a ``string`` or a ``stream``, depending on
+whether the request was fulfilled properly.
+
+A return value of ``1\n`` indicates the server is not configured to serve
+this data. If this is seen by the client, they may not have verified the
+``stream`` capability is set before making the request.
+
+A return value of ``2\n`` indicates the server was unable to lock the
+repository to generate data.
+
+All other responses are a ``stream`` of bytes. The first line of this data
+contains 2 space-delimited integers corresponding to the path count and
+payload size, respectively::
+
+    <path count> <payload size>\n
+
+The ``<payload size>`` is the total size of path data: it does not include
+the size of the per-path header lines.
+
+Following that header are ``<path count>`` entries. Each entry consists of a
+line with metadata followed by raw revlog data. The line consists of::
+
+    <store path>\0<size>\n
+
+The ``<store path>`` is the encoded store path of the data that follows.
+``<size>`` is the amount of data for this store path/revlog that follows the
+newline.
+
+There is no trailer to indicate end of data. Instead, the client should stop
+reading after ``<path count>`` entries are consumed.
+
+unbundle
+--------
+
+Send a bundle containing data (usually changegroup data) to the server.
+
+Accepts the argument ``heads``, which is a space-delimited list of hex nodes
+corresponding to server repository heads observed by the client. This is used
+to detect race conditions and abort push operations before a server performs
+too much work or a client transfers too much data.
+
+The request payload consists of a bundle to be applied to the repository,
+similarly to as if :hg:`unbundle` were called.
+
+In most scenarios, a special ``push response`` type is returned. This type
+contains an integer describing the change in heads as a result of the
+operation. A value of ``0`` indicates nothing changed. ``1`` means the number
+of heads remained the same. Values ``2`` and larger indicate the number of
+added heads minus 1. e.g. ``3`` means 2 heads were added. Negative values
+indicate the number of fewer heads, also off by 1. e.g. ``-2`` means there
+is 1 fewer head.
+
+The encoding of the ``push response`` type varies by transport.
+
+For the SSH transport, this type is composed of 2 ``string`` responses: an
+empty response (``0\n``) followed by the integer result value. e.g.
+``1\n2``. So the full response might be ``0\n1\n2``.
+
+For the HTTP transport, the response is a ``string`` type composed of an
+integer result value followed by a newline (``\n``) followed by string
+content holding server output that should be displayed on the client (output
+hooks, etc).
+
+In some cases, the server may respond with a ``bundle2`` bundle. In this
+case, the response type is ``stream``. For the HTTP transport, the response
+is zlib compressed.
+
+The server may also respond with a generic error type, which contains a string
+indicating the failure.

mercurial/mpatch.h

@@ -0,0 +1,26 b''
+#ifndef _HG_MPATCH_H_
+#define _HG_MPATCH_H_
+
+#define MPATCH_ERR_NO_MEM -3
+#define MPATCH_ERR_CANNOT_BE_DECODED -2
+#define MPATCH_ERR_INVALID_PATCH -1
+
+struct mpatch_frag {
+	int start, end, len;
+	const char *data;
+};
+
+struct mpatch_flist {
+	struct mpatch_frag *base, *head, *tail;
+};
+
+int mpatch_decode(const char *bin, ssize_t len, struct mpatch_flist** res);
+ssize_t mpatch_calcsize(ssize_t len, struct mpatch_flist *l);
+void mpatch_lfree(struct mpatch_flist *a);
+int mpatch_apply(char *buf, const char *orig, ssize_t len,
+	struct mpatch_flist *l);
+struct mpatch_flist *mpatch_fold(void *bins,
+	struct mpatch_flist* (*get_next_item)(void*, ssize_t),
+	ssize_t start, ssize_t end);
+
+#endif

mercurial/profiling.py

@@ -0,0 +1,164 b''
+# profiling.py - profiling functions
+#
+# Copyright 2016 Gregory Szorc <[e-mail]>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+from __future__ import absolute_import, print_function
+
+import contextlib
+import os
+import sys
+import time
+
+from .i18n import _
+from . import (
+    error,
+    util,
+)
+
[e-mail]
+def lsprofile(ui, fp):
+    format = ui.config('profiling', 'format', default='text')
+    field = ui.config('profiling', 'sort', default='inlinetime')
+    limit = ui.configint('profiling', 'limit', default=30)
+    climit = ui.configint('profiling', 'nested', default=0)
+
+    if format not in ['text', 'kcachegrind']:
+        ui.warn(_("unrecognized profiling format '%s'"
+                    " - Ignored\n") % format)
+        format = 'text'
+
+    try:
+        from . import lsprof
+    except ImportError:
+        raise error.Abort(_(
+            'lsprof not available - install from '
+            'http://codespeak.net/svn/user/arigo/hack/misc/lsprof/'))
+    p = lsprof.Profiler()
+    p.enable(subcalls=True)
+    try:
+        yield
+    finally:
+        p.disable()
+
+        if format == 'kcachegrind':
+            from . import lsprofcalltree
+            calltree = lsprofcalltree.KCacheGrind(p)
+            calltree.output(fp)
+        else:
+            # format == 'text'
+            stats = lsprof.Stats(p.getstats())
+            stats.sort(field)
+            stats.pprint(limit=limit, file=fp, climit=climit)
+
[e-mail]
+def flameprofile(ui, fp):
+    try:
+        from flamegraph import flamegraph
+    except ImportError:
+        raise error.Abort(_(
+            'flamegraph not available - install from '
+            'https://github.com/evanhempel/python-flamegraph'))
+    # developer config: profiling.freq
+    freq = ui.configint('profiling', 'freq', default=1000)
+    filter_ = None
+    collapse_recursion = True
+    thread = flamegraph.ProfileThread(fp, 1.0 / freq,
+                                      filter_, collapse_recursion)
+    start_time = time.clock()
+    try:
+        thread.start()
+        yield
+    finally:
+        thread.stop()
+        thread.join()
+        print('Collected %d stack frames (%d unique) in %2.2f seconds.' % (
+            time.clock() - start_time, thread.num_frames(),
+            thread.num_frames(unique=True)))
+
[e-mail]
+def statprofile(ui, fp):
+    try:
+        import statprof
+    except ImportError:
+        raise error.Abort(_(
+            'statprof not available - install using "easy_install statprof"'))
+
+    freq = ui.configint('profiling', 'freq', default=1000)
+    if freq > 0:
+        # Cannot reset when profiler is already active. So silently no-op.
+        if statprof.state.profile_level == 0:
+            statprof.reset(freq)
+    else:
+        ui.warn(_("invalid sampling frequency '%s' - ignoring\n") % freq)
+
+    statprof.start()
+    try:
+        yield
+    finally:
+        statprof.stop()
+        statprof.display(fp)
+
[e-mail]
+def profile(ui):
+    """Start profiling.
+
+    Profiling is active when the context manager is active. When the context
+    manager exits, profiling results will be written to the configured output.
+    """
+    profiler = os.getenv('HGPROF')
+    if profiler is None:
+        profiler = ui.config('profiling', 'type', default='ls')
+    if profiler not in ('ls', 'stat', 'flame'):
+        ui.warn(_("unrecognized profiler '%s' - ignored\n") % profiler)
+        profiler = 'ls'
+
+    output = ui.config('profiling', 'output')
+
+    if output == 'blackbox':
+        fp = util.stringio()
+    elif output:
+        path = ui.expandpath(output)
+        fp = open(path, 'wb')
+    else:
+        fp = sys.stderr
+
+    try:
+        if profiler == 'ls':
+            proffn = lsprofile
+        elif profiler == 'flame':
+            proffn = flameprofile
+        else:
+            proffn = statprofile
+
+        with proffn(ui, fp):
+            yield
+
+    finally:
+        if output:
+            if output == 'blackbox':
+                val = 'Profile:\n%s' % fp.getvalue()
+                # ui.log treats the input as a format string,
+                # so we need to escape any % signs.
+                val = val.replace('%', '%%')
+                ui.log('profile', val)
+            fp.close()
+
[e-mail]
+def maybeprofile(ui):
+    """Profile if enabled, else do nothing.
+
+    This context manager can be used to optionally profile if profiling
+    is enabled. Otherwise, it does nothing.
+
+    The purpose of this context manager is to make calling code simpler:
+    just use a single code path for calling into code you may want to profile
+    and this function determines whether to start profiling.
+    """
+    if ui.configbool('profiling', 'enabled'):
+        with profile(ui):
+            yield
+    else:
+        yield

Makefile

@@ -164,10 +164,12 b' osx:'
 	  --install-lib=/Library/Python/2.7/site-packages/
 	make -C doc all install DESTDIR="$(PWD)/build/mercurial/"
 	mkdir -p $${OUTPUTDIR:-dist}
-	pkgbuild --root build/mercurial/ --identifier org.mercurial-scm.mercurial \
-	  build/mercurial.pkg
 	HGVER=$$((cat build/mercurial/Library/Python/2.7/site-packages/mercurial/__version__.py; echo 'print(version)') | python) && \
 	OSXVER=$$(sw_vers -productVersion | cut -d. -f1,2) && \
+	pkgbuild --root build/mercurial/ \
+	  --identifier org.mercurial-scm.mercurial \
+	  --version "$${HGVER}" \
+	  build/mercurial.pkg && \
 	productbuild --distribution contrib/macosx/distribution.xml \
 	  --package-path build/ \
 	  --version "$${HGVER}" \

contrib/bash_completion

@@ -89,9 +89,11 b' shopt -s extglob'
 
 _hg_status()
 {
-    local files="$(_hg_cmd status -n$1 "glob:$cur**")"
-    local IFS=$'\n'
-    COMPREPLY=(${COMPREPLY[@]:-} $(compgen -W '$files' -- "$cur"))
+    if [ -z "$HGCOMPLETE_NOSTATUS" ]; then
+        local files="$(_hg_cmd status -n$1 "glob:$cur**")"
+        local IFS=$'\n'
+        COMPREPLY=(${COMPREPLY[@]:-} $(compgen -W '$files' -- "$cur"))
+    fi
 }
 
 _hg_branches()

contrib/check-code.py

@@ -237,7 +237,7 b' pypats = ['
      "tuple parameter unpacking not available in Python 3+"),
     (r'(?<!def)\s+(cmp)\(', "cmp is not available in Python 3+"),
     (r'\breduce\s*\(.*', "reduce is not available in Python 3+"),
-    (r'dict\(.*=', 'dict() is different in Py2 and 3 and is slower than {}',
+    (r'\bdict\(.*=', 'dict() is different in Py2 and 3 and is slower than {}',
      'dict-from-generator'),
     (r'\.has_key\b', "dict.has_key is not available in Python 3+"),
     (r'\s<>\s', '<> operator is not available in Python 3+, use !='),
@@ -295,7 +295,7 b' pypats = ['
      "comparison with singleton, use 'is' or 'is not' instead"),
     (r'^\s*(while|if) [01]:',
      "use True/False for constant Boolean expression"),
-    (r'(?:(?<!def)\s+|\()hasattr',
+    (r'(?:(?<!def)\s+|\()hasattr\(',
      'hasattr(foo, bar) is broken, use util.safehasattr(foo, bar) instead'),
     (r'opener\([^)]*\).read\(',
      "use opener.read() instead"),

contrib/check-commit

@@ -35,13 +35,18 b' errors = ['
      "summary line doesn't start with 'topic: '"),
     (afterheader + r"[A-Z][a-z]\S+", "don't capitalize summary lines"),
     (afterheader + r"[^\n]*: *[A-Z][a-z]\S+", "don't capitalize summary lines"),
-    (afterheader + r"\S*[^A-Za-z0-9-]\S*: ",
+    (afterheader + r"\S*[^A-Za-z0-9-_]\S*: ",
      "summary keyword should be most user-relevant one-word command or topic"),
     (afterheader + r".*\.\s*\n", "don't add trailing period on summary line"),
     (afterheader + r".{79,}", "summary line too long (limit is 78)"),
     (r"\n\+\n( |\+)\n", "adds double empty line"),
     (r"\n \n\+\n", "adds double empty line"),
-    (r"\n\+[ \t]+def [a-z]+_[a-z]", "adds a function with foo_bar naming"),
+    # Forbid "_" in function name.
+    #
+    # We skip the check for cffi related functions. They use names mapping the
+    # name of the C function. C function names may contain "_".
+    (r"\n\+[ \t]+def (?!cffi)[a-z]+_[a-z]",
+     "adds a function with foo_bar naming"),
 ]
 
 word = re.compile('\S')

contrib/check-py3-compat.py

@@ -10,7 +10,6 b''
 from __future__ import absolute_import, print_function
 
 import ast
-import imp
 import os
 import sys
 import traceback
@@ -41,6 +40,7 b' def check_compat_py2(f):'
 
 def check_compat_py3(f):
     """Check Python 3 compatibility of a file with Python 3."""
+    import importlib  # not available on Python 2.6
     with open(f, 'rb') as fh:
         content = fh.read()
 
@@ -55,34 +55,33 b' def check_compat_py3(f):'
     # out module paths for things not in a package can be confusing.
     if f.startswith(('hgext/', 'mercurial/')) and not f.endswith('__init__.py'):
         assert f.endswith('.py')
-        name = f.replace('/', '.')[:-3]
-        with open(f, 'r') as fh:
-            try:
-                imp.load_module(name, fh, '', ('py', 'r', imp.PY_SOURCE))
-            except Exception as e:
-                exc_type, exc_value, tb = sys.exc_info()
-                # We walk the stack and ignore frames from our custom importer,
-                # import mechanisms, and stdlib modules. This kinda/sorta
-                # emulates CPython behavior in import.c while also attempting
-                # to pin blame on a Mercurial file.
-                for frame in reversed(traceback.extract_tb(tb)):
-                    if frame.name == '_call_with_frames_removed':
-                        continue
-                    if 'importlib' in frame.filename:
-                        continue
-                    if 'mercurial/__init__.py' in frame.filename:
-                        continue
-                    if frame.filename.startswith(sys.prefix):
-                        continue
-                    break
+        name = f.replace('/', '.')[:-3].replace('.pure.', '.')
+        try:
+            importlib.import_module(name)
+        except Exception as e:
+            exc_type, exc_value, tb = sys.exc_info()
+            # We walk the stack and ignore frames from our custom importer,
+            # import mechanisms, and stdlib modules. This kinda/sorta
+            # emulates CPython behavior in import.c while also attempting
+            # to pin blame on a Mercurial file.
+            for frame in reversed(traceback.extract_tb(tb)):
+                if frame.name == '_call_with_frames_removed':
+                    continue
+                if 'importlib' in frame.filename:
+                    continue
+                if 'mercurial/__init__.py' in frame.filename:
+                    continue
+                if frame.filename.startswith(sys.prefix):
+                    continue
+                break
 
-                if frame.filename:
-                    filename = os.path.basename(frame.filename)
-                    print('%s: error importing: <%s> %s (error at %s:%d)' % (
-                          f, type(e).__name__, e, filename, frame.lineno))
-                else:
-                    print('%s: error importing module: <%s> %s (line %d)' % (
-                          f, type(e).__name__, e, frame.lineno))
+            if frame.filename:
+                filename = os.path.basename(frame.filename)
+                print('%s: error importing: <%s> %s (error at %s:%d)' % (
+                      f, type(e).__name__, e, filename, frame.lineno))
+            else:
+                print('%s: error importing module: <%s> %s (line %d)' % (
+                      f, type(e).__name__, e, frame.lineno))
 
 if __name__ == '__main__':
     if sys.version_info[0] == 2:

contrib/chg/hgclient.c

@@ -126,15 +126,10 b' static void readchannel(hgclient_t *hgc)'
 		return;  /* assumes input request */
 
 	size_t cursize = 0;
-	int emptycount = 0;
 	while (cursize < hgc->ctx.datasize) {
 		rsize = recv(hgc->sockfd, hgc->ctx.data + cursize,
 			     hgc->ctx.datasize - cursize, 0);
-		/* rsize == 0 normally indicates EOF, while it's also a valid
-		 * packet size for unix socket. treat it as EOF and abort if
-		 * we get many empty responses in a row. */
-		emptycount = (rsize == 0 ? emptycount + 1 : 0);
-		if (rsize < 0 || emptycount > 20)
+		if (rsize < 1)
 			abortmsg("failed to read data block");
 		cursize += rsize;
 	}

contrib/perf.py

@@ -25,6 +25,7 b' import random'
 import sys
 import time
 from mercurial import (
+    changegroup,
     cmdutil,
     commands,
     copies,
@@ -130,15 +131,53 b' def gettimer(ui, opts=None):'
 
     # enforce an idle period before execution to counteract power management
     # experimental config: perf.presleep
-    time.sleep(ui.configint("perf", "presleep", 1))
+    time.sleep(getint(ui, "perf", "presleep", 1))
 
     if opts is None:
         opts = {}
     # redirect all to stderr
     ui = ui.copy()
-    ui.fout = ui.ferr
+    uifout = safeattrsetter(ui, 'fout', ignoremissing=True)
+    if uifout:
+        # for "historical portability":
+        # ui.fout/ferr have been available since 1.9 (or 4e1ccd4c2b6d)
+        uifout.set(ui.ferr)
+
     # get a formatter
-    fm = ui.formatter('perf', opts)
+    uiformatter = getattr(ui, 'formatter', None)
+    if uiformatter:
+        fm = uiformatter('perf', opts)
+    else:
+        # for "historical portability":
+        # define formatter locally, because ui.formatter has been
+        # available since 2.2 (or ae5f92e154d3)
+        from mercurial import node
+        class defaultformatter(object):
+            """Minimized composition of baseformatter and plainformatter
+            """
+            def __init__(self, ui, topic, opts):
+                self._ui = ui
+                if ui.debugflag:
+                    self.hexfunc = node.hex
+                else:
+                    self.hexfunc = node.short
+            def __nonzero__(self):
+                return False
+            def startitem(self):
+                pass
+            def data(self, **data):
+                pass
+            def write(self, fields, deftext, *fielddata, **opts):
+                self._ui.write(deftext % fielddata, **opts)
+            def condwrite(self, cond, fields, deftext, *fielddata, **opts):
+                if cond:
+                    self._ui.write(deftext % fielddata, **opts)
+            def plain(self, text, **opts):
+                self._ui.write(text, **opts)
+            def end(self):
+                pass
+        fm = defaultformatter(ui, 'perf', opts)
+
     # stub function, runs code only once instead of in a loop
     # experimental config: perf.stub
     if ui.configbool("perf", "stub"):
@@ -181,6 +220,121 b' def _timer(fm, func, title=None):'
     fm.write('count',  ' (best of %d)', count)
     fm.plain('\n')
 
+# utilities for historical portability
+
+def getint(ui, section, name, default):
+    # for "historical portability":
+    # ui.configint has been available since 1.9 (or fa2b596db182)
+    v = ui.config(section, name, None)
+    if v is None:
+        return default
+    try:
+        return int(v)
+    except ValueError:
+        raise error.ConfigError(("%s.%s is not an integer ('%s')")
+                                % (section, name, v))
+
+def safeattrsetter(obj, name, ignoremissing=False):
+    """Ensure that 'obj' has 'name' attribute before subsequent setattr
+
+    This function is aborted, if 'obj' doesn't have 'name' attribute
+    at runtime. This avoids overlooking removal of an attribute, which
+    breaks assumption of performance measurement, in the future.
+
+    This function returns the object to (1) assign a new value, and
+    (2) restore an original value to the attribute.
+
+    If 'ignoremissing' is true, missing 'name' attribute doesn't cause
+    abortion, and this function returns None. This is useful to
+    examine an attribute, which isn't ensured in all Mercurial
+    versions.
+    """
+    if not util.safehasattr(obj, name):
+        if ignoremissing:
+            return None
+        raise error.Abort(("missing attribute %s of %s might break assumption"
+                           " of performance measurement") % (name, obj))
+
+    origvalue = getattr(obj, name)
+    class attrutil(object):
+        def set(self, newvalue):
+            setattr(obj, name, newvalue)
+        def restore(self):
+            setattr(obj, name, origvalue)
+
+    return attrutil()
+
+# utilities to examine each internal API changes
+
+def getbranchmapsubsettable():
+    # for "historical portability":
+    # subsettable is defined in:
+    # - branchmap since 2.9 (or 175c6fd8cacc)
+    # - repoview since 2.5 (or 59a9f18d4587)
+    for mod in (branchmap, repoview):
+        subsettable = getattr(mod, 'subsettable', None)
+        if subsettable:
+            return subsettable
+
+    # bisecting in bcee63733aad::59a9f18d4587 can reach here (both
+    # branchmap and repoview modules exist, but subsettable attribute
+    # doesn't)
+    raise error.Abort(("perfbranchmap not available with this Mercurial"),
+                      hint="use 2.5 or later")
+
+def getsvfs(repo):
+    """Return appropriate object to access files under .hg/store
+    """
+    # for "historical portability":
+    # repo.svfs has been available since 2.3 (or 7034365089bf)
+    svfs = getattr(repo, 'svfs', None)
+    if svfs:
+        return svfs
+    else:
+        return getattr(repo, 'sopener')
+
+def getvfs(repo):
+    """Return appropriate object to access files under .hg
+    """
+    # for "historical portability":
+    # repo.vfs has been available since 2.3 (or 7034365089bf)
+    vfs = getattr(repo, 'vfs', None)
+    if vfs:
+        return vfs
+    else:
+        return getattr(repo, 'opener')
+
+def repocleartagscachefunc(repo):
+    """Return the function to clear tags cache according to repo internal API
+    """
+    if util.safehasattr(repo, '_tagscache'): # since 2.0 (or 9dca7653b525)
+        # in this case, setattr(repo, '_tagscache', None) or so isn't
+        # correct way to clear tags cache, because existing code paths
+        # expect _tagscache to be a structured object.
+        def clearcache():
+            # _tagscache has been filteredpropertycache since 2.5 (or
+            # 98c867ac1330), and delattr() can't work in such case
+            if '_tagscache' in vars(repo):
+                del repo.__dict__['_tagscache']
+        return clearcache
+
+    repotags = safeattrsetter(repo, '_tags', ignoremissing=True)
+    if repotags: # since 1.4 (or 5614a628d173)
+        return lambda : repotags.set(None)
+
+    repotagscache = safeattrsetter(repo, 'tagscache', ignoremissing=True)
+    if repotagscache: # since 0.6 (or d7df759d0e97)
+        return lambda : repotagscache.set(None)
+
+    # Mercurial earlier than 0.6 (or d7df759d0e97) logically reaches
+    # this point, but it isn't so problematic, because:
+    # - repo.tags of such Mercurial isn't "callable", and repo.tags()
+    #   in perftags() causes failure soon
+    # - perf.py itself has been available since 1.1 (or eb240755386d)
+    raise error.Abort(("tags API of this hg command is unknown"))
+
+# perf commands
+
 @command('perfwalk', formatteropts)
 def perfwalk(ui, repo, *pats, **opts):
     timer, fm = gettimer(ui, opts)
@@ -249,10 +403,12 b' def perftags(ui, repo, **opts):'
     import mercurial.changelog
     import mercurial.manifest
     timer, fm = gettimer(ui, opts)
+    svfs = getsvfs(repo)
+    repocleartagscache = repocleartagscachefunc(repo)
     def t():
-        repo.changelog = mercurial.changelog.changelog(repo.svfs)
-        repo.manifest = mercurial.manifest.manifest(repo.svfs)
-        repo._tags = None
+        repo.changelog = mercurial.changelog.changelog(svfs)
+        repo.manifest = mercurial.manifest.manifest(svfs)
+        repocleartagscache()
         return len(repo.tags())
     timer(t)
     fm.end()
@@ -279,6 +435,37 b' def perfancestorset(ui, repo, revset, **'
     timer(d)
     fm.end()
 
+@command('perfchangegroupchangelog', formatteropts +
+         [('', 'version', '02', 'changegroup version'),
+          ('r', 'rev', '', 'revisions to add to changegroup')])
+def perfchangegroupchangelog(ui, repo, version='02', rev=None, **opts):
+    """Benchmark producing a changelog group for a changegroup.
+
+    This measures the time spent processing the changelog during a
+    bundle operation. This occurs during `hg bundle` and on a server
+    processing a `getbundle` wire protocol request (handles clones
+    and pull requests).
+
+    By default, all revisions are added to the changegroup.
+    """
+    cl = repo.changelog
+    revs = [cl.lookup(r) for r in repo.revs(rev or 'all()')]
+    bundler = changegroup.getbundler(version, repo)
+
+    def lookup(node):
+        # The real bundler reads the revision in order to access the
+        # manifest node and files list. Do that here.
+        cl.read(node)
+        return node
+
+    def d():
+        for chunk in bundler.group(revs, cl, lookup):
+            pass
+
+    timer, fm = gettimer(ui, opts)
+    timer(d)
+    fm.end()
+
 @command('perfdirs', formatteropts)
 def perfdirs(ui, repo, **opts):
     timer, fm = gettimer(ui, opts)
@@ -399,8 +586,9 b' def perfindex(ui, repo, **opts):'
     timer, fm = gettimer(ui, opts)
     mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
     n = repo["tip"].node()
+    svfs = getsvfs(repo)
     def d():
-        cl = mercurial.revlog.revlog(repo.svfs, "00changelog.i")
+        cl = mercurial.revlog.revlog(svfs, "00changelog.i")
         cl.rev(n)
     timer(d)
     fm.end()
@@ -423,7 +611,7 b' def perfparents(ui, repo, **opts):'
     timer, fm = gettimer(ui, opts)
     # control the number of commits perfparents iterates over
     # experimental config: perf.parentscount
-    count = ui.configint("perf", "parentscount", 1000)
+    count = getint(ui, "perf", "parentscount", 1000)
     if len(repo.changelog) < count:
         raise error.Abort("repo needs %d commits for this test" % count)
     repo = repo.unfiltered()
@@ -472,7 +660,7 b' def perfnodelookup(ui, repo, rev, **opts'
     import mercurial.revlog
     mercurial.revlog._prereadsize = 2**24 # disable lazy parser in old hg
     n = repo[rev].node()
-    cl = mercurial.revlog.revlog(repo.svfs, "00changelog.i")
+    cl = mercurial.revlog.revlog(getsvfs(repo), "00changelog.i")
     def d():
         cl.rev(n)
         clearcaches(cl)
@@ -543,8 +731,8 b' def perffncachewrite(ui, repo, **opts):'
         s.fncache._dirty = True
         s.fncache.write(tr)
     timer(d)
+    tr.close()
     lock.release()
-    tr.close()
     fm.end()
 
 @command('perffncacheencode', formatteropts)
@@ -580,9 +768,10 b' def perfdiffwd(ui, repo, **opts):'
 
 @command('perfrevlog', revlogopts + formatteropts +
          [('d', 'dist', 100, 'distance between the revisions'),
-          ('s', 'startrev', 0, 'revision to start reading at')],
+          ('s', 'startrev', 0, 'revision to start reading at'),
+          ('', 'reverse', False, 'read in reverse')],
          '-c|-m|FILE')
-def perfrevlog(ui, repo, file_=None, startrev=0, **opts):
+def perfrevlog(ui, repo, file_=None, startrev=0, reverse=False, **opts):
     """Benchmark reading a series of revisions from a revlog.
 
     By default, we read every ``-d/--dist`` revision from 0 to tip of
@@ -591,11 +780,20 b' def perfrevlog(ui, repo, file_=None, sta'
     The start revision can be defined via ``-s/--startrev``.
     """
     timer, fm = gettimer(ui, opts)
-    dist = opts['dist']
     _len = getlen(ui)
+
     def d():
         r = cmdutil.openrevlog(repo, 'perfrevlog', file_, opts)
-        for x in xrange(startrev, _len(r), dist):
+
+        startrev = 0
+        endrev = _len(r)
+        dist = opts['dist']
+
+        if reverse:
+            startrev, endrev = endrev, startrev
+            dist = -1 * dist
+
+        for x in xrange(startrev, endrev, dist):
             r.revision(r.node(x))
 
     timer(d)
@@ -772,10 +970,11 b' def perfbranchmap(ui, repo, full=False, '
         return d
     # add filter in smaller subset to bigger subset
     possiblefilters = set(repoview.filtertable)
+    subsettable = getbranchmapsubsettable()
     allfilters = []
     while possiblefilters:
         for name in possiblefilters:
-            subset = branchmap.subsettable.get(name)
+            subset = subsettable.get(name)
             if subset not in possiblefilters:
                 break
         else:
@@ -789,16 +988,17 b' def perfbranchmap(ui, repo, full=False, '
             repo.filtered(name).branchmap()
     # add unfiltered
     allfilters.append(None)
-    oldread = branchmap.read
-    oldwrite = branchmap.branchcache.write
+
+    branchcacheread = safeattrsetter(branchmap, 'read')
+    branchcachewrite = safeattrsetter(branchmap.branchcache, 'write')
+    branchcacheread.set(lambda repo: None)
+    branchcachewrite.set(lambda bc, repo: None)
     try:
-        branchmap.read = lambda repo: None
-        branchmap.write = lambda repo: None
         for name in allfilters:
             timer(getbranchmap(name), title=str(name))
     finally:
-        branchmap.read = oldread
-        branchmap.branchcache.write = oldwrite
+        branchcacheread.restore()
+        branchcachewrite.restore()
     fm.end()
 
 @command('perfloadmarkers')
@@ -807,7 +1007,8 b' def perfloadmarkers(ui, repo):'
 
     Result is the number of markers in the repo."""
     timer, fm = gettimer(ui)
-    timer(lambda: len(obsolete.obsstore(repo.svfs)))
+    svfs = getsvfs(repo)
+    timer(lambda: len(obsolete.obsstore(svfs)))
     fm.end()
 
 @command('perflrucachedict', formatteropts +

contrib/synthrepo.py

@@ -62,11 +62,11 b' from mercurial import ('
     util,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)

contrib/wix/help.wxs

@@ -42,6 +42,7 b''
             <File Id="internals.changegroups.txt" Name="changegroups.txt" />
             <File Id="internals.requirements.txt" Name="requirements.txt" />
             <File Id="internals.revlogs.txt"      Name="revlogs.txt" />
+            <File Id="internals.wireprotocol.txt" Name="wireprotocol.txt" />
           </Component>
         </Directory>
 

contrib/zsh_completion

@@ -371,7 +371,7 b' typeset -A _hg_cmd_globals'
 
 # Common options
 _hg_global_opts=(
-    '(--repository -R)'{-R+,--repository}'[repository root directory]:repository:_files -/'
+    '(--repository -R)'{-R+,--repository=}'[repository root directory]:repository:_files -/'
     '--cwd[change working directory]:new working directory:_files -/'
     '(--noninteractive -y)'{-y,--noninteractive}'[do not prompt, assume yes for any required answers]'
     '(--verbose -v)'{-v,--verbose}'[enable additional output]'
@@ -390,8 +390,8 b' typeset -A _hg_cmd_globals'
 )
 
 _hg_pat_opts=(
-  '*'{-I+,--include}'[include names matching the given patterns]:dir:_files -W $(_hg_cmd root) -/'
-  '*'{-X+,--exclude}'[exclude names matching the given patterns]:dir:_files -W $(_hg_cmd root) -/')
+  '*'{-I+,--include=}'[include names matching the given patterns]:dir:_files -W $(_hg_cmd root) -/'
+  '*'{-X+,--exclude=}'[exclude names matching the given patterns]:dir:_files -W $(_hg_cmd root) -/')
 
 _hg_clone_opts=(
   $_hg_remote_opts
@@ -402,8 +402,8 b' typeset -A _hg_cmd_globals'
 _hg_date_user_opts=(
   '(--currentdate -D)'{-D,--currentdate}'[record the current date as commit date]'
   '(--currentuser -U)'{-U,--currentuser}'[record the current user as committer]'
-  '(--date -d)'{-d+,--date}'[record the specified date as commit date]:date:'
-  '(--user -u)'{-u+,--user}'[record the specified user as committer]:user:')
+  '(--date -d)'{-d+,--date=}'[record the specified date as commit date]:date:'
+  '(--user -u)'{-u+,--user=}'[record the specified user as committer]:user:')
 
 _hg_gitlike_opts=(
   '(--git -g)'{-g,--git}'[use git extended diff format]')
@@ -414,7 +414,7 b' typeset -A _hg_cmd_globals'
   '--nodates[omit dates from diff headers]')
 
 _hg_mergetool_opts=(
-  '(--tool -t)'{-t+,--tool}'[specify merge tool]:tool:')
+  '(--tool -t)'{-t+,--tool=}'[specify merge tool]:tool:')
 
 _hg_dryrun_opts=(
   '(--dry-run -n)'{-n,--dry-run}'[do not perform actions, just print output]')
@@ -430,7 +430,7 b' typeset -A _hg_cmd_globals'
 
 _hg_log_opts=(
   $_hg_global_opts $_hg_style_opts $_hg_gitlike_opts
-  '(--limit -l)'{-l+,--limit}'[limit number of changes displayed]:'
+  '(--limit -l)'{-l+,--limit=}'[limit number of changes displayed]:'
   '(--no-merges -M)'{-M,--no-merges}'[do not show merges]'
   '(--patch -p)'{-p,--patch}'[show patch]'
   '--stat[output diffstat-style summary of changes]'
@@ -438,16 +438,16 b' typeset -A _hg_cmd_globals'
 
 _hg_commit_opts=(
   '(-m --message -l --logfile --edit -e)'{-e,--edit}'[edit commit message]'
-  '(-e --edit -l --logfile --message -m)'{-m+,--message}'[use <text> as commit message]:message:'
-  '(-e --edit -m --message --logfile -l)'{-l+,--logfile}'[read the commit message from <file>]:log file:_files')
+  '(-e --edit -l --logfile --message -m)'{-m+,--message=}'[use <text> as commit message]:message:'
+  '(-e --edit -m --message --logfile -l)'{-l+,--logfile=}'[read the commit message from <file>]:log file:_files')
 
 _hg_remote_opts=(
-  '(--ssh -e)'{-e+,--ssh}'[specify ssh command to use]:'
+  '(--ssh -e)'{-e+,--ssh=}'[specify ssh command to use]:'
   '--remotecmd[specify hg command to run on the remote side]:')
 
 _hg_branch_bmark_opts=(
-  '(--bookmark -B)'{-B+,--bookmark}'[specify bookmark(s)]:bookmark:_hg_bookmarks'
-  '(--branch -b)'{-b+,--branch}'[specify branch(es)]:branch:_hg_branches'
+  '(--bookmark -B)'{-B+,--bookmark=}'[specify bookmark(s)]:bookmark:_hg_bookmarks'
+  '(--branch -b)'{-b+,--branch=}'[specify branch(es)]:branch:_hg_branches'
 )
 
 _hg_subrepos_opts=(
@@ -464,13 +464,13 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_addremove() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts $_hg_dryrun_opts \
-  '(--similarity -s)'{-s+,--similarity}'[guess renamed files by similarity (0<=s<=100)]:' \
+  '(--similarity -s)'{-s+,--similarity=}'[guess renamed files by similarity (0<=s<=100)]:' \
   '*:unknown or missing files:_hg_addremove'
 }
 
 _hg_cmd_annotate() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts \
-  '(--rev -r)'{-r+,--rev}'[annotate the specified revision]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[annotate the specified revision]:revision:_hg_labels' \
   '(--follow -f)'{-f,--follow}'[follow file copies and renames]' \
   '(--text -a)'{-a,--text}'[treat all files as text]' \
   '(--user -u)'{-u,--user}'[list the author]' \
@@ -483,21 +483,21 b' typeset -A _hg_cmd_globals'
 _hg_cmd_archive() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts $_hg_subrepos_opts \
   '--no-decode[do not pass files through decoders]' \
-  '(--prefix -p)'{-p+,--prefix}'[directory prefix for files in archive]:' \
-  '(--rev -r)'{-r+,--rev}'[revision to distribute]:revision:_hg_labels' \
-  '(--type -t)'{-t+,--type}'[type of distribution to create]:archive type:(files tar tbz2 tgz uzip zip)' \
+  '(--prefix -p)'{-p+,--prefix=}'[directory prefix for files in archive]:' \
+  '(--rev -r)'{-r+,--rev=}'[revision to distribute]:revision:_hg_labels' \
+  '(--type -t)'{-t+,--type=}'[type of distribution to create]:archive type:(files tar tbz2 tgz uzip zip)' \
   '*:destination:_files'
 }
 
 _hg_cmd_backout() {
   _arguments -s -w : $_hg_global_opts $_hg_mergetool_opts $_hg_pat_opts \
     '--merge[merge with old dirstate parent after backout]' \
-    '(--date -d)'{-d+,--date}'[record datecode as commit date]:date code:' \
+    '(--date -d)'{-d+,--date=}'[record datecode as commit date]:date code:' \
     '--parent[parent to choose when backing out merge]' \
-    '(--user -u)'{-u+,--user}'[record user as commiter]:user:' \
-    '(--rev -r)'{-r+,--rev}'[revision]:revision:_hg_labels' \
-    '(--message -m)'{-m+,--message}'[use <text> as commit message]:text:' \
-    '(--logfile -l)'{-l+,--logfile}'[read commit message from <file>]:log file:_files'
+    '(--user -u)'{-u+,--user=}'[record user as commiter]:user:' \
+    '(--rev -r)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
+    '(--message -m)'{-m+,--message=}'[use <text> as commit message]:text:' \
+    '(--logfile -l)'{-l+,--logfile=}'[read commit message from <file>]:log file:_files'
 }
 
 _hg_cmd_bisect() {
@@ -507,7 +507,7 b' typeset -A _hg_cmd_globals'
   '(--good -g --bad -b --skip -s --reset -r)'{-g,--good}'[mark changeset good]'::revision:_hg_labels \
   '(--good -g --bad -b --skip -s --reset -r)'{-b,--bad}'[mark changeset bad]'::revision:_hg_labels \
   '(--good -g --bad -b --skip -s --reset -r)'{-s,--skip}'[skip testing changeset]' \
-  '(--command -c --noupdate -U)'{-c+,--command}'[use command to check changeset state]':commands:_command_names \
+  '(--command -c --noupdate -U)'{-c+,--command=}'[use command to check changeset state]':commands:_command_names \
   '(--command -c --noupdate -U)'{-U,--noupdate}'[do not update to target]'
 }
 
@@ -515,9 +515,9 b' typeset -A _hg_cmd_globals'
   _arguments -s -w : $_hg_global_opts \
   '(--force -f)'{-f,--force}'[force]' \
   '(--inactive -i)'{-i,--inactive}'[mark a bookmark inactive]' \
-  '(--rev -r --delete -d --rename -m)'{-r+,--rev}'[revision]:revision:_hg_labels' \
+  '(--rev -r --delete -d --rename -m)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
   '(--rev -r --delete -d --rename -m)'{-d,--delete}'[delete a given bookmark]' \
-  '(--rev -r --delete -d --rename -m)'{-m+,--rename}'[rename a given bookmark]:bookmark:_hg_bookmarks' \
+  '(--rev -r --delete -d --rename -m)'{-m+,--rename=}'[rename a given bookmark]:bookmark:_hg_bookmarks' \
   ':bookmark:_hg_bookmarks'
 }
 
@@ -537,8 +537,8 b' typeset -A _hg_cmd_globals'
   _arguments -s -w : $_hg_global_opts $_hg_remote_opts \
   '(--force -f)'{-f,--force}'[run even when remote repository is unrelated]' \
   '(2)*--base[a base changeset to specify instead of a destination]:revision:_hg_labels' \
-  '(--branch -b)'{-b+,--branch}'[a specific branch to bundle]' \
-  '(--rev -r)'{-r+,--rev}'[changeset(s) to bundle]:' \
+  '(--branch -b)'{-b+,--branch=}'[a specific branch to bundle]:' \
+  '(--rev -r)'{-r+,--rev=}'[changeset(s) to bundle]:' \
   '--all[bundle all changesets in the repository]' \
   ':output file:_files' \
   ':destination repository:_files -/'
@@ -546,17 +546,17 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_cat() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts \
-  '(--output -o)'{-o+,--output}'[print output to file with formatted name]:filespec:' \
-  '(--rev -r)'{-r+,--rev}'[revision]:revision:_hg_labels' \
+  '(--output -o)'{-o+,--output=}'[print output to file with formatted name]:filespec:' \
+  '(--rev -r)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
   '--decode[apply any matching decode filter]' \
   '*:file:_hg_files'
 }
 
 _hg_cmd_clone() {
   _arguments -s -w : $_hg_global_opts $_hg_clone_opts \
-  '(--rev -r)'{-r+,--rev}'[a changeset you would like to have after cloning]:' \
-  '(--updaterev -u)'{-u+,--updaterev}'[revision, tag or branch to check out]' \
-  '(--branch -b)'{-b+,--branch}'[clone only the specified branch]' \
+  '(--rev -r)'{-r+,--rev=}'[a changeset you would like to have after cloning]:' \
+  '(--updaterev -u)'{-u+,--updaterev=}'[revision, tag or branch to check out]:' \
+  '(--branch -b)'{-b+,--branch=}'[clone only the specified branch]:' \
   ':source repository:_hg_remote' \
   ':destination:_hg_clone_dest'
 }
@@ -564,10 +564,10 b' typeset -A _hg_cmd_globals'
 _hg_cmd_commit() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts $_hg_subrepos_opts \
   '(--addremove -A)'{-A,--addremove}'[mark new/missing files as added/removed before committing]' \
-  '(--message -m)'{-m+,--message}'[use <text> as commit message]:text:' \
-  '(--logfile -l)'{-l+,--logfile}'[read commit message from <file>]:log file:_files' \
-  '(--date -d)'{-d+,--date}'[record datecode as commit date]:date code:' \
-  '(--user -u)'{-u+,--user}'[record user as commiter]:user:' \
+  '(--message -m)'{-m+,--message=}'[use <text> as commit message]:text:' \
+  '(--logfile -l)'{-l+,--logfile=}'[read commit message from <file>]:log file:_files' \
+  '(--date -d)'{-d+,--date=}'[record datecode as commit date]:date code:' \
+  '(--user -u)'{-u+,--user=}'[record user as commiter]:user:' \
   '--amend[amend the parent of the working dir]' \
   '--close-branch[mark a branch as closed]' \
   '*:file:_hg_files'
@@ -584,12 +584,12 b' typeset -A _hg_cmd_globals'
   typeset -A opt_args
   _arguments -s -w : $_hg_global_opts $_hg_diff_opts $_hg_ignore_space_opts \
                      $_hg_pat_opts $_hg_subrepos_opts \
-  '*'{-r,--rev}'+[revision]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[revision]:revision:_hg_revrange' \
   '(--show-function -p)'{-p,--show-function}'[show which function each change is in]' \
-  '(--change -c)'{-c,--change}'[change made by revision]' \
+  '(--change -c)'{-c+,--change=}'[change made by revision]:' \
   '(--text -a)'{-a,--text}'[treat all files as text]' \
   '--reverse[produce a diff that undoes the changes]' \
-  '(--unified -U)'{-U,--unified}'[number of lines of context to show]' \
+  '(--unified -U)'{-U+,--unified=}'[number of lines of context to show]:' \
   '--stat[output diffstat-style summary of changes]' \
   '*:file:->diff_files'
 
@@ -606,9 +606,9 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_export() {
   _arguments -s -w : $_hg_global_opts $_hg_diff_opts \
-  '(--outout -o)'{-o+,--output}'[print output to file with formatted name]:filespec:' \
+  '(--outout -o)'{-o+,--output=}'[print output to file with formatted name]:filespec:' \
   '--switch-parent[diff against the second parent]' \
-  '(--rev -r)'{-r+,--rev}'[revision]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
   '*:revision:_hg_labels'
 }
 
@@ -634,7 +634,7 b' typeset -A _hg_cmd_globals'
   '(--ignore-case -i)'{-i,--ignore-case}'[ignore case when matching]' \
   '(--files-with-matches -l)'{-l,--files-with-matches}'[print only filenames and revs that match]' \
   '(--line-number -n)'{-n,--line-number}'[print matching line numbers]' \
-  '*'{-r+,--rev}'[search in given revision range]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[search in given revision range]:revision:_hg_revrange' \
   '(--user -u)'{-u,--user}'[print user who committed change]' \
   '(--date -d)'{-d,--date}'[print date of a changeset]' \
   '1:search pattern:' \
@@ -645,7 +645,7 b' typeset -A _hg_cmd_globals'
   _arguments -s -w : $_hg_global_opts $_hg_style_opts \
   '(--topo -t)'{-t,--topo}'[show topological heads only]' \
   '(--closed -c)'{-c,--closed}'[show normal and closed branch heads]' \
-  '(--rev -r)'{-r+,--rev}'[show only heads which are descendants of rev]:revision:_hg_labels'
+  '(--rev -r)'{-r+,--rev=}'[show only heads which are descendants of rev]:revision:_hg_labels'
 }
 
 _hg_cmd_help() {
@@ -658,25 +658,25 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_identify() {
   _arguments -s -w : $_hg_global_opts $_hg_remote_opts \
-  '(--rev -r)'{-r+,--rev}'[identify the specified rev]:revision:_hg_labels' \
-  '(--num -n)'{-n+,--num}'[show local revision number]' \
-  '(--id -i)'{-i+,--id}'[show global revision id]' \
-  '(--branch -b)'{-b+,--branch}'[show branch]' \
-  '(--bookmark -B)'{-B+,--bookmark}'[show bookmarks]' \
-  '(--tags -t)'{-t+,--tags}'[show tags]'
+  '(--rev -r)'{-r+,--rev=}'[identify the specified rev]:revision:_hg_labels' \
+  '(--num -n)'{-n,--num}'[show local revision number]' \
+  '(--id -i)'{-i,--id}'[show global revision id]' \
+  '(--branch -b)'{-b,--branch}'[show branch]' \
+  '(--bookmark -B)'{-B,--bookmark}'[show bookmarks]' \
+  '(--tags -t)'{-t,--tags}'[show tags]'
 }
 
 _hg_cmd_import() {
   _arguments -s -w : $_hg_global_opts $_hg_commit_opts \
-  '(--strip -p)'{-p+,--strip}'[directory strip option for patch (default: 1)]:count:' \
+  '(--strip -p)'{-p+,--strip=}'[directory strip option for patch (default: 1)]:count:' \
   '(--force -f)'{-f,--force}'[skip check for outstanding uncommitted changes]' \
   '--bypass[apply patch without touching the working directory]' \
   '--no-commit[do not commit, just update the working directory]' \
   '--exact[apply patch to the nodes from which it was generated]' \
   '--import-branch[use any branch information in patch (implied by --exact)]' \
-  '(--date -d)'{-d+,--date}'[record datecode as commit date]:date code:' \
-  '(--user -u)'{-u+,--user}'[record user as commiter]:user:' \
-  '(--similarity -s)'{-s+,--similarity}'[guess renamed files by similarity (0<=s<=100)]:' \
+  '(--date -d)'{-d+,--date=}'[record datecode as commit date]:date code:' \
+  '(--user -u)'{-u+,--user=}'[record user as commiter]:user:' \
+  '(--similarity -s)'{-s+,--similarity=}'[guess renamed files by similarity (0<=s<=100)]:' \
   '*:patch:_files'
 }
 
@@ -684,7 +684,7 b' typeset -A _hg_cmd_globals'
   _arguments -s -w : $_hg_log_opts $_hg_branch_bmark_opts $_hg_remote_opts \
                      $_hg_subrepos_opts \
   '(--force -f)'{-f,--force}'[run even when the remote repository is unrelated]' \
-  '(--rev -r)'{-r+,--rev}'[a specific revision up to which you would like to pull]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[a specific revision up to which you would like to pull]:revision:_hg_labels' \
   '(--newest-first -n)'{-n,--newest-first}'[show newest record first]' \
   '--bundle[file to store the bundles into]:bundle file:_files' \
   ':source:_hg_remote'
@@ -697,7 +697,7 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_locate() {
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts \
-  '(--rev -r)'{-r+,--rev}'[search repository as it stood at revision]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[search repository as it stood at revision]:revision:_hg_labels' \
   '(--print0 -0)'{-0,--print0}'[end filenames with NUL, for use with xargs]' \
   '(--fullpath -f)'{-f,--fullpath}'[print complete paths]' \
   '*:search pattern:_hg_files'
@@ -709,27 +709,27 b' typeset -A _hg_cmd_globals'
   '(-f --follow)--follow-first[only follow the first parent of merge changesets]' \
   '(--copies -C)'{-C,--copies}'[show copied files]' \
   '(--keyword -k)'{-k+,--keyword}'[search for a keyword]:' \
-  '*'{-r,--rev}'[show the specified revision or revset]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[show the specified revision or revset]:revision:_hg_revrange' \
   '(--only-merges -m)'{-m,--only-merges}'[show only merges]' \
-  '(--prune -P)'{-P+,--prune}'[do not display revision or any of its ancestors]:revision:_hg_labels' \
-  '(--graph -G)'{-G+,--graph}'[show the revision DAG]' \
-  '(--branch -b)'{-b+,--branch}'[show changesets within the given named branch]:branch:_hg_branches' \
-  '(--user -u)'{-u+,--user}'[revisions committed by user]:user:' \
-  '(--date -d)'{-d+,--date}'[show revisions matching date spec]:date:' \
+  '(--prune -P)'{-P+,--prune=}'[do not display revision or any of its ancestors]:revision:_hg_labels' \
+  '(--graph -G)'{-G,--graph}'[show the revision DAG]' \
+  '(--branch -b)'{-b+,--branch=}'[show changesets within the given named branch]:branch:_hg_branches' \
+  '(--user -u)'{-u+,--user=}'[revisions committed by user]:user:' \
+  '(--date -d)'{-d+,--date=}'[show revisions matching date spec]:date:' \
   '*:files:_hg_files'
 }
 
 _hg_cmd_manifest() {
   _arguments -s -w : $_hg_global_opts \
   '--all[list files from all revisions]' \
-  '(--rev -r)'{-r+,--rev}'[revision to display]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[revision to display]:revision:_hg_labels' \
   ':revision:_hg_labels'
 }
 
 _hg_cmd_merge() {
   _arguments -s -w : $_hg_global_opts $_hg_mergetool_opts \
   '(--force -f)'{-f,--force}'[force a merge with outstanding changes]' \
-  '(--rev -r 1)'{-r,--rev}'[revision to merge]:revision:_hg_mergerevs' \
+  '(--rev -r 1)'{-r+,--rev=}'[revision to merge]:revision:_hg_mergerevs' \
   '(--preview -P)'{-P,--preview}'[review revisions to merge (no merge is performed)]' \
   ':revision:_hg_mergerevs'
 }
@@ -738,14 +738,14 b' typeset -A _hg_cmd_globals'
   _arguments -s -w : $_hg_log_opts $_hg_branch_bmark_opts $_hg_remote_opts \
                      $_hg_subrepos_opts \
   '(--force -f)'{-f,--force}'[run even when the remote repository is unrelated]' \
-  '*'{-r,--rev}'[a specific revision you would like to push]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[a specific revision you would like to push]:revision:_hg_revrange' \
   '(--newest-first -n)'{-n,--newest-first}'[show newest record first]' \
   ':destination:_hg_remote'
 }
 
 _hg_cmd_parents() {
   _arguments -s -w : $_hg_global_opts $_hg_style_opts \
-  '(--rev -r)'{-r+,--rev}'[show parents of the specified rev]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[show parents of the specified rev]:revision:_hg_labels' \
   ':last modified file:_hg_files'
 }
 
@@ -760,7 +760,7 b' typeset -A _hg_cmd_globals'
   '(--draft -d)'{-d,--draft}'[set changeset phase to draft]' \
   '(--secret -s)'{-s,--secret}'[set changeset phase to secret]' \
   '(--force -f)'{-f,--force}'[allow to move boundary backward]' \
-  '(--rev -r)'{-r+,--rev}'[target revision]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[target revision]:revision:_hg_labels' \
   ':revision:_hg_labels'
 }
 
@@ -775,7 +775,7 b' typeset -A _hg_cmd_globals'
 _hg_cmd_push() {
   _arguments -s -w : $_hg_global_opts $_hg_branch_bmark_opts $_hg_remote_opts \
   '(--force -f)'{-f,--force}'[force push]' \
-  '(--rev -r)'{-r+,--rev}'[a specific revision you would like to push]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[a specific revision you would like to push]:revision:_hg_labels' \
   '--new-branch[allow pushing a new branch]' \
   ':destination:_hg_remote'
 }
@@ -819,9 +819,9 b' typeset -A _hg_cmd_globals'
 
   _arguments -s -w : $_hg_global_opts $_hg_pat_opts $_hg_dryrun_opts \
   '(--all -a :)'{-a,--all}'[revert all changes when no arguments given]' \
-  '(--rev -r)'{-r+,--rev}'[revision to revert to]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[revision to revert to]:revision:_hg_labels' \
   '(--no-backup -C)'{-C,--no-backup}'[do not save backup copies of files]' \
-  '(--date -d)'{-d+,--date}'[tipmost revision matching date]:date code:' \
+  '(--date -d)'{-d+,--date=}'[tipmost revision matching date]:date code:' \
   '*:file:->diff_files'
 
   if [[ $state == 'diff_files' ]]
@@ -844,13 +844,13 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_serve() {
   _arguments -s -w : $_hg_global_opts \
-  '(--accesslog -A)'{-A+,--accesslog}'[name of access log file]:log file:_files' \
-  '(--errorlog -E)'{-E+,--errorlog}'[name of error log file]:log file:_files' \
+  '(--accesslog -A)'{-A+,--accesslog=}'[name of access log file]:log file:_files' \
+  '(--errorlog -E)'{-E+,--errorlog=}'[name of error log file]:log file:_files' \
   '(--daemon -d)'{-d,--daemon}'[run server in background]' \
-  '(--port -p)'{-p+,--port}'[listen port]:listen port:' \
-  '(--address -a)'{-a+,--address}'[interface address]:interface address:' \
+  '(--port -p)'{-p+,--port=}'[listen port]:listen port:' \
+  '(--address -a)'{-a+,--address=}'[interface address]:interface address:' \
   '--prefix[prefix path to serve from]:directory:_files' \
-  '(--name -n)'{-n+,--name}'[name to show in web pages]:repository name:' \
+  '(--name -n)'{-n+,--name=}'[name to show in web pages]:repository name:' \
   '--web-conf[name of the hgweb config file]:webconf_file:_files' \
   '--pid-file[name of file to write process ID to]:pid_file:_files' \
   '--cmdserver[cmdserver mode]:mode:' \
@@ -863,7 +863,7 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_showconfig() {
   _arguments -s -w : $_hg_global_opts \
-  '(--untrusted -u)'{-u+,--untrusted}'[show untrusted configuration options]' \
+  '(--untrusted -u)'{-u,--untrusted}'[show untrusted configuration options]' \
   ':config item:_hg_config'
 }
 
@@ -893,10 +893,10 b' typeset -A _hg_cmd_globals'
 _hg_cmd_tag() {
   _arguments -s -w : $_hg_global_opts \
   '(--local -l)'{-l,--local}'[make the tag local]' \
-  '(--message -m)'{-m+,--message}'[message for tag commit log entry]:message:' \
-  '(--date -d)'{-d+,--date}'[record datecode as commit date]:date code:' \
-  '(--user -u)'{-u+,--user}'[record user as commiter]:user:' \
-  '(--rev -r)'{-r+,--rev}'[revision to tag]:revision:_hg_labels' \
+  '(--message -m)'{-m+,--message=}'[message for tag commit log entry]:message:' \
+  '(--date -d)'{-d+,--date=}'[record datecode as commit date]:date code:' \
+  '(--user -u)'{-u+,--user=}'[record user as commiter]:user:' \
+  '(--rev -r)'{-r+,--rev=}'[revision to tag]:revision:_hg_labels' \
   '(--force -f)'{-f,--force}'[force tag]' \
   '--remove[remove a tag]' \
   '(--edit -e)'{-e,--edit}'[edit commit message]' \
@@ -917,9 +917,9 b' typeset -A _hg_cmd_globals'
 _hg_cmd_update() {
   _arguments -s -w : $_hg_global_opts \
   '(--clean -C)'{-C,--clean}'[overwrite locally modified files]' \
-  '(--rev -r)'{-r+,--rev}'[revision]:revision:_hg_labels' \
+  '(--rev -r)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
   '(--check -c)'{-c,--check}'[update across branches if no uncommitted changes]' \
-  '(--date -d)'{-d+,--date}'[tipmost revision matching date]' \
+  '(--date -d)'{-d+,--date=}'[tipmost revision matching date]:' \
   ':revision:_hg_labels'
 }
 
@@ -928,7 +928,7 b' typeset -A _hg_cmd_globals'
 # HGK
 _hg_cmd_view() {
   _arguments -s -w : $_hg_global_opts \
-  '(--limit -l)'{-l+,--limit}'[limit number of changes displayed]:' \
+  '(--limit -l)'{-l+,--limit=}'[limit number of changes displayed]:' \
   ':revision range:_hg_labels'
 }
 
@@ -989,7 +989,7 b' typeset -A _hg_cmd_globals'
 
 _hg_cmd_qclone() {
   _arguments -s -w : $_hg_global_opts $_hg_remote_opts $_hg_clone_opts \
-  '(--patches -p)'{-p+,--patches}'[location of source patch repository]' \
+  '(--patches -p)'{-p+,--patches=}'[location of source patch repository]:' \
   ':source repository:_hg_remote' \
   ':destination:_hg_clone_dest'
 }
@@ -997,7 +997,7 b' typeset -A _hg_cmd_globals'
 _hg_cmd_qdelete() {
   _arguments -s -w : $_hg_global_opts \
   '(--keep -k)'{-k,--keep}'[keep patch file]' \
-  '*'{-r+,--rev}'[stop managing a revision]:applied patch:_hg_revrange' \
+  '*'{-r+,--rev=}'[stop managing a revision]:applied patch:_hg_revrange' \
   '*:unapplied patch:_hg_qdeletable'
 }
 
@@ -1046,7 +1046,7 b' typeset -A _hg_cmd_globals'
   '(--existing -e)'{-e,--existing}'[import file in patch dir]' \
   '(--name -n 2)'{-n+,--name}'[patch file name]:name:' \
   '(--force -f)'{-f,--force}'[overwrite existing files]' \
-  '*'{-r+,--rev}'[place existing revisions under mq control]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[place existing revisions under mq control]:revision:_hg_revrange' \
   '(--push -P)'{-P,--push}'[qpush after importing]' \
   '*:patch:_files'
 }
@@ -1125,8 +1125,8 b' typeset -A _hg_cmd_globals'
   '(--force -f)'{-f,--force}'[force removal, discard uncommitted changes, no backup]' \
   '(--no-backup -n)'{-n,--no-backup}'[no backups]' \
   '(--keep -k)'{-k,--keep}'[do not modify working copy during strip]' \
-  '(--bookmark -B)'{-B+,--bookmark}'[remove revs only reachable from given bookmark]:bookmark:_hg_bookmarks' \
-  '(--rev -r)'{-r+,--rev}'[revision]:revision:_hg_labels' \
+  '(--bookmark -B)'{-B+,--bookmark=}'[remove revs only reachable from given bookmark]:bookmark:_hg_bookmarks' \
+  '(--rev -r)'{-r+,--rev=}'[revision]:revision:_hg_labels' \
   ':revision:_hg_labels'
 }
 
@@ -1138,7 +1138,7 b' typeset -A _hg_cmd_globals'
   '(--outgoing -o)'{-o,--outgoing}'[send changes not found in the target repository]' \
   '(--bundle -b)'{-b,--bundle}'[send changes not in target as a binary bundle]' \
   '--bundlename[name of the bundle attachment file (default: bundle)]:' \
-  '*'{-r+,--rev}'[search in given revision range]:revision:_hg_revrange' \
+  '*'{-r+,--rev=}'[search in given revision range]:revision:_hg_revrange' \
   '--force[run even when remote repository is unrelated (with -b/--bundle)]' \
   '*--base[a base changeset to specify instead of a destination (with -b/--bundle)]:revision:_hg_labels' \
   '--intro[send an introduction email for a single patch]' \
@@ -1163,10 +1163,10 b' typeset -A _hg_cmd_globals'
 # Rebase
 _hg_cmd_rebase() {
   _arguments -s -w : $_hg_global_opts $_hg_commit_opts $_hg_mergetool_opts \
-  '*'{-r,--rev}'[rebase these revisions]:revision:_hg_revrange' \
-  '(--source -s)'{-s+,--source}'[rebase from the specified changeset]:revision:_hg_labels' \
-  '(--base -b)'{-b+,--base}'[rebase from the base of the specified changeset]:revision:_hg_labels' \
-  '(--dest -d)'{-d+,--dest}'[rebase onto the specified changeset]:revision:_hg_labels' \
+  '*'{-r+,--rev=}'[rebase these revisions]:revision:_hg_revrange' \
+  '(--source -s)'{-s+,--source=}'[rebase from the specified changeset]:revision:_hg_labels' \
+  '(--base -b)'{-b+,--base=}'[rebase from the base of the specified changeset]:revision:_hg_labels' \
+  '(--dest -d)'{-d+,--dest=}'[rebase onto the specified changeset]:revision:_hg_labels' \
   '--collapse[collapse the rebased changeset]' \
   '--keep[keep original changeset]' \
   '--keepbranches[keep original branch name]' \
@@ -1181,8 +1181,8 b' typeset -A _hg_cmd_globals'
   '(--addremove -A)'{-A,--addremove}'[mark new/missing files as added/removed before committing]' \
   '--close-branch[mark a branch as closed, hiding it from the branch list]' \
   '--amend[amend the parent of the working dir]' \
-  '(--date -d)'{-d+,--date}'[record the specified date as commit date]:date:' \
-  '(--user -u)'{-u+,--user}'[record the specified user as committer]:user:'
+  '(--date -d)'{-d+,--date=}'[record the specified date as commit date]:date:' \
+  '(--user -u)'{-u+,--user=}'[record the specified user as committer]:user:'
 }
 
 _hg_cmd_qrecord() {
@@ -1195,8 +1195,8 b' typeset -A _hg_cmd_globals'
 _arguments -s -w : $_hg_global_opts \
   '(--source-type -s)'{-s,--source-type}'[source repository type]' \
   '(--dest-type -d)'{-d,--dest-type}'[destination repository type]' \
-  '(--rev -r)'{-r+,--rev}'[import up to target revision]:revision:' \
-  '(--authormap -A)'{-A+,--authormap}'[remap usernames using this file]:file:_files' \
+  '(--rev -r)'{-r+,--rev=}'[import up to target revision]:revision:' \
+  '(--authormap -A)'{-A+,--authormap=}'[remap usernames using this file]:file:_files' \
   '--filemap[remap file names using contents of file]:file:_files' \
   '--splicemap[splice synthesized history into place]:file:_files' \
   '--branchmap[change branch names while converting]:file:_files' \

doc/hgmanpage.py

@@ -57,7 +57,6 b' try:'
     import roman
 except ImportError:
     from docutils.utils import roman
-import inspect
 
 FIELD_LIST_INDENT = 7
 DEFINITION_LIST_INDENT = 7
@@ -289,10 +288,10 b' class Translator(nodes.NodeVisitor):'
         text = node.astext()
         text = text.replace('\\','\\e')
         replace_pairs = [
-            (u'-', ur'\-'),
-            (u'\'', ur'\(aq'),
-            (u'´', ur'\''),
-            (u'`', ur'\(ga'),
+            (u'-', u'\\-'),
+            (u"'", u'\\(aq'),
+            (u'´', u"\\'"),
+            (u'`', u'\\(ga'),
             ]
         for (in_char, out_markup) in replace_pairs:
             text = text.replace(in_char, out_markup)

hgext/acl.py

@@ -204,11 +204,11 b' from mercurial import ('
 
 urlreq = util.urlreq
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def _getusers(ui, group):
 

hgext/blackbox.py

@@ -51,11 +51,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 lastui = None
 
 filehandles = {}

hgext/bugzilla.py

@@ -294,11 +294,11 b' from mercurial import ('
 urlparse = util.urlparse
 xmlrpclib = util.xmlrpclib
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 class bzaccess(object):
     '''Base class for access to Bugzilla.'''

hgext/censor.py

@@ -42,11 +42,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('censor',
     [('r', 'rev', '', _('censor file from specified revision'), _('REV')),

hgext/chgserver.py

@@ -63,11 +63,11 b' from mercurial import ('
     util,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 _log = commandserver.log
 

hgext/children.py

@@ -26,11 +26,11 b' templateopts = commands.templateopts'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('children',
     [('r', 'rev', '',

hgext/churn.py

@@ -26,11 +26,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def maketemplater(ui, repo, tmpl):
     return cmdutil.changeset_templater(ui, repo, False, None, tmpl, None, False)

hgext/clonebundles.py

@@ -169,7 +169,7 b' from mercurial import ('
     wireproto,
 )
 
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def capabilities(orig, repo, proto):
     caps = orig(repo, proto)

hgext/color.py

@@ -29,6 +29,15 b" ECMA-48 mode, the options are 'bold', 'i"
 Some may not be available for a given terminal type, and will be
 silently ignored.
 
+If the terminfo entry for your terminal is missing codes for an effect
+or has the wrong codes, you can add or override those codes in your
+configuration::
+
+  [color]
+  terminfo.dim = \E[2m
+
+where '\E' is substituted with an escape character.
+
 Labels
 ------
 
@@ -170,11 +179,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # start and stop parameters for effects
 _effects = {'none': 0, 'black': 30, 'red': 31, 'green': 32, 'yellow': 33,
@@ -196,9 +205,12 b' def _terminfosetup(ui, mode):'
     if mode not in ('auto', 'terminfo'):
         return
 
-    _terminfo_params.update((key[6:], (False, int(val)))
+    _terminfo_params.update((key[6:], (False, int(val), ''))
         for key, val in ui.configitems('color')
         if key.startswith('color.'))
+    _terminfo_params.update((key[9:], (True, '', val.replace('\\E', '\x1b')))
+        for key, val in ui.configitems('color')
+        if key.startswith('terminfo.'))
 
     try:
         curses.setupterm()
@@ -206,10 +218,10 b' def _terminfosetup(ui, mode):'
         _terminfo_params = {}
         return
 
-    for key, (b, e) in _terminfo_params.items():
+    for key, (b, e, c) in _terminfo_params.items():
         if not b:
             continue
-        if not curses.tigetstr(e):
+        if not c and not curses.tigetstr(e):
             # Most terminals don't support dim, invis, etc, so don't be
             # noisy and use ui.debug().
             ui.debug("no terminfo entry for %s\n" % e)
@@ -290,26 +302,26 b' def _modesetup(ui, coloropt):'
 
 try:
     import curses
-    # Mapping from effect name to terminfo attribute name or color number.
-    # This will also force-load the curses module.
-    _terminfo_params = {'none': (True, 'sgr0'),
-                        'standout': (True, 'smso'),
-                        'underline': (True, 'smul'),
-                        'reverse': (True, 'rev'),
-                        'inverse': (True, 'rev'),
-                        'blink': (True, 'blink'),
-                        'dim': (True, 'dim'),
-                        'bold': (True, 'bold'),
-                        'invisible': (True, 'invis'),
-                        'italic': (True, 'sitm'),
-                        'black': (False, curses.COLOR_BLACK),
-                        'red': (False, curses.COLOR_RED),
-                        'green': (False, curses.COLOR_GREEN),
-                        'yellow': (False, curses.COLOR_YELLOW),
-                        'blue': (False, curses.COLOR_BLUE),
-                        'magenta': (False, curses.COLOR_MAGENTA),
-                        'cyan': (False, curses.COLOR_CYAN),
-                        'white': (False, curses.COLOR_WHITE)}
+    # Mapping from effect name to terminfo attribute name (or raw code) or
+    # color number.  This will also force-load the curses module.
+    _terminfo_params = {'none': (True, 'sgr0', ''),
+                        'standout': (True, 'smso', ''),
+                        'underline': (True, 'smul', ''),
+                        'reverse': (True, 'rev', ''),
+                        'inverse': (True, 'rev', ''),
+                        'blink': (True, 'blink', ''),
+                        'dim': (True, 'dim', ''),
+                        'bold': (True, 'bold', ''),
+                        'invisible': (True, 'invis', ''),
+                        'italic': (True, 'sitm', ''),
+                        'black': (False, curses.COLOR_BLACK, ''),
+                        'red': (False, curses.COLOR_RED, ''),
+                        'green': (False, curses.COLOR_GREEN, ''),
+                        'yellow': (False, curses.COLOR_YELLOW, ''),
+                        'blue': (False, curses.COLOR_BLUE, ''),
+                        'magenta': (False, curses.COLOR_MAGENTA, ''),
+                        'cyan': (False, curses.COLOR_CYAN, ''),
+                        'white': (False, curses.COLOR_WHITE, '')}
 except ImportError:
     _terminfo_params = {}
 
@@ -375,9 +387,15 b' def _effect_str(effect):'
     if effect.endswith('_background'):
         bg = True
         effect = effect[:-11]
-    attr, val = _terminfo_params[effect]
+    try:
+        attr, val, termcode = _terminfo_params[effect]
+    except KeyError:
+        return ''
     if attr:
-        return curses.tigetstr(val)
+        if termcode:
+            return termcode
+        else:
+            return curses.tigetstr(val)
     elif bg:
         return curses.tparm(curses.tigetstr('setab'), val)
     else:
@@ -412,7 +430,7 b' def valideffect(effect):'
 
 def configstyles(ui):
     for status, cfgeffects in ui.configitems('color'):
-        if '.' not in status or status.startswith('color.'):
+        if '.' not in status or status.startswith(('color.', 'terminfo.')):
             continue
         cfgeffects = ui.configlist('color', status)
         if cfgeffects:
@@ -524,10 +542,16 b' def debugcolor(ui, repo, **opts):'
     _styles = {}
     for effect in _effects.keys():
         _styles[effect] = effect
+    if _terminfo_params:
+        for k, v in ui.configitems('color'):
+            if k.startswith('color.'):
+                _styles[k] = k[6:]
+            elif k.startswith('terminfo.'):
+                _styles[k] = k[9:]
     ui.write(('color mode: %s\n') % ui._colormode)
     ui.write(_('available colors:\n'))
-    for label, colors in _styles.items():
-        ui.write(('%s\n') % colors, label=label)
+    for colorname, label in _styles.items():
+        ui.write(('%s\n') % colorname, label=label)
 
 if os.name != 'nt':
     w32effects = None
@@ -558,8 +582,8 b' else:'
                     ('srWindow', _SMALL_RECT),
                     ('dwMaximumWindowSize', _COORD)]
 
-    _STD_OUTPUT_HANDLE = 0xfffffff5L # (DWORD)-11
-    _STD_ERROR_HANDLE = 0xfffffff4L  # (DWORD)-12
+    _STD_OUTPUT_HANDLE = 0xfffffff5 # (DWORD)-11
+    _STD_ERROR_HANDLE = 0xfffffff4  # (DWORD)-12
 
     _FOREGROUND_BLUE = 0x0001
     _FOREGROUND_GREEN = 0x0002

hgext/convert/__init__.py

@@ -23,11 +23,11 b' from . import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # Commands definition was moved elsewhere to ease demandload job.
 

hgext/convert/subversion.py

@@ -531,8 +531,8 b' class svn_source(converter_source):'
     def checkrevformat(self, revstr, mapname='splicemap'):
         """ fails if revision format does not match the correct format"""
         if not re.match(r'svn:[0-9a-f]{8,8}-[0-9a-f]{4,4}-'
-                              '[0-9a-f]{4,4}-[0-9a-f]{4,4}-[0-9a-f]'
-                              '{12,12}(.*)\@[0-9]+$',revstr):
+                              r'[0-9a-f]{4,4}-[0-9a-f]{4,4}-[0-9a-f]'
+                              r'{12,12}(.*)\@[0-9]+$',revstr):
             raise error.Abort(_('%s entry %s is not a valid revision'
                                ' identifier') % (mapname, revstr))
 

hgext/eol.py

@@ -104,11 +104,11 b' from mercurial import ('
     util,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # Matches a lone LF, i.e., one that is not part of CRLF.
 singlelf = re.compile('(^|[^\r])\n')
@@ -175,25 +175,27 b' class eolfile(object):'
 
         include = []
         exclude = []
+        self.patterns = []
         for pattern, style in self.cfg.items('patterns'):
             key = style.upper()
             if key == 'BIN':
                 exclude.append(pattern)
             else:
                 include.append(pattern)
+            m = match.match(root, '', [pattern])
+            self.patterns.append((pattern, key, m))
         # This will match the files for which we need to care
         # about inconsistent newlines.
         self.match = match.match(root, '', [], include, exclude)
 
     def copytoui(self, ui):
-        for pattern, style in self.cfg.items('patterns'):
-            key = style.upper()
+        for pattern, key, m in self.patterns:
             try:
                 ui.setconfig('decode', pattern, self._decode[key], 'eol')
                 ui.setconfig('encode', pattern, self._encode[key], 'eol')
             except KeyError:
                 ui.warn(_("ignoring unknown EOL style '%s' from %s\n")
-                        % (style, self.cfg.source('patterns', pattern)))
+                        % (key, self.cfg.source('patterns', pattern)))
         # eol.only-consistent can be specified in ~/.hgrc or .hgeol
         for k, v in self.cfg.items('eol'):
             ui.setconfig('eol', k, v, 'eol')
@@ -203,10 +205,10 b' class eolfile(object):'
         for f in (files or ctx.files()):
             if f not in ctx:
                 continue
-            for pattern, style in self.cfg.items('patterns'):
-                if not match.match(repo.root, '', [pattern])(f):
+            for pattern, key, m in self.patterns:
+                if not m(f):
                     continue
-                target = self._encode[style.upper()]
+                target = self._encode[key]
                 data = ctx[f].data()
                 if (target == "to-lf" and "\r\n" in data
                     or target == "to-crlf" and singlelf.search(data)):
@@ -305,15 +307,20 b' def reposetup(ui, repo):'
             return eol.match
 
         def _hgcleardirstate(self):
-            self._eolfile = self.loadeol([None, 'tip'])
-            if not self._eolfile:
-                self._eolfile = util.never
+            self._eolmatch = self.loadeol([None, 'tip'])
+            if not self._eolmatch:
+                self._eolmatch = util.never
                 return
 
+            oldeol = None
             try:
                 cachemtime = os.path.getmtime(self.join("eol.cache"))
             except OSError:
                 cachemtime = 0
+            else:
+                olddata = self.vfs.read("eol.cache")
+                if olddata:
+                    oldeol = eolfile(self.ui, self.root, olddata)
 
             try:
                 eolmtime = os.path.getmtime(self.wjoin(".hgeol"))
@@ -322,18 +329,37 b' def reposetup(ui, repo):'
 
             if eolmtime > cachemtime:
                 self.ui.debug("eol: detected change in .hgeol\n")
+
+                hgeoldata = self.wvfs.read('.hgeol')
+                neweol = eolfile(self.ui, self.root, hgeoldata)
+
                 wlock = None
                 try:
                     wlock = self.wlock()
                     for f in self.dirstate:
-                        if self.dirstate[f] == 'n':
-                            # all normal files need to be looked at
-                            # again since the new .hgeol file might no
-                            # longer match a file it matched before
-                            self.dirstate.normallookup(f)
-                    # Create or touch the cache to update mtime
-                    self.vfs("eol.cache", "w").close()
-                    wlock.release()
+                        if self.dirstate[f] != 'n':
+                            continue
+                        if oldeol is not None:
+                            if not oldeol.match(f) and not neweol.match(f):
+                                continue
+                            oldkey = None
+                            for pattern, key, m in oldeol.patterns:
+                                if m(f):
+                                    oldkey = key
+                                    break
+                            newkey = None
+                            for pattern, key, m in neweol.patterns:
+                                if m(f):
+                                    newkey = key
+                                    break
+                            if oldkey == newkey:
+                                continue
+                        # all normal files need to be looked at again since
+                        # the new .hgeol file specify a different filter
+                        self.dirstate.normallookup(f)
+                    # Write the cache to update mtime and cache .hgeol
+                    with self.vfs("eol.cache", "w") as f:
+                        f.write(hgeoldata)
                 except error.LockUnavailable:
                     # If we cannot lock the repository and clear the
                     # dirstate, then a commit might not see all files
@@ -341,10 +367,13 b' def reposetup(ui, repo):'
                     # repository, then we can also not make a commit,
                     # so ignore the error.
                     pass
+                finally:
+                    if wlock is not None:
+                        wlock.release()
 
         def commitctx(self, ctx, haserror=False):
             for f in sorted(ctx.added() + ctx.modified()):
-                if not self._eolfile(f):
+                if not self._eolmatch(f):
                     continue
                 fctx = ctx[f]
                 if fctx is None:

hgext/extdiff.py

@@ -84,11 +84,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def snapshot(ui, repo, files, node, tmproot, listsubrepos):
     '''snapshot files as of some revision
@@ -324,6 +324,34 b' def extdiff(ui, repo, *pats, **opts):'
     cmdline = ' '.join(map(util.shellquote, [program] + option))
     return dodiff(ui, repo, cmdline, pats, opts)
 
+class savedcmd(object):
+    """use external program to diff repository (or selected files)
+
+    Show differences between revisions for the specified files, using
+    the following program::
+
+        %(path)s
+
+    When two revision arguments are given, then changes are shown
+    between those revisions. If only one revision is specified then
+    that revision is compared to the working directory, and, when no
+    revisions are specified, the working directory files are compared
+    to its parent.
+    """
+
+    def __init__(self, path, cmdline):
+        # We can't pass non-ASCII through docstrings (and path is
+        # in an unknown encoding anyway)
+        docpath = path.encode("string-escape")
+        self.__doc__ = self.__doc__ % {'path': util.uirepr(docpath)}
+        self._cmdline = cmdline
+
+    def __call__(self, ui, repo, *pats, **opts):
+        options = ' '.join(map(util.shellquote, opts['option']))
+        if options:
+            options = ' ' + options
+        return dodiff(ui, repo, self._cmdline + options, pats, opts)
+
 def uisetup(ui):
     for cmd, path in ui.configitems('extdiff'):
         path = util.expandpath(path)
@@ -357,28 +385,8 b' def uisetup(ui):'
                    ui.config('merge-tools', cmd+'.diffargs')
             if args:
                 cmdline += ' ' + args
-        def save(cmdline):
-            '''use closure to save diff command to use'''
-            def mydiff(ui, repo, *pats, **opts):
-                options = ' '.join(map(util.shellquote, opts['option']))
-                if options:
-                    options = ' ' + options
-                return dodiff(ui, repo, cmdline + options, pats, opts)
-            # We can't pass non-ASCII through docstrings (and path is
-            # in an unknown encoding anyway)
-            docpath = path.encode("string-escape")
-            mydiff.__doc__ = '''\
-use %(path)s to diff repository (or selected files)
+        command(cmd, extdiffopts[:], _('hg %s [OPTION]... [FILE]...') % cmd,
+                inferrepo=True)(savedcmd(path, cmdline))
 
-    Show differences between revisions for the specified files, using
-    the %(path)s program.
-
-    When two revision arguments are given, then changes are shown
-    between those revisions. If only one revision is specified then
-    that revision is compared to the working directory, and, when no
-    revisions are specified, the working directory files are compared
-    to its parent.\
-''' % {'path': util.uirepr(docpath)}
-            return mydiff
-        command(cmd, extdiffopts[:], _('hg %s [OPTION]... [FILE]...') % cmd,
-                inferrepo=True)(save(cmdline))
+# tell hggettext to extract docstrings from these functions:
+i18nfunctions = [savedcmd]

hgext/fetch.py

@@ -26,11 +26,11 b' from mercurial import ('
 release = lock.release
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('fetch',
     [('r', 'rev', [],

hgext/fsmonitor/__init__.py

@@ -113,11 +113,11 b' from . import ('
     watchmanclient,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # This extension is incompatible with the following blacklisted extensions
 # and will disable itself when encountering one of these:

hgext/gpg.py

@@ -23,11 +23,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 class gpg(object):
     def __init__(self, path, key=None):

hgext/graphlog.py

@@ -25,11 +25,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('glog',
     [('f', 'follow', None,

hgext/hgk.py

@@ -54,11 +54,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('debug-diff-tree',
     [('p', 'patch', None, _('generate patch')),

hgext/highlight/__init__.py

@@ -41,11 +41,11 b' from mercurial import ('
     fileset,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def pygmentize(web, field, fctx, tmpl):
     style = web.config('web', 'pygments_style', 'colorful')

hgext/histedit.py

@@ -201,23 +201,11 b' release = lock.release'
 cmdtable = {}
 command = cmdutil.command(cmdtable)
 
-class _constraints(object):
-    # aborts if there are multiple rules for one node
-    noduplicates = 'noduplicates'
-    # abort if the node does belong to edited stack
-    forceother = 'forceother'
-    # abort if the node doesn't belong to edited stack
-    noother = 'noother'
-
-    @classmethod
-    def known(cls):
-        return set([v for k, v in cls.__dict__.items() if k[0] != '_'])
-
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 actiontable = {}
 primaryactions = set()
@@ -403,7 +391,7 b' class histeditaction(object):'
             raise error.ParseError("invalid changeset %s" % rulehash)
         return cls(state, rev)
 
-    def verify(self, prev):
+    def verify(self, prev, expected, seen):
         """ Verifies semantic correctness of the rule"""
         repo = self.repo
         ha = node.hex(self.node)
@@ -412,6 +400,19 b' class histeditaction(object):'
         except error.RepoError:
             raise error.ParseError(_('unknown changeset %s listed')
                               % ha[:12])
+        if self.node is not None:
+            self._verifynodeconstraints(prev, expected, seen)
+
+    def _verifynodeconstraints(self, prev, expected, seen):
+        # by default command need a node in the edited list
+        if self.node not in expected:
+            raise error.ParseError(_('%s "%s" changeset was not a candidate')
+                                   % (self.verb, node.short(self.node)),
+                                   hint=_('only use listed changesets'))
+        # and only one command per node
+        if self.node in seen:
+            raise error.ParseError(_('duplicated command for changeset %s') %
+                                   node.short(self.node))
 
     def torule(self):
         """build a histedit rule line for an action
@@ -434,19 +435,6 b' class histeditaction(object):'
         """
         return "%s\n%s" % (self.verb, node.hex(self.node))
 
-    def constraints(self):
-        """Return a set of constrains that this action should be verified for
-        """
-        return set([_constraints.noduplicates, _constraints.noother])
-
-    def nodetoverify(self):
-        """Returns a node associated with the action that will be used for
-        verification purposes.
-
-        If the action doesn't correspond to node it should return None
-        """
-        return self.node
-
     def run(self):
         """Runs the action. The default behavior is simply apply the action's
         rulectx onto the current parentctx."""
@@ -573,18 +561,7 b' def collapse(repo, first, last, commitop'
     copied = copies.pathcopies(base, last)
 
     # prune files which were reverted by the updates
-    def samefile(f):
-        if f in last.manifest():
-            a = last.filectx(f)
-            if f in base.manifest():
-                b = base.filectx(f)
-                return (a.data() == b.data()
-                        and a.flags() == b.flags())
-            else:
-                return False
-        else:
-            return f not in base.manifest()
-    files = [f for f in files if not samefile(f)]
+    files = [f for f in files if not cmdutil.samefile(f, last, base)]
     # commit version of these files as defined by head
     headmf = last.manifest()
     def filectxfn(repo, ctx, path):
@@ -683,9 +660,9 b' class edit(histeditaction):'
 @action(['fold', 'f'],
         _('use commit, but combine it with the one above'))
 class fold(histeditaction):
-    def verify(self, prev):
+    def verify(self, prev, expected, seen):
         """ Verifies semantic correctness of the fold rule"""
-        super(fold, self).verify(prev)
+        super(fold, self).verify(prev, expected, seen)
         repo = self.repo
         if not prev:
             c = repo[self.node].parents()[0]
@@ -795,8 +772,6 b' class fold(histeditaction):'
         return repo[n], replacements
 
 class base(histeditaction):
-    def constraints(self):
-        return set([_constraints.forceother])
 
     def run(self):
         if self.repo['.'].node() != self.node:
@@ -811,6 +786,14 b' class base(histeditaction):'
         basectx = self.repo['.']
         return basectx, []
 
+    def _verifynodeconstraints(self, prev, expected, seen):
+        # base can only be use with a node not in the edited set
+        if self.node in expected:
+            msg = _('%s "%s" changeset was an edited list candidate')
+            raise error.ParseError(
+                msg % (self.verb, node.short(self.node)),
+                hint=_('base must only use unlisted changesets'))
+
 @action(['_multifold'],
         _(
     """fold subclass used for when multiple folds happen in a row
@@ -871,7 +854,7 b' def findoutgoing(ui, repo, remote=None, '
     roots = list(repo.revs("roots(%ln)", outgoing.missing))
     if 1 < len(roots):
         msg = _('there are ambiguous outgoing revisions')
-        hint = _('see "hg help histedit" for more detail')
+        hint = _("see 'hg help histedit' for more detail")
         raise error.Abort(msg, hint=hint)
     return repo.lookup(roots[0])
 
@@ -1210,8 +1193,8 b' def _edithisteditplan(ui, repo, state, r'
     else:
         rules = _readfile(rules)
     actions = parserules(rules, state)
-    ctxs = [repo[act.nodetoverify()] \
-            for act in state.actions if act.nodetoverify()]
+    ctxs = [repo[act.node] \
+            for act in state.actions if act.node]
     warnverifyactions(ui, repo, actions, state, ctxs)
     state.actions = actions
     state.write()
@@ -1307,7 +1290,7 b' def between(repo, old, new, keep):'
         root = ctxs[0] # list is already sorted by repo.set
         if not root.mutable():
             raise error.Abort(_('cannot edit public changeset: %s') % root,
-                             hint=_('see "hg help phases" for details'))
+                             hint=_("see 'hg help phases' for details"))
     return [c.node() for c in ctxs]
 
 def ruleeditor(repo, ui, actions, editcomment=""):
@@ -1396,36 +1379,14 b' def verifyactions(actions, state, ctxs):'
     Will abort if there are to many or too few rules, a malformed rule,
     or a rule on a changeset outside of the user-given range.
     """
-    expected = set(c.hex() for c in ctxs)
+    expected = set(c.node() for c in ctxs)
     seen = set()
     prev = None
     for action in actions:
-        action.verify(prev)
+        action.verify(prev, expected, seen)
         prev = action
-        constraints = action.constraints()
-        for constraint in constraints:
-            if constraint not in _constraints.known():
-                raise error.ParseError(_('unknown constraint "%s"') %
-                        constraint)
-
-        nodetoverify = action.nodetoverify()
-        if nodetoverify is not None:
-            ha = node.hex(nodetoverify)
-            if _constraints.noother in constraints and ha not in expected:
-                raise error.ParseError(
-                    _('%s "%s" changeset was not a candidate')
-                     % (action.verb, ha[:12]),
-                    hint=_('only use listed changesets'))
-            if _constraints.forceother in constraints and ha in expected:
-                raise error.ParseError(
-                    _('%s "%s" changeset was not an edited list candidate')
-                     % (action.verb, ha[:12]),
-                    hint=_('only use listed changesets'))
-            if _constraints.noduplicates in constraints and ha in seen:
-                raise error.ParseError(_(
-                        'duplicated command for changeset %s') %
-                        ha[:12])
-            seen.add(ha)
+        if action.node is not None:
+            seen.add(action.node)
     missing = sorted(expected - seen)  # sort to stabilize output
 
     if state.repo.ui.configbool('histedit', 'dropmissing'):
@@ -1433,15 +1394,16 b' def verifyactions(actions, state, ctxs):'
             raise error.ParseError(_('no rules provided'),
                     hint=_('use strip extension to remove commits'))
 
-        drops = [drop(state, node.bin(n)) for n in missing]
+        drops = [drop(state, n) for n in missing]
         # put the in the beginning so they execute immediately and
         # don't show in the edit-plan in the future
         actions[:0] = drops
     elif missing:
         raise error.ParseError(_('missing rules for changeset %s') %
-                missing[0][:12],
+                node.short(missing[0]),
                 hint=_('use "drop %s" to discard, see also: '
-                       '"hg help -e histedit.config"') % missing[0][:12])
+                       "'hg help -e histedit.config'")
+                       % node.short(missing[0]))
 
 def adjustreplacementsfrommarkers(repo, oldreplacements):
     """Adjust replacements from obsolescense markers
@@ -1608,10 +1570,9 b' def stripwrapper(orig, ui, repo, nodelis'
     if os.path.exists(os.path.join(repo.path, 'histedit-state')):
         state = histeditstate(repo)
         state.read()
-        histedit_nodes = set([action.nodetoverify() for action
-                             in state.actions if action.nodetoverify()])
-        strip_nodes = set([repo[n].node() for n in nodelist])
-        common_nodes = histedit_nodes & strip_nodes
+        histedit_nodes = set([action.node for action
+                             in state.actions if action.node])
+        common_nodes = histedit_nodes & set(nodelist)
         if common_nodes:
             raise error.Abort(_("histedit in progress, can't strip %s")
                              % ', '.join(node.short(x) for x in common_nodes))

hgext/journal.py

@@ -24,7 +24,6 b' from mercurial import ('
     bookmarks,
     cmdutil,
     commands,
-    dirstate,
     dispatch,
     error,
     extensions,
@@ -40,11 +39,11 b' from . import share'
 cmdtable = {}
 command = cmdutil.command(cmdtable)
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # storage format version; increment when the format changes
 storageversion = 0
@@ -63,8 +62,6 b' def extsetup(ui):'
     extensions.wrapfunction(dispatch, 'runcommand', runcommand)
     extensions.wrapfunction(bookmarks.bmstore, '_write', recordbookmarks)
     extensions.wrapfunction(
-        dirstate.dirstate, '_writedirstate', recorddirstateparents)
-    extensions.wrapfunction(
         localrepo.localrepository.dirstate, 'func', wrapdirstate)
     extensions.wrapfunction(hg, 'postshare', wrappostshare)
     extensions.wrapfunction(hg, 'copystore', unsharejournal)
@@ -84,34 +81,19 b' def wrapdirstate(orig, repo):'
     dirstate = orig(repo)
     if util.safehasattr(repo, 'journal'):
         dirstate.journalstorage = repo.journal
+        dirstate.addparentchangecallback('journal', recorddirstateparents)
     return dirstate
 
-def recorddirstateparents(orig, dirstate, dirstatefp):
+def recorddirstateparents(dirstate, old, new):
     """Records all dirstate parent changes in the journal."""
+    old = list(old)
+    new = list(new)
     if util.safehasattr(dirstate, 'journalstorage'):
-        old = [node.nullid, node.nullid]
-        nodesize = len(node.nullid)
-        try:
-            # The only source for the old state is in the dirstate file still
-            # on disk; the in-memory dirstate object only contains the new
-            # state. dirstate._opendirstatefile() switches beteen .hg/dirstate
-            # and .hg/dirstate.pending depending on the transaction state.
-            with dirstate._opendirstatefile() as fp:
-                state = fp.read(2 * nodesize)
-            if len(state) == 2 * nodesize:
-                old = [state[:nodesize], state[nodesize:]]
-        except IOError:
-            pass
-
-        new = dirstate.parents()
-        if old != new:
-            # only record two hashes if there was a merge
-            oldhashes = old[:1] if old[1] == node.nullid else old
-            newhashes = new[:1] if new[1] == node.nullid else new
-            dirstate.journalstorage.record(
-                wdirparenttype, '.', oldhashes, newhashes)
-
-    return orig(dirstate, dirstatefp)
+        # only record two hashes if there was a merge
+        oldhashes = old[:1] if old[1] == node.nullid else old
+        newhashes = new[:1] if new[1] == node.nullid else new
+        dirstate.journalstorage.record(
+            wdirparenttype, '.', oldhashes, newhashes)
 
 # hooks to record bookmark changes (both local and remote)
 def recordbookmarks(orig, store, fp):
@@ -165,9 +147,10 b' def _mergeentriesiter(*iterables, **kwar'
 
 def wrappostshare(orig, sourcerepo, destrepo, **kwargs):
     """Mark this shared working copy as sharing journal information"""
-    orig(sourcerepo, destrepo, **kwargs)
-    with destrepo.vfs('shared', 'a') as fp:
-        fp.write('journal\n')
+    with destrepo.wlock():
+        orig(sourcerepo, destrepo, **kwargs)
+        with destrepo.vfs('shared', 'a') as fp:
+            fp.write('journal\n')
 
 def unsharejournal(orig, ui, repo, repopath):
     """Copy shared journal entries into this repo when unsharing"""
@@ -179,12 +162,12 b' def unsharejournal(orig, ui, repo, repop'
             # there is a shared repository and there are shared journal entries
             # to copy. move shared date over from source to destination but
             # move the local file first
-            if repo.vfs.exists('journal'):
-                journalpath = repo.join('journal')
+            if repo.vfs.exists('namejournal'):
+                journalpath = repo.join('namejournal')
                 util.rename(journalpath, journalpath + '.bak')
             storage = repo.journal
             local = storage._open(
-                repo.vfs, filename='journal.bak', _newestfirst=False)
+                repo.vfs, filename='namejournal.bak', _newestfirst=False)
             shared = (
                 e for e in storage._open(sharedrepo.vfs, _newestfirst=False)
                 if sharednamespaces.get(e.namespace) in sharedfeatures)
@@ -194,8 +177,8 b' def unsharejournal(orig, ui, repo, repop'
     return orig(ui, repo, repopath)
 
 class journalentry(collections.namedtuple(
-        'journalentry',
-        'timestamp user command namespace name oldhashes newhashes')):
+        u'journalentry',
+        u'timestamp user command namespace name oldhashes newhashes')):
     """Individual journal entry
 
     * timestamp: a mercurial (time, timezone) tuple
@@ -284,19 +267,31 b' class journalstorage(object):'
         # with a non-local repo (cloning for example).
         cls._currentcommand = fullargs
 
+    def _currentlock(self, lockref):
+        """Returns the lock if it's held, or None if it's not.
+
+        (This is copied from the localrepo class)
+        """
+        if lockref is None:
+            return None
+        l = lockref()
+        if l is None or not l.held:
+            return None
+        return l
+
     def jlock(self, vfs):
         """Create a lock for the journal file"""
-        if self._lockref and self._lockref():
+        if self._currentlock(self._lockref) is not None:
             raise error.Abort(_('journal lock does not support nesting'))
         desc = _('journal of %s') % vfs.base
         try:
-            l = lock.lock(vfs, 'journal.lock', 0, desc=desc)
+            l = lock.lock(vfs, 'namejournal.lock', 0, desc=desc)
         except error.LockHeld as inst:
             self.ui.warn(
                 _("waiting for lock on %s held by %r\n") % (desc, inst.locker))
             # default to 600 seconds timeout
             l = lock.lock(
-                vfs, 'journal.lock',
+                vfs, 'namejournal.lock',
                 int(self.ui.config("ui", "timeout", "600")), desc=desc)
             self.ui.warn(_("got lock after %s seconds\n") % l.delay)
         self._lockref = weakref.ref(l)
@@ -336,7 +331,7 b' class journalstorage(object):'
         with self.jlock(vfs):
             version = None
             # open file in amend mode to ensure it is created if missing
-            with vfs('journal', mode='a+b', atomictemp=True) as f:
+            with vfs('namejournal', mode='a+b', atomictemp=True) as f:
                 f.seek(0, os.SEEK_SET)
                 # Read just enough bytes to get a version number (up to 2
                 # digits plus separator)
@@ -394,7 +389,7 b' class journalstorage(object):'
             if sharednamespaces.get(e.namespace) in self.sharedfeatures)
         return _mergeentriesiter(local, shared)
 
-    def _open(self, vfs, filename='journal', _newestfirst=True):
+    def _open(self, vfs, filename='namejournal', _newestfirst=True):
         if not vfs.exists(filename):
             return
 
@@ -475,8 +470,10 b' def journal(ui, repo, *args, **opts):'
     for count, entry in enumerate(repo.journal.filtered(name=name)):
         if count == limit:
             break
-        newhashesstr = ','.join([node.short(hash) for hash in entry.newhashes])
-        oldhashesstr = ','.join([node.short(hash) for hash in entry.oldhashes])
+        newhashesstr = fm.formatlist(map(fm.hexfunc, entry.newhashes),
+                                     name='node', sep=',')
+        oldhashesstr = fm.formatlist(map(fm.hexfunc, entry.oldhashes),
+                                     name='node', sep=',')
 
         fm.startitem()
         fm.condwrite(ui.verbose, 'oldhashes', '%s -> ', oldhashesstr)
@@ -486,7 +483,7 b' def journal(ui, repo, *args, **opts):'
             opts.get('all') or name.startswith('re:'),
             'name', '  %-8s', entry.name)
 
-        timestring = util.datestr(entry.timestamp, '%Y-%m-%d %H:%M %1%2')
+        timestring = fm.formatdate(entry.timestamp, '%Y-%m-%d %H:%M %1%2')
         fm.condwrite(ui.verbose, 'date', ' %s', timestring)
         fm.write('command', '  %s\n', entry.command)
 

hgext/keyword.py

@@ -24,7 +24,7 b''
 # Files to act upon/ignore are specified in the [keyword] section.
 # Customized keyword template mappings in the [keywordmaps] section.
 #
-# Run "hg help keyword" and "hg kwdemo" to get info on configuration.
+# Run 'hg help keyword' and 'hg kwdemo' to get info on configuration.
 
 '''expand keywords in tracked files
 
@@ -112,11 +112,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # hg commands that do not act on keywords
 nokwcommands = ('add addremove annotate bundle export grep incoming init log'

hgext/largefiles/__init__.py

@@ -119,11 +119,11 b' from . import ('
     uisetup as uisetupmod,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 reposetup = reposetup.reposetup
 

hgext/largefiles/basestore.py

@@ -91,15 +91,13 b' class basestore(object):'
         storefilename = lfutil.storepath(self.repo, hash)
 
         tmpname = storefilename + '.tmp'
-        tmpfile = util.atomictempfile(tmpname,
-                                      createmode=self.repo.store.createmode)
-
-        try:
-            gothash = self._getfile(tmpfile, filename, hash)
-        except StoreError as err:
-            self.ui.warn(err.longmessage())
-            gothash = ""
-        tmpfile.close()
+        with util.atomictempfile(tmpname,
+                createmode=self.repo.store.createmode) as tmpfile:
+            try:
+                gothash = self._getfile(tmpfile, filename, hash)
+            except StoreError as err:
+                self.ui.warn(err.longmessage())
+                gothash = ""
 
         if gothash != hash:
             if gothash != "":

hgext/largefiles/lfcommands.py

@@ -515,9 +515,13 b' def updatelfiles(ui, repo, filelist=None'
             rellfile = lfile
             relstandin = lfutil.standin(lfile)
             if wvfs.exists(relstandin):
-                mode = wvfs.stat(relstandin).st_mode
-                if mode != wvfs.stat(rellfile).st_mode:
-                    wvfs.chmod(rellfile, mode)
+                standinexec = wvfs.stat(relstandin).st_mode & 0o100
+                st = wvfs.stat(rellfile).st_mode
+                if standinexec != st & 0o100:
+                    st &= ~0o111
+                    if standinexec:
+                        st |= (st >> 2) & 0o111 & ~util.umask
+                    wvfs.chmod(rellfile, st)
                     update1 = 1
 
             updated += update1

hgext/largefiles/lfutil.py

@@ -54,10 +54,10 b' def link(src, dest):'
         util.oslink(src, dest)
     except OSError:
         # if hardlinks fail, fallback on atomic copy
-        dst = util.atomictempfile(dest)
-        for chunk in util.filechunkiter(open(src, 'rb')):
-            dst.write(chunk)
-        dst.close()
+        with open(src, 'rb') as srcf:
+            with util.atomictempfile(dest) as dstf:
+                for chunk in util.filechunkiter(srcf):
+                    dstf.write(chunk)
         os.chmod(dest, os.stat(src).st_mode)
 
 def usercachepath(ui, hash):
@@ -231,7 +231,8 b' def copyfromcache(repo, hash, filename):'
     # don't use atomic writes in the working copy.
     with open(path, 'rb') as srcfd:
         with wvfs(filename, 'wb') as destfd:
-            gothash = copyandhash(srcfd, destfd)
+            gothash = copyandhash(
+                util.filechunkiter(srcfd), destfd)
     if gothash != hash:
         repo.ui.warn(_('%s: data corruption in %s with hash %s\n')
                      % (filename, path, gothash))
@@ -264,11 +265,11 b' def copytostoreabsolute(repo, file, hash'
         link(usercachepath(repo.ui, hash), storepath(repo, hash))
     else:
         util.makedirs(os.path.dirname(storepath(repo, hash)))
-        dst = util.atomictempfile(storepath(repo, hash),
-                                  createmode=repo.store.createmode)
-        for chunk in util.filechunkiter(open(file, 'rb')):
-            dst.write(chunk)
-        dst.close()
+        with open(file, 'rb') as srcf:
+            with util.atomictempfile(storepath(repo, hash),
+                                     createmode=repo.store.createmode) as dstf:
+                for chunk in util.filechunkiter(srcf):
+                    dstf.write(chunk)
         linktousercache(repo, hash)
 
 def linktousercache(repo, hash):
@@ -370,10 +371,9 b' def hashfile(file):'
     if not os.path.exists(file):
         return ''
     hasher = hashlib.sha1('')
-    fd = open(file, 'rb')
-    for data in util.filechunkiter(fd, 128 * 1024):
-        hasher.update(data)
-    fd.close()
+    with open(file, 'rb') as fd:
+        for data in util.filechunkiter(fd):
+            hasher.update(data)
     return hasher.hexdigest()
 
 def getexecutable(filename):

hgext/largefiles/localstore.py

@@ -10,6 +10,7 b''
 from __future__ import absolute_import
 
 from mercurial.i18n import _
+from mercurial import util
 
 from . import (
     basestore,
@@ -42,7 +43,8 b' class localstore(basestore.basestore):'
             raise basestore.StoreError(filename, hash, self.url,
                 _("can't get file locally"))
         with open(path, 'rb') as fd:
-            return lfutil.copyandhash(fd, tmpfile)
+            return lfutil.copyandhash(
+                util.filechunkiter(fd), tmpfile)
 
     def _verifyfiles(self, contents, filestocheck):
         failed = False

hgext/largefiles/overrides.py

@@ -883,11 +883,8 b' def hgclone(orig, ui, opts, *args, **kwa'
 
         # If largefiles is required for this repo, permanently enable it locally
         if 'largefiles' in repo.requirements:
-            fp = repo.vfs('hgrc', 'a', text=True)
-            try:
+            with repo.vfs('hgrc', 'a', text=True) as fp:
                 fp.write('\n[extensions]\nlargefiles=\n')
-            finally:
-                fp.close()
 
         # Caching is implicitly limited to 'rev' option, since the dest repo was
         # truncated at that point.  The user may expect a download count with
@@ -1339,30 +1336,28 b' def overridecat(orig, ui, repo, file1, *'
     m.visitdir = lfvisitdirfn
 
     for f in ctx.walk(m):
-        fp = cmdutil.makefileobj(repo, opts.get('output'), ctx.node(),
-                                 pathname=f)
-        lf = lfutil.splitstandin(f)
-        if lf is None or origmatchfn(f):
-            # duplicating unreachable code from commands.cat
-            data = ctx[f].data()
-            if opts.get('decode'):
-                data = repo.wwritedata(f, data)
-            fp.write(data)
-        else:
-            hash = lfutil.readstandin(repo, lf, ctx.rev())
-            if not lfutil.inusercache(repo.ui, hash):
-                store = storefactory.openstore(repo)
-                success, missing = store.get([(lf, hash)])
-                if len(success) != 1:
-                    raise error.Abort(
-                        _('largefile %s is not in cache and could not be '
-                          'downloaded')  % lf)
-            path = lfutil.usercachepath(repo.ui, hash)
-            fpin = open(path, "rb")
-            for chunk in util.filechunkiter(fpin, 128 * 1024):
-                fp.write(chunk)
-            fpin.close()
-        fp.close()
+        with cmdutil.makefileobj(repo, opts.get('output'), ctx.node(),
+                                 pathname=f) as fp:
+            lf = lfutil.splitstandin(f)
+            if lf is None or origmatchfn(f):
+                # duplicating unreachable code from commands.cat
+                data = ctx[f].data()
+                if opts.get('decode'):
+                    data = repo.wwritedata(f, data)
+                fp.write(data)
+            else:
+                hash = lfutil.readstandin(repo, lf, ctx.rev())
+                if not lfutil.inusercache(repo.ui, hash):
+                    store = storefactory.openstore(repo)
+                    success, missing = store.get([(lf, hash)])
+                    if len(success) != 1:
+                        raise error.Abort(
+                            _('largefile %s is not in cache and could not be '
+                              'downloaded')  % lf)
+                path = lfutil.usercachepath(repo.ui, hash)
+                with open(path, "rb") as fpin:
+                    for chunk in util.filechunkiter(fpin):
+                        fp.write(chunk)
         err = 0
     return err
 

hgext/largefiles/proto.py

@@ -134,7 +134,7 b' def wirereposetup(ui, repo):'
                                                 length))
 
             # SSH streams will block if reading more than length
-            for chunk in util.filechunkiter(stream, 128 * 1024, length):
+            for chunk in util.filechunkiter(stream, limit=length):
                 yield chunk
             # HTTP streams must hit the end to process the last empty
             # chunk of Chunked-Encoding so the connection can be reused.

hgext/largefiles/remotestore.py

@@ -45,17 +45,13 b' class remotestore(basestore.basestore):'
 
     def sendfile(self, filename, hash):
         self.ui.debug('remotestore: sendfile(%s, %s)\n' % (filename, hash))
-        fd = None
         try:
-            fd = lfutil.httpsendfile(self.ui, filename)
-            return self._put(hash, fd)
+            with lfutil.httpsendfile(self.ui, filename) as fd:
+                return self._put(hash, fd)
         except IOError as e:
             raise error.Abort(
                 _('remotestore: could not open file %s: %s')
                 % (filename, str(e)))
-        finally:
-            if fd:
-                fd.close()
 
     def _getfile(self, tmpfile, filename, hash):
         try:
@@ -122,7 +118,7 b' class remotestore(basestore.basestore):'
         raise NotImplementedError('abstract method')
 
     def _get(self, hash):
-        '''Get file with the given hash from the remote store.'''
+        '''Get a iterator for content with the given hash.'''
         raise NotImplementedError('abstract method')
 
     def _stat(self, hashes):

hgext/logtoprocess.py

@@ -40,11 +40,11 b' import platform'
 import subprocess
 import sys
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def uisetup(ui):
     if platform.system() == 'Windows':

hgext/mq.py

@@ -99,11 +99,11 b" seriesopts = [('s', 'summary', None, _('"
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # force load strip extension formerly included in mq and import some utility
 try:
@@ -1562,7 +1562,7 b' class queue(object):'
             if not repo[self.applied[-1].node].mutable():
                 raise error.Abort(
                     _("popping would remove a public revision"),
-                    hint=_('see "hg help phases" for details'))
+                    hint=_("see 'hg help phases' for details"))
 
             # we know there are no local changes, so we can make a simplified
             # form of hg.update.
@@ -1631,7 +1631,7 b' class queue(object):'
                 raise error.Abort(_("cannot qrefresh a revision with children"))
             if not repo[top].mutable():
                 raise error.Abort(_("cannot qrefresh public revision"),
-                                 hint=_('see "hg help phases" for details'))
+                                 hint=_("see 'hg help phases' for details"))
 
             cparents = repo.changelog.parents(top)
             patchparent = self.qparents(repo, top)
@@ -1840,7 +1840,7 b' class queue(object):'
 
                     self.applied.append(statusentry(n, patchfn))
                 finally:
-                    lockmod.release(lock, tr)
+                    lockmod.release(tr, lock)
             except: # re-raises
                 ctx = repo[cparents[0]]
                 repo.dirstate.rebuild(ctx.node(), ctx.manifest())
@@ -2117,7 +2117,7 b' class queue(object):'
                 for r in rev:
                     if not repo[r].mutable():
                         raise error.Abort(_('revision %d is not mutable') % r,
-                                         hint=_('see "hg help phases" '
+                                         hint=_("see 'hg help phases' "
                                                 'for details'))
                     p1, p2 = repo.changelog.parentrevs(r)
                     n = repo.changelog.node(r)
@@ -3354,53 +3354,54 b' def qqueue(ui, repo, name=None, **opts):'
         raise error.Abort(
                 _('invalid queue name, may not contain the characters ":\\/."'))
 
-    existing = _getqueues()
-
-    if opts.get('create'):
-        if name in existing:
-            raise error.Abort(_('queue "%s" already exists') % name)
-        if _noqueues():
-            _addqueue(_defaultqueue)
-        _addqueue(name)
-        _setactive(name)
-    elif opts.get('rename'):
-        current = _getcurrent()
-        if name == current:
-            raise error.Abort(_('can\'t rename "%s" to its current name')
-                              % name)
-        if name in existing:
-            raise error.Abort(_('queue "%s" already exists') % name)
-
-        olddir = _queuedir(current)
-        newdir = _queuedir(name)
-
-        if os.path.exists(newdir):
-            raise error.Abort(_('non-queue directory "%s" already exists') %
-                    newdir)
-
-        fh = repo.vfs('patches.queues.new', 'w')
-        for queue in existing:
-            if queue == current:
-                fh.write('%s\n' % (name,))
-                if os.path.exists(olddir):
-                    util.rename(olddir, newdir)
-            else:
-                fh.write('%s\n' % (queue,))
-        fh.close()
-        util.rename(repo.join('patches.queues.new'), repo.join(_allqueues))
-        _setactivenocheck(name)
-    elif opts.get('delete'):
-        _delete(name)
-    elif opts.get('purge'):
-        if name in existing:
+    with repo.wlock():
+        existing = _getqueues()
+
+        if opts.get('create'):
+            if name in existing:
+                raise error.Abort(_('queue "%s" already exists') % name)
+            if _noqueues():
+                _addqueue(_defaultqueue)
+            _addqueue(name)
+            _setactive(name)
+        elif opts.get('rename'):
+            current = _getcurrent()
+            if name == current:
+                raise error.Abort(_('can\'t rename "%s" to its current name')
+                                  % name)
+            if name in existing:
+                raise error.Abort(_('queue "%s" already exists') % name)
+
+            olddir = _queuedir(current)
+            newdir = _queuedir(name)
+
+            if os.path.exists(newdir):
+                raise error.Abort(_('non-queue directory "%s" already exists') %
+                        newdir)
+
+            fh = repo.vfs('patches.queues.new', 'w')
+            for queue in existing:
+                if queue == current:
+                    fh.write('%s\n' % (name,))
+                    if os.path.exists(olddir):
+                        util.rename(olddir, newdir)
+                else:
+                    fh.write('%s\n' % (queue,))
+            fh.close()
+            util.rename(repo.join('patches.queues.new'), repo.join(_allqueues))
+            _setactivenocheck(name)
+        elif opts.get('delete'):
             _delete(name)
-        qdir = _queuedir(name)
-        if os.path.exists(qdir):
-            shutil.rmtree(qdir)
-    else:
-        if name not in existing:
-            raise error.Abort(_('use --create to create a new queue'))
-        _setactive(name)
+        elif opts.get('purge'):
+            if name in existing:
+                _delete(name)
+            qdir = _queuedir(name)
+            if os.path.exists(qdir):
+                shutil.rmtree(qdir)
+        else:
+            if name not in existing:
+                raise error.Abort(_('use --create to create a new queue'))
+            _setactive(name)
 
 def mqphasedefaults(repo, roots):
     """callback used to set mq changeset as secret when no phase data exists"""

hgext/notify.py

@@ -148,11 +148,11 b' from mercurial import ('
     util,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # template for single changeset can include email headers.
 single_template = '''

hgext/pager.py

@@ -10,7 +10,7 b''
 #   [extension]
 #   pager =
 #
-# Run "hg help pager" to get info on configuration.
+# Run 'hg help pager' to get info on configuration.
 
 '''browse command output with an external pager
 
@@ -75,11 +75,11 b' from mercurial import ('
     util,
     )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def _runpager(ui, p):
     pager = subprocess.Popen(p, shell=True, bufsize=-1,

hgext/patchbomb.py

@@ -87,11 +87,11 b' stringio = util.stringio'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def _addpullheader(seq, ctx):
     """Add a header pointing to a public URL where the changeset is available

hgext/purge.py

@@ -38,11 +38,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('purge|clean',
     [('a', 'abort-on-err', None, _('abort if an error occurs')),

hgext/rebase.py

@@ -66,11 +66,11 b' revskipped = (revignored, revprecursor, '
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def _nothingtorebase():
     return 1
@@ -296,7 +296,7 b' class rebaseruntime(object):'
         if not self.keepf and not self.repo[root].mutable():
             raise error.Abort(_("can't rebase public changeset %s")
                              % self.repo[root],
-                             hint=_('see "hg help phases" for details'))
+                             hint=_("see 'hg help phases' for details"))
 
         (self.originalwd, self.target, self.state) = result
         if self.collapsef:
@@ -335,8 +335,9 b' class rebaseruntime(object):'
         if self.activebookmark:
             bookmarks.deactivate(repo)
 
-        sortedrevs = sorted(self.state)
-        total = len(self.state)
+        sortedrevs = repo.revs('sort(%ld, -topo)', self.state)
+        cands = [k for k, v in self.state.iteritems() if v == revtodo]
+        total = len(cands)
         pos = 0
         for rev in sortedrevs:
             ctx = repo[rev]
@@ -345,8 +346,8 b' class rebaseruntime(object):'
             names = repo.nodetags(ctx.node()) + repo.nodebookmarks(ctx.node())
             if names:
                 desc += ' (%s)' % ' '.join(names)
-            pos += 1
             if self.state[rev] == revtodo:
+                pos += 1
                 ui.status(_('rebasing %s\n') % desc)
                 ui.progress(_("rebasing"), pos, ("%d:%s" % (rev, ctx)),
                             _('changesets'), total)
@@ -1127,7 +1128,7 b' def abort(repo, originalwd, target, stat'
         if immutable:
             repo.ui.warn(_("warning: can't clean up public changesets %s\n")
                         % ', '.join(str(repo[r]) for r in immutable),
-                        hint=_('see "hg help phases" for details'))
+                        hint=_("see 'hg help phases' for details"))
             cleanup = False
 
         descendants = set()
@@ -1197,7 +1198,7 b' def buildstate(repo, dest, rebaseset, co'
                 repo.ui.debug('source is a child of destination\n')
                 return None
 
-        repo.ui.debug('rebase onto %d starting from %s\n' % (dest, root))
+        repo.ui.debug('rebase onto %s starting from %s\n' % (dest, root))
         state.update(dict.fromkeys(rebaseset, revtodo))
         # Rebase tries to turn <dest> into a parent of <root> while
         # preserving the number of parents of rebased changesets:

hgext/record.py

@@ -22,11 +22,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 
 @command("record",
@@ -70,7 +70,7 b' def record(ui, repo, *pats, **opts):'
     backup = ui.backupconfig('experimental', 'crecord')
     try:
         ui.setconfig('experimental', 'crecord', False, 'record')
-        commands.commit(ui, repo, *pats, **opts)
+        return commands.commit(ui, repo, *pats, **opts)
     finally:
         ui.restoreconfig(backup)
 

hgext/relink.py

@@ -21,11 +21,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('relink', [], _('[ORIGIN]'))
 def relink(ui, repo, origin=None, **opts):

hgext/schemes.py

@@ -56,11 +56,11 b' from mercurial import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 
 class ShortRepository(object):

hgext/share.py

@@ -56,11 +56,11 b' parseurl = hg.parseurl'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 @command('share',
     [('U', 'noupdate', None, _('do not create a working directory')),

hgext/shelve.py

@@ -54,11 +54,11 b' from . import ('
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 backupdir = 'shelve-backup'
 shelvedir = 'shelved'

hgext/strip.py

@@ -23,11 +23,11 b' release = lockmod.release'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 def checksubstate(repo, baserev=None):
     '''return list of subrepos at a different revision than substate.

hgext/transplant.py

@@ -40,11 +40,11 b' class TransplantError(error.Abort):'
 
 cmdtable = {}
 command = cmdutil.command(cmdtable)
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 class transplantentry(object):
     def __init__(self, lnode, rnode):

hgext/win32mbcs.py

@@ -55,11 +55,11 b' from mercurial import ('
     error,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 _encoding = None                                # see extsetup
 
@@ -148,8 +148,8 b' def wrapname(name, wrapper):'
 # NOTE: os.path.dirname() and os.path.basename() are safe because
 #       they use result of os.path.split()
 funcs = '''os.path.join os.path.split os.path.splitext
- os.path.normpath os.makedirs
- mercurial.util.endswithsep mercurial.util.splitpath mercurial.util.checkcase
+ os.path.normpath os.makedirs mercurial.util.endswithsep
+ mercurial.util.splitpath mercurial.util.fscasesensitive
  mercurial.util.fspath mercurial.util.pconvert mercurial.util.normpath
  mercurial.util.checkwinfilename mercurial.util.checkosfilename
  mercurial.util.split'''

hgext/win32text.py

@@ -52,11 +52,11 b' from mercurial import ('
     util,
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # regexp for single LF without CR preceding.
 re_single_lf = re.compile('(^|[^\r])\n', re.MULTILINE)

hgext/zeroconf/__init__.py

@@ -40,11 +40,11 b' from mercurial.hgweb import ('
     server as servermod
 )
 
-# Note for extension authors: ONLY specify testedwith = 'internal' for
+# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
 # be specifying the version(s) of Mercurial they are tested with, or
 # leave the attribute unspecified.
-testedwith = 'internal'
+testedwith = 'ships-with-hg-core'
 
 # publish
 

i18n/hggettext

@@ -114,7 +114,7 b' def docstrings(path):'
         if func.__doc__:
             src = inspect.getsource(func)
             name = "%s.%s" % (path, func.__name__)
-            lineno = func.func_code.co_firstlineno
+            lineno = inspect.getsourcelines(func)[1]
             doc = func.__doc__
             if rstrip:
                 doc = doc.rstrip()

mercurial/__init__.py

@@ -170,7 +170,7 b' if sys.version_info[0] >= 3:'
             spec.loader = hgloader(spec.name, spec.origin)
             return spec
 
-    def replacetokens(tokens):
+    def replacetokens(tokens, fullname):
         """Transform a stream of tokens from raw to Python 3.
 
         It is called by the custom module loading machinery to rewrite
@@ -184,6 +184,57 b' if sys.version_info[0] >= 3:'
         REMEMBER TO CHANGE ``BYTECODEHEADER`` WHEN CHANGING THIS FUNCTION
         OR CACHED FILES WON'T GET INVALIDATED PROPERLY.
         """
+        futureimpline = False
+
+        # The following utility functions access the tokens list and i index of
+        # the for i, t enumerate(tokens) loop below
+        def _isop(j, *o):
+            """Assert that tokens[j] is an OP with one of the given values"""
+            try:
+                return tokens[j].type == token.OP and tokens[j].string in o
+            except IndexError:
+                return False
+
+        def _findargnofcall(n):
+            """Find arg n of a call expression (start at 0)
+
+            Returns index of the first token of that argument, or None if
+            there is not that many arguments.
+
+            Assumes that token[i + 1] is '('.
+
+            """
+            nested = 0
+            for j in range(i + 2, len(tokens)):
+                if _isop(j, ')', ']', '}'):
+                    # end of call, tuple, subscription or dict / set
+                    nested -= 1
+                    if nested < 0:
+                        return None
+                elif n == 0:
+                    # this is the starting position of arg
+                    return j
+                elif _isop(j, '(', '[', '{'):
+                    nested += 1
+                elif _isop(j, ',') and nested == 0:
+                    n -= 1
+
+            return None
+
+        def _ensureunicode(j):
+            """Make sure the token at j is a unicode string
+
+            This rewrites a string token to include the unicode literal prefix
+            so the string transformer won't add the byte prefix.
+
+            Ignores tokens that are not strings. Assumes bounds checking has
+            already been done.
+
+            """
+            st = tokens[j]
+            if st.type == token.STRING and st.string.startswith(("'", '"')):
+                tokens[j] = st._replace(string='u%s' % st.string)
+
         for i, t in enumerate(tokens):
             # Convert most string literals to byte literals. String literals
             # in Python 2 are bytes. String literals in Python 3 are unicode.
@@ -213,64 +264,61 b' if sys.version_info[0] >= 3:'
                     continue
 
                 # String literal. Prefix to make a b'' string.
-                yield tokenize.TokenInfo(t.type, 'b%s' % s, t.start, t.end,
-                                          t.line)
+                yield t._replace(string='b%s' % t.string)
                 continue
 
-            try:
-                nexttoken = tokens[i + 1]
-            except IndexError:
-                nexttoken = None
-
-            try:
-                prevtoken = tokens[i - 1]
-            except IndexError:
-                prevtoken = None
+            # Insert compatibility imports at "from __future__ import" line.
+            # No '\n' should be added to preserve line numbers.
+            if (t.type == token.NAME and t.string == 'import' and
+                all(u.type == token.NAME for u in tokens[i - 2:i]) and
+                [u.string for u in tokens[i - 2:i]] == ['from', '__future__']):
+                futureimpline = True
+            if t.type == token.NEWLINE and futureimpline:
+                futureimpline = False
+                if fullname == 'mercurial.pycompat':
+                    yield t
+                    continue
+                r, c = t.start
+                l = (b'; from mercurial.pycompat import '
+                     b'delattr, getattr, hasattr, setattr, xrange\n')
+                for u in tokenize.tokenize(io.BytesIO(l).readline):
+                    if u.type in (tokenize.ENCODING, token.ENDMARKER):
+                        continue
+                    yield u._replace(
+                        start=(r, c + u.start[1]), end=(r, c + u.end[1]))
+                continue
 
             # This looks like a function call.
-            if (t.type == token.NAME and nexttoken and
-                nexttoken.type == token.OP and nexttoken.string == '('):
+            if t.type == token.NAME and _isop(i + 1, '('):
                 fn = t.string
 
                 # *attr() builtins don't accept byte strings to 2nd argument.
-                # Rewrite the token to include the unicode literal prefix so
-                # the string transformer above doesn't add the byte prefix.
-                if fn in ('getattr', 'setattr', 'hasattr', 'safehasattr'):
-                    try:
-                        # (NAME, 'getattr')
-                        # (OP, '(')
-                        # (NAME, 'foo')
-                        # (OP, ',')
-                        # (NAME|STRING, foo)
-                        st = tokens[i + 4]
-                        if (st.type == token.STRING and
-                            st.string[0] in ("'", '"')):
-                            rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
-                                                    st.start, st.end, st.line)
-                            tokens[i + 4] = rt
-                    except IndexError:
-                        pass
+                if (fn in ('getattr', 'setattr', 'hasattr', 'safehasattr') and
+                        not _isop(i - 1, '.')):
+                    arg1idx = _findargnofcall(1)
+                    if arg1idx is not None:
+                        _ensureunicode(arg1idx)
 
                 # .encode() and .decode() on str/bytes/unicode don't accept
-                # byte strings on Python 3. Rewrite the token to include the
-                # unicode literal prefix so the string transformer above doesn't
-                # add the byte prefix.
-                if (fn in ('encode', 'decode') and
-                    prevtoken.type == token.OP and prevtoken.string == '.'):
-                    # (OP, '.')
-                    # (NAME, 'encode')
-                    # (OP, '(')
-                    # (STRING, 'utf-8')
-                    # (OP, ')')
-                    try:
-                        st = tokens[i + 2]
-                        if (st.type == token.STRING and
-                            st.string[0] in ("'", '"')):
-                            rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
-                                                    st.start, st.end, st.line)
-                            tokens[i + 2] = rt
-                    except IndexError:
-                        pass
+                # byte strings on Python 3.
+                elif fn in ('encode', 'decode') and _isop(i - 1, '.'):
+                    for argn in range(2):
+                        argidx = _findargnofcall(argn)
+                        if argidx is not None:
+                            _ensureunicode(argidx)
+
+                # Bare open call (not an attribute on something else), the
+                # second argument (mode) must be a string, not bytes
+                elif fn == 'open' and not _isop(i - 1, '.'):
+                    arg1idx = _findargnofcall(1)
+                    if arg1idx is not None:
+                        _ensureunicode(arg1idx)
+
+                # It changes iteritems to items as iteritems is not
+                # present in Python 3 world.
+                elif fn == 'iteritems':
+                    yield t._replace(string='items')
+                    continue
 
             # Emit unmodified token.
             yield t
@@ -279,7 +327,7 b' if sys.version_info[0] >= 3:'
     # ``replacetoken`` or any mechanism that changes semantics of module
     # loading is changed. Otherwise cached bytecode may get loaded without
     # the new transformation mechanisms applied.
-    BYTECODEHEADER = b'HG\x00\x01'
+    BYTECODEHEADER = b'HG\x00\x06'
 
     class hgloader(importlib.machinery.SourceFileLoader):
         """Custom module loader that transforms source code.
@@ -338,7 +386,7 b' if sys.version_info[0] >= 3:'
             """Perform token transformation before compilation."""
             buf = io.BytesIO(data)
             tokens = tokenize.tokenize(buf.readline)
-            data = tokenize.untokenize(replacetokens(list(tokens)))
+            data = tokenize.untokenize(replacetokens(list(tokens), self.name))
             # Python's built-in importer strips frames from exceptions raised
             # for this code. Unfortunately, that mechanism isn't extensible
             # and our frame will be blamed for the import failure. There

mercurial/archival.py

@@ -231,7 +231,7 b' class zipit(object):'
         if islink:
             mode = 0o777
             ftype = _UNX_IFLNK
-        i.external_attr = (mode | ftype) << 16L
+        i.external_attr = (mode | ftype) << 16
         # add "extended-timestamp" extra block, because zip archives
         # without this will be extracted with unexpected timestamp,
         # if TZ is not configured as GMT

mercurial/bdiff_module.c

@@ -17,6 +17,7 b''
 
 #include "bdiff.h"
 #include "bitmanipulation.h"
+#include "util.h"
 
 
 static PyObject *blocks(PyObject *self, PyObject *args)

mercurial/branchmap.py

@@ -470,8 +470,12 b' class revbranchcache(object):'
     def write(self, tr=None):
         """Save branch cache if it is dirty."""
         repo = self._repo
-        if self._rbcnamescount < len(self._names):
-            try:
+        wlock = None
+        step = ''
+        try:
+            if self._rbcnamescount < len(self._names):
+                step = ' names'
+                wlock = repo.wlock(wait=False)
                 if self._rbcnamescount != 0:
                     f = repo.vfs.open(_rbcnames, 'ab')
                     if f.tell() == self._rbcsnameslen:
@@ -489,16 +493,15 b' class revbranchcache(object):'
                                   for b in self._names[self._rbcnamescount:]))
                 self._rbcsnameslen = f.tell()
                 f.close()
-            except (IOError, OSError, error.Abort) as inst:
-                repo.ui.debug("couldn't write revision branch cache names: "
-                              "%s\n" % inst)
-                return
-            self._rbcnamescount = len(self._names)
+                self._rbcnamescount = len(self._names)
 
-        start = self._rbcrevslen * _rbcrecsize
-        if start != len(self._rbcrevs):
-            revs = min(len(repo.changelog), len(self._rbcrevs) // _rbcrecsize)
-            try:
+            start = self._rbcrevslen * _rbcrecsize
+            if start != len(self._rbcrevs):
+                step = ''
+                if wlock is None:
+                    wlock = repo.wlock(wait=False)
+                revs = min(len(repo.changelog),
+                           len(self._rbcrevs) // _rbcrecsize)
                 f = repo.vfs.open(_rbcrevs, 'ab')
                 if f.tell() != start:
                     repo.ui.debug("truncating %s to %s\n" % (_rbcrevs, start))
@@ -510,8 +513,10 b' class revbranchcache(object):'
                 end = revs * _rbcrecsize
                 f.write(self._rbcrevs[start:end])
                 f.close()
-            except (IOError, OSError, error.Abort) as inst:
-                repo.ui.debug("couldn't write revision branch cache: %s\n" %
-                              inst)
-                return
-            self._rbcrevslen = revs
+                self._rbcrevslen = revs
+        except (IOError, OSError, error.Abort, error.LockError) as inst:
+            repo.ui.debug("couldn't write revision branch cache%s: %s\n"
+                          % (step, inst))
+        finally:
+            if wlock is not None:
+                wlock.release()

mercurial/bundle2.py

@@ -159,6 +159,7 b' from . import ('
     error,
     obsolete,
     pushkey,
+    pycompat,
     tags,
     url,
     util,
@@ -572,7 +573,9 b' class bundle20(object):'
             yield param
         # starting compression
         for chunk in self._getcorechunk():
-            yield self._compressor.compress(chunk)
+            data = self._compressor.compress(chunk)
+            if data:
+                yield data
         yield self._compressor.flush()
 
     def _paramchunk(self):
@@ -996,7 +999,10 b' class bundlepart(object):'
             outdebug(ui, 'closing payload chunk')
             # abort current part payload
             yield _pack(_fpayloadsize, 0)
-            raise exc_info[0], exc_info[1], exc_info[2]
+            if pycompat.ispy3:
+                raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])
+            else:
+                exec("""raise exc_info[0], exc_info[1], exc_info[2]""")
         # end of payload
         outdebug(ui, 'closing payload chunk')
         yield _pack(_fpayloadsize, 0)
@@ -1320,7 +1326,9 b' def writebundle(ui, cg, filename, bundle'
         def chunkiter():
             yield header
             for chunk in subchunkiter:
-                yield z.compress(chunk)
+                data = z.compress(chunk)
+                if data:
+                    yield data
             yield z.flush()
         chunkiter = chunkiter()
 

mercurial/bundlerepo.py

@@ -56,10 +56,8 b' class bundlerevlog(revlog.revlog):'
         self.repotiprev = n - 1
         chain = None
         self.bundlerevs = set() # used by 'bundle()' revset expression
-        while True:
-            chunkdata = bundle.deltachunk(chain)
-            if not chunkdata:
-                break
+        getchunk = lambda: bundle.deltachunk(chain)
+        for chunkdata in iter(getchunk, {}):
             node = chunkdata['node']
             p1 = chunkdata['p1']
             p2 = chunkdata['p2']
@@ -190,22 +188,36 b' class bundlechangelog(bundlerevlog, chan'
             self.filteredrevs = oldfilter
 
 class bundlemanifest(bundlerevlog, manifest.manifest):
-    def __init__(self, opener, bundle, linkmapper):
-        manifest.manifest.__init__(self, opener)
+    def __init__(self, opener, bundle, linkmapper, dirlogstarts=None, dir=''):
+        manifest.manifest.__init__(self, opener, dir=dir)
         bundlerevlog.__init__(self, opener, self.indexfile, bundle,
                               linkmapper)
+        if dirlogstarts is None:
+            dirlogstarts = {}
+            if self.bundle.version == "03":
+                dirlogstarts = _getfilestarts(self.bundle)
+        self._dirlogstarts = dirlogstarts
+        self._linkmapper = linkmapper
 
     def baserevision(self, nodeorrev):
         node = nodeorrev
         if isinstance(node, int):
             node = self.node(node)
 
-        if node in self._mancache:
-            result = self._mancache[node][0].text()
+        if node in self.fulltextcache:
+            result = self.fulltextcache[node].tostring()
         else:
             result = manifest.manifest.revision(self, nodeorrev)
         return result
 
+    def dirlog(self, d):
+        if d in self._dirlogstarts:
+            self.bundle.seek(self._dirlogstarts[d])
+            return bundlemanifest(
+                self.opener, self.bundle, self._linkmapper,
+                self._dirlogstarts, dir=d)
+        return super(bundlemanifest, self).dirlog(d)
+
 class bundlefilelog(bundlerevlog, filelog.filelog):
     def __init__(self, opener, path, bundle, linkmapper):
         filelog.filelog.__init__(self, opener, path)
@@ -236,6 +248,15 b' class bundlephasecache(phases.phasecache'
         self.invalidate()
         self.dirty = True
 
+def _getfilestarts(bundle):
+    bundlefilespos = {}
+    for chunkdata in iter(bundle.filelogheader, {}):
+        fname = chunkdata['filename']
+        bundlefilespos[fname] = bundle.tell()
+        for chunk in iter(lambda: bundle.deltachunk(None), {}):
+            pass
+    return bundlefilespos
+
 class bundlerepository(localrepo.localrepository):
     def __init__(self, ui, path, bundlename):
         def _writetempbundle(read, suffix, header=''):
@@ -283,7 +304,8 b' class bundlerepository(localrepo.localre'
                                                   "multiple changegroups")
                     cgstream = part
                     version = part.params.get('version', '01')
-                    if version not in changegroup.allsupportedversions(ui):
+                    legalcgvers = changegroup.supportedincomingversions(self)
+                    if version not in legalcgvers:
                         msg = _('Unsupported changegroup version: %s')
                         raise error.Abort(msg % version)
                     if self.bundle.compressed():
@@ -328,10 +350,6 b' class bundlerepository(localrepo.localre'
         self.bundle.manifestheader()
         linkmapper = self.unfiltered().changelog.rev
         m = bundlemanifest(self.svfs, self.bundle, linkmapper)
-        # XXX: hack to work with changegroup3, but we still don't handle
-        # tree manifests correctly
-        if self.bundle.version == "03":
-            self.bundle.filelogheader()
         self.filestart = self.bundle.tell()
         return m
 
@@ -351,16 +369,7 b' class bundlerepository(localrepo.localre'
     def file(self, f):
         if not self.bundlefilespos:
             self.bundle.seek(self.filestart)
-            while True:
-                chunkdata = self.bundle.filelogheader()
-                if not chunkdata:
-                    break
-                fname = chunkdata['filename']
-                self.bundlefilespos[fname] = self.bundle.tell()
-                while True:
-                    c = self.bundle.deltachunk(None)
-                    if not c:
-                        break
+            self.bundlefilespos = _getfilestarts(self.bundle)
 
         if f in self.bundlefilespos:
             self.bundle.seek(self.bundlefilespos[f])
@@ -480,7 +489,10 b' def getremotechanges(ui, repo, other, on'
     if bundlename or not localrepo:
         # create a bundle (uncompressed if other repo is not local)
 
-        canbundle2 = (ui.configbool('experimental', 'bundle2-exp', True)
+        # developer config: devel.legacy.exchange
+        legexc = ui.configlist('devel', 'legacy.exchange')
+        forcebundle1 = 'bundle2' not in legexc and 'bundle1' in legexc
+        canbundle2 = (not forcebundle1
                       and other.capable('getbundle')
                       and other.capable('bundle2'))
         if canbundle2:

mercurial/changegroup.py

@@ -15,7 +15,6 b' import weakref'
 from .i18n import _
 from .node import (
     hex,
-    nullid,
     nullrev,
     short,
 )
@@ -94,7 +93,9 b' def writechunks(ui, chunks, filename, vf'
             if vfs:
                 fh = vfs.open(filename, "wb")
             else:
-                fh = open(filename, "wb")
+                # Increase default buffer size because default is usually
+                # small (4k is common on Linux).
+                fh = open(filename, "wb", 131072)
         else:
             fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
             fh = os.fdopen(fd, "wb")
@@ -333,7 +334,7 b' class cg1unpacker(object):'
                     for cset in xrange(clstart, clend):
                         mfnode = repo.changelog.read(
                             repo.changelog.node(cset))[0]
-                        mfest = repo.manifest.readdelta(mfnode)
+                        mfest = repo.manifestlog[mfnode].readdelta()
                         # store file nodes we must see
                         for f, n in mfest.iteritems():
                             needfiles.setdefault(f, set()).add(n)
@@ -404,6 +405,7 b' class cg1unpacker(object):'
                         # coming call to `destroyed` will repair it.
                         # In other case we can safely update cache on
                         # disk.
+                        repo.ui.debug('updating the branch cache\n')
                         branchmap.updatecache(repo.filtered('served'))
 
                     def runhooks():
@@ -413,8 +415,6 b' class cg1unpacker(object):'
                         if clstart >= len(repo):
                             return
 
-                        # forcefully update the on-disk branch cache
-                        repo.ui.debug("updating the branch cache\n")
                         repo.hook("changegroup", **hookargs)
 
                         for n in added:
@@ -475,10 +475,7 b' class cg3unpacker(cg2unpacker):'
     def _unpackmanifests(self, repo, revmap, trp, prog, numchanges):
         super(cg3unpacker, self)._unpackmanifests(repo, revmap, trp, prog,
                                                   numchanges)
-        while True:
-            chunkdata = self.filelogheader()
-            if not chunkdata:
-                break
+        for chunkdata in iter(self.filelogheader, {}):
             # If we get here, there are directory manifests in the changegroup
             d = chunkdata["filename"]
             repo.ui.debug("adding %s revisions\n" % d)
@@ -823,11 +820,24 b' class cg2packer(cg1packer):'
 
     def deltaparent(self, revlog, rev, p1, p2, prev):
         dp = revlog.deltaparent(rev)
-        # avoid storing full revisions; pick prev in those cases
-        # also pick prev when we can't be sure remote has dp
-        if dp == nullrev or (dp != p1 and dp != p2 and dp != prev):
+        if dp == nullrev and revlog.storedeltachains:
+            # Avoid sending full revisions when delta parent is null. Pick prev
+            # in that case. It's tempting to pick p1 in this case, as p1 will
+            # be smaller in the common case. However, computing a delta against
+            # p1 may require resolving the raw text of p1, which could be
+            # expensive. The revlog caches should have prev cached, meaning
+            # less CPU for changegroup generation. There is likely room to add
+            # a flag and/or config option to control this behavior.
             return prev
-        return dp
+        elif dp == nullrev:
+            # revlog is configured to use full snapshot for a reason,
+            # stick to full snapshot.
+            return nullrev
+        elif dp not in (p1, p2, prev):
+            # Pick prev when we can't be sure remote has the base revision.
+            return prev
+        else:
+            return dp
 
     def builddeltaheader(self, node, p1n, p2n, basenode, linknode, flags):
         # Do nothing with flags, it is implicitly 0 in cg1 and cg2
@@ -946,17 +956,7 b' def changegroupsubset(repo, roots, heads'
     Another wrinkle is doing the reverse, figuring out which changeset in
     the changegroup a particular filenode or manifestnode belongs to.
     """
-    cl = repo.changelog
-    if not roots:
-        roots = [nullid]
-    discbases = []
-    for n in roots:
-        discbases.extend([p for p in cl.parents(n) if p != nullid])
-    # TODO: remove call to nodesbetween.
-    csets, roots, heads = cl.nodesbetween(roots, heads)
-    included = set(csets)
-    discbases = [n for n in discbases if n not in included]
-    outgoing = discovery.outgoing(cl, discbases, heads)
+    outgoing = discovery.outgoing(repo, missingroots=roots, missingheads=heads)
     bundler = getbundler(version, repo)
     return getsubset(repo, outgoing, bundler, source)
 
@@ -982,26 +982,7 b' def getlocalchangegroup(repo, source, ou'
     bundler = getbundler(version, repo, bundlecaps)
     return getsubset(repo, outgoing, bundler, source)
 
-def computeoutgoing(repo, heads, common):
-    """Computes which revs are outgoing given a set of common
-    and a set of heads.
-
-    This is a separate function so extensions can have access to
-    the logic.
-
-    Returns a discovery.outgoing object.
-    """
-    cl = repo.changelog
-    if common:
-        hasnode = cl.hasnode
-        common = [n for n in common if hasnode(n)]
-    else:
-        common = [nullid]
-    if not heads:
-        heads = cl.heads()
-    return discovery.outgoing(cl, common, heads)
-
-def getchangegroup(repo, source, heads=None, common=None, bundlecaps=None,
+def getchangegroup(repo, source, outgoing, bundlecaps=None,
                    version='01'):
     """Like changegroupsubset, but returns the set difference between the
     ancestors of heads and the ancestors common.
@@ -1011,7 +992,6 b' def getchangegroup(repo, source, heads=N'
     The nodes in common might not all be known locally due to the way the
     current discovery protocol works.
     """
-    outgoing = computeoutgoing(repo, heads, common)
     return getlocalchangegroup(repo, source, outgoing, bundlecaps=bundlecaps,
                                version=version)
 
@@ -1022,10 +1002,7 b' def changegroup(repo, basenodes, source)'
 def _addchangegroupfiles(repo, source, revmap, trp, expectedfiles, needfiles):
     revisions = 0
     files = 0
-    while True:
-        chunkdata = source.filelogheader()
-        if not chunkdata:
-            break
+    for chunkdata in iter(source.filelogheader, {}):
         files += 1
         f = chunkdata["filename"]
         repo.ui.debug("adding %s revisions\n" % f)

mercurial/changelog.py

@@ -124,7 +124,7 b' class appender(object):'
 
 def _divertopener(opener, target):
     """build an opener that writes in 'target.a' instead of 'target'"""
-    def _divert(name, mode='r'):
+    def _divert(name, mode='r', checkambig=False):
         if name != target:
             return opener(name, mode)
         return opener(name + ".a", mode)
@@ -132,15 +132,16 b' def _divertopener(opener, target):'
 
 def _delayopener(opener, target, buf):
     """build an opener that stores chunks in 'buf' instead of 'target'"""
-    def _delay(name, mode='r'):
+    def _delay(name, mode='r', checkambig=False):
         if name != target:
             return opener(name, mode)
         return appender(opener, name, mode, buf)
     return _delay
 
-_changelogrevision = collections.namedtuple('changelogrevision',
-                                            ('manifest', 'user', 'date',
-                                             'files', 'description', 'extra'))
+_changelogrevision = collections.namedtuple(u'changelogrevision',
+                                            (u'manifest', u'user', u'date',
+                                             u'files', u'description',
+                                             u'extra'))
 
 class changelogrevision(object):
     """Holds results of a parsed changelog revision.
@@ -151,8 +152,8 b' class changelogrevision(object):'
     """
 
     __slots__ = (
-        '_offsets',
-        '_text',
+        u'_offsets',
+        u'_text',
     )
 
     def __new__(cls, text):
@@ -256,11 +257,18 b' class changelogrevision(object):'
 
 class changelog(revlog.revlog):
     def __init__(self, opener):
-        revlog.revlog.__init__(self, opener, "00changelog.i")
+        revlog.revlog.__init__(self, opener, "00changelog.i",
+                               checkambig=True)
         if self._initempty:
             # changelogs don't benefit from generaldelta
             self.version &= ~revlog.REVLOGGENERALDELTA
             self._generaldelta = False
+
+        # Delta chains for changelogs tend to be very small because entries
+        # tend to be small and don't delta well with each. So disable delta
+        # chains.
+        self.storedeltachains = False
+
         self._realopener = opener
         self._delayed = False
         self._delaybuf = None
@@ -381,9 +389,9 b' class changelog(revlog.revlog):'
             tmpname = self.indexfile + ".a"
             nfile = self.opener.open(tmpname)
             nfile.close()
-            self.opener.rename(tmpname, self.indexfile)
+            self.opener.rename(tmpname, self.indexfile, checkambig=True)
         elif self._delaybuf:
-            fp = self.opener(self.indexfile, 'a')
+            fp = self.opener(self.indexfile, 'a', checkambig=True)
             fp.write("".join(self._delaybuf))
             fp.close()
             self._delaybuf = None

mercurial/cmdutil.py

@@ -499,6 +499,12 b' class _unclosablefile(object):'
     def __getattr__(self, attr):
         return getattr(self._fp, attr)
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, exc_tb):
+        pass
+
 def makefileobj(repo, pat, node=None, desc=None, total=None,
                 seqno=None, revwidth=None, mode='wb', modemap=None,
                 pathname=None):
@@ -549,7 +555,7 b' def openrevlog(repo, cmd, file_, opts):'
             if 'treemanifest' not in repo.requirements:
                 raise error.Abort(_("--dir can only be used on repos with "
                                    "treemanifest enabled"))
-            dirlog = repo.dirlog(dir)
+            dirlog = repo.manifest.dirlog(dir)
             if len(dirlog):
                 r = dirlog
         elif mf:
@@ -640,8 +646,26 b' def copy(ui, repo, pats, opts, rename=Fa'
 
         if not after and exists or after and state in 'mn':
             if not opts['force']:
-                ui.warn(_('%s: not overwriting - file exists\n') %
-                        reltarget)
+                if state in 'mn':
+                    msg = _('%s: not overwriting - file already committed\n')
+                    if after:
+                        flags = '--after --force'
+                    else:
+                        flags = '--force'
+                    if rename:
+                        hint = _('(hg rename %s to replace the file by '
+                                 'recording a rename)\n') % flags
+                    else:
+                        hint = _('(hg copy %s to replace the file by '
+                                 'recording a copy)\n') % flags
+                else:
+                    msg = _('%s: not overwriting - file exists\n')
+                    if rename:
+                        hint = _('(hg rename --after to record the rename)\n')
+                    else:
+                        hint = _('(hg copy --after to record the copy)\n')
+                ui.warn(msg % reltarget)
+                ui.warn(hint)
                 return
 
         if after:
@@ -1611,25 +1635,26 b' def show_changeset(ui, repo, opts, buffe'
 
     return changeset_templater(ui, repo, matchfn, opts, tmpl, mapfile, buffered)
 
-def showmarker(ui, marker, index=None):
+def showmarker(fm, marker, index=None):
     """utility function to display obsolescence marker in a readable way
 
     To be used by debug function."""
     if index is not None:
-        ui.write("%i " % index)
-    ui.write(hex(marker.precnode()))
-    for repl in marker.succnodes():
-        ui.write(' ')
-        ui.write(hex(repl))
-    ui.write(' %X ' % marker.flags())
+        fm.write('index', '%i ', index)
+    fm.write('precnode', '%s ', hex(marker.precnode()))
+    succs = marker.succnodes()
+    fm.condwrite(succs, 'succnodes', '%s ',
+                 fm.formatlist(map(hex, succs), name='node'))
+    fm.write('flag', '%X ', marker.flags())
     parents = marker.parentnodes()
     if parents is not None:
-        ui.write('{%s} ' % ', '.join(hex(p) for p in parents))
-    ui.write('(%s) ' % util.datestr(marker.date()))
-    ui.write('{%s}' % (', '.join('%r: %r' % t for t in
-                                 sorted(marker.metadata().items())
-                                 if t[0] != 'date')))
-    ui.write('\n')
+        fm.write('parentnodes', '{%s} ',
+                 fm.formatlist(map(hex, parents), name='node', sep=', '))
+    fm.write('date', '(%s) ', fm.formatdate(marker.date()))
+    meta = marker.metadata().copy()
+    meta.pop('date', None)
+    fm.write('metadata', '{%s}', fm.formatdict(meta, fmt='%r: %r', sep=', '))
+    fm.plain('\n')
 
 def finddate(ui, repo, date):
     """Find the tipmost changeset that matches the given date spec"""
@@ -1940,7 +1965,7 b' def _makefollowlogfilematcher(repo, file'
     # --follow, we want the names of the ancestors of FILE in the
     # revision, stored in "fcache". "fcache" is populated by
     # reproducing the graph traversal already done by --follow revset
-    # and relating linkrevs to file names (which is not "correct" but
+    # and relating revs to file names (which is not "correct" but
     # good enough).
     fcache = {}
     fcacheready = [False]
@@ -1948,9 +1973,10 b' def _makefollowlogfilematcher(repo, file'
 
     def populate():
         for fn in files:
-            for i in ((pctx[fn],), pctx[fn].ancestors(followfirst=followfirst)):
-                for c in i:
-                    fcache.setdefault(c.linkrev(), set()).add(c.path())
+            fctx = pctx[fn]
+            fcache.setdefault(fctx.introrev(), set()).add(fctx.path())
+            for c in fctx.ancestors(followfirst=followfirst):
+                fcache.setdefault(c.rev(), set()).add(c.path())
 
     def filematcher(rev):
         if not fcacheready[0]:
@@ -2151,15 +2177,8 b' def getgraphlogrevs(repo, pats, opts):'
         if not (revs.isdescending() or revs.istopo()):
             revs.sort(reverse=True)
     if expr:
-        # Revset matchers often operate faster on revisions in changelog
-        # order, because most filters deal with the changelog.
-        revs.reverse()
-        matcher = revset.match(repo.ui, expr)
-        # Revset matches can reorder revisions. "A or B" typically returns
-        # returns the revision matching A then the revision matching B. Sort
-        # again to fix that.
+        matcher = revset.match(repo.ui, expr, order=revset.followorder)
         revs = matcher(repo, revs)
-        revs.sort(reverse=True)
     if limit is not None:
         limitedrevs = []
         for idx, rev in enumerate(revs):
@@ -2184,23 +2203,8 b' def getlogrevs(repo, pats, opts):'
         return revset.baseset([]), None, None
     expr, filematcher = _makelogrevset(repo, pats, opts, revs)
     if expr:
-        # Revset matchers often operate faster on revisions in changelog
-        # order, because most filters deal with the changelog.
-        if not opts.get('rev'):
-            revs.reverse()
-        matcher = revset.match(repo.ui, expr)
-        # Revset matches can reorder revisions. "A or B" typically returns
-        # returns the revision matching A then the revision matching B. Sort
-        # again to fix that.
-        fixopts = ['branch', 'only_branch', 'keyword', 'user']
-        oldrevs = revs
+        matcher = revset.match(repo.ui, expr, order=revset.followorder)
         revs = matcher(repo, revs)
-        if not opts.get('rev'):
-            revs.sort(reverse=True)
-        elif len(pats) > 1 or any(len(opts.get(op, [])) > 1 for op in fixopts):
-            # XXX "A or B" is known to change the order; fix it by filtering
-            # matched set again (issue5100)
-            revs = oldrevs & revs
     if limit is not None:
         limitedrevs = []
         for idx, r in enumerate(revs):
@@ -2415,14 +2419,10 b' def files(ui, ctx, m, fm, fmt, subrepos)'
         ret = 0
 
     for subpath in sorted(ctx.substate):
-        def matchessubrepo(subpath):
-            return (m.exact(subpath)
-                    or any(f.startswith(subpath + '/') for f in m.files()))
-
-        if subrepos or matchessubrepo(subpath):
+        submatch = matchmod.subdirmatcher(subpath, m)
+        if (subrepos or m.exact(subpath) or any(submatch.files())):
             sub = ctx.sub(subpath)
             try:
-                submatch = matchmod.subdirmatcher(subpath, m)
                 recurse = m.exact(subpath) or subrepos
                 if sub.printfiles(ui, submatch, fm, fmt, recurse) == 0:
                     ret = 0
@@ -2450,21 +2450,12 b' def remove(ui, repo, m, prefix, after, f'
     total = len(subs)
     count = 0
     for subpath in subs:
-        def matchessubrepo(matcher, subpath):
-            if matcher.exact(subpath):
-                return True
-            for f in matcher.files():
-                if f.startswith(subpath):
-                    return True
-            return False
-
         count += 1
-        if subrepos or matchessubrepo(m, subpath):
+        submatch = matchmod.subdirmatcher(subpath, m)
+        if subrepos or m.exact(subpath) or any(submatch.files()):
             ui.progress(_('searching'), count, total=total, unit=_('subrepos'))
-
             sub = wctx.sub(subpath)
             try:
-                submatch = matchmod.subdirmatcher(subpath, m)
                 if sub.removefiles(submatch, prefix, after, force, subrepos,
                                    warnings):
                     ret = 1
@@ -2530,8 +2521,8 b' def remove(ui, repo, m, prefix, after, f'
         for f in added:
             count += 1
             ui.progress(_('skipping'), count, total=total, unit=_('files'))
-            warnings.append(_('not removing %s: file has been marked for add'
-                      ' (use forget to undo)\n') % m.rel(f))
+            warnings.append(_("not removing %s: file has been marked for add"
+                      " (use 'hg forget' to undo add)\n") % m.rel(f))
             ret = 1
         ui.progress(_('skipping'), None)
 
@@ -2581,14 +2572,7 b' def cat(ui, repo, ctx, matcher, prefix, '
             write(file)
             return 0
 
-    # Don't warn about "missing" files that are really in subrepos
-    def badfn(path, msg):
-        for subpath in ctx.substate:
-            if path.startswith(subpath + '/'):
-                return
-        matcher.bad(path, msg)
-
-    for abs in ctx.walk(matchmod.badmatch(matcher, badfn)):
+    for abs in ctx.walk(matcher):
         write(abs)
         err = 0
 
@@ -2623,6 +2607,18 b' def commit(ui, repo, commitfunc, pats, o'
 
     return commitfunc(ui, repo, message, matcher, opts)
 
+def samefile(f, ctx1, ctx2):
+    if f in ctx1.manifest():
+        a = ctx1.filectx(f)
+        if f in ctx2.manifest():
+            b = ctx2.filectx(f)
+            return (not a.cmp(b)
+                    and a.flags() == b.flags())
+        else:
+            return False
+    else:
+        return f not in ctx2.manifest()
+
 def amend(ui, repo, commitfunc, old, extra, pats, opts):
     # avoid cycle context -> subrepo -> cmdutil
     from . import context
@@ -2706,19 +2702,7 b' def amend(ui, repo, commitfunc, old, ext'
                 # we can discard X from our list of files. Likewise if X
                 # was deleted, it's no longer relevant
                 files.update(ctx.files())
-
-                def samefile(f):
-                    if f in ctx.manifest():
-                        a = ctx.filectx(f)
-                        if f in base.manifest():
-                            b = base.filectx(f)
-                            return (not a.cmp(b)
-                                    and a.flags() == b.flags())
-                        else:
-                            return False
-                    else:
-                        return f not in base.manifest()
-                files = [f for f in files if not samefile(f)]
+                files = [f for f in files if not samefile(f, ctx, base)]
 
                 def filectxfn(repo, ctx_, path):
                     try:
@@ -3542,10 +3526,11 b' class dirstateguard(object):'
 
     def __init__(self, repo, name):
         self._repo = repo
+        self._active = False
+        self._closed = False
         self._suffix = '.backup.%s.%d' % (name, id(self))
         repo.dirstate.savebackup(repo.currenttransaction(), self._suffix)
         self._active = True
-        self._closed = False
 
     def __del__(self):
         if self._active: # still active

mercurial/commands.py

@@ -441,12 +441,12 b' def annotate(ui, repo, *pats, **opts):'
     if linenumber and (not opts.get('changeset')) and (not opts.get('number')):
         raise error.Abort(_('at least one of -n/-c is required for -l'))
 
-    if fm:
+    if fm.isplain():
+        def makefunc(get, fmt):
+            return lambda x: fmt(get(x))
+    else:
         def makefunc(get, fmt):
             return get
-    else:
-        def makefunc(get, fmt):
-            return lambda x: fmt(get(x))
     funcmap = [(makefunc(get, fmt), sep) for op, sep, get, fmt in opmap
                if opts.get(op)]
     funcmap[0] = (funcmap[0][0], '') # no separator in front of first column
@@ -476,12 +476,12 b' def annotate(ui, repo, *pats, **opts):'
 
         for f, sep in funcmap:
             l = [f(n) for n, dummy in lines]
-            if fm:
-                formats.append(['%s' for x in l])
-            else:
+            if fm.isplain():
                 sizes = [encoding.colwidth(x) for x in l]
                 ml = max(sizes)
                 formats.append([sep + ' ' * (ml - w) + '%s' for w in sizes])
+            else:
+                formats.append(['%s' for x in l])
             pieces.append(l)
 
         for f, p, l in zip(zip(*formats), zip(*pieces), lines):
@@ -835,57 +835,6 b' def bisect(ui, repo, rev=None, extra=Non'
 
     Returns 0 on success.
     """
-    def extendbisectrange(nodes, good):
-        # bisect is incomplete when it ends on a merge node and
-        # one of the parent was not checked.
-        parents = repo[nodes[0]].parents()
-        if len(parents) > 1:
-            if good:
-                side = state['bad']
-            else:
-                side = state['good']
-            num = len(set(i.node() for i in parents) & set(side))
-            if num == 1:
-                return parents[0].ancestor(parents[1])
-        return None
-
-    def print_result(nodes, good):
-        displayer = cmdutil.show_changeset(ui, repo, {})
-        if len(nodes) == 1:
-            # narrowed it down to a single revision
-            if good:
-                ui.write(_("The first good revision is:\n"))
-            else:
-                ui.write(_("The first bad revision is:\n"))
-            displayer.show(repo[nodes[0]])
-            extendnode = extendbisectrange(nodes, good)
-            if extendnode is not None:
-                ui.write(_('Not all ancestors of this changeset have been'
-                           ' checked.\nUse bisect --extend to continue the '
-                           'bisection from\nthe common ancestor, %s.\n')
-                         % extendnode)
-        else:
-            # multiple possible revisions
-            if good:
-                ui.write(_("Due to skipped revisions, the first "
-                        "good revision could be any of:\n"))
-            else:
-                ui.write(_("Due to skipped revisions, the first "
-                        "bad revision could be any of:\n"))
-            for n in nodes:
-                displayer.show(repo[n])
-        displayer.close()
-
-    def check_state(state, interactive=True):
-        if not state['good'] or not state['bad']:
-            if (good or bad or skip or reset) and interactive:
-                return
-            if not state['good']:
-                raise error.Abort(_('cannot bisect (no known good revisions)'))
-            else:
-                raise error.Abort(_('cannot bisect (no known bad revisions)'))
-        return True
-
     # backward compatibility
     if rev in "good bad reset init".split():
         ui.warn(_("(use of 'hg bisect <cmd>' is deprecated)\n"))
@@ -902,13 +851,36 b' def bisect(ui, repo, rev=None, extra=Non'
     cmdutil.checkunfinished(repo)
 
     if reset:
-        p = repo.join("bisect.state")
-        if os.path.exists(p):
-            os.unlink(p)
+        hbisect.resetstate(repo)
         return
 
     state = hbisect.load_state(repo)
 
+    # update state
+    if good or bad or skip:
+        if rev:
+            nodes = [repo.lookup(i) for i in scmutil.revrange(repo, [rev])]
+        else:
+            nodes = [repo.lookup('.')]
+        if good:
+            state['good'] += nodes
+        elif bad:
+            state['bad'] += nodes
+        elif skip:
+            state['skip'] += nodes
+        hbisect.save_state(repo, state)
+        if not (state['good'] and state['bad']):
+            return
+
+    def mayupdate(repo, node, show_stats=True):
+        """common used update sequence"""
+        if noupdate:
+            return
+        cmdutil.bailifchanged(repo)
+        return hg.clean(repo, node, show_stats=show_stats)
+
+    displayer = cmdutil.show_changeset(ui, repo, {})
+
     if command:
         changesets = 1
         if noupdate:
@@ -921,6 +893,8 b' def bisect(ui, repo, rev=None, extra=Non'
             node, p2 = repo.dirstate.parents()
             if p2 != nullid:
                 raise error.Abort(_('current bisect revision is a merge'))
+        if rev:
+            node = repo[scmutil.revsingle(repo, rev, node)].node()
         try:
             while changesets:
                 # update state
@@ -938,61 +912,38 b' def bisect(ui, repo, rev=None, extra=Non'
                     raise error.Abort(_("%s killed") % command)
                 else:
                     transition = "bad"
-                ctx = scmutil.revsingle(repo, rev, node)
-                rev = None # clear for future iterations
-                state[transition].append(ctx.node())
+                state[transition].append(node)
+                ctx = repo[node]
                 ui.status(_('changeset %d:%s: %s\n') % (ctx, ctx, transition))
-                check_state(state, interactive=False)
+                hbisect.checkstate(state)
                 # bisect
                 nodes, changesets, bgood = hbisect.bisect(repo.changelog, state)
                 # update to next check
                 node = nodes[0]
-                if not noupdate:
-                    cmdutil.bailifchanged(repo)
-                    hg.clean(repo, node, show_stats=False)
+                mayupdate(repo, node, show_stats=False)
         finally:
             state['current'] = [node]
             hbisect.save_state(repo, state)
-        print_result(nodes, bgood)
+        hbisect.printresult(ui, repo, state, displayer, nodes, bgood)
         return
 
-    # update state
-
-    if rev:
-        nodes = [repo.lookup(i) for i in scmutil.revrange(repo, [rev])]
-    else:
-        nodes = [repo.lookup('.')]
-
-    if good or bad or skip:
-        if good:
-            state['good'] += nodes
-        elif bad:
-            state['bad'] += nodes
-        elif skip:
-            state['skip'] += nodes
-        hbisect.save_state(repo, state)
-
-    if not check_state(state):
-        return
+    hbisect.checkstate(state)
 
     # actually bisect
     nodes, changesets, good = hbisect.bisect(repo.changelog, state)
     if extend:
         if not changesets:
-            extendnode = extendbisectrange(nodes, good)
+            extendnode = hbisect.extendrange(repo, state, nodes, good)
             if extendnode is not None:
                 ui.write(_("Extending search to changeset %d:%s\n")
                          % (extendnode.rev(), extendnode))
                 state['current'] = [extendnode.node()]
                 hbisect.save_state(repo, state)
-                if noupdate:
-                    return
-                cmdutil.bailifchanged(repo)
-                return hg.clean(repo, extendnode.node())
+                return mayupdate(repo, extendnode.node())
         raise error.Abort(_("nothing to extend"))
 
     if changesets == 0:
-        print_result(nodes, good)
+        hbisect.printresult(ui, repo, state, displayer, nodes, good)
     else:
         assert len(nodes) == 1 # only a single node can be tested next
         node = nodes[0]
@@ -1006,9 +957,7 b' def bisect(ui, repo, rev=None, extra=Non'
                  % (rev, short(node), changesets, tests))
         state['current'] = [node]
         hbisect.save_state(repo, state)
-        if not noupdate:
-            cmdutil.bailifchanged(repo)
-            return hg.clean(repo, node)
+        return mayupdate(repo, node)
 
 @command('bookmarks|bookmark',
     [('f', 'force', False, _('force')),
@@ -1185,7 +1134,7 b' def bookmark(ui, repo, *names, **opts):'
         fm = ui.formatter('bookmarks', opts)
         hexfn = fm.hexfunc
         marks = repo._bookmarks
-        if len(marks) == 0 and not fm:
+        if len(marks) == 0 and fm.isplain():
             ui.status(_("no bookmarks set\n"))
         for bmark, n in sorted(marks.iteritems()):
             active = repo._activebookmark
@@ -1383,8 +1332,8 b' def bundle(ui, repo, fname, dest=None, *'
                 repo, bundletype, strict=False)
     except error.UnsupportedBundleSpecification as e:
         raise error.Abort(str(e),
-                          hint=_('see "hg help bundle" for supported '
-                                 'values for --type'))
+                          hint=_("see 'hg help bundle' for supported "
+                                 "values for --type"))
 
     # Packed bundles are a pseudo bundle format for now.
     if cgversion == 's1':
@@ -1411,10 +1360,11 b' def bundle(ui, repo, fname, dest=None, *'
             raise error.Abort(_("--base is incompatible with specifying "
                                "a destination"))
         common = [repo.lookup(rev) for rev in base]
-        heads = revs and map(repo.lookup, revs) or revs
-        cg = changegroup.getchangegroup(repo, 'bundle', heads=heads,
-                                         common=common, bundlecaps=bundlecaps,
-                                         version=cgversion)
+        heads = revs and map(repo.lookup, revs) or None
+        outgoing = discovery.outgoing(repo, common, heads)
+        cg = changegroup.getchangegroup(repo, 'bundle', outgoing,
+                                        bundlecaps=bundlecaps,
+                                        version=cgversion)
         outgoing = None
     else:
         dest = ui.expandpath(dest or 'default-push', dest or 'default')
@@ -1688,9 +1638,11 b' def commit(ui, repo, *pats, **opts):'
 def _docommit(ui, repo, *pats, **opts):
     if opts.get('interactive'):
         opts.pop('interactive')
-        cmdutil.dorecord(ui, repo, commit, None, False,
-                        cmdutil.recordfilter, *pats, **opts)
-        return
+        ret = cmdutil.dorecord(ui, repo, commit, None, False,
+                               cmdutil.recordfilter, *pats, **opts)
+        # ret can be 0 (no changes to record) or the value returned by
+        # commit(), 1 if nothing changed or None on success.
+        return 1 if ret == 0 else ret
 
     if opts.get('subrepos'):
         if opts.get('amend'):
@@ -1787,7 +1739,7 b' def _docommit(ui, repo, *pats, **opts):'
     [('u', 'untrusted', None, _('show untrusted configuration options')),
      ('e', 'edit', None, _('edit user config')),
      ('l', 'local', None, _('edit repository config')),
-     ('g', 'global', None, _('edit global config'))],
+     ('g', 'global', None, _('edit global config'))] + formatteropts,
     _('[-u] [NAME]...'),
     optionalrepo=True)
 def config(ui, repo, *values, **opts):
@@ -1848,6 +1800,7 b' def config(ui, repo, *values, **opts):'
                   onerr=error.Abort, errprefix=_("edit failed"))
         return
 
+    fm = ui.formatter('config', opts)
     for f in scmutil.rcpath():
         ui.debug('read config from: %s\n' % f)
     untrusted = bool(opts.get('untrusted'))
@@ -1858,25 +1811,32 b' def config(ui, repo, *values, **opts):'
             raise error.Abort(_('only one config item permitted'))
     matched = False
     for section, name, value in ui.walkconfig(untrusted=untrusted):
-        value = str(value).replace('\n', '\\n')
-        sectname = section + '.' + name
+        value = str(value)
+        if fm.isplain():
+            value = value.replace('\n', '\\n')
+        entryname = section + '.' + name
         if values:
             for v in values:
                 if v == section:
-                    ui.debug('%s: ' %
-                             ui.configsource(section, name, untrusted))
-                    ui.write('%s=%s\n' % (sectname, value))
+                    fm.startitem()
+                    fm.condwrite(ui.debugflag, 'source', '%s: ',
+                                 ui.configsource(section, name, untrusted))
+                    fm.write('name value', '%s=%s\n', entryname, value)
                     matched = True
-                elif v == sectname:
-                    ui.debug('%s: ' %
-                             ui.configsource(section, name, untrusted))
-                    ui.write(value, '\n')
+                elif v == entryname:
+                    fm.startitem()
+                    fm.condwrite(ui.debugflag, 'source', '%s: ',
+                                 ui.configsource(section, name, untrusted))
+                    fm.write('value', '%s\n', value)
+                    fm.data(name=entryname)
                     matched = True
         else:
-            ui.debug('%s: ' %
-                     ui.configsource(section, name, untrusted))
-            ui.write('%s=%s\n' % (sectname, value))
+            fm.startitem()
+            fm.condwrite(ui.debugflag, 'source', '%s: ',
+                         ui.configsource(section, name, untrusted))
+            fm.write('name value', '%s=%s\n', entryname, value)
             matched = True
+    fm.end()
     if matched:
         return 0
     return 1
@@ -1987,8 +1947,9 b' def debugbuilddag(ui, repo, text=None,'
 
     tags = []
 
-    lock = tr = None
+    wlock = lock = tr = None
     try:
+        wlock = repo.wlock()
         lock = repo.lock()
         tr = repo.transaction("builddag")
 
@@ -2073,7 +2034,7 b' def debugbuilddag(ui, repo, text=None,'
             repo.vfs.write("localtags", "".join(tags))
     finally:
         ui.progress(_('building'), None)
-        release(tr, lock)
+        release(tr, lock, wlock)
 
 @command('debugbundle',
         [('a', 'all', None, _('show all details')),
@@ -2102,10 +2063,7 b' def _debugchangegroup(ui, gen, all=None,'
         def showchunks(named):
             ui.write("\n%s%s\n" % (indent_string, named))
             chain = None
-            while True:
-                chunkdata = gen.deltachunk(chain)
-                if not chunkdata:
-                    break
+            for chunkdata in iter(lambda: gen.deltachunk(chain), {}):
                 node = chunkdata['node']
                 p1 = chunkdata['p1']
                 p2 = chunkdata['p2']
@@ -2121,10 +2079,7 b' def _debugchangegroup(ui, gen, all=None,'
         showchunks("changelog")
         chunkdata = gen.manifestheader()
         showchunks("manifest")
-        while True:
-            chunkdata = gen.filelogheader()
-            if not chunkdata:
-                break
+        for chunkdata in iter(gen.filelogheader, {}):
             fname = chunkdata['filename']
             showchunks(fname)
     else:
@@ -2132,10 +2087,7 b' def _debugchangegroup(ui, gen, all=None,'
             raise error.Abort(_('use debugbundle2 for this file'))
         chunkdata = gen.changelogheader()
         chain = None
-        while True:
-            chunkdata = gen.deltachunk(chain)
-            if not chunkdata:
-                break
+        for chunkdata in iter(lambda: gen.deltachunk(chain), {}):
             node = chunkdata['node']
             ui.write("%s%s\n" % (indent_string, hex(node)))
             chain = node
@@ -2398,12 +2350,15 b' def debugdiscovery(ui, repo, remoteurl="'
 def debugextensions(ui, **opts):
     '''show information about active extensions'''
     exts = extensions.extensions(ui)
+    hgver = util.version()
     fm = ui.formatter('debugextensions', opts)
     for extname, extmod in sorted(exts, key=operator.itemgetter(0)):
+        isinternal = extensions.ismoduleinternal(extmod)
         extsource = extmod.__file__
-        exttestedwith = getattr(extmod, 'testedwith', None)
-        if exttestedwith is not None:
-            exttestedwith = exttestedwith.split()
+        if isinternal:
+            exttestedwith = []  # never expose magic string to users
+        else:
+            exttestedwith = getattr(extmod, 'testedwith', '').split()
         extbuglink = getattr(extmod, 'buglink', None)
 
         fm.startitem()
@@ -2412,21 +2367,24 b' def debugextensions(ui, **opts):'
             fm.write('name', '%s\n', extname)
         else:
             fm.write('name', '%s', extname)
-            if not exttestedwith:
+            if isinternal or hgver in exttestedwith:
+                fm.plain('\n')
+            elif not exttestedwith:
                 fm.plain(_(' (untested!)\n'))
             else:
-                if exttestedwith == ['internal'] or \
-                                util.version() in exttestedwith:
-                    fm.plain('\n')
-                else:
-                    lasttestedversion = exttestedwith[-1]
-                    fm.plain(' (%s!)\n' % lasttestedversion)
+                lasttestedversion = exttestedwith[-1]
+                fm.plain(' (%s!)\n' % lasttestedversion)
 
         fm.condwrite(ui.verbose and extsource, 'source',
                  _('  location: %s\n'), extsource or "")
 
+        if ui.verbose:
+            fm.plain(_('  bundled: %s\n') % ['no', 'yes'][isinternal])
+        fm.data(bundled=isinternal)
+
         fm.condwrite(ui.verbose and exttestedwith, 'testedwith',
-                 _('  tested with: %s\n'), ' '.join(exttestedwith or []))
+                     _('  tested with: %s\n'),
+                     fm.formatlist(exttestedwith, name='ver'))
 
         fm.condwrite(ui.verbose and extbuglink, 'buglink',
                  _('  bug reporting: %s\n'), extbuglink or "")
@@ -2453,7 +2411,7 b' def debugfsinfo(ui, path="."):'
     ui.write(('exec: %s\n') % (util.checkexec(path) and 'yes' or 'no'))
     ui.write(('symlink: %s\n') % (util.checklink(path) and 'yes' or 'no'))
     ui.write(('hardlink: %s\n') % (util.checknlink(path) and 'yes' or 'no'))
-    ui.write(('case-sensitive: %s\n') % (util.checkcase('.debugfsinfo')
+    ui.write(('case-sensitive: %s\n') % (util.fscasesensitive('.debugfsinfo')
                                 and 'yes' or 'no'))
     os.unlink('.debugfsinfo')
 
@@ -2832,7 +2790,7 b' def debuginstall(ui, **opts):'
     if not problems:
         fm.data(problems=problems)
     fm.condwrite(problems, 'problems',
-                 _("%s problems detected,"
+                 _("%d problems detected,"
                    " please check your install!\n"), problems)
     fm.end()
 
@@ -3054,7 +3012,7 b' def debuglocks(ui, repo, **opts):'
          ('r', 'rev', [], _('display markers relevant to REV')),
          ('', 'index', False, _('display index of the marker')),
          ('', 'delete', [], _('delete markers specified by indices')),
-        ] + commitopts2,
+        ] + commitopts2 + formatteropts,
          _('[OBSOLETED [REPLACEMENT ...]]'))
 def debugobsolete(ui, repo, precursor=None, *successors, **opts):
     """create arbitrary obsolete marker
@@ -3142,6 +3100,7 b' def debugobsolete(ui, repo, precursor=No'
             markerset = set(markers)
             isrelevant = lambda m: m in markerset
 
+        fm = ui.formatter('debugobsolete', opts)
         for i, m in enumerate(markerstoiter):
             if not isrelevant(m):
                 # marker can be irrelevant when we're iterating over a set
@@ -3152,8 +3111,10 b' def debugobsolete(ui, repo, precursor=No'
                 # to get the correct indices, but only display the ones that
                 # are relevant to --rev value
                 continue
+            fm.startitem()
             ind = i if opts.get('index') else None
-            cmdutil.showmarker(ui, m, index=ind)
+            cmdutil.showmarker(fm, m, index=ind)
+        fm.end()
 
 @command('debugpathcomplete',
          [('f', 'full', None, _('complete an entire path')),
@@ -3382,9 +3343,9 b' def debugrevlog(ui, repo, file_=None, **'
     nump2prev = 0
     chainlengths = []
 
-    datasize = [None, 0, 0L]
-    fullsize = [None, 0, 0L]
-    deltasize = [None, 0, 0L]
+    datasize = [None, 0, 0]
+    fullsize = [None, 0, 0]
+    deltasize = [None, 0, 0]
 
     def addsize(size, l):
         if l[0] is None or size < l[0]:
@@ -3509,29 +3470,92 b' def debugrevlog(ui, repo, file_=None, **'
                                                              numdeltas))
 
 @command('debugrevspec',
-    [('', 'optimize', None, _('print parsed tree after optimizing'))],
+    [('', 'optimize', None,
+      _('print parsed tree after optimizing (DEPRECATED)')),
+     ('p', 'show-stage', [],
+      _('print parsed tree at the given stage'), _('NAME')),
+     ('', 'no-optimized', False, _('evaluate tree without optimization')),
+     ('', 'verify-optimized', False, _('verify optimized result')),
+     ],
     ('REVSPEC'))
 def debugrevspec(ui, repo, expr, **opts):
     """parse and apply a revision specification
 
-    Use --verbose to print the parsed tree before and after aliases
-    expansion.
+    Use -p/--show-stage option to print the parsed tree at the given stages.
+    Use -p all to print tree at every stage.
+
+    Use --verify-optimized to compare the optimized result with the unoptimized
+    one. Returns 1 if the optimized result differs.
     """
-    if ui.verbose:
-        tree = revset.parse(expr, lookup=repo.__contains__)
-        ui.note(revset.prettyformat(tree), "\n")
-        newtree = revset.expandaliases(ui, tree)
-        if newtree != tree:
-            ui.note(("* expanded:\n"), revset.prettyformat(newtree), "\n")
-        tree = newtree
-        newtree = revset.foldconcat(tree)
-        if newtree != tree:
-            ui.note(("* concatenated:\n"), revset.prettyformat(newtree), "\n")
-        if opts["optimize"]:
-            optimizedtree = revset.optimize(newtree)
-            ui.note(("* optimized:\n"),
-                    revset.prettyformat(optimizedtree), "\n")
-    func = revset.match(ui, expr, repo)
+    stages = [
+        ('parsed', lambda tree: tree),
+        ('expanded', lambda tree: revset.expandaliases(ui, tree)),
+        ('concatenated', revset.foldconcat),
+        ('analyzed', revset.analyze),
+        ('optimized', revset.optimize),
+    ]
+    if opts['no_optimized']:
+        stages = stages[:-1]
+    if opts['verify_optimized'] and opts['no_optimized']:
+        raise error.Abort(_('cannot use --verify-optimized with '
+                            '--no-optimized'))
+    stagenames = set(n for n, f in stages)
+
+    showalways = set()
+    showchanged = set()
+    if ui.verbose and not opts['show_stage']:
+        # show parsed tree by --verbose (deprecated)
+        showalways.add('parsed')
+        showchanged.update(['expanded', 'concatenated'])
+        if opts['optimize']:
+            showalways.add('optimized')
+    if opts['show_stage'] and opts['optimize']:
+        raise error.Abort(_('cannot use --optimize with --show-stage'))
+    if opts['show_stage'] == ['all']:
+        showalways.update(stagenames)
+    else:
+        for n in opts['show_stage']:
+            if n not in stagenames:
+                raise error.Abort(_('invalid stage name: %s') % n)
+        showalways.update(opts['show_stage'])
+
+    treebystage = {}
+    printedtree = None
+    tree = revset.parse(expr, lookup=repo.__contains__)
+    for n, f in stages:
+        treebystage[n] = tree = f(tree)
+        if n in showalways or (n in showchanged and tree != printedtree):
+            if opts['show_stage'] or n != 'parsed':
+                ui.write(("* %s:\n") % n)
+            ui.write(revset.prettyformat(tree), "\n")
+            printedtree = tree
+
+    if opts['verify_optimized']:
+        arevs = revset.makematcher(treebystage['analyzed'])(repo)
+        brevs = revset.makematcher(treebystage['optimized'])(repo)
+        if ui.verbose:
+            ui.note(("* analyzed set:\n"), revset.prettyformatset(arevs), "\n")
+            ui.note(("* optimized set:\n"), revset.prettyformatset(brevs), "\n")
+        arevs = list(arevs)
+        brevs = list(brevs)
+        if arevs == brevs:
+            return 0
+        ui.write(('--- analyzed\n'), label='diff.file_a')
+        ui.write(('+++ optimized\n'), label='diff.file_b')
+        sm = difflib.SequenceMatcher(None, arevs, brevs)
+        for tag, alo, ahi, blo, bhi in sm.get_opcodes():
+            if tag in ('delete', 'replace'):
+                for c in arevs[alo:ahi]:
+                    ui.write('-%s\n' % c, label='diff.deleted')
+            if tag in ('insert', 'replace'):
+                for c in brevs[blo:bhi]:
+                    ui.write('+%s\n' % c, label='diff.inserted')
+            if tag == 'equal':
+                for c in arevs[alo:ahi]:
+                    ui.write(' %s\n' % c)
+        return 1
+
+    func = revset.makematcher(tree)
     revs = func(repo)
     if ui.verbose:
         ui.note(("* set:\n"), revset.prettyformatset(revs), "\n")
@@ -3914,16 +3938,16 b' def export(ui, repo, *changesets, **opts'
     [('r', 'rev', '', _('search the repository as it is in REV'), _('REV')),
      ('0', 'print0', None, _('end filenames with NUL, for use with xargs')),
     ] + walkopts + formatteropts + subrepoopts,
-    _('[OPTION]... [PATTERN]...'))
+    _('[OPTION]... [FILE]...'))
 def files(ui, repo, *pats, **opts):
     """list tracked files
 
     Print files under Mercurial control in the working directory or
-    specified revision whose names match the given patterns (excluding
-    removed files).
-
-    If no patterns are given to match, this command prints the names
-    of all files under Mercurial control in the working directory.
+    specified revision for given files (excluding removed files).
+    Files can be specified as filenames or filesets.
+
+    If no files are given to match, this command prints the names
+    of all files under Mercurial control.
 
     .. container:: verbose
 
@@ -3964,15 +3988,11 b' def files(ui, repo, *pats, **opts):'
     end = '\n'
     if opts.get('print0'):
         end = '\0'
-    fm = ui.formatter('files', opts)
     fmt = '%s' + end
 
     m = scmutil.match(ctx, pats, opts)
-    ret = cmdutil.files(ui, ctx, m, fm, fmt, opts.get('subrepos'))
-
-    fm.end()
-
-    return ret
+    with ui.formatter('files', opts) as fm:
+        return cmdutil.files(ui, ctx, m, fm, fmt, opts.get('subrepos'))
 
 @command('^forget', walkopts, _('[OPTION]... FILE...'), inferrepo=True)
 def forget(ui, repo, *pats, **opts):
@@ -4142,9 +4162,7 b' def _dograft(ui, repo, *revs, **opts):'
         # check for ancestors of dest branch
         crev = repo['.'].rev()
         ancestors = repo.changelog.ancestors([crev], inclusive=True)
-        # Cannot use x.remove(y) on smart set, this has to be a list.
         # XXX make this lazy in the future
-        revs = list(revs)
         # don't mutate while iterating, create a copy
         for rev in list(revs):
             if rev in ancestors:
@@ -4284,7 +4302,7 b' def _dograft(ui, repo, *revs, **opts):'
      _('only search files changed within revision range'), _('REV')),
     ('u', 'user', None, _('list the author (long with -v)')),
     ('d', 'date', None, _('list the date (short with -q)')),
-    ] + walkopts,
+    ] + formatteropts + walkopts,
     _('[OPTION]... PATTERN [FILE]...'),
     inferrepo=True)
 def grep(ui, repo, pattern, *pats, **opts):
@@ -4349,19 +4367,16 b' def grep(ui, repo, pattern, *pats, **opt'
         def __eq__(self, other):
             return self.line == other.line
 
-        def __iter__(self):
-            yield (self.line[:self.colstart], '')
-            yield (self.line[self.colstart:self.colend], 'grep.match')
-            rest = self.line[self.colend:]
-            while rest != '':
-                match = regexp.search(rest)
-                if not match:
-                    yield (rest, '')
+        def findpos(self):
+            """Iterate all (start, end) indices of matches"""
+            yield self.colstart, self.colend
+            p = self.colend
+            while p < len(self.line):
+                m = regexp.search(self.line, p)
+                if not m:
                     break
-                mstart, mend = match.span()
-                yield (rest[:mstart], '')
-                yield (rest[mstart:mend], 'grep.match')
-                rest = rest[mend:]
+                yield m.span()
+                p = m.end()
 
     matches = {}
     copies = {}
@@ -4387,50 +4402,76 b' def grep(ui, repo, pattern, *pats, **opt'
                 for i in xrange(blo, bhi):
                     yield ('+', b[i])
 
-    def display(fn, ctx, pstates, states):
+    def display(fm, fn, ctx, pstates, states):
         rev = ctx.rev()
+        if fm.isplain():
+            formatuser = ui.shortuser
+        else:
+            formatuser = str
         if ui.quiet:
-            datefunc = util.shortdate
+            datefmt = '%Y-%m-%d'
         else:
-            datefunc = util.datestr
+            datefmt = '%a %b %d %H:%M:%S %Y %1%2'
         found = False
         @util.cachefunc
         def binary():
             flog = getfile(fn)
             return util.binary(flog.read(ctx.filenode(fn)))
 
+        fieldnamemap = {'filename': 'file', 'linenumber': 'line_number'}
         if opts.get('all'):
             iter = difflinestates(pstates, states)
         else:
             iter = [('', l) for l in states]
         for change, l in iter:
-            cols = [(fn, 'grep.filename'), (str(rev), 'grep.rev')]
-
-            if opts.get('line_number'):
-                cols.append((str(l.linenum), 'grep.linenumber'))
+            fm.startitem()
+            fm.data(node=fm.hexfunc(ctx.node()))
+            cols = [
+                ('filename', fn, True),
+                ('rev', rev, True),
+                ('linenumber', l.linenum, opts.get('line_number')),
+            ]
             if opts.get('all'):
-                cols.append((change, 'grep.change'))
-            if opts.get('user'):
-                cols.append((ui.shortuser(ctx.user()), 'grep.user'))
-            if opts.get('date'):
-                cols.append((datefunc(ctx.date()), 'grep.date'))
-            for col, label in cols[:-1]:
-                ui.write(col, label=label)
-                ui.write(sep, label='grep.sep')
-            ui.write(cols[-1][0], label=cols[-1][1])
+                cols.append(('change', change, True))
+            cols.extend([
+                ('user', formatuser(ctx.user()), opts.get('user')),
+                ('date', fm.formatdate(ctx.date(), datefmt), opts.get('date')),
+            ])
+            lastcol = next(name for name, data, cond in reversed(cols) if cond)
+            for name, data, cond in cols:
+                field = fieldnamemap.get(name, name)
+                fm.condwrite(cond, field, '%s', data, label='grep.%s' % name)
+                if cond and name != lastcol:
+                    fm.plain(sep, label='grep.sep')
             if not opts.get('files_with_matches'):
-                ui.write(sep, label='grep.sep')
+                fm.plain(sep, label='grep.sep')
                 if not opts.get('text') and binary():
-                    ui.write(_(" Binary file matches"))
+                    fm.plain(_(" Binary file matches"))
                 else:
-                    for s, label in l:
-                        ui.write(s, label=label)
-            ui.write(eol)
+                    displaymatches(fm.nested('texts'), l)
+            fm.plain(eol)
             found = True
             if opts.get('files_with_matches'):
                 break
         return found
 
+    def displaymatches(fm, l):
+        p = 0
+        for s, e in l.findpos():
+            if p < s:
+                fm.startitem()
+                fm.write('text', '%s', l.line[p:s])
+                fm.data(matched=False)
+            fm.startitem()
+            fm.write('text', '%s', l.line[s:e], label='grep.match')
+            fm.data(matched=True)
+            p = e
+        if p < len(l.line):
+            fm.startitem()
+            fm.write('text', '%s', l.line[p:])
+            fm.data(matched=False)
+        fm.end()
+
     skip = {}
     revfiles = {}
     matchfn = scmutil.match(repo[None], pats, opts)
@@ -4472,6 +4513,7 b' def grep(ui, repo, pattern, *pats, **opt'
                 except error.LookupError:
                     pass
 
+    fm = ui.formatter('grep', opts)
     for ctx in cmdutil.walkchangerevs(repo, matchfn, opts, prep):
         rev = ctx.rev()
         parent = ctx.p1().rev()
@@ -4484,7 +4526,7 b' def grep(ui, repo, pattern, *pats, **opt'
                 continue
             pstates = matches.get(parent, {}).get(copy or fn, [])
             if pstates or states:
-                r = display(fn, ctx, pstates, states)
+                r = display(fm, fn, ctx, pstates, states)
                 found = found or r
                 if r and not opts.get('all'):
                     skip[fn] = True
@@ -4492,6 +4534,7 b' def grep(ui, repo, pattern, *pats, **opt'
                         skip[copy] = True
         del matches[rev]
         del revfiles[rev]
+    fm.end()
 
     return not found
 
@@ -4607,12 +4650,15 b' def help_(ui, name=None, **opts):'
     section = None
     subtopic = None
     if name and '.' in name:
-        name, section = name.split('.', 1)
-        section = encoding.lower(section)
-        if '.' in section:
-            subtopic, section = section.split('.', 1)
+        name, remaining = name.split('.', 1)
+        remaining = encoding.lower(remaining)
+        if '.' in remaining:
+            subtopic, section = remaining.split('.', 1)
         else:
-            subtopic = section
+            if name in help.subtopics:
+                subtopic = remaining
+            else:
+                section = remaining
 
     text = help.help_(ui, name, subtopic=subtopic, **opts)
 
@@ -5437,7 +5483,9 b' def merge(ui, repo, node=None, **opts):'
         # ui.forcemerge is an internal variable, do not document
         repo.ui.setconfig('ui', 'forcemerge', opts.get('tool', ''), 'merge')
         force = opts.get('force')
-        return hg.merge(repo, node, force=force, mergeforce=force)
+        labels = ['working copy', 'merge rev']
+        return hg.merge(repo, node, force=force, mergeforce=force,
+                        labels=labels)
     finally:
         ui.setconfig('ui', 'forcemerge', '', 'merge')
 
@@ -5611,10 +5659,10 b' def paths(ui, repo, search=None, **opts)'
         pathitems = sorted(ui.paths.iteritems())
 
     fm = ui.formatter('paths', opts)
-    if fm:
+    if fm.isplain():
+        hidepassword = util.hidepassword
+    else:
         hidepassword = str
-    else:
-        hidepassword = util.hidepassword
     if ui.quiet:
         namefmt = '%s\n'
     else:
@@ -5936,7 +5984,7 b' def push(ui, repo, dest=None, **opts):'
     path = ui.paths.getpath(dest, default=('default-push', 'default'))
     if not path:
         raise error.Abort(_('default repository not configured!'),
-                         hint=_('see the "path" section in "hg help config"'))
+                         hint=_("see 'hg help config.paths'"))
     dest = path.pushloc or path.loc
     branches = (path.branch, opts.get('branch') or [])
     ui.status(_('pushing to %s\n') % util.hidepassword(dest))
@@ -6459,7 +6507,7 b' def root(ui, repo):'
     ('n', 'name', '',
      _('name to show in web pages (default: working directory)'), _('NAME')),
     ('', 'web-conf', '',
-     _('name of the hgweb config file (see "hg help hgweb")'), _('FILE')),
+     _("name of the hgweb config file (see 'hg help hgweb')"), _('FILE')),
     ('', 'webdir-conf', '', _('name of the hgweb config file (DEPRECATED)'),
      _('FILE')),
     ('', 'pid-file', '', _('name of file to write process ID to'), _('FILE')),
@@ -7247,37 +7295,48 b' def verify(ui, repo):'
     """
     return hg.verify(repo)
 
-@command('version', [], norepo=True)
-def version_(ui):
+@command('version', [] + formatteropts, norepo=True)
+def version_(ui, **opts):
     """output version and copyright information"""
-    ui.write(_("Mercurial Distributed SCM (version %s)\n")
-             % util.version())
-    ui.status(_(
+    fm = ui.formatter("version", opts)
+    fm.startitem()
+    fm.write("ver", _("Mercurial Distributed SCM (version %s)\n"),
+             util.version())
+    license = _(
         "(see https://mercurial-scm.org for more information)\n"
         "\nCopyright (C) 2005-2016 Matt Mackall and others\n"
         "This is free software; see the source for copying conditions. "
         "There is NO\nwarranty; "
         "not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n"
-    ))
-
-    ui.note(_("\nEnabled extensions:\n\n"))
+    )
+    if not ui.quiet:
+        fm.plain(license)
+
     if ui.verbose:
-        # format names and versions into columns
-        names = []
-        vers = []
-        place = []
-        for name, module in extensions.extensions():
-            names.append(name)
-            vers.append(extensions.moduleversion(module))
-            if extensions.ismoduleinternal(module):
-                place.append(_("internal"))
-            else:
-                place.append(_("external"))
-        if names:
-            maxnamelen = max(len(n) for n in names)
-            for i, name in enumerate(names):
-                ui.write("  %-*s  %s  %s\n" %
-                         (maxnamelen, name, place[i], vers[i]))
+        fm.plain(_("\nEnabled extensions:\n\n"))
+    # format names and versions into columns
+    names = []
+    vers = []
+    isinternals = []
+    for name, module in extensions.extensions():
+        names.append(name)
+        vers.append(extensions.moduleversion(module) or None)
+        isinternals.append(extensions.ismoduleinternal(module))
+    fn = fm.nested("extensions")
+    if names:
+        namefmt = "  %%-%ds  " % max(len(n) for n in names)
+        places = [_("external"), _("internal")]
+        for n, v, p in zip(names, vers, isinternals):
+            fn.startitem()
+            fn.condwrite(ui.verbose, "name", namefmt, n)
+            if ui.verbose:
+                fn.plain("%s  " % places[p])
+            fn.data(bundled=p)
+            fn.condwrite(ui.verbose and v, "ver", "%s", v)
+            if ui.verbose:
+                fn.plain("\n")
+    fn.end()
+    fm.end()
 
 def loadcmdtable(ui, name, cmdtable):
     """Load command functions from specified cmdtable

mercurial/context.py

@@ -528,11 +528,12 b' class changectx(basectx):'
 
     @propertycache
     def _manifest(self):
-        return self._repo.manifest.read(self._changeset.manifest)
+        return self._repo.manifestlog[self._changeset.manifest].read()
 
     @propertycache
     def _manifestdelta(self):
-        return self._repo.manifest.readdelta(self._changeset.manifest)
+        mfnode = self._changeset.manifest
+        return self._repo.manifestlog[mfnode].readdelta()
 
     @propertycache
     def _parents(self):
@@ -823,7 +824,7 b' class basefilectx(object):'
         """
         repo = self._repo
         cl = repo.unfiltered().changelog
-        ma = repo.manifest
+        mfl = repo.manifestlog
         # fetch the linkrev
         fr = filelog.rev(fnode)
         lkr = filelog.linkrev(fr)
@@ -848,7 +849,7 b' class basefilectx(object):'
                 if path in ac[3]: # checking the 'files' field.
                     # The file has been touched, check if the content is
                     # similar to the one we search for.
-                    if fnode == ma.readfast(ac[0]).get(path):
+                    if fnode == mfl[ac[0]].readfast().get(path):
                         return a
             # In theory, we should never get out of that loop without a result.
             # But if manifest uses a buggy file revision (not children of the
@@ -929,7 +930,7 b' class basefilectx(object):'
         def lines(text):
             if text.endswith("\n"):
                 return text.count("\n")
-            return text.count("\n") + 1
+            return text.count("\n") + int(bool(text))
 
         if linenumber:
             def decorate(text, rev):
@@ -939,8 +940,7 b' class basefilectx(object):'
                 return ([(rev, False)] * lines(text), text)
 
         def pair(parent, child):
-            blocks = mdiff.allblocks(parent[1], child[1], opts=diffopts,
-                                     refine=True)
+            blocks = mdiff.allblocks(parent[1], child[1], opts=diffopts)
             for (a1, a2, b1, b2), t in blocks:
                 # Changed blocks ('!') or blocks made only of blank lines ('~')
                 # belong to the child.
@@ -1508,7 +1508,7 b' class workingctx(committablectx):'
 
         # Only a case insensitive filesystem needs magic to translate user input
         # to actual case in the filesystem.
-        if not util.checkcase(r.root):
+        if not util.fscasesensitive(r.root):
             return matchmod.icasefsmatcher(r.root, r.getcwd(), pats, include,
                                            exclude, default, r.auditor, self,
                                            listsubrepos=listsubrepos,

mercurial/copies.py

@@ -231,30 +231,34 b' def pathcopies(x, y, match=None):'
     return _chain(x, y, _backwardrenames(x, a),
                   _forwardcopies(a, y, match=match))
 
-def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2):
+def _computenonoverlap(repo, c1, c2, addedinm1, addedinm2, baselabel=''):
     """Computes, based on addedinm1 and addedinm2, the files exclusive to c1
     and c2. This is its own function so extensions can easily wrap this call
     to see what files mergecopies is about to process.
 
     Even though c1 and c2 are not used in this function, they are useful in
     other extensions for being able to read the file nodes of the changed files.
+
+    "baselabel" can be passed to help distinguish the multiple computations
+    done in the graft case.
     """
     u1 = sorted(addedinm1 - addedinm2)
     u2 = sorted(addedinm2 - addedinm1)
 
+    header = "  unmatched files in %s"
+    if baselabel:
+        header += ' (from %s)' % baselabel
     if u1:
-        repo.ui.debug("  unmatched files in local:\n   %s\n"
-                      % "\n   ".join(u1))
+        repo.ui.debug("%s:\n   %s\n" % (header % 'local', "\n   ".join(u1)))
     if u2:
-        repo.ui.debug("  unmatched files in other:\n   %s\n"
-                      % "\n   ".join(u2))
+        repo.ui.debug("%s:\n   %s\n" % (header % 'other', "\n   ".join(u2)))
     return u1, u2
 
 def _makegetfctx(ctx):
-    """return a 'getfctx' function suitable for checkcopies usage
+    """return a 'getfctx' function suitable for _checkcopies usage
 
     We have to re-setup the function building 'filectx' for each
-    'checkcopies' to ensure the linkrev adjustment is properly setup for
+    '_checkcopies' to ensure the linkrev adjustment is properly setup for
     each. Linkrev adjustment is important to avoid bug in rename
     detection. Moreover, having a proper '_ancestrycontext' setup ensures
     the performance impact of this adjustment is kept limited. Without it,
@@ -285,10 +289,26 b' def _makegetfctx(ctx):'
         return fctx
     return util.lrucachefunc(makectx)
 
-def mergecopies(repo, c1, c2, ca):
+def _combinecopies(copyfrom, copyto, finalcopy, diverge, incompletediverge):
+    """combine partial copy paths"""
+    remainder = {}
+    for f in copyfrom:
+        if f in copyto:
+            finalcopy[copyto[f]] = copyfrom[f]
+            del copyto[f]
+    for f in incompletediverge:
+        assert f not in diverge
+        ic = incompletediverge[f]
+        if ic[0] in copyto:
+            diverge[f] = [copyto[ic[0]], ic[1]]
+        else:
+            remainder[f] = ic
+    return remainder
+
+def mergecopies(repo, c1, c2, base):
     """
     Find moves and copies between context c1 and c2 that are relevant
-    for merging.
+    for merging. 'base' will be used as the merge base.
 
     Returns four dicts: "copy", "movewithdir", "diverge", and
     "renamedelete".
@@ -321,6 +341,27 b' def mergecopies(repo, c1, c2, ca):'
     if repo.ui.configbool('experimental', 'disablecopytrace'):
         return {}, {}, {}, {}
 
+    # In certain scenarios (e.g. graft, update or rebase), base can be
+    # overridden We still need to know a real common ancestor in this case We
+    # can't just compute _c1.ancestor(_c2) and compare it to ca, because there
+    # can be multiple common ancestors, e.g. in case of bidmerge.  Because our
+    # caller may not know if the revision passed in lieu of the CA is a genuine
+    # common ancestor or not without explicitly checking it, it's better to
+    # determine that here.
+    #
+    # base.descendant(wc) and base.descendant(base) are False, work around that
+    _c1 = c1.p1() if c1.rev() is None else c1
+    _c2 = c2.p1() if c2.rev() is None else c2
+    # an endpoint is "dirty" if it isn't a descendant of the merge base
+    # if we have a dirty endpoint, we need to trigger graft logic, and also
+    # keep track of which endpoint is dirty
+    dirtyc1 = not (base == _c1 or base.descendant(_c1))
+    dirtyc2 = not (base== _c2 or base.descendant(_c2))
+    graft = dirtyc1 or dirtyc2
+    tca = base
+    if graft:
+        tca = _c1.ancestor(_c2)
+
     limit = _findlimit(repo, c1.rev(), c2.rev())
     if limit is None:
         # no common ancestor, no copies
@@ -329,28 +370,63 b' def mergecopies(repo, c1, c2, ca):'
 
     m1 = c1.manifest()
     m2 = c2.manifest()
-    ma = ca.manifest()
+    mb = base.manifest()
 
-    copy1, copy2, = {}, {}
-    movewithdir1, movewithdir2 = {}, {}
-    fullcopy1, fullcopy2 = {}, {}
-    diverge = {}
+    # gather data from _checkcopies:
+    # - diverge = record all diverges in this dict
+    # - copy = record all non-divergent copies in this dict
+    # - fullcopy = record all copies in this dict
+    # - incomplete = record non-divergent partial copies here
+    # - incompletediverge = record divergent partial copies here
+    diverge = {} # divergence data is shared
+    incompletediverge  = {}
+    data1 = {'copy': {},
+             'fullcopy': {},
+             'incomplete': {},
+             'diverge': diverge,
+             'incompletediverge': incompletediverge,
+            }
+    data2 = {'copy': {},
+             'fullcopy': {},
+             'incomplete': {},
+             'diverge': diverge,
+             'incompletediverge': incompletediverge,
+            }
 
     # find interesting file sets from manifests
-    addedinm1 = m1.filesnotin(ma)
-    addedinm2 = m2.filesnotin(ma)
-    u1, u2 = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
+    addedinm1 = m1.filesnotin(mb)
+    addedinm2 = m2.filesnotin(mb)
     bothnew = sorted(addedinm1 & addedinm2)
+    if tca == base:
+        # unmatched file from base
+        u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2)
+        u1u, u2u = u1r, u2r
+    else:
+        # unmatched file from base (DAG rotation in the graft case)
+        u1r, u2r = _computenonoverlap(repo, c1, c2, addedinm1, addedinm2,
+                                      baselabel='base')
+        # unmatched file from topological common ancestors (no DAG rotation)
+        # need to recompute this for directory move handling when grafting
+        mta = tca.manifest()
+        u1u, u2u = _computenonoverlap(repo, c1, c2, m1.filesnotin(mta),
+                                                    m2.filesnotin(mta),
+                                      baselabel='topological common ancestor')
 
-    for f in u1:
-        checkcopies(c1, f, m1, m2, ca, limit, diverge, copy1, fullcopy1)
+    for f in u1u:
+        _checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, data1)
+
+    for f in u2u:
+        _checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, data2)
 
-    for f in u2:
-        checkcopies(c2, f, m2, m1, ca, limit, diverge, copy2, fullcopy2)
+    copy = dict(data1['copy'].items() + data2['copy'].items())
+    fullcopy = dict(data1['fullcopy'].items() + data2['fullcopy'].items())
 
-    copy = dict(copy1.items() + copy2.items())
-    movewithdir = dict(movewithdir1.items() + movewithdir2.items())
-    fullcopy = dict(fullcopy1.items() + fullcopy2.items())
+    if dirtyc1:
+        _combinecopies(data2['incomplete'], data1['incomplete'], copy, diverge,
+                       incompletediverge)
+    else:
+        _combinecopies(data1['incomplete'], data2['incomplete'], copy, diverge,
+                       incompletediverge)
 
     renamedelete = {}
     renamedeleteset = set()
@@ -369,10 +445,44 b' def mergecopies(repo, c1, c2, ca):'
     if bothnew:
         repo.ui.debug("  unmatched files new in both:\n   %s\n"
                       % "\n   ".join(bothnew))
-    bothdiverge, _copy, _fullcopy = {}, {}, {}
+    bothdiverge = {}
+    bothincompletediverge = {}
+    remainder = {}
+    both1 = {'copy': {},
+             'fullcopy': {},
+             'incomplete': {},
+             'diverge': bothdiverge,
+             'incompletediverge': bothincompletediverge
+            }
+    both2 = {'copy': {},
+             'fullcopy': {},
+             'incomplete': {},
+             'diverge': bothdiverge,
+             'incompletediverge': bothincompletediverge
+            }
     for f in bothnew:
-        checkcopies(c1, f, m1, m2, ca, limit, bothdiverge, _copy, _fullcopy)
-        checkcopies(c2, f, m2, m1, ca, limit, bothdiverge, _copy, _fullcopy)
+        _checkcopies(c1, f, m1, m2, base, tca, dirtyc1, limit, both1)
+        _checkcopies(c2, f, m2, m1, base, tca, dirtyc2, limit, both2)
+    if dirtyc1:
+        # incomplete copies may only be found on the "dirty" side for bothnew
+        assert not both2['incomplete']
+        remainder = _combinecopies({}, both1['incomplete'], copy, bothdiverge,
+                                   bothincompletediverge)
+    elif dirtyc2:
+        assert not both1['incomplete']
+        remainder = _combinecopies({}, both2['incomplete'], copy, bothdiverge,
+                                   bothincompletediverge)
+    else:
+        # incomplete copies and divergences can't happen outside grafts
+        assert not both1['incomplete']
+        assert not both2['incomplete']
+        assert not bothincompletediverge
+    for f in remainder:
+        assert f not in bothdiverge
+        ic = remainder[f]
+        if ic[0] in (m1 if dirtyc1 else m2):
+            # backed-out rename on one side, but watch out for deleted files
+            bothdiverge[f] = ic
     for of, fl in bothdiverge.items():
         if len(fl) == 2 and fl[0] == fl[1]:
             copy[fl[0]] = of # not actually divergent, just matching renames
@@ -393,7 +503,7 b' def mergecopies(repo, c1, c2, ca):'
     del divergeset
 
     if not fullcopy:
-        return copy, movewithdir, diverge, renamedelete
+        return copy, {}, diverge, renamedelete
 
     repo.ui.debug("  checking for directory renames\n")
 
@@ -431,14 +541,15 b' def mergecopies(repo, c1, c2, ca):'
     del d1, d2, invalid
 
     if not dirmove:
-        return copy, movewithdir, diverge, renamedelete
+        return copy, {}, diverge, renamedelete
 
     for d in dirmove:
         repo.ui.debug("   discovered dir src: '%s' -> dst: '%s'\n" %
                       (d, dirmove[d]))
 
+    movewithdir = {}
     # check unaccounted nonoverlapping files against directory moves
-    for f in u1 + u2:
+    for f in u1r + u2r:
         if f not in fullcopy:
             for d in dirmove:
                 if f.startswith(d):
@@ -452,55 +563,74 b' def mergecopies(repo, c1, c2, ca):'
 
     return copy, movewithdir, diverge, renamedelete
 
-def checkcopies(ctx, f, m1, m2, ca, limit, diverge, copy, fullcopy):
+def _related(f1, f2, limit):
+    """return True if f1 and f2 filectx have a common ancestor
+
+    Walk back to common ancestor to see if the two files originate
+    from the same file. Since workingfilectx's rev() is None it messes
+    up the integer comparison logic, hence the pre-step check for
+    None (f1 and f2 can only be workingfilectx's initially).
+    """
+
+    if f1 == f2:
+        return f1 # a match
+
+    g1, g2 = f1.ancestors(), f2.ancestors()
+    try:
+        f1r, f2r = f1.linkrev(), f2.linkrev()
+
+        if f1r is None:
+            f1 = next(g1)
+        if f2r is None:
+            f2 = next(g2)
+
+        while True:
+            f1r, f2r = f1.linkrev(), f2.linkrev()
+            if f1r > f2r:
+                f1 = next(g1)
+            elif f2r > f1r:
+                f2 = next(g2)
+            elif f1 == f2:
+                return f1 # a match
+            elif f1r == f2r or f1r < limit or f2r < limit:
+                return False # copy no longer relevant
+    except StopIteration:
+        return False
+
+def _checkcopies(ctx, f, m1, m2, base, tca, remotebase, limit, data):
     """
     check possible copies of f from m1 to m2
 
     ctx = starting context for f in m1
-    f = the filename to check
+    f = the filename to check (as in m1)
     m1 = the source manifest
     m2 = the destination manifest
-    ca = the changectx of the common ancestor
+    base = the changectx used as a merge base
+    tca = topological common ancestor for graft-like scenarios
+    remotebase = True if base is outside tca::ctx, False otherwise
     limit = the rev number to not search beyond
-    diverge = record all diverges in this dict
-    copy = record all non-divergent copies in this dict
-    fullcopy = record all copies in this dict
+    data = dictionary of dictionary to store copy data. (see mergecopies)
+
+    note: limit is only an optimization, and there is no guarantee that
+    irrelevant revisions will not be limited
+    there is no easy way to make this algorithm stop in a guaranteed way
+    once it "goes behind a certain revision".
     """
 
-    ma = ca.manifest()
+    mb = base.manifest()
+    mta = tca.manifest()
+    # Might be true if this call is about finding backward renames,
+    # This happens in the case of grafts because the DAG is then rotated.
+    # If the file exists in both the base and the source, we are not looking
+    # for a rename on the source side, but on the part of the DAG that is
+    # traversed backwards.
+    #
+    # In the case there is both backward and forward renames (before and after
+    # the base) this is more complicated as we must detect a divergence.
+    # We use 'backwards = False' in that case.
+    backwards = not remotebase and base != tca and f in mb
     getfctx = _makegetfctx(ctx)
 
-    def _related(f1, f2, limit):
-        # Walk back to common ancestor to see if the two files originate
-        # from the same file. Since workingfilectx's rev() is None it messes
-        # up the integer comparison logic, hence the pre-step check for
-        # None (f1 and f2 can only be workingfilectx's initially).
-
-        if f1 == f2:
-            return f1 # a match
-
-        g1, g2 = f1.ancestors(), f2.ancestors()
-        try:
-            f1r, f2r = f1.linkrev(), f2.linkrev()
-
-            if f1r is None:
-                f1 = next(g1)
-            if f2r is None:
-                f2 = next(g2)
-
-            while True:
-                f1r, f2r = f1.linkrev(), f2.linkrev()
-                if f1r > f2r:
-                    f1 = next(g1)
-                elif f2r > f1r:
-                    f2 = next(g2)
-                elif f1 == f2:
-                    return f1 # a match
-                elif f1r == f2r or f1r < limit or f2r < limit:
-                    return False # copy no longer relevant
-        except StopIteration:
-            return False
-
     of = None
     seen = set([f])
     for oc in getfctx(f, m1[f]).ancestors():
@@ -513,20 +643,47 b' def checkcopies(ctx, f, m1, m2, ca, limi'
             continue
         seen.add(of)
 
-        fullcopy[f] = of # remember for dir rename detection
+        # remember for dir rename detection
+        if backwards:
+            data['fullcopy'][of] = f # grafting backwards through renames
+        else:
+            data['fullcopy'][f] = of
         if of not in m2:
             continue # no match, keep looking
-        if m2[of] == ma.get(of):
-            break # no merge needed, quit early
+        if m2[of] == mb.get(of):
+            return # no merge needed, quit early
         c2 = getfctx(of, m2[of])
-        cr = _related(oc, c2, ca.rev())
+        # c2 might be a plain new file on added on destination side that is
+        # unrelated to the droids we are looking for.
+        cr = _related(oc, c2, tca.rev())
         if cr and (of == f or of == c2.path()): # non-divergent
-            copy[f] = of
-            of = None
-            break
+            if backwards:
+                data['copy'][of] = f
+            elif of in mb:
+                data['copy'][f] = of
+            elif remotebase: # special case: a <- b <- a -> b "ping-pong" rename
+                data['copy'][of] = f
+                del data['fullcopy'][f]
+                data['fullcopy'][of] = f
+            else: # divergence w.r.t. graft CA on one side of topological CA
+                for sf in seen:
+                    if sf in mb:
+                        assert sf not in data['diverge']
+                        data['diverge'][sf] = [f, of]
+                        break
+            return
 
-    if of in ma:
-        diverge.setdefault(of, []).append(f)
+    if of in mta:
+        if backwards or remotebase:
+            data['incomplete'][of] = f
+        else:
+            for sf in seen:
+                if sf in mb:
+                    if tca == base:
+                        data['diverge'].setdefault(sf, []).append(f)
+                    else:
+                        data['incompletediverge'][sf] = [of, f]
+                    return
 
 def duplicatecopies(repo, rev, fromrev, skiprev=None):
     '''reproduce copies from fromrev to rev in the dirstate

mercurial/crecord.py

@@ -28,7 +28,7 b' stringio = util.stringio'
 
 # This is required for ncurses to display non-ASCII characters in default user
 # locale encoding correctly.  --immerrr
-locale.setlocale(locale.LC_ALL, '')
+locale.setlocale(locale.LC_ALL, u'')
 
 # patch comments based on the git one
 diffhelptext = _("""# To remove '-' lines, make them ' ' lines (context).
@@ -719,7 +719,7 b' class curseschunkselector(object):'
         "scroll the screen to fully show the currently-selected"
         selstart = self.selecteditemstartline
         selend = self.selecteditemendline
-        #selnumlines = selend - selstart
+
         padstart = self.firstlineofpadtoprint
         padend = padstart + self.yscreensize - self.numstatuslines - 1
         # 'buffered' pad start/end values which scroll with a certain
@@ -1263,7 +1263,6 b' class curseschunkselector(object):'
             self.statuswin.resize(self.numstatuslines, self.xscreensize)
             self.numpadlines = self.getnumlinesdisplayed(ignorefolding=True) + 1
             self.chunkpad = curses.newpad(self.numpadlines, self.xscreensize)
-            # todo: try to resize commit message window if possible
         except curses.error:
             pass
 
@@ -1339,6 +1338,7 b' the following are valid keystrokes:'
  shift-left-arrow   [H] : go to parent header / fold selected header
                       f : fold / unfold item, hiding/revealing its children
                       F : fold / unfold parent item and all of its ancestors
+                 ctrl-l : scroll the selected line to the top of the screen
                       m : edit / resume editing the commit message
                       e : edit the currently selected hunk
                       a : toggle amend mode, only with commit -i
@@ -1583,13 +1583,17 b' are you sure you want to review/edit and'
             self.helpwindow()
             self.stdscr.clear()
             self.stdscr.refresh()
+        elif curses.unctrl(keypressed) in ["^L"]:
+            # scroll the current line to the top of the screen
+            self.scrolllines(self.selecteditemstartline)
 
     def main(self, stdscr):
         """
         method to be wrapped by curses.wrapper() for selecting chunks.
         """
 
-        signal.signal(signal.SIGWINCH, self.sigwinchhandler)
+        origsigwinchhandler = signal.signal(signal.SIGWINCH,
+                                            self.sigwinchhandler)
         self.stdscr = stdscr
         # error during initialization, cannot be printed in the curses
         # interface, it should be printed by the calling code
@@ -1640,3 +1644,4 b' are you sure you want to review/edit and'
                 keypressed = "foobar"
             if self.handlekeypressed(keypressed):
                 break
+        signal.signal(signal.SIGWINCH, origsigwinchhandler)

mercurial/demandimport.py

@@ -64,8 +64,12 b' def _hgextimport(importfunc, name, globa'
         return importfunc(hgextname, globals, *args, **kwargs)
 
 class _demandmod(object):
-    """module demand-loader and proxy"""
-    def __init__(self, name, globals, locals, level=level):
+    """module demand-loader and proxy
+
+    Specify 1 as 'level' argument at construction, to import module
+    relatively.
+    """
+    def __init__(self, name, globals, locals, level):
         if '.' in name:
             head, rest = name.split('.', 1)
             after = [rest]
@@ -117,7 +121,8 b' class _demandmod(object):'
                 if '.' in p:
                     h, t = p.split('.', 1)
                 if getattr(mod, h, nothing) is nothing:
-                    setattr(mod, h, _demandmod(p, mod.__dict__, mod.__dict__))
+                    setattr(mod, h, _demandmod(p, mod.__dict__, mod.__dict__,
+                                               level=1))
                 elif t:
                     subload(getattr(mod, h), t)
 
@@ -186,11 +191,16 b' def _demandimport(name, globals=None, lo'
         def processfromitem(mod, attr):
             """Process an imported symbol in the import statement.
 
-            If the symbol doesn't exist in the parent module, it must be a
-            module. We set missing modules up as _demandmod instances.
+            If the symbol doesn't exist in the parent module, and if the
+            parent module is a package, it must be a module. We set missing
+            modules up as _demandmod instances.
             """
             symbol = getattr(mod, attr, nothing)
+            nonpkg = getattr(mod, '__path__', nothing) is nothing
             if symbol is nothing:
+                if nonpkg:
+                    # do not try relative import, which would raise ValueError
+                    raise ImportError('cannot import name %s' % attr)
                 mn = '%s.%s' % (mod.__name__, attr)
                 if mn in ignore:
                     importfunc = _origimport
@@ -210,8 +220,8 b' def _demandimport(name, globals=None, lo'
             mod = rootmod
             for comp in modname.split('.')[1:]:
                 if getattr(mod, comp, nothing) is nothing:
-                    setattr(mod, comp,
-                            _demandmod(comp, mod.__dict__, mod.__dict__))
+                    setattr(mod, comp, _demandmod(comp, mod.__dict__,
+                                                  mod.__dict__, level=1))
                 mod = getattr(mod, comp)
             return mod
 
@@ -259,6 +269,7 b' ignore = ['
     '_imp',
     '_xmlplus',
     'fcntl',
+    'nt', # pathlib2 tests the existence of built-in 'nt' module
     'win32com.gen_py',
     '_winreg', # 2.7 mimetypes needs immediate ImportError
     'pythoncom',
@@ -279,9 +290,17 b' ignore = ['
     'mimetools',
     'sqlalchemy.events', # has import-time side effects (issue5085)
     # setuptools 8 expects this module to explode early when not on windows
-    'distutils.msvc9compiler'
+    'distutils.msvc9compiler',
+    '__builtin__',
+    'builtins',
     ]
 
+if _pypy:
+    ignore.extend([
+        # _ctypes.pointer is shadowed by "from ... import pointer" (PyPy 5)
+        '_ctypes.pointer',
+    ])
+
 def isenabled():
     return builtins.__import__ == _demandimport
 

mercurial/destutil.py

@@ -416,8 +416,8 b' def _statusotherbranchheads(ui, repo):'
                         'updating to a closed head\n') %
                       (currentbranch))
             if otherheads:
-                ui.warn(_('(committing will reopen the head, '
-                            'use `hg heads .` to see %i other heads)\n') %
+                ui.warn(_("(committing will reopen the head, "
+                            "use 'hg heads .' to see %i other heads)\n") %
                           (len(otherheads)))
             else:
                 ui.warn(_('(committing will reopen branch "%s")\n') %

mercurial/dirs.c

@@ -11,6 +11,12 b''
 #include <Python.h>
 #include "util.h"
 
+#ifdef IS_PY3K
+#define PYLONG_VALUE(o) ((PyLongObject *)o)->ob_digit[1]
+#else
+#define PYLONG_VALUE(o) PyInt_AS_LONG(o)
+#endif
+
 /*
  * This is a multiset of directory names, built from the files that
  * appear in a dirstate or manifest.
@@ -41,11 +47,20 b' static inline Py_ssize_t _finddir(const '
 
 static int _addpath(PyObject *dirs, PyObject *path)
 {
-	const char *cpath = PyString_AS_STRING(path);
-	Py_ssize_t pos = PyString_GET_SIZE(path);
+	const char *cpath = PyBytes_AS_STRING(path);
+	Py_ssize_t pos = PyBytes_GET_SIZE(path);
 	PyObject *key = NULL;
 	int ret = -1;
 
+	/* This loop is super critical for performance. That's why we inline
+	* access to Python structs instead of going through a supported API.
+	* The implementation, therefore, is heavily dependent on CPython
+	* implementation details. We also commit violations of the Python
+	* "protocol" such as mutating immutable objects. But since we only
+	* mutate objects created in this function or in other well-defined
+	* locations, the references are known so these violations should go
+	* unnoticed. The code for adjusting the length of a PyBytesObject is
+	* essentially a minimal version of _PyBytes_Resize. */
 	while ((pos = _finddir(cpath, pos - 1)) != -1) {
 		PyObject *val;
 
@@ -53,30 +68,36 b' static int _addpath(PyObject *dirs, PyOb'
 		   in our dict. Try to avoid allocating and
 		   deallocating a string for each prefix we check. */
 		if (key != NULL)
-			((PyStringObject *)key)->ob_shash = -1;
+			((PyBytesObject *)key)->ob_shash = -1;
 		else {
 			/* Force Python to not reuse a small shared string. */
-			key = PyString_FromStringAndSize(cpath,
+			key = PyBytes_FromStringAndSize(cpath,
 							 pos < 2 ? 2 : pos);
 			if (key == NULL)
 				goto bail;
 		}
-		PyString_GET_SIZE(key) = pos;
-		PyString_AS_STRING(key)[pos] = '\0';
+		/* Py_SIZE(o) refers to the ob_size member of the struct. Yes,
+		* assigning to what looks like a function seems wrong. */
+		Py_SIZE(key) = pos;
+		((PyBytesObject *)key)->ob_sval[pos] = '\0';
 
 		val = PyDict_GetItem(dirs, key);
 		if (val != NULL) {
-			PyInt_AS_LONG(val) += 1;
+			PYLONG_VALUE(val) += 1;
 			break;
 		}
 
 		/* Force Python to not reuse a small shared int. */
+#ifdef IS_PY3K
+		val = PyLong_FromLong(0x1eadbeef);
+#else
 		val = PyInt_FromLong(0x1eadbeef);
+#endif
 
 		if (val == NULL)
 			goto bail;
 
-		PyInt_AS_LONG(val) = 1;
+		PYLONG_VALUE(val) = 1;
 		ret = PyDict_SetItem(dirs, key, val);
 		Py_DECREF(val);
 		if (ret == -1)
@@ -93,15 +114,15 b' bail:'
 
 static int _delpath(PyObject *dirs, PyObject *path)
 {
-	char *cpath = PyString_AS_STRING(path);
-	Py_ssize_t pos = PyString_GET_SIZE(path);
+	char *cpath = PyBytes_AS_STRING(path);
+	Py_ssize_t pos = PyBytes_GET_SIZE(path);
 	PyObject *key = NULL;
 	int ret = -1;
 
 	while ((pos = _finddir(cpath, pos - 1)) != -1) {
 		PyObject *val;
 
-		key = PyString_FromStringAndSize(cpath, pos);
+		key = PyBytes_FromStringAndSize(cpath, pos);
 
 		if (key == NULL)
 			goto bail;
@@ -113,7 +134,7 b' static int _delpath(PyObject *dirs, PyOb'
 			goto bail;
 		}
 
-		if (--PyInt_AS_LONG(val) <= 0) {
+		if (--PYLONG_VALUE(val) <= 0) {
 			if (PyDict_DelItem(dirs, key) == -1)
 				goto bail;
 		} else
@@ -134,7 +155,7 b' static int dirs_fromdict(PyObject *dirs,'
 	Py_ssize_t pos = 0;
 
 	while (PyDict_Next(source, &pos, &key, &value)) {
-		if (!PyString_Check(key)) {
+		if (!PyBytes_Check(key)) {
 			PyErr_SetString(PyExc_TypeError, "expected string key");
 			return -1;
 		}
@@ -165,7 +186,7 b' static int dirs_fromiter(PyObject *dirs,'
 		return -1;
 
 	while ((item = PyIter_Next(iter)) != NULL) {
-		if (!PyString_Check(item)) {
+		if (!PyBytes_Check(item)) {
 			PyErr_SetString(PyExc_TypeError, "expected string");
 			break;
 		}
@@ -224,7 +245,7 b' PyObject *dirs_addpath(dirsObject *self,'
 {
 	PyObject *path;
 
-	if (!PyArg_ParseTuple(args, "O!:addpath", &PyString_Type, &path))
+	if (!PyArg_ParseTuple(args, "O!:addpath", &PyBytes_Type, &path))
 		return NULL;
 
 	if (_addpath(self->dict, path) == -1)
@@ -237,7 +258,7 b' static PyObject *dirs_delpath(dirsObject'
 {
 	PyObject *path;
 
-	if (!PyArg_ParseTuple(args, "O!:delpath", &PyString_Type, &path))
+	if (!PyArg_ParseTuple(args, "O!:delpath", &PyBytes_Type, &path))
 		return NULL;
 
 	if (_delpath(self->dict, path) == -1)
@@ -248,7 +269,7 b' static PyObject *dirs_delpath(dirsObject'
 
 static int dirs_contains(dirsObject *self, PyObject *value)
 {
-	return PyString_Check(value) ? PyDict_Contains(self->dict, value) : 0;
+	return PyBytes_Check(value) ? PyDict_Contains(self->dict, value) : 0;
 }
 
 static void dirs_dealloc(dirsObject *self)
@@ -270,7 +291,7 b' static PyMethodDef dirs_methods[] = {'
 	{NULL} /* Sentinel */
 };
 
-static PyTypeObject dirsType = { PyObject_HEAD_INIT(NULL) };
+static PyTypeObject dirsType = { PyVarObject_HEAD_INIT(NULL, 0) };
 
 void dirs_module_init(PyObject *mod)
 {

mercurial/dirstate.py

@@ -74,8 +74,6 b' def _trypending(root, vfs, filename):'
                 raise
     return (vfs(filename), False)
 
-_token = object()
-
 class dirstate(object):
 
     def __init__(self, opener, ui, root, validate):
@@ -103,6 +101,8 b' class dirstate(object):'
         self._parentwriters = 0
         self._filename = 'dirstate'
         self._pendingfilename = '%s.pending' % self._filename
+        self._plchangecallbacks = {}
+        self._origpl = None
 
         # for consistent view between _pl() and _read() invocations
         self._pendingmode = None
@@ -227,7 +227,7 b' class dirstate(object):'
 
     @propertycache
     def _checkcase(self):
-        return not util.checkcase(self._join('.hg'))
+        return not util.fscasesensitive(self._join('.hg'))
 
     def _join(self, f):
         # much faster than os.path.join()
@@ -349,6 +349,8 b' class dirstate(object):'
 
         self._dirty = self._dirtypl = True
         oldp2 = self._pl[1]
+        if self._origpl is None:
+            self._origpl = self._pl
         self._pl = p1, p2
         copies = {}
         if oldp2 != nullid and p2 == nullid:
@@ -444,6 +446,7 b' class dirstate(object):'
         self._lastnormaltime = 0
         self._dirty = False
         self._parentwriters = 0
+        self._origpl = None
 
     def copy(self, source, dest):
         """Mark dest as a copy of source. Unmark dest if source is None."""
@@ -677,37 +680,23 b' class dirstate(object):'
             self.clear()
             self._lastnormaltime = lastnormaltime
 
+        if self._origpl is None:
+            self._origpl = self._pl
+        self._pl = (parent, nullid)
         for f in changedfiles:
-            mode = 0o666
-            if f in allfiles and 'x' in allfiles.flags(f):
-                mode = 0o777
-
             if f in allfiles:
-                self._map[f] = dirstatetuple('n', mode, -1, 0)
+                self.normallookup(f)
             else:
-                self._map.pop(f, None)
-                if f in self._nonnormalset:
-                    self._nonnormalset.remove(f)
+                self.drop(f)
 
-        self._pl = (parent, nullid)
         self._dirty = True
 
-    def write(self, tr=_token):
+    def write(self, tr):
         if not self._dirty:
             return
 
         filename = self._filename
-        if tr is _token: # not explicitly specified
-            self._ui.deprecwarn('use dirstate.write with '
-                               'repo.currenttransaction()',
-                               '3.9')
-
-            if self._opener.lexists(self._pendingfilename):
-                # if pending file already exists, in-memory changes
-                # should be written into it, because it has priority
-                # to '.hg/dirstate' at reading under HG_PENDING mode
-                filename = self._pendingfilename
-        elif tr:
+        if tr:
             # 'dirstate.write()' is not only for writing in-memory
             # changes out, but also for dropping ambiguous timestamp.
             # delayed writing re-raise "ambiguous timestamp issue".
@@ -733,7 +722,23 b' class dirstate(object):'
         st = self._opener(filename, "w", atomictemp=True, checkambig=True)
         self._writedirstate(st)
 
+    def addparentchangecallback(self, category, callback):
+        """add a callback to be called when the wd parents are changed
+
+        Callback will be called with the following arguments:
+            dirstate, (oldp1, oldp2), (newp1, newp2)
+
+        Category is a unique identifier to allow overwriting an old callback
+        with a newer callback.
+        """
+        self._plchangecallbacks[category] = callback
+
     def _writedirstate(self, st):
+        # notify callbacks about parents change
+        if self._origpl is not None and self._origpl != self._pl:
+            for c, callback in sorted(self._plchangecallbacks.iteritems()):
+                callback(self, self._origpl, self._pl)
+            self._origpl = None
         # use the modification time of the newly created temporary file as the
         # filesystem's notion of 'now'
         now = util.fstat(st).st_mtime & _rangemask

mercurial/discovery.py

@@ -76,10 +76,29 b' class outgoing(object):'
     The sets are computed on demand from the heads, unless provided upfront
     by discovery.'''
 
-    def __init__(self, revlog, commonheads, missingheads):
+    def __init__(self, repo, commonheads=None, missingheads=None,
+                 missingroots=None):
+        # at least one of them must not be set
+        assert None in (commonheads, missingroots)
+        cl = repo.changelog
+        if missingheads is None:
+            missingheads = cl.heads()
+        if missingroots:
+            discbases = []
+            for n in missingroots:
+                discbases.extend([p for p in cl.parents(n) if p != nullid])
+            # TODO remove call to nodesbetween.
+            # TODO populate attributes on outgoing instance instead of setting
+            # discbases.
+            csets, roots, heads = cl.nodesbetween(missingroots, missingheads)
+            included = set(csets)
+            missingheads = heads
+            commonheads = [n for n in discbases if n not in included]
+        elif not commonheads:
+            commonheads = [nullid]
         self.commonheads = commonheads
         self.missingheads = missingheads
-        self._revlog = revlog
+        self._revlog = cl
         self._common = None
         self._missing = None
         self.excluded = []
@@ -116,7 +135,7 b' def findcommonoutgoing(repo, other, only'
     If portable is given, compute more conservative common and missingheads,
     to make bundles created from the instance more portable.'''
     # declare an empty outgoing object to be filled later
-    og = outgoing(repo.changelog, None, None)
+    og = outgoing(repo, None, None)
 
     # get common set if not provided
     if commoninc is None:
@@ -382,7 +401,7 b' def checkheads(pushop):'
                     errormsg = (_("push creates new branch '%s' "
                                   "with multiple heads") % (branch))
                     hint = _("merge or"
-                             " see \"hg help push\" for details about"
+                             " see 'hg help push' for details about"
                              " pushing new heads")
         elif len(newhs) > len(oldhs):
             # remove bookmarked or existing remote heads from the new heads list
@@ -401,11 +420,11 b' def checkheads(pushop):'
                                  ) % short(dhs[0])
                 if unsyncedheads:
                     hint = _("pull and merge or"
-                             " see \"hg help push\" for details about"
+                             " see 'hg help push' for details about"
                              " pushing new heads")
                 else:
                     hint = _("merge or"
-                             " see \"hg help push\" for details about"
+                             " see 'hg help push' for details about"
                              " pushing new heads")
             if branch is None:
                 repo.ui.note(_("new remote heads:\n"))

mercurial/dispatch.py

@@ -34,6 +34,7 b' from . import ('
     fileset,
     hg,
     hook,
+    profiling,
     revset,
     templatefilters,
     templatekw,
@@ -150,7 +151,7 b' def _runcatch(req):'
     except ValueError:
         pass # happens if called in a thread
 
-    try:
+    def _runcatchfunc():
         try:
             debugger = 'pdb'
             debugtrace = {
@@ -212,6 +213,16 b' def _runcatch(req):'
             ui.traceback()
             raise
 
+    return callcatch(ui, _runcatchfunc)
+
+def callcatch(ui, func):
+    """call func() with global exception handling
+
+    return func() if no exception happens. otherwise do some error handling
+    and return an exit code accordingly.
+    """
+    try:
+        return func()
     # Global exception handling, alphabetically
     # Mercurial-specific first, followed by built-in and library exceptions
     except error.AmbiguousCommand as inst:
@@ -489,6 +500,8 b' class cmdalias(object):'
             ui.debug("alias '%s' shadows command '%s'\n" %
                      (self.name, self.cmdname))
 
+        ui.log('commandalias', "alias '%s' expands to '%s'\n",
+               self.name, self.definition)
         if util.safehasattr(self, 'shell'):
             return self.fn(ui, *args, **opts)
         else:
@@ -545,7 +558,7 b' def _parse(ui, args):'
         c.append((o[0], o[1], options[o[1]], o[3]))
 
     try:
-        args = fancyopts.fancyopts(args, c, cmdoptions, True)
+        args = fancyopts.fancyopts(args, c, cmdoptions, gnu=True)
     except fancyopts.getopt.GetoptError as inst:
         raise error.CommandError(cmd, inst)
 
@@ -761,7 +774,8 b' def _dispatch(req):'
     # Check abbreviation/ambiguity of shell alias.
     shellaliasfn = _checkshellalias(lui, ui, args)
     if shellaliasfn:
-        return shellaliasfn()
+        with profiling.maybeprofile(lui):
+            return shellaliasfn()
 
     # check for fallback encoding
     fallback = lui.config('ui', 'fallbackencoding')
@@ -808,6 +822,10 b' def _dispatch(req):'
             for ui_ in uis:
                 ui_.setconfig('ui', opt, val, '--' + opt)
 
+    if options['profile']:
+        for ui_ in uis:
+            ui_.setconfig('profiling', 'enabled', 'true', '--profile')
+
     if options['traceback']:
         for ui_ in uis:
             ui_.setconfig('ui', 'traceback', 'on', '--traceback')
@@ -827,187 +845,70 b' def _dispatch(req):'
     elif not cmd:
         return commands.help_(ui, 'shortlist')
 
-    repo = None
-    cmdpats = args[:]
-    if not _cmdattr(ui, cmd, func, 'norepo'):
-        # use the repo from the request only if we don't have -R
-        if not rpath and not cwd:
-            repo = req.repo
-
-        if repo:
-            # set the descriptors of the repo ui to those of ui
-            repo.ui.fin = ui.fin
-            repo.ui.fout = ui.fout
-            repo.ui.ferr = ui.ferr
-        else:
-            try:
-                repo = hg.repository(ui, path=path)
-                if not repo.local():
-                    raise error.Abort(_("repository '%s' is not local") % path)
-                repo.ui.setconfig("bundle", "mainreporoot", repo.root, 'repo')
-            except error.RequirementError:
-                raise
-            except error.RepoError:
-                if rpath and rpath[-1]: # invalid -R path
-                    raise
-                if not _cmdattr(ui, cmd, func, 'optionalrepo'):
-                    if (_cmdattr(ui, cmd, func, 'inferrepo') and
-                        args and not path):
-                        # try to infer -R from command args
-                        repos = map(cmdutil.findrepo, args)
-                        guess = repos[0]
-                        if guess and repos.count(guess) == len(repos):
-                            req.args = ['--repository', guess] + fullargs
-                            return _dispatch(req)
-                    if not path:
-                        raise error.RepoError(_("no repository found in '%s'"
-                                                " (.hg not found)")
-                                              % os.getcwd())
-                    raise
-        if repo:
-            ui = repo.ui
-            if options['hidden']:
-                repo = repo.unfiltered()
-        args.insert(0, repo)
-    elif rpath:
-        ui.warn(_("warning: --repository ignored\n"))
-
-    msg = ' '.join(' ' in a and repr(a) or a for a in fullargs)
-    ui.log("command", '%s\n', msg)
-    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
-    try:
-        return runcommand(lui, repo, cmd, fullargs, ui, options, d,
-                          cmdpats, cmdoptions)
-    finally:
-        if repo and repo != req.repo:
-            repo.close()
-
-def lsprofile(ui, func, fp):
-    format = ui.config('profiling', 'format', default='text')
-    field = ui.config('profiling', 'sort', default='inlinetime')
-    limit = ui.configint('profiling', 'limit', default=30)
-    climit = ui.configint('profiling', 'nested', default=0)
-
-    if format not in ['text', 'kcachegrind']:
-        ui.warn(_("unrecognized profiling format '%s'"
-                    " - Ignored\n") % format)
-        format = 'text'
+    with profiling.maybeprofile(lui):
+        repo = None
+        cmdpats = args[:]
+        if not _cmdattr(ui, cmd, func, 'norepo'):
+            # use the repo from the request only if we don't have -R
+            if not rpath and not cwd:
+                repo = req.repo
 
-    try:
-        from . import lsprof
-    except ImportError:
-        raise error.Abort(_(
-            'lsprof not available - install from '
-            'http://codespeak.net/svn/user/arigo/hack/misc/lsprof/'))
-    p = lsprof.Profiler()
-    p.enable(subcalls=True)
-    try:
-        return func()
-    finally:
-        p.disable()
-
-        if format == 'kcachegrind':
-            from . import lsprofcalltree
-            calltree = lsprofcalltree.KCacheGrind(p)
-            calltree.output(fp)
-        else:
-            # format == 'text'
-            stats = lsprof.Stats(p.getstats())
-            stats.sort(field)
-            stats.pprint(limit=limit, file=fp, climit=climit)
+            if repo:
+                # set the descriptors of the repo ui to those of ui
+                repo.ui.fin = ui.fin
+                repo.ui.fout = ui.fout
+                repo.ui.ferr = ui.ferr
+            else:
+                try:
+                    repo = hg.repository(ui, path=path)
+                    if not repo.local():
+                        raise error.Abort(_("repository '%s' is not local")
+                                          % path)
+                    repo.ui.setconfig("bundle", "mainreporoot", repo.root,
+                                      'repo')
+                except error.RequirementError:
+                    raise
+                except error.RepoError:
+                    if rpath and rpath[-1]: # invalid -R path
+                        raise
+                    if not _cmdattr(ui, cmd, func, 'optionalrepo'):
+                        if (_cmdattr(ui, cmd, func, 'inferrepo') and
+                            args and not path):
+                            # try to infer -R from command args
+                            repos = map(cmdutil.findrepo, args)
+                            guess = repos[0]
+                            if guess and repos.count(guess) == len(repos):
+                                req.args = ['--repository', guess] + fullargs
+                                return _dispatch(req)
+                        if not path:
+                            raise error.RepoError(_("no repository found in"
+                                                    " '%s' (.hg not found)")
+                                                  % os.getcwd())
+                        raise
+            if repo:
+                ui = repo.ui
+                if options['hidden']:
+                    repo = repo.unfiltered()
+            args.insert(0, repo)
+        elif rpath:
+            ui.warn(_("warning: --repository ignored\n"))
 
-def flameprofile(ui, func, fp):
-    try:
-        from flamegraph import flamegraph
-    except ImportError:
-        raise error.Abort(_(
-            'flamegraph not available - install from '
-            'https://github.com/evanhempel/python-flamegraph'))
-    # developer config: profiling.freq
-    freq = ui.configint('profiling', 'freq', default=1000)
-    filter_ = None
-    collapse_recursion = True
-    thread = flamegraph.ProfileThread(fp, 1.0 / freq,
-                                      filter_, collapse_recursion)
-    start_time = time.clock()
-    try:
-        thread.start()
-        func()
-    finally:
-        thread.stop()
-        thread.join()
-        print('Collected %d stack frames (%d unique) in %2.2f seconds.' % (
-            time.clock() - start_time, thread.num_frames(),
-            thread.num_frames(unique=True)))
-
-
-def statprofile(ui, func, fp):
-    try:
-        import statprof
-    except ImportError:
-        raise error.Abort(_(
-            'statprof not available - install using "easy_install statprof"'))
-
-    freq = ui.configint('profiling', 'freq', default=1000)
-    if freq > 0:
-        statprof.reset(freq)
-    else:
-        ui.warn(_("invalid sampling frequency '%s' - ignoring\n") % freq)
-
-    statprof.start()
-    try:
-        return func()
-    finally:
-        statprof.stop()
-        statprof.display(fp)
+        msg = ' '.join(' ' in a and repr(a) or a for a in fullargs)
+        ui.log("command", '%s\n', msg)
+        d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
+        try:
+            return runcommand(lui, repo, cmd, fullargs, ui, options, d,
+                              cmdpats, cmdoptions)
+        finally:
+            if repo and repo != req.repo:
+                repo.close()
 
 def _runcommand(ui, options, cmd, cmdfunc):
-    """Enables the profiler if applicable.
-
-    ``profiling.enabled`` - boolean config that enables or disables profiling
-    """
-    def checkargs():
-        try:
-            return cmdfunc()
-        except error.SignatureError:
-            raise error.CommandError(cmd, _("invalid arguments"))
-
-    if options['profile'] or ui.configbool('profiling', 'enabled'):
-        profiler = os.getenv('HGPROF')
-        if profiler is None:
-            profiler = ui.config('profiling', 'type', default='ls')
-        if profiler not in ('ls', 'stat', 'flame'):
-            ui.warn(_("unrecognized profiler '%s' - ignored\n") % profiler)
-            profiler = 'ls'
-
-        output = ui.config('profiling', 'output')
-
-        if output == 'blackbox':
-            fp = util.stringio()
-        elif output:
-            path = ui.expandpath(output)
-            fp = open(path, 'wb')
-        else:
-            fp = sys.stderr
-
-        try:
-            if profiler == 'ls':
-                return lsprofile(ui, checkargs, fp)
-            elif profiler == 'flame':
-                return flameprofile(ui, checkargs, fp)
-            else:
-                return statprofile(ui, checkargs, fp)
-        finally:
-            if output:
-                if output == 'blackbox':
-                    val = "Profile:\n%s" % fp.getvalue()
-                    # ui.log treats the input as a format string,
-                    # so we need to escape any % signs.
-                    val = val.replace('%', '%%')
-                    ui.log('profile', val)
-                fp.close()
-    else:
-        return checkargs()
+    """Run a command function, possibly with profiling enabled."""
+    try:
+        return cmdfunc()
+    except error.SignatureError:
+        raise error.CommandError(cmd, _('invalid arguments'))
 
 def _exceptionwarning(ui):
     """Produce a warning message for the current active exception"""
@@ -1031,7 +932,7 b' def _exceptionwarning(ui):'
                 break
 
             # Never blame on extensions bundled with Mercurial.
-            if testedwith == 'internal':
+            if extensions.ismoduleinternal(mod):
                 continue
 
             tested = [util.versiontuple(t, 2) for t in testedwith.split()]

mercurial/encoding.py

@@ -10,14 +10,16 b' from __future__ import absolute_import'
 import array
 import locale
 import os
-import sys
 import unicodedata
 
 from . import (
     error,
+    pycompat,
 )
 
-if sys.version_info[0] >= 3:
+_sysstr = pycompat.sysstr
+
+if pycompat.ispy3:
     unichr = chr
 
 # These unicode characters are ignored by HFS+ (Apple Technote 1150,
@@ -27,7 +29,7 b' if sys.version_info[0] >= 3:'
            "200c 200d 200e 200f 202a 202b 202c 202d 202e "
            "206a 206b 206c 206d 206e 206f feff".split()]
 # verify the next function will work
-if sys.version_info[0] >= 3:
+if pycompat.ispy3:
     assert set(i[0] for i in _ignore) == set([ord(b'\xe2'), ord(b'\xef')])
 else:
     assert set(i[0] for i in _ignore) == set(["\xe2", "\xef"])
@@ -45,6 +47,19 b' def hfsignoreclean(s):'
             s = s.replace(c, '')
     return s
 
+# encoding.environ is provided read-only, which may not be used to modify
+# the process environment
+_nativeenviron = (not pycompat.ispy3 or os.supports_bytes_environ)
+if not pycompat.ispy3:
+    environ = os.environ
+elif _nativeenviron:
+    environ = os.environb
+else:
+    # preferred encoding isn't known yet; use utf-8 to avoid unicode error
+    # and recreate it once encoding is settled
+    environ = dict((k.encode(u'utf-8'), v.encode(u'utf-8'))
+                   for k, v in os.environ.items())
+
 def _getpreferredencoding():
     '''
     On darwin, getpreferredencoding ignores the locale environment and
@@ -76,13 +91,13 b' def _getpreferredencoding():'
 }
 
 try:
-    encoding = os.environ.get("HGENCODING")
+    encoding = environ.get("HGENCODING")
     if not encoding:
         encoding = locale.getpreferredencoding() or 'ascii'
         encoding = _encodingfixers.get(encoding, lambda: encoding)()
 except locale.Error:
     encoding = 'ascii'
-encodingmode = os.environ.get("HGENCODINGMODE", "strict")
+encodingmode = environ.get("HGENCODINGMODE", "strict")
 fallbackencoding = 'ISO-8859-1'
 
 class localstr(str):
@@ -136,23 +151,24 b' def tolocal(s):'
             if encoding == 'UTF-8':
                 # fast path
                 return s
-            r = u.encode(encoding, "replace")
-            if u == r.decode(encoding):
+            r = u.encode(_sysstr(encoding), u"replace")
+            if u == r.decode(_sysstr(encoding)):
                 # r is a safe, non-lossy encoding of s
                 return r
             return localstr(s, r)
         except UnicodeDecodeError:
             # we should only get here if we're looking at an ancient changeset
             try:
-                u = s.decode(fallbackencoding)
-                r = u.encode(encoding, "replace")
-                if u == r.decode(encoding):
+                u = s.decode(_sysstr(fallbackencoding))
+                r = u.encode(_sysstr(encoding), u"replace")
+                if u == r.decode(_sysstr(encoding)):
                     # r is a safe, non-lossy encoding of s
                     return r
                 return localstr(u.encode('UTF-8'), r)
             except UnicodeDecodeError:
                 u = s.decode("utf-8", "replace") # last ditch
-                return u.encode(encoding, "replace") # can't round-trip
+                # can't round-trip
+                return u.encode(_sysstr(encoding), u"replace")
     except LookupError as k:
         raise error.Abort(k, hint="please check your locale settings")
 
@@ -172,20 +188,27 b' def fromlocal(s):'
         return s._utf8
 
     try:
-        return s.decode(encoding, encodingmode).encode("utf-8")
+        u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
+        return u.encode("utf-8")
     except UnicodeDecodeError as inst:
         sub = s[max(0, inst.start - 10):inst.start + 10]
         raise error.Abort("decoding near '%s': %s!" % (sub, inst))
     except LookupError as k:
         raise error.Abort(k, hint="please check your locale settings")
 
+if not _nativeenviron:
+    # now encoding and helper functions are available, recreate the environ
+    # dict to be exported to other modules
+    environ = dict((tolocal(k.encode(u'utf-8')), tolocal(v.encode(u'utf-8')))
+                   for k, v in os.environ.items())
+
 # How to treat ambiguous-width characters. Set to 'wide' to treat as wide.
-wide = (os.environ.get("HGENCODINGAMBIGUOUS", "narrow") == "wide"
+wide = (environ.get("HGENCODINGAMBIGUOUS", "narrow") == "wide"
         and "WFA" or "WF")
 
 def colwidth(s):
     "Find the column width of a string for display in the local encoding"
-    return ucolwidth(s.decode(encoding, 'replace'))
+    return ucolwidth(s.decode(_sysstr(encoding), u'replace'))
 
 def ucolwidth(d):
     "Find the column width of a Unicode string for display"
@@ -265,7 +288,7 b" def trim(s, width, ellipsis='', leftside"
     +
     """
     try:
-        u = s.decode(encoding)
+        u = s.decode(_sysstr(encoding))
     except UnicodeDecodeError:
         if len(s) <= width: # trimming is not needed
             return s
@@ -292,7 +315,7 b" def trim(s, width, ellipsis='', leftside"
     for i in xrange(1, len(u)):
         usub = uslice(i)
         if ucolwidth(usub) <= width:
-            return concat(usub.encode(encoding))
+            return concat(usub.encode(_sysstr(encoding)))
     return ellipsis # no enough room for multi-column characters
 
 def _asciilower(s):
@@ -337,12 +360,12 b' def lower(s):'
         if isinstance(s, localstr):
             u = s._utf8.decode("utf-8")
         else:
-            u = s.decode(encoding, encodingmode)
+            u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
 
         lu = u.lower()
         if u == lu:
             return s # preserve localstring
-        return lu.encode(encoding)
+        return lu.encode(_sysstr(encoding))
     except UnicodeError:
         return s.lower() # we don't know how to fold this except in ASCII
     except LookupError as k:
@@ -360,12 +383,12 b' def upperfallback(s):'
         if isinstance(s, localstr):
             u = s._utf8.decode("utf-8")
         else:
-            u = s.decode(encoding, encodingmode)
+            u = s.decode(_sysstr(encoding), _sysstr(encodingmode))
 
         uu = u.upper()
         if u == uu:
             return s # preserve localstring
-        return uu.encode(encoding)
+        return uu.encode(_sysstr(encoding))
     except UnicodeError:
         return s.upper() # we don't know how to fold this except in ASCII
     except LookupError as k:

mercurial/exchange.py

@@ -257,13 +257,40 b' def buildobsmarkerspart(bundler, markers'
         return bundler.newpart('obsmarkers', data=stream)
     return None
 
-def _canusebundle2(op):
-    """return true if a pull/push can use bundle2
+def _computeoutgoing(repo, heads, common):
+    """Computes which revs are outgoing given a set of common
+    and a set of heads.
+
+    This is a separate function so extensions can have access to
+    the logic.
 
-    Feel free to nuke this function when we drop the experimental option"""
-    return (op.repo.ui.configbool('experimental', 'bundle2-exp', True)
-            and op.remote.capable('bundle2'))
+    Returns a discovery.outgoing object.
+    """
+    cl = repo.changelog
+    if common:
+        hasnode = cl.hasnode
+        common = [n for n in common if hasnode(n)]
+    else:
+        common = [nullid]
+    if not heads:
+        heads = cl.heads()
+    return discovery.outgoing(repo, common, heads)
 
+def _forcebundle1(op):
+    """return true if a pull/push must use bundle1
+
+    This function is used to allow testing of the older bundle version"""
+    ui = op.repo.ui
+    forcebundle1 = False
+    # The goal is this config is to allow developper to choose the bundle
+    # version used during exchanged. This is especially handy during test.
+    # Value is a list of bundle version to be picked from, highest version
+    # should be used.
+    #
+    # developer config: devel.legacy.exchange
+    exchange = ui.configlist('devel', 'legacy.exchange')
+    forcebundle1 = 'bundle2' not in exchange and 'bundle1' in exchange
+    return forcebundle1 or not op.remote.capable('bundle2')
 
 class pushoperation(object):
     """A object that represent a single push operation
@@ -417,7 +444,7 b' def push(repo, remote, force=False, revs'
         # bundle2 push may receive a reply bundle touching bookmarks or other
         # things requiring the wlock. Take it now to ensure proper ordering.
         maypushback = pushop.ui.configbool('experimental', 'bundle2.pushback')
-        if _canusebundle2(pushop) and maypushback:
+        if (not _forcebundle1(pushop)) and maypushback:
             localwlock = pushop.repo.wlock()
         locallock = pushop.repo.lock()
         pushop.locallocked = True
@@ -442,7 +469,7 b' def push(repo, remote, force=False, revs'
             lock = pushop.remote.lock()
         try:
             _pushdiscovery(pushop)
-            if _canusebundle2(pushop):
+            if not _forcebundle1(pushop):
                 _pushbundle2(pushop)
             _pushchangeset(pushop)
             _pushsyncphase(pushop)
@@ -1100,7 +1127,7 b' class pulloperation(object):'
 
     @util.propertycache
     def canusebundle2(self):
-        return _canusebundle2(self)
+        return not _forcebundle1(self)
 
     @util.propertycache
     def remotebundle2caps(self):
@@ -1174,8 +1201,10 b' def pull(repo, remote, heads=None, force'
                     " %s") % (', '.join(sorted(missing)))
             raise error.Abort(msg)
 
-    lock = pullop.repo.lock()
+    wlock = lock = None
     try:
+        wlock = pullop.repo.wlock()
+        lock = pullop.repo.lock()
         pullop.trmanager = transactionmanager(repo, 'pull', remote.url())
         streamclone.maybeperformlegacystreamclone(pullop)
         # This should ideally be in _pullbundle2(). However, it needs to run
@@ -1190,8 +1219,7 b' def pull(repo, remote, heads=None, force'
         _pullobsolete(pullop)
         pullop.trmanager.close()
     finally:
-        pullop.trmanager.release()
-        lock.release()
+        lockmod.release(pullop.trmanager, lock, wlock)
 
     return pullop
 
@@ -1504,20 +1532,14 b' def bundle2requested(bundlecaps):'
         return any(cap.startswith('HG2') for cap in bundlecaps)
     return False
 
-def getbundle(repo, source, heads=None, common=None, bundlecaps=None,
-              **kwargs):
-    """return a full bundle (with potentially multiple kind of parts)
+def getbundlechunks(repo, source, heads=None, common=None, bundlecaps=None,
+                    **kwargs):
+    """Return chunks constituting a bundle's raw data.
 
     Could be a bundle HG10 or a bundle HG20 depending on bundlecaps
-    passed. For now, the bundle can contain only changegroup, but this will
-    changes when more part type will be available for bundle2.
+    passed.
 
-    This is different from changegroup.getchangegroup that only returns an HG10
-    changegroup bundle. They may eventually get reunited in the future when we
-    have a clearer idea of the API we what to query different data.
-
-    The implementation is at a very early stage and will get massive rework
-    when the API of bundle is refined.
+    Returns an iterator over raw chunks (of varying sizes).
     """
     usebundle2 = bundle2requested(bundlecaps)
     # bundle10 case
@@ -1528,8 +1550,9 b' def getbundle(repo, source, heads=None, '
         if kwargs:
             raise ValueError(_('unsupported getbundle arguments: %s')
                              % ', '.join(sorted(kwargs.keys())))
-        return changegroup.getchangegroup(repo, source, heads=heads,
-                                          common=common, bundlecaps=bundlecaps)
+        outgoing = _computeoutgoing(repo, heads, common)
+        bundler = changegroup.getbundler('01', repo, bundlecaps)
+        return changegroup.getsubsetraw(repo, outgoing, bundler, source)
 
     # bundle20 case
     b2caps = {}
@@ -1547,7 +1570,7 b' def getbundle(repo, source, heads=None, '
         func(bundler, repo, source, bundlecaps=bundlecaps, b2caps=b2caps,
              **kwargs)
 
-    return util.chunkbuffer(bundler.getchunks())
+    return bundler.getchunks()
 
 @getbundle2partsgenerator('changegroup')
 def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
@@ -1564,7 +1587,7 b' def _getbundlechangegrouppart(bundler, r'
             if not cgversions:
                 raise ValueError(_('no common changegroup version'))
             version = max(cgversions)
-        outgoing = changegroup.computeoutgoing(repo, heads, common)
+        outgoing = _computeoutgoing(repo, heads, common)
         cg = changegroup.getlocalchangegroupraw(repo, source, outgoing,
                                                 bundlecaps=bundlecaps,
                                                 version=version)
@@ -1617,7 +1640,7 b' def _getbundletagsfnodes(bundler, repo, '
     if not (kwargs.get('cg', True) and 'hgtagsfnodes' in b2caps):
         return
 
-    outgoing = changegroup.computeoutgoing(repo, heads, common)
+    outgoing = _computeoutgoing(repo, heads, common)
 
     if not outgoing.missingheads:
         return

mercurial/extensions.py

@@ -22,6 +22,7 b' from . import ('
 )
 
 _extensions = {}
+_disabledextensions = {}
 _aftercallbacks = {}
 _order = []
 _builtin = set(['hbisect', 'bookmarks', 'parentrevspec', 'progress', 'interhg',
@@ -79,7 +80,29 b' def _importh(name):'
         mod = getattr(mod, comp)
     return mod
 
+def _importext(name, path=None, reportfunc=None):
+    if path:
+        # the module will be loaded in sys.modules
+        # choose an unique name so that it doesn't
+        # conflicts with other modules
+        mod = loadpath(path, 'hgext.%s' % name)
+    else:
+        try:
+            mod = _importh("hgext.%s" % name)
+        except ImportError as err:
+            if reportfunc:
+                reportfunc(err, "hgext.%s" % name, "hgext3rd.%s" % name)
+            try:
+                mod = _importh("hgext3rd.%s" % name)
+            except ImportError as err:
+                if reportfunc:
+                    reportfunc(err, "hgext3rd.%s" % name, name)
+                mod = _importh(name)
+    return mod
+
 def _reportimporterror(ui, err, failed, next):
+    # note: this ui.debug happens before --debug is processed,
+    #       Use --config ui.debug=1 to see them.
     ui.debug('could not import %s (%s): trying %s\n'
              % (failed, err, next))
     if ui.debugflag:
@@ -95,21 +118,7 b' def load(ui, name, path):'
     if shortname in _extensions:
         return _extensions[shortname]
     _extensions[shortname] = None
-    if path:
-        # the module will be loaded in sys.modules
-        # choose an unique name so that it doesn't
-        # conflicts with other modules
-        mod = loadpath(path, 'hgext.%s' % name)
-    else:
-        try:
-            mod = _importh("hgext.%s" % name)
-        except ImportError as err:
-            _reportimporterror(ui, err, "hgext.%s" % name, name)
-            try:
-                mod = _importh("hgext3rd.%s" % name)
-            except ImportError as err:
-                _reportimporterror(ui, err, "hgext3rd.%s" % name, name)
-                mod = _importh(name)
+    mod = _importext(name, path, bind(_reportimporterror, ui))
 
     # Before we do anything with the extension, check against minimum stated
     # compatibility. This gives extension authors a mechanism to have their
@@ -148,6 +157,7 b' def loadall(ui):'
     for (name, path) in result:
         if path:
             if path[0] == '!':
+                _disabledextensions[name] = path[1:]
                 continue
         try:
             load(ui, name, path)
@@ -210,11 +220,13 b' def bind(func, *args):'
         return func(*(args + a), **kw)
     return closure
 
-def _updatewrapper(wrap, origfn):
-    '''Copy attributes to wrapper function'''
+def _updatewrapper(wrap, origfn, unboundwrapper):
+    '''Copy and add some useful attributes to wrapper'''
     wrap.__module__ = getattr(origfn, '__module__')
     wrap.__doc__ = getattr(origfn, '__doc__')
     wrap.__dict__.update(getattr(origfn, '__dict__', {}))
+    wrap._origfunc = origfn
+    wrap._unboundwrapper = unboundwrapper
 
 def wrapcommand(table, command, wrapper, synopsis=None, docstring=None):
     '''Wrap the command named `command' in table
@@ -254,7 +266,7 b' def wrapcommand(table, command, wrapper,'
 
     origfn = entry[0]
     wrap = bind(util.checksignature(wrapper), util.checksignature(origfn))
-    _updatewrapper(wrap, origfn)
+    _updatewrapper(wrap, origfn, wrapper)
     if docstring is not None:
         wrap.__doc__ += docstring
 
@@ -303,10 +315,46 b' def wrapfunction(container, funcname, wr'
     origfn = getattr(container, funcname)
     assert callable(origfn)
     wrap = bind(wrapper, origfn)
-    _updatewrapper(wrap, origfn)
+    _updatewrapper(wrap, origfn, wrapper)
     setattr(container, funcname, wrap)
     return origfn
 
+def unwrapfunction(container, funcname, wrapper=None):
+    '''undo wrapfunction
+
+    If wrappers is None, undo the last wrap. Otherwise removes the wrapper
+    from the chain of wrappers.
+
+    Return the removed wrapper.
+    Raise IndexError if wrapper is None and nothing to unwrap; ValueError if
+    wrapper is not None but is not found in the wrapper chain.
+    '''
+    chain = getwrapperchain(container, funcname)
+    origfn = chain.pop()
+    if wrapper is None:
+        wrapper = chain[0]
+    chain.remove(wrapper)
+    setattr(container, funcname, origfn)
+    for w in reversed(chain):
+        wrapfunction(container, funcname, w)
+    return wrapper
+
+def getwrapperchain(container, funcname):
+    '''get a chain of wrappers of a function
+
+    Return a list of functions: [newest wrapper, ..., oldest wrapper, origfunc]
+
+    The wrapper functions are the ones passed to wrapfunction, whose first
+    argument is origfunc.
+    '''
+    result = []
+    fn = getattr(container, funcname)
+    while fn:
+        assert callable(fn)
+        result.append(getattr(fn, '_unboundwrapper', fn))
+        fn = getattr(fn, '_origfunc', None)
+    return result
+
 def _disabledpaths(strip_init=False):
     '''find paths of disabled extensions. returns a dict of {name: path}
     removes /__init__.py from packages if strip_init is True'''
@@ -332,6 +380,7 b' def _disabledpaths(strip_init=False):'
         if name in exts or name in _order or name == '__init__':
             continue
         exts[name] = path
+    exts.update(_disabledextensions)
     return exts
 
 def _moduledoc(file):
@@ -494,4 +543,4 b' def moduleversion(module):'
 
 def ismoduleinternal(module):
     exttestedwith = getattr(module, 'testedwith', None)
-    return exttestedwith == "internal"
+    return exttestedwith == "ships-with-hg-core"

mercurial/fancyopts.py

@@ -12,6 +12,17 b' import getopt'
 from .i18n import _
 from . import error
 
+# Set of flags to not apply boolean negation logic on
+nevernegate = set([
+    # avoid --no-noninteractive
+    'noninteractive',
+    # These two flags are special because they cause hg to do one
+    # thing and then exit, and so aren't suitable for use in things
+    # like aliases anyway.
+    'help',
+    'version',
+    ])
+
 def gnugetopt(args, options, longoptions):
     """Parse options mostly like getopt.gnu_getopt.
 
@@ -64,6 +75,8 b' def fancyopts(args, options, state, gnu='
     shortlist = ''
     argmap = {}
     defmap = {}
+    negations = {}
+    alllong = set(o[1] for o in options)
 
     for option in options:
         if len(option) == 5:
@@ -91,6 +104,18 b' def fancyopts(args, options, state, gnu='
                 short += ':'
             if oname:
                 oname += '='
+        elif oname not in nevernegate:
+            if oname.startswith('no-'):
+                insert = oname[3:]
+            else:
+                insert = 'no-' + oname
+            # backout (as a practical example) has both --commit and
+            # --no-commit options, so we don't want to allow the
+            # negations of those flags.
+            if insert not in alllong:
+                assert ('--' + oname) not in negations
+                negations['--' + insert] = '--' + oname
+                namelist.append(insert)
         if short:
             shortlist += short
         if name:
@@ -105,6 +130,11 b' def fancyopts(args, options, state, gnu='
 
     # transfer result to state
     for opt, val in opts:
+        boolval = True
+        negation = negations.get(opt, False)
+        if negation:
+            opt = negation
+            boolval = False
         name = argmap[opt]
         obj = defmap[name]
         t = type(obj)
@@ -121,7 +151,7 b' def fancyopts(args, options, state, gnu='
         elif t is type([]):
             state[name].append(val)
         elif t is type(None) or t is type(False):
-            state[name] = True
+            state[name] = boolval
 
     # return unparsed args
     return args

mercurial/filemerge.py

@@ -19,6 +19,7 b' from . import ('
     error,
     formatter,
     match,
+    pycompat,
     scmutil,
     simplemerge,
     tagmerge,
@@ -93,7 +94,8 b' def internaltool(name, mergetype, onfail'
     '''return a decorator for populating internal merge tool table'''
     def decorator(func):
         fullname = ':' + name
-        func.__doc__ = "``%s``\n" % fullname + func.__doc__.strip()
+        func.__doc__ = (pycompat.sysstr("``%s``\n" % fullname)
+                        + func.__doc__.strip())
         internals[fullname] = func
         internals['internal:' + name] = func
         internalsdoc[fullname] = func
@@ -230,50 +232,56 b' def _matcheol(file, origfile):'
                 util.writefile(file, newdata)
 
 @internaltool('prompt', nomerge)
-def _iprompt(repo, mynode, orig, fcd, fco, fca, toolconf):
+def _iprompt(repo, mynode, orig, fcd, fco, fca, toolconf, labels=None):
     """Asks the user which of the local `p1()` or the other `p2()` version to
     keep as the merged version."""
     ui = repo.ui
     fd = fcd.path()
 
+    prompts = partextras(labels)
+    prompts['fd'] = fd
     try:
         if fco.isabsent():
             index = ui.promptchoice(
-                _("local changed %s which remote deleted\n"
+                _("local%(l)s changed %(fd)s which other%(o)s deleted\n"
                   "use (c)hanged version, (d)elete, or leave (u)nresolved?"
-                  "$$ &Changed $$ &Delete $$ &Unresolved") % fd, 2)
+                  "$$ &Changed $$ &Delete $$ &Unresolved") % prompts, 2)
             choice = ['local', 'other', 'unresolved'][index]
         elif fcd.isabsent():
             index = ui.promptchoice(
-                _("remote changed %s which local deleted\n"
+                _("other%(o)s changed %(fd)s which local%(l)s deleted\n"
                   "use (c)hanged version, leave (d)eleted, or "
                   "leave (u)nresolved?"
-                  "$$ &Changed $$ &Deleted $$ &Unresolved") % fd, 2)
+                  "$$ &Changed $$ &Deleted $$ &Unresolved") % prompts, 2)
             choice = ['other', 'local', 'unresolved'][index]
         else:
             index = ui.promptchoice(
-                _("no tool found to merge %s\n"
-                  "keep (l)ocal, take (o)ther, or leave (u)nresolved?"
-                  "$$ &Local $$ &Other $$ &Unresolved") % fd, 2)
+                _("no tool found to merge %(fd)s\n"
+                  "keep (l)ocal%(l)s, take (o)ther%(o)s, or leave (u)nresolved?"
+                  "$$ &Local $$ &Other $$ &Unresolved") % prompts, 2)
             choice = ['local', 'other', 'unresolved'][index]
 
         if choice == 'other':
-            return _iother(repo, mynode, orig, fcd, fco, fca, toolconf)
+            return _iother(repo, mynode, orig, fcd, fco, fca, toolconf,
+                           labels)
         elif choice == 'local':
-            return _ilocal(repo, mynode, orig, fcd, fco, fca, toolconf)
+            return _ilocal(repo, mynode, orig, fcd, fco, fca, toolconf,
+                           labels)
         elif choice == 'unresolved':
-            return _ifail(repo, mynode, orig, fcd, fco, fca, toolconf)
+            return _ifail(repo, mynode, orig, fcd, fco, fca, toolconf,
+                          labels)
     except error.ResponseExpected:
         ui.write("\n")
-        return _ifail(repo, mynode, orig, fcd, fco, fca, toolconf)
+        return _ifail(repo, mynode, orig, fcd, fco, fca, toolconf,
+                      labels)
 
 @internaltool('local', nomerge)
-def _ilocal(repo, mynode, orig, fcd, fco, fca, toolconf):
+def _ilocal(repo, mynode, orig, fcd, fco, fca, toolconf, labels=None):
     """Uses the local `p1()` version of files as the merged version."""
     return 0, fcd.isabsent()
 
 @internaltool('other', nomerge)
-def _iother(repo, mynode, orig, fcd, fco, fca, toolconf):
+def _iother(repo, mynode, orig, fcd, fco, fca, toolconf, labels=None):
     """Uses the other `p2()` version of files as the merged version."""
     if fco.isabsent():
         # local changed, remote deleted -- 'deleted' picked
@@ -285,7 +293,7 b' def _iother(repo, mynode, orig, fcd, fco'
     return 0, deleted
 
 @internaltool('fail', nomerge)
-def _ifail(repo, mynode, orig, fcd, fco, fca, toolconf):
+def _ifail(repo, mynode, orig, fcd, fco, fca, toolconf, labels=None):
     """
     Rather than attempting to merge files that were modified on both
     branches, it marks them as unresolved. The resolve command must be
@@ -508,11 +516,11 b' def _formatconflictmarker(repo, ctx, tem'
     # 8 for the prefix of conflict marker lines (e.g. '<<<<<<< ')
     return util.ellipsis(mark, 80 - 8)
 
-_defaultconflictmarker = ('{node|short} ' +
-    '{ifeq(tags, "tip", "", "{tags} ")}' +
-    '{if(bookmarks, "{bookmarks} ")}' +
-    '{ifeq(branch, "default", "", "{branch} ")}' +
-    '- {author|user}: {desc|firstline}')
+_defaultconflictmarker = ('{node|short} '
+                          '{ifeq(tags, "tip", "", "{tags} ")}'
+                          '{if(bookmarks, "{bookmarks} ")}'
+                          '{ifeq(branch, "default", "", "{branch} ")}'
+                          '- {author|user}: {desc|firstline}')
 
 _defaultconflictlabels = ['local', 'other']
 
@@ -537,6 +545,22 b' def _formatlabels(repo, fcd, fco, fca, l'
         newlabels.append(_formatconflictmarker(repo, ca, tmpl, labels[2], pad))
     return newlabels
 
+def partextras(labels):
+    """Return a dictionary of extra labels for use in prompts to the user
+
+    Intended use is in strings of the form "(l)ocal%(l)s".
+    """
+    if labels is None:
+        return {
+            "l": "",
+            "o": "",
+        }
+
+    return {
+        "l": " [%s]" % labels[0],
+        "o": " [%s]" % labels[1],
+    }
+
 def _filemerge(premerge, repo, mynode, orig, fcd, fco, fca, labels=None):
     """perform a 3-way merge in the working directory
 
@@ -588,7 +612,7 b' def _filemerge(premerge, repo, mynode, o'
     toolconf = tool, toolpath, binary, symlink
 
     if mergetype == nomerge:
-        r, deleted = func(repo, mynode, orig, fcd, fco, fca, toolconf)
+        r, deleted = func(repo, mynode, orig, fcd, fco, fca, toolconf, labels)
         return True, r, deleted
 
     if premerge:

mercurial/fileset.py

@@ -345,10 +345,10 b' def _sizetomax(s):'
 def size(mctx, x):
     """File size matches the given expression. Examples:
 
-    - 1k (files from 1024 to 2047 bytes)
-    - < 20k (files less than 20480 bytes)
-    - >= .5MB (files at least 524288 bytes)
-    - 4k - 1MB (files from 4096 bytes to 1048576 bytes)
+    - size('1k') - files from 1024 to 2047 bytes
+    - size('< 20k') - files less than 20480 bytes
+    - size('>= .5MB') - files at least 524288 bytes
+    - size('4k - 1MB') - files from 4096 bytes to 1048576 bytes
     """
 
     # i18n: "size" is a keyword

mercurial/formatter.py

@@ -18,25 +18,45 b' from .node import ('
 from . import (
     encoding,
     error,
+    templatekw,
     templater,
     util,
 )
 
 pickle = util.pickle
 
+class _nullconverter(object):
+    '''convert non-primitive data types to be processed by formatter'''
+    @staticmethod
+    def formatdate(date, fmt):
+        '''convert date tuple to appropriate format'''
+        return date
+    @staticmethod
+    def formatdict(data, key, value, fmt, sep):
+        '''convert dict or key-value pairs to appropriate dict format'''
+        # use plain dict instead of util.sortdict so that data can be
+        # serialized as a builtin dict in pickle output
+        return dict(data)
+    @staticmethod
+    def formatlist(data, name, fmt, sep):
+        '''convert iterable to appropriate list format'''
+        return list(data)
+
 class baseformatter(object):
-    def __init__(self, ui, topic, opts):
+    def __init__(self, ui, topic, opts, converter):
         self._ui = ui
         self._topic = topic
         self._style = opts.get("style")
         self._template = opts.get("template")
+        self._converter = converter
         self._item = None
         # function to convert node to string suitable for this output
         self.hexfunc = hex
-    def __nonzero__(self):
-        '''return False if we're not doing real templating so we can
-        skip extra work'''
-        return True
+    def __enter__(self):
+        return self
+    def __exit__(self, exctype, excvalue, traceback):
+        if exctype is None:
+            self.end()
     def _showitem(self):
         '''show a formatted item once all data is collected'''
         pass
@@ -45,6 +65,17 b' class baseformatter(object):'
         if self._item is not None:
             self._showitem()
         self._item = {}
+    def formatdate(self, date, fmt='%a %b %d %H:%M:%S %Y %1%2'):
+        '''convert date tuple to appropriate format'''
+        return self._converter.formatdate(date, fmt)
+    def formatdict(self, data, key='key', value='value', fmt='%s=%s', sep=' '):
+        '''convert dict or key-value pairs to appropriate dict format'''
+        return self._converter.formatdict(data, key, value, fmt, sep)
+    def formatlist(self, data, name, fmt='%s', sep=' '):
+        '''convert iterable to appropriate list format'''
+        # name is mandatory argument for now, but it could be optional if
+        # we have default template keyword, e.g. {item}
+        return self._converter.formatlist(data, name, fmt, sep)
     def data(self, **data):
         '''insert data into item that's not shown in default output'''
         self._item.update(data)
@@ -61,21 +92,55 b' class baseformatter(object):'
     def plain(self, text, **opts):
         '''show raw text for non-templated mode'''
         pass
+    def isplain(self):
+        '''check for plain formatter usage'''
+        return False
+    def nested(self, field):
+        '''sub formatter to store nested data in the specified field'''
+        self._item[field] = data = []
+        return _nestedformatter(self._ui, self._converter, data)
     def end(self):
         '''end output for the formatter'''
         if self._item is not None:
             self._showitem()
 
+class _nestedformatter(baseformatter):
+    '''build sub items and store them in the parent formatter'''
+    def __init__(self, ui, converter, data):
+        baseformatter.__init__(self, ui, topic='', opts={}, converter=converter)
+        self._data = data
+    def _showitem(self):
+        self._data.append(self._item)
+
+def _iteritems(data):
+    '''iterate key-value pairs in stable order'''
+    if isinstance(data, dict):
+        return sorted(data.iteritems())
+    return data
+
+class _plainconverter(object):
+    '''convert non-primitive data types to text'''
+    @staticmethod
+    def formatdate(date, fmt):
+        '''stringify date tuple in the given format'''
+        return util.datestr(date, fmt)
+    @staticmethod
+    def formatdict(data, key, value, fmt, sep):
+        '''stringify key-value pairs separated by sep'''
+        return sep.join(fmt % (k, v) for k, v in _iteritems(data))
+    @staticmethod
+    def formatlist(data, name, fmt, sep):
+        '''stringify iterable separated by sep'''
+        return sep.join(fmt % e for e in data)
+
 class plainformatter(baseformatter):
     '''the default text output scheme'''
     def __init__(self, ui, topic, opts):
-        baseformatter.__init__(self, ui, topic, opts)
+        baseformatter.__init__(self, ui, topic, opts, _plainconverter)
         if ui.debugflag:
             self.hexfunc = hex
         else:
             self.hexfunc = short
-    def __nonzero__(self):
-        return False
     def startitem(self):
         pass
     def data(self, **data):
@@ -88,12 +153,17 b' class plainformatter(baseformatter):'
             self._ui.write(deftext % fielddata, **opts)
     def plain(self, text, **opts):
         self._ui.write(text, **opts)
+    def isplain(self):
+        return True
+    def nested(self, field):
+        # nested data will be directly written to ui
+        return self
     def end(self):
         pass
 
 class debugformatter(baseformatter):
     def __init__(self, ui, topic, opts):
-        baseformatter.__init__(self, ui, topic, opts)
+        baseformatter.__init__(self, ui, topic, opts, _nullconverter)
         self._ui.write("%s = [\n" % self._topic)
     def _showitem(self):
         self._ui.write("    " + repr(self._item) + ",\n")
@@ -103,7 +173,7 b' class debugformatter(baseformatter):'
 
 class pickleformatter(baseformatter):
     def __init__(self, ui, topic, opts):
-        baseformatter.__init__(self, ui, topic, opts)
+        baseformatter.__init__(self, ui, topic, opts, _nullconverter)
         self._data = []
     def _showitem(self):
         self._data.append(self._item)
@@ -112,7 +182,11 b' class pickleformatter(baseformatter):'
         self._ui.write(pickle.dumps(self._data))
 
 def _jsonifyobj(v):
-    if isinstance(v, tuple):
+    if isinstance(v, dict):
+        xs = ['"%s": %s' % (encoding.jsonescape(k), _jsonifyobj(u))
+              for k, u in sorted(v.iteritems())]
+        return '{' + ', '.join(xs) + '}'
+    elif isinstance(v, (list, tuple)):
         return '[' + ', '.join(_jsonifyobj(e) for e in v) + ']'
     elif v is None:
         return 'null'
@@ -127,7 +201,7 b' def _jsonifyobj(v):'
 
 class jsonformatter(baseformatter):
     def __init__(self, ui, topic, opts):
-        baseformatter.__init__(self, ui, topic, opts)
+        baseformatter.__init__(self, ui, topic, opts, _nullconverter)
         self._ui.write("[")
         self._ui._first = True
     def _showitem(self):
@@ -149,9 +223,32 b' class jsonformatter(baseformatter):'
         baseformatter.end(self)
         self._ui.write("\n]\n")
 
+class _templateconverter(object):
+    '''convert non-primitive data types to be processed by templater'''
+    @staticmethod
+    def formatdate(date, fmt):
+        '''return date tuple'''
+        return date
+    @staticmethod
+    def formatdict(data, key, value, fmt, sep):
+        '''build object that can be evaluated as either plain string or dict'''
+        data = util.sortdict(_iteritems(data))
+        def f():
+            yield _plainconverter.formatdict(data, key, value, fmt, sep)
+        return templatekw._hybrid(f(), data, lambda k: {key: k, value: data[k]},
+                                  lambda d: fmt % (d[key], d[value]))
+    @staticmethod
+    def formatlist(data, name, fmt, sep):
+        '''build object that can be evaluated as either plain string or list'''
+        data = list(data)
+        def f():
+            yield _plainconverter.formatlist(data, name, fmt, sep)
+        return templatekw._hybrid(f(), data, lambda x: {name: x},
+                                  lambda d: fmt % d[name])
+
 class templateformatter(baseformatter):
     def __init__(self, ui, topic, opts):
-        baseformatter.__init__(self, ui, topic, opts)
+        baseformatter.__init__(self, ui, topic, opts, _templateconverter)
         self._topic = topic
         self._t = gettemplater(ui, topic, opts.get('template', ''))
     def _showitem(self):

mercurial/hbisect.py

@@ -139,6 +139,19 b' def bisect(changelog, state):'
 
     return ([best_node], tot, good)
 
+def extendrange(repo, state, nodes, good):
+    # bisect is incomplete when it ends on a merge node and
+    # one of the parent was not checked.
+    parents = repo[nodes[0]].parents()
+    if len(parents) > 1:
+        if good:
+            side = state['bad']
+        else:
+            side = state['good']
+        num = len(set(i.node() for i in parents) & set(side))
+        if num == 1:
+            return parents[0].ancestor(parents[1])
+    return None
 
 def load_state(repo):
     state = {'current': [], 'good': [], 'bad': [], 'skip': []}
@@ -159,6 +172,22 b' def save_state(repo, state):'
                 f.write("%s %s\n" % (kind, hex(node)))
         f.close()
 
+def resetstate(repo):
+    """remove any bisect state from the repository"""
+    if repo.vfs.exists("bisect.state"):
+        repo.vfs.unlink("bisect.state")
+
+def checkstate(state):
+    """check we have both 'good' and 'bad' to define a range
+
+    Raise Abort exception otherwise."""
+    if state['good'] and state['bad']:
+        return True
+    if not state['good']:
+        raise error.Abort(_('cannot bisect (no known good revisions)'))
+    else:
+        raise error.Abort(_('cannot bisect (no known bad revisions)'))
+
 def get(repo, status):
     """
     Return a list of revision(s) that match the given status:
@@ -261,3 +290,29 b' def shortlabel(label):'
         return label[0].upper()
 
     return None
+
+def printresult(ui, repo, state, displayer, nodes, good):
+    if len(nodes) == 1:
+        # narrowed it down to a single revision
+        if good:
+            ui.write(_("The first good revision is:\n"))
+        else:
+            ui.write(_("The first bad revision is:\n"))
+        displayer.show(repo[nodes[0]])
+        extendnode = extendrange(repo, state, nodes, good)
+        if extendnode is not None:
+            ui.write(_('Not all ancestors of this changeset have been'
+                       ' checked.\nUse bisect --extend to continue the '
+                       'bisection from\nthe common ancestor, %s.\n')
+                     % extendnode)
+    else:
+        # multiple possible revisions
+        if good:
+            ui.write(_("Due to skipped revisions, the first "
+                    "good revision could be any of:\n"))
+        else:
+            ui.write(_("Due to skipped revisions, the first "
+                    "bad revision could be any of:\n"))
+        for n in nodes:
+            displayer.show(repo[n])
+    displayer.close()

mercurial/help.py

@@ -184,14 +184,16 b' def loaddoc(topic, subdir=None):'
     return loader
 
 internalstable = sorted([
-    (['bundles'], _('container for exchange of repository data'),
+    (['bundles'], _('Bundles'),
      loaddoc('bundles', subdir='internals')),
-    (['changegroups'], _('representation of revlog data'),
+    (['changegroups'], _('Changegroups'),
      loaddoc('changegroups', subdir='internals')),
-    (['requirements'], _('repository requirements'),
+    (['requirements'], _('Repository Requirements'),
      loaddoc('requirements', subdir='internals')),
-    (['revlogs'], _('revision storage mechanism'),
+    (['revlogs'], _('Revision Logs'),
      loaddoc('revlogs', subdir='internals')),
+    (['wireprotocol'], _('Wire Protocol'),
+     loaddoc('wireprotocol', subdir='internals')),
 ])
 
 def internalshelp(ui):
@@ -356,8 +358,8 b' def help_(ui, name, unknowncmd=False, fu'
             mod = extensions.find(name)
             doc = gettext(mod.__doc__) or ''
             if '\n' in doc.strip():
-                msg = _('(use "hg help -e %s" to show help for '
-                        'the %s extension)') % (name, name)
+                msg = _("(use 'hg help -e %s' to show help for "
+                        "the %s extension)") % (name, name)
                 rst.append('\n%s\n' % msg)
         except KeyError:
             pass
@@ -372,7 +374,7 b' def help_(ui, name, unknowncmd=False, fu'
 
         if not ui.verbose:
             if not full:
-                rst.append(_('\n(use "hg %s -h" to show more help)\n')
+                rst.append(_("\n(use 'hg %s -h' to show more help)\n")
                            % name)
             elif not ui.quiet:
                 rst.append(_('\n(some details hidden, use --verbose '
@@ -448,21 +450,21 b' def help_(ui, name, unknowncmd=False, fu'
             rst.append('\n%s\n' % optrst(_("global options"),
                                          commands.globalopts, ui.verbose))
             if name == 'shortlist':
-                rst.append(_('\n(use "hg help" for the full list '
-                             'of commands)\n'))
+                rst.append(_("\n(use 'hg help' for the full list "
+                             "of commands)\n"))
         else:
             if name == 'shortlist':
-                rst.append(_('\n(use "hg help" for the full list of commands '
-                             'or "hg -v" for details)\n'))
+                rst.append(_("\n(use 'hg help' for the full list of commands "
+                             "or 'hg -v' for details)\n"))
             elif name and not full:
-                rst.append(_('\n(use "hg help %s" to show the full help '
-                             'text)\n') % name)
+                rst.append(_("\n(use 'hg help %s' to show the full help "
+                             "text)\n") % name)
             elif name and cmds and name in cmds.keys():
-                rst.append(_('\n(use "hg help -v -e %s" to show built-in '
-                             'aliases and global options)\n') % name)
+                rst.append(_("\n(use 'hg help -v -e %s' to show built-in "
+                             "aliases and global options)\n") % name)
             else:
-                rst.append(_('\n(use "hg help -v%s" to show built-in aliases '
-                             'and global options)\n')
+                rst.append(_("\n(use 'hg help -v%s' to show built-in aliases "
+                             "and global options)\n")
                            % (name and " " + name or ""))
         return rst
 
@@ -496,8 +498,8 b' def help_(ui, name, unknowncmd=False, fu'
 
         try:
             cmdutil.findcmd(name, commands.table)
-            rst.append(_('\nuse "hg help -c %s" to see help for '
-                       'the %s command\n') % (name, name))
+            rst.append(_("\nuse 'hg help -c %s' to see help for "
+                       "the %s command\n") % (name, name))
         except error.UnknownCommand:
             pass
         return rst
@@ -534,8 +536,8 b' def help_(ui, name, unknowncmd=False, fu'
             modcmds = set([c.partition('|')[0] for c in ct])
             rst.extend(helplist(modcmds.__contains__))
         else:
-            rst.append(_('(use "hg help extensions" for information on enabling'
-                       ' extensions)\n'))
+            rst.append(_("(use 'hg help extensions' for information on enabling"
+                       " extensions)\n"))
         return rst
 
     def helpextcmd(name, subtopic=None):
@@ -547,8 +549,8 b' def help_(ui, name, unknowncmd=False, fu'
                               "extension:") % cmd, {ext: doc}, indent=4,
                        showdeprecated=True)
         rst.append('\n')
-        rst.append(_('(use "hg help extensions" for information on enabling '
-                   'extensions)\n'))
+        rst.append(_("(use 'hg help extensions' for information on enabling "
+                   "extensions)\n"))
         return rst
 
 
@@ -573,7 +575,7 b' def help_(ui, name, unknowncmd=False, fu'
                 rst.append('\n')
         if not rst:
             msg = _('no matches')
-            hint = _('try "hg help" for a list of topics')
+            hint = _("try 'hg help' for a list of topics")
             raise error.Abort(msg, hint=hint)
     elif name and name != 'shortlist':
         queries = []
@@ -596,7 +598,7 b' def help_(ui, name, unknowncmd=False, fu'
                 raise error.UnknownCommand(name)
             else:
                 msg = _('no such help topic: %s') % name
-                hint = _('try "hg help --keyword %s"') % name
+                hint = _("try 'hg help --keyword %s'") % name
                 raise error.Abort(msg, hint=hint)
     else:
         # program name

mercurial/help/config.txt

@@ -1393,6 +1393,12 b" collected during profiling, while 'profi"
 statistical text report generated from the profiling data. The
 profiling is done using lsprof.
 
+``enabled``
+    Enable the profiler.
+    (default: false)
+
+    This is equivalent to passing ``--profile`` on the command line.
+
 ``type``
     The type of profiler to use.
     (default: ls)
@@ -1557,6 +1563,21 b' Controls generic server settings.'
     repositories to the exchange format required by the bundle1 data
     format can consume a lot of CPU.
 
+``zliblevel``
+    Integer between ``-1`` and ``9`` that controls the zlib compression level
+    for wire protocol commands that send zlib compressed output (notably the
+    commands that send repository history data).
+
+    The default (``-1``) uses the default zlib compression level, which is
+    likely equivalent to ``6``. ``0`` means no compression. ``9`` means
+    maximum compression.
+
+    Setting this option allows server operators to make trade-offs between
+    bandwidth and CPU used. Lowering the compression lowers CPU utilization
+    but sends more bytes to clients.
+
+    This option only impacts the HTTP server.
+
 ``smtp``
 --------
 

mercurial/help/internals/bundles.txt

@@ -1,6 +1,3 b''
-Bundles
-=======
-
 A bundle is a container for repository data.
 
 Bundles are used as standalone files as well as the interchange format
@@ -8,7 +5,7 b' over the wire protocol used when two Mer'
 each other.
 
 Headers
--------
+=======
 
 Bundles produced since Mercurial 0.7 (September 2005) have a 4 byte
 header identifying the major bundle type. The header always begins with

mercurial/help/internals/changegroups.txt

@@ -1,6 +1,3 b''
-Changegroups
-============
-
 Changegroups are representations of repository revlog data, specifically
 the changelog, manifest, and filelogs.
 
@@ -35,7 +32,7 b' There is a special case chunk that has 0'
 call this an *empty chunk*.
 
 Delta Groups
-------------
+============
 
 A *delta group* expresses the content of a revlog as a series of deltas,
 or patches against previous revisions.
@@ -111,21 +108,21 b' changegroup. This allows the delta to be'
 which can result in smaller deltas and more efficient encoding of data.
 
 Changeset Segment
------------------
+=================
 
 The *changeset segment* consists of a single *delta group* holding
 changelog data. It is followed by an *empty chunk* to denote the
 boundary to the *manifests segment*.
 
 Manifest Segment
-----------------
+================
 
 The *manifest segment* consists of a single *delta group* holding
 manifest data. It is followed by an *empty chunk* to denote the boundary
 to the *filelogs segment*.
 
 Filelogs Segment
-----------------
+================
 
 The *filelogs* segment consists of multiple sub-segments, each
 corresponding to an individual file whose data is being described::
@@ -154,4 +151,3 b' Each filelog sub-segment consists of the'
 
 That is, a *chunk* consisting of the filename (not terminated or padded)
 followed by N chunks constituting the *delta group* for this file.
-

mercurial/help/internals/requirements.txt

@@ -1,5 +1,3 b''
-Requirements
-============
 
 Repositories contain a file (``.hg/requires``) containing a list of
 features/capabilities that are *required* for clients to interface
@@ -19,7 +17,7 b' The following sections describe the requ'
 Mercurial core distribution.
 
 revlogv1
---------
+========
 
 When present, revlogs are version 1 (RevlogNG). RevlogNG was introduced
 in 2006. The ``revlogv1`` requirement has been enabled by default
@@ -28,7 +26,7 b' since the ``requires`` file was introduc'
 If this requirement is not present, version 0 revlogs are assumed.
 
 store
------
+=====
 
 The *store* repository layout should be used.
 
@@ -36,7 +34,7 b' This requirement has been enabled by def'
 was introduced in Mercurial 0.9.2.
 
 fncache
--------
+=======
 
 The *fncache* repository layout should be used.
 
@@ -48,7 +46,7 b' enabled (which is the default behavior).'
 1.1 (released December 2008).
 
 shared
-------
+======
 
 Denotes that the store for a repository is shared from another location
 (defined by the ``.hg/sharedpath`` file).
@@ -58,7 +56,7 b' This requirement is set when a repositor'
 The requirement was added in Mercurial 1.3 (released July 2009).
 
 dotencode
----------
+=========
 
 The *dotencode* repository layout should be used.
 
@@ -70,7 +68,7 b' is enabled (which is the default behavio'
 Mercurial 1.7 (released November 2010).
 
 parentdelta
------------
+===========
 
 Denotes a revlog delta encoding format that was experimental and
 replaced by *generaldelta*. It should not be seen in the wild because
@@ -80,7 +78,7 b' This requirement was added in Mercurial '
 1.9.
 
 generaldelta
-------------
+============
 
 Revlogs should be created with the *generaldelta* flag enabled. The
 generaldelta flag will cause deltas to be encoded against a parent
@@ -91,7 +89,7 b' July 2011). The requirement was disabled'
 default until Mercurial 3.7 (released February 2016).
 
 manifestv2
-----------
+==========
 
 Denotes that version 2 of manifests are being used.
 
@@ -100,7 +98,7 b' May 2015). The requirement is currently '
 by default.
 
 treemanifest
-------------
+============
 
 Denotes that tree manifests are being used. Tree manifests are
 one manifest per directory (as opposed to a single flat manifest).

mercurial/help/internals/revlogs.txt

@@ -1,6 +1,3 b''
-Revlogs
-=======
-
 Revision logs - or *revlogs* - are an append only data structure for
 storing discrete entries, or *revisions*. They are the primary storage
 mechanism of repository data.
@@ -28,7 +25,7 b' revision #0 and the second is revision #'
 used to mean *does not exist* or *not defined*.
 
 File Format
------------
+===========
 
 A revlog begins with a 32-bit big endian integer holding version info
 and feature flags. This integer is shared with the first revision
@@ -77,7 +74,7 b' possibly located between index entries. '
 below.
 
 RevlogNG Format
----------------
+===============
 
 RevlogNG (version 1) begins with an index describing the revisions in
 the revlog. If the ``inline`` flag is set, revision data is stored inline,
@@ -129,7 +126,7 b' The first 4 bytes of the revlog are shar'
 and the 6 byte absolute offset field from the first revlog entry.
 
 Delta Chains
-------------
+============
 
 Revision data is encoded as a chain of *chunks*. Each chain begins with
 the compressed original full text for that revision. Each subsequent
@@ -153,7 +150,7 b' by default in Mercurial 3.7) activates t'
 computed against an arbitrary revision (almost certainly a parent revision).
 
 File Storage
-------------
+============
 
 Revlogs logically consist of an index (metadata of entries) and
 revision data. This data may be stored together in a single file or in
@@ -172,7 +169,7 b' The actual layout of revlog files on dis'
 (possibly containing inline data) and a ``.d`` file holds the revision data.
 
 Revision Entries
-----------------
+================
 
 Revision entries consist of an optional 1 byte header followed by an
 encoding of the revision data. The headers are as follows:
@@ -187,7 +184,7 b' x (0x78)'
    The 0x78 value is actually the first byte of the zlib header (CMF byte).
 
 Hash Computation
-----------------
+================
 
 The hash of the revision is stored in the index and is used both as a primary
 key and for data integrity verification.

mercurial/help/revsets.txt

@@ -12,11 +12,17 b' Special characters can be used in quoted'
 e.g., ``\n`` is interpreted as a newline. To prevent them from being
 interpreted, strings can be prefixed with ``r``, e.g. ``r'...'``.
 
+Prefix
+======
+
 There is a single prefix operator:
 
 ``not x``
   Changesets not in x. Short form is ``! x``.
 
+Infix
+=====
+
 These are the supported infix operators:
 
 ``x::y``
@@ -55,16 +61,40 b' These are the supported infix operators:'
 ``x~n``
   The nth first ancestor of x; ``x~0`` is x; ``x~3`` is ``x^^^``.
 
+``x ## y``
+  Concatenate strings and identifiers into one string.
+
+  All other prefix, infix and postfix operators have lower priority than
+  ``##``. For example, ``a1 ## a2~2`` is equivalent to ``(a1 ## a2)~2``.
+
+  For example::
+
+    [revsetalias]
+    issue(a1) = grep(r'\bissue[ :]?' ## a1 ## r'\b|\bbug\(' ## a1 ## r'\)')
+
+    ``issue(1234)`` is equivalent to
+    ``grep(r'\bissue[ :]?1234\b|\bbug\(1234\)')``
+    in this case. This matches against all of "issue 1234", "issue:1234",
+    "issue1234" and "bug(1234)".
+
+Postfix
+=======
+
 There is a single postfix operator:
 
 ``x^``
   Equivalent to ``x^1``, the first parent of each changeset in x.
 
+Predicates
+==========
 
 The following predicates are supported:
 
 .. predicatesmarker
 
+Aliases
+=======
+
 New predicates (known as "aliases") can be defined, using any combination of
 existing predicates or other aliases. An alias definition looks like::
 
@@ -86,18 +116,8 b' For example,'
 defines three aliases, ``h``, ``d``, and ``rs``. ``rs(0:tip, author)`` is
 exactly equivalent to ``reverse(sort(0:tip, author))``.
 
-An infix operator ``##`` can concatenate strings and identifiers into
-one string. For example::
-
-  [revsetalias]
-  issue(a1) = grep(r'\bissue[ :]?' ## a1 ## r'\b|\bbug\(' ## a1 ## r'\)')
-
-``issue(1234)`` is equivalent to ``grep(r'\bissue[ :]?1234\b|\bbug\(1234\)')``
-in this case. This matches against all of "issue 1234", "issue:1234",
-"issue1234" and "bug(1234)".
-
-All other prefix, infix and postfix operators have lower priority than
-``##``. For example, ``a1 ## a2~2`` is equivalent to ``(a1 ## a2)~2``.
+Equivalents
+===========
 
 Command line equivalents for :hg:`log`::
 
@@ -110,6 +130,9 b' Command line equivalents for :hg:`log`::'
   -P x  ->  !::x
   -l x  ->  limit(expr, x)
 
+Examples
+========
+
 Some sample queries:
 
 - Changesets on the default branch::

mercurial/help/templates.txt

@@ -43,6 +43,15 b' In addition to filters, there are some b'
 
 .. functionsmarker
 
+We provide a limited set of infix arithmetic operations on integers::
+
+  + for addition
+  - for subtraction
+  * for multiplication
+  / for floor division (division rounded to integer nearest -infinity)
+
+Division fulfils the law x = x / y + mod(x, y).
+
 Also, for any expression that returns a list, there is a list operator::
 
     expr % "{template}"
@@ -95,6 +104,10 b' Some sample command line templates:'
 
    $ hg log -r 0 --template "files: {join(files, ', ')}\n"
 
+- Join the list of files ending with ".py" with a ", "::
+
+   $ hg log -r 0 --template "pythonfiles: {join(files('**.py'), ', ')}\n"
+
 - Separate non-empty arguments by a " "::
 
    $ hg log -r 0 --template "{separate(' ', node, bookmarks, tags}\n"

mercurial/hg.py

@@ -195,7 +195,7 b' def defaultdest(source):'
         return ''
     return os.path.basename(os.path.normpath(path))
 
-def share(ui, source, dest=None, update=True, bookmarks=True):
+def share(ui, source, dest=None, update=True, bookmarks=True, defaultpath=None):
     '''create a shared repository'''
 
     if not islocal(source):
@@ -240,10 +240,10 b' def share(ui, source, dest=None, update='
     destvfs.write('sharedpath', sharedpath)
 
     r = repository(ui, destwvfs.base)
-    postshare(srcrepo, r, bookmarks=bookmarks)
+    postshare(srcrepo, r, bookmarks=bookmarks, defaultpath=defaultpath)
     _postshareupdate(r, update, checkout=checkout)
 
-def postshare(sourcerepo, destrepo, bookmarks=True):
+def postshare(sourcerepo, destrepo, bookmarks=True, defaultpath=None):
     """Called after a new shared repo is created.
 
     The new repo only has a requirements file and pointer to the source.
@@ -252,17 +252,18 b' def postshare(sourcerepo, destrepo, book'
     Extensions can wrap this function and write additional entries to
     destrepo/.hg/shared to indicate additional pieces of data to be shared.
     """
-    default = sourcerepo.ui.config('paths', 'default')
+    default = defaultpath or sourcerepo.ui.config('paths', 'default')
     if default:
         fp = destrepo.vfs("hgrc", "w", text=True)
         fp.write("[paths]\n")
         fp.write("default = %s\n" % default)
         fp.close()
 
-    if bookmarks:
-        fp = destrepo.vfs('shared', 'w')
-        fp.write(sharedbookmarks + '\n')
-        fp.close()
+    with destrepo.wlock():
+        if bookmarks:
+            fp = destrepo.vfs('shared', 'w')
+            fp.write(sharedbookmarks + '\n')
+            fp.close()
 
 def _postshareupdate(repo, update, checkout=None):
     """Maybe perform a working directory update after a shared repo is created.
@@ -373,8 +374,15 b' def clonewithshare(ui, peeropts, sharepa'
             clone(ui, peeropts, source, dest=sharepath, pull=True,
                   rev=rev, update=False, stream=stream)
 
+    # Resolve the value to put in [paths] section for the source.
+    if islocal(source):
+        defaultpath = os.path.abspath(util.urllocalpath(source))
+    else:
+        defaultpath = source
+
     sharerepo = repository(ui, path=sharepath)
-    share(ui, sharerepo, dest=dest, update=False, bookmarks=False)
+    share(ui, sharerepo, dest=dest, update=False, bookmarks=False,
+          defaultpath=defaultpath)
 
     # We need to perform a pull against the dest repo to fetch bookmarks
     # and other non-store data that isn't shared by default. In the case of
@@ -737,20 +745,22 b' def updatetotally(ui, repo, checkout, br'
             if movemarkfrom == repo['.'].node():
                 pass # no-op update
             elif bookmarks.update(repo, [movemarkfrom], repo['.'].node()):
-                ui.status(_("updating bookmark %s\n") % repo._activebookmark)
+                b = ui.label(repo._activebookmark, 'bookmarks.active')
+                ui.status(_("updating bookmark %s\n") % b)
             else:
                 # this can happen with a non-linear update
-                ui.status(_("(leaving bookmark %s)\n") %
-                          repo._activebookmark)
+                b = ui.label(repo._activebookmark, 'bookmarks')
+                ui.status(_("(leaving bookmark %s)\n") % b)
                 bookmarks.deactivate(repo)
         elif brev in repo._bookmarks:
             if brev != repo._activebookmark:
-                ui.status(_("(activating bookmark %s)\n") % brev)
+                b = ui.label(brev, 'bookmarks.active')
+                ui.status(_("(activating bookmark %s)\n") % b)
             bookmarks.activate(repo, brev)
         elif brev:
             if repo._activebookmark:
-                ui.status(_("(leaving bookmark %s)\n") %
-                          repo._activebookmark)
+                b = ui.label(repo._activebookmark, 'bookmarks')
+                ui.status(_("(leaving bookmark %s)\n") % b)
             bookmarks.deactivate(repo)
 
         if warndest:
@@ -758,10 +768,11 b' def updatetotally(ui, repo, checkout, br'
 
     return ret
 
-def merge(repo, node, force=None, remind=True, mergeforce=False):
+def merge(repo, node, force=None, remind=True, mergeforce=False, labels=None):
     """Branch merge with node, resolving changes. Return true if any
     unresolved conflicts."""
-    stats = mergemod.update(repo, node, True, force, mergeforce=mergeforce)
+    stats = mergemod.update(repo, node, True, force, mergeforce=mergeforce,
+                            labels=labels)
     _showstats(repo, stats)
     if stats[3]:
         repo.ui.status(_("use 'hg resolve' to retry unresolved file merges "

mercurial/hgweb/hgweb_mod.py

@@ -28,6 +28,7 b' from .. import ('
     error,
     hg,
     hook,
+    profiling,
     repoview,
     templatefilters,
     templater,
@@ -305,8 +306,9 b' class hgweb(object):'
         should be using instances of this class as the WSGI application.
         """
         with self._obtainrepo() as repo:
-            for r in self._runwsgi(req, repo):
-                yield r
+            with profiling.maybeprofile(repo.ui):
+                for r in self._runwsgi(req, repo):
+                    yield r
 
     def _runwsgi(self, req, repo):
         rctx = requestcontext(self, repo)

mercurial/hgweb/hgwebdir_mod.py

@@ -31,6 +31,7 b' from .. import ('
     encoding,
     error,
     hg,
+    profiling,
     scmutil,
     templater,
     ui as uimod,
@@ -217,6 +218,11 b' class hgwebdir(object):'
         return False
 
     def run_wsgi(self, req):
+        with profiling.maybeprofile(self.ui):
+            for r in self._runwsgi(req):
+                yield r
+
+    def _runwsgi(self, req):
         try:
             self.refresh()
 

mercurial/hgweb/protocol.py

@@ -73,14 +73,30 b' class webproto(wireproto.abstractserverp'
         val = self.ui.fout.getvalue()
         self.ui.ferr, self.ui.fout = self.oldio
         return val
-    def groupchunks(self, cg):
-        z = zlib.compressobj()
-        while True:
-            chunk = cg.read(4096)
-            if not chunk:
-                break
-            yield z.compress(chunk)
+
+    def groupchunks(self, fh):
+        def getchunks():
+            while True:
+                chunk = fh.read(32768)
+                if not chunk:
+                    break
+                yield chunk
+
+        return self.compresschunks(getchunks())
+
+    def compresschunks(self, chunks):
+        # Don't allow untrusted settings because disabling compression or
+        # setting a very high compression level could lead to flooding
+        # the server's network or CPU.
+        z = zlib.compressobj(self.ui.configint('server', 'zliblevel', -1))
+        for chunk in chunks:
+            data = z.compress(chunk)
+            # Not all calls to compress() emit data. It is cheaper to inspect
+            # that here than to send it via the generator.
+            if data:
+                yield data
         yield z.flush()
+
     def _client(self):
         return 'remote:%s:%s:%s' % (
             self.req.env.get('wsgi.url_scheme') or 'http',

mercurial/hgweb/server.py

@@ -263,7 +263,7 b' def openlog(opt, default):'
         return open(opt, 'a')
     return default
 
-class MercurialHTTPServer(object, _mixin, httpservermod.httpserver):
+class MercurialHTTPServer(_mixin, httpservermod.httpserver, object):
 
     # SO_REUSEADDR has broken semantics on windows
     if os.name == 'nt':

mercurial/hgweb/webcommands.py

@@ -31,7 +31,6 b' from .. import ('
     encoding,
     error,
     graphmod,
-    patch,
     revset,
     scmutil,
     templatefilters,
@@ -861,8 +860,6 b' def annotate(web, req, tmpl):'
     fctx = webutil.filectx(web.repo, req)
     f = fctx.path()
     parity = paritygen(web.stripecount)
-    diffopts = patch.difffeatureopts(web.repo.ui, untrusted=True,
-                                     section='annotate', whitespace=True)
 
     def parents(f):
         for p in f.parents():
@@ -877,8 +874,8 b' def annotate(web, req, tmpl):'
                   or 'application/octet-stream')
             lines = [((fctx.filectx(fctx.filerev()), 1), '(binary:%s)' % mt)]
         else:
-            lines = fctx.annotate(follow=True, linenumber=True,
-                                  diffopts=diffopts)
+            lines = webutil.annotate(fctx, web.repo.ui)
+
         previousrev = None
         blockparitygen = paritygen(1)
         for lineno, ((f, targetline), l) in enumerate(lines):

mercurial/hgweb/webutil.py

@@ -164,6 +164,11 b' class _siblings(object):'
     def __len__(self):
         return len(self.siblings)
 
+def annotate(fctx, ui):
+    diffopts = patch.difffeatureopts(ui, untrusted=True,
+                                     section='annotate', whitespace=True)
+    return fctx.annotate(follow=True, linenumber=True, diffopts=diffopts)
+
 def parents(ctx, hide=None):
     if isinstance(ctx, context.basefilectx):
         introrev = ctx.introrev()

mercurial/httpconnection.py

@@ -58,6 +58,12 b' class httpsendfile(object):'
                          unit=_('kb'), total=self._total)
         return ret
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+
 # moved here from url.py to avoid a cycle
 def readauthforuri(ui, uri, user):
     # Read configuration

mercurial/i18n.py

@@ -12,7 +12,10 b' import locale'
 import os
 import sys
 
-from . import encoding
+from . import (
+    encoding,
+    pycompat,
+)
 
 # modelled after templater.templatepath:
 if getattr(sys, 'frozen', None) is not None:
@@ -27,10 +30,10 b' except NameError:'
 
 _languages = None
 if (os.name == 'nt'
-    and 'LANGUAGE' not in os.environ
-    and 'LC_ALL' not in os.environ
-    and 'LC_MESSAGES' not in os.environ
-    and 'LANG' not in os.environ):
+    and 'LANGUAGE' not in encoding.environ
+    and 'LC_ALL' not in encoding.environ
+    and 'LC_MESSAGES' not in encoding.environ
+    and 'LANG' not in encoding.environ):
     # Try to detect UI language by "User Interface Language Management" API
     # if no locale variables are set. Note that locale.getdefaultlocale()
     # uses GetLocaleInfo(), which may be different from UI language.
@@ -46,7 +49,7 b" if (os.name == 'nt'"
 _ugettext = None
 
 def setdatapath(datapath):
-    localedir = os.path.join(datapath, 'locale')
+    localedir = os.path.join(datapath, pycompat.sysstr('locale'))
     t = gettextmod.translation('hg', localedir, _languages, fallback=True)
     global _ugettext
     try:
@@ -85,16 +88,18 b' def gettext(message):'
             # means u.encode(sys.getdefaultencoding()).decode(enc). Since
             # the Python encoding defaults to 'ascii', this fails if the
             # translated string use non-ASCII characters.
-            _msgcache[message] = u.encode(encoding.encoding, "replace")
+            encodingstr = pycompat.sysstr(encoding.encoding)
+            _msgcache[message] = u.encode(encodingstr, "replace")
         except LookupError:
             # An unknown encoding results in a LookupError.
             _msgcache[message] = message
     return _msgcache[message]
 
 def _plain():
-    if 'HGPLAIN' not in os.environ and 'HGPLAINEXCEPT' not in os.environ:
+    if ('HGPLAIN' not in encoding.environ
+        and 'HGPLAINEXCEPT' not in encoding.environ):
         return False
-    exceptions = os.environ.get('HGPLAINEXCEPT', '').strip().split(',')
+    exceptions = encoding.environ.get('HGPLAINEXCEPT', '').strip().split(',')
     return 'i18n' not in exceptions
 
 if _plain():

mercurial/localrepo.py

@@ -149,14 +149,18 b' class localpeer(peer.peerrepository):'
 
     def getbundle(self, source, heads=None, common=None, bundlecaps=None,
                   **kwargs):
-        cg = exchange.getbundle(self._repo, source, heads=heads,
-                                common=common, bundlecaps=bundlecaps, **kwargs)
+        chunks = exchange.getbundlechunks(self._repo, source, heads=heads,
+                                          common=common, bundlecaps=bundlecaps,
+                                          **kwargs)
+        cb = util.chunkbuffer(chunks)
+
         if bundlecaps is not None and 'HG20' in bundlecaps:
             # When requesting a bundle2, getbundle returns a stream to make the
             # wire level function happier. We need to build a proper object
             # from it in local peer.
-            cg = bundle2.getunbundler(self.ui, cg)
-        return cg
+            return bundle2.getunbundler(self.ui, cb)
+        else:
+            return changegroup.getunbundler('01', cb, None)
 
     # TODO We might want to move the next two calls into legacypeer and add
     # unbundle instead.
@@ -504,8 +508,9 b' class localrepository(object):'
     def manifest(self):
         return manifest.manifest(self.svfs)
 
-    def dirlog(self, dir):
-        return self.manifest.dirlog(dir)
+    @property
+    def manifestlog(self):
+        return manifest.manifestlog(self.svfs, self)
 
     @repofilecache('dirstate')
     def dirstate(self):
@@ -1007,8 +1012,7 b' class localrepository(object):'
     def transaction(self, desc, report=None):
         if (self.ui.configbool('devel', 'all-warnings')
                 or self.ui.configbool('devel', 'check-locks')):
-            l = self._lockref and self._lockref()
-            if l is None or not l.held:
+            if self._currentlock(self._lockref) is None:
                 raise RuntimeError('programming error: transaction requires '
                                    'locking')
         tr = self.currenttransaction()
@@ -1246,6 +1250,13 b' class localrepository(object):'
             delattr(self.unfiltered(), 'dirstate')
 
     def invalidate(self, clearfilecache=False):
+        '''Invalidates both store and non-store parts other than dirstate
+
+        If a transaction is running, invalidation of store is omitted,
+        because discarding in-memory changes might cause inconsistency
+        (e.g. incomplete fncache causes unintentional failure, but
+        redundant one doesn't).
+        '''
         unfiltered = self.unfiltered() # all file caches are stored unfiltered
         for k in self._filecache.keys():
             # dirstate is invalidated separately in invalidatedirstate()
@@ -1259,7 +1270,11 b' class localrepository(object):'
             except AttributeError:
                 pass
         self.invalidatecaches()
-        self.store.invalidatecaches()
+        if not self.currenttransaction():
+            # TODO: Changing contents of store outside transaction
+            # causes inconsistency. We should make in-memory store
+            # changes detectable, and abort if changed.
+            self.store.invalidatecaches()
 
     def invalidateall(self):
         '''Fully invalidates both store and non-store parts, causing the
@@ -1268,6 +1283,7 b' class localrepository(object):'
         self.invalidate()
         self.invalidatedirstate()
 
+    @unfilteredmethod
     def _refreshfilecachestats(self, tr):
         """Reload stats of cached files so that they are flagged as valid"""
         for k, ce in self._filecache.items():
@@ -1290,8 +1306,15 b' class localrepository(object):'
         except error.LockHeld as inst:
             if not wait:
                 raise
-            self.ui.warn(_("waiting for lock on %s held by %r\n") %
-                         (desc, inst.locker))
+            # show more details for new-style locks
+            if ':' in inst.locker:
+                host, pid = inst.locker.split(":", 1)
+                self.ui.warn(
+                    _("waiting for lock on %s held by process %r "
+                      "on host %r\n") % (desc, pid, host))
+            else:
+                self.ui.warn(_("waiting for lock on %s held by %r\n") %
+                             (desc, inst.locker))
             # default to 600 seconds timeout
             l = lockmod.lock(vfs, lockname,
                              int(self.ui.config("ui", "timeout", "600")),
@@ -1320,8 +1343,8 b' class localrepository(object):'
 
         If both 'lock' and 'wlock' must be acquired, ensure you always acquires
         'wlock' first to avoid a dead-lock hazard.'''
-        l = self._lockref and self._lockref()
-        if l is not None and l.held:
+        l = self._currentlock(self._lockref)
+        if l is not None:
             l.lock()
             return l
 
@@ -1352,8 +1375,7 b' class localrepository(object):'
         # acquisition would not cause dead-lock as they would just fail.
         if wait and (self.ui.configbool('devel', 'all-warnings')
                      or self.ui.configbool('devel', 'check-locks')):
-            l = self._lockref and self._lockref()
-            if l is not None and l.held:
+            if self._currentlock(self._lockref) is not None:
                 self.ui.develwarn('"wlock" acquired after "lock"')
 
         def unlock():
@@ -1607,8 +1629,8 b' class localrepository(object):'
             ms = mergemod.mergestate.read(self)
 
             if list(ms.unresolved()):
-                raise error.Abort(_('unresolved merge conflicts '
-                                    '(see "hg help resolve")'))
+                raise error.Abort(_("unresolved merge conflicts "
+                                    "(see 'hg help resolve')"))
             if ms.mdstate() != 's' or list(ms.driverresolved()):
                 raise error.Abort(_('driver-resolved merge conflicts'),
                                   hint=_('run "hg resolve --all" to resolve'))
@@ -1714,9 +1736,9 b' class localrepository(object):'
                 drop = [f for f in removed if f in m]
                 for f in drop:
                     del m[f]
-                mn = self.manifest.add(m, trp, linkrev,
-                                       p1.manifestnode(), p2.manifestnode(),
-                                       added, drop)
+                mn = self.manifestlog.add(m, trp, linkrev,
+                                          p1.manifestnode(), p2.manifestnode(),
+                                          added, drop)
                 files = changed + removed
             else:
                 mn = p1.manifestnode()

mercurial/mail.py

@@ -8,6 +8,8 b''
 from __future__ import absolute_import, print_function
 
 import email
+import email.charset
+import email.header
 import os
 import quopri
 import smtplib
@@ -23,7 +25,7 b' from . import ('
     util,
 )
 
-_oldheaderinit = email.Header.Header.__init__
+_oldheaderinit = email.header.Header.__init__
 def _unifiedheaderinit(self, *args, **kw):
     """
     Python 2.7 introduces a backwards incompatible change
@@ -203,24 +205,33 b' def validateconfig(ui):'
             raise error.Abort(_('%r specified as email transport, '
                                'but not in PATH') % method)
 
+def codec2iana(cs):
+    ''''''
+    cs = email.charset.Charset(cs).input_charset.lower()
+
+    # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1"
+    if cs.startswith("iso") and not cs.startswith("iso-"):
+        return "iso-" + cs[3:]
+    return cs
+
 def mimetextpatch(s, subtype='plain', display=False):
     '''Return MIME message suitable for a patch.
-    Charset will be detected as utf-8 or (possibly fake) us-ascii.
+    Charset will be detected by first trying to decode as us-ascii, then utf-8,
+    and finally the global encodings. If all those fail, fall back to
+    ISO-8859-1, an encoding with that allows all byte sequences.
     Transfer encodings will be used if necessary.'''
 
-    cs = 'us-ascii'
-    if not display:
+    cs = ['us-ascii', 'utf-8', encoding.encoding, encoding.fallbackencoding]
+    if display:
+        return mimetextqp(s, subtype, 'us-ascii')
+    for charset in cs:
         try:
-            s.decode('us-ascii')
+            s.decode(charset)
+            return mimetextqp(s, subtype, codec2iana(charset))
         except UnicodeDecodeError:
-            try:
-                s.decode('utf-8')
-                cs = 'utf-8'
-            except UnicodeDecodeError:
-                # We'll go with us-ascii as a fallback.
-                pass
+            pass
 
-    return mimetextqp(s, subtype, cs)
+    return mimetextqp(s, subtype, "iso-8859-1")
 
 def mimetextqp(body, subtype, charset):
     '''Return MIME message.
@@ -279,7 +290,7 b' def headencode(ui, s, charsets=None, dis'
     if not display:
         # split into words?
         s, cs = _encode(ui, s, charsets)
-        return str(email.Header.Header(s, cs))
+        return str(email.header.Header(s, cs))
     return s
 
 def _addressencode(ui, name, addr, charsets=None):
@@ -330,7 +341,7 b' def mimeencode(ui, s, charsets=None, dis'
 def headdecode(s):
     '''Decodes RFC-2047 header'''
     uparts = []
-    for part, charset in email.Header.decode_header(s):
+    for part, charset in email.header.decode_header(s):
         if charset is not None:
             try:
                 uparts.append(part.decode(charset))

mercurial/manifest.c

@@ -56,10 +56,10 b' static PyObject *nodeof(line *l) {'
 	}
 	if (l->hash_suffix != '\0') {
 		char newhash[21];
-		memcpy(newhash, PyString_AsString(hash), 20);
+		memcpy(newhash, PyBytes_AsString(hash), 20);
 		Py_DECREF(hash);
 		newhash[20] = l->hash_suffix;
-		hash = PyString_FromStringAndSize(newhash, 21);
+		hash = PyBytes_FromStringAndSize(newhash, 21);
 	}
 	return hash;
 }
@@ -79,7 +79,7 b' static PyObject *hashflags(line *l)'
 
 	if (!hash)
 		return NULL;
-	flags = PyString_FromStringAndSize(s + hplen - 1, flen);
+	flags = PyBytes_FromStringAndSize(s + hplen - 1, flen);
 	if (!flags) {
 		Py_DECREF(hash);
 		return NULL;
@@ -144,7 +144,7 b' static int lazymanifest_init(lazymanifes'
 	if (!PyArg_ParseTuple(args, "S", &pydata)) {
 		return -1;
 	}
-	err = PyString_AsStringAndSize(pydata, &data, &len);
+	err = PyBytes_AsStringAndSize(pydata, &data, &len);
 
 	self->dirty = false;
 	if (err == -1)
@@ -238,10 +238,10 b' static PyObject *lmiter_iterentriesnext('
 		goto done;
 	}
 	pl = pathlen(l);
-	path = PyString_FromStringAndSize(l->start, pl);
+	path = PyBytes_FromStringAndSize(l->start, pl);
 	hash = nodeof(l);
 	consumed = pl + 41;
-	flags = PyString_FromStringAndSize(l->start + consumed,
+	flags = PyBytes_FromStringAndSize(l->start + consumed,
 					   l->len - consumed - 1);
 	if (!path || !hash || !flags) {
 		goto done;
@@ -254,9 +254,15 b' done:'
 	return ret;
 }
 
+#ifdef IS_PY3K
+#define LAZYMANIFESTENTRIESITERATOR_TPFLAGS Py_TPFLAGS_DEFAULT
+#else
+#define LAZYMANIFESTENTRIESITERATOR_TPFLAGS Py_TPFLAGS_DEFAULT \
+	| Py_TPFLAGS_HAVE_ITER
+#endif
+
 static PyTypeObject lazymanifestEntriesIterator = {
-	PyObject_HEAD_INIT(NULL)
-	0,                               /*ob_size */
+	PyVarObject_HEAD_INIT(NULL, 0)
 	"parsers.lazymanifest.entriesiterator", /*tp_name */
 	sizeof(lmIter),                  /*tp_basicsize */
 	0,                               /*tp_itemsize */
@@ -275,9 +281,7 b' static PyTypeObject lazymanifestEntriesI'
 	0,                               /*tp_getattro */
 	0,                               /*tp_setattro */
 	0,                               /*tp_as_buffer */
-	/* tp_flags: Py_TPFLAGS_HAVE_ITER tells python to
-	   use tp_iter and tp_iternext fields. */
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_ITER,
+	LAZYMANIFESTENTRIESITERATOR_TPFLAGS, /* tp_flags */
 	"Iterator for 3-tuples in a lazymanifest.",  /* tp_doc */
 	0,                               /* tp_traverse */
 	0,                               /* tp_clear */
@@ -295,12 +299,18 b' static PyObject *lmiter_iterkeysnext(PyO'
 		return NULL;
 	}
 	pl = pathlen(l);
-	return PyString_FromStringAndSize(l->start, pl);
+	return PyBytes_FromStringAndSize(l->start, pl);
 }
 
+#ifdef IS_PY3K
+#define LAZYMANIFESTKEYSITERATOR_TPFLAGS Py_TPFLAGS_DEFAULT
+#else
+#define LAZYMANIFESTKEYSITERATOR_TPFLAGS Py_TPFLAGS_DEFAULT \
+	| Py_TPFLAGS_HAVE_ITER
+#endif
+
 static PyTypeObject lazymanifestKeysIterator = {
-	PyObject_HEAD_INIT(NULL)
-	0,                               /*ob_size */
+	PyVarObject_HEAD_INIT(NULL, 0)
 	"parsers.lazymanifest.keysiterator", /*tp_name */
 	sizeof(lmIter),                  /*tp_basicsize */
 	0,                               /*tp_itemsize */
@@ -319,9 +329,7 b' static PyTypeObject lazymanifestKeysIter'
 	0,                               /*tp_getattro */
 	0,                               /*tp_setattro */
 	0,                               /*tp_as_buffer */
-	/* tp_flags: Py_TPFLAGS_HAVE_ITER tells python to
-	   use tp_iter and tp_iternext fields. */
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_ITER,
+	LAZYMANIFESTKEYSITERATOR_TPFLAGS, /* tp_flags */
 	"Keys iterator for a lazymanifest.",  /* tp_doc */
 	0,                               /* tp_traverse */
 	0,                               /* tp_clear */
@@ -388,12 +396,12 b' static PyObject *lazymanifest_getitem(la'
 {
 	line needle;
 	line *hit;
-	if (!PyString_Check(key)) {
+	if (!PyBytes_Check(key)) {
 		PyErr_Format(PyExc_TypeError,
 			     "getitem: manifest keys must be a string.");
 		return NULL;
 	}
-	needle.start = PyString_AsString(key);
+	needle.start = PyBytes_AsString(key);
 	hit = bsearch(&needle, self->lines, self->numlines, sizeof(line),
 		      &linecmp);
 	if (!hit || hit->deleted) {
@@ -407,12 +415,12 b' static int lazymanifest_delitem(lazymani'
 {
 	line needle;
 	line *hit;
-	if (!PyString_Check(key)) {
+	if (!PyBytes_Check(key)) {
 		PyErr_Format(PyExc_TypeError,
 			     "delitem: manifest keys must be a string.");
 		return -1;
 	}
-	needle.start = PyString_AsString(key);
+	needle.start = PyBytes_AsString(key);
 	hit = bsearch(&needle, self->lines, self->numlines, sizeof(line),
 		      &linecmp);
 	if (!hit || hit->deleted) {
@@ -476,7 +484,7 b' static int lazymanifest_setitem('
 	char *dest;
 	int i;
 	line new;
-	if (!PyString_Check(key)) {
+	if (!PyBytes_Check(key)) {
 		PyErr_Format(PyExc_TypeError,
 			     "setitem: manifest keys must be a string.");
 		return -1;
@@ -489,17 +497,17 b' static int lazymanifest_setitem('
 			     "Manifest values must be a tuple of (node, flags).");
 		return -1;
 	}
-	if (PyString_AsStringAndSize(key, &path, &plen) == -1) {
+	if (PyBytes_AsStringAndSize(key, &path, &plen) == -1) {
 		return -1;
 	}
 
 	pyhash = PyTuple_GetItem(value, 0);
-	if (!PyString_Check(pyhash)) {
+	if (!PyBytes_Check(pyhash)) {
 		PyErr_Format(PyExc_TypeError,
 			     "node must be a 20-byte string");
 		return -1;
 	}
-	hlen = PyString_Size(pyhash);
+	hlen = PyBytes_Size(pyhash);
 	/* Some parts of the codebase try and set 21 or 22
 	 * byte "hash" values in order to perturb things for
 	 * status. We have to preserve at least the 21st
@@ -511,15 +519,15 b' static int lazymanifest_setitem('
 			     "node must be a 20-byte string");
 		return -1;
 	}
-	hash = PyString_AsString(pyhash);
+	hash = PyBytes_AsString(pyhash);
 
 	pyflags = PyTuple_GetItem(value, 1);
-	if (!PyString_Check(pyflags) || PyString_Size(pyflags) > 1) {
+	if (!PyBytes_Check(pyflags) || PyBytes_Size(pyflags) > 1) {
 		PyErr_Format(PyExc_TypeError,
 			     "flags must a 0 or 1 byte string");
 		return -1;
 	}
-	if (PyString_AsStringAndSize(pyflags, &flags, &flen) == -1) {
+	if (PyBytes_AsStringAndSize(pyflags, &flags, &flen) == -1) {
 		return -1;
 	}
 	/* one null byte and one newline */
@@ -564,12 +572,12 b' static int lazymanifest_contains(lazyman'
 {
 	line needle;
 	line *hit;
-	if (!PyString_Check(key)) {
+	if (!PyBytes_Check(key)) {
 		/* Our keys are always strings, so if the contains
 		 * check is for a non-string, just return false. */
 		return 0;
 	}
-	needle.start = PyString_AsString(key);
+	needle.start = PyBytes_AsString(key);
 	hit = bsearch(&needle, self->lines, self->numlines, sizeof(line),
 		      &linecmp);
 	if (!hit || hit->deleted) {
@@ -609,10 +617,10 b' static int compact(lazymanifest *self) {'
 			need += self->lines[i].len;
 		}
 	}
-	pydata = PyString_FromStringAndSize(NULL, need);
+	pydata = PyBytes_FromStringAndSize(NULL, need);
 	if (!pydata)
 		return -1;
-	data = PyString_AsString(pydata);
+	data = PyBytes_AsString(pydata);
 	if (!data) {
 		return -1;
 	}
@@ -747,7 +755,7 b' static PyObject *lazymanifest_diff(lazym'
 		return NULL;
 	}
 	listclean = (!pyclean) ? false : PyObject_IsTrue(pyclean);
-	es = PyString_FromString("");
+	es = PyBytes_FromString("");
 	if (!es) {
 		goto nomem;
 	}
@@ -787,8 +795,8 b' static PyObject *lazymanifest_diff(lazym'
 			result = linecmp(left, right);
 		}
 		key = result <= 0 ?
-			PyString_FromString(left->start) :
-			PyString_FromString(right->start);
+			PyBytes_FromString(left->start) :
+			PyBytes_FromString(right->start);
 		if (!key)
 			goto nomem;
 		if (result < 0) {
@@ -873,9 +881,14 b' static PyMethodDef lazymanifest_methods['
 	{NULL},
 };
 
+#ifdef IS_PY3K
+#define LAZYMANIFEST_TPFLAGS Py_TPFLAGS_DEFAULT
+#else
+#define LAZYMANIFEST_TPFLAGS Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_SEQUENCE_IN
+#endif
+
 static PyTypeObject lazymanifestType = {
-	PyObject_HEAD_INIT(NULL)
-	0,                                                /* ob_size */
+	PyVarObject_HEAD_INIT(NULL, 0)
 	"parsers.lazymanifest",                           /* tp_name */
 	sizeof(lazymanifest),                             /* tp_basicsize */
 	0,                                                /* tp_itemsize */
@@ -894,7 +907,7 b' static PyTypeObject lazymanifestType = {'
 	0,                                                /* tp_getattro */
 	0,                                                /* tp_setattro */
 	0,                                                /* tp_as_buffer */
-	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_SEQUENCE_IN, /* tp_flags */
+	LAZYMANIFEST_TPFLAGS,                             /* tp_flags */
 	"TODO(augie)",                                    /* tp_doc */
 	0,                                                /* tp_traverse */
 	0,                                                /* tp_clear */

mercurial/manifest.py

@@ -104,69 +104,300 b' def _textv2(it):'
     _checkforbidden(files)
     return ''.join(lines)
 
-class _lazymanifest(dict):
-    """This is the pure implementation of lazymanifest.
+class lazymanifestiter(object):
+    def __init__(self, lm):
+        self.pos = 0
+        self.lm = lm
 
-    It has not been optimized *at all* and is not lazy.
-    """
+    def __iter__(self):
+        return self
 
-    def __init__(self, data):
-        dict.__init__(self)
-        for f, n, fl in _parse(data):
-            self[f] = n, fl
+    def next(self):
+        try:
+            data, pos = self.lm._get(self.pos)
+        except IndexError:
+            raise StopIteration
+        if pos == -1:
+            self.pos += 1
+            return data[0]
+        self.pos += 1
+        zeropos = data.find('\x00', pos)
+        return data[pos:zeropos]
 
-    def __setitem__(self, k, v):
-        node, flag = v
-        assert node is not None
-        if len(node) > 21:
-            node = node[:21] # match c implementation behavior
-        dict.__setitem__(self, k, (node, flag))
+class lazymanifestiterentries(object):
+    def __init__(self, lm):
+        self.lm = lm
+        self.pos = 0
 
     def __iter__(self):
-        return iter(sorted(dict.keys(self)))
+        return self
+
+    def next(self):
+        try:
+            data, pos = self.lm._get(self.pos)
+        except IndexError:
+            raise StopIteration
+        if pos == -1:
+            self.pos += 1
+            return data
+        zeropos = data.find('\x00', pos)
+        hashval = unhexlify(data, self.lm.extrainfo[self.pos],
+                            zeropos + 1, 40)
+        flags = self.lm._getflags(data, self.pos, zeropos)
+        self.pos += 1
+        return (data[pos:zeropos], hashval, flags)
+
+def unhexlify(data, extra, pos, length):
+    s = data[pos:pos + length].decode('hex')
+    if extra:
+        s += chr(extra & 0xff)
+    return s
+
+def _cmp(a, b):
+    return (a > b) - (a < b)
+
+class _lazymanifest(object):
+    def __init__(self, data, positions=None, extrainfo=None, extradata=None):
+        if positions is None:
+            self.positions = self.findlines(data)
+            self.extrainfo = [0] * len(self.positions)
+            self.data = data
+            self.extradata = []
+        else:
+            self.positions = positions[:]
+            self.extrainfo = extrainfo[:]
+            self.extradata = extradata[:]
+            self.data = data
+
+    def findlines(self, data):
+        if not data:
+            return []
+        pos = data.find("\n")
+        if pos == -1 or data[-1] != '\n':
+            raise ValueError("Manifest did not end in a newline.")
+        positions = [0]
+        prev = data[:data.find('\x00')]
+        while pos < len(data) - 1 and pos != -1:
+            positions.append(pos + 1)
+            nexts = data[pos + 1:data.find('\x00', pos + 1)]
+            if nexts < prev:
+                raise ValueError("Manifest lines not in sorted order.")
+            prev = nexts
+            pos = data.find("\n", pos + 1)
+        return positions
+
+    def _get(self, index):
+        # get the position encoded in pos:
+        #   positive number is an index in 'data'
+        #   negative number is in extrapieces
+        pos = self.positions[index]
+        if pos >= 0:
+            return self.data, pos
+        return self.extradata[-pos - 1], -1
+
+    def _getkey(self, pos):
+        if pos >= 0:
+            return self.data[pos:self.data.find('\x00', pos + 1)]
+        return self.extradata[-pos - 1][0]
+
+    def bsearch(self, key):
+        first = 0
+        last = len(self.positions) - 1
+
+        while first <= last:
+            midpoint = (first + last)//2
+            nextpos = self.positions[midpoint]
+            candidate = self._getkey(nextpos)
+            r = _cmp(key, candidate)
+            if r == 0:
+                return midpoint
+            else:
+                if r < 0:
+                    last = midpoint - 1
+                else:
+                    first = midpoint + 1
+        return -1
 
-    def iterkeys(self):
-        return iter(sorted(dict.keys(self)))
+    def bsearch2(self, key):
+        # same as the above, but will always return the position
+        # done for performance reasons
+        first = 0
+        last = len(self.positions) - 1
+
+        while first <= last:
+            midpoint = (first + last)//2
+            nextpos = self.positions[midpoint]
+            candidate = self._getkey(nextpos)
+            r = _cmp(key, candidate)
+            if r == 0:
+                return (midpoint, True)
+            else:
+                if r < 0:
+                    last = midpoint - 1
+                else:
+                    first = midpoint + 1
+        return (first, False)
+
+    def __contains__(self, key):
+        return self.bsearch(key) != -1
+
+    def _getflags(self, data, needle, pos):
+        start = pos + 41
+        end = data.find("\n", start)
+        if end == -1:
+            end = len(data) - 1
+        if start == end:
+            return ''
+        return self.data[start:end]
 
-    def iterentries(self):
-        return ((f, e[0], e[1]) for f, e in sorted(self.iteritems()))
+    def __getitem__(self, key):
+        if not isinstance(key, str):
+            raise TypeError("getitem: manifest keys must be a string.")
+        needle = self.bsearch(key)
+        if needle == -1:
+            raise KeyError
+        data, pos = self._get(needle)
+        if pos == -1:
+            return (data[1], data[2])
+        zeropos = data.find('\x00', pos)
+        assert 0 <= needle <= len(self.positions)
+        assert len(self.extrainfo) == len(self.positions)
+        hashval = unhexlify(data, self.extrainfo[needle], zeropos + 1, 40)
+        flags = self._getflags(data, needle, zeropos)
+        return (hashval, flags)
+
+    def __delitem__(self, key):
+        needle, found = self.bsearch2(key)
+        if not found:
+            raise KeyError
+        cur = self.positions[needle]
+        self.positions = self.positions[:needle] + self.positions[needle + 1:]
+        self.extrainfo = self.extrainfo[:needle] + self.extrainfo[needle + 1:]
+        if cur >= 0:
+            self.data = self.data[:cur] + '\x00' + self.data[cur + 1:]
+
+    def __setitem__(self, key, value):
+        if not isinstance(key, str):
+            raise TypeError("setitem: manifest keys must be a string.")
+        if not isinstance(value, tuple) or len(value) != 2:
+            raise TypeError("Manifest values must be a tuple of (node, flags).")
+        hashval = value[0]
+        if not isinstance(hashval, str) or not 20 <= len(hashval) <= 22:
+            raise TypeError("node must be a 20-byte string")
+        flags = value[1]
+        if len(hashval) == 22:
+            hashval = hashval[:-1]
+        if not isinstance(flags, str) or len(flags) > 1:
+            raise TypeError("flags must a 0 or 1 byte string, got %r", flags)
+        needle, found = self.bsearch2(key)
+        if found:
+            # put the item
+            pos = self.positions[needle]
+            if pos < 0:
+                self.extradata[-pos - 1] = (key, hashval, value[1])
+            else:
+                # just don't bother
+                self.extradata.append((key, hashval, value[1]))
+                self.positions[needle] = -len(self.extradata)
+        else:
+            # not found, put it in with extra positions
+            self.extradata.append((key, hashval, value[1]))
+            self.positions = (self.positions[:needle] + [-len(self.extradata)]
+                              + self.positions[needle:])
+            self.extrainfo = (self.extrainfo[:needle] + [0] +
+                              self.extrainfo[needle:])
 
     def copy(self):
-        c = _lazymanifest('')
-        c.update(self)
-        return c
+        # XXX call _compact like in C?
+        return _lazymanifest(self.data, self.positions, self.extrainfo,
+            self.extradata)
+
+    def _compact(self):
+        # hopefully not called TOO often
+        if len(self.extradata) == 0:
+            return
+        l = []
+        last_cut = 0
+        i = 0
+        offset = 0
+        self.extrainfo = [0] * len(self.positions)
+        while i < len(self.positions):
+            if self.positions[i] >= 0:
+                cur = self.positions[i]
+                last_cut = cur
+                while True:
+                    self.positions[i] = offset
+                    i += 1
+                    if i == len(self.positions) or self.positions[i] < 0:
+                        break
+                    offset += self.positions[i] - cur
+                    cur = self.positions[i]
+                end_cut = self.data.find('\n', cur)
+                if end_cut != -1:
+                    end_cut += 1
+                offset += end_cut - cur
+                l.append(self.data[last_cut:end_cut])
+            else:
+                while i < len(self.positions) and self.positions[i] < 0:
+                    cur = self.positions[i]
+                    t = self.extradata[-cur - 1]
+                    l.append(self._pack(t))
+                    self.positions[i] = offset
+                    if len(t[1]) > 20:
+                        self.extrainfo[i] = ord(t[1][21])
+                    offset += len(l[-1])
+                    i += 1
+        self.data = ''.join(l)
+        self.extradata = []
+
+    def _pack(self, d):
+        return d[0] + '\x00' + d[1][:20].encode('hex') + d[2] + '\n'
+
+    def text(self):
+        self._compact()
+        return self.data
 
     def diff(self, m2, clean=False):
         '''Finds changes between the current manifest and m2.'''
+        # XXX think whether efficiency matters here
         diff = {}
 
-        for fn, e1 in self.iteritems():
+        for fn, e1, flags in self.iterentries():
             if fn not in m2:
-                diff[fn] = e1, (None, '')
+                diff[fn] = (e1, flags), (None, '')
             else:
                 e2 = m2[fn]
-                if e1 != e2:
-                    diff[fn] = e1, e2
+                if (e1, flags) != e2:
+                    diff[fn] = (e1, flags), e2
                 elif clean:
                     diff[fn] = None
 
-        for fn, e2 in m2.iteritems():
+        for fn, e2, flags in m2.iterentries():
             if fn not in self:
-                diff[fn] = (None, ''), e2
+                diff[fn] = (None, ''), (e2, flags)
 
         return diff
 
+    def iterentries(self):
+        return lazymanifestiterentries(self)
+
+    def iterkeys(self):
+        return lazymanifestiter(self)
+
+    def __iter__(self):
+        return lazymanifestiter(self)
+
+    def __len__(self):
+        return len(self.positions)
+
     def filtercopy(self, filterfn):
+        # XXX should be optimized
         c = _lazymanifest('')
         for f, n, fl in self.iterentries():
             if filterfn(f):
                 c[f] = n, fl
         return c
 
-    def text(self):
-        """Get the full data of this manifest as a bytestring."""
-        return _textv1(self.iterentries())
-
 try:
     _lazymanifest = parsers.lazymanifest
 except AttributeError:
@@ -882,6 +1113,8 b' class treemanifest(object):'
 
     def writesubtrees(self, m1, m2, writesubtree):
         self._load() # for consistency; should never have any effect here
+        m1._load()
+        m2._load()
         emptytree = treemanifest()
         for d, subm in self._dirs.iteritems():
             subp1 = m1._dirs.get(d, emptytree)._node
@@ -890,12 +1123,11 b' class treemanifest(object):'
                 subp1, subp2 = subp2, subp1
             writesubtree(subm, subp1, subp2)
 
-class manifest(revlog.revlog):
+class manifestrevlog(revlog.revlog):
+    '''A revlog that stores manifest texts. This is responsible for caching the
+    full-text manifest contents.
+    '''
     def __init__(self, opener, dir='', dirlogcache=None):
-        '''The 'dir' and 'dirlogcache' arguments are for internal use by
-        manifest.manifest only. External users should create a root manifest
-        log with manifest.manifest(opener) and call dirlog() on it.
-        '''
         # During normal operations, we expect to deal with not more than four
         # revs at a time (such as during commit --amend). When rebasing large
         # stacks of commits, the number can go up, hence the config knob below.
@@ -907,17 +1139,18 b' class manifest(revlog.revlog):'
             cachesize = opts.get('manifestcachesize', cachesize)
             usetreemanifest = opts.get('treemanifest', usetreemanifest)
             usemanifestv2 = opts.get('manifestv2', usemanifestv2)
-        self._mancache = util.lrucachedict(cachesize)
-        self._treeinmem = usetreemanifest
+
         self._treeondisk = usetreemanifest
         self._usemanifestv2 = usemanifestv2
+
+        self._fulltextcache = util.lrucachedict(cachesize)
+
         indexfile = "00manifest.i"
         if dir:
-            assert self._treeondisk
+            assert self._treeondisk, 'opts is %r' % opts
             if not dir.endswith('/'):
                 dir = dir + '/'
             indexfile = "meta/" + dir + "00manifest.i"
-        revlog.revlog.__init__(self, opener, indexfile)
         self._dir = dir
         # The dirlogcache is kept on the root manifest log
         if dir:
@@ -925,12 +1158,281 b' class manifest(revlog.revlog):'
         else:
             self._dirlogcache = {'': self}
 
+        super(manifestrevlog, self).__init__(opener, indexfile,
+                                             checkambig=bool(dir))
+
+    @property
+    def fulltextcache(self):
+        return self._fulltextcache
+
+    def clearcaches(self):
+        super(manifestrevlog, self).clearcaches()
+        self._fulltextcache.clear()
+        self._dirlogcache = {'': self}
+
+    def dirlog(self, dir):
+        if dir:
+            assert self._treeondisk
+        if dir not in self._dirlogcache:
+            self._dirlogcache[dir] = manifestrevlog(self.opener, dir,
+                                                    self._dirlogcache)
+        return self._dirlogcache[dir]
+
+    def add(self, m, transaction, link, p1, p2, added, removed):
+        if (p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta')
+            and not self._usemanifestv2):
+            # If our first parent is in the manifest cache, we can
+            # compute a delta here using properties we know about the
+            # manifest up-front, which may save time later for the
+            # revlog layer.
+
+            _checkforbidden(added)
+            # combine the changed lists into one sorted iterator
+            work = heapq.merge([(x, False) for x in added],
+                               [(x, True) for x in removed])
+
+            arraytext, deltatext = m.fastdelta(self.fulltextcache[p1], work)
+            cachedelta = self.rev(p1), deltatext
+            text = util.buffer(arraytext)
+            n = self.addrevision(text, transaction, link, p1, p2, cachedelta)
+        else:
+            # The first parent manifest isn't already loaded, so we'll
+            # just encode a fulltext of the manifest and pass that
+            # through to the revlog layer, and let it handle the delta
+            # process.
+            if self._treeondisk:
+                m1 = self.read(p1)
+                m2 = self.read(p2)
+                n = self._addtree(m, transaction, link, m1, m2)
+                arraytext = None
+            else:
+                text = m.text(self._usemanifestv2)
+                n = self.addrevision(text, transaction, link, p1, p2)
+                arraytext = array.array('c', text)
+
+        if arraytext is not None:
+            self.fulltextcache[n] = arraytext
+
+        return n
+
+    def _addtree(self, m, transaction, link, m1, m2):
+        # If the manifest is unchanged compared to one parent,
+        # don't write a new revision
+        if m.unmodifiedsince(m1) or m.unmodifiedsince(m2):
+            return m.node()
+        def writesubtree(subm, subp1, subp2):
+            sublog = self.dirlog(subm.dir())
+            sublog.add(subm, transaction, link, subp1, subp2, None, None)
+        m.writesubtrees(m1, m2, writesubtree)
+        text = m.dirtext(self._usemanifestv2)
+        # Double-check whether contents are unchanged to one parent
+        if text == m1.dirtext(self._usemanifestv2):
+            n = m1.node()
+        elif text == m2.dirtext(self._usemanifestv2):
+            n = m2.node()
+        else:
+            n = self.addrevision(text, transaction, link, m1.node(), m2.node())
+        # Save nodeid so parent manifest can calculate its nodeid
+        m.setnode(n)
+        return n
+
+class manifestlog(object):
+    """A collection class representing the collection of manifest snapshots
+    referenced by commits in the repository.
+
+    In this situation, 'manifest' refers to the abstract concept of a snapshot
+    of the list of files in the given commit. Consumers of the output of this
+    class do not care about the implementation details of the actual manifests
+    they receive (i.e. tree or flat or lazily loaded, etc)."""
+    def __init__(self, opener, repo):
+        self._repo = repo
+
+        usetreemanifest = False
+
+        opts = getattr(opener, 'options', None)
+        if opts is not None:
+            usetreemanifest = opts.get('treemanifest', usetreemanifest)
+        self._treeinmem = usetreemanifest
+
+        # We'll separate this into it's own cache once oldmanifest is no longer
+        # used
+        self._mancache = repo.manifest._mancache
+
+    @property
+    def _revlog(self):
+        return self._repo.manifest
+
+    def __getitem__(self, node):
+        """Retrieves the manifest instance for the given node. Throws a KeyError
+        if not found.
+        """
+        if node in self._mancache:
+            cachemf = self._mancache[node]
+            # The old manifest may put non-ctx manifests in the cache, so skip
+            # those since they don't implement the full api.
+            if (isinstance(cachemf, manifestctx) or
+                isinstance(cachemf, treemanifestctx)):
+                return cachemf
+
+        if self._treeinmem:
+            m = treemanifestctx(self._revlog, '', node)
+        else:
+            m = manifestctx(self._revlog, node)
+        if node != revlog.nullid:
+            self._mancache[node] = m
+        return m
+
+    def add(self, m, transaction, link, p1, p2, added, removed):
+        return self._revlog.add(m, transaction, link, p1, p2, added, removed)
+
+class manifestctx(object):
+    """A class representing a single revision of a manifest, including its
+    contents, its parent revs, and its linkrev.
+    """
+    def __init__(self, revlog, node):
+        self._revlog = revlog
+        self._data = None
+
+        self._node = node
+
+        # TODO: We eventually want p1, p2, and linkrev exposed on this class,
+        # but let's add it later when something needs it and we can load it
+        # lazily.
+        #self.p1, self.p2 = revlog.parents(node)
+        #rev = revlog.rev(node)
+        #self.linkrev = revlog.linkrev(rev)
+
+    def node(self):
+        return self._node
+
+    def read(self):
+        if not self._data:
+            if self._node == revlog.nullid:
+                self._data = manifestdict()
+            else:
+                text = self._revlog.revision(self._node)
+                arraytext = array.array('c', text)
+                self._revlog._fulltextcache[self._node] = arraytext
+                self._data = manifestdict(text)
+        return self._data
+
+    def readfast(self):
+        rl = self._revlog
+        r = rl.rev(self._node)
+        deltaparent = rl.deltaparent(r)
+        if deltaparent != revlog.nullrev and deltaparent in rl.parentrevs(r):
+            return self.readdelta()
+        return self.read()
+
+    def readdelta(self):
+        revlog = self._revlog
+        if revlog._usemanifestv2:
+            # Need to perform a slow delta
+            r0 = revlog.deltaparent(revlog.rev(self._node))
+            m0 = manifestctx(revlog, revlog.node(r0)).read()
+            m1 = self.read()
+            md = manifestdict()
+            for f, ((n0, fl0), (n1, fl1)) in m0.diff(m1).iteritems():
+                if n1:
+                    md[f] = n1
+                    if fl1:
+                        md.setflag(f, fl1)
+            return md
+
+        r = revlog.rev(self._node)
+        d = mdiff.patchtext(revlog.revdiff(revlog.deltaparent(r), r))
+        return manifestdict(d)
+
+class treemanifestctx(object):
+    def __init__(self, revlog, dir, node):
+        revlog = revlog.dirlog(dir)
+        self._revlog = revlog
+        self._dir = dir
+        self._data = None
+
+        self._node = node
+
+        # TODO: Load p1/p2/linkrev lazily. They need to be lazily loaded so that
+        # we can instantiate treemanifestctx objects for directories we don't
+        # have on disk.
+        #self.p1, self.p2 = revlog.parents(node)
+        #rev = revlog.rev(node)
+        #self.linkrev = revlog.linkrev(rev)
+
+    def read(self):
+        if not self._data:
+            if self._node == revlog.nullid:
+                self._data = treemanifest()
+            elif self._revlog._treeondisk:
+                m = treemanifest(dir=self._dir)
+                def gettext():
+                    return self._revlog.revision(self._node)
+                def readsubtree(dir, subm):
+                    return treemanifestctx(self._revlog, dir, subm).read()
+                m.read(gettext, readsubtree)
+                m.setnode(self._node)
+                self._data = m
+            else:
+                text = self._revlog.revision(self._node)
+                arraytext = array.array('c', text)
+                self._revlog.fulltextcache[self._node] = arraytext
+                self._data = treemanifest(dir=self._dir, text=text)
+
+        return self._data
+
+    def node(self):
+        return self._node
+
+    def readdelta(self):
+        # Need to perform a slow delta
+        revlog = self._revlog
+        r0 = revlog.deltaparent(revlog.rev(self._node))
+        m0 = treemanifestctx(revlog, self._dir, revlog.node(r0)).read()
+        m1 = self.read()
+        md = treemanifest(dir=self._dir)
+        for f, ((n0, fl0), (n1, fl1)) in m0.diff(m1).iteritems():
+            if n1:
+                md[f] = n1
+                if fl1:
+                    md.setflag(f, fl1)
+        return md
+
+    def readfast(self):
+        rl = self._revlog
+        r = rl.rev(self._node)
+        deltaparent = rl.deltaparent(r)
+        if deltaparent != revlog.nullrev and deltaparent in rl.parentrevs(r):
+            return self.readdelta()
+        return self.read()
+
+class manifest(manifestrevlog):
+    def __init__(self, opener, dir='', dirlogcache=None):
+        '''The 'dir' and 'dirlogcache' arguments are for internal use by
+        manifest.manifest only. External users should create a root manifest
+        log with manifest.manifest(opener) and call dirlog() on it.
+        '''
+        # During normal operations, we expect to deal with not more than four
+        # revs at a time (such as during commit --amend). When rebasing large
+        # stacks of commits, the number can go up, hence the config knob below.
+        cachesize = 4
+        usetreemanifest = False
+        opts = getattr(opener, 'options', None)
+        if opts is not None:
+            cachesize = opts.get('manifestcachesize', cachesize)
+            usetreemanifest = opts.get('treemanifest', usetreemanifest)
+        self._mancache = util.lrucachedict(cachesize)
+        self._treeinmem = usetreemanifest
+        super(manifest, self).__init__(opener, dir=dir, dirlogcache=dirlogcache)
+
     def _newmanifest(self, data=''):
         if self._treeinmem:
             return treemanifest(self._dir, data)
         return manifestdict(data)
 
     def dirlog(self, dir):
+        """This overrides the base revlog implementation to allow construction
+        'manifest' types instead of manifestrevlog types. This is only needed
+        until we migrate off the 'manifest' type."""
         if dir:
             assert self._treeondisk
         if dir not in self._dirlogcache:
@@ -973,20 +1475,6 b' class manifest(revlog.revlog):'
         d = mdiff.patchtext(self.revdiff(self.deltaparent(r), r))
         return manifestdict(d)
 
-    def readfast(self, node):
-        '''use the faster of readdelta or read
-
-        This will return a manifest which is either only the files
-        added/modified relative to p1, or all files in the
-        manifest. Which one is returned depends on the codepath used
-        to retrieve the data.
-        '''
-        r = self.rev(node)
-        deltaparent = self.deltaparent(r)
-        if deltaparent != revlog.nullrev and deltaparent in self.parentrevs(r):
-            return self.readdelta(node)
-        return self.read(node)
-
     def readshallowfast(self, node):
         '''like readfast(), but calls readshallowdelta() instead of readdelta()
         '''
@@ -1000,7 +1488,11 b' class manifest(revlog.revlog):'
         if node == revlog.nullid:
             return self._newmanifest() # don't upset local cache
         if node in self._mancache:
-            return self._mancache[node][0]
+            cached = self._mancache[node]
+            if (isinstance(cached, manifestctx) or
+                isinstance(cached, treemanifestctx)):
+                cached = cached.read()
+            return cached
         if self._treeondisk:
             def gettext():
                 return self.revision(node)
@@ -1014,7 +1506,9 b' class manifest(revlog.revlog):'
             text = self.revision(node)
             m = self._newmanifest(text)
             arraytext = array.array('c', text)
-        self._mancache[node] = (m, arraytext)
+        self._mancache[node] = m
+        if arraytext is not None:
+            self.fulltextcache[node] = arraytext
         return m
 
     def readshallow(self, node):
@@ -1033,64 +1527,6 b' class manifest(revlog.revlog):'
         except KeyError:
             return None, None
 
-    def add(self, m, transaction, link, p1, p2, added, removed):
-        if (p1 in self._mancache and not self._treeinmem
-            and not self._usemanifestv2):
-            # If our first parent is in the manifest cache, we can
-            # compute a delta here using properties we know about the
-            # manifest up-front, which may save time later for the
-            # revlog layer.
-
-            _checkforbidden(added)
-            # combine the changed lists into one sorted iterator
-            work = heapq.merge([(x, False) for x in added],
-                               [(x, True) for x in removed])
-
-            arraytext, deltatext = m.fastdelta(self._mancache[p1][1], work)
-            cachedelta = self.rev(p1), deltatext
-            text = util.buffer(arraytext)
-            n = self.addrevision(text, transaction, link, p1, p2, cachedelta)
-        else:
-            # The first parent manifest isn't already loaded, so we'll
-            # just encode a fulltext of the manifest and pass that
-            # through to the revlog layer, and let it handle the delta
-            # process.
-            if self._treeondisk:
-                m1 = self.read(p1)
-                m2 = self.read(p2)
-                n = self._addtree(m, transaction, link, m1, m2)
-                arraytext = None
-            else:
-                text = m.text(self._usemanifestv2)
-                n = self.addrevision(text, transaction, link, p1, p2)
-                arraytext = array.array('c', text)
-
-        self._mancache[n] = (m, arraytext)
-
-        return n
-
-    def _addtree(self, m, transaction, link, m1, m2):
-        # If the manifest is unchanged compared to one parent,
-        # don't write a new revision
-        if m.unmodifiedsince(m1) or m.unmodifiedsince(m2):
-            return m.node()
-        def writesubtree(subm, subp1, subp2):
-            sublog = self.dirlog(subm.dir())
-            sublog.add(subm, transaction, link, subp1, subp2, None, None)
-        m.writesubtrees(m1, m2, writesubtree)
-        text = m.dirtext(self._usemanifestv2)
-        # Double-check whether contents are unchanged to one parent
-        if text == m1.dirtext(self._usemanifestv2):
-            n = m1.node()
-        elif text == m2.dirtext(self._usemanifestv2):
-            n = m2.node()
-        else:
-            n = self.addrevision(text, transaction, link, m1.node(), m2.node())
-        # Save nodeid so parent manifest can calculate its nodeid
-        m.setnode(n)
-        return n
-
     def clearcaches(self):
         super(manifest, self).clearcaches()
         self._mancache.clear()
-        self._dirlogcache = {'': self}

mercurial/mdiff.py

@@ -113,12 +113,11 b' def splitblock(base1, lines1, base2, lin'
         s1 = i1
         s2 = i2
 
-def allblocks(text1, text2, opts=None, lines1=None, lines2=None, refine=False):
+def allblocks(text1, text2, opts=None, lines1=None, lines2=None):
     """Return (block, type) tuples, where block is an mdiff.blocks
     line entry. type is '=' for blocks matching exactly one another
     (bdiff blocks), '!' for non-matching blocks and '~' for blocks
-    matching only after having filtered blank lines. If refine is True,
-    then '~' blocks are refined and are only made of blank lines.
+    matching only after having filtered blank lines.
     line1 and line2 are text1 and text2 split with splitnewlines() if
     they are already available.
     """

mercurial/merge.py

@@ -475,10 +475,12 b' class mergestate(object):'
         flo = fco.flags()
         fla = fca.flags()
         if 'x' in flags + flo + fla and 'l' not in flags + flo + fla:
-            if fca.node() == nullid:
+            if fca.node() == nullid and flags != flo:
                 if preresolve:
                     self._repo.ui.warn(
-                        _('warning: cannot merge flags for %s\n') % afile)
+                        _('warning: cannot merge flags for %s '
+                          'without common ancestor - keeping local flags\n')
+                        % afile)
             elif flags == fla:
                 flags = flo
         if preresolve:
@@ -781,7 +783,7 b' def driverconclude(repo, ms, wctx, label'
 def manifestmerge(repo, wctx, p2, pa, branchmerge, force, matcher,
                   acceptremote, followcopies):
     """
-    Merge p1 and p2 with ancestor pa and generate merge action list
+    Merge wctx and p2 with ancestor pa and generate merge action list
 
     branchmerge and force are as passed in to update
     matcher = matcher to filter file lists
@@ -1036,6 +1038,12 b' def batchremove(repo, actions):'
     unlink = util.unlinkpath
     wjoin = repo.wjoin
     audit = repo.wvfs.audit
+    try:
+        cwd = os.getcwd()
+    except OSError as err:
+        if err.errno != errno.ENOENT:
+            raise
+        cwd = None
     i = 0
     for f, args, msg in actions:
         repo.ui.debug(" %s: %s -> r\n" % (f, msg))
@@ -1053,6 +1061,18 b' def batchremove(repo, actions):'
         i += 1
     if i > 0:
         yield i, f
+    if cwd:
+        # cwd was present before we started to remove files
+        # let's check if it is present after we removed them
+        try:
+            os.getcwd()
+        except OSError as err:
+            if err.errno != errno.ENOENT:
+                raise
+            # Print a warning if cwd was deleted
+            repo.ui.warn(_("current directory was removed\n"
+                           "(consider changing to repo root: %s)\n") %
+                         repo.root)
 
 def batchget(repo, mctx, actions):
     """apply gets to the working directory
@@ -1150,7 +1170,7 b' def applyupdates(repo, actions, wctx, mc'
     numupdates = sum(len(l) for m, l in actions.items() if m != 'k')
 
     if [a for a in actions['r'] if a[0] == '.hgsubstate']:
-        subrepo.submerge(repo, wctx, mctx, wctx, overwrite)
+        subrepo.submerge(repo, wctx, mctx, wctx, overwrite, labels)
 
     # remove in parallel (must come first)
     z = 0
@@ -1168,7 +1188,7 b' def applyupdates(repo, actions, wctx, mc'
     updated = len(actions['g'])
 
     if [a for a in actions['g'] if a[0] == '.hgsubstate']:
-        subrepo.submerge(repo, wctx, mctx, wctx, overwrite)
+        subrepo.submerge(repo, wctx, mctx, wctx, overwrite, labels)
 
     # forget (manifest only, just log it) (must come first)
     for f, args, msg in actions['f']:
@@ -1253,7 +1273,7 b' def applyupdates(repo, actions, wctx, mc'
         progress(_updating, z, item=f, total=numupdates, unit=_files)
         if f == '.hgsubstate': # subrepo states need updating
             subrepo.submerge(repo, wctx, mctx, wctx.ancestor(mctx),
-                             overwrite)
+                             overwrite, labels)
             continue
         audit(f)
         complete, r = ms.preresolve(f, wctx)
@@ -1286,8 +1306,29 b' def applyupdates(repo, actions, wctx, mc'
     removed += msremoved
 
     extraactions = ms.actions()
-    for k, acts in extraactions.iteritems():
-        actions[k].extend(acts)
+    if extraactions:
+        mfiles = set(a[0] for a in actions['m'])
+        for k, acts in extraactions.iteritems():
+            actions[k].extend(acts)
+            # Remove these files from actions['m'] as well. This is important
+            # because in recordupdates, files in actions['m'] are processed
+            # after files in other actions, and the merge driver might add
+            # files to those actions via extraactions above. This can lead to a
+            # file being recorded twice, with poor results. This is especially
+            # problematic for actions['r'] (currently only possible with the
+            # merge driver in the initial merge process; interrupted merges
+            # don't go through this flow).
+            #
+            # The real fix here is to have indexes by both file and action so
+            # that when the action for a file is changed it is automatically
+            # reflected in the other action lists. But that involves a more
+            # complex data structure, so this will do for now.
+            #
+            # We don't need to do the same operation for 'dc' and 'cd' because
+            # those lists aren't consulted again.
+            mfiles.difference_update(a[0] for a in acts)
+
+        actions['m'] = [a for a in actions['m'] if a[0] in mfiles]
 
     progress(_updating, None, total=numupdates, unit=_files)
 
@@ -1514,15 +1555,16 b' def update(repo, node, branchmerge, forc'
                     pas = [p1]
 
         # deprecated config: merge.followcopies
-        followcopies = False
+        followcopies = repo.ui.configbool('merge', 'followcopies', True)
         if overwrite:
             pas = [wc]
+            followcopies = False
         elif pas == [p2]: # backwards
-            pas = [wc.p1()]
-        elif not branchmerge and not wc.dirty(missing=True):
-            pass
-        elif pas[0] and repo.ui.configbool('merge', 'followcopies', True):
-            followcopies = True
+            pas = [p1]
+        elif not pas[0]:
+            followcopies = False
+        if not branchmerge and not wc.dirty(missing=True):
+            followcopies = False
 
         ### calculate phase
         actionbyfile, diverge, renamedelete = calculateupdates(
@@ -1535,11 +1577,13 b' def update(repo, node, branchmerge, forc'
         if '.hgsubstate' in actionbyfile:
             f = '.hgsubstate'
             m, args, msg = actionbyfile[f]
+            prompts = filemerge.partextras(labels)
+            prompts['f'] = f
             if m == 'cd':
                 if repo.ui.promptchoice(
-                    _("local changed %s which remote deleted\n"
+                    _("local%(l)s changed %(f)s which other%(o)s deleted\n"
                       "use (c)hanged version or (d)elete?"
-                      "$$ &Changed $$ &Delete") % f, 0):
+                      "$$ &Changed $$ &Delete") % prompts, 0):
                     actionbyfile[f] = ('r', None, "prompt delete")
                 elif f in p1:
                     actionbyfile[f] = ('am', None, "prompt keep")
@@ -1549,9 +1593,9 b' def update(repo, node, branchmerge, forc'
                 f1, f2, fa, move, anc = args
                 flags = p2[f2].flags()
                 if repo.ui.promptchoice(
-                    _("remote changed %s which local deleted\n"
+                    _("other%(o)s changed %(f)s which local%(l)s deleted\n"
                       "use (c)hanged version or leave (d)eleted?"
-                      "$$ &Changed $$ &Deleted") % f, 0) == 0:
+                      "$$ &Changed $$ &Deleted") % prompts, 0) == 0:
                     actionbyfile[f] = ('g', (flags, False), "prompt recreating")
                 else:
                     del actionbyfile[f]
@@ -1563,7 +1607,7 b' def update(repo, node, branchmerge, forc'
                 actions[m] = []
             actions[m].append((f, args, msg))
 
-        if not util.checkcase(repo.path):
+        if not util.fscasesensitive(repo.path):
             # check collision between files only in p2 for clean update
             if (not branchmerge and
                 (force or not wc.dirty(missing=True, branch=False))):

mercurial/mpatch.c

@@ -20,49 +20,33 b''
  of the GNU General Public License, incorporated herein by reference.
 */
 
-#define PY_SSIZE_T_CLEAN
-#include <Python.h>
 #include <stdlib.h>
 #include <string.h>
 
-#include "util.h"
 #include "bitmanipulation.h"
-
-static char mpatch_doc[] = "Efficient binary patching.";
-static PyObject *mpatch_Error;
+#include "compat.h"
+#include "mpatch.h"
 
-struct frag {
-	int start, end, len;
-	const char *data;
-};
-
-struct flist {
-	struct frag *base, *head, *tail;
-};
-
-static struct flist *lalloc(Py_ssize_t size)
+static struct mpatch_flist *lalloc(ssize_t size)
 {
-	struct flist *a = NULL;
+	struct mpatch_flist *a = NULL;
 
 	if (size < 1)
 		size = 1;
 
-	a = (struct flist *)malloc(sizeof(struct flist));
+	a = (struct mpatch_flist *)malloc(sizeof(struct mpatch_flist));
 	if (a) {
-		a->base = (struct frag *)malloc(sizeof(struct frag) * size);
+		a->base = (struct mpatch_frag *)malloc(sizeof(struct mpatch_frag) * size);
 		if (a->base) {
 			a->head = a->tail = a->base;
 			return a;
 		}
 		free(a);
-		a = NULL;
 	}
-	if (!PyErr_Occurred())
-		PyErr_NoMemory();
 	return NULL;
 }
 
-static void lfree(struct flist *a)
+void mpatch_lfree(struct mpatch_flist *a)
 {
 	if (a) {
 		free(a->base);
@@ -70,7 +54,7 b' static void lfree(struct flist *a)'
 	}
 }
 
-static Py_ssize_t lsize(struct flist *a)
+static ssize_t lsize(struct mpatch_flist *a)
 {
 	return a->tail - a->head;
 }
@@ -78,9 +62,10 b' static Py_ssize_t lsize(struct flist *a)'
 /* move hunks in source that are less cut to dest, compensating
    for changes in offset. the last hunk may be split if necessary.
 */
-static int gather(struct flist *dest, struct flist *src, int cut, int offset)
+static int gather(struct mpatch_flist *dest, struct mpatch_flist *src, int cut,
+	int offset)
 {
-	struct frag *d = dest->tail, *s = src->head;
+	struct mpatch_frag *d = dest->tail, *s = src->head;
 	int postend, c, l;
 
 	while (s != src->tail) {
@@ -123,9 +108,9 b' static int gather(struct flist *dest, st'
 }
 
 /* like gather, but with no output list */
-static int discard(struct flist *src, int cut, int offset)
+static int discard(struct mpatch_flist *src, int cut, int offset)
 {
-	struct frag *s = src->head;
+	struct mpatch_frag *s = src->head;
 	int postend, c, l;
 
 	while (s != src->tail) {
@@ -160,10 +145,11 b' static int discard(struct flist *src, in'
 
 /* combine hunk lists a and b, while adjusting b for offset changes in a/
    this deletes a and b and returns the resultant list. */
-static struct flist *combine(struct flist *a, struct flist *b)
+static struct mpatch_flist *combine(struct mpatch_flist *a,
+	struct mpatch_flist *b)
 {
-	struct flist *c = NULL;
-	struct frag *bh, *ct;
+	struct mpatch_flist *c = NULL;
+	struct mpatch_frag *bh, *ct;
 	int offset = 0, post;
 
 	if (a && b)
@@ -189,26 +175,26 b' static struct flist *combine(struct flis'
 		}
 
 		/* hold on to tail from a */
-		memcpy(c->tail, a->head, sizeof(struct frag) * lsize(a));
+		memcpy(c->tail, a->head, sizeof(struct mpatch_frag) * lsize(a));
 		c->tail += lsize(a);
 	}
 
-	lfree(a);
-	lfree(b);
+	mpatch_lfree(a);
+	mpatch_lfree(b);
 	return c;
 }
 
 /* decode a binary patch into a hunk list */
-static struct flist *decode(const char *bin, Py_ssize_t len)
+int mpatch_decode(const char *bin, ssize_t len, struct mpatch_flist **res)
 {
-	struct flist *l;
-	struct frag *lt;
+	struct mpatch_flist *l;
+	struct mpatch_frag *lt;
 	int pos = 0;
 
 	/* assume worst case size, we won't have many of these lists */
 	l = lalloc(len / 12 + 1);
 	if (!l)
-		return NULL;
+		return MPATCH_ERR_NO_MEM;
 
 	lt = l->tail;
 
@@ -224,28 +210,24 b' static struct flist *decode(const char *'
 	}
 
 	if (pos != len) {
-		if (!PyErr_Occurred())
-			PyErr_SetString(mpatch_Error, "patch cannot be decoded");
-		lfree(l);
-		return NULL;
+		mpatch_lfree(l);
+		return MPATCH_ERR_CANNOT_BE_DECODED;
 	}
 
 	l->tail = lt;
-	return l;
+	*res = l;
+	return 0;
 }
 
 /* calculate the size of resultant text */
-static Py_ssize_t calcsize(Py_ssize_t len, struct flist *l)
+ssize_t mpatch_calcsize(ssize_t len, struct mpatch_flist *l)
 {
-	Py_ssize_t outlen = 0, last = 0;
-	struct frag *f = l->head;
+	ssize_t outlen = 0, last = 0;
+	struct mpatch_frag *f = l->head;
 
 	while (f != l->tail) {
 		if (f->start < last || f->end > len) {
-			if (!PyErr_Occurred())
-				PyErr_SetString(mpatch_Error,
-				                "invalid patch");
-			return -1;
+			return MPATCH_ERR_INVALID_PATCH;
 		}
 		outlen += f->start - last;
 		last = f->end;
@@ -257,18 +239,16 b' static Py_ssize_t calcsize(Py_ssize_t le'
 	return outlen;
 }
 
-static int apply(char *buf, const char *orig, Py_ssize_t len, struct flist *l)
+int mpatch_apply(char *buf, const char *orig, ssize_t len,
+	struct mpatch_flist *l)
 {
-	struct frag *f = l->head;
+	struct mpatch_frag *f = l->head;
 	int last = 0;
 	char *p = buf;
 
 	while (f != l->tail) {
 		if (f->start < last || f->end > len) {
-			if (!PyErr_Occurred())
-				PyErr_SetString(mpatch_Error,
-				                "invalid patch");
-			return 0;
+			return MPATCH_ERR_INVALID_PATCH;
 		}
 		memcpy(p, orig + last, f->start - last);
 		p += f->start - last;
@@ -278,146 +258,23 b' static int apply(char *buf, const char *'
 		f++;
 	}
 	memcpy(p, orig + last, len - last);
-	return 1;
+	return 0;
 }
 
 /* recursively generate a patch of all bins between start and end */
-static struct flist *fold(PyObject *bins, Py_ssize_t start, Py_ssize_t end)
+struct mpatch_flist *mpatch_fold(void *bins,
+	struct mpatch_flist* (*get_next_item)(void*, ssize_t),
+	ssize_t start, ssize_t end)
 {
-	Py_ssize_t len, blen;
-	const char *buffer;
+	ssize_t len;
 
 	if (start + 1 == end) {
 		/* trivial case, output a decoded list */
-		PyObject *tmp = PyList_GetItem(bins, start);
-		if (!tmp)
-			return NULL;
-		if (PyObject_AsCharBuffer(tmp, &buffer, &blen))
-			return NULL;
-		return decode(buffer, blen);
+		return get_next_item(bins, start);
 	}
 
 	/* divide and conquer, memory management is elsewhere */
 	len = (end - start) / 2;
-	return combine(fold(bins, start, start + len),
-		       fold(bins, start + len, end));
-}
-
-static PyObject *
-patches(PyObject *self, PyObject *args)
-{
-	PyObject *text, *bins, *result;
-	struct flist *patch;
-	const char *in;
-	char *out;
-	Py_ssize_t len, outlen, inlen;
-
-	if (!PyArg_ParseTuple(args, "OO:mpatch", &text, &bins))
-		return NULL;
-
-	len = PyList_Size(bins);
-	if (!len) {
-		/* nothing to do */
-		Py_INCREF(text);
-		return text;
-	}
-
-	if (PyObject_AsCharBuffer(text, &in, &inlen))
-		return NULL;
-
-	patch = fold(bins, 0, len);
-	if (!patch)
-		return NULL;
-
-	outlen = calcsize(inlen, patch);
-	if (outlen < 0) {
-		result = NULL;
-		goto cleanup;
-	}
-	result = PyBytes_FromStringAndSize(NULL, outlen);
-	if (!result) {
-		result = NULL;
-		goto cleanup;
-	}
-	out = PyBytes_AsString(result);
-	if (!apply(out, in, inlen, patch)) {
-		Py_DECREF(result);
-		result = NULL;
-	}
-cleanup:
-	lfree(patch);
-	return result;
+	return combine(mpatch_fold(bins, get_next_item, start, start + len),
+		       mpatch_fold(bins, get_next_item, start + len, end));
 }
-
-/* calculate size of a patched file directly */
-static PyObject *
-patchedsize(PyObject *self, PyObject *args)
-{
-	long orig, start, end, len, outlen = 0, last = 0, pos = 0;
-	Py_ssize_t patchlen;
-	char *bin;
-
-	if (!PyArg_ParseTuple(args, "ls#", &orig, &bin, &patchlen))
-		return NULL;
-
-	while (pos >= 0 && pos < patchlen) {
-		start = getbe32(bin + pos);
-		end = getbe32(bin + pos + 4);
-		len = getbe32(bin + pos + 8);
-		if (start > end)
-			break; /* sanity check */
-		pos += 12 + len;
-		outlen += start - last;
-		last = end;
-		outlen += len;
-	}
-
-	if (pos != patchlen) {
-		if (!PyErr_Occurred())
-			PyErr_SetString(mpatch_Error, "patch cannot be decoded");
-		return NULL;
-	}
-
-	outlen += orig - last;
-	return Py_BuildValue("l", outlen);
-}
-
-static PyMethodDef methods[] = {
-	{"patches", patches, METH_VARARGS, "apply a series of patches\n"},
-	{"patchedsize", patchedsize, METH_VARARGS, "calculed patched size\n"},
-	{NULL, NULL}
-};
-
-#ifdef IS_PY3K
-static struct PyModuleDef mpatch_module = {
-	PyModuleDef_HEAD_INIT,
-	"mpatch",
-	mpatch_doc,
-	-1,
-	methods
-};
-
-PyMODINIT_FUNC PyInit_mpatch(void)
-{
-	PyObject *m;
-
-	m = PyModule_Create(&mpatch_module);
-	if (m == NULL)
-		return NULL;
-
-	mpatch_Error = PyErr_NewException("mercurial.mpatch.mpatchError",
-					  NULL, NULL);
-	Py_INCREF(mpatch_Error);
-	PyModule_AddObject(m, "mpatchError", mpatch_Error);
-
-	return m;
-}
-#else
-PyMODINIT_FUNC
-initmpatch(void)
-{
-	Py_InitModule3("mpatch", methods, mpatch_doc);
-	mpatch_Error = PyErr_NewException("mercurial.mpatch.mpatchError",
-					  NULL, NULL);
-}
-#endif

mercurial/mpatch_module.c

@@ -27,288 +27,54 b''
 
 #include "util.h"
 #include "bitmanipulation.h"
+#include "compat.h"
+#include "mpatch.h"
 
 static char mpatch_doc[] = "Efficient binary patching.";
 static PyObject *mpatch_Error;
 
-struct frag {
-	int start, end, len;
-	const char *data;
-};
-
-struct flist {
-	struct frag *base, *head, *tail;
-};
-
-static struct flist *lalloc(Py_ssize_t size)
+static void setpyerr(int r)
 {
-	struct flist *a = NULL;
-
-	if (size < 1)
-		size = 1;
-
-	a = (struct flist *)malloc(sizeof(struct flist));
-	if (a) {
-		a->base = (struct frag *)malloc(sizeof(struct frag) * size);
-		if (a->base) {
-			a->head = a->tail = a->base;
-			return a;
-		}
-		free(a);
-		a = NULL;
-	}
-	if (!PyErr_Occurred())
+	switch (r) {
+	case MPATCH_ERR_NO_MEM:
 		PyErr_NoMemory();
-	return NULL;
-}
-
-static void lfree(struct flist *a)
-{
-	if (a) {
-		free(a->base);
-		free(a);
+		break;
+	case MPATCH_ERR_CANNOT_BE_DECODED:
+		PyErr_SetString(mpatch_Error, "patch cannot be decoded");
+		break;
+	case MPATCH_ERR_INVALID_PATCH:
+		PyErr_SetString(mpatch_Error, "invalid patch");
+		break;
 	}
 }
 
-static Py_ssize_t lsize(struct flist *a)
-{
-	return a->tail - a->head;
-}
-
-/* move hunks in source that are less cut to dest, compensating
-   for changes in offset. the last hunk may be split if necessary.
-*/
-static int gather(struct flist *dest, struct flist *src, int cut, int offset)
+struct mpatch_flist *cpygetitem(void *bins, ssize_t pos)
 {
-	struct frag *d = dest->tail, *s = src->head;
-	int postend, c, l;
-
-	while (s != src->tail) {
-		if (s->start + offset >= cut)
-			break; /* we've gone far enough */
-
-		postend = offset + s->start + s->len;
-		if (postend <= cut) {
-			/* save this hunk */
-			offset += s->start + s->len - s->end;
-			*d++ = *s++;
-		}
-		else {
-			/* break up this hunk */
-			c = cut - offset;
-			if (s->end < c)
-				c = s->end;
-			l = cut - offset - s->start;
-			if (s->len < l)
-				l = s->len;
-
-			offset += s->start + l - c;
-
-			d->start = s->start;
-			d->end = c;
-			d->len = l;
-			d->data = s->data;
-			d++;
-			s->start = c;
-			s->len = s->len - l;
-			s->data = s->data + l;
-
-			break;
-		}
-	}
-
-	dest->tail = d;
-	src->head = s;
-	return offset;
-}
-
-/* like gather, but with no output list */
-static int discard(struct flist *src, int cut, int offset)
-{
-	struct frag *s = src->head;
-	int postend, c, l;
-
-	while (s != src->tail) {
-		if (s->start + offset >= cut)
-			break;
-
-		postend = offset + s->start + s->len;
-		if (postend <= cut) {
-			offset += s->start + s->len - s->end;
-			s++;
-		}
-		else {
-			c = cut - offset;
-			if (s->end < c)
-				c = s->end;
-			l = cut - offset - s->start;
-			if (s->len < l)
-				l = s->len;
+	const char *buffer;
+	struct mpatch_flist *res;
+	ssize_t blen;
+	int r;
 
-			offset += s->start + l - c;
-			s->start = c;
-			s->len = s->len - l;
-			s->data = s->data + l;
-
-			break;
-		}
-	}
-
-	src->head = s;
-	return offset;
-}
-
-/* combine hunk lists a and b, while adjusting b for offset changes in a/
-   this deletes a and b and returns the resultant list. */
-static struct flist *combine(struct flist *a, struct flist *b)
-{
-	struct flist *c = NULL;
-	struct frag *bh, *ct;
-	int offset = 0, post;
-
-	if (a && b)
-		c = lalloc((lsize(a) + lsize(b)) * 2);
-
-	if (c) {
-
-		for (bh = b->head; bh != b->tail; bh++) {
-			/* save old hunks */
-			offset = gather(c, a, bh->start, offset);
-
-			/* discard replaced hunks */
-			post = discard(a, bh->end, offset);
-
-			/* insert new hunk */
-			ct = c->tail;
-			ct->start = bh->start - offset;
-			ct->end = bh->end - post;
-			ct->len = bh->len;
-			ct->data = bh->data;
-			c->tail++;
-			offset = post;
-		}
-
-		/* hold on to tail from a */
-		memcpy(c->tail, a->head, sizeof(struct frag) * lsize(a));
-		c->tail += lsize(a);
-	}
-
-	lfree(a);
-	lfree(b);
-	return c;
-}
-
-/* decode a binary patch into a hunk list */
-static struct flist *decode(const char *bin, Py_ssize_t len)
-{
-	struct flist *l;
-	struct frag *lt;
-	int pos = 0;
-
-	/* assume worst case size, we won't have many of these lists */
-	l = lalloc(len / 12 + 1);
-	if (!l)
+	PyObject *tmp = PyList_GetItem((PyObject*)bins, pos);
+	if (!tmp)
 		return NULL;
-
-	lt = l->tail;
-
-	while (pos >= 0 && pos < len) {
-		lt->start = getbe32(bin + pos);
-		lt->end = getbe32(bin + pos + 4);
-		lt->len = getbe32(bin + pos + 8);
-		lt->data = bin + pos + 12;
-		pos += 12 + lt->len;
-		if (lt->start > lt->end || lt->len < 0)
-			break; /* sanity check */
-		lt++;
-	}
-
-	if (pos != len) {
+	if (PyObject_AsCharBuffer(tmp, &buffer, (Py_ssize_t*)&blen))
+		return NULL;
+	if ((r = mpatch_decode(buffer, blen, &res)) < 0) {
 		if (!PyErr_Occurred())
-			PyErr_SetString(mpatch_Error, "patch cannot be decoded");
-		lfree(l);
+			setpyerr(r);
 		return NULL;
 	}
-
-	l->tail = lt;
-	return l;
-}
-
-/* calculate the size of resultant text */
-static Py_ssize_t calcsize(Py_ssize_t len, struct flist *l)
-{
-	Py_ssize_t outlen = 0, last = 0;
-	struct frag *f = l->head;
-
-	while (f != l->tail) {
-		if (f->start < last || f->end > len) {
-			if (!PyErr_Occurred())
-				PyErr_SetString(mpatch_Error,
-				                "invalid patch");
-			return -1;
-		}
-		outlen += f->start - last;
-		last = f->end;
-		outlen += f->len;
-		f++;
-	}
-
-	outlen += len - last;
-	return outlen;
-}
-
-static int apply(char *buf, const char *orig, Py_ssize_t len, struct flist *l)
-{
-	struct frag *f = l->head;
-	int last = 0;
-	char *p = buf;
-
-	while (f != l->tail) {
-		if (f->start < last || f->end > len) {
-			if (!PyErr_Occurred())
-				PyErr_SetString(mpatch_Error,
-				                "invalid patch");
-			return 0;
-		}
-		memcpy(p, orig + last, f->start - last);
-		p += f->start - last;
-		memcpy(p, f->data, f->len);
-		last = f->end;
-		p += f->len;
-		f++;
-	}
-	memcpy(p, orig + last, len - last);
-	return 1;
-}
-
-/* recursively generate a patch of all bins between start and end */
-static struct flist *fold(PyObject *bins, Py_ssize_t start, Py_ssize_t end)
-{
-	Py_ssize_t len, blen;
-	const char *buffer;
-
-	if (start + 1 == end) {
-		/* trivial case, output a decoded list */
-		PyObject *tmp = PyList_GetItem(bins, start);
-		if (!tmp)
-			return NULL;
-		if (PyObject_AsCharBuffer(tmp, &buffer, &blen))
-			return NULL;
-		return decode(buffer, blen);
-	}
-
-	/* divide and conquer, memory management is elsewhere */
-	len = (end - start) / 2;
-	return combine(fold(bins, start, start + len),
-		       fold(bins, start + len, end));
+	return res;
 }
 
 static PyObject *
 patches(PyObject *self, PyObject *args)
 {
 	PyObject *text, *bins, *result;
-	struct flist *patch;
+	struct mpatch_flist *patch;
 	const char *in;
+	int r = 0;
 	char *out;
 	Py_ssize_t len, outlen, inlen;
 
@@ -325,12 +91,16 b' patches(PyObject *self, PyObject *args)'
 	if (PyObject_AsCharBuffer(text, &in, &inlen))
 		return NULL;
 
-	patch = fold(bins, 0, len);
-	if (!patch)
+	patch = mpatch_fold(bins, cpygetitem, 0, len);
+	if (!patch) { /* error already set or memory error */
+		if (!PyErr_Occurred())
+			PyErr_NoMemory();
 		return NULL;
+	}
 
-	outlen = calcsize(inlen, patch);
+	outlen = mpatch_calcsize(inlen, patch);
 	if (outlen < 0) {
+		r = (int)outlen;
 		result = NULL;
 		goto cleanup;
 	}
@@ -340,12 +110,14 b' patches(PyObject *self, PyObject *args)'
 		goto cleanup;
 	}
 	out = PyBytes_AsString(result);
-	if (!apply(out, in, inlen, patch)) {
+	if ((r = mpatch_apply(out, in, inlen, patch)) < 0) {
 		Py_DECREF(result);
 		result = NULL;
 	}
 cleanup:
-	lfree(patch);
+	mpatch_lfree(patch);
+	if (!result && !PyErr_Occurred())
+		setpyerr(r);
 	return result;
 }
 

mercurial/obsolete.py

@@ -42,9 +42,9 b' Examples:'
 
     (A, ())
 
-- When changeset A is split into B and C, a single marker are used:
+- When changeset A is split into B and C, a single marker is used:
 
-    (A, (C, C))
+    (A, (B, C))
 
   We use a single marker to distinguish the "split" case from the "divergence"
   case. If two independent operations rewrite the same changeset A in to A' and
@@ -1236,7 +1236,7 b' def createmarkers(repo, relations, flag='
             if not prec.mutable():
                 raise error.Abort(_("cannot obsolete public changeset: %s")
                                  % prec,
-                                 hint='see "hg help phases" for details')
+                                 hint="see 'hg help phases' for details")
             nprec = prec.node()
             nsucs = tuple(s.node() for s in sucs)
             npare = None

mercurial/osutil.c

@@ -63,11 +63,19 b' struct listdir_stat {'
 };
 #endif
 
+#ifdef IS_PY3K
+#define listdir_slot(name) \
+	static PyObject *listdir_stat_##name(PyObject *self, void *x) \
+	{ \
+		return PyLong_FromLong(((struct listdir_stat *)self)->st.name); \
+	}
+#else
 #define listdir_slot(name) \
 	static PyObject *listdir_stat_##name(PyObject *self, void *x) \
 	{ \
 		return PyInt_FromLong(((struct listdir_stat *)self)->st.name); \
 	}
+#endif
 
 listdir_slot(st_dev)
 listdir_slot(st_mode)
@@ -624,7 +632,7 b' static PyObject *statfiles(PyObject *sel'
 		pypath = PySequence_GetItem(names, i);
 		if (!pypath)
 			goto bail;
-		path = PyString_AsString(pypath);
+		path = PyBytes_AsString(pypath);
 		if (path == NULL) {
 			Py_DECREF(pypath);
 			PyErr_SetString(PyExc_TypeError, "not a string");
@@ -706,7 +714,7 b' static PyObject *recvfds(PyObject *self,'
 	if (!rfdslist)
 		goto bail;
 	for (i = 0; i < rfdscount; i++) {
-		PyObject *obj = PyInt_FromLong(rfds[i]);
+		PyObject *obj = PyLong_FromLong(rfds[i]);
 		if (!obj)
 			goto bail;
 		PyList_SET_ITEM(rfdslist, i, obj);

mercurial/parser.py

@@ -65,7 +65,7 b' class parser(object):'
             # handle infix rules, take as suffix if unambiguous
             infix, suffix = self._elements[token][3:]
             if suffix and not (infix and self._hasnewterm()):
-                expr = (suffix[0], expr)
+                expr = (suffix, expr)
             elif infix:
                 expr = (infix[0], expr, self._parseoperand(*infix[1:]))
             else:

mercurial/parsers.c

@@ -15,6 +15,18 b''
 #include "util.h"
 #include "bitmanipulation.h"
 
+#ifdef IS_PY3K
+/* The mapping of Python types is meant to be temporary to get Python
+ * 3 to compile. We should remove this once Python 3 support is fully
+ * supported and proper types are used in the extensions themselves. */
+#define PyInt_Type PyLong_Type
+#define PyInt_Check PyLong_Check
+#define PyInt_FromLong PyLong_FromLong
+#define PyInt_FromSsize_t PyLong_FromSsize_t
+#define PyInt_AS_LONG PyLong_AS_LONG
+#define PyInt_AsLong PyLong_AsLong
+#endif
+
 static char *versionerrortext = "Python minor version mismatch";
 
 static int8_t hextable[256] = {
@@ -610,37 +622,37 b' static PyObject *pack_dirstate(PyObject '
 	/* Figure out how much we need to allocate. */
 	for (nbytes = 40, pos = 0; PyDict_Next(map, &pos, &k, &v);) {
 		PyObject *c;
-		if (!PyString_Check(k)) {
+		if (!PyBytes_Check(k)) {
 			PyErr_SetString(PyExc_TypeError, "expected string key");
 			goto bail;
 		}
-		nbytes += PyString_GET_SIZE(k) + 17;
+		nbytes += PyBytes_GET_SIZE(k) + 17;
 		c = PyDict_GetItem(copymap, k);
 		if (c) {
-			if (!PyString_Check(c)) {
+			if (!PyBytes_Check(c)) {
 				PyErr_SetString(PyExc_TypeError,
 						"expected string key");
 				goto bail;
 			}
-			nbytes += PyString_GET_SIZE(c) + 1;
+			nbytes += PyBytes_GET_SIZE(c) + 1;
 		}
 	}
 
-	packobj = PyString_FromStringAndSize(NULL, nbytes);
+	packobj = PyBytes_FromStringAndSize(NULL, nbytes);
 	if (packobj == NULL)
 		goto bail;
 
-	p = PyString_AS_STRING(packobj);
+	p = PyBytes_AS_STRING(packobj);
 
 	pn = PySequence_ITEM(pl, 0);
-	if (PyString_AsStringAndSize(pn, &s, &l) == -1 || l != 20) {
+	if (PyBytes_AsStringAndSize(pn, &s, &l) == -1 || l != 20) {
 		PyErr_SetString(PyExc_TypeError, "expected a 20-byte hash");
 		goto bail;
 	}
 	memcpy(p, s, l);
 	p += 20;
 	pn = PySequence_ITEM(pl, 1);
-	if (PyString_AsStringAndSize(pn, &s, &l) == -1 || l != 20) {
+	if (PyBytes_AsStringAndSize(pn, &s, &l) == -1 || l != 20) {
 		PyErr_SetString(PyExc_TypeError, "expected a 20-byte hash");
 		goto bail;
 	}
@@ -685,21 +697,21 b' static PyObject *pack_dirstate(PyObject '
 		putbe32((uint32_t)mtime, p + 8);
 		t = p + 12;
 		p += 16;
-		len = PyString_GET_SIZE(k);
-		memcpy(p, PyString_AS_STRING(k), len);
+		len = PyBytes_GET_SIZE(k);
+		memcpy(p, PyBytes_AS_STRING(k), len);
 		p += len;
 		o = PyDict_GetItem(copymap, k);
 		if (o) {
 			*p++ = '\0';
-			l = PyString_GET_SIZE(o);
-			memcpy(p, PyString_AS_STRING(o), l);
+			l = PyBytes_GET_SIZE(o);
+			memcpy(p, PyBytes_AS_STRING(o), l);
 			p += l;
 			len += l + 1;
 		}
 		putbe32((uint32_t)len, t);
 	}
 
-	pos = p - PyString_AS_STRING(packobj);
+	pos = p - PyBytes_AS_STRING(packobj);
 	if (pos != nbytes) {
 		PyErr_Format(PyExc_SystemError, "bad dirstate size: %ld != %ld",
                              (long)pos, (long)nbytes);
@@ -796,7 +808,7 b' static const char *index_deref(indexObje'
 		return self->offsets[pos];
 	}
 
-	return PyString_AS_STRING(self->data) + pos * v1_hdrsize;
+	return PyBytes_AS_STRING(self->data) + pos * v1_hdrsize;
 }
 
 static inline int index_get_parents(indexObject *self, Py_ssize_t rev,
@@ -926,7 +938,7 b' static const char *index_node(indexObjec'
 		PyObject *tuple, *str;
 		tuple = PyList_GET_ITEM(self->added, pos - self->length + 1);
 		str = PyTuple_GetItem(tuple, 7);
-		return str ? PyString_AS_STRING(str) : NULL;
+		return str ? PyBytes_AS_STRING(str) : NULL;
 	}
 
 	data = index_deref(self, pos);
@@ -937,7 +949,7 b' static int nt_insert(indexObject *self, '
 
 static int node_check(PyObject *obj, char **node, Py_ssize_t *nodelen)
 {
-	if (PyString_AsStringAndSize(obj, node, nodelen) == -1)
+	if (PyBytes_AsStringAndSize(obj, node, nodelen) == -1)
 		return -1;
 	if (*nodelen == 20)
 		return 0;
@@ -1825,7 +1837,7 b' static PyObject *index_partialmatch(inde'
 	case -2:
 		Py_RETURN_NONE;
 	case -1:
-		return PyString_FromStringAndSize(nullid, 20);
+		return PyBytes_FromStringAndSize(nullid, 20);
 	}
 
 	fullnode = index_node(self, rev);
@@ -1834,7 +1846,7 b' static PyObject *index_partialmatch(inde'
 			     "could not access rev %d", rev);
 		return NULL;
 	}
-	return PyString_FromStringAndSize(fullnode, 20);
+	return PyBytes_FromStringAndSize(fullnode, 20);
 }
 
 static PyObject *index_m_get(indexObject *self, PyObject *args)
@@ -2247,7 +2259,7 b' static void nt_invalidate_added(indexObj'
 		PyObject *tuple = PyList_GET_ITEM(self->added, i);
 		PyObject *node = PyTuple_GET_ITEM(tuple, 7);
 
-		nt_insert(self, PyString_AS_STRING(node), -1);
+		nt_insert(self, PyBytes_AS_STRING(node), -1);
 	}
 
 	if (start == 0)
@@ -2264,7 +2276,12 b' static int index_slice_del(indexObject *'
 	Py_ssize_t length = index_length(self);
 	int ret = 0;
 
+/* Argument changed from PySliceObject* to PyObject* in Python 3. */
+#ifdef IS_PY3K
+	if (PySlice_GetIndicesEx(item, length,
+#else
 	if (PySlice_GetIndicesEx((PySliceObject*)item, length,
+#endif
 				 &start, &stop, &step, &slicelength) < 0)
 		return -1;
 
@@ -2372,9 +2389,9 b' static int index_assign_subscript(indexO'
  */
 static Py_ssize_t inline_scan(indexObject *self, const char **offsets)
 {
-	const char *data = PyString_AS_STRING(self->data);
+	const char *data = PyBytes_AS_STRING(self->data);
 	Py_ssize_t pos = 0;
-	Py_ssize_t end = PyString_GET_SIZE(self->data);
+	Py_ssize_t end = PyBytes_GET_SIZE(self->data);
 	long incr = v1_hdrsize;
 	Py_ssize_t len = 0;
 
@@ -2416,11 +2433,11 b' static int index_init(indexObject *self,'
 
 	if (!PyArg_ParseTuple(args, "OO", &data_obj, &inlined_obj))
 		return -1;
-	if (!PyString_Check(data_obj)) {
+	if (!PyBytes_Check(data_obj)) {
 		PyErr_SetString(PyExc_TypeError, "data is not a string");
 		return -1;
 	}
-	size = PyString_GET_SIZE(data_obj);
+	size = PyBytes_GET_SIZE(data_obj);
 
 	self->inlined = inlined_obj && PyObject_IsTrue(inlined_obj);
 	self->data = data_obj;
@@ -2516,8 +2533,7 b' static PyGetSetDef index_getset[] = {'
 };
 
 static PyTypeObject indexType = {
-	PyObject_HEAD_INIT(NULL)
-	0,                         /* ob_size */
+	PyVarObject_HEAD_INIT(NULL, 0)
 	"parsers.index",           /* tp_name */
 	sizeof(indexObject),       /* tp_basicsize */
 	0,                         /* tp_itemsize */
@@ -2613,7 +2629,7 b' static PyObject *readshas('
 		return NULL;
 	}
 	for (i = 0; i < num; i++) {
-		PyObject *hash = PyString_FromStringAndSize(source, hashwidth);
+		PyObject *hash = PyBytes_FromStringAndSize(source, hashwidth);
 		if (hash == NULL) {
 			Py_DECREF(list);
 			return NULL;
@@ -2669,7 +2685,7 b' static PyObject *fm1readmarker(const cha'
 	if (data + hashwidth > dataend) {
 		goto overflow;
 	}
-	prec = PyString_FromStringAndSize(data, hashwidth);
+	prec = PyBytes_FromStringAndSize(data, hashwidth);
 	data += hashwidth;
 	if (prec == NULL) {
 		goto bail;
@@ -2712,9 +2728,9 b' static PyObject *fm1readmarker(const cha'
 		if (meta + leftsize + rightsize > dataend) {
 			goto overflow;
 		}
-		left = PyString_FromStringAndSize(meta, leftsize);
+		left = PyBytes_FromStringAndSize(meta, leftsize);
 		meta += leftsize;
-		right = PyString_FromStringAndSize(meta, rightsize);
+		right = PyBytes_FromStringAndSize(meta, rightsize);
 		meta += rightsize;
 		tmp = PyTuple_New(2);
 		if (!left || !right || !tmp) {
@@ -2880,7 +2896,7 b' PyMODINIT_FUNC PyInit_parsers(void)'
 	PyObject *mod;
 
 	if (check_python_version() == -1)
-		return;
+		return NULL;
 	mod = PyModule_Create(&parsers_module);
 	module_init(mod);
 	return mod;

mercurial/patch.py

@@ -410,11 +410,7 b' class linereader(object):'
         return self.fp.readline()
 
     def __iter__(self):
-        while True:
-            l = self.readline()
-            if not l:
-                break
-            yield l
+        return iter(self.readline, '')
 
 class abstractbackend(object):
     def __init__(self, ui):
@@ -673,6 +669,8 b' class patchfile(object):'
                 self.mode = (False, False)
         if self.missing:
             self.ui.warn(_("unable to find '%s' for patching\n") % self.fname)
+            self.ui.warn(_("(use '--prefix' to apply patch relative to the "
+                           "current directory)\n"))
 
         self.hash = {}
         self.dirty = 0
@@ -1688,10 +1686,7 b' def scanpatch(fp):'
     def scanwhile(first, p):
         """scan lr while predicate holds"""
         lines = [first]
-        while True:
-            line = lr.readline()
-            if not line:
-                break
+        for line in iter(lr.readline, ''):
             if p(line):
                 lines.append(line)
             else:
@@ -1699,10 +1694,7 b' def scanpatch(fp):'
                 break
         return lines
 
-    while True:
-        line = lr.readline()
-        if not line:
-            break
+    for line in iter(lr.readline, ''):
         if line.startswith('diff --git a/') or line.startswith('diff -r '):
             def notheader(line):
                 s = line.split(None, 1)
@@ -1772,10 +1764,7 b' def iterhunks(fp):'
     context = None
     lr = linereader(fp)
 
-    while True:
-        x = lr.readline()
-        if not x:
-            break
+    for x in iter(lr.readline, ''):
         if state == BFILE and (
             (not context and x[0] == '@')
             or (context is not False and x.startswith('***************'))
@@ -1963,8 +1952,10 b' def _applydiff(ui, fp, patcher, backend,'
                 data, mode = None, None
                 if gp.op in ('RENAME', 'COPY'):
                     data, mode = store.getfile(gp.oldpath)[:2]
-                    # FIXME: failing getfile has never been handled here
-                    assert data is not None
+                    if data is None:
+                        # This means that the old path does not exist
+                        raise PatchError(_("source file '%s' does not exist")
+                                           % gp.oldpath)
                 if gp.mode:
                     mode = gp.mode
                     if gp.op == 'ADD':
@@ -2155,7 +2146,14 b' def difffeatureopts(ui, opts=None, untru'
     def get(key, name=None, getter=ui.configbool, forceplain=None):
         if opts:
             v = opts.get(key)
-            if v:
+            # diffopts flags are either None-default (which is passed
+            # through unchanged, so we can identify unset values), or
+            # some other falsey default (eg --unified, which defaults
+            # to an empty string). We only want to override the config
+            # entries from hgrc with command line values if they
+            # appear to have been set, which is any truthy value,
+            # True, or False.
+            if v or isinstance(v, bool):
                 return v
         if forceplain is not None and ui.plain():
             return forceplain

mercurial/pathencode.c

@@ -156,7 +156,7 b' PyObject *encodedir(PyObject *self, PyOb'
 	if (!PyArg_ParseTuple(args, "O:encodedir", &pathobj))
 		return NULL;
 
-	if (PyString_AsStringAndSize(pathobj, &path, &len) == -1) {
+	if (PyBytes_AsStringAndSize(pathobj, &path, &len) == -1) {
 		PyErr_SetString(PyExc_TypeError, "expected a string");
 		return NULL;
 	}
@@ -168,11 +168,12 b' PyObject *encodedir(PyObject *self, PyOb'
 		return pathobj;
 	}
 
-	newobj = PyString_FromStringAndSize(NULL, newlen);
+	newobj = PyBytes_FromStringAndSize(NULL, newlen);
 
 	if (newobj) {
-		PyString_GET_SIZE(newobj)--;
-		_encodedir(PyString_AS_STRING(newobj), newlen, path,
+		assert(PyBytes_Check(newobj));
+		Py_SIZE(newobj)--;
+		_encodedir(PyBytes_AS_STRING(newobj), newlen, path,
 			   len + 1);
 	}
 
@@ -515,9 +516,9 b' PyObject *lowerencode(PyObject *self, Py'
 		return NULL;
 
 	newlen = _lowerencode(NULL, 0, path, len);
-	ret = PyString_FromStringAndSize(NULL, newlen);
+	ret = PyBytes_FromStringAndSize(NULL, newlen);
 	if (ret)
-		_lowerencode(PyString_AS_STRING(ret), newlen, path, len);
+		_lowerencode(PyBytes_AS_STRING(ret), newlen, path, len);
 
 	return ret;
 }
@@ -568,11 +569,11 b' static PyObject *hashmangle(const char *'
 	if (lastdot >= 0)
 		destsize += len - lastdot - 1;
 
-	ret = PyString_FromStringAndSize(NULL, destsize);
+	ret = PyBytes_FromStringAndSize(NULL, destsize);
 	if (ret == NULL)
 		return NULL;
 
-	dest = PyString_AS_STRING(ret);
+	dest = PyBytes_AS_STRING(ret);
 	memcopy(dest, &destlen, destsize, "dh/", 3);
 
 	/* Copy up to dirprefixlen bytes of each path component, up to
@@ -638,7 +639,8 b' static PyObject *hashmangle(const char *'
 		memcopy(dest, &destlen, destsize, &src[lastdot],
 			len - lastdot - 1);
 
-	PyString_GET_SIZE(ret) = destlen;
+	assert(PyBytes_Check(ret));
+	Py_SIZE(ret) = destlen;
 
 	return ret;
 }
@@ -653,7 +655,7 b' static int sha1hash(char hash[20], const'
 	PyObject *shaobj, *hashobj;
 
 	if (shafunc == NULL) {
-		PyObject *hashlib, *name = PyString_FromString("hashlib");
+		PyObject *hashlib, *name = PyBytes_FromString("hashlib");
 
 		if (name == NULL)
 			return -1;
@@ -686,14 +688,14 b' static int sha1hash(char hash[20], const'
 	if (hashobj == NULL)
 		return -1;
 
-	if (!PyString_Check(hashobj) || PyString_GET_SIZE(hashobj) != 20) {
+	if (!PyBytes_Check(hashobj) || PyBytes_GET_SIZE(hashobj) != 20) {
 		PyErr_SetString(PyExc_TypeError,
 				"result of digest is not a 20-byte hash");
 		Py_DECREF(hashobj);
 		return -1;
 	}
 
-	memcpy(hash, PyString_AS_STRING(hashobj), 20);
+	memcpy(hash, PyBytes_AS_STRING(hashobj), 20);
 	Py_DECREF(hashobj);
 	return 0;
 }
@@ -731,7 +733,7 b' PyObject *pathencode(PyObject *self, PyO'
 	if (!PyArg_ParseTuple(args, "O:pathencode", &pathobj))
 		return NULL;
 
-	if (PyString_AsStringAndSize(pathobj, &path, &len) == -1) {
+	if (PyBytes_AsStringAndSize(pathobj, &path, &len) == -1) {
 		PyErr_SetString(PyExc_TypeError, "expected a string");
 		return NULL;
 	}
@@ -747,11 +749,12 b' PyObject *pathencode(PyObject *self, PyO'
 			return pathobj;
 		}
 
-		newobj = PyString_FromStringAndSize(NULL, newlen);
+		newobj = PyBytes_FromStringAndSize(NULL, newlen);
 
 		if (newobj) {
-			PyString_GET_SIZE(newobj)--;
-			basicencode(PyString_AS_STRING(newobj), newlen, path,
+			assert(PyBytes_Check(newobj));
+			Py_SIZE(newobj)--;
+			basicencode(PyBytes_AS_STRING(newobj), newlen, path,
 				    len + 1);
 		}
 	}

mercurial/pathutil.py

@@ -40,7 +40,7 b' class pathauditor(object):'
         self.root = root
         self._realfs = realfs
         self.callback = callback
-        if os.path.lexists(root) and not util.checkcase(root):
+        if os.path.lexists(root) and not util.fscasesensitive(root):
             self.normcase = util.normcase
         else:
             self.normcase = lambda x: x

mercurial/pure/bdiff.py

@@ -12,6 +12,10 b' import difflib'
 import re
 import struct
 
+from . import policy
+policynocffi = policy.policynocffi
+modulepolicy = policy.policy
+
 def splitnewlines(text):
     '''like str.splitlines, but only split on newlines.'''
     lines = [l + '\n' for l in text.split('\n')]
@@ -96,3 +100,70 b' def fixws(text, allws):'
         text = re.sub('[ \t\r]+', ' ', text)
         text = text.replace(' \n', '\n')
     return text
+
+if modulepolicy not in policynocffi:
+    try:
+        from _bdiff_cffi import ffi, lib
+    except ImportError:
+        if modulepolicy == 'cffi': # strict cffi import
+            raise
+    else:
+        def blocks(sa, sb):
+            a = ffi.new("struct bdiff_line**")
+            b = ffi.new("struct bdiff_line**")
+            ac = ffi.new("char[]", str(sa))
+            bc = ffi.new("char[]", str(sb))
+            l = ffi.new("struct bdiff_hunk*")
+            try:
+                an = lib.bdiff_splitlines(ac, len(sa), a)
+                bn = lib.bdiff_splitlines(bc, len(sb), b)
+                if not a[0] or not b[0]:
+                    raise MemoryError
+                count = lib.bdiff_diff(a[0], an, b[0], bn, l)
+                if count < 0:
+                    raise MemoryError
+                rl = [None] * count
+                h = l.next
+                i = 0
+                while h:
+                    rl[i] = (h.a1, h.a2, h.b1, h.b2)
+                    h = h.next
+                    i += 1
+            finally:
+                lib.free(a[0])
+                lib.free(b[0])
+                lib.bdiff_freehunks(l.next)
+            return rl
+
+        def bdiff(sa, sb):
+            a = ffi.new("struct bdiff_line**")
+            b = ffi.new("struct bdiff_line**")
+            ac = ffi.new("char[]", str(sa))
+            bc = ffi.new("char[]", str(sb))
+            l = ffi.new("struct bdiff_hunk*")
+            try:
+                an = lib.bdiff_splitlines(ac, len(sa), a)
+                bn = lib.bdiff_splitlines(bc, len(sb), b)
+                if not a[0] or not b[0]:
+                    raise MemoryError
+                count = lib.bdiff_diff(a[0], an, b[0], bn, l)
+                if count < 0:
+                    raise MemoryError
+                rl = []
+                h = l.next
+                la = lb = 0
+                while h:
+                    if h.a1 != la or h.b1 != lb:
+                        lgt = (b[0] + h.b1).l - (b[0] + lb).l
+                        rl.append(struct.pack(">lll", (a[0] + la).l - a[0].l,
+                            (a[0] + h.a1).l - a[0].l, lgt))
+                        rl.append(str(ffi.buffer((b[0] + lb).l, lgt)))
+                    la = h.a2
+                    lb = h.b2
+                    h = h.next
+
+            finally:
+                lib.free(a[0])
+                lib.free(b[0])
+                lib.bdiff_freehunks(l.next)
+            return "".join(rl)

mercurial/pure/mpatch.py

@@ -9,8 +9,10 b' from __future__ import absolute_import'
 
 import struct
 
-from . import pycompat
+from . import policy, pycompat
 stringio = pycompat.stringio
+modulepolicy = policy.policy
+policynocffi = policy.policynocffi
 
 class mpatchError(Exception):
     """error raised when a delta cannot be decoded
@@ -125,3 +127,44 b' def patchedsize(orig, delta):'
 
     outlen += orig - last
     return outlen
+
+if modulepolicy not in policynocffi:
+    try:
+        from _mpatch_cffi import ffi, lib
+    except ImportError:
+        if modulepolicy == 'cffi': # strict cffi import
+            raise
+    else:
+        @ffi.def_extern()
+        def cffi_get_next_item(arg, pos):
+            all, bins = ffi.from_handle(arg)
+            container = ffi.new("struct mpatch_flist*[1]")
+            to_pass = ffi.new("char[]", str(bins[pos]))
+            all.append(to_pass)
+            r = lib.mpatch_decode(to_pass, len(to_pass) - 1, container)
+            if r < 0:
+                return ffi.NULL
+            return container[0]
+
+        def patches(text, bins):
+            lgt = len(bins)
+            all = []
+            if not lgt:
+                return text
+            arg = (all, bins)
+            patch = lib.mpatch_fold(ffi.new_handle(arg),
+                                    lib.cffi_get_next_item, 0, lgt)
+            if not patch:
+                raise mpatchError("cannot decode chunk")
+            outlen = lib.mpatch_calcsize(len(text), patch)
+            if outlen < 0:
+                lib.mpatch_lfree(patch)
+                raise mpatchError("inconsistency detected")
+            buf = ffi.new("char[]", outlen)
+            if lib.mpatch_apply(buf, text, len(text), patch) < 0:
+                lib.mpatch_lfree(patch)
+                raise mpatchError("error applying patches")
+            res = ffi.buffer(buf, outlen)[:]
+            lib.mpatch_lfree(patch)
+            return res
+

mercurial/pure/osutil.py

@@ -120,13 +120,14 b" if sys.platform == 'darwin' and ffi is n"
                 if skip == name and tp == statmod.S_ISDIR:
                     return []
                 if stat:
-                    mtime = cur.time.tv_sec
+                    mtime = cur.mtime.tv_sec
                     mode = (cur.accessmask & ~lib.S_IFMT)| tp
                     ret.append((name, tp, stat_res(st_mode=mode, st_mtime=mtime,
                                 st_size=cur.datalength)))
                 else:
                     ret.append((name, tp))
-                cur += lgt
+                cur = ffi.cast("val_attrs_t*", int(ffi.cast("intptr_t", cur))
+                    + lgt)
         return ret
 
     def listdir(path, stat=False, skip=None):
@@ -173,30 +174,30 b" if os.name != 'nt':"
 
     class _iovec(ctypes.Structure):
         _fields_ = [
-            ('iov_base', ctypes.c_void_p),
-            ('iov_len', ctypes.c_size_t),
+            (u'iov_base', ctypes.c_void_p),
+            (u'iov_len', ctypes.c_size_t),
         ]
 
     class _msghdr(ctypes.Structure):
         _fields_ = [
-            ('msg_name', ctypes.c_void_p),
-            ('msg_namelen', _socklen_t),
-            ('msg_iov', ctypes.POINTER(_iovec)),
-            ('msg_iovlen', _msg_iovlen_t),
-            ('msg_control', ctypes.c_void_p),
-            ('msg_controllen', _msg_controllen_t),
-            ('msg_flags', ctypes.c_int),
+            (u'msg_name', ctypes.c_void_p),
+            (u'msg_namelen', _socklen_t),
+            (u'msg_iov', ctypes.POINTER(_iovec)),
+            (u'msg_iovlen', _msg_iovlen_t),
+            (u'msg_control', ctypes.c_void_p),
+            (u'msg_controllen', _msg_controllen_t),
+            (u'msg_flags', ctypes.c_int),
         ]
 
     class _cmsghdr(ctypes.Structure):
         _fields_ = [
-            ('cmsg_len', _cmsg_len_t),
-            ('cmsg_level', ctypes.c_int),
-            ('cmsg_type', ctypes.c_int),
-            ('cmsg_data', ctypes.c_ubyte * 0),
+            (u'cmsg_len', _cmsg_len_t),
+            (u'cmsg_level', ctypes.c_int),
+            (u'cmsg_type', ctypes.c_int),
+            (u'cmsg_data', ctypes.c_ubyte * 0),
         ]
 
-    _libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
+    _libc = ctypes.CDLL(ctypes.util.find_library(u'c'), use_errno=True)
     _recvmsg = getattr(_libc, 'recvmsg', None)
     if _recvmsg:
         _recvmsg.restype = getattr(ctypes, 'c_ssize_t', ctypes.c_long)

mercurial/pycompat.py

@@ -12,7 +12,9 b' from __future__ import absolute_import'
 
 import sys
 
-if sys.version_info[0] < 3:
+ispy3 = (sys.version_info[0] >= 3)
+
+if not ispy3:
     import cPickle as pickle
     import cStringIO as io
     import httplib
@@ -29,36 +31,84 b' else:'
     import urllib.parse as urlparse
     import xmlrpc.client as xmlrpclib
 
+if ispy3:
+    import builtins
+    import functools
+    import os
+    fsencode = os.fsencode
+
+    def sysstr(s):
+        """Return a keyword str to be passed to Python functions such as
+        getattr() and str.encode()
+
+        This never raises UnicodeDecodeError. Non-ascii characters are
+        considered invalid and mapped to arbitrary but unique code points
+        such that 'sysstr(a) != sysstr(b)' for all 'a != b'.
+        """
+        if isinstance(s, builtins.str):
+            return s
+        return s.decode(u'latin-1')
+
+    def _wrapattrfunc(f):
+        @functools.wraps(f)
+        def w(object, name, *args):
+            return f(object, sysstr(name), *args)
+        return w
+
+    # these wrappers are automagically imported by hgloader
+    delattr = _wrapattrfunc(builtins.delattr)
+    getattr = _wrapattrfunc(builtins.getattr)
+    hasattr = _wrapattrfunc(builtins.hasattr)
+    setattr = _wrapattrfunc(builtins.setattr)
+    xrange = builtins.range
+
+else:
+    def sysstr(s):
+        return s
+
+    # Partial backport from os.py in Python 3, which only accepts bytes.
+    # In Python 2, our paths should only ever be bytes, a unicode path
+    # indicates a bug.
+    def fsencode(filename):
+        if isinstance(filename, str):
+            return filename
+        else:
+            raise TypeError(
+                "expect str, not %s" % type(filename).__name__)
+
 stringio = io.StringIO
 empty = _queue.Empty
 queue = _queue.Queue
 
 class _pycompatstub(object):
-    pass
-
-def _alias(alias, origin, items):
-    """ populate a _pycompatstub
+    def __init__(self):
+        self._aliases = {}
 
-    copies items from origin to alias
-    """
-    def hgcase(item):
-        return item.replace('_', '').lower()
-    for item in items:
+    def _registeraliases(self, origin, items):
+        """Add items that will be populated at the first access"""
+        items = map(sysstr, items)
+        self._aliases.update(
+            (item.replace(sysstr('_'), sysstr('')).lower(), (origin, item))
+            for item in items)
+
+    def __getattr__(self, name):
         try:
-            setattr(alias, hgcase(item), getattr(origin, item))
-        except AttributeError:
-            pass
+            origin, item = self._aliases[name]
+        except KeyError:
+            raise AttributeError(name)
+        self.__dict__[name] = obj = getattr(origin, item)
+        return obj
 
 httpserver = _pycompatstub()
 urlreq = _pycompatstub()
 urlerr = _pycompatstub()
-try:
+if not ispy3:
     import BaseHTTPServer
     import CGIHTTPServer
     import SimpleHTTPServer
     import urllib2
     import urllib
-    _alias(urlreq, urllib, (
+    urlreq._registeraliases(urllib, (
         "addclosehook",
         "addinfourl",
         "ftpwrapper",
@@ -71,9 +121,8 b' try:'
         "unquote",
         "url2pathname",
         "urlencode",
-        "urlencode",
     ))
-    _alias(urlreq, urllib2, (
+    urlreq._registeraliases(urllib2, (
         "AbstractHTTPHandler",
         "BaseHandler",
         "build_opener",
@@ -89,24 +138,24 b' try:'
         "Request",
         "urlopen",
     ))
-    _alias(urlerr, urllib2, (
+    urlerr._registeraliases(urllib2, (
         "HTTPError",
         "URLError",
     ))
-    _alias(httpserver, BaseHTTPServer, (
+    httpserver._registeraliases(BaseHTTPServer, (
         "HTTPServer",
         "BaseHTTPRequestHandler",
     ))
-    _alias(httpserver, SimpleHTTPServer, (
+    httpserver._registeraliases(SimpleHTTPServer, (
         "SimpleHTTPRequestHandler",
     ))
-    _alias(httpserver, CGIHTTPServer, (
+    httpserver._registeraliases(CGIHTTPServer, (
         "CGIHTTPRequestHandler",
     ))
 
-except ImportError:
+else:
     import urllib.request
-    _alias(urlreq, urllib.request, (
+    urlreq._registeraliases(urllib.request, (
         "AbstractHTTPHandler",
         "addclosehook",
         "addinfourl",
@@ -134,20 +183,14 b' except ImportError:'
         "urlopen",
     ))
     import urllib.error
-    _alias(urlerr, urllib.error, (
+    urlerr._registeraliases(urllib.error, (
         "HTTPError",
         "URLError",
     ))
     import http.server
-    _alias(httpserver, http.server, (
+    httpserver._registeraliases(http.server, (
         "HTTPServer",
         "BaseHTTPRequestHandler",
         "SimpleHTTPRequestHandler",
         "CGIHTTPRequestHandler",
     ))
-
-try:
-    xrange
-except NameError:
-    import builtins
-    builtins.xrange = range

mercurial/registrar.py

@@ -8,6 +8,7 b''
 from __future__ import absolute_import
 
 from . import (
+    pycompat,
     util,
 )
 
@@ -108,6 +109,9 b' class revsetpredicate(_funcregistrarbase'
     Optional argument 'safe' indicates whether a predicate is safe for
     DoS attack (False by default).
 
+    Optional argument 'takeorder' indicates whether a predicate function
+    takes ordering policy as the last argument.
+
     'revsetpredicate' instance in example above can be used to
     decorate multiple functions.
 
@@ -118,10 +122,11 b' class revsetpredicate(_funcregistrarbase'
     Otherwise, explicit 'revset.loadpredicate()' is needed.
     """
     _getname = _funcregistrarbase._parsefuncdecl
-    _docformat = "``%s``\n    %s"
+    _docformat = pycompat.sysstr("``%s``\n    %s")
 
-    def _extrasetup(self, name, func, safe=False):
+    def _extrasetup(self, name, func, safe=False, takeorder=False):
         func._safe = safe
+        func._takeorder = takeorder
 
 class filesetpredicate(_funcregistrarbase):
     """Decorator to register fileset predicate
@@ -156,7 +161,7 b' class filesetpredicate(_funcregistrarbas'
     Otherwise, explicit 'fileset.loadpredicate()' is needed.
     """
     _getname = _funcregistrarbase._parsefuncdecl
-    _docformat = "``%s``\n    %s"
+    _docformat = pycompat.sysstr("``%s``\n    %s")
 
     def _extrasetup(self, name, func, callstatus=False, callexisting=False):
         func._callstatus = callstatus
@@ -165,7 +170,7 b' class filesetpredicate(_funcregistrarbas'
 class _templateregistrarbase(_funcregistrarbase):
     """Base of decorator to register functions as template specific one
     """
-    _docformat = ":%s: %s"
+    _docformat = pycompat.sysstr(":%s: %s")
 
 class templatekeyword(_templateregistrarbase):
     """Decorator to register template keyword

mercurial/repair.py

@@ -147,9 +147,10 b' def strip(ui, repo, nodelist, backup=Tru'
                        vfs.join(backupfile))
         repo.ui.log("backupbundle", "saved backup bundle to %s\n",
                     vfs.join(backupfile))
-    if saveheads or savebases:
-        # do not compress partial bundle if we remove it from disk later
-        chgrpfile = _bundle(repo, savebases, saveheads, node, 'temp',
+    tmpbundlefile = None
+    if saveheads:
+        # do not compress temporary bundle if we remove it from disk later
+        tmpbundlefile = _bundle(repo, savebases, saveheads, node, 'temp',
                             compress=False)
 
     mfst = repo.manifest
@@ -173,32 +174,34 b' def strip(ui, repo, nodelist, backup=Tru'
                     if (unencoded.startswith('meta/') and
                         unencoded.endswith('00manifest.i')):
                         dir = unencoded[5:-12]
-                        repo.dirlog(dir).strip(striprev, tr)
+                        repo.manifest.dirlog(dir).strip(striprev, tr)
             for fn in files:
                 repo.file(fn).strip(striprev, tr)
             tr.endgroup()
 
             for i in xrange(offset, len(tr.entries)):
                 file, troffset, ignore = tr.entries[i]
-                repo.svfs(file, 'a').truncate(troffset)
+                with repo.svfs(file, 'a', checkambig=True) as fp:
+                    fp.truncate(troffset)
                 if troffset == 0:
                     repo.store.markremoved(file)
 
-        if saveheads or savebases:
+        if tmpbundlefile:
             ui.note(_("adding branch\n"))
-            f = vfs.open(chgrpfile, "rb")
-            gen = exchange.readbundle(ui, f, chgrpfile, vfs)
+            f = vfs.open(tmpbundlefile, "rb")
+            gen = exchange.readbundle(ui, f, tmpbundlefile, vfs)
             if not repo.ui.verbose:
                 # silence internal shuffling chatter
                 repo.ui.pushbuffer()
             if isinstance(gen, bundle2.unbundle20):
                 with repo.transaction('strip') as tr:
                     tr.hookargs = {'source': 'strip',
-                                   'url': 'bundle:' + vfs.join(chgrpfile)}
+                                   'url': 'bundle:' + vfs.join(tmpbundlefile)}
                     bundle2.applybundle(repo, gen, tr, source='strip',
-                                        url='bundle:' + vfs.join(chgrpfile))
+                                        url='bundle:' + vfs.join(tmpbundlefile))
             else:
-                gen.apply(repo, 'strip', 'bundle:' + vfs.join(chgrpfile), True)
+                gen.apply(repo, 'strip', 'bundle:' + vfs.join(tmpbundlefile),
+                          True)
             if not repo.ui.verbose:
                 repo.ui.popbuffer()
             f.close()
@@ -227,16 +230,18 b' def strip(ui, repo, nodelist, backup=Tru'
 
     except: # re-raises
         if backupfile:
-            ui.warn(_("strip failed, full bundle stored in '%s'\n")
+            ui.warn(_("strip failed, backup bundle stored in '%s'\n")
                     % vfs.join(backupfile))
-        elif saveheads:
-            ui.warn(_("strip failed, partial bundle stored in '%s'\n")
-                    % vfs.join(chgrpfile))
+        if tmpbundlefile:
+            ui.warn(_("strip failed, unrecovered changes stored in '%s'\n")
+                    % vfs.join(tmpbundlefile))
+            ui.warn(_("(fix the problem, then recover the changesets with "
+                      "\"hg unbundle '%s'\")\n") % vfs.join(tmpbundlefile))
         raise
     else:
-        if saveheads or savebases:
-            # Remove partial backup only if there were no exceptions
-            vfs.unlink(chgrpfile)
+        if tmpbundlefile:
+            # Remove temporary bundle only if there were no exceptions
+            vfs.unlink(tmpbundlefile)
 
     repo.destroyed()
 

mercurial/revlog.py

@@ -212,8 +212,11 b' class revlog(object):'
     fashion, which means we never need to rewrite a file to insert or
     remove data, and can use some simple techniques to avoid the need
     for locking while reading.
+
+    If checkambig, indexfile is opened with checkambig=True at
+    writing, to avoid file stat ambiguity.
     """
-    def __init__(self, opener, indexfile):
+    def __init__(self, opener, indexfile, checkambig=False):
         """
         create a revlog object
 
@@ -223,11 +226,13 b' class revlog(object):'
         self.indexfile = indexfile
         self.datafile = indexfile[:-2] + ".d"
         self.opener = opener
+        #  When True, indexfile is opened with checkambig=True at writing, to
+        #  avoid file stat ambiguity.
+        self._checkambig = checkambig
         # 3-tuple of (node, rev, text) for a raw revision.
         self._cache = None
-        # 2-tuple of (rev, baserev) defining the base revision the delta chain
-        # begins at for a revision.
-        self._basecache = None
+        # Maps rev to chain base rev.
+        self._chainbasecache = util.lrucachedict(100)
         # 2-tuple of (offset, data) of raw data from the revlog at an offset.
         self._chunkcache = (0, '')
         # How much data to read and cache into the raw revlog data cache.
@@ -292,6 +297,8 b' class revlog(object):'
             raise RevlogError(_("index %s unknown format %d")
                               % (self.indexfile, fmt))
 
+        self.storedeltachains = True
+
         self._io = revlogio()
         if self.version == REVLOGV0:
             self._io = revlogoldio()
@@ -340,7 +347,7 b' class revlog(object):'
 
     def clearcaches(self):
         self._cache = None
-        self._basecache = None
+        self._chainbasecache.clear()
         self._chunkcache = (0, '')
         self._pcache = {}
 
@@ -390,11 +397,17 b' class revlog(object):'
     def length(self, rev):
         return self.index[rev][1]
     def chainbase(self, rev):
+        base = self._chainbasecache.get(rev)
+        if base is not None:
+            return base
+
         index = self.index
         base = index[rev][3]
         while base != rev:
             rev = base
             base = index[rev][3]
+
+        self._chainbasecache[rev] = base
         return base
     def chainlen(self, rev):
         return self._chaininfo(rev)[0]
@@ -1271,7 +1284,8 b' class revlog(object):'
         finally:
             df.close()
 
-        fp = self.opener(self.indexfile, 'w', atomictemp=True)
+        fp = self.opener(self.indexfile, 'w', atomictemp=True,
+                         checkambig=self._checkambig)
         self.version &= ~(REVLOGNGINLINEDATA)
         self._inline = False
         for i in self:
@@ -1314,7 +1328,7 b' class revlog(object):'
         dfh = None
         if not self._inline:
             dfh = self.opener(self.datafile, "a+")
-        ifh = self.opener(self.indexfile, "a+")
+        ifh = self.opener(self.indexfile, "a+", checkambig=self._checkambig)
         try:
             return self._addrevision(node, text, transaction, link, p1, p2,
                                      REVIDX_DEFAULT_FLAGS, cachedelta, ifh, dfh)
@@ -1428,30 +1442,24 b' class revlog(object):'
                         fh = dfh
                     ptext = self.revision(self.node(rev), _df=fh)
                     delta = mdiff.textdiff(ptext, t)
-            data = self.compress(delta)
-            l = len(data[1]) + len(data[0])
-            if basecache[0] == rev:
-                chainbase = basecache[1]
-            else:
-                chainbase = self.chainbase(rev)
-            dist = l + offset - self.start(chainbase)
+            header, data = self.compress(delta)
+            deltalen = len(header) + len(data)
+            chainbase = self.chainbase(rev)
+            dist = deltalen + offset - self.start(chainbase)
             if self._generaldelta:
                 base = rev
             else:
                 base = chainbase
             chainlen, compresseddeltalen = self._chaininfo(rev)
             chainlen += 1
-            compresseddeltalen += l
-            return dist, l, data, base, chainbase, chainlen, compresseddeltalen
+            compresseddeltalen += deltalen
+            return (dist, deltalen, (header, data), base,
+                    chainbase, chainlen, compresseddeltalen)
 
         curr = len(self)
         prev = curr - 1
-        base = chainbase = curr
         offset = self.end(prev)
         delta = None
-        if self._basecache is None:
-            self._basecache = (prev, self.chainbase(prev))
-        basecache = self._basecache
         p1r, p2r = self.rev(p1), self.rev(p2)
 
         # full versions are inserted when the needed deltas
@@ -1463,8 +1471,12 b' class revlog(object):'
             textlen = len(text)
 
         # should we try to build a delta?
-        if prev != nullrev:
+        if prev != nullrev and self.storedeltachains:
             tested = set()
+            # This condition is true most of the time when processing
+            # changegroup data into a generaldelta repo. The only time it
+            # isn't true is if this is the first revision in a delta chain
+            # or if ``format.generaldelta=true`` disabled ``lazydeltabase``.
             if cachedelta and self._generaldelta and self._lazydeltabase:
                 # Assume what we received from the server is a good choice
                 # build delta will reuse the cache
@@ -1515,7 +1527,7 b' class revlog(object):'
 
         if type(text) == str: # only accept immutable objects
             self._cache = (node, curr, text)
-        self._basecache = (curr, chainbase)
+        self._chainbasecache[curr] = chainbase
         return node
 
     def _writeentry(self, transaction, ifh, dfh, entry, data, link, offset):
@@ -1569,7 +1581,7 b' class revlog(object):'
         end = 0
         if r:
             end = self.end(r - 1)
-        ifh = self.opener(self.indexfile, "a+")
+        ifh = self.opener(self.indexfile, "a+", checkambig=self._checkambig)
         isize = r * self._io.size
         if self._inline:
             transaction.add(self.indexfile, end + isize, r)
@@ -1585,10 +1597,7 b' class revlog(object):'
         try:
             # loop through our set of deltas
             chain = None
-            while True:
-                chunkdata = cg.deltachunk(chain)
-                if not chunkdata:
-                    break
+            for chunkdata in iter(lambda: cg.deltachunk(chain), {}):
                 node = chunkdata['node']
                 p1 = chunkdata['p1']
                 p2 = chunkdata['p2']
@@ -1646,7 +1655,8 b' class revlog(object):'
                     # reopen the index
                     ifh.close()
                     dfh = self.opener(self.datafile, "a+")
-                    ifh = self.opener(self.indexfile, "a+")
+                    ifh = self.opener(self.indexfile, "a+",
+                                      checkambig=self._checkambig)
         finally:
             if dfh:
                 dfh.close()

mercurial/revset.py

@@ -9,6 +9,7 b' from __future__ import absolute_import'
 
 import heapq
 import re
+import string
 
 from .i18n import _
 from . import (
@@ -22,6 +23,7 b' from . import ('
     parser,
     pathutil,
     phases,
+    pycompat,
     registrar,
     repoview,
     util,
@@ -149,18 +151,16 b' elements = {'
     "(": (21, None, ("group", 1, ")"), ("func", 1, ")"), None),
     "##": (20, None, None, ("_concat", 20), None),
     "~": (18, None, None, ("ancestor", 18), None),
-    "^": (18, None, None, ("parent", 18), ("parentpost", 18)),
+    "^": (18, None, None, ("parent", 18), "parentpost"),
     "-": (5, None, ("negate", 19), ("minus", 5), None),
-    "::": (17, None, ("dagrangepre", 17), ("dagrange", 17),
-           ("dagrangepost", 17)),
-    "..": (17, None, ("dagrangepre", 17), ("dagrange", 17),
-           ("dagrangepost", 17)),
-    ":": (15, "rangeall", ("rangepre", 15), ("range", 15), ("rangepost", 15)),
+    "::": (17, None, ("dagrangepre", 17), ("dagrange", 17), "dagrangepost"),
+    "..": (17, None, ("dagrangepre", 17), ("dagrange", 17), "dagrangepost"),
+    ":": (15, "rangeall", ("rangepre", 15), ("range", 15), "rangepost"),
     "not": (10, None, ("not", 10), None, None),
     "!": (10, None, ("not", 10), None, None),
     "and": (5, None, None, ("and", 5), None),
     "&": (5, None, None, ("and", 5), None),
-    "%": (5, None, None, ("only", 5), ("onlypost", 5)),
+    "%": (5, None, None, ("only", 5), "onlypost"),
     "or": (4, None, None, ("or", 4), None),
     "|": (4, None, None, ("or", 4), None),
     "+": (4, None, None, ("or", 4), None),
@@ -175,12 +175,12 b' elements = {'
 keywords = set(['and', 'or', 'not'])
 
 # default set of valid characters for the initial letter of symbols
-_syminitletters = set(c for c in [chr(i) for i in xrange(256)]
-                      if c.isalnum() or c in '._@' or ord(c) > 127)
+_syminitletters = set(
+    string.ascii_letters +
+    string.digits + pycompat.sysstr('._@')) | set(map(chr, xrange(128, 256)))
 
 # default set of valid characters for non-initial letters of symbols
-_symletters = set(c for c in  [chr(i) for i in xrange(256)]
-                  if c.isalnum() or c in '-._/@' or ord(c) > 127)
+_symletters = _syminitletters | set(pycompat.sysstr('-/'))
 
 def tokenize(program, lookup=None, syminitletters=None, symletters=None):
     '''
@@ -362,14 +362,22 b' def stringset(repo, subset, x):'
         return baseset([x])
     return baseset()
 
-def rangeset(repo, subset, x, y):
+def rangeset(repo, subset, x, y, order):
     m = getset(repo, fullreposet(repo), x)
     n = getset(repo, fullreposet(repo), y)
 
     if not m or not n:
         return baseset()
-    m, n = m.first(), n.last()
-
+    return _makerangeset(repo, subset, m.first(), n.last(), order)
+
+def rangepre(repo, subset, y, order):
+    # ':y' can't be rewritten to '0:y' since '0' may be hidden
+    n = getset(repo, fullreposet(repo), y)
+    if not n:
+        return baseset()
+    return _makerangeset(repo, subset, 0, n.last(), order)
+
+def _makerangeset(repo, subset, m, n, order):
     if m == n:
         r = baseset([m])
     elif n == node.wdirrev:
@@ -380,35 +388,43 b' def rangeset(repo, subset, x, y):'
         r = spanset(repo, m, n + 1)
     else:
         r = spanset(repo, m, n - 1)
-    # XXX We should combine with subset first: 'subset & baseset(...)'. This is
-    # necessary to ensure we preserve the order in subset.
-    #
-    # This has performance implication, carrying the sorting over when possible
-    # would be more efficient.
-    return r & subset
-
-def dagrange(repo, subset, x, y):
+
+    if order == defineorder:
+        return r & subset
+    else:
+        # carrying the sorting over when possible would be more efficient
+        return subset & r
+
+def dagrange(repo, subset, x, y, order):
     r = fullreposet(repo)
     xs = reachableroots(repo, getset(repo, r, x), getset(repo, r, y),
                          includepath=True)
     return subset & xs
 
-def andset(repo, subset, x, y):
+def andset(repo, subset, x, y, order):
     return getset(repo, getset(repo, subset, x), y)
 
-def differenceset(repo, subset, x, y):
+def differenceset(repo, subset, x, y, order):
     return getset(repo, subset, x) - getset(repo, subset, y)
 
-def orset(repo, subset, *xs):
+def _orsetlist(repo, subset, xs):
     assert xs
     if len(xs) == 1:
         return getset(repo, subset, xs[0])
     p = len(xs) // 2
-    a = orset(repo, subset, *xs[:p])
-    b = orset(repo, subset, *xs[p:])
+    a = _orsetlist(repo, subset, xs[:p])
+    b = _orsetlist(repo, subset, xs[p:])
     return a + b
 
-def notset(repo, subset, x):
+def orset(repo, subset, x, order):
+    xs = getlist(x)
+    if order == followorder:
+        # slow path to take the subset order
+        return subset & _orsetlist(repo, fullreposet(repo), xs)
+    else:
+        return _orsetlist(repo, subset, xs)
+
+def notset(repo, subset, x, order):
     return subset - getset(repo, subset, x)
 
 def listset(repo, subset, *xs):
@@ -418,10 +434,13 b' def listset(repo, subset, *xs):'
 def keyvaluepair(repo, subset, k, v):
     raise error.ParseError(_("can't use a key-value pair in this context"))
 
-def func(repo, subset, a, b):
+def func(repo, subset, a, b, order):
     f = getsymbol(a)
     if f in symbols:
-        return symbols[f](repo, subset, b)
+        fn = symbols[f]
+        if getattr(fn, '_takeorder', False):
+            return fn(repo, subset, b, order)
+        return fn(repo, subset, b)
 
     keep = lambda fn: getattr(fn, '__doc__', None) is not None
 
@@ -515,7 +534,7 b' def _firstancestors(repo, subset, x):'
     # Like ``ancestors(set)`` but follows only the first parents.
     return _ancestors(repo, subset, x, followfirst=True)
 
-def ancestorspec(repo, subset, x, n):
+def ancestorspec(repo, subset, x, n, order):
     """``set~n``
     Changesets that are the Nth ancestor (first parents only) of a changeset
     in set.
@@ -1001,12 +1020,21 b' def first(repo, subset, x):'
     return limit(repo, subset, x)
 
 def _follow(repo, subset, x, name, followfirst=False):
-    l = getargs(x, 0, 1, _("%s takes no arguments or a pattern") % name)
+    l = getargs(x, 0, 2, _("%s takes no arguments or a pattern "
+                           "and an optional revset") % name)
     c = repo['.']
     if l:
         x = getstring(l[0], _("%s expected a pattern") % name)
+        rev = None
+        if len(l) >= 2:
+            revs = getset(repo, fullreposet(repo), l[1])
+            if len(revs) != 1:
+                raise error.RepoLookupError(
+                        _("%s expected one starting revision") % name)
+            rev = revs.last()
+            c = repo[rev]
         matcher = matchmod.match(repo.root, repo.getcwd(), [x],
-                                 ctx=repo[None], default='path')
+                                 ctx=repo[rev], default='path')
 
         files = c.manifest().walk(matcher)
 
@@ -1021,20 +1049,20 b' def _follow(repo, subset, x, name, follo'
 
     return subset & s
 
-@predicate('follow([pattern])', safe=True)
+@predicate('follow([pattern[, startrev]])', safe=True)
 def follow(repo, subset, x):
     """
     An alias for ``::.`` (ancestors of the working directory's first parent).
     If pattern is specified, the histories of files matching given
-    pattern is followed, including copies.
+    pattern in the revision given by startrev are followed, including copies.
     """
     return _follow(repo, subset, x, 'follow')
 
 @predicate('_followfirst', safe=True)
 def _followfirst(repo, subset, x):
-    # ``followfirst([pattern])``
-    # Like ``follow([pattern])`` but follows only the first parent of
-    # every revisions or files revisions.
+    # ``followfirst([pattern[, startrev]])``
+    # Like ``follow([pattern[, startrev]])`` but follows only the first parent
+    # of every revisions or files revisions.
     return _follow(repo, subset, x, '_followfirst', followfirst=True)
 
 @predicate('all()', safe=True)
@@ -1519,6 +1547,9 b' def p2(repo, subset, x):'
     # some optimisations from the fact this is a baseset.
     return subset & ps
 
+def parentpost(repo, subset, x, order):
+    return p1(repo, subset, x)
+
 @predicate('parents([set])', safe=True)
 def parents(repo, subset, x):
     """
@@ -1569,7 +1600,7 b' def secret(repo, subset, x):'
     target = phases.secret
     return _phase(repo, subset, target)
 
-def parentspec(repo, subset, x, n):
+def parentspec(repo, subset, x, n, order):
     """``set^0``
     The set.
     ``set^1`` (or ``set^``), ``set^2``
@@ -1590,7 +1621,7 b' def parentspec(repo, subset, x, n):'
             ps.add(cl.parentrevs(r)[0])
         elif n == 2:
             parents = cl.parentrevs(r)
-            if len(parents) > 1:
+            if parents[1] != node.nullrev:
                 ps.add(parents[1])
     return subset & ps
 
@@ -1813,12 +1844,13 b' def matching(repo, subset, x):'
 
     return subset.filter(matches, condrepr=('<matching%r %r>', fields, revs))
 
-@predicate('reverse(set)', safe=True)
-def reverse(repo, subset, x):
+@predicate('reverse(set)', safe=True, takeorder=True)
+def reverse(repo, subset, x, order):
     """Reverse order of set.
     """
     l = getset(repo, subset, x)
-    l.reverse()
+    if order == defineorder:
+        l.reverse()
     return l
 
 @predicate('roots(set)', safe=True)
@@ -1880,8 +1912,8 b' def _getsortargs(x):'
 
     return args['set'], keyflags, opts
 
-@predicate('sort(set[, [-]key... [, ...]])', safe=True)
-def sort(repo, subset, x):
+@predicate('sort(set[, [-]key... [, ...]])', safe=True, takeorder=True)
+def sort(repo, subset, x, order):
     """Sort set by keys. The default sort order is ascending, specify a key
     as ``-key`` to sort in descending order.
 
@@ -1902,7 +1934,7 b' def sort(repo, subset, x):'
     s, keyflags, opts = _getsortargs(x)
     revs = getset(repo, subset, s)
 
-    if not keyflags:
+    if not keyflags or order != defineorder:
         return revs
     if len(keyflags) == 1 and keyflags[0][0] == "rev":
         revs.sort(reverse=keyflags[0][1])
@@ -2233,9 +2265,7 b' def wdir(repo, subset, x):'
         return baseset([node.wdirrev])
     return baseset()
 
-# for internal use
-@predicate('_list', safe=True)
-def _list(repo, subset, x):
+def _orderedlist(repo, subset, x):
     s = getstring(x, "internal error")
     if not s:
         return baseset()
@@ -2264,8 +2294,15 b' def _list(repo, subset, x):'
     return baseset(ls)
 
 # for internal use
-@predicate('_intlist', safe=True)
-def _intlist(repo, subset, x):
+@predicate('_list', safe=True, takeorder=True)
+def _list(repo, subset, x, order):
+    if order == followorder:
+        # slow path to take the subset order
+        return subset & _orderedlist(repo, fullreposet(repo), x)
+    else:
+        return _orderedlist(repo, subset, x)
+
+def _orderedintlist(repo, subset, x):
     s = getstring(x, "internal error")
     if not s:
         return baseset()
@@ -2274,8 +2311,15 b' def _intlist(repo, subset, x):'
     return baseset([r for r in ls if r in s])
 
 # for internal use
-@predicate('_hexlist', safe=True)
-def _hexlist(repo, subset, x):
+@predicate('_intlist', safe=True, takeorder=True)
+def _intlist(repo, subset, x, order):
+    if order == followorder:
+        # slow path to take the subset order
+        return subset & _orderedintlist(repo, fullreposet(repo), x)
+    else:
+        return _orderedintlist(repo, subset, x)
+
+def _orderedhexlist(repo, subset, x):
     s = getstring(x, "internal error")
     if not s:
         return baseset()
@@ -2284,8 +2328,18 b' def _hexlist(repo, subset, x):'
     s = subset
     return baseset([r for r in ls if r in s])
 
+# for internal use
+@predicate('_hexlist', safe=True, takeorder=True)
+def _hexlist(repo, subset, x, order):
+    if order == followorder:
+        # slow path to take the subset order
+        return subset & _orderedhexlist(repo, fullreposet(repo), x)
+    else:
+        return _orderedhexlist(repo, subset, x)
+
 methods = {
     "range": rangeset,
+    "rangepre": rangepre,
     "dagrange": dagrange,
     "string": stringset,
     "symbol": stringset,
@@ -2298,7 +2352,51 b' methods = {'
     "func": func,
     "ancestor": ancestorspec,
     "parent": parentspec,
-    "parentpost": p1,
+    "parentpost": parentpost,
+}
+
+# Constants for ordering requirement, used in _analyze():
+#
+# If 'define', any nested functions and operations can change the ordering of
+# the entries in the set. If 'follow', any nested functions and operations
+# should take the ordering specified by the first operand to the '&' operator.
+#
+# For instance,
+#
+#   X & (Y | Z)
+#   ^   ^^^^^^^
+#   |   follow
+#   define
+#
+# will be evaluated as 'or(y(x()), z(x()))', where 'x()' can change the order
+# of the entries in the set, but 'y()', 'z()' and 'or()' shouldn't.
+#
+# 'any' means the order doesn't matter. For instance,
+#
+#   X & !Y
+#        ^
+#        any
+#
+# 'y()' can either enforce its ordering requirement or take the ordering
+# specified by 'x()' because 'not()' doesn't care the order.
+#
+# Transition of ordering requirement:
+#
+# 1. starts with 'define'
+# 2. shifts to 'follow' by 'x & y'
+# 3. changes back to 'define' on function call 'f(x)' or function-like
+#    operation 'x (f) y' because 'f' may have its own ordering requirement
+#    for 'x' and 'y' (e.g. 'first(x)')
+#
+anyorder = 'any'        # don't care the order
+defineorder = 'define'  # should define the order
+followorder = 'follow'  # must follow the current order
+
+# transition table for 'x & y', from the current expression 'x' to 'y'
+_tofolloworder = {
+    anyorder: anyorder,
+    defineorder: followorder,
+    followorder: followorder,
 }
 
 def _matchonly(revs, bases):
@@ -2316,6 +2414,97 b' def _matchonly(revs, bases):'
         and getsymbol(bases[1][1]) == 'ancestors'):
         return ('list', revs[2], bases[1][2])
 
+def _fixops(x):
+    """Rewrite raw parsed tree to resolve ambiguous syntax which cannot be
+    handled well by our simple top-down parser"""
+    if not isinstance(x, tuple):
+        return x
+
+    op = x[0]
+    if op == 'parent':
+        # x^:y means (x^) : y, not x ^ (:y)
+        # x^:  means (x^) :,   not x ^ (:)
+        post = ('parentpost', x[1])
+        if x[2][0] == 'dagrangepre':
+            return _fixops(('dagrange', post, x[2][1]))
+        elif x[2][0] == 'rangepre':
+            return _fixops(('range', post, x[2][1]))
+        elif x[2][0] == 'rangeall':
+            return _fixops(('rangepost', post))
+    elif op == 'or':
+        # make number of arguments deterministic:
+        # x + y + z -> (or x y z) -> (or (list x y z))
+        return (op, _fixops(('list',) + x[1:]))
+
+    return (op,) + tuple(_fixops(y) for y in x[1:])
+
+def _analyze(x, order):
+    if x is None:
+        return x
+
+    op = x[0]
+    if op == 'minus':
+        return _analyze(('and', x[1], ('not', x[2])), order)
+    elif op == 'only':
+        t = ('func', ('symbol', 'only'), ('list', x[1], x[2]))
+        return _analyze(t, order)
+    elif op == 'onlypost':
+        return _analyze(('func', ('symbol', 'only'), x[1]), order)
+    elif op == 'dagrangepre':
+        return _analyze(('func', ('symbol', 'ancestors'), x[1]), order)
+    elif op == 'dagrangepost':
+        return _analyze(('func', ('symbol', 'descendants'), x[1]), order)
+    elif op == 'rangeall':
+        return _analyze(('rangepre', ('string', 'tip')), order)
+    elif op == 'rangepost':
+        return _analyze(('range', x[1], ('string', 'tip')), order)
+    elif op == 'negate':
+        s = getstring(x[1], _("can't negate that"))
+        return _analyze(('string', '-' + s), order)
+    elif op in ('string', 'symbol'):
+        return x
+    elif op == 'and':
+        ta = _analyze(x[1], order)
+        tb = _analyze(x[2], _tofolloworder[order])
+        return (op, ta, tb, order)
+    elif op == 'or':
+        return (op, _analyze(x[1], order), order)
+    elif op == 'not':
+        return (op, _analyze(x[1], anyorder), order)
+    elif op in ('rangepre', 'parentpost'):
+        return (op, _analyze(x[1], defineorder), order)
+    elif op == 'group':
+        return _analyze(x[1], order)
+    elif op in ('dagrange', 'range', 'parent', 'ancestor'):
+        ta = _analyze(x[1], defineorder)
+        tb = _analyze(x[2], defineorder)
+        return (op, ta, tb, order)
+    elif op == 'list':
+        return (op,) + tuple(_analyze(y, order) for y in x[1:])
+    elif op == 'keyvalue':
+        return (op, x[1], _analyze(x[2], order))
+    elif op == 'func':
+        f = getsymbol(x[1])
+        d = defineorder
+        if f == 'present':
+            # 'present(set)' is known to return the argument set with no
+            # modification, so forward the current order to its argument
+            d = order
+        return (op, x[1], _analyze(x[2], d), order)
+    raise ValueError('invalid operator %r' % op)
+
+def analyze(x, order=defineorder):
+    """Transform raw parsed tree to evaluatable tree which can be fed to
+    optimize() or getset()
+
+    All pseudo operations should be mapped to real operations or functions
+    defined in methods or symbols table respectively.
+
+    'order' specifies how the current expression 'x' is ordered (see the
+    constants defined above.)
+    """
+    return _analyze(x, order)
+
 def _optimize(x, small):
     if x is None:
         return 0, x
@@ -2325,47 +2514,29 b' def _optimize(x, small):'
         smallbonus = .5
 
     op = x[0]
-    if op == 'minus':
-        return _optimize(('and', x[1], ('not', x[2])), small)
-    elif op == 'only':
-        t = ('func', ('symbol', 'only'), ('list', x[1], x[2]))
-        return _optimize(t, small)
-    elif op == 'onlypost':
-        return _optimize(('func', ('symbol', 'only'), x[1]), small)
-    elif op == 'dagrangepre':
-        return _optimize(('func', ('symbol', 'ancestors'), x[1]), small)
-    elif op == 'dagrangepost':
-        return _optimize(('func', ('symbol', 'descendants'), x[1]), small)
-    elif op == 'rangeall':
-        return _optimize(('range', ('string', '0'), ('string', 'tip')), small)
-    elif op == 'rangepre':
-        return _optimize(('range', ('string', '0'), x[1]), small)
-    elif op == 'rangepost':
-        return _optimize(('range', x[1], ('string', 'tip')), small)
-    elif op == 'negate':
-        s = getstring(x[1], _("can't negate that"))
-        return _optimize(('string', '-' + s), small)
-    elif op in 'string symbol negate':
+    if op in ('string', 'symbol'):
         return smallbonus, x # single revisions are small
     elif op == 'and':
         wa, ta = _optimize(x[1], True)
         wb, tb = _optimize(x[2], True)
+        order = x[3]
         w = min(wa, wb)
 
         # (::x and not ::y)/(not ::y and ::x) have a fast path
         tm = _matchonly(ta, tb) or _matchonly(tb, ta)
         if tm:
-            return w, ('func', ('symbol', 'only'), tm)
+            return w, ('func', ('symbol', 'only'), tm, order)
 
         if tb is not None and tb[0] == 'not':
-            return wa, ('difference', ta, tb[1])
+            return wa, ('difference', ta, tb[1], order)
 
         if wa > wb:
-            return w, (op, tb, ta)
-        return w, (op, ta, tb)
+            return w, (op, tb, ta, order)
+        return w, (op, ta, tb, order)
     elif op == 'or':
         # fast path for machine-generated expression, that is likely to have
         # lots of trivial revisions: 'a + b + c()' to '_list(a b) + c()'
+        order = x[2]
         ws, ts, ss = [], [], []
         def flushss():
             if not ss:
@@ -2374,12 +2545,12 b' def _optimize(x, small):'
                 w, t = ss[0]
             else:
                 s = '\0'.join(t[1] for w, t in ss)
-                y = ('func', ('symbol', '_list'), ('string', s))
+                y = ('func', ('symbol', '_list'), ('string', s), order)
                 w, t = _optimize(y, False)
             ws.append(w)
             ts.append(t)
             del ss[:]
-        for y in x[1:]:
+        for y in getlist(x[1]):
             w, t = _optimize(y, False)
             if t is not None and (t[0] == 'string' or t[0] == 'symbol'):
                 ss.append((w, t))
@@ -2393,33 +2564,27 b' def _optimize(x, small):'
         # we can't reorder trees by weight because it would change the order.
         # ("sort(a + b)" == "sort(b + a)", but "a + b" != "b + a")
         #   ts = tuple(t for w, t in sorted(zip(ws, ts), key=lambda wt: wt[0]))
-        return max(ws), (op,) + tuple(ts)
+        return max(ws), (op, ('list',) + tuple(ts), order)
     elif op == 'not':
         # Optimize not public() to _notpublic() because we have a fast version
-        if x[1] == ('func', ('symbol', 'public'), None):
-            newsym = ('func', ('symbol', '_notpublic'), None)
+        if x[1][:3] == ('func', ('symbol', 'public'), None):
+            order = x[1][3]
+            newsym = ('func', ('symbol', '_notpublic'), None, order)
             o = _optimize(newsym, not small)
             return o[0], o[1]
         else:
             o = _optimize(x[1], not small)
-            return o[0], (op, o[1])
-    elif op == 'parentpost':
+            order = x[2]
+            return o[0], (op, o[1], order)
+    elif op in ('rangepre', 'parentpost'):
         o = _optimize(x[1], small)
-        return o[0], (op, o[1])
-    elif op == 'group':
-        return _optimize(x[1], small)
-    elif op in 'dagrange range parent ancestorspec':
-        if op == 'parent':
-            # x^:y means (x^) : y, not x ^ (:y)
-            post = ('parentpost', x[1])
-            if x[2][0] == 'dagrangepre':
-                return _optimize(('dagrange', post, x[2][1]), small)
-            elif x[2][0] == 'rangepre':
-                return _optimize(('range', post, x[2][1]), small)
-
+        order = x[2]
+        return o[0], (op, o[1], order)
+    elif op in ('dagrange', 'range', 'parent', 'ancestor'):
         wa, ta = _optimize(x[1], small)
         wb, tb = _optimize(x[2], small)
-        return wa + wb, (op, ta, tb)
+        order = x[3]
+        return wa + wb, (op, ta, tb, order)
     elif op == 'list':
         ws, ts = zip(*(_optimize(y, small) for y in x[1:]))
         return sum(ws), (op,) + ts
@@ -2429,32 +2594,36 b' def _optimize(x, small):'
     elif op == 'func':
         f = getsymbol(x[1])
         wa, ta = _optimize(x[2], small)
-        if f in ("author branch closed date desc file grep keyword "
-                 "outgoing user"):
+        if f in ('author', 'branch', 'closed', 'date', 'desc', 'file', 'grep',
+                 'keyword', 'outgoing', 'user', 'destination'):
             w = 10 # slow
-        elif f in "modifies adds removes":
+        elif f in ('modifies', 'adds', 'removes'):
             w = 30 # slower
         elif f == "contains":
             w = 100 # very slow
         elif f == "ancestor":
             w = 1 * smallbonus
-        elif f in "reverse limit first _intlist":
+        elif f in ('reverse', 'limit', 'first', '_intlist'):
             w = 0
-        elif f in "sort":
+        elif f == "sort":
             w = 10 # assume most sorts look at changelog
         else:
             w = 1
-        return w + wa, (op, x[1], ta)
-    return 1, x
+        order = x[3]
+        return w + wa, (op, x[1], ta, order)
+    raise ValueError('invalid operator %r' % op)
 
 def optimize(tree):
+    """Optimize evaluatable tree
+
+    All pseudo operations should be transformed beforehand.
+    """
     _weight, newtree = _optimize(tree, small=True)
     return newtree
 
 # the set of valid characters for the initial letter of symbols in
 # alias declarations and definitions
-_aliassyminitletters = set(c for c in [chr(i) for i in xrange(256)]
-                           if c.isalnum() or c in '._@$' or ord(c) > 127)
+_aliassyminitletters = _syminitletters | set(pycompat.sysstr('$'))
 
 def _parsewith(spec, lookup=None, syminitletters=None):
     """Generate a parse tree of given spec with given tokenizing options
@@ -2475,7 +2644,7 b' def _parsewith(spec, lookup=None, symini'
                                  syminitletters=syminitletters))
     if pos != len(spec):
         raise error.ParseError(_('invalid token'), pos)
-    return parser.simplifyinfixops(tree, ('list', 'or'))
+    return _fixops(parser.simplifyinfixops(tree, ('list', 'or')))
 
 class _aliasrules(parser.basealiasrules):
     """Parsing and expansion rule set of revset aliases"""
@@ -2496,15 +2665,14 b' class _aliasrules(parser.basealiasrules)'
         if tree[0] == 'func' and tree[1][0] == 'symbol':
             return tree[1][1], getlist(tree[2])
 
-def expandaliases(ui, tree, showwarning=None):
+def expandaliases(ui, tree):
     aliases = _aliasrules.buildmap(ui.configitems('revsetalias'))
     tree = _aliasrules.expand(aliases, tree)
-    if showwarning:
-        # warn about problematic (but not referred) aliases
-        for name, alias in sorted(aliases.iteritems()):
-            if alias.error and not alias.warned:
-                showwarning(_('warning: %s\n') % (alias.error))
-                alias.warned = True
+    # warn about problematic (but not referred) aliases
+    for name, alias in sorted(aliases.iteritems()):
+        if alias.error and not alias.warned:
+            ui.warn(_('warning: %s\n') % (alias.error))
+            alias.warned = True
     return tree
 
 def foldconcat(tree):
@@ -2535,13 +2703,21 b' def posttreebuilthook(tree, repo):'
     # hook for extensions to execute code on the optimized tree
     pass
 
-def match(ui, spec, repo=None):
-    """Create a matcher for a single revision spec."""
-    return matchany(ui, [spec], repo=repo)
-
-def matchany(ui, specs, repo=None):
+def match(ui, spec, repo=None, order=defineorder):
+    """Create a matcher for a single revision spec
+
+    If order=followorder, a matcher takes the ordering specified by the input
+    set.
+    """
+    return matchany(ui, [spec], repo=repo, order=order)
+
+def matchany(ui, specs, repo=None, order=defineorder):
     """Create a matcher that will include any revisions matching one of the
-    given specs"""
+    given specs
+
+    If order=followorder, a matcher takes the ordering specified by the input
+    set.
+    """
     if not specs:
         def mfunc(repo, subset=None):
             return baseset()
@@ -2554,15 +2730,18 b' def matchany(ui, specs, repo=None):'
     if len(specs) == 1:
         tree = parse(specs[0], lookup)
     else:
-        tree = ('or',) + tuple(parse(s, lookup) for s in specs)
-    return _makematcher(ui, tree, repo)
-
-def _makematcher(ui, tree, repo):
+        tree = ('or', ('list',) + tuple(parse(s, lookup) for s in specs))
+
     if ui:
-        tree = expandaliases(ui, tree, showwarning=ui.warn)
+        tree = expandaliases(ui, tree)
     tree = foldconcat(tree)
+    tree = analyze(tree, order)
     tree = optimize(tree)
     posttreebuilthook(tree, repo)
+    return makematcher(tree)
+
+def makematcher(tree):
+    """Create a matcher from an evaluatable tree"""
     def mfunc(repo, subset=None):
         if subset is None:
             subset = fullreposet(repo)