wix: functionality to automate building WiX installers

Like we did for Inno Setup, we want to make it easier to
produce WiX installers. This commit does that.

We introduce a new hgpackaging.wix module for performing
all the high-level tasks required to produce WiX installers.
This required miscellaneous enhancements to existing code in
hgpackaging, including support for signing binaries.

A new build.py script for calling into the module APIs has been
created. It behaves very similarly to the Inno Setup build.py
script.

Unlike Inno Setup, we didn't have code in the repo previously
to generate WiX installers. It appears that all existing
automation for building WiX installers lives in the
https://bitbucket.org/tortoisehg/thg-winbuild repository - most
notably in its setup.py file. My strategy for inventing the
code in this commit was to step through the code in that repo's
setup.py and observe what it was doing. Despite the length of
setup.py in that repository, the actual amount of steps required
to produce a WiX installer is actually quite low. It consists
of a basic py2exe build plus invocations of candle.exe and
light.exe to produce the MSI.

One rabbit hole that gave me fits was locating the Visual Studio
9 C Runtime merge modules. These merge modules are only present
on your system if you have a full Visual Studio 2008 installation.
Fortunately, I have a copy of Visual Studio 2008 and was able
to install all the required updates. I then uploaded these merge
modules to a personal repository on GitHub. That is where the
added code references them from. We probably don't need to
ship the merge modules. But that is for another day.

The installs from the MSIs produced with the new automation
differ from the last official MSI in the following ways:

* Our HTML manual pages have UNIX line endings instead of Windows.
* We ship modules in the mercurial.pure package. It appears the
  upstream packaging code is not including this package due to
  omission (they supply an explicit list of packages that has
  drifted out of sync with our setup.py).
* We do not ship various distutils.* modules. This is because
  virtualenvs have a custom distutils/__init__.py that automagically
  imports distutils from its original location and py2exe gets
  confused by this. We don't use distutils in core Mercurial and
  don't provide a usable python.exe, so this omission should be
  acceptable.
* The version of the enum package is different and we ship
  an enum.pyc instead of an enum/__init__.py.
* The version of the docutils package is different and we
  ship a different set of files.
* The version of Sphinx is drastically newer and we ship a
  number of files the old version did not. (I'm not sure why
  we ship Sphinx - I think it is a side-effect of the way the
  THG code was installing dependencies.)
* We ship the idna package (dependent of requests which is a
  dependency of newer versions of Sphinx).
* The version of imagesize is different and we ship an
  imagesize.pyc instead of an imagesize/__init__.pyc.
* The version of the jinja2 package is different and the sets
  of files differs.
* We ship the packaging package, which is a dependency for Sphinx.
* The version of the pygments package is different and the sets
  of files differs.
* We ship the requests package, which is a dependency for Sphinx.
* We ship the snowballstemmer package, which is a dependency for
  Sphinx.
* We ship the urllib3 package, which is a dependency for requests,
  which is a dependency for Sphinx.
* We ship a newer version of the futures package, which includes a
  handful of extra modules that match Python 3 module names.

# no-check-commit because foo_bar naming

Differential Revision: https://phab.mercurial-scm.org/D6097

contrib/packaging/hgpackaging/wix.py

@@ -0,0 +1,248 b''
+# wix.py - WiX installer functionality
+#
+# Copyright 2019 Gregory Szorc <[e-mail]>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+# no-check-code because Python 3 native.
+
+import os
+import pathlib
+import re
+import subprocess
+
+from .downloads import (
+    download_entry,
+)
+from .py2exe import (
+    build_py2exe,
+)
+from .util import (
+    extract_zip_to_directory,
+    sign_with_signtool,
+)
+
+
+SUPPORT_WXS = [
+    ('contrib.wxs', r'contrib'),
+    ('dist.wxs', r'dist'),
+    ('doc.wxs', r'doc'),
+    ('help.wxs', r'mercurial\help'),
+    ('i18n.wxs', r'i18n'),
+    ('locale.wxs', r'mercurial\locale'),
+    ('templates.wxs', r'mercurial\templates'),
+]
+
+
+EXTRA_PACKAGES = {
+    'distutils',
+    'enum',
+    'imagesize',
+    'pygments',
+    'sphinx',
+}
+
+
+EXCLUDES = {
+    # Python 3 only.
+    'jinja2.asyncsupport',
+}
+
+
+def find_version(source_dir: pathlib.Path):
+    version_py = source_dir / 'mercurial' / '__version__.py'
+
+    with version_py.open('r', encoding='utf-8') as fh:
+        source = fh.read().strip()
+
+    m = re.search('version = b"(.*)"', source)
+    return m.group(1)
+
+
+def normalize_version(version):
+    """Normalize Mercurial version string so WiX accepts it.
+
+    Version strings have to be numeric X.Y.Z.
+    """
+
+    if '+' in version:
+        version, extra = version.split('+', 1)
+    else:
+        extra = None
+
+    # 4.9rc0
+    if version[:-1].endswith('rc'):
+        version = version[:-3]
+
+    versions = [int(v) for v in version.split('.')]
+    while len(versions) < 3:
+        versions.append(0)
+
+    major, minor, build = versions[:3]
+
+    if extra:
+        # <commit count>-<hash>+<date>
+        build = int(extra.split('-')[0])
+
+    return '.'.join('%d' % x for x in (major, minor, build))
+
+
+def ensure_vc90_merge_modules(build_dir):
+    x86 = (
+        download_entry('vc9-crt-x86-msm', build_dir,
+                       local_name='microsoft.vcxx.crt.x86_msm.msm')[0],
+        download_entry('vc9-crt-x86-msm-policy', build_dir,
+                       local_name='policy.x.xx.microsoft.vcxx.crt.x86_msm.msm')[0]
+    )
+
+    x64 = (
+        download_entry('vc9-crt-x64-msm', build_dir,
+                       local_name='microsoft.vcxx.crt.x64_msm.msm')[0],
+        download_entry('vc9-crt-x64-msm-policy', build_dir,
+                       local_name='policy.x.xx.microsoft.vcxx.crt.x64_msm.msm')[0]
+    )
+    return {
+        'x86': x86,
+        'x64': x64,
+    }
+
+
+def run_candle(wix, cwd, wxs, source_dir, defines=None):
+    args = [
+        str(wix / 'candle.exe'),
+        '-nologo',
+        str(wxs),
+        '-dSourceDir=%s' % source_dir,
+    ]
+
+    if defines:
+        args.extend('-d%s=%s' % define for define in sorted(defines.items()))
+
+    subprocess.run(args, cwd=str(cwd), check=True)
+
+
+def make_post_build_signing_fn(name, subject_name=None, cert_path=None,
+                               cert_password=None, timestamp_url=None):
+    """Create a callable that will use signtool to sign hg.exe."""
+
+    def post_build_sign(source_dir, build_dir, dist_dir, version):
+        description = '%s %s' % (name, version)
+
+        sign_with_signtool(dist_dir / 'hg.exe', description,
+                           subject_name=subject_name, cert_path=cert_path,
+                           cert_password=cert_password,
+                           timestamp_url=timestamp_url)
+
+    return post_build_sign
+
+
+def build_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
+                    msi_name='mercurial', version=None, post_build_fn=None):
+    """Build a WiX MSI installer.
+
+    ``source_dir`` is the path to the Mercurial source tree to use.
+    ``arch`` is the target architecture. either ``x86`` or ``x64``.
+    ``python_exe`` is the path to the Python executable to use/bundle.
+    ``version`` is the Mercurial version string. If not defined,
+    ``mercurial/__version__.py`` will be consulted.
+    ``post_build_fn`` is a callable that will be called after building
+    Mercurial but before invoking WiX. It can be used to e.g. facilitate
+    signing. It is passed the paths to the Mercurial source, build, and
+    dist directories and the resolved Mercurial version.
+    """
+    arch = 'x64' if r'\x64' in os.environ.get('LIB', '') else 'x86'
+
+    hg_build_dir = source_dir / 'build'
+    dist_dir = source_dir / 'dist'
+
+    requirements_txt = (source_dir / 'contrib' / 'packaging' /
+                        'wix' / 'requirements.txt')
+
+    build_py2exe(source_dir, hg_build_dir,
+                 python_exe, 'wix', requirements_txt,
+                 extra_packages=EXTRA_PACKAGES, extra_excludes=EXCLUDES)
+
+    version = version or normalize_version(find_version(source_dir))
+    print('using version string: %s' % version)
+
+    if post_build_fn:
+        post_build_fn(source_dir, hg_build_dir, dist_dir, version)
+
+    build_dir = hg_build_dir / ('wix-%s' % arch)
+
+    build_dir.mkdir(exist_ok=True)
+
+    wix_pkg, wix_entry = download_entry('wix', hg_build_dir)
+    wix_path = hg_build_dir / ('wix-%s' % wix_entry['version'])
+
+    if not wix_path.exists():
+        extract_zip_to_directory(wix_pkg, wix_path)
+
+    ensure_vc90_merge_modules(hg_build_dir)
+
+    source_build_rel = pathlib.Path(os.path.relpath(source_dir, build_dir))
+
+    defines = {'Platform': arch}
+
+    for wxs, rel_path in SUPPORT_WXS:
+        wxs = source_dir / 'contrib' / 'packaging' / 'wix' / wxs
+        wxs_source_dir = source_dir / rel_path
+        run_candle(wix_path, build_dir, wxs, wxs_source_dir, defines=defines)
+
+    source = source_dir / 'contrib' / 'packaging' / 'wix' / 'mercurial.wxs'
+    defines['Version'] = version
+    defines['Comments'] = 'Installs Mercurial version %s' % version
+    defines['VCRedistSrcDir'] = str(hg_build_dir)
+
+    run_candle(wix_path, build_dir, source, source_build_rel, defines=defines)
+
+    msi_path = source_dir / 'dist' / (
+        '%s-%s-%s.msi' % (msi_name, version, arch))
+
+    args = [
+        str(wix_path / 'light.exe'),
+        '-nologo',
+        '-ext', 'WixUIExtension',
+        '-sw1076',
+        '-spdb',
+        '-o', str(msi_path),
+    ]
+
+    for source, rel_path in SUPPORT_WXS:
+        assert source.endswith('.wxs')
+        args.append(str(build_dir / ('%s.wixobj' % source[:-4])))
+
+    args.append(str(build_dir / 'mercurial.wixobj'))
+
+    subprocess.run(args, cwd=str(source_dir), check=True)
+
+    print('%s created' % msi_path)
+
+    return {
+        'msi_path': msi_path,
+    }
+
+
+def build_signed_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
+                           name: str, version=None, subject_name=None,
+                           cert_path=None, cert_password=None,
+                           timestamp_url=None):
+    """Build an installer with signed executables."""
+
+    post_build_fn = make_post_build_signing_fn(
+        name,
+        subject_name=subject_name,
+        cert_path=cert_path,
+        cert_password=cert_password,
+        timestamp_url=timestamp_url)
+
+    info = build_installer(source_dir, python_exe=python_exe,
+                           msi_name=name.lower(), version=version,
+                           post_build_fn=post_build_fn)
+
+    description = '%s %s' % (name, version)
+
+    sign_with_signtool(info['msi_path'], description,
+                       subject_name=subject_name, cert_path=cert_path,
+                       cert_password=cert_password, timestamp_url=timestamp_url)

contrib/packaging/wix/build.py

@@ -0,0 +1,65 b''
+#!/usr/bin/env python3
+# Copyright 2019 Gregory Szorc <[e-mail]>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+# no-check-code because Python 3 native.
+
+"""Code to build Mercurial WiX installer."""
+
+import argparse
+import os
+import pathlib
+import sys
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument('--name',
+                        help='Application name',
+                        default='Mercurial')
+    parser.add_argument('--python',
+                        help='Path to Python executable to use',
+                        required=True)
+    parser.add_argument('--sign-sn',
+                        help='Subject name (or fragment thereof) of certificate '
+                             'to use for signing')
+    parser.add_argument('--sign-cert',
+                        help='Path to certificate to use for signing')
+    parser.add_argument('--sign-password',
+                        help='Password for signing certificate')
+    parser.add_argument('--sign-timestamp-url',
+                        help='URL of timestamp server to use for signing')
+    parser.add_argument('--version',
+                        help='Version string to use')
+
+    args = parser.parse_args()
+
+    here = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
+    source_dir = here.parent.parent.parent
+
+    sys.path.insert(0, str(source_dir / 'contrib' / 'packaging'))
+
+    from hgpackaging.wix import (
+        build_installer,
+        build_signed_installer,
+    )
+
+    fn = build_installer
+    kwargs = {
+        'source_dir': source_dir,
+        'python_exe': pathlib.Path(args.python),
+        'version': args.version,
+    }
+
+    if args.sign_sn or args.sign_cert:
+        fn = build_signed_installer
+        kwargs['name'] = args.name
+        kwargs['subject_name'] = args.sign_sn
+        kwargs['cert_path'] = args.sign_cert
+        kwargs['cert_password'] = args.sign_password
+        kwargs['timestamp_url'] = args.sign_timestamp_url
+
+    fn(**kwargs)

contrib/packaging/wix/requirements.txt

@@ -0,0 +1,132 b''
+#
+# This file is autogenerated by pip-compile
+# To update, run:
+#
+#    pip-compile --generate-hashes contrib/packaging/wix/requirements.txt.in -o contrib/packaging/wix/requirements.txt -U
+#
+alabaster==0.7.12 \
+    --hash=sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359 \
+    --hash=sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02 \
+    # via sphinx
+babel==2.6.0 \
+    --hash=sha256:6778d85147d5d85345c14a26aada5e478ab04e39b078b0745ee6870c2b5cf669 \
+    --hash=sha256:8cba50f48c529ca3fa18cf81fa9403be176d374ac4d60738b839122dfaaa3d23 \
+    # via sphinx
+certifi==2018.11.29 \
+    --hash=sha256:47f9c83ef4c0c621eaef743f133f09fa8a74a9b75f037e8624f83bd1b6626cb7 \
+    --hash=sha256:993f830721089fef441cdfeb4b2c8c9df86f0c63239f06bd025a76a7daddb033 \
+    # via requests
+chardet==3.0.4 \
+    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
+    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
+    # via requests
+colorama==0.4.1 \
+    --hash=sha256:05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d \
+    --hash=sha256:f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48 \
+    # via sphinx
+docutils==0.14 \
+    --hash=sha256:02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6 \
+    --hash=sha256:51e64ef2ebfb29cae1faa133b3710143496eca21c530f3f71424d77687764274 \
+    --hash=sha256:7a4bd47eaf6596e1295ecb11361139febe29b084a87bf005bf899f9a42edc3c6
+enum==0.4.7 \
+    --hash=sha256:8c7cf3587eda51008bcc1eed99ea2c331ccd265c231dbaa95ec5258d3dc03100
+future==0.17.1 \
+    --hash=sha256:67045236dcfd6816dc439556d009594abf643e5eb48992e36beac09c2ca659b8
+idna==2.8 \
+    --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \
+    --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c \
+    # via requests
+imagesize==1.1.0 \
+    --hash=sha256:3f349de3eb99145973fefb7dbe38554414e5c30abd0c8e4b970a7c9d09f3a1d8 \
+    --hash=sha256:f3832918bc3c66617f92e35f5d70729187676313caa60c187eb0f28b8fe5e3b5 \
+    # via sphinx
+jinja2==2.10 \
+    --hash=sha256:74c935a1b8bb9a3947c50a54766a969d4846290e1e788ea44c1392163723c3bd \
+    --hash=sha256:f84be1bb0040caca4cea721fcbbbbd61f9be9464ca236387158b0feea01914a4 \
+    # via sphinx
+markupsafe==1.1.1 \
+    --hash=sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473 \
+    --hash=sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161 \
+    --hash=sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235 \
+    --hash=sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5 \
+    --hash=sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff \
+    --hash=sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b \
+    --hash=sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1 \
+    --hash=sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e \
+    --hash=sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183 \
+    --hash=sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66 \
+    --hash=sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1 \
+    --hash=sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1 \
+    --hash=sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e \
+    --hash=sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b \
+    --hash=sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905 \
+    --hash=sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735 \
+    --hash=sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d \
+    --hash=sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e \
+    --hash=sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d \
+    --hash=sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c \
+    --hash=sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21 \
+    --hash=sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2 \
+    --hash=sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5 \
+    --hash=sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b \
+    --hash=sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6 \
+    --hash=sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f \
+    --hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
+    --hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
+    # via jinja2
+packaging==19.0 \
+    --hash=sha256:0c98a5d0be38ed775798ece1b9727178c4469d9c3b4ada66e8e6b7849f8732af \
+    --hash=sha256:9e1cbf8c12b1f1ce0bb5344b8d7ecf66a6f8a6e91bcb0c84593ed6d3ab5c4ab3 \
+    # via sphinx
+pygments==2.3.1 \
+    --hash=sha256:5ffada19f6203563680669ee7f53b64dabbeb100eb51b61996085e99c03b284a \
+    --hash=sha256:e8218dd399a61674745138520d0d4cf2621d7e032439341bc3f647bff125818d
+pyparsing==2.3.1 \
+    --hash=sha256:66c9268862641abcac4a96ba74506e594c884e3f57690a696d21ad8210ed667a \
+    --hash=sha256:f6c5ef0d7480ad048c054c37632c67fca55299990fff127850181659eea33fc3 \
+    # via packaging
+pypiwin32==223 \
+    --hash=sha256:67adf399debc1d5d14dffc1ab5acacb800da569754fafdc576b2a039485aa775 \
+    --hash=sha256:71be40c1fbd28594214ecaecb58e7aa8b708eabfa0125c8a109ebd51edbd776a
+pytz==2018.9 \
+    --hash=sha256:32b0891edff07e28efe91284ed9c31e123d84bea3fd98e1f72be2508f43ef8d9 \
+    --hash=sha256:d5f05e487007e29e03409f9398d074e158d920d36eb82eaf66fb1136b0c5374c \
+    # via babel
+pywin32==224 \
+    --hash=sha256:22e218832a54ed206452c8f3ca9eff07ef327f8e597569a4c2828be5eaa09a77 \
+    --hash=sha256:32b37abafbfeddb0fe718008d6aada5a71efa2874f068bee1f9e703983dcc49a \
+    --hash=sha256:35451edb44162d2f603b5b18bd427bc88fcbc74849eaa7a7e7cfe0f507e5c0c8 \
+    --hash=sha256:4eda2e1e50faa706ff8226195b84fbcbd542b08c842a9b15e303589f85bfb41c \
+    --hash=sha256:5f265d72588806e134c8e1ede8561739071626ea4cc25c12d526aa7b82416ae5 \
+    --hash=sha256:6852ceac5fdd7a146b570655c37d9eacd520ed1eaeec051ff41c6fc94243d8bf \
+    --hash=sha256:6dbc4219fe45ece6a0cc6baafe0105604fdee551b5e876dc475d3955b77190ec \
+    --hash=sha256:9bd07746ce7f2198021a9fa187fa80df7b221ec5e4c234ab6f00ea355a3baf99 \
+    # via pypiwin32
+requests==2.21.0 \
+    --hash=sha256:502a824f31acdacb3a35b6690b5fbf0bc41d63a24a45c4004352b0242707598e \
+    --hash=sha256:7bf2a778576d825600030a110f3c0e3e8edc51dfaafe1c146e39a2027784957b \
+    # via sphinx
+six==1.12.0 \
+    --hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
+    --hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73 \
+    # via packaging, sphinx
+snowballstemmer==1.2.1 \
+    --hash=sha256:919f26a68b2c17a7634da993d91339e288964f93c274f1343e3bbbe2096e1128 \
+    --hash=sha256:9f3bcd3c401c3e862ec0ebe6d2c069ebc012ce142cce209c098ccb5b09136e89 \
+    # via sphinx
+sphinx==1.8.4 \
+    --hash=sha256:b53904fa7cb4b06a39409a492b949193a1b68cc7241a1a8ce9974f86f0d24287 \
+    --hash=sha256:c1c00fc4f6e8b101a0d037065043460dffc2d507257f2f11acaed71fd2b0c83c
+sphinxcontrib-websupport==1.1.0 \
+    --hash=sha256:68ca7ff70785cbe1e7bccc71a48b5b6d965d79ca50629606c7861a21b206d9dd \
+    --hash=sha256:9de47f375baf1ea07cdb3436ff39d7a9c76042c10a769c52353ec46e4e8fc3b9 \
+    # via sphinx
+typing==3.6.6 \
+    --hash=sha256:4027c5f6127a6267a435201981ba156de91ad0d1d98e9ddc2aa173453453492d \
+    --hash=sha256:57dcf675a99b74d64dacf6fba08fb17cf7e3d5fdff53d4a30ea2a5e7e52543d4 \
+    --hash=sha256:a4c8473ce11a65999c8f59cb093e70686b6c84c98df58c1dae9b3b196089858a \
+    # via sphinx
+urllib3==1.24.1 \
+    --hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \
+    --hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 \
+    # via requests

contrib/packaging/wix/requirements.txt.in

@@ -0,0 +1,6 b''
+docutils
+enum
+future
+pygments
+pypiwin32
+sphinx

contrib/packaging/hgpackaging/downloads.py

@@ -1,145 +1,175 b''
 # downloads.py - Code for downloading dependencies.
 #
 # Copyright 2019 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 # no-check-code because Python 3 native.
 
 import gzip
 import hashlib
 import pathlib
 import urllib.request
 
 
 DOWNLOADS = {
     'gettext': {
         'url': 'https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-bin.zip',
         'size': 1606131,
         'sha256': '60b9ef26bc5cceef036f0424e542106cf158352b2677f43a01affd6d82a1d641',
         'version': '0.14.4',
     },
     'gettext-dep': {
         'url': 'https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-dep.zip',
         'size': 715086,
         'sha256': '411f94974492fd2ecf52590cb05b1023530aec67e64154a88b1e4ebcd9c28588',
     },
     'py2exe': {
         'url': 'https://versaweb.dl.sourceforge.net/project/py2exe/py2exe/0.6.9/py2exe-0.6.9.zip',
         'size': 149687,
         'sha256': '6bd383312e7d33eef2e43a5f236f9445e4f3e0f6b16333c6f183ed445c44ddbd',
         'version': '0.6.9',
     },
+    # The VC9 CRT merge modules aren't readily available on most systems because
+    # they are only installed as part of a full Visual Studio 2008 install.
+    # While we could potentially extract them from a Visual Studio 2008
+    # installer, it is easier to just fetch them from a known URL.
+    'vc9-crt-x86-msm': {
+        'url': 'https://github.com/indygreg/vc90-merge-modules/raw/9232f8f0b2135df619bf7946eaa176b4ac35ccff/Microsoft_VC90_CRT_x86.msm',
+        'size': 615424,
+        'sha256': '837e887ef31b332feb58156f429389de345cb94504228bb9a523c25a9dd3d75e',
+    },
+    'vc9-crt-x86-msm-policy': {
+        'url': 'https://github.com/indygreg/vc90-merge-modules/raw/9232f8f0b2135df619bf7946eaa176b4ac35ccff/policy_9_0_Microsoft_VC90_CRT_x86.msm',
+        'size': 71168,
+        'sha256': '3fbcf92e3801a0757f36c5e8d304e134a68d5cafd197a6df7734ae3e8825c940',
+    },
+    'vc9-crt-x64-msm': {
+        'url': 'https://github.com/indygreg/vc90-merge-modules/raw/9232f8f0b2135df619bf7946eaa176b4ac35ccff/Microsoft_VC90_CRT_x86_x64.msm',
+        'size': 662528,
+        'sha256': '50d9639b5ad4844a2285269c7551bf5157ec636e32396ddcc6f7ec5bce487a7c',
+    },
+    'vc9-crt-x64-msm-policy': {
+        'url': 'https://github.com/indygreg/vc90-merge-modules/raw/9232f8f0b2135df619bf7946eaa176b4ac35ccff/policy_9_0_Microsoft_VC90_CRT_x86_x64.msm',
+        'size': 71168,
+        'sha256': '0550ea1929b21239134ad3a678c944ba0f05f11087117b6cf0833e7110686486',
+    },
     'virtualenv': {
         'url': 'https://files.pythonhosted.org/packages/37/db/89d6b043b22052109da35416abc3c397655e4bd3cff031446ba02b9654fa/virtualenv-16.4.3.tar.gz',
         'size': 3713208,
         'sha256': '984d7e607b0a5d1329425dd8845bd971b957424b5ba664729fab51ab8c11bc39',
         'version': '16.4.3',
     },
+    'wix': {
+        'url': 'https://github.com/wixtoolset/wix3/releases/download/wix3111rtm/wix311-binaries.zip',
+        'size': 34358269,
+        'sha256': '37f0a533b0978a454efb5dc3bd3598becf9660aaf4287e55bf68ca6b527d051d',
+        'version': '3.11.1',
+    },
 }
 
 
 def hash_path(p: pathlib.Path):
     h = hashlib.sha256()
 
     with p.open('rb') as fh:
         while True:
             chunk = fh.read(65536)
             if not chunk:
                 break
 
             h.update(chunk)
 
     return h.hexdigest()
 
 
 class IntegrityError(Exception):
     """Represents an integrity error when downloading a URL."""
 
 
 def secure_download_stream(url, size, sha256):
     """Securely download a URL to a stream of chunks.
 
     If the integrity of the download fails, an IntegrityError is
     raised.
     """
     h = hashlib.sha256()
     length = 0
 
     with urllib.request.urlopen(url) as fh:
         if not url.endswith('.gz') and fh.info().get('Content-Encoding') == 'gzip':
             fh = gzip.GzipFile(fileobj=fh)
 
         while True:
             chunk = fh.read(65536)
             if not chunk:
                 break
 
             h.update(chunk)
             length += len(chunk)
 
             yield chunk
 
     digest = h.hexdigest()
 
     if length != size:
         raise IntegrityError('size mismatch on %s: wanted %d; got %d' % (
             url, size, length))
 
     if digest != sha256:
         raise IntegrityError('sha256 mismatch on %s: wanted %s; got %s' % (
             url, sha256, digest))
 
 
 def download_to_path(url: str, path: pathlib.Path, size: int, sha256: str):
     """Download a URL to a filesystem path, possibly with verification."""
 
     # We download to a temporary file and rename at the end so there's
     # no chance of the final file being partially written or containing
     # bad data.
     print('downloading %s to %s' % (url, path))
 
     if path.exists():
         good = True
 
         if path.stat().st_size != size:
             print('existing file size is wrong; removing')
             good = False
 
         if good:
             if hash_path(path) != sha256:
                 print('existing file hash is wrong; removing')
                 good = False
 
         if good:
             print('%s exists and passes integrity checks' % path)
             return
 
         path.unlink()
 
     tmp = path.with_name('%s.tmp' % path.name)
 
     try:
         with tmp.open('wb') as fh:
             for chunk in secure_download_stream(url, size, sha256):
                 fh.write(chunk)
     except IntegrityError:
         tmp.unlink()
         raise
 
     tmp.rename(path)
     print('successfully downloaded %s' % url)
 
 
 def download_entry(name: dict, dest_path: pathlib.Path, local_name=None) -> pathlib.Path:
     entry = DOWNLOADS[name]
 
     url = entry['url']
 
     local_name = local_name or url[url.rindex('/') + 1:]
 
     local_path = dest_path / local_name
     download_to_path(url, local_path, entry['size'], entry['sha256'])
 
     return local_path, entry

contrib/packaging/hgpackaging/util.py

@@ -1,73 +1,157 b''
 # util.py - Common packaging utility code.
 #
 # Copyright 2019 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 # no-check-code because Python 3 native.
 
 import distutils.version
+import getpass
 import os
 import pathlib
 import subprocess
 import tarfile
 import zipfile
 
 
 def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
     with tarfile.open(source, 'r') as tf:
         tf.extractall(dest)
 
 
 def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
     with zipfile.ZipFile(source, 'r') as zf:
         zf.extractall(dest)
 
 
 def find_vc_runtime_files(x64=False):
     """Finds Visual C++ Runtime DLLs to include in distribution."""
     winsxs = pathlib.Path(os.environ['SYSTEMROOT']) / 'WinSxS'
 
     prefix = 'amd64' if x64 else 'x86'
 
     candidates = sorted(p for p in os.listdir(winsxs)
                   if p.lower().startswith('%s_microsoft.vc90.crt_' % prefix))
 
     for p in candidates:
         print('found candidate VC runtime: %s' % p)
 
     # Take the newest version.
     version = candidates[-1]
 
     d = winsxs / version
 
     return [
         d / 'msvcm90.dll',
         d / 'msvcp90.dll',
         d / 'msvcr90.dll',
         winsxs / 'Manifests' / ('%s.manifest' % version),
     ]
 
 
+def windows_10_sdk_info():
+    """Resolves information about the Windows 10 SDK."""
+
+    base = pathlib.Path(os.environ['ProgramFiles(x86)']) / 'Windows Kits' / '10'
+
+    if not base.is_dir():
+        raise Exception('unable to find Windows 10 SDK at %s' % base)
+
+    # Find the latest version.
+    bin_base = base / 'bin'
+
+    versions = [v for v in os.listdir(bin_base) if v.startswith('10.')]
+    version = sorted(versions, reverse=True)[0]
+
+    bin_version = bin_base / version
+
+    return {
+        'root': base,
+        'version': version,
+        'bin_root': bin_version,
+        'bin_x86': bin_version / 'x86',
+        'bin_x64': bin_version / 'x64'
+    }
+
+
+def find_signtool():
+    """Find signtool.exe from the Windows SDK."""
+    sdk = windows_10_sdk_info()
+
+    for key in ('bin_x64', 'bin_x86'):
+        p = sdk[key] / 'signtool.exe'
+
+        if p.exists():
+            return p
+
+    raise Exception('could not find signtool.exe in Windows 10 SDK')
+
+
+def sign_with_signtool(file_path, description, subject_name=None,
+                       cert_path=None, cert_password=None,
+                       timestamp_url=None):
+    """Digitally sign a file with signtool.exe.
+
+    ``file_path`` is file to sign.
+    ``description`` is text that goes in the signature.
+
+    The signing certificate can be specified by ``cert_path`` or
+    ``subject_name``. These correspond to the ``/f`` and ``/n`` arguments
+    to signtool.exe, respectively.
+
+    The certificate password can be specified via ``cert_password``. If
+    not provided, you will be prompted for the password.
+
+    ``timestamp_url`` is the URL of a RFC 3161 timestamp server (``/tr``
+    argument to signtool.exe).
+    """
+    if cert_path and subject_name:
+        raise ValueError('cannot specify both cert_path and subject_name')
+
+    while cert_path and not cert_password:
+        cert_password = getpass.getpass('password for %s: ' % cert_path)
+
+    args = [
+        str(find_signtool()), 'sign',
+        '/v',
+        '/fd', 'sha256',
+        '/d', description,
+    ]
+
+    if cert_path:
+        args.extend(['/f', str(cert_path), '/p', cert_password])
+    elif subject_name:
+        args.extend(['/n', subject_name])
+
+    if timestamp_url:
+        args.extend(['/tr', timestamp_url, '/td', 'sha256'])
+
+    args.append(str(file_path))
+
+    print('signing %s' % file_path)
+    subprocess.run(args, check=True)
+
+
 PRINT_PYTHON_INFO = '''
 import platform; print("%s:%s" % (platform.architecture()[0], platform.python_version()))
 '''.strip()
 
 
 def python_exe_info(python_exe: pathlib.Path):
     """Obtain information about a Python executable."""
 
     res = subprocess.run(
         [str(python_exe), '-c', PRINT_PYTHON_INFO],
         capture_output=True, check=True)
 
     arch, version = res.stdout.decode('utf-8').split(':')
 
     version = distutils.version.LooseVersion(version)
 
     return {
         'arch': arch,
         'version': version,
         'py3': version >= distutils.version.LooseVersion('3'),
     }

contrib/packaging/wix/readme.rst

@@ -1,29 +1,71 b''
-WiX installer source files
+WiX Installer
+=============
+
+The files in this directory are used to produce an MSI installer using
+the WiX Toolset (http://wixtoolset.org/).
+
+The MSI installers require elevated (admin) privileges due to the
+installation of MSVC CRT libraries into the Windows system store. See
+the Inno Setup installers in the ``inno`` sibling directory for installers
+that do not have this requirement.
+
+Requirements
+============
+
+Building the WiX installers requires a Windows machine. The following
+dependencies must be installed:
+
+* Python 2.7 (download from https://www.python.org/downloads/)
+* Microsoft Visual C++ Compiler for Python 2.7
+  (https://www.microsoft.com/en-us/download/details.aspx?id=44266)
+* Python 3.5+ (to run the ``build.py`` script)
+
+Building
+========
+
+The ``build.py`` script automates the process of producing an MSI
+installer. It manages fetching and configuring non-system dependencies
+(such as py2exe, gettext, and various Python packages).
+
+The script requires an activated ``Visual C++ 2008`` command prompt.
+A shortcut to such a prompt was installed with ``Microsoft Visual
+C++ Compiler for Python 2.7``. From your Start Menu, look for
+``Microsoft Visual C++ Compiler Package for Python 2.7`` then
+launch either ``Visual C++ 2008 32-bit Command Prompt`` or
+``Visual C++ 2008 64-bit Command Prompt``.
+
+From the prompt, change to the Mercurial source directory. e.g.
+``cd c:\src\hg``.
+
+Next, invoke ``build.py`` to produce an MSI installer. You will need
+to supply the path to the Python interpreter to use.::
+
+   $ python3 contrib\packaging\wix\build.py \
+      --python c:\python27\python.exe
+
+.. note::
+
+   The script validates that the Visual C++ environment is active and
+   that the architecture of the specified Python interpreter matches the
+   Visual C++ environment. An error is raised otherwise.
+
+If everything runs as intended, dependencies will be fetched and
+configured into the ``build`` sub-directory, Mercurial will be built,
+and an installer placed in the ``dist`` sub-directory. The final line
+of output should print the name of the generated installer.
+
+Additional options may be configured. Run ``build.py --help`` to see
+a list of program flags.
+
+Relationship to TortoiseHG
 ==========================
 
-The files in this folder are used by the thg-winbuild [1] package
-building architecture to create a Mercurial MSI installer.   These files
-are versioned within the Mercurial source tree because the WXS files
-must kept up to date with distribution changes within their branch.  In
-other words, the default branch WXS files are expected to diverge from
-the stable branch WXS files.  Storing them within the same repository is
-the only sane way to keep the source tree and the installer in sync.
-
-The MSI installer builder uses only the mercurial.ini file from the
-contrib/win32 folder.
+TortoiseHG uses the WiX files in this directory.
 
-The MSI packages built by thg-winbuild require elevated (admin)
-privileges to be installed due to the installation of MSVC CRT libraries
-under the C:\WINDOWS\WinSxS folder.  Thus the InnoSetup installers may
-still be useful to some users.
+The code for building TortoiseHG installers lives at
+https://bitbucket.org/tortoisehg/thg-winbuild and is maintained by
+Steve Borho ([e-mail]).
 
-To build your own MSI packages, clone the thg-winbuild [1] repository
-and follow the README.txt [2] instructions closely.  There are fewer
-prerequisites for a WiX [3] installer than an InnoSetup installer, but
-they are more specific.
-
-Direct questions or comments to Steve Borho <[e-mail]>
-
-[1] http://bitbucket.org/tortoisehg/thg-winbuild
-[2] http://bitbucket.org/tortoisehg/thg-winbuild/src/tip/README.txt
-[3] http://wix.sourceforge.net/
+When changing behavior of the WiX installer, be sure to notify
+the TortoiseHG Project of the changes so they have ample time
+provide feedback and react to those changes.

tests/test-check-code.t

@@ -1,74 +1,76 b''
 #require test-repo
 
   $ . "$TESTDIR/helpers-testrepo.sh"
   $ check_code="$TESTDIR"/../contrib/check-code.py
   $ cd "$TESTDIR"/..
 
 New errors are not allowed. Warnings are strongly discouraged.
 (The writing "no-che?k-code" is for not skipping this file when checking.)
 
   $ testrepohg locate \
   > -X contrib/python-zstandard \
   > -X hgext/fsmonitor/pywatchman \
   > -X mercurial/thirdparty \
   > | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false
   Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)
+  Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob)
   Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob)
+  Skipping contrib/packaging/wix/build.py it has no-che?k-code (glob)
   Skipping i18n/polib.py it has no-che?k-code (glob)
   Skipping mercurial/statprof.py it has no-che?k-code (glob)
   Skipping tests/badserverext.py it has no-che?k-code (glob)
 
 @commands in debugcommands.py should be in alphabetical order.
 
   >>> import re
   >>> commands = []
   >>> with open('mercurial/debugcommands.py', 'rb') as fh:
   ...     for line in fh:
   ...         m = re.match(br"^@command\('([a-z]+)", line)
   ...         if m:
   ...             commands.append(m.group(1))
   >>> scommands = list(sorted(commands))
   >>> for i, command in enumerate(scommands):
   ...     if command != commands[i]:
   ...         print('commands in debugcommands.py not sorted; first differing '
   ...               'command is %s; expected %s' % (commands[i], command))
   ...         break
 
 Prevent adding new files in the root directory accidentally.
 
   $ testrepohg files 'glob:*'
   .arcconfig
   .clang-format
   .editorconfig
   .hgignore
   .hgsigs
   .hgtags
   .jshintrc
   CONTRIBUTING
   CONTRIBUTORS
   COPYING
   Makefile
   README.rst
   hg
   hgeditor
   hgweb.cgi
   setup.py
 
 Prevent adding modules which could be shadowed by ancient .so/.dylib.
 
   $ testrepohg files \
   > mercurial/base85.py \
   > mercurial/bdiff.py \
   > mercurial/diffhelpers.py \
   > mercurial/mpatch.py \
   > mercurial/osutil.py \
   > mercurial/parsers.py \
   > mercurial/zstd.py
   [1]
 
 Keep python3 tests sorted:
   $ sort < contrib/python3-whitelist > $TESTTMP/py3sorted
   $ cmp contrib/python3-whitelist $TESTTMP/py3sorted || echo 'Please sort passing tests!'

tests/test-check-py3-compat.t

@@ -1,58 +1,59 b''
 #require test-repo
 
   $ . "$TESTDIR/helpers-testrepo.sh"
   $ cd "$TESTDIR"/..
 
 #if no-py3
   $ testrepohg files 'set:(**.py)' \
   > -X contrib/packaging/hgpackaging/ \
   > -X contrib/packaging/inno/ \
+  > -X contrib/packaging/wix/ \
   > -X hgdemandimport/demandimportpy2.py \
   > -X mercurial/thirdparty/cbor \
   > | sed 's|\\|/|g' | xargs "$PYTHON" contrib/check-py3-compat.py
   contrib/python-zstandard/setup.py not using absolute_import
   contrib/python-zstandard/setup_zstd.py not using absolute_import
   contrib/python-zstandard/tests/common.py not using absolute_import
   contrib/python-zstandard/tests/test_buffer_util.py not using absolute_import
   contrib/python-zstandard/tests/test_compressor.py not using absolute_import
   contrib/python-zstandard/tests/test_compressor_fuzzing.py not using absolute_import
   contrib/python-zstandard/tests/test_data_structures.py not using absolute_import
   contrib/python-zstandard/tests/test_data_structures_fuzzing.py not using absolute_import
   contrib/python-zstandard/tests/test_decompressor.py not using absolute_import
   contrib/python-zstandard/tests/test_decompressor_fuzzing.py not using absolute_import
   contrib/python-zstandard/tests/test_estimate_sizes.py not using absolute_import
   contrib/python-zstandard/tests/test_module_attributes.py not using absolute_import
   contrib/python-zstandard/tests/test_train_dictionary.py not using absolute_import
   setup.py not using absolute_import
 #endif
 
 #if py3
   $ testrepohg files 'set:(**.py) - grep(pygments)' \
   > -X hgdemandimport/demandimportpy2.py \
   > -X hgext/fsmonitor/pywatchman \
   > -X mercurial/cffi \
   > -X mercurial/thirdparty \
   > | sed 's|\\|/|g' | xargs "$PYTHON" contrib/check-py3-compat.py \
   > | sed 's/[0-9][0-9]*)$/*)/'
   contrib/python-zstandard/tests/test_compressor.py:324: SyntaxWarning: invalid escape sequence \( (py38 !)
     with self.assertRaisesRegexp(zstd.ZstdError, 'cannot call compress\(\) after compressor'): (py38 !)
   contrib/python-zstandard/tests/test_compressor.py:1329: SyntaxWarning: invalid escape sequence \( (py38 !)
     'cannot call compress\(\) after compression finished'): (py38 !)
   contrib/python-zstandard/tests/test_compressor.py:1341: SyntaxWarning: invalid escape sequence \( (py38 !)
     'cannot call flush\(\) after compression finished'): (py38 !)
   contrib/python-zstandard/tests/test_compressor.py:1353: SyntaxWarning: invalid escape sequence \( (py38 !)
     'cannot call finish\(\) after compression finished'): (py38 !)
   hgext/convert/transport.py: error importing: <*Error> No module named 'svn.client' (error at transport.py:*) (glob) (?)
   hgext/infinitepush/sqlindexapi.py: error importing: <*Error> No module named 'mysql' (error at sqlindexapi.py:*) (glob) (?)
   mercurial/scmwindows.py: error importing: <ValueError> _type_ 'v' not supported (error at win32.py:*) (no-windows !)
   mercurial/win32.py: error importing: <ValueError> _type_ 'v' not supported (error at win32.py:*) (no-windows !)
   mercurial/windows.py: error importing: <ModuleNotFoundError> No module named 'msvcrt' (error at windows.py:*) (no-windows !)
   mercurial/posix.py: error importing: <ModuleNotFoundError> No module named 'fcntl' (error at posix.py:*) (windows !)
   mercurial/scmposix.py: error importing: <ModuleNotFoundError> No module named 'fcntl' (error at scmposix.py:*) (windows !)
 #endif
 
 #if py3 pygments
   $ testrepohg files 'set:(**.py) and grep(pygments)' | sed 's|\\|/|g' \
   > | xargs "$PYTHON" contrib/check-py3-compat.py \
   > | sed 's/[0-9][0-9]*)$/*)/'
 #endif