packaging: support building Inno installer with PyOxidizer

We want to start distributing Mercurial on Python 3 on
Windows. PyOxidizer will be our vehicle for achieving that.

This commit implements basic support for producing Inno
installers using PyOxidizer.

While it is an eventual goal of PyOxidizer to produce
installers, those features aren't yet implemented. So our
strategy for producing Mercurial installers is similar to
what we've been doing with py2exe: invoke a build system to
produce files then stage those files into a directory so they
can be turned into an installer.

We had to make significant alterations to the pyoxidizer.bzl
config file to get it to produce the files that we desire for
a Windows install. This meant differentiating the build targets
so we can target Windows specifically.

We've added a new module to hgpackaging to deal with interacting
with PyOxidizer. It is similar to pyexe: we invoke a build process
then copy files to a staging directory. Ideally these extra
files would be defined in pyoxidizer.bzl. But I don't think it
is worth doing at this time, as PyOxidizer's config files are
lacking some features to make this turnkey.

The rest of the change is introducing a variant of the
Inno installer code that invokes PyOxidizer instead of
py2exe.

Comparing the Python 2.7 based Inno installers with this
one, the following changes were observed:

* No lib/*.{pyd, dll} files
* No Microsoft.VC90.CRT.manifest
* No msvc{m,p,r}90.dll files
* python27.dll replaced with python37.dll
* Add vcruntime140.dll file

The disappearance of the .pyd and .dll files is acceptable, as
PyOxidizer has embedded these in hg.exe and loads them from
memory.

The disappearance of the *90* files is acceptable because those
provide the Visual C++ 9 runtime, as required by Python 2.7.
Similarly, the appearance of vcruntime140.dll is a requirement
of Python 3.7.

Differential Revision: https://phab.mercurial-scm.org/D8473

contrib/packaging/hgpackaging/pyoxidizer.py

@@ -0,0 +1,145
+# pyoxidizer.py - Packaging support for PyOxidizer
+#
+# Copyright 2020 Gregory Szorc <[e-mail]>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+# no-check-code because Python 3 native.
+
+import os
+import pathlib
+import shutil
+import subprocess
+import sys
+
+from .downloads import download_entry
+from .util import (
+    extract_zip_to_directory,
+    process_install_rules,
+    find_vc_runtime_dll,
+)
+
+
+STAGING_RULES_WINDOWS = [
+    ('contrib/bash_completion', 'contrib/'),
+    ('contrib/hgk', 'contrib/hgk.tcl'),
+    ('contrib/hgweb.fcgi', 'contrib/'),
+    ('contrib/hgweb.wsgi', 'contrib/'),
+    ('contrib/logo-droplets.svg', 'contrib/'),
+    ('contrib/mercurial.el', 'contrib/'),
+    ('contrib/mq.el', 'contrib/'),
+    ('contrib/tcsh_completion', 'contrib/'),
+    ('contrib/tcsh_completion_build.sh', 'contrib/'),
+    ('contrib/vim/*', 'contrib/vim/'),
+    ('contrib/win32/postinstall.txt', 'ReleaseNotes.txt'),
+    ('contrib/win32/ReadMe.html', 'ReadMe.html'),
+    ('contrib/xml.rnc', 'contrib/'),
+    ('contrib/zsh_completion', 'contrib/'),
+    ('doc/*.html', 'doc/'),
+    ('doc/style.css', 'doc/'),
+    ('COPYING', 'Copying.txt'),
+]
+
+STAGING_RULES_APP = [
+    ('mercurial/helptext/**/*.txt', 'helptext/'),
+    ('mercurial/defaultrc/*.rc', 'defaultrc/'),
+    ('mercurial/locale/**/*', 'locale/'),
+    ('mercurial/templates/**/*', 'templates/'),
+]
+
+STAGING_EXCLUDES_WINDOWS = [
+    "doc/hg-ssh.8.html",
+]
+
+
+def run_pyoxidizer(
+    source_dir: pathlib.Path,
+    build_dir: pathlib.Path,
+    out_dir: pathlib.Path,
+    target_triple: str,
+):
+    """Build Mercurial with PyOxidizer and copy additional files into place.
+
+    After successful completion, ``out_dir`` contains files constituting a
+    Mercurial install.
+    """
+    # We need to make gettext binaries available for compiling i18n files.
+    gettext_pkg, gettext_entry = download_entry('gettext', build_dir)
+    gettext_dep_pkg = download_entry('gettext-dep', build_dir)[0]
+
+    gettext_root = build_dir / ('gettext-win-%s' % gettext_entry['version'])
+
+    if not gettext_root.exists():
+        extract_zip_to_directory(gettext_pkg, gettext_root)
+        extract_zip_to_directory(gettext_dep_pkg, gettext_root)
+
+    env = dict(os.environ)
+    env["PATH"] = "%s%s%s" % (
+        env["PATH"],
+        os.pathsep,
+        str(gettext_root / "bin"),
+    )
+
+    args = [
+        "pyoxidizer",
+        "build",
+        "--path",
+        str(source_dir / "rust" / "hgcli"),
+        "--release",
+        "--target-triple",
+        target_triple,
+    ]
+
+    subprocess.run(args, env=env, check=True)
+
+    if "windows" in target_triple:
+        target = "app_windows"
+    else:
+        target = "app_posix"
+
+    build_dir = (
+        source_dir / "build" / "pyoxidizer" / target_triple / "release" / target
+    )
+
+    if out_dir.exists():
+        print("purging %s" % out_dir)
+        shutil.rmtree(out_dir)
+
+    # Now assemble all the files from PyOxidizer into the staging directory.
+    shutil.copytree(build_dir, out_dir)
+
+    # Move some of those files around.
+    process_install_rules(STAGING_RULES_APP, build_dir, out_dir)
+    # Nuke the mercurial/* directory, as we copied resources
+    # to an appropriate location just above.
+    shutil.rmtree(out_dir / "mercurial")
+
+    # We also need to run setup.py build_doc to produce html files,
+    # as they aren't built as part of ``pip install``.
+    # This will fail if docutils isn't installed.
+    subprocess.run(
+        [sys.executable, str(source_dir / "setup.py"), "build_doc", "--html"],
+        cwd=str(source_dir),
+        check=True,
+    )
+
+    if "windows" in target_triple:
+        process_install_rules(STAGING_RULES_WINDOWS, source_dir, out_dir)
+
+        # Write out a default editor.rc file to configure notepad as the
+        # default editor.
+        with (out_dir / "defaultrc" / "editor.rc").open(
+            "w", encoding="utf-8"
+        ) as fh:
+            fh.write("[ui]\neditor = notepad\n")
+
+        for f in STAGING_EXCLUDES_WINDOWS:
+            p = out_dir / f
+            if p.exists():
+                print("removing %s" % p)
+                p.unlink()
+
+        # Add vcruntimeXXX.dll next to executable.
+        vc_runtime_dll = find_vc_runtime_dll(x64="x86_64" in target_triple)
+        shutil.copy(vc_runtime_dll, out_dir / vc_runtime_dll.name)

contrib/packaging/hgpackaging/cli.py

@@ -1,153 +1,166
 # cli.py - Command line interface for automation
 #
 # Copyright 2019 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 # no-check-code because Python 3 native.
 
 import argparse
 import os
 import pathlib
 
 from . import (
     inno,
     wix,
 )
 
 HERE = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
 SOURCE_DIR = HERE.parent.parent.parent
 
 
-def build_inno(python=None, iscc=None, version=None):
-    if not os.path.isabs(python):
+def build_inno(pyoxidizer_target=None, python=None, iscc=None, version=None):
+    if not pyoxidizer_target and not python:
+        raise Exception("--python required unless building with PyOxidizer")
+
+    if python and not os.path.isabs(python):
         raise Exception("--python arg must be an absolute path")
 
     if iscc:
         iscc = pathlib.Path(iscc)
     else:
         iscc = (
             pathlib.Path(os.environ["ProgramFiles(x86)"])
             / "Inno Setup 5"
             / "ISCC.exe"
         )
 
     build_dir = SOURCE_DIR / "build"
 
-    inno.build(
-        SOURCE_DIR, build_dir, pathlib.Path(python), iscc, version=version,
-    )
+    if pyoxidizer_target:
+        inno.build_with_pyoxidizer(
+            SOURCE_DIR, build_dir, pyoxidizer_target, iscc, version=version
+        )
+    else:
+        inno.build_with_py2exe(
+            SOURCE_DIR, build_dir, pathlib.Path(python), iscc, version=version,
+        )
 
 
 def build_wix(
     name=None,
     python=None,
     version=None,
     sign_sn=None,
     sign_cert=None,
     sign_password=None,
     sign_timestamp_url=None,
     extra_packages_script=None,
     extra_wxs=None,
     extra_features=None,
 ):
     fn = wix.build_installer
     kwargs = {
         "source_dir": SOURCE_DIR,
         "python_exe": pathlib.Path(python),
         "version": version,
     }
 
     if not os.path.isabs(python):
         raise Exception("--python arg must be an absolute path")
 
     if extra_packages_script:
         kwargs["extra_packages_script"] = extra_packages_script
     if extra_wxs:
         kwargs["extra_wxs"] = dict(
             thing.split("=") for thing in extra_wxs.split(",")
         )
     if extra_features:
         kwargs["extra_features"] = extra_features.split(",")
 
     if sign_sn or sign_cert:
         fn = wix.build_signed_installer
         kwargs["name"] = name
         kwargs["subject_name"] = sign_sn
         kwargs["cert_path"] = sign_cert
         kwargs["cert_password"] = sign_password
         kwargs["timestamp_url"] = sign_timestamp_url
 
     fn(**kwargs)
 
 
 def get_parser():
     parser = argparse.ArgumentParser()
 
     subparsers = parser.add_subparsers()
 
     sp = subparsers.add_parser("inno", help="Build Inno Setup installer")
-    sp.add_argument("--python", required=True, help="path to python.exe to use")
+    sp.add_argument(
+        "--pyoxidizer-target",
+        choices={"i686-pc-windows-msvc", "x86_64-pc-windows-msvc"},
+        help="Build with PyOxidizer targeting this host triple",
+    )
+    sp.add_argument("--python", help="path to python.exe to use")
     sp.add_argument("--iscc", help="path to iscc.exe to use")
     sp.add_argument(
         "--version",
         help="Mercurial version string to use "
         "(detected from __version__.py if not defined",
     )
     sp.set_defaults(func=build_inno)
 
     sp = subparsers.add_parser(
         "wix", help="Build Windows installer with WiX Toolset"
     )
     sp.add_argument("--name", help="Application name", default="Mercurial")
     sp.add_argument(
         "--python", help="Path to Python executable to use", required=True
     )
     sp.add_argument(
         "--sign-sn",
         help="Subject name (or fragment thereof) of certificate "
         "to use for signing",
     )
     sp.add_argument(
         "--sign-cert", help="Path to certificate to use for signing"
     )
     sp.add_argument("--sign-password", help="Password for signing certificate")
     sp.add_argument(
         "--sign-timestamp-url",
         help="URL of timestamp server to use for signing",
     )
     sp.add_argument("--version", help="Version string to use")
     sp.add_argument(
         "--extra-packages-script",
         help=(
             "Script to execute to include extra packages in " "py2exe binary."
         ),
     )
     sp.add_argument(
         "--extra-wxs", help="CSV of path_to_wxs_file=working_dir_for_wxs_file"
     )
     sp.add_argument(
         "--extra-features",
         help=(
             "CSV of extra feature names to include "
             "in the installer from the extra wxs files"
         ),
     )
     sp.set_defaults(func=build_wix)
 
     return parser
 
 
 def main():
     parser = get_parser()
     args = parser.parse_args()
 
     if not hasattr(args, "func"):
         parser.print_help()
         return
 
     kwargs = dict(vars(args))
     del kwargs["func"]
 
     args.func(**kwargs)

contrib/packaging/hgpackaging/inno.py

@@ -1,197 +1,227
 # inno.py - Inno Setup functionality.
 #
 # Copyright 2019 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 # no-check-code because Python 3 native.
 
 import os
 import pathlib
 import shutil
 import subprocess
 
 import jinja2
 
 from .py2exe import (
     build_py2exe,
     stage_install,
 )
+from .pyoxidizer import run_pyoxidizer
 from .util import (
-    find_vc_runtime_files,
+    find_legacy_vc_runtime_files,
     normalize_windows_version,
     process_install_rules,
     read_version_py,
 )
 
 EXTRA_PACKAGES = {
     'dulwich',
     'keyring',
     'pygments',
     'win32ctypes',
 }
 
 EXTRA_INSTALL_RULES = [
     ('contrib/win32/mercurial.ini', 'defaultrc/mercurial.rc'),
 ]
 
 PACKAGE_FILES_METADATA = {
     'ReadMe.html': 'Flags: isreadme',
 }
 
 
-def build(
+def build_with_py2exe(
     source_dir: pathlib.Path,
     build_dir: pathlib.Path,
     python_exe: pathlib.Path,
     iscc_exe: pathlib.Path,
     version=None,
 ):
-    """Build the Inno installer.
+    """Build the Inno installer using py2exe.
 
     Build files will be placed in ``build_dir``.
 
     py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
     for finding the Python 2.7 toolchain. So, we require the environment
     to already be configured with an active toolchain.
     """
     if not iscc_exe.exists():
         raise Exception('%s does not exist' % iscc_exe)
 
     vc_x64 = r'\x64' in os.environ.get('LIB', '')
     arch = 'x64' if vc_x64 else 'x86'
     inno_build_dir = build_dir / ('inno-py2exe-%s' % arch)
     staging_dir = inno_build_dir / 'stage'
 
     requirements_txt = (
         source_dir / 'contrib' / 'packaging' / 'requirements_win32.txt'
     )
 
     inno_build_dir.mkdir(parents=True, exist_ok=True)
 
     build_py2exe(
         source_dir,
         build_dir,
         python_exe,
         'inno',
         requirements_txt,
         extra_packages=EXTRA_PACKAGES,
     )
 
     # Purge the staging directory for every build so packaging is
     # pristine.
     if staging_dir.exists():
         print('purging %s' % staging_dir)
         shutil.rmtree(staging_dir)
 
     # Now assemble all the packaged files into the staging directory.
     stage_install(source_dir, staging_dir)
 
     # We also install some extra files.
     process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir)
 
     # hg.exe depends on VC9 runtime DLLs. Copy those into place.
-    for f in find_vc_runtime_files(vc_x64):
+    for f in find_legacy_vc_runtime_files(vc_x64):
         if f.name.endswith('.manifest'):
             basename = 'Microsoft.VC90.CRT.manifest'
         else:
             basename = f.name
 
         dest_path = staging_dir / basename
 
         print('copying %s to %s' % (f, dest_path))
         shutil.copyfile(f, dest_path)
 
     build_installer(
         source_dir,
         inno_build_dir,
         staging_dir,
         iscc_exe,
         version,
         arch="x64" if vc_x64 else None,
     )
 
 
+def build_with_pyoxidizer(
+    source_dir: pathlib.Path,
+    build_dir: pathlib.Path,
+    target_triple: str,
+    iscc_exe: pathlib.Path,
+    version=None,
+):
+    """Build the Inno installer using PyOxidizer."""
+    if not iscc_exe.exists():
+        raise Exception("%s does not exist" % iscc_exe)
+
+    inno_build_dir = build_dir / ("inno-pyoxidizer-%s" % target_triple)
+    staging_dir = inno_build_dir / "stage"
+
+    inno_build_dir.mkdir(parents=True, exist_ok=True)
+    run_pyoxidizer(source_dir, inno_build_dir, staging_dir, target_triple)
+
+    process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir)
+
+    build_installer(
+        source_dir,
+        inno_build_dir,
+        staging_dir,
+        iscc_exe,
+        version,
+        arch="x64" if "x86_64" in target_triple else None,
+    )
+
+
 def build_installer(
     source_dir: pathlib.Path,
     inno_build_dir: pathlib.Path,
     staging_dir: pathlib.Path,
     iscc_exe: pathlib.Path,
     version,
     arch=None,
 ):
     """Build an Inno installer from staged Mercurial files.
 
     This function is agnostic about how to build Mercurial. It just
     cares that Mercurial files are in ``staging_dir``.
     """
     inno_source_dir = source_dir / "contrib" / "packaging" / "inno"
 
     # The final package layout is simply a mirror of the staging directory.
     package_files = []
     for root, dirs, files in os.walk(staging_dir):
         dirs.sort()
 
         root = pathlib.Path(root)
 
         for f in sorted(files):
             full = root / f
             rel = full.relative_to(staging_dir)
             if str(rel.parent) == '.':
                 dest_dir = '{app}'
             else:
                 dest_dir = '{app}\\%s' % rel.parent
 
             package_files.append(
                 {
                     'source': rel,
                     'dest_dir': dest_dir,
                     'metadata': PACKAGE_FILES_METADATA.get(str(rel), None),
                 }
             )
 
     print('creating installer')
 
     # Install Inno files by rendering a template.
     jinja_env = jinja2.Environment(
         loader=jinja2.FileSystemLoader(str(inno_source_dir)),
         # Need to change these to prevent conflict with Inno Setup.
         comment_start_string='{##',
         comment_end_string='##}',
     )
 
     try:
         template = jinja_env.get_template('mercurial.iss')
     except jinja2.TemplateSyntaxError as e:
         raise Exception(
             'template syntax error at %s:%d: %s'
             % (e.name, e.lineno, e.message,)
         )
 
     content = template.render(package_files=package_files)
 
     with (inno_build_dir / 'mercurial.iss').open('w', encoding='utf-8') as fh:
         fh.write(content)
 
     # Copy additional files used by Inno.
     for p in ('mercurial.ico', 'postinstall.txt'):
         shutil.copyfile(
             source_dir / 'contrib' / 'win32' / p, inno_build_dir / p
         )
 
     args = [str(iscc_exe)]
 
     if arch:
         args.append('/dARCH=%s' % arch)
 
     if not version:
         version = read_version_py(source_dir)
 
     args.append('/dVERSION=%s' % version)
     args.append('/dQUAD_VERSION=%s' % normalize_windows_version(version))
 
     args.append('/Odist')
     args.append(str(inno_build_dir / 'mercurial.iss'))
 
     subprocess.run(args, cwd=str(source_dir), check=True)

contrib/packaging/hgpackaging/util.py

@@ -1,286 +1,338
 # util.py - Common packaging utility code.
 #
 # Copyright 2019 Gregory Szorc <[e-mail]>
 #
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
 # no-check-code because Python 3 native.
 
 import distutils.version
 import getpass
 import glob
 import os
 import pathlib
 import re
 import shutil
 import subprocess
 import tarfile
 import zipfile
 
 
 def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
     with tarfile.open(source, 'r') as tf:
         tf.extractall(dest)
 
 
 def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
     with zipfile.ZipFile(source, 'r') as zf:
         zf.extractall(dest)
 
 
-def find_vc_runtime_files(x64=False):
+def find_vc_runtime_dll(x64=False):
+    """Finds Visual C++ Runtime DLL to include in distribution."""
+    # We invoke vswhere to find the latest Visual Studio install.
+    vswhere = (
+        pathlib.Path(os.environ["ProgramFiles(x86)"])
+        / "Microsoft Visual Studio"
+        / "Installer"
+        / "vswhere.exe"
+    )
+
+    if not vswhere.exists():
+        raise Exception(
+            "could not find vswhere.exe: %s does not exist" % vswhere
+        )
+
+    args = [
+        str(vswhere),
+        # -products * is necessary to return results from Build Tools
+        # (as opposed to full IDE installs).
+        "-products",
+        "*",
+        "-requires",
+        "Microsoft.VisualCpp.Redist.14.Latest",
+        "-latest",
+        "-property",
+        "installationPath",
+    ]
+
+    vs_install_path = pathlib.Path(
+        os.fsdecode(subprocess.check_output(args).strip())
+    )
+
+    # This just gets us a path like
+    # C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
+    # Actually vcruntime140.dll is under a path like:
+    # VC\Redist\MSVC\<version>\<arch>\Microsoft.VC14<X>.CRT\vcruntime140.dll.
+
+    arch = "x64" if x64 else "x86"
+
+    search_glob = (
+        r"%s\VC\Redist\MSVC\*\%s\Microsoft.VC14*.CRT\vcruntime140.dll"
+        % (vs_install_path, arch)
+    )
+
+    candidates = glob.glob(search_glob, recursive=True)
+
+    for candidate in reversed(candidates):
+        return pathlib.Path(candidate)
+
+    raise Exception("could not find vcruntime140.dll")
+
+
+def find_legacy_vc_runtime_files(x64=False):
     """Finds Visual C++ Runtime DLLs to include in distribution."""
     winsxs = pathlib.Path(os.environ['SYSTEMROOT']) / 'WinSxS'
 
     prefix = 'amd64' if x64 else 'x86'
 
     candidates = sorted(
         p
         for p in os.listdir(winsxs)
         if p.lower().startswith('%s_microsoft.vc90.crt_' % prefix)
     )
 
     for p in candidates:
         print('found candidate VC runtime: %s' % p)
 
     # Take the newest version.
     version = candidates[-1]
 
     d = winsxs / version
 
     return [
         d / 'msvcm90.dll',
         d / 'msvcp90.dll',
         d / 'msvcr90.dll',
         winsxs / 'Manifests' / ('%s.manifest' % version),
     ]
 
 
 def windows_10_sdk_info():
     """Resolves information about the Windows 10 SDK."""
 
     base = pathlib.Path(os.environ['ProgramFiles(x86)']) / 'Windows Kits' / '10'
 
     if not base.is_dir():
         raise Exception('unable to find Windows 10 SDK at %s' % base)
 
     # Find the latest version.
     bin_base = base / 'bin'
 
     versions = [v for v in os.listdir(bin_base) if v.startswith('10.')]
     version = sorted(versions, reverse=True)[0]
 
     bin_version = bin_base / version
 
     return {
         'root': base,
         'version': version,
         'bin_root': bin_version,
         'bin_x86': bin_version / 'x86',
         'bin_x64': bin_version / 'x64',
     }
 
 
 def normalize_windows_version(version):
     """Normalize Mercurial version string so WiX/Inno accepts it.
 
     Version strings have to be numeric ``A.B.C[.D]`` to conform with MSI's
     requirements.
 
     We normalize RC version or the commit count to a 4th version component.
     We store this in the 4th component because ``A.B.C`` releases do occur
     and we want an e.g. ``5.3rc0`` version to be semantically less than a
     ``5.3.1rc2`` version. This requires always reserving the 3rd version
     component for the point release and the ``X.YrcN`` release is always
     point release 0.
 
     In the case of an RC and presence of ``+`` suffix data, we can't use both
     because the version format is limited to 4 components. We choose to use
     RC and throw away the commit count in the suffix. This means we could
     produce multiple installers with the same normalized version string.
 
     >>> normalize_windows_version("5.3")
     '5.3.0'
 
     >>> normalize_windows_version("5.3rc0")
     '5.3.0.0'
 
     >>> normalize_windows_version("5.3rc1")
     '5.3.0.1'
 
     >>> normalize_windows_version("5.3rc1+2-abcdef")
     '5.3.0.1'
 
     >>> normalize_windows_version("5.3+2-abcdef")
     '5.3.0.2'
     """
     if '+' in version:
         version, extra = version.split('+', 1)
     else:
         extra = None
 
     # 4.9rc0
     if version[:-1].endswith('rc'):
         rc = int(version[-1:])
         version = version[:-3]
     else:
         rc = None
 
     # Ensure we have at least X.Y version components.
     versions = [int(v) for v in version.split('.')]
     while len(versions) < 3:
         versions.append(0)
 
     if len(versions) < 4:
         if rc is not None:
             versions.append(rc)
         elif extra:
             # <commit count>-<hash>+<date>
             versions.append(int(extra.split('-')[0]))
 
     return '.'.join('%d' % x for x in versions[0:4])
 
 
 def find_signtool():
     """Find signtool.exe from the Windows SDK."""
     sdk = windows_10_sdk_info()
 
     for key in ('bin_x64', 'bin_x86'):
         p = sdk[key] / 'signtool.exe'
 
         if p.exists():
             return p
 
     raise Exception('could not find signtool.exe in Windows 10 SDK')
 
 
 def sign_with_signtool(
     file_path,
     description,
     subject_name=None,
     cert_path=None,
     cert_password=None,
     timestamp_url=None,
 ):
     """Digitally sign a file with signtool.exe.
 
     ``file_path`` is file to sign.
     ``description`` is text that goes in the signature.
 
     The signing certificate can be specified by ``cert_path`` or
     ``subject_name``. These correspond to the ``/f`` and ``/n`` arguments
     to signtool.exe, respectively.
 
     The certificate password can be specified via ``cert_password``. If
     not provided, you will be prompted for the password.
 
     ``timestamp_url`` is the URL of a RFC 3161 timestamp server (``/tr``
     argument to signtool.exe).
     """
     if cert_path and subject_name:
         raise ValueError('cannot specify both cert_path and subject_name')
 
     while cert_path and not cert_password:
         cert_password = getpass.getpass('password for %s: ' % cert_path)
 
     args = [
         str(find_signtool()),
         'sign',
         '/v',
         '/fd',
         'sha256',
         '/d',
         description,
     ]
 
     if cert_path:
         args.extend(['/f', str(cert_path), '/p', cert_password])
     elif subject_name:
         args.extend(['/n', subject_name])
 
     if timestamp_url:
         args.extend(['/tr', timestamp_url, '/td', 'sha256'])
 
     args.append(str(file_path))
 
     print('signing %s' % file_path)
     subprocess.run(args, check=True)
 
 
 PRINT_PYTHON_INFO = '''
 import platform; print("%s:%s" % (platform.architecture()[0], platform.python_version()))
 '''.strip()
 
 
 def python_exe_info(python_exe: pathlib.Path):
     """Obtain information about a Python executable."""
 
     res = subprocess.check_output([str(python_exe), '-c', PRINT_PYTHON_INFO])
 
     arch, version = res.decode('utf-8').split(':')
 
     version = distutils.version.LooseVersion(version)
 
     return {
         'arch': arch,
         'version': version,
         'py3': version >= distutils.version.LooseVersion('3'),
     }
 
 
 def process_install_rules(
     rules: list, source_dir: pathlib.Path, dest_dir: pathlib.Path
 ):
     for source, dest in rules:
         if '*' in source:
             if not dest.endswith('/'):
                 raise ValueError('destination must end in / when globbing')
 
             # We strip off the source path component before the first glob
             # character to construct the relative install path.
             prefix_end_index = source[: source.index('*')].rindex('/')
             relative_prefix = source_dir / source[0:prefix_end_index]
 
             for res in glob.glob(str(source_dir / source), recursive=True):
                 source_path = pathlib.Path(res)
 
                 if source_path.is_dir():
                     continue
 
                 rel_path = source_path.relative_to(relative_prefix)
 
                 dest_path = dest_dir / dest[:-1] / rel_path
 
                 dest_path.parent.mkdir(parents=True, exist_ok=True)
                 print('copying %s to %s' % (source_path, dest_path))
                 shutil.copy(source_path, dest_path)
 
         # Simple file case.
         else:
             source_path = pathlib.Path(source)
 
             if dest.endswith('/'):
                 dest_path = pathlib.Path(dest) / source_path.name
             else:
                 dest_path = pathlib.Path(dest)
 
             full_source_path = source_dir / source_path
             full_dest_path = dest_dir / dest_path
 
             full_dest_path.parent.mkdir(parents=True, exist_ok=True)
             shutil.copy(full_source_path, full_dest_path)
             print('copying %s to %s' % (full_source_path, full_dest_path))
 
 
 def read_version_py(source_dir):
     """Read the mercurial/__version__.py file to resolve the version string."""
     p = source_dir / 'mercurial' / '__version__.py'
 
     with p.open('r', encoding='utf-8') as fh:
         m = re.search('version = b"([^"]+)"', fh.read(), re.MULTILINE)
 
         if not m:
             raise Exception('could not parse %s' % p)
 
         return m.group(1)

rust/hgcli/pyoxidizer.bzl

@@ -1,59 +1,104
 ROOT = CWD + "/../.."
 
-def make_exe():
-    dist = default_python_distribution()
+# Code to run in Python interpreter.
+RUN_CODE = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
+
+
+set_build_path(ROOT + "/build/pyoxidizer")
+
 
-    code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
+def make_distribution():
+    return default_python_distribution()
+
 
+def make_distribution_windows():
+    return default_python_distribution(flavor="standalone_dynamic")
+
+
+def make_exe(dist):
     config = PythonInterpreterConfig(
         raw_allocator = "system",
-        run_eval = code,
+        run_eval = RUN_CODE,
         # We want to let the user load extensions from the file system
         filesystem_importer = True,
         # We need this to make resourceutil happy, since it looks for sys.frozen.
         sys_frozen = True,
         legacy_windows_stdio = True,
     )
 
     exe = dist.to_python_executable(
         name = "hg",
         resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
         config = config,
         # Extension may depend on any Python functionality. Include all
         # extensions.
         extension_module_filter = "all",
     )
 
-    exe.add_python_resources(dist.pip_install([ROOT]))
+    # Add Mercurial to resources.
+    for resource in dist.pip_install(["--verbose", ROOT]):
+        # This is a bit wonky and worth explaining.
+        #
+        # Various parts of Mercurial don't yet support loading package
+        # resources via the ResourceReader interface. Or, not having
+        # file-based resources would be too inconvenient for users.
+        #
+        # So, for package resources, we package them both in the
+        # filesystem as well as in memory. If both are defined,
+        # PyOxidizer will prefer the in-memory location. So even
+        # if the filesystem file isn't packaged in the location
+        # specified here, we should never encounter an errors as the
+        # resource will always be available in memory.
+        if type(resource) == "PythonPackageResource":
+            exe.add_filesystem_relative_python_resource(".", resource)
+            exe.add_in_memory_python_resource(resource)
+        else:
+            exe.add_python_resource(resource)
+
+    # On Windows, we install extra packages for convenience.
+    if "windows" in BUILD_TARGET_TRIPLE:
+        exe.add_python_resources(
+            dist.pip_install(["-r", ROOT + "/contrib/packaging/requirements_win32.txt"])
+        )
 
     return exe
 
-def make_install(exe):
+
+def make_manifest(dist, exe):
     m = FileManifest()
-
-    # `hg` goes in root directory.
     m.add_python_resource(".", exe)
 
-    templates = glob(
-        include = [ROOT + "/mercurial/templates/**/*"],
-        strip_prefix = ROOT + "/mercurial/",
-    )
-    m.add_manifest(templates)
+    return m
 
-    return m
 
 def make_embedded_resources(exe):
     return exe.to_embedded_resources()
 
-register_target("exe", make_exe)
-register_target("app", make_install, depends = ["exe"], default = True)
-register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True)
+
+register_target("distribution_posix", make_distribution)
+register_target("distribution_windows", make_distribution_windows)
+
+register_target("exe_posix", make_exe, depends = ["distribution_posix"])
+register_target("exe_windows", make_exe, depends = ["distribution_windows"])
+
+register_target(
+    "app_posix",
+    make_manifest,
+    depends = ["distribution_posix", "exe_posix"],
+    default = "windows" not in BUILD_TARGET_TRIPLE,
+)
+register_target(
+    "app_windows",
+    make_manifest,
+    depends = ["distribution_windows", "exe_windows"],
+    default = "windows" in BUILD_TARGET_TRIPLE,
+)
+
 resolve_targets()
 
 # END OF COMMON USER-ADJUSTED SETTINGS.
 #
 # Everything below this is typically managed by PyOxidizer and doesn't need
 # to be updated by people.
 
-PYOXIDIZER_VERSION = "0.7.0-pre"
-PYOXIDIZER_COMMIT = "c772a1379c3026314eda1c8ea244b86c0658951d"
+PYOXIDIZER_VERSION = "0.7.0"

tests/test-check-code.t

@@ -1,93 +1,94
 #require test-repo
 
   $ . "$TESTDIR/helpers-testrepo.sh"
   $ check_code="$TESTDIR"/../contrib/check-code.py
   $ cd "$TESTDIR"/..
 
 New errors are not allowed. Warnings are strongly discouraged.
 (The writing "no-che?k-code" is for not skipping this file when checking.)
 
   $ testrepohg locate \
   > -X contrib/python-zstandard \
   > -X hgext/fsmonitor/pywatchman \
   > -X mercurial/thirdparty \
   > | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false
   Skipping contrib/automation/hgautomation/__init__.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/aws.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/cli.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/linux.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/pypi.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/ssh.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/try_server.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/windows.py it has no-che?k-code (glob)
   Skipping contrib/automation/hgautomation/winrm.py it has no-che?k-code (glob)
   Skipping contrib/fuzz/FuzzedDataProvider.h it has no-che?k-code (glob)
   Skipping contrib/fuzz/standalone_fuzz_target_runner.cc it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/cli.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob)
+  Skipping contrib/packaging/hgpackaging/pyoxidizer.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)
   Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob)
   Skipping i18n/polib.py it has no-che?k-code (glob)
   Skipping mercurial/statprof.py it has no-che?k-code (glob)
   Skipping tests/badserverext.py it has no-che?k-code (glob)
 
 @commands in debugcommands.py should be in alphabetical order.
 
   >>> import re
   >>> commands = []
   >>> with open('mercurial/debugcommands.py', 'rb') as fh:
   ...     for line in fh:
   ...         m = re.match(br"^@command\('([a-z]+)", line)
   ...         if m:
   ...             commands.append(m.group(1))
   >>> scommands = list(sorted(commands))
   >>> for i, command in enumerate(scommands):
   ...     if command != commands[i]:
   ...         print('commands in debugcommands.py not sorted; first differing '
   ...               'command is %s; expected %s' % (commands[i], command))
   ...         break
 
 Prevent adding new files in the root directory accidentally.
 
   $ testrepohg files 'glob:*'
   .arcconfig
   .clang-format
   .editorconfig
   .hgignore
   .hgsigs
   .hgtags
   .jshintrc
   CONTRIBUTING
   CONTRIBUTORS
   COPYING
   Makefile
   README.rst
   black.toml
   hg
   hgeditor
   hgweb.cgi
   setup.py
 
 Prevent adding modules which could be shadowed by ancient .so/.dylib.
 
   $ testrepohg files \
   > mercurial/base85.py \
   > mercurial/bdiff.py \
   > mercurial/diffhelpers.py \
   > mercurial/mpatch.py \
   > mercurial/osutil.py \
   > mercurial/parsers.py \
   > mercurial/zstd.py
   [1]
 
 Keep python3 tests sorted:
   $ sort < contrib/python3-whitelist > $TESTTMP/py3sorted
   $ cmp contrib/python3-whitelist $TESTTMP/py3sorted || echo 'Please sort passing tests!'
 
 Keep Windows line endings in check
 
   $ hg files 'set:eol(dos)'
   contrib/win32/hg.bat
   contrib/win32/mercurial.ini