pyoxidizer: support producing MSI installers

Newer versions of PyOxidizer have support for building WiX MSI
installers "natively." Essentially, you can script the definition
of your WiX installer via Starlark and PyOxidizer can invoke
WiX tools to produce the installer.

This commit teaches our PyOxidizer config file to produce MSI
installers similarly to how
`contrib/packaging/packging.py wix` would do it.

We had to make a very minor change to `mercurial.wxs` to reflect
different paths depending on who builds. This is because when
PyOxidizer builds WiX installers, it does so from an isolated
directory, not Mercurial's source directory. We simply copy the
files into the build environment so they are accessible.

After this change, running `pyoxidizer build msi` produces a nearly
identical install layout to what the previous method produces.
When I applied this series on top of the 5.8 tag, here is the
list of differences and explanations:

* docs/*.html files are missing from the new installer because the
  Python build environment doesn't have docutils.
* .pyd and .exe files differ, likely because I'm using a different
  Visual Studio toolchain on my local computer than the official build
  environment.
* Various .dist-info/ directories have different names. This is
  because older versions of PyOxidizer had buggy behavior and weren't
  properly normalizing package names in .dist-info/ directories. e.g.
  we went from `cached-property-1.5.2.dist-info` to
  `cached_property-1.5.2.dist-info`.
* Translations (.mo files) may be missing if gettext isn't in %Path%.
  This is because the packaging.py code installs gettext and ensures
  it can be found.
* Some *.dist-info/RECORD files vary due to SHA-256 content digest
  divergence due to build environment differences. (This should be
  harmless.)
* The new install layout ships a python3.dll because newer versions
  of PyOxidizer ship this file.
* The new install layout has a different vcruntime140.dll and also a
  vcruntime140_1.dll because newer versions of PyOxidizer ship a
  newer version of the Visual C++ Redistributable Runtime.

The new PyOxidizer functionality is not yet integrated with
packaging.py. This will come in a subsequent commit. So for now, the
new functionality introduced here is unused.

Differential Revision: https://phab.mercurial-scm.org/D10683

contrib/packaging/wix/mercurial.wxs

@@ -1,153 +1,157 b''
 <?xml version='1.0' encoding='windows-1252'?>
 <Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'>
 
   <!-- Copyright 2010 Steve Borho <[e-mail]>
 
   This software may be used and distributed according to the terms of the
   GNU General Public License version 2 or any later version. -->
 
   <?include guids.wxi ?>
   <?include defines.wxi ?>
 
   <?if $(var.Platform) = "x64" ?>
     <?define PFolder = ProgramFiles64Folder ?>
   <?else?>
     <?define PFolder = ProgramFilesFolder ?>
   <?endif?>
 
   <Product Id='*'
     Name='Mercurial $(var.Version) ($(var.Platform))'
     UpgradeCode='$(var.ProductUpgradeCode)'
     Language='1033' Codepage='1252' Version='$(var.Version)'
     Manufacturer='Olivia Mackall and others'>
 
     <Package Id='*'
       Keywords='Installer'
       Description="Mercurial distributed SCM (version $(var.Version))"
       Comments='$(var.Comments)'
       Platform='$(var.Platform)'
       Manufacturer='Olivia Mackall and others'
       InstallerVersion='300' Languages='1033' Compressed='yes' SummaryCodepage='1252' />
 
     <Media Id='1' Cabinet='mercurial.cab' EmbedCab='yes' DiskPrompt='CD-ROM #1'
            CompressionLevel='high' />
     <Property Id='DiskPrompt' Value="Mercurial $(var.Version) Installation [1]" />
 
     <Condition Message='Mercurial MSI installers require Windows XP or higher'>
         VersionNT >= 501
     </Condition>
 
     <Property Id="INSTALLDIR">
       <ComponentSearch Id='SearchForMainExecutableComponent'
                        Guid='$(var.ComponentMainExecutableGUID)'
                        Type='directory' />
     </Property>
 
     <!--Property Id='ARPCOMMENTS'>any comments</Property-->
     <Property Id='ARPCONTACT'>[e-mail]</Property>
     <Property Id='ARPHELPLINK'>https://mercurial-scm.org/wiki/</Property>
     <Property Id='ARPURLINFOABOUT'>https://mercurial-scm.org/about/</Property>
     <Property Id='ARPURLUPDATEINFO'>https://mercurial-scm.org/downloads/</Property>
     <Property Id='ARPHELPTELEPHONE'>https://mercurial-scm.org/wiki/Support</Property>
     <Property Id='ARPPRODUCTICON'>hgIcon.ico</Property>
 
     <Property Id='INSTALLEDMERCURIALPRODUCTS' Secure='yes'></Property>
     <Property Id='REINSTALLMODE'>amus</Property>
 
     <!--Auto-accept the license page-->
     <Property Id='LicenseAccepted'>1</Property>
 
     <Directory Id='TARGETDIR' Name='SourceDir'>
       <Directory Id='$(var.PFolder)' Name='PFiles'>
         <Directory Id='INSTALLDIR' Name='Mercurial'>
           <Component Id='MainExecutable' Guid='$(var.ComponentMainExecutableGUID)' Win64='$(var.IsX64)'>
             <CreateFolder />
             <Environment Id="Environment" Name="PATH" Part="last" System="yes"
                          Permanent="no" Value="[INSTALLDIR]" Action="set" />
           </Component>
         </Directory>
       </Directory>
 
       <Directory Id="ProgramMenuFolder" Name="Programs">
         <Directory Id="ProgramMenuDir" Name="Mercurial $(var.Version)">
           <Component Id="ProgramMenuDir" Guid="$(var.ProgramMenuDir.guid)" Win64='$(var.IsX64)'>
             <RemoveFolder Id='ProgramMenuDir' On='uninstall' />
             <RegistryValue Root='HKCU' Key='Software\Mercurial\InstallDir' Type='string'
                            Value='[INSTALLDIR]' KeyPath='yes' />
             <Shortcut Id='UrlShortcut' Directory='ProgramMenuDir' Name='Mercurial Web Site'
                       Target='[ARPHELPLINK]' Icon="hgIcon.ico" IconIndex='0' />
           </Component>
         </Directory>
       </Directory>
 
       <!-- Install VCRedist merge modules on Python 2. On Python 3,
            vcruntimeXXX.dll is part of the install layout and gets picked up
            as a regular file. -->
       <?if $(var.PythonVersion) = "2" ?>
         <?if $(var.Platform) = "x86" ?>
           <Merge Id='VCRuntime' DiskId='1' Language='1033'
                 SourceFile='$(var.VCRedistSrcDir)\microsoft.vcxx.crt.x86_msm.msm' />
           <Merge Id='VCRuntimePolicy' DiskId='1' Language='1033'
                 SourceFile='$(var.VCRedistSrcDir)\policy.x.xx.microsoft.vcxx.crt.x86_msm.msm' />
         <?else?>
           <Merge Id='VCRuntime' DiskId='1' Language='1033'
                 SourceFile='$(var.VCRedistSrcDir)\microsoft.vcxx.crt.x64_msm.msm' />
           <Merge Id='VCRuntimePolicy' DiskId='1' Language='1033'
                 SourceFile='$(var.VCRedistSrcDir)\policy.x.xx.microsoft.vcxx.crt.x64_msm.msm' />
         <?endif?>
       <?endif?>
     </Directory>
 
     <Feature Id='Complete' Title='Mercurial' Description='The complete package'
         Display='expand' Level='1' ConfigurableDirectory='INSTALLDIR' >
       <Feature Id='MainProgram' Title='Program' Description='Mercurial command line app'
              Level='1' Absent='disallow' >
         <ComponentRef Id='MainExecutable' />
         <ComponentRef Id='ProgramMenuDir' />
         <ComponentGroupRef Id="hg.group.ROOT" />
         <ComponentGroupRef Id="hg.group.defaultrc" />
         <ComponentGroupRef Id="hg.group.helptext" />
         <?ifdef MercurialHasLib?>
           <ComponentGroupRef Id="hg.group.lib" />
         <?endif?>
         <ComponentGroupRef Id="hg.group.templates" />
         <?if $(var.PythonVersion) = "2" ?>
           <MergeRef Id='VCRuntime' />
           <MergeRef Id='VCRuntimePolicy' />
         <?endif?>
       </Feature>
       <?ifdef MercurialExtraFeatures?>
         <?foreach EXTRAFEAT in $(var.MercurialExtraFeatures)?>
           <FeatureRef Id="$(var.EXTRAFEAT)" />
         <?endforeach?>
       <?endif?>
       <Feature Id='Locales' Title='Translations' Description='Translations' Level='1'>
         <ComponentGroupRef Id="hg.group.locale" />
       </Feature>
       <Feature Id='Documentation' Title='Documentation' Description='HTML man pages' Level='1'>
         <ComponentGroupRef Id="hg.group.doc" />
       </Feature>
       <Feature Id='Misc' Title='Miscellaneous' Description='Contributed scripts' Level='1'>
         <ComponentGroupRef Id="hg.group.contrib" />
       </Feature>
     </Feature>
 
     <UIRef Id="WixUI_FeatureTree" />
     <UIRef Id="WixUI_ErrorProgressText" />
 
+    <?ifdef PyOxidizer?>
+    <WixVariable Id="WixUILicenseRtf" Value="COPYING.rtf" />
+    <Icon Id="hgIcon.ico" SourceFile="mercurial.ico" />
+    <?else?>
     <WixVariable Id="WixUILicenseRtf" Value="contrib\packaging\wix\COPYING.rtf" />
-
     <Icon Id="hgIcon.ico" SourceFile="contrib/win32/mercurial.ico" />
+    <?endif?>
 
     <Upgrade Id='$(var.ProductUpgradeCode)'>
       <UpgradeVersion
         IncludeMinimum='yes' Minimum='0.0.0' IncludeMaximum='no' OnlyDetect='no'
         Property='INSTALLEDMERCURIALPRODUCTS' />
     </Upgrade>
 
     <InstallExecuteSequence>
       <RemoveExistingProducts After='InstallInitialize'/>
     </InstallExecuteSequence>
 
   </Product>
 </Wix>

rust/hgcli/pyoxidizer.bzl

@@ -1,87 +1,238 b''
+# The following variables can be passed in as parameters:
+#
+# VERSION
+#   Version string of program being produced.
+#
+# MSI_NAME
+#   Root name of MSI installer.
+#
+# EXTRA_MSI_FEATURES
+#   ; delimited string of extra features to advertise in the built MSA.
+
 ROOT = CWD + "/../.."
 
+VERSION = VARS.get("VERSION", "5.8")
+MSI_NAME = VARS.get("MSI_NAME", "mercurial")
+EXTRA_MSI_FEATURES = VARS.get("EXTRA_MSI_FEATURES")
+
 IS_WINDOWS = "windows" in BUILD_TARGET_TRIPLE
 
 # Code to run in Python interpreter.
 RUN_CODE = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
 
 set_build_path(ROOT + "/build/pyoxidizer")
 
 def make_distribution():
     return default_python_distribution(python_version = "3.8")
 
 def resource_callback(policy, resource):
     if not IS_WINDOWS:
         resource.add_location = "in-memory"
         return
 
     # We use a custom resource routing policy to influence where things are loaded
     # from.
     #
     # For Python modules and resources, we load from memory if they are in
     # the standard library and from the filesystem if not. This is because
     # parts of Mercurial and some 3rd party packages aren't yet compatible
     # with memory loading.
     #
     # For Python extension modules, we load from the filesystem because
     # this yields greatest compatibility.
     if type(resource) in ("PythonModuleSource", "PythonPackageResource", "PythonPackageDistributionResource"):
         if resource.is_stdlib:
             resource.add_location = "in-memory"
         else:
             resource.add_location = "filesystem-relative:lib"
 
     elif type(resource) == "PythonExtensionModule":
         resource.add_location = "filesystem-relative:lib"
 
 def make_exe(dist):
     """Builds a Rust-wrapped Mercurial binary."""
     packaging_policy = dist.make_python_packaging_policy()
 
     # Extension may depend on any Python functionality. Include all
     # extensions.
     packaging_policy.extension_module_filter = "all"
     packaging_policy.resources_location = "in-memory"
     if IS_WINDOWS:
         packaging_policy.resources_location_fallback = "filesystem-relative:lib"
     packaging_policy.register_resource_callback(resource_callback)
 
     config = dist.make_python_interpreter_config()
     config.allocator_backend = "default"
     config.run_command = RUN_CODE
 
     # We want to let the user load extensions from the file system
     config.filesystem_importer = True
 
     # We need this to make resourceutil happy, since it looks for sys.frozen.
     config.sys_frozen = True
     config.legacy_windows_stdio = True
 
     exe = dist.to_python_executable(
         name = "hg",
         packaging_policy = packaging_policy,
         config = config,
     )
 
     # Add Mercurial to resources.
     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
 
     # On Windows, we install extra packages for convenience.
     if IS_WINDOWS:
         exe.add_python_resources(
             exe.pip_install(["-r", ROOT + "/contrib/packaging/requirements-windows-py3.txt"]),
         )
 
     return exe
 
 def make_manifest(dist, exe):
     m = FileManifest()
     m.add_python_resource(".", exe)
 
     return m
 
+
+# This adjusts the InstallManifest produced from exe generation to provide
+# additional files found in a Windows install layout.
+def make_windows_install_layout(manifest):
+    # Copy various files to new install locations. This can go away once
+    # we're using the importlib resource reader.
+    RECURSIVE_COPIES = {
+        "lib/mercurial/locale/": "locale/",
+        "lib/mercurial/templates/": "templates/",
+    }
+    for (search, replace) in RECURSIVE_COPIES.items():
+        for path in manifest.paths():
+            if path.startswith(search):
+                new_path = path.replace(search, replace)
+                print("copy %s to %s" % (path, new_path))
+                file = manifest.get_file(path)
+                manifest.add_file(file, path = new_path)
+
+    # Similar to above, but with filename pattern matching.
+    # lib/mercurial/helptext/**/*.txt -> helptext/
+    # lib/mercurial/defaultrc/*.rc -> defaultrc/
+    for path in manifest.paths():
+        if path.startswith("lib/mercurial/helptext/") and path.endswith(".txt"):
+            new_path = path[len("lib/mercurial/"):]
+        elif path.startswith("lib/mercurial/defaultrc/") and path.endswith(".rc"):
+            new_path = path[len("lib/mercurial/"):]
+        else:
+            continue
+
+        print("copying %s to %s" % (path, new_path))
+        manifest.add_file(manifest.get_file(path), path = new_path)
+
+    # We also install a handful of additional files.
+    EXTRA_CONTRIB_FILES = [
+        "bash_completion",
+        "hgweb.fcgi",
+        "hgweb.wsgi",
+        "logo-droplets.svg",
+        "mercurial.el",
+        "mq.el",
+        "tcsh_completion",
+        "tcsh_completion_build.sh",
+        "xml.rnc",
+        "zsh_completion",
+    ]
+
+    for f in EXTRA_CONTRIB_FILES:
+        manifest.add_file(FileContent(path = ROOT + "/contrib/" + f), directory = "contrib")
+
+    # Individual files with full source to destination path mapping.
+    EXTRA_FILES = {
+        "contrib/hgk": "contrib/hgk.tcl",
+        "contrib/win32/postinstall.txt": "ReleaseNotes.txt",
+        "contrib/win32/ReadMe.html": "ReadMe.html",
+        "doc/style.css": "doc/style.css",
+        "COPYING": "Copying.txt",
+    }
+
+    for source, dest in EXTRA_FILES.items():
+        print("adding extra file %s" % dest)
+        manifest.add_file(FileContent(path = ROOT + "/" + source), path = dest)
+
+    # And finally some wildcard matches.
+    manifest.add_manifest(glob(
+        include = [ROOT + "/contrib/vim/*"],
+        strip_prefix = ROOT + "/"
+    ))
+    manifest.add_manifest(glob(
+        include = [ROOT + "/doc/*.html"],
+        strip_prefix = ROOT + "/"
+    ))
+
+    # But we don't ship hg-ssh on Windows, so exclude its documentation.
+    manifest.remove("doc/hg-ssh.8.html")
+
+    return manifest
+
+
+def make_msi(manifest):
+    manifest = make_windows_install_layout(manifest)
+
+    if "x86_64" in BUILD_TARGET_TRIPLE:
+        platform = "x64"
+    else:
+        platform = "x86"
+
+    manifest.add_file(
+        FileContent(path = ROOT + "/contrib/packaging/wix/COPYING.rtf"),
+        path = "COPYING.rtf",
+    )
+    manifest.remove("Copying.txt")
+    manifest.add_file(
+        FileContent(path = ROOT + "/contrib/win32/mercurial.ini"),
+        path = "defaultrc/mercurial.rc",
+    )
+    manifest.add_file(
+        FileContent(filename = "editor.rc", content = "[ui]\neditor = notepad\n"),
+        path = "defaultrc/editor.rc",
+    )
+
+    wix = WiXInstaller("hg", "%s-%s.msi" % (MSI_NAME, VERSION))
+
+    # Materialize files in the manifest to the install layout.
+    wix.add_install_files(manifest)
+
+    # From mercurial.wxs.
+    wix.install_files_root_directory_id = "INSTALLDIR"
+
+    # Pull in our custom .wxs files.
+    defines = {
+        "PyOxidizer": "1",
+        "Platform": platform,
+        "Version": VERSION,
+        "Comments": "Installs Mercurial version %s" % VERSION,
+        "PythonVersion": "3",
+        "MercurialHasLib": "1",
+    }
+
+    if EXTRA_MSI_FEATURES:
+        defines["MercurialExtraFeatures"] = EXTRA_MSI_FEATURES
+
+    wix.add_wxs_file(
+        ROOT + "/contrib/packaging/wix/mercurial.wxs",
+        preprocessor_parameters=defines,
+    )
+
+    # Our .wxs references to other files. Pull those into the build environment.
+    for f in ("defines.wxi", "guids.wxi", "COPYING.rtf"):
+        wix.add_build_file(f, ROOT + "/contrib/packaging/wix/" + f)
+
+    wix.add_build_file("mercurial.ico", ROOT + "/contrib/win32/mercurial.ico")
+
+    return wix
+
+
 register_target("distribution", make_distribution)
 register_target("exe", make_exe, depends = ["distribution"])
 register_target("app", make_manifest, depends = ["distribution", "exe"], default = True)
+register_target("msi", make_msi, depends = ["app"])
 
 resolve_targets()