hgcli: customize for Mercurial

Now that we have a shiny new PyOxidizer-based hgcli project, let's
customize it for Mercurial!

This commit replaces the auto-generated pyoxidizer.bzl with one
that installs Mercurial from the local source repository.

A README.md with build instructions has been added.

The Cargo.toml file has been updated to reflect the proper license
and reference the added README.md.

In my Linux environment, running the test suite yields 27 failures.

It's worth noting the run time of the test harness on Linux on my
Ryzen 3950X:

before: 378s wall; 9982s user; 1195s sys
after:  353s wall; 8996s user;  958s sys
% orig: 93.4 wall;  90.1  user; 80.2 sys

While I haven't measured explicitly, I suspect the performance win is
due to in-memory resource loading (which is known to be faster than
Python's filesystem importer).

Differential Revision: https://phab.mercurial-scm.org/D8351

rust/hgcli/README.md

@@ -0,0 +1,50
+# Oxidized Mercurial
+
+This project provides a Rust implementation of the Mercurial (`hg`)
+version control tool.
+
+Under the hood, the project uses
+[PyOxidizer](https://github.com/indygreg/PyOxidizer) to embed a Python
+interpreter in a binary built with Rust. At run-time, the Rust `fn main()`
+is called and Rust code handles initial process startup. An in-process
+Python interpreter is started (if needed) to provide additional
+functionality.
+
+# Building
+
+This project currently requires an unreleased version of PyOxidizer
+(0.7.0-pre). For best results, build the exact PyOxidizer commit
+as defined in the `pyoxidizer.bzl` file:
+
+    $ git clone https://github.com/indygreg/PyOxidizer.git
+    $ cd PyOxidizer
+    $ git checkout <Git commit from pyoxidizer.bzl>
+    $ cargo build --release
+
+Then build this Rust project using the built `pyoxidizer` executable::
+
+    $ /path/to/pyoxidizer/target/release/pyoxidizer build
+
+If all goes according to plan, there should be an assembled application
+under `build/<arch>/debug/app/` with an `hg` executable:
+
+    $ build/x86_64-unknown-linux-gnu/debug/app/hg version
+    Mercurial Distributed SCM (version 5.3.1+433-f99cd77d53dc+20200331)
+    (see https://mercurial-scm.org for more information)
+
+    Copyright (C) 2005-2020 Matt Mackall and others
+    This is free software; see the source for copying conditions. There is NO
+    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+# Running Tests
+
+To run tests with a built `hg` executable, you can use the `--with-hg`
+argument to `run-tests.py`. But there's a wrinkle: many tests run custom
+Python scripts that need to `import` modules provided by Mercurial. Since
+these modules are embedded in the produced `hg` executable, a regular
+Python interpreter can't access them! To work around this, set `PYTHONPATH`
+to the Mercurial source directory. e.g.:
+
+    $ cd /path/to/hg/src/tests
+    $ PYTHONPATH=`pwd`/.. python3.7 run-tests.py \
+        --with-hg `pwd`/../rust/hgcli/build/x86_64-unknown-linux-gnu/debug/app/hg

rust/hgcli/Cargo.toml

@@ -1,20 +1,24
 [package]
 name = "hgcli"
 version = "0.1.0"
 build = "build.rs"
 authors = ["Gregory Szorc <[e-mail]>"]
 edition = "2018"
+license = "GPL-2.0"
+readme = "README.md"
 
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+[[bin]]
+name = "hg"
+path = "src/main.rs"
 
 [dependencies]
 jemallocator-global = { version = "0.3", optional = true }
 pyembed = { git = "https://github.com/indygreg/PyOxidizer.git", rev = "c772a1379c3026314eda1c8ea244b86c0658951d", default-features=false }
 
 [features]
 default = ["build-mode-pyoxidizer-exe"]
 jemalloc = ["jemallocator-global", "pyembed/jemalloc"]
 build-mode-pyoxidizer-exe = ["pyembed/build-mode-pyoxidizer-exe"]
 build-mode-prebuilt-artifacts = ["pyembed/build-mode-prebuilt-artifacts"]
 cpython-link-unresolved-static = ["pyembed/cpython-link-unresolved-static"]
 cpython-link-default = ["pyembed/cpython-link-default"]

rust/hgcli/pyoxidizer.bzl

@@ -1,153 +1,59
-# This file defines how PyOxidizer application building and packaging is
-# performed. See the pyoxidizer crate's documentation for extensive
-# documentation on this file format.
+ROOT = CWD + "/../.."
 
-# Obtain the default PythonDistribution for our build target. We link
-# this distribution into our produced executable and extract the Python
-# standard library from it.
-def make_dist():
-    return default_python_distribution()
+def make_exe():
+    dist = default_python_distribution()
+
+    code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
 
-# Configuration files consist of functions which define build "targets."
-# This function creates a Python executable and installs it in a destination
-# directory.
-def make_exe(dist):
-    # This variable defines the configuration of the
-    # embedded Python interpreter.
-    python_config = PythonInterpreterConfig(
-        #     bytes_warning=0,
-        #     dont_write_bytecode=True,
-        #     ignore_environment=True,
-        #     inspect=False,
-        #     interactive=False,
-        #     isolated=False,
-        #     legacy_windows_fs_encoding=False,
-        #     legacy_windows_stdio=False,
-        #     no_site=True,
-        #     no_user_site_directory=True,
-        #     optimize_level=0,
-        #     parser_debug=False,
-        #     stdio_encoding=None,
-        #     unbuffered_stdio=False,
-        #     filesystem_importer=False,
-        #     sys_frozen=False,
-        #     sys_meipass=False,
-        #     sys_paths=None,
-        #     raw_allocator=None,
-        #     terminfo_resolution="dynamic",
-        #     terminfo_dirs=None,
-        #     use_hash_seed=False,
-        #     verbose=0,
-        #     write_modules_directory_env=None,
-        #     run_eval=None,
-        #     run_module=None,
-        #     run_noop=False,
-        #     run_repl=True,
+    config = PythonInterpreterConfig(
+        raw_allocator = "system",
+        run_eval = code,
+        # We want to let the user load extensions from the file system
+        filesystem_importer = True,
+        # We need this to make resourceutil happy, since it looks for sys.frozen.
+        sys_frozen = True,
+        legacy_windows_stdio = True,
     )
 
-    # The run_eval, run_module, run_noop, and run_repl arguments are mutually
-    # exclusive controls over what the interpreter should do once it initializes.
-    #
-    # run_eval -- Run the specified string value via `eval()`.
-    # run_module -- Import the specified module as __main__ and run it.
-    # run_noop -- Do nothing.
-    # run_repl -- Start a Python REPL.
-    #
-    # These arguments can be ignored if you are providing your own Rust code for
-    # starting the interpreter, as Rust code has full control over interpreter
-    # behavior.
-
-    # Produce a PythonExecutable from a Python distribution, embedded
-    # resources, and other options. The returned object represents the
-    # standalone executable that will be built.
     exe = dist.to_python_executable(
-        name = "hgcli",
-        config = python_config,
-        # Embed all extension modules, making this a fully-featured Python.
+        name = "hg",
+        resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
+        config = config,
+        # Extension may depend on any Python functionality. Include all
+        # extensions.
         extension_module_filter = "all",
-
-        # Only package the minimal set of extension modules needed to initialize
-        # a Python interpreter. Many common packages in Python's standard
-        # library won't work with this setting.
-        #extension_module_filter='minimal',
-
-        # Only package extension modules that don't require linking against
-        # non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
-        # other features that require external libraries.
-        #extension_module_filter='no-libraries',
-
-        # Only package extension modules that don't link against GPL licensed
-        # libraries.
-        #extension_module_filter='no-gpl',
-
-        # Include Python module sources. This isn't strictly required and it does
-        # make binary sizes larger. But having the sources can be useful for
-        # activities such as debugging.
-        include_sources = True,
-
-        # Whether to include non-module resource data/files.
-        include_resources = False,
-
-        # Do not include functionality for testing Python itself.
-        include_test = False,
     )
 
-    # Invoke `pip install` with our Python distribution to install a single package.
-    # `pip_install()` returns objects representing installed files.
-    # `add_in_memory_python_resources()` adds these objects to the binary,
-    # marking them for in-memory loading.
-    #exe.add_in_memory_python_resources(dist.pip_install(["appdirs"]))
+    exe.add_python_resources(dist.pip_install([ROOT]))
 
-    # Invoke `pip install` using a requirements file and add the collected resources
-    # to our binary.
-    #exe.add_in_memory_python_resources(dist.pip_install(["-r", "requirements.txt"]))
+    return exe
+
+def make_install(exe):
+    m = FileManifest()
 
-    # Read Python files from a local directory and add them to our embedded
-    # context, taking just the resources belonging to the `foo` and `bar`
-    # Python packages.
-    #exe.add_in_memory_python_resources(dist.read_package_root(
-    #    path="/src/mypackage",
-    #    packages=["foo", "bar"],
-    #))
+    # `hg` goes in root directory.
+    m.add_python_resource(".", exe)
 
-    # Discover Python files from a virtualenv and add them to our embedded
-    # context.
-    #exe.add_in_memory_python_resources(dist.read_virtualenv(path="/path/to/venv"))
+    templates = glob(
+        include = [ROOT + "/mercurial/templates/**/*"],
+        strip_prefix = ROOT + "/mercurial/",
+    )
+    m.add_manifest(templates)
 
-    # Filter all resources collected so far through a filter of names
-    # in a file.
-    #exe.filter_from_files(files=["/path/to/filter-file"]))
-
-    # Return our `PythonExecutable` instance so it can be built and
-    # referenced by other consumers of this target.
-    return exe
+    return m
 
 def make_embedded_resources(exe):
     return exe.to_embedded_resources()
 
-def make_install(exe):
-    # Create an object that represents our installed application file layout.
-    files = FileManifest()
-
-    # Add the generated executable to our install layout in the root directory.
-    files.add_python_resource(".", exe)
-
-    return files
-
-# Tell PyOxidizer about the build targets defined above.
-register_target("dist", make_dist)
-register_target("exe", make_exe, depends = ["dist"], default = True)
-register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True)
-register_target("install", make_install, depends = ["exe"])
-
-# Resolve whatever targets the invoker of this configuration file is requesting
-# be resolved.
+register_target("exe", make_exe)
+register_target("app", make_install, depends = ["exe"], default = True)
+register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True)
 resolve_targets()
 
 # END OF COMMON USER-ADJUSTED SETTINGS.
 #
 # Everything below this is typically managed by PyOxidizer and doesn't need
 # to be updated by people.
 
 PYOXIDIZER_VERSION = "0.7.0-pre"
 PYOXIDIZER_COMMIT = "c772a1379c3026314eda1c8ea244b86c0658951d"