rhg: Add support for automatic fallback to Python

`rhg` is a command-line application that can do a small subset of what
`hg` can. It is written entirely in Rust, which avoids the cost of starting
a Python interpreter and importing many Python modules.
In a script that runs many `hg` commands, this cost can add up.

However making users decide when to use `rhg` instead of `hg` is
not practical as we want the subset of supported functionality
to grow over time.

Instead we introduce "fallback" behavior where, when `rhg` encounters
something (a sub-command, a repository format, …) that is not implemented
in Rust-only, it does nothing but silently start a subprocess of
Python-based `hg` running the same command.
That way `rhg` becomes a drop-in replacement for `hg` that sometimes
goes faster. Whether Python is used should be an implementation detail
not apparent to users (other than through speed).

A new `fallback` value is added to the previously introduced
`rhg.on-unsupported` configuration key. When in this mode, the new
`rhg.fallback-executable` config is determine what command to use
to run a Python-based `hg`.

The previous `rhg.on-unsupported = abort-silent` configuration was designed
to let a wrapper script call `rhg` and then fall back to `hg` based on the
exit code. This is still available, but having fallback behavior built-in
in rhg might be easier for users instead of leaving that script "as an
exercise for the reader".

Using a subprocess like this is not idea, especially when `rhg` is to be
installed in `$PATH` as `hg`, since the other `hg.py` executable needs
to still be available… somewhere. Eventually this could be replaced
by using PyOxidizer to a have a single executable that embeds a Python
interpreter, but only starts it when needed.

Differential Revision: https://phab.mercurial-scm.org/D10093

rust/rhg/src/main.rs

@@ -1,286 +1,336
 extern crate log;
 use crate::ui::Ui;
 use clap::App;
 use clap::AppSettings;
 use clap::Arg;
 use clap::ArgMatches;
 use format_bytes::format_bytes;
 use hg::config::Config;
 use hg::repo::{Repo, RepoError};
 use hg::utils::files::{get_bytes_from_os_str, get_path_from_bytes};
 use hg::utils::SliceExt;
 use std::ffi::OsString;
 use std::path::PathBuf;
+use std::process::Command;
 
 mod blackbox;
 mod error;
 mod exitcode;
 mod ui;
 use error::CommandError;
 
 fn main_with_result(
     process_start_time: &blackbox::ProcessStartTime,
     ui: &ui::Ui,
     repo: Result<&Repo, &NoRepoInCwdError>,
     config: &Config,
 ) -> Result<(), CommandError> {
     let app = App::new("rhg")
         .global_setting(AppSettings::AllowInvalidUtf8)
         .setting(AppSettings::SubcommandRequired)
         .setting(AppSettings::VersionlessSubcommands)
         .arg(
             Arg::with_name("repository")
                 .help("repository root directory")
                 .short("-R")
                 .long("--repository")
                 .value_name("REPO")
                 .takes_value(true)
                 // Both ok: `hg -R ./foo log` or `hg log -R ./foo`
                 .global(true),
         )
         .arg(
             Arg::with_name("config")
                 .help("set/override config option (use 'section.name=value')")
                 .long("--config")
                 .value_name("CONFIG")
                 .takes_value(true)
                 .global(true)
                 // Ok: `--config section.key1=val --config section.key2=val2`
                 .multiple(true)
                 // Not ok: `--config section.key1=val section.key2=val2`
                 .number_of_values(1),
         )
         .version("0.0.1");
     let app = add_subcommand_args(app);
 
     let matches = app.clone().get_matches_safe()?;
 
     let (subcommand_name, subcommand_matches) = matches.subcommand();
     let run = subcommand_run_fn(subcommand_name)
         .expect("unknown subcommand name from clap despite AppSettings::SubcommandRequired");
     let subcommand_args = subcommand_matches
         .expect("no subcommand arguments from clap despite AppSettings::SubcommandRequired");
 
     let invocation = CliInvocation {
         ui,
         subcommand_args,
         config,
         repo,
     };
     let blackbox = blackbox::Blackbox::new(&invocation, process_start_time)?;
     blackbox.log_command_start();
     let result = run(&invocation);
     blackbox.log_command_end(exit_code(&result));
     result
 }
 
 fn main() {
     // Run this first, before we find out if the blackbox extension is even
     // enabled, in order to include everything in-between in the duration
     // measurements. Reading config files can be slow if they’re on NFS.
     let process_start_time = blackbox::ProcessStartTime::now();
 
     env_logger::init();
     let ui = ui::Ui::new();
 
     let early_args = EarlyArgs::parse(std::env::args_os());
     let non_repo_config =
         Config::load(early_args.config).unwrap_or_else(|error| {
             // Normally this is decided based on config, but we don’t have that
             // available. As of this writing config loading never returns an
             // "unsupported" error but that is not enforced by the type system.
             let on_unsupported = OnUnsupported::Abort;
 
             exit(&ui, on_unsupported, Err(error.into()))
         });
 
     let repo_path = early_args.repo.as_deref().map(get_path_from_bytes);
     let repo_result = match Repo::find(&non_repo_config, repo_path) {
         Ok(repo) => Ok(repo),
         Err(RepoError::NotFound { at }) if repo_path.is_none() => {
             // Not finding a repo is not fatal yet, if `-R` was not given
             Err(NoRepoInCwdError { cwd: at })
         }
         Err(error) => exit(
             &ui,
             OnUnsupported::from_config(&non_repo_config),
             Err(error.into()),
         ),
     };
 
     let config = if let Ok(repo) = &repo_result {
         repo.config()
     } else {
         &non_repo_config
     };
 
     let result = main_with_result(
         &process_start_time,
         &ui,
         repo_result.as_ref(),
         config,
     );
     exit(&ui, OnUnsupported::from_config(config), result)
 }
 
 fn exit_code(result: &Result<(), CommandError>) -> i32 {
     match result {
         Ok(()) => exitcode::OK,
         Err(CommandError::Abort { .. }) => exitcode::ABORT,
 
         // Exit with a specific code and no error message to let a potential
         // wrapper script fallback to Python-based Mercurial.
         Err(CommandError::UnsupportedFeature { .. }) => {
             exitcode::UNIMPLEMENTED
         }
     }
 }
 
 fn exit(
     ui: &Ui,
-    on_unsupported: OnUnsupported,
+    mut on_unsupported: OnUnsupported,
     result: Result<(), CommandError>,
 ) -> ! {
+    if let (
+        OnUnsupported::Fallback { executable },
+        Err(CommandError::UnsupportedFeature { .. }),
+    ) = (&on_unsupported, &result)
+    {
+        let mut args = std::env::args_os();
+        let executable_path = get_path_from_bytes(&executable);
+        let this_executable = args.next().expect("exepcted argv[0] to exist");
+        if executable_path == &PathBuf::from(this_executable) {
+            // Avoid spawning infinitely many processes until resource
+            // exhaustion.
+            let _ = ui.write_stderr(&format_bytes!(
+                b"Blocking recursive fallback. The 'rhg.fallback-executable = {}' config \
+                points to `rhg` itself.\n",
+                executable
+            ));
+            on_unsupported = OnUnsupported::Abort
+        } else {
+            // `args` is now `argv[1..]` since we’ve already consumed `argv[0]`
+            let result = Command::new(executable_path).args(args).status();
+            match result {
+                Ok(status) => std::process::exit(
+                    status.code().unwrap_or(exitcode::ABORT),
+                ),
+                Err(error) => {
+                    let _ = ui.write_stderr(&format_bytes!(
+                        b"tried to fall back to a '{}' sub-process but got error {}\n",
+                        executable, format_bytes::Utf8(error)
+                    ));
+                    on_unsupported = OnUnsupported::Abort
+                }
+            }
+        }
+    }
     match &result {
         Ok(_) => {}
         Err(CommandError::Abort { message }) => {
             if !message.is_empty() {
                 // Ignore errors when writing to stderr, we’re already exiting
                 // with failure code so there’s not much more we can do.
                 let _ =
                     ui.write_stderr(&format_bytes!(b"abort: {}\n", message));
             }
         }
         Err(CommandError::UnsupportedFeature { message }) => {
             match on_unsupported {
                 OnUnsupported::Abort => {
                     let _ = ui.write_stderr(&format_bytes!(
                         b"unsupported feature: {}\n",
                         message
                     ));
                 }
                 OnUnsupported::AbortSilent => {}
+                OnUnsupported::Fallback { .. } => unreachable!(),
             }
         }
     }
     std::process::exit(exit_code(&result))
 }
 
 macro_rules! subcommands {
     ($( $command: ident )+) => {
         mod commands {
             $(
                 pub mod $command;
             )+
         }
 
         fn add_subcommand_args<'a, 'b>(app: App<'a, 'b>) -> App<'a, 'b> {
             app
             $(
                 .subcommand(commands::$command::args())
             )+
         }
 
         pub type RunFn = fn(&CliInvocation) -> Result<(), CommandError>;
 
         fn subcommand_run_fn(name: &str) -> Option<RunFn> {
             match name {
                 $(
                     stringify!($command) => Some(commands::$command::run),
                 )+
                 _ => None,
             }
         }
     };
 }
 
 subcommands! {
     cat
     debugdata
     debugrequirements
     files
     root
     config
 }
 pub struct CliInvocation<'a> {
     ui: &'a Ui,
     subcommand_args: &'a ArgMatches<'a>,
     config: &'a Config,
     /// References inside `Result` is a bit peculiar but allow
     /// `invocation.repo?` to work out with `&CliInvocation` since this
     /// `Result` type is `Copy`.
     repo: Result<&'a Repo, &'a NoRepoInCwdError>,
 }
 
 struct NoRepoInCwdError {
     cwd: PathBuf,
 }
 
 /// CLI arguments to be parsed "early" in order to be able to read
 /// configuration before using Clap. Ideally we would also use Clap for this,
 /// see <https://github.com/clap-rs/clap/discussions/2366>.
 ///
 /// These arguments are still declared when we do use Clap later, so that Clap
 /// does not return an error for their presence.
 struct EarlyArgs {
     /// Values of all `--config` arguments. (Possibly none)
     config: Vec<Vec<u8>>,
     /// Value of the `-R` or `--repository` argument, if any.
     repo: Option<Vec<u8>>,
 }
 
 impl EarlyArgs {
     fn parse(args: impl IntoIterator<Item = OsString>) -> Self {
         let mut args = args.into_iter().map(get_bytes_from_os_str);
         let mut config = Vec::new();
         let mut repo = None;
         // Use `while let` instead of `for` so that we can also call
         // `args.next()` inside the loop.
         while let Some(arg) = args.next() {
             if arg == b"--config" {
                 if let Some(value) = args.next() {
                     config.push(value)
                 }
             } else if let Some(value) = arg.drop_prefix(b"--config=") {
                 config.push(value.to_owned())
             }
 
             if arg == b"--repository" || arg == b"-R" {
                 if let Some(value) = args.next() {
                     repo = Some(value)
                 }
             } else if let Some(value) = arg.drop_prefix(b"--repository=") {
                 repo = Some(value.to_owned())
             } else if let Some(value) = arg.drop_prefix(b"-R") {
                 repo = Some(value.to_owned())
             }
         }
         Self { config, repo }
     }
 }
 
 /// What to do when encountering some unsupported feature.
 ///
 /// See `HgError::UnsupportedFeature` and `CommandError::UnsupportedFeature`.
 enum OnUnsupported {
     /// Print an error message describing what feature is not supported,
     /// and exit with code 252.
     Abort,
     /// Silently exit with code 252.
     AbortSilent,
+    /// Try running a Python implementation
+    Fallback { executable: Vec<u8> },
 }
 
 impl OnUnsupported {
+    const DEFAULT: Self = OnUnsupported::Abort;
+    const DEFAULT_FALLBACK_EXECUTABLE: &'static [u8] = b"hg";
+
     fn from_config(config: &Config) -> Self {
-        let default = OnUnsupported::Abort;
-        match config.get(b"rhg", b"on-unsupported") {
+        match config
+            .get(b"rhg", b"on-unsupported")
+            .map(|value| value.to_ascii_lowercase())
+            .as_deref()
+        {
             Some(b"abort") => OnUnsupported::Abort,
             Some(b"abort-silent") => OnUnsupported::AbortSilent,
-            None => default,
+            Some(b"fallback") => OnUnsupported::Fallback {
+                executable: config
+                    .get(b"rhg", b"fallback-executable")
+                    .unwrap_or(Self::DEFAULT_FALLBACK_EXECUTABLE)
+                    .to_owned(),
+            },
+            None => Self::DEFAULT,
             Some(_) => {
                 // TODO: warn about unknown config value
-                default
+                Self::DEFAULT
             }
         }
     }
 }

tests/test-rhg.t

@@ -1,287 +1,309
 #require rust
 
 Define an rhg function that will only run if rhg exists
+  $ RHG="$RUNTESTDIR/../rust/target/release/rhg"
   $ rhg() {
-  > if [ -f "$RUNTESTDIR/../rust/target/release/rhg" ]; then
-  >   "$RUNTESTDIR/../rust/target/release/rhg" "$@"
+  > if [ -f "$RHG" ]; then
+  >   "$RHG" "$@"
   > else
   >   echo "skipped: Cannot find rhg. Try to run cargo build in rust/rhg."
   >   exit 80
   > fi
   > }
 
 Unimplemented command
   $ rhg unimplemented-command
   unsupported feature: error: Found argument 'unimplemented-command' which wasn't expected, or isn't valid in this context
   
   USAGE:
       rhg [OPTIONS] <SUBCOMMAND>
   
   For more information try --help
   
   [252]
   $ rhg unimplemented-command --config rhg.on-unsupported=abort-silent
   [252]
 
 Finding root
   $ rhg root
   abort: no repository found in '$TESTTMP' (.hg not found)!
   [255]
 
   $ hg init repository
   $ cd repository
   $ rhg root
   $TESTTMP/repository
 
 Reading and setting configuration
   $ echo "[ui]" >> $HGRCPATH
   $ echo "username = user1" >> $HGRCPATH
   $ rhg config ui.username
   user1
   $ echo "[ui]" >> .hg/hgrc
   $ echo "username = user2" >> .hg/hgrc
   $ rhg config ui.username
   user2
   $ rhg --config ui.username=user3 config ui.username
   user3
 
 Unwritable file descriptor
   $ rhg root > /dev/full
   abort: No space left on device (os error 28)
   [255]
 
 Deleted repository
   $ rm -rf `pwd`
   $ rhg root
   abort: $ENOENT$: current directory
   [255]
 
 Listing tracked files
   $ cd $TESTTMP
   $ hg init repository
   $ cd repository
   $ for i in 1 2 3; do
   >   echo $i >> file$i
   >   hg add file$i
   > done
   > hg commit -m "commit $i" -q
 
 Listing tracked files from root
   $ rhg files
   file1
   file2
   file3
 
 Listing tracked files from subdirectory
   $ mkdir -p path/to/directory
   $ cd path/to/directory
   $ rhg files
   ../../../file1
   ../../../file2
   ../../../file3
 
 Listing tracked files through broken pipe
   $ rhg files | head -n 1
   ../../../file1
 
 Debuging data in inline index
   $ cd $TESTTMP
   $ rm -rf repository
   $ hg init repository
   $ cd repository
   $ for i in 1 2 3 4 5 6; do
   >   echo $i >> file-$i
   >   hg add file-$i
   >   hg commit -m "Commit $i" -q
   > done
   $ rhg debugdata -c 2
   8d0267cb034247ebfa5ee58ce59e22e57a492297
   test
   0 0
   file-3
   
   Commit 3 (no-eol)
   $ rhg debugdata -m 2
   file-1\x00b8e02f6433738021a065f94175c7cd23db5f05be (esc)
   file-2\x005d9299349fc01ddd25d0070d149b124d8f10411e (esc)
   file-3\x002661d26c649684b482d10f91960cc3db683c38b4 (esc)
 
 Debuging with full node id
   $ rhg debugdata -c `hg log -r 0 -T '{node}'`
   d1d1c679d3053e8926061b6f45ca52009f011e3f
   test
   0 0
   file-1
   
   Commit 1 (no-eol)
 
 Specifying revisions by changeset ID
   $ hg log -T '{node}\n'
   c6ad58c44207b6ff8a4fbbca7045a5edaa7e908b
   d654274993d0149eecc3cc03214f598320211900
   f646af7e96481d3a5470b695cf30ad8e3ab6c575
   cf8b83f14ead62b374b6e91a0e9303b85dfd9ed7
   91c6f6e73e39318534dc415ea4e8a09c99cd74d6
   6ae9681c6d30389694d8701faf24b583cf3ccafe
   $ rhg files -r cf8b83
   file-1
   file-2
   file-3
   $ rhg cat -r cf8b83 file-2
   2
   $ rhg cat -r c file-2
   abort: ambiguous revision identifier c
   [255]
   $ rhg cat -r d file-2
   2
 
 Cat files
   $ cd $TESTTMP
   $ rm -rf repository
   $ hg init repository
   $ cd repository
   $ echo "original content" > original
   $ hg add original
   $ hg commit -m "add original" original
   $ rhg cat -r 0 original
   original content
 Cat copied file should not display copy metadata
   $ hg copy original copy_of_original
   $ hg commit -m "add copy of original"
   $ rhg cat -r 1 copy_of_original
   original content
 
+Fallback to Python
+  $ rhg cat original
+  unsupported feature: `rhg cat` without `--rev` / `-r`
+  [252]
+  $ FALLBACK="--config rhg.on-unsupported=fallback"
+  $ rhg cat original $FALLBACK
+  original content
+
+  $ rhg cat original $FALLBACK --config rhg.fallback-executable=false
+  [1]
+
+  $ rhg cat original $FALLBACK --config rhg.fallback-executable=hg-non-existent
+  tried to fall back to a 'hg-non-existent' sub-process but got error $ENOENT$
+  unsupported feature: `rhg cat` without `--rev` / `-r`
+  [252]
+
+  $ rhg cat original $FALLBACK --config rhg.fallback-executable="$RHG"
+  Blocking recursive fallback. The 'rhg.fallback-executable = */rust/target/release/rhg' config points to `rhg` itself. (glob)
+  unsupported feature: `rhg cat` without `--rev` / `-r`
+  [252]
+
 Requirements
   $ rhg debugrequirements
   dotencode
   fncache
   generaldelta
   revlogv1
   sparserevlog
   store
 
   $ echo indoor-pool >> .hg/requires
   $ rhg files
   unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
   [252]
 
   $ rhg cat -r 1 copy_of_original
   unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
   [252]
 
   $ rhg debugrequirements
   unsupported feature: repository requires feature unknown to this Mercurial: indoor-pool
   [252]
 
   $ echo -e '\xFF' >> .hg/requires
   $ rhg debugrequirements
   abort: corrupted repository: parse error in 'requires' file
   [255]
 
 Persistent nodemap
   $ cd $TESTTMP
   $ rm -rf repository
   $ hg init repository
   $ cd repository
   $ rhg debugrequirements | grep nodemap
   [1]
   $ hg debugbuilddag .+5000 --overwritten-file --config "storage.revlog.nodemap.mode=warn"
   $ hg id -r tip
   c3ae8dec9fad tip
   $ ls .hg/store/00changelog*
   .hg/store/00changelog.d
   .hg/store/00changelog.i
   $ rhg files -r c3ae8dec9fad
   of
 
   $ cd $TESTTMP
   $ rm -rf repository
   $ hg --config format.use-persistent-nodemap=True init repository
   $ cd repository
   $ rhg debugrequirements | grep nodemap
   persistent-nodemap
   $ hg debugbuilddag .+5000 --overwritten-file --config "storage.revlog.nodemap.mode=warn"
   $ hg id -r tip
   c3ae8dec9fad tip
   $ ls .hg/store/00changelog*
   .hg/store/00changelog-*.nd (glob)
   .hg/store/00changelog.d
   .hg/store/00changelog.i
   .hg/store/00changelog.n
 
 Specifying revisions by changeset ID
   $ rhg files -r c3ae8dec9fad
   of
   $ rhg cat -r c3ae8dec9fad of
   r5000
 
 Crate a shared repository
 
   $ echo "[extensions]"      >> $HGRCPATH
   $ echo "share = "          >> $HGRCPATH
 
   $ cd $TESTTMP
   $ hg init repo1
   $ echo a > repo1/a
   $ hg -R repo1 commit -A -m'init'
   adding a
 
   $ hg share repo1 repo2
   updating working directory
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 And check that basic rhg commands work with sharing
 
   $ rhg files -R repo2
   repo2/a
   $ rhg -R repo2 cat -r 0 repo2/a
   a
 
 Same with relative sharing
 
   $ hg share repo2 repo3 --relative
   updating working directory
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
   $ rhg files -R repo3
   repo3/a
   $ rhg -R repo3 cat -r 0 repo3/a
   a
 
 Same with share-safe
 
   $ echo "[format]"         >> $HGRCPATH
   $ echo "use-share-safe = True" >> $HGRCPATH
 
   $ cd $TESTTMP
   $ hg init repo4
   $ cd repo4
   $ echo a > a
   $ hg commit -A -m'init'
   adding a
 
   $ cd ..
   $ hg share repo4 repo5
   updating working directory
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
 And check that basic rhg commands work with sharing
 
   $ cd repo5
   $ rhg files
   a
   $ rhg cat -r 0 a
   a
 
 The blackbox extension is supported
 
   $ echo "[extensions]" >> $HGRCPATH
   $ echo "blackbox =" >> $HGRCPATH
   $ echo "[blackbox]" >> $HGRCPATH
   $ echo "maxsize = 1" >> $HGRCPATH
   $ rhg files > /dev/null
   $ cat .hg/blackbox.log
   ????/??/?? ??:??:??.??? * @d3873e73d99ef67873dac33fbcc66268d5d2b6f4 (*)> (rust) files exited 0 after 0.??? seconds (glob)
   $ cat .hg/blackbox.log.1
   ????/??/?? ??:??:??.??? * @d3873e73d99ef67873dac33fbcc66268d5d2b6f4 (*)> (rust) files (glob)