inline-changelog: fix a critical bug in write_pending that delete data

Since a93e52f0b6ff we no longer use inline-revlog for the changelog. The goal there was to
solve the lack of testing for the two variants (inline vs split) and reduce the
complexity of the interaction with "diverted-write" on the changelog level.

However many existing repository still have inline-changelog and we
automatically move them to normal revlog as soon as we have the chances.

Unfortunately This conversion is buggy and can result in the destruction of the
changelog.i if hook triggers the "write pending" mechanism.

The bugs comes from the "revlog splitting" logic and the "write_pending" logic
stepping over each other. Ironically the change in a93e52f0b6ff aims at no
longer having this kind of problem.

This changesets fix this issue and add associated tests.

Fixing this reveal that the transaction hooks end up not seeing the pending
transaction content, because the name is not right ("changelog.i.s.a" instead of
"changelog.i.s") we fix this in the next changeset.

tests/test-split-legacy-inline-changelog.t

@@ -0,0 +1,307 b''
+======================================================
+Test operation on repository with an inlined changelog
+======================================================
+
+Inlined revlog has been a bag of complexity for a long time and the combination
+with special transaction logic on the changelog was a long source of bugs
+poorly covered by the test suites.
+
+We stopped doing any usage of inlined-revlog for changelog in a93e52f0b6ff,
+upgrading legacy inlined version as soon as possible when we see them. However
+this Mercurial does not produce such inlined-changelog that case is very poorly
+covered in the test suites. This test file aims at covering these cases.
+
+Double checking test data
+=========================
+
+We should have a repository around
+
+  $ mkdir sanity-check
+  $ cd sanity-check
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ cd inlined-changelog
+  $ hg root
+  $TESTTMP/sanity-check/inlined-changelog
+
+The repository should not be corrupted initially
+
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 1 changesets with 1 changes to 1 files
+
+The changelog of that repository MUST be inlined
+
+  $ hg debugrevlog -c | grep -E '^flags\b'
+  flags  : inline
+
+Touching that repository MUST split that inlined changelog
+
+  $ hg branch foo --quiet
+  $ hg commit -m foo --quiet
+  $ hg debugrevlog -c | grep -E '^flags\b'
+  flags  : (none)
+
+  $ cd ../..
+
+Test doing a simple commit
+==========================
+
+Simple commit
+-------------
+
+  $ mkdir simple-commit
+  $ cd simple-commit
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ cd inlined-changelog
+  $ hg up --quiet
+  $ hg log -GT '[{rev}] {desc}\n'
+  @  [0] first commit
+  
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -GT '[{rev}] {desc}\n'
+  @  [1] second changeset
+  |
+  o  [0] first commit
+  
+  $ cd ../..
+
+Simple commit with a pretxn hook configured
+-------------------------------------------
+
+Before 6.7.3 this used to delete the changelog index
+
+  $ mkdir pretxnclose-commit
+  $ cd pretxnclose-commit
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ cat >> inlined-changelog/.hg/hgrc <<EOF
+  > [hooks]
+  > pretxnclose=hg log -r tip -T "pre-txn tip rev: {rev}\n"
+  > EOF
+  $ cd inlined-changelog
+  $ hg up --quiet
+  $ hg log -GT '[{rev}] {desc}\n'
+  @  [0] first commit
+  
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  pre-txn tip rev: 1 (missing-correct-output !)
+  warning: ignoring unknown working parent 11b63e930bf2! (known-bad-output !)
+  pre-txn tip rev: 0 (known-bad-output !)
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -GT '[{rev}] {desc}\n'
+  @  [1] second changeset
+  |
+  o  [0] first commit
+  
+  $ cd ../..
+
+Test pushing to a repository with a repository revlog
+=====================================================
+
+Simple local push
+-----------------
+
+  $ mkdir simple-local-push
+  $ cd simple-local-push
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [0] first commit
+
+  $ hg clone --pull inlined-changelog client
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  new changesets 827f11bfd362
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ cd client
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  $ hg push
+  pushing to $TESTTMP/*/inlined-changelog (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  $ cd ..
+
+  $ hg verify -R inlined-changelog
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [1] second changeset
+  [0] first commit
+  $ cd ..
+
+Simple local push with a pretxnchangegroup hook
+-----------------------------------------------
+
+Before 6.7.3 this used to delete the server changelog
+
+  $ mkdir pretxnchangegroup-local-push
+  $ cd pretxnchangegroup-local-push
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ cat >> inlined-changelog/.hg/hgrc <<EOF
+  > [hooks]
+  > pretxnchangegroup=hg log -r tip -T "pre-txn tip rev: {rev}\n"
+  > EOF
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [0] first commit
+
+  $ hg clone --pull inlined-changelog client
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  new changesets 827f11bfd362
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ cd client
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  $ hg push
+  pushing to $TESTTMP/*/inlined-changelog (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  pre-txn tip rev: 1 (missing-correct-output !)
+  pre-txn tip rev: 0 (known-bad-output !)
+  added 1 changesets with 1 changes to 1 files
+  $ cd ..
+
+  $ hg verify -R inlined-changelog
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [1] second changeset
+  [0] first commit
+  $ cd ..
+
+Simple ssh push
+-----------------
+
+  $ mkdir simple-ssh-push
+  $ cd simple-ssh-push
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [0] first commit
+
+  $ hg clone ssh://user@dummy/"`pwd`"/inlined-changelog client
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  new changesets 827f11bfd362
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ cd client
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  $ hg push
+  pushing to ssh://user@dummy/$TESTTMP/simple-ssh-push/inlined-changelog
+  searching for changes
+  remote: adding changesets
+  remote: adding manifests
+  remote: adding file changes
+  remote: added 1 changesets with 1 changes to 1 files
+  $ cd ..
+
+  $ hg verify -R inlined-changelog
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [1] second changeset
+  [0] first commit
+  $ cd ..
+
+Simple ssh push with a pretxnchangegroup hook
+-----------------------------------------------
+
+Before 6.7.3 this used to delete the server changelog
+
+  $ mkdir pretxnchangegroup-ssh-push
+  $ cd pretxnchangegroup-ssh-push
+  $ tar xf $TESTDIR/bundles/inlined-changelog.tar
+  $ cat >> inlined-changelog/.hg/hgrc <<EOF
+  > [hooks]
+  > pretxnchangegroup=hg log -r tip -T "pre-txn tip rev: {rev}\n"
+  > EOF
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [0] first commit
+
+  $ hg clone ssh://user@dummy/"`pwd`"/inlined-changelog client
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  new changesets 827f11bfd362
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ cd client
+  $ echo b > b
+  $ hg add b
+  $ hg commit -m "second changeset"
+  $ hg push
+  pushing to ssh://user@dummy/$TESTTMP/pretxnchangegroup-ssh-push/inlined-changelog
+  searching for changes
+  remote: adding changesets
+  remote: adding manifests
+  remote: adding file changes
+  remote: pre-txn tip rev: 1 (missing-correct-output !)
+  remote: pre-txn tip rev: 0 (known-bad-output !)
+  remote: added 1 changesets with 1 changes to 1 files
+  $ cd ..
+
+  $ hg verify -R inlined-changelog
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checking dirstate
+  checked 2 changesets with 2 changes to 2 files
+  $ hg log -R inlined-changelog -T '[{rev}] {desc}\n'
+  [1] second changeset
+  [0] first commit
+  $ cd ..

mercurial/changelog.py

@@ -357,8 +357,10 b' class changelog(revlog.revlog):'
             if new_index is not None:
                 self._indexfile = new_index
                 tr.registertmp(new_index)
-        tr.addpending(b'cl-%i' % id(self), self._writepending)
-        tr.addfinalize(b'cl-%i' % id(self), self._finalize)
+        # use "000" as prefix to make sure we run before the spliting of legacy
+        # inline changelog..
+        tr.addpending(b'000-cl-%i' % id(self), self._writepending)
+        tr.addfinalize(b'000-cl-%i' % id(self), self._finalize)
 
     def _finalize(self, tr):
         """finalize index updates"""

mercurial/revlog.py

@@ -2895,10 +2895,13 b' class revlog:'
                 maybe_self._inner.index_file = old_index_file_path
 
         tr.registertmp(new_index_file_path)
+        # we use 001 here to make this this happens after the finalisation of
+        # pending changelog write (using 000). Otherwise the two finalizer
+        # would step over each other and delete the changelog.i file.
         if self.target[1] is not None:
-            callback_id = b'000-revlog-split-%d-%s' % self.target
+            callback_id = b'001-revlog-split-%d-%s' % self.target
         else:
-            callback_id = b'000-revlog-split-%d' % self.target[0]
+            callback_id = b'001-revlog-split-%d' % self.target[0]
         tr.addfinalize(callback_id, finalize_callback)
         tr.addabort(callback_id, abort_callback)
 

tests/test-bookmarks.t

@@ -1082,9 +1082,9 b' ensure changelog is written before bookm'
   >            time.sleep(0.5)
   >        if os.path.exists(b"$TESTTMP/unpause"):
   >            os.remove(b"$TESTTMP/unpause")
-  >    # It is important that this finalizer start with 'a', so it runs before
-  >    # the changelog finalizer appends to the changelog.
-  >    tr.addfinalize(b'a-sleep', sleep)
+  >    # It is important that this finalizer start with '000-a', so it runs
+  >    # before the changelog finalizer appends to the changelog.
+  >    tr.addfinalize(b'000-a-sleep', sleep)
   >    return tr
   > 
   > def extsetup(ui):