largefiles: update largefiles even if rebase is aborted by conflict

Before this patch, largefiles in the working directory aren't updated
correctly, if rebase is aborted by conflict. This prevents users from
viewing appropriate largefiles while resolving conflicts.

While rebase, largefiles in the working directory are updated only at
successful committing in the special code path of
"lfilesrepo.commit()".

To update largefiles even if rebase is aborted by conflict, this patch
centralizes the logic of updating largefiles in the working directory
into the "mergeupdate" wrapping "merge.update".


This is a temporary way to fix with less changes. For fundamental
resolution of this kind of problems in the future, largefiles in the
working directory should be updated with other (normal) files
simultaneously while "merge.update" execution: maybe by hooking
"applyupdates".

"Action list based updating" introduced by hooking "applyupdates" will
also improve performance of updating, because it automatically
decreases target files to be checked.


Just after this patch, there are some improper things in "Case 0" code
path of "lfilesrepo.commit()":

  - "updatelfiles" invocation is redundant for rebase
  - detailed comment doesn't meet to rebase behavior

These will be resolved after the subsequent patch for transplant,
because this code path is shared with transplant.


Even though replacing "merge.update" in rebase extension by "hg.merge"
can also avoid this problem, this patch chooses centralizing the logic
into "mergeupdate", because:

  - "merge.update" invocation in rebase extension can't be directly
    replaced by "hg.merge", because:

    - rebase requires some extra arguments, which "hg.merge" doesn't
      take (e.g. "ancestor")

    - rebase doesn't require statistics information forcibly displayed
      in "hg.merge"

  - introducing "mergeupdate" can resolve also problem of some other
    code paths directly using "merge.update"

    largefiles in the working directory aren't updated regardless of
    the result of commands below, before this patch:

    - backout (for revisions other than the parent revision of the
      working directory without "--merge")

    - graft

    - histedit (for revisions other than the parent of the working
      directory


When "partial" is specified, "merge.update" doesn't update dirstate
entries for standins, even though standins themselves are updated.

In this case, "normallookup" should be used to mark largefiles as
"possibly dirty" forcibly, because applying "normal" on lfdirstate
treats them as "clean" unexpectedly.

This is reason why "normallookup=partial" is specified for
"lfcommands.updatelfiles".


This patch doesn't test "hg rebase --continue", because it doesn't
work correctly if largefiles in the working directory are modified
manually while resolving conflicts. This will be fixed in the next
step of refactoring for largefiles.

All changes of tests/*.t files other than test-largefiles-update.t in
this patch come from invoking "updatelfiles" not after but before
statistics output of "hg.update", "hg.clean" and "hg.merge".

hgext/largefiles/overrides.py

@@ -714,47 +714,6 b' def overriderevert(orig, ui, repo, *pats'
     finally:
         wlock.release()
 
-def hgupdaterepo(orig, repo, node, overwrite):
-    wlock = repo.wlock()
-    try:
-        return _hgupdaterepo(orig, repo, node, overwrite)
-    finally:
-        wlock.release()
-
-def _hgupdaterepo(orig, repo, node, overwrite):
-    if not overwrite:
-        # update standins for linear merge
-        lfdirstate = lfutil.openlfdirstate(repo.ui, repo)
-        s = lfdirstate.status(match_.always(repo.root, repo.getcwd()),
-                              [], False, False, False)
-        unsure, modified, added = s[:3]
-        for lfile in unsure + modified + added:
-            lfutil.updatestandin(repo, lfutil.standin(lfile))
-
-        # Only call updatelfiles on the standins that have changed to save time
-        oldstandins = lfutil.getstandinsstate(repo)
-
-    result = orig(repo, node, overwrite)
-
-    filelist = None
-    if not overwrite:
-        newstandins = lfutil.getstandinsstate(repo)
-        filelist = lfutil.getlfilestoupdate(oldstandins, newstandins)
-    lfcommands.updatelfiles(repo.ui, repo, filelist=filelist)
-    return result
-
-def hgmerge(orig, repo, node, force=None, remind=True):
-    wlock = repo.wlock()
-    try:
-        return _hgmerge(orig, repo, node, force, remind)
-    finally:
-        wlock.release()
-
-def _hgmerge(orig, repo, node, force, remind):
-    result = orig(repo, node, force, remind)
-    lfcommands.updatelfiles(repo.ui, repo)
-    return result
-
 # When we rebase a repository with remotely changed largefiles, we need to
 # take some extra care so that the largefiles are correctly updated in the
 # working copy
@@ -1305,3 +1264,57 b' def mercurialsinkbefore(orig, sink):'
 def mercurialsinkafter(orig, sink):
     sink.repo._isconverting = False
     orig(sink)
+
+def mergeupdate(orig, repo, node, branchmerge, force, partial,
+                *args, **kwargs):
+    wlock = repo.wlock()
+    try:
+        # branch |       |         |
+        #  merge | force | partial | action
+        # -------+-------+---------+--------------
+        #    x   |   x   |    x    | linear-merge
+        #    o   |   x   |    x    | branch-merge
+        #    x   |   o   |    x    | overwrite (as clean update)
+        #    o   |   o   |    x    | force-branch-merge (*1)
+        #    x   |   x   |    o    |   (*)
+        #    o   |   x   |    o    |   (*)
+        #    x   |   o   |    o    | overwrite (as revert)
+        #    o   |   o   |    o    |   (*)
+        #
+        # (*) don't care
+        # (*1) deprecated, but used internally (e.g: "rebase --collapse")
+
+        linearmerge = not branchmerge and not force and not partial
+
+        if linearmerge or (branchmerge and force and not partial):
+            # update standins for linear-merge or force-branch-merge,
+            # because largefiles in the working directory may be modified
+            lfdirstate = lfutil.openlfdirstate(repo.ui, repo)
+            s = lfdirstate.status(match_.always(repo.root, repo.getcwd()),
+                                  [], False, False, False)
+            unsure, modified, added = s[:3]
+            for lfile in unsure + modified + added:
+                lfutil.updatestandin(repo, lfutil.standin(lfile))
+
+        if linearmerge:
+            # Only call updatelfiles on the standins that have changed
+            # to save time
+            oldstandins = lfutil.getstandinsstate(repo)
+
+        result = orig(repo, node, branchmerge, force, partial, *args, **kwargs)
+
+        filelist = None
+        if linearmerge:
+            newstandins = lfutil.getstandinsstate(repo)
+            filelist = lfutil.getlfilestoupdate(oldstandins, newstandins)
+
+        # suppress status message while automated committing
+        printmessage = not (getattr(repo, "_isrebasing", False) or
+                            getattr(repo, "_istransplanting", False))
+        lfcommands.updatelfiles(repo.ui, repo, filelist=filelist,
+                                printmessage=printmessage,
+                                normallookup=partial)
+
+        return result
+    finally:
+        wlock.release()

hgext/largefiles/uisetup.py

@@ -101,6 +101,8 b' def uisetup(ui):'
                                     overrides.overridecalculateupdates)
     entry = extensions.wrapfunction(merge, 'recordupdates',
                                     overrides.mergerecordupdates)
+    entry = extensions.wrapfunction(merge, 'update',
+                                    overrides.mergeupdate)
     entry = extensions.wrapfunction(filemerge, 'filemerge',
                                     overrides.overridefilemerge)
     entry = extensions.wrapfunction(cmdutil, 'copy',
@@ -117,9 +119,6 b' def uisetup(ui):'
     entry = extensions.wrapfunction(commands, 'revert',
                                     overrides.overriderevert)
 
-    extensions.wrapfunction(hg, 'updaterepo', overrides.hgupdaterepo)
-    extensions.wrapfunction(hg, 'merge', overrides.hgmerge)
-
     extensions.wrapfunction(archival, 'archive', overrides.overridearchive)
     extensions.wrapfunction(subrepo.hgsubrepo, 'archive',
                             overrides.hgsubrepoarchive)

tests/test-issue3084.t

@@ -29,10 +29,10 b' Normal file in the working copy, keeping'
 
   $ echo "n" | hg merge --config ui.interactive=Yes
   remote turned local normal file foo into a largefile
-  use (l)argefile or keep (n)ormal file? 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  use (l)argefile or keep (n)ormal file? getting changed largefiles
+  0 largefiles updated, 0 removed
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
 
   $ hg status
   $ cat foo
@@ -43,10 +43,10 b' Normal file in the working copy, keeping'
   $ hg update -q -C
   $ echo "l" | hg merge --config ui.interactive=Yes
   remote turned local normal file foo into a largefile
-  use (l)argefile or keep (n)ormal file? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  use (l)argefile or keep (n)ormal file? getting changed largefiles
+  1 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
 
   $ hg status
   M foo
@@ -71,10 +71,10 b' Largefile in the working copy, keeping t'
   $ hg update -q -C -r 1
   $ echo "n" | hg merge --config ui.interactive=Yes
   remote turned local largefile foo into a normal file
-  keep (l)argefile or use (n)ormal file? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  keep (l)argefile or use (n)ormal file? getting changed largefiles
+  0 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
 
   $ hg status
   M foo
@@ -99,10 +99,10 b' Largefile in the working copy, keeping t'
   $ hg update -q -C -r 1
   $ echo "l" | hg merge --config ui.interactive=Yes
   remote turned local largefile foo into a normal file
-  keep (l)argefile or use (n)ormal file? 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  keep (l)argefile or use (n)ormal file? getting changed largefiles
+  1 largefiles updated, 0 removed
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
 
   $ hg status
 
@@ -206,10 +206,10 b' Ancestor: normal Parent: normal= Paren'
 
   $ hg up -Cqr normal=
   $ hg merge -r large
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -217,10 +217,10 b' swap'
 
   $ hg up -Cqr large
   $ hg merge -r normal=
+  getting changed largefiles
+  0 largefiles updated, 0 removed
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -233,10 +233,10 b' Ancestor: normal Parent: normal2 Paren'
   use (c)hanged version or (d)elete? c
   remote turned local normal file f into a largefile
   use (l)argefile or keep (n)ormal file? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -244,20 +244,20 b' Ancestor: normal Parent: normal2 Paren'
   $ ( echo c; echo n ) | hg merge -r large --config ui.interactive=Yes
   local changed f which remote deleted
   use (c)hanged version or (d)elete? remote turned local normal file f into a largefile
-  use (l)argefile or keep (n)ormal file? 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  use (l)argefile or keep (n)ormal file? getting changed largefiles
+  0 largefiles updated, 0 removed
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   normal2
 
   $ hg up -Cqr normal2
   $ echo d | hg merge -r large --config ui.interactive=Yes
   local changed f which remote deleted
-  use (c)hanged version or (d)elete? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  use (c)hanged version or (d)elete? getting changed largefiles
+  1 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -269,10 +269,10 b' swap'
   use (c)hanged version or leave (d)eleted? c
   remote turned local largefile f into a normal file
   keep (l)argefile or use (n)ormal file? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -280,20 +280,20 b' swap'
   $ ( echo c; echo n ) | hg merge -r normal2 --config ui.interactive=Yes
   remote changed f which local deleted
   use (c)hanged version or leave (d)eleted? remote turned local largefile f into a normal file
-  keep (l)argefile or use (n)ormal file? 2 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  keep (l)argefile or use (n)ormal file? getting changed largefiles
+  0 largefiles updated, 0 removed
+  2 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   normal2
 
   $ hg up -Cqr large
   $ echo d | hg merge -r normal2 --config ui.interactive=Yes
   remote changed f which local deleted
-  use (c)hanged version or leave (d)eleted? 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  use (c)hanged version or leave (d)eleted? getting changed largefiles
+  0 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   large
 
@@ -301,10 +301,10 b' Ancestor: large Parent: large= Paren'
 
   $ hg up -Cqr large=
   $ hg merge -r normal
+  getting changed largefiles
+  0 largefiles updated, 0 removed
   1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   normal
 
@@ -326,20 +326,20 b' Ancestor: large Parent: large2 Paren'
   use (c)hanged version or (d)elete? c
   remote turned local largefile f into a normal file
   keep (l)argefile or use (n)ormal file? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   0 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large2
 
   $ hg up -Cqr large2
   $ echo d | hg merge -r normal --config ui.interactive=Yes
   local changed .hglf/f which remote deleted
-  use (c)hanged version or (d)elete? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  use (c)hanged version or (d)elete? getting changed largefiles
+  0 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cat f
   normal
 
@@ -351,10 +351,10 b' swap'
   use (c)hanged version or leave (d)eleted? c
   remote turned local normal file f into a largefile
   use (l)argefile or keep (n)ormal file? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   2 files updated, 0 files merged, 1 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f
   large2
 

tests/test-largefiles-misc.t

@@ -659,10 +659,10 b" merge action 'd' for 'local renamed dire"
   R d1/f
   $ hg merge
   merging d2/f and d1/f to d2/f
+  getting changed largefiles
+  0 largefiles updated, 0 removed
   1 files updated, 1 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  0 largefiles updated, 0 removed
   $ cd ..
 
 
@@ -725,10 +725,10 b' Merge conflicts:'
   ancestor was 09d2af8dd22201dd8d48e5dcfcaed281ff9422c7
   keep (l)ocal e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e or
   take (o)ther 7448d8798a4380162d4b56f9b452e2f6f9e24e7a? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   0 files updated, 4 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ cat f-different
   1
   $ cat f-same

tests/test-largefiles-update.t

@@ -36,10 +36,10 b' Test that "hg merge" updates largefiles '
   $ cat .hglf/large1
   4669e532d5b2c093a78eca010077e708a071bb64
   $ hg merge --config debug.dirstate.delaywrite=2
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ hg status -A large1
   M large1
   $ cat large1
@@ -67,10 +67,10 b' Test that "hg merge" updates largefiles '
   take (o)ther 58e24f733a964da346e2407a2bee99d9001184f5? merging normal1
   warning: conflicts during merge.
   merging normal1 incomplete! (edit conflicts, then use 'hg resolve --mark')
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   0 files updated, 1 files merged, 0 files removed, 1 files unresolved
   use 'hg resolve' to retry unresolved file merges or 'hg update -C .' to abandon
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   [1]
   $ hg status -A large1
   M large1
@@ -456,3 +456,33 b' Test that the internal linear merging wo'
   d7591fe9be0f6227d90bddf3e4f52ff41fc1f544
 
   $ cd ..
+  $ cd repo
+
+Test that rebase updates largefiles in the working directory even if
+it is aborted by conflict.
+
+  $ hg update -q -C 3
+  $ cat .hglf/large1
+  e5bb990443d6a92aaf7223813720f7566c9dd05b
+  $ cat large1
+  large1 in #3
+  $ hg rebase -s 1 -d 3 --keep --config ui.interactive=True <<EOF
+  > o
+  > EOF
+  largefile large1 has a merge conflict
+  ancestor was 4669e532d5b2c093a78eca010077e708a071bb64
+  keep (l)ocal e5bb990443d6a92aaf7223813720f7566c9dd05b or
+  take (o)ther 58e24f733a964da346e2407a2bee99d9001184f5? merging normal1
+  warning: conflicts during merge.
+  merging normal1 incomplete! (edit conflicts, then use 'hg resolve --mark')
+  unresolved conflicts (see hg resolve, then hg rebase --continue)
+  [1]
+  $ cat .hglf/large1
+  58e24f733a964da346e2407a2bee99d9001184f5
+  $ cat large1
+  large1 in #1
+
+  $ hg rebase -q --abort
+  rebase aborted
+
+  $ cd ..

tests/test-largefiles.t

@@ -1603,11 +1603,11 b' Merge with revision with missing largefi'
   A f
   created new head
   $ hg merge -r 6
-  4 files updated, 0 files merged, 0 files removed, 0 files unresolved
-  (branch merge, don't forget to commit)
   getting changed largefiles
   large3: largefile 7838695e10da2bb75ac1156565f40a2595fa2fa0 not available from file:/*/$TESTTMP/d (glob)
   1 largefiles updated, 0 removed
+  4 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
 
   $ hg rollback -q
   $ hg up -Cq
@@ -1661,10 +1661,10 b' correctly.'
   ancestor was 971fb41e78fea4f8e0ba5244784239371cb00591
   keep (l)ocal d846f26643bfa8ec210be40cc93cc6b7ff1128ea or
   take (o)ther e166e74c7303192238d60af5a9c4ce9bef0b7928? l
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   3 files updated, 1 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ hg commit -m "Merge repos e and f"
   Invoking status precommit hook
   M normal3
@@ -1695,10 +1695,10 b' Test status after merging with a branch '
   M normal3
   created new head
   $ hg merge
+  getting changed largefiles
+  1 largefiles updated, 0 removed
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   (branch merge, don't forget to commit)
-  getting changed largefiles
-  1 largefiles updated, 0 removed
   $ hg status
   M large
 
@@ -1742,7 +1742,7 b' Test that transplanting a largefile chan'
   adding file changes
   added 1 changesets with 2 changes to 2 files
   getting changed largefiles
-  1 largefiles updated, 0 removed
+  0 largefiles updated, 0 removed
   $ hg log --template '{rev}:{node|short}  {desc|firstline}\n'
   9:598410d3eb9a  modify normal file largefile in repo d
   8:a381d2c8c80e  modify normal file and largefile in repo b