test-lfs: add tests around corrupted lfs objects

These are mostly tests against file:// based remote stores, because that's what
we have the most control over.

The test uploading a corrupt blob to lfs-test-server demonstrates an overly
broad exception handler in the retry loop.  A corrupt blob is actually
transferred in a download, but eventually caught when it is accessed (only after
it leaves the corrupt file in a couple places locally).  I don't think we want
to trust random 3rd party implementations, and this would be a problem if there
were a `debuglfsdownload` command that simply cached the files.  And given the
cryptic errors, we should probably validate the file hash locally before
uploading, and also after downloading.

tests/test-lfs-test-server.t

@@ -105,6 +105,55 b' Clear the cache to force a download'
   lfs: found 37a65ab78d5ecda767e8622c248b5dbff1e68b1678ab0e730d5eb8601ec8ad19 in the local lfs store
   3 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
+Test a corrupt file download, but clear the cache first to force a download
+
+XXX: ideally, the validation would occur before polluting the usercache and
+local store, with a clearer error message.
+
+  $ rm -rf `hg config lfs.usercache`
+  $ cp $TESTTMP/lfs-content/d1/1e/1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 blob
+  $ echo 'damage' > $TESTTMP/lfs-content/d1/1e/1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998
+  $ rm ../repo1/.hg/store/lfs/objects/d1/1e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998
+  $ rm ../repo1/*
+  $ hg --repo ../repo1 update -C tip -v
+  resolving manifests
+  getting a
+  lfs: found 31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b in the local lfs store
+  getting b
+  lfs: found 31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b in the local lfs store
+  getting c
+  lfs: downloading d11e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 (19 bytes)
+  lfs: adding d11e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 to the usercache
+  lfs: processed: d11e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998
+  lfs: found d11e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 in the local lfs store
+  abort: integrity check failed on data/c.i:0!
+  [255]
+
+BUG: the corrupted blob was added to the usercache and local store
+
+  $ cat ../repo1/.hg/store/lfs/objects/d1/1e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 | $TESTDIR/f --sha256
+  sha256=fa82ca222fc9813afad3559637960bf311170cdd80ed35287f4623eb2320a660
+  $ cat `hg config lfs.usercache`/d1/1e1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998 | $TESTDIR/f --sha256
+  sha256=fa82ca222fc9813afad3559637960bf311170cdd80ed35287f4623eb2320a660
+  $ cp blob $TESTTMP/lfs-content/d1/1e/1a642b60813aee592094109b406089b8dff4cb157157f753418ec7857998
+
+Test a corrupted file upload
+
+  $ echo 'another lfs blob' > b
+  $ hg ci -m 'another blob'
+  $ echo 'damage' > .hg/store/lfs/objects/e6/59058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0
+  $ hg push -v ../repo1
+  pushing to ../repo1
+  searching for changes
+  lfs: uploading e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0 (17 bytes)
+  lfs: failed: LfsRemoteError('HTTP error: HTTP Error 500: Internal Server Error (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)',) (remaining retry 5)
+  lfs: failed: LfsRemoteError('HTTP error: HTTP Error 404: Not Found (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)',) (remaining retry 4)
+  lfs: failed: LfsRemoteError('HTTP error: HTTP Error 404: Not Found (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)',) (remaining retry 3)
+  lfs: failed: LfsRemoteError('HTTP error: HTTP Error 404: Not Found (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)',) (remaining retry 2)
+  lfs: failed: LfsRemoteError('HTTP error: HTTP Error 404: Not Found (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)',) (remaining retry 1)
+  abort: HTTP error: HTTP Error 404: Not Found (oid=e659058e26b07b39d2a9c7145b3f99b41f797b6621c8076600e9cb7ee88291f0, action=upload)!
+  [255]
+
 Check error message when the remote missed a blob:
 
   $ echo FFFFF > b

tests/test-lfs.t

@@ -605,6 +605,193 b' remote store.'
   added 3 changesets with 2 changes to 1 files
   $ hg -R repo14 -q verify
 
+Test damaged file scenarios.  (This also damages the usercache because of the
+hardlinks.)
+
+  $ echo 'damage' >> repo5/.hg/store/lfs/objects/66/100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e
+
+Repo with damaged lfs objects in any revision will fail verification.
+
+  $ hg -R repo5 verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+   l@1: unpacking 46a2f24864bc: integrity check failed on data/l.i:0
+   large@0: unpacking 2c531e0992ff: integrity check failed on data/large.i:0
+  4 files, 5 changesets, 10 total revisions
+  2 integrity errors encountered!
+  (first damaged changeset appears to be 0)
+  [1]
+
+Updates work after cloning a damaged repo, if the damaged lfs objects aren't in
+the update destination.  Those objects won't be added to the new repo's store
+because they aren't accessed.
+
+  $ hg clone -v repo5 fromcorrupt
+  updating to branch default
+  resolving manifests
+  getting l
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the usercache
+  getting s
+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ test -f fromcorrupt/.hg/store/lfs/objects/66/100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e
+  [1]
+
+Verify will copy/link all lfs objects into the local store that aren't already
+present.  Bypass the corrupted usercache to show that verify works when fed by
+the (uncorrupted) remote store.
+
+  $ hg -R fromcorrupt --config lfs.usercache=emptycache verify -v
+  repository uses revlog format 1
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  lfs: adding 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e to the usercache
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+  lfs: adding 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 to the usercache
+  lfs: found 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 in the local lfs store
+  lfs: adding b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c to the usercache
+  lfs: found b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c in the local lfs store
+  4 files, 5 changesets, 10 total revisions
+
+BUG: Verify will copy/link a corrupted file from the usercache into the local
+store, and poison it.  (The verify with a good remote now fails.)
+
+  $ rm -r fromcorrupt/.hg/store/lfs/objects/66/100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e
+  $ hg -R fromcorrupt verify -v
+  repository uses revlog format 1
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the usercache
+   l@1: unpacking 46a2f24864bc: integrity check failed on data/l.i:0
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+   large@0: unpacking 2c531e0992ff: integrity check failed on data/large.i:0
+  lfs: found 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 in the local lfs store
+  lfs: found b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c in the local lfs store
+  4 files, 5 changesets, 10 total revisions
+  2 integrity errors encountered!
+  (first damaged changeset appears to be 0)
+  [1]
+  $ hg -R fromcorrupt --config lfs.usercache=emptycache verify -v
+  repository uses revlog format 1
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+   l@1: unpacking 46a2f24864bc: integrity check failed on data/l.i:0
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+   large@0: unpacking 2c531e0992ff: integrity check failed on data/large.i:0
+  lfs: found 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 in the local lfs store
+  lfs: found b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c in the local lfs store
+  4 files, 5 changesets, 10 total revisions
+  2 integrity errors encountered!
+  (first damaged changeset appears to be 0)
+  [1]
+
+Damaging a file required by the update destination fails the update.
+
+  $ echo 'damage' >> $TESTTMP/dummy-remote/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b
+  $ hg --config lfs.usercache=emptycache clone -v repo5 fromcorrupt2
+  updating to branch default
+  resolving manifests
+  getting l
+  lfs: adding 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b to the usercache
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+  abort: integrity check failed on data/l.i:3!
+  [255]
+
+BUG: A corrupted lfs blob either shouldn't be created after a transfer from a
+file://remotestore, or it shouldn't be left behind.
+
+  $ cat emptycache/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b | $TESTDIR/f --sha256
+  sha256=40f67c7e91d554db4bc500f8f62c2e40f9f61daa5b62388e577bbae26f5396ff
+  $ cat fromcorrupt2/.hg/store/lfs/objects/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b | $TESTDIR/f --sha256
+  sha256=40f67c7e91d554db4bc500f8f62c2e40f9f61daa5b62388e577bbae26f5396ff
+
+  $ hg -R fromcorrupt2 verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+   l@1: unpacking 46a2f24864bc: integrity check failed on data/l.i:0
+   l@4: unpacking 6f1ff1f39c11: integrity check failed on data/l.i:3
+   large@0: unpacking 2c531e0992ff: integrity check failed on data/large.i:0
+  4 files, 5 changesets, 10 total revisions
+  3 integrity errors encountered!
+  (first damaged changeset appears to be 0)
+  [1]
+
+BUG: push will happily send corrupt files upstream.  (The alternate dummy remote
+avoids the corrupt lfs object in the original remote.)
+
+  $ mkdir $TESTTMP/dummy-remote2
+  $ hg init dest
+  $ hg -R fromcorrupt2 --config lfs.url=file:///$TESTTMP/dummy-remote2 push -v dest
+  pushing to dest
+  searching for changes
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+  lfs: found 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 in the local lfs store
+  lfs: found b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c in the local lfs store
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+  5 changesets found
+  uncompressed size of bundle content:
+       997 (changelog)
+      1032 (manifests)
+       841  l
+       272  large
+       788  s
+       139  small
+  adding changesets
+  adding manifests
+  adding file changes
+  added 5 changesets with 10 changes to 4 files
+
+  $ hg -R fromcorrupt2 --config lfs.url=file:///$TESTTMP/dummy-remote2 verify -v
+  repository uses revlog format 1
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+   l@1: unpacking 46a2f24864bc: integrity check failed on data/l.i:0
+  lfs: found 22f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b in the local lfs store
+   l@4: unpacking 6f1ff1f39c11: integrity check failed on data/l.i:3
+  lfs: found 66100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e in the local lfs store
+   large@0: unpacking 2c531e0992ff: integrity check failed on data/large.i:0
+  lfs: found 89b6070915a3d573ff3599d1cda305bc5e38549b15c4847ab034169da66e1ca8 in the local lfs store
+  lfs: found b1a6ea88da0017a0e77db139a54618986e9a2489bee24af9fe596de9daac498c in the local lfs store
+  4 files, 5 changesets, 10 total revisions
+  3 integrity errors encountered!
+  (first damaged changeset appears to be 0)
+  [1]
+
+  $ cat $TESTTMP/dummy-remote2/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b | $TESTDIR/f --sha256
+  sha256=40f67c7e91d554db4bc500f8f62c2e40f9f61daa5b62388e577bbae26f5396ff
+  $ cat fromcorrupt2/.hg/store/lfs/objects/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b | $TESTDIR/f --sha256
+  sha256=40f67c7e91d554db4bc500f8f62c2e40f9f61daa5b62388e577bbae26f5396ff
+
+  $ cat $TESTTMP/dummy-remote2/66/100b384bf761271b407d79fc30cdd0554f3b2c5d944836e936d584b88ce88e
+  LONGER-THAN-TEN-BYTES-WILL-TRIGGER-LFS
+  damage
+  $ cat $TESTTMP/dummy-remote2/22/f66a3fc0b9bf3f012c814303995ec07099b3a9ce02a7af84b5970811074a3b
+  RESTORE-TO-BE-LARGE
+  damage
+
+Accessing a corrupt file will complain
+
+  $ hg --cwd fromcorrupt2 cat -r 0 large
+  abort: integrity check failed on data/large.i:0!
+  [255]
+
 lfs -> normal -> lfs round trip conversions are possible.  The threshold for the
 lfs destination is specified here because it was originally listed in the local
 .hgrc, and the global one is too high to trigger lfs usage.  For lfs -> normal,