##// END OF EJS Templates
procutil: make stream detection in make_line_buffered more correct and strict...
procutil: make stream detection in make_line_buffered more correct and strict In make_line_buffered(), we don’t want to wrap the stream if we know that lines get flushed to the underlying raw stream already. Previously, the heuristic was too optimistic. It assumed that any stream which is not an instance of io.BufferedIOBase doesn’t need wrapping. However, there are buffered streams that aren’t instances of io.BufferedIOBase, like Mercurial’s own winstdout. The new logic is different in two ways: First, only for the check, if unwraps any combination of WriteAllWrapper and winstdout. Second, it skips wrapping the stream only if it is an instance of io.RawIOBase (or already wrapped). If it is an instance of io.BufferedIOBase, it gets wrapped. In any other case, the function raises an exception. This ensures that, if an unknown stream is passed or we add another wrapper in the future, we don’t wrap the stream if it’s already line buffered or not wrap the stream if it’s not line buffered. In fact, this was already helpful during development of this change. Without it, I possibly would have forgot that WriteAllWrapper needs to be ignored for the check, leading to unnecessary wrapping if stdout is unbuffered. The alternative would have been to always wrap unknown streams. However, I don’t think that anyone would benefit from being less strict. We can expect streams from the standard library to be subclassing either io.RawIOBase or io.BufferedIOBase, so running Mercurial in the standard way should not regress by this change. Py2exe might replace sys.stdout and sys.stderr, but that currently breaks Mercurial anyway and also these streams don’t claim to be interactive, so this function is not called for them.

File last commit:

r44747:87167caa default
r50658:094a5fa3 6.2 stable
Show More
TODO.rst
192 lines | 7.5 KiB | text/x-rst | RstLexer

Prior to removing (EXPERIMENTAL)

These things affect UI and/or behavior, and should probably be implemented (or ruled out) prior to taking off the experimental shrinkwrap.

  1. Finish the hg convert story
    • Add an argument to accept a rules file to apply during conversion? Currently lfs.track is the only way to affect the conversion.
    • drop lfs.track config settings
    • splice in .hglfs file for normal repo -> lfs conversions?
  2. Stop uploading blobs when pushing between local repos
    • Could probably hardlink directly to the other local repo's store
    • Support inferring lfs.url for local push/pull (currently only supports http)
  3. Stop uploading blobs on strip/amend/histedit/etc.
    • This seems to be a side effect of doing it for hg bundle, which probably makes sense.
  4. Handle a server with the extension loaded and a client without the extension more gracefully.
  5. Remove lfs.retry hack in client? This came from FB, but it's not clear why it is/was needed.
  6. hg export currently writes out the LFS blob. Should it write the pointer instead?
    • hg diff is similar, and probably shouldn't see the pointer file
  7. Show to-be-applied rules with hg files -r 'wdir()' 'set:lfs()'
    • debugignore can show file + line number, so a dedicated command could be useful too.
  8. Filesets, revsets and templates
    • A dedicated revset should be faster than 'file(set:lfs())'
    • Attach {lfsoid} and {lfspointer} to general keywords, IFF the file is a blob
    • Drop existing items that would be redundant with general support
  9. Can grep avoid downloading most things?
    • Add a command option to skip LFS blobs?
  10. Add a flag that's visible in hg files -v to indicate external storage?
  11. Server side issues
  12. Add locks on cache and blob store
    • This is complicated with a global store, and multiple potentially unrelated local repositories that reference the same blob.
    • Alternately, maybe just handle collisions when trying to create the same blob in the store somehow.
  13. Are proper file sizes reported in debugupgraderepo?
  14. Finish prefetching files
    • -T {data} (other than cat?)
    • verify
    • grep
  15. Output cleanup
    • Can we print the url when connecting to the blobstore? (A sudden connection refused after pulling commits looks confusing.) Problem is, 'pushing to main url' is printed, and then lfs wants to upload before going back to the main repo transfer, so then that could be confusing with extra output. (This is kinda improved with 380f5131ee7b and 9f78d10742af.)
    • Add more progress indicators? Uploading a large repo looks idle for a long time while it scans for blobs in each outgoing revision.
    • Print filenames instead of hashes in error messages
      • subrepo aware paths, where necessary
    • Is existing output at the right status/note/debug level?
  16. Can verify be done without downloading everything?
    • If we know that we are talking to an hg server, we can leverage the fact that it validates in the Batch API portion, and skip d/l altogether. OTOH, maybe we should download the files unconditionally for forensics. The alternative is to define a custom transfer handler that definitively verifies without transferring, and then cache those results. When verify comes looking, look in the cache instead of actually opening the file and processing it.
    • Yuya has concerns about when blob fetch takes place vs when revlog is verified. Since the visible hash matches the blob content, I don't think there's a way to verify the pointer file that's actually stored in the filelog (other than basic JSON checks). Full verification requires the blob. See https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116133.html
    • Opening a corrupt pointer file aborts. It probably shouldn't for verify.

Future ideas/features/polishing

These aren't in any particular order, and are things that don't have obvious BC concerns.

  1. Garbage collection (issue5790)
    • This gets complicated because of the global cache, which may or may not consist of hardlinks to the repo, and may be in use by other repos. (So the gc may be pointless.)
  2. Compress blobs
    • 700MB repo becomes 2.5GB with all lfs blobs
    • What implications are there for filesystem paths that don't indicate compression? (i.e. how to share with global cache and other local repos?)
    • Probably needs to be stored under .hg/store/lfs/zstd, with a repo requirement.
    • Allow tuneable compression type and settings?
    • Support compression over the wire if both sides understand the compression?
    • debugupgraderepo to convert?
    • Probably not worth supporting compressed and uncompressed concurrently
  3. Determine things to upload with readfast()
    • Significantly faster when pushing an entire large repo to http.
    • Causes test changes to fileset and templates; may need both this and current methods of lookup.
  4. Is a command to download everything needed? This would allow copying the whole to a portable drive. Currently this can be effected by running hg verify.
  5. Stop reading in entire file into one buffer when passing through filelog interface
  6. Keep corrupt files around in 'store/lfs/incoming' for forensics?
    • Files should be downloaded to 'incoming', and moved to normal location when done.
  7. Client side path enhancements
  8. Server enhancements
  9. Handle 3rd party server storage.
    • Teach client to handle lfs verify action. This is needed after the server instructs the client to upload the file to another server, in order to tell the server that the upload completed.
    • Teach the server to send redirects if configured, and process verify requests.
  10. Is any hg-git work needed?