##// END OF EJS Templates
lfs: dedent documentation section about .hglfs file...
Denis Laxalde -
r43597:26caf96a default
parent child Browse files
Show More
@@ -1,406 +1,404 b''
1 1 # lfs - hash-preserving large file support using Git-LFS protocol
2 2 #
3 3 # Copyright 2017 Facebook, Inc.
4 4 #
5 5 # This software may be used and distributed according to the terms of the
6 6 # GNU General Public License version 2 or any later version.
7 7
8 8 """lfs - large file support (EXPERIMENTAL)
9 9
10 10 This extension allows large files to be tracked outside of the normal
11 11 repository storage and stored on a centralized server, similar to the
12 12 ``largefiles`` extension. The ``git-lfs`` protocol is used when
13 13 communicating with the server, so existing git infrastructure can be
14 14 harnessed. Even though the files are stored outside of the repository,
15 15 they are still integrity checked in the same manner as normal files.
16 16
17 17 The files stored outside of the repository are downloaded on demand,
18 18 which reduces the time to clone, and possibly the local disk usage.
19 19 This changes fundamental workflows in a DVCS, so careful thought
20 20 should be given before deploying it. :hg:`convert` can be used to
21 21 convert LFS repositories to normal repositories that no longer
22 22 require this extension, and do so without changing the commit hashes.
23 23 This allows the extension to be disabled if the centralized workflow
24 24 becomes burdensome. However, the pre and post convert clones will
25 25 not be able to communicate with each other unless the extension is
26 26 enabled on both.
27 27
28 28 To start a new repository, or to add LFS files to an existing one, just
29 29 create an ``.hglfs`` file as described below in the root directory of
30 30 the repository. Typically, this file should be put under version
31 31 control, so that the settings will propagate to other repositories with
32 32 push and pull. During any commit, Mercurial will consult this file to
33 33 determine if an added or modified file should be stored externally. The
34 34 type of storage depends on the characteristics of the file at each
35 35 commit. A file that is near a size threshold may switch back and forth
36 36 between LFS and normal storage, as needed.
37 37
38 38 Alternately, both normal repositories and largefile controlled
39 39 repositories can be converted to LFS by using :hg:`convert` and the
40 40 ``lfs.track`` config option described below. The ``.hglfs`` file
41 41 should then be created and added, to control subsequent LFS selection.
42 42 The hashes are also unchanged in this case. The LFS and non-LFS
43 43 repositories can be distinguished because the LFS repository will
44 44 abort any command if this extension is disabled.
45 45
46 46 Committed LFS files are held locally, until the repository is pushed.
47 47 Prior to pushing the normal repository data, the LFS files that are
48 48 tracked by the outgoing commits are automatically uploaded to the
49 49 configured central server. No LFS files are transferred on
50 50 :hg:`pull` or :hg:`clone`. Instead, the files are downloaded on
51 51 demand as they need to be read, if a cached copy cannot be found
52 52 locally. Both committing and downloading an LFS file will link the
53 53 file to a usercache, to speed up future access. See the `usercache`
54 54 config setting described below.
55 55
56 .hglfs::
57
58 56 The extension reads its configuration from a versioned ``.hglfs``
59 57 configuration file found in the root of the working directory. The
60 58 ``.hglfs`` file uses the same syntax as all other Mercurial
61 59 configuration files. It uses a single section, ``[track]``.
62 60
63 61 The ``[track]`` section specifies which files are stored as LFS (or
64 62 not). Each line is keyed by a file pattern, with a predicate value.
65 63 The first file pattern match is used, so put more specific patterns
66 64 first. The available predicates are ``all()``, ``none()``, and
67 65 ``size()``. See "hg help filesets.size" for the latter.
68 66
69 67 Example versioned ``.hglfs`` file::
70 68
71 69 [track]
72 70 # No Makefile or python file, anywhere, will be LFS
73 71 **Makefile = none()
74 72 **.py = none()
75 73
76 74 **.zip = all()
77 75 **.exe = size(">1MB")
78 76
79 77 # Catchall for everything not matched above
80 78 ** = size(">10MB")
81 79
82 80 Configs::
83 81
84 82 [lfs]
85 83 # Remote endpoint. Multiple protocols are supported:
86 84 # - http(s)://user:pass@example.com/path
87 85 # git-lfs endpoint
88 86 # - file:///tmp/path
89 87 # local filesystem, usually for testing
90 88 # if unset, lfs will assume the remote repository also handles blob storage
91 89 # for http(s) URLs. Otherwise, lfs will prompt to set this when it must
92 90 # use this value.
93 91 # (default: unset)
94 92 url = https://example.com/repo.git/info/lfs
95 93
96 94 # Which files to track in LFS. Path tests are "**.extname" for file
97 95 # extensions, and "path:under/some/directory" for path prefix. Both
98 96 # are relative to the repository root.
99 97 # File size can be tested with the "size()" fileset, and tests can be
100 98 # joined with fileset operators. (See "hg help filesets.operators".)
101 99 #
102 100 # Some examples:
103 101 # - all() # everything
104 102 # - none() # nothing
105 103 # - size(">20MB") # larger than 20MB
106 104 # - !**.txt # anything not a *.txt file
107 105 # - **.zip | **.tar.gz | **.7z # some types of compressed files
108 106 # - path:bin # files under "bin" in the project root
109 107 # - (**.php & size(">2MB")) | (**.js & size(">5MB")) | **.tar.gz
110 108 # | (path:bin & !path:/bin/README) | size(">1GB")
111 109 # (default: none())
112 110 #
113 111 # This is ignored if there is a tracked '.hglfs' file, and this setting
114 112 # will eventually be deprecated and removed.
115 113 track = size(">10M")
116 114
117 115 # how many times to retry before giving up on transferring an object
118 116 retry = 5
119 117
120 118 # the local directory to store lfs files for sharing across local clones.
121 119 # If not set, the cache is located in an OS specific cache location.
122 120 usercache = /path/to/global/cache
123 121 """
124 122
125 123 from __future__ import absolute_import
126 124
127 125 import sys
128 126
129 127 from mercurial.i18n import _
130 128
131 129 from mercurial import (
132 130 config,
133 131 context,
134 132 error,
135 133 exchange,
136 134 extensions,
137 135 exthelper,
138 136 filelog,
139 137 filesetlang,
140 138 localrepo,
141 139 minifileset,
142 140 node,
143 141 pycompat,
144 142 revlog,
145 143 scmutil,
146 144 templateutil,
147 145 util,
148 146 )
149 147
150 148 from mercurial.interfaces import repository
151 149
152 150 from . import (
153 151 blobstore,
154 152 wireprotolfsserver,
155 153 wrapper,
156 154 )
157 155
158 156 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
159 157 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
160 158 # be specifying the version(s) of Mercurial they are tested with, or
161 159 # leave the attribute unspecified.
162 160 testedwith = b'ships-with-hg-core'
163 161
164 162 eh = exthelper.exthelper()
165 163 eh.merge(wrapper.eh)
166 164 eh.merge(wireprotolfsserver.eh)
167 165
168 166 cmdtable = eh.cmdtable
169 167 configtable = eh.configtable
170 168 extsetup = eh.finalextsetup
171 169 uisetup = eh.finaluisetup
172 170 filesetpredicate = eh.filesetpredicate
173 171 reposetup = eh.finalreposetup
174 172 templatekeyword = eh.templatekeyword
175 173
176 174 eh.configitem(
177 175 b'experimental', b'lfs.serve', default=True,
178 176 )
179 177 eh.configitem(
180 178 b'experimental', b'lfs.user-agent', default=None,
181 179 )
182 180 eh.configitem(
183 181 b'experimental', b'lfs.disableusercache', default=False,
184 182 )
185 183 eh.configitem(
186 184 b'experimental', b'lfs.worker-enable', default=False,
187 185 )
188 186
189 187 eh.configitem(
190 188 b'lfs', b'url', default=None,
191 189 )
192 190 eh.configitem(
193 191 b'lfs', b'usercache', default=None,
194 192 )
195 193 # Deprecated
196 194 eh.configitem(
197 195 b'lfs', b'threshold', default=None,
198 196 )
199 197 eh.configitem(
200 198 b'lfs', b'track', default=b'none()',
201 199 )
202 200 eh.configitem(
203 201 b'lfs', b'retry', default=5,
204 202 )
205 203
206 204 lfsprocessor = (
207 205 wrapper.readfromstore,
208 206 wrapper.writetostore,
209 207 wrapper.bypasscheckhash,
210 208 )
211 209
212 210
213 211 def featuresetup(ui, supported):
214 212 # don't die on seeing a repo with the lfs requirement
215 213 supported |= {b'lfs'}
216 214
217 215
218 216 @eh.uisetup
219 217 def _uisetup(ui):
220 218 localrepo.featuresetupfuncs.add(featuresetup)
221 219
222 220
223 221 @eh.reposetup
224 222 def _reposetup(ui, repo):
225 223 # Nothing to do with a remote repo
226 224 if not repo.local():
227 225 return
228 226
229 227 repo.svfs.lfslocalblobstore = blobstore.local(repo)
230 228 repo.svfs.lfsremoteblobstore = blobstore.remote(repo)
231 229
232 230 class lfsrepo(repo.__class__):
233 231 @localrepo.unfilteredmethod
234 232 def commitctx(self, ctx, error=False, origctx=None):
235 233 repo.svfs.options[b'lfstrack'] = _trackedmatcher(self)
236 234 return super(lfsrepo, self).commitctx(ctx, error, origctx=origctx)
237 235
238 236 repo.__class__ = lfsrepo
239 237
240 238 if b'lfs' not in repo.requirements:
241 239
242 240 def checkrequireslfs(ui, repo, **kwargs):
243 241 if b'lfs' in repo.requirements:
244 242 return 0
245 243
246 244 last = kwargs.get(r'node_last')
247 245 _bin = node.bin
248 246 if last:
249 247 s = repo.set(b'%n:%n', _bin(kwargs[r'node']), _bin(last))
250 248 else:
251 249 s = repo.set(b'%n', _bin(kwargs[r'node']))
252 250 match = repo._storenarrowmatch
253 251 for ctx in s:
254 252 # TODO: is there a way to just walk the files in the commit?
255 253 if any(
256 254 ctx[f].islfs() for f in ctx.files() if f in ctx and match(f)
257 255 ):
258 256 repo.requirements.add(b'lfs')
259 257 repo.features.add(repository.REPO_FEATURE_LFS)
260 258 repo._writerequirements()
261 259 repo.prepushoutgoinghooks.add(b'lfs', wrapper.prepush)
262 260 break
263 261
264 262 ui.setconfig(b'hooks', b'commit.lfs', checkrequireslfs, b'lfs')
265 263 ui.setconfig(
266 264 b'hooks', b'pretxnchangegroup.lfs', checkrequireslfs, b'lfs'
267 265 )
268 266 else:
269 267 repo.prepushoutgoinghooks.add(b'lfs', wrapper.prepush)
270 268
271 269
272 270 def _trackedmatcher(repo):
273 271 """Return a function (path, size) -> bool indicating whether or not to
274 272 track a given file with lfs."""
275 273 if not repo.wvfs.exists(b'.hglfs'):
276 274 # No '.hglfs' in wdir. Fallback to config for now.
277 275 trackspec = repo.ui.config(b'lfs', b'track')
278 276
279 277 # deprecated config: lfs.threshold
280 278 threshold = repo.ui.configbytes(b'lfs', b'threshold')
281 279 if threshold:
282 280 filesetlang.parse(trackspec) # make sure syntax errors are confined
283 281 trackspec = b"(%s) | size('>%d')" % (trackspec, threshold)
284 282
285 283 return minifileset.compile(trackspec)
286 284
287 285 data = repo.wvfs.tryread(b'.hglfs')
288 286 if not data:
289 287 return lambda p, s: False
290 288
291 289 # Parse errors here will abort with a message that points to the .hglfs file
292 290 # and line number.
293 291 cfg = config.config()
294 292 cfg.parse(b'.hglfs', data)
295 293
296 294 try:
297 295 rules = [
298 296 (minifileset.compile(pattern), minifileset.compile(rule))
299 297 for pattern, rule in cfg.items(b'track')
300 298 ]
301 299 except error.ParseError as e:
302 300 # The original exception gives no indicator that the error is in the
303 301 # .hglfs file, so add that.
304 302
305 303 # TODO: See if the line number of the file can be made available.
306 304 raise error.Abort(_(b'parse error in .hglfs: %s') % e)
307 305
308 306 def _match(path, size):
309 307 for pat, rule in rules:
310 308 if pat(path, size):
311 309 return rule(path, size)
312 310
313 311 return False
314 312
315 313 return _match
316 314
317 315
318 316 # Called by remotefilelog
319 317 def wrapfilelog(filelog):
320 318 wrapfunction = extensions.wrapfunction
321 319
322 320 wrapfunction(filelog, 'addrevision', wrapper.filelogaddrevision)
323 321 wrapfunction(filelog, 'renamed', wrapper.filelogrenamed)
324 322 wrapfunction(filelog, 'size', wrapper.filelogsize)
325 323
326 324
327 325 @eh.wrapfunction(localrepo, b'resolverevlogstorevfsoptions')
328 326 def _resolverevlogstorevfsoptions(orig, ui, requirements, features):
329 327 opts = orig(ui, requirements, features)
330 328 for name, module in extensions.extensions(ui):
331 329 if module is sys.modules[__name__]:
332 330 if revlog.REVIDX_EXTSTORED in opts[b'flagprocessors']:
333 331 msg = (
334 332 _(b"cannot register multiple processors on flag '%#x'.")
335 333 % revlog.REVIDX_EXTSTORED
336 334 )
337 335 raise error.Abort(msg)
338 336
339 337 opts[b'flagprocessors'][revlog.REVIDX_EXTSTORED] = lfsprocessor
340 338 break
341 339
342 340 return opts
343 341
344 342
345 343 @eh.extsetup
346 344 def _extsetup(ui):
347 345 wrapfilelog(filelog.filelog)
348 346
349 347 context.basefilectx.islfs = wrapper.filectxislfs
350 348
351 349 scmutil.fileprefetchhooks.add(b'lfs', wrapper._prefetchfiles)
352 350
353 351 # Make bundle choose changegroup3 instead of changegroup2. This affects
354 352 # "hg bundle" command. Note: it does not cover all bundle formats like
355 353 # "packed1". Using "packed1" with lfs will likely cause trouble.
356 354 exchange._bundlespeccontentopts[b"v2"][b"cg.version"] = b"03"
357 355
358 356
359 357 @eh.filesetpredicate(b'lfs()')
360 358 def lfsfileset(mctx, x):
361 359 """File that uses LFS storage."""
362 360 # i18n: "lfs" is a keyword
363 361 filesetlang.getargs(x, 0, 0, _(b"lfs takes no arguments"))
364 362 ctx = mctx.ctx
365 363
366 364 def lfsfilep(f):
367 365 return wrapper.pointerfromctx(ctx, f, removed=True) is not None
368 366
369 367 return mctx.predicate(lfsfilep, predrepr=b'<lfs>')
370 368
371 369
372 370 @eh.templatekeyword(b'lfs_files', requires={b'ctx'})
373 371 def lfsfiles(context, mapping):
374 372 """List of strings. All files modified, added, or removed by this
375 373 changeset."""
376 374 ctx = context.resource(mapping, b'ctx')
377 375
378 376 pointers = wrapper.pointersfromctx(ctx, removed=True) # {path: pointer}
379 377 files = sorted(pointers.keys())
380 378
381 379 def pointer(v):
382 380 # In the file spec, version is first and the other keys are sorted.
383 381 sortkeyfunc = lambda x: (x[0] != b'version', x)
384 382 items = sorted(pycompat.iteritems(pointers[v]), key=sortkeyfunc)
385 383 return util.sortdict(items)
386 384
387 385 makemap = lambda v: {
388 386 b'file': v,
389 387 b'lfsoid': pointers[v].oid() if pointers[v] else None,
390 388 b'lfspointer': templateutil.hybriddict(pointer(v)),
391 389 }
392 390
393 391 # TODO: make the separator ', '?
394 392 f = templateutil._showcompatlist(context, mapping, b'lfs_file', files)
395 393 return templateutil.hybrid(f, files, makemap, pycompat.identity)
396 394
397 395
398 396 @eh.command(
399 397 b'debuglfsupload',
400 398 [(b'r', b'rev', [], _(b'upload large files introduced by REV'))],
401 399 )
402 400 def debuglfsupload(ui, repo, **opts):
403 401 """upload lfs blobs added by the working copy parent or given revisions"""
404 402 revs = opts.get(r'rev', [])
405 403 pointers = wrapper.extractpointers(repo, scmutil.revrange(repo, revs))
406 404 wrapper.uploadblobs(repo, pointers)
General Comments 0
You need to be logged in to leave comments. Login now