##// END OF EJS Templates
lfs: remove callstatus property from 'lfs()' fileset...
Yuya Nishihara -
r38836:7790cd70 default
parent child Browse files
Show More
@@ -1,401 +1,401 b''
1 1 # lfs - hash-preserving large file support using Git-LFS protocol
2 2 #
3 3 # Copyright 2017 Facebook, Inc.
4 4 #
5 5 # This software may be used and distributed according to the terms of the
6 6 # GNU General Public License version 2 or any later version.
7 7
8 8 """lfs - large file support (EXPERIMENTAL)
9 9
10 10 This extension allows large files to be tracked outside of the normal
11 11 repository storage and stored on a centralized server, similar to the
12 12 ``largefiles`` extension. The ``git-lfs`` protocol is used when
13 13 communicating with the server, so existing git infrastructure can be
14 14 harnessed. Even though the files are stored outside of the repository,
15 15 they are still integrity checked in the same manner as normal files.
16 16
17 17 The files stored outside of the repository are downloaded on demand,
18 18 which reduces the time to clone, and possibly the local disk usage.
19 19 This changes fundamental workflows in a DVCS, so careful thought
20 20 should be given before deploying it. :hg:`convert` can be used to
21 21 convert LFS repositories to normal repositories that no longer
22 22 require this extension, and do so without changing the commit hashes.
23 23 This allows the extension to be disabled if the centralized workflow
24 24 becomes burdensome. However, the pre and post convert clones will
25 25 not be able to communicate with each other unless the extension is
26 26 enabled on both.
27 27
28 28 To start a new repository, or to add LFS files to an existing one, just
29 29 create an ``.hglfs`` file as described below in the root directory of
30 30 the repository. Typically, this file should be put under version
31 31 control, so that the settings will propagate to other repositories with
32 32 push and pull. During any commit, Mercurial will consult this file to
33 33 determine if an added or modified file should be stored externally. The
34 34 type of storage depends on the characteristics of the file at each
35 35 commit. A file that is near a size threshold may switch back and forth
36 36 between LFS and normal storage, as needed.
37 37
38 38 Alternately, both normal repositories and largefile controlled
39 39 repositories can be converted to LFS by using :hg:`convert` and the
40 40 ``lfs.track`` config option described below. The ``.hglfs`` file
41 41 should then be created and added, to control subsequent LFS selection.
42 42 The hashes are also unchanged in this case. The LFS and non-LFS
43 43 repositories can be distinguished because the LFS repository will
44 44 abort any command if this extension is disabled.
45 45
46 46 Committed LFS files are held locally, until the repository is pushed.
47 47 Prior to pushing the normal repository data, the LFS files that are
48 48 tracked by the outgoing commits are automatically uploaded to the
49 49 configured central server. No LFS files are transferred on
50 50 :hg:`pull` or :hg:`clone`. Instead, the files are downloaded on
51 51 demand as they need to be read, if a cached copy cannot be found
52 52 locally. Both committing and downloading an LFS file will link the
53 53 file to a usercache, to speed up future access. See the `usercache`
54 54 config setting described below.
55 55
56 56 .hglfs::
57 57
58 58 The extension reads its configuration from a versioned ``.hglfs``
59 59 configuration file found in the root of the working directory. The
60 60 ``.hglfs`` file uses the same syntax as all other Mercurial
61 61 configuration files. It uses a single section, ``[track]``.
62 62
63 63 The ``[track]`` section specifies which files are stored as LFS (or
64 64 not). Each line is keyed by a file pattern, with a predicate value.
65 65 The first file pattern match is used, so put more specific patterns
66 66 first. The available predicates are ``all()``, ``none()``, and
67 67 ``size()``. See "hg help filesets.size" for the latter.
68 68
69 69 Example versioned ``.hglfs`` file::
70 70
71 71 [track]
72 72 # No Makefile or python file, anywhere, will be LFS
73 73 **Makefile = none()
74 74 **.py = none()
75 75
76 76 **.zip = all()
77 77 **.exe = size(">1MB")
78 78
79 79 # Catchall for everything not matched above
80 80 ** = size(">10MB")
81 81
82 82 Configs::
83 83
84 84 [lfs]
85 85 # Remote endpoint. Multiple protocols are supported:
86 86 # - http(s)://user:pass@example.com/path
87 87 # git-lfs endpoint
88 88 # - file:///tmp/path
89 89 # local filesystem, usually for testing
90 90 # if unset, lfs will assume the remote repository also handles blob storage
91 91 # for http(s) URLs. Otherwise, lfs will prompt to set this when it must
92 92 # use this value.
93 93 # (default: unset)
94 94 url = https://example.com/repo.git/info/lfs
95 95
96 96 # Which files to track in LFS. Path tests are "**.extname" for file
97 97 # extensions, and "path:under/some/directory" for path prefix. Both
98 98 # are relative to the repository root.
99 99 # File size can be tested with the "size()" fileset, and tests can be
100 100 # joined with fileset operators. (See "hg help filesets.operators".)
101 101 #
102 102 # Some examples:
103 103 # - all() # everything
104 104 # - none() # nothing
105 105 # - size(">20MB") # larger than 20MB
106 106 # - !**.txt # anything not a *.txt file
107 107 # - **.zip | **.tar.gz | **.7z # some types of compressed files
108 108 # - path:bin # files under "bin" in the project root
109 109 # - (**.php & size(">2MB")) | (**.js & size(">5MB")) | **.tar.gz
110 110 # | (path:bin & !path:/bin/README) | size(">1GB")
111 111 # (default: none())
112 112 #
113 113 # This is ignored if there is a tracked '.hglfs' file, and this setting
114 114 # will eventually be deprecated and removed.
115 115 track = size(">10M")
116 116
117 117 # how many times to retry before giving up on transferring an object
118 118 retry = 5
119 119
120 120 # the local directory to store lfs files for sharing across local clones.
121 121 # If not set, the cache is located in an OS specific cache location.
122 122 usercache = /path/to/global/cache
123 123 """
124 124
125 125 from __future__ import absolute_import
126 126
127 127 from mercurial.i18n import _
128 128
129 129 from mercurial import (
130 130 bundle2,
131 131 changegroup,
132 132 cmdutil,
133 133 config,
134 134 context,
135 135 error,
136 136 exchange,
137 137 extensions,
138 138 filelog,
139 139 fileset,
140 140 hg,
141 141 localrepo,
142 142 minifileset,
143 143 node,
144 144 pycompat,
145 145 registrar,
146 146 revlog,
147 147 scmutil,
148 148 templateutil,
149 149 upgrade,
150 150 util,
151 151 vfs as vfsmod,
152 152 wireprotoserver,
153 153 wireprotov1server,
154 154 )
155 155
156 156 from . import (
157 157 blobstore,
158 158 wireprotolfsserver,
159 159 wrapper,
160 160 )
161 161
162 162 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
163 163 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
164 164 # be specifying the version(s) of Mercurial they are tested with, or
165 165 # leave the attribute unspecified.
166 166 testedwith = 'ships-with-hg-core'
167 167
168 168 configtable = {}
169 169 configitem = registrar.configitem(configtable)
170 170
171 171 configitem('experimental', 'lfs.serve',
172 172 default=True,
173 173 )
174 174 configitem('experimental', 'lfs.user-agent',
175 175 default=None,
176 176 )
177 177 configitem('experimental', 'lfs.disableusercache',
178 178 default=False,
179 179 )
180 180 configitem('experimental', 'lfs.worker-enable',
181 181 default=False,
182 182 )
183 183
184 184 configitem('lfs', 'url',
185 185 default=None,
186 186 )
187 187 configitem('lfs', 'usercache',
188 188 default=None,
189 189 )
190 190 # Deprecated
191 191 configitem('lfs', 'threshold',
192 192 default=None,
193 193 )
194 194 configitem('lfs', 'track',
195 195 default='none()',
196 196 )
197 197 configitem('lfs', 'retry',
198 198 default=5,
199 199 )
200 200
201 201 cmdtable = {}
202 202 command = registrar.command(cmdtable)
203 203
204 204 templatekeyword = registrar.templatekeyword()
205 205 filesetpredicate = registrar.filesetpredicate()
206 206
207 207 def featuresetup(ui, supported):
208 208 # don't die on seeing a repo with the lfs requirement
209 209 supported |= {'lfs'}
210 210
211 211 def uisetup(ui):
212 212 localrepo.featuresetupfuncs.add(featuresetup)
213 213
214 214 def reposetup(ui, repo):
215 215 # Nothing to do with a remote repo
216 216 if not repo.local():
217 217 return
218 218
219 219 repo.svfs.lfslocalblobstore = blobstore.local(repo)
220 220 repo.svfs.lfsremoteblobstore = blobstore.remote(repo)
221 221
222 222 class lfsrepo(repo.__class__):
223 223 @localrepo.unfilteredmethod
224 224 def commitctx(self, ctx, error=False):
225 225 repo.svfs.options['lfstrack'] = _trackedmatcher(self)
226 226 return super(lfsrepo, self).commitctx(ctx, error)
227 227
228 228 repo.__class__ = lfsrepo
229 229
230 230 if 'lfs' not in repo.requirements:
231 231 def checkrequireslfs(ui, repo, **kwargs):
232 232 if 'lfs' not in repo.requirements:
233 233 last = kwargs.get(r'node_last')
234 234 _bin = node.bin
235 235 if last:
236 236 s = repo.set('%n:%n', _bin(kwargs[r'node']), _bin(last))
237 237 else:
238 238 s = repo.set('%n', _bin(kwargs[r'node']))
239 239 match = repo.narrowmatch()
240 240 for ctx in s:
241 241 # TODO: is there a way to just walk the files in the commit?
242 242 if any(ctx[f].islfs() for f in ctx.files()
243 243 if f in ctx and match(f)):
244 244 repo.requirements.add('lfs')
245 245 repo._writerequirements()
246 246 repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
247 247 break
248 248
249 249 ui.setconfig('hooks', 'commit.lfs', checkrequireslfs, 'lfs')
250 250 ui.setconfig('hooks', 'pretxnchangegroup.lfs', checkrequireslfs, 'lfs')
251 251 else:
252 252 repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
253 253
254 254 def _trackedmatcher(repo):
255 255 """Return a function (path, size) -> bool indicating whether or not to
256 256 track a given file with lfs."""
257 257 if not repo.wvfs.exists('.hglfs'):
258 258 # No '.hglfs' in wdir. Fallback to config for now.
259 259 trackspec = repo.ui.config('lfs', 'track')
260 260
261 261 # deprecated config: lfs.threshold
262 262 threshold = repo.ui.configbytes('lfs', 'threshold')
263 263 if threshold:
264 264 fileset.parse(trackspec) # make sure syntax errors are confined
265 265 trackspec = "(%s) | size('>%d')" % (trackspec, threshold)
266 266
267 267 return minifileset.compile(trackspec)
268 268
269 269 data = repo.wvfs.tryread('.hglfs')
270 270 if not data:
271 271 return lambda p, s: False
272 272
273 273 # Parse errors here will abort with a message that points to the .hglfs file
274 274 # and line number.
275 275 cfg = config.config()
276 276 cfg.parse('.hglfs', data)
277 277
278 278 try:
279 279 rules = [(minifileset.compile(pattern), minifileset.compile(rule))
280 280 for pattern, rule in cfg.items('track')]
281 281 except error.ParseError as e:
282 282 # The original exception gives no indicator that the error is in the
283 283 # .hglfs file, so add that.
284 284
285 285 # TODO: See if the line number of the file can be made available.
286 286 raise error.Abort(_('parse error in .hglfs: %s') % e)
287 287
288 288 def _match(path, size):
289 289 for pat, rule in rules:
290 290 if pat(path, size):
291 291 return rule(path, size)
292 292
293 293 return False
294 294
295 295 return _match
296 296
297 297 def wrapfilelog(filelog):
298 298 wrapfunction = extensions.wrapfunction
299 299
300 300 wrapfunction(filelog, 'addrevision', wrapper.filelogaddrevision)
301 301 wrapfunction(filelog, 'renamed', wrapper.filelogrenamed)
302 302 wrapfunction(filelog, 'size', wrapper.filelogsize)
303 303
304 304 def extsetup(ui):
305 305 wrapfilelog(filelog.filelog)
306 306
307 307 wrapfunction = extensions.wrapfunction
308 308
309 309 wrapfunction(cmdutil, '_updatecatformatter', wrapper._updatecatformatter)
310 310 wrapfunction(scmutil, 'wrapconvertsink', wrapper.convertsink)
311 311
312 312 wrapfunction(upgrade, '_finishdatamigration',
313 313 wrapper.upgradefinishdatamigration)
314 314
315 315 wrapfunction(upgrade, 'preservedrequirements',
316 316 wrapper.upgraderequirements)
317 317
318 318 wrapfunction(upgrade, 'supporteddestrequirements',
319 319 wrapper.upgraderequirements)
320 320
321 321 wrapfunction(changegroup,
322 322 'allsupportedversions',
323 323 wrapper.allsupportedversions)
324 324
325 325 wrapfunction(exchange, 'push', wrapper.push)
326 326 wrapfunction(wireprotov1server, '_capabilities', wrapper._capabilities)
327 327 wrapfunction(wireprotoserver, 'handlewsgirequest',
328 328 wireprotolfsserver.handlewsgirequest)
329 329
330 330 wrapfunction(context.basefilectx, 'cmp', wrapper.filectxcmp)
331 331 wrapfunction(context.basefilectx, 'isbinary', wrapper.filectxisbinary)
332 332 context.basefilectx.islfs = wrapper.filectxislfs
333 333
334 334 revlog.addflagprocessor(
335 335 revlog.REVIDX_EXTSTORED,
336 336 (
337 337 wrapper.readfromstore,
338 338 wrapper.writetostore,
339 339 wrapper.bypasscheckhash,
340 340 ),
341 341 )
342 342
343 343 wrapfunction(hg, 'clone', wrapper.hgclone)
344 344 wrapfunction(hg, 'postshare', wrapper.hgpostshare)
345 345
346 346 scmutil.fileprefetchhooks.add('lfs', wrapper._prefetchfiles)
347 347
348 348 # Make bundle choose changegroup3 instead of changegroup2. This affects
349 349 # "hg bundle" command. Note: it does not cover all bundle formats like
350 350 # "packed1". Using "packed1" with lfs will likely cause trouble.
351 351 exchange._bundlespeccontentopts["v2"]["cg.version"] = "03"
352 352
353 353 # bundlerepo uses "vfsmod.readonlyvfs(othervfs)", we need to make sure lfs
354 354 # options and blob stores are passed from othervfs to the new readonlyvfs.
355 355 wrapfunction(vfsmod.readonlyvfs, '__init__', wrapper.vfsinit)
356 356
357 357 # when writing a bundle via "hg bundle" command, upload related LFS blobs
358 358 wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)
359 359
360 @filesetpredicate('lfs()', callstatus=True)
360 @filesetpredicate('lfs()')
361 361 def lfsfileset(mctx, x):
362 362 """File that uses LFS storage."""
363 363 # i18n: "lfs" is a keyword
364 364 fileset.getargs(x, 0, 0, _("lfs takes no arguments"))
365 365 ctx = mctx.ctx
366 366 def lfsfilep(f):
367 367 return wrapper.pointerfromctx(ctx, f, removed=True) is not None
368 368 return mctx.predicate(lfsfilep, predrepr='<lfs>')
369 369
370 370 @templatekeyword('lfs_files', requires={'ctx'})
371 371 def lfsfiles(context, mapping):
372 372 """List of strings. All files modified, added, or removed by this
373 373 changeset."""
374 374 ctx = context.resource(mapping, 'ctx')
375 375
376 376 pointers = wrapper.pointersfromctx(ctx, removed=True) # {path: pointer}
377 377 files = sorted(pointers.keys())
378 378
379 379 def pointer(v):
380 380 # In the file spec, version is first and the other keys are sorted.
381 381 sortkeyfunc = lambda x: (x[0] != 'version', x)
382 382 items = sorted(pointers[v].iteritems(), key=sortkeyfunc)
383 383 return util.sortdict(items)
384 384
385 385 makemap = lambda v: {
386 386 'file': v,
387 387 'lfsoid': pointers[v].oid() if pointers[v] else None,
388 388 'lfspointer': templateutil.hybriddict(pointer(v)),
389 389 }
390 390
391 391 # TODO: make the separator ', '?
392 392 f = templateutil._showcompatlist(context, mapping, 'lfs_file', files)
393 393 return templateutil.hybrid(f, files, makemap, pycompat.identity)
394 394
395 395 @command('debuglfsupload',
396 396 [('r', 'rev', [], _('upload large files introduced by REV'))])
397 397 def debuglfsupload(ui, repo, **opts):
398 398 """upload lfs blobs added by the working copy parent or given revisions"""
399 399 revs = opts.get(r'rev', [])
400 400 pointers = wrapper.extractpointers(repo, scmutil.revrange(repo, revs))
401 401 wrapper.uploadblobs(repo, pointers)
General Comments 0
You need to be logged in to leave comments. Login now