##// END OF EJS Templates
lfs: drop unused import...
Gregory Szorc -
r39904:b06303a2 default
parent child Browse files
Show More
@@ -1,402 +1,401 b''
1 1 # lfs - hash-preserving large file support using Git-LFS protocol
2 2 #
3 3 # Copyright 2017 Facebook, Inc.
4 4 #
5 5 # This software may be used and distributed according to the terms of the
6 6 # GNU General Public License version 2 or any later version.
7 7
8 8 """lfs - large file support (EXPERIMENTAL)
9 9
10 10 This extension allows large files to be tracked outside of the normal
11 11 repository storage and stored on a centralized server, similar to the
12 12 ``largefiles`` extension. The ``git-lfs`` protocol is used when
13 13 communicating with the server, so existing git infrastructure can be
14 14 harnessed. Even though the files are stored outside of the repository,
15 15 they are still integrity checked in the same manner as normal files.
16 16
17 17 The files stored outside of the repository are downloaded on demand,
18 18 which reduces the time to clone, and possibly the local disk usage.
19 19 This changes fundamental workflows in a DVCS, so careful thought
20 20 should be given before deploying it. :hg:`convert` can be used to
21 21 convert LFS repositories to normal repositories that no longer
22 22 require this extension, and do so without changing the commit hashes.
23 23 This allows the extension to be disabled if the centralized workflow
24 24 becomes burdensome. However, the pre and post convert clones will
25 25 not be able to communicate with each other unless the extension is
26 26 enabled on both.
27 27
28 28 To start a new repository, or to add LFS files to an existing one, just
29 29 create an ``.hglfs`` file as described below in the root directory of
30 30 the repository. Typically, this file should be put under version
31 31 control, so that the settings will propagate to other repositories with
32 32 push and pull. During any commit, Mercurial will consult this file to
33 33 determine if an added or modified file should be stored externally. The
34 34 type of storage depends on the characteristics of the file at each
35 35 commit. A file that is near a size threshold may switch back and forth
36 36 between LFS and normal storage, as needed.
37 37
38 38 Alternately, both normal repositories and largefile controlled
39 39 repositories can be converted to LFS by using :hg:`convert` and the
40 40 ``lfs.track`` config option described below. The ``.hglfs`` file
41 41 should then be created and added, to control subsequent LFS selection.
42 42 The hashes are also unchanged in this case. The LFS and non-LFS
43 43 repositories can be distinguished because the LFS repository will
44 44 abort any command if this extension is disabled.
45 45
46 46 Committed LFS files are held locally, until the repository is pushed.
47 47 Prior to pushing the normal repository data, the LFS files that are
48 48 tracked by the outgoing commits are automatically uploaded to the
49 49 configured central server. No LFS files are transferred on
50 50 :hg:`pull` or :hg:`clone`. Instead, the files are downloaded on
51 51 demand as they need to be read, if a cached copy cannot be found
52 52 locally. Both committing and downloading an LFS file will link the
53 53 file to a usercache, to speed up future access. See the `usercache`
54 54 config setting described below.
55 55
56 56 .hglfs::
57 57
58 58 The extension reads its configuration from a versioned ``.hglfs``
59 59 configuration file found in the root of the working directory. The
60 60 ``.hglfs`` file uses the same syntax as all other Mercurial
61 61 configuration files. It uses a single section, ``[track]``.
62 62
63 63 The ``[track]`` section specifies which files are stored as LFS (or
64 64 not). Each line is keyed by a file pattern, with a predicate value.
65 65 The first file pattern match is used, so put more specific patterns
66 66 first. The available predicates are ``all()``, ``none()``, and
67 67 ``size()``. See "hg help filesets.size" for the latter.
68 68
69 69 Example versioned ``.hglfs`` file::
70 70
71 71 [track]
72 72 # No Makefile or python file, anywhere, will be LFS
73 73 **Makefile = none()
74 74 **.py = none()
75 75
76 76 **.zip = all()
77 77 **.exe = size(">1MB")
78 78
79 79 # Catchall for everything not matched above
80 80 ** = size(">10MB")
81 81
82 82 Configs::
83 83
84 84 [lfs]
85 85 # Remote endpoint. Multiple protocols are supported:
86 86 # - http(s)://user:pass@example.com/path
87 87 # git-lfs endpoint
88 88 # - file:///tmp/path
89 89 # local filesystem, usually for testing
90 90 # if unset, lfs will assume the remote repository also handles blob storage
91 91 # for http(s) URLs. Otherwise, lfs will prompt to set this when it must
92 92 # use this value.
93 93 # (default: unset)
94 94 url = https://example.com/repo.git/info/lfs
95 95
96 96 # Which files to track in LFS. Path tests are "**.extname" for file
97 97 # extensions, and "path:under/some/directory" for path prefix. Both
98 98 # are relative to the repository root.
99 99 # File size can be tested with the "size()" fileset, and tests can be
100 100 # joined with fileset operators. (See "hg help filesets.operators".)
101 101 #
102 102 # Some examples:
103 103 # - all() # everything
104 104 # - none() # nothing
105 105 # - size(">20MB") # larger than 20MB
106 106 # - !**.txt # anything not a *.txt file
107 107 # - **.zip | **.tar.gz | **.7z # some types of compressed files
108 108 # - path:bin # files under "bin" in the project root
109 109 # - (**.php & size(">2MB")) | (**.js & size(">5MB")) | **.tar.gz
110 110 # | (path:bin & !path:/bin/README) | size(">1GB")
111 111 # (default: none())
112 112 #
113 113 # This is ignored if there is a tracked '.hglfs' file, and this setting
114 114 # will eventually be deprecated and removed.
115 115 track = size(">10M")
116 116
117 117 # how many times to retry before giving up on transferring an object
118 118 retry = 5
119 119
120 120 # the local directory to store lfs files for sharing across local clones.
121 121 # If not set, the cache is located in an OS specific cache location.
122 122 usercache = /path/to/global/cache
123 123 """
124 124
125 125 from __future__ import absolute_import
126 126
127 127 from mercurial.i18n import _
128 128
129 129 from mercurial import (
130 130 bundle2,
131 131 changegroup,
132 132 cmdutil,
133 133 config,
134 134 context,
135 135 error,
136 136 exchange,
137 137 extensions,
138 138 filelog,
139 139 filesetlang,
140 hg,
141 140 localrepo,
142 141 minifileset,
143 142 node,
144 143 pycompat,
145 144 registrar,
146 145 repository,
147 146 revlog,
148 147 scmutil,
149 148 templateutil,
150 149 upgrade,
151 150 util,
152 151 vfs as vfsmod,
153 152 wireprotoserver,
154 153 wireprotov1server,
155 154 )
156 155
157 156 from . import (
158 157 blobstore,
159 158 wireprotolfsserver,
160 159 wrapper,
161 160 )
162 161
163 162 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
164 163 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
165 164 # be specifying the version(s) of Mercurial they are tested with, or
166 165 # leave the attribute unspecified.
167 166 testedwith = 'ships-with-hg-core'
168 167
169 168 configtable = {}
170 169 configitem = registrar.configitem(configtable)
171 170
172 171 configitem('experimental', 'lfs.serve',
173 172 default=True,
174 173 )
175 174 configitem('experimental', 'lfs.user-agent',
176 175 default=None,
177 176 )
178 177 configitem('experimental', 'lfs.disableusercache',
179 178 default=False,
180 179 )
181 180 configitem('experimental', 'lfs.worker-enable',
182 181 default=False,
183 182 )
184 183
185 184 configitem('lfs', 'url',
186 185 default=None,
187 186 )
188 187 configitem('lfs', 'usercache',
189 188 default=None,
190 189 )
191 190 # Deprecated
192 191 configitem('lfs', 'threshold',
193 192 default=None,
194 193 )
195 194 configitem('lfs', 'track',
196 195 default='none()',
197 196 )
198 197 configitem('lfs', 'retry',
199 198 default=5,
200 199 )
201 200
202 201 cmdtable = {}
203 202 command = registrar.command(cmdtable)
204 203
205 204 templatekeyword = registrar.templatekeyword()
206 205 filesetpredicate = registrar.filesetpredicate()
207 206
208 207 def featuresetup(ui, supported):
209 208 # don't die on seeing a repo with the lfs requirement
210 209 supported |= {'lfs'}
211 210
212 211 def uisetup(ui):
213 212 localrepo.featuresetupfuncs.add(featuresetup)
214 213
215 214 def reposetup(ui, repo):
216 215 # Nothing to do with a remote repo
217 216 if not repo.local():
218 217 return
219 218
220 219 repo.svfs.lfslocalblobstore = blobstore.local(repo)
221 220 repo.svfs.lfsremoteblobstore = blobstore.remote(repo)
222 221
223 222 class lfsrepo(repo.__class__):
224 223 @localrepo.unfilteredmethod
225 224 def commitctx(self, ctx, error=False):
226 225 repo.svfs.options['lfstrack'] = _trackedmatcher(self)
227 226 return super(lfsrepo, self).commitctx(ctx, error)
228 227
229 228 repo.__class__ = lfsrepo
230 229
231 230 if 'lfs' not in repo.requirements:
232 231 def checkrequireslfs(ui, repo, **kwargs):
233 232 if 'lfs' not in repo.requirements:
234 233 last = kwargs.get(r'node_last')
235 234 _bin = node.bin
236 235 if last:
237 236 s = repo.set('%n:%n', _bin(kwargs[r'node']), _bin(last))
238 237 else:
239 238 s = repo.set('%n', _bin(kwargs[r'node']))
240 239 match = repo.narrowmatch()
241 240 for ctx in s:
242 241 # TODO: is there a way to just walk the files in the commit?
243 242 if any(ctx[f].islfs() for f in ctx.files()
244 243 if f in ctx and match(f)):
245 244 repo.requirements.add('lfs')
246 245 repo.features.add(repository.REPO_FEATURE_LFS)
247 246 repo._writerequirements()
248 247 repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
249 248 break
250 249
251 250 ui.setconfig('hooks', 'commit.lfs', checkrequireslfs, 'lfs')
252 251 ui.setconfig('hooks', 'pretxnchangegroup.lfs', checkrequireslfs, 'lfs')
253 252 else:
254 253 repo.prepushoutgoinghooks.add('lfs', wrapper.prepush)
255 254
256 255 def _trackedmatcher(repo):
257 256 """Return a function (path, size) -> bool indicating whether or not to
258 257 track a given file with lfs."""
259 258 if not repo.wvfs.exists('.hglfs'):
260 259 # No '.hglfs' in wdir. Fallback to config for now.
261 260 trackspec = repo.ui.config('lfs', 'track')
262 261
263 262 # deprecated config: lfs.threshold
264 263 threshold = repo.ui.configbytes('lfs', 'threshold')
265 264 if threshold:
266 265 filesetlang.parse(trackspec) # make sure syntax errors are confined
267 266 trackspec = "(%s) | size('>%d')" % (trackspec, threshold)
268 267
269 268 return minifileset.compile(trackspec)
270 269
271 270 data = repo.wvfs.tryread('.hglfs')
272 271 if not data:
273 272 return lambda p, s: False
274 273
275 274 # Parse errors here will abort with a message that points to the .hglfs file
276 275 # and line number.
277 276 cfg = config.config()
278 277 cfg.parse('.hglfs', data)
279 278
280 279 try:
281 280 rules = [(minifileset.compile(pattern), minifileset.compile(rule))
282 281 for pattern, rule in cfg.items('track')]
283 282 except error.ParseError as e:
284 283 # The original exception gives no indicator that the error is in the
285 284 # .hglfs file, so add that.
286 285
287 286 # TODO: See if the line number of the file can be made available.
288 287 raise error.Abort(_('parse error in .hglfs: %s') % e)
289 288
290 289 def _match(path, size):
291 290 for pat, rule in rules:
292 291 if pat(path, size):
293 292 return rule(path, size)
294 293
295 294 return False
296 295
297 296 return _match
298 297
299 298 def wrapfilelog(filelog):
300 299 wrapfunction = extensions.wrapfunction
301 300
302 301 wrapfunction(filelog, 'addrevision', wrapper.filelogaddrevision)
303 302 wrapfunction(filelog, 'renamed', wrapper.filelogrenamed)
304 303 wrapfunction(filelog, 'size', wrapper.filelogsize)
305 304
306 305 def extsetup(ui):
307 306 wrapfilelog(filelog.filelog)
308 307
309 308 wrapfunction = extensions.wrapfunction
310 309
311 310 wrapfunction(localrepo, 'makefilestorage', wrapper.localrepomakefilestorage)
312 311
313 312 wrapfunction(cmdutil, '_updatecatformatter', wrapper._updatecatformatter)
314 313 wrapfunction(scmutil, 'wrapconvertsink', wrapper.convertsink)
315 314
316 315 wrapfunction(upgrade, '_finishdatamigration',
317 316 wrapper.upgradefinishdatamigration)
318 317
319 318 wrapfunction(upgrade, 'preservedrequirements',
320 319 wrapper.upgraderequirements)
321 320
322 321 wrapfunction(upgrade, 'supporteddestrequirements',
323 322 wrapper.upgraderequirements)
324 323
325 324 wrapfunction(changegroup,
326 325 'allsupportedversions',
327 326 wrapper.allsupportedversions)
328 327
329 328 wrapfunction(exchange, 'push', wrapper.push)
330 329 wrapfunction(wireprotov1server, '_capabilities', wrapper._capabilities)
331 330 wrapfunction(wireprotoserver, 'handlewsgirequest',
332 331 wireprotolfsserver.handlewsgirequest)
333 332
334 333 wrapfunction(context.basefilectx, 'cmp', wrapper.filectxcmp)
335 334 wrapfunction(context.basefilectx, 'isbinary', wrapper.filectxisbinary)
336 335 context.basefilectx.islfs = wrapper.filectxislfs
337 336
338 337 revlog.addflagprocessor(
339 338 revlog.REVIDX_EXTSTORED,
340 339 (
341 340 wrapper.readfromstore,
342 341 wrapper.writetostore,
343 342 wrapper.bypasscheckhash,
344 343 ),
345 344 )
346 345
347 346 scmutil.fileprefetchhooks.add('lfs', wrapper._prefetchfiles)
348 347
349 348 # Make bundle choose changegroup3 instead of changegroup2. This affects
350 349 # "hg bundle" command. Note: it does not cover all bundle formats like
351 350 # "packed1". Using "packed1" with lfs will likely cause trouble.
352 351 exchange._bundlespeccontentopts["v2"]["cg.version"] = "03"
353 352
354 353 # bundlerepo uses "vfsmod.readonlyvfs(othervfs)", we need to make sure lfs
355 354 # options and blob stores are passed from othervfs to the new readonlyvfs.
356 355 wrapfunction(vfsmod.readonlyvfs, '__init__', wrapper.vfsinit)
357 356
358 357 # when writing a bundle via "hg bundle" command, upload related LFS blobs
359 358 wrapfunction(bundle2, 'writenewbundle', wrapper.writenewbundle)
360 359
361 360 @filesetpredicate('lfs()')
362 361 def lfsfileset(mctx, x):
363 362 """File that uses LFS storage."""
364 363 # i18n: "lfs" is a keyword
365 364 filesetlang.getargs(x, 0, 0, _("lfs takes no arguments"))
366 365 ctx = mctx.ctx
367 366 def lfsfilep(f):
368 367 return wrapper.pointerfromctx(ctx, f, removed=True) is not None
369 368 return mctx.predicate(lfsfilep, predrepr='<lfs>')
370 369
371 370 @templatekeyword('lfs_files', requires={'ctx'})
372 371 def lfsfiles(context, mapping):
373 372 """List of strings. All files modified, added, or removed by this
374 373 changeset."""
375 374 ctx = context.resource(mapping, 'ctx')
376 375
377 376 pointers = wrapper.pointersfromctx(ctx, removed=True) # {path: pointer}
378 377 files = sorted(pointers.keys())
379 378
380 379 def pointer(v):
381 380 # In the file spec, version is first and the other keys are sorted.
382 381 sortkeyfunc = lambda x: (x[0] != 'version', x)
383 382 items = sorted(pointers[v].iteritems(), key=sortkeyfunc)
384 383 return util.sortdict(items)
385 384
386 385 makemap = lambda v: {
387 386 'file': v,
388 387 'lfsoid': pointers[v].oid() if pointers[v] else None,
389 388 'lfspointer': templateutil.hybriddict(pointer(v)),
390 389 }
391 390
392 391 # TODO: make the separator ', '?
393 392 f = templateutil._showcompatlist(context, mapping, 'lfs_file', files)
394 393 return templateutil.hybrid(f, files, makemap, pycompat.identity)
395 394
396 395 @command('debuglfsupload',
397 396 [('r', 'rev', [], _('upload large files introduced by REV'))])
398 397 def debuglfsupload(ui, repo, **opts):
399 398 """upload lfs blobs added by the working copy parent or given revisions"""
400 399 revs = opts.get(r'rev', [])
401 400 pointers = wrapper.extractpointers(repo, scmutil.revrange(repo, revs))
402 401 wrapper.uploadblobs(repo, pointers)
General Comments 0
You need to be logged in to leave comments. Login now