##// END OF EJS Templates
fsmonitor: add support for extra `hg debuginstall` data...
Augie Fackler -
r42876:3358dc6e default draft
parent child Browse files
Show More
@@ -1,832 +1,850 b''
1 1 # __init__.py - fsmonitor initialization and overrides
2 2 #
3 3 # Copyright 2013-2016 Facebook, Inc.
4 4 #
5 5 # This software may be used and distributed according to the terms of the
6 6 # GNU General Public License version 2 or any later version.
7 7
8 8 '''Faster status operations with the Watchman file monitor (EXPERIMENTAL)
9 9
10 10 Integrates the file-watching program Watchman with Mercurial to produce faster
11 11 status results.
12 12
13 13 On a particular Linux system, for a real-world repository with over 400,000
14 14 files hosted on ext4, vanilla `hg status` takes 1.3 seconds. On the same
15 15 system, with fsmonitor it takes about 0.3 seconds.
16 16
17 17 fsmonitor requires no configuration -- it will tell Watchman about your
18 18 repository as necessary. You'll need to install Watchman from
19 19 https://facebook.github.io/watchman/ and make sure it is in your PATH.
20 20
21 21 fsmonitor is incompatible with the largefiles and eol extensions, and
22 22 will disable itself if any of those are active.
23 23
24 24 The following configuration options exist:
25 25
26 26 ::
27 27
28 28 [fsmonitor]
29 29 mode = {off, on, paranoid}
30 30
31 31 When `mode = off`, fsmonitor will disable itself (similar to not loading the
32 32 extension at all). When `mode = on`, fsmonitor will be enabled (the default).
33 33 When `mode = paranoid`, fsmonitor will query both Watchman and the filesystem,
34 34 and ensure that the results are consistent.
35 35
36 36 ::
37 37
38 38 [fsmonitor]
39 39 timeout = (float)
40 40
41 41 A value, in seconds, that determines how long fsmonitor will wait for Watchman
42 42 to return results. Defaults to `2.0`.
43 43
44 44 ::
45 45
46 46 [fsmonitor]
47 47 blacklistusers = (list of userids)
48 48
49 49 A list of usernames for which fsmonitor will disable itself altogether.
50 50
51 51 ::
52 52
53 53 [fsmonitor]
54 54 walk_on_invalidate = (boolean)
55 55
56 56 Whether or not to walk the whole repo ourselves when our cached state has been
57 57 invalidated, for example when Watchman has been restarted or .hgignore rules
58 58 have been changed. Walking the repo in that case can result in competing for
59 59 I/O with Watchman. For large repos it is recommended to set this value to
60 60 false. You may wish to set this to true if you have a very fast filesystem
61 61 that can outpace the IPC overhead of getting the result data for the full repo
62 62 from Watchman. Defaults to false.
63 63
64 64 ::
65 65
66 66 [fsmonitor]
67 67 warn_when_unused = (boolean)
68 68
69 69 Whether to print a warning during certain operations when fsmonitor would be
70 70 beneficial to performance but isn't enabled.
71 71
72 72 ::
73 73
74 74 [fsmonitor]
75 75 warn_update_file_count = (integer)
76 76
77 77 If ``warn_when_unused`` is set and fsmonitor isn't enabled, a warning will
78 78 be printed during working directory updates if this many files will be
79 79 created.
80 80 '''
81 81
82 82 # Platforms Supported
83 83 # ===================
84 84 #
85 85 # **Linux:** *Stable*. Watchman and fsmonitor are both known to work reliably,
86 86 # even under severe loads.
87 87 #
88 88 # **Mac OS X:** *Stable*. The Mercurial test suite passes with fsmonitor
89 89 # turned on, on case-insensitive HFS+. There has been a reasonable amount of
90 90 # user testing under normal loads.
91 91 #
92 92 # **Solaris, BSD:** *Alpha*. watchman and fsmonitor are believed to work, but
93 93 # very little testing has been done.
94 94 #
95 95 # **Windows:** *Alpha*. Not in a release version of watchman or fsmonitor yet.
96 96 #
97 97 # Known Issues
98 98 # ============
99 99 #
100 100 # * fsmonitor will disable itself if any of the following extensions are
101 101 # enabled: largefiles, inotify, eol; or if the repository has subrepos.
102 102 # * fsmonitor will produce incorrect results if nested repos that are not
103 103 # subrepos exist. *Workaround*: add nested repo paths to your `.hgignore`.
104 104 #
105 105 # The issues related to nested repos and subrepos are probably not fundamental
106 106 # ones. Patches to fix them are welcome.
107 107
108 108 from __future__ import absolute_import
109 109
110 110 import codecs
111 111 import hashlib
112 112 import os
113 113 import stat
114 114 import sys
115 import tempfile
115 116 import weakref
116 117
117 118 from mercurial.i18n import _
118 119 from mercurial.node import (
119 120 hex,
120 121 )
121 122
122 123 from mercurial import (
123 124 context,
124 125 encoding,
125 126 error,
126 127 extensions,
127 128 localrepo,
128 129 merge,
129 130 pathutil,
130 131 pycompat,
131 132 registrar,
132 133 scmutil,
133 134 util,
134 135 )
135 136 from mercurial import match as matchmod
136 137
137 138 from . import (
138 139 pywatchman,
139 140 state,
140 141 watchmanclient,
141 142 )
142 143
143 144 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
144 145 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
145 146 # be specifying the version(s) of Mercurial they are tested with, or
146 147 # leave the attribute unspecified.
147 148 testedwith = 'ships-with-hg-core'
148 149
149 150 configtable = {}
150 151 configitem = registrar.configitem(configtable)
151 152
152 153 configitem('fsmonitor', 'mode',
153 154 default='on',
154 155 )
155 156 configitem('fsmonitor', 'walk_on_invalidate',
156 157 default=False,
157 158 )
158 159 configitem('fsmonitor', 'timeout',
159 160 default='2',
160 161 )
161 162 configitem('fsmonitor', 'blacklistusers',
162 163 default=list,
163 164 )
164 165 configitem('fsmonitor', 'watchman_exe',
165 166 default='watchman',
166 167 )
167 168 configitem('fsmonitor', 'verbose',
168 169 default=True,
169 170 )
170 171 configitem('experimental', 'fsmonitor.transaction_notify',
171 172 default=False,
172 173 )
173 174
174 175 # This extension is incompatible with the following blacklisted extensions
175 176 # and will disable itself when encountering one of these:
176 177 _blacklist = ['largefiles', 'eol']
177 178
179 def debuginstall(ui, fm):
180 fm.write("fsmonitor-watchman",
181 _("fsmonitor checking for watchman binary... (%s)\n"),
182 ui.configpath("fsmonitor", "watchman_exe"))
183 root = tempfile.mkdtemp()
184 c = watchmanclient.client(ui, root)
185 err = None
186 try:
187 v = c.command("version")
188 fm.write("fsmonitor-watchman-version",
189 _(" watchman binary version %s\n"), v["version"])
190 except watchmanclient.Unavailable as e:
191 err = str(e)
192 fm.condwrite(err, "fsmonitor-watchman-error",
193 _(" watchman binary missing or broken: %s\n"), err)
194 return 1 if err else 0
195
178 196 def _handleunavailable(ui, state, ex):
179 197 """Exception handler for Watchman interaction exceptions"""
180 198 if isinstance(ex, watchmanclient.Unavailable):
181 199 # experimental config: fsmonitor.verbose
182 200 if ex.warn and ui.configbool('fsmonitor', 'verbose'):
183 201 if 'illegal_fstypes' not in str(ex):
184 202 ui.warn(str(ex) + '\n')
185 203 if ex.invalidate:
186 204 state.invalidate()
187 205 # experimental config: fsmonitor.verbose
188 206 if ui.configbool('fsmonitor', 'verbose'):
189 207 ui.log('fsmonitor', 'Watchman unavailable: %s\n', ex.msg)
190 208 else:
191 209 ui.log('fsmonitor', 'Watchman exception: %s\n', ex)
192 210
193 211 def _hashignore(ignore):
194 212 """Calculate hash for ignore patterns and filenames
195 213
196 214 If this information changes between Mercurial invocations, we can't
197 215 rely on Watchman information anymore and have to re-scan the working
198 216 copy.
199 217
200 218 """
201 219 sha1 = hashlib.sha1()
202 220 sha1.update(repr(ignore))
203 221 return sha1.hexdigest()
204 222
205 223 _watchmanencoding = pywatchman.encoding.get_local_encoding()
206 224 _fsencoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
207 225 _fixencoding = codecs.lookup(_watchmanencoding) != codecs.lookup(_fsencoding)
208 226
209 227 def _watchmantofsencoding(path):
210 228 """Fix path to match watchman and local filesystem encoding
211 229
212 230 watchman's paths encoding can differ from filesystem encoding. For example,
213 231 on Windows, it's always utf-8.
214 232 """
215 233 try:
216 234 decoded = path.decode(_watchmanencoding)
217 235 except UnicodeDecodeError as e:
218 236 raise error.Abort(str(e), hint='watchman encoding error')
219 237
220 238 try:
221 239 encoded = decoded.encode(_fsencoding, 'strict')
222 240 except UnicodeEncodeError as e:
223 241 raise error.Abort(str(e))
224 242
225 243 return encoded
226 244
227 245 def overridewalk(orig, self, match, subrepos, unknown, ignored, full=True):
228 246 '''Replacement for dirstate.walk, hooking into Watchman.
229 247
230 248 Whenever full is False, ignored is False, and the Watchman client is
231 249 available, use Watchman combined with saved state to possibly return only a
232 250 subset of files.'''
233 251 def bail(reason):
234 252 self._ui.debug('fsmonitor: fallback to core status, %s\n' % reason)
235 253 return orig(match, subrepos, unknown, ignored, full=True)
236 254
237 255 if full:
238 256 return bail('full rewalk requested')
239 257 if ignored:
240 258 return bail('listing ignored files')
241 259 if not self._watchmanclient.available():
242 260 return bail('client unavailable')
243 261 state = self._fsmonitorstate
244 262 clock, ignorehash, notefiles = state.get()
245 263 if not clock:
246 264 if state.walk_on_invalidate:
247 265 return bail('no clock')
248 266 # Initial NULL clock value, see
249 267 # https://facebook.github.io/watchman/docs/clockspec.html
250 268 clock = 'c:0:0'
251 269 notefiles = []
252 270
253 271 ignore = self._ignore
254 272 dirignore = self._dirignore
255 273 if unknown:
256 274 if _hashignore(ignore) != ignorehash and clock != 'c:0:0':
257 275 # ignore list changed -- can't rely on Watchman state any more
258 276 if state.walk_on_invalidate:
259 277 return bail('ignore rules changed')
260 278 notefiles = []
261 279 clock = 'c:0:0'
262 280 else:
263 281 # always ignore
264 282 ignore = util.always
265 283 dirignore = util.always
266 284
267 285 matchfn = match.matchfn
268 286 matchalways = match.always()
269 287 dmap = self._map
270 288 if util.safehasattr(dmap, '_map'):
271 289 # for better performance, directly access the inner dirstate map if the
272 290 # standard dirstate implementation is in use.
273 291 dmap = dmap._map
274 292 nonnormalset = self._map.nonnormalset
275 293
276 294 copymap = self._map.copymap
277 295 getkind = stat.S_IFMT
278 296 dirkind = stat.S_IFDIR
279 297 regkind = stat.S_IFREG
280 298 lnkkind = stat.S_IFLNK
281 299 join = self._join
282 300 normcase = util.normcase
283 301 fresh_instance = False
284 302
285 303 exact = skipstep3 = False
286 304 if match.isexact(): # match.exact
287 305 exact = True
288 306 dirignore = util.always # skip step 2
289 307 elif match.prefix(): # match.match, no patterns
290 308 skipstep3 = True
291 309
292 310 if not exact and self._checkcase:
293 311 # note that even though we could receive directory entries, we're only
294 312 # interested in checking if a file with the same name exists. So only
295 313 # normalize files if possible.
296 314 normalize = self._normalizefile
297 315 skipstep3 = False
298 316 else:
299 317 normalize = None
300 318
301 319 # step 1: find all explicit files
302 320 results, work, dirsnotfound = self._walkexplicit(match, subrepos)
303 321
304 322 skipstep3 = skipstep3 and not (work or dirsnotfound)
305 323 work = [d for d in work if not dirignore(d[0])]
306 324
307 325 if not work and (exact or skipstep3):
308 326 for s in subrepos:
309 327 del results[s]
310 328 del results['.hg']
311 329 return results
312 330
313 331 # step 2: query Watchman
314 332 try:
315 333 # Use the user-configured timeout for the query.
316 334 # Add a little slack over the top of the user query to allow for
317 335 # overheads while transferring the data
318 336 self._watchmanclient.settimeout(state.timeout + 0.1)
319 337 result = self._watchmanclient.command('query', {
320 338 'fields': ['mode', 'mtime', 'size', 'exists', 'name'],
321 339 'since': clock,
322 340 'expression': [
323 341 'not', [
324 342 'anyof', ['dirname', '.hg'],
325 343 ['name', '.hg', 'wholename']
326 344 ]
327 345 ],
328 346 'sync_timeout': int(state.timeout * 1000),
329 347 'empty_on_fresh_instance': state.walk_on_invalidate,
330 348 })
331 349 except Exception as ex:
332 350 _handleunavailable(self._ui, state, ex)
333 351 self._watchmanclient.clearconnection()
334 352 return bail('exception during run')
335 353 else:
336 354 # We need to propagate the last observed clock up so that we
337 355 # can use it for our next query
338 356 state.setlastclock(result['clock'])
339 357 if result['is_fresh_instance']:
340 358 if state.walk_on_invalidate:
341 359 state.invalidate()
342 360 return bail('fresh instance')
343 361 fresh_instance = True
344 362 # Ignore any prior noteable files from the state info
345 363 notefiles = []
346 364
347 365 # for file paths which require normalization and we encounter a case
348 366 # collision, we store our own foldmap
349 367 if normalize:
350 368 foldmap = dict((normcase(k), k) for k in results)
351 369
352 370 switch_slashes = pycompat.ossep == '\\'
353 371 # The order of the results is, strictly speaking, undefined.
354 372 # For case changes on a case insensitive filesystem we may receive
355 373 # two entries, one with exists=True and another with exists=False.
356 374 # The exists=True entries in the same response should be interpreted
357 375 # as being happens-after the exists=False entries due to the way that
358 376 # Watchman tracks files. We use this property to reconcile deletes
359 377 # for name case changes.
360 378 for entry in result['files']:
361 379 fname = entry['name']
362 380 if _fixencoding:
363 381 fname = _watchmantofsencoding(fname)
364 382 if switch_slashes:
365 383 fname = fname.replace('\\', '/')
366 384 if normalize:
367 385 normed = normcase(fname)
368 386 fname = normalize(fname, True, True)
369 387 foldmap[normed] = fname
370 388 fmode = entry['mode']
371 389 fexists = entry['exists']
372 390 kind = getkind(fmode)
373 391
374 392 if '/.hg/' in fname or fname.endswith('/.hg'):
375 393 return bail('nested-repo-detected')
376 394
377 395 if not fexists:
378 396 # if marked as deleted and we don't already have a change
379 397 # record, mark it as deleted. If we already have an entry
380 398 # for fname then it was either part of walkexplicit or was
381 399 # an earlier result that was a case change
382 400 if fname not in results and fname in dmap and (
383 401 matchalways or matchfn(fname)):
384 402 results[fname] = None
385 403 elif kind == dirkind:
386 404 if fname in dmap and (matchalways or matchfn(fname)):
387 405 results[fname] = None
388 406 elif kind == regkind or kind == lnkkind:
389 407 if fname in dmap:
390 408 if matchalways or matchfn(fname):
391 409 results[fname] = entry
392 410 elif (matchalways or matchfn(fname)) and not ignore(fname):
393 411 results[fname] = entry
394 412 elif fname in dmap and (matchalways or matchfn(fname)):
395 413 results[fname] = None
396 414
397 415 # step 3: query notable files we don't already know about
398 416 # XXX try not to iterate over the entire dmap
399 417 if normalize:
400 418 # any notable files that have changed case will already be handled
401 419 # above, so just check membership in the foldmap
402 420 notefiles = set((normalize(f, True, True) for f in notefiles
403 421 if normcase(f) not in foldmap))
404 422 visit = set((f for f in notefiles if (f not in results and matchfn(f)
405 423 and (f in dmap or not ignore(f)))))
406 424
407 425 if not fresh_instance:
408 426 if matchalways:
409 427 visit.update(f for f in nonnormalset if f not in results)
410 428 visit.update(f for f in copymap if f not in results)
411 429 else:
412 430 visit.update(f for f in nonnormalset
413 431 if f not in results and matchfn(f))
414 432 visit.update(f for f in copymap
415 433 if f not in results and matchfn(f))
416 434 else:
417 435 if matchalways:
418 436 visit.update(f for f, st in dmap.iteritems() if f not in results)
419 437 visit.update(f for f in copymap if f not in results)
420 438 else:
421 439 visit.update(f for f, st in dmap.iteritems()
422 440 if f not in results and matchfn(f))
423 441 visit.update(f for f in copymap
424 442 if f not in results and matchfn(f))
425 443
426 444 audit = pathutil.pathauditor(self._root, cached=True).check
427 445 auditpass = [f for f in visit if audit(f)]
428 446 auditpass.sort()
429 447 auditfail = visit.difference(auditpass)
430 448 for f in auditfail:
431 449 results[f] = None
432 450
433 451 nf = iter(auditpass).next
434 452 for st in util.statfiles([join(f) for f in auditpass]):
435 453 f = nf()
436 454 if st or f in dmap:
437 455 results[f] = st
438 456
439 457 for s in subrepos:
440 458 del results[s]
441 459 del results['.hg']
442 460 return results
443 461
444 462 def overridestatus(
445 463 orig, self, node1='.', node2=None, match=None, ignored=False,
446 464 clean=False, unknown=False, listsubrepos=False):
447 465 listignored = ignored
448 466 listclean = clean
449 467 listunknown = unknown
450 468
451 469 def _cmpsets(l1, l2):
452 470 try:
453 471 if 'FSMONITOR_LOG_FILE' in encoding.environ:
454 472 fn = encoding.environ['FSMONITOR_LOG_FILE']
455 473 f = open(fn, 'wb')
456 474 else:
457 475 fn = 'fsmonitorfail.log'
458 476 f = self.vfs.open(fn, 'wb')
459 477 except (IOError, OSError):
460 478 self.ui.warn(_('warning: unable to write to %s\n') % fn)
461 479 return
462 480
463 481 try:
464 482 for i, (s1, s2) in enumerate(zip(l1, l2)):
465 483 if set(s1) != set(s2):
466 484 f.write('sets at position %d are unequal\n' % i)
467 485 f.write('watchman returned: %s\n' % s1)
468 486 f.write('stat returned: %s\n' % s2)
469 487 finally:
470 488 f.close()
471 489
472 490 if isinstance(node1, context.changectx):
473 491 ctx1 = node1
474 492 else:
475 493 ctx1 = self[node1]
476 494 if isinstance(node2, context.changectx):
477 495 ctx2 = node2
478 496 else:
479 497 ctx2 = self[node2]
480 498
481 499 working = ctx2.rev() is None
482 500 parentworking = working and ctx1 == self['.']
483 501 match = match or matchmod.always()
484 502
485 503 # Maybe we can use this opportunity to update Watchman's state.
486 504 # Mercurial uses workingcommitctx and/or memctx to represent the part of
487 505 # the workingctx that is to be committed. So don't update the state in
488 506 # that case.
489 507 # HG_PENDING is set in the environment when the dirstate is being updated
490 508 # in the middle of a transaction; we must not update our state in that
491 509 # case, or we risk forgetting about changes in the working copy.
492 510 updatestate = (parentworking and match.always() and
493 511 not isinstance(ctx2, (context.workingcommitctx,
494 512 context.memctx)) and
495 513 'HG_PENDING' not in encoding.environ)
496 514
497 515 try:
498 516 if self._fsmonitorstate.walk_on_invalidate:
499 517 # Use a short timeout to query the current clock. If that
500 518 # takes too long then we assume that the service will be slow
501 519 # to answer our query.
502 520 # walk_on_invalidate indicates that we prefer to walk the
503 521 # tree ourselves because we can ignore portions that Watchman
504 522 # cannot and we tend to be faster in the warmer buffer cache
505 523 # cases.
506 524 self._watchmanclient.settimeout(0.1)
507 525 else:
508 526 # Give Watchman more time to potentially complete its walk
509 527 # and return the initial clock. In this mode we assume that
510 528 # the filesystem will be slower than parsing a potentially
511 529 # very large Watchman result set.
512 530 self._watchmanclient.settimeout(
513 531 self._fsmonitorstate.timeout + 0.1)
514 532 startclock = self._watchmanclient.getcurrentclock()
515 533 except Exception as ex:
516 534 self._watchmanclient.clearconnection()
517 535 _handleunavailable(self.ui, self._fsmonitorstate, ex)
518 536 # boo, Watchman failed. bail
519 537 return orig(node1, node2, match, listignored, listclean,
520 538 listunknown, listsubrepos)
521 539
522 540 if updatestate:
523 541 # We need info about unknown files. This may make things slower the
524 542 # first time, but whatever.
525 543 stateunknown = True
526 544 else:
527 545 stateunknown = listunknown
528 546
529 547 if updatestate:
530 548 ps = poststatus(startclock)
531 549 self.addpostdsstatus(ps)
532 550
533 551 r = orig(node1, node2, match, listignored, listclean, stateunknown,
534 552 listsubrepos)
535 553 modified, added, removed, deleted, unknown, ignored, clean = r
536 554
537 555 if not listunknown:
538 556 unknown = []
539 557
540 558 # don't do paranoid checks if we're not going to query Watchman anyway
541 559 full = listclean or match.traversedir is not None
542 560 if self._fsmonitorstate.mode == 'paranoid' and not full:
543 561 # run status again and fall back to the old walk this time
544 562 self.dirstate._fsmonitordisable = True
545 563
546 564 # shut the UI up
547 565 quiet = self.ui.quiet
548 566 self.ui.quiet = True
549 567 fout, ferr = self.ui.fout, self.ui.ferr
550 568 self.ui.fout = self.ui.ferr = open(os.devnull, 'wb')
551 569
552 570 try:
553 571 rv2 = orig(
554 572 node1, node2, match, listignored, listclean, listunknown,
555 573 listsubrepos)
556 574 finally:
557 575 self.dirstate._fsmonitordisable = False
558 576 self.ui.quiet = quiet
559 577 self.ui.fout, self.ui.ferr = fout, ferr
560 578
561 579 # clean isn't tested since it's set to True above
562 580 with self.wlock():
563 581 _cmpsets(
564 582 [modified, added, removed, deleted, unknown, ignored, clean],
565 583 rv2)
566 584 modified, added, removed, deleted, unknown, ignored, clean = rv2
567 585
568 586 return scmutil.status(
569 587 modified, added, removed, deleted, unknown, ignored, clean)
570 588
571 589 class poststatus(object):
572 590 def __init__(self, startclock):
573 591 self._startclock = startclock
574 592
575 593 def __call__(self, wctx, status):
576 594 clock = wctx.repo()._fsmonitorstate.getlastclock() or self._startclock
577 595 hashignore = _hashignore(wctx.repo().dirstate._ignore)
578 596 notefiles = (status.modified + status.added + status.removed +
579 597 status.deleted + status.unknown)
580 598 wctx.repo()._fsmonitorstate.set(clock, hashignore, notefiles)
581 599
582 600 def makedirstate(repo, dirstate):
583 601 class fsmonitordirstate(dirstate.__class__):
584 602 def _fsmonitorinit(self, repo):
585 603 # _fsmonitordisable is used in paranoid mode
586 604 self._fsmonitordisable = False
587 605 self._fsmonitorstate = repo._fsmonitorstate
588 606 self._watchmanclient = repo._watchmanclient
589 607 self._repo = weakref.proxy(repo)
590 608
591 609 def walk(self, *args, **kwargs):
592 610 orig = super(fsmonitordirstate, self).walk
593 611 if self._fsmonitordisable:
594 612 return orig(*args, **kwargs)
595 613 return overridewalk(orig, self, *args, **kwargs)
596 614
597 615 def rebuild(self, *args, **kwargs):
598 616 self._fsmonitorstate.invalidate()
599 617 return super(fsmonitordirstate, self).rebuild(*args, **kwargs)
600 618
601 619 def invalidate(self, *args, **kwargs):
602 620 self._fsmonitorstate.invalidate()
603 621 return super(fsmonitordirstate, self).invalidate(*args, **kwargs)
604 622
605 623 dirstate.__class__ = fsmonitordirstate
606 624 dirstate._fsmonitorinit(repo)
607 625
608 626 def wrapdirstate(orig, self):
609 627 ds = orig(self)
610 628 # only override the dirstate when Watchman is available for the repo
611 629 if util.safehasattr(self, '_fsmonitorstate'):
612 630 makedirstate(self, ds)
613 631 return ds
614 632
615 633 def extsetup(ui):
616 634 extensions.wrapfilecache(
617 635 localrepo.localrepository, 'dirstate', wrapdirstate)
618 636 if pycompat.isdarwin:
619 637 # An assist for avoiding the dangling-symlink fsevents bug
620 638 extensions.wrapfunction(os, 'symlink', wrapsymlink)
621 639
622 640 extensions.wrapfunction(merge, 'update', wrapupdate)
623 641
624 642 def wrapsymlink(orig, source, link_name):
625 643 ''' if we create a dangling symlink, also touch the parent dir
626 644 to encourage fsevents notifications to work more correctly '''
627 645 try:
628 646 return orig(source, link_name)
629 647 finally:
630 648 try:
631 649 os.utime(os.path.dirname(link_name), None)
632 650 except OSError:
633 651 pass
634 652
635 653 class state_update(object):
636 654 ''' This context manager is responsible for dispatching the state-enter
637 655 and state-leave signals to the watchman service. The enter and leave
638 656 methods can be invoked manually (for scenarios where context manager
639 657 semantics are not possible). If parameters oldnode and newnode are None,
640 658 they will be populated based on current working copy in enter and
641 659 leave, respectively. Similarly, if the distance is none, it will be
642 660 calculated based on the oldnode and newnode in the leave method.'''
643 661
644 662 def __init__(self, repo, name, oldnode=None, newnode=None, distance=None,
645 663 partial=False):
646 664 self.repo = repo.unfiltered()
647 665 self.name = name
648 666 self.oldnode = oldnode
649 667 self.newnode = newnode
650 668 self.distance = distance
651 669 self.partial = partial
652 670 self._lock = None
653 671 self.need_leave = False
654 672
655 673 def __enter__(self):
656 674 self.enter()
657 675
658 676 def enter(self):
659 677 # Make sure we have a wlock prior to sending notifications to watchman.
660 678 # We don't want to race with other actors. In the update case,
661 679 # merge.update is going to take the wlock almost immediately. We are
662 680 # effectively extending the lock around several short sanity checks.
663 681 if self.oldnode is None:
664 682 self.oldnode = self.repo['.'].node()
665 683
666 684 if self.repo.currentwlock() is None:
667 685 if util.safehasattr(self.repo, 'wlocknostateupdate'):
668 686 self._lock = self.repo.wlocknostateupdate()
669 687 else:
670 688 self._lock = self.repo.wlock()
671 689 self.need_leave = self._state(
672 690 'state-enter',
673 691 hex(self.oldnode))
674 692 return self
675 693
676 694 def __exit__(self, type_, value, tb):
677 695 abort = True if type_ else False
678 696 self.exit(abort=abort)
679 697
680 698 def exit(self, abort=False):
681 699 try:
682 700 if self.need_leave:
683 701 status = 'failed' if abort else 'ok'
684 702 if self.newnode is None:
685 703 self.newnode = self.repo['.'].node()
686 704 if self.distance is None:
687 705 self.distance = calcdistance(
688 706 self.repo, self.oldnode, self.newnode)
689 707 self._state(
690 708 'state-leave',
691 709 hex(self.newnode),
692 710 status=status)
693 711 finally:
694 712 self.need_leave = False
695 713 if self._lock:
696 714 self._lock.release()
697 715
698 716 def _state(self, cmd, commithash, status='ok'):
699 717 if not util.safehasattr(self.repo, '_watchmanclient'):
700 718 return False
701 719 try:
702 720 self.repo._watchmanclient.command(cmd, {
703 721 'name': self.name,
704 722 'metadata': {
705 723 # the target revision
706 724 'rev': commithash,
707 725 # approximate number of commits between current and target
708 726 'distance': self.distance if self.distance else 0,
709 727 # success/failure (only really meaningful for state-leave)
710 728 'status': status,
711 729 # whether the working copy parent is changing
712 730 'partial': self.partial,
713 731 }})
714 732 return True
715 733 except Exception as e:
716 734 # Swallow any errors; fire and forget
717 735 self.repo.ui.log(
718 736 'watchman', 'Exception %s while running %s\n', e, cmd)
719 737 return False
720 738
721 739 # Estimate the distance between two nodes
722 740 def calcdistance(repo, oldnode, newnode):
723 741 anc = repo.changelog.ancestor(oldnode, newnode)
724 742 ancrev = repo[anc].rev()
725 743 distance = (abs(repo[oldnode].rev() - ancrev)
726 744 + abs(repo[newnode].rev() - ancrev))
727 745 return distance
728 746
729 747 # Bracket working copy updates with calls to the watchman state-enter
730 748 # and state-leave commands. This allows clients to perform more intelligent
731 749 # settling during bulk file change scenarios
732 750 # https://facebook.github.io/watchman/docs/cmd/subscribe.html#advanced-settling
733 751 def wrapupdate(orig, repo, node, branchmerge, force, ancestor=None,
734 752 mergeancestor=False, labels=None, matcher=None, **kwargs):
735 753
736 754 distance = 0
737 755 partial = True
738 756 oldnode = repo['.'].node()
739 757 newnode = repo[node].node()
740 758 if matcher is None or matcher.always():
741 759 partial = False
742 760 distance = calcdistance(repo.unfiltered(), oldnode, newnode)
743 761
744 762 with state_update(repo, name="hg.update", oldnode=oldnode, newnode=newnode,
745 763 distance=distance, partial=partial):
746 764 return orig(
747 765 repo, node, branchmerge, force, ancestor, mergeancestor,
748 766 labels, matcher, **kwargs)
749 767
750 768 def repo_has_depth_one_nested_repo(repo):
751 769 for f in repo.wvfs.listdir():
752 770 if os.path.isdir(os.path.join(repo.root, f, '.hg')):
753 771 msg = 'fsmonitor: sub-repository %r detected, fsmonitor disabled\n'
754 772 repo.ui.debug(msg % f)
755 773 return True
756 774 return False
757 775
758 776 def reposetup(ui, repo):
759 777 # We don't work with largefiles or inotify
760 778 exts = extensions.enabled()
761 779 for ext in _blacklist:
762 780 if ext in exts:
763 781 ui.warn(_('The fsmonitor extension is incompatible with the %s '
764 782 'extension and has been disabled.\n') % ext)
765 783 return
766 784
767 785 if repo.local():
768 786 # We don't work with subrepos either.
769 787 #
770 788 # if repo[None].substate can cause a dirstate parse, which is too
771 789 # slow. Instead, look for a file called hgsubstate,
772 790 if repo.wvfs.exists('.hgsubstate') or repo.wvfs.exists('.hgsub'):
773 791 return
774 792
775 793 if repo_has_depth_one_nested_repo(repo):
776 794 return
777 795
778 796 fsmonitorstate = state.state(repo)
779 797 if fsmonitorstate.mode == 'off':
780 798 return
781 799
782 800 try:
783 801 client = watchmanclient.client(repo.ui, repo._root)
784 802 except Exception as ex:
785 803 _handleunavailable(ui, fsmonitorstate, ex)
786 804 return
787 805
788 806 repo._fsmonitorstate = fsmonitorstate
789 807 repo._watchmanclient = client
790 808
791 809 dirstate, cached = localrepo.isfilecached(repo, 'dirstate')
792 810 if cached:
793 811 # at this point since fsmonitorstate wasn't present,
794 812 # repo.dirstate is not a fsmonitordirstate
795 813 makedirstate(repo, dirstate)
796 814
797 815 class fsmonitorrepo(repo.__class__):
798 816 def status(self, *args, **kwargs):
799 817 orig = super(fsmonitorrepo, self).status
800 818 return overridestatus(orig, self, *args, **kwargs)
801 819
802 820 def wlocknostateupdate(self, *args, **kwargs):
803 821 return super(fsmonitorrepo, self).wlock(*args, **kwargs)
804 822
805 823 def wlock(self, *args, **kwargs):
806 824 l = super(fsmonitorrepo, self).wlock(*args, **kwargs)
807 825 if not ui.configbool(
808 826 "experimental", "fsmonitor.transaction_notify"):
809 827 return l
810 828 if l.held != 1:
811 829 return l
812 830 origrelease = l.releasefn
813 831
814 832 def staterelease():
815 833 if origrelease:
816 834 origrelease()
817 835 if l.stateupdate:
818 836 l.stateupdate.exit()
819 837 l.stateupdate = None
820 838
821 839 try:
822 840 l.stateupdate = None
823 841 l.stateupdate = state_update(self, name="hg.transaction")
824 842 l.stateupdate.enter()
825 843 l.releasefn = staterelease
826 844 except Exception as e:
827 845 # Swallow any errors; fire and forget
828 846 self.ui.log(
829 847 'watchman', 'Exception in state update %s\n', e)
830 848 return l
831 849
832 850 repo.__class__ = fsmonitorrepo
@@ -1,268 +1,278 b''
1 1 hg debuginstall
2 2 $ hg debuginstall
3 3 checking encoding (ascii)...
4 4 checking Python executable (*) (glob)
5 5 checking Python version (2.*) (glob) (no-py3 !)
6 6 checking Python version (3.*) (glob) (py3 !)
7 7 checking Python lib (.*[Ll]ib.*)... (re)
8 8 checking Python security support (*) (glob)
9 9 TLS 1.2 not supported by Python install; network connections lack modern security (?)
10 10 SNI not supported by Python install; may have connectivity issues with some servers (?)
11 11 checking Mercurial version (*) (glob)
12 12 checking Mercurial custom build (*) (glob)
13 13 checking module policy (*) (glob)
14 14 checking installed modules (*mercurial)... (glob)
15 15 checking registered compression engines (*zlib*) (glob)
16 16 checking available compression engines (*zlib*) (glob)
17 17 checking available compression engines for wire protocol (*zlib*) (glob)
18 18 checking "re2" regexp engine \((available|missing)\) (re)
19 19 checking templates (*mercurial?templates)... (glob)
20 20 checking default template (*mercurial?templates?map-cmdline.default) (glob)
21 21 checking commit editor... (*) (glob)
22 22 checking username (test)
23 23 no problems detected
24 24
25 25 hg debuginstall JSON
26 26 $ hg debuginstall -Tjson | sed 's|\\\\|\\|g'
27 27 [
28 28 {
29 29 "compengines": ["bz2", "bz2truncated", "none", "zlib"*], (glob)
30 30 "compenginesavail": ["bz2", "bz2truncated", "none", "zlib"*], (glob)
31 31 "compenginesserver": [*"zlib"*], (glob)
32 32 "defaulttemplate": "*mercurial?templates?map-cmdline.default", (glob)
33 33 "defaulttemplateerror": null,
34 34 "defaulttemplatenotfound": "default",
35 35 "editor": "*", (glob)
36 36 "editornotfound": false,
37 37 "encoding": "ascii",
38 38 "encodingerror": null,
39 39 "extensionserror": null, (no-pure !)
40 40 "hgmodulepolicy": "*", (glob)
41 41 "hgmodules": "*mercurial", (glob)
42 42 "hgver": "*", (glob)
43 43 "hgverextra": "*", (glob)
44 44 "problems": 0,
45 45 "pythonexe": "*", (glob)
46 46 "pythonlib": "*", (glob)
47 47 "pythonsecurity": [*], (glob)
48 48 "pythonver": "*.*.*", (glob)
49 49 "re2": (true|false), (re)
50 50 "templatedirs": "*mercurial?templates", (glob)
51 51 "username": "test",
52 52 "usernameerror": null,
53 53 "vinotfound": false
54 54 }
55 55 ]
56 56
57 57 hg debuginstall with no username
58 58 $ HGUSER= hg debuginstall
59 59 checking encoding (ascii)...
60 60 checking Python executable (*) (glob)
61 61 checking Python version (2.*) (glob) (no-py3 !)
62 62 checking Python version (3.*) (glob) (py3 !)
63 63 checking Python lib (.*[Ll]ib.*)... (re)
64 64 checking Python security support (*) (glob)
65 65 TLS 1.2 not supported by Python install; network connections lack modern security (?)
66 66 SNI not supported by Python install; may have connectivity issues with some servers (?)
67 67 checking Mercurial version (*) (glob)
68 68 checking Mercurial custom build (*) (glob)
69 69 checking module policy (*) (glob)
70 70 checking installed modules (*mercurial)... (glob)
71 71 checking registered compression engines (*zlib*) (glob)
72 72 checking available compression engines (*zlib*) (glob)
73 73 checking available compression engines for wire protocol (*zlib*) (glob)
74 74 checking "re2" regexp engine \((available|missing)\) (re)
75 75 checking templates (*mercurial?templates)... (glob)
76 76 checking default template (*mercurial?templates?map-cmdline.default) (glob)
77 77 checking commit editor... (*) (glob)
78 78 checking username...
79 79 no username supplied
80 80 (specify a username in your configuration file)
81 81 1 problems detected, please check your install!
82 82 [1]
83 83
84 84 hg debuginstall with invalid encoding
85 85 $ HGENCODING=invalidenc hg debuginstall | grep encoding
86 86 checking encoding (invalidenc)...
87 87 unknown encoding: invalidenc
88 88
89 89 exception message in JSON
90 90
91 91 $ HGENCODING=invalidenc HGUSER= hg debuginstall -Tjson | grep error
92 92 "defaulttemplateerror": null,
93 93 "encodingerror": "unknown encoding: invalidenc",
94 94 "extensionserror": null, (no-pure !)
95 95 "usernameerror": "no username supplied",
96 96
97 97 path variables are expanded (~ is the same as $TESTTMP)
98 98 $ mkdir tools
99 99 $ touch tools/testeditor.exe
100 100 #if execbit
101 101 $ chmod 755 tools/testeditor.exe
102 102 #endif
103 103 $ HGEDITOR="~/tools/testeditor.exe" hg debuginstall
104 104 checking encoding (ascii)...
105 105 checking Python executable (*) (glob)
106 106 checking Python version (2.*) (glob) (no-py3 !)
107 107 checking Python version (3.*) (glob) (py3 !)
108 108 checking Python lib (.*[Ll]ib.*)... (re)
109 109 checking Python security support (*) (glob)
110 110 TLS 1.2 not supported by Python install; network connections lack modern security (?)
111 111 SNI not supported by Python install; may have connectivity issues with some servers (?)
112 112 checking Mercurial version (*) (glob)
113 113 checking Mercurial custom build (*) (glob)
114 114 checking module policy (*) (glob)
115 115 checking installed modules (*mercurial)... (glob)
116 116 checking registered compression engines (*zlib*) (glob)
117 117 checking available compression engines (*zlib*) (glob)
118 118 checking available compression engines for wire protocol (*zlib*) (glob)
119 119 checking "re2" regexp engine \((available|missing)\) (re)
120 120 checking templates (*mercurial?templates)... (glob)
121 121 checking default template (*mercurial?templates?map-cmdline.default) (glob)
122 122 checking commit editor... ($TESTTMP/tools/testeditor.exe)
123 123 checking username (test)
124 124 no problems detected
125 125
126 126 print out the binary post-shlexsplit in the error message when commit editor is
127 127 not found (this is intentionally using backslashes to mimic a windows usecase).
128 128 $ HGEDITOR="c:\foo\bar\baz.exe -y -z" hg debuginstall
129 129 checking encoding (ascii)...
130 130 checking Python executable (*) (glob)
131 131 checking Python version (2.*) (glob) (no-py3 !)
132 132 checking Python version (3.*) (glob) (py3 !)
133 133 checking Python lib (.*[Ll]ib.*)... (re)
134 134 checking Python security support (*) (glob)
135 135 TLS 1.2 not supported by Python install; network connections lack modern security (?)
136 136 SNI not supported by Python install; may have connectivity issues with some servers (?)
137 137 checking Mercurial version (*) (glob)
138 138 checking Mercurial custom build (*) (glob)
139 139 checking module policy (*) (glob)
140 140 checking installed modules (*mercurial)... (glob)
141 141 checking registered compression engines (*zlib*) (glob)
142 142 checking available compression engines (*zlib*) (glob)
143 143 checking available compression engines for wire protocol (*zlib*) (glob)
144 144 checking "re2" regexp engine \((available|missing)\) (re)
145 145 checking templates (*mercurial?templates)... (glob)
146 146 checking default template (*mercurial?templates?map-cmdline.default) (glob)
147 147 checking commit editor... (c:\foo\bar\baz.exe) (windows !)
148 148 Can't find editor 'c:\foo\bar\baz.exe' in PATH (windows !)
149 149 checking commit editor... (c:foobarbaz.exe) (no-windows !)
150 150 Can't find editor 'c:foobarbaz.exe' in PATH (no-windows !)
151 151 (specify a commit editor in your configuration file)
152 152 checking username (test)
153 153 1 problems detected, please check your install!
154 154 [1]
155 155
156 debuginstall extension support
157 $ hg debuginstall --config extensions.fsmonitor= --config fsmonitor.watchman_exe=false | grep atchman
158 fsmonitor checking for watchman binary... (false)
159 watchman binary missing or broken: warning: Watchman unavailable: watchman exited with code 1
160 Verify the json works too:
161 $ hg debuginstall --config extensions.fsmonitor= --config fsmonitor.watchman_exe=false -Tjson | grep atchman
162 "fsmonitor-watchman": "false",
163 "fsmonitor-watchman-error": "warning: Watchman unavailable: watchman exited with code 1",
164
165
156 166 #if test-repo
157 167 $ . "$TESTDIR/helpers-testrepo.sh"
158 168
159 169 $ cat >> wixxml.py << EOF
160 170 > import os
161 171 > import subprocess
162 172 > import sys
163 173 > import xml.etree.ElementTree as ET
164 174 > from mercurial import pycompat
165 175 >
166 176 > # MSYS mangles the path if it expands $TESTDIR
167 177 > testdir = os.environ['TESTDIR']
168 178 > ns = {'wix' : 'http://schemas.microsoft.com/wix/2006/wi'}
169 179 >
170 180 > def directory(node, relpath):
171 181 > '''generator of files in the xml node, rooted at relpath'''
172 182 > dirs = node.findall('./{%(wix)s}Directory' % ns)
173 183 >
174 184 > for d in dirs:
175 185 > for subfile in directory(d, relpath + d.attrib['Name'] + '/'):
176 186 > yield subfile
177 187 >
178 188 > files = node.findall('./{%(wix)s}Component/{%(wix)s}File' % ns)
179 189 >
180 190 > for f in files:
181 191 > yield pycompat.sysbytes(relpath + f.attrib['Name'])
182 192 >
183 193 > def hgdirectory(relpath):
184 194 > '''generator of tracked files, rooted at relpath'''
185 195 > hgdir = "%s/../mercurial" % (testdir)
186 196 > args = ['hg', '--cwd', hgdir, 'files', relpath]
187 197 > proc = subprocess.Popen(args, stdout=subprocess.PIPE,
188 198 > stderr=subprocess.PIPE)
189 199 > output = proc.communicate()[0]
190 200 >
191 201 > for line in output.splitlines():
192 202 > if os.name == 'nt':
193 203 > yield line.replace(pycompat.sysbytes(os.sep), b'/')
194 204 > else:
195 205 > yield line
196 206 >
197 207 > tracked = [f for f in hgdirectory(sys.argv[1])]
198 208 >
199 209 > xml = ET.parse("%s/../contrib/packaging/wix/%s.wxs" % (testdir, sys.argv[1]))
200 210 > root = xml.getroot()
201 211 > dir = root.find('.//{%(wix)s}DirectoryRef' % ns)
202 212 >
203 213 > installed = [f for f in directory(dir, '')]
204 214 >
205 215 > print('Not installed:')
206 216 > for f in sorted(set(tracked) - set(installed)):
207 217 > print(' %s' % pycompat.sysstr(f))
208 218 >
209 219 > print('Not tracked:')
210 220 > for f in sorted(set(installed) - set(tracked)):
211 221 > print(' %s' % pycompat.sysstr(f))
212 222 > EOF
213 223
214 224 $ ( testrepohgenv; "$PYTHON" wixxml.py help )
215 225 Not installed:
216 226 help/common.txt
217 227 help/hg-ssh.8.txt
218 228 help/hg.1.txt
219 229 help/hgignore.5.txt
220 230 help/hgrc.5.txt
221 231 Not tracked:
222 232
223 233 $ ( testrepohgenv; "$PYTHON" wixxml.py templates )
224 234 Not installed:
225 235 Not tracked:
226 236
227 237 #endif
228 238
229 239 #if virtualenv
230 240
231 241 Verify that Mercurial is installable with pip. Note that this MUST be
232 242 the last test in this file, because we do some nasty things to the
233 243 shell environment in order to make the virtualenv work reliably.
234 244
235 245 $ cd $TESTTMP
236 246 Note: --no-site-packages is deprecated, but some places have an
237 247 ancient virtualenv from their linux distro or similar and it's not yet
238 248 the default for them.
239 249 $ unset PYTHONPATH
240 250 $ "$PYTHON" -m virtualenv --no-site-packages --never-download installenv >> pip.log
241 251 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. (?)
242 252 Note: we use this weird path to run pip and hg to avoid platform differences,
243 253 since it's bin on most platforms but Scripts on Windows.
244 254 $ ./installenv/*/pip install --no-index $TESTDIR/.. >> pip.log
245 255 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. (?)
246 256 $ ./installenv/*/hg debuginstall || cat pip.log
247 257 checking encoding (ascii)...
248 258 checking Python executable (*) (glob)
249 259 checking Python version (2.*) (glob) (no-py3 !)
250 260 checking Python version (3.*) (glob) (py3 !)
251 261 checking Python lib (*)... (glob)
252 262 checking Python security support (*) (glob)
253 263 TLS 1.2 not supported by Python install; network connections lack modern security (?)
254 264 SNI not supported by Python install; may have connectivity issues with some servers (?)
255 265 checking Mercurial version (*) (glob)
256 266 checking Mercurial custom build (*) (glob)
257 267 checking module policy (*) (glob)
258 268 checking installed modules (*/mercurial)... (glob)
259 269 checking registered compression engines (*) (glob)
260 270 checking available compression engines (*) (glob)
261 271 checking available compression engines for wire protocol (*) (glob)
262 272 checking "re2" regexp engine \((available|missing)\) (re)
263 273 checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob)
264 274 checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob)
265 275 checking commit editor... (*) (glob)
266 276 checking username (test)
267 277 no problems detected
268 278 #endif
General Comments 0
You need to be logged in to leave comments. Login now