##// END OF EJS Templates
fsmonitor: fix traceback about bytes and str incompatibility...
Raphaël Gomès -
r49126:6fb282ce default
parent child Browse files
Show More
@@ -1,1001 +1,1001
1 # __init__.py - fsmonitor initialization and overrides
1 # __init__.py - fsmonitor initialization and overrides
2 #
2 #
3 # Copyright 2013-2016 Facebook, Inc.
3 # Copyright 2013-2016 Facebook, Inc.
4 #
4 #
5 # This software may be used and distributed according to the terms of the
5 # This software may be used and distributed according to the terms of the
6 # GNU General Public License version 2 or any later version.
6 # GNU General Public License version 2 or any later version.
7
7
8 '''Faster status operations with the Watchman file monitor (EXPERIMENTAL)
8 '''Faster status operations with the Watchman file monitor (EXPERIMENTAL)
9
9
10 Integrates the file-watching program Watchman with Mercurial to produce faster
10 Integrates the file-watching program Watchman with Mercurial to produce faster
11 status results.
11 status results.
12
12
13 On a particular Linux system, for a real-world repository with over 400,000
13 On a particular Linux system, for a real-world repository with over 400,000
14 files hosted on ext4, vanilla `hg status` takes 1.3 seconds. On the same
14 files hosted on ext4, vanilla `hg status` takes 1.3 seconds. On the same
15 system, with fsmonitor it takes about 0.3 seconds.
15 system, with fsmonitor it takes about 0.3 seconds.
16
16
17 fsmonitor requires no configuration -- it will tell Watchman about your
17 fsmonitor requires no configuration -- it will tell Watchman about your
18 repository as necessary. You'll need to install Watchman from
18 repository as necessary. You'll need to install Watchman from
19 https://facebook.github.io/watchman/ and make sure it is in your PATH.
19 https://facebook.github.io/watchman/ and make sure it is in your PATH.
20
20
21 fsmonitor is incompatible with the largefiles and eol extensions, and
21 fsmonitor is incompatible with the largefiles and eol extensions, and
22 will disable itself if any of those are active.
22 will disable itself if any of those are active.
23
23
24 The following configuration options exist:
24 The following configuration options exist:
25
25
26 ::
26 ::
27
27
28 [fsmonitor]
28 [fsmonitor]
29 mode = {off, on, paranoid}
29 mode = {off, on, paranoid}
30
30
31 When `mode = off`, fsmonitor will disable itself (similar to not loading the
31 When `mode = off`, fsmonitor will disable itself (similar to not loading the
32 extension at all). When `mode = on`, fsmonitor will be enabled (the default).
32 extension at all). When `mode = on`, fsmonitor will be enabled (the default).
33 When `mode = paranoid`, fsmonitor will query both Watchman and the filesystem,
33 When `mode = paranoid`, fsmonitor will query both Watchman and the filesystem,
34 and ensure that the results are consistent.
34 and ensure that the results are consistent.
35
35
36 ::
36 ::
37
37
38 [fsmonitor]
38 [fsmonitor]
39 timeout = (float)
39 timeout = (float)
40
40
41 A value, in seconds, that determines how long fsmonitor will wait for Watchman
41 A value, in seconds, that determines how long fsmonitor will wait for Watchman
42 to return results. Defaults to `2.0`.
42 to return results. Defaults to `2.0`.
43
43
44 ::
44 ::
45
45
46 [fsmonitor]
46 [fsmonitor]
47 blacklistusers = (list of userids)
47 blacklistusers = (list of userids)
48
48
49 A list of usernames for which fsmonitor will disable itself altogether.
49 A list of usernames for which fsmonitor will disable itself altogether.
50
50
51 ::
51 ::
52
52
53 [fsmonitor]
53 [fsmonitor]
54 walk_on_invalidate = (boolean)
54 walk_on_invalidate = (boolean)
55
55
56 Whether or not to walk the whole repo ourselves when our cached state has been
56 Whether or not to walk the whole repo ourselves when our cached state has been
57 invalidated, for example when Watchman has been restarted or .hgignore rules
57 invalidated, for example when Watchman has been restarted or .hgignore rules
58 have been changed. Walking the repo in that case can result in competing for
58 have been changed. Walking the repo in that case can result in competing for
59 I/O with Watchman. For large repos it is recommended to set this value to
59 I/O with Watchman. For large repos it is recommended to set this value to
60 false. You may wish to set this to true if you have a very fast filesystem
60 false. You may wish to set this to true if you have a very fast filesystem
61 that can outpace the IPC overhead of getting the result data for the full repo
61 that can outpace the IPC overhead of getting the result data for the full repo
62 from Watchman. Defaults to false.
62 from Watchman. Defaults to false.
63
63
64 ::
64 ::
65
65
66 [fsmonitor]
66 [fsmonitor]
67 warn_when_unused = (boolean)
67 warn_when_unused = (boolean)
68
68
69 Whether to print a warning during certain operations when fsmonitor would be
69 Whether to print a warning during certain operations when fsmonitor would be
70 beneficial to performance but isn't enabled.
70 beneficial to performance but isn't enabled.
71
71
72 ::
72 ::
73
73
74 [fsmonitor]
74 [fsmonitor]
75 warn_update_file_count = (integer)
75 warn_update_file_count = (integer)
76 # or when mercurial is built with rust support
76 # or when mercurial is built with rust support
77 warn_update_file_count_rust = (integer)
77 warn_update_file_count_rust = (integer)
78
78
79 If ``warn_when_unused`` is set and fsmonitor isn't enabled, a warning will
79 If ``warn_when_unused`` is set and fsmonitor isn't enabled, a warning will
80 be printed during working directory updates if this many files will be
80 be printed during working directory updates if this many files will be
81 created.
81 created.
82 '''
82 '''
83
83
84 # Platforms Supported
84 # Platforms Supported
85 # ===================
85 # ===================
86 #
86 #
87 # **Linux:** *Stable*. Watchman and fsmonitor are both known to work reliably,
87 # **Linux:** *Stable*. Watchman and fsmonitor are both known to work reliably,
88 # even under severe loads.
88 # even under severe loads.
89 #
89 #
90 # **Mac OS X:** *Stable*. The Mercurial test suite passes with fsmonitor
90 # **Mac OS X:** *Stable*. The Mercurial test suite passes with fsmonitor
91 # turned on, on case-insensitive HFS+. There has been a reasonable amount of
91 # turned on, on case-insensitive HFS+. There has been a reasonable amount of
92 # user testing under normal loads.
92 # user testing under normal loads.
93 #
93 #
94 # **Solaris, BSD:** *Alpha*. watchman and fsmonitor are believed to work, but
94 # **Solaris, BSD:** *Alpha*. watchman and fsmonitor are believed to work, but
95 # very little testing has been done.
95 # very little testing has been done.
96 #
96 #
97 # **Windows:** *Alpha*. Not in a release version of watchman or fsmonitor yet.
97 # **Windows:** *Alpha*. Not in a release version of watchman or fsmonitor yet.
98 #
98 #
99 # Known Issues
99 # Known Issues
100 # ============
100 # ============
101 #
101 #
102 # * fsmonitor will disable itself if any of the following extensions are
102 # * fsmonitor will disable itself if any of the following extensions are
103 # enabled: largefiles, inotify, eol; or if the repository has subrepos.
103 # enabled: largefiles, inotify, eol; or if the repository has subrepos.
104 # * fsmonitor will produce incorrect results if nested repos that are not
104 # * fsmonitor will produce incorrect results if nested repos that are not
105 # subrepos exist. *Workaround*: add nested repo paths to your `.hgignore`.
105 # subrepos exist. *Workaround*: add nested repo paths to your `.hgignore`.
106 #
106 #
107 # The issues related to nested repos and subrepos are probably not fundamental
107 # The issues related to nested repos and subrepos are probably not fundamental
108 # ones. Patches to fix them are welcome.
108 # ones. Patches to fix them are welcome.
109
109
110 from __future__ import absolute_import
110 from __future__ import absolute_import
111
111
112 import codecs
112 import codecs
113 import os
113 import os
114 import stat
114 import stat
115 import sys
115 import sys
116 import tempfile
116 import tempfile
117 import weakref
117 import weakref
118
118
119 from mercurial.i18n import _
119 from mercurial.i18n import _
120 from mercurial.node import hex
120 from mercurial.node import hex
121 from mercurial.pycompat import open
121 from mercurial.pycompat import open
122 from mercurial import (
122 from mercurial import (
123 context,
123 context,
124 encoding,
124 encoding,
125 error,
125 error,
126 extensions,
126 extensions,
127 localrepo,
127 localrepo,
128 merge,
128 merge,
129 pathutil,
129 pathutil,
130 pycompat,
130 pycompat,
131 registrar,
131 registrar,
132 scmutil,
132 scmutil,
133 util,
133 util,
134 )
134 )
135 from mercurial import match as matchmod
135 from mercurial import match as matchmod
136 from mercurial.utils import (
136 from mercurial.utils import (
137 hashutil,
137 hashutil,
138 stringutil,
138 stringutil,
139 )
139 )
140
140
141 from . import (
141 from . import (
142 pywatchman,
142 pywatchman,
143 state,
143 state,
144 watchmanclient,
144 watchmanclient,
145 )
145 )
146
146
147 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
147 # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
148 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
148 # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
149 # be specifying the version(s) of Mercurial they are tested with, or
149 # be specifying the version(s) of Mercurial they are tested with, or
150 # leave the attribute unspecified.
150 # leave the attribute unspecified.
151 testedwith = b'ships-with-hg-core'
151 testedwith = b'ships-with-hg-core'
152
152
153 configtable = {}
153 configtable = {}
154 configitem = registrar.configitem(configtable)
154 configitem = registrar.configitem(configtable)
155
155
156 configitem(
156 configitem(
157 b'fsmonitor',
157 b'fsmonitor',
158 b'mode',
158 b'mode',
159 default=b'on',
159 default=b'on',
160 )
160 )
161 configitem(
161 configitem(
162 b'fsmonitor',
162 b'fsmonitor',
163 b'walk_on_invalidate',
163 b'walk_on_invalidate',
164 default=False,
164 default=False,
165 )
165 )
166 configitem(
166 configitem(
167 b'fsmonitor',
167 b'fsmonitor',
168 b'timeout',
168 b'timeout',
169 default=b'2',
169 default=b'2',
170 )
170 )
171 configitem(
171 configitem(
172 b'fsmonitor',
172 b'fsmonitor',
173 b'blacklistusers',
173 b'blacklistusers',
174 default=list,
174 default=list,
175 )
175 )
176 configitem(
176 configitem(
177 b'fsmonitor',
177 b'fsmonitor',
178 b'watchman_exe',
178 b'watchman_exe',
179 default=b'watchman',
179 default=b'watchman',
180 )
180 )
181 configitem(
181 configitem(
182 b'fsmonitor',
182 b'fsmonitor',
183 b'verbose',
183 b'verbose',
184 default=True,
184 default=True,
185 experimental=True,
185 experimental=True,
186 )
186 )
187 configitem(
187 configitem(
188 b'experimental',
188 b'experimental',
189 b'fsmonitor.transaction_notify',
189 b'fsmonitor.transaction_notify',
190 default=False,
190 default=False,
191 )
191 )
192
192
193 # This extension is incompatible with the following blacklisted extensions
193 # This extension is incompatible with the following blacklisted extensions
194 # and will disable itself when encountering one of these:
194 # and will disable itself when encountering one of these:
195 _blacklist = [b'largefiles', b'eol']
195 _blacklist = [b'largefiles', b'eol']
196
196
197
197
198 def debuginstall(ui, fm):
198 def debuginstall(ui, fm):
199 fm.write(
199 fm.write(
200 b"fsmonitor-watchman",
200 b"fsmonitor-watchman",
201 _(b"fsmonitor checking for watchman binary... (%s)\n"),
201 _(b"fsmonitor checking for watchman binary... (%s)\n"),
202 ui.configpath(b"fsmonitor", b"watchman_exe"),
202 ui.configpath(b"fsmonitor", b"watchman_exe"),
203 )
203 )
204 root = tempfile.mkdtemp()
204 root = tempfile.mkdtemp()
205 c = watchmanclient.client(ui, root)
205 c = watchmanclient.client(ui, root)
206 err = None
206 err = None
207 try:
207 try:
208 v = c.command(b"version")
208 v = c.command(b"version")
209 fm.write(
209 fm.write(
210 b"fsmonitor-watchman-version",
210 b"fsmonitor-watchman-version",
211 _(b" watchman binary version %s\n"),
211 _(b" watchman binary version %s\n"),
212 pycompat.bytestr(v["version"]),
212 pycompat.bytestr(v["version"]),
213 )
213 )
214 except watchmanclient.Unavailable as e:
214 except watchmanclient.Unavailable as e:
215 err = stringutil.forcebytestr(e)
215 err = stringutil.forcebytestr(e)
216 fm.condwrite(
216 fm.condwrite(
217 err,
217 err,
218 b"fsmonitor-watchman-error",
218 b"fsmonitor-watchman-error",
219 _(b" watchman binary missing or broken: %s\n"),
219 _(b" watchman binary missing or broken: %s\n"),
220 err,
220 err,
221 )
221 )
222 return 1 if err else 0
222 return 1 if err else 0
223
223
224
224
225 def _handleunavailable(ui, state, ex):
225 def _handleunavailable(ui, state, ex):
226 """Exception handler for Watchman interaction exceptions"""
226 """Exception handler for Watchman interaction exceptions"""
227 if isinstance(ex, watchmanclient.Unavailable):
227 if isinstance(ex, watchmanclient.Unavailable):
228 # experimental config: fsmonitor.verbose
228 # experimental config: fsmonitor.verbose
229 if ex.warn and ui.configbool(b'fsmonitor', b'verbose'):
229 if ex.warn and ui.configbool(b'fsmonitor', b'verbose'):
230 if b'illegal_fstypes' not in stringutil.forcebytestr(ex):
230 if b'illegal_fstypes' not in stringutil.forcebytestr(ex):
231 ui.warn(stringutil.forcebytestr(ex) + b'\n')
231 ui.warn(stringutil.forcebytestr(ex) + b'\n')
232 if ex.invalidate:
232 if ex.invalidate:
233 state.invalidate()
233 state.invalidate()
234 # experimental config: fsmonitor.verbose
234 # experimental config: fsmonitor.verbose
235 if ui.configbool(b'fsmonitor', b'verbose'):
235 if ui.configbool(b'fsmonitor', b'verbose'):
236 ui.log(
236 ui.log(
237 b'fsmonitor',
237 b'fsmonitor',
238 b'Watchman unavailable: %s\n',
238 b'Watchman unavailable: %s\n',
239 stringutil.forcebytestr(ex.msg),
239 stringutil.forcebytestr(ex.msg),
240 )
240 )
241 else:
241 else:
242 ui.log(
242 ui.log(
243 b'fsmonitor',
243 b'fsmonitor',
244 b'Watchman exception: %s\n',
244 b'Watchman exception: %s\n',
245 stringutil.forcebytestr(ex),
245 stringutil.forcebytestr(ex),
246 )
246 )
247
247
248
248
249 def _hashignore(ignore):
249 def _hashignore(ignore):
250 """Calculate hash for ignore patterns and filenames
250 """Calculate hash for ignore patterns and filenames
251
251
252 If this information changes between Mercurial invocations, we can't
252 If this information changes between Mercurial invocations, we can't
253 rely on Watchman information anymore and have to re-scan the working
253 rely on Watchman information anymore and have to re-scan the working
254 copy.
254 copy.
255
255
256 """
256 """
257 sha1 = hashutil.sha1()
257 sha1 = hashutil.sha1()
258 sha1.update(pycompat.byterepr(ignore))
258 sha1.update(pycompat.byterepr(ignore))
259 return pycompat.sysbytes(sha1.hexdigest())
259 return pycompat.sysbytes(sha1.hexdigest())
260
260
261
261
262 _watchmanencoding = pywatchman.encoding.get_local_encoding()
262 _watchmanencoding = pywatchman.encoding.get_local_encoding()
263 _fsencoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
263 _fsencoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
264 _fixencoding = codecs.lookup(_watchmanencoding) != codecs.lookup(_fsencoding)
264 _fixencoding = codecs.lookup(_watchmanencoding) != codecs.lookup(_fsencoding)
265
265
266
266
267 def _watchmantofsencoding(path):
267 def _watchmantofsencoding(path):
268 """Fix path to match watchman and local filesystem encoding
268 """Fix path to match watchman and local filesystem encoding
269
269
270 watchman's paths encoding can differ from filesystem encoding. For example,
270 watchman's paths encoding can differ from filesystem encoding. For example,
271 on Windows, it's always utf-8.
271 on Windows, it's always utf-8.
272 """
272 """
273 try:
273 try:
274 decoded = path.decode(_watchmanencoding)
274 decoded = path.decode(_watchmanencoding)
275 except UnicodeDecodeError as e:
275 except UnicodeDecodeError as e:
276 raise error.Abort(
276 raise error.Abort(
277 stringutil.forcebytestr(e), hint=b'watchman encoding error'
277 stringutil.forcebytestr(e), hint=b'watchman encoding error'
278 )
278 )
279
279
280 try:
280 try:
281 encoded = decoded.encode(_fsencoding, 'strict')
281 encoded = decoded.encode(_fsencoding, 'strict')
282 except UnicodeEncodeError as e:
282 except UnicodeEncodeError as e:
283 raise error.Abort(stringutil.forcebytestr(e))
283 raise error.Abort(stringutil.forcebytestr(e))
284
284
285 return encoded
285 return encoded
286
286
287
287
288 def overridewalk(orig, self, match, subrepos, unknown, ignored, full=True):
288 def overridewalk(orig, self, match, subrepos, unknown, ignored, full=True):
289 """Replacement for dirstate.walk, hooking into Watchman.
289 """Replacement for dirstate.walk, hooking into Watchman.
290
290
291 Whenever full is False, ignored is False, and the Watchman client is
291 Whenever full is False, ignored is False, and the Watchman client is
292 available, use Watchman combined with saved state to possibly return only a
292 available, use Watchman combined with saved state to possibly return only a
293 subset of files."""
293 subset of files."""
294
294
295 def bail(reason):
295 def bail(reason):
296 self._ui.debug(b'fsmonitor: fallback to core status, %s\n' % reason)
296 self._ui.debug(b'fsmonitor: fallback to core status, %s\n' % reason)
297 return orig(match, subrepos, unknown, ignored, full=True)
297 return orig(match, subrepos, unknown, ignored, full=True)
298
298
299 if full:
299 if full:
300 return bail(b'full rewalk requested')
300 return bail(b'full rewalk requested')
301 if ignored:
301 if ignored:
302 return bail(b'listing ignored files')
302 return bail(b'listing ignored files')
303 if not self._watchmanclient.available():
303 if not self._watchmanclient.available():
304 return bail(b'client unavailable')
304 return bail(b'client unavailable')
305 state = self._fsmonitorstate
305 state = self._fsmonitorstate
306 clock, ignorehash, notefiles = state.get()
306 clock, ignorehash, notefiles = state.get()
307 if not clock:
307 if not clock:
308 if state.walk_on_invalidate:
308 if state.walk_on_invalidate:
309 return bail(b'no clock')
309 return bail(b'no clock')
310 # Initial NULL clock value, see
310 # Initial NULL clock value, see
311 # https://facebook.github.io/watchman/docs/clockspec.html
311 # https://facebook.github.io/watchman/docs/clockspec.html
312 clock = b'c:0:0'
312 clock = b'c:0:0'
313 notefiles = []
313 notefiles = []
314
314
315 ignore = self._ignore
315 ignore = self._ignore
316 dirignore = self._dirignore
316 dirignore = self._dirignore
317 if unknown:
317 if unknown:
318 if _hashignore(ignore) != ignorehash and clock != b'c:0:0':
318 if _hashignore(ignore) != ignorehash and clock != b'c:0:0':
319 # ignore list changed -- can't rely on Watchman state any more
319 # ignore list changed -- can't rely on Watchman state any more
320 if state.walk_on_invalidate:
320 if state.walk_on_invalidate:
321 return bail(b'ignore rules changed')
321 return bail(b'ignore rules changed')
322 notefiles = []
322 notefiles = []
323 clock = b'c:0:0'
323 clock = b'c:0:0'
324 else:
324 else:
325 # always ignore
325 # always ignore
326 ignore = util.always
326 ignore = util.always
327 dirignore = util.always
327 dirignore = util.always
328
328
329 matchfn = match.matchfn
329 matchfn = match.matchfn
330 matchalways = match.always()
330 matchalways = match.always()
331 dmap = self._map
331 dmap = self._map
332 if util.safehasattr(dmap, b'_map'):
332 if util.safehasattr(dmap, b'_map'):
333 # for better performance, directly access the inner dirstate map if the
333 # for better performance, directly access the inner dirstate map if the
334 # standard dirstate implementation is in use.
334 # standard dirstate implementation is in use.
335 dmap = dmap._map
335 dmap = dmap._map
336 nonnormalset = self._map.nonnormalset
336 nonnormalset = self._map.nonnormalset
337
337
338 copymap = self._map.copymap
338 copymap = self._map.copymap
339 getkind = stat.S_IFMT
339 getkind = stat.S_IFMT
340 dirkind = stat.S_IFDIR
340 dirkind = stat.S_IFDIR
341 regkind = stat.S_IFREG
341 regkind = stat.S_IFREG
342 lnkkind = stat.S_IFLNK
342 lnkkind = stat.S_IFLNK
343 join = self._join
343 join = self._join
344 normcase = util.normcase
344 normcase = util.normcase
345 fresh_instance = False
345 fresh_instance = False
346
346
347 exact = skipstep3 = False
347 exact = skipstep3 = False
348 if match.isexact(): # match.exact
348 if match.isexact(): # match.exact
349 exact = True
349 exact = True
350 dirignore = util.always # skip step 2
350 dirignore = util.always # skip step 2
351 elif match.prefix(): # match.match, no patterns
351 elif match.prefix(): # match.match, no patterns
352 skipstep3 = True
352 skipstep3 = True
353
353
354 if not exact and self._checkcase:
354 if not exact and self._checkcase:
355 # note that even though we could receive directory entries, we're only
355 # note that even though we could receive directory entries, we're only
356 # interested in checking if a file with the same name exists. So only
356 # interested in checking if a file with the same name exists. So only
357 # normalize files if possible.
357 # normalize files if possible.
358 normalize = self._normalizefile
358 normalize = self._normalizefile
359 skipstep3 = False
359 skipstep3 = False
360 else:
360 else:
361 normalize = None
361 normalize = None
362
362
363 # step 1: find all explicit files
363 # step 1: find all explicit files
364 results, work, dirsnotfound = self._walkexplicit(match, subrepos)
364 results, work, dirsnotfound = self._walkexplicit(match, subrepos)
365
365
366 skipstep3 = skipstep3 and not (work or dirsnotfound)
366 skipstep3 = skipstep3 and not (work or dirsnotfound)
367 work = [d for d in work if not dirignore(d[0])]
367 work = [d for d in work if not dirignore(d[0])]
368
368
369 if not work and (exact or skipstep3):
369 if not work and (exact or skipstep3):
370 for s in subrepos:
370 for s in subrepos:
371 del results[s]
371 del results[s]
372 del results[b'.hg']
372 del results[b'.hg']
373 return results
373 return results
374
374
375 # step 2: query Watchman
375 # step 2: query Watchman
376 try:
376 try:
377 # Use the user-configured timeout for the query.
377 # Use the user-configured timeout for the query.
378 # Add a little slack over the top of the user query to allow for
378 # Add a little slack over the top of the user query to allow for
379 # overheads while transferring the data
379 # overheads while transferring the data
380 self._watchmanclient.settimeout(state.timeout + 0.1)
380 self._watchmanclient.settimeout(state.timeout + 0.1)
381 result = self._watchmanclient.command(
381 result = self._watchmanclient.command(
382 b'query',
382 b'query',
383 {
383 {
384 b'fields': [b'mode', b'mtime', b'size', b'exists', b'name'],
384 b'fields': [b'mode', b'mtime', b'size', b'exists', b'name'],
385 b'since': clock,
385 b'since': clock,
386 b'expression': [
386 b'expression': [
387 b'not',
387 b'not',
388 [
388 [
389 b'anyof',
389 b'anyof',
390 [b'dirname', b'.hg'],
390 [b'dirname', b'.hg'],
391 [b'name', b'.hg', b'wholename'],
391 [b'name', b'.hg', b'wholename'],
392 ],
392 ],
393 ],
393 ],
394 b'sync_timeout': int(state.timeout * 1000),
394 b'sync_timeout': int(state.timeout * 1000),
395 b'empty_on_fresh_instance': state.walk_on_invalidate,
395 b'empty_on_fresh_instance': state.walk_on_invalidate,
396 },
396 },
397 )
397 )
398 except Exception as ex:
398 except Exception as ex:
399 _handleunavailable(self._ui, state, ex)
399 _handleunavailable(self._ui, state, ex)
400 self._watchmanclient.clearconnection()
400 self._watchmanclient.clearconnection()
401 return bail(b'exception during run')
401 return bail(b'exception during run')
402 else:
402 else:
403 # We need to propagate the last observed clock up so that we
403 # We need to propagate the last observed clock up so that we
404 # can use it for our next query
404 # can use it for our next query
405 state.setlastclock(pycompat.sysbytes(result[b'clock']))
405 state.setlastclock(pycompat.sysbytes(result[b'clock']))
406 if result[b'is_fresh_instance']:
406 if result[b'is_fresh_instance']:
407 if state.walk_on_invalidate:
407 if state.walk_on_invalidate:
408 state.invalidate()
408 state.invalidate()
409 return bail(b'fresh instance')
409 return bail(b'fresh instance')
410 fresh_instance = True
410 fresh_instance = True
411 # Ignore any prior noteable files from the state info
411 # Ignore any prior noteable files from the state info
412 notefiles = []
412 notefiles = []
413
413
414 # for file paths which require normalization and we encounter a case
414 # for file paths which require normalization and we encounter a case
415 # collision, we store our own foldmap
415 # collision, we store our own foldmap
416 if normalize:
416 if normalize:
417 foldmap = {normcase(k): k for k in results}
417 foldmap = {normcase(k): k for k in results}
418
418
419 switch_slashes = pycompat.ossep == b'\\'
419 switch_slashes = pycompat.ossep == b'\\'
420 # The order of the results is, strictly speaking, undefined.
420 # The order of the results is, strictly speaking, undefined.
421 # For case changes on a case insensitive filesystem we may receive
421 # For case changes on a case insensitive filesystem we may receive
422 # two entries, one with exists=True and another with exists=False.
422 # two entries, one with exists=True and another with exists=False.
423 # The exists=True entries in the same response should be interpreted
423 # The exists=True entries in the same response should be interpreted
424 # as being happens-after the exists=False entries due to the way that
424 # as being happens-after the exists=False entries due to the way that
425 # Watchman tracks files. We use this property to reconcile deletes
425 # Watchman tracks files. We use this property to reconcile deletes
426 # for name case changes.
426 # for name case changes.
427 for entry in result[b'files']:
427 for entry in result[b'files']:
428 fname = entry[b'name']
428 fname = entry[b'name']
429
429
430 # Watchman always give us a str. Normalize to bytes on Python 3
430 # Watchman always give us a str. Normalize to bytes on Python 3
431 # using Watchman's encoding, if needed.
431 # using Watchman's encoding, if needed.
432 if not isinstance(fname, bytes):
432 if not isinstance(fname, bytes):
433 fname = fname.encode(_watchmanencoding)
433 fname = fname.encode(_watchmanencoding)
434
434
435 if _fixencoding:
435 if _fixencoding:
436 fname = _watchmantofsencoding(fname)
436 fname = _watchmantofsencoding(fname)
437
437
438 if switch_slashes:
438 if switch_slashes:
439 fname = fname.replace(b'\\', b'/')
439 fname = fname.replace(b'\\', b'/')
440 if normalize:
440 if normalize:
441 normed = normcase(fname)
441 normed = normcase(fname)
442 fname = normalize(fname, True, True)
442 fname = normalize(fname, True, True)
443 foldmap[normed] = fname
443 foldmap[normed] = fname
444 fmode = entry[b'mode']
444 fmode = entry[b'mode']
445 fexists = entry[b'exists']
445 fexists = entry[b'exists']
446 kind = getkind(fmode)
446 kind = getkind(fmode)
447
447
448 if b'/.hg/' in fname or fname.endswith(b'/.hg'):
448 if b'/.hg/' in fname or fname.endswith(b'/.hg'):
449 return bail(b'nested-repo-detected')
449 return bail(b'nested-repo-detected')
450
450
451 if not fexists:
451 if not fexists:
452 # if marked as deleted and we don't already have a change
452 # if marked as deleted and we don't already have a change
453 # record, mark it as deleted. If we already have an entry
453 # record, mark it as deleted. If we already have an entry
454 # for fname then it was either part of walkexplicit or was
454 # for fname then it was either part of walkexplicit or was
455 # an earlier result that was a case change
455 # an earlier result that was a case change
456 if (
456 if (
457 fname not in results
457 fname not in results
458 and fname in dmap
458 and fname in dmap
459 and (matchalways or matchfn(fname))
459 and (matchalways or matchfn(fname))
460 ):
460 ):
461 results[fname] = None
461 results[fname] = None
462 elif kind == dirkind:
462 elif kind == dirkind:
463 if fname in dmap and (matchalways or matchfn(fname)):
463 if fname in dmap and (matchalways or matchfn(fname)):
464 results[fname] = None
464 results[fname] = None
465 elif kind == regkind or kind == lnkkind:
465 elif kind == regkind or kind == lnkkind:
466 if fname in dmap:
466 if fname in dmap:
467 if matchalways or matchfn(fname):
467 if matchalways or matchfn(fname):
468 results[fname] = entry
468 results[fname] = entry
469 elif (matchalways or matchfn(fname)) and not ignore(fname):
469 elif (matchalways or matchfn(fname)) and not ignore(fname):
470 results[fname] = entry
470 results[fname] = entry
471 elif fname in dmap and (matchalways or matchfn(fname)):
471 elif fname in dmap and (matchalways or matchfn(fname)):
472 results[fname] = None
472 results[fname] = None
473
473
474 # step 3: query notable files we don't already know about
474 # step 3: query notable files we don't already know about
475 # XXX try not to iterate over the entire dmap
475 # XXX try not to iterate over the entire dmap
476 if normalize:
476 if normalize:
477 # any notable files that have changed case will already be handled
477 # any notable files that have changed case will already be handled
478 # above, so just check membership in the foldmap
478 # above, so just check membership in the foldmap
479 notefiles = {
479 notefiles = {
480 normalize(f, True, True)
480 normalize(f, True, True)
481 for f in notefiles
481 for f in notefiles
482 if normcase(f) not in foldmap
482 if normcase(f) not in foldmap
483 }
483 }
484 visit = {
484 visit = {
485 f
485 f
486 for f in notefiles
486 for f in notefiles
487 if (f not in results and matchfn(f) and (f in dmap or not ignore(f)))
487 if (f not in results and matchfn(f) and (f in dmap or not ignore(f)))
488 }
488 }
489
489
490 if not fresh_instance:
490 if not fresh_instance:
491 if matchalways:
491 if matchalways:
492 visit.update(f for f in nonnormalset if f not in results)
492 visit.update(f for f in nonnormalset if f not in results)
493 visit.update(f for f in copymap if f not in results)
493 visit.update(f for f in copymap if f not in results)
494 else:
494 else:
495 visit.update(
495 visit.update(
496 f for f in nonnormalset if f not in results and matchfn(f)
496 f for f in nonnormalset if f not in results and matchfn(f)
497 )
497 )
498 visit.update(f for f in copymap if f not in results and matchfn(f))
498 visit.update(f for f in copymap if f not in results and matchfn(f))
499 else:
499 else:
500 if matchalways:
500 if matchalways:
501 visit.update(
501 visit.update(
502 f for f, st in pycompat.iteritems(dmap) if f not in results
502 f for f, st in pycompat.iteritems(dmap) if f not in results
503 )
503 )
504 visit.update(f for f in copymap if f not in results)
504 visit.update(f for f in copymap if f not in results)
505 else:
505 else:
506 visit.update(
506 visit.update(
507 f
507 f
508 for f, st in pycompat.iteritems(dmap)
508 for f, st in pycompat.iteritems(dmap)
509 if f not in results and matchfn(f)
509 if f not in results and matchfn(f)
510 )
510 )
511 visit.update(f for f in copymap if f not in results and matchfn(f))
511 visit.update(f for f in copymap if f not in results and matchfn(f))
512
512
513 audit = pathutil.pathauditor(self._root, cached=True).check
513 audit = pathutil.pathauditor(self._root, cached=True).check
514 auditpass = [f for f in visit if audit(f)]
514 auditpass = [f for f in visit if audit(f)]
515 auditpass.sort()
515 auditpass.sort()
516 auditfail = visit.difference(auditpass)
516 auditfail = visit.difference(auditpass)
517 for f in auditfail:
517 for f in auditfail:
518 results[f] = None
518 results[f] = None
519
519
520 nf = iter(auditpass)
520 nf = iter(auditpass)
521 for st in util.statfiles([join(f) for f in auditpass]):
521 for st in util.statfiles([join(f) for f in auditpass]):
522 f = next(nf)
522 f = next(nf)
523 if st or f in dmap:
523 if st or f in dmap:
524 results[f] = st
524 results[f] = st
525
525
526 for s in subrepos:
526 for s in subrepos:
527 del results[s]
527 del results[s]
528 del results[b'.hg']
528 del results[b'.hg']
529 return results
529 return results
530
530
531
531
532 def overridestatus(
532 def overridestatus(
533 orig,
533 orig,
534 self,
534 self,
535 node1=b'.',
535 node1=b'.',
536 node2=None,
536 node2=None,
537 match=None,
537 match=None,
538 ignored=False,
538 ignored=False,
539 clean=False,
539 clean=False,
540 unknown=False,
540 unknown=False,
541 listsubrepos=False,
541 listsubrepos=False,
542 ):
542 ):
543 listignored = ignored
543 listignored = ignored
544 listclean = clean
544 listclean = clean
545 listunknown = unknown
545 listunknown = unknown
546
546
547 def _cmpsets(l1, l2):
547 def _cmpsets(l1, l2):
548 try:
548 try:
549 if b'FSMONITOR_LOG_FILE' in encoding.environ:
549 if b'FSMONITOR_LOG_FILE' in encoding.environ:
550 fn = encoding.environ[b'FSMONITOR_LOG_FILE']
550 fn = encoding.environ[b'FSMONITOR_LOG_FILE']
551 f = open(fn, b'wb')
551 f = open(fn, b'wb')
552 else:
552 else:
553 fn = b'fsmonitorfail.log'
553 fn = b'fsmonitorfail.log'
554 f = self.vfs.open(fn, b'wb')
554 f = self.vfs.open(fn, b'wb')
555 except (IOError, OSError):
555 except (IOError, OSError):
556 self.ui.warn(_(b'warning: unable to write to %s\n') % fn)
556 self.ui.warn(_(b'warning: unable to write to %s\n') % fn)
557 return
557 return
558
558
559 try:
559 try:
560 for i, (s1, s2) in enumerate(zip(l1, l2)):
560 for i, (s1, s2) in enumerate(zip(l1, l2)):
561 if set(s1) != set(s2):
561 if set(s1) != set(s2):
562 f.write(b'sets at position %d are unequal\n' % i)
562 f.write(b'sets at position %d are unequal\n' % i)
563 f.write(b'watchman returned: %s\n' % s1)
563 f.write(b'watchman returned: %r\n' % s1)
564 f.write(b'stat returned: %s\n' % s2)
564 f.write(b'stat returned: %r\n' % s2)
565 finally:
565 finally:
566 f.close()
566 f.close()
567
567
568 if isinstance(node1, context.changectx):
568 if isinstance(node1, context.changectx):
569 ctx1 = node1
569 ctx1 = node1
570 else:
570 else:
571 ctx1 = self[node1]
571 ctx1 = self[node1]
572 if isinstance(node2, context.changectx):
572 if isinstance(node2, context.changectx):
573 ctx2 = node2
573 ctx2 = node2
574 else:
574 else:
575 ctx2 = self[node2]
575 ctx2 = self[node2]
576
576
577 working = ctx2.rev() is None
577 working = ctx2.rev() is None
578 parentworking = working and ctx1 == self[b'.']
578 parentworking = working and ctx1 == self[b'.']
579 match = match or matchmod.always()
579 match = match or matchmod.always()
580
580
581 # Maybe we can use this opportunity to update Watchman's state.
581 # Maybe we can use this opportunity to update Watchman's state.
582 # Mercurial uses workingcommitctx and/or memctx to represent the part of
582 # Mercurial uses workingcommitctx and/or memctx to represent the part of
583 # the workingctx that is to be committed. So don't update the state in
583 # the workingctx that is to be committed. So don't update the state in
584 # that case.
584 # that case.
585 # HG_PENDING is set in the environment when the dirstate is being updated
585 # HG_PENDING is set in the environment when the dirstate is being updated
586 # in the middle of a transaction; we must not update our state in that
586 # in the middle of a transaction; we must not update our state in that
587 # case, or we risk forgetting about changes in the working copy.
587 # case, or we risk forgetting about changes in the working copy.
588 updatestate = (
588 updatestate = (
589 parentworking
589 parentworking
590 and match.always()
590 and match.always()
591 and not isinstance(ctx2, (context.workingcommitctx, context.memctx))
591 and not isinstance(ctx2, (context.workingcommitctx, context.memctx))
592 and b'HG_PENDING' not in encoding.environ
592 and b'HG_PENDING' not in encoding.environ
593 )
593 )
594
594
595 try:
595 try:
596 if self._fsmonitorstate.walk_on_invalidate:
596 if self._fsmonitorstate.walk_on_invalidate:
597 # Use a short timeout to query the current clock. If that
597 # Use a short timeout to query the current clock. If that
598 # takes too long then we assume that the service will be slow
598 # takes too long then we assume that the service will be slow
599 # to answer our query.
599 # to answer our query.
600 # walk_on_invalidate indicates that we prefer to walk the
600 # walk_on_invalidate indicates that we prefer to walk the
601 # tree ourselves because we can ignore portions that Watchman
601 # tree ourselves because we can ignore portions that Watchman
602 # cannot and we tend to be faster in the warmer buffer cache
602 # cannot and we tend to be faster in the warmer buffer cache
603 # cases.
603 # cases.
604 self._watchmanclient.settimeout(0.1)
604 self._watchmanclient.settimeout(0.1)
605 else:
605 else:
606 # Give Watchman more time to potentially complete its walk
606 # Give Watchman more time to potentially complete its walk
607 # and return the initial clock. In this mode we assume that
607 # and return the initial clock. In this mode we assume that
608 # the filesystem will be slower than parsing a potentially
608 # the filesystem will be slower than parsing a potentially
609 # very large Watchman result set.
609 # very large Watchman result set.
610 self._watchmanclient.settimeout(self._fsmonitorstate.timeout + 0.1)
610 self._watchmanclient.settimeout(self._fsmonitorstate.timeout + 0.1)
611 startclock = self._watchmanclient.getcurrentclock()
611 startclock = self._watchmanclient.getcurrentclock()
612 except Exception as ex:
612 except Exception as ex:
613 self._watchmanclient.clearconnection()
613 self._watchmanclient.clearconnection()
614 _handleunavailable(self.ui, self._fsmonitorstate, ex)
614 _handleunavailable(self.ui, self._fsmonitorstate, ex)
615 # boo, Watchman failed. bail
615 # boo, Watchman failed. bail
616 return orig(
616 return orig(
617 node1,
617 node1,
618 node2,
618 node2,
619 match,
619 match,
620 listignored,
620 listignored,
621 listclean,
621 listclean,
622 listunknown,
622 listunknown,
623 listsubrepos,
623 listsubrepos,
624 )
624 )
625
625
626 if updatestate:
626 if updatestate:
627 # We need info about unknown files. This may make things slower the
627 # We need info about unknown files. This may make things slower the
628 # first time, but whatever.
628 # first time, but whatever.
629 stateunknown = True
629 stateunknown = True
630 else:
630 else:
631 stateunknown = listunknown
631 stateunknown = listunknown
632
632
633 if updatestate:
633 if updatestate:
634 ps = poststatus(startclock)
634 ps = poststatus(startclock)
635 self.addpostdsstatus(ps)
635 self.addpostdsstatus(ps)
636
636
637 r = orig(
637 r = orig(
638 node1, node2, match, listignored, listclean, stateunknown, listsubrepos
638 node1, node2, match, listignored, listclean, stateunknown, listsubrepos
639 )
639 )
640 modified, added, removed, deleted, unknown, ignored, clean = r
640 modified, added, removed, deleted, unknown, ignored, clean = r
641
641
642 if not listunknown:
642 if not listunknown:
643 unknown = []
643 unknown = []
644
644
645 # don't do paranoid checks if we're not going to query Watchman anyway
645 # don't do paranoid checks if we're not going to query Watchman anyway
646 full = listclean or match.traversedir is not None
646 full = listclean or match.traversedir is not None
647 if self._fsmonitorstate.mode == b'paranoid' and not full:
647 if self._fsmonitorstate.mode == b'paranoid' and not full:
648 # run status again and fall back to the old walk this time
648 # run status again and fall back to the old walk this time
649 self.dirstate._fsmonitordisable = True
649 self.dirstate._fsmonitordisable = True
650
650
651 # shut the UI up
651 # shut the UI up
652 quiet = self.ui.quiet
652 quiet = self.ui.quiet
653 self.ui.quiet = True
653 self.ui.quiet = True
654 fout, ferr = self.ui.fout, self.ui.ferr
654 fout, ferr = self.ui.fout, self.ui.ferr
655 self.ui.fout = self.ui.ferr = open(os.devnull, b'wb')
655 self.ui.fout = self.ui.ferr = open(os.devnull, b'wb')
656
656
657 try:
657 try:
658 rv2 = orig(
658 rv2 = orig(
659 node1,
659 node1,
660 node2,
660 node2,
661 match,
661 match,
662 listignored,
662 listignored,
663 listclean,
663 listclean,
664 listunknown,
664 listunknown,
665 listsubrepos,
665 listsubrepos,
666 )
666 )
667 finally:
667 finally:
668 self.dirstate._fsmonitordisable = False
668 self.dirstate._fsmonitordisable = False
669 self.ui.quiet = quiet
669 self.ui.quiet = quiet
670 self.ui.fout, self.ui.ferr = fout, ferr
670 self.ui.fout, self.ui.ferr = fout, ferr
671
671
672 # clean isn't tested since it's set to True above
672 # clean isn't tested since it's set to True above
673 with self.wlock():
673 with self.wlock():
674 _cmpsets(
674 _cmpsets(
675 [modified, added, removed, deleted, unknown, ignored, clean],
675 [modified, added, removed, deleted, unknown, ignored, clean],
676 rv2,
676 rv2,
677 )
677 )
678 modified, added, removed, deleted, unknown, ignored, clean = rv2
678 modified, added, removed, deleted, unknown, ignored, clean = rv2
679
679
680 return scmutil.status(
680 return scmutil.status(
681 modified, added, removed, deleted, unknown, ignored, clean
681 modified, added, removed, deleted, unknown, ignored, clean
682 )
682 )
683
683
684
684
685 class poststatus(object):
685 class poststatus(object):
686 def __init__(self, startclock):
686 def __init__(self, startclock):
687 self._startclock = pycompat.sysbytes(startclock)
687 self._startclock = pycompat.sysbytes(startclock)
688
688
689 def __call__(self, wctx, status):
689 def __call__(self, wctx, status):
690 clock = wctx.repo()._fsmonitorstate.getlastclock() or self._startclock
690 clock = wctx.repo()._fsmonitorstate.getlastclock() or self._startclock
691 hashignore = _hashignore(wctx.repo().dirstate._ignore)
691 hashignore = _hashignore(wctx.repo().dirstate._ignore)
692 notefiles = (
692 notefiles = (
693 status.modified
693 status.modified
694 + status.added
694 + status.added
695 + status.removed
695 + status.removed
696 + status.deleted
696 + status.deleted
697 + status.unknown
697 + status.unknown
698 )
698 )
699 wctx.repo()._fsmonitorstate.set(clock, hashignore, notefiles)
699 wctx.repo()._fsmonitorstate.set(clock, hashignore, notefiles)
700
700
701
701
702 def makedirstate(repo, dirstate):
702 def makedirstate(repo, dirstate):
703 class fsmonitordirstate(dirstate.__class__):
703 class fsmonitordirstate(dirstate.__class__):
704 def _fsmonitorinit(self, repo):
704 def _fsmonitorinit(self, repo):
705 # _fsmonitordisable is used in paranoid mode
705 # _fsmonitordisable is used in paranoid mode
706 self._fsmonitordisable = False
706 self._fsmonitordisable = False
707 self._fsmonitorstate = repo._fsmonitorstate
707 self._fsmonitorstate = repo._fsmonitorstate
708 self._watchmanclient = repo._watchmanclient
708 self._watchmanclient = repo._watchmanclient
709 self._repo = weakref.proxy(repo)
709 self._repo = weakref.proxy(repo)
710
710
711 def walk(self, *args, **kwargs):
711 def walk(self, *args, **kwargs):
712 orig = super(fsmonitordirstate, self).walk
712 orig = super(fsmonitordirstate, self).walk
713 if self._fsmonitordisable:
713 if self._fsmonitordisable:
714 return orig(*args, **kwargs)
714 return orig(*args, **kwargs)
715 return overridewalk(orig, self, *args, **kwargs)
715 return overridewalk(orig, self, *args, **kwargs)
716
716
717 def rebuild(self, *args, **kwargs):
717 def rebuild(self, *args, **kwargs):
718 self._fsmonitorstate.invalidate()
718 self._fsmonitorstate.invalidate()
719 return super(fsmonitordirstate, self).rebuild(*args, **kwargs)
719 return super(fsmonitordirstate, self).rebuild(*args, **kwargs)
720
720
721 def invalidate(self, *args, **kwargs):
721 def invalidate(self, *args, **kwargs):
722 self._fsmonitorstate.invalidate()
722 self._fsmonitorstate.invalidate()
723 return super(fsmonitordirstate, self).invalidate(*args, **kwargs)
723 return super(fsmonitordirstate, self).invalidate(*args, **kwargs)
724
724
725 dirstate.__class__ = fsmonitordirstate
725 dirstate.__class__ = fsmonitordirstate
726 dirstate._fsmonitorinit(repo)
726 dirstate._fsmonitorinit(repo)
727
727
728
728
729 def wrapdirstate(orig, self):
729 def wrapdirstate(orig, self):
730 ds = orig(self)
730 ds = orig(self)
731 # only override the dirstate when Watchman is available for the repo
731 # only override the dirstate when Watchman is available for the repo
732 if util.safehasattr(self, b'_fsmonitorstate'):
732 if util.safehasattr(self, b'_fsmonitorstate'):
733 makedirstate(self, ds)
733 makedirstate(self, ds)
734 return ds
734 return ds
735
735
736
736
737 def extsetup(ui):
737 def extsetup(ui):
738 extensions.wrapfilecache(
738 extensions.wrapfilecache(
739 localrepo.localrepository, b'dirstate', wrapdirstate
739 localrepo.localrepository, b'dirstate', wrapdirstate
740 )
740 )
741 if pycompat.isdarwin:
741 if pycompat.isdarwin:
742 # An assist for avoiding the dangling-symlink fsevents bug
742 # An assist for avoiding the dangling-symlink fsevents bug
743 extensions.wrapfunction(os, b'symlink', wrapsymlink)
743 extensions.wrapfunction(os, b'symlink', wrapsymlink)
744
744
745 extensions.wrapfunction(merge, b'_update', wrapupdate)
745 extensions.wrapfunction(merge, b'_update', wrapupdate)
746
746
747
747
748 def wrapsymlink(orig, source, link_name):
748 def wrapsymlink(orig, source, link_name):
749 """if we create a dangling symlink, also touch the parent dir
749 """if we create a dangling symlink, also touch the parent dir
750 to encourage fsevents notifications to work more correctly"""
750 to encourage fsevents notifications to work more correctly"""
751 try:
751 try:
752 return orig(source, link_name)
752 return orig(source, link_name)
753 finally:
753 finally:
754 try:
754 try:
755 os.utime(os.path.dirname(link_name), None)
755 os.utime(os.path.dirname(link_name), None)
756 except OSError:
756 except OSError:
757 pass
757 pass
758
758
759
759
760 class state_update(object):
760 class state_update(object):
761 """This context manager is responsible for dispatching the state-enter
761 """This context manager is responsible for dispatching the state-enter
762 and state-leave signals to the watchman service. The enter and leave
762 and state-leave signals to the watchman service. The enter and leave
763 methods can be invoked manually (for scenarios where context manager
763 methods can be invoked manually (for scenarios where context manager
764 semantics are not possible). If parameters oldnode and newnode are None,
764 semantics are not possible). If parameters oldnode and newnode are None,
765 they will be populated based on current working copy in enter and
765 they will be populated based on current working copy in enter and
766 leave, respectively. Similarly, if the distance is none, it will be
766 leave, respectively. Similarly, if the distance is none, it will be
767 calculated based on the oldnode and newnode in the leave method."""
767 calculated based on the oldnode and newnode in the leave method."""
768
768
769 def __init__(
769 def __init__(
770 self,
770 self,
771 repo,
771 repo,
772 name,
772 name,
773 oldnode=None,
773 oldnode=None,
774 newnode=None,
774 newnode=None,
775 distance=None,
775 distance=None,
776 partial=False,
776 partial=False,
777 ):
777 ):
778 self.repo = repo.unfiltered()
778 self.repo = repo.unfiltered()
779 self.name = name
779 self.name = name
780 self.oldnode = oldnode
780 self.oldnode = oldnode
781 self.newnode = newnode
781 self.newnode = newnode
782 self.distance = distance
782 self.distance = distance
783 self.partial = partial
783 self.partial = partial
784 self._lock = None
784 self._lock = None
785 self.need_leave = False
785 self.need_leave = False
786
786
787 def __enter__(self):
787 def __enter__(self):
788 self.enter()
788 self.enter()
789
789
790 def enter(self):
790 def enter(self):
791 # Make sure we have a wlock prior to sending notifications to watchman.
791 # Make sure we have a wlock prior to sending notifications to watchman.
792 # We don't want to race with other actors. In the update case,
792 # We don't want to race with other actors. In the update case,
793 # merge.update is going to take the wlock almost immediately. We are
793 # merge.update is going to take the wlock almost immediately. We are
794 # effectively extending the lock around several short sanity checks.
794 # effectively extending the lock around several short sanity checks.
795 if self.oldnode is None:
795 if self.oldnode is None:
796 self.oldnode = self.repo[b'.'].node()
796 self.oldnode = self.repo[b'.'].node()
797
797
798 if self.repo.currentwlock() is None:
798 if self.repo.currentwlock() is None:
799 if util.safehasattr(self.repo, b'wlocknostateupdate'):
799 if util.safehasattr(self.repo, b'wlocknostateupdate'):
800 self._lock = self.repo.wlocknostateupdate()
800 self._lock = self.repo.wlocknostateupdate()
801 else:
801 else:
802 self._lock = self.repo.wlock()
802 self._lock = self.repo.wlock()
803 self.need_leave = self._state(b'state-enter', hex(self.oldnode))
803 self.need_leave = self._state(b'state-enter', hex(self.oldnode))
804 return self
804 return self
805
805
806 def __exit__(self, type_, value, tb):
806 def __exit__(self, type_, value, tb):
807 abort = True if type_ else False
807 abort = True if type_ else False
808 self.exit(abort=abort)
808 self.exit(abort=abort)
809
809
810 def exit(self, abort=False):
810 def exit(self, abort=False):
811 try:
811 try:
812 if self.need_leave:
812 if self.need_leave:
813 status = b'failed' if abort else b'ok'
813 status = b'failed' if abort else b'ok'
814 if self.newnode is None:
814 if self.newnode is None:
815 self.newnode = self.repo[b'.'].node()
815 self.newnode = self.repo[b'.'].node()
816 if self.distance is None:
816 if self.distance is None:
817 self.distance = calcdistance(
817 self.distance = calcdistance(
818 self.repo, self.oldnode, self.newnode
818 self.repo, self.oldnode, self.newnode
819 )
819 )
820 self._state(b'state-leave', hex(self.newnode), status=status)
820 self._state(b'state-leave', hex(self.newnode), status=status)
821 finally:
821 finally:
822 self.need_leave = False
822 self.need_leave = False
823 if self._lock:
823 if self._lock:
824 self._lock.release()
824 self._lock.release()
825
825
826 def _state(self, cmd, commithash, status=b'ok'):
826 def _state(self, cmd, commithash, status=b'ok'):
827 if not util.safehasattr(self.repo, b'_watchmanclient'):
827 if not util.safehasattr(self.repo, b'_watchmanclient'):
828 return False
828 return False
829 try:
829 try:
830 self.repo._watchmanclient.command(
830 self.repo._watchmanclient.command(
831 cmd,
831 cmd,
832 {
832 {
833 b'name': self.name,
833 b'name': self.name,
834 b'metadata': {
834 b'metadata': {
835 # the target revision
835 # the target revision
836 b'rev': commithash,
836 b'rev': commithash,
837 # approximate number of commits between current and target
837 # approximate number of commits between current and target
838 b'distance': self.distance if self.distance else 0,
838 b'distance': self.distance if self.distance else 0,
839 # success/failure (only really meaningful for state-leave)
839 # success/failure (only really meaningful for state-leave)
840 b'status': status,
840 b'status': status,
841 # whether the working copy parent is changing
841 # whether the working copy parent is changing
842 b'partial': self.partial,
842 b'partial': self.partial,
843 },
843 },
844 },
844 },
845 )
845 )
846 return True
846 return True
847 except Exception as e:
847 except Exception as e:
848 # Swallow any errors; fire and forget
848 # Swallow any errors; fire and forget
849 self.repo.ui.log(
849 self.repo.ui.log(
850 b'watchman', b'Exception %s while running %s\n', e, cmd
850 b'watchman', b'Exception %s while running %s\n', e, cmd
851 )
851 )
852 return False
852 return False
853
853
854
854
855 # Estimate the distance between two nodes
855 # Estimate the distance between two nodes
856 def calcdistance(repo, oldnode, newnode):
856 def calcdistance(repo, oldnode, newnode):
857 anc = repo.changelog.ancestor(oldnode, newnode)
857 anc = repo.changelog.ancestor(oldnode, newnode)
858 ancrev = repo[anc].rev()
858 ancrev = repo[anc].rev()
859 distance = abs(repo[oldnode].rev() - ancrev) + abs(
859 distance = abs(repo[oldnode].rev() - ancrev) + abs(
860 repo[newnode].rev() - ancrev
860 repo[newnode].rev() - ancrev
861 )
861 )
862 return distance
862 return distance
863
863
864
864
865 # Bracket working copy updates with calls to the watchman state-enter
865 # Bracket working copy updates with calls to the watchman state-enter
866 # and state-leave commands. This allows clients to perform more intelligent
866 # and state-leave commands. This allows clients to perform more intelligent
867 # settling during bulk file change scenarios
867 # settling during bulk file change scenarios
868 # https://facebook.github.io/watchman/docs/cmd/subscribe.html#advanced-settling
868 # https://facebook.github.io/watchman/docs/cmd/subscribe.html#advanced-settling
869 def wrapupdate(
869 def wrapupdate(
870 orig,
870 orig,
871 repo,
871 repo,
872 node,
872 node,
873 branchmerge,
873 branchmerge,
874 force,
874 force,
875 ancestor=None,
875 ancestor=None,
876 mergeancestor=False,
876 mergeancestor=False,
877 labels=None,
877 labels=None,
878 matcher=None,
878 matcher=None,
879 **kwargs
879 **kwargs
880 ):
880 ):
881
881
882 distance = 0
882 distance = 0
883 partial = True
883 partial = True
884 oldnode = repo[b'.'].node()
884 oldnode = repo[b'.'].node()
885 newnode = repo[node].node()
885 newnode = repo[node].node()
886 if matcher is None or matcher.always():
886 if matcher is None or matcher.always():
887 partial = False
887 partial = False
888 distance = calcdistance(repo.unfiltered(), oldnode, newnode)
888 distance = calcdistance(repo.unfiltered(), oldnode, newnode)
889
889
890 with state_update(
890 with state_update(
891 repo,
891 repo,
892 name=b"hg.update",
892 name=b"hg.update",
893 oldnode=oldnode,
893 oldnode=oldnode,
894 newnode=newnode,
894 newnode=newnode,
895 distance=distance,
895 distance=distance,
896 partial=partial,
896 partial=partial,
897 ):
897 ):
898 return orig(
898 return orig(
899 repo,
899 repo,
900 node,
900 node,
901 branchmerge,
901 branchmerge,
902 force,
902 force,
903 ancestor,
903 ancestor,
904 mergeancestor,
904 mergeancestor,
905 labels,
905 labels,
906 matcher,
906 matcher,
907 **kwargs
907 **kwargs
908 )
908 )
909
909
910
910
911 def repo_has_depth_one_nested_repo(repo):
911 def repo_has_depth_one_nested_repo(repo):
912 for f in repo.wvfs.listdir():
912 for f in repo.wvfs.listdir():
913 if os.path.isdir(os.path.join(repo.root, f, b'.hg')):
913 if os.path.isdir(os.path.join(repo.root, f, b'.hg')):
914 msg = b'fsmonitor: sub-repository %r detected, fsmonitor disabled\n'
914 msg = b'fsmonitor: sub-repository %r detected, fsmonitor disabled\n'
915 repo.ui.debug(msg % f)
915 repo.ui.debug(msg % f)
916 return True
916 return True
917 return False
917 return False
918
918
919
919
920 def reposetup(ui, repo):
920 def reposetup(ui, repo):
921 # We don't work with largefiles or inotify
921 # We don't work with largefiles or inotify
922 exts = extensions.enabled()
922 exts = extensions.enabled()
923 for ext in _blacklist:
923 for ext in _blacklist:
924 if ext in exts:
924 if ext in exts:
925 ui.warn(
925 ui.warn(
926 _(
926 _(
927 b'The fsmonitor extension is incompatible with the %s '
927 b'The fsmonitor extension is incompatible with the %s '
928 b'extension and has been disabled.\n'
928 b'extension and has been disabled.\n'
929 )
929 )
930 % ext
930 % ext
931 )
931 )
932 return
932 return
933
933
934 if repo.local():
934 if repo.local():
935 # We don't work with subrepos either.
935 # We don't work with subrepos either.
936 #
936 #
937 # if repo[None].substate can cause a dirstate parse, which is too
937 # if repo[None].substate can cause a dirstate parse, which is too
938 # slow. Instead, look for a file called hgsubstate,
938 # slow. Instead, look for a file called hgsubstate,
939 if repo.wvfs.exists(b'.hgsubstate') or repo.wvfs.exists(b'.hgsub'):
939 if repo.wvfs.exists(b'.hgsubstate') or repo.wvfs.exists(b'.hgsub'):
940 return
940 return
941
941
942 if repo_has_depth_one_nested_repo(repo):
942 if repo_has_depth_one_nested_repo(repo):
943 return
943 return
944
944
945 fsmonitorstate = state.state(repo)
945 fsmonitorstate = state.state(repo)
946 if fsmonitorstate.mode == b'off':
946 if fsmonitorstate.mode == b'off':
947 return
947 return
948
948
949 try:
949 try:
950 client = watchmanclient.client(repo.ui, repo.root)
950 client = watchmanclient.client(repo.ui, repo.root)
951 except Exception as ex:
951 except Exception as ex:
952 _handleunavailable(ui, fsmonitorstate, ex)
952 _handleunavailable(ui, fsmonitorstate, ex)
953 return
953 return
954
954
955 repo._fsmonitorstate = fsmonitorstate
955 repo._fsmonitorstate = fsmonitorstate
956 repo._watchmanclient = client
956 repo._watchmanclient = client
957
957
958 dirstate, cached = localrepo.isfilecached(repo, b'dirstate')
958 dirstate, cached = localrepo.isfilecached(repo, b'dirstate')
959 if cached:
959 if cached:
960 # at this point since fsmonitorstate wasn't present,
960 # at this point since fsmonitorstate wasn't present,
961 # repo.dirstate is not a fsmonitordirstate
961 # repo.dirstate is not a fsmonitordirstate
962 makedirstate(repo, dirstate)
962 makedirstate(repo, dirstate)
963
963
964 class fsmonitorrepo(repo.__class__):
964 class fsmonitorrepo(repo.__class__):
965 def status(self, *args, **kwargs):
965 def status(self, *args, **kwargs):
966 orig = super(fsmonitorrepo, self).status
966 orig = super(fsmonitorrepo, self).status
967 return overridestatus(orig, self, *args, **kwargs)
967 return overridestatus(orig, self, *args, **kwargs)
968
968
969 def wlocknostateupdate(self, *args, **kwargs):
969 def wlocknostateupdate(self, *args, **kwargs):
970 return super(fsmonitorrepo, self).wlock(*args, **kwargs)
970 return super(fsmonitorrepo, self).wlock(*args, **kwargs)
971
971
972 def wlock(self, *args, **kwargs):
972 def wlock(self, *args, **kwargs):
973 l = super(fsmonitorrepo, self).wlock(*args, **kwargs)
973 l = super(fsmonitorrepo, self).wlock(*args, **kwargs)
974 if not ui.configbool(
974 if not ui.configbool(
975 b"experimental", b"fsmonitor.transaction_notify"
975 b"experimental", b"fsmonitor.transaction_notify"
976 ):
976 ):
977 return l
977 return l
978 if l.held != 1:
978 if l.held != 1:
979 return l
979 return l
980 origrelease = l.releasefn
980 origrelease = l.releasefn
981
981
982 def staterelease():
982 def staterelease():
983 if origrelease:
983 if origrelease:
984 origrelease()
984 origrelease()
985 if l.stateupdate:
985 if l.stateupdate:
986 l.stateupdate.exit()
986 l.stateupdate.exit()
987 l.stateupdate = None
987 l.stateupdate = None
988
988
989 try:
989 try:
990 l.stateupdate = None
990 l.stateupdate = None
991 l.stateupdate = state_update(self, name=b"hg.transaction")
991 l.stateupdate = state_update(self, name=b"hg.transaction")
992 l.stateupdate.enter()
992 l.stateupdate.enter()
993 l.releasefn = staterelease
993 l.releasefn = staterelease
994 except Exception as e:
994 except Exception as e:
995 # Swallow any errors; fire and forget
995 # Swallow any errors; fire and forget
996 self.ui.log(
996 self.ui.log(
997 b'watchman', b'Exception in state update %s\n', e
997 b'watchman', b'Exception in state update %s\n', e
998 )
998 )
999 return l
999 return l
1000
1000
1001 repo.__class__ = fsmonitorrepo
1001 repo.__class__ = fsmonitorrepo
General Comments 0
You need to be logged in to leave comments. Login now