##// END OF EJS Templates
clone-bundles: add an option to generate bundles in the background...
marmoute -
r51307:3973b1dc default
parent child Browse files
Show More
@@ -1,996 +1,1022 b''
1 # This software may be used and distributed according to the terms of the
1 # This software may be used and distributed according to the terms of the
2 # GNU General Public License version 2 or any later version.
2 # GNU General Public License version 2 or any later version.
3
3
4 """advertise pre-generated bundles to seed clones
4 """advertise pre-generated bundles to seed clones
5
5
6 "clonebundles" is a server-side extension used to advertise the existence
6 "clonebundles" is a server-side extension used to advertise the existence
7 of pre-generated, externally hosted bundle files to clients that are
7 of pre-generated, externally hosted bundle files to clients that are
8 cloning so that cloning can be faster, more reliable, and require less
8 cloning so that cloning can be faster, more reliable, and require less
9 resources on the server. "pullbundles" is a related feature for sending
9 resources on the server. "pullbundles" is a related feature for sending
10 pre-generated bundle files to clients as part of pull operations.
10 pre-generated bundle files to clients as part of pull operations.
11
11
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
12 Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
13 the server, in response to a client's request to clone, dynamically generates
13 the server, in response to a client's request to clone, dynamically generates
14 a bundle containing the entire repository content and sends it to the client.
14 a bundle containing the entire repository content and sends it to the client.
15 There is no caching on the server and the server will have to redundantly
15 There is no caching on the server and the server will have to redundantly
16 generate the same outgoing bundle in response to each clone request. For
16 generate the same outgoing bundle in response to each clone request. For
17 servers with large repositories or with high clone volume, the load from
17 servers with large repositories or with high clone volume, the load from
18 clones can make scaling the server challenging and costly.
18 clones can make scaling the server challenging and costly.
19
19
20 This extension provides server operators the ability to offload
20 This extension provides server operators the ability to offload
21 potentially expensive clone load to an external service. Pre-generated
21 potentially expensive clone load to an external service. Pre-generated
22 bundles also allow using more CPU intensive compression, reducing the
22 bundles also allow using more CPU intensive compression, reducing the
23 effective bandwidth requirements.
23 effective bandwidth requirements.
24
24
25 Here's how clone bundles work:
25 Here's how clone bundles work:
26
26
27 1. A server operator establishes a mechanism for making bundle files available
27 1. A server operator establishes a mechanism for making bundle files available
28 on a hosting service where Mercurial clients can fetch them.
28 on a hosting service where Mercurial clients can fetch them.
29 2. A manifest file listing available bundle URLs and some optional metadata
29 2. A manifest file listing available bundle URLs and some optional metadata
30 is added to the Mercurial repository on the server.
30 is added to the Mercurial repository on the server.
31 3. A client initiates a clone against a clone bundles aware server.
31 3. A client initiates a clone against a clone bundles aware server.
32 4. The client sees the server is advertising clone bundles and fetches the
32 4. The client sees the server is advertising clone bundles and fetches the
33 manifest listing available bundles.
33 manifest listing available bundles.
34 5. The client filters and sorts the available bundles based on what it
34 5. The client filters and sorts the available bundles based on what it
35 supports and prefers.
35 supports and prefers.
36 6. The client downloads and applies an available bundle from the
36 6. The client downloads and applies an available bundle from the
37 server-specified URL.
37 server-specified URL.
38 7. The client reconnects to the original server and performs the equivalent
38 7. The client reconnects to the original server and performs the equivalent
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The
39 of :hg:`pull` to retrieve all repository data not in the bundle. (The
40 repository could have been updated between when the bundle was created
40 repository could have been updated between when the bundle was created
41 and when the client started the clone.) This may use "pullbundles".
41 and when the client started the clone.) This may use "pullbundles".
42
42
43 Instead of the server generating full repository bundles for every clone
43 Instead of the server generating full repository bundles for every clone
44 request, it generates full bundles once and they are subsequently reused to
44 request, it generates full bundles once and they are subsequently reused to
45 bootstrap new clones. The server may still transfer data at clone time.
45 bootstrap new clones. The server may still transfer data at clone time.
46 However, this is only data that has been added/changed since the bundle was
46 However, this is only data that has been added/changed since the bundle was
47 created. For large, established repositories, this can reduce server load for
47 created. For large, established repositories, this can reduce server load for
48 clones to less than 1% of original.
48 clones to less than 1% of original.
49
49
50 Here's how pullbundles work:
50 Here's how pullbundles work:
51
51
52 1. A manifest file listing available bundles and describing the revisions
52 1. A manifest file listing available bundles and describing the revisions
53 is added to the Mercurial repository on the server.
53 is added to the Mercurial repository on the server.
54 2. A new-enough client informs the server that it supports partial pulls
54 2. A new-enough client informs the server that it supports partial pulls
55 and initiates a pull.
55 and initiates a pull.
56 3. If the server has pull bundles enabled and sees the client advertising
56 3. If the server has pull bundles enabled and sees the client advertising
57 partial pulls, it checks for a matching pull bundle in the manifest.
57 partial pulls, it checks for a matching pull bundle in the manifest.
58 A bundle matches if the format is supported by the client, the client
58 A bundle matches if the format is supported by the client, the client
59 has the required revisions already and needs something from the bundle.
59 has the required revisions already and needs something from the bundle.
60 4. If there is at least one matching bundle, the server sends it to the client.
60 4. If there is at least one matching bundle, the server sends it to the client.
61 5. The client applies the bundle and notices that the server reply was
61 5. The client applies the bundle and notices that the server reply was
62 incomplete. It initiates another pull.
62 incomplete. It initiates another pull.
63
63
64 To work, this extension requires the following of server operators:
64 To work, this extension requires the following of server operators:
65
65
66 * Generating bundle files of repository content (typically periodically,
66 * Generating bundle files of repository content (typically periodically,
67 such as once per day).
67 such as once per day).
68 * Clone bundles: A file server that clients have network access to and that
68 * Clone bundles: A file server that clients have network access to and that
69 Python knows how to talk to through its normal URL handling facility
69 Python knows how to talk to through its normal URL handling facility
70 (typically an HTTP/HTTPS server).
70 (typically an HTTP/HTTPS server).
71 * A process for keeping the bundles manifest in sync with available bundle
71 * A process for keeping the bundles manifest in sync with available bundle
72 files.
72 files.
73
73
74 Strictly speaking, using a static file hosting server isn't required: a server
74 Strictly speaking, using a static file hosting server isn't required: a server
75 operator could use a dynamic service for retrieving bundle data. However,
75 operator could use a dynamic service for retrieving bundle data. However,
76 static file hosting services are simple and scalable and should be sufficient
76 static file hosting services are simple and scalable and should be sufficient
77 for most needs.
77 for most needs.
78
78
79 Bundle files can be generated with the :hg:`bundle` command. Typically
79 Bundle files can be generated with the :hg:`bundle` command. Typically
80 :hg:`bundle --all` is used to produce a bundle of the entire repository.
80 :hg:`bundle --all` is used to produce a bundle of the entire repository.
81
81
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special
82 :hg:`debugcreatestreamclonebundle` can be used to produce a special
83 *streaming clonebundle*. These are bundle files that are extremely efficient
83 *streaming clonebundle*. These are bundle files that are extremely efficient
84 to produce and consume (read: fast). However, they are larger than
84 to produce and consume (read: fast). However, they are larger than
85 traditional bundle formats and require that clients support the exact set
85 traditional bundle formats and require that clients support the exact set
86 of repository data store formats in use by the repository that created them.
86 of repository data store formats in use by the repository that created them.
87 Typically, a newer server can serve data that is compatible with older clients.
87 Typically, a newer server can serve data that is compatible with older clients.
88 However, *streaming clone bundles* don't have this guarantee. **Server
88 However, *streaming clone bundles* don't have this guarantee. **Server
89 operators need to be aware that newer versions of Mercurial may produce
89 operators need to be aware that newer versions of Mercurial may produce
90 streaming clone bundles incompatible with older Mercurial versions.**
90 streaming clone bundles incompatible with older Mercurial versions.**
91
91
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
92 A server operator is responsible for creating a ``.hg/clonebundles.manifest``
93 file containing the list of available bundle files suitable for seeding
93 file containing the list of available bundle files suitable for seeding
94 clones. If this file does not exist, the repository will not advertise the
94 clones. If this file does not exist, the repository will not advertise the
95 existence of clone bundles when clients connect. For pull bundles,
95 existence of clone bundles when clients connect. For pull bundles,
96 ``.hg/pullbundles.manifest`` is used.
96 ``.hg/pullbundles.manifest`` is used.
97
97
98 The manifest file contains a newline (\\n) delimited list of entries.
98 The manifest file contains a newline (\\n) delimited list of entries.
99
99
100 Each line in this file defines an available bundle. Lines have the format:
100 Each line in this file defines an available bundle. Lines have the format:
101
101
102 <URL> [<key>=<value>[ <key>=<value>]]
102 <URL> [<key>=<value>[ <key>=<value>]]
103
103
104 That is, a URL followed by an optional, space-delimited list of key=value
104 That is, a URL followed by an optional, space-delimited list of key=value
105 pairs describing additional properties of this bundle. Both keys and values
105 pairs describing additional properties of this bundle. Both keys and values
106 are URI encoded.
106 are URI encoded.
107
107
108 For pull bundles, the URL is a path under the ``.hg`` directory of the
108 For pull bundles, the URL is a path under the ``.hg`` directory of the
109 repository.
109 repository.
110
110
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
111 Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
112 All non-uppercase keys can be used by site installations. An example use
112 All non-uppercase keys can be used by site installations. An example use
113 for custom properties is to use the *datacenter* attribute to define which
113 for custom properties is to use the *datacenter* attribute to define which
114 data center a file is hosted in. Clients could then prefer a server in the
114 data center a file is hosted in. Clients could then prefer a server in the
115 data center closest to them.
115 data center closest to them.
116
116
117 The following reserved keys are currently defined:
117 The following reserved keys are currently defined:
118
118
119 BUNDLESPEC
119 BUNDLESPEC
120 A "bundle specification" string that describes the type of the bundle.
120 A "bundle specification" string that describes the type of the bundle.
121
121
122 These are string values that are accepted by the "--type" argument of
122 These are string values that are accepted by the "--type" argument of
123 :hg:`bundle`.
123 :hg:`bundle`.
124
124
125 The values are parsed in strict mode, which means they must be of the
125 The values are parsed in strict mode, which means they must be of the
126 "<compression>-<type>" form. See
126 "<compression>-<type>" form. See
127 mercurial.exchange.parsebundlespec() for more details.
127 mercurial.exchange.parsebundlespec() for more details.
128
128
129 :hg:`debugbundle --spec` can be used to print the bundle specification
129 :hg:`debugbundle --spec` can be used to print the bundle specification
130 string for a bundle file. The output of this command can be used verbatim
130 string for a bundle file. The output of this command can be used verbatim
131 for the value of ``BUNDLESPEC`` (it is already escaped).
131 for the value of ``BUNDLESPEC`` (it is already escaped).
132
132
133 Clients will automatically filter out specifications that are unknown or
133 Clients will automatically filter out specifications that are unknown or
134 unsupported so they won't attempt to download something that likely won't
134 unsupported so they won't attempt to download something that likely won't
135 apply.
135 apply.
136
136
137 The actual value doesn't impact client behavior beyond filtering:
137 The actual value doesn't impact client behavior beyond filtering:
138 clients will still sniff the bundle type from the header of downloaded
138 clients will still sniff the bundle type from the header of downloaded
139 files.
139 files.
140
140
141 **Use of this key is highly recommended**, as it allows clients to
141 **Use of this key is highly recommended**, as it allows clients to
142 easily skip unsupported bundles. If this key is not defined, an old
142 easily skip unsupported bundles. If this key is not defined, an old
143 client may attempt to apply a bundle that it is incapable of reading.
143 client may attempt to apply a bundle that it is incapable of reading.
144
144
145 REQUIRESNI
145 REQUIRESNI
146 Whether Server Name Indication (SNI) is required to connect to the URL.
146 Whether Server Name Indication (SNI) is required to connect to the URL.
147 SNI allows servers to use multiple certificates on the same IP. It is
147 SNI allows servers to use multiple certificates on the same IP. It is
148 somewhat common in CDNs and other hosting providers. Older Python
148 somewhat common in CDNs and other hosting providers. Older Python
149 versions do not support SNI. Defining this attribute enables clients
149 versions do not support SNI. Defining this attribute enables clients
150 with older Python versions to filter this entry without experiencing
150 with older Python versions to filter this entry without experiencing
151 an opaque SSL failure at connection time.
151 an opaque SSL failure at connection time.
152
152
153 If this is defined, it is important to advertise a non-SNI fallback
153 If this is defined, it is important to advertise a non-SNI fallback
154 URL or clients running old Python releases may not be able to clone
154 URL or clients running old Python releases may not be able to clone
155 with the clonebundles facility.
155 with the clonebundles facility.
156
156
157 Value should be "true".
157 Value should be "true".
158
158
159 REQUIREDRAM
159 REQUIREDRAM
160 Value specifies expected memory requirements to decode the payload.
160 Value specifies expected memory requirements to decode the payload.
161 Values can have suffixes for common bytes sizes. e.g. "64MB".
161 Values can have suffixes for common bytes sizes. e.g. "64MB".
162
162
163 This key is often used with zstd-compressed bundles using a high
163 This key is often used with zstd-compressed bundles using a high
164 compression level / window size, which can require 100+ MB of memory
164 compression level / window size, which can require 100+ MB of memory
165 to decode.
165 to decode.
166
166
167 heads
167 heads
168 Used for pull bundles. This contains the ``;`` separated changeset
168 Used for pull bundles. This contains the ``;`` separated changeset
169 hashes of the heads of the bundle content.
169 hashes of the heads of the bundle content.
170
170
171 bases
171 bases
172 Used for pull bundles. This contains the ``;`` separated changeset
172 Used for pull bundles. This contains the ``;`` separated changeset
173 hashes of the roots of the bundle content. This can be skipped if
173 hashes of the roots of the bundle content. This can be skipped if
174 the bundle was created without ``--base``.
174 the bundle was created without ``--base``.
175
175
176 Manifests can contain multiple entries. Assuming metadata is defined, clients
176 Manifests can contain multiple entries. Assuming metadata is defined, clients
177 will filter entries from the manifest that they don't support. The remaining
177 will filter entries from the manifest that they don't support. The remaining
178 entries are optionally sorted by client preferences
178 entries are optionally sorted by client preferences
179 (``ui.clonebundleprefers`` config option). The client then attempts
179 (``ui.clonebundleprefers`` config option). The client then attempts
180 to fetch the bundle at the first URL in the remaining list.
180 to fetch the bundle at the first URL in the remaining list.
181
181
182 **Errors when downloading a bundle will fail the entire clone operation:
182 **Errors when downloading a bundle will fail the entire clone operation:
183 clients do not automatically fall back to a traditional clone.** The reason
183 clients do not automatically fall back to a traditional clone.** The reason
184 for this is that if a server is using clone bundles, it is probably doing so
184 for this is that if a server is using clone bundles, it is probably doing so
185 because the feature is necessary to help it scale. In other words, there
185 because the feature is necessary to help it scale. In other words, there
186 is an assumption that clone load will be offloaded to another service and
186 is an assumption that clone load will be offloaded to another service and
187 that the Mercurial server isn't responsible for serving this clone load.
187 that the Mercurial server isn't responsible for serving this clone load.
188 If that other service experiences issues and clients start mass falling back to
188 If that other service experiences issues and clients start mass falling back to
189 the original Mercurial server, the added clone load could overwhelm the server
189 the original Mercurial server, the added clone load could overwhelm the server
190 due to unexpected load and effectively take it offline. Not having clients
190 due to unexpected load and effectively take it offline. Not having clients
191 automatically fall back to cloning from the original server mitigates this
191 automatically fall back to cloning from the original server mitigates this
192 scenario.
192 scenario.
193
193
194 Because there is no automatic Mercurial server fallback on failure of the
194 Because there is no automatic Mercurial server fallback on failure of the
195 bundle hosting service, it is important for server operators to view the bundle
195 bundle hosting service, it is important for server operators to view the bundle
196 hosting service as an extension of the Mercurial server in terms of
196 hosting service as an extension of the Mercurial server in terms of
197 availability and service level agreements: if the bundle hosting service goes
197 availability and service level agreements: if the bundle hosting service goes
198 down, so does the ability for clients to clone. Note: clients will see a
198 down, so does the ability for clients to clone. Note: clients will see a
199 message informing them how to bypass the clone bundles facility when a failure
199 message informing them how to bypass the clone bundles facility when a failure
200 occurs. So server operators should prepare for some people to follow these
200 occurs. So server operators should prepare for some people to follow these
201 instructions when a failure occurs, thus driving more load to the original
201 instructions when a failure occurs, thus driving more load to the original
202 Mercurial server when the bundle hosting service fails.
202 Mercurial server when the bundle hosting service fails.
203
203
204
204
205 auto-generation of clone bundles
205 auto-generation of clone bundles
206 --------------------------------
206 --------------------------------
207
207
208 It is possible to set Mercurial to automatically re-generate clone bundles when
208 It is possible to set Mercurial to automatically re-generate clone bundles when
209 enough new content is available.
209 enough new content is available.
210
210
211 Mercurial will take care of the process asynchronously. The defined list of
211 Mercurial will take care of the process asynchronously. The defined list of
212 bundle-type will be generated, uploaded, and advertised. Older bundles will get
212 bundle-type will be generated, uploaded, and advertised. Older bundles will get
213 decommissioned as newer ones replace them.
213 decommissioned as newer ones replace them.
214
214
215 Bundles Generation:
215 Bundles Generation:
216 ...................
216 ...................
217
217
218 The extension can generate multiple variants of the clone bundle. Each
218 The extension can generate multiple variants of the clone bundle. Each
219 different variant will be defined by the "bundle-spec" they use::
219 different variant will be defined by the "bundle-spec" they use::
220
220
221 [clone-bundles]
221 [clone-bundles]
222 auto-generate.formats= zstd-v2, gzip-v2
222 auto-generate.formats= zstd-v2, gzip-v2
223
223
224 See `hg help bundlespec` for details about available options.
224 See `hg help bundlespec` for details about available options.
225
225
226 By default, new bundles are generated when 5% of the repository contents or at
226 By default, new bundles are generated when 5% of the repository contents or at
227 least 1000 revisions are not contained in the cached bundles. This option can
227 least 1000 revisions are not contained in the cached bundles. This option can
228 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
228 be controlled by the `clone-bundles.trigger.below-bundled-ratio` option
229 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
229 (default 0.95) and the `clone-bundles.trigger.revs` option (default 1000)::
230
230
231 [clone-bundles]
231 [clone-bundles]
232 trigger.below-bundled-ratio=0.95
232 trigger.below-bundled-ratio=0.95
233 trigger.revs=1000
233 trigger.revs=1000
234
234
235 This logic can be manually triggered using the `admin::clone-bundles-refresh`
235 This logic can be manually triggered using the `admin::clone-bundles-refresh`
236 command, or automatically on each repository change if
236 command, or automatically on each repository change if
237 `clone-bundles.auto-generate.on-change` is set to `yes`.
237 `clone-bundles.auto-generate.on-change` is set to `yes`.
238
238
239 [clone-bundles]
239 [clone-bundles]
240 auto-generate.on-change=yes
240 auto-generate.on-change=yes
241 auto-generate.formats= zstd-v2, gzip-v2
241 auto-generate.formats= zstd-v2, gzip-v2
242
242
243 Bundles Upload and Serving:
243 Bundles Upload and Serving:
244 ...........................
244 ...........................
245
245
246 The generated bundles need to be made available to users through a "public" URL.
246 The generated bundles need to be made available to users through a "public" URL.
247 This should be donne through `clone-bundles.upload-command` configuration. The
247 This should be donne through `clone-bundles.upload-command` configuration. The
248 value of this command should be a shell command. It will have access to the
248 value of this command should be a shell command. It will have access to the
249 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
249 bundle file path through the `$HGCB_BUNDLE_PATH` variable. And the expected
250 basename in the "public" URL is accessible at::
250 basename in the "public" URL is accessible at::
251
251
252 [clone-bundles]
252 [clone-bundles]
253 upload-command=sftp put $HGCB_BUNDLE_PATH \
253 upload-command=sftp put $HGCB_BUNDLE_PATH \
254 sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
254 sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
255
255
256 If the file was already uploaded, the command must still succeed.
256 If the file was already uploaded, the command must still succeed.
257
257
258 After upload, the file should be available at an url defined by
258 After upload, the file should be available at an url defined by
259 `clone-bundles.url-template`.
259 `clone-bundles.url-template`.
260
260
261 [clone-bundles]
261 [clone-bundles]
262 url-template=https://bundles.host/cache/clone-bundles/{basename}
262 url-template=https://bundles.host/cache/clone-bundles/{basename}
263
263
264 Old bundles cleanup:
264 Old bundles cleanup:
265 ....................
265 ....................
266
266
267 When new bundles are generated, the older ones are no longer necessary and can
267 When new bundles are generated, the older ones are no longer necessary and can
268 be removed from storage. This is done through the `clone-bundles.delete-command`
268 be removed from storage. This is done through the `clone-bundles.delete-command`
269 configuration. The command is given the url of the artifact to delete through
269 configuration. The command is given the url of the artifact to delete through
270 the `$HGCB_BUNDLE_URL` environment variable.
270 the `$HGCB_BUNDLE_URL` environment variable.
271
271
272 [clone-bundles]
272 [clone-bundles]
273 delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
273 delete-command=sftp rm sftp://bundles.host/clone-bundles/$HGCB_BUNDLE_BASENAME
274
274
275 If the file was already deleted, the command must still succeed.
275 If the file was already deleted, the command must still succeed.
276 """
276 """
277
277
278
278
279 import os
279 import os
280 import weakref
280 import weakref
281
281
282 from mercurial.i18n import _
282 from mercurial.i18n import _
283
283
284 from mercurial import (
284 from mercurial import (
285 bundlecaches,
285 bundlecaches,
286 commands,
286 commands,
287 error,
287 error,
288 extensions,
288 extensions,
289 localrepo,
289 localrepo,
290 lock,
290 lock,
291 node,
291 node,
292 registrar,
292 registrar,
293 util,
293 util,
294 wireprotov1server,
294 wireprotov1server,
295 )
295 )
296
296
297
297
298 from mercurial.utils import (
298 from mercurial.utils import (
299 procutil,
299 procutil,
300 )
300 )
301
301
302 testedwith = b'ships-with-hg-core'
302 testedwith = b'ships-with-hg-core'
303
303
304
304
305 def capabilities(orig, repo, proto):
305 def capabilities(orig, repo, proto):
306 caps = orig(repo, proto)
306 caps = orig(repo, proto)
307
307
308 # Only advertise if a manifest exists. This does add some I/O to requests.
308 # Only advertise if a manifest exists. This does add some I/O to requests.
309 # But this should be cheaper than a wasted network round trip due to
309 # But this should be cheaper than a wasted network round trip due to
310 # missing file.
310 # missing file.
311 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
311 if repo.vfs.exists(bundlecaches.CB_MANIFEST_FILE):
312 caps.append(b'clonebundles')
312 caps.append(b'clonebundles')
313
313
314 return caps
314 return caps
315
315
316
316
317 def extsetup(ui):
317 def extsetup(ui):
318 extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities)
318 extensions.wrapfunction(wireprotov1server, b'_capabilities', capabilities)
319
319
320
320
321 # logic for bundle auto-generation
321 # logic for bundle auto-generation
322
322
323
323
324 configtable = {}
324 configtable = {}
325 configitem = registrar.configitem(configtable)
325 configitem = registrar.configitem(configtable)
326
326
327 cmdtable = {}
327 cmdtable = {}
328 command = registrar.command(cmdtable)
328 command = registrar.command(cmdtable)
329
329
330 configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
330 configitem(b'clone-bundles', b'auto-generate.on-change', default=False)
331 configitem(b'clone-bundles', b'auto-generate.formats', default=list)
331 configitem(b'clone-bundles', b'auto-generate.formats', default=list)
332 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
332 configitem(b'clone-bundles', b'trigger.below-bundled-ratio', default=0.95)
333 configitem(b'clone-bundles', b'trigger.revs', default=1000)
333 configitem(b'clone-bundles', b'trigger.revs', default=1000)
334
334
335 configitem(b'clone-bundles', b'upload-command', default=None)
335 configitem(b'clone-bundles', b'upload-command', default=None)
336
336
337 configitem(b'clone-bundles', b'delete-command', default=None)
337 configitem(b'clone-bundles', b'delete-command', default=None)
338
338
339 configitem(b'clone-bundles', b'url-template', default=None)
339 configitem(b'clone-bundles', b'url-template', default=None)
340
340
341 configitem(b'devel', b'debug.clonebundles', default=False)
341 configitem(b'devel', b'debug.clonebundles', default=False)
342
342
343
343
344 # category for the post-close transaction hooks
344 # category for the post-close transaction hooks
345 CAT_POSTCLOSE = b"clonebundles-autobundles"
345 CAT_POSTCLOSE = b"clonebundles-autobundles"
346
346
347 # template for bundle file names
347 # template for bundle file names
348 BUNDLE_MASK = (
348 BUNDLE_MASK = (
349 b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
349 b"full-%(bundle_type)s-%(revs)d_revs-%(tip_short)s_tip-%(op_id)s.hg"
350 )
350 )
351
351
352
352
353 # file in .hg/ use to track clonebundles being auto-generated
353 # file in .hg/ use to track clonebundles being auto-generated
354 AUTO_GEN_FILE = b'clonebundles.auto-gen'
354 AUTO_GEN_FILE = b'clonebundles.auto-gen'
355
355
356
356
357 class BundleBase(object):
357 class BundleBase(object):
358 """represents the core of properties that matters for us in a bundle
358 """represents the core of properties that matters for us in a bundle
359
359
360 :bundle_type: the bundlespec (see hg help bundlespec)
360 :bundle_type: the bundlespec (see hg help bundlespec)
361 :revs: the number of revisions in the repo at bundle creation time
361 :revs: the number of revisions in the repo at bundle creation time
362 :tip_rev: the rev-num of the tip revision
362 :tip_rev: the rev-num of the tip revision
363 :tip_node: the node id of the tip-most revision in the bundle
363 :tip_node: the node id of the tip-most revision in the bundle
364
364
365 :ready: True if the bundle is ready to be served
365 :ready: True if the bundle is ready to be served
366 """
366 """
367
367
368 ready = False
368 ready = False
369
369
370 def __init__(self, bundle_type, revs, tip_rev, tip_node):
370 def __init__(self, bundle_type, revs, tip_rev, tip_node):
371 self.bundle_type = bundle_type
371 self.bundle_type = bundle_type
372 self.revs = revs
372 self.revs = revs
373 self.tip_rev = tip_rev
373 self.tip_rev = tip_rev
374 self.tip_node = tip_node
374 self.tip_node = tip_node
375
375
376 def valid_for(self, repo):
376 def valid_for(self, repo):
377 """is this bundle applicable to the current repository
377 """is this bundle applicable to the current repository
378
378
379 This is useful for detecting bundles made irrelevant by stripping.
379 This is useful for detecting bundles made irrelevant by stripping.
380 """
380 """
381 tip_node = node.bin(self.tip_node)
381 tip_node = node.bin(self.tip_node)
382 return repo.changelog.index.get_rev(tip_node) == self.tip_rev
382 return repo.changelog.index.get_rev(tip_node) == self.tip_rev
383
383
384 def __eq__(self, other):
384 def __eq__(self, other):
385 left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
385 left = (self.ready, self.bundle_type, self.tip_rev, self.tip_node)
386 right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
386 right = (other.ready, other.bundle_type, other.tip_rev, other.tip_node)
387 return left == right
387 return left == right
388
388
389 def __neq__(self, other):
389 def __neq__(self, other):
390 return not self == other
390 return not self == other
391
391
392 def __cmp__(self, other):
392 def __cmp__(self, other):
393 if self == other:
393 if self == other:
394 return 0
394 return 0
395 return -1
395 return -1
396
396
397
397
398 class RequestedBundle(BundleBase):
398 class RequestedBundle(BundleBase):
399 """A bundle that should be generated.
399 """A bundle that should be generated.
400
400
401 Additional attributes compared to BundleBase
401 Additional attributes compared to BundleBase
402 :heads: list of head revisions (as rev-num)
402 :heads: list of head revisions (as rev-num)
403 :op_id: a "unique" identifier for the operation triggering the change
403 :op_id: a "unique" identifier for the operation triggering the change
404 """
404 """
405
405
406 def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
406 def __init__(self, bundle_type, revs, tip_rev, tip_node, head_revs, op_id):
407 self.head_revs = head_revs
407 self.head_revs = head_revs
408 self.op_id = op_id
408 self.op_id = op_id
409 super(RequestedBundle, self).__init__(
409 super(RequestedBundle, self).__init__(
410 bundle_type,
410 bundle_type,
411 revs,
411 revs,
412 tip_rev,
412 tip_rev,
413 tip_node,
413 tip_node,
414 )
414 )
415
415
416 @property
416 @property
417 def suggested_filename(self):
417 def suggested_filename(self):
418 """A filename that can be used for the generated bundle"""
418 """A filename that can be used for the generated bundle"""
419 data = {
419 data = {
420 b'bundle_type': self.bundle_type,
420 b'bundle_type': self.bundle_type,
421 b'revs': self.revs,
421 b'revs': self.revs,
422 b'heads': self.head_revs,
422 b'heads': self.head_revs,
423 b'tip_rev': self.tip_rev,
423 b'tip_rev': self.tip_rev,
424 b'tip_node': self.tip_node,
424 b'tip_node': self.tip_node,
425 b'tip_short': self.tip_node[:12],
425 b'tip_short': self.tip_node[:12],
426 b'op_id': self.op_id,
426 b'op_id': self.op_id,
427 }
427 }
428 return BUNDLE_MASK % data
428 return BUNDLE_MASK % data
429
429
430 def generate_bundle(self, repo, file_path):
430 def generate_bundle(self, repo, file_path):
431 """generate the bundle at `filepath`"""
431 """generate the bundle at `filepath`"""
432 commands.bundle(
432 commands.bundle(
433 repo.ui,
433 repo.ui,
434 repo,
434 repo,
435 file_path,
435 file_path,
436 base=[b"null"],
436 base=[b"null"],
437 rev=self.head_revs,
437 rev=self.head_revs,
438 type=self.bundle_type,
438 type=self.bundle_type,
439 quiet=True,
439 quiet=True,
440 )
440 )
441
441
442 def generating(self, file_path, hostname=None, pid=None):
442 def generating(self, file_path, hostname=None, pid=None):
443 """return a GeneratingBundle object from this object"""
443 """return a GeneratingBundle object from this object"""
444 if pid is None:
444 if pid is None:
445 pid = os.getpid()
445 pid = os.getpid()
446 if hostname is None:
446 if hostname is None:
447 hostname = lock._getlockprefix()
447 hostname = lock._getlockprefix()
448 return GeneratingBundle(
448 return GeneratingBundle(
449 self.bundle_type,
449 self.bundle_type,
450 self.revs,
450 self.revs,
451 self.tip_rev,
451 self.tip_rev,
452 self.tip_node,
452 self.tip_node,
453 hostname,
453 hostname,
454 pid,
454 pid,
455 file_path,
455 file_path,
456 )
456 )
457
457
458
458
459 class GeneratingBundle(BundleBase):
459 class GeneratingBundle(BundleBase):
460 """A bundle being generated
460 """A bundle being generated
461
461
462 extra attributes compared to BundleBase:
462 extra attributes compared to BundleBase:
463
463
464 :hostname: the hostname of the machine generating the bundle
464 :hostname: the hostname of the machine generating the bundle
465 :pid: the pid of the process generating the bundle
465 :pid: the pid of the process generating the bundle
466 :filepath: the target filename of the bundle
466 :filepath: the target filename of the bundle
467
467
468 These attributes exist to help detect stalled generation processes.
468 These attributes exist to help detect stalled generation processes.
469 """
469 """
470
470
471 ready = False
471 ready = False
472
472
473 def __init__(
473 def __init__(
474 self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
474 self, bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
475 ):
475 ):
476 self.hostname = hostname
476 self.hostname = hostname
477 self.pid = pid
477 self.pid = pid
478 self.filepath = filepath
478 self.filepath = filepath
479 super(GeneratingBundle, self).__init__(
479 super(GeneratingBundle, self).__init__(
480 bundle_type, revs, tip_rev, tip_node
480 bundle_type, revs, tip_rev, tip_node
481 )
481 )
482
482
483 @classmethod
483 @classmethod
484 def from_line(cls, line):
484 def from_line(cls, line):
485 """create an object by deserializing a line from AUTO_GEN_FILE"""
485 """create an object by deserializing a line from AUTO_GEN_FILE"""
486 assert line.startswith(b'PENDING-v1 ')
486 assert line.startswith(b'PENDING-v1 ')
487 (
487 (
488 __,
488 __,
489 bundle_type,
489 bundle_type,
490 revs,
490 revs,
491 tip_rev,
491 tip_rev,
492 tip_node,
492 tip_node,
493 hostname,
493 hostname,
494 pid,
494 pid,
495 filepath,
495 filepath,
496 ) = line.split()
496 ) = line.split()
497 hostname = util.urlreq.unquote(hostname)
497 hostname = util.urlreq.unquote(hostname)
498 filepath = util.urlreq.unquote(filepath)
498 filepath = util.urlreq.unquote(filepath)
499 revs = int(revs)
499 revs = int(revs)
500 tip_rev = int(tip_rev)
500 tip_rev = int(tip_rev)
501 pid = int(pid)
501 pid = int(pid)
502 return cls(
502 return cls(
503 bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
503 bundle_type, revs, tip_rev, tip_node, hostname, pid, filepath
504 )
504 )
505
505
506 def to_line(self):
506 def to_line(self):
507 """serialize the object to include as a line in AUTO_GEN_FILE"""
507 """serialize the object to include as a line in AUTO_GEN_FILE"""
508 templ = b"PENDING-v1 %s %d %d %s %s %d %s"
508 templ = b"PENDING-v1 %s %d %d %s %s %d %s"
509 data = (
509 data = (
510 self.bundle_type,
510 self.bundle_type,
511 self.revs,
511 self.revs,
512 self.tip_rev,
512 self.tip_rev,
513 self.tip_node,
513 self.tip_node,
514 util.urlreq.quote(self.hostname),
514 util.urlreq.quote(self.hostname),
515 self.pid,
515 self.pid,
516 util.urlreq.quote(self.filepath),
516 util.urlreq.quote(self.filepath),
517 )
517 )
518 return templ % data
518 return templ % data
519
519
520 def __eq__(self, other):
520 def __eq__(self, other):
521 if not super(GeneratingBundle, self).__eq__(other):
521 if not super(GeneratingBundle, self).__eq__(other):
522 return False
522 return False
523 left = (self.hostname, self.pid, self.filepath)
523 left = (self.hostname, self.pid, self.filepath)
524 right = (other.hostname, other.pid, other.filepath)
524 right = (other.hostname, other.pid, other.filepath)
525 return left == right
525 return left == right
526
526
527 def uploaded(self, url, basename):
527 def uploaded(self, url, basename):
528 """return a GeneratedBundle from this object"""
528 """return a GeneratedBundle from this object"""
529 return GeneratedBundle(
529 return GeneratedBundle(
530 self.bundle_type,
530 self.bundle_type,
531 self.revs,
531 self.revs,
532 self.tip_rev,
532 self.tip_rev,
533 self.tip_node,
533 self.tip_node,
534 url,
534 url,
535 basename,
535 basename,
536 )
536 )
537
537
538
538
539 class GeneratedBundle(BundleBase):
539 class GeneratedBundle(BundleBase):
540 """A bundle that is done being generated and can be served
540 """A bundle that is done being generated and can be served
541
541
542 extra attributes compared to BundleBase:
542 extra attributes compared to BundleBase:
543
543
544 :file_url: the url where the bundle is available.
544 :file_url: the url where the bundle is available.
545 :basename: the "basename" used to upload (useful for deletion)
545 :basename: the "basename" used to upload (useful for deletion)
546
546
547 These attributes exist to generate a bundle manifest
547 These attributes exist to generate a bundle manifest
548 (.hg/pullbundles.manifest)
548 (.hg/pullbundles.manifest)
549 """
549 """
550
550
551 ready = True
551 ready = True
552
552
553 def __init__(
553 def __init__(
554 self, bundle_type, revs, tip_rev, tip_node, file_url, basename
554 self, bundle_type, revs, tip_rev, tip_node, file_url, basename
555 ):
555 ):
556 self.file_url = file_url
556 self.file_url = file_url
557 self.basename = basename
557 self.basename = basename
558 super(GeneratedBundle, self).__init__(
558 super(GeneratedBundle, self).__init__(
559 bundle_type, revs, tip_rev, tip_node
559 bundle_type, revs, tip_rev, tip_node
560 )
560 )
561
561
562 @classmethod
562 @classmethod
563 def from_line(cls, line):
563 def from_line(cls, line):
564 """create an object by deserializing a line from AUTO_GEN_FILE"""
564 """create an object by deserializing a line from AUTO_GEN_FILE"""
565 assert line.startswith(b'DONE-v1 ')
565 assert line.startswith(b'DONE-v1 ')
566 (
566 (
567 __,
567 __,
568 bundle_type,
568 bundle_type,
569 revs,
569 revs,
570 tip_rev,
570 tip_rev,
571 tip_node,
571 tip_node,
572 file_url,
572 file_url,
573 basename,
573 basename,
574 ) = line.split()
574 ) = line.split()
575 revs = int(revs)
575 revs = int(revs)
576 tip_rev = int(tip_rev)
576 tip_rev = int(tip_rev)
577 file_url = util.urlreq.unquote(file_url)
577 file_url = util.urlreq.unquote(file_url)
578 return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
578 return cls(bundle_type, revs, tip_rev, tip_node, file_url, basename)
579
579
580 def to_line(self):
580 def to_line(self):
581 """serialize the object to include as a line in AUTO_GEN_FILE"""
581 """serialize the object to include as a line in AUTO_GEN_FILE"""
582 templ = b"DONE-v1 %s %d %d %s %s %s"
582 templ = b"DONE-v1 %s %d %d %s %s %s"
583 data = (
583 data = (
584 self.bundle_type,
584 self.bundle_type,
585 self.revs,
585 self.revs,
586 self.tip_rev,
586 self.tip_rev,
587 self.tip_node,
587 self.tip_node,
588 util.urlreq.quote(self.file_url),
588 util.urlreq.quote(self.file_url),
589 self.basename,
589 self.basename,
590 )
590 )
591 return templ % data
591 return templ % data
592
592
593 def manifest_line(self):
593 def manifest_line(self):
594 """serialize the object to include as a line in pullbundles.manifest"""
594 """serialize the object to include as a line in pullbundles.manifest"""
595 templ = b"%s BUNDLESPEC=%s REQUIRESNI=true"
595 templ = b"%s BUNDLESPEC=%s REQUIRESNI=true"
596 return templ % (self.file_url, self.bundle_type)
596 return templ % (self.file_url, self.bundle_type)
597
597
598 def __eq__(self, other):
598 def __eq__(self, other):
599 if not super(GeneratedBundle, self).__eq__(other):
599 if not super(GeneratedBundle, self).__eq__(other):
600 return False
600 return False
601 return self.file_url == other.file_url
601 return self.file_url == other.file_url
602
602
603
603
604 def parse_auto_gen(content):
604 def parse_auto_gen(content):
605 """parse the AUTO_GEN_FILE to return a list of Bundle object"""
605 """parse the AUTO_GEN_FILE to return a list of Bundle object"""
606 bundles = []
606 bundles = []
607 for line in content.splitlines():
607 for line in content.splitlines():
608 if line.startswith(b'PENDING-v1 '):
608 if line.startswith(b'PENDING-v1 '):
609 bundles.append(GeneratingBundle.from_line(line))
609 bundles.append(GeneratingBundle.from_line(line))
610 elif line.startswith(b'DONE-v1 '):
610 elif line.startswith(b'DONE-v1 '):
611 bundles.append(GeneratedBundle.from_line(line))
611 bundles.append(GeneratedBundle.from_line(line))
612 return bundles
612 return bundles
613
613
614
614
615 def dumps_auto_gen(bundles):
615 def dumps_auto_gen(bundles):
616 """serialize a list of Bundle as a AUTO_GEN_FILE content"""
616 """serialize a list of Bundle as a AUTO_GEN_FILE content"""
617 lines = []
617 lines = []
618 for b in bundles:
618 for b in bundles:
619 lines.append(b"%s\n" % b.to_line())
619 lines.append(b"%s\n" % b.to_line())
620 lines.sort()
620 lines.sort()
621 return b"".join(lines)
621 return b"".join(lines)
622
622
623
623
624 def read_auto_gen(repo):
624 def read_auto_gen(repo):
625 """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
625 """read the AUTO_GEN_FILE for the <repo> a list of Bundle object"""
626 data = repo.vfs.tryread(AUTO_GEN_FILE)
626 data = repo.vfs.tryread(AUTO_GEN_FILE)
627 if not data:
627 if not data:
628 return []
628 return []
629 return parse_auto_gen(data)
629 return parse_auto_gen(data)
630
630
631
631
632 def write_auto_gen(repo, bundles):
632 def write_auto_gen(repo, bundles):
633 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
633 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
634 assert repo._cb_lock_ref is not None
634 assert repo._cb_lock_ref is not None
635 data = dumps_auto_gen(bundles)
635 data = dumps_auto_gen(bundles)
636 with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
636 with repo.vfs(AUTO_GEN_FILE, mode=b'wb', atomictemp=True) as f:
637 f.write(data)
637 f.write(data)
638
638
639
639
640 def generate_manifest(bundles):
640 def generate_manifest(bundles):
641 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
641 """write a list of Bundle objects into the repo's AUTO_GEN_FILE"""
642 bundles = list(bundles)
642 bundles = list(bundles)
643 bundles.sort(key=lambda b: b.bundle_type)
643 bundles.sort(key=lambda b: b.bundle_type)
644 lines = []
644 lines = []
645 for b in bundles:
645 for b in bundles:
646 lines.append(b"%s\n" % b.manifest_line())
646 lines.append(b"%s\n" % b.manifest_line())
647 return b"".join(lines)
647 return b"".join(lines)
648
648
649
649
650 def update_ondisk_manifest(repo):
650 def update_ondisk_manifest(repo):
651 """update the clonebundle manifest with latest url"""
651 """update the clonebundle manifest with latest url"""
652 with repo.clonebundles_lock():
652 with repo.clonebundles_lock():
653 bundles = read_auto_gen(repo)
653 bundles = read_auto_gen(repo)
654
654
655 per_types = {}
655 per_types = {}
656 for b in bundles:
656 for b in bundles:
657 if not (b.ready and b.valid_for(repo)):
657 if not (b.ready and b.valid_for(repo)):
658 continue
658 continue
659 current = per_types.get(b.bundle_type)
659 current = per_types.get(b.bundle_type)
660 if current is not None and current.revs >= b.revs:
660 if current is not None and current.revs >= b.revs:
661 continue
661 continue
662 per_types[b.bundle_type] = b
662 per_types[b.bundle_type] = b
663 manifest = generate_manifest(per_types.values())
663 manifest = generate_manifest(per_types.values())
664 with repo.vfs(
664 with repo.vfs(
665 bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
665 bundlecaches.CB_MANIFEST_FILE, mode=b"wb", atomictemp=True
666 ) as f:
666 ) as f:
667 f.write(manifest)
667 f.write(manifest)
668
668
669
669
670 def update_bundle_list(repo, new_bundles=(), del_bundles=()):
670 def update_bundle_list(repo, new_bundles=(), del_bundles=()):
671 """modify the repo's AUTO_GEN_FILE
671 """modify the repo's AUTO_GEN_FILE
672
672
673 This method also regenerates the clone bundle manifest when needed"""
673 This method also regenerates the clone bundle manifest when needed"""
674 with repo.clonebundles_lock():
674 with repo.clonebundles_lock():
675 bundles = read_auto_gen(repo)
675 bundles = read_auto_gen(repo)
676 if del_bundles:
676 if del_bundles:
677 bundles = [b for b in bundles if b not in del_bundles]
677 bundles = [b for b in bundles if b not in del_bundles]
678 new_bundles = [b for b in new_bundles if b not in bundles]
678 new_bundles = [b for b in new_bundles if b not in bundles]
679 bundles.extend(new_bundles)
679 bundles.extend(new_bundles)
680 write_auto_gen(repo, bundles)
680 write_auto_gen(repo, bundles)
681 all_changed = []
681 all_changed = []
682 all_changed.extend(new_bundles)
682 all_changed.extend(new_bundles)
683 all_changed.extend(del_bundles)
683 all_changed.extend(del_bundles)
684 if any(b.ready for b in all_changed):
684 if any(b.ready for b in all_changed):
685 update_ondisk_manifest(repo)
685 update_ondisk_manifest(repo)
686
686
687
687
688 def cleanup_tmp_bundle(repo, target):
688 def cleanup_tmp_bundle(repo, target):
689 """remove a GeneratingBundle file and entry"""
689 """remove a GeneratingBundle file and entry"""
690 assert not target.ready
690 assert not target.ready
691 with repo.clonebundles_lock():
691 with repo.clonebundles_lock():
692 repo.vfs.tryunlink(target.filepath)
692 repo.vfs.tryunlink(target.filepath)
693 update_bundle_list(repo, del_bundles=[target])
693 update_bundle_list(repo, del_bundles=[target])
694
694
695
695
696 def finalize_one_bundle(repo, target):
696 def finalize_one_bundle(repo, target):
697 """upload a generated bundle and advertise it in the clonebundles.manifest"""
697 """upload a generated bundle and advertise it in the clonebundles.manifest"""
698 with repo.clonebundles_lock():
698 with repo.clonebundles_lock():
699 bundles = read_auto_gen(repo)
699 bundles = read_auto_gen(repo)
700 if target in bundles and target.valid_for(repo):
700 if target in bundles and target.valid_for(repo):
701 result = upload_bundle(repo, target)
701 result = upload_bundle(repo, target)
702 update_bundle_list(repo, new_bundles=[result])
702 update_bundle_list(repo, new_bundles=[result])
703 cleanup_tmp_bundle(repo, target)
703 cleanup_tmp_bundle(repo, target)
704
704
705
705
706 def find_outdated_bundles(repo, bundles):
706 def find_outdated_bundles(repo, bundles):
707 """finds outdated bundles"""
707 """finds outdated bundles"""
708 olds = []
708 olds = []
709 per_types = {}
709 per_types = {}
710 for b in bundles:
710 for b in bundles:
711 if not b.valid_for(repo):
711 if not b.valid_for(repo):
712 olds.append(b)
712 olds.append(b)
713 continue
713 continue
714 l = per_types.setdefault(b.bundle_type, [])
714 l = per_types.setdefault(b.bundle_type, [])
715 l.append(b)
715 l.append(b)
716 for key in sorted(per_types):
716 for key in sorted(per_types):
717 all = per_types[key]
717 all = per_types[key]
718 if len(all) > 1:
718 if len(all) > 1:
719 all.sort(key=lambda b: b.revs, reverse=True)
719 all.sort(key=lambda b: b.revs, reverse=True)
720 olds.extend(all[1:])
720 olds.extend(all[1:])
721 return olds
721 return olds
722
722
723
723
724 def collect_garbage(repo):
724 def collect_garbage(repo):
725 """finds outdated bundles and get them deleted"""
725 """finds outdated bundles and get them deleted"""
726 with repo.clonebundles_lock():
726 with repo.clonebundles_lock():
727 bundles = read_auto_gen(repo)
727 bundles = read_auto_gen(repo)
728 olds = find_outdated_bundles(repo, bundles)
728 olds = find_outdated_bundles(repo, bundles)
729 for o in olds:
729 for o in olds:
730 delete_bundle(repo, o)
730 delete_bundle(repo, o)
731 update_bundle_list(repo, del_bundles=olds)
731 update_bundle_list(repo, del_bundles=olds)
732
732
733
733
734 def upload_bundle(repo, bundle):
734 def upload_bundle(repo, bundle):
735 """upload the result of a GeneratingBundle and return a GeneratedBundle
735 """upload the result of a GeneratingBundle and return a GeneratedBundle
736
736
737 The upload is done using the `clone-bundles.upload-command`
737 The upload is done using the `clone-bundles.upload-command`
738 """
738 """
739 cmd = repo.ui.config(b'clone-bundles', b'upload-command')
739 cmd = repo.ui.config(b'clone-bundles', b'upload-command')
740 url = repo.ui.config(b'clone-bundles', b'url-template')
740 url = repo.ui.config(b'clone-bundles', b'url-template')
741 basename = repo.vfs.basename(bundle.filepath)
741 basename = repo.vfs.basename(bundle.filepath)
742 filepath = procutil.shellquote(bundle.filepath)
742 filepath = procutil.shellquote(bundle.filepath)
743 variables = {
743 variables = {
744 b'HGCB_BUNDLE_PATH': filepath,
744 b'HGCB_BUNDLE_PATH': filepath,
745 b'HGCB_BUNDLE_BASENAME': basename,
745 b'HGCB_BUNDLE_BASENAME': basename,
746 }
746 }
747 env = procutil.shellenviron(environ=variables)
747 env = procutil.shellenviron(environ=variables)
748 ret = repo.ui.system(cmd, environ=env)
748 ret = repo.ui.system(cmd, environ=env)
749 if ret:
749 if ret:
750 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
750 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
751 url = (
751 url = (
752 url.decode('utf8')
752 url.decode('utf8')
753 .format(basename=basename.decode('utf8'))
753 .format(basename=basename.decode('utf8'))
754 .encode('utf8')
754 .encode('utf8')
755 )
755 )
756 return bundle.uploaded(url, basename)
756 return bundle.uploaded(url, basename)
757
757
758
758
759 def delete_bundle(repo, bundle):
759 def delete_bundle(repo, bundle):
760 """delete a bundle from storage"""
760 """delete a bundle from storage"""
761 assert bundle.ready
761 assert bundle.ready
762 msg = b'clone-bundles: deleting bundle %s\n'
762 msg = b'clone-bundles: deleting bundle %s\n'
763 msg %= bundle.basename
763 msg %= bundle.basename
764 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
764 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
765 repo.ui.write(msg)
765 repo.ui.write(msg)
766 else:
766 else:
767 repo.ui.debug(msg)
767 repo.ui.debug(msg)
768
768
769 cmd = repo.ui.config(b'clone-bundles', b'delete-command')
769 cmd = repo.ui.config(b'clone-bundles', b'delete-command')
770 variables = {
770 variables = {
771 b'HGCB_BUNDLE_URL': bundle.file_url,
771 b'HGCB_BUNDLE_URL': bundle.file_url,
772 b'HGCB_BASENAME': bundle.basename,
772 b'HGCB_BASENAME': bundle.basename,
773 }
773 }
774 env = procutil.shellenviron(environ=variables)
774 env = procutil.shellenviron(environ=variables)
775 ret = repo.ui.system(cmd, environ=env)
775 ret = repo.ui.system(cmd, environ=env)
776 if ret:
776 if ret:
777 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
777 raise error.Abort(b"command returned status %d: %s" % (ret, cmd))
778
778
779
779
780 def auto_bundle_needed_actions(repo, bundles, op_id):
780 def auto_bundle_needed_actions(repo, bundles, op_id):
781 """find the list of bundles that need action
781 """find the list of bundles that need action
782
782
783 returns a list of RequestedBundle objects that need to be generated and
783 returns a list of RequestedBundle objects that need to be generated and
784 uploaded."""
784 uploaded."""
785 create_bundles = []
785 create_bundles = []
786 delete_bundles = []
786 delete_bundles = []
787 repo = repo.filtered(b"immutable")
787 repo = repo.filtered(b"immutable")
788 targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
788 targets = repo.ui.configlist(b'clone-bundles', b'auto-generate.formats')
789 ratio = float(
789 ratio = float(
790 repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
790 repo.ui.config(b'clone-bundles', b'trigger.below-bundled-ratio')
791 )
791 )
792 abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
792 abs_revs = repo.ui.configint(b'clone-bundles', b'trigger.revs')
793 revs = len(repo.changelog)
793 revs = len(repo.changelog)
794 generic_data = {
794 generic_data = {
795 'revs': revs,
795 'revs': revs,
796 'head_revs': repo.changelog.headrevs(),
796 'head_revs': repo.changelog.headrevs(),
797 'tip_rev': repo.changelog.tiprev(),
797 'tip_rev': repo.changelog.tiprev(),
798 'tip_node': node.hex(repo.changelog.tip()),
798 'tip_node': node.hex(repo.changelog.tip()),
799 'op_id': op_id,
799 'op_id': op_id,
800 }
800 }
801 for t in targets:
801 for t in targets:
802 if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
802 if new_bundle_needed(repo, bundles, ratio, abs_revs, t, revs):
803 data = generic_data.copy()
803 data = generic_data.copy()
804 data['bundle_type'] = t
804 data['bundle_type'] = t
805 b = RequestedBundle(**data)
805 b = RequestedBundle(**data)
806 create_bundles.append(b)
806 create_bundles.append(b)
807 delete_bundles.extend(find_outdated_bundles(repo, bundles))
807 delete_bundles.extend(find_outdated_bundles(repo, bundles))
808 return create_bundles, delete_bundles
808 return create_bundles, delete_bundles
809
809
810
810
811 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
811 def new_bundle_needed(repo, bundles, ratio, abs_revs, bundle_type, revs):
812 """consider the current cached content and trigger new bundles if needed"""
812 """consider the current cached content and trigger new bundles if needed"""
813 threshold = max((revs * ratio), (revs - abs_revs))
813 threshold = max((revs * ratio), (revs - abs_revs))
814 for b in bundles:
814 for b in bundles:
815 if not b.valid_for(repo) or b.bundle_type != bundle_type:
815 if not b.valid_for(repo) or b.bundle_type != bundle_type:
816 continue
816 continue
817 if b.revs > threshold:
817 if b.revs > threshold:
818 return False
818 return False
819 return True
819 return True
820
820
821
821
822 def start_one_bundle(repo, bundle):
822 def start_one_bundle(repo, bundle):
823 """start the generation of a single bundle file
823 """start the generation of a single bundle file
824
824
825 the `bundle` argument should be a RequestedBundle object.
825 the `bundle` argument should be a RequestedBundle object.
826
826
827 This data is passed to the `debugmakeclonebundles` "as is".
827 This data is passed to the `debugmakeclonebundles` "as is".
828 """
828 """
829 data = util.pickle.dumps(bundle)
829 data = util.pickle.dumps(bundle)
830 cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
830 cmd = [procutil.hgexecutable(), b'--cwd', repo.path, INTERNAL_CMD]
831 env = procutil.shellenviron()
831 env = procutil.shellenviron()
832 msg = b'clone-bundles: starting bundle generation: %s\n'
832 msg = b'clone-bundles: starting bundle generation: %s\n'
833 stdout = None
833 stdout = None
834 stderr = None
834 stderr = None
835 waits = []
835 waits = []
836 record_wait = None
836 record_wait = None
837 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
837 if repo.ui.configbool(b'devel', b'debug.clonebundles'):
838 stdout = procutil.stdout
838 stdout = procutil.stdout
839 stderr = procutil.stderr
839 stderr = procutil.stderr
840 repo.ui.write(msg % bundle.bundle_type)
840 repo.ui.write(msg % bundle.bundle_type)
841 record_wait = waits.append
841 record_wait = waits.append
842 else:
842 else:
843 repo.ui.debug(msg % bundle.bundle_type)
843 repo.ui.debug(msg % bundle.bundle_type)
844 bg = procutil.runbgcommand
844 bg = procutil.runbgcommand
845 bg(
845 bg(
846 cmd,
846 cmd,
847 env,
847 env,
848 stdin_bytes=data,
848 stdin_bytes=data,
849 stdout=stdout,
849 stdout=stdout,
850 stderr=stderr,
850 stderr=stderr,
851 record_wait=record_wait,
851 record_wait=record_wait,
852 )
852 )
853 for f in waits:
853 for f in waits:
854 f()
854 f()
855
855
856
856
857 INTERNAL_CMD = b'debug::internal-make-clone-bundles'
857 INTERNAL_CMD = b'debug::internal-make-clone-bundles'
858
858
859
859
860 @command(INTERNAL_CMD, [], b'')
860 @command(INTERNAL_CMD, [], b'')
861 def debugmakeclonebundles(ui, repo):
861 def debugmakeclonebundles(ui, repo):
862 """Internal command to auto-generate debug bundles"""
862 """Internal command to auto-generate debug bundles"""
863 requested_bundle = util.pickle.load(procutil.stdin)
863 requested_bundle = util.pickle.load(procutil.stdin)
864 procutil.stdin.close()
864 procutil.stdin.close()
865
865
866 collect_garbage(repo)
866 collect_garbage(repo)
867
867
868 fname = requested_bundle.suggested_filename
868 fname = requested_bundle.suggested_filename
869 fpath = repo.vfs.makedirs(b'tmp-bundles')
869 fpath = repo.vfs.makedirs(b'tmp-bundles')
870 fpath = repo.vfs.join(b'tmp-bundles', fname)
870 fpath = repo.vfs.join(b'tmp-bundles', fname)
871 bundle = requested_bundle.generating(fpath)
871 bundle = requested_bundle.generating(fpath)
872 update_bundle_list(repo, new_bundles=[bundle])
872 update_bundle_list(repo, new_bundles=[bundle])
873
873
874 requested_bundle.generate_bundle(repo, fpath)
874 requested_bundle.generate_bundle(repo, fpath)
875
875
876 repo.invalidate()
876 repo.invalidate()
877 finalize_one_bundle(repo, bundle)
877 finalize_one_bundle(repo, bundle)
878
878
879
879
880 def make_auto_bundler(source_repo):
880 def make_auto_bundler(source_repo):
881 reporef = weakref.ref(source_repo)
881 reporef = weakref.ref(source_repo)
882
882
883 def autobundle(tr):
883 def autobundle(tr):
884 repo = reporef()
884 repo = reporef()
885 assert repo is not None
885 assert repo is not None
886 bundles = read_auto_gen(repo)
886 bundles = read_auto_gen(repo)
887 new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
887 new, __ = auto_bundle_needed_actions(repo, bundles, b"%d_txn" % id(tr))
888 for data in new:
888 for data in new:
889 start_one_bundle(repo, data)
889 start_one_bundle(repo, data)
890 return None
890 return None
891
891
892 return autobundle
892 return autobundle
893
893
894
894
895 def reposetup(ui, repo):
895 def reposetup(ui, repo):
896 """install the two pieces needed for automatic clonebundle generation
896 """install the two pieces needed for automatic clonebundle generation
897
897
898 - add a "post-close" hook that fires bundling when needed
898 - add a "post-close" hook that fires bundling when needed
899 - introduce a clone-bundle lock to let multiple processes meddle with the
899 - introduce a clone-bundle lock to let multiple processes meddle with the
900 state files.
900 state files.
901 """
901 """
902 if not repo.local():
902 if not repo.local():
903 return
903 return
904
904
905 class autobundlesrepo(repo.__class__):
905 class autobundlesrepo(repo.__class__):
906 def transaction(self, *args, **kwargs):
906 def transaction(self, *args, **kwargs):
907 tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
907 tr = super(autobundlesrepo, self).transaction(*args, **kwargs)
908 enabled = repo.ui.configbool(
908 enabled = repo.ui.configbool(
909 b'clone-bundles',
909 b'clone-bundles',
910 b'auto-generate.on-change',
910 b'auto-generate.on-change',
911 )
911 )
912 targets = repo.ui.configlist(
912 targets = repo.ui.configlist(
913 b'clone-bundles', b'auto-generate.formats'
913 b'clone-bundles', b'auto-generate.formats'
914 )
914 )
915 if enabled and targets:
915 if enabled and targets:
916 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
916 tr.addpostclose(CAT_POSTCLOSE, make_auto_bundler(self))
917 return tr
917 return tr
918
918
919 @localrepo.unfilteredmethod
919 @localrepo.unfilteredmethod
920 def clonebundles_lock(self, wait=True):
920 def clonebundles_lock(self, wait=True):
921 '''Lock the repository file related to clone bundles'''
921 '''Lock the repository file related to clone bundles'''
922 if not util.safehasattr(self, '_cb_lock_ref'):
922 if not util.safehasattr(self, '_cb_lock_ref'):
923 self._cb_lock_ref = None
923 self._cb_lock_ref = None
924 l = self._currentlock(self._cb_lock_ref)
924 l = self._currentlock(self._cb_lock_ref)
925 if l is not None:
925 if l is not None:
926 l.lock()
926 l.lock()
927 return l
927 return l
928
928
929 l = self._lock(
929 l = self._lock(
930 vfs=self.vfs,
930 vfs=self.vfs,
931 lockname=b"clonebundleslock",
931 lockname=b"clonebundleslock",
932 wait=wait,
932 wait=wait,
933 releasefn=None,
933 releasefn=None,
934 acquirefn=None,
934 acquirefn=None,
935 desc=_(b'repository %s') % self.origroot,
935 desc=_(b'repository %s') % self.origroot,
936 )
936 )
937 self._cb_lock_ref = weakref.ref(l)
937 self._cb_lock_ref = weakref.ref(l)
938 return l
938 return l
939
939
940 repo._wlockfreeprefix.add(AUTO_GEN_FILE)
940 repo._wlockfreeprefix.add(AUTO_GEN_FILE)
941 repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
941 repo._wlockfreeprefix.add(bundlecaches.CB_MANIFEST_FILE)
942 repo.__class__ = autobundlesrepo
942 repo.__class__ = autobundlesrepo
943
943
944
944
945 @command(b'admin::clone-bundles-refresh', [], b'')
945 @command(
946 def cmd_admin_clone_bundles_refresh(ui, repo: localrepo.localrepository):
946 b'admin::clone-bundles-refresh',
947 [
948 (
949 b'',
950 b'background',
951 False,
952 _(b'start bundle generation in the background'),
953 ),
954 ],
955 b'',
956 )
957 def cmd_admin_clone_bundles_refresh(
958 ui,
959 repo: localrepo.localrepository,
960 background=False,
961 ):
947 """generate clone bundles according to the configuration
962 """generate clone bundles according to the configuration
948
963
949 This runs the logic for automatic generation, removing outdated bundles and
964 This runs the logic for automatic generation, removing outdated bundles and
950 generating new ones if necessary. See :hg:`help -e clone-bundles` for
965 generating new ones if necessary. See :hg:`help -e clone-bundles` for
951 details about how to configure this feature.
966 details about how to configure this feature.
952 """
967 """
953 debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
968 debug = repo.ui.configbool(b'devel', b'debug.clonebundles')
954 bundles = read_auto_gen(repo)
969 bundles = read_auto_gen(repo)
955 op_id = b"%d_acbr" % os.getpid()
970 op_id = b"%d_acbr" % os.getpid()
956 create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
971 create, delete = auto_bundle_needed_actions(repo, bundles, op_id)
957
972
958 # we clean up outdated bundle before generating new one to keep the last
973 # if some bundles are scheduled for creation in the background, they will
959 # two version of the bundle around for a while and avoid having to deal
974 # deal with garbage collection too, so no need to synchroniously do it.
960 # client that just got served a manifest.
975 #
976 # However if no bundles are scheduled for creation, we need to explicitly do
977 # it here.
978 if not (background and create):
979 # we clean up outdated bundles before generating new ones to keep the
980 # last two versions of the bundle around for a while and avoid having to
981 # deal with clients that just got served a manifest.
961 for o in delete:
982 for o in delete:
962 delete_bundle(repo, o)
983 delete_bundle(repo, o)
963 update_bundle_list(repo, del_bundles=delete)
984 update_bundle_list(repo, del_bundles=delete)
964
985
965 if create:
986 if create:
966 fpath = repo.vfs.makedirs(b'tmp-bundles')
987 fpath = repo.vfs.makedirs(b'tmp-bundles')
988
989 if background:
990 for requested_bundle in create:
991 start_one_bundle(repo, requested_bundle)
992 else:
967 for requested_bundle in create:
993 for requested_bundle in create:
968 if debug:
994 if debug:
969 msg = b'clone-bundles: starting bundle generation: %s\n'
995 msg = b'clone-bundles: starting bundle generation: %s\n'
970 repo.ui.write(msg % requested_bundle.bundle_type)
996 repo.ui.write(msg % requested_bundle.bundle_type)
971 fname = requested_bundle.suggested_filename
997 fname = requested_bundle.suggested_filename
972 fpath = repo.vfs.join(b'tmp-bundles', fname)
998 fpath = repo.vfs.join(b'tmp-bundles', fname)
973 generating_bundle = requested_bundle.generating(fpath)
999 generating_bundle = requested_bundle.generating(fpath)
974 update_bundle_list(repo, new_bundles=[generating_bundle])
1000 update_bundle_list(repo, new_bundles=[generating_bundle])
975 requested_bundle.generate_bundle(repo, fpath)
1001 requested_bundle.generate_bundle(repo, fpath)
976 result = upload_bundle(repo, generating_bundle)
1002 result = upload_bundle(repo, generating_bundle)
977 update_bundle_list(repo, new_bundles=[result])
1003 update_bundle_list(repo, new_bundles=[result])
978 update_ondisk_manifest(repo)
1004 update_ondisk_manifest(repo)
979 cleanup_tmp_bundle(repo, generating_bundle)
1005 cleanup_tmp_bundle(repo, generating_bundle)
980
1006
981
1007
982 @command(b'admin::clone-bundles-clear', [], b'')
1008 @command(b'admin::clone-bundles-clear', [], b'')
983 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
1009 def cmd_admin_clone_bundles_clear(ui, repo: localrepo.localrepository):
984 """remove existing clone bundle caches
1010 """remove existing clone bundle caches
985
1011
986 See `hg help admin::clone-bundles-refresh` for details on how to regenerate
1012 See `hg help admin::clone-bundles-refresh` for details on how to regenerate
987 them.
1013 them.
988
1014
989 This command will only affect bundles currently available, it will not
1015 This command will only affect bundles currently available, it will not
990 affect bundles being asynchronously generated.
1016 affect bundles being asynchronously generated.
991 """
1017 """
992 bundles = read_auto_gen(repo)
1018 bundles = read_auto_gen(repo)
993 delete = [b for b in bundles if b.ready]
1019 delete = [b for b in bundles if b.ready]
994 for o in delete:
1020 for o in delete:
995 delete_bundle(repo, o)
1021 delete_bundle(repo, o)
996 update_bundle_list(repo, del_bundles=delete)
1022 update_bundle_list(repo, del_bundles=delete)
@@ -1,337 +1,360 b''
1
1
2 #require no-reposimplestore no-chg
2 #require no-reposimplestore no-chg
3
3
4 initial setup
4 initial setup
5
5
6 $ hg init server
6 $ hg init server
7 $ cat >> server/.hg/hgrc << EOF
7 $ cat >> server/.hg/hgrc << EOF
8 > [extensions]
8 > [extensions]
9 > clonebundles =
9 > clonebundles =
10 >
10 >
11 > [clone-bundles]
11 > [clone-bundles]
12 > auto-generate.on-change = yes
12 > auto-generate.on-change = yes
13 > auto-generate.formats = v2
13 > auto-generate.formats = v2
14 > upload-command = cp "\$HGCB_BUNDLE_PATH" "$TESTTMP"/final-upload/
14 > upload-command = cp "\$HGCB_BUNDLE_PATH" "$TESTTMP"/final-upload/
15 > delete-command = rm -f "$TESTTMP/final-upload/\$HGCB_BASENAME"
15 > delete-command = rm -f "$TESTTMP/final-upload/\$HGCB_BASENAME"
16 > url-template = file://$TESTTMP/final-upload/{basename}
16 > url-template = file://$TESTTMP/final-upload/{basename}
17 >
17 >
18 > [devel]
18 > [devel]
19 > debug.clonebundles=yes
19 > debug.clonebundles=yes
20 > EOF
20 > EOF
21
21
22 $ mkdir final-upload
22 $ mkdir final-upload
23 $ hg clone server client
23 $ hg clone server client
24 updating to branch default
24 updating to branch default
25 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
25 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
26 $ cd client
26 $ cd client
27
27
28 Test bundles are generated on push
28 Test bundles are generated on push
29 ==================================
29 ==================================
30
30
31 $ touch foo
31 $ touch foo
32 $ hg -q commit -A -m 'add foo'
32 $ hg -q commit -A -m 'add foo'
33 $ touch bar
33 $ touch bar
34 $ hg -q commit -A -m 'add bar'
34 $ hg -q commit -A -m 'add bar'
35 $ hg push
35 $ hg push
36 pushing to $TESTTMP/server
36 pushing to $TESTTMP/server
37 searching for changes
37 searching for changes
38 adding changesets
38 adding changesets
39 adding manifests
39 adding manifests
40 adding file changes
40 adding file changes
41 2 changesets found
41 2 changesets found
42 added 2 changesets with 2 changes to 2 files
42 added 2 changesets with 2 changes to 2 files
43 clone-bundles: starting bundle generation: v2
43 clone-bundles: starting bundle generation: v2
44 $ cat ../server/.hg/clonebundles.manifest
44 $ cat ../server/.hg/clonebundles.manifest
45 file:/*/$TESTTMP/final-upload/full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
45 file:/*/$TESTTMP/final-upload/full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
46 $ ls -1 ../final-upload
46 $ ls -1 ../final-upload
47 full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
47 full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
48 $ ls -1 ../server/.hg/tmp-bundles
48 $ ls -1 ../server/.hg/tmp-bundles
49
49
50 Newer bundles are generated with more pushes
50 Newer bundles are generated with more pushes
51 --------------------------------------------
51 --------------------------------------------
52
52
53 $ touch baz
53 $ touch baz
54 $ hg -q commit -A -m 'add baz'
54 $ hg -q commit -A -m 'add baz'
55 $ touch buz
55 $ touch buz
56 $ hg -q commit -A -m 'add buz'
56 $ hg -q commit -A -m 'add buz'
57 $ hg push
57 $ hg push
58 pushing to $TESTTMP/server
58 pushing to $TESTTMP/server
59 searching for changes
59 searching for changes
60 adding changesets
60 adding changesets
61 adding manifests
61 adding manifests
62 adding file changes
62 adding file changes
63 4 changesets found
63 4 changesets found
64 added 2 changesets with 2 changes to 2 files
64 added 2 changesets with 2 changes to 2 files
65 clone-bundles: starting bundle generation: v2
65 clone-bundles: starting bundle generation: v2
66
66
67 $ cat ../server/.hg/clonebundles.manifest
67 $ cat ../server/.hg/clonebundles.manifest
68 file:/*/$TESTTMP/final-upload/full-v2-4_revs-6427147b985a_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
68 file:/*/$TESTTMP/final-upload/full-v2-4_revs-6427147b985a_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
69 $ ls -1 ../final-upload
69 $ ls -1 ../final-upload
70 full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
70 full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
71 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
71 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
72 $ ls -1 ../server/.hg/tmp-bundles
72 $ ls -1 ../server/.hg/tmp-bundles
73
73
74 Older bundles are cleaned up with more pushes
74 Older bundles are cleaned up with more pushes
75 ---------------------------------------------
75 ---------------------------------------------
76
76
77 $ touch faz
77 $ touch faz
78 $ hg -q commit -A -m 'add faz'
78 $ hg -q commit -A -m 'add faz'
79 $ touch fuz
79 $ touch fuz
80 $ hg -q commit -A -m 'add fuz'
80 $ hg -q commit -A -m 'add fuz'
81 $ hg push
81 $ hg push
82 pushing to $TESTTMP/server
82 pushing to $TESTTMP/server
83 searching for changes
83 searching for changes
84 adding changesets
84 adding changesets
85 adding manifests
85 adding manifests
86 adding file changes
86 adding file changes
87 clone-bundles: deleting bundle full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
87 clone-bundles: deleting bundle full-v2-2_revs-aaff8d2ffbbf_tip-*_txn.hg (glob)
88 6 changesets found
88 6 changesets found
89 added 2 changesets with 2 changes to 2 files
89 added 2 changesets with 2 changes to 2 files
90 clone-bundles: starting bundle generation: v2
90 clone-bundles: starting bundle generation: v2
91
91
92 $ cat ../server/.hg/clonebundles.manifest
92 $ cat ../server/.hg/clonebundles.manifest
93 file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
93 file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
94 $ ls -1 ../final-upload
94 $ ls -1 ../final-upload
95 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
95 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
96 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
96 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
97 $ ls -1 ../server/.hg/tmp-bundles
97 $ ls -1 ../server/.hg/tmp-bundles
98
98
99 Test conditions to get them generated
99 Test conditions to get them generated
100 =====================================
100 =====================================
101
101
102 Check ratio
102 Check ratio
103
103
104 $ cat >> ../server/.hg/hgrc << EOF
104 $ cat >> ../server/.hg/hgrc << EOF
105 > [clone-bundles]
105 > [clone-bundles]
106 > trigger.below-bundled-ratio = 0.5
106 > trigger.below-bundled-ratio = 0.5
107 > EOF
107 > EOF
108 $ touch far
108 $ touch far
109 $ hg -q commit -A -m 'add far'
109 $ hg -q commit -A -m 'add far'
110 $ hg push
110 $ hg push
111 pushing to $TESTTMP/server
111 pushing to $TESTTMP/server
112 searching for changes
112 searching for changes
113 adding changesets
113 adding changesets
114 adding manifests
114 adding manifests
115 adding file changes
115 adding file changes
116 added 1 changesets with 1 changes to 1 files
116 added 1 changesets with 1 changes to 1 files
117 $ cat ../server/.hg/clonebundles.manifest
117 $ cat ../server/.hg/clonebundles.manifest
118 file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
118 file:/*/$TESTTMP/final-upload/full-v2-6_revs-b1010e95ea00_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
119 $ ls -1 ../final-upload
119 $ ls -1 ../final-upload
120 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
120 full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
121 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
121 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
122 $ ls -1 ../server/.hg/tmp-bundles
122 $ ls -1 ../server/.hg/tmp-bundles
123
123
124 Check absolute number of revisions
124 Check absolute number of revisions
125
125
126 $ cat >> ../server/.hg/hgrc << EOF
126 $ cat >> ../server/.hg/hgrc << EOF
127 > [clone-bundles]
127 > [clone-bundles]
128 > trigger.revs = 2
128 > trigger.revs = 2
129 > EOF
129 > EOF
130 $ touch bur
130 $ touch bur
131 $ hg -q commit -A -m 'add bur'
131 $ hg -q commit -A -m 'add bur'
132 $ hg push
132 $ hg push
133 pushing to $TESTTMP/server
133 pushing to $TESTTMP/server
134 searching for changes
134 searching for changes
135 adding changesets
135 adding changesets
136 adding manifests
136 adding manifests
137 adding file changes
137 adding file changes
138 clone-bundles: deleting bundle full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
138 clone-bundles: deleting bundle full-v2-4_revs-6427147b985a_tip-*_txn.hg (glob)
139 8 changesets found
139 8 changesets found
140 added 1 changesets with 1 changes to 1 files
140 added 1 changesets with 1 changes to 1 files
141 clone-bundles: starting bundle generation: v2
141 clone-bundles: starting bundle generation: v2
142 $ cat ../server/.hg/clonebundles.manifest
142 $ cat ../server/.hg/clonebundles.manifest
143 file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
143 file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
144 $ ls -1 ../final-upload
144 $ ls -1 ../final-upload
145 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
145 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
146 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
146 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
147 $ ls -1 ../server/.hg/tmp-bundles
147 $ ls -1 ../server/.hg/tmp-bundles
148
148
149 (that one would not generate new bundles)
149 (that one would not generate new bundles)
150
150
151 $ touch tur
151 $ touch tur
152 $ hg -q commit -A -m 'add tur'
152 $ hg -q commit -A -m 'add tur'
153 $ hg push
153 $ hg push
154 pushing to $TESTTMP/server
154 pushing to $TESTTMP/server
155 searching for changes
155 searching for changes
156 adding changesets
156 adding changesets
157 adding manifests
157 adding manifests
158 adding file changes
158 adding file changes
159 added 1 changesets with 1 changes to 1 files
159 added 1 changesets with 1 changes to 1 files
160 $ cat ../server/.hg/clonebundles.manifest
160 $ cat ../server/.hg/clonebundles.manifest
161 file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
161 file:/*/$TESTTMP/final-upload/full-v2-8_revs-8353e8af1306_tip-*_txn.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
162 $ ls -1 ../final-upload
162 $ ls -1 ../final-upload
163 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
163 full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
164 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
164 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
165 $ ls -1 ../server/.hg/tmp-bundles
165 $ ls -1 ../server/.hg/tmp-bundles
166
166
167 Test generation through the dedicated command
167 Test generation through the dedicated command
168 =============================================
168 =============================================
169
169
170 $ cat >> ../server/.hg/hgrc << EOF
170 $ cat >> ../server/.hg/hgrc << EOF
171 > [clone-bundles]
171 > [clone-bundles]
172 > auto-generate.on-change = no
172 > auto-generate.on-change = no
173 > EOF
173 > EOF
174
174
175 Check the command can generate content when needed
175 Check the command can generate content when needed
176 --------------------------------------------------
176 --------------------------------------------------
177
177
178 Do a push that makes the condition fulfilled,
178 Do a push that makes the condition fulfilled,
179 Yet it should not automatically generate a bundle with
179 Yet it should not automatically generate a bundle with
180 "auto-generate.on-change" not set.
180 "auto-generate.on-change" not set.
181
181
182 $ touch quoi
182 $ touch quoi
183 $ hg -q commit -A -m 'add quoi'
183 $ hg -q commit -A -m 'add quoi'
184
184
185 $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
185 $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
186 $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
186 $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
187 $ ls -1 ../server/.hg/tmp-bundles
187 $ ls -1 ../server/.hg/tmp-bundles
188
188
189 $ hg push
189 $ hg push
190 pushing to $TESTTMP/server
190 pushing to $TESTTMP/server
191 searching for changes
191 searching for changes
192 adding changesets
192 adding changesets
193 adding manifests
193 adding manifests
194 adding file changes
194 adding file changes
195 added 1 changesets with 1 changes to 1 files
195 added 1 changesets with 1 changes to 1 files
196
196
197 $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
197 $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
198 $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
198 $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
199 $ ls -1 ../server/.hg/tmp-bundles
199 $ ls -1 ../server/.hg/tmp-bundles
200 $ test "$pre_push_manifest" = "$post_push_manifest"
200 $ test "$pre_push_manifest" = "$post_push_manifest"
201 $ test "$pre_push_upload" = "$post_push_upload"
201 $ test "$pre_push_upload" = "$post_push_upload"
202
202
203 Running the command should detect the stale bundles, and do the full automatic
203 Running the command should detect the stale bundles, and do the full automatic
204 generation logic.
204 generation logic.
205
205
206 $ hg -R ../server/ admin::clone-bundles-refresh
206 $ hg -R ../server/ admin::clone-bundles-refresh
207 clone-bundles: deleting bundle full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
207 clone-bundles: deleting bundle full-v2-6_revs-b1010e95ea00_tip-*_txn.hg (glob)
208 clone-bundles: starting bundle generation: v2
208 clone-bundles: starting bundle generation: v2
209 10 changesets found
209 10 changesets found
210 $ cat ../server/.hg/clonebundles.manifest
210 $ cat ../server/.hg/clonebundles.manifest
211 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
211 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
212 $ ls -1 ../final-upload
212 $ ls -1 ../final-upload
213 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
213 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
214 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
214 full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
215 $ ls -1 ../server/.hg/tmp-bundles
215 $ ls -1 ../server/.hg/tmp-bundles
216
216
217 Check the command cleans up older bundles when possible
217 Check the command cleans up older bundles when possible
218 -------------------------------------------------------
218 -------------------------------------------------------
219
219
220 $ hg -R ../server/ admin::clone-bundles-refresh
220 $ hg -R ../server/ admin::clone-bundles-refresh
221 clone-bundles: deleting bundle full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
221 clone-bundles: deleting bundle full-v2-8_revs-8353e8af1306_tip-*_txn.hg (glob)
222 $ cat ../server/.hg/clonebundles.manifest
222 $ cat ../server/.hg/clonebundles.manifest
223 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
223 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
224 $ ls -1 ../final-upload
224 $ ls -1 ../final-upload
225 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
225 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
226 $ ls -1 ../server/.hg/tmp-bundles
226 $ ls -1 ../server/.hg/tmp-bundles
227
227
228 Nothing is generated when the bundles are sufficiently up to date
228 Nothing is generated when the bundles are sufficiently up to date
229 -----------------------------------------------------------------
229 -----------------------------------------------------------------
230
230
231 $ touch feur
231 $ touch feur
232 $ hg -q commit -A -m 'add feur'
232 $ hg -q commit -A -m 'add feur'
233
233
234 $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
234 $ pre_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
235 $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
235 $ pre_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
236 $ ls -1 ../server/.hg/tmp-bundles
236 $ ls -1 ../server/.hg/tmp-bundles
237
237
238 $ hg push
238 $ hg push
239 pushing to $TESTTMP/server
239 pushing to $TESTTMP/server
240 searching for changes
240 searching for changes
241 adding changesets
241 adding changesets
242 adding manifests
242 adding manifests
243 adding file changes
243 adding file changes
244 added 1 changesets with 1 changes to 1 files
244 added 1 changesets with 1 changes to 1 files
245
245
246 $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
246 $ post_push_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
247 $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
247 $ post_push_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
248 $ ls -1 ../server/.hg/tmp-bundles
248 $ ls -1 ../server/.hg/tmp-bundles
249 $ test "$pre_push_manifest" = "$post_push_manifest"
249 $ test "$pre_push_manifest" = "$post_push_manifest"
250 $ test "$pre_push_upload" = "$post_push_upload"
250 $ test "$pre_push_upload" = "$post_push_upload"
251
251
252 $ hg -R ../server/ admin::clone-bundles-refresh
252 $ hg -R ../server/ admin::clone-bundles-refresh
253
253
254 $ post_refresh_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
254 $ post_refresh_manifest=`cat ../server/.hg/clonebundles.manifest|f --sha256 | sed 's/.*=//' | cat`
255 $ post_refresh_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
255 $ post_refresh_upload=`ls -1 ../final-upload|f --sha256 | sed 's/.*=//' | cat`
256 $ ls -1 ../server/.hg/tmp-bundles
256 $ ls -1 ../server/.hg/tmp-bundles
257 $ test "$pre_push_manifest" = "$post_refresh_manifest"
257 $ test "$pre_push_manifest" = "$post_refresh_manifest"
258 $ test "$pre_push_upload" = "$post_refresh_upload"
258 $ test "$pre_push_upload" = "$post_refresh_upload"
259
259
260 Test modification of configuration
260 Test modification of configuration
261 ==================================
261 ==================================
262
262
263 Testing that later runs adapt to configuration changes even if the repository is
263 Testing that later runs adapt to configuration changes even if the repository is
264 unchanged.
264 unchanged.
265
265
266 adding more formats
266 adding more formats
267 -------------------
267 -------------------
268
268
269 bundle for added formats should be generated
269 bundle for added formats should be generated
270
270
271 change configuration
271 change configuration
272
272
273 $ cat >> ../server/.hg/hgrc << EOF
273 $ cat >> ../server/.hg/hgrc << EOF
274 > [clone-bundles]
274 > [clone-bundles]
275 > auto-generate.formats = v1, v2
275 > auto-generate.formats = v1, v2
276 > EOF
276 > EOF
277
277
278 refresh the bundles
278 refresh the bundles
279
279
280 $ hg -R ../server/ admin::clone-bundles-refresh
280 $ hg -R ../server/ admin::clone-bundles-refresh
281 clone-bundles: starting bundle generation: v1
281 clone-bundles: starting bundle generation: v1
282 11 changesets found
282 11 changesets found
283
283
284 the bundle for the "new" format should have been added
284 the bundle for the "new" format should have been added
285
285
286 $ cat ../server/.hg/clonebundles.manifest
286 $ cat ../server/.hg/clonebundles.manifest
287 file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
287 file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
288 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
288 file:/*/$TESTTMP/final-upload/full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
289 $ ls -1 ../final-upload
289 $ ls -1 ../final-upload
290 full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
290 full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
291 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
291 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
292 $ ls -1 ../server/.hg/tmp-bundles
292 $ ls -1 ../server/.hg/tmp-bundles
293
293
294 Changing the ratio
294 Changing the ratio
295 ------------------
295 ------------------
296
296
297 Changing the ratio to something that would have triggered a bundle during the last push.
297 Changing the ratio to something that would have triggered a bundle during the last push.
298
298
299 $ cat >> ../server/.hg/hgrc << EOF
299 $ cat >> ../server/.hg/hgrc << EOF
300 > [clone-bundles]
300 > [clone-bundles]
301 > trigger.below-bundled-ratio = 0.95
301 > trigger.below-bundled-ratio = 0.95
302 > EOF
302 > EOF
303
303
304 refresh the bundles
304 refresh the bundles
305
305
306 $ hg -R ../server/ admin::clone-bundles-refresh
306 $ hg -R ../server/ admin::clone-bundles-refresh
307 clone-bundles: starting bundle generation: v2
307 clone-bundles: starting bundle generation: v2
308 11 changesets found
308 11 changesets found
309
309
310
310
311 the "outdated' bundle should be refreshed
311 the "outdated' bundle should be refreshed
312
312
313 $ cat ../server/.hg/clonebundles.manifest
313 $ cat ../server/.hg/clonebundles.manifest
314 file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
314 file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
315 file:/*/$TESTTMP/final-upload/full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
315 file:/*/$TESTTMP/final-upload/full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
316 $ ls -1 ../final-upload
316 $ ls -1 ../final-upload
317 full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
317 full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
318 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
318 full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
319 full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
319 full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
320 $ ls -1 ../server/.hg/tmp-bundles
320 $ ls -1 ../server/.hg/tmp-bundles
321
321
322 Test more command options
322 Test more command options
323 =========================
323 =========================
324
324
325 bundle clearing
325 bundle clearing
326 ---------------
326 ---------------
327
327
328 $ hg -R ../server/ admin::clone-bundles-clear
328 $ hg -R ../server/ admin::clone-bundles-clear
329 clone-bundles: deleting bundle full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
329 clone-bundles: deleting bundle full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
330 clone-bundles: deleting bundle full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
330 clone-bundles: deleting bundle full-v2-10_revs-3b6f57f17d70_tip-*_acbr.hg (glob)
331 clone-bundles: deleting bundle full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
331 clone-bundles: deleting bundle full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
332
332
333 Nothing should remain
333 Nothing should remain
334
334
335 $ cat ../server/.hg/clonebundles.manifest
335 $ cat ../server/.hg/clonebundles.manifest
336 $ ls -1 ../final-upload
336 $ ls -1 ../final-upload
337 $ ls -1 ../server/.hg/tmp-bundles
337 $ ls -1 ../server/.hg/tmp-bundles
338
339 background generation
340 ---------------------
341
342 generate bundle using background subprocess
343 (since we are in devel mode, the command will still wait for the background
344 process to end)
345
346 $ hg -R ../server/ admin::clone-bundles-refresh --background
347 11 changesets found
348 11 changesets found
349 clone-bundles: starting bundle generation: v1
350 clone-bundles: starting bundle generation: v2
351
352 bundles should have been generated
353
354 $ cat ../server/.hg/clonebundles.manifest
355 file:/*/$TESTTMP/final-upload/full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v1 REQUIRESNI=true (glob)
356 file:/*/$TESTTMP/final-upload/full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg BUNDLESPEC=v2 REQUIRESNI=true (glob)
357 $ ls -1 ../final-upload
358 full-v1-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
359 full-v2-11_revs-4226b1cd5fda_tip-*_acbr.hg (glob)
360 $ ls -1 ../server/.hg/tmp-bundles
General Comments 0
You need to be logged in to leave comments. Login now