##// END OF EJS Templates
internals: correct capitalization of 'compression' stream level parameter
Kim Alvefur -
r37823:8c8b6d13 stable
parent child Browse files
Show More
@@ -1,677 +1,677 b''
1 1 Bundle2 refers to a data format that is used for both on-disk storage
2 2 and over-the-wire transfer of repository data and state.
3 3
4 4 The data format allows the capture of multiple components of
5 5 repository data. Contrast with the initial bundle format, which
6 6 only captured *changegroup* data (and couldn't store bookmarks,
7 7 phases, etc).
8 8
9 9 Bundle2 is used for:
10 10
11 11 * Transferring data from a repository (e.g. as part of an ``hg clone``
12 12 or ``hg pull`` operation).
13 13 * Transferring data to a repository (e.g. as part of an ``hg push``
14 14 operation).
15 15 * Storing data on disk (e.g. the result of an ``hg bundle``
16 16 operation).
17 17 * Transferring the results of a repository operation (e.g. the
18 18 reply to an ``hg push`` operation).
19 19
20 20 At its highest level, a bundle2 payload is a stream that begins
21 21 with some metadata and consists of a series of *parts*, with each
22 22 part describing repository data or state or the result of an
23 23 operation. New bundle2 parts are introduced over time when there is
24 24 a need to capture a new form of data. A *capabilities* mechanism
25 25 exists to allow peers to understand which bundle2 parts the other
26 26 understands.
27 27
28 28 Stream Format
29 29 =============
30 30
31 31 A bundle2 payload consists of a magic string (``HG20``) followed by
32 32 stream level parameters, followed by any number of payload *parts*.
33 33
34 34 It may help to think of the stream level parameters as *headers* and the
35 35 payload parts as the *body*.
36 36
37 37 Stream Level Parameters
38 38 -----------------------
39 39
40 40 Following the magic string is data that defines parameters applicable to the
41 41 entire payload.
42 42
43 43 Stream level parameters begin with a 32-bit unsigned big-endian integer.
44 44 The value of this integer defines the number of bytes of stream level
45 45 parameters that follow.
46 46
47 47 The *N* bytes of raw data contains a space separated list of parameters.
48 48 Each parameter consists of a required name and an optional value.
49 49
50 50 Parameters have the form ``<name>`` or ``<name>=<value>``.
51 51
52 52 Both the parameter name and value are URL quoted.
53 53
54 54 Names MUST start with a letter. If the first letter is lower case, the
55 55 parameter is advisory and can safely be ignored. If the first letter
56 56 is upper case, the parameter is mandatory and the handler MUST stop if
57 57 it is unable to process it.
58 58
59 59 Stream level parameters apply to the entire bundle2 payload. Lower-level
60 60 options should go into a bundle2 part instead.
61 61
62 62 The following stream level parameters are defined:
63 63
64 compression
64 Compression
65 65 Compression format of payload data. ``GZ`` denotes zlib. ``BZ``
66 66 denotes bzip2. ``ZS`` denotes zstandard.
67 67
68 68 When defined, all bytes after the stream level parameters are
69 69 compressed using the compression format defined by this parameter.
70 70
71 71 If this parameter isn't present, data is raw/uncompressed.
72 72
73 73 This parameter MUST be mandatory because attempting to consume
74 74 streams without knowing how to decode the underlying bytes will
75 75 result in errors.
76 76
77 77 Payload Part
78 78 ------------
79 79
80 80 Following the stream level parameters are 0 or more payload parts. Each
81 81 payload part consists of a header and a body.
82 82
83 83 The payload part header consists of a 32-bit unsigned big-endian integer
84 84 defining the number of bytes in the header that follow. The special
85 85 value ``0`` indicates the end of the bundle2 stream.
86 86
87 87 The binary format of the part header is as follows:
88 88
89 89 * 8-bit unsigned size of the part name
90 90 * N-bytes alphanumeric part name
91 91 * 32-bit unsigned big-endian part ID
92 92 * N bytes part parameter data
93 93
94 94 The *part name* identifies the type of the part. A part name with an
95 95 UPPERCASE letter is mandatory. Otherwise, the part is advisory. A
96 96 consumer should abort if it encounters a mandatory part it doesn't know
97 97 how to process. See the sections below for each defined part type.
98 98
99 99 The *part ID* is a unique identifier within the bundle used to refer to a
100 100 specific part. It should be unique within the bundle2 payload.
101 101
102 102 Part parameter data consists of:
103 103
104 104 * 1 byte number of mandatory parameters
105 105 * 1 byte number of advisory parameters
106 106 * 2 * N bytes of sizes of parameter key and values
107 107 * N * M blobs of values for parameter key and values
108 108
109 109 Following the 2 bytes of mandatory and advisory parameter counts are
110 110 2-tuples of bytes of the sizes of each parameter. e.g.
111 111 (<key size>, <value size>).
112 112
113 113 Following that are the raw values, without padding. Mandatory parameters
114 114 come first, followed by advisory parameters.
115 115
116 116 Each parameter's key MUST be unique within the part.
117 117
118 118 Following the part parameter data is the part payload. The part payload
119 119 consists of a series of framed chunks. The frame header is a 32-bit
120 120 big-endian integer defining the size of the chunk. The N bytes of raw
121 121 payload data follows.
122 122
123 123 The part payload consists of 0 or more chunks.
124 124
125 125 A chunk with size ``0`` denotes the end of the part payload. Therefore,
126 126 there will always be at least 1 32-bit integer following the payload
127 127 part header.
128 128
129 129 A chunk size of ``-1`` is used to signal an *interrupt*. If such a chunk
130 130 size is seen, the stream processor should process the next bytes as a new
131 131 payload part. After this payload part, processing of the original,
132 132 interrupted part should resume.
133 133
134 134 Capabilities
135 135 ============
136 136
137 137 Bundle2 is a dynamic format that can evolve over time. For example,
138 138 when a new repository data concept is invented, a new bundle2 part
139 139 is typically invented to hold that data. In addition, parts performing
140 140 similar functionality may come into existence if there is a better
141 141 mechanism for performing certain functionality.
142 142
143 143 Because the bundle2 format evolves over time, peers need to understand
144 144 what bundle2 features the other can understand. The *capabilities*
145 145 mechanism is how those features are expressed.
146 146
147 147 Bundle2 capabilities are logically expressed as a dictionary of
148 148 string key-value pairs where the keys are strings and the values
149 149 are lists of strings.
150 150
151 151 Capabilities are encoded for exchange between peers. The encoded
152 152 capabilities blob consists of a newline (``\n``) delimited list of
153 153 entries. Each entry has the form ``<key>`` or ``<key>=<value>``,
154 154 depending if the capability has a value.
155 155
156 156 The capability name is URL quoted (``%XX`` encoding of URL unsafe
157 157 characters).
158 158
159 159 The value, if present, is formed by URL quoting each value in
160 160 the capability list and concatenating the result with a comma (``,``).
161 161
162 162 For example, the capabilities ``novaluekey`` and ``listvaluekey``
163 163 with values ``value 1`` and ``value 2``. This would be encoded as:
164 164
165 165 listvaluekey=value%201,value%202\nnovaluekey
166 166
167 167 The sections below detail the defined bundle2 capabilities.
168 168
169 169 HG20
170 170 ----
171 171
172 172 Denotes that the peer supports the bundle2 data format.
173 173
174 174 bookmarks
175 175 ---------
176 176
177 177 Denotes that the peer supports the ``bookmarks`` part.
178 178
179 179 Peers should not issue mandatory ``bookmarks`` parts unless this
180 180 capability is present.
181 181
182 182 changegroup
183 183 -----------
184 184
185 185 Denotes which versions of the *changegroup* format the peer can
186 186 receive. Values include ``01``, ``02``, and ``03``.
187 187
188 188 The peer should not generate changegroup data for a version not
189 189 specified by this capability.
190 190
191 191 checkheads
192 192 ----------
193 193
194 194 Denotes which forms of heads checking the peer supports.
195 195
196 196 If ``related`` is in the value, then the peer supports the ``check:heads``
197 197 part and the peer is capable of detecting race conditions when applying
198 198 changelog data.
199 199
200 200 digests
201 201 -------
202 202
203 203 Denotes which hashing formats the peer supports.
204 204
205 205 Values are names of hashing function. Values include ``md5``, ``sha1``,
206 206 and ``sha512``.
207 207
208 208 error
209 209 -----
210 210
211 211 Denotes which ``error:`` parts the peer supports.
212 212
213 213 Value is a list of strings of ``error:`` part names. Valid values
214 214 include ``abort``, ``unsupportecontent``, ``pushraced``, and ``pushkey``.
215 215
216 216 Peers should not issue an ``error:`` part unless the type of that
217 217 part is listed as supported by this capability.
218 218
219 219 listkeys
220 220 --------
221 221
222 222 Denotes that the peer supports the ``listkeys`` part.
223 223
224 224 hgtagsfnodes
225 225 ------------
226 226
227 227 Denotes that the peer supports the ``hgtagsfnodes`` part.
228 228
229 229 obsmarkers
230 230 ----------
231 231
232 232 Denotes that the peer supports the ``obsmarker`` part and which versions
233 233 of the obsolescence data format it can receive. Values are strings like
234 234 ``V<N>``. e.g. ``V1``.
235 235
236 236 phases
237 237 ------
238 238
239 239 Denotes that the peer supports the ``phases`` part.
240 240
241 241 pushback
242 242 --------
243 243
244 244 Denotes that the peer supports sending/receiving bundle2 data in response
245 245 to a bundle2 request.
246 246
247 247 This capability is typically used by servers that employ server-side
248 248 rewriting of pushed repository data. For example, a server may wish to
249 249 automatically rebase pushed changesets. When this capability is present,
250 250 the server can send a bundle2 response containing the rewritten changeset
251 251 data and the client will apply it.
252 252
253 253 pushkey
254 254 -------
255 255
256 256 Denotes that the peer supports the ``puskey`` part.
257 257
258 258 remote-changegroup
259 259 ------------------
260 260
261 261 Denotes that the peer supports the ``remote-changegroup`` part and
262 262 which protocols it can use to fetch remote changegroup data.
263 263
264 264 Values are protocol names. e.g. ``http`` and ``https``.
265 265
266 266 stream
267 267 ------
268 268
269 269 Denotes that the peer supports ``stream*`` parts in order to support
270 270 *stream clone*.
271 271
272 272 Values are which ``stream*`` parts the peer supports. ``v2`` denotes
273 273 support for the ``stream2`` part.
274 274
275 275 Bundle2 Part Types
276 276 ==================
277 277
278 278 The sections below detail the various bundle2 part types.
279 279
280 280 bookmarks
281 281 ---------
282 282
283 283 The ``bookmarks`` part holds bookmarks information.
284 284
285 285 This part has no parameters.
286 286
287 287 The payload consists of entries defining bookmarks. Each entry consists of:
288 288
289 289 * 20 bytes binary changeset node.
290 290 * 2 bytes big endian short defining bookmark name length.
291 291 * N bytes defining bookmark name.
292 292
293 293 Receivers typically update bookmarks to match the state specified in
294 294 this part.
295 295
296 296 changegroup
297 297 -----------
298 298
299 299 The ``changegroup`` part contains *changegroup* data (changelog, manifestlog,
300 300 and filelog revision data).
301 301
302 302 The following part parameters are defined for this part.
303 303
304 304 version
305 305 Changegroup version string. e.g. ``01``, ``02``, and ``03``. This parameter
306 306 determines how to interpret the changegroup data within the part.
307 307
308 308 nbchanges
309 309 The number of changesets in this changegroup. This parameter can be used
310 310 to aid in the display of progress bars, etc during part application.
311 311
312 312 treemanifest
313 313 Whether the changegroup contains tree manifests.
314 314
315 315 targetphase
316 316 The target phase of changesets in this part. Value is an integer of
317 317 the target phase.
318 318
319 319 The payload of this part is raw changegroup data. See
320 320 :hg:`help internals.changegroups` for the format of changegroup data.
321 321
322 322 check:bookmarks
323 323 ---------------
324 324
325 325 The ``check:bookmarks`` part is inserted into a bundle as a means for the
326 326 receiver to validate that the sender's known state of bookmarks matches
327 327 the receiver's.
328 328
329 329 This part has no parameters.
330 330
331 331 The payload is a binary stream of bookmark data. Each entry in the stream
332 332 consists of:
333 333
334 334 * 20 bytes binary node that bookmark is associated with
335 335 * 2 bytes unsigned short defining length of bookmark name
336 336 * N bytes containing the bookmark name
337 337
338 338 If all bits in the node value are ``1``, then this signifies a missing
339 339 bookmark.
340 340
341 341 When the receiver encounters this part, for each bookmark in the part
342 342 payload, it should validate that the current bookmark state matches
343 343 the specified state. If it doesn't, then the receiver should take
344 344 appropriate action. (In the case of pushes, this mismatch signifies
345 345 a race condition and the receiver should consider rejecting the push.)
346 346
347 347 check:heads
348 348 -----------
349 349
350 350 The ``check:heads`` part is a means to validate that the sender's state
351 351 of DAG heads matches the receiver's.
352 352
353 353 This part has no parameters.
354 354
355 355 The body of this part is an array of 20 byte binary nodes representing
356 356 changeset heads.
357 357
358 358 Receivers should compare the set of heads defined in this part to the
359 359 current set of repo heads and take action if there is a mismatch in that
360 360 set.
361 361
362 362 Note that this part applies to *all* heads in the repo.
363 363
364 364 check:phases
365 365 ------------
366 366
367 367 The ``check:phases`` part validates that the sender's state of phase
368 368 boundaries matches the receiver's.
369 369
370 370 This part has no parameters.
371 371
372 372 The payload consists of an array of 24 byte entries. Each entry is
373 373 a big endian 32-bit integer defining the phase integer and 20 byte
374 374 binary node value.
375 375
376 376 For each changeset defined in this part, the receiver should validate
377 377 that its current phase matches the phase defined in this part. The
378 378 receiver should take appropriate action if a mismatch occurs.
379 379
380 380 check:updated-heads
381 381 -------------------
382 382
383 383 The ``check:updated-heads`` part validates that the sender's state of
384 384 DAG heads updated by this bundle matches the receiver's.
385 385
386 386 This type is nearly identical to ``check:heads`` except the heads
387 387 in the payload are only a subset of heads in the repository. The
388 388 receiver should validate that all nodes specified by the sender are
389 389 branch heads and take appropriate action if not.
390 390
391 391 error:abort
392 392 -----------
393 393
394 394 The ``error:abort`` part conveys a fatal error.
395 395
396 396 The following part parameters are defined:
397 397
398 398 message
399 399 The string content of the error message.
400 400
401 401 hint
402 402 Supplemental string giving a hint on how to fix the problem.
403 403
404 404 error:pushkey
405 405 -------------
406 406
407 407 The ``error:pushkey`` part conveys an error in the *pushkey* protocol.
408 408
409 409 The following part parameters are defined:
410 410
411 411 namespace
412 412 The pushkey domain that exhibited the error.
413 413
414 414 key
415 415 The key whose update failed.
416 416
417 417 new
418 418 The value we tried to set the key to.
419 419
420 420 old
421 421 The old value of the key (as supplied by the client).
422 422
423 423 ret
424 424 The integer result code for the pushkey request.
425 425
426 426 in-reply-to
427 427 Part ID that triggered this error.
428 428
429 429 This part is generated if there was an error applying *pushkey* data.
430 430 Pushkey data includes bookmarks, phases, and obsolescence markers.
431 431
432 432 error:pushraced
433 433 ---------------
434 434
435 435 The ``error:pushraced`` part conveys that an error occurred and
436 436 the likely cause is losing a race with another pusher.
437 437
438 438 The following part parameters are defined:
439 439
440 440 message
441 441 String error message.
442 442
443 443 This part is typically emitted when a receiver examining ``check:*``
444 444 parts encountered inconsistency between incoming state and local state.
445 445 The likely cause of that inconsistency is another repository change
446 446 operation (often another client performing an ``hg push``).
447 447
448 448 error:unsupportedcontent
449 449 ------------------------
450 450
451 451 The ``error:unsupportedcontent`` part conveys that a bundle2 receiver
452 452 encountered a part or content it was not able to handle.
453 453
454 454 The following part parameters are defined:
455 455
456 456 parttype
457 457 The name of the part that triggered this error.
458 458
459 459 params
460 460 ``\0`` delimited list of parameters.
461 461
462 462 hgtagsfnodes
463 463 ------------
464 464
465 465 The ``hgtagsfnodes`` type defines file nodes for the ``.hgtags`` file
466 466 for various changesets.
467 467
468 468 This part has no parameters.
469 469
470 470 The payload is an array of pairs of 20 byte binary nodes. The first node
471 471 is a changeset node. The second node is the ``.hgtags`` file node.
472 472
473 473 Resolving tags requires resolving the ``.hgtags`` file node for changesets.
474 474 On large repositories, this can be expensive. Repositories cache the
475 475 mapping of changeset to ``.hgtags`` file node on disk as a performance
476 476 optimization. This part allows that cached data to be transferred alongside
477 477 changeset data.
478 478
479 479 Receivers should update their ``.hgtags`` cache file node mappings with
480 480 the incoming data.
481 481
482 482 listkeys
483 483 --------
484 484
485 485 The ``listkeys`` part holds content for a *pushkey* namespace.
486 486
487 487 The following part parameters are defined:
488 488
489 489 namespace
490 490 The pushkey domain this data belongs to.
491 491
492 492 The part payload contains a newline (``\n``) delimited list of
493 493 tab (``\t``) delimited key-value pairs defining entries in this pushkey
494 494 namespace.
495 495
496 496 obsmarkers
497 497 ----------
498 498
499 499 The ``obsmarkers`` part defines obsolescence markers.
500 500
501 501 This part has no parameters.
502 502
503 503 The payload consists of obsolescence markers using the on-disk markers
504 504 format. The first byte defines the version format.
505 505
506 506 The receiver should apply the obsolescence markers defined in this
507 507 part. A ``reply:obsmarkers`` part should be sent to the sender, if possible.
508 508
509 509 output
510 510 ------
511 511
512 512 The ``output`` part is used to display output on the receiver.
513 513
514 514 This part has no parameters.
515 515
516 516 The payload consists of raw data to be printed on the receiver.
517 517
518 518 phase-heads
519 519 -----------
520 520
521 521 The ``phase-heads`` part defines phase boundaries.
522 522
523 523 This part has no parameters.
524 524
525 525 The payload consists of an array of 24 byte entries. Each entry is
526 526 a big endian 32-bit integer defining the phase integer and 20 byte
527 527 binary node value.
528 528
529 529 pushkey
530 530 -------
531 531
532 532 The ``pushkey`` part communicates an intent to perform a ``pushkey``
533 533 request.
534 534
535 535 The following part parameters are defined:
536 536
537 537 namespace
538 538 The pushkey domain to operate on.
539 539
540 540 key
541 541 The key within the pushkey namespace that is being changed.
542 542
543 543 old
544 544 The old value for the key being changed.
545 545
546 546 new
547 547 The new value for the key being changed.
548 548
549 549 This part has no payload.
550 550
551 551 The receiver should perform a pushkey operation as described by this
552 552 part's parameters.
553 553
554 554 If the pushey operation fails, a ``reply:pushkey`` part should be sent
555 555 back to the sender, if possible. The ``in-reply-to`` part parameter
556 556 should reference the source part.
557 557
558 558 pushvars
559 559 --------
560 560
561 561 The ``pushvars`` part defines environment variables that should be
562 562 set when processing this bundle2 payload.
563 563
564 564 The part's advisory parameters define environment variables.
565 565
566 566 There is no part payload.
567 567
568 568 When received, part parameters are prefixed with ``USERVAR_`` and the
569 569 resulting variables are defined in the hooks context for the current
570 570 bundle2 application. This part provides a mechanism for senders to
571 571 inject extra state into the hook execution environment on the receiver.
572 572
573 573 remote-changegroup
574 574 ------------------
575 575
576 576 The ``remote-changegroup`` part defines an external location of a bundle
577 577 to apply. This part can be used by servers to serve pre-generated bundles
578 578 hosted at arbitrary URLs.
579 579
580 580 The following part parameters are defined:
581 581
582 582 url
583 583 The URL of the remote bundle.
584 584
585 585 size
586 586 The size in bytes of the remote bundle.
587 587
588 588 digests
589 589 A space separated list of the digest types provided in additional
590 590 part parameters.
591 591
592 592 digest:<type>
593 593 The hexadecimal representation of the digest (hash) of the remote bundle.
594 594
595 595 There is no payload for this part type.
596 596
597 597 When encountered, clients should attempt to fetch the URL being advertised
598 598 and read and apply it as a bundle.
599 599
600 600 The ``size`` and ``digest:<type>`` parameters should be used to validate
601 601 that the downloaded bundle matches what was advertised. If a mismatch occurs,
602 602 the client should abort.
603 603
604 604 reply:changegroup
605 605 -----------------
606 606
607 607 The ``reply:changegroup`` part conveys the results of application of a
608 608 ``changegroup`` part.
609 609
610 610 The following part parameters are defined:
611 611
612 612 return
613 613 Integer return code from changegroup application.
614 614
615 615 in-reply-to
616 616 Part ID of part this reply is in response to.
617 617
618 618 reply:obsmarkers
619 619 ----------------
620 620
621 621 The ``reply:obsmarkers`` part conveys the results of applying an
622 622 ``obsmarkers`` part.
623 623
624 624 The following part parameters are defined:
625 625
626 626 new
627 627 The integer number of new markers that were applied.
628 628
629 629 in-reply-to
630 630 The part ID that this part is in reply to.
631 631
632 632 reply:pushkey
633 633 -------------
634 634
635 635 The ``reply:pushkey`` part conveys the result of a *pushkey* operation.
636 636
637 637 The following part parameters are defined:
638 638
639 639 return
640 640 Integer result code from pushkey operation.
641 641
642 642 in-reply-to
643 643 Part ID that triggered this pushkey operation.
644 644
645 645 This part has no payload.
646 646
647 647 replycaps
648 648 ---------
649 649
650 650 The ``replycaps`` part notifies the receiver that a reply bundle should
651 651 be created.
652 652
653 653 This part has no parameters.
654 654
655 655 The payload consists of a bundle2 capabilities blob.
656 656
657 657 stream2
658 658 -------
659 659
660 660 The ``stream2`` part contains *streaming clone* version 2 data.
661 661
662 662 The following part parameters are defined:
663 663
664 664 requirements
665 665 URL quoted repository requirements string. Requirements are delimited by a
666 666 command (``,``).
667 667
668 668 filecount
669 669 The total number of files being transferred in the payload.
670 670
671 671 bytecount
672 672 The total size of file content being transferred in the payload.
673 673
674 674 The payload consists of raw stream clone version 2 data.
675 675
676 676 The ``filecount`` and ``bytecount`` parameters can be used for progress and
677 677 reporting purposes. The values may not be exact.
General Comments 0
You need to be logged in to leave comments. Login now