##// END OF EJS Templates
help: align description of 'base rev' with reality [issue5488]...
Kim Alvefur -
r31214:167b69cc default
parent child Browse files
Show More
@@ -1,215 +1,215 b''
1 Revision logs - or *revlogs* - are an append only data structure for
1 Revision logs - or *revlogs* - are an append only data structure for
2 storing discrete entries, or *revisions*. They are the primary storage
2 storing discrete entries, or *revisions*. They are the primary storage
3 mechanism of repository data.
3 mechanism of repository data.
4
4
5 Revlogs effectively model a directed acyclic graph (DAG). Each node
5 Revlogs effectively model a directed acyclic graph (DAG). Each node
6 has edges to 1 or 2 *parent* nodes. Each node contains metadata and
6 has edges to 1 or 2 *parent* nodes. Each node contains metadata and
7 the raw value for that node.
7 the raw value for that node.
8
8
9 Revlogs consist of entries which have metadata and revision data.
9 Revlogs consist of entries which have metadata and revision data.
10 Metadata includes the hash of the revision's content, sizes, and
10 Metadata includes the hash of the revision's content, sizes, and
11 links to its *parent* entries. The collective metadata is referred
11 links to its *parent* entries. The collective metadata is referred
12 to as the *index* and the revision data is the *data*.
12 to as the *index* and the revision data is the *data*.
13
13
14 Revision data is stored as a series of compressed deltas against previous
14 Revision data is stored as a series of compressed deltas against previous
15 revisions.
15 revisions.
16
16
17 Revlogs are written in an append-only fashion. We never need to rewrite
17 Revlogs are written in an append-only fashion. We never need to rewrite
18 a file to insert nor do we need to remove data. Rolling back in-progress
18 a file to insert nor do we need to remove data. Rolling back in-progress
19 writes can be performed by truncating files. Read locks can be avoided
19 writes can be performed by truncating files. Read locks can be avoided
20 using simple techniques. This means that references to other data in
20 using simple techniques. This means that references to other data in
21 the same revlog *always* refer to a previous entry.
21 the same revlog *always* refer to a previous entry.
22
22
23 Revlogs can be modeled as 0-indexed arrays. The first revision is
23 Revlogs can be modeled as 0-indexed arrays. The first revision is
24 revision #0 and the second is revision #1. The revision -1 is typically
24 revision #0 and the second is revision #1. The revision -1 is typically
25 used to mean *does not exist* or *not defined*.
25 used to mean *does not exist* or *not defined*.
26
26
27 File Format
27 File Format
28 ===========
28 ===========
29
29
30 A revlog begins with a 32-bit big endian integer holding version info
30 A revlog begins with a 32-bit big endian integer holding version info
31 and feature flags. This integer is shared with the first revision
31 and feature flags. This integer is shared with the first revision
32 entry.
32 entry.
33
33
34 This integer is logically divided into 2 16-bit shorts. The least
34 This integer is logically divided into 2 16-bit shorts. The least
35 significant half of the integer is the format/version short. The other
35 significant half of the integer is the format/version short. The other
36 short holds feature flags that dictate behavior of the revlog.
36 short holds feature flags that dictate behavior of the revlog.
37
37
38 Only 1 bit of the format/version short is currently used. Remaining
38 Only 1 bit of the format/version short is currently used. Remaining
39 bits are reserved for future use.
39 bits are reserved for future use.
40
40
41 The following values for the format/version short are defined:
41 The following values for the format/version short are defined:
42
42
43 0
43 0
44 The original revlog version.
44 The original revlog version.
45 1
45 1
46 RevlogNG (*next generation*). It replaced version 0 when it was
46 RevlogNG (*next generation*). It replaced version 0 when it was
47 implemented in 2006.
47 implemented in 2006.
48
48
49 The feature flags short consists of bit flags. Where 0 is the least
49 The feature flags short consists of bit flags. Where 0 is the least
50 significant bit, the following bit offsets define flags:
50 significant bit, the following bit offsets define flags:
51
51
52 0
52 0
53 Store revision data inline.
53 Store revision data inline.
54 1
54 1
55 Generaldelta encoding.
55 Generaldelta encoding.
56
56
57 2-15
57 2-15
58 Reserved for future use.
58 Reserved for future use.
59
59
60 The following header values are common:
60 The following header values are common:
61
61
62 00 00 00 01
62 00 00 00 01
63 RevlogNG
63 RevlogNG
64 00 01 00 01
64 00 01 00 01
65 RevlogNG + inline
65 RevlogNG + inline
66 00 02 00 01
66 00 02 00 01
67 RevlogNG + generaldelta
67 RevlogNG + generaldelta
68 00 03 00 01
68 00 03 00 01
69 RevlogNG + inline + generaldelta
69 RevlogNG + inline + generaldelta
70
70
71 Following the 32-bit header is the remainder of the first index entry.
71 Following the 32-bit header is the remainder of the first index entry.
72 Following that are remaining *index* data. Inlined revision data is
72 Following that are remaining *index* data. Inlined revision data is
73 possibly located between index entries. More on this layout is described
73 possibly located between index entries. More on this layout is described
74 below.
74 below.
75
75
76 RevlogNG Format
76 RevlogNG Format
77 ===============
77 ===============
78
78
79 RevlogNG (version 1) begins with an index describing the revisions in
79 RevlogNG (version 1) begins with an index describing the revisions in
80 the revlog. If the ``inline`` flag is set, revision data is stored inline,
80 the revlog. If the ``inline`` flag is set, revision data is stored inline,
81 or between index entries (as opposed to in a separate container).
81 or between index entries (as opposed to in a separate container).
82
82
83 Each index entry is 64 bytes. The byte layout of each entry is as
83 Each index entry is 64 bytes. The byte layout of each entry is as
84 follows, with byte 0 being the first byte (all data stored as big endian):
84 follows, with byte 0 being the first byte (all data stored as big endian):
85
85
86 0-3 (4 bytes) (rev 0 only)
86 0-3 (4 bytes) (rev 0 only)
87 Revlog header
87 Revlog header
88
88
89 0-5 (6 bytes)
89 0-5 (6 bytes)
90 Absolute offset of revision data from beginning of revlog.
90 Absolute offset of revision data from beginning of revlog.
91
91
92 6-7 (2 bytes)
92 6-7 (2 bytes)
93 Bit flags impacting revision behavior. The following bit offsets define:
93 Bit flags impacting revision behavior. The following bit offsets define:
94
94
95 0: REVIDX_ISCENSORED revision has censor metadata, must be verified.
95 0: REVIDX_ISCENSORED revision has censor metadata, must be verified.
96
96
97 1: REVIDX_ELLIPSIS revision hash does not match its data. Used by
97 1: REVIDX_ELLIPSIS revision hash does not match its data. Used by
98 narrowhg
98 narrowhg
99
99
100 2: REVIDX_EXTSTORED revision data is stored externally.
100 2: REVIDX_EXTSTORED revision data is stored externally.
101
101
102 8-11 (4 bytes)
102 8-11 (4 bytes)
103 Compressed length of revision data / chunk as stored in revlog.
103 Compressed length of revision data / chunk as stored in revlog.
104
104
105 12-15 (4 bytes)
105 12-15 (4 bytes)
106 Uncompressed length of revision data. This is the size of the full
106 Uncompressed length of revision data. This is the size of the full
107 revision data, not the size of the chunk post decompression.
107 revision data, not the size of the chunk post decompression.
108
108
109 16-19 (4 bytes)
109 16-19 (4 bytes)
110 Base or previous revision this revision's delta was produced against.
110 Base or previous revision this revision's delta was produced against.
111 -1 means this revision holds full text (as opposed to a delta).
111 This revision holds full text (as opposed to a delta) if it points to
112 For generaldelta repos, this is the previous revision in the delta
112 itself. For generaldelta repos, this is the previous revision in the
113 chain. For non-generaldelta repos, this is the base or first
113 delta chain. For non-generaldelta repos, this is the base or first
114 revision in the delta chain.
114 revision in the delta chain.
115
115
116 20-23 (4 bytes)
116 20-23 (4 bytes)
117 A revision this revision is *linked* to. This allows a revision in
117 A revision this revision is *linked* to. This allows a revision in
118 one revlog to be forever associated with a revision in another
118 one revlog to be forever associated with a revision in another
119 revlog. For example, a file's revlog may point to the changelog
119 revlog. For example, a file's revlog may point to the changelog
120 revision that introduced it.
120 revision that introduced it.
121
121
122 24-27 (4 bytes)
122 24-27 (4 bytes)
123 Revision of 1st parent. -1 indicates no parent.
123 Revision of 1st parent. -1 indicates no parent.
124
124
125 28-31 (4 bytes)
125 28-31 (4 bytes)
126 Revision of 2nd parent. -1 indicates no 2nd parent.
126 Revision of 2nd parent. -1 indicates no 2nd parent.
127
127
128 32-63 (32 bytes)
128 32-63 (32 bytes)
129 Hash of revision's full text. Currently, SHA-1 is used and only
129 Hash of revision's full text. Currently, SHA-1 is used and only
130 the first 20 bytes of this field are used. The rest of the bytes
130 the first 20 bytes of this field are used. The rest of the bytes
131 are ignored and should be stored as \0.
131 are ignored and should be stored as \0.
132
132
133 If inline revision data is being stored, the compressed revision data
133 If inline revision data is being stored, the compressed revision data
134 (of length from bytes offset 8-11 from the index entry) immediately
134 (of length from bytes offset 8-11 from the index entry) immediately
135 follows the index entry. There is no header on the revision data. There
135 follows the index entry. There is no header on the revision data. There
136 is no padding between it and the index entries before and after.
136 is no padding between it and the index entries before and after.
137
137
138 If revision data is not inline, then raw revision data is stored in a
138 If revision data is not inline, then raw revision data is stored in a
139 separate byte container. The offsets from bytes 0-5 and the compressed
139 separate byte container. The offsets from bytes 0-5 and the compressed
140 length from bytes 8-11 define how to access this data.
140 length from bytes 8-11 define how to access this data.
141
141
142 The first 4 bytes of the revlog are shared between the revlog header
142 The first 4 bytes of the revlog are shared between the revlog header
143 and the 6 byte absolute offset field from the first revlog entry.
143 and the 6 byte absolute offset field from the first revlog entry.
144
144
145 Delta Chains
145 Delta Chains
146 ============
146 ============
147
147
148 Revision data is encoded as a chain of *chunks*. Each chain begins with
148 Revision data is encoded as a chain of *chunks*. Each chain begins with
149 the compressed original full text for that revision. Each subsequent
149 the compressed original full text for that revision. Each subsequent
150 *chunk* is a *delta* against the previous revision. We therefore call
150 *chunk* is a *delta* against the previous revision. We therefore call
151 these chains of chunks/deltas *delta chains*.
151 these chains of chunks/deltas *delta chains*.
152
152
153 The full text for a revision is reconstructed by loading the original
153 The full text for a revision is reconstructed by loading the original
154 full text for the base revision of a *delta chain* and then applying
154 full text for the base revision of a *delta chain* and then applying
155 *deltas* until the target revision is reconstructed.
155 *deltas* until the target revision is reconstructed.
156
156
157 *Delta chains* are limited in length so lookup time is bound. They are
157 *Delta chains* are limited in length so lookup time is bound. They are
158 limited to ~2x the length of the revision's data. The linear distance
158 limited to ~2x the length of the revision's data. The linear distance
159 between the base chunk and the final chunk is also limited so the
159 between the base chunk and the final chunk is also limited so the
160 amount of read I/O to load all chunks in the delta chain is bound.
160 amount of read I/O to load all chunks in the delta chain is bound.
161
161
162 Deltas and delta chains are either computed against the previous
162 Deltas and delta chains are either computed against the previous
163 revision in the revlog or another revision (almost certainly one of
163 revision in the revlog or another revision (almost certainly one of
164 the parents of the revision). Historically, deltas were computed against
164 the parents of the revision). Historically, deltas were computed against
165 the previous revision. The *generaldelta* revlog feature flag (enabled
165 the previous revision. The *generaldelta* revlog feature flag (enabled
166 by default in Mercurial 3.7) activates the mode where deltas are
166 by default in Mercurial 3.7) activates the mode where deltas are
167 computed against an arbitrary revision (almost certainly a parent revision).
167 computed against an arbitrary revision (almost certainly a parent revision).
168
168
169 File Storage
169 File Storage
170 ============
170 ============
171
171
172 Revlogs logically consist of an index (metadata of entries) and
172 Revlogs logically consist of an index (metadata of entries) and
173 revision data. This data may be stored together in a single file or in
173 revision data. This data may be stored together in a single file or in
174 separate files. The mechanism used is indicated by the ``inline`` feature
174 separate files. The mechanism used is indicated by the ``inline`` feature
175 flag on the revlog.
175 flag on the revlog.
176
176
177 Mercurial's behavior is to use inline storage until a revlog reaches a
177 Mercurial's behavior is to use inline storage until a revlog reaches a
178 certain size, at which point it will be converted to non-inline. The
178 certain size, at which point it will be converted to non-inline. The
179 reason there is a size limit on inline storage is to establish an upper
179 reason there is a size limit on inline storage is to establish an upper
180 bound on how much data must be read to load the index. It would be a waste
180 bound on how much data must be read to load the index. It would be a waste
181 to read tens or hundreds of extra megabytes of data just to access the
181 to read tens or hundreds of extra megabytes of data just to access the
182 index data.
182 index data.
183
183
184 The actual layout of revlog files on disk is governed by the repository's
184 The actual layout of revlog files on disk is governed by the repository's
185 *store format*. Typically, a ``.i`` file represents the index revlog
185 *store format*. Typically, a ``.i`` file represents the index revlog
186 (possibly containing inline data) and a ``.d`` file holds the revision data.
186 (possibly containing inline data) and a ``.d`` file holds the revision data.
187
187
188 Revision Entries
188 Revision Entries
189 ================
189 ================
190
190
191 Revision entries consist of an optional 1 byte header followed by an
191 Revision entries consist of an optional 1 byte header followed by an
192 encoding of the revision data. The headers are as follows:
192 encoding of the revision data. The headers are as follows:
193
193
194 \0 (0x00)
194 \0 (0x00)
195 Revision data is the entirety of the entry, including this header.
195 Revision data is the entirety of the entry, including this header.
196 u (0x75)
196 u (0x75)
197 Raw revision data follows.
197 Raw revision data follows.
198 x (0x78)
198 x (0x78)
199 zlib (RFC 1950) data.
199 zlib (RFC 1950) data.
200
200
201 The 0x78 value is actually the first byte of the zlib header (CMF byte).
201 The 0x78 value is actually the first byte of the zlib header (CMF byte).
202
202
203 Hash Computation
203 Hash Computation
204 ================
204 ================
205
205
206 The hash of the revision is stored in the index and is used both as a primary
206 The hash of the revision is stored in the index and is used both as a primary
207 key and for data integrity verification.
207 key and for data integrity verification.
208
208
209 Currently, SHA-1 is the only supported hashing algorithm. To obtain the SHA-1
209 Currently, SHA-1 is the only supported hashing algorithm. To obtain the SHA-1
210 hash of a revision:
210 hash of a revision:
211
211
212 1. Hash the parent nodes
212 1. Hash the parent nodes
213 2. Hash the fulltext of the revision
213 2. Hash the fulltext of the revision
214
214
215 The 20 byte node ids of the parents are fed into the hasher in ascending order.
215 The 20 byte node ids of the parents are fed into the hasher in ascending order.
General Comments 0
You need to be logged in to leave comments. Login now