##// END OF EJS Templates
deltas: set estimated compression upper bound to "3x" instead of "10x"...
deltas: set estimated compression upper bound to "3x" instead of "10x" In pratice, we very rarely observer compression better than "3x" on manifest deltas. Having a more aggressive estimate significantly helps our pathological use case on a private repository. Here are a comparison of timings using different upper bound. Estimated compression | ø | ×10 | ×5 | ×3 | timing | 14.11 | 2.61 | 1.96 | 1.53 | We also tested the impact of this series on an array of public repositories. This shown no impact in either size nor timing. Full data set below for those interested. Size ---- Regarding size, not significant impact have been noticed on neither public nor private repositories. Here are the number we gathered on public repositories: zlib/upperbound | no | 10x | 5x | 3x mercurial | 5 875 730 | 5 875 730 | 5 875 730 | 5 875 730 pypy | 27 782 913 | 27 782 913 | 27 782 913 | 27 782 913 netbeans | 159 161 207 | 159 161 207 | 159 161 207 | 159 959 879 (+0.5%) mozilla-central | 323 841 642 | 323 841 642 | 323 841 642 | 319 867 519 (-2.5%) mozilla-try | 746 649 123 | 746 649 123 | 746 649 123 | 741 155 568 (-0.7%) private-repo | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%) zstd/upperbound | no | 10x | 5x | 3x mercurial | 5 895 206 | 5 895 206 | 5 895 206 | 5 895 206 pypy | 28 689 230 | 28 689 230 | 28 689 230 | 28 689 230 netbeans | 157 636 387 | 157 636 387 | 157 636 387 | 159 692 678 (+1.3%) mozilla-central | 317 650 281 | 317 650 281 | 317 650 281 | 319 613 603 (+0.6%) mozilla-try | 737 555 275 | 737 555 275 | 737 555 275 | 738 079 473 (+0.1%) private-repo | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%) Speed ------ Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds. mercurial zlib | no | 10x | 5x | 3x | total | 65.551783 | 65.388887 | 65.260658 | 65.321199 | max | 0.034544 | 0.034571 | 0.034659 | 0.034521 | 99.99% | 0.034544 | 0.034571 | 0.034659 | 0.034521 | zstd | no | 10x | 5x | 3x | total | 49.118449 | 49.054062 | 48.753588 | 48.740230 | max | 0.009338 | 0.009239 | 0.009202 | 0.009178 | 99.99% | 0.007618 | 0.007639 | 0.007626 | 0.007621 | pypy zlib | no | 10x | 5x | 3x | total | 560.865984 | 558.983817 | 559.083815 | 559.349152 | max | 0.219614 | 0.215922 | 0.218112 | 0.218107 | 99.99% | 0.219614 | 0.215922 | 0.218112 | 0.218107 | zstd | no | 10x | 5x | 3x | total | 349.393280 | 347.395819 | 347.185407 | 345.643985 | max | 0.084143 | 0.083536 | 0.081834 | 0.082178 | 99.99% | 0.039445 | 0.039639 | 0.039612 | 0.039175 | netbeans zlib | no | 10x | 5x | 3x | total | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 | max | 2.666852 | 2.672059 | 2.662453 | 2.662936 | 99.99% | 2.058772 | 2.070429 | 2.069569 | 2.064653 | zstd | no | 10x | 5x | 3x | total | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 | max | 2.063482 | 2.062851 | 2.065229 | 2.060147 | 99.99% | 1.146647 | 1.143794 | 1.142933 | 1.146529 | mozilla zlib | no | 10x | 5x | 3x | total | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 | max | 3.383474 | 3.387400 | 3.405711 | 3.387316 | 99.99% | 1.006755 | 1.005954 | 1.007700 | 1.007373 | zstd | no | 10x | 5x | 3x | total | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 | max | 1.460822 | 1.449640 | 1.439747 | 1.465304 | 99.99% | 0.527111 | 0.527377 | 0.527807 | 0.527226 |

File last commit:

r42354:be073355 default
r42669:4a3abb33 default
Show More
dagops.rs
276 lines | 8.8 KiB | application/rls-services+xml | RustLexer
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278 // dagops.rs
//
// Copyright 2019 Georges Racinet <georges.racinet@octobus.net>
//
// This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version.
//! Miscellaneous DAG operations
//!
//! # Terminology
//! - By *relative heads* of a collection of revision numbers (`Revision`),
//! we mean those revisions that have no children among the collection.
//! - Similarly *relative roots* of a collection of `Revision`, we mean
//! those whose parents, if any, don't belong to the collection.
use super::{Graph, GraphError, Revision, NULL_REVISION};
Georges Racinet
rust-dagops: range of revisions...
r42353 use crate::ancestors::AncestorsIterator;
use std::collections::{BTreeSet, HashSet};
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278
fn remove_parents(
graph: &impl Graph,
rev: Revision,
set: &mut HashSet<Revision>,
) -> Result<(), GraphError> {
for parent in graph.parents(rev)?.iter() {
if *parent != NULL_REVISION {
set.remove(parent);
}
}
Ok(())
}
/// Relative heads out of some revisions, passed as an iterator.
///
/// These heads are defined as those revisions that have no children
/// among those emitted by the iterator.
///
/// # Performance notes
/// Internally, this clones the iterator, and builds a `HashSet` out of it.
///
/// This function takes an `Iterator` instead of `impl IntoIterator` to
/// guarantee that cloning the iterator doesn't result in cloning the full
/// construct it comes from.
pub fn heads<'a>(
graph: &impl Graph,
iter_revs: impl Clone + Iterator<Item = &'a Revision>,
) -> Result<HashSet<Revision>, GraphError> {
let mut heads: HashSet<Revision> = iter_revs.clone().cloned().collect();
heads.remove(&NULL_REVISION);
for rev in iter_revs {
Georges Racinet
rust: itering less on MissingAncestors.bases for max()...
r41866 if *rev != NULL_REVISION {
remove_parents(graph, *rev, &mut heads)?;
}
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278 }
Ok(heads)
}
/// Retain in `revs` only its relative heads.
///
/// This is an in-place operation, so that control of the incoming
/// set is left to the caller.
/// - a direct Python binding would probably need to build its own `HashSet`
/// from an incoming iterable, even if its sole purpose is to extract the
/// heads.
/// - a Rust caller can decide whether cloning beforehand is appropriate
///
/// # Performance notes
/// Internally, this function will store a full copy of `revs` in a `Vec`.
pub fn retain_heads(
graph: &impl Graph,
revs: &mut HashSet<Revision>,
) -> Result<(), GraphError> {
revs.remove(&NULL_REVISION);
// we need to construct an iterable copy of revs to avoid itering while
// mutating
let as_vec: Vec<Revision> = revs.iter().cloned().collect();
for rev in as_vec {
Georges Racinet
rust: itering less on MissingAncestors.bases for max()...
r41866 if rev != NULL_REVISION {
remove_parents(graph, rev, revs)?;
}
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278 }
Ok(())
}
Georges Racinet
rust-dagops: roots...
r42354 /// Roots of `revs`, passed as a `HashSet`
///
/// They are returned in arbitrary order
pub fn roots<G: Graph>(
graph: &G,
revs: &HashSet<Revision>,
) -> Result<Vec<Revision>, GraphError> {
let mut roots: Vec<Revision> = Vec::new();
for rev in revs {
if graph
.parents(*rev)?
.iter()
.filter(|p| **p != NULL_REVISION)
.all(|p| !revs.contains(p))
{
roots.push(*rev);
}
}
Ok(roots)
}
Georges Racinet
rust-dagops: range of revisions...
r42353 /// Compute the topological range between two collections of revisions
///
/// This is equivalent to the revset `<roots>::<heads>`.
///
/// Currently, the given `Graph` has to implement `Clone`, which means
/// actually cloning just a reference-counted Python pointer if
/// it's passed over through `rust-cpython`. This is due to the internal
/// use of `AncestorsIterator`
///
/// # Algorithmic details
///
/// This is a two-pass swipe inspired from what `reachableroots2` from
/// `mercurial.cext.parsers` does to obtain the same results.
///
/// - first, we climb up the DAG from `heads` in topological order, keeping
/// them in the vector `heads_ancestors` vector, and adding any element of
/// `roots` we find among them to the resulting range.
/// - Then, we iterate on that recorded vector so that a revision is always
/// emitted after its parents and add all revisions whose parents are already
/// in the range to the results.
///
/// # Performance notes
///
/// The main difference with the C implementation is that
/// the latter uses a flat array with bit flags, instead of complex structures
/// like `HashSet`, making it faster in most scenarios. In theory, it's
/// possible that the present implementation could be more memory efficient
/// for very large repositories with many branches.
pub fn range(
graph: &(impl Graph + Clone),
roots: impl IntoIterator<Item = Revision>,
heads: impl IntoIterator<Item = Revision>,
) -> Result<BTreeSet<Revision>, GraphError> {
let mut range = BTreeSet::new();
let roots: HashSet<Revision> = roots.into_iter().collect();
let min_root: Revision = match roots.iter().cloned().min() {
None => {
return Ok(range);
}
Some(r) => r,
};
// Internally, AncestorsIterator currently maintains a `HashSet`
// of all seen revision, which is also what we record, albeit in an ordered
// way. There's room for improvement on this duplication.
let ait = AncestorsIterator::new(graph.clone(), heads, min_root, true)?;
let mut heads_ancestors: Vec<Revision> = Vec::new();
for revres in ait {
let rev = revres?;
if roots.contains(&rev) {
range.insert(rev);
}
heads_ancestors.push(rev);
}
for rev in heads_ancestors.into_iter().rev() {
for parent in graph.parents(rev)?.iter() {
if *parent != NULL_REVISION && range.contains(parent) {
range.insert(rev);
}
}
}
Ok(range)
}
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278 #[cfg(test)]
mod tests {
use super::*;
use crate::testing::SampleGraph;
/// Apply `retain_heads()` to the given slice and return as a sorted `Vec`
fn retain_heads_sorted(
graph: &impl Graph,
revs: &[Revision],
) -> Result<Vec<Revision>, GraphError> {
let mut revs: HashSet<Revision> = revs.iter().cloned().collect();
retain_heads(graph, &mut revs)?;
let mut as_vec: Vec<Revision> = revs.iter().cloned().collect();
as_vec.sort();
Ok(as_vec)
}
#[test]
fn test_retain_heads() -> Result<(), GraphError> {
assert_eq!(retain_heads_sorted(&SampleGraph, &[4, 5, 6])?, vec![5, 6]);
assert_eq!(
retain_heads_sorted(&SampleGraph, &[4, 1, 6, 12, 0])?,
vec![1, 6, 12]
);
assert_eq!(
retain_heads_sorted(&SampleGraph, &[1, 2, 3, 4, 5, 6, 7, 8, 9])?,
vec![3, 5, 8, 9]
);
Ok(())
}
/// Apply `heads()` to the given slice and return as a sorted `Vec`
fn heads_sorted(
graph: &impl Graph,
revs: &[Revision],
) -> Result<Vec<Revision>, GraphError> {
let heads = heads(graph, revs.iter())?;
let mut as_vec: Vec<Revision> = heads.iter().cloned().collect();
as_vec.sort();
Ok(as_vec)
}
#[test]
fn test_heads() -> Result<(), GraphError> {
assert_eq!(heads_sorted(&SampleGraph, &[4, 5, 6])?, vec![5, 6]);
assert_eq!(
heads_sorted(&SampleGraph, &[4, 1, 6, 12, 0])?,
vec![1, 6, 12]
);
assert_eq!(
heads_sorted(&SampleGraph, &[1, 2, 3, 4, 5, 6, 7, 8, 9])?,
vec![3, 5, 8, 9]
);
Ok(())
}
Georges Racinet
rust-dagops: roots...
r42354 /// Apply `roots()` and sort the result for easier comparison
fn roots_sorted(
graph: &impl Graph,
revs: &[Revision],
) -> Result<Vec<Revision>, GraphError> {
let mut as_vec = roots(graph, &revs.iter().cloned().collect())?;
as_vec.sort();
Ok(as_vec)
}
#[test]
fn test_roots() -> Result<(), GraphError> {
assert_eq!(roots_sorted(&SampleGraph, &[4, 5, 6])?, vec![4]);
assert_eq!(
roots_sorted(&SampleGraph, &[4, 1, 6, 12, 0])?,
vec![0, 4, 12]
);
assert_eq!(
roots_sorted(&SampleGraph, &[1, 2, 3, 4, 5, 6, 7, 8, 9])?,
vec![1, 8]
);
Ok(())
}
Georges Racinet
rust-dagops: range of revisions...
r42353 /// Apply `range()` and convert the result into a Vec for easier comparison
fn range_vec(
graph: impl Graph + Clone,
roots: &[Revision],
heads: &[Revision],
) -> Result<Vec<Revision>, GraphError> {
range(&graph, roots.iter().cloned(), heads.iter().cloned())
.map(|bs| bs.into_iter().collect())
}
#[test]
fn test_range() -> Result<(), GraphError> {
assert_eq!(range_vec(SampleGraph, &[0], &[4])?, vec![0, 1, 2, 4]);
assert_eq!(range_vec(SampleGraph, &[0], &[8])?, vec![]);
assert_eq!(
range_vec(SampleGraph, &[5, 6], &[10, 11, 13])?,
vec![5, 10]
);
assert_eq!(
range_vec(SampleGraph, &[5, 6], &[10, 12])?,
vec![5, 6, 9, 10, 12]
);
Ok(())
}
Georges Racinet on ishtar.racinet.fr
rust: dagop.headrevs() Rust counterparts...
r41278 }