Simple API

compress(data) compresses and returns data as a one-shot operation.:

cctx = zstd.ZstdCompressor()
compressed = cctx.compress(b'data to compress')

The data argument can be any object that implements the buffer protocol.

Unless compression_params or dict_data are passed to the ZstdCompressor, each invocation of compress() will calculate the optimal compression parameters for the configured compression level and input data size (some parameters are fine-tuned for small input sizes).

If a compression dictionary is being used, the compression parameters determined from the first input's size will be reused for subsequent operations.

There is currently a deficiency in zstd's C APIs that makes it difficult to round trip empty inputs when write_content_size=True. Attempting this will raise a ValueError unless allow_empty=True is passed to compress().

Streaming Input API

write_to(fh) (which behaves as a context manager) allows you to stream data into a compressor.:

cctx = zstd.ZstdCompressor(level=10)
with cctx.write_to(fh) as compressor:
    compressor.write(b'chunk 0')
    compressor.write(b'chunk 1')
    ...

The argument to write_to() must have a write(data) method. As compressed data is available, write() will be called with the compressed data as its argument. Many common Python types implement write(), including open file handles and io.BytesIO.

write_to() returns an object representing a streaming compressor instance. It must be used as a context manager. That object's write(data) method is used to feed data into the compressor.

A flush() method can be called to evict whatever data remains within the compressor's internal state into the output object. This may result in 0 or more write() calls to the output object.

Both write() and flush() return the number of bytes written to the object's write(). In many cases, small inputs do not accumulate enough data to cause a write and write() will return 0.

If the size of the data being fed to this streaming compressor is known, you can declare it before compression begins:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh, size=data_len) as compressor:
    compressor.write(chunk0)
    compressor.write(chunk1)
    ...

Declaring the size of the source data allows compression parameters to be tuned. And if write_content_size is used, it also results in the content size being written into the frame header of the output data.

The size of chunks being write() to the destination can be specified:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh, write_size=32768) as compressor:
    ...

To see how much memory is being used by the streaming compressor:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh) as compressor:
    ...
    byte_size = compressor.memory_size()

Streaming Output API

read_from(reader) provides a mechanism to stream data out of a compressor as an iterator of data chunks.:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh):
     # Do something with emitted data.

read_from() accepts an object that has a read(size) method or conforms to the buffer protocol. (bytes and memoryview are 2 common types that provide the buffer protocol.)

Uncompressed data is fetched from the source either by calling read(size) or by fetching a slice of data from the object directly (in the case where the buffer protocol is being used). The returned iterator consists of chunks of compressed data.

If reading from the source via read(), read() will be called until it raises or returns an empty bytes (b''). It is perfectly valid for the source to deliver fewer bytes than were what requested by read(size).

Like write_to(), read_from() also accepts a size argument declaring the size of the input stream:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh, size=some_int):
    pass

You can also control the size that data is read() from the source and the ideal size of output chunks:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh, read_size=16384, write_size=8192):
    pass

Unlike write_to(), read_from() does not give direct control over the sizes of chunks fed into the compressor. Instead, chunk sizes will be whatever the object being read from delivers. These will often be of a uniform size.

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data between 2 streams while compressing it.:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh)

For example, say you wish to compress a file:

cctx = zstd.ZstdCompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    cctx.copy_stream(ifh, ofh)

It is also possible to declare the size of the source stream:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, size=len_of_input)

You can also specify how large the chunks that are read() and write() from and to the streams:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, read_size=32768, write_size=16384)

The stream copier returns a 2-tuple of bytes read and written:

cctx = zstd.ZstdCompressor()
read_count, write_count = cctx.copy_stream(ifh, ofh)

Compressor API

compressobj() returns an object that exposes compress(data) and flush() methods. Each returns compressed data or an empty bytes.

The purpose of compressobj() is to provide an API-compatible interface with zlib.compressobj and bz2.BZ2Compressor. This allows callers to swap in different compressor objects while using the same API.

flush() accepts an optional argument indicating how to end the stream. zstd.COMPRESSOBJ_FLUSH_FINISH (the default) ends the compression stream. Once this type of flush is performed, compress() and flush() can no longer be called. This type of flush must be called to end the compression context. If not called, returned data may be incomplete.

A zstd.COMPRESSOBJ_FLUSH_BLOCK argument to flush() will flush a zstd block. Flushes of this type can be performed multiple times. The next call to compress() will begin a new zstd block.

Here is how this API should be used:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj()
data = cobj.compress(b'raw input 0')
data = cobj.compress(b'raw input 1')
data = cobj.flush()

Or to flush blocks:

cctx.zstd.ZstdCompressor()
cobj = cctx.compressobj()
data = cobj.compress(b'chunk in first block')
data = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)
data = cobj.compress(b'chunk in second block')
data = cobj.flush()

For best performance results, keep input chunks under 256KB. This avoids extra allocations for a large output object.

It is possible to declare the input size of the data that will be fed into the compressor:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj(size=6)
data = cobj.compress(b'foobar')
data = cobj.flush()

Batch Compression API

(Experimental. Not yet supported in CFFI bindings.)

multi_compress_to_buffer(data, [threads=0]) performs compression of multiple inputs as a single operation.

Data to be compressed can be passed as a BufferWithSegmentsCollection, a BufferWithSegments, or a list containing byte like objects. Each element of the container will be compressed individually using the configured parameters on the ZstdCompressor instance.

The threads argument controls how many threads to use for compression. The default is 0 which means to use a single thread. Negative values use the number of logical CPUs in the machine.

The function returns a BufferWithSegmentsCollection. This type represents N discrete memory allocations, eaching holding 1 or more compressed frames.

Output data is written to shared memory buffers. This means that unlike regular Python objects, a reference to any object within the collection keeps the shared buffer and therefore memory backing it alive. This can have undesirable effects on process memory usage.

The API and behavior of this function is experimental and will likely change. Known deficiencies include:

• If asked to use multiple threads, it will always spawn that many threads, even if the input is too small to use them. It should automatically lower the thread count when the extra threads would just add overhead.
• The buffer allocation strategy is fixed. There is room to make it dynamic, perhaps even to allow one output buffer per input, facilitating a variation of the API to return a list without the adverse effects of shared memory buffers.

Simple API

decompress(data) can be used to decompress an entire compressed zstd frame in a single operation.:

dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(data)

By default, decompress(data) will only work on data written with the content size encoded in its header. This can be achieved by creating a ZstdCompressor with write_content_size=True. If compressed data without an embedded content size is seen, zstd.ZstdError will be raised.

If the compressed data doesn't have its content size embedded within it, decompression can be attempted by specifying the max_output_size argument.:

dctx = zstd.ZstdDecompressor()
uncompressed = dctx.decompress(data, max_output_size=1048576)

Ideally, max_output_size will be identical to the decompressed output size.

If max_output_size is too small to hold the decompressed data, zstd.ZstdError will be raised.

If max_output_size is larger than the decompressed data, the allocated output buffer will be resized to only use the space required.

Please note that an allocation of the requested max_output_size will be performed every time the method is called. Setting to a very large value could result in a lot of work for the memory allocator and may result in MemoryError being raised if the allocation fails.

If the exact size of decompressed data is unknown, it is strongly recommended to use a streaming API.

Streaming Input API

write_to(fh) can be used to incrementally send compressed data to a decompressor.:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh) as decompressor:
    decompressor.write(compressed_data)

This behaves similarly to zstd.ZstdCompressor: compressed data is written to the decompressor by calling write(data) and decompressed output is written to the output object by calling its write(data) method.

Calls to write() will return the number of bytes written to the output object. Not all inputs will result in bytes being written, so return values of 0 are possible.

The size of chunks being write() to the destination can be specified:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh, write_size=16384) as decompressor:
    pass

You can see how much memory is being used by the decompressor:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh) as decompressor:
    byte_size = decompressor.memory_size()

Streaming Output API

read_from(fh) provides a mechanism to stream decompressed data out of a compressed source as an iterator of data chunks.:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh):
    # Do something with original data.

read_from() accepts a) an object with a read(size) method that will return compressed bytes b) an object conforming to the buffer protocol that can expose its data as a contiguous range of bytes. The bytes and memoryview types expose this buffer protocol.

read_from() returns an iterator whose elements are chunks of the decompressed data.

The size of requested read() from the source can be specified:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh, read_size=16384):
    pass

It is also possible to skip leading bytes in the input data:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh, skip_bytes=1):
    pass

Skipping leading bytes is useful if the source data contains extra header data but you want to avoid the overhead of making a buffer copy or allocating a new memoryview object in order to decompress the data.

Similarly to ZstdCompressor.read_from(), the consumer of the iterator controls when data is decompressed. If the iterator isn't consumed, decompression is put on hold.

When read_from() is passed an object conforming to the buffer protocol, the behavior may seem similar to what occurs when the simple decompression API is used. However, this API works when the decompressed size is unknown. Furthermore, if feeding large inputs, the decompressor will work in chunks instead of performing a single operation.

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data across 2 streams while performing decompression.:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh)

e.g. to decompress a file to another file:

dctx = zstd.ZstdDecompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    dctx.copy_stream(ifh, ofh)

The size of chunks being read() and write() from and to the streams can be specified:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh, read_size=8192, write_size=16384)

Decompressor API

decompressobj() returns an object that exposes a decompress(data) methods. Compressed data chunks are fed into decompress(data) and uncompressed output (or an empty bytes) is returned. Output from subsequent calls needs to be concatenated to reassemble the full decompressed byte sequence.

The purpose of decompressobj() is to provide an API-compatible interface with zlib.decompressobj and bz2.BZ2Decompressor. This allows callers to swap in different decompressor objects while using the same API.

Each object is single use: once an input frame is decoded, decompress() can no longer be called.

Here is how this API should be used:

dctx = zstd.ZstdDeompressor()
dobj = cctx.decompressobj()
data = dobj.decompress(compressed_chunk_0)
data = dobj.decompress(compressed_chunk_1)

Batch Decompression API

(Experimental. Not yet supported in CFFI bindings.)

multi_decompress_to_buffer() performs decompression of multiple frames as a single operation and returns a BufferWithSegmentsCollection containing decompressed data for all inputs.

Compressed frames can be passed to the function as a BufferWithSegments, a BufferWithSegmentsCollection, or as a list containing objects that conform to the buffer protocol. For best performance, pass a BufferWithSegmentsCollection or a BufferWithSegments, as minimal input validation will be done for that type. If calling from Python (as opposed to C), constructing one of these instances may add overhead cancelling out the performance overhead of validation for list inputs.

The decompressed size of each frame must be discoverable. It can either be embedded within the zstd frame (write_content_size=True argument to ZstdCompressor) or passed in via the decompressed_sizes argument.

The decompressed_sizes argument is an object conforming to the buffer protocol which holds an array of 64-bit unsigned integers in the machine's native format defining the decompressed sizes of each frame. If this argument is passed, it avoids having to scan each frame for its decompressed size. This frame scanning can add noticeable overhead in some scenarios.

The threads argument controls the number of threads to use to perform decompression operations. The default (0) or the value 1 means to use a single thread. Negative values use the number of logical CPUs in the machine.

This function is logically equivalent to performing dctx.decompress() on each input frame and returning the result.

This function exists to perform decompression on multiple frames as fast as possible by having as little overhead as possible. Since decompression is performed as a single operation and since the decompressed output is stored in a single buffer, extra memory allocations, Python objects, and Python function calls are avoided. This is ideal for scenarios where callers need to access decompressed data for multiple frames.

Currently, the implementation always spawns multiple threads when requested, even if the amount of work to do is small. In the future, it will be smarter about avoiding threads and their associated overhead when the amount of work to do is small.

Content-Only Dictionary Chain Decompression

decompress_content_dict_chain(frames) performs decompression of a list of zstd frames produced using chained content-only dictionary compression. Such a list of frames is produced by compressing discrete inputs where each non-initial input is compressed with a content-only dictionary consisting of the content of the previous input.

For example, say you have the following inputs:

inputs = [b'input 1', b'input 2', b'input 3']

The zstd frame chain consists of:

1. b'input 1' compressed in standalone/discrete mode
2. b'input 2' compressed using b'input 1' as a content-only dictionary
3. b'input 3' compressed using b'input 2' as a content-only dictionary

Each zstd frame must have the content size written.

The following Python code can be used to produce a content-only dictionary chain:

def make_chain(inputs):
    frames = []

    # First frame is compressed in standalone/discrete mode.
    zctx = zstd.ZstdCompressor(write_content_size=True)
    frames.append(zctx.compress(inputs[0]))

    # Subsequent frames use the previous fulltext as a content-only dictionary
    for i, raw in enumerate(inputs[1:]):
        dict_data = zstd.ZstdCompressionDict(inputs[i])
        zctx = zstd.ZstdCompressor(write_content_size=True, dict_data=dict_data)
        frames.append(zctx.compress(raw))

    return frames

decompress_content_dict_chain() returns the uncompressed data of the last element in the input chain.

It is possible to implement content-only dictionary chain decompression on top of other Python APIs. However, this function will likely be significantly faster, especially for long input chains, as it avoids the overhead of instantiating and passing around intermediate objects between C and Python.

Cover Dictionaries

An alternate dictionary training mechanism named cover is also available. More details about this training mechanism are available in the paper Effective Construction of Relative Lempel-Ziv Dictionaries (authors: Liao, Petri, Moffat, Wirth).

To use this mechanism, use zstd.train_cover_dictionary() instead of zstd.train_dictionary(). The function behaves nearly the same except its arguments are different and the returned dictionary will contain k and d attributes reflecting the parameters to the cover algorithm.

The segment and dmer size parameters to the cover algorithm can either be specified manually or you can ask train_cover_dictionary() to try multiple values and pick the best one, where best means the smallest compressed data size.

In manual mode, the k and d arguments must be specified or a ZstdError will be raised.

In automatic mode (triggered by specifying optimize=True), k and d are optional. If a value isn't specified, then default values for both are tested. The steps argument can control the number of steps through k values. The level argument defines the compression level that will be used when testing the compressed size. And threads can specify the number of threads to use for concurrent operation.

This function takes the following arguments:

dict_size

Target size in bytes of the dictionary to generate.

samples

A list of bytes holding samples the dictionary will be trained from.

k

Parameter to cover algorithm defining the segment size. A reasonable range is [16, 2048+].

d

Parameter to cover algorithm defining the dmer size. A reasonable range is [6, 16]. d must be less than or equal to k.

dict_id

Integer dictionary ID for the produced dictionary. Default is 0, which uses a random value.

optimize

When true, test dictionary generation with multiple parameters.

level

Integer target compression level when testing compression with optimize=True. Default is 1.

steps

Number of steps through k values to perform when optimize=True. Default is 32.

threads

Number of threads to use when optimize=True. Default is 0, which means to use a single thread. A negative value can be specified to use as many threads as there are detected logical CPUs.

notifications

Controls writing of informational messages to stderr. See the documentation for train_dictionary() for more.

estimate_compression_context_size(CompressionParameters)

Given a CompressionParameters struct, estimate the memory size required to perform compression.

estimate_decompression_context_size()

Estimate the memory size requirements for a decompressor instance.

BufferWithSegments

The BufferWithSegments type represents a memory buffer containing N discrete items of known lengths (segments). It is essentially a fixed size memory address and an array of 2-tuples of (offset, length) 64-bit unsigned native endian integers defining the byte offset and length of each segment within the buffer.

Instances behave like containers.

len() returns the number of segments within the instance.

o[index] or __getitem__ obtains a BufferSegment representing an individual segment within the backing buffer. That returned object references (not copies) memory. This means that iterating all objects doesn't copy data within the buffer.

The .size attribute contains the total size in bytes of the backing buffer.

Instances conform to the buffer protocol. So a reference to the backing bytes can be obtained via memoryview(o). A copy of the backing bytes can also be obtained via .tobytes().

The .segments attribute exposes the array of (offset, length) for segments within the buffer. It is a BufferSegments type.

BufferSegment

The BufferSegment type represents a segment within a BufferWithSegments. It is essentially a reference to N bytes within a BufferWithSegments.

len() returns the length of the segment in bytes.

.offset contains the byte offset of this segment within its parent BufferWithSegments instance.

The object conforms to the buffer protocol. .tobytes() can be called to obtain a bytes instance with a copy of the backing bytes.

BufferSegments

This type represents an array of (offset, length) integers defining segments within a BufferWithSegments.

The array members are 64-bit unsigned integers using host/native bit order.

Instances conform to the buffer protocol.

BufferWithSegmentsCollection

The BufferWithSegmentsCollection type represents a virtual spanning view of multiple BufferWithSegments instances.

Instances are constructed from 1 or more BufferWithSegments instances. The resulting object behaves like an ordered sequence whose members are the segments within each BufferWithSegments.

len() returns the number of segments within all BufferWithSegments instances.

o[index] and __getitem__(index) return the BufferSegment at that offset as if all BufferWithSegments instances were a single entity.

If the object is composed of 2 BufferWithSegments instances with the first having 2 segments and the second have 3 segments, then b[0] and b[1] access segments in the first object and b[2], b[3], and b[4] access segments from the second.