How Does Content-Defined Chunking Reduce Duplicate Backup Data?

Content-defined chunking improves backup deduplication by choosing chunk boundaries from file contents, allowing unchanged regions to remain reusable after insertions or deletions.

Incremental backups often contain large files that are mostly unchanged between versions: virtual-disk images, mail archives, databases copied as files, project bundles, and exported media libraries. If every chunk begins at a fixed byte offset, inserting a small header near the front can shift every later boundary even though the later bytes are identical. Content-defined chunking makes the segmentation follow local byte patterns instead of absolute positions, so the backup can re-synchronize with old chunks after the changed region.

Fixed-Size Boundaries Can Turn One Small Edit Into Many New Chunks

A fixed-size chunker cuts at positions such as every 1 MiB regardless of what the bytes contain. When bytes are inserted near the beginning, the old and new streams are offset from one another, so each later fixed chunk contains a different combination of bytes even if almost all underlying content remained the same.

Content-defined chunking was developed for deduplication because content-derived cut points detect redundancy that fixed offsets can miss after local edits. The benefit is not that CDC predicts which files are similar; it gives the deduplicator a segmentation that can survive shifted positions.

If an entire file is replaced with unrelated bytes, no chunking algorithm can manufacture duplicate content. CDC helps most when versions share large unchanged byte regions but those regions have moved relative to the start of the file.

A Rolling Fingerprint Searches the Byte Stream for Local Cut Points

CDC moves a window across the input and updates a fingerprint as bytes enter and leave that window. A boundary is declared when the fingerprint satisfies a configured condition, subject to minimum and maximum chunk-size rules that prevent pathological tiny or huge chunks.

Borg's chunker uses a rolling content fingerprint so the cost of evaluating the next candidate boundary does not require hashing the whole window from scratch. Because the fingerprint depends on nearby bytes, the same local sequence can trigger the same cut even when its absolute file offset has changed.

The rolling fingerprint is therefore a boundary-finding mechanism, not the final identity of stored backup data. Treating those two hashes as interchangeable would weaken the explanation of where deduplication actually decides reuse.

Minimum, maximum, and average chunk sizes also shape the boundary search. They control how often candidate cuts are considered and how much metadata the repository must manage.

CDC Re-Synchronizes After an Edit Instead of Staying Shifted Forever

After an insertion or deletion, the rolling window initially sees different bytes and produces different chunk boundaries around the edit. Once it moves fully into a sufficiently long unchanged region, it can encounter the same local content patterns and resume cutting at positions that align with the older version.

Borg notes that content-defined boundaries can remain stable relative to unchanged content even when bytes are inserted or removed elsewhere. That resynchronization is what confines many edits to a small number of new chunks instead of invalidating the rest of the file.

Restic similarly splits files into variable-length blobs using a sliding fingerprint, so unchanged variable-length blobs can be referenced again across snapshots. The repository still needs an index to recognize those already stored blobs.

The resynchronization distance depends on the chunking parameters and the changed byte pattern, so CDC does not promise that exactly one new chunk appears for every edit. Its advantage is statistical locality: changes are less likely to shift every later boundary.

Featured

ZimaCube 2 Personal Cloud Home NAS

ZimaCube2

$799.00 - $2,499.00

Buy Now

A Strong Chunk Identifier Decides Reuse After the Boundary Is Chosen

Finding a boundary only answers where one candidate chunk ends; the repository must still decide whether the complete chunk contents already exist. That second decision uses a stronger content identifier or authenticated hash over the finished chunk and looks it up in the repository index.

Borg explicitly separates its boundary hash from the cryptographic chunk identity used as the deduplication criterion. Restic likewise references stored blobs by a strong content hash rather than treating the rolling fingerprint as proof that two chunks are identical.

This two-stage design explains the storage path clearly: rolling hash chooses candidate segmentation; content hash identifies the resulting chunk; repository lookup decides store versus reuse. Deduplication savings happen only at the last two stages, even though CDC makes those matches far more likely to survive edits.

Chunk Size and Data Transformation Set the Compute-to-Savings Boundary

Smaller average chunks isolate changes more precisely, but they increase the number of fingerprints, index entries, lookups, metadata objects, and storage references. Larger chunks lower indexing overhead but allow a small edit to invalidate a larger unit of reusable data.

FastCDC focuses on reducing the rolling-hash CPU overhead while retaining strong redundancy detection, showing that chunking itself can become a meaningful cost before any duplicated bytes are eliminated. The best parameters balance chunking work, index size, and the similarity pattern of the backup set.

Transformation before chunking can also remove the byte similarity CDC depends on. Encryption with different nonces, formats that rewrite most of a file after a small logical change, or some compression layouts can make two logically similar versions look unrelated at the byte level.

ZimaSpace's analysis of deduplication index overhead covers the other side of this trade-off: finer reuse requires more metadata and memory to track what already exists. CDC is valuable when the recovered storage exceeds that extra chunking and index cost, not merely because variable-size chunks sound more sophisticated.