Part 4 showed that containers are a prerequisite for CDC on cloud object storage, collapsing per-operation costs by orders of magnitude. But the major providers still charge for every PUT, GET, and byte of egress. Can we do better? A newer generation of S3-compatible services has emerged with pricing models that eliminate or sharply reduce these costs, and caching can cut read expenses further still. This post explores both, then wraps up the series with a look at what motivated this deep dive in the first place.

The Cost Comparison Continued

The dominance of per-operation costs on major cloud providers is what makes container packing essential. But a newer generation of S3-compatible storage services has emerged with pricing models that eliminate or sharply reduce the very cost dimensions that punish CDC. Cloudflare R2 charges zero egress. Backblaze B2 offers free uploads and storage at a fraction of S3’s price. Wasabi charges no per-operation fees and no egress fees at all. The explorer below applies the same workload to these challengers.

The savings over established providers are substantial. Each challenger eliminates a different cost dimension: R2 kills egress, B2 offers free uploads and cheap storage, and Wasabi removes both operations and egress fees entirely. The explorer below puts all six providers side by side so you can compare directly.

Egress dominates the established provider bills, and the challengers that eliminate it see the largest absolute savings. Wasabi’s model is the most aggressive: with no per-operation or egress fees, the only cost is storage itself. However, Wasabi’s pricing comes with constraints. There is a 90-day minimum storage duration (deleting data sooner still incurs the full charge), a 1 TB minimum storage volume, and a fair-use egress policy that caps monthly egress at your total storage volume. For read-heavy workloads where egress significantly exceeds stored data, the “free egress” claim may not hold.

Reducing Costs through Caching

The cost explorers above model a direct path: chunks flow from the writer to storage and from storage to the reader. But production systems rarely work that way. A read-through cache between readers and the storage backend can dramatically reduce both operations costs and egress, the two cost dimensions that dominate at scale.

CDC chunks are unusually well-suited for caching. Every chunk is immutable and content-addressed: its hash is its identity. There is no invalidation problem, because a chunk’s content never changes. If chunk a7f3e9... is in the cache, it will be correct forever. And because deduplication shrinks the working set (many files share the same chunks), the effective cache hit rate is higher than it would be for opaque file-based caching. Popular files that share chunks with other popular files all benefit from the same cached data.

A key question for any cache is how much data must be stored to achieve a given hit rate. The answer depends on the access distribution. Breslau et al. showed that web request frequencies follow a Zipf distribution, where the k-th most popular item is accessed with probability proportional to 1/k^α.[37] The α parameter controls how skewed the popularity curve is. At α = 0, every item is equally popular and caching provides no advantage. As α increases, popularity concentrates: a small number of items account for a disproportionate share of requests, which is exactly the condition where caching thrives. Breslau et al. measured α values between 0.64 and 0.83 for web traffic. More recent measurements by Berger et al. on real CDN and web application traces found α values between 0.85 and 1.0, indicating that modern access patterns are even more skewed.[38]

The measure of skewness, α, can be seen by dragging the skewness slider in the visualization below. High α values represent the condition in which a high percentage of requests occur for a few very popular items (high skew), while low α values represent traffic spread more equally across all items (low skew).

In the visualization below, each bar represents an item, like a file or chunk, ranked by popularity as a measure of how frequently it is requested. The height of each bar is its overall percentage share of total requests.

That skewed distribution is exactly why caching works. If you cache only the most popular items, you can serve a disproportionate share of requests without touching the storage backend. Measured α values for web and CDN traffic typically fall between 0.64 and 1.0, but not all workloads follow a Zipf distribution, and yours may differ. Measuring α for a specific workload is feasible but out of scope for this post; see the footnote^† for pointers on how it’s done. The next visualization shows this relationship directly: given a skewness level and a target hit rate, how much unique data do you actually need to cache?

Under a Zipf distribution, the cache size needed for a target hit rate h is approximately h^1/(1-α) of the total unique data. The explorers below use α = 0.6 (below the measured range, deliberately conservative), giving a cache fraction of h^2.5. This overstates how much cache capacity is needed: with Berger’s higher α values, real caches would require less data for the same hit rate.

The relationship between hit rate and cache size is worth pausing on, because it is not immediately intuitive. A 50% hit rate means serving half of all requests from cache. Because access patterns are skewed, the most popular 18% of unique data accounts for 50% of all requests – those chunks get hit over and over. To reach a 90% hit rate, you need to also cache the moderately popular long tail, which requires about 77% of unique data. And reaching 99% means caching nearly everything (98%), because that last 9% of requests comes from rarely-accessed chunks that each contribute only a small share of traffic.

The cost impact depends heavily on the pricing model. Established cache providers charge for provisioned capacity: you pay for memory whether it is hit or not. Challenger providers charge per-request: you pay only for the operations you use, with no idle cost.

The challenger cache providers invert the cost structure. Instead of provisioning memory upfront, you pay for each cache read (hit) and each cache write (miss that populates the cache). Storage costs, if any, scale with the actual cached data volume.

All Costs Considered

The individual explorers above isolate several cost dimensions: storage provider pricing, per-operation and egress fees, cache provider models, hit rates, and their sensitivity to the Zipf access distribution. Six storage providers, nine cache options, chunk size, and container packing create a large enough configuration space that cost-based decisions are difficult to make by intuition alone. The comprehensive model below puts all of these dimensions into a single view.

The cost landscape has a few clear takeaways. First, provider choice dominates: challenger storage providers with free egress and zero per-operation fees can reduce the monthly bill by 90% or more compared to established providers at the same chunk size and container configuration. Second, caching interacts with provider choice in non-obvious ways. A CDN cache in front of an established provider offloads expensive egress and read operations, producing large absolute savings. But the same cache in front of a free-egress provider like R2 or Wasabi adds cost without offsetting much, because there was little egress cost to begin with. Third, container packing remains essential regardless of provider or cache layer. Without it, per-operation costs at small chunk sizes overwhelm every other line item. The container abstraction from Part 4 is not optional; it is a prerequisite for making CDC economically viable on any cloud object store.

Why I Care About This

This series grew out of my master’s thesis research, where I’m evaluating structure-aware chunking as a deduplication strategy for source code files on large version control platforms. Source code is a particularly interesting domain for chunking because individual files are typically small[27] and edits tend to be localized, small changes concentrated in specific functions or blocks[28]. This means even smaller chunk sizes may be appropriate since the overhead is bounded by the small file sizes involved.

If edits concentrate in specific functions and blocks, the natural extension of content-defined chunking is to define boundaries using the structure of the source code itself: functions, methods, classes, and modules. Instead of using a rolling hash window to identify chunk boundaries, you can parse the code into its syntactic units and chunk along those boundaries directly. cAST (chunking via abstract syntax tree; Zhang et al., 2025)[14] does exactly this in the context of retrieval-augmented code generation (RAG): It parses source code into an AST, recursively splitting large nodes and merging small siblings to produce chunks that align with function, class, and module boundaries. The result is semantically coherent chunks aligned to syntax nodes that improve both retrieval precision and generation quality across diverse programming languages.

My thesis asks whether aligning chunk boundaries to syntactic structures like functions, classes, and modules via AST parsing can outperform byte-level CDC for deduplicating source code across versions on large version control platforms. To understand the benefits cAST offers for source code deduplication, I’m comparing it against whole-file content-addressable storage as a baseline, modeling Git’s approach without its packfile and delta compression layers, and FastCDC v2020 from the BSW family for byte-level content-defined chunking. The comparison spans ten programming languages across hundreds of public repositories with large commit histories, over a range of target chunk sizes, building an empirical cost model for the traditional tradeoffs between system resources (CPU and memory), network, and storage. The cost model should help clarify if and when the overhead of parsing and smaller chunks outweighs the core costs, and cloud costs, to achieve better deduplication. I’m also grounding these costs in two user-facing scenarios, file view and diff view, measuring time to first useful byte (TTFUB) to understand how each deduplication strategy’s overhead translates into latency that users actually feel.

Conclusion

Every solution at one layer of abstraction creates problems at the next. Content-defined chunking solves fixed-size chunking’s fatal sensitivity to insertions and deletions by letting data determine its own boundaries. But deploying CDC at scale reveals that the chunking algorithm is only the beginning. Chunk size controls a web of interacting costs: storage efficiency, per-operation pricing, network egress, CPU overhead, and memory pressure. Containers decouple logical chunk granularity from physical object count, but at the cost of fragmentation, complex garbage collection, and a design space where container size, rewriting strategy, and GC policy all interact. Caching under Zipf access patterns can dramatically reduce read and egress costs, but the savings depend on provider pricing models that vary by orders of magnitude.

The deeper insight running through this series is that CDC’s power comes from its modularity. The chunking layer and the storage layer are cleanly separated. A chunking algorithm does not need to know whether its output will be stored as individual objects or packed into containers, cached at the edge or served from origin, priced per-operation or per-gigabyte. This separation of concerns is why the same core idea from Rabin’s 1981 fingerprinting still works in a 2025 cloud storage system with container packing, locality-preserved caching, and piggybacked GC-defragmentation. Each layer can evolve independently, and both have.

That modularity also points toward where the field goes next. Structure-aware chunking, like cAST’s use of abstract syntax trees for source code, raises an obvious question: what other domains have exploitable structure? Document formats, configuration files, database snapshots, and serialized protocol buffers all have internal structure that byte-level chunking ignores. On the performance side, VectorCDC’s SIMD acceleration shows that hardware-aware algorithm design can push throughput far beyond what scalar implementations achieve, and as instruction sets widen further, the gap will only grow. Beyond text and code, deduplication for images, video, and other binary formats remains largely unexplored territory where content-aware boundary detection could take entirely different forms.

Perhaps the most consequential open question is what role deduplication plays in the AI revolution ahead. Retrieval-augmented generation systems depend on chunking strategies that balance retrieval precision against chunk coherence. Model checkpointing, distributed training state, and inference caching all generate enormous volumes of partially redundant data. As AI workloads continue to scale, the economics of storing and transferring deduplicated data will only become more critical. The algorithm that lets data decide its own boundaries may have its most important work still ahead.

References

In Part 3, we built a deduplication pipeline, explored its cost tradeoffs, and saw where CDC is deployed today. The deduplication ratios looked great, but we left a critical question unanswered: what happens when you actually store those billions of variable-size chunks on cloud object storage? Cloud providers charge per API operation, and the answer can be shocking.

This post starts with the cost problem that naive cloud chunk storage creates, then introduces the solution: containers, grouping many small chunks into larger, fixed-size storage objects. But every solution at one layer of abstraction creates problems at the next. Containers introduce fragmentation, make garbage collection hard, and create a design space where container size, rewriting strategy, and GC policy all interact. We’ll reference the decade of research focused on how to best solve these problems along the way.

The Cloud Cost Problem

Established cloud object storage providers charge per API operation. Every PUT writes one object; every GET reads one object. If you take a textbook CDC pipeline and store each unique chunk as its own object on S3, you get one PUT per chunk on the write path and one GET per chunk on the read path. With billions of small chunks, the per-operation costs add up fast.

The explorer below models exactly this scenario: a naive architecture where every chunk is a separate object. Use the same workload assumptions from Part 3 (100M users, 1 PB total data, 1B document reads per month, 50 edits per user per month) and drag the chunk size slider to see what happens.

At small chunk sizes, operations cost orders of magnitude more than storage. At 4 KB average chunk size, PUT and GET costs run into the tens of millions of dollars per month across all three providers, while storage itself costs only a few thousand. At 1 KB it gets even worse. Smaller chunks give better deduplication ratios, but the per-operation pricing model punishes you for using them.

This is the fundamental tension: smaller chunks improve deduplication, but cloud object storage charges you per object. Storing each chunk as its own object means the number of API operations scales with the number of chunks, not the number of bytes. The solution is to decouple the two.

Reducing Costs through Containers

Most developers know “container” as a Linux, Docker, or OCI concept. In the deduplication literature, the word means something entirely different. Zhu et al. (FAST ‘08) defined the container abstraction in the Data Domain paper as a way to decouple chunk granularity from the number of objects written to storage.[16] A deduplication container is a self-describing, immutable, fixed-size storage unit, typically a few megabytes, that groups many variable-size chunks together. Instead of storing each chunk as its own object, the system packs hundreds or thousands of chunks into a single container and writes that container as one I/O operation.

A container has two sections. The metadata section contains the fingerprint (cryptographic hash) of every chunk in the container, along with each chunk’s byte offset and length within the data section. The data section holds the actual chunk bytes, often compressed. The metadata section is compact enough to be read independently, which lets the system know what a container holds without reading the full data. This separation is critical for the optimizations that make container-based deduplication practical.

The write path works as follows. Incoming unique chunks, those that pass the deduplication check and are confirmed to be new, are buffered in memory. When enough chunks have accumulated to fill a container, they are packed together (metadata header followed by chunk data), and the entire container is written as a single I/O operation: one PUT on object storage, one sequential write on disk. After the write, the chunk index is updated to map each chunk’s hash to a tuple of (container_id, byte_offset, length). The buffering amortizes the cost of a write across many chunks. At 4 KB average chunk size and 4 MB container size, a single write covers roughly 1,000 chunks.

The read path inverts this. To retrieve a chunk, the system looks up its container_id, offset, and length in the chunk index, then issues a byte-range request on that container (a range GET on S3, or a positioned read on local disk). If multiple chunks from the same container are needed, a single read can fetch them all. In practice, restoring a file often requires chunks from several containers, so the system issues a set of range reads in parallel.

Zhu et al. introduced a key optimization that exploits container structure: locality-preserved caching.[16] When the system reads a container’s metadata section to check whether it contains a particular chunk, it caches all the fingerprints from that container in memory. Because write streams have locality, consecutive chunks in the input tend to end up in the same or nearby containers, this prefetches fingerprints for upcoming deduplication lookups. Combined with a Bloom filter for quickly skipping chunks that are definitely new (called a “summary vector” in the Data Domain paper), this locality-preserved caching eliminated 99% of on-disk index lookups during deduplication. The Bloom filter handles the common case (most incoming chunks are new and can be skipped without a disk seek), while the container metadata cache handles the remaining case (chunks that might be duplicates are likely co-located with recently seen duplicates).

A note on terminology: the deduplication research literature consistently uses the term “container” for this abstraction (Zhu et al., Lillibridge et al., Xia et al., and others).[16][18][15] Backup tools like Restic and Git use “packfile” for a similar concept. Git’s packfile format serves a somewhat different purpose (it includes delta compression between objects, not just packing), so this post uses the literature’s term to avoid conflation.

The key insight of the container abstraction is that it decouples logical chunk granularity from physical object count. You can have billions of 4 KB chunks but only millions of 4 MB containers. The chunk index grows with the number of unique chunks, but the storage layer deals with far fewer, larger objects. The CDC algorithm does not need to change. The same Gear hash or Rabin fingerprint that produces variable-size chunks feeds into the same deduplication lookup. The container is purely a storage-layer concern, invisible to the chunking logic above it.

The Impact of Containers on Cost

Now that we understand the container abstraction, let’s revisit the cost picture. The explorer below shows both with and without container totals. Compare them to see the magnitude of cost savings that containers provide.

The savings are dramatic. At 4 KB average chunk size with 4 MB containers, each container holds roughly 1,000 chunks. That means PUT operations drop by a factor of 1,000, and GET operations drop similarly (assuming reasonable locality, where chunks needed for a read tend to cluster in the same containers). Storage cost is identical in both modes because the same bytes are stored either way. But operations cost goes from dominating the monthly bill to being negligible.

This is why container packing is not an optimization. It is a prerequisite for running CDC-based deduplication on cloud object storage at any meaningful scale. Without it, the per-operation pricing model of S3, GCS, and Azure Blob Storage makes fine-grained chunking economically impossible. With it, the system can use whatever chunk size gives the best deduplication ratio, because the storage layer absorbs the object-count explosion transparently.

More Containers More Problems

Containers solve the operations cost problem, but they create three new problems at the next layer of abstraction. First, fragmentation: because deduplication shares chunks across versions, the chunks needed to reconstruct any single version become scattered across containers written at different times, degrading read performance. Second, garbage collection: deleting old versions only removes references to chunks, but a container can only be freed when every chunk in it is unreferenced, so dead space accumulates. Third, design tradeoffs: container size, chunk size, rewriting aggressiveness, and GC policy all interact in ways that make tuning surprisingly difficult. Each of these has been the subject of over a decade of focused research.

The Fragmentation Problem

The setup is straightforward. Deduplication works by sharing chunks across versions. If a chunk from the current version already exists in the store, the new version simply references it rather than storing it again. This sharing is the entire point of deduplication. But it means the chunks needed to reconstruct the latest version of a file are physically scattered across containers that were written at different times, alongside chunks from unrelated files and earlier versions.

This fragmentation worsens predictably over time. When you write a file for the first time, all of its chunks are new, so they go into containers sequentially with perfect locality. Reading that file back means reading containers in order, each one packed with relevant data. The second version deduplicates most of its chunks against the first, so very few new containers are written. The chunks that are new land in fresh containers alongside other new chunks from that write cycle. So far, so good. But by the hundredth version, the latest version’s chunks are scattered across containers written during every prior cycle. Some containers hold mostly chunks from version 1. Others hold chunks from version 47. Reconstructing the current version now requires reading dozens or hundreds of containers, each containing mostly irrelevant chunks that happen to share a container with one or two chunks you need. This is read amplification: to get the bytes you want, you must read far more bytes than you need.

Lillibridge et al. (FAST ‘13) quantified this problem precisely.[30] They found that read speeds for the most recent version can degrade by orders of magnitude over a system’s lifetime as fragmentation accumulates. Each new version scatters its few new chunks across fresh containers, while the majority of its chunks (the deduplicated ones) point back to increasingly dispersed old containers.

The first mitigation came from Kaczmarczyk et al. (SYSTOR ‘12), who proposed Context-Based Rewriting (CBR).[29] The idea is to intervene at ingest time: as chunks are being written, identify those that would land in containers with poor locality (containers where most other chunks are not part of the current write stream), and selectively rewrite those duplicate chunks into new containers alongside their neighbors. The key insight is that rewriting a small fraction of duplicates can dramatically improve restore locality. CBR rewrote only about 5% of duplicate chunks and reduced the restore performance degradation from 12-55% to only 4-7%.[29] The cost is a modest reduction in deduplication ratio (since those rewritten chunks now exist in two places), but the restore improvement far outweighs the extra storage.

Lillibridge et al. (FAST ‘13) proposed container capping, which directly limits how many distinct old containers a new version is allowed to reference.[30] The intuition is that restore performance degrades with the number of containers that must be read: if the current version’s chunks are spread across hundreds of containers, restoring it requires hundreds of reads, most of which fetch containers full of irrelevant data. Container capping sets a ceiling on that container count. When the incoming data stream would reference too many distinct containers, the system selectively rewrites some duplicate chunks into fresh containers alongside their neighbors, bringing the count under the cap. This is the same rewrite-to-improve-locality idea as CBR, but the trigger is different: CBR decides per-chunk based on the locality of the target container, while capping decides globally based on the total container count for the current version. Container capping sacrifices roughly 8% of the deduplication ratio in exchange for a 2-6x restore speed improvement.[30] The same paper also proposed forward assembly, a complementary technique that optimizes memory management during restores rather than the storage layout itself, achieving 2-4x restore improvement by using the file manifest to plan cache eviction decisions ahead of time.[30]

Fu et al. (ATC ‘14) proposed History-Aware Rewriting (HAR), which exploits version history to identify fragmented chunks more precisely.[31] Rather than applying a uniform rewriting threshold, HAR improves on CBR by analyzing past versions to determine which specific chunks are causing the worst read performance and rewrites only those. This targeted approach achieves similar restore improvements to CBR and container capping while rewriting fewer chunks, preserving more of the deduplication ratio.

The most recent major advance is MFDedup by Zou et al. (FAST ‘21), which claimed to resolve the deduplication-versus-locality dilemma that had defined the previous decade of research.[32] Their key observation was that most duplicate chunks in a given version come from the immediately previous version, not from distant history. MFDedup exploits this with two techniques: NDF (Neighbor-Duplicate-Focus) indexing, which deduplicates primarily against the previous version rather than the entire history, and AVAR (Across-Version-Aware Reorganization), which rearranges chunks after ingest into a compact sequential layout optimized for restore. The result: 1.1-2.2x higher deduplication ratio and 2.6-11.6x faster restore compared to previous approaches.[32] MFDedup achieves this by recognizing that the traditional approach of deduplicating against all prior versions creates unnecessarily scattered references, when deduplicating against only the most recent version captures most of the savings while preserving far better locality.

The progression from CBR (2012) through container capping (2013) to HAR (2014) to MFDedup (2021) shows a field converging on a clear principle: ingest-time layout decisions are the primary lever for controlling fragmentation. You cannot fix fragmentation after the fact without rewriting data. The question is how much to rewrite and when. Early approaches (CBR, capping) used simple heuristics applied uniformly. Later approaches (HAR) added historical context for more targeted rewriting. MFDedup went further by rethinking the deduplication target itself, shifting from “deduplicate against everything” to “deduplicate against the most relevant prior version.” Each step reduced the tradeoff between deduplication ratio and restore locality.

Garbage Collection

Containers solve the operations cost problem but make deletion hard. In a non-deduplicated system, deleting a file or an old version frees its storage immediately. In a deduplicated system, deleting a file only removes references to chunks. A chunk can only be freed when no remaining file or version references it. And because chunks live inside containers, a container can only be freed when every chunk in it is unreferenced. A single surviving chunk, perhaps shared by one remaining version, keeps the entire container alive. As old versions are deleted over time, containers accumulate dead space: unreferenced chunks that waste storage but cannot be reclaimed because they share a container with at least one live chunk.

Logical garbage collection is the traditional approach.[33] Walk the file-level metadata (version manifests) to determine which chunks are still referenced by at least one live file or version. Mark all unreferenced chunks. Then identify containers where all chunks are unreferenced and reclaim those containers entirely. Containers with a mix of live and dead chunks present a choice: leave them as-is (wasting the dead space) or “compact” them by reading out the live chunks, packing them into new containers, and freeing the old ones. Compaction reclaims dead space but costs I/O: you must read the old containers, write new ones, and update the chunk index. At scale, the metadata walk itself is expensive because it must traverse every manifest in the system.

Douglis et al. (FAST ‘17) introduced a fundamentally different approach: physical garbage collection.[33] Instead of walking metadata top-down (enumerating all references to determine which chunks are live), physical GC walks the raw container storage bottom-up with sequential I/O. For each container, it reads the metadata section and checks which chunks are still live by consulting a compact liveness structure. This inverts the access pattern: instead of many random lookups into metadata, the system does one sequential scan of container storage. Physical GC was up to two orders of magnitude faster than logical GC for extreme workloads (very high dedup ratios or millions of small files) and 10-60% faster in common cases.[33] The key insight is that when dedup ratios are high, the number of containers is much smaller than the number of metadata references, so scanning containers is cheaper than scanning references.

The connection between GC and fragmentation is deeper than it first appears. GC rewrites containers to reclaim dead chunks. Defragmentation rewrites containers to improve chunk locality for restores. Both involve reading, filtering, and rewriting containers. They are, in a sense, the same operation applied with different goals.

Liu et al. (EuroSys ‘25) made this connection explicit with GCCDF (Garbage-Collection-Collaborative Defragmentation), which unifies GC and defragmentation into a single pass.[34] The insight is that GC already reads partially-dead containers, extracts live chunks, and writes them into new containers. That rewriting step is exactly the opportunity to also fix fragmentation: instead of packing live chunks in arbitrary order, the system reorders them by restore locality, grouping chunks that will be read together into the same new container. One pass through the data, two benefits: reclaimed dead space and improved read performance. Running GC and defragmentation as separate operations would double the I/O cost, since each would independently read, filter, and rewrite containers. Piggybacking eliminates that redundancy.

On cloud object storage, GC and fragmentation take on additional economic significance. Dead chunks in containers waste storage cost (you pay per GB for bytes that no live file references). But the bigger lever is operations cost. Fragmentation scatters the live chunks needed to restore a file across more containers, which translates to more GET requests. GC and compaction consolidate live chunks into fewer, denser containers, improving locality and reducing the number of operations required to materialize a file. However, GC and compaction are not free, and can incur substantial operations costs to perform. There is a break-even point where the monthly savings from improved locality and reclaimed storage justify these operations costs. Finding that point requires modeling your application domain (read/write rates, change ratios, data growth) alongside the pricing scheme of your cloud provider. That analysis is out of scope for this series, but ideally the cost structure outlined here gives you enough context to start one for your application.

Container Size as the Primary Lever

Part 3 identified chunk size as the single parameter that ties all costs together. On cloud object storage, container size supplants it. Container size determines the number of objects in the bucket, the number of API operations per write and read, and the degree of read amplification when materializing files. Fu et al. (FAST ‘15) systematically explored how container size, chunk size, and rewriting aggressiveness interact for on-premises storage, finding that no single configuration dominates across all workloads.[35] Duggal et al. (ATC ‘19) confirmed that the container abstraction translates to cloud storage, where operations have higher latency, GC must factor in the cost of compaction itself, and every GET has a price.[36]

Containers dramatically reduce costs on major cloud providers, but the cost story does not end there. A newer generation of S3-compatible services has emerged with pricing models that eliminate or sharply reduce per-operation and egress fees, and caching can cut read costs further still. Part 5 explores how these challenger providers and caching layers can drive costs down well beyond what containers alone achieve.

References

In Part 1, we explored why content-defined chunking exists and surveyed three algorithm families. In Part 2, we took a deep dive into FastCDC’s Gear hash, normalized chunking, and how average byte targets affect chunk distribution. In this post, we bring the pieces together to see deduplication in action, examine where CDC is used in practice today and where it is not, and explore the cost tradeoffs that shape real-world systems.

Deduplication in Action

Imagine you are building a system to store files that change over time. Each new version is mostly the same as the last, with only a small edit here or there. As we saw in Part 1, storing a complete copy of every version wastes storage on identical content. Content-defined chunking offers a way out: split each version into chunks based on content, fingerprint each chunk, and only store chunks you have not seen before. But what does that actually look like in practice? The explorer below runs FastCDC on editable text. A small edit has already been saved to show deduplication at work: most chunks are recognized as duplicates and never stored again. Try making your own edits to see how content-defined boundaries respond.

As the demo shows, even a small edit only produces a handful of new chunks while the rest are shared across versions. But how does this work under the hood?

The Deduplication Pipeline

Recall the system to store files that change over time, where the goal is to avoid writing identical content twice. Implementations vary widely, but any CDC-based deduplication system needs the same core ingredients: a way to split data into chunks, a way to fingerprint each chunk, and a way to check whether that fingerprint has been seen before. The visualization below walks through these ingredients as a simple linear pipeline, though real systems will likely optimize by reordering, parallelizing, or batching these steps.

Walking through the stages: the raw file bytes enter the pipeline and are split into variable-size chunks by the CDC algorithm (here, FastCDC). Each chunk is then fingerprinted with a cryptographic hash like BLAKE3. The system looks up each fingerprint in the chunk store. Chunks that already exist are skipped with only a reference recorded, while genuinely new chunks are written to storage and their fingerprints registered for future lookups. The result is that storing a new version of a file costs only the new chunks, not a full copy.

Each stage in the pipeline maps to just a few lines of code, but together they form a system where redundant data is identified and eliminated before it ever reaches disk or network. When a file changes, only the chunks that were actually modified produce new hashes. The rest match what is already in the store, so they are never written again.

The Core Cost Tradeoffs

Deduplication is not free. Every stage of the pipeline above consumes resources, and the central engineering challenge is deciding where to spend and where to save.[15] The core CDC costs fall into four categories, and they all interact.

The explorer below visualizes these four dimensions as you move the chunk size slider. Small chunks push CPU, memory, and metadata overhead up while improving deduplication and network efficiency. Large chunks do the reverse. The sweet spot depends on your workload.

If you experimented with the Parametric Chunking Explorer in Part 2, you saw this tradeoff firsthand: smaller average sizes produced more chunks with tighter size distributions, while larger averages produced fewer, more variable chunks. Those demos showed the statistical effect. In production, the right balance depends on your workload: the volume of duplicated content, the rate at which that content changes, and how each cost dimension (CPU, memory, network, storage) maps onto your constraints. These are the competing forces that determine whether CDC is a valuable strategy for your system, and if so, what average chunk size best balances them. That answer depends on your domain, and only you as the expert in your particular system can make that call. My hope is that the intuitions developed here help you make a more informed decision.

Where CDC Lives Today

Content-defined chunking has become infrastructure, often invisible but always essential. It shows up across three broad categories: backup and archival, file sync and distribution, and content-addressable storage.

Backup and archival tools were the earliest adopters. Restic uses Rabin fingerprints with configurable chunk sizes,[29] Borg uses Buzhash with a secret seed (preventing attackers from predicting chunk boundaries based on known content),[30] and newer tools like Kopia,[31] Duplicacy,[32] Bupstash,[33] and Tarsnap[34] all rely on CDC to deduplicate across snapshots. The pattern is the same in each: split data into content-defined chunks, fingerprint each chunk, and store only the unique ones.

File sync and software distribution use CDC to minimize transfer sizes. Riot Games rebuilt the League of Legends patcher around FastCDC, replacing an older binary-delta system and achieving a tenfold improvement in patching speeds.[27] casync, created by Lennart Poettering, applies CDC to OS and container image distribution, chunking across file boundaries so that updates to a filesystem image only transfer the chunks that actually changed.[35]

Content-addressable storage systems like IPFS use CDC to split files into variable-size blocks before distributing them across a peer-to-peer network.[28] Because chunk boundaries are determined by content rather than position, identical regions of different files naturally converge on the same chunks and the same content addresses.

When CDC Is Not the Right Choice

Not every system chooses CDC, and the cost tradeoffs help explain why. CDC optimizes for one thing above all: stable chunk boundaries across edits. That stability enables fine-grained deduplication, but it comes at a cost, and not every application prioritizes deduplication over other concerns.

Dropbox is the most prominent example. Their architecture uses fixed-size 4 MiB blocks with SHA-256 hashing, and has since the early days of the product.[23] Dropbox’s primary engineering challenge was not deduplication, it was transport: syncing files across hundreds of millions of devices as fast as possible while keeping infrastructure costs predictable.

Fixed-size blocks give Dropbox properties that CDC cannot. Block N always starts at offset N * 4 MiB, so a client can request any block without first receiving a boundary list. Upload work can be split across threads by byte offset with zero coordination, because boundaries are known before the content is read. The receiver knows when each block ends, enabling Dropbox’s streaming sync architecture where downloads begin before the upload finishes, achieving up to 2x improvement on large file sync.[23] And because every block is exactly 4 MiB (except the last), memory allocation, I/O scheduling, and storage alignment are all simple to model and predict at scale.

There is also the metadata question. CDC’s chunk index must be backed by a persistent, highly available data store once it outgrows a single machine. For Dropbox, serving hundreds of millions of users, the difference between a fixed-size block index and a variable-size CDC chunk index is not just memory; it is the size and complexity of the metadata infrastructure required to support it. Fixed-size blocks produce fewer, more predictable index entries, which simplifies that infrastructure considerably.

The tradeoff is real. The QuickSync study found that a minor edit in Dropbox can generate sync traffic 10x the size of the actual modification, because insertions shift every subsequent block boundary.[25] This is precisely the boundary-shift problem that CDC was designed to solve, as we explored in Part 1. But Dropbox chose to absorb that cost and compensate elsewhere: their Broccoli compression encoder achieves ~33% upload bandwidth savings[24], and the streaming sync architecture pipelines work so effectively that the extra bytes matter less than they otherwise would.

In short, Dropbox traded storage efficiency for transport speed and operational simplicity. Fixed-size blocks mean a predictable object count, making it straightforward to forecast API costs on cloud storage where every PUT and GET is a line item. The ability to parallelize everything without content-dependent coordination was worth more than the deduplication gains CDC would have provided.

Seafile, an open-source file sync platform, takes the opposite approach: it uses Rabin fingerprint-based CDC with ~1 MB average chunks to achieve block-level deduplication across file versions and libraries.[26] Where Dropbox chose to optimize purely for transport, Seafile shows that CDC-based sync systems can work in practice. Part 4 explores how the container abstraction makes this economically viable.

Why Cloud Storage is the Cost that Matters

The four dimensions above assume a local or self-managed storage backend where the cost of writing and reading an object is just disk I/O. Most production systems today do not work that way. They store chunks on cloud object storage (S3, GCS, or Azure Blob Storage), and cloud providers turn every one of those engineering costs into a line item on a bill.

Cloud providers charge not just per GB stored but also per API operation. Every PUT and every GET has a price. When each chunk is its own object, the number of API calls scales with the number of chunks, and that operations cost can dominate the bill entirely. The same knob that the explorer above illustrates (smaller chunks improve dedup but increase chunk count) takes on a new, financially painful dimension: more chunks means more API calls means a larger cloud bill, even if the total bytes stored are fewer.

This is the problem that Part 4: CDC in the Cloud tackles head-on. Grouping chunks into larger, fixed-size containers collapses the object count and makes CDC viable on cloud storage. But containers introduce their own challenges: fragmentation, garbage collection complexity, and restore performance degradation. Part 5 then takes a deep dive into the full cost picture, exploring how different storage providers, caching layers, and container configurations combine to determine the real monthly bill.

References

Tools & Implementations

• fastcdc-rs on GitHub

The interactive animations in this post are available for experimentation. Try modifying the input text, adjusting chunk size parameters, and watching how CDC adapts to your changes.

Part 1 introduced the deduplication problem, showed why fixed-size chunking fails, and surveyed three CDC algorithm families: Basic Sliding Window (BSW), Local Extrema, and Statistical. This post focuses on FastCDC, the most widely adopted BSW algorithm, exploring its Gear hash, normalized chunking strategy, and tunable parameters through interactive demos.

A Closer Look at BSW via FastCDC

The Basic Sliding Window family includes FastCDC, which is true to its name: it is fast. Where Rabin used polynomial arithmetic and Buzhash used cyclic shifts, FastCDC’s Gear hash strips the rolling hash down to its simplest possible form. That speed, combined with normalized chunking for tighter chunk-size distributions, has made FastCDC one of the most widely implemented CDC algorithms today, with mature libraries in Rust, Go, Python, Java, and C++. We’ll explore both the 2016[5] and 2020[6] versions in detail, using the fastcdc-rs Rust crate as our reference implementation.

The Gear Hash

At FastCDC’s core is the Gear hash, a rolling hash reduced to two operations. For each byte, you:

1. Left-shift the current hash by one bit, dropping the most significant bit
2. Add the value from the Gear table keyed by the current byte, a pre-computed 64-bit random value, to the hash

That’s it. No XOR with outgoing bytes, no polynomial division. Just shift and add.

The visualization below uses 32-bit values for compactness. The real FastCDC implementation uses 64-bit Gear table entries and a 64-bit hash accumulator, as shown in the code samples that follow. The algorithm works identically at either width – only the bit count and mask positions change. To internalize the algorithm, try advancing the animation one step at a time and observe the Gear table lookup, the rolling hash manipulation, and the binary addition at each position. Playing the animation at full speed is also useful for seeing the overall flow, but stepping through it frame by frame is the best way to build intuition.

The Gear table maps each of the 256 possible byte values to a pre-computed 64-bit random number:

1
2
3
4
5
6
7
8
// GEAR table: 256 random 64-bit values
GEAR[256] = generate_random_table()

function gear_hash(data, start, end):
    hash = 0
    for i from start to end:
        hash = (hash << 1) + GEAR[data[i]]
    return hash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// From fastcdc-rs v2020
const GEAR: [u64; 256] = [
    0x3b5d3c7d207e37dc, 0x784d68ba91123086,
    0xcd52880f882e7298, 0xecc4917415d5c696,
    // ... 252 more values
];

fn gear_hash(data: &[u8]) -> u64 {
    let mut hash: u64 = 0;
    for &byte in data {
        hash = hash.wrapping_shl(1)
                   .wrapping_add(GEAR[byte as usize]);
    }
    hash
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// GEAR table (first few values shown)
const GEAR: bigint[] = [
    0x3b5d3c7d207e37dcn, 0x784d68ba91123086n,
    0xcd52880f882e7298n, 0xecc4917415d5c696n,
    // ... 252 more values
];

function gearHash(data: Uint8Array): bigint {
    let hash = 0n;
    for (const byte of data) {
        hash = ((hash << 1n) + GEAR[byte]) & 0xFFFFFFFFFFFFFFFFn;
    }
    return hash;
}

The simplicity of this hash is the point. A single left-shift and a single addition per byte gives the Gear hash its speed advantage over Rabin (which requires polynomial division) and Buzhash (which requires XOR with both the incoming and outgoing byte). But a fast hash is only half the story. The other half is deciding when the hash value signals a chunk boundary.

Finding Chunk Boundaries

With the Gear hash updating for each byte, how do we decide where to cut?

The classic approach: check if the low N bits of the hash are zero. If we want an average chunk size of 8KB, we check if hash & 0x1FFF == 0 (the low 13 bits).

Why does this work? The Gear hash produces pseudo-random values, so the probability that any N bits are all zero is $1/2^N$. For 13 bits, that’s $1/2^{13} = 1/8192$, meaning on average, one in every 8,192 bytes triggers a boundary. The mask is the chunk size control: more bits mean larger average chunks, fewer bits mean smaller ones.

This is the heart of every BSW algorithm. The algorithm doesn’t search for patterns in the content directly. Instead, it feeds each byte through the Gear table, lets the rolling hash mix the values together, and checks whether certain bits of the result happen to be zero. The content determines the hash, the mask selects which bits to check, and a boundary is placed wherever those bits are all zero.

But basic masking has a problem, and FastCDC does something more clever: normalized chunking with dual masks.

The problem with basic masking is chunk sizes follow an exponential distribution. You get many small chunks and occasional very large ones. This hurts deduplication because small chunks increase metadata overhead, and large chunks reduce sharing opportunities.

Why exponential? Because the probability of hitting a boundary is the same at every byte position. Each byte has a $1/2^N$ chance of triggering a cut, regardless of how many bytes have already been consumed into the current chunk. Short chunks are always more likely than long ones, for the same reason that flipping heads on the first try is more likely than waiting until the tenth.

The fix is intuitive: vary the probability based on how many bytes have been consumed into the current chunk so far. If only a few bytes have been consumed, make boundaries harder to find so you don’t produce tiny chunks. Once you’ve consumed past the target average, make boundaries easier to find so chunks don’t grow too large.

FastCDC’s solution:

1. Below the average size: Use a stricter mask (more bits must be zero)
2. Above the average size: Use a looser mask (fewer bits must be zero)

FastCDC was published in two rounds: a 2016 paper that introduced normalized chunking, and a 2020 revision that added a performance optimization by processing two bytes per iteration. Let’s walk through both versions to see how the dual-mask idea translates into concrete code, and how the algorithm evolved between the two papers.

The 2016 Algorithm

This is the version illustrated above: it processes one byte at a time, shifting and adding through the Gear table exactly as we saw in the Gear Hash in Action animation, and switches between the strict and loose mask depending on how far into the current chunk it has consumed. Here’s the complete loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
function find_chunk_boundary(data, min_size, avg_size, max_size):
    // Skip to minimum size (cut-point skipping)
    position = min_size
    hash = 0

    // Calculate masks based on average size
    bits = log2(avg_size)
    mask_s = MASKS[bits + 1]  // Stricter (1 more bit)
    mask_l = MASKS[bits - 1]  // Looser  (1 fewer bit)

    // Phase 1: Strict mask until average size
    while position < avg_size AND position < data.length:
        hash = (hash << 1) + GEAR[data[position]]
        if (hash & mask_s) == 0:
            return position  // Found boundary!
        position += 1

    // Phase 2: Loose mask until maximum size
    while position < max_size AND position < data.length:
        hash = (hash << 1) + GEAR[data[position]]
        if (hash & mask_l) == 0:
            return position  // Found boundary!
        position += 1

    // Hit maximum size without finding boundary
    return min(max_size, data.length)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
// From fastcdc-rs v2016 module
pub fn cut(
    source: &[u8],
    min_size: usize,
    avg_size: usize,
    max_size: usize,
    mask_s: u64,
    mask_l: u64,
) -> usize {
    let mut remaining = source.len();
    if remaining <= min_size {
        return remaining;
    }

    let mut center = avg_size;
    if remaining > max_size {
        remaining = max_size;
    } else if remaining < center {
        center = remaining;
    }

    let mut index = min_size;
    let mut hash: u64 = 0;

    // Phase 1: strict mask until center (average size)
    while index < center {
        hash = (hash << 1).wrapping_add(GEAR[source[index] as usize]);
        if (hash & mask_s) == 0 {
            return index;
        }
        index += 1;
    }

    // Phase 2: loose mask until max_size
    while index < remaining {
        hash = (hash << 1).wrapping_add(GEAR[source[index] as usize]);
        if (hash & mask_l) == 0 {
            return index;
        }
        index += 1;
    }

    remaining
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
function findChunkBoundary(
    data: Uint8Array,
    minSize: number,
    avgSize: number,
    maxSize: number,
): number {
    if (data.length <= minSize) {
        return data.length;
    }

    const bits = Math.floor(Math.log2(avgSize));
    const maskS = MASKS[bits + 1]; // Stricter
    const maskL = MASKS[bits - 1]; // Looser

    let remaining = Math.min(data.length, maxSize);
    let center = Math.min(avgSize, remaining);
    let index = minSize;
    let hash = 0n;

    // Phase 1: strict mask until center
    while (index < center) {
        hash = ((hash << 1n) + GEAR[data[index]]) & MASK_64;
        if ((hash & maskS) === 0n) {
            return index;
        }
        index++;
    }

    // Phase 2: loose mask until max
    while (index < remaining) {
        hash = ((hash << 1n) + GEAR[data[index]]) & MASK_64;
        if ((hash & maskL) === 0n) {
            return index;
        }
        index++;
    }

    return remaining;
}

The 2016 algorithm’s one-byte sliding window is already fast, but the 2020 revision found a way to cut the number of loop iterations in half.

The 2020 Enhancement: Rolling Two Bytes

The 2020 paper introduces a simple optimization: process two adjacent bytes per step as the sliding window moves across the data. Since two consecutive single-bit shifts are equivalent to one two-bit shift, the hash updates for two adjacent bytes can be collapsed:

Instead of two separate steps:

hash = (hash << 1) + GEAR[byte1]     // step 1
hash = (hash << 1) + GEAR[byte2]     // step 2

Collapse into one:

hash = (hash << 2) + GEAR_LS[byte1]  // Left-shifted GEAR table
hash = hash + GEAR[byte2]            // Regular GEAR table

The boundary check still happens after each byte, but each check uses a different mask. After the first byte, the hash bits sit one position higher than they would in the 2016 algorithm, so a left-shifted mask (mask_s_ls) is needed to check the equivalent bits. After the second byte, the bits realign with the original positions, so the regular mask (mask_s) is used. The results are identical to processing one byte at a time.

Where does the speedup come from? Each time the sliding window advances one step, the CPU must increment the position, evaluate whether the window has reached a size limit, and branch back to process the next position. By processing two bytes per step, this bookkeeping happens half as often. The two single-bit shifts also collapse into one two-bit shift instruction. These savings are small per step, but across millions of bytes they add up.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// Pre-compute left-shifted GEAR table
GEAR_LS[i] = GEAR[i] << 1  // for all i

function find_chunk_boundary_2020(data, min_size, avg_size, max_size):
    position = min_size / 2  // Start at half (we process 2 bytes/iter)
    hash = 0

    // Phase 1: Strict mask, two bytes at a time
    while position < avg_size / 2:
        byte_pos = position * 2

        // First byte: shift by 2, add left-shifted value
        hash = (hash << 2) + GEAR_LS[data[byte_pos]]
        if (hash & mask_s_ls) == 0:
            return byte_pos

        // Second byte: add regular value
        hash = hash + GEAR[data[byte_pos + 1]]
        if (hash & mask_s) == 0:
            return byte_pos + 1

        position += 1

    // Phase 2: Loose mask (similar structure)
    // ... same pattern with mask_l

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
// From fastcdc-rs v2020 module
pub fn cut_gear(
    source: &[u8],
    min_size: usize,
    avg_size: usize,
    max_size: usize,
    mask_s: u64,
    mask_l: u64,
    mask_s_ls: u64,        // Left-shifted strict mask
    mask_l_ls: u64,        // Left-shifted loose mask
    gear: &[u64; 256],
    gear_ls: &[u64; 256],  // Left-shifted GEAR table
) -> (u64, usize) {
    let mut remaining = source.len();
    if remaining <= min_size {
        return (0, remaining);
    }

    let mut center = avg_size;
    if remaining > max_size {
        remaining = max_size;
    } else if remaining < center {
        center = remaining;
    }

    let mut index = min_size / 2;
    let mut hash: u64 = 0;

    // Phase 1: strict mask, two bytes per iteration
    while index < center / 2 {
        let a = index * 2;

        // First byte
        hash = (hash << 2).wrapping_add(gear_ls[source[a] as usize]);
        if (hash & mask_s_ls) == 0 {
            return (hash, a);
        }

        // Second byte
        hash = hash.wrapping_add(gear[source[a + 1] as usize]);
        if (hash & mask_s) == 0 {
            return (hash, a + 1);
        }

        index += 1;
    }

    // Phase 2: loose mask (same pattern)
    while index < remaining / 2 {
        let a = index * 2;
        hash = (hash << 2).wrapping_add(gear_ls[source[a] as usize]);
        if (hash & mask_l_ls) == 0 {
            return (hash, a);
        }
        hash = hash.wrapping_add(gear[source[a + 1] as usize]);
        if (hash & mask_l) == 0 {
            return (hash, a + 1);
        }
        index += 1;
    }

    (hash, remaining)
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// Pre-computed left-shifted table
const GEAR_LS: bigint[] = GEAR.map(g => (g << 1n) & MASK_64);

function findChunkBoundary2020(
    data: Uint8Array,
    minSize: number,
    avgSize: number,
    maxSize: number,
): number {
    if (data.length <= minSize) {
        return data.length;
    }

    const bits = Math.floor(Math.log2(avgSize));
    const maskS = MASKS[bits + 1];
    const maskL = MASKS[bits - 1];
    const maskSLs = maskS << 1n;  // Left-shifted masks
    const maskLLs = maskL << 1n;

    let remaining = Math.min(data.length, maxSize);
    let center = Math.min(avgSize, remaining);
    let index = Math.floor(minSize / 2);
    let hash = 0n;

    // Phase 1: strict mask, two bytes at a time
    while (index < Math.floor(center / 2)) {
        const a = index * 2;

        // First byte
        hash = ((hash << 2n) + GEAR_LS[data[a]]) & MASK_64;
        if ((hash & maskSLs) === 0n) return a;

        // Second byte
        hash = (hash + GEAR[data[a + 1]]) & MASK_64;
        if ((hash & maskS) === 0n) return a + 1;

        index++;
    }

    // Phase 2: loose mask
    while (index < Math.floor(remaining / 2)) {
        const a = index * 2;
        hash = ((hash << 2n) + GEAR_LS[data[a]]) & MASK_64;
        if ((hash & maskLLs) === 0n) return a;
        hash = (hash + GEAR[data[a + 1]]) & MASK_64;
        if ((hash & maskL) === 0n) return a + 1;
        index++;
    }

    return remaining;
}

Both versions produce the same chunk boundaries for the same input and parameters. The difference is purely mechanical: the 2020 version reaches those boundaries faster by doing two bytes of work per loop iteration. With the algorithm itself understood, the next question is practical: how do the parameters you choose actually affect the chunks that come out?

Exploring the Parameters

The target average chunk size is the primary parameter when configuring FastCDC. A smaller average means more chunks (better deduplication granularity but more metadata), while a larger average means fewer chunks (less overhead but coarser deduplication). Drag the slider below to see how FastCDC re-chunks the same text at different target sizes:

Notice how the dual-mask strategy keeps chunk sizes clustered around the target average. At small targets (16-32 bytes) you get many chunks, and the distribution chart reveals the normalized shape. At large targets (96-128 bytes) the entire text may collapse into just a few chunks.

To see why normalization matters, consider what a single mask does. Basic CDC checks the same bit pattern from minimum size all the way to maximum size. Each byte after the minimum has an independent 1/avgSize probability of triggering a boundary. Because the algorithm cuts at the first match, most chunks end early: the chance of reaching any given byte without a match drops exponentially. This produces a geometric distribution skewed toward small chunks, with a few chunks surviving long enough to reach the maximum size. The FastCDC paper addresses this with normalization levels (NC1 through NC3), which control how aggressively the two masks differ from the base probability. At NC2 (the paper’s recommended level), the strict mask is 4x harder to trigger than the single-mask baseline, suppressing early cuts below the target average, while the loose mask is 4x easier, catching chunks shortly after they pass it. The result is a tight cluster around the target rather than a skewed spread.

Compare the two approaches below. Both chunk the same 8 KB of pseudo-random bytes (generated from a fixed seed), using the same Gear hash and the same target parameters. The only difference is how the mask is applied. Random data makes the statistical properties of each algorithm clearly visible because natural language text has too much structure to reveal the distribution shapes at this scale.

The density curve beneath each chunked block view shows the distribution of chunk sizes: the horizontal axis is chunk size in bytes, the vertical axis is how likely a chunk of that size is, and the dashed line marks the target average. A tall, narrow peak means most chunks land near the same size; a long tail trailing to the right means many chunks end up much larger than the target:

Basic CDC’s single mask produces chunks that follow an exponential distribution: many small chunks and a long tail of large ones. FastCDC’s dual-mask normalization clusters chunks tightly around the target average, reducing both extremes. This narrower distribution means less wasted metadata on tiny chunks and fewer oversized chunks that dilute deduplication.

FastCDC gives us a chunking algorithm that is fast, produces well-distributed chunk sizes, and, most importantly, generates stable boundaries that survive local edits. But chunking is only the first stage of a deduplication pipeline. Once data is split into chunks, each chunk needs a cryptographic fingerprint, those fingerprints need to be indexed and looked up efficiently, and duplicate chunks need to be eliminated during storage or transmission. The choices made at each stage (hash function, index structure, storage layout) interact with the chunking layer in ways that matter for real-world performance.

In the next post, Part 3: Deduplication in Action, we’ll build on FastCDC to walk through the deduplication pipeline end to end: fingerprinting chunks, detecting duplicates, and examining the cost tradeoffs that shape how these systems perform in practice.

References

Have you ever considered what it takes to store large amounts of user content at scale? Especially content like files or source code, where multiple versions of the same document exist with only minor changes between them. It’s easy to take document storage for granted, but storing content at scale efficiently is a surprisingly nuanced problem. While there are numerous aspects to this problem space, this series focuses on one in particular: deduplication. How do you avoid storing the same unchanged bytes of a file over and over again as that file evolves? Ideally, a storage system would only store the minimum set of unique bytes. Is that even possible? This series will help answer that question, and more.

What is deduplication? At its heart, it is the separation of content that changed from content that has not, with varying levels of precision and accuracy. But how do you identify what changed? That’s where the content-defined chunking (CDC) family of algorithms offers help. These algorithms share a common goal: splitting a file into smaller chunks at byte boundaries determined by the content’s structure.

Ideally, a small edit to a file should minimize the impact to the surrounding chunks whose content has not changed, resulting in new chunks that capture the edit while preserving existing chunks. Diverse content brings unique structure, and not all content can be chunked the same way. As we will see, each CDC algorithm and its approach affords benefits and tradeoffs.

This series walks you through an overview of all CDC algorithms, grounded by their underlying 40 years of CDC research. Next, you’ll deep dive into a popular and general-purpose CDC algorithm, FastCDC, to deeply understand how it works and what makes it so fast. To better contextualize how CDC works in production systems, you’ll walk through a general deduplication pipeline. This sets the stage to begin understanding CDC’s costs and tradeoffs, what factors dominate deduplication system costs at scale, and how to navigate cloud storage decisions.

Along the way, there are several interactive visualizations provided to help build mental models for the algorithms, their constraints, and costs associated with deduplication systems. You can step through how FastCDC identifies chunk boundaries, see the impact of normalized chunking, and even see how chunk size impacts storage pressure while making edits to a file. As the discussion focuses more on costs, the visualizations allow for manipulating container capacities and cache hit rates, all contextualized within the pricing schemes of popular cloud object storage and cache providers, making it easy to see the severe impacts levers like chunk size and container size have on operating costs for deduplication systems at scale.

Motivating the Problem

Imagine you’re building a backup system. A user stores a 500MB file, then modifies a single paragraph and saves it again. In a naive system, this results in two nearly identical copies of the same file. Despite the small change of a single paragraph, the storage system grew from 500MB to 1GB. Surely we can do better.

This is the deduplication problem, and it shows up in many familiar places: cloud blob storage providers managing petabytes of user files (e.g. Amazon S3 or Azure Blob Storage), cloud file servers like Google Drive or iCloud, and software backup tools like Restic and Borg.

The simplest form of deduplication is whole-file comparison: hash the entire file, and if two files produce the same hash, store only one copy. This works well for exact duplicates, but falls apart with even the smallest edit. Change a single byte and the hash changes completely, so the system treats the original and the edited version as two entirely different files.

One fix is to reduce the granularity of comparison. Instead of hashing a file as a single unit, split it into smaller segments called chunks and hash each chunk independently. A small edit now only affects the chunks near the change, leaving the rest unchanged. Those unchanged chunks can be recognized as duplicates and stored only once. The question then becomes: how should we decide where to split?

Why Not Just Use Fixed-Size Chunks?

The naive approach to chunking is fixed-size splitting: choose a chunk size, say 4KB, and split the file at every 4KB boundary. A 1MB file becomes 256 chunks of 4KB each. This approach is conceptually simple, but is problematic if we want to prevent change amplification – the invalidation of unchanged chunks when small edits occur. Using this naive chunking strategy, let’s see what happens to unchanged chunks when a small edit occurs at the beginning of a file:

Inserting “NEW INTRO.” at the beginning of the file causes every chunk boundary to shift, invalidating all five original chunks. The result is five new chunks and zero unchanged chunks, producing a deduplication ratio of 0%. In practice, this means the entire file would need to be stored again, even though most of its content did not change. We need a chunking strategy whose boundaries are not determined by fixed byte offsets, and that offers more flexibility to identify split points that better preserve unchanged chunks.

The Core Idea: Content as the Arbiter

Instead of using fixed-length byte windows to split a file into chunks, what if we could use patterns or structure in the file’s content to identify chunk boundaries? This is the core problem a family of algorithms known as content-defined chunking (CDC) attempts to solve.

How does CDC decide where to split? The details vary across algorithms, but the core principle is the same: examine a small region of data at each position, and declare a boundary when the content at that position satisfies some condition. Different algorithms use different strategies for this. Some compute a hash of a sliding window, some look for local extrema in the byte values, and some use statistical properties of the data. What they all share is that the boundary decision, or split point, is dependent on the content itself.

Let’s revisit the same example from before, but this time we split the text at sentence boundaries. Each sentence ending (a period, exclamation mark, or question mark followed by a space) defines a chunk boundary. Because the boundary is determined by the content itself, not by a fixed byte count, inserting text at the beginning of the file does not invalidate existing unchanged chunks.

Inserting “NEW INTRO.” creates just one new chunk. The original five sentences are unchanged, so their chunks are identical to before. The result is a much higher deduplication ratio, meaning we only need to store the new chunk and can reference the existing chunks for the rest of the file.

Three Families of CDC

Origins

The story begins with Turing Award winner Michael Rabin, who introduced polynomial fingerprinting in 1981.[1] His key insight: represent a sequence of bytes as a polynomial and evaluate it at a random point to get a “fingerprint” that uniquely identifies the content with high probability. More importantly, this fingerprint could be computed incrementally as a rolling hash, making it efficient to slide across data.

For a sequence of bytes $b_0, b_1, \ldots, b_{n-1}$, the fingerprint is:

where $p$ is an irreducible polynomial over $GF(2)$.

Twenty years later, the Low-Bandwidth File System (LBFS) at MIT became the first major system to use CDC in practice.[2] LBFS slid a 48-byte window across the data and computed a Rabin fingerprint at each position. Whenever the low 13 bits of the fingerprint equaled a magic constant, it declared a chunk boundary, producing an average chunk size of about 8KB. The breakthrough was demonstrating that CDC could achieve dramatic bandwidth savings for real file workloads. Modifying a single paragraph in a large document transmitted only the changed chunk, not the entire file.

1
2
3
4
5
6
// Simplified LBFS boundary check
if ((fingerprint % 8192) == 0x78) {
    // This is a chunk boundary
    emit_chunk(start, current_position);
    start = current_position;
}

The deduplication era of 2005-2015 drove an explosion of CDC research. Many successful systems built deduplication pipelines using CDC based on advances in research that produced faster hash functions, better chunk size distributions, and new ways of finding chunk boundaries. By the mid-2010s, what had been a single technique branched into a family of algorithms with fundamentally different strategies.

A Taxonomy of CDC Algorithms

A comprehensive 2024 survey by Gregoriadis et al.[12] organizes the CDC landscape into three distinct families based on their core mechanism for finding chunk boundaries. This taxonomy clarifies a field that can otherwise feel like a confusing proliferation of acronyms.

Algorithmic Timeline

1
2
3
4
5
6
7
8
9
// Rabin fingerprint: rolling hash over GF(2)
uint64_t fp = 0;
for (size_t i = 0; i < len; i++) {
    fp ^= shift_table[window[i % w]];  // remove outgoing byte
    fp = (fp << 8) | data[i];          // shift in new byte
    if (fp & HIGH_BIT) fp ^= poly;     // reduce mod irreducible poly
    window[i % w] = data[i];
    if ((fp % D) == r) return i;       // boundary!
}

1
2
3
4
5
6
7
8
9
10
11
// Buzhash: cyclic polynomial rolling hash
uint32_t table[256];                            // random values, initialized once

uint32_t h = 0;
for (size_t i = 0; i < len; i++) {
    h = ROTATE_LEFT(h, 1);                      // cyclic shift by 1
    h ^= ROTATE_LEFT(table[window[i % w]], w);  // remove outgoing
    h ^= table[data[i]];                        // add incoming
    window[i % w] = data[i];
    if ((h % D) == r) return i;                 // boundary!
}

1
2
3
4
5
6
7
8
9
// Gear hash: feedforward, no window needed
uint64_t gear_table[256]; // random 64-bit values

uint64_t hash = 0;
for (size_t i = min_size; i < len; i++) {
    hash = (hash << 1) + gear_table[data[i]];
    if ((hash & mask) == 0)
        return i; // boundary!
}

1
2
3
4
5
6
7
8
9
10
// AE: boundary when max byte is at window's right edge
for (size_t i = min_size; i < len; i++) {
    uint8_t max_val = 0;
    size_t max_pos = 0;
    for (int j = i - w + 1; j <= i; j++) {
        if (data[j] >= max_val)
            { max_val = data[j]; max_pos = j; }
    }
    if (max_pos == i) return i; // boundary!
}

1
2
3
4
5
6
7
8
9
10
11
12
// FastCDC: Gear hash + normalized chunking
uint64_t hash = 0;
size_t i = min;
for (; i < avg && i < len; i++) {              // phase 1: strict mask
    hash = (hash << 1) + gear_table[data[i]];
    if (!(hash & mask_s)) return i;
}
for (; i < max && i < len; i++) {              // phase 2: loose mask
    hash = (hash << 1) + gear_table[data[i]];
    if (!(hash & mask_l)) return i;
}
return i;                                      // hit max chunk size

1
2
3
4
5
6
7
8
9
10
11
12
13
// RAM: skip to the max, don't scan past it
size_t i = min;
while (i < len) {
    size_t max_pos = i;
    for (size_t j = i+1; j <= i+ahead; j++)       // scan lookahead
        if (data[j] >= data[max_pos]) max_pos = j;
    if (max_pos != i) { i = max_pos; continue; }  // skip!
    bool ok = 1;
    for (size_t j = i-back; j < i; j++)           // check lookback
        if (data[j] > data[i]) { ok = 0; break; }
    if (ok) return i;                             // boundary!
    i++;
}

1
2
3
4
5
6
7
8
9
10
11
12
// MII: boundary candidates are independent of chunk size
size_t last = 0;
for (size_t i = 0; i < len; i++) {
    bool is_max = 1;
    for (int j = -W/2; j <= W/2; j++)      // large symmetric window
        if (data[i+j] > data[i]) { is_max = 0; break; }
    if (!is_max) continue;
    if (i - last >= min) {                 // filter by min chunk size
        last = i;
        emit_boundary(i);                  // boundary!
    }
}

1
2
3
4
5
6
7
8
9
// PCI: boundary when bit-population exceeds threshold
int pop_sum = 0;
for (size_t i = 0; i < len; i++) {
    pop_sum += __builtin_popcount(data[i]);       // add incoming
    if (i >= w)
        pop_sum -= __builtin_popcount(data[i-w]); // remove outgoing
    if (i >= min && pop_sum >= threshold)
        return i; // boundary!
}

1
2
3
4
5
6
7
8
9
10
11
// BFBC Phase 1: build digram frequency table
uint32_t freq[256][256] = {0};
for (size_t i = 0; i + 1 < len; i++)
    freq[data[i]][data[i+1]]++;
bool is_cut[256][256] = select_top_k(freq, k);

// Phase 2: chunk using frequent digrams as boundaries
for (size_t i = min; i < len - 1; i++) {
    if (is_cut[data[i]][data[i+1]])
        return i + 1; // boundary after digram
}

1
2
3
4
5
6
7
8
9
10
11
// VectorCDC: SIMD-accelerated local max (AVX2)
for (; i + 32 <= len; i += 32) {
    __m256i curr = _mm256_loadu_si256(data + i);
    __m256i prev = _mm256_loadu_si256(data + i - 1);
    __m256i next = _mm256_loadu_si256(data + i + 1);
    __m256i is_max = _mm256_and_si256(
        _mm256_cmpeq_epi8(curr, _mm256_max_epu8(curr, prev)),
        _mm256_cmpeq_epi8(curr, _mm256_max_epu8(curr, next)));
    uint32_t mask = _mm256_movemask_epi8(is_max);
    if (mask) return i + __builtin_ctz(mask); // first local max
}

In the next post, Part 2: A Deep Dive into FastCDC, we’ll take a closer look at the BSW family through FastCDC, an algorithm that combines Gear hashing with normalized chunking to achieve both high throughput and excellent deduplication.

References

What started as fun idea, and low-stakes desire to add a subtle wind movement background animation to my homepage, turned into an exploration of JavaScript canvas rendering, noise functions, planar waves, and the surprisingly rich parameter space of moving lines.

This post walks through the mental model behind the line field animation used as the background for this website, how it evolved from a static grid to an organic wind simulation, and learnings and reflection on creative coding through conversational iteration with LLM’s.

The Mental Model

At its core, the animation is simple: draw a field of parallel lines on a canvas, then displace each point along those lines based on mathematical functions that change over time to simulate wind.

The key insight is that lines are just sequences of points. If you can control where each point sits, you can make the line wave, ripple, or flow.

How Wind “Moves”

Wind isn’t modeled as emanating from a point source. Instead, it’s a traveling planar wave. A useful reference is ocean waves approaching a beach rather than ripples from a dropped stone. Each point’s position is projected onto the wind direction axis, and a sine wave sweeps along that axis over time:

1
  displacement = sin(position_along_wind_axis + time)

As time advances, the sine wave pattern shifts in the wind direction, creating the illusion of wind sweeping across the field. With two wind sources at different angles, the waves interfere to create complex, naturalistic patterns.

Layered Displacement

The final displacement of each point combines several layers:

1. Simplex noise — Organic, spatially-coherent randomness that scrolls in the wind direction
2. Traveling sine waves — Parallel wavefronts for each wind source
3. Per-line modifiers — Jitter (phase randomization), whisp (amplitude variation), and gust envelopes

Each layer operates at different spatial frequencies and serves a different purpose, but they all scroll or animate over time to create cohesive movement.

The Evolution: From Static to Organic

The animation started simple, just a field of lines. Each turn in the session increased the complexity by adding additional parameters and controls. Below is a timeline of how the animation evolved along with the prompt used for each step.

Stage 1: A Grid of Lines

The first version was trivially simple: parallel lines at a fixed angle, evenly spaced.

The LLM (opus 4-5) provided this initial skeleton including a canvas element, a render loop, and controls for the requested parameters.

Controls:

• Line Direction — The angle at which lines are drawn across the canvas
• Line Spacing — Distance between parallel lines (smaller = denser field)
• Line Width — Stroke thickness of each line
• Opacity — Transparency of the lines
• Line Color — The color of the lines

Stage 2: Wind and Movement

Wind is modeled as a sine wave traveling in a direction causing displacement of the points making up the field of lines, resulting in the illusion of movement. Points along the wind axis move together, creating the illusion of cohesive wind pushing through the field. Simplex noise adds organic variation—each point samples a 2D noise field, shifting perpendicular to the line direction.

Controls:

• Wind Speed — How fast the wave travels through the field
• Wind Direction — The direction the wave propagates
• Wind Size — Wavelength of the wind pattern (larger = broader, gentler curves)
• Jitter Amount — Randomizes each line’s wave phase (0 = lines move in sync, higher = lines move independently/chaotically)
• Jitter Diameter — Overall displacement magnitude (larger = points move further from origin)

Stage 3: Dual Wind Sources

Adding a second wind with independent direction and speed created interference patterns. The interaction between winds produces complex, naturalistic movement that neither wind creates alone.

New controls:

• Wind 2 Speed/Direction — A second independent wind source that combines with the first
• Wind Density — How many wave cycles fit in the viewport (higher = more turbulent appearance)

Stage 4: Depth Effect

Using the noise displacement value to modulate opacity: points that displace more appear “closer” and more opaque. This simple trick adds surprising depth to a 2D animation.

New control:

• Depth Effect — How strongly displacement affects opacity (0 = uniform opacity, 1 = maximum depth variation)

Stage 5: Whisp Effect

I left this prompt intentionally vague to see how the model would respond. First it attempted to add per-point turbulence, but this was too noisy. Eventually, we settled on this concept of “whisp”, which controls the intensity of wave amplitude on the displacement of clusters of lines.

Whisp as a control works well with jitter, but they are distinct controls. Jitter is a parameter that influences lines independently, and alters the line’s offset in relation to a wave. Whisp is a source of noise that moves slowly over the field and influences clusters of lines. As whisp noise moves over clusters of lines, it multiplies the amplitude of independent waves (i.e. wind) interacting with the same cluster of lines. This means whisp is an independent temporal noise source that moves through the field lines, independently of the two wind sources, allowing for temporary stronger wind effects.

New control:

• Whisp — Per-line variation in wind response (0 = all lines move uniformly, higher = some lines and their neighbors catch more wind than others)

Stage 6: Gusts

Wind speed sets the maximum intensity. Gust modulates how much of that maximum is active in different regions. Slow-moving spatial noise modulates each wind’s intensity. At low gust values, you see calm areas punctuated by gusts sweeping through.

New controls:

• Wind 1 Gust — Oscillation envelope for the first wind (10 = constant, 1 = mostly calm with occasional gusts)
• Wind 2 Gust — Oscillation envelope for the second wind (independent of Wind 1)

Prompting for Creative Code

What surprised me most was how well conversational iteration works for this kind of project. Each prompt built on the last, and the AI maintained context about what we’d built.

A few patterns that worked well:

Start vague, then refine: “Add a subtle animation” -> “Make it wave” -> “Add wind direction” -> “Two wind sources”

Incorporate feeling without controlling implementation: “Make it feel more whispy” would often lead to a better solution than “add turbulence to each point.”

Iterate via feedback: When the first whisp implementation felt wrong, describing why (“too choppy, no gradual movement”) helped find a better approach.

Use a sandbox: Adjusting parameters live and experimenting at the extreme ends of values is often more intuitive than reading the code alone.

The Sandbox: Playing with Parameters

I find the general recipe of build a sandbox first, then refine through iteration works surprisingly well across a broad range of problems, not just for creative coding and play like this animation exercise. The underlying principle, which echoes themes from REPL-driven development, is the faster your feedback loop, the more you can explore and sample from the solution space to find an ideal solution.

If you want to play with the sandbox I used to create this animation yourself, visit the animations sandbox and experiment. And if you build something interesting, I’d love to see it.

Future Directions

This line field animation was just a fun idea I had, but there are others I would like to explore:

Genetic Animations

What if parameters evolved over time based on fitness functions? Lines that “survive” based on aesthetic criteria, gradually evolving toward interesting configurations.

Layered Animations

Multiple animation layers with different parameters, composited with blend modes. A fast, fine-grained layer over a slow, broad layer could create rich depth.

Interactive Response

Animations that respond to mouse position or cursor movement. Wind that flows away from the cursor, or lines that orient toward it.

Graph-Based Visualizations

Instead of parallel lines, what about connected graphs? Nodes that drift with noise, edges that stretch and compress, creating organic network visualizations.

Audio-Reactive

Parameters modulated by audio input—bass driving wind speed, treble affecting jitter. The animation becomes a visualizer.

LLM’s open new dimensions

What started as an amusing simple background animation experiment quickly became an exploration of noise functions, wave interference, and the expressive power of parameterized systems.

I would never have spent the time learning and playing with traveling planar waves, simplex noise, or linear interpolation, if it weren’t for the ease by which LLM’s enable this form of play. Just as each parameter in the animation opens a dimension of variation, LLM’s open a dimension of infinite creativity and exploration.

While I contributed to GitHub’s Rails monolith, most of my work focused on building code intelligence services external to it.

This included contributing to Tree-sitter grammars and runtime, helping build a research-oriented program analysis library in Haskell called Semantic, and shipping GitHub’s first code navigation service powering jump-to-definition and find-all-references features.

Later, I helped evolve that system into zero-config, precise code navigation using Stack Graphs.

With the rise of generative AI, I contributed to Copilot Chat for the web and authored the prompt-building library behind its dynamic context logic.

Most recently, I spent the last two years working on GitHub’s code search system, Blackbird.

I’ve captured the 10 lessons that mattered most, one for each year, that I’m taking with me.

1. The core is the moat

GitHub’s platform is indispensable, but only because its core experience is stable, fast, and reliable.

People often describe “moats” in terms of features, data, or network effects. But none of that matters if your foundation is broken. A product with 1,000 features has little value if 900 are buggy or slow.

The real moat? Relentlessly nailing the core experience. Every time.

2. Build first for customers

Before GitHub’s 2018 acquisition by Microsoft, the Dear GitHub letter captured widespread frustration with the platform. Under Nat Friedman’s leadership, a “paper cuts” initiative helped rebuild trust with the open source community by fixing small but painful issues.

The lesson: Dogfooding is good, but over-relying on internal usage can lead to blind spots. You risk overfitting your product to narrow needs and overlooking real customer pain.

If you’re lucky, customers will tell you what’s broken. If you’re not, they’ll leave without saying a word.

3. Make it work. Make it scale. Make it faster.

Kent Beck’s classic advice, “make it work, make it right, make it fast”, holds true. But at scale, I’ve found another ordering works better: “Make it work, make it scale, make it faster.”

When I joined the code search team, our backfill job, responsible for keeping our 140+ million repository index fresh, was taking five days to complete. This bottleneck made experimentation risky and recovery from failure slow.

After rounds of optimization, I cut the process from 5 days to 34 hours, a 72% improvement. It instantly increased trust in our system and unlocked team velocity.

Speed builds trust. Whether it’s internal tools or user-facing products, faster wins.

4. Know your tools, especially the ones you build

In dev tools, there’s no substitute for real usage. If you’re not using your own product, you’re building on assumptions.

Great engineers don’t just use tools. They study, tweak, and master them. They learn constantly, challenge defaults, and experiment with new workflows.

Follow your curiosity. Stay uncomfortable. Keep tinkering. Remember to play. Our craft improves with care.

5. Good telemetry is priceless. Bad telemetry is noise.

If you can’t measure it, you can’t fix it. But if you measure everything, you can’t see anything.

Over-logging and dashboard bloat create fog, not clarity. The best observability isn’t about volume, it’s about relevance.

Keep dashboards lean. Prune aggressively. During every incident, ask:

• What helped us resolve this faster?
• What slowed us down?

Telemetry should evolve with your systems, or it’ll betray you when it matters most.

6. Legacy code is a historic renovation project

Legacy systems carry the business. Customers rely on them. Maintaining them is an honor, not a chore.

In software, unlike architecture, you can renovate without permits. But that takes care: knowing what to preserve and what to rework. This skill is hard to learn, but ensures systems can change and respond to business needs over time.

Leaders who overlook this work during performance reviews risk starving the systems that got them here. Refactoring legacy systems is hard, risky, and essential. It should be celebrated and rewarded.

7. Software is a team sport

No matter how skilled you are, your impact depends on how well you collaborate, within your team and across functions.

Some career-changing lessons I’ve learned:

• Deliver consistently.
• Own what you commit.
• Reflect while building, not just after.
• Learn from teammates’ strengths and share your own.
• Communicate with care, it costs others time and energy.

Growth isn’t just about skill. It’s about trust, connection, and shared momentum.

8. Value is subjective

The best tech doesn’t always win. That’s because value is rarely objective.

Influence matters. And influence starts with making the case:

• Show the business impact.
• Prove it with demos and results.
• Tell a compelling story. Repeat it.

Even then, your best ideas might not win. That’s okay. Influence is a game with uneven footing, some seats are closer to the decision-maker. Play with integrity. Once the decision is made, commit and move forward.

9. Read the research

With the explosion of generative AI, the gap between what’s published and what’s productized is wider than ever.

Amazing ideas are freely available on Arxiv. You don’t have to read everything, just skim abstracts to spot patterns.

If you go deeper, ask:

• What are the key insight(s) behind this result?
• Can I extend this to other domains?
• How does it build on or challenge prior work?
• What new research does this enable?

My two favorite tricks: use AI to summarize a paper’s bibliography and trace its intellectual lineage, and use AI to retrieve other research referencing the paper.

10. Be flexible enough to adapt. Be focused enough to matter.

One of the best pieces of advice I got at GitHub: “Some degree of fungibility is good.”

Adaptability opens doors. Specialization gives you the leverage to walk through them.

If fungibility gets you in the room, depth helps you lead once you’re there.

Final note

Working at GitHub on the world’s largest developer platform has been a privilege. I’m deeply grateful for the experience and opportunity to meaningfully improve developer tooling. As much as the problems were meaningful, the people made GitHub special. I’ll deeply miss working with all my GitHub friends.

During Nat Friedman’s departure from GitHub, he said to consider work at GitHub as “being of service to all developers.” To past, present, and future GitHub devs, thank you for serving all developers and pushing the platform forward. 🙇‍♂️

What’s next

After 10 years specializing in code intelligence, I’m excited to join Nuanced to help apply program analysis and static analysis techniques to improve code generation workflows for AI.

Effective, efficient code context is a significant, unsolved problem, especially for large, complex codebases. I’m excited to bring everything I’ve learned at GitHub to help tackle it.

For this experiment, I created three animations described below. Some technical details about the implementation follow after the animations.

Character-by-character animation

Word-by-word animation

Word-by-word with probability animation

Animating with promises

This experiment started by creating a lightweight combinator abstracting over JavaScript's Promise constructor (i.e. new Promise((resolve, reject) => ...)). My motivation was to both explore animating text streamed from an imaginary generative AI system, and learn about JavaScript promises. My goal for this collection of combinators was to provide a declarative combinator DSL, making it possible to separate the promise-based logic and management from the animation iteration.

Combinator building blocks

At the heart of this experiment is a collection of combinators that abstract promise construction, and resolving promises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Returns a promise that resolves after executing the given operation.
function promise(op) {
  return new Promise((resolve, reject) => op(resolve, reject))
}

// Returns a function that resolves a promise with the given operation.
function resolveWith(op) {
  return ((resolve) => op(resolve));
}

// Returns a function that resolves a promise with the given operation,
// allowing the operation to be called recursively.
function resolveWithFix(op) {
  return ((resolve) => op(op, resolve));
}

The resolveWithFix combinator is the key to enabling recursive operations, allowing the operation to call itself with the same parameters. This mirrors the idea behind fixed-point combinators like the Y-combinator, which allows for recursion without named functions. In this case, the fixed-point combinator enables recursion using anonymous promise-based operations instead of named functions. The resolveWithFix combinator allows recursive promise operations until the recurring operation fully resolves. In this context, the op is one of the three presented iterators. This fixed-point combinator enables stateful iteration without imperative loops.

The combinators alone are not sufficient to create the animations. We need to define how the operations will be executed and how they will interact with the DOM. There are many possible ways to implement this, but I've chosen to focus on recursive async iteration.

Recursive iterators for animation

I wanted to push promises all the way down the stack, and created iterators that use the above combinators to manage animation state and DOM updates in which every iteration is the resolution of a single promise. The simplest of the iterators is the charIterator which iterates over each character in a string, updating the opacity of each character over time. The wordIterator does the same for words, and the wordWithProbabilityIterator adds a probability value to each word, revealing it on hover. Below is the charIterator implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
function charIterator(target, input, externalOpts = {}) {
  // Initialize and close over the iterator state.
  const opts = {
    ...externalOpts,
    index: 0,
    input: input,
    queue: [],
    output: "",
    outputWithOpacity: "",
    initialOpacity: 0, // Initial opacity for characters.
    steps: externalOpts.steps ?? 20, // Number of steps for the animation. Increasing this will make the animation appear "smoother" and its duration longer.
    delay: externalOpts.delay ?? 20, // Delay applied to each recursive call in milliseconds. Increasing this will make the animation appear slower.
    status: "running",
  };
  function reset() {
    opts.index = 0;
    opts.queue = [];
    opts.outputWithOpacity = "";
    opts.status = "paused";
  }
  var domCtx = document.getElementById(target);

  const runner = function(op, next) {
    promise(
      resolveWith((next) => {
        setTimeout(() => {
          promise(
            resolveWith((next) => {
              opts.output = "";
              if (opts.index < opts.input.length) {
                const char = input[opts.index];
                opts.queue.push({ char: char, opacity: opts.initialOpacity });
              }
              return next();
            })
          ).then(() => {
            opts.queue = opts.queue.filter((current) => {
              if (current.opacity >= 100) {
                if (current.char === '\n') {
                  opts.outputWithOpacity += `<br>`;
                  return false;
                }

                opts.outputWithOpacity += `<span class="set" style="opacity: ${current.opacity}%">${current.char}</span>`;
                return false;
              }

              current.opacity += (100 / opts.steps);
              if (current.char === '\n') {
                opts.output += `<br>`;
              } else {
                opts.output += `<span style="opacity: ${current.opacity}%">${current.char}</span>`;
              }

              return true;
            });
          })
          return next();
        }, opts.delay);
      })
    ).then(() =>
      promise(
          resolveWith((next) => {
            domCtx.innerHTML = opts.outputWithOpacity + opts.output;
            opts.index++;
            return next();
          })
      )
    ).then(() => {
      // If the status is paused or canceled, we abort the recursion.
      if (opts.status !== "running") return next();

      // If the index exceeds the input string length and the queue is empty, we can abort the recursion,
      // or we can reset the state of the iterator, and loop the animation endlessly.
      if (opts.index >= opts.input.length && opts.queue.length == 0) {
        //return next(); // This is where we would abort.
        reset();
        opts.status = "running";
        domCtx.innerHTML = "";
      }

      // Otherwise, we use a Y-combinator to continue the recursion.
      op(op, next, opts)
    })
  }

  return { runner, opts, reset };
}

The charIterator is invoked using one last combinator named animate:

1
2
3
4
// Animate a target element using the provided iterator and input string.
function animate(target, input, iterator, opts) {
  return resolveWithFix(iterator(target, input, opts));
}

What I find interesting about this implementation is the side-effect of an iterator for which every iteration is an async operation. This asynchrony affords very precise control over the animation, and the promise chain used by the iterators allows decomposing the animation sequence into separate promise resolutions. The first promise resolution represents a single iteration, management of the work queue and its elements' states. The second promise resolution updates the iterator's index position, and updates the DOM. The third promise resolution applies the recursive step or aborts if the base cases are met. Although the iterator is essentially a sequential, imperative program, it is a good way to exercise asynchronous semantics and to separate concerns across distinct stages of the promise chain.

Why this matters

I'm a strong believer in curiosity-driven development and creative play. This experiment was both a challenging and fun dive into an unusual application of asynchronous programming, modeling iterator evaluation entirely through JavaScript promises. Surprisingly, the combinator-based approach to wrapping and resolving promises allowed for a declarative DSL for managing promises. Exploring the interplay between UI buttons as an asynchronous signal, and the interruption or resuming of a recursive iterator promise chain also revealed how asynchronous execution allows separating layers related to some computation into distinct stages or steps of an async workflow. I hope you found something here that sparked a new idea, or at least enjoyed playing with the animations.

To follow this post, I'm assuming you have a comfortable familiarity with the Y-combinator. If you're new to the Y-combinator, I would suggest first viewing an inspiring and thorough talk given by the late Jim Weirich. The overview he provides of how the Y-combinator works, along with the explanations about the underlying concepts from lambda calculus used by Haskell Curry to formally define the Y-combinator will give you a solid foundation for understanding the concept. However, in a nutshell, the Y-combinator allows us the ability to employ recursion using only anonymous functions.

Let's take a look at a classic example of a Y-combinator application - calculating a factorial. You'll notice that the result is stored as a local variable, but otherwise the recursion is handled entirely via anonymous functions via Ruby's lambda short-hand operator, the -> operator.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Find the factorial of 5
5 * 4 * 3 * 2 * 1 = 120

# Y-combinator solution
result = -> (builder, number) { builder.call(builder, number) }.call(
  -> (recurse, number) {
    return 1 if number == 0
    return number * recurse.call(recurse, number - 1)
  },
  5
)

result # => 120

Given that we have this example of a factorial Y-combinator, let's take a look at what inject looks like given the constraint of only using anonymous functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Ruby inject example
[*0..10].inject(&:+) # => 55

# Y-combinator solution
result = -> (builder, limit, count = 0) { builder.call(builder, limit, count) }.call(
  -> (recurse, limit, count) {
    return 0 if count > limit
    return count + recurse.call(recurse, limit, count + 1)
  },
  10
)

result # => 55

This Y-combinator gives us the correct result, but there are limitations with this solution. The first is that we are required to pass in a limit as an input parameter rather than a collection or a range with arbitrary start and stop points. This solution is also only capable of addition. Should we choose to implement inject-like functionality using multiplication instead, we would have to duplicate most of this solution. Let's make our Y-combinator solution a little more sophisticated, so that it can handle a range or collection as an input parameter, and also support any of the basic arithmetic operations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Ruby inject example
[100, 10].inject(&:*) # => 1000

# Y-combinator solution
result = -> (builder, range, operator) { builder.call(builder, range, operator) }.call(
  -> (recurse, range, operator, count = 0) {
    if range[count] == range[-1]
      return range[count].send(operator, operator == :* || operator == :/ ? 1 : 0)
    else
      return range[count].send(operator, recurse.call(recurse, range, operator, count + 1))
    end
  },
  [100, 10],
  :*
)

result # => 1000

The approach now takes in a range or collection and an operator as input parameters. In order to support a generic Y-combinator that allows for any arithmetic operation, we make use of the various arithmetic identities. Given Ruby's Smalltalk roots, we can send the operator message to each of the elements in the provided range or collection. To prove this works, I've included examples of subtraction and division below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Ruby inject example with subtraction
[100, 10].inject(&:-) # => 90

# Y-combinator solution
result = -> (builder, range, operator) { builder.call(builder, range, operator) }.call(
  -> (recurse, range, operator, count = 0) {
    if range[count] == range[-1]
      return range[count].send(operator, operator == :* || operator == :/ ? 1 : 0)
    else
      return range[count].send(operator, recurse.call(recurse, range, operator, count + 1))
    end
  },
  [100, 10],
  :-
)

result # => 90

# Ruby inject example with division
[100, 10].inject(&:/) # => 10

# Y-combinator solution
result = -> (builder, range, operator) { builder.call(builder, range, operator) }.call(
  -> (recurse, range, operator, count = 0) {
    if range[count] == range[-1]
      return range[count].send(operator, operator == :* || operator == :/ ? 1 : 0)
    else
      return range[count].send(operator, recurse.call(recurse, range, operator, count + 1))
    end
  },
  [100, 10],
  :/
)

result # => 10

Now let's take a look at Ruby's zip method. This one is a little more interesting because it involves two collections.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Ruby zip example
[*0..5].zip([*100..105]) # => [[0, 100], [1, 101], [2, 102], [3, 103], [4, 104], [5, 105]]

result = -> (builder, collection1, collection2, count = 0) { builder.call(builder, collection1, collection2, count) }.call(
  -> (recurse, collection1, collection2, count) {
    return [] if count == collection1.count
    return recurse.call(recurse, collection1, collection2, count + 1).push([collection1[collection1.count - 1 - count], collection2[collection2.count - 1 - count]])
  },
  [*0..5],
  [*100..105]
)

result # => [[0, 100], [1, 101], [2, 102], [3, 103], [4, 104], [5, 105]]

I ran into a problem initially with implementing zip as a Y-combinator. Ruby does not have a cons function similar to cons found in Lisp or Scheme. I reasoned I could create an accumulator variable and pass it along through each recursive call, but the mutation of that accumulator variable is troubling. I reasoned that the base case of the recursive function would instead return an empty array, to which the zipped tuples of both collections would be added.

Just for fun I was curious to see how the Y-combinator zip solution profiled against Ruby's zip method. The results are unsurprising. Ruby's native zip method is far better optimized than recursive lambda calls.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
require 'benchmark'

native_result = Benchmark.measure do
  10000.times do
    [*0..100].zip([*0..100])
  end
end

native_result.real # => 0.177837

y_combinator_result = Benchmark.measure do
  10000.times do
    -> (builder, collection1, collection2, count = 0) { builder.call(builder, collection1, collection2, count) }.call(
      -> (recurse, collection1, collection2, count) {
        return [] if count == collection1.count
        return recurse.call(recurse, collection1, collection2, count + 1).push([collection1[collection1.count - 1 - count], collection2[collection2.count - 1 - count]])
      },
      [*0..100],
      [*0..100]
    )
  end
end

y_combinator_result.real # => 0.61685

(native_result.real - y_combinator_result.real) / native_result.real.to_f * 100 # => -246%

Although the Y-combinator zip method is roughly 246% less performant than Ruby's native zip method, the exercise of implementing two of Ruby's enumerable methods using Y-combinators was good fun. This post was originally inspired by Tom Stuart's amazing programming with nothing, and is also available as a talk.