Skip to main content

Chunking

On the last page you put a small invoice into INVOICES and got it back. The bytes were small enough to feel atomic: one put, one get, done. But NATS messages have a size limit, and an invoice PDF can be far larger than any single message. So how did a file land in a message store at all?

It did not land in one message; the store split it. This page is about that split: how an object becomes chunks, how get puts them back together, and what happens when a put fails partway through.

An object is split into chunks​

A chunk is one message holding a slice of an object. When you put an object, the store doesn't try to fit the whole file in a single message. It reads the bytes in order and cuts them at a fixed boundary called the chunk size, writing each slice as its own message. A 3 MB invoice becomes a sequence of chunk messages, not one giant one.

The default chunk size is 128 KB. You did nothing to enable splitting on the last page: your small invoice fit within a single chunk, well under that 128 KB boundary, so there was nothing to split. A larger file crosses the boundary and produces several chunks.

Put a 3 MB invoice and read the chunk count back:

#!/bin/bash
# Put a large invoice into the INVOICES bucket and watch it get split into
# chunks. The object is `invoice-ord_9x3m.pdf`, a 3 MB PDF for one Acme
# order. You do nothing special to enable chunking: put always splits the
# bytes at the chunk size (128 KB by default) and stores each slice as one
# message.
#
# `order-svc` would run this after a large multi-page invoice is generated.
nats object put INVOICES invoice-ord_9x3m.pdf

# Read the object's metadata. The `Chunks` field is the count of chunk
# messages the bytes were split into; `Size` is the reassembled byte size;
# `Digest` is the SHA-256 the server computed while storing.
#
# At the 128 KB default a 3 MB file lands in roughly 24 chunks.
nats object info INVOICES invoice-ord_9x3m.pdf

# Get it back. The server streams the chunks in order, reassembles them,
# and verifies the SHA-256 against the stored digest before handing you a
# single file. You never reassemble anything yourself.
nats object get INVOICES invoice-ord_9x3m.pdf --output ./invoice-ord_9x3m.pdf

This stores invoice-ord_9x3m.pdf, the large invoice for one Acme order. The nats object info output now carries a Chunks field: the count of chunk messages the bytes were split into. At the 128 KB default, a 3 MB file lands in roughly 24 chunks.

You never ask for chunks and you never see them as separate things after the fact. They're an implementation detail of how the store fits a file into a message log. The object you put and the object you get are the same bytes; the chunking happens in between.

Get reassembles and verifies​

A get is the split run backwards. The store reads the object's chunks in order, concatenates them, and hands you the reassembled bytes, exactly the file you put. You don't reassemble anything yourself; the convenience forms (get-to-bytes, get-to-file, get-to-stream) all give you a whole object.

While it reassembles, the store does one more thing: it recomputes the SHA-256 digest over the bytes and compares it to the digest the store recorded during the put. (You met the digest on the last page: it's the integrity hash put computes as it stores.) If the two digests match, the bytes are intact and you get them. If they don't match (a chunk is missing or corrupted), the get fails instead of handing you a truncated file.

That verification is why get is safe even though the bytes traveled as many separate messages. The digest is computed over the whole object, so a single missing or reordered chunk changes it and the get refuses to return.

The put/get flow looks like this: a put sends the chunks out, then a metadata message; a get reads the metadata in, then the chunks, then runs the digest check.

The metadata message that follows the chunks carries the object's name, size, digest, and chunk count. We define what else it holds (descriptions, headers, links) on the next page. For now it's enough to know the chunks come first and the metadata closes the put.

A failed put leaves no half-objects​

Chunks publish one after another, so a put isn't instantaneous. A network drop or a crashed client can stop a put after some chunks have landed but before the rest do. What happens to the chunks that made it?

They're purged. A put that fails mid-stream removes its partial chunks before it returns the error, so a failure leaves the store as it was, not half-written. There's no leftover sequence of orphan chunks for a later get to stumble on.

This works because each put gets a fresh NUID: a unique identifier generated for that put alone, separate from the object's name. The chunks of one put are tagged with this fresh identity. When you put the same object name twice (a corrected invoice over a draft), the second put's chunks never overlap the first's. The store writes the new chunks under the new identity, points the object's metadata at them, and the old chunks fall away. A re-put replaces the object cleanly rather than merging new bytes into the old ones.

So two failure modes don't happen. A put that fails partway doesn't leave a broken object behind, and a re-put doesn't splice new bytes into old ones. Either you get the whole object you put, or the get returns an error.

Choosing a chunk size​

The chunk size is configurable. You can set it per put, and it changes how many messages the object becomes:

#!/bin/bash
# Set the chunk size on a put and watch the chunk count change. The same
# 3 MB `invoice-ord_9x3m.pdf` splits into more or fewer messages depending
# on where you draw the split boundary.
#
# A small chunk size makes many messages. Here 64 KB roughly doubles the
# chunk count versus the 128 KB default — more per-message overhead.
nats object put INVOICES invoice-ord_9x3m.pdf --chunk-size 64KB
nats object info INVOICES invoice-ord_9x3m.pdf

# A large chunk size makes few messages — until it exceeds the backing
# stream's max message size, which rejects the put. The chunk size is
# clamped to that stream limit, so pushing it too high fails the store
# rather than silently splitting differently.
#
# The full set of chunk-size options is documented in
# [Reference](/reference/). We only need the behavior here.
nats object put INVOICES invoice-ord_9x3m.pdf --chunk-size 1MB
nats object info INVOICES invoice-ord_9x3m.pdf

The same 3 MB invoice splits into more messages at a smaller chunk size and fewer at a larger one. The default of 128 KB is a reasonable middle for most files, and most of the time you leave it alone.

The full set of chunk-size options is documented in Reference. We only need the behavior here.

Pitfalls​

Two traps show up once objects get large. Each is scoped to this page: the chunk size, and the integrity check on get.

A chunk size at the extremes causes problems. Set it too small and a single file becomes thousands of tiny messages. Each chunk is a NATS message that carries its own protocol framing (headers and subject routing on top of the slice of bytes), so very small chunks waste storage on per-message overhead and slow puts and gets down. Set it too large and the chunk exceeds the backing stream's maximum message size, and the put fails outright: the store rejects an oversized value rather than quietly splitting some other way. Don't tune the chunk size to chase a benchmark; the 128 KB default fits almost every file. If you must change it, stay well inside the stream's max message size, covered on Shaping the stream. Don't push the chunk size up to make "fewer messages" your goal; past the stream limit it stops storing anything.

Always check the get result before you use the bytes. A failed get won't hand you a truncated file, so verify success first. Because chunks publish asynchronously, an unchecked failure mid-put can leave an object that fails its digest check on the way back out. A get that hits a missing chunk or a digest mismatch errors instead of returning. So check the get result, and re-put from the source on failure rather than shipping a partial invoice. Don't assume a put "worked" without confirming a get reassembles and verifies it.

Guard the get on its outcome, and re-put on failure:

#!/bin/bash
# Always check the get result before you trust the bytes. A get reassembles
# the chunks and verifies the SHA-256 against the stored digest; if the
# stored object is incomplete (a put that failed mid-stream, or a chunk
# that never landed) the get errors instead of handing you a truncated file.
#
# `nats object get` exits non-zero on a failed reassembly or digest
# mismatch, so guard the next step on its exit code.
if nats object get INVOICES invoice-ord_9x3m.pdf --output ./invoice-ord_9x3m.pdf; then
  echo "invoice retrieved and digest verified"
else
  # The bytes are not safe to use. Re-put from the source and retry rather
  # than shipping a partial invoice.
  echo "get failed: incomplete object or digest mismatch, re-put required" >&2
  exit 1
fi

Where you are​

You now have:

  • A large invoice-ord_9x3m.pdf stored across multiple chunks in INVOICES, with a Chunks count you can read.
  • A mental model of put as split-then-store and get as reassemble-then-verify.
  • The two guarantees that make that safe: a failed put purges its partial chunks, and a re-put under a fresh identity never overlaps old bytes.

The chunks carry the data. The metadata message that closes each put carries everything about the object, which the next page covers.

What's next​

The metadata message named the object, its size, and its digest. It can carry more: a human-readable description, HTTP-style headers, a free-form key/value map, and links from one object to another.

Continue to Metadata and links.

See also​