Snappy Compress and Decompress

What is Snappy?

Snappy is a compression format from Google built for speed rather than small files. Databases, message queues and storage formats use it where the cost of compressing and decompressing matters more than a few extra bytes.

The same algorithm travels in three containers, and this tool reads and writes all three.

Snappy options

Input

Pasted text up to 100 MiB, read as UTF-8, or a file up to 2 GiB. Framed and Kafka files are read in pieces and are not held whole.

Action

Auto (the default), Compress or Decompress. Auto decompresses input that starts with a Snappy stream header or is named .sz or .snappy, and compresses anything else. Input that is plainly another format, such as gzip, is refused with a hint unless you choose Compress.

Format

Auto, Framed, Raw block or Kafka (snappy-java). Auto compresses to the framed format. When you decompress, the stream header decides the format whatever this says, and data without a header is read as a raw block.

Download name

Optional. Compress adds .sz for framed output and .snappy for the other two. Decompress removes one trailing .sz or .snappy. Left blank, an upload keeps its own name and pasted text downloads as output.sz, output.snappy or output.bin.

The three containers differ like this:

Framed

The .sz framing format. It splits data into chunks of up to 64 KiB and checks each one with a CRC-32C checksum.

Raw block

One bare Snappy block with no header or checksum, the form Parquet pages, LevelDB tables and Prometheus remote-write requests carry. It cannot be read in pieces, so it is limited to 100 MiB in and 200 MiB out.

Kafka (snappy-java)

The stream Kafka's Java client writes, a 16-byte header followed by compressed blocks, each with its length in front. This page writes one block per 32 KiB of input, as snappy-java does by default, and reads several streams joined end to end as one.

Why decode Snappy here instead of a script

Snappy data usually turns up as a few bytes copied out of a log, a Kafka message or a database dump, and the three containers look alike to a casual eye. Feeding a Kafka message to a raw decoder, or a raw Parquet page to a framed one, fails with an unhelpful error.

This page reads the header and picks the right decoder, and it tells you when the bytes are really gzip, Zstandard or another format it recognizes.

Snappy alternatives

LZ4 is the closest relative, built for the same kind of speed. Zstandard makes noticeably smaller files and is the better choice when you get to pick the format. gzip compresses better than Snappy but more slowly, and every system opens it.

On your own machine, python-snappy reads the raw and framed forms, and snappy-java reads the Kafka form. Hadoop writes its own framing for .snappy files, which this tool does not read.

Things to know about Snappy

Snappy started inside Google under the name Zippy and was released as open source in 2011.

A raw block begins with the length of the original data, so a decoder knows how much memory it needs before it starts. A damaged or hostile block can claim gigabytes, so this page checks that claim against what the data could produce, and against the size limit, before it reserves anything.