Snappy Compress and Decompress

What is Snappy?

Snappy is a compression format from Google built for speed rather than small files. Databases, message queues and storage formats use it where the cost of compressing and decompressing matters more than a few extra bytes. The same algorithm travels in three containers, and this tool reads and writes all three.

• Input: pasted text up to 100 MiB, read as UTF-8, or a file up to 2 GiB. Framed and Kafka files are read in pieces and are not held whole.
• Action: Auto, Compress or Decompress. Auto is the default. It decompresses input that starts with a Snappy stream header or is named .sz or .snappy, and compresses anything else. Input that is plainly another compression format, such as gzip, is refused with a hint unless you choose Compress.
• Format: Auto, Framed, Raw block or Kafka (snappy-java). Auto compresses to the framed format. When you decompress, the stream header decides the format whatever this says, and data without a header is read as a raw block.
• Framed is the .sz framing format. It splits data into chunks of up to 64 KiB and checks each one with a CRC-32C checksum.
• Raw block is one bare Snappy block with no header or checksum, the form Parquet pages, LevelDB tables and Prometheus remote-write requests carry. It has no framing to read in pieces, so it is limited to 100 MiB in and 200 MiB out.
• Kafka (snappy-java) is the stream that Kafka's Java client writes: a 16-byte header, then compressed blocks, each with its length in front. This page writes one block per 32 KiB of input, as snappy-java does by default. Several of these streams joined end to end are read as one.
• Download name: optional. Compress adds .sz for framed output and .snappy for the other two. Decompress removes one trailing .sz or .snappy. If you upload a file and leave this blank, the upload name is used. Otherwise the download is named output.sz, output.snappy or output.bin.

Why reach for this instead of a script

Snappy data usually turns up as a few bytes copied out of a log, a Kafka message or a database dump, and the three containers look alike to a casual eye. Feeding a Kafka message to a raw decoder, or a raw Parquet page to a framed one, fails with an unhelpful error. This page reads the header and picks the right decoder, and it tells you when the bytes are really gzip, Zstandard or another format it recognizes.

Nothing is uploaded. The file is processed in your browser.

Are there alternatives?

LZ4 is the closest relative, built for the same kind of speed, and has its own tool on this site. Zstandard makes noticeably smaller files and is the better choice when you get to pick the format. gzip compresses better than Snappy but more slowly, and every system opens it.

On your own machine, python-snappy reads the raw and framed forms, and snappy-java reads the Kafka form. Hadoop writes its own framing for .snappy files, which this tool does not read.

Fun facts and more info

Snappy started inside Google under the name Zippy and was released as open source in 2011.

A raw block begins with the length of the original data, so a decoder knows how much memory it needs before it starts. A damaged or hostile block can claim gigabytes. This page checks that claim against what the data could possibly produce, and against the size limit, before it reserves anything.

The checksums in the framed format catch damaged or truncated data. They do not prove who made the file.