ElectronicZoologyfield notes from the garage
Memory • ESP32

Compression
The Tardis Treatment

Board: Any ESP32
Library: miniz
PSRAM required: No
ESP32 Memory Series - Step 8 of 9
⚠ Untested
Part of the ESP32 Memory series → • If your board has PSRAM, see Step 7: Compression with PSRAM →

The Problem

You have more data than you have room for, or you want to squeeze more into your flash budget. Store your data compressed in flash, decompress it in small chunks into RAM as you need it, use it, then discard it.

No PSRAM required. This works on any ESP32. Typical image and bitmap data compresses to 30 to 60 percent of its original size depending on content.

The Library

The easiest compression library for this purpose is miniz. It is small, well supported, and straightforward to use on the ESP32. Add it to your project via the Arduino Library Manager - search for miniz.

Compress On Your Computer

Use this Python script to compress your data file and output it as a C array ready to paste into a header file.

import zlib, sys

with open(sys.argv[1], 'rb') as f:
    data = f.read()

compressed = zlib.compress(data, level=9)

print(f"// Original size: {len(data)} bytes")
print(f"// Compressed size: {len(compressed)} bytes")
print(f"const uint32_t originalSize = {len(data)};")
print(f"const uint32_t compressedSize = {len(compressed)};")
print("const uint8_t compressedData[] PROGMEM = {")
print(', '.join(f'0x{b:02x}' for b in compressed))
print("}")

Run it like this:

python3 compress.py myimage.bmp > image_data.h

This gives you a header file with your compressed data stored in PROGMEM, sitting in flash taking up as little space as possible.

Decompress At Runtime

Place image_data.h in your sketch folder, then use this sketch as a starting point.

#include <miniz.h>
#include "image_data.h"

#define CHUNK_SIZE 256

void setup() {
    Serial.begin(115200);
    delay(1000);

    // Copy compressed data from PROGMEM into a RAM buffer
    uint8_t* compBuf = (uint8_t*) malloc(compressedSize);
    memcpy_P(compBuf, compressedData, compressedSize);

    // Set up the decompressor
    tinfl_decompressor decomp;
    tinfl_init(&decomp);

    uint8_t outChunk[CHUNK_SIZE];
    size_t inPos = 0;
    tinfl_status status;

    do {
        size_t inBytes = compressedSize - inPos;
        size_t outBytes = CHUNK_SIZE;

        status = tinfl_decompress(
            &decomp,
            compBuf + inPos, &inBytes,
            outChunk, outChunk, &outBytes,
            TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF
        );

        inPos += inBytes;

        // outChunk now contains outBytes of decompressed data
        // process it here: send to display, write to serial, whatever you need
        Serial.write(outChunk, outBytes);

    } while (status == TINFL_STATUS_HAS_MORE_OUTPUT);

    free(compBuf);
    Serial.println("\nDone.");
}

void loop() {}

What Is Happening

The compressed data lives in PROGMEM in flash. At runtime you copy it into a small RAM buffer, run the decompressor over it in chunks, and process each chunk as it comes out. Only one chunk exists in RAM at any moment. The rest stays compressed in flash until needed.

This is slower than having everything decompressed and ready to go, but on most projects the decompression is fast enough that it is not noticeable. The flash savings are the point.

Have PSRAM? You can remove the chunk-by-chunk complexity entirely - decompress everything into PSRAM at boot and access it freely with a normal pointer. See Step 7: Compression with PSRAM →
← Back to series