ZMat - Portable Data Compression for MATLAB, Python & C

# ZMat

Portable data compression library for MATLAB/Octave, Python and C/Fortran

v1.0.0 MATLAB / Octave Python C Library Fortran90 GPL v3 NIH Funded

🗜️ What is ZMat?

ZMat provides a fast, portable C compression library (libzmat) with bindings for MATLAB/Octave (MEX) and Python (C extension). A single unified API gives you access to 12 codecs with zero external dependencies.

#### 🏆 NeuroJSON Project

Part of the NeuroJSON project (neurojson.org), funded by NIH grant U24-NS124027. Powers compression in JSONLab, JNIfTI and other NeuroJSON tools.

12
Total Codecs
3
Language Bindings
6+
Years Active
GitHub Examples NeuroJSON

⚙️ Supported Codecs

Method | Speed | Ratio | Best For
--- | --- | --- | ---
lz4 | ⚡⚡⚡⚡⚡ | ⭐⭐ | Real-time / streaming
zlib / gzip | ⚡⚡⚡⚡ | ⭐⭐⭐ | Balanced; wide compatibility
zstd | ⚡⚡⚡ | ⭐⭐⭐⭐ | Modern high-throughput
lz4hc | ⚡⚡⚡ | ⭐⭐⭐ | LZ4 with better ratio
lzma / lzip | ⚡ | ⭐⭐⭐⭐⭐ | Maximum compression
blosc2* | ⚡⚡⚡⚡ | ⭐⭐⭐⭐ | N-D numeric arrays
base64 | ⚡⚡⚡⚡⚡ | — | Text-safe encoding

* blosc2 variants: blosclz, lz4, lz4hc, zlib, zstd

⚡ Performance Guide

#### 🚀 Ultra-Fast — LZ4

- Fastest compress & decompress
- Ideal for real-time pipelines
- Level 1–9

#### ⚖️ Balanced — zlib / zstd

- Industry-standard algorithms
- Python stdlib interoperable
- Level 1–9 (zstd up to 22)

#### 🎯 Smallest — LZMA

- Highest compression ratio
- Best for archival/cold storage
- Level 1–9

#### 🔬 Scientific — Blosc2

- Optimized for N-D arrays
- Byte-shuffle pre-filter
- Multi-threaded (nthread)

## Key Features

One library, every language you need

🔧

### libzmat C Library

Static & dynamic library. Embed directly in any C/C++/Fortran project.

📐

### MATLAB / Octave MEX

Single `zmat()` call for compress, decompress, encode and decode.

🐍

### Python Extension

`pip install zmat` — pure C extension, zero runtime dependencies.

🌍

### Cross-Platform

Windows, Linux, macOS. Pre-compiled MEX & Python wheels available.

📦

### 12 Codecs Built-in

zlib, gzip, lzma, lzip, lz4, lz4hc, zstd, 5× blosc2, base64.

🔀

### Type Preservation

MATLAB `info` struct and Python `info=True` flag restore the original array dtype, shape, and memory order after decompression.

## Code Examples

Choose your language

    % ── Compress with default zlib ──────────────────────────────────
    [compressed, info] = zmat(eye(10));

    % Decompress using the info struct (restores type + dimensions)
    original = zmat(compressed, info);

    % Or decompress manually (iscompress = 0)
    original = zmat(compressed, 0);
    isequal(original, eye(10))   % → 1

    % ── Compression level ───────────────────────────────────────────
    % iscompress = -N sets compression level N (1–9)
    data = rand(500, 500);
    [c1, i1] = zmat(data, -1, 'zlib');   % fast, larger
    [c9, i9] = zmat(data, -9, 'zlib');   % slow, smaller
    fprintf('Level 1: %d B  Level 9: %d B\n', numel(c1), numel(c9));

    % ── Round-trip any array type ───────────────────────────────────
    img = uint8(rand(256,256,3) * 255);
    [cimg, iimg] = zmat(img, 1, 'lz4');
    restored = zmat(cimg, iimg);    % uint8 256×256×3 restored

    % ── Benchmark all codecs on the same data ───────────────────────
    data   = rand(500, 500);       % 2 MB of doubles
    codecs = {'zlib','gzip','lzma','lzip','lz4','lz4hc','zstd'};

    for k = 1:numel(codecs)
        t = tic;
        [cdata, info] = zmat(data, 1, codecs{k});
        tc = toc(t);
        t = tic;
        orig = zmat(cdata, info);
        td = toc(t);
        fprintf('%-8s  %6.1f KB  ratio=%.2f  c=%.3fs  d=%.3fs\n', ...
            codecs{k}, numel(cdata)/1024, numel(cdata)/numel(data)/8, tc, td);
    end

    % ── Decompression with explicit method ──────────────────────────
    [czstd, ~] = zmat(data, 1, 'zstd');
    orig = zmat(czstd, 0, 'zstd');     % pass method explicitly
    assert(isequal(orig, data));

    % ── Blosc2 meta-compressor (optimized for N-D numeric arrays) ───
    vol = uint16(rand(128,128,64) * 65535);   % 3-D volume, uint16

    % Basic blosc2 variants
    [b_lz4,  i1] = zmat(vol, 1, 'blosc2lz4');
    [b_zstd, i2] = zmat(vol, 1, 'blosc2zstd');
    [b_zlib, i3] = zmat(vol, 1, 'blosc2zlib');

    fprintf('lz4: %d  zstd: %d  zlib: %d bytes\n', ...
        numel(b_lz4), numel(b_zstd), numel(b_zlib));

    % Advanced: 4 threads, byte-shuffle, typesize=2 (uint16)
    [best, info] = zmat(vol, -6, 'blosc2zstd', ...
        'nthread', 4, 'shuffle', 1, 'typesize', 2);

    % Decompress — info carries nthread / shuffle / typesize
    restored = zmat(best, info);
    assert(isequal(restored, vol));

    % ── Base64 encode / decode ──────────────────────────────────────
    str = 'ZMat compression test';

    % Encode  (iscompress=2 strips trailing newline)
    encoded = char(zmat(str, 2, 'base64'));
    disp(encoded)   % → 'Wk1hdCBjb21wcmVzc2lvbiB0ZXN0'

    % Decode
    decoded = char(zmat(encoded, 0, 'base64'));
    assert(strcmp(decoded, str));

    % Keep newlines every 76 chars (iscompress=3 = RFC 2045 style)
    encoded_full = char(zmat(str, 3, 'base64'));

    % Chain: compress then base64-encode for JSON embedding
    data  = rand(50, 50);
    [cmp, info] = zmat(data, 1, 'zlib');
    b64  = char(zmat(cmp, 2, 'base64'));   % safe for JSON strings

    % Reverse: base64-decode then decompress
    raw  = zmat(uint8(b64), 0, 'base64');
    orig = zmat(raw, info);
    assert(isequal(orig, data));

    import zmat

    # ── Compress / decompress (default: zlib) ───────────────────────
    data       = b"ZMat compression test data" * 1000
    compressed = zmat.compress(data)
    restored   = zmat.decompress(compressed)
    assert restored == data

    print(ff"Original: {len(data):,} B → Compressed: {len(compressed):,} B  "
          f"(ratio {len(compressed)/len(data):.3f})")

    # ── Compression level ───────────────────────────────────────────
    # low-level zmat() call: iscompress=1 default, -N sets level
    c_fast = zmat.zmat(data, iscompress=1,  method='zlib')   # default
    c_max  = zmat.zmat(data, iscompress=-9, method='zlib')   # max
    print(f"Level 1: {len(c_fast)} B  Level 9: {len(c_max)} B")

    # Both round-trip correctly
    assert zmat.decompress(c_fast) == data
    assert zmat.decompress(c_max)  == data

    # ── bytearray input also works ──────────────────────────────────
    buf = bytearray(b"buffered data" * 500)
    assert zmat.decompress(zmat.compress(buf)) == bytes(buf)

    import zmat, time

    data = bytes(range(256)) * 4000   # ~1 MB, moderately compressible

    methods = ['zlib', 'gzip', 'lzma', 'lzip', 'lz4', 'lz4hc', 'zstd']

    print(f"{'Method':<10} {'Size':>8}  {'Ratio':>6}  {'Comp':>8}  {'Decomp':>8}")
    print("-" * 50)

    for m in methods:
        t0 = time.perf_counter()
        c  = zmat.compress(data, method=m)
        tc = time.perf_counter() - t0

        t0 = time.perf_counter()
        d  = zmat.decompress(c, method=m)
        td = time.perf_counter() - t0

        assert d == data, ff"{m} round-trip failed!"
        print(ff"{m:<10} {len(c):>8}  {len(c)/len(data):>6.3f}  {tc:>7.4f}s  {td:>7.4f}s")

    # ── Method names are case-insensitive ───────────────────────────
    c1 = zmat.compress(data, method='ZLIB')
    c2 = zmat.compress(data, method='zlib')
    assert c1 == c2

    import zmat

    # ── Blosc2 meta-compressor (ideal for numeric arrays) ───────────
    import struct
    # Simulate a 3-D uint16 volume (128×128×64)
    vol = bytes(range(256)) * (128*128*64 * 2 // 256)

    # All five blosc2 variants
    for m in ['blosc2blosclz','blosc2lz4','blosc2lz4hc','blosc2zlib','blosc2zstd']:
        c = zmat.compress(vol, method=m)
        d = zmat.decompress(c, method=m)
        assert d == vol
        print(ff"{m:<18}  {len(c):>8} B  ratio={len(c)/len(vol):.3f}")

    # ── Advanced: multi-threading + byte-shuffle ────────────────────
    # typesize=2 → uint16 elements; shuffle=1 → byte-shuffle filter
    c_adv = zmat.zmat(vol, iscompress=-6, method='blosc2zstd',
                      nthread=4, shuffle=1, typesize=2)
    d_adv = zmat.zmat(c_adv, iscompress=0, method='blosc2zstd')
    assert d_adv == vol
    print(ff"Advanced blosc2zstd: {len(c_adv)} B")

    import numpy as np
    import zmat

    # ── Compress / restore a NumPy array ────────────────────────────
    arr = np.random.rand(1000, 1000)           # 8 MB float64
    raw = arr.tobytes()                          # zmat works on bytes

    c = zmat.compress(raw, method='lz4')
    print(f"lz4: {len(raw):,} → {len(c):,} B")

    restored = np.frombuffer(
        zmat.decompress(c, method='lz4'),
        dtype=arr.dtype
    ).reshape(arr.shape)
    assert np.array_equal(arr, restored)

    # ── Blosc2 + shuffle for integer arrays ─────────────────────────
    vol   = np.arange(128*128*64, dtype=np.uint16).reshape(128,128,64)
    cbest = zmat.zmat(vol.tobytes(), iscompress=1, method='blosc2zstd',
                      typesize=2, shuffle=1)    # typesize = itemsize
    dbest = zmat.zmat(cbest, iscompress=0, method='blosc2zstd')
    vrest = np.frombuffer(dbest, dtype=np.uint16).reshape(vol.shape)
    assert np.array_equal(vol, vrest)

    # ── Structured dtype round-trip ─────────────────────────────────
    dt  = np.dtype([('x', np.float32), ('y', np.float32), ('z', np.float32)])
    pts = np.zeros(100000, dtype=dt)
    c2  = zmat.compress(pts.tobytes(), method='zstd')
    print(f"Struct array: {pts.nbytes:,} → {len(c2):,} B")

    import numpy as np
    import zmat

    # ── Array-info round-trip (mirrors MATLAB [ss,info]=zmat / zmat(ss,info)) ─
    arr = np.eye(5)                      # float64, 5×5 identity

    # compress — capture dtype, shape, order alongside the bytes
    compressed, info = zmat.compress(arr, info=True)
    print(info)
    # {'type': 'float64', 'shape': (5, 5), 'byte': 8, 'method': 'zlib', 'order': 'C'}

    # decompress — array is fully restored: dtype, shape, values
    restored = zmat.decompress(compressed, info=info)
    assert isinstance(restored, np.ndarray)
    assert restored.dtype == arr.dtype
    assert restored.shape == arr.shape
    assert np.array_equal(restored, arr)

    # ── Any dtype, any shape ─────────────────────────────────────────
    img = np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8)
    c_img, i_img = zmat.compress(img, method='lz4', info=True)
    r_img = zmat.decompress(c_img, info=i_img)   # uint8 256×256×3 restored
    assert np.array_equal(r_img, img)

    vol = np.arange(128*128*64, dtype=np.uint16).reshape(128, 128, 64)
    c_vol, i_vol = zmat.compress(vol, method='zstd', info=True)
    r_vol = zmat.decompress(c_vol, info=i_vol)   # uint16 128×128×64 restored
    assert np.array_equal(r_vol, vol)

    # ── Fortran-order arrays are preserved ───────────────────────────
    arr_f = np.asfortranarray(np.eye(100))
    c_f, i_f = zmat.compress(arr_f, info=True)
    r_f = zmat.decompress(c_f, info=i_f)
    assert r_f.flags['F_CONTIGUOUS']         # memory order restored
    assert np.array_equal(r_f, arr_f)

    # ── info dict fields (mirrors MATLAB info struct) ────────────────
    #   'type'   → numpy dtype string, e.g. 'float64'
    #   'shape'  → tuple of dimensions
    #   'byte'   → bytes per element  (arr.itemsize)
    #   'method' → compression method used
    #   'order'  → 'F' for Fortran-contiguous, 'C' otherwise

    # ── non-ndarray input: returns (bytes, None) ────────────────────
    c_raw, info_none = zmat.compress(b"plain bytes" * 100, info=True)
    assert info_none is None                    # no metadata for plain bytes

    import zmat, zlib, gzip, base64, io

    # ── Base64 encode / decode ──────────────────────────────────────
    data    = b"ZMat Python base64 test"
    encoded = zmat.encode(data, method='base64')
    decoded = zmat.decode(encoded, method='base64')
    assert decoded == data

    # zmat base64 matches Python stdlib (strip newlines for comparison)
    assert encoded.replace(b'\n', b'') == base64.b64encode(data)

    # Chain: compress → base64  (useful for JSON / XML embedding)
    payload = b"scientific data" * 5000
    c   = zmat.compress(payload, method='zlib')
    b64 = zmat.encode(c, method='base64')
    print(f"base64 string: {len(b64)} chars")

    # Reverse
    c2  = zmat.decode(b64, method='base64')
    out = zmat.decompress(c2, method='zlib')
    assert out == payload

    # ── stdlib interoperability ─────────────────────────────────────
    text = b"interoperability test" * 500

    # zmat zlib ↔ Python zlib
    assert zlib.decompress(zmat.compress(text, method='zlib')) == text
    assert zmat.decompress(zlib.compress(text), method='zlib')  == text

    # zmat gzip ↔ Python gzip
    cg = zmat.compress(text, method='gzip')
    with gzip.open(io.BytesIO(cg), 'rb') as f:
        assert f.read() == text

    /* test_zmat.c – compress and decompress with libzmat */
    #include "zmatlib.h"
    #include <stdio.h>
    #include <string.h>
    #include <stdlib.h>

    int main() {
        unsigned char input[] = "ZMat C compression example";
        size_t inputsize = strlen((char*)input);
        unsigned char *output  = NULL;
        unsigned char *decoded = NULL;
        size_t outputsize = 0, decodedsize = 0;
        int    status    = 0;

        /* zipid: 0=zlib 1=gzip 3=lzma 5=lz4 7=zstd 8-12=blosc2* */
        /* clevel: -1 to -9 to set level;  0 = decompress          */
        zmat_run(inputsize, input, &outputsize, &output, 0, &status, -6);
        printf("Compressed  : %zu → %zu bytes\n", inputsize, outputsize);

        zmat_run(outputsize, output, &decodedsize, &decoded, 0, &status, 0);
        printf("Decompressed: %s\n", decoded);

        free(output);
        free(decoded);
        return 0;
    }

    ! test_zmat.f90 – Fortran90 binding via zmat_module
    program zmat_demo
        use iso_c_binding
        use zmat_module
        implicit none

        character(len=40)         :: input_str = "ZMat Fortran demo"
        integer(c_size_t)          :: input_len, output_len, decoded_len
        integer(c_int)             :: zipid, status, clevel
        type(c_ptr)                :: out_ptr, dec_ptr
        character(kind=c_char), pointer :: out_arr(:), dec_arr(:)
        integer :: i

        input_len = len_trim(input_str)
        zipid  = 0    ! 0=zlib, 5=lz4, 7=zstd …
        clevel = -6   ! compression level

        ! Compress
        call zmat_run(input_len, input_str, output_len, out_ptr, zipid, status, clevel)
        call c_f_pointer(out_ptr, out_arr, [output_len])
        print *, "Compressed:", input_len, "→", output_len, "bytes"

        ! Decompress (clevel=0)
        call zmat_run(output_len, out_arr, decoded_len, dec_ptr, zipid, status, 0)
        call c_f_pointer(dec_ptr, dec_arr, [decoded_len])
        print *, "Decompressed:", (dec_arr(i), i=1,decoded_len)
    end program zmat_demo

#### GCC / Clang

    gcc -o test_zmat test_zmat.c -I../include -L../lib -lzmat -lpthread -lm

#### CMake

    cmake_minimum_required(VERSION 3.3)
    project(zmat_example C)

    find_library(ZMAT_LIB zmat PATHS ../lib)
    add_executable(test_zmat test_zmat.c)
    target_include_directories(test_zmat PRIVATE ../include)
    target_link_libraries(test_zmat ${ZMAT_LIB} pthread m)

#### Fortran

    gfortran -o test_zmat test_zmat.f90 zmat_fortran.f90 \
        -I../include -L../lib -lzmat -lpthread -lm

## API Reference

Consistent interface across all languages

### MATLAB / Octave

zmat.m / zipmat.mex

- `zmat(data)`compress with default zlib
- `zmat(data, 1, method)`compress with named codec
- `zmat(data, -N, method)`compress at level N (1–9)
- `zmat(data, 0, method)`decompress
- `zmat(data, info)`decompress & restore type/size
- `[out, info] = zmat(…)`info carries type, size, method
- `zmat(…, 'nthread', N)`blosc2 thread count
- `zmat(…, 'typesize', B)`bytes per element (shuffle)

### Python

import zmat

- `zmat.compress(data, method, level)`compress bytes / bytearray
- `zmat.compress(arr, info=True)`compress ndarray → (bytes, info_dict)
- `zmat.decompress(data, method)`decompress bytes
- `zmat.decompress(data, info=info)`decompress → original ndarray (dtype + shape restored)
- `zmat.encode(data, method)`encode (e.g. base64)
- `zmat.decode(data, method)`decode
- `zmat.zmat(data, iscompress, method,`
  `nthread, shuffle, typesize)` low-level full-control interface

### C Library

zmatlib.h

- `zmat_run(insize, in, &outsize, &out,`
  `zipid, &status, clevel)` main compress / decompress call
- `zmat_encode(…)`compress wrapper
- `zmat_decode(…)`decompress wrapper

### Codec Names

zipid (C) / method (MATLAB & Python)

- `0 / 'zlib'`zlib/zip (default)
- `1 / 'gzip'`gzip format
- `2 / 'base64'`base64 encoding
- `3 / 'lzma'`LZMA
- `4 / 'lzip'`lzip format
- `5 / 'lz4'`LZ4 fast
- `6 / 'lz4hc'`LZ4 high compression
- `7 / 'zstd'`Zstandard
- `8–12 / 'blosc2*'`blosclz, lz4, lz4hc, zlib, zstd

## Installation

Pick your platform and language

### 🐍 Python — PyPI

Zero dependencies. Wheels for Linux, macOS, Windows.

pip install zmat

Build from source:

git clone https://github.com/NeuroJSON/zmat.git cd zmat/python && pip install .

### 📐 MATLAB / Octave — Manual

Download ZIP, then add to path:

addpath('/path/to/zmat'); rehash;

Pre-compiled MEX files included for Win/Linux/macOS

### 🐧 Debian / Ubuntu

Install from package manager:

sudo apt-get install octave-zmat matlab-zmat # C library + headers sudo apt-get install libzmat1 libzmat1-dev

### 🎩 Fedora

sudo dnf install octave-zmat # C library sudo dnf install zmat zmat-devel zmat-static

### 🔧 From Source (MEX)

git clone https://github.com/NeuroJSON/zmat.git cd zmat/src make mex # MATLAB make oct # Octave

### 🏗️ CMake

cd zmat/src mkdir build && cd build cmake ../ make

Builds libzmat.a + zipmat.mex automatically

## Ready to Compress?

Fast · Portable · Zero dependencies

1.0.0
Current Version
12
Codecs Built-in
3
Language Bindings
Download Latest Release PyPI Package GitHub Repository

Supported by NIH Grant U24-NS124027 · GPL v3 License

Copyright © 2019–2026 Qianqian Fang <q.fang@neu.edu>