Compression and decompression API

The public API accepts the caller’s tensor and returns a self-describing CompressionResult. Shape conversion, spatial blocking, temporal padding, and shape restoration are handled inside the library.

#include <caesar/models/caesar_compress.h>
#include <caesar/models/caesar_decompress.h>

CompressionConfig config;
config.memory_data = field;  // Original tensor; no caller-side reshape/padding.
config.n_frame = 8;          // Required; only 8 is supported currently.
config.correction_method = caesar::CorrectionMethod::GAE;

Compressor compressor;
auto compressed = compressor.compress(config, 1e-4f);
Decompressor decompressor;
auto restored = decompressor.decompress(compressed);

GAE is the default correction method. LBRC and NGLR are alternatives; config.nglr_options controls NGLR correction training. Requested relative error bounds remain targets to validate on the actual data, not unconditional guarantees. GAE and LBRC are extensively tested on Linux CPU and Linux NVIDIA CUDA. NGLR has more limited validation and may encounter issues. Other backends have method-specific testing status; see the platform matrix, known limitations, and NGLR.

Shapes and ownership

Input

Output

[T,H,W]

[T,H,W]

[S,T,H,W]

[S,T,H,W]

[V,S,T,H,W]

[1,S,T,H,W], containing variable index 0 only

Singleton dimensions are retained. For 5D input, variables 1 through V−1 are not encoded or reconstructed. original_shape retains the full caller shape; shape_info describes the selected variable’s restored shape and internal 5D conversion. Spatial and temporal padding are recorded in compressionMetaData. Noncontiguous tensors are accepted. Compression owns its working copy and does not modify the caller’s tensor.

Input must be a nonempty floating-point tensor with three through five dimensions. The current pipeline computes and returns float32 data; CPU float64 input is converted internally, so very small error targets may be limited by float32 precision. Double-precision input on MPS fails explicitly. Output resides on the installed model’s device. Transfer it with .cpu() if needed.

n_frame has no usable default: leaving it unset or setting it to anything other than 8 throws. Compression and decompression use a fixed internal batch size of 128. The final partial batch uses only the samples present. NGLR’s separate training batch option does not change foundation-model inference batching.

Model identity and lifetime

Constructors infer the device from the installed model. They optionally accept an exact required model identity string, but no device or batch-size argument. The compiled artifacts must target the intended backend; runtime availability cannot make an installation compiled for a different backend compatible. CompressionResult stores model_id, n_frame, correction method, original shape, conversion metadata, encoded streams, and correction payloads. Decompression checks that the installed model matches the result’s identity before decoding. It does not load a different checkpoint automatically.

Keep all three .pt2 packages and all six vbr_* / gs_* probability tables with the matching model installation. The result does not bundle model artifacts. Use a separate compressor/decompressor object in each calling thread; the existing per-thread model cache remains in use. See models.

GPU-to-CPU portability is still pending: latent indexes are regenerated by the hyper-decoder and are not yet serialized. The completed buffer serializer does not resolve or validate that separate entropy-decoding portability issue. Migration of the external ADIOS2 operator to this serializer must be handled and validated in that integration repository.

CompressionResult field reference

These are C++ member types, not a packed memory layout. The declarations live in CAESAR/models/caesar_compress.h, CAESAR/data_utils.h, lbrc.h, and nglr.h. All result fields are owned standard-library values, with no borrowed tensor storage or model handles. Copies copy their containers; use std::move to transfer large results. Keep the complete result for decoding, including metadata for the selected correction method. Unused correction fields can be empty/default; empty latent streams are valid for all-filtered constant data.

Member

C++ type

Meaning

original_shape

std::vector<int64_t>

Full caller shape, including all original variables.

shape_info

PaddingInfo

Shape conversion/restoration metadata for the selected variable.

n_frame

int

Model window length, currently 8.

model_id

std::string

Exact registered checkpoint identity required for decoding.

encoded_latents

std::vector<std::string>

Binary entropy-coded main latent streams, in encoded sample order.

encoded_hyper_latents

std::vector<std::string>

Corresponding binary hyper-latent streams.

compressionMetaData

CompressionMetaData

Normalization, sample placement, filtering, and padding metadata.

correction_method

caesar::CorrectionMethod

Enum with uint8_t underlying type: GAE=0, LBRC=1, NGLR=2.

gaeMetaData

GAEMetaData

GAE correction decoding parameters.

gae_comp_data

std::vector<uint8_t>

GAE binary correction payload.

lbrcMetaData

LBRCMetaData

LBRC correction decoding parameters.

lbrc_blocks

std::vector<LBRCBlock>

LBRC correction blocks in codec order.

nglrMetaData

nglr::NGLRMetaData

NGLR codec parameters and trained correction-network weights.

nglr_comp_data

std::vector<uint8_t>

NGLR binary correction payload.

Shape and foundation metadata

PaddingInfo member

C++ type

Meaning

original_shape

std::vector<int64_t>

Selected variable’s shape to restore.

original_length

int64_t

Selected variable’s element count, checked on restoration.

padded_shape

std::vector<int64_t>

Internal five-axis conversion shape.

H, W

int64_t each

Legacy fields, currently initialized to zero.

was_padded

bool

Legacy flag, currently false; actual padding is recorded below.

CompressionMetaData member

C++ type

Meaning

offsets, scales

std::vector<float> each

Per-sample normalization parameters, where used by the model.

indexes

std::vector<std::vector<int32_t>>

Sample placement tuples [variable, blocked_section, start_t, end_t]; end is exclusive. These are not entropy-decoder latent indexes.

block_info

std::tuple<int32_t, int32_t, std::vector<int32_t>>

Spatial block counts (nH,nW) and padding [top,down,left,right].

data_input_shape

std::vector<int32_t>

Internal blocked input shape used during reconstruction.

filtered_blocks

std::vector<std::pair<int32_t, float>>

Filtered block index and its constant reconstruction value.

global_scale

float

Global range used for GAE normalization.

global_offset

float

Global mean used for GAE normalization.

pad_T

int64_t

Number of appended temporal padding frames.

all_filtered

bool

All blocks were filtered; reconstruction can omit latent decoding.

GAE metadata

GAEMetaData member

C++ type

Meaning

GAE_correction_occur

bool

Whether GAE correction was needed.

padding_recon_info

std::vector<int>

Padding applied when preparing reconstruction for GAE.

pcaBasis

std::vector<std::vector<float>>

PCA correction basis matrix.

uniqueVals

std::vector<float>

Unique quantized coefficient values used to recover coefficients.

quanBin

double

Coefficient quantization bin width.

nVec

int64_t

Process-mask length used by the GAE decoder.

prefixLength

int64_t

Prefix-mask length used by the GAE decoder.

dataBytes

int64_t

Compressed GAE data byte count recorded by the codec.

coeffIntBytes

size_t

Uncompressed coefficient-index byte count needed by lossless decoding.

LBRC metadata and blocks

Structure/member

C++ type

Meaning

LBRCMetaData::lbrc_correction_occur

bool

Whether LBRC correction was needed.

LBRCMetaData::x_mean, scale

float each

Correction normalization mean and scale.

LBRCMetaData::block_size

std::array<int64_t, 3>

Correction block dimensions (T,H,W).

LBRCBlock::step

double

Block quantization step.

LBRCBlock::bit_count

uint32_t

Number of encoded bitplanes.

LBRCBlock::streams

std::vector<std::vector<uint8_t>>

Compressed binary bitplane streams.

NGLR metadata and weights

nglr::NGLRMetaData member

C++ type

Meaning

schema_version

uint32_t

Correction metadata schema, currently 1.

correction_occurred

bool

Whether NGLR correction was needed.

constant_input

bool

Constant-input reconstruction path flag.

quantization

nglr::NGLRMeta

Normalization, quantization, and block settings below.

hidden, q_hidden, model_blocks

int32_t each

Correction network architecture dimensions and block count.

shape

std::vector<int64_t>

Correction tensor shape.

weights

std::vector<nglr::NGLRWeight>

Trained correction network parameters carried with this result.

nglr::NGLRMeta contains double members x_mean (normalization mean), scale (normalization scale), step (quantization step), q_context_scale (context scaling), and delta_scale (residual scaling); block_t, block_h, and block_w are int64_t correction block dimensions. Each nglr::NGLRWeight has name (std::string, parameter name), shape (std::vector<int64_t>, parameter dimensions), and values (std::vector<float>, flattened parameter values). See NGLR.

Working with binary data in memory

// Streams are binary strings: embedded zero bytes are valid.
for (const std::string &stream : compressed.encoded_latents) {
  const char *data = stream.data();
  size_t length = stream.size();
  // Pass data + length to your binary consumer; valid while stream is unchanged.
  (void)data;
  (void)length;
}

// Transport the entire result, including every metadata/correction field.
// Requires <caesar/models/serialization.h>.
std::vector<uint8_t> buffer = caesar::serialize(compressed);
CompressionResult received = caesar::deserialize(buffer);
torch::Tensor reconstructed = decompressor.decompress(received);

Do not memcpy the struct itself: strings and vectors contain process-local pointers. Summing latent sizes alone omits metadata and correction overhead; caesar::serialize(compressed).size() measures the complete serialized result (excluding the separately installed model). The returned tensor is reconstructed scientific data; the latent byte streams are encoded data, not float arrays.

Saving complete results

#include <caesar/models/serialization.h>

// Save every result field in one file, then reload and decompress.
caesar::save(compressed, "field.cae");
auto reloaded = caesar::load("field.cae");
auto restored_from_disk = decompressor.decompress(reloaded);

// Buffer API for ADIOS2 and other integrations.
auto bytes = caesar::serialize(compressed);
auto reloaded_from_buffer = caesar::deserialize(bytes.data(), bytes.size());

The version-1 buffer includes model identity, frame count, original shape, all padding and compression metadata, both latent streams, and all GAE, LBRC, and NGLR metadata and payloads (including NGLR weights). Embedded zero bytes are preserved. The buffer has a magic number and version, uses little-endian scalars and uint64 lengths, and currently requires 32-bit int and 64-bit size_t. Readers reject unsupported versions, truncated data, invalid lengths, invalid booleans/correction methods, and trailing bytes. This is a new format; it does not read the old ADIOS2 field sequence or the CLI’s three-file format.

save overwrites its destination. Model packages and probability tables are still supplied separately by the matching installation. Serialization itself requires no model loading or GPU execution and does not establish GPU-to-CPU entropy-decoding portability.

In ADIOS2, replace the field-by-field writer with a length-prefixed buffer:

auto bytes = caesar::serialize(comp);
WriteParameter(bufferOut, bufferOutOffset, uint64_t(bytes.size()));
std::memcpy(bufferOut + bufferOutOffset, bytes.data(), bytes.size());
bufferOutOffset += bytes.size();

On reading, validate that the length fits within the remaining input bytes, then call caesar::deserialize(bufferIn + bufferInOffset, length) and advance the offset. Pass the returned result to Decompressor::decompress. Bump the ADIOS2 operator buffer version when switching formats; keep its outer shape, data type, threshold handling, and per-variable slicing logic. See the ADIOS2 migration notes. The remote ADIOS2 source must be updated separately.

CLI migration

export CAESAR_MODEL_DIR="$PWD/exported_model"
build/CAESAR/caesar compress input.bin --shape 1,1,20,256,256 \
  --n-frame 8 --error-bound 0.001 --correction gae --output sample.cae
build/CAESAR/caesar decompress sample.cae --output restored.bin \
  --verify --original input.bin

The raw CLI input/output format is float32. Compression requires shape and --n-frame 8. Decompression reads both from metadata; omit --shape and --n-frame, including when verifying against the original file. The --batch-size, --compress-device, and --decompress-device options have been removed. Unknown options fail explicitly. Constant data may produce valid empty .latents and .hyper files.

The CLI still writes three files: .latents, .hyper, and .meta. Metadata now starts with the CAESAPI1 header carrying model identity, frame count, and shape information. Older metadata is rejected with a recompression message; this is a format break. These files retain the existing native binary payload encoding and are not a new portable serialization API. The C/D file-based tests also use the new header. ADIOS migration to the buffer API above remains a separate integration task.