Compression and decompression API
The public API accepts the caller’s tensor and returns a self-describing
CompressionResult. Shape conversion, spatial blocking, temporal padding, and
shape restoration are handled inside the library.
#include <caesar/models/caesar_compress.h>
#include <caesar/models/caesar_decompress.h>
CompressionConfig config;
config.memory_data = field; // Original tensor; no caller-side reshape/padding.
config.n_frame = 8; // Required; only 8 is supported currently.
config.correction_method = caesar::CorrectionMethod::GAE;
Compressor compressor;
auto compressed = compressor.compress(config, 1e-4f);
Decompressor decompressor;
auto restored = decompressor.decompress(compressed);
GAE is the default correction method. LBRC and NGLR are alternatives;
config.nglr_options controls NGLR correction training. Requested relative error
bounds remain targets to validate on the actual data, not unconditional guarantees.
GAE and LBRC are extensively tested on Linux CPU and Linux NVIDIA CUDA. NGLR
has more limited validation and may encounter issues. Other backends have
method-specific testing status; see the platform matrix,
known limitations, and NGLR.
Shapes and ownership
Input |
Output |
|---|---|
|
|
|
|
|
|
Singleton dimensions are retained. For 5D input, variables 1 through V−1 are not
encoded or reconstructed. original_shape retains the full caller shape;
shape_info describes the selected variable’s restored shape and internal 5D
conversion. Spatial and temporal padding are recorded in compressionMetaData.
Noncontiguous tensors are accepted. Compression owns its working copy and does
not modify the caller’s tensor.
Input must be a nonempty floating-point tensor with three through five dimensions.
The current pipeline computes and returns float32 data; CPU float64 input is
converted internally, so very small error targets may be limited by float32
precision. Double-precision input on MPS fails explicitly. Output resides on the
installed model’s device. Transfer it with .cpu() if needed.
n_frame has no usable default: leaving it unset or setting it to anything other
than 8 throws. Compression and decompression use a fixed internal batch size of
128. The final partial batch uses only the samples present. NGLR’s separate
training batch option does not change foundation-model inference batching.
Model identity and lifetime
Constructors infer the device from the installed model. They optionally accept an
exact required model identity string, but no device or batch-size argument.
The compiled artifacts must target the intended backend; runtime availability
cannot make an installation compiled for a different backend compatible.
CompressionResult stores model_id, n_frame, correction method, original
shape, conversion metadata, encoded streams, and correction payloads.
Decompression checks that the installed model matches the result’s identity
before decoding. It does not load a different checkpoint automatically.
Keep all three .pt2 packages and all six vbr_* / gs_* probability tables
with the matching model installation. The result does not bundle model artifacts.
Use a separate compressor/decompressor object in each calling thread; the
existing per-thread model cache remains in use. See models.
GPU-to-CPU portability is still pending: latent indexes are regenerated by the hyper-decoder and are not yet serialized. The completed buffer serializer does not resolve or validate that separate entropy-decoding portability issue. Migration of the external ADIOS2 operator to this serializer must be handled and validated in that integration repository.
CompressionResult field reference
These are C++ member types, not a packed memory layout. The declarations live in
CAESAR/models/caesar_compress.h, CAESAR/data_utils.h, lbrc.h, and nglr.h.
All result fields are owned standard-library values, with no borrowed tensor
storage or model handles. Copies copy their containers; use std::move to transfer
large results. Keep the complete result for decoding, including metadata for the
selected correction method. Unused correction fields can be empty/default;
empty latent streams are valid for all-filtered constant data.
Member |
C++ type |
Meaning |
|---|---|---|
|
|
Full caller shape, including all original variables. |
|
|
Shape conversion/restoration metadata for the selected variable. |
|
|
Model window length, currently 8. |
|
|
Exact registered checkpoint identity required for decoding. |
|
|
Binary entropy-coded main latent streams, in encoded sample order. |
|
|
Corresponding binary hyper-latent streams. |
|
|
Normalization, sample placement, filtering, and padding metadata. |
|
|
Enum with |
|
|
GAE correction decoding parameters. |
|
|
GAE binary correction payload. |
|
|
LBRC correction decoding parameters. |
|
|
LBRC correction blocks in codec order. |
|
|
NGLR codec parameters and trained correction-network weights. |
|
|
NGLR binary correction payload. |
Shape and foundation metadata
|
C++ type |
Meaning |
|---|---|---|
|
|
Selected variable’s shape to restore. |
|
|
Selected variable’s element count, checked on restoration. |
|
|
Internal five-axis conversion shape. |
|
|
Legacy fields, currently initialized to zero. |
|
|
Legacy flag, currently false; actual padding is recorded below. |
|
C++ type |
Meaning |
|---|---|---|
|
|
Per-sample normalization parameters, where used by the model. |
|
|
Sample placement tuples |
|
|
Spatial block counts |
|
|
Internal blocked input shape used during reconstruction. |
|
|
Filtered block index and its constant reconstruction value. |
|
|
Global range used for GAE normalization. |
|
|
Global mean used for GAE normalization. |
|
|
Number of appended temporal padding frames. |
|
|
All blocks were filtered; reconstruction can omit latent decoding. |
GAE metadata
|
C++ type |
Meaning |
|---|---|---|
|
|
Whether GAE correction was needed. |
|
|
Padding applied when preparing reconstruction for GAE. |
|
|
PCA correction basis matrix. |
|
|
Unique quantized coefficient values used to recover coefficients. |
|
|
Coefficient quantization bin width. |
|
|
Process-mask length used by the GAE decoder. |
|
|
Prefix-mask length used by the GAE decoder. |
|
|
Compressed GAE data byte count recorded by the codec. |
|
|
Uncompressed coefficient-index byte count needed by lossless decoding. |
LBRC metadata and blocks
Structure/member |
C++ type |
Meaning |
|---|---|---|
|
|
Whether LBRC correction was needed. |
|
|
Correction normalization mean and scale. |
|
|
Correction block dimensions |
|
|
Block quantization step. |
|
|
Number of encoded bitplanes. |
|
|
Compressed binary bitplane streams. |
NGLR metadata and weights
|
C++ type |
Meaning |
|---|---|---|
|
|
Correction metadata schema, currently 1. |
|
|
Whether NGLR correction was needed. |
|
|
Constant-input reconstruction path flag. |
|
|
Normalization, quantization, and block settings below. |
|
|
Correction network architecture dimensions and block count. |
|
|
Correction tensor shape. |
|
|
Trained correction network parameters carried with this result. |
nglr::NGLRMeta contains double members x_mean (normalization mean),
scale (normalization scale), step (quantization step), q_context_scale
(context scaling), and delta_scale (residual scaling); block_t, block_h,
and block_w are int64_t correction block dimensions.
Each nglr::NGLRWeight has name (std::string, parameter name), shape
(std::vector<int64_t>, parameter dimensions), and values
(std::vector<float>, flattened parameter values). See NGLR.
Working with binary data in memory
// Streams are binary strings: embedded zero bytes are valid.
for (const std::string &stream : compressed.encoded_latents) {
const char *data = stream.data();
size_t length = stream.size();
// Pass data + length to your binary consumer; valid while stream is unchanged.
(void)data;
(void)length;
}
// Transport the entire result, including every metadata/correction field.
// Requires <caesar/models/serialization.h>.
std::vector<uint8_t> buffer = caesar::serialize(compressed);
CompressionResult received = caesar::deserialize(buffer);
torch::Tensor reconstructed = decompressor.decompress(received);
Do not memcpy the struct itself: strings and vectors contain process-local
pointers. Summing latent sizes alone omits metadata and correction overhead;
caesar::serialize(compressed).size() measures the complete serialized result
(excluding the separately installed model). The returned tensor is reconstructed
scientific data; the latent byte streams are encoded data, not float arrays.
Saving complete results
#include <caesar/models/serialization.h>
// Save every result field in one file, then reload and decompress.
caesar::save(compressed, "field.cae");
auto reloaded = caesar::load("field.cae");
auto restored_from_disk = decompressor.decompress(reloaded);
// Buffer API for ADIOS2 and other integrations.
auto bytes = caesar::serialize(compressed);
auto reloaded_from_buffer = caesar::deserialize(bytes.data(), bytes.size());
The version-1 buffer includes model identity, frame count, original shape, all padding and compression metadata, both latent streams, and all GAE, LBRC, and NGLR metadata and payloads (including NGLR weights). Embedded zero bytes are preserved. The buffer has a magic number and version, uses little-endian scalars and uint64 lengths, and currently requires 32-bit int and 64-bit size_t. Readers reject unsupported versions, truncated data, invalid lengths, invalid booleans/correction methods, and trailing bytes. This is a new format; it does not read the old ADIOS2 field sequence or the CLI’s three-file format.
save overwrites its destination. Model packages and probability tables are
still supplied separately by the matching installation. Serialization itself
requires no model loading or GPU execution and does not establish GPU-to-CPU
entropy-decoding portability.
In ADIOS2, replace the field-by-field writer with a length-prefixed buffer:
auto bytes = caesar::serialize(comp);
WriteParameter(bufferOut, bufferOutOffset, uint64_t(bytes.size()));
std::memcpy(bufferOut + bufferOutOffset, bytes.data(), bytes.size());
bufferOutOffset += bytes.size();
On reading, validate that the length fits within the remaining input bytes,
then call caesar::deserialize(bufferIn + bufferInOffset, length) and advance
the offset. Pass the returned result to Decompressor::decompress. Bump the
ADIOS2 operator buffer version when switching formats; keep its outer shape,
data type, threshold handling, and per-variable slicing logic. See the ADIOS2 migration notes. The remote
ADIOS2 source must be updated separately.
CLI migration
export CAESAR_MODEL_DIR="$PWD/exported_model"
build/CAESAR/caesar compress input.bin --shape 1,1,20,256,256 \
--n-frame 8 --error-bound 0.001 --correction gae --output sample.cae
build/CAESAR/caesar decompress sample.cae --output restored.bin \
--verify --original input.bin
The raw CLI input/output format is float32. Compression requires shape and
--n-frame 8. Decompression reads both from metadata; omit --shape and
--n-frame, including when verifying against the original file. The
--batch-size, --compress-device, and --decompress-device options have been
removed. Unknown options fail explicitly. Constant data may produce valid empty
.latents and .hyper files.
The CLI still writes three files: .latents, .hyper, and .meta. Metadata now
starts with the CAESAPI1 header carrying model identity, frame count, and shape
information. Older metadata is rejected with a recompression message; this is a
format break. These files retain the existing native binary payload encoding and
are not a new portable serialization API. The C/D file-based tests also use the
new header. ADIOS migration to the buffer API above remains a separate integration task.