Models and installation
The UFL_MODELS repository owns model_catalog.json. CAESAR reads the catalog directly
from UFL_MODELS/main by default. Checkpoint URLs remain pinned to immutable commits.
The following foundation and domain checkpoints support 3D–5D inputs. CAESAR v1
(registration 1) and CAESAR v2 (registration 2) are foundation models; v2 is the
newer optimized release and remains the default. eelsM1 (registration 3) is a
domain fine-tune trained and tested on one EELS dataset. microscopy (registration
4) is a domain-specific model trained and tested across four microscopy datasets.
Model training has been tested on NVIDIA GPUs only, according to the maintainer. Training has not been tested on CPU, AMD GPUs, Apple MPS, Intel XPU, or any other backend. This is separate from model export and compression/decompression validation; see training and platform testing scope.
Download and compile
python3 model_registry.py --list
python3 model_registry.py caesar_v2
python3 compile_model.py cpu
The shell wrapper compile_model.sh invokes the compilation command.
The Python entrypoints also work on Windows. Use the same Python environment that
contains torch and CAESAR’s dependencies. Choose cpu, cuda (including ROCm),
mps, or xpu; the backend must support AOTI compilation on your platform.
These export choices do not establish equal platform or correction-method
validation. GAE/LBRC are extensively tested on Linux CPU and NVIDIA CUDA;
ROCm GAE has preliminary validation and ROCm LBRC testing is in progress.
MPS is experimental and XPU end-to-end execution is unverified. See the
platform matrix and Windows CPU procedure.
For offline use with a local UFL_MODELS checkout, set the catalog path for downloading and model compilation:
export CAESAR_MODEL_CATALOG="$PWD/../UFL_MODELS/model_catalog.json"
python3 model_registry.py caesar_v2 --source-dir ../UFL_MODELS
python3 compile_model.py cpu
Downloads automatically save the validated UFL catalog inside
pretrained/selected_model.json. Compilation checks that snapshot and the local
checkpoint hash, and exports model_catalog.json with the installation. Exporters
validate locally: no internet access is needed after downloading. CMake only checks
that the compiled packages, entropy tables, and metadata exist; it does not require
a registry catalog or Python. A
later administrator removal does not invalidate an existing download’s snapshot.
For Frontier, run python3 model_registry.py caesar_v2 on the login node, then
run python3 compile_model.py cuda in the GPU job. ROCm uses the cuda device name.
Existing downloads made before snapshots were introduced compile directly from
pretrained/selected_model.json: the complete model record becomes a one-entry
local snapshot, and the checkpoint SHA-256 must still match. No login-node retry
or internet connection is needed.
--catalog overrides the catalog source for a registry command.
CAESAR_MODEL_CATALOG sets the download source and can supply a local catalog
for older selections or installations that lack snapshots. Saved snapshots take
precedence during compilation and installation validation. Snapshots are trusted
local download artifacts, not cryptographically signed registration certificates.
Download accepts an exact ufl:<number>@sha256:<hash> identity as well as a
registered name. --output selects the download directory; pass its
selected_model.json to the compiler with --selection. The compiler’s
--output selects the exported installation directory. Standard CMake packaging
uses exported_model/. CAESAR_MODEL_DIR selects a compiled installation at runtime.
Downloads preserve actual checkpoint filenames, verify SHA-256, and publish a
selection only after verification. Compilation checks the saved
UFL catalog snapshot when present; legacy selections supply their saved model
record instead. A local selection is not proof of upstream registration. All three
exporters use a frozen copy of the same verified checkpoint. A staged export is
published only when all three packages and six entropy tables exist. A failed
export preserves the old installation. Do not replace an installation while a
process is using it. An interrupted installer may leave exported_model.lock;
remove that directory only after confirming the installer is no longer running.
Individual exporter scripts can also be run separately for development, as described below.
Check the current UFL database later
Normal compilation uses the saved selection and checkpoint locally:
python3 compile_model.py cpu
# Or on the target GPU machine:
python3 compile_model.py cuda
An existing registered selection, including an older JSON without a catalog snapshot, does not need to be downloaded or registered again. Its saved registration is assumed locally; compilation still verifies the checkpoint hash and network compatibility. At the start of compilation, CAESAR also tries the current UFL catalog with a three-second network timeout. If this compute node cannot reach UFL, a warning says registration is unconfirmed and compilation continues using the saved local selection. An unavailable database is not treated as proof that the model is unregistered. If UFL is reachable but the checkpoint is absent, CAESAR warns to submit it and continues compilation. This check never changes identity. The complete installer checks once, rather than repeating it for each component.
On a networked login node, explicitly check the current database later:
python3 model_registry.py --check-registration
# Or check another selection:
python3 model_registry.py --check-registration /path/to/selected_model.json
The check uses the current UFL database, not the saved snapshot. It does not
compile, download checkpoints, or change the selection. A missing checkpoint
prints an all-caps warning and submission instructions. A retired checkpoint is
identified separately. A network failure says that registration could not be
checked; it does not claim the model is missing or block later local compilation.
Exit statuses are 0 for present, 1 for absent/retired, and 2 for unavailable.
--catalog PATH can check a local copy of the database instead. A local bypass
selection whose checkpoint hash is already registered reports the UFL identity
and how to select it.
Individual developer and in situ bypass
Each script accepts bypass or --bypass, independently of the device:
source venv_cpu/bin/activate # or your target device's environment
python3 CAESAR_compressor.py cpu bypass
python3 CAESAR_decompressor.py cpu bypass
python3 CAESAR_hyper_decompressor.py cpu bypass
Replace cpu with cuda, mps, or xpu for your target machine. Each command
compiles only that component. All read pretrained/selected_model.json, then load
the exact filename next to that JSON and verify its SHA-256. CAESAR does not scan
pretrained/ for a checkpoint or choose based on file timestamps. The .pt2 files
in exported_model/ are compilation outputs, not the original training checkpoint.
Individual exports default to exported_model/, replacing only the selected
component’s package. Use --output DIRECTORY and --selection PATH to override
either location. They do not update installation metadata or certify a complete
set. When replacing a component in an existing installation, use the same
checkpoint and device as its other components. The compressor also writes the
six entropy tables. Use compile_model.py when changing the installation’s model
or device.
To select your own checkpoint without sending anything to UFL:
python3 model_registry.py --local pretrained/my_model.pt --bypass \
--min-dims 3 --max-dims 5
python3 CAESAR_compressor.py cuda bypass
Local selection records the checkpoint hash, architecture label, and declared supported input ranks. It does not verify that those rank declarations have been tested. To produce a complete installation for in situ use, the wrapper also supports bypass and stages all components together:
python3 compile_model.py cuda bypass --output exported_model
python3 model_registry.py --validate-installation exported_model
Bypass works without internet access. Its UFL check is advisory; it skips catalog approval, keeps the hash and metadata
checks, and still strictly loads the state dictionary. Bypassed installations use
registration 0 and identity local@sha256:<hash> when no matching saved
registration can be validated. A valid saved UFL registration keeps its original
identity even with bypass; it does not trigger the unsupported-model warning.
For unregistered models, Python prints an all-caps unsupported/unsafe warning and
submission instructions; C++ warns when loading a local installation.
The compressed data records the local identity, so decoding must use that same
identity. Registering the same bytes later gives a different UFL identity; keep
the local installation for previously encoded data.
On a networked login node, install Git and GitHub CLI, authenticate, and submit:
gh auth login
python3 register_model.py add \
--name my_model --display-name "My model" \
--description "Training data, intended use, and provenance" \
--checkpoint /path/to/my_model.pt --min-dims 3 --max-dims 5
This opens the UFL_MODELS registration pull request. Bypass never uploads the
checkpoint or opens a pull request. After approval, select the registered model
with python3 model_registry.py my_model and compile without bypass. See
registration setup and submission.
Architecture changes and remaining constraints
Previously, compilation required a catalog snapshot even if the selection and
checkpoint hash were valid. Individual scripts required staging environment
variables from compile_model.py, so they could not be run directly for testing.
The registry checks, checkpoint integrity, and compilation setup were coupled.
Now legacy selection validation works offline, and the shared export_context()
resolves either a staged installation or the arguments of one direct exporter.
The complete installer retains its lock, frozen checkpoint, staging, and rollback.
Local identities are accepted by the Python installation validator and C++ metadata
reader without claiming a positive UFL registration. No network layers, tensor
shapes, entropy coding, or serialized buffer layout changed.
The exporters still duplicate network definitions and construct a fixed
caesar-bcrn-v1 configuration (dim=16, sr_dim=16, one input/output channel).
The architecture label in a JSON file does not make another network compatible:
strict loading rejects missing, extra, or differently shaped parameters. Supporting
a new architecture requires updating all three exporters and checking their
interface against the C++ runners and entropy tables. A future refactor could
share the network definition to prevent these copies drifting apart.
Inspect the selected model
./build/CAESAR/caesar --model-info
This prints the selected compiled installation’s model, registration, checkpoint
identity, dimensions, backend, and installation path. CAESAR_MODEL_DIR selects
the installation using the normal runtime search rules.
During export, compile_model.py writes the compilation hardware name to
model_device.txt beside model_metadata.txt. The executable reads this file for
Build device; it does not detect the current machine’s hardware. For example,
CUDA exports record the GPU name reported by PyTorch, such as NVIDIA B200, and
CPU exports record the compiler machine’s CPU model. Apple MPS exports
(python3 compile_model.py mps) record the Apple chip name, for example
MPS: Apple M4 Max; XPU exports record the device name reported by PyTorch. The file is packaged with
the installation. Older exports without it show “Not recorded”; recompile the
model to record the device. The Device field remains the compiled backend.
Example from a CPU installation (names and paths depend on your machine):
Built with CAESAR model
======================
Name : Xcompact3d_latent_16_16
Registration : 5
Device : cpu
Build device : CPU: 12th Gen Intel(R) Core(TM) i7-1250U
Dimensions : 3D–5D
Architecture : caesar-bcrn-v1
Checkpoint : Xcompact3d_latent_16_16.pt
Identity : ufl:5@sha256:066b4e38c357e9506d2c123d3bc483fdecd796cf7a80183a2cd501dd7c6a38a4
Installation : /path/to/CAESAR/exported_model
python3 model_registry.py --list displays an aligned catalog table, followed by
labeled descriptions and full identities. For example, a catalog containing the
two foundation releases displays:
Available CAESAR models
=======================
Reg. Name Display name Dimensions Default
---- --------- ------------ ---------- -------
1 caesar_v1 CAESAR v1 3D–5D
2 caesar_v2 CAESAR v2 3D–5D *
* Catalog default (the executable uses its compiled installation).
The star marks the catalog download default. It does not mean that the executable
uses that model. To change the runtime model, download and compile the desired
checkpoint, then select that installation with CAESAR_MODEL_DIR before starting
the executable.
Registration and identity
Identity is ufl:<registration_id>@sha256:<checkpoint_sha256>. Registration numbers,
names, hashes, and full identities must be unique. Never reuse a number or replace
a registered checkpoint’s bytes. A new checkpoint gets a new registration, even
when it is another version of the same model family. Names and descriptions help
users choose; the registration plus hash is what compressed data must record.
The device is installation-specific and is not part of checkpoint identity.
Registration commands live in CAESAR; UFL_MODELS is the authoritative catalog
database and stores registered checkpoints. Run python3 register_model.py add --help
to submit a checkpoint or python3 register_model.py remove MODEL_NAME to request
retirement. Both open a pull request to UFL_MODELS from your fork. Its workflow checks
that ownership comes from the pull request author’s stable account ID, that the
checkpoint is a state_dict matching the CAESAR network, and that nothing else changed,
then merges it automatically. Users may retire only their own models; administrator
E53klasky may retire any model. Retirement preserves the full record and reserves its
identity. The checkpoint is deleted from the current branch; Git history retains
older versions, so this does not shrink the full repository history.
See Automated model registration for commands, validation, one-time GitHub setup, admin rules, and consuming catalog updates. Local scripts do not enforce repository permissions; restrict direct catalog and workflow writes to trusted administrators and automation. CAESAR compilation continues to validate the UFL catalog and strictly load compatible state_dicts.
Local metadata and cache
model_metadata.txt records schema version, registration number, model name and
identity, checkpoint hash and original filename, architecture, supported ranks,
and compiled device. Runtime derives the artifact directory from that file’s
location, allowing installation relocation. The training checkpoint need not be
deployed beside the compiled packages. Descriptions and download URLs remain in
the catalog. Old name-only installations must be re-exported;
there is no name-only or implicit CPU fallback.
Metadata initializes once per process using C++ static initialization. Each thread
has its own identity-keyed cache of lazily loaded AOTI runners and probability
tables; MPI ranks have independent process memory. Use a separate Compressor or
Decompressor object in each thread. Calling one from a different thread fails
explicitly. Thread isolation costs extra model memory but avoids sharing a runner
whose concurrent invocation is not guaranteed safe. Cached runners use shared
ownership so clear() cannot invalidate handles held by existing objects. Windows retains the existing
AOTI teardown workaround (cached runners are not destroyed at thread teardown).
One compiled installation is selected per process. Set CAESAR_MODEL_DIR before
first use; restart the process to switch installations. Missing files in an
explicit directory fail rather than falling back to another installation. All
artifacts are loaded from the metadata’s directory, never combined across paths.
The runtime device must match the compilation target and be available. Runtime
accelerator selection cannot retarget compiled artifacts; the CLI provides no
device override. Export for the intended backend with its appropriate PyTorch
build, then use that matching installation. Backend availability alone does not
validate correction methods or achieved NRMSE.
Runtime cache lifetime and clearing
ModelCache is an in-memory runtime cache, separate from downloaded checkpoints
in pretrained/ and compiled artifacts in exported_model/. It does not cache
input tensors, compressed results, or reconstructed fields. Each thread lazily
loads and reuses its own compressor, hyper-decompressor, decoder, and six
probability tables. AOTI package-loader construction is serialized across threads;
that does not make a compressor/decompressor object callable from another thread.
Reuse objects on their creating thread for repeated fields. The first operation that needs a package/table pays its load cost; warm runs reuse it. Each additional worker thread or MPI rank can add model/table memory. There is no automatic LRU, capacity setting, or cross-thread/rank model sharing. Compressor/decompressor objects also hold copies of probability tables and shared runner handles.
Advanced callers can clear the cache of the calling thread:
#include <caesar/models/model_cache.h>
// First destroy compressor/decompressor objects in this thread if the aim is
// to release their model handles as well.
ModelCache::instance().clear();
clear() resets the cached three runner handles and six probability tables.
Existing objects retain their runner handles and table copies and remain usable;
subsequent cache getters reload artifacts. Clearing does not reset process-wide
model metadata, select another model/device, delete files, or clear another
thread’s cache. It does not guarantee an immediate drop in GPU/process memory,
since live handles and backend allocators may retain allocations. On Windows,
the thread-local registry is intentionally retained at teardown to avoid unsafe
AOTI destruction; explicit clearing still resets its entries’ loaded resources.
Serialized model identity and external integration
The version-1 CAESAR serializer stores CompressionResult.model_id with every
result, together with shape, normalization, padding, encoded streams, and
correction metadata/payloads. caesar::serialize/deserialize handle buffers;
caesar::save/load handle single-file storage. Decompression checks the saved
identity against the installed model before decoding and rejects a mismatch.
The model packages and probability tables remain in the matching installation.
See the public API.
The separately maintained ADIOS2 operator can wrap this buffer with a length
prefix and pass the deserialized result to Decompressor::decompress.
ADIOS2 migration notes describe that integration;
its external source and validation are outside this checkout. Existing name-only
files cannot establish an exact checkpoint identity, and the version-1 serializer
does not read legacy ADIOS2 buffers or the CLI’s three-file layout.
Input preparation
Padding and conversion to the model’s 5D input are internal CAESAR responsibilities.
The 3D–5D catalog range describes user input ranks handled through CAESAR’s internal
preparation; it is not an assertion that the exported network accepts arbitrary
shapes directly. The version-1 serializer preserves every PaddingInfo field,
including H, W, and was_padded; changing these fields requires considering
the serialized format as well as input preparation.
Verification
python3 -m unittest discover -s tests -p test_model_registry.py -v
python3 tests/check_registered_checkpoints.py --source-dir ../UFL_MODELS
./build/tests/test_model_metadata
CAESAR_MODEL_DIR="$PWD/exported_model" ./build/tests/test_model_cache
The checkpoint check strictly loads all active registrations into each of the three
export models. The CPU installation and test_caesarCD exercise actual compiled
inference. Metadata tests cover missing/mismatched identities and malformed
manifests; registry tests cover duplicates, unregistered selections, hash failures,
and preserving the previous installation when compilation fails. Cache tests
cover reuse, separate thread runners, and retaining live handles across clear().