Models and installation

The UFL_MODELS repository owns model_catalog.json. CAESAR reads the catalog directly from UFL_MODELS/main by default. Checkpoint URLs remain pinned to immutable commits. The following foundation and domain checkpoints support 3D–5D inputs. CAESAR v1 (registration 1) and CAESAR v2 (registration 2) are foundation models; v2 is the newer optimized release and remains the default. eelsM1 (registration 3) is a domain fine-tune trained and tested on one EELS dataset. microscopy (registration 4) is a domain-specific model trained and tested across four microscopy datasets.

Model training has been tested on NVIDIA GPUs only, according to the maintainer. Training has not been tested on CPU, AMD GPUs, Apple MPS, Intel XPU, or any other backend. This is separate from model export and compression/decompression validation; see training and platform testing scope.

Download and compile

python3 model_registry.py --list
python3 model_registry.py caesar_v2
python3 compile_model.py cpu

The shell wrapper compile_model.sh invokes the compilation command. The Python entrypoints also work on Windows. Use the same Python environment that contains torch and CAESAR’s dependencies. Choose cpu, cuda (including ROCm), mps, or xpu; the backend must support AOTI compilation on your platform. These export choices do not establish equal platform or correction-method validation. GAE/LBRC are extensively tested on Linux CPU and NVIDIA CUDA; ROCm GAE has preliminary validation and ROCm LBRC testing is in progress. MPS is experimental and XPU end-to-end execution is unverified. See the platform matrix and Windows CPU procedure.

For offline use with a local UFL_MODELS checkout, set the catalog path for downloading and model compilation:

export CAESAR_MODEL_CATALOG="$PWD/../UFL_MODELS/model_catalog.json"
python3 model_registry.py caesar_v2 --source-dir ../UFL_MODELS
python3 compile_model.py cpu

Downloads automatically save the validated UFL catalog inside pretrained/selected_model.json. Compilation checks that snapshot and the local checkpoint hash, and exports model_catalog.json with the installation. Exporters validate locally: no internet access is needed after downloading. CMake only checks that the compiled packages, entropy tables, and metadata exist; it does not require a registry catalog or Python. A later administrator removal does not invalidate an existing download’s snapshot.

For Frontier, run python3 model_registry.py caesar_v2 on the login node, then run python3 compile_model.py cuda in the GPU job. ROCm uses the cuda device name. Existing downloads made before snapshots were introduced compile directly from pretrained/selected_model.json: the complete model record becomes a one-entry local snapshot, and the checkpoint SHA-256 must still match. No login-node retry or internet connection is needed.

--catalog overrides the catalog source for a registry command. CAESAR_MODEL_CATALOG sets the download source and can supply a local catalog for older selections or installations that lack snapshots. Saved snapshots take precedence during compilation and installation validation. Snapshots are trusted local download artifacts, not cryptographically signed registration certificates.

Download accepts an exact ufl:<number>@sha256:<hash> identity as well as a registered name. --output selects the download directory; pass its selected_model.json to the compiler with --selection. The compiler’s --output selects the exported installation directory. Standard CMake packaging uses exported_model/. CAESAR_MODEL_DIR selects a compiled installation at runtime.

Downloads preserve actual checkpoint filenames, verify SHA-256, and publish a selection only after verification. Compilation checks the saved UFL catalog snapshot when present; legacy selections supply their saved model record instead. A local selection is not proof of upstream registration. All three exporters use a frozen copy of the same verified checkpoint. A staged export is published only when all three packages and six entropy tables exist. A failed export preserves the old installation. Do not replace an installation while a process is using it. An interrupted installer may leave exported_model.lock; remove that directory only after confirming the installer is no longer running. Individual exporter scripts can also be run separately for development, as described below.

Check the current UFL database later

Normal compilation uses the saved selection and checkpoint locally:

python3 compile_model.py cpu
# Or on the target GPU machine:
python3 compile_model.py cuda

An existing registered selection, including an older JSON without a catalog snapshot, does not need to be downloaded or registered again. Its saved registration is assumed locally; compilation still verifies the checkpoint hash and network compatibility. At the start of compilation, CAESAR also tries the current UFL catalog with a three-second network timeout. If this compute node cannot reach UFL, a warning says registration is unconfirmed and compilation continues using the saved local selection. An unavailable database is not treated as proof that the model is unregistered. If UFL is reachable but the checkpoint is absent, CAESAR warns to submit it and continues compilation. This check never changes identity. The complete installer checks once, rather than repeating it for each component.

On a networked login node, explicitly check the current database later:

python3 model_registry.py --check-registration
# Or check another selection:
python3 model_registry.py --check-registration /path/to/selected_model.json

The check uses the current UFL database, not the saved snapshot. It does not compile, download checkpoints, or change the selection. A missing checkpoint prints an all-caps warning and submission instructions. A retired checkpoint is identified separately. A network failure says that registration could not be checked; it does not claim the model is missing or block later local compilation. Exit statuses are 0 for present, 1 for absent/retired, and 2 for unavailable. --catalog PATH can check a local copy of the database instead. A local bypass selection whose checkpoint hash is already registered reports the UFL identity and how to select it.

Individual developer and in situ bypass

Each script accepts bypass or --bypass, independently of the device:

source venv_cpu/bin/activate  # or your target device's environment
python3 CAESAR_compressor.py cpu bypass
python3 CAESAR_decompressor.py cpu bypass
python3 CAESAR_hyper_decompressor.py cpu bypass

Replace cpu with cuda, mps, or xpu for your target machine. Each command compiles only that component. All read pretrained/selected_model.json, then load the exact filename next to that JSON and verify its SHA-256. CAESAR does not scan pretrained/ for a checkpoint or choose based on file timestamps. The .pt2 files in exported_model/ are compilation outputs, not the original training checkpoint.

Individual exports default to exported_model/, replacing only the selected component’s package. Use --output DIRECTORY and --selection PATH to override either location. They do not update installation metadata or certify a complete set. When replacing a component in an existing installation, use the same checkpoint and device as its other components. The compressor also writes the six entropy tables. Use compile_model.py when changing the installation’s model or device.

To select your own checkpoint without sending anything to UFL:

python3 model_registry.py --local pretrained/my_model.pt --bypass \
  --min-dims 3 --max-dims 5
python3 CAESAR_compressor.py cuda bypass

Local selection records the checkpoint hash, architecture label, and declared supported input ranks. It does not verify that those rank declarations have been tested. To produce a complete installation for in situ use, the wrapper also supports bypass and stages all components together:

python3 compile_model.py cuda bypass --output exported_model
python3 model_registry.py --validate-installation exported_model

Bypass works without internet access. Its UFL check is advisory; it skips catalog approval, keeps the hash and metadata checks, and still strictly loads the state dictionary. Bypassed installations use registration 0 and identity local@sha256:<hash> when no matching saved registration can be validated. A valid saved UFL registration keeps its original identity even with bypass; it does not trigger the unsupported-model warning. For unregistered models, Python prints an all-caps unsupported/unsafe warning and submission instructions; C++ warns when loading a local installation. The compressed data records the local identity, so decoding must use that same identity. Registering the same bytes later gives a different UFL identity; keep the local installation for previously encoded data.

On a networked login node, install Git and GitHub CLI, authenticate, and submit:

gh auth login
python3 register_model.py add \
  --name my_model --display-name "My model" \
  --description "Training data, intended use, and provenance" \
  --checkpoint /path/to/my_model.pt --min-dims 3 --max-dims 5

This opens the UFL_MODELS registration pull request. Bypass never uploads the checkpoint or opens a pull request. After approval, select the registered model with python3 model_registry.py my_model and compile without bypass. See registration setup and submission.

Architecture changes and remaining constraints

Previously, compilation required a catalog snapshot even if the selection and checkpoint hash were valid. Individual scripts required staging environment variables from compile_model.py, so they could not be run directly for testing. The registry checks, checkpoint integrity, and compilation setup were coupled.

Now legacy selection validation works offline, and the shared export_context() resolves either a staged installation or the arguments of one direct exporter. The complete installer retains its lock, frozen checkpoint, staging, and rollback. Local identities are accepted by the Python installation validator and C++ metadata reader without claiming a positive UFL registration. No network layers, tensor shapes, entropy coding, or serialized buffer layout changed.

The exporters still duplicate network definitions and construct a fixed caesar-bcrn-v1 configuration (dim=16, sr_dim=16, one input/output channel). The architecture label in a JSON file does not make another network compatible: strict loading rejects missing, extra, or differently shaped parameters. Supporting a new architecture requires updating all three exporters and checking their interface against the C++ runners and entropy tables. A future refactor could share the network definition to prevent these copies drifting apart.

Inspect the selected model

./build/CAESAR/caesar --model-info

This prints the selected compiled installation’s model, registration, checkpoint identity, dimensions, backend, and installation path. CAESAR_MODEL_DIR selects the installation using the normal runtime search rules.

During export, compile_model.py writes the compilation hardware name to model_device.txt beside model_metadata.txt. The executable reads this file for Build device; it does not detect the current machine’s hardware. For example, CUDA exports record the GPU name reported by PyTorch, such as NVIDIA B200, and CPU exports record the compiler machine’s CPU model. Apple MPS exports (python3 compile_model.py mps) record the Apple chip name, for example MPS: Apple M4 Max; XPU exports record the device name reported by PyTorch. The file is packaged with the installation. Older exports without it show “Not recorded”; recompile the model to record the device. The Device field remains the compiled backend.

Example from a CPU installation (names and paths depend on your machine):

Built with CAESAR model
======================
  Name          : Xcompact3d_latent_16_16
  Registration  : 5
  Device        : cpu
  Build device  : CPU: 12th Gen Intel(R) Core(TM) i7-1250U
  Dimensions    : 3D–5D
  Architecture  : caesar-bcrn-v1
  Checkpoint    : Xcompact3d_latent_16_16.pt
  Identity      : ufl:5@sha256:066b4e38c357e9506d2c123d3bc483fdecd796cf7a80183a2cd501dd7c6a38a4
  Installation  : /path/to/CAESAR/exported_model

python3 model_registry.py --list displays an aligned catalog table, followed by labeled descriptions and full identities. For example, a catalog containing the two foundation releases displays:

Available CAESAR models
=======================
Reg.  Name       Display name  Dimensions  Default
----  ---------  ------------  ----------  -------
1     caesar_v1  CAESAR v1     3D–5D
2     caesar_v2  CAESAR v2     3D–5D       *

* Catalog default (the executable uses its compiled installation).

The star marks the catalog download default. It does not mean that the executable uses that model. To change the runtime model, download and compile the desired checkpoint, then select that installation with CAESAR_MODEL_DIR before starting the executable.

Registration and identity

Identity is ufl:<registration_id>@sha256:<checkpoint_sha256>. Registration numbers, names, hashes, and full identities must be unique. Never reuse a number or replace a registered checkpoint’s bytes. A new checkpoint gets a new registration, even when it is another version of the same model family. Names and descriptions help users choose; the registration plus hash is what compressed data must record. The device is installation-specific and is not part of checkpoint identity.

Registration commands live in CAESAR; UFL_MODELS is the authoritative catalog database and stores registered checkpoints. Run python3 register_model.py add --help to submit a checkpoint or python3 register_model.py remove MODEL_NAME to request retirement. Both open a pull request to UFL_MODELS from your fork. Its workflow checks that ownership comes from the pull request author’s stable account ID, that the checkpoint is a state_dict matching the CAESAR network, and that nothing else changed, then merges it automatically. Users may retire only their own models; administrator E53klasky may retire any model. Retirement preserves the full record and reserves its identity. The checkpoint is deleted from the current branch; Git history retains older versions, so this does not shrink the full repository history.

See Automated model registration for commands, validation, one-time GitHub setup, admin rules, and consuming catalog updates. Local scripts do not enforce repository permissions; restrict direct catalog and workflow writes to trusted administrators and automation. CAESAR compilation continues to validate the UFL catalog and strictly load compatible state_dicts.

Local metadata and cache

model_metadata.txt records schema version, registration number, model name and identity, checkpoint hash and original filename, architecture, supported ranks, and compiled device. Runtime derives the artifact directory from that file’s location, allowing installation relocation. The training checkpoint need not be deployed beside the compiled packages. Descriptions and download URLs remain in the catalog. Old name-only installations must be re-exported; there is no name-only or implicit CPU fallback.

Metadata initializes once per process using C++ static initialization. Each thread has its own identity-keyed cache of lazily loaded AOTI runners and probability tables; MPI ranks have independent process memory. Use a separate Compressor or Decompressor object in each thread. Calling one from a different thread fails explicitly. Thread isolation costs extra model memory but avoids sharing a runner whose concurrent invocation is not guaranteed safe. Cached runners use shared ownership so clear() cannot invalidate handles held by existing objects. Windows retains the existing AOTI teardown workaround (cached runners are not destroyed at thread teardown).

One compiled installation is selected per process. Set CAESAR_MODEL_DIR before first use; restart the process to switch installations. Missing files in an explicit directory fail rather than falling back to another installation. All artifacts are loaded from the metadata’s directory, never combined across paths. The runtime device must match the compilation target and be available. Runtime accelerator selection cannot retarget compiled artifacts; the CLI provides no device override. Export for the intended backend with its appropriate PyTorch build, then use that matching installation. Backend availability alone does not validate correction methods or achieved NRMSE.

Runtime cache lifetime and clearing

ModelCache is an in-memory runtime cache, separate from downloaded checkpoints in pretrained/ and compiled artifacts in exported_model/. It does not cache input tensors, compressed results, or reconstructed fields. Each thread lazily loads and reuses its own compressor, hyper-decompressor, decoder, and six probability tables. AOTI package-loader construction is serialized across threads; that does not make a compressor/decompressor object callable from another thread.

Reuse objects on their creating thread for repeated fields. The first operation that needs a package/table pays its load cost; warm runs reuse it. Each additional worker thread or MPI rank can add model/table memory. There is no automatic LRU, capacity setting, or cross-thread/rank model sharing. Compressor/decompressor objects also hold copies of probability tables and shared runner handles.

Advanced callers can clear the cache of the calling thread:

#include <caesar/models/model_cache.h>

// First destroy compressor/decompressor objects in this thread if the aim is
// to release their model handles as well.
ModelCache::instance().clear();

clear() resets the cached three runner handles and six probability tables. Existing objects retain their runner handles and table copies and remain usable; subsequent cache getters reload artifacts. Clearing does not reset process-wide model metadata, select another model/device, delete files, or clear another thread’s cache. It does not guarantee an immediate drop in GPU/process memory, since live handles and backend allocators may retain allocations. On Windows, the thread-local registry is intentionally retained at teardown to avoid unsafe AOTI destruction; explicit clearing still resets its entries’ loaded resources.

Serialized model identity and external integration

The version-1 CAESAR serializer stores CompressionResult.model_id with every result, together with shape, normalization, padding, encoded streams, and correction metadata/payloads. caesar::serialize/deserialize handle buffers; caesar::save/load handle single-file storage. Decompression checks the saved identity against the installed model before decoding and rejects a mismatch. The model packages and probability tables remain in the matching installation. See the public API.

The separately maintained ADIOS2 operator can wrap this buffer with a length prefix and pass the deserialized result to Decompressor::decompress. ADIOS2 migration notes describe that integration; its external source and validation are outside this checkout. Existing name-only files cannot establish an exact checkpoint identity, and the version-1 serializer does not read legacy ADIOS2 buffers or the CLI’s three-file layout.

Input preparation

Padding and conversion to the model’s 5D input are internal CAESAR responsibilities. The 3D–5D catalog range describes user input ranks handled through CAESAR’s internal preparation; it is not an assertion that the exported network accepts arbitrary shapes directly. The version-1 serializer preserves every PaddingInfo field, including H, W, and was_padded; changing these fields requires considering the serialized format as well as input preparation.

Verification

python3 -m unittest discover -s tests -p test_model_registry.py -v
python3 tests/check_registered_checkpoints.py --source-dir ../UFL_MODELS
./build/tests/test_model_metadata
CAESAR_MODEL_DIR="$PWD/exported_model" ./build/tests/test_model_cache

The checkpoint check strictly loads all active registrations into each of the three export models. The CPU installation and test_caesarCD exercise actual compiled inference. Metadata tests cover missing/mismatched identities and malformed manifests; registry tests cover duplicates, unregistered selections, hash failures, and preserving the previous installation when compilation fails. Cache tests cover reuse, separate thread runners, and retaining live handles across clear().