Reading HDF5 Results Outside Julia

Simulation histories and logs are intended to be useful in ordinary HDF5 tools outside Julia. This page defines the public groups and datasets that external readers can rely on. Additional entries may be present for Julia reconstruction or other implementation needs. Readers should use the documented entries and tolerate extra entries, rather than require an exact set of keys.

SystemsOfSystems defines the structure that identifies runs, models, variables, and their metadata. Sample values use the documented HDF5Vectors storage format. Its portable data and schema entries are also available to external readers; they are not Julia implementation details. The distinction is between documented, externally interpretable data and entries needed only to reconstruct Julia objects.

Format Versions and Compatibility

History metadata and logs have independent integer format versions because either can be used without the other. These describe the stored layout, not the SystemsOfSystems package release. HDF5Vectors versions its nested value encodings separately.

Each loader checks its group's version before interpreting its contents or deserializing Julia objects. A missing history version, a malformed version, or an unsupported explicit version produces an ArgumentError identifying the group and the problem. Unversioned logs are the one deliberate exception: the existing legacy log reader handles them, whether stored at the file root or in a selected group. Missing versions do not provide a general way to infer future layouts.

Newly written history and log groups also carry a scalar string dataset, systems_of_systems_version, identifying the writer's package release. This is provenance, not a compatibility gate. It is optional when reading, including for files written before provenance was recorded. A newer writer package version does not itself make a supported format unreadable.

Saving an in-memory log writes the current log format. Copying an existing HDF5 log preserves its format version and writer provenance, including absent datasets on a legacy log. The same applies when saving history metadata beside an existing log. Neither operation converts the log to a newer format; new history metadata records its own current version and writer independently.

Support Policy

SystemsOfSystems uses independent format versions with bounded support. Within a package major release, newer releases retain the supported-format readers provided by earlier releases in that major. A future package major release may drop support for older formats; permanent backward reading is not promised. The support table above and release notes identify supported formats and any removals. A package major release need not change a format version if the storage layout remains compatible.

Format versions change when the interpretation of public data changes incompatibly. Adding optional entries that readers can ignore, or changing compression and chunk sizes, does not require a new format version. Renaming or removing a public entry, changing its meaning, or changing its representation incompatibly does. Such changes to the existing public writer interface belong in a package major release. Configurable group locations are already part of the format and do not require a version change.

Support applies to the documented layout and its portable data. It does not guarantee reconstruction of arbitrary application types or compatibility of Julia-serialized objects across Julia or dependency versions. Changes to internal reconstruction entries do not automatically require a public format-version change, but they still need to respect the package's supported loading behavior. Nested HDF5Vectors encodings have their own compatibility requirements.

Locating a Run and Its Log

By default, a saved history has two groups:

/history/                  # Run metadata
/log/                      # Root model's logged data

These names are defaults, not mandatory locations. A caller may place either group elsewhere. An external reader starts from the chosen history group and follows its log_path dataset to the log. A standalone log has no history group; its location is supplied directly, with /log as the default for new files.

The following entries are relative to the history group:

All entries except cleanup_errors, model, and the provenance dataset are required in a saved history. New files include cleanup_errors/count, even when it is zero. Within each cleanup entry, context and details are required, while type is optional for compatibility with older files. Type names and descriptions are descriptive text: their wording is not a machine-readable classification scheme.

The Julia history loader constructs a TerminationSummary directly from the six stop datasets. It preserves the results of finished, failed, interrupted, describe, and details, along with the original type name as descriptive text. It does not reconstruct the original termination reason.

Cleanup entries likewise load directly as CleanupErrorSummary values. Their context and details datasets preserve the results of cleanup_context and cleanup_details. The summaries retain the saved type labels and error order when re-saved. No exception objects or stack traces are reconstructed.

Because log_path is within the file, moving or renaming the file preserves it. Moving a log group within the file requires updating any history that refers to it. A log group with is_null = true records that logging was disabled; it has no model tree. Ordinary logs need not have an is_null entry.

Models and Variables

The log group carries the scalar integer dataset log_format_version and, for newly written logs, the scalar string provenance dataset systems_of_systems_version. These datasets describe the whole log and are not repeated on child models. Legacy unversioned logs may lack both datasets.

For an ordinary log, this group also represents the root model. Each model has the following public entries, relative to its own group:

The names datasets are present even when their lists are empty. models and timeseries may be absent when there are no corresponding entries. Readers should follow the name lists rather than infer model order or variable categories from HDF5 group iteration. Logging policies can omit variables, and unsupported constants are omitted with a warning when saving.

For example, the time series position belonging to model /vehicle/controller is at:

/log/models/vehicle/models/controller/timeseries/position/

If the log lives at /runs/first/samples, the same relative path starts there instead. Model paths used by the simulation do not include the storage group names or the repeated models components.

Time Series

Each timeseries/<name> group exposes these entries:

The time and data vectors have the same logical length for a successfully recorded series. Different variables may have different sample times; each series has its own time vector. The times are actual logged samples and need not include the history's final t_stop.

Dimension group membership is reconstructed by splitting group_dimension_labels according to group_dimension_counts, in the order of group_labels. For example, counts [2, 1] divide three labels into a two-dimension group and a one-dimension group. When groups_are_missing is true, grouping was unspecified; this differs from an explicitly empty grouping.

The interpolator's name is useful when inspecting the series, but does not encode all parameters or define how another language should execute a custom interpolator. Reading the recorded samples does not require reconstructing it.

Sample Storage

time and data are vector groups, not necessarily datasets. For the HDF5Vectors version 1 format, each contains public metadata/count, metadata/format_name, metadata/format_version, and metadata/schema entries describing its length and encoding. The physical values are beneath its own data group. This accounts for the repeated data component in the paths below:

timeseries/position/
├── time/
│   ├── metadata/ ...
│   └── data/values             # Float64 sample times
└── data/
    ├── metadata/ ...
    └── data/values             # Scalar or dense-array samples

For scalar numbers, data/data/values is a one-dimensional dataset. For a fixed-size vector sample with three components, the raw HDF5 dataset has shape (sample_count, 3). HDF5.jl reverses dimensions at the HDF5 C boundary: matrix samples of Julia size (2, 3) have raw HDF5 shape (sample_count, 3, 2). This is an array-ordering convention, not a transpose applied to the model's values. The HDF5Vectors storage documentation describes how these extents relate to the Julia view.

Portable record samples use field names beneath data/data instead of a single values dataset. For example, a record with a scalar field temperature has a column at data/data/temperature/values; nested records repeat their field names. Readers can inspect data/metadata/schema/kind and its child schemas to determine the representation, rather than assume every variable uses data/data/values. The HDF5Vectors documentation defines the complete scalar, dense, record, and other layouts.

Some values, such as dynamically sized arrays without declared dimensions, may be encoded with Julia serialization. Their time series and metadata are still discoverable, but their payload cannot be reconstructed by an ordinary HDF5 reader. The public layout does not imply that every possible Julia value has a portable encoding. Applications that need cross-language access should select data types and storage options with portable HDF5Vectors representations.

Constants

Each constants/<name> group contains a value HDF5Vectors group and ordinary metadata datasets:

Constants supplied with VariableDescription also have the grouping and interpolator_type entries described for time series. Ordinary constants need not have those entries. External readers can inspect the value's HDF5Vectors schema without using value_type to reconstruct a Julia type.

Julia Reconstruction Entries

The public entries above are sufficient to navigate a run and read its portably encoded log data. Other entries are not part of the SystemsOfSystems external-reader contract. In particular, serialized_type, serialized_value_type, and serialized_interpolator store Julia reconstruction information. The is_variable_description flag selects a Julia wrapper when loading a constant. These entries may change without changing the public layout.

The public HDF5Vectors groups for logged values and the optional final model can be inspected according to their ordinary stored schemas. SystemsOfSystems does not promise one fixed set of fields for arbitrary application models: the schema describes the value actually saved. Termination information has the fixed layout of six ordinary stop datasets described above. Julia-serialized schemas and payloads remain Julia-specific, even when stored inside an otherwise public group.

Adding entries does not invalidate this layout. Existing public paths and their meanings are compatibility commitments, while unlisted entries may change. Readers should also honor the separate HDF5Vectors format version before interpreting vector storage. This page describes newly written files; older standalone logs may use a root-level model layout or lack metadata introduced in later releases.