API Reference

HDF5Vector{T} <: AbstractVector{T}

An append-only HDF5-backed vector with logical element type T. It supports ordinary AbstractVector reads and grows with push!. Vectors are normally constructed with create_hdf5_vector, copy_to_hdf5_vector, or load_hdf5_vector, and remain usable only while their HDF5 file is open.

create_hdf5_vector(
    group::HDF5.Group,
    name::AbstractString,
    type::Type;
    dims = nothing,
    chunk_length = 1000,
    portable = true,
    serialize_arrays = true,
    serialize_nonconcrete = true,
)

Creates an empty HDF5Vector{type} in the new child name of group. name must be one unused HDF5 path component. Schema inference selects and records the complete physical representation before the vector is returned.

dims declares the fixed shape of a dynamically sized array element. chunk_length controls the extensible dimension of physical HDF5 chunks. portable = true gives bits-type records field-oriented storage. serialize_arrays and serialize_nonconcrete control whether schema inference may select Julia serialization for those categories.

Values later passed to push! must already have exactly the declared element type.

create_hdf5_vector(
    group::HDF5.Group,
    name::AbstractString,
    schema::AbstractSchema;
    chunk_length = 1000,
)

Creates an empty HDF5-backed vector using an already selected schema. The schema determines the logical type, codecs, and physical representation; only chunk_length remains a creation option.

copy_to_hdf5_vector(
    group::HDF5.Group,
    name::AbstractString,
    collection::AbstractVector;
    kwargs...,
)

Creates an HDF5-backed vector and copies collection into it with one recursive bulk write. The declared element type is eltype(collection), and the creation options have the same meaning as for create_hdf5_vector.

Destination validation, schema inference, and encoding of the complete collection finish before the destination group is created. The input must currently be an AbstractVector.

load_hdf5_vector(group::HDF5.Group)
load_hdf5_vector(group::HDF5.Group, type::Type)
load_hdf5_vector(group::HDF5.Group, schema::AbstractSchema)

Opens and validates an existing HDF5 vector group.

The one-argument form deserializes the exact stored schema. The explicit-type form repeats schema inference from the options stored when the vector was created from a type, avoiding schema deserialization. A vector created from an explicit schema still needs that stored schema unless the caller supplies a schema directly. A supplied schema is authoritative: loading validates that it can read the physical layout, but does not require all of its logical details to equal the stored schema. This can help load a file after its Julia type or schema implementation has changed.

The returned vector remains usable only while the HDF5 file is open. Files whose schema or values require Julia deserialization should be trusted.

SchemaPolicy(;
    portable = true,
    serialize_arrays = true,
    serialize_nonconcrete = true,
)

Controls default schema inference without becoming part of storage execution.

portable selects field-oriented records instead of native HDF5 representations for nonzero-size bits types. Arrays without declared dimensions and nonconcrete declared types can use Julia serialization when their corresponding policy fields are true.

infer_schema(type::Type; dims = nothing, policy = SchemaPolicy())

Builds a complete storage schema without opening or modifying an HDF5 file. The result is the representation plan used by later creation, writing, and loading operations.

Applications can extend this function for a logical type and return a schema composed from the built-in physical representations. Recursive inference always returns through this public method, so the extension applies equally to a root vector, a record field, or a dense element type.

For example, a Grade stored as one UInt8 needs only a codec and this selection method:

struct GradeCodec <: HDF5Vectors.AbstractCodec{Grade, UInt8} end

HDF5Vectors.infer_schema(::Type{Grade}; kwargs...) =
    HDF5Vectors.ScalarSchema(GradeCodec())

The codec implements encode_value and decode_value. Its concrete type is stored with the schema, so no package-owned codec-name registry is needed when loading.

json_schema(type::Type)

Returns a scalar schema that stores each value as a JSON string. JSON3 must be loaded before values can be encoded or decoded.

serialization_schema(type::Type)

Builds an explicit Julia-serialization schema for type.

An explicit, recursive description of the logical value type, its encoding, and its physical storage shape.

Schemas contain no open HDF5 objects. They can therefore be inferred, inspected, and tested before an HDF5 destination is created.

A new physical schema implements one vertical protocol:

• encode_value, decode_value, and encoded_value_type describe pure conversion.
• write_schema_node and validate_schema_node describe stored metadata.
• create_store, open_store, and physical_length manage its HDF5 representation.
• initialize_encoded! writes a complete encoded batch into a newly created empty store.
• append_encoded!, read_encoded, and read_encoded_batch provide append and read access.

Batch conversion and storage methods can specialize the generic fallbacks when the physical representation supports more efficient whole-column I/O. The files under representations contain the complete built-in implementations grouped according to this protocol.

ScalarSchema(codec::AbstractCodec)

Describes one logical value encoded as one scalar HDF5-compatible value.

DenseSchema(type, dims, element_codec)

Describes fixed-size array-like values whose elements share one scalar codec. Physical storage stacks the encoded values along one additional final dimension.

RecordSchema(type, names, codec, children)

Describes a field-oriented value with ordered field names, a record codec, and one recursive child schema per field.

BlobSchema(codec::AbstractCodec{T, Vector{UInt8}})

Describes variable-length byte values stored in concatenated bytes and cumulative stops datasets.

ConstantSchema(codec::AbstractCodec{T, Nothing})

Describes a logical type with one stored constant value and no per-element physical data.

logical_type(codec_or_schema)

Returns the Julia value type represented by a codec or schema.

encoded_type(codec_or_schema)

Returns the scalar HDF5-compatible type produced by a scalar or dense codec or schema.

encoded_value_type(schema::AbstractSchema)

Returns the Julia type of one complete encoded logical value. This differs from encoded_type for schemas whose encoded value is a container or record.

encode_batch(schema::AbstractSchema, values::AbstractVector)

Encodes a complete logical vector before physical storage is initialized. The default returns a vector of individually encoded values; schemas can specialize this method when their physical representation has a more efficient batch shape.

decode_batch(schema::AbstractSchema, encoded)

Decodes a batch returned by the schema's physical store. The default decodes an ordinary vector element-by-element.

A pure conversion between a Julia value of type T and an encoded value of type H.

Codecs know nothing about HDF5 objects or vector operations. This makes their round-trip behavior testable independently of physical storage. An application codec needs encode_value and decode_value methods, and its logical type selects the completed schema through infer_schema. The stored schema serializes the codec itself for exact untyped loading; codec_identifier is descriptive metadata rather than a registry.

A pure conversion between a Julia record value and its ordered logical fields.

Each field is encoded recursively by its own schema after decompose runs. compose performs the inverse operation after those fields have been decoded.

encode_value(codec_or_schema, value)

Converts one logical Julia value into the representation described by a codec or schema. Custom codec methods should be pure and should not access HDF5 objects.

decode_value(codec_or_schema, encoded)

Reconstructs one logical Julia value from the representation produced by encode_value.

decompose(codec::AbstractRecordCodec, value)

Returns the ordered logical fields of one record value. The result must contain one value for each child of the corresponding RecordSchema.

compose(codec::AbstractRecordCodec, fields::Tuple)

Reconstructs one logical record value from its ordered, recursively decoded fields.

A codec that stores a supported Julia scalar without conversion.

A codec that stores a Char as its Int32 Unicode code point.

A codec that stores a Symbol as a String.

A codec that stores an enum using its integer base type H.

A codec that stores a logical value of type T as a JSON string using JSON3.

A codec that uses Julia Serialization to encode a value as bytes.

The default record codec for a concrete struct with N fields.

The built-in record codec for a heterogeneous tuple.

The built-in record codec for a named tuple.

The record codec used when a static array cannot use dense storage.

A codec that represents every element with one stored constant value.

schema_identifier(schema::AbstractSchema)

Returns the stable, human-readable identifier stored for a schema implementation. The default uses the implementation type. An application can specialize this function when the identifier must remain stable across a type rename.

codec_identifier(codec)

Returns the stable, human-readable identifier stored for a codec implementation. The default uses the implementation type. An application can specialize this function when the identifier must remain stable across a type rename.

write_schema(group::HDF5.Group, schema::AbstractSchema)

Writes the versioned logical type and complete storage schema into a new metadata child of group. Custom schema and codec implementations are serialized as ordinary Julia metadata and also describe themselves through the extensible schema-node interface.

read_schema(group::HDF5.Group)
read_schema(group::HDF5.Group, type::Type)
read_schema(group::HDF5.Group, schema::AbstractSchema)

Reads a schema and validates it against the ordinary stored schema description. Untyped loading deserializes the stored schema, allowing application-defined codecs to reconstruct without a package-owned registry. Typed loading repeats public schema inference when the vector was created from a type. Supplying an explicit schema treats that schema as authoritative and avoids metadata deserialization entirely. This can allow a caller to load physical data after the corresponding Julia type or schema implementation has changed.

Writes one schema implementation's readable metadata node.

Validates one stored metadata node against a selected schema implementation.

Writes the metadata fields shared by every schema node.

Validates the metadata fields shared by every schema node.

Writes the scalar encoded type used by a scalar or dense schema.

Validates a scalar or dense schema node's stored encoded type.

Writes a schema node's human-readable codec identifier.

Validates a schema node's stored codec identifier.

An open physical HDF5 representation that reads and writes values already encoded by an AbstractSchema. Stores contain no logical type conversion.

Creates an empty physical store beneath an HDF5 vector's data group.

Opens and validates a physical store beneath an HDF5 vector's data group.

Returns a store's logical physical length, or nothing when it has no payload length.

Initializes a newly created empty store from a complete encoded batch.

Appends one encoded value at the supplied next logical index.

Reads one encoded value or an encoded unit range from a physical store.

read_encoded_batch(store::AbstractStore, indices::UnitRange{Int})

Reads the encoded batch consumed by decode_batch. The default uses the store's unit-range read_encoded method.

Validates a prepared batch before a recursively nested store begins initialization.