Expand description
Thin HDF5 payload adapter for tenferro tensors.
This crate stores ordinary tenferro values in HDF5 datasets: a
TypedTensor<T, R, Host>, the same tensor
once it became Dynamic, and the dtype-erased
Tensor. It is an optional adapter: no tenferro
compute crate depends on it, and it never creates an engine, a thread pool or
an HDF5 runtime of its own.
§Features
default = []; the adapter is enabled by the hdf5 feature, which pulls in the
hdf5-rt binding. A default workspace build therefore contains no HDF5 edge and
can run on a host without libhdf5. The crate is unpublished (publish = false)
until hdf5-rt is on crates.io.
§What it does not do
- No hidden transfer. A tensor whose payload lives in a backend buffer is
rejected with [
Error::NotHostResident]; nothing is downloaded, uploaded, synchronized or materialized. - No hidden copies. A compact column-major view is written straight from
its host buffer, and a read allocates exactly one payload buffer, which the
returned tensor adopts. A non-compact view is rejected with
[
Error::NotCompact]; compact it explicitly if that is what you want. - No numeric conversion. The dtype, shape and values round-trip
unchanged, bit patterns included. There is no
f32tof64conversion and no integer widening. - No path policy. The caller owns the [
hdf5::File] and [hdf5::Group] handles, and the adapter only touches the one link name it is given.
§Backend
One backend: hdf5-rt, re-exported
here as [hdf5], so callers always pass handles of the crate this adapter
uses. hdf5-rt loads libhdf5 at run time (Linux: the bare libhdf5.so and
libhdf5_serial.so first, so LD_LIBRARY_PATH and ld.so.conf are honoured;
then the usual serial-install paths), needs no library at build time, and
panics inside the first HDF5 call if nothing loads. On Linux install it with
apt-get install libhdf5-dev or use a conda-forge hdf5.
§Schema (version 1)
One dataset per tensor, linked as name in the caller’s group, with three
scalar attributes: type = "tenferro.Tensor", version = 1, and dtype
one of f32 f64 c32 c64 i32 i64 bool.
The HDF5 dataspace holds the shape reversed (rank 0 becomes a scalar
dataspace). HDF5 extents are C-order and tenferro storage is column-major, so
the reversed extents make the stored payload identical to the column-major
buffer, and both directions are copy-free. In h5py the logical tensor is
therefore np.asarray(dataset).T.
| dtype | HDF5 type | h5py reads it as |
|---|---|---|
f32, f64 | H5T_IEEE_F32LE, F64LE | float32, float64 |
i32, i64 | H5T_STD_I32LE, I64LE | int32, int64 |
c32, c64 | compound {"r": F, "i": F} | complex64, complex128 |
bool | int8 enum {FALSE=0, TRUE=1} | bool |
DType::External payloads have no encoding and are
rejected with [Error::UnsupportedDType]; a Rust TypeId is never persisted.
Overwriting is the caller’s decision: a taken name fails with
[Error::AlreadyExists], and replacing means unlinking the link first.
§Example
The adapter is behind the hdf5 feature (default = []), so this example is
compiled in the --features hdf5 lane:
use tenferro_hdf5::{hdf5, read_info, read_typed, write_typed, ReadOptions};
use tenferro_tensor::{DynRank, Host, TypedTensor};
let dir = tempfile::tempdir()?;
let file = hdf5::File::create(dir.path().join("tensors.h5"))?;
let group = file.create_group("tensors")?;
let tensor: TypedTensor<f64, DynRank, Host> =
TypedTensor::from_host_vec_col_major(vec![2, 3], (0..6).map(f64::from).collect())?;
write_typed(&group, "a", tensor.as_view())?;
let info = read_info(&group, "a")?;
assert_eq!(info.shape, vec![2, 3]);
let back: TypedTensor<f64, DynRank, Host> =
tenferro_hdf5::read_typed_with(&group, "a", &ReadOptions::default().max_elements(1 << 20))?;
assert_eq!(back.as_slice(), tensor.as_slice());