tenferro_hdf5/lib.rs
1//! Thin HDF5 payload adapter for tenferro tensors.
2//!
3//! This crate stores ordinary tenferro values in HDF5 datasets: a
4//! [`TypedTensor<T, R, Host>`](tenferro_tensor::TypedTensor), the same tensor
5//! once it became [`Dynamic`](tenferro_tensor::Dynamic), and the dtype-erased
6//! [`Tensor`](tenferro_tensor::Tensor). It is an optional adapter: no tenferro
7//! compute crate depends on it, and it never creates an engine, a thread pool or
8//! an HDF5 runtime of its own.
9//!
10//! # Features
11//!
12//! `default = []`; the adapter is enabled by the `hdf5` feature, which pulls in the
13//! `hdf5-rt` binding. A default workspace build therefore contains no HDF5 edge and
14//! can run on a host without `libhdf5`. The crate is unpublished (`publish = false`)
15//! until `hdf5-rt` is on crates.io.
16//!
17//! # What it does not do
18//!
19//! - **No hidden transfer.** A tensor whose payload lives in a backend buffer is
20//! rejected with [`Error::NotHostResident`]; nothing is downloaded, uploaded,
21//! synchronized or materialized.
22//! - **No hidden copies.** A compact column-major view is written straight from
23//! its host buffer, and a read allocates exactly one payload buffer, which the
24//! returned tensor adopts. A non-compact view is rejected with
25//! [`Error::NotCompact`]; compact it explicitly if that is what you want.
26//! - **No numeric conversion.** The dtype, shape and values round-trip
27//! unchanged, bit patterns included. There is no `f32` to `f64` conversion and
28//! no integer widening.
29//! - **No path policy.** The caller owns the [`hdf5::File`] and [`hdf5::Group`]
30//! handles, and the adapter only touches the one link name it is given.
31//!
32//! # Backend
33//!
34//! One backend: [`hdf5-rt`](https://github.com/tensor4all/hdf5-rt), re-exported
35//! here as [`hdf5`], so callers always pass handles of the crate this adapter
36//! uses. `hdf5-rt` loads `libhdf5` at run time (Linux: the bare `libhdf5.so` and
37//! `libhdf5_serial.so` first, so `LD_LIBRARY_PATH` and `ld.so.conf` are honoured;
38//! then the usual serial-install paths), needs no library at build time, and
39//! panics inside the first HDF5 call if nothing loads. On Linux install it with
40//! `apt-get install libhdf5-dev` or use a conda-forge `hdf5`.
41//!
42//! # Schema (version 1)
43//!
44//! One dataset per tensor, linked as `name` in the caller's group, with three
45//! scalar attributes: `type = "tenferro.Tensor"`, `version = 1`, and `dtype`
46//! one of `f32 f64 c32 c64 i32 i64 bool`.
47//!
48//! The HDF5 dataspace holds the shape **reversed** (rank 0 becomes a scalar
49//! dataspace). HDF5 extents are C-order and tenferro storage is column-major, so
50//! the reversed extents make the stored payload identical to the column-major
51//! buffer, and both directions are copy-free. In h5py the logical tensor is
52//! therefore `np.asarray(dataset).T`.
53//!
54//! | dtype | HDF5 type | h5py reads it as |
55//! |---|---|---|
56//! | `f32`, `f64` | `H5T_IEEE_F32LE`, `F64LE` | `float32`, `float64` |
57//! | `i32`, `i64` | `H5T_STD_I32LE`, `I64LE` | `int32`, `int64` |
58//! | `c32`, `c64` | compound `{"r": F, "i": F}` | `complex64`, `complex128` |
59//! | `bool` | int8 enum `{FALSE=0, TRUE=1}` | `bool` |
60//!
61//! [`DType::External`](tenferro_tensor::DType) payloads have no encoding and are
62//! rejected with [`Error::UnsupportedDType`]; a Rust `TypeId` is never persisted.
63//! Overwriting is the caller's decision: a taken name fails with
64//! [`Error::AlreadyExists`], and replacing means unlinking the link first.
65//!
66//! # Example
67//!
68//! The adapter is behind the `hdf5` feature (`default = []`), so this example is
69//! compiled in the `--features hdf5` lane:
70//!
71//! ```
72//! # #[cfg(feature = "hdf5")]
73//! # fn example() -> Result<(), Box<dyn std::error::Error>> {
74//! use tenferro_hdf5::{hdf5, read_info, read_typed, write_typed, ReadOptions};
75//! use tenferro_tensor::{DynRank, Host, TypedTensor};
76//!
77//! let dir = tempfile::tempdir()?;
78//! let file = hdf5::File::create(dir.path().join("tensors.h5"))?;
79//! let group = file.create_group("tensors")?;
80//!
81//! let tensor: TypedTensor<f64, DynRank, Host> =
82//! TypedTensor::from_host_vec_col_major(vec![2, 3], (0..6).map(f64::from).collect())?;
83//! write_typed(&group, "a", tensor.as_view())?;
84//!
85//! let info = read_info(&group, "a")?;
86//! assert_eq!(info.shape, vec![2, 3]);
87//!
88//! let back: TypedTensor<f64, DynRank, Host> =
89//! tenferro_hdf5::read_typed_with(&group, "a", &ReadOptions::default().max_elements(1 << 20))?;
90//! assert_eq!(back.as_slice(), tensor.as_slice());
91//! # Ok(())
92//! # }
93//! # #[cfg(feature = "hdf5")]
94//! # fn main() { example().unwrap(); }
95//! # #[cfg(not(feature = "hdf5"))]
96//! # fn main() {}
97//! ```
98#![deny(missing_docs)]
99
100#[cfg(feature = "hdf5")]
101mod error;
102#[cfg(feature = "hdf5")]
103mod read;
104#[cfg(feature = "hdf5")]
105mod schema;
106#[cfg(feature = "hdf5")]
107mod write;
108
109/// The HDF5 binding this adapter is built on.
110///
111/// Callers open files and groups through this re-export, so their handles are
112/// always the ones the adapter expects.
113#[cfg(feature = "hdf5")]
114pub use hdf5_rt as hdf5;
115#[cfg(feature = "hdf5")]
116pub use hdf5_rt::H5Type;
117
118#[cfg(feature = "hdf5")]
119pub use error::{Error, SchemaViolation};
120#[cfg(feature = "hdf5")]
121pub use read::{
122 read_info, read_tensor, read_tensor_with, read_typed, read_typed_with, ReadOptions, TensorInfo,
123};
124#[cfg(feature = "hdf5")]
125pub use write::{write_tensor, write_typed};
126
127/// Value of the `type` attribute that marks a tenferro tensor dataset.
128#[cfg(feature = "hdf5")]
129pub const SCHEMA_TYPE: &str = "tenferro.Tensor";
130
131/// Schema version this adapter writes and reads.
132#[cfg(feature = "hdf5")]
133pub const SCHEMA_VERSION: i64 = 1;