Skip to main content

tenferro_cpu/
batch_policy.rs

1//! Overridable batch execution strategy for batched CPU operations (#2142).
2//!
3//! A batched CPU operation (a strided batched contraction or a
4//! packed-LU batch) splits its independent items over the thread budget the
5//! lower library receives on `tcontract_exec::Exec`. The vocabulary for that
6//! split lives in `tcontract_exec` and is re-exported here so callers keep a
7//! `tenferro_cpu` spelling:
8//!
9//! * [`CpuBatchPolicy`] is `tcontract_exec::BatchPolicy` (a [`CpuBatchStrategy`]
10//!   plus the [`CpuBatchThresholds`] the `Auto` arm uses),
11//! * [`CpuBatchStrategy`] is `tcontract_exec::BatchStrategy`,
12//! * [`CpuBatchThresholds`] is `tcontract_exec::WorkModel`.
13//!
14//! Two variants were renamed when the vocabulary moved into `tcontract-exec`:
15//! the old `OuterParallel` is [`CpuBatchStrategy::Outer`], and the old
16//! `ProviderItems` is [`CpuBatchStrategy::Inner`]. `Sequential` and `Auto` kept
17//! their names. The old vendor-dimension cutoff is deliberately not restored: it
18//! was an inert knob once the decision moved into the lower library.
19//!
20//! # Precedence
21//!
22//! A batched operation resolves its policy as the innermost scoped override,
23//! then the [`crate::CpuEngine`] default:
24//!
25//! 1. the policy of an enclosing [`with_batch_policy`] scope, if any;
26//! 2. otherwise [`crate::CpuEngine::batch_policy`], whose default is
27//!    [`CpuBatchStrategy::Auto`].
28//!
29//! Wrapping one call in [`with_batch_policy`] therefore expresses a
30//! per-operation choice. A non-CPU engine is rejected with a typed
31//! [`tenferro_tensor::ErrorKind::Unsupported`] error; there is no process-global
32//! or thread-local policy outside the scope callback, and each scope restores
33//! the previous policy on return, error and unwind.
34//!
35//! # Route-specific `Auto`
36//!
37//! `Auto` is resolved inside the lower library and keeps that library's legacy
38//! rule until the upstream shared model lands (tcontract-rs#123 step 3):
39//! `tlinalg`'s 64-extent cutoff that counts RHS columns in the linalg lane,
40//! and `cpueinsum`'s width rule in the contraction lane. `CpuBatchThresholds`
41//! feeds those rules where the lower library reads the policy's model; a forced
42//! strategy ignores the model and never falls back.
43//!
44//! # Forced strategies
45//!
46//! A forced strategy is always honoured: the linalg entry resolves it against
47//! the lane budget the caller supplied and never quietly substitutes `Auto`.
48
49/// How the items of one batched operation are split over a thread budget.
50///
51/// This is `tcontract_exec::BatchStrategy`. The old `OuterParallel` variant is
52/// [`CpuBatchStrategy::Outer`] and the old `ProviderItems` variant is
53/// [`CpuBatchStrategy::Inner`]; `Sequential` and `Auto` kept their names.
54///
55/// # Examples
56///
57/// ```rust
58/// use tenferro_cpu::CpuBatchStrategy;
59///
60/// assert_eq!(CpuBatchStrategy::default(), CpuBatchStrategy::Auto);
61/// ```
62pub use cpueinsum::BatchStrategy as CpuBatchStrategy;
63
64/// A batch strategy together with the [`CpuBatchThresholds`] `Auto` applies.
65///
66/// This is `tcontract_exec::BatchPolicy`.
67///
68/// # Examples
69///
70/// ```rust
71/// use tenferro_cpu::{CpuBatchPolicy, CpuBatchStrategy};
72///
73/// let policy = CpuBatchPolicy::new(CpuBatchStrategy::Outer);
74/// assert_eq!(policy.strategy, CpuBatchStrategy::Outer);
75/// ```
76pub use cpueinsum::BatchPolicy as CpuBatchPolicy;
77
78/// The cost and fan-out model [`CpuBatchStrategy::Auto`] applies.
79///
80/// This is `tcontract_exec::WorkModel`; the old vendor-dimension cutoff is not
81/// part of it.
82///
83/// # Examples
84///
85/// ```rust
86/// use tenferro_cpu::CpuBatchThresholds;
87///
88/// assert_eq!(CpuBatchThresholds::default().min_items, 2);
89/// ```
90pub use cpueinsum::WorkModel as CpuBatchThresholds;
91
92/// Run `f` on `engine` with `policy` as the effective CPU batch policy.
93///
94/// This is the scoped override of the precedence described in the [module
95/// docs](self): wrapping a single call expresses a per-operation choice. The
96/// previous policy is restored when `f` returns, returns an error or unwinds,
97/// and scopes nest with the innermost winning. `f` runs on the caller's own
98/// engine, so a forwarding engine keeps its own overrides inside the scope.
99/// Nothing here mutates process-global state.
100///
101/// # Examples
102///
103/// ```rust
104/// use tenferro_cpu::{with_batch_policy, with_cpu_engine, CpuBatchPolicy, CpuBatchStrategy, CpuEngine};
105///
106/// let mut engine = CpuEngine::serial();
107/// let observed = with_batch_policy(
108///     &mut engine,
109///     CpuBatchPolicy::new(CpuBatchStrategy::Outer),
110///     |engine| {
111///         with_cpu_engine(engine, |cpu| {
112///             cpu.with_linalg_pool(|context, _| Ok(context.batch_policy().strategy))
113///         })
114///         .expect("a CPU engine")
115///     },
116/// )??;
117/// assert_eq!(observed, CpuBatchStrategy::Outer);
118/// assert_eq!(engine.batch_policy().strategy, CpuBatchStrategy::Auto);
119/// # Ok::<(), Box<dyn std::error::Error>>(())
120/// ```
121///
122/// # Errors
123///
124/// Returns [`tenferro_tensor::Error::Unsupported`] without running `f` when
125/// `engine` is neither a [`crate::CpuEngine`] nor an engine that forwards one
126/// through [`tenferro_tensor::Engine::native_engine`] (a CUDA or other
127/// custom engine). `f`'s own result is returned unchanged inside `Ok`.
128///
129/// # Panics
130///
131/// A panic in `f` propagates after the previous policy is restored.
132pub fn with_batch_policy<R>(
133    engine: &mut dyn tenferro_tensor::Engine,
134    policy: CpuBatchPolicy,
135    f: impl FnOnce(&mut dyn tenferro_tensor::Engine) -> R,
136) -> tenferro_tensor::Result<R> {
137    let Some(previous) = crate::with_cpu_engine(engine, |cpu| cpu.replace_batch_policy(policy))
138    else {
139        return Err(tenferro_tensor::Error::unsupported(
140            "tenferro_cpu::with_batch_policy",
141            "the engine is not a CPU engine and does not forward one; CPU batch policies apply \
142             only to CpuEngine",
143        ));
144    };
145    // `f` runs on the caller's own engine, so a wrapping engine keeps its
146    // overrides inside the scope. The policy is restored before an unwind
147    // continues; nothing observes the engine between the panic and the
148    // restore, so asserting unwind safety is sound.
149    let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| f(&mut *engine)));
150    // INVARIANT: the same engine yielded a CPU engine above and native tokens
151    // are stable for an engine's lifetime, so this visit runs.
152    let _ = crate::with_cpu_engine(engine, |cpu| cpu.replace_batch_policy(previous));
153    match outcome {
154        Ok(value) => Ok(value),
155        Err(payload) => std::panic::resume_unwind(payload),
156    }
157}