Skip to main content

Parallelism and Caching

tenferro keeps CPU parallelism and execution caches explicit. Use this page to control CPU thread counts, avoid provider oversubscription, and release cached memory in long-running processes.

The caller owns the pool, so thread count and affinity are pool properties. For how to build the pool, affinity/NUMA guidance, and the location of elementwise Rayon work, see CPU Execution and Thread Pools.

For tensor memory layout and column-major buffers, see Memory Order. That is part of the tensor data model, not the parallelism contract.

For measured per-call dispatch and Rayon worker handoff costs, batching and one-thread alternatives, see Dispatch and Engine Cost.

CPU Backend Provider

There is one CPU provider and no feature combination that changes it; thread ownership is described in Choosing A Backend. This page keeps the runtime mechanics below.

CPU Thread Count

Build one pool with the width you want when one engine should carry a fixed CPU parallelism policy:

use tenferro_cpu::{Pool, CpuEngine};

let pool = Pool::builder().threads(4).build()?;
let engine = CpuEngine::new(pool);
assert_eq!(engine.num_threads(), 4);

A Pool built without an explicit threads uses the process-visible CPU count; no environment variable changes it, so the width of a pool is always visible at the call that builds it. Pass the width you want:

let threads = std::env::var("MY_APP_THREADS")?.parse()?;
let engine = CpuEngine::new(Pool::builder().threads(threads).build()?);

The operation’s placement token hands the engine’s worker count to faer-backed kernels as a plain width (Exec::external). A serial engine uses sequential faer execution; a pooled engine uses faer’s parallelism with that width, on the engine’s pool.

CPU Operation Parallelism

The CpuEngine owns the workspace and the buffer pool; the pool it computes on is the Pool the caller passed to CpuEngine::new. Each operation mints one placement token (threadteam::Exec) from that pool. The operation starts on the calling thread and the token enters the pool only when an entry splits the work into more than one task. No ambient pool is ever an implicit fallback, and CpuEngine::serial never enters any pool.

Operation family Threading behavior
Elementwise and analytic ops Shared typed paths use strided-basic, ordinary dtype dispatch uses strided-kernel, and erased fused replay uses strided-fused; all take the operation’s token as their last argument and run on the engine’s pool within its budget.
Reductions sum/product erased replay and typed max/min reductions take the same token and never use an ambient pool. Upstream strided plan coverage determines whether a specific multi-axis reduction shape can actually partition work.
View materialization, transpose/permute, broadcast, convert, and diagonal extraction Strided copy/map kernels take the same token; layout fallback and linalg input materialization are included.
dot_general faer receives Par::Team only for a width above one on a team token (a multi-thread engine); a plain or serial token receives Par::Seq. Direct downstream faer calls use CpuEngine::exec()? and token.team(..) when token.carries_team(); tenferro hands out no faer::Par.
Indexed gather/scatter and dynamic slice/update These delegate to strided erased plans with the operation’s token; upstream strided plan coverage determines whether a specific indexed plan can actually partition work.
Slicing, padding, concatenation, reverse, triangular masks, and embed_diagonal These are dedicated sequential CPU loops today because their per-output indexing patterns do not yet have a strided-kernel/backend-native parallel primitive. They still run on the engine, and source comments mark the intentional sequential path.

CPU affine-strided copy, permutation, broadcast, map, zip-map, and axis reduction delegate to strided-rs, while tenferro supplies operation semantics, validation, dtype and placement checks, error translation, and execution resources. Einsum/dot-general is the benchmark-backed tenferro exception: tenferro owns its planning, optimized preparation, and provider integration.

Even a host-to-host materialization goes through the engine. The engine owns a persistent buffer pool, can allocate an uninitialized output when the copy fully overwrites it, and runs the kernel on the caller’s pool through the operation’s token. A pool-free copy, a temporary buffer pool, or a process-global pool would create a second memory and threading policy. Memory reuse and thread policy are execution resources, not tensor metadata.

Use the engine’s canonicalization operation when a metadata-only view must become compact:

use tenferro_cpu::{Pool, CpuEngine};
use tenferro_tensor::{TensorStructural, TypedTensor};

let tensor = TypedTensor::<f64>::from_vec_col_major(
    vec![2, 3],
    vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
)?;
let transposed = tensor.as_view().transpose_view([1, 0])?.into_tensor_read()?;
let mut engine = CpuEngine::new(Pool::builder().threads(4).build()?);
let compact = engine.to_contiguous_read(transposed)?;

assert_eq!(compact.shape(), &[3, 2]);
assert_eq!(compact.as_slice::<f64>()?, &[1.0, 3.0, 5.0, 2.0, 4.0, 6.0]);

Canonicalization preserves placement; it does not silently upload host data or download device data.

CPU Library Threads

tenferro passes Sequential or the selected pool with a thread budget to the lower numerical library (cpueinsum, tcontract, tlinalg), and the lower library decides its own lanes within that budget. There is no vendor-internal thread pool to configure.

Avoid Oversubscription

Do not accidentally multiply outer application parallelism by inner kernel parallelism. If an outer loop already runs many independent tenferro calls in parallel, use a smaller inner backend:

use tenferro_cpu::{Pool, CpuEngine};

let engine = CpuEngine::new(Pool::builder().threads(1).build()?);
assert_eq!(engine.num_threads(), 1);

For benchmarks, pin all relevant thread counts and report them with the result. Bitwise-identical results are promised at a fixed thread count only, never across thread counts: see Reproducibility.

When the host already partitions coarse work across caller-owned pools, give each partition its own CpuEngine over its own pool. One computation uses one engine at a time (every operation takes &mut), so a partition must admit at most one coarse tenferro operation per pool and account for any physical CPU oversubscription itself. See CPU Execution and Thread Pools.

Reuse Runtime State

Reuse execution objects when you repeat related work:

  • tenferro_runtime::Runtime retains immutable engine/extension registration snapshots, prepared-plan cache entries, and registered runtime cache owners for the Runtime::prepare / Runtime::run pipeline.
  • EagerRuntime retains eager extension plans and compiled inner extension programs across immediate operations. It stores no engine: every eager call takes the one the caller holds and registers it with the runtime on first use.
  • GraphCompiler retains graph lowering and static extension planning caches. Its compiled artifact keeps semantic program plus compiler options, not backend staging.
  • Runtime owns prepared-plan caching and registered runtime cache owners. CPU buffer pools and extension caches remain owned by the caller-held engine that the registration describes.
use tenferro_cpu::{Pool, CpuEngine};
use tenferro_runtime::{GraphCompiler, Runtime};

let mut compiler = GraphCompiler::new();
let engine = CpuEngine::new(Pool::builder().threads(4).build()?);
let mut builder = Runtime::builder();
builder.register_engine(tenferro_cpu::runtime_engine_registration(&engine)?)?;
let runtime = builder.build()?;

Short-lived scripts can usually ignore cache tuning. Services, notebooks, and benchmark harnesses should treat caches as part of runtime resource management.

Cache Limits

Runtime, compiler, eager, and CPU backend caches are bounded by default and can be configured independently.

use std::num::NonZeroUsize;
use tenferro_cpu::CpuEngine;
use tenferro_runtime::extension::ExtensionCacheLimits;
use tenferro_runtime::ExecutionEngine;

let mut engine = CpuEngine::serial();
let mut caches = engine.take_extension_caches();
caches.set_limits(
    ExtensionCacheLimits::new(
        NonZeroUsize::new(128).ok_or("positive cache-entry limit")?,
    )
    .with_max_retained_bytes(
        NonZeroUsize::new(64 * 1024 * 1024).ok_or("positive cache-byte limit")?,
    ),
);
engine.restore_extension_caches(caches);

For CPU engines, the buffer pool has its own retention limit:

use tenferro_cpu::CpuEngine;

let mut engine = CpuEngine::serial();
engine.set_buffer_pool_limit_bytes(32 * 1024 * 1024);

The limit is a byte cap on recycled capacity and defaults to 100 MiB. An output larger than the cap cannot come back through the pool, so every call allocates a fresh destination and pays its first touch inside the operation: on one x86-64 core a fresh 256 MiB f64 destination costs roughly 110 ms of page faults, where the same sqrt over a reused destination takes about 22 ms. Size the limit to the outputs a run keeps cycling through, and leave it small when it only serves small scratch. The cap stays a bound either way; only its value is workload-specific, and TENFERRO_BUFFER_POOL_MAX_RETAINED_BYTES sets it once for every engine the process creates.

Clearing Cached Memory

Clear caches when a long-running process changes workload phase, when a notebook has finished a large experiment, or when memory pressure matters more than reusing old plans and buffers.

use tenferro_ad::EagerRuntime;
use tenferro_runtime::{GraphCompiler, Runtime};

fn main() -> Result<(), Box<dyn std::error::Error>> {
let eager = EagerRuntime::new()?;
let runtime = Runtime::builder().build()?;
let mut compiler = GraphCompiler::new();

runtime.clear_prepared_cache()?;
runtime.clear_caches()?;
compiler.clear_caches();
eager.clear_caches()?;
Ok(())
}
main()?;

For CPU engines, clear the buffer pool through the engine:

use tenferro_cpu::CpuEngine;

let mut engine = CpuEngine::serial();
engine.reset_buffer_pool();

retained_bytes in cache stats is tenferro’s logical retained-data estimate. It is not operating-system RSS and does not include allocator arena slack, thread stacks, or provider-owned memory.

CacheStats also reports cache events when the owner can observe them: hits, misses, evictions, and explicit clears. Extension cache selectors scope event counters the same way they scope entries and retained_bytes, so ExtensionCacheSelector::Family and ExtensionCacheSelector::Cache report only the selected family or cache.

CUDA Transfer Boundaries

CUDA transfers are explicit. Upload once near the boundary of a CUDA workload, run supported operations on CUDA tensors, then download only when host inspection or CPU execution is needed. Repeated upload/download inside tight loops usually dominates runtime. See Devices and GPU for the current CUDA coverage and setup commands.