CPU large-matrix eager AD: gap survey and fixes

• Suite: cpu/large_ad
• Target profile: amd-cpu
• Provider: system MKL, one backend worker thread, MKL_NUM_THREADS=1
• Affinity: pinned to CPU 1 for every row
• Thread budget: 1
• tenferro-rs revision measured: 5782859f (the tree that became #1828), plus the #1835 eigvalsh change measured separately after it merged
• Reference: PyTorch 2.12.0+cpu, torch.set_num_threads(1)

Commands

cargo build -j 16 --release --features system-mkl --bin publication_gate

export LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/latest/lib:/opt/intel/oneapi/compiler/latest/lib
export MKL_NUM_THREADS=1 OMP_NUM_THREADS=1 RAYON_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_DYNAMIC=FALSE
export PUBLICATION_GATE_PROFILE=full PUBLICATION_GATE_SUITE=large \
       PUBLICATION_GATE_TENFERRO_MODE=both BENCH_RUNS=15 BENCH_WARMUPS=3

# Interleave the two backends per repetition so the reference is not inflated by drift.
for rep in 1 2 3; do
  taskset -c 1 target/release/publication_gate > fin$rep-rust.csv 2> fin$rep-rust.stderr
  taskset -c 1 .venv/bin/python scripts/benchmark_cpu_ops_python.py \
    --backend pytorch-cpu --num-threads 1 --output fin$rep-torch.csv
done

The full profile now covers 64/128/256/512/1024 for the large-matrix primal and AD rows in both the Rust gate and the Python runner; that size extension is part of this change.

Scope

Timings are forward + sum + backward for the eager rows, not backward alone. Eager includes its own recording/AD preparation inside the timing; the prepared trace builds and compiles the derivative graph outside the timing and the timed region is one run_prepared call. Input construction and sessions are outside timing in both cases, and outputs are retained until the clock stops. Cross-backend ratios come from interleaved runs, because two sequential suite runs drift more than the effects measured here.

Results

Median of three interleaved repetitions, 15 samples and 3 warmups each.

Provider floor

Measured inside the benchmark binary with an LD_PRELOAD probe at n=1024, so the kernel's own overhead is separable from its LAPACK call:

Every primal row is within about 3% of the provider call, so the remaining cost is the decomposition itself. dsyevr (MRRR) is not an alternative here: 373.6 ms against 292.0 ms for dsyevd (divide and conquer) at n=1024 for all eigenpairs.

Changes that came out of this survey

The eigvalsh pair, interleaved A/B with both rows in one process:

Remaining, not addressed

• Small-size eager AD rows still carry a fixed per-backward cost (64x64 solve backward 2.97x and matmul backward 2.33x versus Torch, about 0.2-0.3 ms in absolute terms). This is latency, not large-matrix throughput.
• Eager rows remain marginally slower than their own prepared-trace counterparts. A dedicated clean-context probe of the n=1024 solve row measured eager 27.0-27.9 ms against trace 24.6 ms, and running both linalg operations inside one entered session was not faster than two separate entries once the pooled workspace was warm. The much larger eager solve numbers that appear inside a full suite run (up to about 45 ms) are a memory/buffer-pool context effect, so that row's absolute value is not a stable artifact.
• The eigenvalue-only workspace zeroing suspected during the eigvalsh analysis was measured at 0.332 ms for the 16.8 MB dsyevd workspace and left unchanged.

Evidence

Raw samples, protocol scripts, provider probes and source snapshots are kept under data/results/amd-cpu/cpu/large-ad/ in the benchmark checkout; that tree is git-ignored by policy. Files of interest: fin{1,2,3}-{rust,torch}.{csv,jsonl}, eighAB-{beforeA,beforeB,afterA,afterB,afterC}.{csv,stderr}, sym-floor.{csv,stderr}, eigh_ab.sh, gemm_probe.c, sym_probe.c, syev_compare.c, getrf_probe.c.

Post-fix multi-thread (1T/4T) provider survey

The 1T survey above remains valid: with one thread the domain is a single CPU, so worker affinity and provider threading are identical before and after the change below. This section records the multi-thread case, which is what exposed the problem and what changed.

Change measured: tenferro-rs 67fdd861 (fix(cpu): confine workers to the domain CPU set, PR #1847). Before it, each Rayon worker was pinned to one CPU; BLAS/LAPACK provider threads inherit the creating thread's mask, so an MKL provider team was confined to that single CPU. After it, every worker is confined to the whole domain CPU set, which provider threads inherit.

Protocol (declared before the run): suite cpu/large_ad, profile full, PUBLICATION_GATE_TENFERRO_MODE=both, 15 samples / 3 warmups, one paired repetition, runs strictly sequential. 1T uses taskset -c 10; 4T uses taskset -c 10-13 with RAYON_NUM_THREADS and the lane's provider thread count set to 4. Lanes: system-openblas (Linux default), system-mkl, cpu-faer; PyTorch reference from the wheel at the same thread counts. Every run exited 0 and produced a stable row count (142 tenferro rows, 86 torch rows).

Measured on the same EPYC 7713P host as the 1T survey, with other users' background load present; treat small ratios as indicative and the >2x ratios as structural.

Tenferro (median over repetitions)

Provider-matched torch reference (median over repetitions)

Run validity

• all runs exited 0 with a stable row count

Affinity guard (4T, one run per lane)

scripts/cpu_provider_affinity_check.py records Cpus_allowed_list per thread while a run executes. A provider thread count does not prove where the threads may run, so this is the regression observable for the failure above.

Before the fix the same probe showed openmp_worker at cpus=10 (one CPU, inherited from the worker pinned to CPU 10). It now reports 10-13, i.e. the whole admitted domain.

Controlled before/after (same binary, MKL, 4T)

A dedicated A/B run toggles only the worker mask (per-worker single CPU vs domain set) with the same binary and interleaved execution, so it isolates the cause from build, harness and drift:

The native rows show the change costs nothing outside the provider path; a separate paired run at 4 workers over 16 CPUs across two CCXs kept every faer-backed row within +-4% with mixed signs, and 2 ms affinity sampling showed 0-0.2 worker migrations/s without a provider.

Provider pairs measured here

The system-mkl lane therefore compares two different MKL builds, and no lane makes both sides use the same library except the provider-matched OpenBLAS image. See CPU provider pairs.

Reproduce

# tenferro lanes (Linux default is system-openblas; use system-mkl / cpu-faer for the others)
cargo build -j 16 --release --features system-openblas --bin publication_gate
export PUBLICATION_GATE_PROFILE=full PUBLICATION_GATE_SUITE=large \
       PUBLICATION_GATE_TENFERRO_MODE=both BENCH_RUNS=15 BENCH_WARMUPS=3
taskset -c 10-13 env RAYON_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 \
  LD_LIBRARY_PATH=/opt/openblas/lib ./target/release/publication_gate > run-4t.csv

# PyTorch reference at the same thread count
.venv/bin/python scripts/benchmark_cpu_ops_python.py --backend pytorch-cpu \
  --num-threads 4 --runs 15 --warmups 3 --output run-4t-torch.csv

# affinity guard: which CPUs were worker and provider threads actually allowed to use
python3 scripts/cpu_provider_affinity_check.py --output guard-4t.json -- \
  taskset -c 10-13 env RAYON_NUM_THREADS=4 MKL_NUM_THREADS=4 \
  LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/latest/lib \
  ./target/release/publication_gate

Raw CSVs, the guard JSON, the campaign driver and the summarizer are kept under data/results/amd-cpu/cpu/large-ad/postfix4t/ (local, git-ignored).