CPU Performance-Issue Workloads

• Suite: cpu/perf_issues
• Raw run: data/results/amd-cpu/cpu/perf_issues/20261008_063605

CPU Information

• Model: AMD Ryzen 9 9955HX 16-Core Processor
• Vendor: AuthenticAMD
• Logical CPUs: 32
• Sockets: 1
• Cores per socket: 16
• Threads per core: 2
• NUMA nodes: 1
• Python platform: Linux-7.0.0-38-generic-x86_64-with-glibc2.39

run_t1

Full metadata: data/results/amd-cpu/cpu/perf_issues/20261008_063605/run_t1.yaml

timestamp: '2026-10-08T06:36:05Z'
tenferro_rs:
  path: /workspaces/tenferro-benchmark/extern/tenferro-rs
  commit: 5cf78c7ec0ad9516dd78bab546d5e7bd42fa3102
  dirty: false
  features:
  - system-mkl
  resolved_path: /workspaces/tenferro-benchmark/extern/tenferro-rs
blas:
  implementation: mkl
  version: 2026.0.1
  root: /opt/intel/oneapi/mkl/latest
  library: /opt/intel/oneapi/mkl/latest/lib/libmkl_rt.so
environment:
  hostname: 14099c68ff8e
  os: Linux-7.0.0-38-generic-x86_64-with-glibc2.39
  arch: x86_64
  cpu: x86_64
  env:
    OMP_NUM_THREADS: '1'
    OMP_THREAD_LIMIT: '1'
    OMP_DYNAMIC: 'FALSE'
    RAYON_NUM_THREADS: '1'
    TENFERRO_CPU_BACKEND_KIND: blas
    BENCHMARK_TARGET_PROFILE: amd-cpu
    BENCHMARK_COMMIT: 2cd0fd6445dcd9aad98990f76c17e33e773c4a7d
    OPENBLAS_ROOT: /opt/openblas
    MKLROOT: /opt/intel/oneapi/mkl/latest
    OPENBLAS_NUM_THREADS: '1'
    GOTO_NUM_THREADS: '1'
    MKL_NUM_THREADS: '1'
    VECLIB_MAXIMUM_THREADS: '1'
    VECLIB_NUM_THREADS: '1'
    NUMEXPR_NUM_THREADS: '1'
    BLIS_NUM_THREADS: '1'
    PJRT_NPROC: '1'
    XLA_FLAGS: --xla_cpu_multi_thread_eigen=false --xla_cpu_experimental_ynn_fusion_type=
      intra_op_parallelism_threads=1
    JULIA_NUM_THREADS: '1'
    USE_CUDA: '0'
    BENCHMARK_BUILD_PROFILE: release
    RUSTC_VERSION: rustc 1.99.0 (b940084d7 2026-09-28)
    MATMUL_NUM_THREADS: '1'
    BENCH_INSTANCE: null

run_t4

Full metadata: data/results/amd-cpu/cpu/perf_issues/20261008_063605/run_t4.yaml

timestamp: '2026-10-08T06:36:57Z'
tenferro_rs:
  path: /workspaces/tenferro-benchmark/extern/tenferro-rs
  commit: 5cf78c7ec0ad9516dd78bab546d5e7bd42fa3102
  dirty: false
  features:
  - system-mkl
  resolved_path: /workspaces/tenferro-benchmark/extern/tenferro-rs
blas:
  implementation: mkl
  version: 2026.0.1
  root: /opt/intel/oneapi/mkl/latest
  library: /opt/intel/oneapi/mkl/latest/lib/libmkl_rt.so
environment:
  hostname: 14099c68ff8e
  os: Linux-7.0.0-38-generic-x86_64-with-glibc2.39
  arch: x86_64
  cpu: x86_64
  env:
    OMP_NUM_THREADS: '4'
    OMP_THREAD_LIMIT: '4'
    OMP_DYNAMIC: 'FALSE'
    RAYON_NUM_THREADS: '4'
    TENFERRO_CPU_BACKEND_KIND: blas
    BENCHMARK_TARGET_PROFILE: amd-cpu
    BENCHMARK_COMMIT: 2cd0fd6445dcd9aad98990f76c17e33e773c4a7d
    OPENBLAS_ROOT: /opt/openblas
    MKLROOT: /opt/intel/oneapi/mkl/latest
    OPENBLAS_NUM_THREADS: '4'
    GOTO_NUM_THREADS: '4'
    MKL_NUM_THREADS: '4'
    VECLIB_MAXIMUM_THREADS: '4'
    VECLIB_NUM_THREADS: '4'
    NUMEXPR_NUM_THREADS: '4'
    BLIS_NUM_THREADS: '4'
    PJRT_NPROC: '4'
    XLA_FLAGS: --xla_cpu_multi_thread_eigen=true --xla_cpu_experimental_ynn_fusion_type=
      intra_op_parallelism_threads=4
    JULIA_NUM_THREADS: '4'
    USE_CUDA: '0'
    BENCHMARK_BUILD_PROFILE: release
    RUSTC_VERSION: rustc 1.99.0 (b940084d7 2026-09-28)
    MATMUL_NUM_THREADS: '4'
    BENCH_INSTANCE: null

Case status

• Manifest: cpu/perf_issues version 2, coverage full, effort standard
• Expected 306, selected 306, executed 306, unsupported 0, failed 0, missing 0, noisy 32.
• Complete: yes — only executed cases count as covered; unsupported, failed, missing and unselected cases never do.
• Noisy: eager_backward_matmul2x2_f64_leaves0@t1, eager_backward_matmul2x2_f64_leaves2@t1, eager_backward_matmul2x2_f64_leaves32@t1, eager_backward_matmul2x2_f64_leaves128@t1, layer_norm_f32_eager-single-call_d1024_len8_b8@t1, rms_norm_f32_eager-composed_d1024_len64_b8@t1, small_contraction_abcd-dbef-acef_f64_per-call-default-pool_d4@t1, masked_softmax_f32_eager-single-call_len64_b8@t1, decode_proj_f32_eager-shared_in1024_out4096_len8@t4, decode_proj_f32_faer-direct_in1024_out4096_len64@t4, decode_proj_f32_host-sgemm_in1024_out4096_len512@t4, conj_dot_f64_pytorch-vdot_len10000@t4, eager_backward_matmul2x2_f64_leaves2@t4, eager_backward_matmul2x2_f64_leaves128@t4, eager_backward_matmul2x2_f64_leaves512@t4, tanh_chain_f32_compiled-prepared_k10_1024x64@t4, tanh_chain_f32_eager-shared_k10_1024x64@t4, layer_norm_f32_pytorch-fused_d1024_len64_b1@t4, layer_norm_f32_eager-single-call_d1024_len64_b1@t4, layer_norm_f32_eager-composed_d1024_len64_b8@t4, layer_norm_f32_pytorch-fused_d1024_len64_b8@t4, layer_norm_f32_eager-single-call_d1024_len64_b8@t4, rms_norm_f32_eager-composed_d1024_len64_b8@t4, rms_norm_f32_pytorch-fused_d1024_len64_b8@t4, rms_norm_f32_eager-single-call_d1024_len64_b8@t4, activation_sigmoid_f32_eager-single-call_1024x64@t4, activation_sigmoid_f32_eager-composed_1024x64@t4, activation_silu_f32_eager-single-call_1024x64@t4, softmax_f32_eager-single-call_len512_b8@t4, softmax_f32_pytorch_len512_b8@t4, log_softmax_f32_eager-composed_len512_b8@t4, masked_softmax_f32_eager-single-call_len512_b8@t4

Numerical checks: 306/306 recorded rows passed (case × thread count).

Steady-state operation rows

Each batch is one wall-clock interval over many operations; every output is retained until the clock stops. Median and inclusive IQR are per operation, in nanoseconds; CoV is sample standard deviation / mean (NOISY means CoV > 10%, a descriptive label only). Setup (inputs, backends, runtimes, session entry, tracing/compilation, preparation) is outside timing; faer-direct and host-sgemm write into a preallocated output, the tenferro and PyTorch arms allocate their output. reference_for in the case file names the case each reference arm is compared with. GPU batches end with one device synchronize (a per-call synchronize where the transfer is the operation); the Threads column is the host thread setting.

Per-call session entry diagnostics (not operation comparisons)

A session (or eager session) is entered and left for every operation, inside the timer.

Eager AD workflow diagnostics

Forward + reduction + backward through the eager runtime, including its internal session entries; gradients accumulate across a batch and are reset outside timing.

Counter diagnostics (no timing)

decode_block_copy_volume_f32_d1024_len8 (#1995, 1 threads)

Source: counting global allocator in this process, per eager op including its own session entry. Unavailable: per-op layout-copy count, copied bytes (pooled buffer reuse is invisible to the host allocator).

decode_block_copy_volume_f32_d1024_len8 (#1995, 4 threads)

Source: counting global allocator in this process, per eager op including its own session entry. Unavailable: per-op layout-copy count, copied bytes (pooled buffer reuse is invisible to the host allocator).

Timing boundaries

• activation / eager-composed — inside: hand-written formula from pre-B1 eager primitives, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• activation / eager-single-call — inside: one activation call, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• activation / pytorch — inside: torch gelu_tanh, output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• activation / session-single-call — inside: one activation call, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, with_backend_session entry (CpuBackend::new()), correctness_check.
• backward_live_leaves / eager-ad — inside: eager matmul forward, reduce_sum, backward (gradients accumulate across a batch), internal eager session entries; outside: eager_runtime_construction, leaf_construction (including the unrelated leaves), correctness_check, gradient reset between cases.
• composed_norm / eager-composed — inside: eager composition (reduce_sum, scale_real, broadcast_in_dim, sub/add/mul, reduce_sum_squares, rsqrt), intermediate and output allocation; outside: eager_runtime_construction, input/weight/bias/eps construction, eager session entry, correctness_check.
• composed_norm / eager-single-call — inside: one EagerSession layer_norm / rms_norm call (composite), intermediate and output allocation; outside: eager_runtime_construction, input/weight/bias/eps construction, eager session entry, correctness_check.
• composed_norm / pytorch-fused — inside: torch.nn.functional.rms_norm (fused), output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• copy_volume / eager-counters — inside: —; outside: everything (counter diagnostic, no timing).
• decode_projection / eager-per-call — inside: eager_session_entry_exit, eager_dot_general, output_allocation; outside: backend_construction, input_construction, correctness_check.
• decode_projection / eager-shared — inside: eager_dot_general, output_allocation; outside: backend_construction, input_construction, correctness_check, eager_session_entry_exit.
• decode_projection / faer-direct — inside: faer_matmul_into_preallocated_output; outside: backend_construction, input_construction, correctness_check, output_allocation.
• decode_projection / host-sgemm — inside: matrixmultiply_sgemm_into_preallocated_output; outside: backend_construction, input_construction, correctness_check, output_allocation.
• decode_projection / pytorch — inside: torch.nn.functional.linear, output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• gemm_mm256 / dot-general-shared — inside: dot_general_read, output_allocation; outside: backend_construction (CpuBackend::with_threads(1)), input_construction, session_entry_exit, correctness_check.
• rank1_dot / dot-general-rank1 — inside: dot_general_read (rank-1, rank-0 tensor output), output_allocation; outside: backend_construction, input_construction, operand_conjugation, session_entry_exit, correctness_check.
• rank1_dot / portable-loop — inside: sequential Rust loop over borrowed slices, scalar result; outside: input_construction, correctness_check.
• rank1_dot / pytorch-vdot — inside: torch.vdot (BLAS ddot/zdotc), scalar tensor allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• reduce_mean / eager-composed — inside: reduce_sum + scale_real, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• reduce_mean / eager-single-call — inside: one reduce_mean call, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• reduce_mean / pytorch — inside: torch.mean, output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• reduce_mean / session-single-call — inside: one reduce_mean call, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, with_backend_session entry (CpuBackend::new()), correctness_check.
• small_contraction_pool_hop / per-call-default-pool — inside: with_backend_session entry/exit per call, dot_general_read, output_allocation; outside: backend_construction, input_construction, correctness_check.
• small_contraction_pool_hop / per-call-threads1 — inside: with_backend_session entry/exit per call, dot_general_read, output_allocation; outside: backend_construction, input_construction, correctness_check.
• small_contraction_pool_hop / shared-session-default-pool — inside: dot_general_read, output_allocation; outside: backend_construction, input_construction, correctness_check.
• small_solves / pytorch-solve-batched — inside: torch.linalg.solve on (batch, k, k), output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• small_solves / pytorch-solve-triangular-batched — inside: torch.linalg.solve_triangular on (batch, k, k), output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• small_solves / solve-batched — inside: one batched solve call ([k, k, batch]), output_allocation; outside: backend_construction, input_construction (items pre-sliced), session_entry_exit, correctness_check.
• small_solves / solve-loop — inside: batch x rank-2 solve calls, output_allocation per item; outside: backend_construction, input_construction (items pre-sliced), session_entry_exit, correctness_check.
• small_solves / triangular-solve-loop — inside: batch x rank-2 solve calls, output_allocation per item; outside: backend_construction, input_construction (items pre-sliced), session_entry_exit, correctness_check.
• softmax / eager-composed — inside: max-subtracted composition (reduce_max, broadcast_in_dim, sub, exp, reduce_sum, div/log; select when masked), intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• softmax / eager-single-call — inside: one softmax-family call (composite, all-masked guard included), intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• softmax / pytorch — inside: torch masked_softmax, output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• softmax / session-single-call — inside: one softmax-family call (composite, all-masked guard included), intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, with_backend_session entry (CpuBackend::new()), correctness_check.
• take_along_axis / eager-gather-prebuilt — inside: gather with prebuilt (row, batch) index tuples, intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• take_along_axis / eager-single-call — inside: one take_along_axis call (index tuples built inside, then gather), intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, eager session entry, correctness_check.
• take_along_axis / pytorch — inside: torch.take_along_dim, output_allocation; outside: input_construction, torch.set_num_threads, correctness_check.
• take_along_axis / session-single-call — inside: one take_along_axis call (index tuples built inside, then gather), intermediate and output allocation; outside: backend/runtime construction, input, constant, mask and index construction, with_backend_session entry (CpuBackend::new()), correctness_check.
• tanh_chain / compiled-prepared — inside: run_prepared (runtime-internal session entry), fused region execution, output_allocation; outside: backend/runtime construction, trace and compile, prepare_compiled, input_construction, eager session entry (eager arm), correctness_check.
• tanh_chain / eager-shared — inside: chain of eager tanh ops, intermediate and output allocation; outside: backend/runtime construction, trace and compile, prepare_compiled, input_construction, eager session entry (eager arm), correctness_check.