CI Cache Trust And Immutable GPU Artifact Reuse

Status: active. Implements issue #1403 under the #1401 CI performance umbrella. Companion contract tests live in scripts/ci/tests/test_workflow_contracts.py and scripts/ci/tests/test_find_archive_artifact.py.

Threat model

GitHub Actions caches are shared mutable state. A cache entry saved into the default-branch scope is readable by every later run, including trusted RunPod GPU runs that execute restored binaries on a paid self-hosted pod. Two paths could publish attacker-influenceable bytes into that scope if left open:

  1. A job that executes PR-controlled code while holding a cache-write credential. runpod-gpu-test.yml runs on the workflow_run event, so github.ref is refs/heads/main and any cache it saved would land in the privileged main scope — while its cuda-archive job compiles PR-controlled sources (build.rs, proc macros) that can execute arbitrary code inside the job.
  2. A job on the self-hosted RunPod pod. The pod executes the checked-out ref as root in the same container as the runner process, so any credential visible to the runner is visible to test code.

pull_request-event workflows are not part of this problem: GitHub isolates their cache writes to the PR’s own merge-ref scope, which cannot be read from main-scope runs, and their workflow definitions never receive RunPod secrets.

Namespace and role model

Actor Cache role Why
ci-cache-publish.yml (push to main, schedule, dispatch from main) Only shared-cache writer Definition and built code are both default-branch content
runpod-gpu-test.yml cuda-archive Reader only Builds PR-controlled code in the main-scoped context
runpod-gpu-execute.yml run-gpu-tests Reader only Trusted reusable lifecycle; runs on the self-hosted pod
ci.yml / ci-pr-workspace-tests.yml PR jobs Platform-isolated writers Saves go to the PR merge-ref scope, never to main scope
ci.yml / ci-pr-workspace-tests.yml push-to-main jobs Trusted writers (CPU lanes) Push runs build default-branch code

Attacker-controlled contents cannot reach the privileged namespace because the only jobs holding a write-capable posture in the main scope never execute non-default-branch code, and every job that does execute PR code or run on the pod uses actions/cache/restore plus rust-cache save-if: false, with read-only workflow permissions (cuda-archive adds actions: read only, for artifact lookup). These invariants are contract-tested.

The trusted hosted lifecycle watcher and independent orphan reaper have actions: write solely to cancel obsolete/expired workflows after confirming pod deletion. They check out the trusted workflow SHA without persisted credentials and never build or execute PR code. The reusable lifecycle’s caller grants that ceiling, while the callee’s default permissions and GPU/build jobs remain read-only. These jobs do not save caches; shared-cache publication stays with ci-cache-publish.yml.

Key derivation

The CUDA/PJRT archive key is derived from material compilation inputs only:

  • Cargo.lock, every Cargo.toml, src/**, tests/**, examples/**, benches/**, and every build.rs (hashed) — everything cargo nextest archive compiles, including required-feature CUDA examples that no non-GPU lane checks,
  • every Markdown file embedded into a binary with include_str! (for example docs/guides/devices-and-gpu.md in the CUDA capability tests) — a contract test discovers these embeds and fails if one is missing from the key,
  • scripts/ci/**, .cargo/**, and rust-toolchain* (hashed) — these are executed or read from the PR checkout during the archive build, so a key match must prove they were identical; without this, a PR that only edits an install script could poison an artifact under an unchanged name,
  • the Rust toolchain version (rustc -V),
  • the cudarc binding and PTX toolkit versions (explicit key prefix),
  • a manually bumped vN component covering the archive build commands and toolkit selection themselves.

Workflow YAML is deliberately excluded: PR-triggered runs always execute the default-branch workflow definition, so hashing it would only cause unrelated invalidation. (runpod_config.json is hashed as part of scripts/ci/**; the occasional spurious rebuild is accepted in exchange for the simpler “hash everything the checkout contributes to the build” rule.) The consumer (runpod-gpu-test.yml) and publisher (ci-cache-publish.yml) compute the key with byte-identical expressions over an identical checkout layout; a contract test compares the two lines.

The cuTENSOR and CUDA-runtime-tree caches key on OS, architecture, component version, and a manual vN. Hosted preparation restores them under the fixed root /opt/tenferro-ci, then stages immutable runtime artifacts for the Pod. The cuTENSOR cuda12-runtime-v3 namespace contains only lib/libcutensor.so*, the single-GPU shared ABI already selected by runtime packaging. Static archives and independent multi-GPU/MPI providers are excluded at installation, before cache publication; restoring them just to discard them wastes cache quota and transfer bytes. The installer requires a nonempty, resolvable libcutensor.so.2. A cache miss downloads the same vendor redistributable. Reader and writer keys rotate together, with no fallback to the larger v2 cache. CUDA runtime trees already exclude compiler/static archives. The minimal-v9 CUDA runtime namespace includes cuFFT for the FFT tests in both the 12.6 and 12.8 tiers. Hosted preparation rejects and rebuilds restored trees missing cuFFT; Pod configuration also requires it before running tests.

The hosted toolkit cache contains the relocatable nvcc/include/library tree for the exact requested CUDA major.minor version. Its key includes Ubuntu version, architecture, toolkit version, installer hash and a manual version. The same absolute /opt/tenferro-ci/cuda-toolkit-<version> path is used by publisher and reader. Only ci-cache-publish.yml saves it. Restore uses no prefix fallback; a miss installs normally. Both restored and newly installed trees must compile a small PTX kernel before use/publication, rather than trusting a hit message. The cache is bounded by Actions eviction/quota and can be deleted through the Actions cache UI/API; neither correctness nor GPU authorization depends on it.

Every cache namespace was rotated (vN bumps across the archive, Rust build, runtime-tree, and cuTENSOR keys) when the trusted publisher took ownership: entries under the previous names could have been populated while write-capable jobs still built PR code, so consumers must never restore them.

Immutable GPU artifact reuse across retries

Every cuda-archive run uploads the two nextest archives as a per-run artifact whose name is the content key. A GPU allocation retry (a fresh workflow_dispatch through recover_runpod_pr.py, or any rerun with unchanged sources) resolves its own content key, then asks scripts/ci/find_archive_artifact.py for an existing unexpired artifact with that exact name before building anything.

The finder only accepts artifacts whose producing run:

  • belongs to this repository (head_repository check),
  • ran a trusted workflow file (runpod-gpu-test.yml or ci-cache-publish.yml),
  • was triggered by an event whose definition comes from the default branch (workflow_run, workflow_dispatch, push, schedule) — never pull_request, whose definitions are PR-controlled,
  • for directly-triggered events (workflow_dispatch, push, schedule), ran on the default branch — a dispatch aimed at another ref is rejected, and the ci-cache-publish.yml jobs additionally refuse to run at all off refs/heads/main.

Name-collision attacks fail on the producer check; content-substitution attacks fail because the key is computed by the trusted workflow from its own pinned checkout, so a name match implies identical build inputs. Reuse is an optimization: any lookup or download failure falls back to a fresh build.

On success the archives are re-uploaded to the current run so the pod’s download-artifact step and per-run retention semantics are unchanged, and no Cargo compilation happens on the retry path.

Observability and bounds

  • The archive job logs its source (cache hit / reused artifact with run id / fresh build) plus archive sizes and free disk. The hosted workspace matrix maps rust-cache to its external CARGO_TARGET_DIR and, on an exact hit, verifies restored faer and strided-rs dependency artifacts under the actual target path; a cache-hit message alone is not treated as evidence.

  • Hosted preparation logs cuTENSOR and runtime-tree cache hit state before falling back to direct downloads; the Pod receives the prepared artifacts.

  • The publisher logs whether a key was already published (no-op) or built.

  • Artifacts retain for 7 days; caches use GitHub’s 7-day LRU eviction with a weekly scheduled publish refresh. Denied writes cannot occur silently: write paths are removed from untrusted jobs entirely, and the contract tests fail if one is reintroduced.

Residual risks

  • The vN key components must be bumped by hand when archive build commands or toolkit selection change; a forgotten bump can reuse a stale archive until any source file changes.
  • actions: read on cuda-archive exposes read access to run metadata and artifacts of this repository to PR-controlled build code. This grants no write capability and the repository is public.
  • End-to-end cache-hit and retry behavior on paid RunPod hardware can only be demonstrated in live CI runs.
  • The manual PR-recovery path (workflow_dispatch with pr_number) builds and runs PR-authored code in a run whose ref is main. If GitHub’s cache service granted that run an implicit main-scope cache-write credential independent of GITHUB_TOKEN permissions, PR build scripts could attempt cache poisoning from inside the job. Observed behavior contradicts this: the read-only-token runs of this workflow had their cache saves rejected (#1403), indicating cache writes follow token permissions here. This pre-existing recovery path is tracked under the #1322 hardening umbrella; revisit if GitHub’s cache-credential model changes.