CI/CD Pipeline Standards
CI/CD Pipeline Standards
Section titled “CI/CD Pipeline Standards”These standards govern how continuous-integration pipelines are structured so they
stay fast, cheap, and trustworthy as a repository grows. They assume the
make-based automation in
arch-02_automation_standards.md and the
workspace model in
arch-06_monorepo_workspace_standards.md.
CI minutes cost real money, and per-job setup overhead is itself a cost. Treat the pipeline as something to be measured and optimized, not left to sprawl.
1. Local Gate Mirrors CI Exactly
Section titled “1. Local Gate Mirrors CI Exactly”- The local gate (
make ci) MUST run the same checks as the required CI gate — same linters, same coverage thresholds, same build. “Green locally, red in CI” (or the reverse) means the gate has drifted; fix the drift, don’t paper over it. - CI SHOULD invoke the same
maketargets a developer runs, so there is one definition of “passing” and no CI-only logic to keep in sync.
2. Consolidate the Required Gate; Build Once
Section titled “2. Consolidate the Required Gate; Build Once”- Prefer one consolidated required gate over a sprawl of overlapping per-member workflows that each cold-install dependencies and rebuild shared packages. A task runner with a dependency graph (turbo, nx, bazel, gradle) runs each unit of work once across the workspace.
- Build each artifact exactly once per run. Two jobs building the same output in parallel waste minutes and can race on a shared output directory. Produce the build once and let dependent jobs consume it.
- Keep the required gate minimal and fast. Move advisory checks (dependency CVE audit, license scan, link-checking) into separate jobs that do not block merge — a transitive advisory should not wedge the required gate.
3. Don’t Run Everything on Every PR
Section titled “3. Don’t Run Everything on Every PR”- Cancel superseded runs. Set a concurrency group keyed on workflow + ref with cancel-in-progress, so a new push cancels the previous run instead of paying for both.
- Path filters / affected-only. Only run a member’s gate when files in its dependency closure changed. Document each member’s filter (and why it is what it is) next to the filter.
- Tier the heavy suites. Expensive matrices — full cross-browser E2E, visual-regression across themes, multi-OS builds — run on a reduced set per PR (e.g. one browser) and the full matrix on a nightly schedule plus manual dispatch. State the reduction explicitly; never let “we only test one browser on PRs” be a silent gap.
4. Caching
Section titled “4. Caching”- Cache the expensive, deterministic inputs: dependency stores, compiled artifacts, browser binaries, task-runner outputs.
- Pin cache-action and tool versions to an exact version — never a floating major tag — so a cache key or action upgrade can’t silently change behavior mid-stream.
- Key caches precisely (OS + tool version + lockfile hash + browser set). Use
restore-keysto fall back from a narrow set to a superset, never the reverse. - Give remote/task caches a bounded budget with eviction (LRU to a size cap on a schedule). An unbounded cache grows until it costs more than it saves.
- On a persistent self-hosted runner, prefer the runner’s local on-disk store over re-downloading a cache tarball every job — the redundant fetch can dominate setup time.
5. Artifacts & Retention
Section titled “5. Artifacts & Retention”- Set an explicit short retention on build/test artifacts (screenshots, logs, coverage reports) — they accumulate storage cost. A week is a reasonable default unless a longer window is needed for a specific audit.
- Upload diagnostic artifacts (failing screenshots, logs) on failure so a red run is debuggable without re-running.
6. Self-Hosted Runners
Section titled “6. Self-Hosted Runners”Self-hosted runners trade per-minute cost for operational responsibility. If used:
- A job pinned to a self-hosted label queues indefinitely if no matching runner is online — there is no hosted fallback. A check stuck “queued/pending” on a self-hosted job means the runner is down, not that the code is slow. Document the runner’s labels and who owns it.
runs-onlabels must match the runner’s actual auto-labels (a macOS runner has nolinuxlabel). A mismatch means the job never schedules.- Pin the toolchain the runner self-provisions (e.g. via a
rust-toolchain.toml,.nvmrc,.tool-versions) so a fresh runner reproduces the gate deterministically. - Avoid shared mutable state between concurrent runners on one host. Two
runners self-installing into the same shared tool directory will race and corrupt
it (e.g.
ENOTEMPTYduring a parallel dependency-manager install). Give each runner its own home/work directory, or reduce concurrency, or move the self-install into per-runner scope. - Isolate every runner on a shared host. Each runner MUST get its own
HOME, tool-cache, and_workdirectory. Never share package-manager setup dirs (e.g.~/setup-pnpm) across runners on one machine — sharing causes duplicate-listener_diagcollisions andpnpm/action-setupraces (ENOTEMPTY/reflink) on large fan-out. - Diagnose by failure mode. An offline runner yields a run that stays queued/waiting (nothing schedules it). A collision/misconfig yields a run that fails at “Set up job” within seconds. Same-looking red check, opposite cause — read which one before touching the code.
7. Deploy Channels & Promotion
Section titled “7. Deploy Channels & Promotion”For products that deploy from CI, separate staging from production by channel and gate promotion explicitly:
- Pushes to the integration branch deploy to a staging/preview channel; production
deploys are gated on an explicit promotion (a long-lived
releasebranch, a tag, or a manual approval). Never deploy production off arbitrary feature branches. - If the platform’s branch controls take no wildcard, use a single long-lived
promotion branch (
release), not a pattern (release/*), and promote by fast-forwarding into it. - A green pipeline is not proof production updated. The release job MUST end
with an automated production smoke check — fetch the live deployment and assert
the new version/build — not merely trust that the promote step exited 0. Deploy
credentials and branch protections live outside the repo and fail invisibly (a
rolled/deleted build token, a force-push-protected
releasebranch blocking a fast-forward promote). An in-repo green is necessary but not sufficient. - See proc-02_git_version_control_standards.md §5 for the release-version vs build-version model and the scripted release PR that drives promotion.