ci: Actions re-run the full local gate on every push, burning runner time

hexajon commented

2026-08-17 15:52:26 +00:00

Owner

Problem

Every push and merge triggers CI Actions that re-run the same ./scripts/gate.sh plus conformance that the contributor already ran locally before finish (the collapsed gate plus conformance check set, #362). The runner is a VM; if its Go caches are not persisted between runs, it rebuilds and re-tests from scratch each time. Even a docs-only change re-runs the full heavy analyzer suite and conformance.

Concrete instance: #373 was gated green locally - build, vet, gofmt, staticcheck, govulncheck, gosec, test-race, mcp-conformance - and CI then re-ran the identical set on merge.

Why it matters

CI time is a real cost: runner occupancy, wall-clock before a merge reads as "done", and it compounds because the workflow is many small branch-to-finish cycles, not a few large ones. The same state is confirmed two or three times (local gate, sometimes a manual re-run, then CI).

Candidate levers (directions, not a decided design)

Persist/warm the Go caches in the runner (GOCACHE/GOMODCACHE across runs). Likely the single biggest win with zero coverage loss - a cold cache rebuilds and re-tests everything.
Path-filter the jobs: a docs-only diff skips the code analyzers and conformance; a code diff skips nothing. No coverage loss on the paths that matter.
Affected-package-only tests where safe (Go's test cache already helps once the cache persists).
Measure first: capture where the CI minutes actually go (cold rebuild vs analyzers vs conformance) before optimizing, so the fix targets the real cost.

Open question I need to settle

Should CI ever TRUST the local gate and skip re-verification for a locally-gated merge? That trades CI's independence - it verifies the merged state regardless of a committer's local claim, and it catches environment drift - for speed. This is a CI trust-boundary question, not an efficiency tweak, and I am not going to answer it as a side effect of a speed fix. The cache and path-filter levers above need no such trade, so they come first.

Acceptance criteria

CI wall-clock for a routine code merge is measurably reduced, with the heavy checks still running on code changes.
A docs-only change no longer runs the code analyzers or conformance.
The check SET on code changes is unchanged (no coverage dropped), unless I settle the trust-boundary question above otherwise.
The measurement of where CI time went is recorded on this issue, so the chosen fix is grounded.

## Problem Every push and merge triggers CI Actions that re-run the same `./scripts/gate.sh` plus conformance that the contributor already ran locally before `finish` (the collapsed gate plus conformance check set, #362). The runner is a VM; if its Go caches are not persisted between runs, it rebuilds and re-tests from scratch each time. Even a docs-only change re-runs the full heavy analyzer suite and conformance. Concrete instance: #373 was gated green locally - build, vet, gofmt, staticcheck, govulncheck, gosec, test-race, mcp-conformance - and CI then re-ran the identical set on merge. ## Why it matters CI time is a real cost: runner occupancy, wall-clock before a merge reads as "done", and it compounds because the workflow is many small branch-to-finish cycles, not a few large ones. The same state is confirmed two or three times (local gate, sometimes a manual re-run, then CI). ## Candidate levers (directions, not a decided design) - **Persist/warm the Go caches in the runner** (`GOCACHE`/`GOMODCACHE` across runs). Likely the single biggest win with zero coverage loss - a cold cache rebuilds and re-tests everything. - **Path-filter the jobs**: a docs-only diff skips the code analyzers and conformance; a code diff skips nothing. No coverage loss on the paths that matter. - **Affected-package-only tests** where safe (Go's test cache already helps once the cache persists). - **Measure first**: capture where the CI minutes actually go (cold rebuild vs analyzers vs conformance) before optimizing, so the fix targets the real cost. ## Open question I need to settle Should CI ever TRUST the local gate and skip re-verification for a locally-gated merge? That trades CI's independence - it verifies the merged state regardless of a committer's local claim, and it catches environment drift - for speed. This is a CI trust-boundary question, not an efficiency tweak, and I am not going to answer it as a side effect of a speed fix. The cache and path-filter levers above need no such trade, so they come first. ## Acceptance criteria - CI wall-clock for a routine code merge is measurably reduced, with the heavy checks still running on code changes. - A docs-only change no longer runs the code analyzers or conformance. - The check SET on code changes is unchanged (no coverage dropped), unless I settle the trust-boundary question above otherwise. - The measurement of where CI time went is recorded on this issue, so the chosen fix is grounded.

hexajon added the

status/needs-decision

label

2026-08-17 15:52:26 +00:00

hexajon commented

2026-08-17 15:52:35 +00:00

Author

Owner

The no-trade levers come first, because neither costs any coverage: persist and warm the runner Go caches (GOCACHE/GOMODCACHE) across runs, which is likely the biggest single win, and path-filter the jobs so a docs-only diff skips the code analyzers and conformance while a code diff skips nothing. Measuring where the CI minutes actually go should precede either, so the fix targets the real cost instead of the suspected one.

Plan: land the path-filter first, since it is repo-only and loses no coverage, and file the runner cache-persist plus a measurement pass separately.

The trust-boundary question in the body stays open and separate. It is not part of this speed work.

The no-trade levers come first, because neither costs any coverage: persist and warm the runner Go caches (`GOCACHE`/`GOMODCACHE`) across runs, which is likely the biggest single win, and path-filter the jobs so a docs-only diff skips the code analyzers and conformance while a code diff skips nothing. Measuring where the CI minutes actually go should precede either, so the fix targets the real cost instead of the suspected one. Plan: land the path-filter first, since it is repo-only and loses no coverage, and file the runner cache-persist plus a measurement pass separately. The trust-boundary question in the body stays open and separate. It is not part of this speed work.

hexajon self-assigned this

2026-08-17 17:45:48 +00:00

hexajon commented

2026-08-17 19:06:56 +00:00

Author

Owner

Grounded the where-does-the-time-go question against the config. Findings:

CI is a re-run of the local gate: the gate job is ./scripts/gate.sh --no-conformance (ci.yml), with mcp-conformance as its own job, so conformance runs once per CI run, not twice.
The caches ARE persisted, just not on PRs. GOCACHE/GOMODCACHE are runner docker volumes mounted only when the event is not pull_request (the #272 fork-cache-poisoning guard). So push/schedule runs are warm; every pull_request run compiles cold and re-runs the full heavy suite. That is the slow path, and it is the one now on the merge critical path.
Single org runner, jobs serialize (no needs:, but one runner). No concurrency: block, so a superseded push keeps running instead of being cancelled.
No paths: filters, so a docs-only diff still runs the full analyzer suite.
Required-check interaction (new): main now requires ci / gate (pull_request) + ci / mcp-conformance (pull_request). A path-filter that SKIPS the gate job on a docs-only diff would leave the required check unreported and deadlock the merge, so a path-filter has to report success while no-oping the heavy steps, not skip the job.

First no-trade lever going in (PR incoming, refs this issue): a concurrency: cancel-in-progress block, cancelling superseded runs on non-default refs while letting main runs complete. No coverage or trust change.

Still open for a ruling, unchanged:

Warming the PR cache means relaxing the #272 fork-poisoning guard, a trust boundary that matters more once the repo is public with external forks.
The path-filter, in the report-but-no-op shape the required-check interaction above forces.
The CI-trust-the-local-gate question already parked here.

Grounded the where-does-the-time-go question against the config. Findings: - CI is a re-run of the local gate: the `gate` job is `./scripts/gate.sh --no-conformance` (ci.yml), with `mcp-conformance` as its own job, so conformance runs once per CI run, not twice. - **The caches ARE persisted, just not on PRs.** `GOCACHE`/`GOMODCACHE` are runner docker volumes mounted only when the event is not `pull_request` (the #272 fork-cache-poisoning guard). So push/schedule runs are warm; every `pull_request` run compiles cold and re-runs the full heavy suite. That is the slow path, and it is the one now on the merge critical path. - Single org runner, jobs serialize (no `needs:`, but one runner). No `concurrency:` block, so a superseded push keeps running instead of being cancelled. - No `paths:` filters, so a docs-only diff still runs the full analyzer suite. - Required-check interaction (new): main now requires `ci / gate (pull_request)` + `ci / mcp-conformance (pull_request)`. A path-filter that SKIPS the gate job on a docs-only diff would leave the required check unreported and deadlock the merge, so a path-filter has to report success while no-oping the heavy steps, not skip the job. First no-trade lever going in (PR incoming, refs this issue): a `concurrency: cancel-in-progress` block, cancelling superseded runs on non-default refs while letting main runs complete. No coverage or trust change. Still open for a ruling, unchanged: - Warming the PR cache means relaxing the #272 fork-poisoning guard, a trust boundary that matters more once the repo is public with external forks. - The path-filter, in the report-but-no-op shape the required-check interaction above forces. - The CI-trust-the-local-gate question already parked here.

hexajon referenced this issue from a commit

2026-08-17 19:08:45 +00:00

ci: cancel superseded in-progress runs

hexajon referenced this issue

2026-08-17 19:08:51 +00:00

ci: cancel superseded in-progress runs #433

hexajon referenced this issue

2026-08-17 20:28:16 +00:00

ci: skip the redundant push re-run of an already-gated SHA #435

hexajon commented

2026-08-17 20:28:53 +00:00

Author

Owner

Added a lever beyond the ones listed above: SHA-dedup of the push re-run (PR #435, auto-merges on green).

Because merges are fast-forward-only, main's tip is the PR head commit byte for byte, and CI already ran the full gate on that exact SHA as a pull_request event. The gate and mcp-conformance jobs now check, on push only, for a prior (pull_request) success on the SHA and skip the heavy steps when it exists. It fails safe: a PR run is never skipped, and any doubt (no prior success, status API unreachable) runs the full job, so a direct-to-main push still gets the gate.

Narrow ruling for this lever only: it trusts a prior CI run on the identical commit, not a contributor's local gate, so I am treating it as no-trade. That PR run is also the cold one (no cache, #272), i.e. the stronger check, so nothing is lost.

This does NOT settle the open question in the body. "Should CI ever trust the local gate and skip re-verification" stays open and keeps status/needs-decision. The path-filter, the PR-cache trade, and the measurement pass are all still unchanged.

Added a lever beyond the ones listed above: SHA-dedup of the push re-run (PR #435, auto-merges on green). Because merges are fast-forward-only, main's tip is the PR head commit byte for byte, and CI already ran the full gate on that exact SHA as a `pull_request` event. The `gate` and `mcp-conformance` jobs now check, on `push` only, for a prior `(pull_request)` success on the SHA and skip the heavy steps when it exists. It fails safe: a PR run is never skipped, and any doubt (no prior success, status API unreachable) runs the full job, so a direct-to-main push still gets the gate. Narrow ruling for this lever only: it trusts a prior CI run on the identical commit, not a contributor's local gate, so I am treating it as no-trade. That PR run is also the cold one (no cache, #272), i.e. the stronger check, so nothing is lost. This does NOT settle the open question in the body. "Should CI ever trust the local gate and skip re-verification" stays open and keeps `status/needs-decision`. The path-filter, the PR-cache trade, and the measurement pass are all still unchanged.

hexajon commented

2026-09-08 20:53:50 +00:00

Author

Owner

Decided: routine internal CI maintenance, not a standing product decision. Clearing needs-decision.

This is madtea's own repository CI (the gate + conformance re-run), so it is project devops, not a product capability. It does not need product-decision ceremony; the safe posture is also the live-correct one, so I am applying it.

Posture:

CI stays independent: it re-verifies the merged state and does not trust a committer's local gate.
The #272 fork-cache-poisoning guard stays in place. madtea is public, so external fork PRs are a live threat; fork PRs keep compiling cold with no mounted cache, and the fork-PR cache is not warmed.

Speed comes from the no-trade levers, none of which trade coverage:

concurrency cancel-in-progress on non-default refs (landed);
SHA-dedup: a push skips the heavy steps when a prior pull_request run already passed on the identical fast-forward SHA, trusting a prior CI run rather than a local claim (PR #435, landed);
a docs-only path-filter in report-success-but-no-op shape, so the required gate and conformance checks stay reported and the merge never deadlocks;
a measurement pass recording where the runner minutes go, so further tuning targets the real cost.

The remaining levers are execution, not a decision.

Decided: routine internal CI maintenance, not a standing product decision. Clearing needs-decision. This is madtea's own repository CI (the gate + conformance re-run), so it is project devops, not a product capability. It does not need product-decision ceremony; the safe posture is also the live-correct one, so I am applying it. Posture: - CI stays independent: it re-verifies the merged state and does not trust a committer's local gate. - The #272 fork-cache-poisoning guard stays in place. madtea is public, so external fork PRs are a live threat; fork PRs keep compiling cold with no mounted cache, and the fork-PR cache is not warmed. Speed comes from the no-trade levers, none of which trade coverage: - concurrency cancel-in-progress on non-default refs (landed); - SHA-dedup: a push skips the heavy steps when a prior pull_request run already passed on the identical fast-forward SHA, trusting a prior CI run rather than a local claim (PR #435, landed); - a docs-only path-filter in report-success-but-no-op shape, so the required gate and conformance checks stay reported and the merge never deadlocks; - a measurement pass recording where the runner minutes go, so further tuning targets the real cost. The remaining levers are execution, not a decision.

hexajon removed the

status/needs-decision

label

2026-09-08 20:53:50 +00:00

Rows
Columns

ci: Actions re-run the full local gate on every push, burning runner time #426

Problem

Why it matters

Candidate levers (directions, not a decided design)

Open question I need to settle

Acceptance criteria