feat(gate): self-heal preflight for a poisoned module cache - resolution canary, one loud purge-and-retry #132

Closed
opened 2026-07-15 23:06:52 +00:00 by hexajon · 0 comments
hexajon commented 2026-07-15 23:06:52 +00:00 (Migrated from codeberg.org)

A machine-side cache incident (2026-07-15: an external cache GC file-pruned the shared GOMODCACHE and hollowed out extracted module dirs) made ./scripts/gate.sh fail with cryptic per-package errors ("no required module provides package github.com/spf13/cobra" with a perfectly correct go.mod). Go auto-fetches MISSING modules, but a mutated-yet-present cache is trusted by contract - so the gate red-fails with no hint the cache is the culprit, and recovery took human archaeology.

Ask

gate.sh gains a cheap preflight before the check matrix:

  1. Canary: resolve dependencies for one known import (e.g. go list -deps ./internal/cmdresult or go build -n on a tiny package). Cost when healthy: sub-second.
  2. On canary failure: print a loud, single-line diagnosis naming the module cache as the suspect, run go clean -modcache (and -cache if the second attempt still fails), re-run the canary once, and continue if healed. One retry, never a loop.
  3. If still failing after the purge: fail with the canary error AND the purge already ruled out - so I start at the real cause, not the cache.

Acceptance

  • A deliberately corrupted GOMODCACHE (test can simulate by deleting files inside an extracted module dir in a throwaway GOMODCACHE) yields a gate run that heals itself and passes.
  • Healthy-path overhead is negligible; the purge only ever triggers on canary failure.
  • The heal event is unmistakable in gate output (not silent).
A machine-side cache incident (2026-07-15: an external cache GC file-pruned the shared GOMODCACHE and hollowed out extracted module dirs) made ./scripts/gate.sh fail with cryptic per-package errors ("no required module provides package github.com/spf13/cobra" with a perfectly correct go.mod). Go auto-fetches MISSING modules, but a mutated-yet-present cache is trusted by contract - so the gate red-fails with no hint the cache is the culprit, and recovery took human archaeology. ## Ask gate.sh gains a cheap preflight before the check matrix: 1. Canary: resolve dependencies for one known import (e.g. `go list -deps ./internal/cmdresult` or `go build -n` on a tiny package). Cost when healthy: sub-second. 2. On canary failure: print a loud, single-line diagnosis naming the module cache as the suspect, run `go clean -modcache` (and `-cache` if the second attempt still fails), re-run the canary once, and continue if healed. One retry, never a loop. 3. If still failing after the purge: fail with the canary error AND the purge already ruled out - so I start at the real cause, not the cache. ## Acceptance - A deliberately corrupted GOMODCACHE (test can simulate by deleting files inside an extracted module dir in a throwaway GOMODCACHE) yields a gate run that heals itself and passes. - Healthy-path overhead is negligible; the purge only ever triggers on canary failure. - The heal event is unmistakable in gate output (not silent).
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
sixfold-space/madtea#132
No description provided.