forge-test gate: keep GOCACHE off the virtiofs cache mount (Go mmap fails) #730

Closed
opened 2026-10-06 17:00:58 +00:00 by hexajon · 1 comment
Owner

The forge-test Incus VM mounts a persistent host cache over virtiofs so caches survive the per-run snapshot reset. A Go build that points GOCACHE or GOMODCACHE at that mount fails: every import reports "could not import (no such device)" (ENXIO). Go memory-maps its cache artifacts and the virtiofs mount rejects mmap; a plain read of the same file works, so the failure looks like a filesystem fault rather than a build error.

In testing, a go test -tags forge build with the caches on the VM's local disk passes; the same build with the caches on the virtiofs mount fails at import. The forge matrix passes end to end only with local-disk caches.

What to do

The gate step that runs the forge tests in the VM must set GOCACHE and GOMODCACHE to a VM-local path, not the virtiofs cache mount. The downloaded forge binaries are executed, and the kernel memory-maps an executable, so they are likely to hit the same limit on virtiofs; keep the resolved-binary cache off the virtiofs mount too, or enable virtiofs DAX on the mount and confirm a build passes.

Acceptance

  • The forge-test gate step runs go test -tags forge in the VM with the build and module caches on VM-local disk, and it passes.
  • The persistent-cache design still warms what it can across runs without breaking the build, and the doc states which caches live where.

Blocks #692.

The forge-test Incus VM mounts a persistent host cache over virtiofs so caches survive the per-run snapshot reset. A Go build that points GOCACHE or GOMODCACHE at that mount fails: every import reports "could not import <pkg> (no such device)" (ENXIO). Go memory-maps its cache artifacts and the virtiofs mount rejects mmap; a plain read of the same file works, so the failure looks like a filesystem fault rather than a build error. In testing, a `go test -tags forge` build with the caches on the VM's local disk passes; the same build with the caches on the virtiofs mount fails at import. The forge matrix passes end to end only with local-disk caches. ## What to do The gate step that runs the forge tests in the VM must set GOCACHE and GOMODCACHE to a VM-local path, not the virtiofs cache mount. The downloaded forge binaries are executed, and the kernel memory-maps an executable, so they are likely to hit the same limit on virtiofs; keep the resolved-binary cache off the virtiofs mount too, or enable virtiofs DAX on the mount and confirm a build passes. ## Acceptance - The forge-test gate step runs `go test -tags forge` in the VM with the build and module caches on VM-local disk, and it passes. - The persistent-cache design still warms what it can across runs without breaking the build, and the doc states which caches live where. Blocks #692.
Author
Owner

Resolved. The persistent virtiofs cache mount this is about is gone. It was a stray gocache disk device (a host cache dir mounted at /cache); provision.sh now strips any device that is not the NIC or root disk and asserts the set. The runner keeps GOCACHE, GOMODCACHE, and the binary cache at the test user's $HOME on VM-local disk, and go test -tags forge ./internal/testforge/... passes green in the VM (PR #760). ADR 0052 accepted that local-disk caches give up cross-run warming for a correct build, so no virtiofs cache mount is left to keep GOCACHE off.

Resolved. The persistent virtiofs cache mount this is about is gone. It was a stray `gocache` disk device (a host cache dir mounted at /cache); provision.sh now strips any device that is not the NIC or root disk and asserts the set. The runner keeps GOCACHE, GOMODCACHE, and the binary cache at the test user's $HOME on VM-local disk, and `go test -tags forge ./internal/testforge/...` passes green in the VM (PR #760). ADR 0052 accepted that local-disk caches give up cross-run warming for a correct build, so no virtiofs cache mount is left to keep GOCACHE off.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
sixfold-space/madtea#730
No description provided.