bug(forge-test): the throwaway-forge binary download times out after a base-snapshot reset #796

New issue

Closed

opened 2026-10-17 16:48:28 +00:00 by hexajon · 0 comments

hexajon commented

2026-10-17 16:48:28 +00:00

Owner

What happens

run-forge-tests.sh resets the VM to its base snapshot before every run. The snapshot predates any downloaded forge binaries, so the binary cache is cold each run and the binaries download fresh. On 2026-10-16 the Forgejo baseline download stalled and the test hit the 10-minute timeout:

panic: test timed out after 10m0s
    running tests:
        TestForgeProvenRemoval (10m0s)
        TestForgeProvenRemoval/forgejo/baseline (10m0s)
...
madtea.lol/internal/testforge.ResolveBinary(...) internal/testforge/binary.go:112
FAIL  madtea.lol/internal/testforge  600.010s

The stall is in the binary fetch, not a forge operation.

Why it matters

This test is the acceptance proof for the forge-proof-before-removal guard (#666). The timeout produced no verdict, so #666 cannot close on a real forge run until the suite completes.

Possible directions

Keep the binaries on a cache that survives the snapshot reset (a VM-local cache outside the snapshot, or a snapshot taken after a warm download). The cache cannot live on a virtiofs mount because of mmap.
Or pre-stage the pinned binaries into the base snapshot at provision time, or raise the per-run timeout.

Relates to #677, #692, #730.

## What happens run-forge-tests.sh resets the VM to its base snapshot before every run. The snapshot predates any downloaded forge binaries, so the binary cache is cold each run and the binaries download fresh. On 2026-10-16 the Forgejo baseline download stalled and the test hit the 10-minute timeout: panic: test timed out after 10m0s running tests: TestForgeProvenRemoval (10m0s) TestForgeProvenRemoval/forgejo/baseline (10m0s) ... madtea.lol/internal/testforge.ResolveBinary(...) internal/testforge/binary.go:112 FAIL madtea.lol/internal/testforge 600.010s The stall is in the binary fetch, not a forge operation. ## Why it matters This test is the acceptance proof for the forge-proof-before-removal guard (#666). The timeout produced no verdict, so #666 cannot close on a real forge run until the suite completes. ## Possible directions - Keep the binaries on a cache that survives the snapshot reset (a VM-local cache outside the snapshot, or a snapshot taken after a warm download). The cache cannot live on a virtiofs mount because of mmap. - Or pre-stage the pinned binaries into the base snapshot at provision time, or raise the per-run timeout. Relates to #677, #692, #730.

hexajon added the

bug

label

2026-10-17 16:48:28 +00:00

hexajon referenced this issue

2026-10-17 16:49:03 +00:00

guard: prove every removal with the forge's answer, not local state #666

hexajon referenced this issue from a pull request that will close it,

2026-10-18 01:56:09 +00:00

test(forge-test): #796 cache pinned binaries in a host store and copy them into the VM #799

hexajon referenced this issue from a commit

2026-10-18 01:56:11 +00:00