sixfold-space/madtea

SHA256

Fork 0

ci: fix Forgejo Actions on the org runner - CI has never been green on Codeberg #47

New issue

Closed

opened 2026-07-13 21:04:45 +00:00 by hexajon · 4 comments

hexajon commented

2026-07-13 21:04:45 +00:00

(Migrated from codeberg.org)

Move CI/CD off the current .forgejo/workflows/ Actions setup (ci.yml + drift.yml) onto a proper Woodpecker CI pipeline — the Codeberg-native CI. Target: soon, not blocking current work.

Scope:

.woodpecker/ pipeline(s) covering what the Actions jobs run today — the full ./scripts/gate.sh check set (build/gofmt/vet/gofix/modtidy/hook-tests + the heavy analyzers: test-race, staticcheck, govulncheck, gosec, modernize, docs-verify, cross-compile, mcp-conformance) plus drift's surface-diff.
Reuse the gate script rather than re-encoding checks in pipeline YAML where practical (one source of truth for what "green" means).
Image choice: glibc-based Go image (the Alpine/musl "node: not found" failure class from #28 should be structurally avoided, and #28 may be resolved by this migration instead of patched in Actions).
Decide what happens to the .forgejo/workflows/ files once Woodpecker is green (remove, or keep drift.yml's gated forgejo-smoke if Woodpecker can't cover it).
Document the CI setup in docs/contributing/ (how to see runs, retry, required checks).

Acceptance:

A PR against main runs the Woodpecker pipeline and reports status on the PR.
Pipeline green ⇔ ./scripts/gate.sh green locally (no check lost in translation).
#28 closed or superseded with a note.

Move CI/CD off the current `.forgejo/workflows/` Actions setup (ci.yml + drift.yml) onto a proper Woodpecker CI pipeline — the Codeberg-native CI. Target: soon, not blocking current work. Scope: - `.woodpecker/` pipeline(s) covering what the Actions jobs run today — the full `./scripts/gate.sh` check set (build/gofmt/vet/gofix/modtidy/hook-tests + the heavy analyzers: test-race, staticcheck, govulncheck, gosec, modernize, docs-verify, cross-compile, mcp-conformance) plus drift's surface-diff. - Reuse the gate script rather than re-encoding checks in pipeline YAML where practical (one source of truth for what "green" means). - Image choice: glibc-based Go image (the Alpine/musl "node: not found" failure class from #28 should be structurally avoided, and #28 may be resolved by this migration instead of patched in Actions). - Decide what happens to the `.forgejo/workflows/` files once Woodpecker is green (remove, or keep drift.yml's gated forgejo-smoke if Woodpecker can't cover it). - Document the CI setup in docs/contributing/ (how to see runs, retry, required checks). Acceptance: - A PR against main runs the Woodpecker pipeline and reports status on the PR. - Pipeline green ⇔ `./scripts/gate.sh` green locally (no check lost in translation). - #28 closed or superseded with a note.

hexajon commented

2026-07-15 22:15:45 +00:00

(Migrated from codeberg.org)

Reframed: the Woodpecker migration is dropped; this issue is now "make the existing Forgejo Actions setup green on our own org runner".

Why not Woodpecker / Codeberg CI

Codeberg CI (ci.codeberg.org) needs a per-account access request - not enabled, not pursued.
A locally-hosted Woodpecker loses on security in every topology: it adds a public ingress we operate (the server's webhook endpoint), a stored forge OAuth credential on that surface, and fork-PR code execution on our hardware. Forgejo Actions on a self-hosted runner avoids all three - the runner polls outbound only and holds a registration token that can only pull jobs.
Direction: keep Forgejo Actions, run everything on the org's own runner (labels: docker, ubuntu-latest; forgejo-runner v12.13.0 in an isolated VM) - not Codeberg-hosted runners. The already-running Woodpecker VM predates this direction; decommissioning it is tracked outside this issue.

Root cause of the red (reproduced 2026-07-15)

Every Actions task in this repo's history failed - 1083 failures, 7 cancelled, 0 successes across 219 runs. Reproduced locally with forgejo-runner exec -W .forgejo/workflows/ci.yml -j vet inside the runner VM:

docker exec cmd=[node .../dist/restore/index.js]
OCI runtime exec failed: exec: "node": executable file not found in $PATH
exitcode '127': command not found

The JS-implemented actions (actions/checkout, actions/cache pinned from code.forgejo.org) execute node INSIDE the job container, and golang:1.26-trixie ships no node. Every job dies in 3-6s at its first uses: step. This corrects #28's Alpine/musl hypothesis: glibc did not help - no plain golang image carries node.

Plan

De-node the workflows (chosen over swapping to a node-bearing third-party image: keeps the pinned official golang image and removes the JS-action supply chain entirely):
- checkout via plain git in-container (git is present in golang:1.26-trixie; repo is public - anonymous fetch of the exact sha/PR ref).
- caching via runner-local named docker volumes for GOMODCACHE/GOCACHE (valid_volumes in runner config) instead of actions/cache - we own the runner, a shared volume is simpler and faster than the cache protocol.
Keep runs-on: docker (matches the runner's labels); fix drift.yml's self-hosted job label or retire it deliberately.
Reuse scripts/gate.sh as the single source of truth for what green means, per the original issue.
Document the runner setup + how to see/retry runs in docs/contributing/.

Acceptance (revised)

A PR against main runs green on the org runner; pipeline green matches ./scripts/gate.sh green locally.
No workflow step depends on node or fetched actions.
#28 closed or superseded with a note; docs updated.

Reframed: the Woodpecker migration is dropped; this issue is now "make the existing Forgejo Actions setup green on our own org runner". ## Why not Woodpecker / Codeberg CI - Codeberg CI (ci.codeberg.org) needs a per-account access request - not enabled, not pursued. - A locally-hosted Woodpecker loses on security in every topology: it adds a public ingress we operate (the server's webhook endpoint), a stored forge OAuth credential on that surface, and fork-PR code execution on our hardware. Forgejo Actions on a self-hosted runner avoids all three - the runner polls outbound only and holds a registration token that can only pull jobs. - Direction: keep Forgejo Actions, run everything on the org's own runner (labels: docker, ubuntu-latest; forgejo-runner v12.13.0 in an isolated VM) - not Codeberg-hosted runners. The already-running Woodpecker VM predates this direction; decommissioning it is tracked outside this issue. ## Root cause of the red (reproduced 2026-07-15) Every Actions task in this repo's history failed - 1083 failures, 7 cancelled, 0 successes across 219 runs. Reproduced locally with `forgejo-runner exec -W .forgejo/workflows/ci.yml -j vet` inside the runner VM: docker exec cmd=[node .../dist/restore/index.js] OCI runtime exec failed: exec: "node": executable file not found in $PATH exitcode '127': command not found The JS-implemented actions (actions/checkout, actions/cache pinned from code.forgejo.org) execute `node` INSIDE the job container, and `golang:1.26-trixie` ships no node. Every job dies in 3-6s at its first `uses:` step. This corrects #28's Alpine/musl hypothesis: glibc did not help - no plain golang image carries node. ## Plan 1. De-node the workflows (chosen over swapping to a node-bearing third-party image: keeps the pinned official golang image and removes the JS-action supply chain entirely): - checkout via plain git in-container (git is present in golang:1.26-trixie; repo is public - anonymous fetch of the exact sha/PR ref). - caching via runner-local named docker volumes for GOMODCACHE/GOCACHE (valid_volumes in runner config) instead of actions/cache - we own the runner, a shared volume is simpler and faster than the cache protocol. 2. Keep runs-on: docker (matches the runner's labels); fix drift.yml's self-hosted job label or retire it deliberately. 3. Reuse scripts/gate.sh as the single source of truth for what green means, per the original issue. 4. Document the runner setup + how to see/retry runs in docs/contributing/. ## Acceptance (revised) - A PR against main runs green on the org runner; pipeline green matches ./scripts/gate.sh green locally. - No workflow step depends on node or fetched actions. - #28 closed or superseded with a note; docs updated.

hexajon commented

2026-07-15 22:29:19 +00:00

(Migrated from codeberg.org)

Addendum to the plan: the mcp-conformance job's apt-get install -y nodejs npm step is itself a defect. Debian trixie's archive offers nodejs 20.19.2 + npm 9.2.0 (verified in the pinned golang:1.26-trixie image, 2026-07-15), and the project floor is Node 24.18.0 LTS - nothing may install an older runtime.

Replacement: install Node from the official nodejs.org dist tarball - pinned exact version (>= 24.18.0), SHA-256 verified against the published SHASUMS256.txt, extracted to /usr/local - instead of apt. nodejs.org is reachable from the runner VM (egress ACL is public-internet-only; confirmed with a 200 from the job network path). npm ships inside the Node tarball, which also retires the ancient apt npm 9.

Acceptance addition: no workflow step installs Node < 24.18.0 or uses Debian's nodejs/npm packages; the conformance job logs node --version >= 24.18.0 before npm ci.

Addendum to the plan: the mcp-conformance job's `apt-get install -y nodejs npm` step is itself a defect. Debian trixie's archive offers nodejs 20.19.2 + npm 9.2.0 (verified in the pinned golang:1.26-trixie image, 2026-07-15), and the project floor is Node 24.18.0 LTS - nothing may install an older runtime. Replacement: install Node from the official nodejs.org dist tarball - pinned exact version (>= 24.18.0), SHA-256 verified against the published SHASUMS256.txt, extracted to /usr/local - instead of apt. nodejs.org is reachable from the runner VM (egress ACL is public-internet-only; confirmed with a 200 from the job network path). npm ships inside the Node tarball, which also retires the ancient apt npm 9. Acceptance addition: no workflow step installs Node < 24.18.0 or uses Debian's nodejs/npm packages; the conformance job logs `node --version` >= 24.18.0 before `npm ci`.

hexajon commented

2026-07-15 22:32:53 +00:00

(Migrated from codeberg.org)

Plan additions:

gate.sh gains a first-class check (single source of truth - runs identically locally, in CI, and under release.sh): go env GOVERSION must exactly equal go.mod's go directive; loud failure naming both values otherwise.
CI sets GOTOOLCHAIN=local, so the image's own toolchain is what actually runs - a stale image digest then fails the honesty check instead of being papered over by an auto-download.
The conformance job's node check (>= 24.18.0, from the earlier addendum) joins the same pattern: assert-then-run, never assume.

Acceptance addition: gate.sh fails when run with a toolchain that does not match go.mod; CI proves it runs the pinned image's toolchain (GOTOOLCHAIN=local + the gate check green).

Second addendum: version-honesty gate. Today nothing asserts that the toolchain running the checks is the toolchain the project declares - the image digest (go1.26.5), the local toolchain (go1.26.5), and go.mod (go 1.26.5) merely happen to agree. GOTOOLCHAIN=auto makes drift silent: a stale go binary auto-downloads the declared toolchain and proceeds, so a rotted environment (stale image digest, stale /usr/local/go, wrong PATH go) is never surfaced. Plan additions: 1. gate.sh gains a first-class check (single source of truth - runs identically locally, in CI, and under release.sh): `go env GOVERSION` must exactly equal go.mod's `go` directive; loud failure naming both values otherwise. 2. CI sets GOTOOLCHAIN=local, so the image's own toolchain is what actually runs - a stale image digest then fails the honesty check instead of being papered over by an auto-download. 3. The conformance job's node check (>= 24.18.0, from the earlier addendum) joins the same pattern: assert-then-run, never assume. Acceptance addition: gate.sh fails when run with a toolchain that does not match go.mod; CI proves it runs the pinned image's toolchain (GOTOOLCHAIN=local + the gate check green).

hexajon commented

2026-07-16 00:59:39 +00:00

(Migrated from codeberg.org)

Green, live, on our own runner: ci.yml run #5591792 (https://codeberg.org/sixfold_space/madtea/actions/runs/246 era, run id 5591792) completed success on the org runner with every job passing - the first fully green Actions run in this repository's history (previously 1083/1083 tasks failed). What it took, all merged: de-node'd workflows (PR #135 - shell-git checkout, docker-volume Go caches, Node 24.18.0 pinned tarball for conformance, GOTOOLCHAIN=local + the gate's toolchain-honesty check), the sha256 object-format fix for checkout (PR #136 - this repo is a sha256 repo and a bare git init creates a sha1 client), root-skips for the 0o555 permission-barrier test family (PRs #137/#138), and hermetic committer identity in the initRepoAt fixture (PR #138). Runner-side valid_volumes allowlist applied and documented in docs/contributing/ci.md. drift.yml's smoke job now targets a real label. Pipeline green == gate.sh green, per the revised acceptance.

hexajon referenced this issue

2026-08-17 15:47:45 +00:00

release: adopt sigstore (cosign) signing for release artifacts #413

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

sixfold-space/madtea#47

No description provided.

Rows
Columns