COMPAT.md asserts platform behaviour that nothing verifies, and the claims are duplicated into shipped strings #543

New issue

Open

opened 2026-09-17 04:50:00 +00:00 by hexajon · 0 comments

hexajon commented

2026-09-17 04:50:00 +00:00

Owner

docs/COMPAT.md is a hand-written table of Gitea-versus-Forgejo behaviour: 105 lines, 56 table rows, four explicit "Gitea-only" claims. Nothing checks that any of it is true.

It actively misleads readers, including agents, because its assertions are presented as settled fact and are wired into surfaces that carry authority.

The claims are not confined to the doc

internal/mcp/tools_actions.go:197 embeds the assertion directly in the tool description:

Forgejo/Gitea Actions (CI/CD): workflows, runs, jobs, logs, runners, artifacts, dispatch, and tasks. On Forgejo only runs, run_watch, dispatch, runners, secrets, variables, and tasks are available - the workflows, jobs/logs, rerun, artifacts, and run_delete families are Gitea-only (see docs/COMPAT.md).

That string is also generated into internal/mcp/help_generated.go:2352. So the doc's hand-maintained claim is what an agent reads when deciding whether an operation is possible, and it points back at the doc as its authority. Deleting the file alone would not remove the claim; it would only remove the pointer.

Nothing verifies it

The only test touching the file is internal/docs/compat_deps_test.go, and it caps table row length to stop the Notes column ballooning. It is a formatting guard. No test compares any claim in COMPAT.md against a live platform or against the vendored specs.

So every row is an assertion someone made once, ageing silently.

Evidence that it drifts

#504 measured live Gitea 1.27.3 against live Forgejo 15.0.6 and found a whole class of divergence that COMPAT.md does not capture at all: different field names for branch, commit, start and stop times, run number, workflow, title, and outcome. The doc says nothing about any of it.
#538 found the vendored spec that would be the natural reference for these claims is itself a minor version stale, with 8 Gitea paths missing.
One of those 8 missing paths is /repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt}/jobs, which is in the exact family COMPAT.md calls "Gitea-only". The claim is in the right direction, but the supporting data is out of date and the boundary has moved.

Ask

Retire the hand-maintained form. A compatibility claim should be derived from something checkable, or it should not be stated.

The design question worth deciding, rather than one I should pick:

Generate the platform-difference table from the live or vendored specs, so it cannot assert something the specs contradict.
Or delete it and let the runtime behaviour speak, since madtea already wraps a confirmed-unimplemented endpoint family as *gitea.ErrUnsupportedOnForgejo and degrades gracefully where a plain equivalent exists.

Either way the shipped strings in internal/mcp/tools_actions.go have to move with it. Under ADR 0023 they are self-contained, so they cannot quietly rely on a file that no longer exists or no longer says what they claim.

Acceptance

No hand-maintained assertion about platform capability survives without something that verifies it.
The MCP tool descriptions do not state a platform capability that nothing checks.
A claim that becomes false fails a test rather than misleading a reader.

Related: #538 (vendored specs unvalidated against live installs), #504 (field-shape divergence COMPAT.md misses), #542 (the live-forge check that would catch this class has never run).

`docs/COMPAT.md` is a hand-written table of Gitea-versus-Forgejo behaviour: 105 lines, 56 table rows, four explicit "Gitea-only" claims. Nothing checks that any of it is true. It actively misleads readers, including agents, because its assertions are presented as settled fact and are wired into surfaces that carry authority. ## The claims are not confined to the doc `internal/mcp/tools_actions.go:197` embeds the assertion directly in the tool description: > Forgejo/Gitea Actions (CI/CD): workflows, runs, jobs, logs, runners, artifacts, dispatch, and tasks. On Forgejo only runs, run_watch, dispatch, runners, secrets, variables, and tasks are available - the workflows, jobs/logs, rerun, artifacts, and run_delete families are Gitea-only (see docs/COMPAT.md). That string is also generated into `internal/mcp/help_generated.go:2352`. So the doc's hand-maintained claim is what an agent reads when deciding whether an operation is possible, and it points back at the doc as its authority. Deleting the file alone would not remove the claim; it would only remove the pointer. ## Nothing verifies it The only test touching the file is `internal/docs/compat_deps_test.go`, and it caps table row length to stop the Notes column ballooning. It is a formatting guard. No test compares any claim in COMPAT.md against a live platform or against the vendored specs. So every row is an assertion someone made once, ageing silently. ## Evidence that it drifts * #504 measured live Gitea 1.27.3 against live Forgejo 15.0.6 and found a whole class of divergence that COMPAT.md does not capture at all: different field names for branch, commit, start and stop times, run number, workflow, title, and outcome. The doc says nothing about any of it. * #538 found the vendored spec that would be the natural reference for these claims is itself a minor version stale, with 8 Gitea paths missing. * One of those 8 missing paths is `/repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt}/jobs`, which is in the exact family COMPAT.md calls "Gitea-only". The claim is in the right direction, but the supporting data is out of date and the boundary has moved. ## Ask Retire the hand-maintained form. A compatibility claim should be derived from something checkable, or it should not be stated. The design question worth deciding, rather than one I should pick: * Generate the platform-difference table from the live or vendored specs, so it cannot assert something the specs contradict. * Or delete it and let the runtime behaviour speak, since madtea already wraps a confirmed-unimplemented endpoint family as `*gitea.ErrUnsupportedOnForgejo` and degrades gracefully where a plain equivalent exists. Either way the shipped strings in `internal/mcp/tools_actions.go` have to move with it. Under ADR 0023 they are self-contained, so they cannot quietly rely on a file that no longer exists or no longer says what they claim. ## Acceptance * No hand-maintained assertion about platform capability survives without something that verifies it. * The MCP tool descriptions do not state a platform capability that nothing checks. * A claim that becomes false fails a test rather than misleading a reader. Related: #538 (vendored specs unvalidated against live installs), #504 (field-shape divergence COMPAT.md misses), #542 (the live-forge check that would catch this class has never run).

hexajon added the

bug

label

2026-09-17 04:50:00 +00:00

hexajon referenced this issue

2026-09-17 04:50:14 +00:00

api-specs: the vendored specs are hand-refreshed and Gitea's is a minor version stale #538

hexajon referenced this issue

2026-09-17 05:51:01 +00:00

Epic: madtea's external contracts are asserted, not verified #544

hexajon added a new dependency

2026-09-17 05:51:05 +00:00