api-specs: the vendored specs are hand-refreshed and Gitea's is a minor version stale

hexajon commented

2026-09-17 02:24:32 +00:00

Owner

surface-diff checks madtea's endpoint surface against docs/api-specs/gitea-1.26.1.json and docs/api-specs/forgejo-15.0.json. Those files are hand-vendored snapshots. Nothing verifies they match a live install, so the check is only as good as the last manual refresh.

Measured on 2026-09-17 against live instances:

platform	vendored	live	path drift
Forgejo	15.0.1	15.0.6	none, 314 vs 314
Gitea	1.26.1	1.27.3	8 paths missing from the vendored copy

Missing from the vendored Gitea spec:

/repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt}
/repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt}/jobs
/repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs
/repos/{owner}/{repo}/assignees/{assignee}
/repos/{owner}/{repo}/issues/{index}/assignees
/repos/{owner}/{repo}/issues/{index}/assignees/{assignee}
/repos/{owner}/{repo}/pulls/{index}/comments/{id}/replies
/token

Why it matters

TestAPISurfaceMatchesSpec classifies an endpoint missing from every spec as orphaned, which it treats as a bug. Against a stale spec that verdict is wrong: an endpoint that exists in Gitea 1.27 reads as nonexistent, and the obvious response to a failing test is to delete the calling code. A check that produces confident false verdicts is worse than no check.

The Forgejo copy happens to be accurate because patch releases do not move the API surface. That is recency, not a guarantee. The exposure is a minor or major bump, which is exactly what happened on the Gitea side.

Why it went unnoticed

The refresh is a manual instruction in a code comment: "Refresh the JSON in docs/api-specs/ when a new stable lands." Manual steps rot.
Nothing asserts the vendored files are current.
forgejo-smoke does exercise a live Forgejo, but it is gated on vars.RUN_FORGEJO_SMOKE == 'true' and this repository defines no Actions variables at all, so it has never run.
There is no live Gitea check of any kind.

Ask

Both platforms serve their own spec at /swagger.v1.json, so this is mechanical rather than aspirational. Fetching it from a live instance of each platform and comparing against the vendored copy takes one request per platform, and it is how the vendored files should have been validated all along.

Check each platform independently. They diverge, that divergence is the entire reason two specs are vendored, and a single combined check would hide exactly the case that just occurred.

Acceptance

Vendored specs are verified against a live instance of each platform, not by hand.
A vendored spec that no longer matches its platform fails, and the failure names the platform and the version.
Refreshing a spec is one command.
The check states which version of each platform it validated against.

Related: #537 covers the same class of gap for the git CLI surface, which has no generated guarantee at all.

`surface-diff` checks madtea's endpoint surface against `docs/api-specs/gitea-1.26.1.json` and `docs/api-specs/forgejo-15.0.json`. Those files are hand-vendored snapshots. Nothing verifies they match a live install, so the check is only as good as the last manual refresh. Measured on 2026-09-17 against live instances: | platform | vendored | live | path drift | |---|---|---|---| | Forgejo | 15.0.1 | 15.0.6 | none, 314 vs 314 | | Gitea | **1.26.1** | **1.27.3** | **8 paths missing from the vendored copy** | Missing from the vendored Gitea spec: ``` /repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt} /repos/{owner}/{repo}/actions/runs/{run}/attempts/{attempt}/jobs /repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs /repos/{owner}/{repo}/assignees/{assignee} /repos/{owner}/{repo}/issues/{index}/assignees /repos/{owner}/{repo}/issues/{index}/assignees/{assignee} /repos/{owner}/{repo}/pulls/{index}/comments/{id}/replies /token ``` ## Why it matters `TestAPISurfaceMatchesSpec` classifies an endpoint missing from every spec as `orphaned`, which it treats as a bug. Against a stale spec that verdict is wrong: an endpoint that exists in Gitea 1.27 reads as nonexistent, and the obvious response to a failing test is to delete the calling code. A check that produces confident false verdicts is worse than no check. The Forgejo copy happens to be accurate because patch releases do not move the API surface. That is recency, not a guarantee. The exposure is a minor or major bump, which is exactly what happened on the Gitea side. ## Why it went unnoticed * The refresh is a manual instruction in a code comment: "Refresh the JSON in `docs/api-specs/` when a new stable lands." Manual steps rot. * Nothing asserts the vendored files are current. * `forgejo-smoke` does exercise a live Forgejo, but it is gated on `vars.RUN_FORGEJO_SMOKE == 'true'` and this repository defines no Actions variables at all, so it has never run. * There is no live Gitea check of any kind. ## Ask Both platforms serve their own spec at `/swagger.v1.json`, so this is mechanical rather than aspirational. Fetching it from a live instance of each platform and comparing against the vendored copy takes one request per platform, and it is how the vendored files should have been validated all along. Check each platform independently. They diverge, that divergence is the entire reason two specs are vendored, and a single combined check would hide exactly the case that just occurred. ## Acceptance * Vendored specs are verified against a live instance of each platform, not by hand. * A vendored spec that no longer matches its platform fails, and the failure names the platform and the version. * Refreshing a spec is one command. * The check states which version of each platform it validated against. Related: #537 covers the same class of gap for the git CLI surface, which has no generated guarantee at all.

hexajon added the

bug

label

2026-09-17 02:24:32 +00:00

hexajon commented

2026-09-17 02:45:54 +00:00

Author

Owner

A second consequence of the vendored specs, which strengthens the case for fetching them live: surface-diff on the daily cron is deterministic and cannot discover anything.

It compares the endpoint surface extracted from the code against spec files in docs/api-specs/. Both inputs live in the commit. Neither can change between cron runs, so a scheduled run computes an identical result every day. If something did change, that change arrived in a commit, and the push trigger already ran the check.

So the daily run currently does this:

job	on cron	effect
`govulncheck`	yes	genuine. The vulnerability database moves under a static commit, so the answer really does change between pushes.
`surface-diff`	yes	no-op. Deterministic given the commit.
`forgejo-smoke`	gated	never runs. Needs `RUN_FORGEJO_SMOKE`, and this repository defines no Actions variables.

drift.yml's stated purpose is "catch regressions that appear WITHOUT the code changing". surface-diff cannot serve that purpose while its contract is a file in the same commit as the code. A drift check whose reference is frozen in the repository is not a drift check.

Fetching each platform's spec from a live instance fixes both symptoms at once. The stale Gitea copy stops being possible, and the cron run starts answering a question it currently only appears to answer: did Gitea or Forgejo change under us since yesterday.

Worth deciding alongside the fix: whether surface-diff should run on the cron at all until its reference is live. Today it consumes a slot on the single runner to recompute a known answer.

A second consequence of the vendored specs, which strengthens the case for fetching them live: **`surface-diff` on the daily cron is deterministic and cannot discover anything.** It compares the endpoint surface extracted from the code against spec files in `docs/api-specs/`. Both inputs live in the commit. Neither can change between cron runs, so a scheduled run computes an identical result every day. If something did change, that change arrived in a commit, and the `push` trigger already ran the check. So the daily run currently does this: | job | on cron | effect | |---|---|---| | `govulncheck` | yes | genuine. The vulnerability database moves under a static commit, so the answer really does change between pushes. | | `surface-diff` | yes | **no-op.** Deterministic given the commit. | | `forgejo-smoke` | gated | never runs. Needs `RUN_FORGEJO_SMOKE`, and this repository defines no Actions variables. | `drift.yml`'s stated purpose is "catch regressions that appear WITHOUT the code changing". `surface-diff` cannot serve that purpose while its contract is a file in the same commit as the code. A drift check whose reference is frozen in the repository is not a drift check. Fetching each platform's spec from a live instance fixes both symptoms at once. The stale Gitea copy stops being possible, and the cron run starts answering a question it currently only appears to answer: did Gitea or Forgejo change under us since yesterday. Worth deciding alongside the fix: whether `surface-diff` should run on the cron at all until its reference is live. Today it consumes a slot on the single runner to recompute a known answer.

hexajon referenced this issue

2026-09-17 02:46:56 +00:00

ci: a failing scheduled run notifies nobody #540

hexajon referenced this issue

2026-09-17 03:06:00 +00:00

ci: the forgejo-smoke job has never run #542

hexajon referenced this issue

2026-09-17 04:50:00 +00:00

COMPAT.md asserts platform behaviour that nothing verifies, and the claims are duplicated into shipped strings #543

hexajon commented

2026-09-17 04:50:14 +00:00

Author

Owner

Cross-link: #504 already recorded this on 2026-09-15, two days before I measured it here. Its closing note reads "The vendored specs are older than the live servers (docs/api-specs/gitea-1.26.1.json against a live 1.27.3)", and its own ask ends with "Check both live forges before merge, not only the vendored specs."

So the staleness was observed, written down as an aside inside a different issue, and then found again independently. That is worth noting on its own: the finding existed and still did not get acted on, because it was a supporting detail in an issue about something else rather than a tracked item of its own.

#543 is the third instance of the same root cause: hand-maintained claims about platform behaviour with nothing verifying them.

Cross-link: #504 already recorded this on 2026-09-15, two days before I measured it here. Its closing note reads "The vendored specs are older than the live servers (`docs/api-specs/gitea-1.26.1.json` against a live 1.27.3)", and its own ask ends with "Check both live forges before merge, not only the vendored specs." So the staleness was observed, written down as an aside inside a different issue, and then found again independently. That is worth noting on its own: the finding existed and still did not get acted on, because it was a supporting detail in an issue about something else rather than a tracked item of its own. #543 is the third instance of the same root cause: hand-maintained claims about platform behaviour with nothing verifying them.

hexajon referenced this issue

2026-09-17 05:51:01 +00:00

Epic: madtea's external contracts are asserted, not verified #544

hexajon added a new dependency

2026-09-17 05:51:05 +00:00

#544 Epic: madtea's external contracts are asserted, not verified

hexajon referenced this issue

2026-09-29 16:37:06 +00:00

guard: declare the effects of every operation and every spec endpoint #654

hexajon added a new dependency

2026-09-29 16:37:45 +00:00

#654 guard: declare the effects of every operation and every spec endpoint