Epic: madtea's external contracts are asserted, not verified #544

New issue

Open

opened 2026-09-17 05:51:01 +00:00 by hexajon · 0 comments

hexajon commented

2026-09-17 05:51:01 +00:00

Owner

This is one root cause with several symptoms, found over a single day of investigating an unrelated failure.

The state, measured 2026-09-17

contract	what checks it	what it checks against	verified?
Forgejo API	`surface-diff`	`docs/api-specs/forgejo-15.0.json`, hand-vendored	accurate today by recency, nothing enforces it
Gitea API	`surface-diff`	`docs/api-specs/gitea-1.26.1.json`, hand-vendored	stale by a minor version, 8 paths missing
platform differences	nothing	`docs/COMPAT.md`, hand-written, 56 rows	no
platform field shapes	nothing	nothing	no
live forge behaviour	`forgejo-smoke`	a real Forgejo	never executed
git CLI	nothing	nothing	no

Two further properties make the above worse rather than merely incomplete:

surface-diff runs on a daily cron, but it compares repository files against repository files. Its result cannot change between commits, so the scheduled run recomputes a known answer.
A failing scheduled run reaches nobody. The repository has no webhooks. A job that failed every morning would keep failing every morning in silence.

Why this matters more than the individual bugs

A check anchored to a stale artifact does not fail safe. TestAPISurfaceMatchesSpec classifies an endpoint missing from every spec as orphaned and treats it as a bug. Against the stale Gitea spec, an endpoint that exists in 1.27 reads as nonexistent, and the natural response to that failure is to delete working code.

The same shape appears in COMPAT.md, whose assertions are copied verbatim into MCP tool descriptions. An agent reading that a capability is unavailable has no way to tell whether that is true today or was true once.

So the failure mode is not "we lack coverage". It is "we report verdicts we cannot support", which is worse, because it is trusted.

Exit criterion

No claim madtea makes about an external contract survives without something that checks it against that contract, and a check that breaks reaches a person.

Children

#504 - Gitea and Forgejo run fields diverge and nothing maps them
#537 - the git CLI surface has no generated check at all
#538 - vendored specs are hand-refreshed and Gitea's is a minor stale
#540 - a failing scheduled run notifies nobody
#542 - the live-forge smoke job has never run
#543 - COMPAT.md asserts platform behaviour nothing verifies

How this surfaced

None of it was found by looking. It surfaced because a CI job merged without ever having been run, failed, and could not be diagnosed. Each attempt to diagnose it hit another check that turned out to be decorative. That is the real signal here: the defects were individually invisible, and only became visible when something adjacent broke loudly enough to force an investigation.

madtea depends on three external contracts: the git binary's CLI, the Gitea API, and the Forgejo API. Every check that claims to verify one of them is anchored to something hand-maintained, frozen in the repository, switched off, or absent. Several of them report confident verdicts anyway. This is one root cause with several symptoms, found over a single day of investigating an unrelated failure. ## The state, measured 2026-09-17 | contract | what checks it | what it checks against | verified? | |---|---|---|---| | Forgejo API | `surface-diff` | `docs/api-specs/forgejo-15.0.json`, hand-vendored | accurate today by recency, nothing enforces it | | Gitea API | `surface-diff` | `docs/api-specs/gitea-1.26.1.json`, hand-vendored | **stale by a minor version, 8 paths missing** | | platform differences | nothing | `docs/COMPAT.md`, hand-written, 56 rows | **no** | | platform field shapes | nothing | nothing | **no** | | live forge behaviour | `forgejo-smoke` | a real Forgejo | **never executed** | | git CLI | nothing | nothing | **no** | Two further properties make the above worse rather than merely incomplete: * `surface-diff` runs on a daily cron, but it compares repository files against repository files. Its result cannot change between commits, so the scheduled run recomputes a known answer. * A failing scheduled run reaches nobody. The repository has no webhooks. A job that failed every morning would keep failing every morning in silence. ## Why this matters more than the individual bugs A check anchored to a stale artifact does not fail safe. `TestAPISurfaceMatchesSpec` classifies an endpoint missing from every spec as `orphaned` and treats it as a bug. Against the stale Gitea spec, an endpoint that exists in 1.27 reads as nonexistent, and the natural response to that failure is to delete working code. The same shape appears in `COMPAT.md`, whose assertions are copied verbatim into MCP tool descriptions. An agent reading that a capability is unavailable has no way to tell whether that is true today or was true once. So the failure mode is not "we lack coverage". It is "we report verdicts we cannot support", which is worse, because it is trusted. ## Exit criterion No claim madtea makes about an external contract survives without something that checks it against that contract, and a check that breaks reaches a person. ## Children * #504 - Gitea and Forgejo run fields diverge and nothing maps them * #537 - the git CLI surface has no generated check at all * #538 - vendored specs are hand-refreshed and Gitea's is a minor stale * #540 - a failing scheduled run notifies nobody * #542 - the live-forge smoke job has never run * #543 - COMPAT.md asserts platform behaviour nothing verifies ## How this surfaced None of it was found by looking. It surfaced because a CI job merged without ever having been run, failed, and could not be diagnosed. Each attempt to diagnose it hit another check that turned out to be decorative. That is the real signal here: the defects were individually invisible, and only became visible when something adjacent broke loudly enough to force an investigation.

hexajon added a new dependency

2026-09-17 05:51:04 +00:00

#504 actions runs: madtea reads only Gitea's run fields, so Forgejo runs show no branch or start time

hexajon added a new dependency

2026-09-17 05:51:05 +00:00

#537 git: generate and verify the git CLI surface, the way surface-diff does for the forge API

hexajon added a new dependency

2026-09-17 05:51:05 +00:00

#538 api-specs: the vendored specs are hand-refreshed and Gitea's is a minor version stale

hexajon added a new dependency

2026-09-17 05:51:05 +00:00

#540 ci: a failing scheduled run notifies nobody