git: generate and verify the git CLI surface, the way surface-diff does for the forge API #537

New issue

Open

opened 2026-09-17 02:21:48 +00:00 by hexajon · 1 comment

hexajon commented

2026-09-17 02:21:48 +00:00

Owner

This is the counterpart #525 asked for and never built. surface-diff already does exactly this for the forge: extractEndpoints walks the gitea client AST, loadSpec loads the vendored specs in docs/api-specs/, and a miss on every platform is a bug while a miss on some is drift. Git has no equivalent.

The point is not "does madtea run on old git". The point is a generated guarantee that everything we hand to git is valid for the git we pin, so that when git moves for a CVE we can re-run one check and see what broke.

Shape

Mirror the forge side rather than invent something:

forge side (exists)	git side (this issue)
`docs/api-specs/gitea-1.26.1.json`, `forgejo-15.0.json`	`docs/git-specs/git-<version>.json`
`extractEndpoints` AST walk of the client	AST walk of every git argv madtea builds
`TestAPISurfaceMatchesSpec`	`TestGitSurfaceMatchesSpec`
refresh the JSON when a new stable lands	regenerate when the pinned git moves

A generator belongs in cmd/madtea-dev, alongside docs generate, so the spec is produced from a real installed git rather than hand-maintained.

Where the option surface comes from, measured

git can emit its own option list, but the coverage is uneven. Measured against git 2.47.3 on 2026-09-17:

command	`--git-completion-helper` options	usable
`grep`	75	yes
`branch`	49	yes
`status`	28	yes
`cat-file`	20	yes
`merge-base`	6	yes
`log`	15	no
`rev-parse`	1	no
`worktree`	0	no, dispatches to subcommands

The helper only covers a command's own parse_options table. log, show and diff inherit most of their options from the revision-walking and diff machinery, so the helper omits them: --format=, --graph, --grep=, --max-count, --no-merges and --first-parent are all absent from both git log --git-completion-helper and git log -h, and madtea uses every one of them.

worktree needs per-subcommand queries (git worktree add --git-completion-helper returns 17 options, list 7, prune 7, remove 0).

So a check built on the completion helper alone would produce false positives on exactly the commands madtea leans on hardest. The complete source is git's own documentation: Documentation/git-log.adoc pulls in rev-list-options.adoc, which is where the missing flags are defined. A usable generator likely combines the helper where it is rich with the documentation where it is not, and the generator must state which source each entry came from.

Open, and blocking under the no-deferred-checks rule

The option-source strategy above is not settled. Whatever this lands as must not pin a value while that is unresolved, and must not claim coverage it does not have. If the generator cannot cover a command, that command is reported as uncovered rather than silently passing.

Acceptance

One command regenerates the git spec from an installed git.
A test fails when madtea passes a flag the pinned git does not accept.
Commands the generator cannot cover are listed as uncovered, not treated as clean.
The check names the git version it verified against.

Refs #525, #536.

madtea shells out to git across 42 distinct subcommands and hundreds of flags. Nothing verifies that any of those argv shapes are valid for the git version we pin. The test suite proves the paths the tests walk; it says nothing about the flags on the paths they do not. This is the counterpart #525 asked for and never built. `surface-diff` already does exactly this for the forge: `extractEndpoints` walks the gitea client AST, `loadSpec` loads the vendored specs in `docs/api-specs/`, and a miss on every platform is a bug while a miss on some is drift. Git has no equivalent. The point is not "does madtea run on old git". The point is a generated guarantee that everything we hand to git is valid for the git we pin, so that when git moves for a CVE we can re-run one check and see what broke. ## Shape Mirror the forge side rather than invent something: | forge side (exists) | git side (this issue) | |---|---| | `docs/api-specs/gitea-1.26.1.json`, `forgejo-15.0.json` | `docs/git-specs/git-<version>.json` | | `extractEndpoints` AST walk of the client | AST walk of every git argv madtea builds | | `TestAPISurfaceMatchesSpec` | `TestGitSurfaceMatchesSpec` | | refresh the JSON when a new stable lands | regenerate when the pinned git moves | A generator belongs in `cmd/madtea-dev`, alongside `docs generate`, so the spec is produced from a real installed git rather than hand-maintained. ## Where the option surface comes from, measured git can emit its own option list, but the coverage is uneven. Measured against git 2.47.3 on 2026-09-17: | command | `--git-completion-helper` options | usable | |---|---|---| | `grep` | 75 | yes | | `branch` | 49 | yes | | `status` | 28 | yes | | `cat-file` | 20 | yes | | `merge-base` | 6 | yes | | `log` | 15 | **no** | | `rev-parse` | 1 | **no** | | `worktree` | 0 | **no**, dispatches to subcommands | The helper only covers a command's own `parse_options` table. `log`, `show` and `diff` inherit most of their options from the revision-walking and diff machinery, so the helper omits them: `--format=`, `--graph`, `--grep=`, `--max-count`, `--no-merges` and `--first-parent` are all absent from both `git log --git-completion-helper` and `git log -h`, and madtea uses every one of them. `worktree` needs per-subcommand queries (`git worktree add --git-completion-helper` returns 17 options, `list` 7, `prune` 7, `remove` 0). So a check built on the completion helper alone would produce false positives on exactly the commands madtea leans on hardest. The complete source is git's own documentation: `Documentation/git-log.adoc` pulls in `rev-list-options.adoc`, which is where the missing flags are defined. A usable generator likely combines the helper where it is rich with the documentation where it is not, and the generator must state which source each entry came from. ## Open, and blocking under the no-deferred-checks rule The option-source strategy above is not settled. Whatever this lands as must not pin a value while that is unresolved, and must not claim coverage it does not have. If the generator cannot cover a command, that command is reported as uncovered rather than silently passing. ## Acceptance - One command regenerates the git spec from an installed git. - A test fails when madtea passes a flag the pinned git does not accept. - Commands the generator cannot cover are listed as uncovered, not treated as clean. - The check names the git version it verified against. Refs #525, #536.

hexajon added the

enhancement

label

2026-09-17 02:21:48 +00:00

hexajon referenced this issue

2026-09-17 02:24:32 +00:00

api-specs: the vendored specs are hand-refreshed and Gitea's is a minor version stale #538

hexajon referenced this issue

2026-09-17 02:42:09 +00:00

ci(drift): remove the git-matrix job #539

hexajon commented

2026-09-17 04:42:00 +00:00

Author

Owner

One more measurement worth recording here, because it bears on how much a generated check can actually promise.

git's own test suite does not cover grep --end-of-options. In git 2.55.0, fourteen files under t/ exercise that flag:

t0040-parse-options.sh      t4300-merge-tree.sh      t7102-reset.sh
t1091-sparse-checkout-builtin.sh  t5000-tar-tree.sh  t7400-submodule-basic.sh
t1418-reflog-exists.sh      t6000-rev-list-misc.sh   t9350-fast-export.sh
t1503-rev-parse-verify.sh   t6017-rev-list-stdin.sh
t1506-rev-parse-diagnosis.sh  t3800-mktag.sh         t4202-log.sh

log is covered. grep is not, and neither are cat-file, merge-base or diff. madtea uses --end-of-options on all four of those plus rev-parse and the tree/show path.

gitcli(7) documents the flag as a general convention across commands, so madtea's usage is correct against the documented contract. But upstream only tests it on some commands, which is consistent with what 2.43.1's note describes: the form "failed to interpret --rev as a rev" for some commands and was "fixed for many programs like reset and checkout". Documented universally, implemented unevenly, tested partially.

The implication for this issue: a generated surface check can verify that a flag exists on a command. It cannot verify that the flag behaves as documented on that command, and for --end-of-options specifically that gap is where the 2.38.0 failure lived. git grep --git-completion-helper lists --end-of-options quite happily on a git where using it fails.

So the check should not be described as proving the invocations work. It proves they are accepted. Behaviour still needs a real test against a real repo, which is what TestMadTRefInspectGrepAtLocalOnlyRef does for this one case.

One more measurement worth recording here, because it bears on how much a generated check can actually promise. **git's own test suite does not cover `grep --end-of-options`.** In git 2.55.0, fourteen files under `t/` exercise that flag: ``` t0040-parse-options.sh t4300-merge-tree.sh t7102-reset.sh t1091-sparse-checkout-builtin.sh t5000-tar-tree.sh t7400-submodule-basic.sh t1418-reflog-exists.sh t6000-rev-list-misc.sh t9350-fast-export.sh t1503-rev-parse-verify.sh t6017-rev-list-stdin.sh t1506-rev-parse-diagnosis.sh t3800-mktag.sh t4202-log.sh ``` `log` is covered. `grep` is not, and neither are `cat-file`, `merge-base` or `diff`. madtea uses `--end-of-options` on all four of those plus `rev-parse` and the tree/show path. `gitcli(7)` documents the flag as a general convention across commands, so madtea's usage is correct against the documented contract. But upstream only tests it on some commands, which is consistent with what 2.43.1's note describes: the form "failed to interpret `--rev` as a rev" for some commands and was "fixed for many programs like `reset` and `checkout`". Documented universally, implemented unevenly, tested partially. The implication for this issue: a generated surface check can verify that a flag **exists** on a command. It cannot verify that the flag **behaves** as documented on that command, and for `--end-of-options` specifically that gap is where the 2.38.0 failure lived. `git grep --git-completion-helper` lists `--end-of-options` quite happily on a git where using it fails. So the check should not be described as proving the invocations work. It proves they are accepted. Behaviour still needs a real test against a real repo, which is what `TestMadTRefInspectGrepAtLocalOnlyRef` does for this one case.

hexajon referenced this issue

2026-09-17 05:51:01 +00:00

Epic: madtea's external contracts are asserted, not verified #544

hexajon added a new dependency

2026-09-17 05:51:05 +00:00

#544 Epic: madtea's external contracts are asserted, not verified