Replace the TypeScript MCP conformance harness with Go schema-based validation #750

New issue

Open

opened 2026-10-09 21:18:49 +00:00 by hexajon · 0 comments

hexajon commented

2026-10-09 21:18:49 +00:00

Owner

While fixing the missing resultType on tool results (#740), I confirmed a deeper problem with how I test MCP conformance here.

Problem

The client-conformance harness (conformance/mcp-client) pulls the official TypeScript MCP SDK into this Go repo to drive madtea serve as a real client. That one dependency installs about 93 transitive npm packages (a full HTTP server and OAuth stack) to use only a stdio client. The harness is dev-only and node_modules is gitignored, but it is still a large surface to maintain, keep current, and scan.

madtea does not implement the MCP protocol envelope. The go-sdk does: it frames every result and sets resultType at the session layer. So the harness mostly validates the go-sdk's layer, not madtea's own code. The only MCP surface madtea controls is the tool schemas it defines (each tool's inputSchema/outputSchema).

Proposal

The authority for compliance is the official MCP schema (schema.json), which is language-neutral. Validate against it in Go:

A Go test validates madtea's advertised tools/list schemas and its tool-call result envelopes against the official MCP schema.json for each supported protocol revision.
Remove the TypeScript harness, its package.json/lock, and the TS step in scripts/mcp-conformance.sh, unless a concrete gap is proven first.
Add a gate step that warns (never fails) when the pinned go-sdk is behind its latest release. Vulnerabilities are already gated by govulncheck and osv-scanner.
Remove the stopgap internal/mcp/result_type_test.go once the schema validator covers resultType.

Tradeoff

Schema validation catches spec violations. It does not reproduce a specific client's extra strictness beyond the spec. By the spec-is-authority principle, a client that rejects spec-compliant output has its own bug, which I would report upstream rather than encode here.

Acceptance criteria

A Go test validates madtea's tool schemas and result envelopes against the official schema.json for the supported protocol revision(s), and fails on a real violation.
The TS harness, its npm dependencies, and the TS step in scripts/mcp-conformance.sh are removed (or a documented gap justifies keeping a minimal piece).
A gate step warns, without failing, when the pinned go-sdk is behind the latest release.
internal/mcp/result_type_test.go is removed once the validator covers resultType.

While fixing the missing `resultType` on tool results (#740), I confirmed a deeper problem with how I test MCP conformance here. ## Problem The client-conformance harness (`conformance/mcp-client`) pulls the official TypeScript MCP SDK into this Go repo to drive `madtea serve` as a real client. That one dependency installs about 93 transitive npm packages (a full HTTP server and OAuth stack) to use only a stdio client. The harness is dev-only and `node_modules` is gitignored, but it is still a large surface to maintain, keep current, and scan. madtea does not implement the MCP protocol envelope. The go-sdk does: it frames every result and sets `resultType` at the session layer. So the harness mostly validates the go-sdk's layer, not madtea's own code. The only MCP surface madtea controls is the tool schemas it defines (each tool's `inputSchema`/`outputSchema`). ## Proposal The authority for compliance is the official MCP schema (`schema.json`), which is language-neutral. Validate against it in Go: - A Go test validates madtea's advertised `tools/list` schemas and its tool-call result envelopes against the official MCP `schema.json` for each supported protocol revision. - Remove the TypeScript harness, its `package.json`/lock, and the TS step in `scripts/mcp-conformance.sh`, unless a concrete gap is proven first. - Add a gate step that warns (never fails) when the pinned go-sdk is behind its latest release. Vulnerabilities are already gated by govulncheck and osv-scanner. - Remove the stopgap `internal/mcp/result_type_test.go` once the schema validator covers `resultType`. ## Tradeoff Schema validation catches spec violations. It does not reproduce a specific client's extra strictness beyond the spec. By the spec-is-authority principle, a client that rejects spec-compliant output has its own bug, which I would report upstream rather than encode here. ## Acceptance criteria - [ ] A Go test validates madtea's tool schemas and result envelopes against the official `schema.json` for the supported protocol revision(s), and fails on a real violation. - [ ] The TS harness, its npm dependencies, and the TS step in `scripts/mcp-conformance.sh` are removed (or a documented gap justifies keeping a minimal piece). - [ ] A gate step warns, without failing, when the pinned go-sdk is behind the latest release. - [ ] `internal/mcp/result_type_test.go` is removed once the validator covers `resultType`.

hexajon referenced this issue

2026-10-20 23:06:10 +00:00

chore: replace the npm MCP conformance harness with a Go check, and drop npm from the project #854

hexajon referenced this issue from a commit

2026-10-20 23:06:11 +00:00

chore: remove the npm conformance harness and every node step

hexajon referenced this issue from a commit

2026-10-20 23:06:11 +00:00

docs: ADR 0036 names adr check --skip-forge-checks, and ADR 0044 drops the removed npm lockfile

hexajon referenced this issue from a commit

2026-10-20 23:06:11 +00:00

chore: drop the mcp-conformance CI job and the last npm references, and pass --skip-forge-checks to adr check