Phantom APIs: a cautionary tale

Earlier generations of this project’s test suite were written against fastmcp.testing.MCPTestClient and pytest.current_item.

Neither API has ever existed. Not in any version of FastMCP, not in any version of pytest.

The anatomy of a phantom

Both names are plausible. Test frameworks often ship a testing module; MCPTestClient follows the naming convention of real things like Starlette’s TestClient; pytest exposes plenty of pytest.* attributes. They’re exactly the APIs you’d expect to exist — which is why generated code reached for them, and why review didn’t catch them. A phantom API is not a random hallucination; it’s a statistically reasonable interpolation of how libraries usually look. That’s what makes it dangerous.

Around 100 tests imported those names. Every one of them died at import time, on every run, from the day it was written.

The worse part

Broken imports weren’t the real damage. The real damage was what grew around them: since the “real” test suite couldn’t run, the gap was papered over with standalone scripts that only checked tool registration — scripts that always passed, because checking that 19 names appear in a list is very hard to fail. The project had the artifacts of testing (files in tests/, green output somewhere) with almost none of the substance. Confidence and coverage had fully decoupled.

The eventual triage rebuilt the suite against APIs verified to exist — FastMCP’s real in-memory Client, real pytest fixtures — and kept only tests that could demonstrably fail.

The discipline that prevents it

Three habits, all cheap, any one of which would have caught this:

1. If a test can’t fail, it isn’t testing anything. Run every new test and watch it fail — break the code, or assert the wrong value, before trusting it green. A phantom-API test never reaches the failure you designed; it fails at import, for the wrong reason, or is quietly skipped.
2. Verify APIs against the installed library, not against plausibility. python -c "from fastmcp.testing import MCPTestClient" takes two seconds and returns an unambiguous verdict. Documentation, and the actual import, outrank pattern-matching on what the API should be — this matters double when code is generated by a model whose training data is full of libraries that look almost like yours.
3. Treat “always green” as a smell. A suite that has never failed deserves the same suspicion as one that always does. The skip-accounting rules in the testing philosophy exist precisely so that “tests didn’t really run” is detectable.

The general moral for LLM-built systems: generation produces convincing code, and convincingness is uncorrelated with existence. Measure before matching — the two-second import check beats any amount of plausibility.