The two-tier testing philosophy

The test suite (how to run it) rests on one principle with several consequences: mock the backend, never the framework.

The principle

Every fake in a test is a claim: “the real thing behaves like this.” Claims about your own boundary with a backend (what FreeCAD returns for a bounding box) are cheap to keep honest. Claims about a framework’s internals (how FastMCP serializes results, validates schemas, or raises errors) are expensive and constantly drifting — and if you mock them, your tests verify your model of FastMCP rather than FastMCP.

So the suite draws the line at the FreeCAD import boundary:

• FreeCAD is mocked at the sys.modules level — a shared mock module installed before src.freecad_server is imported (the server binds FreeCAD = None at first import, so ordering is load-bearing).
• FastMCP runs for real, via its in-memory client: async with Client(server). Every test call traverses genuine schema validation, parameter hydration, serialization, and error wrapping. When FastMCP 3.x changed behaviors, these tests broke — which is exactly what tests are for.

A subtle enforcement mechanism falls out of running the real framework: mock return values must be JSON-serializable, because the real client serializes results. A MagicMock accidentally left in a return dict fails the call. The framework polices the mocks’ honesty.

Why two tiers instead of one

Mocks can’t testify that FreeCAD actually opens a DXF and computes real geometry. That evidence has to come from the real thing — which needs the Docker image, a virtual display, and seconds-to-minutes of runtime. Hence the split:

Tier	Backend	Where	Speed	What it proves
Mocked	fake FreeCAD, real FastMCP	anywhere	~2 s	tool logic, validation, envelope handling, error paths
Integration	real FreeCAD	Docker container	slow	files actually open, conversions actually produce bytes on disk

The fast tier runs on every change; the real tier runs where the real dependency exists. Neither substitutes for the other.

Skips as contract, and canary thinking

The seam between tiers is managed by reasoned skips: integration tests skip cleanly without FreeCAD, Docker tests skip without the daemon or image. Two rules make the skips trustworthy rather than noise:

1. A failure is never acceptable on a clean checkout. Green must mean green.
2. A skip that should have run is a bug. If the Docker image exists and the Docker tests skipped anyway, the gating logic broke — silently removing coverage while the summary still looks healthy.

That second rule is the canary discipline: the thing most likely to fail silently is the test infrastructure itself, so the suite treats “did the right tests actually run?” as an assertable property (pytest -rs lists every skip with its reason). The same discipline applies to new tests: run them and watch them fail first — a test you’ve never seen fail is a test you know nothing about. The project learned that one the hard way; see Phantom APIs.

The fixture zoo

The real-file tier draws on a deliberately curated menagerie rather than one convenient sample file. Three groups earn their disk space:

• A DXF version ladder — the same drawing saved as R12, 2000, 2010, and 2018 in ASCII, plus 2018 in binary encoding. Format-generation regressions and the binary-DXF path get exercised against every rung.
• Two negative-path fixtures that bracket recover-mode. One file’s defect (a nameless block) kills strict parsing but recovers cleanly; the other’s structural damage (literal newlines inside MTEXT values, courtesy of a converter bug) kills both loaders. Together they document exactly what lenient loading can and cannot save — a boundary worth pinning with real files, not assumptions.
• Contrasting real-world profiles: a 941-layer sparse civil plan set against a 2-layer, 16,000-entity mechanical drawing, plus a real 12 MB STEP solid for the FreeCAD tier. Tools that behave on one profile routinely misbehave on its opposite.

The fixtures live in a gitignored data/ directory — real-world drawings don’t belong in version control — and tests skip cleanly when they’re absent, under the same skip contract as everything else.

Small on purpose

The suite was triaged from ~170 generated tests down to ~97 collected, of which 63 run everywhere. Tests were deleted for testing mocks against mocks, asserting things no user observes, or duplicating framework guarantees. What remains earns its runtime: each test verifies something that could genuinely be wrong. A smaller honest suite beats a large one that manufactures confidence.