AI testing checklist for LLM features before you ship

Shipping an LLM feature without tests is like shipping payments without receipts. It might work in the demo—and quietly fail for users in ways your logs never catch.

Use this checklist before you flip the switch. Steal it, print it, argue with it—but don’t skip it.

1. Define “wrong” before you define “smart”

Write 10–20 examples of bad outputs you refuse to ship: invented facts, policy violations, empty answers, wrong tool calls. Those become your first eval set.

2. Cover the critical path, not the whole model

• Smoke tests for the non-AI plumbing (auth, routing, persistence)
• Golden-set evals for the AI steps that touch users or data
• Regression pack that runs on every prompt or tool change

3. Gate high-stakes actions

If the feature can send email, update CRM, refund, or delete—require a human approval or a strict schema. Testing alone won’t save you from an unbounded write.

4. Baseline latency and cost

Measure p50/p95 latency and tokens per successful task under a realistic load. Set a budget alarm. “It felt fine in the hallway demo” is not a baseline.

5. Watch production like a product surface

• Log inputs/outputs with privacy redaction
• Sample failures weekly into the eval set
• Track refusal rate, tool-error rate, and human overrides

The goal isn’t perfect AI. It’s knowing when it’s wrong—and stopping the damage.

Need this built into your release process?

wohnmohr’s AI testing work sets up evals, regression gates, and release checks so your team can change prompts without fear. Pair it with the vibe-code health scorecard if the surrounding product is also AI-generated and fragile.