Building LLM agents that don't embarrass you

The first agent you build works beautifully on the five prompts you tested in the meeting. The second week of production introduces it to the long tail — the typo-laden, ambiguous, adversarial, edge-case reality of real users — and that's when the screenshots start arriving in Slack.

The fix isn't a better prompt. It's treating the agent like any other piece of software that can regress: you need an evaluation set, you need guardrails, and you need a human in the loop for the decisions that actually matter. Everything else is theatre.

Write the evals before the agent

Before we wire a single tool, we write the cases the agent must get right — and, just as importantly, the cases where the correct answer is 'I don't know, let me get a human.' That refusal set is the most undervalued artifact in applied AI. An agent that knows its limits is trustworthy; an agent that guesses confidently is a liability.

• A golden set of inputs with expected outcomes, versioned in the repo.
• A refusal set: inputs that must escalate, never auto-act.
• A regression gate in CI: the agent's score may not drop between commits.
• A live sample: a percentage of production traces re-scored every night.

Guardrails are architecture, not a wrapper

We bound what an agent can do with the same seriousness we bound a junior engineer's deploy permissions. Tools are typed, rate-limited, and idempotent. Destructive actions require confirmation. Retrieval is grounded, and the model is forced to cite the document it pulled from — and we verify the citation actually contains the claim before we show it.

Confidence without a citation is just a fluent guess. In production, fluent guesses become support tickets, and support tickets become churn.

None of this is glamorous. It's harness code, eval runners, and a lot of boring plumbing. But it's exactly the plumbing that turns a startling demo into a system your customers can rely on — and that's the only kind worth shipping.