The problem
A B2B SaaS wanted an AI support agent that could talk to customers from day one — not a demo that answers one question and melts on the second. Their knowledge base had 40k documents, conflicting versions, and no labelled notion of what "good" meant.
What we did
We built a hybrid retrieval pipeline — keyword (BM25) and vector (pgvector) — with metadata filters and re-ranking, over a knowledge base that was cleaned and versioned first. The labelled eval set came from real support tickets, not a design doc, and it ran in CI before every change.
Answers stream to the customer with citations. A guardrail layer doesn't block low-confidence answers — it routes them: answer when confident, offer a callback form when medium, page a human when low. Cost-per-output is instrumented per ticket.
Impact
Resolution time dropped an order of magnitude while hallucination stayed under 2% on the eval set — and when the guardrails fired, they fired quietly, so customers saw calm competence instead of a wall.