A common complaint about LLM systems is that they’re opaque (which they are), black boxes (which they are not, but it’s still hard for us to probe them), and that basically trusting them is a leap of faith. The problem feels circular: a system that can generate coherent, fluent language, but does so on a scale where human review alone cannot keep up.
What’s the answer? In reality, there are many answers, layered like walls that protect a city: when working with commercial AI providers, they already offer a first line of defence, filtering away violent or unsafe queries. But this leaves out an enormous number of ’legitimate’ queries that are not admissible for a public administration chatbot. What about fairness? Non-discrimination, role integrity, refusal to impersonate or change its style and tone… and a myriad of potential attacks or malicious queries that could potentially turn a nationwide system into a PR nightmare?
Even more than this, how do we even measure if the assistant is able to give out proper information about the public tenders at all (since we are talking about Camilla x INPA)? How do we make sure that the LLM system does not conflate its own (possibly stale) pre-trained knowledge with what is actually written in a fresh public tender? In a context where the user is a citizen, seeking a job, and who will act according to what the system says. The right to work is universally recognized, and it’s incredibly delicate to intertwine it, or gate it, with a system that answers automatically, and absolutely has to adhere to official sources.
Still, we should not be too pessimistic. On the other hand, an incredible opportunity arises: a source of fresh information, available 24/7, with no delay, no human hours wasted, that democratizes access to public work, previously buried in bureaucracy-filled tenders (which they need to be, but are often hard to read).
So in order to make these two worlds meet, the need for safety and observability on one side, and the realm of possibilities opened by deploying AI systems in the public space on the other, I embarked on a journey of evaluating and red-teaming a national-scale chatbot.
The result? 200+ red-teaming scenarios tested daily; hundreds of synthetic questions derived from each new tender and posed to the chatbot, which is scored independently on each tender (and a new tender doesn’t release in prod until it has a good enough score); metadata re-verifications against the tender’s text (e.g. which degrees or work locations it mentions); rigid tool-use contracts that require the assistant to use the provided sources instead of hallucinating.
All of this allowed us to observe and measure the behavior and performance of the assistant along the two most important axes: factual correctness and safety.
It is the third and final article in a series on production LLM systems for the public administration, after the ones on Camilla and on reusable tools via MCP.