Automated testing and evaluation pipeline I built end-to-end for Camilla, the tender-assistance chatbot deployed at national scale for Italian Public Administration. At its core is a task-oriented evaluator: a DAG-based scheduler with dependency rules declared in PostgreSQL and FastAPI workers that scale horizontally, so the assistant’s behavior can be tested on many tenders in parallel. The volume of data added to the chatbot, all of it needing testing, matches every public tender published on INPA: roughly 100 new tenders a day.
On the orchestration side I deliberately took the simplest route that would hold the requirements, without integrating any workflow framework: Postgres is the system’s single point of contact. The container that submits the run and the workers executing the tasks never talk to each other: everyone reads and writes state on the database, and the DAG dependencies are resolved by a SQL function that unlocks tasks as their prerequisites complete. Every task has retries, a timeout and an explicit lifecycle (queued, running, completed, failed, canceled), and resources are governed in the most prosaic way possible: docker compose –scale to add or remove workers. Less orchestration overhead makes it easier to understand why a run stopped, and to start it back up.
run start (Docker) FastAPI workers × N
(docker compose --scale)
│ ▲ │
▼ │ ▼
Postgres: queue, task state, DAG in a SQL function
│
│ per tender: evidence ──► ok/ko verdict
│ global: red-teaming, Weaviate diagnostics
▼
dashboard + email to the service
If a task fails, the healthy ones are still carried to completion before the run halts: relaunch it, and only the failed tasks are automatically picked up again.
Evaluation uses LLM-as-a-judge with multiple rubrics, each specific to a test type: pseudo-ground-truth (verifiable facts extracted from source documents and re-queried against the bot), multi-turn conversations with dynamic scope switching, search-engine behavior checks, embedding-collection quality tests. On top of that, correct use of resources is verified through a “required tool” contract on every turn: a claim produced without invoking the corresponding retrieval tool is flagged as a potential grounding failure.
On top of this infrastructure I designed a suite of 200+ multi-turn scenarios probing institutional role preservation under adversarial pressure: prompt injection and jailbreak resistance, information-boundary protection, fairness and non-discrimination, temporal integrity, recovery to baseline behavior after an attack. The taxonomy is my own design, derived from production interactions and complemented with the OWASP Top 10 for LLM Applications. Cost and latency of every evaluation cycle are tracked in a final dashboard that lets the service govern and monitor the chatbot’s performance over hundreds of new documents a day.
Surprisingly, one of the hardest things to get right while designing and testing this pipeline was metadata verification. As explained on the search engine page, Camilla’s end-to-end behavior rests on each tender being correctly tagged with metadata: which degrees and requirements it demands, work locations and so on. By the time the evaluator tests a tender, that metadata has already been attached to it, yet the service kept reporting inaccuracies and omissions with respect to what was actually written in the tender text. So I added a dedicated task that runs entity extraction over the full tender text and diffs what it finds against the metadata already attached to the tender, flagging every discrepancy.
The difficulties showed up when I had to face the real data: public tenders are incredibly heterogeneous documents, but above all the entities we cared about (geographic ones, degrees) very often appeared with a meaning different from the expected one. For instance, the name of a city where the written exams would take place (but not a work location), or degrees that granted preference points in the ranking without being “required”. On top of that, the data often sat in tables spanning page after page, and came in every imaginable format (degrees especially): with a degree code, without one, with or without a field of study, in broken-up lists or lists stating legal equivalences.
A multi-step approach turned out to be essential: first locating the articles of the tender that contained information about requirements, locations or job profiles. From there, we use regex wherever possible to extract a series of text “islands” (not too disconnected from each other, so as to preserve the surrounding context), and only then run entity extraction, whose system prompts have been revised dozens of times to spell out every business rule needed. The comparison between extracted metadata and the metadata attached to the tender is currently done deterministically wherever possible (e.g. removing degrees that match on degree code from both sides, and other regex-based rules), leaving only the genuinely ambiguous cases to the LLM.
A first description of the pipeline is in the article on Camilla (in Italian); a dedicated article on red-teaming and behavioral evaluation is forthcoming on Agenda Digitale.