Camilla’s retrieval engine, built from scratch: 101K production queries in the first 6 months of rollout, 2.3 seconds average end-to-end latency, and a defect rate on the order of one client-reported miss every several weeks of live traffic.
The search engine is a tool available to the Camilla chatbot/agent, and it handles real queries from real users, turning the questions of people looking for tenders by the most varied criteria into verifiable answers, grounded in the tenders that actually match what they asked for. The goal was never to offer ‘suggestions’, but to give answers as exact as possible.
When the system was still in its early days, I stopped to think about what users would expect from the chatbot, especially when they asked “which tenders are there for…” and stated their own preferences. The idea of using dense retrieval, that is a purely semantic comparison between tender text and user request, didn’t convince me. The user was giving us information far richer than what an embedding can represent. If they said “I have a Master’s degree in aerospace engineering”, they were stating a formal requirement, and the same went for the city they were willing to work in, and so on.
This first aspect brought back to mind a paper by Wen et al. from 2017 presenting what were then called Task-oriented Dialogue Systems, assistants like Siri, Cortana, Alexa and the like. The language models were seq2seq rather than GPT as in our case, but one parallel was clear: separating the retrieval of structured information (in the paper it could be finding restaurants, or booking flights) from the encoding/decoding of natural language.
That paradigm seems to have evolved and found its natural continuation in agent architectures, with the separation between orchestrator and tools. In a sense, this search engine is a tool of the agent, or rather a subagent, because it takes a raw user query and breaks it apart (entity extraction), resolves each entity against a dictionary (ISTAT sources for geographic areas, the MIUR catalog of the Italian Ministry of Education for degrees), produces a set of candidate tenders by applying those filters, and finally reranks/filters further with full-text search on Postgres.
On top of that, this system had to hold up under a data scale that would grow day by day, eventually passing tens of thousands of tenders, each made of dozens of pages with tables, lists, references to laws and regulations. Betting on dense retrieval, or even hybrid, for this task would have forced us to entrust scalability to vector-space indexes (HNSW, for instance) over a huge, noisy mass of data. It would also have put far more stress on the systems at inference/query time, because less structured data is by definition harder to organize and search quickly.
Instead we relied on structured data for tender metadata, using a relational DB (Postgres) to list the tenders, the degrees they required, the work locations, the job profiles, and normalizing all these entities against canonical dictionaries. That way, what used to be hundreds of thousands of pages of text was reduced to a few tens of thousands of structured records, navigable in milliseconds through classic B-Tree indexes.
Finally, the relational structure was plainly there once you looked at the entities we were working with. Countries -> Regions -> Provinces -> Municipalities, degrees that could be stated generically (e.g. ‘bachelor’s degree’) or specifically (e.g. ‘LM-18’ or ‘Master’s degree in Computer Science’). The lexical variations seemed endless, but the underlying structure was clear, and capturing it meant following its philosophy. In no other way could we have expanded ‘Bachelor’s degree in Mathematics’ into holding the requirements ‘L-35’, ‘Bachelor’s degree (generic)’, and all the legally equivalent degrees, according to the MIUR catalog.
Managing the administrative complexity by mapping both sides (tender, user query) onto those dictionaries let us make this domain governable in a clear and explainable way. In fact, every user query comes back with the list of tenders found and the exact set of filters that were applied. This was the vision that pushed me firmly toward this hybrid solution from the start, even more than the latency concerns, which mattered but could have been solved by throwing resources at them.
In production, though, every link in this chain eventually jams: the LLM endpoint erroring out under a request spike or a content filter firing on nothing, a public body or a degree written in a form that exists in no dictionary, a perfectly legitimate combination of filters that nonetheless zeroes out the results. The design rule is that no failure interrupts the search: everything falls back to free text in the FTS string. If the LLM doesn’t answer (after retries on the recoverable errors), extraction falls back to plain regexes; an entity that doesn’t resolve is demoted to free text; if the exact filters zero out the set, they all dissolve into the FTS string with a LIMIT 3 to guarantee an answer anyway. So recall never drops to zero through our own pipeline’s fault, and Postgres FTS (word_similarity over a precomputed, GIN-indexed text surface) acts as a keyword filter and reranker in the normal case, and as the safety net in all the others.
user query
│
▼
entity extraction (LLM) ── LLM down ──► regex ──┐
│ │
▼ ▼
dictionary resolution FTS string
│ (ISTAT, MIUR) ── unknown entity ───────►│
▼ │
exact Postgres filters │
│ (+ expansions) ── 0 tenders ───────────►│
▼ │
FTS: keyword filter and rerank ◄────────────────┘
│ (word_similarity, GIN index)
▼
agent
Upstream of it all there’s also a shortcut: if the query matches a tender’s title almost verbatim (FTS similarity > 0.95), the engine answers right away, skipping the whole pipeline.
All of this, though, solves only half the problem: figuring out which tender the user is talking about. Once the tender is pinned down, the question changes nature (“how many exam stages are there?”, “can I bring a calculator?”) and there semantic matching makes sense again. The second stage runs hybrid search on Weaviate, BM25 plus embeddings, restricted to the content of that one tender: a few dozen pages instead of hundreds of thousands, where semantic recall actually earns its keep and the noise is under control.
The underlying lesson was to simplify the search surface (the available tenders) as much as possible, shifting all the load we could (namely building indexes and structures) to the moment tenders were ingested into the system (at night, to avoid disruption), and at the same time to retain as much information as possible from the user query, exploiting the richness and taxonomy of the domain we were operating in, that is public tenders.
On the testing and evaluation side, we built an internal regression suite of several hundred annotated queries, which today passes at 97%: every change to the engine is re-tested against those queries before reaching production.
The architecture and governance choices are told in the article on Agenda Digitale (in Italian).