An Italian-language phone agent that answers calls on your behalf: it talks to the caller in natural language, streams what’s being said in real time to a paired mobile app (where you can let it carry on, take over, or hang up), and once the call ends produces a transcript, a summary and a classification. Fully open source (AGPL) and built end to end by me: backend, audio DSP, React Native app. The idea was born as a personal project during Human-Computer Interaction at OMSCS (Georgia Tech). I had built Figma prototypes, the professors and a few classmates liked the idea, and that gave me the confidence I needed to embark on the project.
At first it looked like a breeze: Twilio SDK, a realtime model wired in the middle, done. Then I started raising the bar on myself. I don’t want to be tied to Twilio: fine, I’ll handle the WebSocket myself and abstract the telephony provider. I don’t want to depend on OpenAI and Realtime forever: fine, now I need a full AI provider factory. I can’t afford realtime-API pricing for my users: and that’s where the chaos began.
Turn taking in a telephony setting is extremely hard. I had issues with echo, with low-quality 8 kHz phone audio, with barge-in, with energy-level computation, with poor-quality snippets, and the list goes on. The typical case: the assistant’s audio message gets buffered and streamed out, and speech detection triggers either too early, so the assistant’s own echo opens a turn and interrupts its own message, or too late, and pieces of what the user is saying get lost. I spent far too much time debugging those triggers; what came out of it is a stateful audio gate, with dual-threshold barge-in confirmation, stricter thresholds while the assistant is speaking, an echo-absorption window after each response, and spectral echo suppression running in parallel.
caller ─► Twilio ─► audio in ─► AudioGate ─► STT ─┐
▲ ▲ VAD, dual │
│ │ threshold, states ▼
│ echo│ LLM
│ │ sentence by │
│ │ sentence ▼
└── Twilio ◄──── audio out ◄───────────────── TTS
The Realtime engine replaces the whole block from AudioGate to TTS with a single audio-to-audio stream.
I got it to a satisfactory state: under a second to first audio thanks to double streaming (streaming STT, generation and per-sentence synthesis in parallel), though still less natural than Realtime’s audio-to-audio, at 4–5× lower cost. And for me this was no optimization exercise: it was the difference between a product with a future and one without, because token costs would have eaten the whole margin. The lesson of this project lives right there: the costs, and the complexity of handcrafting a realtime pipeline as a team of one. The two engines remain interchangeable behind a single interface, selectable per deployment and per individual user, sharing the same call orchestrator and per-call cost ledger.
A detail I care about: the transparency required by the EU AI Act (Art. 50) is not a disclaimer bolted on top, but a state in the barge-in state machine. While the opening disclosure is playing (“this is an artificial intelligence…”), interruption is blocked, so every caller hears the notice in full, and the user cannot turn it off. The format choices follow the same philosophy of removing chances for error: Twilio speaks μ-law at 8 kHz and the model is configured in the same format, zero audio transcodings in the call path.
On the spam front, I shelved the idea of screening the caller’s number before even answering: I was excited to build it and it was technically feasible, but it comes too close to profiling the caller and it wasn’t legally defensible (besides being ethically questionable). What remains is dynamic behavior driven by the topics the user flags as spam, plus a personal blocklist as a shield. In the meantime the project drifted from “spam filter” to something closer to a proper assistant, with entity extraction from conversations and calendar hooks (scaffolded, not yet wired up).
Before considering it releasable I did something I would recommend to anyone who develops alone: a formal security review of the repo, complete with a score, a list of blocking MUST FIXes (input validation, prompt injection through custom instructions, sensitive-data logging) and an explicit gate on production. The infrastructure was already pointed in that direction: non-root containers with read-only filesystems and dropped capabilities, multi-tier Redis rate limiting with sliding windows and a circuit breaker, CI with Bandit, Semgrep, Trufflehog and Trivy. It is a working, deployable implementation, not a product: the repo states this explicitly, disclaimer included.