A phone line that answers in English or Spanish, knows the listings, and writes the lead to the CRM

The job

The brokerage wanted its main number answered by something that knew its listings and could route callers to the right agent. Earlier attempts with scripted phone systems had failed the moment a caller said something that was not on the menu: "something near Wynwood, maybe a warehouse, maybe to buy". The brief was a receptionist that could hold that conversation in either language, then do something useful with it.

What I built

• A streaming pipeline over WebSockets: Twilio carries the call, Deepgram turns speech into text and decides when a turn has ended, Claude decides what to say and calls tools, ElevenLabs speaks the reply in the phone network's own audio format.
• Turn-taking that sounds like a person: speculative replies that start on a probable end of turn and cancel if the caller resumes, backchannels ("uh-huh", "sí claro") that do not interrupt, and a barge-in that cuts the agent's memory at the last word the caller actually heard.
• Hearing for a noisy phone line: a vocabulary boost for neighbourhoods and property terms, and a deliberately narrow mishearing table that only applies while a specific question is open.
• Knowing the business: CRM listings held in memory and refreshed on a schedule, a hand-built map of South Florida neighbourhoods and highway nicknames, and prompt rules written for the ear (street numbers in pairs, no abbreviations).
• Texting instead of spelling: callers get listings by text and reply with their email, behind a consent flow built to carrier rules, with STOP honoured in code rather than in the prompt.
• Guardrails in code, not prompts: the model can only say "I sent it" when the provider's answer says so.

What I learned, written up

The brokerage's own agents spent long sessions trying to break it and I listened to a lot of recordings. Most of what went wrong was behavioural and was fixed in the prompt; the rest became the two libraries below, each with the story of the bug that produced it.

Open source from this project

• Turn-taking for phone agents voice-engine

  People interrupt, pause and say "uh-huh". This handles turn-taking, so your agent knows when to talk, when to stop, and what the caller actually heard.

• SMS consent, STOP and one send gate sms-consent

  Opt-in, STOP in any language and quiet hours sit behind one gate, so your AI can never claim it sent a text it didn't, or send one it shouldn't.

• Email for agents: send once, tell the truth email-kit

  It retries only when it should, never double-sends, skips people who unsubscribed, and reads a spoken email address back before it sends anything.