Kothon: Building Voice AI for Contact Centres
Project Type
AI Product
User Audience
Contact Centres
Industry
Conversational AI
Team Size
14
Service Used
Engagement Length
2024 - Ongoing
About Kothon
Kothon is a voice AI platform developed by Vivasoft. It runs AI agents that handle live customer calls in Bangla, Nepali, Sinhala and Malay.
Kothon answers inbound calls, runs outbound campaigns, and scores every conversation for quality automatically, so contact centres don’t have to review calls by hand.
Project Overview
Contact centres across South and Southeast Asia lose agent time to repetitive calls. Balance checks, order status and delivery updates take up time that should go to harder conversations.
Existing voice AI tools presented limitations for these languages and use cases. Many existing tools are primarily optimized for English and provided limited support for Bangla, Nepali and Sinhala. They also break down when callers mix English into their sentences, which is how people actually talk in these markets.
Vivasoft decided to develop its own speech stack to support the required languages and use cases. The team trained a speech recognition model on real telephony audio, built a local text-to-speech layer, and grounded every response in the client’s own knowledge base. A SIP-to-WebRTC bridge connects the pipeline to the phone lines a client already uses, so nothing needs to be replaced.
Kothon has moved past pilot stage: calls are being answered and scored live, with one deployment still being validated before go-live. The underlying speech models were trained and are hosted in-house.
Technologies Used
Vivasoft’s AI development team used tools listed below to meet Kothon’s latency, language and deployment requirements, supporting each part of the pipeline, from telephony bridging to
speech processing and data handling.
Go
Python
TypeScript
Gin
GORM
FastAPI
Cobra CLI
React
WebRTC
FastRTC
PJSUA2
MySQL
MongoDB
RabbitMQ
Redis
Docker
Loki
NGINX
Langfuse
JWT
Challenges Faced by Vivasoft
Building Kothon required the team to address several technical challenges related to these languages and contact-centre environments. Here are some of the challenges the development team faced while building it:
1. No Usable Bangla Speech Data
No large, clean, telephony-quality Bangla dataset existed. The team had to assemble and label its own data from real 8kHz phone audio, including regional pronunciation differences that shift word boundaries.
2. Code-Switching Across Languages
Callers mix English into Bangla and Nepali sentences, sometimes writing English words in Latin script. A keyword or topic system built only in English would have missed most of what callers actually said.
3. A Tight Latency Budget
A speech-to-speech conversation needs recognition, reasoning and synthesis to run in under 1500 milliseconds, or the conversation starts to feel broken. Every stage of the pipeline had to run inside that window.
4. Unreliable SIP-to-WebRTC Bridging
Real telephony is messy. Sample rates and codecs differ across carriers, and PBX behavior varies between systems like Asterisk and 3CX. The bridge had to stay stable through dropped connections without losing the call.
5. Hallucination Risk in Regulated Industries
Banking and telecom clients could not accept an agent inventing a rate or a policy detail. One client came to Vivasoft specifically because a previous chatbot had quoted customers the wrong bank rate.
6. Data Residency Requirements
Banks and telecom operators under regulatory supervision could not send call audio to a foreign cloud region. This ruled out most global voice AI providers by default.
7. Unit Economics at Call-Centre Volume
Running every call through a third-party cloud API would have made the pricing unworkable at real contact-centre volume.
Solutions by Vivasoft
Vivasoft addressed each challenge by owning more of the stack, not less. Here are the solutions Vivasoft implemented to address these challenges during development:
1. Self-Hosted ASR Model
Vivasoft trained its own speech recognition model on telephony-domain audio it assembled in-house. This enabled the platform to transcribe Bangla telephony audio using the internally trained model.
2. Trilingual Keyword and Topic Matching
Vivasoft built topic and keyword detection that runs across native script, English and Latin transliteration together. This reflects how callers actually speak, instead of how a single-language model expects them to.
3. An Optimized Speech-to-Speech Pipeline
Vivasoft tuned every stage of the pipeline, including voice activity detection and smart turn detection, to fit inside the 1500-millisecond budget. Turn detection tells the agent exactly when a caller has finished speaking.
4. A Resilient SIP-to-WebRTC Bridge
Vivasoft built a bridge that handles sample-rate and codec differences across carriers and PBX systems, and recovers from dropped peer connections mid-call. This allows Kothon to sit alongside existing telephony without a carrier or PBX change.
5. RAG-Grounded Responses with Hard Limits
Vivasoft grounded the reasoning layer in a retrieval scope the client controls. The agent answers only from that scope and stops at its edge, removing the hallucination risk that had already cost one client trust in a previous chatbot.
6. Fully In-Country Deployment
Because Vivasoft owns the model layer, the entire stack, models included, can run inside a client’s own data center. This met data residency rules that had ruled out global providers.
7. Self-Hosted GPU Inference
Vivasoft moved inference onto GPU infrastructure it owns and operates, instead of renting third-party cloud capacity. This removed the largest variable cost in the platform and made the pricing work at real call volumes.
Outcomes of the Product
Kothon is currently handling live customer traffic across multiple markets. Here are the key outcomes achieved through its development and rollout:
Reduced Agent Workload
Before Kothon, agents handled every balance check, order status and delivery update by hand. Now the AI agent takes on around 30 percent of that repetitive inbound volume.
Full QA Coverage
Most contact centres review 1 to 2 percent of calls manually. Kothon transcribes and scores 100 percent of calls instead, so compliance issues no longer go unseen.
Faster Time to Live
Deployments have reached live production traffic in around 8 weeks after contract signing, running alongside the client’s existing telephony rather than replacing it.
Round the Clock Coverage
Calls outside business hours or during traffic spikes used to go unanswered or sit in a queue. Kothon is available to answer calls 24 hours a day with no queue abandonment.
Lower Churn
Real-time frustration detection routes at-risk calls to escalation before the customer hangs up, driving an average churn reduction of around 20 percent.
Coverage Across Five Languages
Kothon runs Bangla, Nepali, Sinhala, Malay and English on one platform, so entering a new market does not mean rebuilding the contact centre.
Team Involvement
| Resources | Count |
|---|---|
| Front-End | 2 |
| Back-End | 3 |
| Full-Stack / Architecture | 1 |
| AI / ML Specialists | 3 |
| DevOps | 1 |
| Designers | 1 |
| Software QA | 2 |
| Product | 1 |
Core Features of the Software
Kothon is designed to handle live customer calls and provide conversation data for contact-centre operations. Here’s how the features work together to do that:
Self-Hosted Speech Recognition
A speech recognition model trained in-house on telephony-domain audio, trained for Bangla and being extended to Nepali, Khmer and Malay.
Sub-1500ms Speech-to-Speech Pipeline
Recognition, reasoning and synthesis run as one chain, with voice activity detection and smart turn detection keeping response time inside the latency budget.
RAG-Grounded Dialogue
The agent answers only from a knowledge base the client controls, and stops at the edge of that scope instead of guessing.
Context-Aware Warm Handoff
When a call needs a person, both the voice path and the full conversation history move to the human agent, and real-time frustration detection can trigger the handoff before it’s asked for.
Automated Call Analytics
Every call is transcribed and scored for script compliance, sentiment and topic, with keyword matching that works across native script, English and transliteration.
Flexible Deployment
Kothon runs as managed cloud, on-premise, or hybrid, sits alongside existing SIP and PBX infrastructure, and includes an outbound campaign engine with multi-tenant role-based access control.
Development Timeline
Project Start Time
2024
Project End time
Ongoing
Ready to Build Your Own Voice AI?
