# Self-Hosted AI Calling Platform — Phase 1: Architecture & Folder Structure Status: **awaiting approval** · Replaces: Hunar AI integration (`apps/ai_calls/hunar.py`) Owner: TA-ATS · Only external dependency: **SIP trunk provider** (PSTN access) --- ## 1. Goals & non-goals **Goals** - Own the full voice pipeline: telephony, STT, dialog, LLM, TTS, scoring, reports. - Keep the recruiter flow identical: import → select → "Start AI Screening" → live statuses → transcript/score/PDF → stage updates. - Reuse the ATS integration surface that already exists (`apps/ai_calls`): the platform emits the *same* status/transcript/completed events Hunar did, so the ATS side barely changes. - Scale horizontally: 100 → 1000+ concurrent calls by adding workers/GPUs, not by rewriting. **Non-goals** - Self-hosting PSTN (impossible) — we buy SIP trunking (e.g. Twilio Elastic SIP / Telnyx / Airtel SIP in India). - Building our own STT/LLM/TTS models — we run open-source models on our own hardware. --- ## 2. Environment reality check (read first) | Assumption in the brief | Reality in this repo/machine | Consequence | |---|---|---| | Celery + Redis available | **Not installed**, offline venv | Phases 1–2 code runs on the existing thread-queue shim (`ai_calls/queue.py`); real Celery lands with Docker in Phase 13 | | MinIO | Not present; media is filesystem (`frontend/public/media`) | Recordings/PDFs stay on the media volume until MinIO joins the compose stack | | Next.js 15 | Repo runs **Next 16.2.9** | No impact; pages follow the repo's existing patterns | | Docker available | Offline dev box — images can't be pulled here | Phases 3–10 are developed as code + compose files here, but **must be run/tested on a connected Docker host with a GPU** | | GPU | Unknown | Sizing table in §9; minimum 1× 24 GB GPU for a meaningful pilot | Every service therefore ships with a **loopback/mock mode** so the ATS remains demoable on this box (exactly like today's `HUNAR_MOCK`). --- ## 3. High-level architecture ``` ┌──────────────────────── ATS (exists today) ───────────────────────┐ │ Next.js UI ── Django REST ── PostgreSQL ── PDF reports (pdf_builder)│ └──────────────┬────────────────────────────────────▲─────────────────┘ │ REST: start/retry/cancel │ events: status/transcript/ ▼ │ completed (+ WS fan-out) ┌──────────────────────────┐ │ │ scheduler_service │── Redis queues ────────┤ │ (dial pacing, retries, │ │ │ concurrency budgets) │ │ └──────────┬───────────────┘ │ ▼ │ ┌─ SIP trunk ◄── FreeSWITCH (telephony_service) ── mod_audio_fork ──►│ 8 kHz PCM over WS │ (external) │ ESL control: originate, audio_stream_service (per-call bridge) │ │ hangup, DTMF, transfer │ ▲ └───────────────────┘ ▼ │ 22 kHz PCM RNNoise → Silero VAD → faster-whisper│ │ Piper TTS ◄─┐ (transcription_service, streaming) │ │ (tts_service)│ ▼ │ │ conversation_service (LangGraph) │ state machine + dialog policy ───┘ │ llm_service (vLLM · Qwen3-8B-Instruct) question gen · follow-ups · NLU intents │ Qdrant (BGE-small embeddings: question bank, JD context retrieval, answer similarity) │ scoring_service ── report_service ── notification/websocket ``` **Control plane**: Django owns candidates/jobs/calls in PostgreSQL and stays the source of truth. **Media plane**: FreeSWITCH ↔ audio_stream_service ↔ STT/TTS never touches Django — only events do. --- ## 4. Key technology decisions (and why) | Concern | Choice | Why (vs. alternative) | |---|---|---| | Telephony | **FreeSWITCH** | `mod_audio_fork` streams call audio to a WebSocket in real time and accepts injected audio back — exactly the AI-agent shape. Asterisk's ARI `externalMedia` works too but needs more glue (RTP socket handling); FreeSWITCH ESL gives cleaner per-call control. | | Trunk | Any SIP provider | Config-only (gateway XML). Twilio/Telnyx for global, Indian SIP trunk for domestic rates. | | STT | **faster-whisper** (CTranslate2) | 4× realtime on GPU for `small`/`medium`; word timestamps; Hindi + English in one model (whisper is natively multilingual → the bilingual requirement is free). | | VAD | **Silero VAD** | Tiny (1 MB), 1 ms/chunk on CPU, the de-facto standard for endpointing + barge-in. | | Denoise | **RNNoise** | Cheap CPU pre-filter; improves whisper accuracy on telephony audio noticeably. | | LLM runtime | **vLLM** (prod) / **Ollama** (dev) | vLLM's continuous batching is what makes 100+ concurrent dialogs affordable on one GPU; Ollama is the friction-free dev runtime. Same OpenAI-compatible API → one client. | | Model | **Qwen3-8B-Instruct** | Strong Hindi/Hinglish; good instruction following at 8B; fits 24 GB with fp8/awq. Llama-3.x-8B is the drop-in fallback (config value). | | Dialog | **LangGraph** | The call state machine (§6) is literally a graph; LangGraph gives us checkpointing (context survives service restarts mid-call), interrupt handling and deterministic transitions with LLM nodes only where needed. | | TTS | **Piper** | ~10× realtime on CPU (no GPU stolen from LLM/STT), stable latency, has Hindi + Indian-English voices. Coqui XTTS sounds better but is GPU-hungry and slower — offered as a per-call config flag later. | | Embeddings + vector DB | **BGE-small + Qdrant** | Question bank retrieval (per-skill question templates), JD chunk retrieval for grounded follow-ups, and answer-vs-JD similarity as a scoring feature. | | Events/queues | **Redis** (streams + pub/sub) + Celery | Call events flow through Redis streams; Celery handles non-realtime jobs (scoring, PDF, notifications). Realtime audio never goes through a queue. | | Realtime UI | **websocket_service** (FastAPI + uvicorn/websockets) | Subscribes to Redis pub/sub, fans out the §11 event list to browsers. Polling stays as fallback (already built). | | Monitoring | Prometheus + Grafana | Every service exposes `/metrics`; dashboards for call success rate, turn latency percentiles, GPU/queue saturation. | **Latency budget per conversational turn** (target < 1.5 s, hard cap 2.5 s): endpoint detect (Silero, 300 ms silence) → streaming STT finalize ~250 ms → LLM first token ~350 ms (vLLM, 8B) → Piper first audio chunk ~150 ms → playback starts. Barge-in: VAD speech-start event immediately kills TTS playback and flushes the audio queue. --- ## 5. Repository & folder structure New top-level `platform/` (self-hosted calling platform) beside the existing apps; the ATS keeps talking to it through `apps/ai_calls`. ``` TA-ATS-interns/ ├── backend/ # existing Django (control plane, unchanged owner of data) │ └── apps/ai_calls/ │ ├── models.py # AICall (exists) + CallTranscriptUtterance + CallEvaluation (Phase 2) │ ├── providers/ # Phase 2: provider abstraction │ │ ├── base.py # start_call/cancel_call interface + event dataclasses │ │ ├── mock.py # today's simulator (moved from hunar.py) │ │ └── selfhosted.py # POSTs to scheduler_service; verifies signed events │ ├── events.py # handle_status/transcript/completed (exists, stays) │ └── ... ├── platform/ │ ├── docker-compose.yml # whole platform, one command (Phase 13) │ ├── docker-compose.gpu.yml # vLLM/whisper GPU overrides │ ├── .env.example │ ├── shared/ # pip-installable `atsvoice-shared` │ │ ├── events.py # CallEvent schema (pydantic) — single contract for ALL services │ │ ├── redis_streams.py # publish/consume helpers, consumer groups │ │ ├── auth.py # HMAC signing for service↔Django callbacks │ │ └── audio.py # PCM helpers, resampling (8k↔16k↔22k) │ └── services/ │ ├── scheduler_service/ # FastAPI + Celery beat: dial queue, pacing, retry policy, │ │ │ # concurrency budgets per trunk, quiet-hours windows │ ├── telephony_service/ # FreeSWITCH container (dialplan, gateway XML, Lua hooks) │ │ └── esl_controller/ # Python ESL app: originate, answer detect (AMD), hangup causes │ ├── audio_stream_service/ # per-call WS bridge: mod_audio_fork ⇄ STT/TTS; RNNoise here │ ├── transcription_service/ # faster-whisper workers (GPU), Silero endpointing, partials │ ├── conversation_service/ # LangGraph graph = §6 state machine; dialog policy; timers │ ├── llm_service/ # vLLM (prod) / Ollama (dev) + prompt library + question generator │ ├── tts_service/ # Piper HTTP; per-language voice map (en-IN, hi-IN); audio cache │ ├── scoring_service/ # Celery worker: rubric scoring via LLM + heuristic features │ ├── report_service/ # Celery worker: PDF via backend's pdf_builder rubric layout │ ├── websocket_service/ # browser fan-out of call events (FastAPI WS) │ └── notification_service/ # email/webhook notifications on completion/failures ├── frontend/ # existing Next.js — AI screening page gains a WS client (Phase 12) └── docs/ai-calling-platform/ # this document + per-phase design notes ``` Each service: `app/` (code), `tests/`, `Dockerfile`, `README.md`, `/healthz` + `/metrics` endpoints. --- ## 6. Call state machine (single source: `shared/events.py`) ``` QUEUED → RINGING → CONNECTED → INTRODUCTION → QUESTIONING ⇄ FOLLOW_UP │ SUMMARIZATION → COMPLETED error edges from any state: FAILED · BUSY · NO_ANSWER · DROPPED ``` - Owned by **conversation_service** (LangGraph graph nodes = states; checkpointed to Redis so a worker crash resumes the call context). - Every transition publishes a `CallEvent` to Redis → Django callback (signed) + websocket_service. - Dialog rules encoded as graph edges: identity not verified twice → polite exit; reschedule intent → `RESCHEDULE_REQUESTED` terminal event (scheduler re-queues); silence > 6 s → reprompt, twice → summarize and end; total duration timer 7 min → forced SUMMARIZATION. - Interruptions: VAD speech-start during TTS → cancel playback, mark utterance `interrupted`, treat new speech as the answer. - Language: greeting detects reply language (whisper gives us the language id per segment); the graph carries `lang ∈ {en, hi}` and prompts/TTS voice switch accordingly. --- ## 7. Data model (Phase 2 — Django migrations) Existing `AICall` **is** the `Call` table from the brief (already has status/started/ended/duration/score/recommendation/summary + candidate/job/application FKs). We extend rather than duplicate: ``` CallTranscriptUtterance CallEvaluation (1:1 with AICall) id id call FK → AICall call OneToOne → AICall sequence int technical_score 0-100 speaker AGENT | CANDIDATE communication_score message text experience_score started_at ms offset into call confidence_score ended_at ms offset overall_score language en | hi classification Strong Match | Potential Match | Not Suitable interrupted bool recommendation Proceed | Hold | Reject strengths / weaknesses / summary (text/JSON) rubric JSON (per-question scores + evidence quotes) ``` - `AICall.transcript` (text blob) stays for backward compatibility and is assembled from utterances on completion. - Candidate-level `ai_*` fields remain **derived** from the latest AICall (as today) — no candidate migration. - Stage mapping unchanged: AI Calling → AI Screened / AI Qualified (threshold on `overall_score`). - Classification/recommendation mapping: Strong Match ≥ 75 → Proceed · Potential 55–74 → Hold (recruiter review) · else Not Suitable → Reject (stage stays AI Screened; LOST stages remain a human decision). --- ## 8. Contracts **ATS → platform** (Django `providers/selfhosted.py` → scheduler_service, HMAC-signed): ``` POST /v1/calls { call_id, phone, lang_hint, variables{candidate/job...}, max_duration_s, callback_url } POST /v1/calls/{id}/cancel GET /v1/calls/{id} ``` **Platform → ATS** (same three handlers that exist today, plus signature header `X-ATS-Signature`): ``` POST /api/v1/ai-calls/webhook { call_id, status, reason? } POST /api/v1/ai-calls/transcript { call_id, utterances: [...] } # batched every ~5 s POST /api/v1/ai-calls/completed { call_id, duration, transcript, summary, evaluation{...} } ``` **Platform → browsers** (websocket_service, event names per the brief): `call_queued · call_started · call_connected · transcript_updated · question_asked · call_completed · report_generated · call_failed` --- ## 9. Scalability model Per-call steady-state cost: 1 FreeSWITCH channel (light), 1 audio bridge task (asyncio, ~2 MB), streaming STT (~0.3× GPU-second/second on `small`), 1 LLM turn every ~15 s, TTS on CPU. | Concurrent calls | FreeSWITCH | STT (faster-whisper small) | LLM (vLLM Qwen3-8B) | TTS | |---|---|---|---|---| | 100 | 1 node | 1× 24 GB GPU (shared) | same GPU (batched) | 8 CPU cores | | 500 | 1–2 nodes | 2× 24 GB | 1× 80 GB or 2× 24 GB | 32 cores | | 1000+ | SIP load-balance (Kamailio, later) | 4× 24 GB | 2× 80 GB | 64 cores | - scheduler_service enforces a **dial-rate budget** (trunk CPS limit) and per-tenant concurrency; the queue never dumps 1000 originates at once. - All workers are stateless (state in Redis/Postgres) → Kubernetes-ready; compose file maps 1:1 to a Helm chart later. - UI never blocks: start returns immediately (exists today); updates via WS with polling fallback. --- ## 10. Security - SIP trunk credentials only inside telephony_service; FreeSWITCH not exposed publicly except trunk IP allowlist. - Service↔Django callbacks HMAC-signed (`shared/auth.py`) — replaces `HUNAR_WEBHOOK_SECRET`. - Call recordings/transcripts: PII — MinIO bucket with retention policy (Phase 13); browser WS requires the ATS JWT. --- ## 11. Migration & rollback `AI_CALL_PROVIDER = mock | hunar | selfhosted` (settings/env). The provider interface is extracted in Phase 2 so: - today's Hunar code becomes `providers/hunar.py` (kept until decommission), - the mock stays for dev/demo, - flipping back to Hunar is a config change during the pilot. --- ## 12. Phase plan (each ends with tests + your approval) | Phase | Deliverable | Runs on this offline box? | |---|---|---| | 1 | This document | ✅ | | 2 | Provider abstraction, new models/migrations, contracts (`shared/events.py`), Django APIs + tests | ✅ fully | | 3 | telephony_service: FreeSWITCH config, ESL controller, originate→answer→hangup lifecycle | 🟡 code+config here; needs Docker host to run | | 4 | audio_stream_service: mod_audio_fork WS bridge, RNNoise, resampling, loopback test harness | 🟡 harness runs here with WAV fixtures | | 5 | transcription_service: faster-whisper + Silero endpointing, partial/final events | 🟡 needs models downloaded | | 6 | conversation_service: LangGraph graph, dialog policy, question generator prompts | 🟡 logic tests run here with a stub LLM | | 7 | tts_service: Piper, voice map, streaming chunks, cache | 🟡 | | 8 | transcript storage: utterance batching → Django, blob assembly | ✅ | | 9 | scoring_service: rubric prompts + heuristic features → CallEvaluation | ✅ (stub LLM) / 🟡 (real) | | 10 | report_service: rubric PDF layout on the existing pdf_builder | ✅ | | 11 | ATS pages: evaluation panel, per-question scores, live transcript view | ✅ | | 12 | websocket_service + frontend WS client (polling fallback kept) | ✅ (uvicorn+websockets already in venv) | | 13 | docker-compose (+ GPU override), Prometheus/Grafana, MinIO, Celery/Redis swap-in | 🟡 authored here, verified on Docker host | ```