# TA-ATS — Root Causes & Fixes

Every issue hit during the first production bring-up, **in the order they appeared**,
with the underlying cause (not just the symptom) and the permanent fix.

---

## 1. Login page: `ERR_NAME_NOT_RESOLVED` + intermittent 503
**Symptom:** `POST /api/v1/auth/login/` failed; DevTools showed the request going to
the wrong host.

**Root cause:** The frontend reads the API base from `NEXT_PUBLIC_API_URL`, and
**Next.js bakes `NEXT_PUBLIC_*` variables into the bundle at BUILD time** (not runtime).
The production build was made **without** that variable, so the shipped JS still
pointed at the dev default `http://localhost:8000/api/v1`. The visitor's browser then
tried to resolve a host that doesn't exist publicly.

**Fix:** Set it before building, then rebuild:
```
# frontend/.env.production
NEXT_PUBLIC_API_URL=https://ats.indovisionconsultancy.in/api/v1
```
```
npm run build && (restart frontend)
```
See `frontend/src/lib/api.ts` (`API_BASE_URL`).

---

## 2. Login: `{"message":"MFA service unavailable. Try again later."}`
**Symptom:** Login reached the backend but returned 503 with that message.

**Root cause:** The Django backend delegates password/MFA to the **MCP service** over
`settings.MCP_URL`. The value in `backend/.env` had a **one-character typo**:
`http://127.0.0.0:9000/mcp` — `127.0.0.**0**` is not the loopback host (`127.0.0.**1**`),
so every MCP call failed → `MCPServiceError` → the 503 message (see
`backend/core/exceptions.py` and `backend/apps/authentication/mcp_client.py`).

**Fix:**
```
# backend/.env
MCP_URL=http://127.0.0.1:9000/mcp
```
Then restart the backend so it reloads `.env`.

**How to confirm the MCP itself is fine:** a healthy MCP answers a plain
`curl http://127.0.0.1:9000/mcp` with
`{"jsonrpc":"2.0",...,"Not Acceptable: Client must accept text/event-stream"}` — that
response means it is **up** (the Django client sends the correct `Accept` header).

---

## 3. `address already in use` (ports 8000/8002/3000/9000)
**Symptom:** Services "started" but logs showed `[Errno 98] address already in use`;
`/health` returned a `301` (Django) instead of the gateway's JSON.

**Root cause:** **Multiple processes competing for the same port.** Manual `nohup`
starts, leftover runs, and — critically — a **systemd unit (`ats.service`)** were all
launching Django/gateway processes. Whatever grabbed the port first won; the rest
crashed silently. The `301` was Django's `SECURE_SSL_REDIRECT`, proving Django (not the
gateway) held port 8000.

**Fix:** Exactly **one process per port**. Stop everything and free the ports before
starting, and know which manager owns each service:
```
./run_linux.sh stop
fuser -k 8000/tcp 8002/tcp 3000/tcp 9000/tcp
```
On this server, **`ats.service` (systemd) owns Django on :8000** — do not also start a
backend on 8000 from a script.
```
systemctl status ats.service
systemctl restart ats.service
```

---

## 4. Gateway won't start: `No module named uvicorn`
**Root cause:** The `gateway/` folder had **no virtualenv** on the server (only `app/`
and `requirements.txt`). Scripts fell back to a Python that lacked FastAPI/uvicorn.

**Fix:** Give the gateway its own venv (only needed where the gateway is actually run):
```
cd gateway
python3.12 -m venv venv
./venv/bin/pip install -r requirements.txt
```

---

## 5. `ats.service` served STALE config
**Symptom:** A direct `verify_login` test succeeded, but the live web login still
returned "MFA service unavailable".

**Root cause:** `ats.service` (the process actually serving `/api` via nginx) had been
**started before** the `MCP_URL` typo was fixed, so it still held the bad value in
memory. Editing `.env` does nothing until the process reloads it.

**Fix:** `systemctl restart ats.service` after any `.env`/code change.

---

## 6. Seeder crash: `duplicate key ... pipeline_stages_name_key`
**Symptom:** `seed_master_defaults.py` aborted on `Candidate Responded`.

**Root cause:** The seeder did `get_or_create(code=...)`, but existing rows had a
**different/empty `code`**, so the lookup missed and it tried to **create** a row with
a `name` that already exists (name is unique) → `IntegrityError`. A non-idempotent
seeder.

**Fix (code):** key on the unique column and update:
```python
PipelineStage.objects.update_or_create(name=name, defaults={"code": code, ...})
```
**Fix (already-deployed DB):** align existing codes first, then re-run:
```python
for n,c in codes.items():
    PipelineStage.objects.filter(name=n).update(code=c)
```

---

## 7. Frontend build fails: `Module not found: Can't resolve 'xlsx'`
**Root cause:** A `git pull` brought **new code that imports `xlsx`**, but
`npm install` was not run afterward, so the dependency was missing. Build failed →
no `.next` → the old/broken build kept serving.

**Fix:** Always run `npm install` after pulling, and install the missing dep:
```
npm install
npm install xlsx
npm run build
```

---

## 8. Dashboard: "This page couldn't load" (THE dashboard bug)
**Symptom:** Login worked (API returned 200 + tokens), all dashboard APIs returned 200,
all JS chunks 200 — yet the dashboard showed Next.js's error screen.

**Root cause:** The dashboard opened a WebSocket to
`ws://<host>:8000/api/ws/activity`. The page is served over **HTTPS**, and browsers
**block insecure `ws://` from an HTTPS page** ("Mixed Content"). `new WebSocket(...)`
then throws a `SecurityError` **synchronously during React render**, which was
uncaught → the whole page crashed into the error boundary. Nothing to do with the
build, seed, MCP, or APIs.

**Fix:**
1. Use `wss://` (and the same origin, so nginx can proxy it) when on HTTPS:
   ```
   wss://<host>/api/ws/activity      # instead of ws://<host>:8000/...
   ```
   Implemented via `frontend/src/lib/ws.ts` (`activityWsUrl()`), used by
   `AdminDashboard.tsx` and `Header.tsx`.
2. Never let a socket crash the page — construct it defensively (`safeWebSocket()` in
   the same file returns `null` instead of throwing).
3. For the live feed to actually connect, nginx must proxy `/api/ws/activity` with
   WebSocket **Upgrade** headers to the gateway (see SETUP_LINUX.md → "WebSocket").

---

## Quick diagnosis cheat-sheet

| Symptom | Look at | Likely cause |
|---|---|---|
| `ERR_NAME_NOT_RESOLVED` on login | DevTools request URL | frontend built without `NEXT_PUBLIC_API_URL` (#1) |
| "MFA service unavailable" | `backend/.env` `MCP_URL`; is MCP on :9000? | typo `127.0.0.0` or MCP down (#2, #5) |
| `address already in use` | `ss -ltnp \| grep :PORT` | duplicate/stale process (#3) |
| gateway `No module named uvicorn` | `gateway/venv` exists? | missing venv (#4) |
| seeder `duplicate key` | seeder uses `get_or_create(code=)` | non-idempotent seeder (#6) |
| build `Module not found` | did you `npm install` after pull? | missing dep (#7) |
| "This page couldn't load" (page 200) | DevTools **Console** | `ws://` on HTTPS crash (#8) |