# Architecture This document explains the key design decisions in this system and why they were made — intended for whoever maintains this after handover (you, a future developer, or the client's own team). ## High-level request flow ``` Client app | v Nginx (SSL termination, rate limiting, reverse proxy) | v Gunicorn + Uvicorn workers (FastAPI) -- handles HTTP, validation, auth | v Redis cache check -- cache hit? return immediately | (cache miss) v Celery task queued -- API returns 202 + task_id immediately | v Celery worker process (separate from web workers) | +-- Feature builder constructs the model input vector +-- 6 models run in parallel via ThreadPoolExecutor +-- Conflict detection compares model outputs +-- LLM call (Claude) generates the human-readable explanation +-- Result written to PostgreSQL (audit trail) and Redis (cache) | v Client polls GET /api/v1/tasks/{task_id} until status is "completed" ``` ## Why async task queue instead of synchronous request handling The most important architectural decision in this system. Running 6 ML models plus an LLM call synchronously inside an HTTP request handler means that request holds a web worker thread for 2-5+ seconds. Under real concurrent load (the client's stated requirement: "high scale from day one"), this is the single fastest way to exhaust your worker pool and start queuing or rejecting requests. By queuing the work to Celery and returning a task_id immediately: - Web workers stay free to handle the next request in milliseconds - ML compute (CPU-bound) is isolated in its own process pool, scaled independently from web traffic (I/O-bound) - A slow or stuck model inference never blocks unrelated requests - You can scale Celery workers horizontally (more processes/VPS instances) without touching the web tier at all The tradeoff: the client's frontend must poll for results rather than getting a synchronous response. This is documented in the API contract and is standard practice for any API doing non-trivial compute. ## Why models are loaded once per process, not per request Loading 6 model files from disk (joblib deserialization) takes real time — tens to hundreds of milliseconds depending on model size. Doing this on every request would add that latency to every single analysis and would not scale. Instead: - The FastAPI app loads all models once at startup (see `app/main.py` lifespan) — this is used for nothing except the health check exposing `loaded_model_keys`, since the API process itself never runs inference. - Each Celery worker process loads all models once when it starts (see `app/workers/celery_app.py` `init_worker` signal) and keeps them in memory for the life of the process (`worker_max_tasks_per_child=200` recycles the process periodically to avoid memory creep, then it reloads models fresh). ## Why fail-fast on startup If any required model file is missing or corrupt, or required reference data (crops.json/districts.json) is absent, the application refuses to start entirely — see `app/models_loader/registry.py` and `app/services/reference_data.py`. This is deliberate. The alternative — starting in a degraded state and discovering the gap when a real user's request fails — is worse for a client-facing production system. A crash at deploy time is visible immediately in `systemctl status` / CI logs. A silent gap discovered three weeks later from a support ticket is not. ## Why every model runner catches its own exceptions Production ML systems WILL occasionally have one model fail — a SARIMAX model that doesn't exist yet for a new district, a malformed feature vector edge case, a transient numerical issue. See `app/services/model_runners.py`. Each `run_*` function is designed to never raise to its caller. Instead it returns a result dict with `"degraded": true` and an `"error"` key. This means a single model failing produces a partial, clearly-labeled result rather than a total 500 error for the whole analysis. The aggregator collects which models degraded and surfaces that to both the LLM (so its explanation is honest about what's missing) and the client response. ## Why Postgres AND Redis, not just one They serve different purposes and neither substitutes for the other: - **Redis** is the cache (ephemeral, TTL-based, fast) and the Celery broker/result backend. Data here is not meant to be permanent or queryable in complex ways. - **PostgreSQL** is the permanent audit trail (`AnalysisRecord`, `FailedRequestLog`). The client will want usage analytics, historical query patterns, and a record of every analysis served — that needs a real relational store with indexes, not a key-value cache with expiring data. ## Why the cache key includes model_version See `app/services/cache_service.build_cache_key`. When models are retrained and redeployed (new `MODEL_VERSION` in config), old cached results automatically become unreachable — no manual cache flush step required at deploy time, and no risk of accidentally serving stale predictions from a previous model generation. ## Why crops/districts are JSON lookup files, not hardcoded See `data/lookup/crops.json` and `data/lookup/districts.json`, loaded by `app/services/reference_data.py`. The project coordinator or client can add a new supported crop or district by editing a JSON file and redeploying — no code change needed. If the client later wants self-service management via an admin UI, this becomes a database table with the same read interface; calling code does not change. ## Known v1 limitations (documented, not hidden) - **Yield uncertainty** is currently a placeholder heuristic (`expected_yield * 0.15`) in the live inference path (`app/services/model_runners.run_yield_model`), even though the training script (`ml_pipeline/training/train_xgb_yield.py`) already trains a bootstrap ensemble capable of producing a real uncertainty estimate. Wiring the bootstrap ensemble into live inference is a straightforward follow-up — the artifact is already saved, the inference code just needs to use it. - **Price volatility** in the market model output is currently `"unknown"` — computing it properly requires tracking SARIMAX residual variance, not yet implemented in `run_market_model`. - **Trend model `slope`** field is currently always `null` in the API response — the Decision Tree classifier predicts a trend category but the underlying numeric slope isn't passed through. Fixable by adding it as a stored feature looked up at inference time. None of these break the system — they degrade gracefully to `null` or a placeholder, clearly documented in code comments at each site. They are listed here so they are a visible roadmap item, not a surprise.