# Data sourcing and licensing

This system requires three categories of real-world data. This document
covers where to get each one, the realistic timeline, and — critically
— the licensing considerations for a **commercial client product**,
not just a research/personal project.

## Why this document exists

Using government/research data sources in a paid product the client
sells or operates commercially is a different legal situation than
using the same data for a personal project or thesis. Some of these
sources are free for research but require a paid commercial license.
This must be resolved BEFORE the client goes live with real users, not
discovered after.

## 1. Climate data — IMD (India Meteorological Department)

**Source**: https://dsp.imdpune.gov.in/ (IMD Pune Data Supply Portal)

**What to get**: District-level monthly rainfall and temperature,
historical (10-30 years) for every district the client wants to support.

**Licensing**: IMD distinguishes between research/academic use (often
free or low-cost) and commercial use (typically requires a paid data
license, negotiated directly with IMD). **Contact IMD's licensing desk
directly before launch** — email imdpune@gov.in or use their official
licensing request form. Do not assume free-tier access extends to a
commercial product.

**Stopgap while licensing is in progress**: `data.gov.in` publishes
some open rainfall datasets that may be usable for an initial pilot or
demo, with looser licensing terms (verify the specific dataset's
license on data.gov.in itself — terms vary by dataset). This lets
backend/ML development proceed in parallel with the IMD licensing
conversation, but should be replaced with properly licensed data before
a paid client launch on data the client is relying on commercially.

## 2. Crop yield data — ICRISAT / Government APY

**Primary recommended source**: APY (Area, Production, Yield) database
at https://aps.dac.gov.in — published by the Government of India's
Department of Agriculture. This is official government statistics,
generally available for public/commercial use, and more current than
ICRISAT (which can lag by a few years).

**Secondary/historical backfill**: ICRISAT VDSA database at
http://vdsa.icrisat.ac.in — free for research use; check ICRISAT's
specific terms if used in a commercial product, as research-data
licenses commonly restrict commercial redistribution even when the
underlying analysis/product built on it is fine. When in doubt, prefer
the government APY source as primary and treat ICRISAT as a
cross-validation reference rather than the production data source.

## 3. Market price data — Agmarknet via data.gov.in

**Source**: https://data.gov.in's Agmarknet mirror, accessed via their
public REST API (register for a free API key at data.gov.in).

**Licensing**: data.gov.in publishes most datasets under the Government
Open Data License – India (GODL), which is generally permissive,
including for commercial use, but **verify the specific license badge
on the exact dataset/resource ID you use** — it is shown on the
dataset's data.gov.in page. This is the most straightforward of the
three sources licensing-wise.

**Technical access**: `ml_pipeline/data_collection/fetch_agmarknet.py`
in this codebase already implements the API client against this source.

## Realistic timeline for a client launch

| Step | Source | Typical lead time |
|---|---|---|
| Register data.gov.in API key | Agmarknet | Same day |
| Pull APY crop yield data | Government APY portal | 1-3 days (manual CSV export per district) |
| Start IMD commercial licensing conversation | IMD | **Start immediately — can take weeks** |
| Use data.gov.in rainfall as stopgap | data.gov.in | Same day, lower data quality/coverage than IMD |

**Recommendation given to the project coordinator**: do not let IMD's
licensing timeline block the rest of development. Build and test the
full pipeline against the stopgap/synthetic data now, and swap in
properly licensed IMD data before the client's real go-live date. Flag
the IMD licensing cost and timeline to the client directly and early —
it is a cost outside your development scope, not a development delay
you are responsible for.

## Updating data after initial launch

The pipeline in `ml_pipeline/` is designed to be re-run periodically as
fresh data becomes available:

- Climate models: retrain yearly (see `docs/architecture.md` for the
  full retraining checklist) as new IMD data is released
- Market models: retrain quarterly, since price patterns shift faster
  than climate patterns
- Crop yield models: retrain after each new agricultural census/year of
  APY data becomes available

Run `python -m ml_pipeline.training.run_full_pipeline` with updated raw
data paths, then `python -m ml_pipeline.evaluation.validate_model_bundle`
to confirm the new model bundle is deployment-ready before replacing
the production `data/models/v1/` directory.