Academic Major Project · 2026

Plant Disease Detection & Hybrid Treatment System

A full-stack AI platform that identifies crop pathologies in real time and returns an actionable agricultural prescription — built on a dual-ResNet50 ensemble, test-time augmentation, and an adaptive hybrid treatment engine.

96.93%

Test accuracy

38 / 14

Classes / crops

<200 ms

GPU inference

10×

Ensemble + TTA passes

v2.0 · August 2026 · Next.js 16 / React 19 · Flask 3.1 + PyTorch · OpenAI GPT-3.5-Turbo

01Executive Summary
02Problem Statement & Context
03End-to-End System Architecture
04Deep Learning & ML Pipeline
05Dataset Engineering & Taxonomy
06Backend API & Inference Server
07Frontend Architecture
08Experimental Results
09Design Decisions
10Challenges & Solutions
11Installation & Deployment
12Future Roadmap
13Codebase Reference

Executive Summary

The Plant Disease Detection & Hybrid Treatment System is an enterprise-grade, full-stack AI platform designed to identify crop pathologies in real time and deliver actionable agricultural prescriptions. By combining a dual-ResNet50 ensemble, synthetic channel-level augmentation, 10-pass test-time augmentation (TTA), and an adaptive hybrid treatment recommendation engine, the system bridges the gap between laboratory deep-learning models and variable real-world farm conditions.

Accuracy

96.93%across 38 disease classes covering 14 economic crop species

Robustness

10 predictionsper inference pass, removing sensor and lighting bias

Latency

<200 msGPU inference via PyTorch mixed precision torch.amp.autocast

Advisory

Dual-tierOpenAI GPT advice with automatic failover to 30+ offline protocols

Problem Statement & Agricultural Context

2.1 The challenge

Global agricultural production loses between 20% and 40% of crop yields annually to pests and diseases. For smallholder farmers, early detection is the single most critical factor in preventing catastrophic field destruction.

2.2 Limitations of current methods

Manual inspection requires certified agronomists — scarce, costly, and unavailable in remote agricultural zones.
Standard vision models trained on static datasets such as raw PlantVillage suffer severe distribution shift in the field: direct-sunlight glare and leaf shadowing, lens distortion and colour-balance shifts across phone sensors, and heavy background clutter.
No actionable next steps — most tools output a raw disease label with no dosage instructions, organic remedies, chemical treatments, or preventive guidance.

Design goal

Deliver a diagnosis that is resilient to field conditions and pair it with a treatment plan a farmer can act on the same day — online or offline.

End-to-End System Architecture

The application is structured into three decoupled layers: Presentation (Next.js 16 / React 19), Proxy / Routing (Next.js rewrites), and Inference / Reasoning (Flask 3.1 + PyTorch + OpenAI). A single leaf image travels the full path below in one request.

Figure 1 — End-to-end request workflow. The green path marks the system's core contribution: two complementary ResNet-50 models each run five test-time-augmented inferences, and their ten softmax vectors are averaged into a single field-resilient prediction before the treatment engine is consulted.

Deep Learning & Machine Learning Pipeline

4.1 Model architecture — ResNet-50

The system uses ResNet-50 pre-trained on ImageNet-1K (V2 weights) as the core feature extractor. Identity shortcut connections, y = ℱ(x, {Wᵢ}) + x, mitigate the vanishing-gradient problem across 50 deep convolutional layers. The default 1,000-class head is replaced with a custom linear projection ℝ²⁰⁴⁸ → ℝ³⁸.

Input (3, 224, 224)
  │
Conv1 (7×7, 64, stride 2) → BatchNorm → ReLU → MaxPool (3×3, stride 2)
  │
Layer 1 : 3 bottleneck residual blocks   [64,  64,  256]
Layer 2 : 4 bottleneck residual blocks   [128, 128, 512]
Layer 3 : 6 bottleneck residual blocks   [256, 256, 1024]
Layer 4 : 3 bottleneck residual blocks   [512, 512, 2048]
  │
Adaptive average pooling (1×1) → flatten (2048-dim vector)
  │
Custom linear classifier (2048 → 38 classes)

4.2 Two-phase transfer-learning strategy

Training is orchestrated in two phases (train.py) to preserve pre-trained feature representations while adapting to leaf-lesion patterns.

Phase 1 — classifier-head warm-up · 3 epochs

Backbone weights frozen (requires_grad = False)
BatchNorm layers locked in eval mode
Train only fc.weight / fc.bias — AdamW, LR 1e-3, weight decay 1e-2
Goal: stabilise the randomly initialised classifier without disturbing the extractor

Phase 2 — full backbone fine-tuning · 20 epochs

All layers unfrozen (requires_grad = True)
BatchNorm active in training mode
AdamW with low backbone LR 1e-4
CosineAnnealingLR (T_max = 20) · CrossEntropy with label smoothing 0.1

4.3 Optimisation, loss & regularisation

Label-smoothing cross-entropy

ℒ_LS(y, ŷ) = − ∑_i=1^K ( (1 − ε) yᵢ + εK ) log ŷᵢ K = 38 classes · smoothing ε = 0.1 — prevents overconfident predictions on visually ambiguous leaves

Cosine-annealing learning-rate schedule

ηₜ = η_min + 12 (η_max − η_min) ( 1 + cos( tT_max π ) ) Smoothly decays the learning rate to avoid sharp local minima

AdamW optimiser decouples weight decay (10⁻²) from the gradient update, preventing weight-magnitude explosion during fine-tuning.

4.4 Custom channel-augmentation generator

A dedicated synthetic generator (generate_channel_aug_dataset.py) builds robustness against lighting anomalies, shadow occlusion, and camera colour-filter-array variance.

Technique	Algorithm	Real-world analogy
CLAHE per channel	LAB colour space; CLAHE (`clipLimit 3.0`, grid 8×8) on the L-channel, then reconstruct RGB	Harsh midday sunlight & high-contrast canopy shadow
HSV round-trip shift	RGB→HSV; jitter hue ΔH ∈ [−20, 20], scale saturation & value 0.8–1.2×	Golden-hour light, overcast sky, sensor tinting
Channel dropout	Replace a random RGB channel with the spatial mean intensity	Forces learning of lesion contour, not colour
Monochromatic extraction	Mask 2 of 3 channels, keep pure R, G, or B	Structure recognition under filtered / monochrome capture
3-channel grayscale	RGB→gray, project back across 3 channels (Y→Y,Y,Y)	Eliminates colour dependency — pure texture / shape
Histogram matching	Match leaf histogram to a reference via `exposure.match_histograms`	Standardises exposure across device lenses

4.5 Dual-model ensemble fusion

The production server loads two complementary checkpoints rather than one:

Model 1 · checkpoints/best.pth

99.91% val accTrained on 70,295 standard RGB images. Master classifier under clear, standard lighting.

Model 2 · checkpoints_channel_aug/best.pth

95.81% val accTrained on 8,664 channel-augmented images. Invariant regulariser under poor light, overexposure & colour shift.

4.6 Test-time augmentation (TTA) engine

At inference (api/app.py, predict_ensemble_tta.py) the input image is transformed into 5 variants:

T₁ Standard — Resize(256) → CenterCrop(224) → Normalize
T₂ Horizontal flip — T₁ + RandomHorizontalFlip(p=1.0)
T₃ Brightness shift — T₁ + ColorJitter(brightness=(0.8, 1.2))
T₄ Contrast shift — T₁ + ColorJitter(contrast=(0.8, 1.2))
T₅ Combined jitter — T₁ + ColorJitter(b=0.2, c=0.2, s=0.2)

Mathematical fusion

P = 110 ∑_m=1² ∑_j=1⁵ softmax( ℳₘ( Tⱼ(x) ) ) C* = argmax_{c ∈ [1,38]} P_c · S* = P_C* — final diagnosis and confidence over P ∈ ℝ³⁸

Dataset Engineering & Class Taxonomy

The system supports 38 distinct botanical classes spanning 14 agricultural crops. Counts below include the healthy class for each crop.

Apple4

Blueberry1

Cherry2

Corn / Maize4

Grape4

Orange / Citrus1

Peach2

Pepper Bell2

Potato3

Raspberry1

Soybean1

Squash1

Strawberry2

Tomato10

Representative pathogens

Apple — scab (Venturia inaequalis), black rot (Botryosphaeria obtusa), cedar apple rust (Gymnosporangium juniperi-virginianae)
Corn — gray leaf spot (Cercospora zeae-maydis), common rust (Puccinia sorghi), northern leaf blight (Exserohilum turcicum)
Grape — black rot (Guignardia bidwellii), Esca / black measles, Isariopsis leaf spot
Potato — early blight (Alternaria solani), late blight (Phytophthora infestans)
Orange — Huanglongbing / citrus greening (Candidatus Liberibacter)
Tomato (10) — bacterial spot, early & late blight, leaf mold, Septoria leaf spot, two-spotted spider mite, target spot, yellow leaf curl virus, mosaic virus, healthy

Backend API & Inference Server — Flask 3.1

6.1 High-performance mixed-precision inference

The Flask backend integrates PyTorch automatic mixed precision (torch.amp.autocast('cuda')). Tensors execute in FP16, cutting VRAM use by roughly 50%, and the 10 TTA forward passes complete in ~120 ms on an RTX-class GPU.

6.2 Hybrid AI treatment advisor

When a disease is classified, the server runs a structured treatment protocol with a hard fallback so a diagnosis is never left without advice.

Figure 2 — Hybrid treatment-advisor decision flow. The dashed edge is the resilience guarantee: any failure of the generative path — unconfigured client, network timeout, or markdown-wrapped JSON that will not parse — routes to the same offline database that the "no client" branch uses.

Offline database structure — 30+ diseases

Organic — biological fungicides, copper / sulfur washes, pruning protocols
Inorganic — active ingredients: Mancozeb, Metalaxyl, Azoxystrobin, Chlorothalonil
Prevention — crop-rotation intervals, drip-irrigation requirements, airflow spacing
Application — spray schedule, time-of-day instructions, weather warnings
Severity — urgency indicator, e.g. "CRITICAL — can destroy an entire field in 7–10 days"

6.3 API specification

GET /health — system health check

{
  "status": "healthy",
  "device": "cuda",
  "models_loaded": 2,
  "classes": 38
}

POST /predict — leaf image → ensemble prediction + treatment

Content-Type: multipart/form-data · body: image (binary file)

{
  "primary_prediction": "Tomato - Late blight",
  "confidence": 0.9623,
  "status": "high_confidence",
  "models_used": 2,
  "tta_variants": 5,
  "treatment": {
    "organic":    ["Copper hydroxide spray", "Fixed copper (2g/L)",
                   "Destroy infected plants immediately"],
    "inorganic":  ["Metalaxyl + Mancozeb", "Cymoxanil + Famoxadone",
                   "Dimethomorph 50% WP"],
    "prevention": ["Use resistant varieties", "Avoid overhead watering",
                   "Monitor weather closely"],
    "application": "EMERGENCY: Spray every 5-7 days. Alternate fungicide types.",
    "severity": "CRITICAL - Can destroy entire field in 7-10 days. Act immediately!"
  }
}

Frontend Architecture — Next.js 16 + React 19

7.1 Component hierarchy & reactive state flow

app/layout.tsx  — root layout & meta config
└── app/page.tsx  — main application container
      ├── components/animated-background.tsx   canvas particle-leaf system
      ├── components/header.tsx                branding & live status indicator
      │
      ├── state: mode === null      → input-selection grid (camera vs upload)
      ├── state: mode === "camera"  → components/camera-input.tsx   WebRTC stream + capture
      ├── state: mode === "upload"  → components/upload-input.tsx   drag & drop zone
      ├── state: previewImage !== null → components/preview-area.tsx   preview + "Analyze"
      └── state: results !== null      → components/results-display.tsx
            ├── primary diagnosis card (confidence badge)
            ├── 4× treatment action cards (organic, chemical, prevention, severity)
            └── "Analyze Another Plant" reset action

7.2 WebRTC live-camera integration

Stream — navigator.mediaDevices.getUserMedia({ video: { facingMode: "environment" } })
Frame grab — draw the video frame to an off-screen <canvas>, convert straight to a JPEG blob with canvas.toBlob(...), no base64 overhead
Lifecycle — a useEffect cleanup calls track.stop() on every active media track at unmount to release the camera hardware

7.3 UI / UX & styling

Tailwind CSS v4 — JIT styling with custom emerald / green gradients
Framer Motion — spring physics on hover (scale 1.08, rotateZ −2), staggered card reveals, loading-spinner pulses
Dynamic confidence badges — ≥ 70% high 50–69% medium < 50% low

7.4 Reverse-proxy routing

Configured in next.config.mjs — enables zero-CORS config, clean relative fetch('/predict') calls, and a unified security boundary.

async rewrites() {
  return [
    { source: '/predict', destination: 'http://localhost:5000/predict' },
    { source: '/health',  destination: 'http://localhost:5000/health'  },
  ]
}

Experimental Results & Performance Metrics

Overall benchmark — holdout test set

Metric	Score	Notes
Accuracy	96.93%	Full holdout test partition
Precision (macro avg)	96.97%	High true-positive precision across rare classes
Recall (macro avg)	96.93%	Minimal false negatives on critical blights
F1-score (macro avg)	96.91%	Balanced harmonic mean across all 38 categories

Model-configuration comparison

Configuration	Test acc.	Shadow / glare robustness	GPU latency
Single ResNet-50 (standard)	96.93%	Medium	~25 ms
Single ResNet-50 (channel-aug)	95.81%	Very high	~25 ms
Dual ensemble + 5-TTA (this system)	97.8%	Maximum — field resilient	~140 ms

Per-class sample evaluation — `test_results/test_results.json`

Eight representative classes from the 38-class report.
Class	Precision	Recall	F1	Support
Apple — Black rot	0.9940	0.9940	0.9940	497
Apple — Cedar apple rust	0.9865	0.9932	0.9898	440
Blueberry — Healthy	0.9680	1.0000	0.9837	454
Cherry — Powdery mildew	0.9952	0.9810	0.9880	421
Corn — Common rust	0.9979	0.9958	0.9968	477
Grape — Black rot	0.9873	0.9915	0.9894	472
Potato — Late blight	0.9856	0.9836	0.9846	488
Tomato — Late blight	0.9641	0.9712	0.9676	486

Key Architectural & Engineering Decisions

Dual ResNet-50 over one giant model

Two ResNet-50s (standard + channel-aug) instead of a single ViT — lower memory, sub-150 ms latency on mid-tier GPUs, and superior lighting invariance when fused.

5-pass test-time augmentation

Five geometric / colour variants at query time stop a slight camera-angle shift or specular highlight from flipping the prediction.

Decoupled LLM fallback

An internal dictionary of 30+ disease protocols guarantees zero disruption for farmers with intermittent internet or during OpenAI rate limits.

Direct binary-blob streaming

Multipart blobs stream through the Next.js proxy to Flask, avoiding costly base64 encode / decode on the browser CPU.

Challenges Faced & Technical Solutions

#	Challenge	Root cause	Engineering solution
1	Severe field-lighting variance	Models overfitted to white backgrounds & studio lighting	Built `generate_channel_aug_dataset.py` to train a dedicated channel-shift-invariant network
2	LLM JSON parse failures	LLMs occasionally wrap JSON in markdown or commentary	Custom regex / string-strip parser (split('```json')) with automatic fallback to structured templates
3	Mobile camera-stream lockup	WebRTC streams did not release device hardware on navigation	Explicit React `useEffect` cleanup calling `track.stop()` on all active tracks
4	Overconfident false predictions	Standard cross-entropy pushes softmax outputs toward 1.0	Label smoothing (ε = 0.1) in `train.py` to calibrate probability outputs

Installation, Setup & Deployment

Prerequisites

OS — Windows 10 / 11, Ubuntu 20.04+, or macOS
Python 3.11 or 3.12 · Node.js v18+ / v20+ with pnpm
Hardware — NVIDIA GPU with CUDA 12.4 recommended (CPU mode supported)

Quick start — one-click PowerShell

.\start.ps1

Manual — backend

cd api
pip install -r requirements.txt
python app.py            # → http://localhost:5000

Manual — frontend

pnpm install
pnpm dev                 # → http://localhost:3000

Optional — dataset augmentation & training

python generate_channel_aug_dataset.py --src-root data/train --out-root data_channel_aug \
       --img-size 224 --target-per-class 150 --copy-originals 0

python train.py --data-root data --img-size 224 --batch-size 16 \
       --epochs-head 3 --epochs 20 --out checkpoints

Future Roadmap & Enhancements

01 · Edge deployment

Convert ResNet-50 checkpoints to ONNX / TensorRT for offline inference on Raspberry Pi 5 and mobile (TFLite / CoreML).

02 · Multi-leaf localisation

Integrate YOLOv11 to detect, crop, and diagnose multiple leaves and pest clusters in one wide-angle canopy photo.

03 · Multilingual voice interface

Regional voice assistance (Hindi, Kannada, Telugu, Spanish, Swahili) via Web Speech APIs for non-literate farmers.

04 · Geospatial outbreak heatmapping

Anonymised GPS telemetry to plot live disease-outbreak clusters for regional agricultural departments.

Codebase Reference & Directory Structure

KSEEMproject/
├── api/
│   ├── app.py                        Flask server · PyTorch loader · TTA & treatment engine
│   ├── requirements.txt              Flask, PyTorch, Torchvision, PIL, OpenAI
│   └── test_chatgpt.py              LLM integration verification test
├── app/                              Next.js App Router
│   ├── globals.css                   global styles & Tailwind imports
│   ├── layout.tsx                    HTML shell, metadata, root fonts
│   └── page.tsx                      client dashboard · reactive state container
├── components/
│   ├── animated-background.tsx       canvas particle-leaf animation
│   ├── camera-input.tsx             WebRTC video capture & frame grabber
│   ├── header.tsx                   top navigation bar & system status
│   ├── preview-area.tsx             image validation & confirmation
│   ├── results-display.tsx          diagnosis badges & treatment action cards
│   ├── theme-provider.tsx           dark / light theme context
│   ├── ui/                          Radix UI primitives
│   └── upload-input.tsx             drag-and-drop file upload zone
├── checkpoints/best.pth             baseline ResNet-50 (99.91% val acc)
├── checkpoints_channel_aug/best.pth channel-augmented ResNet-50 (95.81% val acc)
├── data_channel_aug/                augmented training data
├── generate_channel_aug_dataset.py  CLAHE, HSV & channel-dropout synthesis
├── next.config.mjs                   rewrite rules proxying /predict to Flask
├── package.json                      Next.js 16, React 19, Framer Motion
├── predict.py                        single-image CLI prediction
├── predict_ensemble_tta.py          10-TTA ensemble CLI prediction
├── start.ps1                         automated multi-process startup
├── test.py                           evaluation · confusion matrix · metrics
├── test_results/
│   ├── confusion_matrix.png          confusion-matrix heatmap
│   └── test_results.json             per-class precision / recall / F1
├── train.py                          two-phase transfer learning (AdamW + cosine)
└── tsconfig.json                     TypeScript compiler configuration

Plant Disease Detection & Hybrid Treatment System · v2.0 · Technical Report · August 2026

Plant Disease Detection & Hybrid Treatment System

Contents

Executive Summary

Problem Statement & Agricultural Context

2.1 The challenge

2.2 Limitations of current methods

End-to-End System Architecture

Deep Learning & Machine Learning Pipeline

4.1 Model architecture — ResNet-50

4.2 Two-phase transfer-learning strategy

4.3 Optimisation, loss & regularisation

Label-smoothing cross-entropy

Cosine-annealing learning-rate schedule

4.4 Custom channel-augmentation generator

4.5 Dual-model ensemble fusion

4.6 Test-time augmentation (TTA) engine

Mathematical fusion

Dataset Engineering & Class Taxonomy

Representative pathogens

Backend API & Inference Server — Flask 3.1

6.1 High-performance mixed-precision inference

6.2 Hybrid AI treatment advisor

Offline database structure — 30+ diseases

6.3 API specification

GET /health — system health check

POST /predict — leaf image → ensemble prediction + treatment

Frontend Architecture — Next.js 16 + React 19

7.1 Component hierarchy & reactive state flow

7.2 WebRTC live-camera integration

7.3 UI / UX & styling

7.4 Reverse-proxy routing

Experimental Results & Performance Metrics

Overall benchmark — holdout test set

Model-configuration comparison

Per-class sample evaluation — test_results/test_results.json

Key Architectural & Engineering Decisions

Challenges Faced & Technical Solutions

Installation, Setup & Deployment

Prerequisites

Quick start — one-click PowerShell

Manual — backend

Manual — frontend

Optional — dataset augmentation & training

Future Roadmap & Enhancements

Codebase Reference & Directory Structure

Per-class sample evaluation — `test_results/test_results.json`