w4ashabii

SmolVLM256M_CropDisease

README

License: apache-2.0

Model Details

Table

Base model	HuggingFaceTB/SmolVLM-256M-Instruct (~256M params)
Fine-tuning method	LoRA (adapters only, base weights frozen)
Trainable params	3,769,344 / 260,254,272 (1.45%)
Framework	🤗 `transformers` `Trainer` + Unsloth `FastVisionModel`
LoRA targets	Attention + MLP modules across both vision encoder and language model
LoRA config	r=8, alpha=16, dropout=0.05, bias="none"
Precision	fp16
Hardware	1x NVIDIA T4 (Google Colab)
Language(s)	English (Q&A text); images of Nepali-grown crops
License	Apache 2.0 (inherited from base model)

Training Data

Dataset: w4ashabii/nepali_crop_data
Format: ShareGPT-style conversations (sharegpt.jsonl), one user turn (image + question) → one assistant turn (crop + disease answer)
Train examples used: 39,987
Eval examples used: 500 (held-out validation split)
Image preprocessing: resized to longest_edge=512, image splitting disabled (not needed for single-leaf/crop photos)

Training Procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Epochs	2
Per-device batch size	8
Gradient accumulation steps	2
Effective batch size	16
Learning rate	1e-4
LR schedule	cosine, 3% warmup
Optimizer	AdamW (`Trainer` default)
Total steps	5,000
Total training time

Checkpoints were saved locally every 500 steps and mirrored to this Hub repo roughly every hour during training via a custom TrainerCallback (each upload replaced the repo's prior contents), with a final clean upload of just the adapter + processor after training completed.

Training / Validation Loss

Table with columns: Step, Training Loss, Validation Loss
Step	Training Loss	Validation Loss
500	0.1948	0.1918
1000	0.1842	0.1871
1500	0.1878	0.1858
2000	0.1827	0.1865
2500	0.1685	0.1856
3000	0.1809	0.1841

Final train loss: 0.2363 (mean over all logged steps, includes early-training values before it stabilized). Validation loss plateaued around 0.183 from step ~3500 onward, suggesting the model converged with room left in the schedule rather than overfitting.

Usage

python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

BASE_MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
ADAPTER_REPO = "w4ashabii/SmolVLM256M_CropDisease"

processor = AutoProcessor.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForVision2Seq.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).to("cuda")
model.eval()

image = Image.open("your_crop_photo.jpg").convert("RGB")
question = "What crop is shown in this image, and does it have any disease? If so, name the disease."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=64)

answer = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer.strip())

With Unsloth (faster inference load)

python
from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained("w4ashabii/SmolVLM256M_CropDisease")
FastVisionModel.for_inference(model)
# ... same generate() call as above

Intended Use

Assistive identification of crop type and visible disease symptoms from photos, for agricultural extension, research, or educational tooling focused on Nepali-grown crops.
Not intended as a sole basis for treatment or pesticide decisions — outputs should be verified by an agronomist or local agricultural extension service, especially for high-stakes crop management decisions.

Limitations

Small base model (256M params) and LoRA-only tuning trade some accuracy for speed/size; expect a lighter-weight, less nuanced answer than larger VLMs.
Loss computed over the full sequence (question + answer) rather than answer-only, so the model was also lightly trained to reproduce the (fixed) question text — this doesn't appear to have hurt convergence here but is a simplification versus masked/answer-only supervision.
Coverage limited to the crops and disease classes present in w4ashabii/nepali_crop_data; performance on out-of-distribution crops, lighting conditions, or camera angles is untested.
Not evaluated for robustness to adversarial or low-quality images (blur, occlusion, multiple crops in frame, etc.).

Training Framework

Fine-tuned with 🤗 transformers.Trainer and LoRA adapters loaded/attached via Unsloth's FastVisionModel (use_gradient_checkpointing="unsloth") for reduced VRAM use and faster steps on a single T4 GPU.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

w4ashabii

Model Tree

Base

HuggingFaceTB/SmolVLM-256M-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table

Base model	HuggingFaceTB/SmolVLM-256M-Instruct (~256M params)
Fine-tuning method	LoRA (adapters only, base weights frozen)
Trainable params	3,769,344 / 260,254,272 (1.45%)
Framework	🤗 `transformers` `Trainer` + Unsloth `FastVisionModel`
LoRA targets	Attention + MLP modules across both vision encoder and language model
LoRA config	r=8, alpha=16, dropout=0.05, bias="none"
Precision	fp16
Hardware	1x NVIDIA T4 (Google Colab)
Language(s)	English (Q&A text); images of Nepali-grown crops
License	Apache 2.0 (inherited from base model)

Training Data

Dataset: w4ashabii/nepali_crop_data
Format: ShareGPT-style conversations (sharegpt.jsonl), one user turn (image + question) → one assistant turn (crop + disease answer)
Train examples used: 39,987
Eval examples used: 500 (held-out validation split)
Image preprocessing: resized to longest_edge=512, image splitting disabled (not needed for single-leaf/crop photos)

Training Procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Epochs	2
Per-device batch size	8
Gradient accumulation steps	2
Effective batch size	16
Learning rate	1e-4
LR schedule	cosine, 3% warmup
Optimizer	AdamW (`Trainer` default)
Total steps	5,000
Total training time

Training / Validation Loss

Table with columns: Step, Training Loss, Validation Loss
Step	Training Loss	Validation Loss
500	0.1948	0.1918
1000	0.1842	0.1871
1500	0.1878	0.1858
2000	0.1827	0.1865
2500	0.1685	0.1856
3000	0.1809	0.1841

Usage

python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

BASE_MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
ADAPTER_REPO = "w4ashabii/SmolVLM256M_CropDisease"

processor = AutoProcessor.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForVision2Seq.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).to("cuda")
model.eval()

image = Image.open("your_crop_photo.jpg").convert("RGB")
question = "What crop is shown in this image, and does it have any disease? If so, name the disease."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=64)

answer = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer.strip())

With Unsloth (faster inference load)

python
from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained("w4ashabii/SmolVLM256M_CropDisease")
FastVisionModel.for_inference(model)
# ... same generate() call as above

Intended Use

Assistive identification of crop type and visible disease symptoms from photos, for agricultural extension, research, or educational tooling focused on Nepali-grown crops.
Not intended as a sole basis for treatment or pesticide decisions — outputs should be verified by an agronomist or local agricultural extension service, especially for high-stakes crop management decisions.

Limitations

Small base model (256M params) and LoRA-only tuning trade some accuracy for speed/size; expect a lighter-weight, less nuanced answer than larger VLMs.
Loss computed over the full sequence (question + answer) rather than answer-only, so the model was also lightly trained to reproduce the (fixed) question text — this doesn't appear to have hurt convergence here but is a simplification versus masked/answer-only supervision.
Coverage limited to the crops and disease classes present in w4ashabii/nepali_crop_data; performance on out-of-distribution crops, lighting conditions, or camera angles is untested.
Not evaluated for robustness to adversarial or low-quality images (blur, occlusion, multiple crops in frame, etc.).