GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
aihpi
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
aihpi
Model Tree
Qwen/Qwen2-VL-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
The model takes an after-meal photo and the served ingredients with their weights as text. It answers with the returned weight per ingredient in grams, as JSON.
Prompt, in the exact format used in training (the ingredient JSON is json.dumps(..., indent=2)):
text
Given this initial meal data in JSON format: ```json{ "ingredients": [ { "name": "Kartoffelpüree ", "weight": 120 }, { "name": "Grüne Bohnen ", "weight": 80 } ], "total_weight": 200}``` Please provide the amount of food returned for each ingredient in JSON format.Answer:
text
Here are the ingredients with their initial weights and returned amounts: ```json{ "ingredients": [ { "name": "Kartoffelpüree ", "initial_weight": 120, "returned_weight": 0.0 }, { "name": "Grüne Bohnen ", "initial_weight": 80, "returned_weight": 26.0 } ], "total_initial_weight": 200}```python
import jsonfrom PIL import Imagefrom transformers import AutoProcessor, Qwen2VLForConditionalGeneration model_id = "aihpi/food-waste-vlm"model = Qwen2VLForConditionalGeneration.from_pretrained(model_id, torch_dtype="auto", device_map="auto")# Training used at most 262144 pixels per image; match it at inference.processor = AutoProcessor.from_pretrained(model_id, max_pixels=262144) served = {"ingredients": [{"name": "Kartoffelpüree ", "weight": 120}, {"name": "Grüne Bohnen ", "weight": 80}], "total_weight": 200}prompt = ("Given this initial meal data in JSON format:\n\n```json\n" + json.dumps(served, indent=2) + "\n```\n\nPlease provide the amount of food returned for each ingredient in JSON format.") messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[Image.open("plate.jpg")], return_tensors="pt").to(model.device) output = model.generate(**inputs, max_new_tokens=512)print(processor.batch_decode(output[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])Trained with LLaMA-Factory (config: qwen2vl_food_waste.yaml).
| Setting | Value |
|---|---|
| Data | Train split: 215 images from 12 receipts (dataset revision 7f75954a), 10% held out for validation |
| Method | LoRA, rank 8, all linear layers |
| Epochs | 3 |
| Learning rate | 1e-4, cosine schedule, 10% warm-up |
| Batch size | 1 per device × 4 gradient accumulation |
| Precision | bf16 |
| Max sequence length | 2048 tokens |
| Max image size | 262144 pixels |
The model has not yet been evaluated on held-out data. The dataset's test split (160 images from 4 receipts) was published after this model was trained, and none of its receipts are in the train split, so it is the intended benchmark.
This model is licensed under Apache 2.0. It is a modified version of Qwen/Qwen2-VL-7B-Instruct, which is also licensed under Apache 2.0.
It was trained on AI-ServicesBB/food-waste-dataset by the AI Service Centre Berlin-Brandenburg, Hasso-Plattner-Institut für Digital Engineering gGmbH, licensed under CC BY 4.0.
The AI Service Centre Berlin-Brandenburg is funded by the German Federal Ministry of Research, Technology and Space under the funding code "KI-Servicezentrum Berlin-Brandenburg" 16IS22092.
Given this initial meal data in JSON format:```json{ "ingredients": [ { "name": "Kartoffelpüree ", "weight": 120 }, { "name": "Grüne Bohnen ", "weight": 80 } ], "total_weight": 200}```Please provide the amount of food returned for each ingredient in JSON format.Here are the ingredients with their initial weights and returned amounts:```json{ "ingredients": [ { "name": "Kartoffelpüree ", "initial_weight": 120, "returned_weight": 0.0 }, { "name": "Grüne Bohnen ", "initial_weight": 80, "returned_weight": 26.0 } ], "total_initial_weight": 200}```import jsonfrom PIL import Imagefrom transformers import AutoProcessor, Qwen2VLForConditionalGeneration model_id = "aihpi/food-waste-vlm"model = Qwen2VLForConditionalGeneration.from_pretrained(model_id, torch_dtype="auto", device_map="auto")# Training used at most 262144 pixels per image; match it at inference.processor = AutoProcessor.from_pretrained(model_id, max_pixels=262144) served = {"ingredients": [{"name": "Kartoffelpüree ", "weight": 120}, {"name": "Grüne Bohnen ", "weight": 80}], "total_weight": 200}prompt = ("Given this initial meal data in JSON format:\n\n```json\n" + json.dumps(served, indent=2) + "\n```\n\nPlease provide the amount of food returned for each ingredient in JSON format.") messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[Image.open("plate.jpg")], return_tensors="pt").to(model.device) output = model.generate(**inputs, max_new_tokens=512)print(processor.batch_decode(output[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])