GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nberkc16
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nberkc16
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
Four fixed sections, about 250–350 words plus a table:
text
**Kernaussagen <Monat Jahr>** (at most 3 points)**Abweichungen im Detail** (Position | Ist | Vergleich | Abweichung | Erklärung | Quelle oder Hypothese)**Hinweise auf mögliche Verzerrungen****Offene Fragen an Buchhaltung / Steuerberater**Example input and output for one test month (Muster GmbH, July 2025, a sector never seen in training):
example_A1-v00_2025-07_messages.json → example_A1-v00_2025-07_output.md.
The model expects exactly the prompt it was trained on (prompt version 1.1):
FACTS: the month as JSON, computed by code: actual, previous month, same month of the previous year,
cumulative, deviations, ratios, significance and distortion flags;WISSEN: general explanations of BWA lines and a guide to typical distortions (background only, not a source);KONTEXTNOTIZEN: the company profile plus the top-8 retrieved company notes, each with a source ID,
month and duration (dauerhaft / temporär).See example_A1-v00_2025-07_messages.json for a complete message pair. Outside this format (other
languages, free questions, other reports) the behaviour is untested.
llama-server)This is the path used for the evaluation below.
gemma4:e4b-it-q4_K_M. After ollama pull gemma4:e4b-it-q4_K_M, it is
in Ollama's models/blobs/ folder as sha256-370c2879f17648d337b0ab4e208fdce348e24d1060064e93d7645daafd56d39c
(5.5 GB). Other GGUF conversions of google/gemma-4-E4B-it should work but were not tested.adapter_rag.gguf from this repository (Files and versions).llama-server build with Gemma 4 support. We used the one bundled with Ollama 0.40.10
(Linux: /usr/local/lib/ollama/llama-server).bash
llama-server --model <base>.gguf --lora adapter_rag.gguf -c 12288 -ngl 999 --port 8080python request_llama_server.py example_A1-v00_2025-07_prompt.txt > kommentar.mdexample_..._prompt.txt is the Gemma 4 chat template already applied: system and user turn, generation
prompt <|turn>model\n, thinking off. It has no leading <bos>, because the server adds it.google/gemma-4-E4B-it with
enable_thinking=False and remove the leading <bos>.-ngl) and reduce -c; prompts are 4–6k tokens.adapter_model.safetensors and adapter_config.json are the same adapter in PEFT format (trained with Unsloth on
unsloth/gemma-4-E4B-it-unsloth-bnb-4bit). They are provided for reference and further training; only the GGUF
path above was evaluated.
| Method | QLoRA (4-bit base), Unsloth 2026.10.8 + TRL 1.15.0, loss on the answer only |
| LoRA | r 16, alpha 16, dropout 0, attention and MLP of the language layers (34.9 M parameters); vision and audio untouched |
| Schedule | 1 epoch, LR 2e-4 linear, warmup 5 %, batch 1 × gradient accumulation 8 (113 steps), AdamW 8-bit, max length 7168 tokens |
| Data | 898 synthetic company-months: wholesale (Sanitär/Heizung) and garden-furniture manufacturer, 20 variants each |
| Targets | rule-based content plan from the generator's ground truth (what to say, which note explains what) + German wording by a teacher model (Mistral Large 3) + automatic checks; rejected texts were regenerated or dropped |
| Validation loss | 0.834 → 0.130 (32 held-out examples) |
| Compute | one Kaggle T4, 7.8 h |
The training targets never contain the true cause of an event unless a visible note states it; otherwise they ask an open question or give a marked hypothesis.
720 test months: 240 from a new sector never seen in training (metal-working supplier) and 480 from new companies of the trained sectors. Same base model file, prompt and sampling for all rows; only the adapter differs.
| Metric | Base model | Base + RAG | Fine-tuned, no RAG ¹ | This model (RAG + FT) |
|---|---|---|---|---|
| Output format correct | 39 % | 30 % | 100 % | 97 % |
| Texts without validation error | 18 % | 4 % | 93 % | 82 % |
| Numbers matching the facts | 95 % | 94 % | 100 % | 99 % |
| Unmarked causes per text (neither source nor hypothesis) | 0.16 | 1.01 | 0 | 0.06 |
| Cited notes that really explain a deviation (precision) | – | 54 % | – | 75 % |
| Expected notes cited (recall) | – | 34 % | – | 59 % |
| Citations on the right table row | – | 37 % | – | 64 % |
| Distortion warnings: recall / precision (ground truth) | 66 % / 86 % | 64 % / 85 % | 84 % / 88 % | 82 % / 87 % |
¹ A separate adapter trained without context notes (not published here).
Apache 2.0, like the base model Gemma 4. Training targets were written with Mistral Large 3 via the Mistral API.
| File | Content |
|---|---|
adapter_rag.gguf | LoRA adapter for llama.cpp (tested) |
adapter_model.safetensors, adapter_config.json | same adapter, PEFT format |
example_A1-v00_2025-07_messages.json | input of one test month (system + user message) |
example_A1-v00_2025-07_prompt.txt | the same input with the chat template applied |
example_A1-v00_2025-07_output.md | the model's output for it (from the evaluation run) |
request_llama_server.py | sends a prompt file to llama-server with the evaluation settings |
REGU AI is a pre-seed startup in Munich. We help European companies move their AI off the cloud and onto infrastructure they own: hardware for running AI locally, digital products that integrate with it, and LLMs customised and trained for a company's own work.
This model is an example of the last part. It is small enough to run on a single GPU that a company can host itself (in the evaluation, each server ran on one 16 GB T4). We publish it to show our approach: numbers come from code, context from retrieval, wording from a small fine-tuned model, and a validation layer checks every draft before a human approves it.
Website: https://www.reguai.tech
**Kernaussagen <Monat Jahr>** (at most 3 points)**Abweichungen im Detail** (Position | Ist | Vergleich | Abweichung | Erklärung | Quelle oder Hypothese)**Hinweise auf mögliche Verzerrungen****Offene Fragen an Buchhaltung / Steuerberater**llama-server --model <base>.gguf --lora adapter_rag.gguf -c 12288 -ngl 999 --port 8080python request_llama_server.py example_A1-v00_2025-07_prompt.txt > kommentar.md