GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
manifesta
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
manifesta
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
google/gemma-4-31B-it, 10,956,800 trainable parameters, 43.9 MB.google/gemma-4-31B-it is ungated and Apache-2.0.The base string in adapter_config.json is google/gemma-4-31B-it, which resolves, is ungated, and is Apache-2.0 licensed. No workaround needed here. (The two chart adapters in this project both shipped broken base strings, so this is worth stating explicitly.)
Gemma 4 is a conditional-generation checkpoint with a vision tower, so use the image-text-to-text auto class even when you only want text.
python
import torchfrom transformers import AutoModelForImageTextToText, AutoProcessorfrom peft import PeftModelBASE = "google/gemma-4-31B-it"ADAPTER = "manifesta/adaption_verified_math_code_instruct"processor = AutoProcessor.from_pretrained(BASE)base = AutoModelForImageTextToText.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, ADAPTER)model.eval()messages = [{"role": "user","content": [{"type": "text", "text":"A factory makes 1,440 pencils in 8 hours. Two machines are down, cutting output ""by 40%. How many pencils are made in a 6 hour shift? Show your work."}],}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True,return_dict=True, return_tensors="pt",).to(model.device)with torch.inference_mode():out = model.generate(**inputs, max_new_tokens=512, do_sample=False)print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Because the base string is correct, the one-line loader works too:
python
from peft import AutoPeftModelForCausalLMmodel = AutoPeftModelForCausalLM.from_pretrained("manifesta/adaption_verified_math_code_instruct")
Optional merge:
python
model = model.merge_and_unload()
Given the evaluation result, the most useful thing to do with this snippet is run the same prompt through the base with and without the adapter attached and compare. That is what the judge did, and the base won.
Read straight out of adapter_model.safetensors and adapter_config.json.
| Property | Value |
|---|---|
| PEFT type | LoRA, task_type: CAUSAL_LM |
Rank r | 8 |
lora_alpha | 8 (scaling factor alpha/r is exactly 1.0) |
lora_dropout | 0.0 |
bias | none |
use_rslora / |
v_proj is missing on exactly ten layers: 5, 11, 17, 23, 29, 35, 41, 47, 53 and 59. Every sixth layer.
Those are precisely the layers listed as full_attention in the base model's layer_types. The other fifty are sliding_attention. Checking the base model's own weight map confirms it: model.layers.5.self_attn contains q_proj, k_proj, o_proj, q_norm and k_norm, and no v_proj.weight at all, while model.layers.4.self_attn does have one.
The reason is in the base config: attention_k_eq_v: true, with num_global_key_value_heads: 4 and global_head_dim: 512 on the global layers. On full-attention layers the key and value projections are shared, so there is no separate v_proj module for PEFT to wrap. It asked for v_proj on all 60 layers and silently got it on 50.
Nothing failed. But it does mean the adapter covers less of the attention stack than target_modules: [q_proj, v_proj] suggests, and the layers it covers least are the global-attention layers, which are the ones carrying long-range context. For multi-step derivations, long-range context is the part that matters most.
Gemma 4 31B is multimodal, with a 27-layer vision tower. This adapter touched none of it. For a text-only math and code task that is the right call, so it is recorded here for completeness rather than as a criticism. It is a different story on the chart QA adapter, where the frozen vision tower is the whole explanation for that run's null result.
Judged by Adaption's AutoScientist evaluator on a held-out split, against the same base model it was trained from. Head-to-head, blind.
| Metric | Base | Adapted | Change |
|---|---|---|---|
| Win rate | 54 | 46 | -8 points |
| Held-out eval loss (first eval) | 2.3657 | ||
| Held-out eval loss (final) | 1.2200 | -1.146 | |
| Training loss | 3.482 (step 1) | 1.238 (step 59) | -2.244 |
Eval loss across the five checkpoints: 2.3657, 1.5618, 1.3206, 1.2417, 1.2200.
Read those two rows together, because they disagree. Loss dropped by 64% and the model still lost the head-to-head by 8 points. A falling loss curve says the adapter learned to reproduce the training distribution. It says nothing about whether reproducing that distribution is an improvement over what the base already did. Here it was not.
This is the single most useful thing in this repo: eval loss is not the headline metric, and if you optimise against it you can ship a regression with a beautiful chart attached.
Two causes, and they compound.
1. The base was already good, so there was no room. Adaptive Data scored the source corpus at 8.0 out of 10 before adaptation, grade B, the highest starting quality of the four datasets built in this project. Adaptation moved it to 8.2, a +2.5% lift, and the grade did not change. Compare the agronomy corpus at 5.0 to 8.4 (+68.0%) and the chart corpus at 6.0 to 7.1 (+18.3%).
Gemma 4 31B is instruction-tuned and already writes competent step by step derivations for exactly this kind of problem. A rank-8 adapter on two projections, roughly 0.035% of parameters, cannot add mathematical capability the base lacks. What it can do is change style and formatting. When the base's existing style is already good, changing it is as likely to hurt as help, and a blind judge comparing two competent derivations will punish any drift toward the training corpus's quirks.
2. Two thirds of the updates were clipped. See below.
3. The global-attention layers got the least adaptation. As covered above, the ten full_attention layers received q_proj only. Those are the layers handling long-range dependency, which is what a multi-step derivation depends on to keep an intermediate result consistent from line four to line twelve.
What would need to change: a lower learning rate or a longer warmup to stop the gradient spikes, a higher rank if the goal is genuinely new capability rather than style transfer, and a harder corpus. Problems the base already solves at 8.0 out of 10 are the wrong training signal. The value should sit in problems it gets wrong.
This run was not stable.
| Gradient norm statistic | Value |
|---|---|
Clipping threshold (max_grad_norm) | 2 |
| Peak gradient norm | 677.21 (step 38) |
| Median gradient norm | 1.97 |
| Minimum | 0.389 |
| Steps above 1 | 39 of 59 (66%) |
| Steps above the clip threshold of 2 | 29 of 59 (49%) |
| Steps above 10 | 9 of 59 (15%) |
| Steps above 100 | 5 of 59 (8%) |
Largest spikes: step 38 at 677.2, step 15 at 645.0, step 40 at 389.4, step 14 at 172.5, step 26 at 138.6. The final step of the run, step 59, was still at 40.1.
What clipping at 2 does when the true norm is 677 is keep the direction and throw away the magnitude. The update becomes a fixed-size step along whatever direction that batch happened to point in, and the relative weighting between batches is gone. When that happens on half the steps of a 59-step run, the optimizer is not really following the loss surface any more, it is taking a sequence of equal-length steps in noisy directions.
The loss still came down, which is worth sitting with. A falling loss curve does not certify a healthy run.
For contrast, the chart QA adapter trained on the same platform with the same clip threshold peaked at 0.72 and clipped on zero of its 34 steps. Same settings, completely different numerical behaviour, which points at the data and the base rather than the configuration.
(The Adaption run dashboard reports the peak as 744.9. The 677.21 above is the maximum in trainer_state.json in this repo, which is the number you can verify yourself. Both are two to three orders of magnitude over the threshold and the conclusion is the same.)
Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.
| Run | Base | Params | Rows ingested | Steps | Rows per step | Win rate (adapted vs base) |
|---|---|---|---|---|---|---|
| Chart QA, first build | Qwen3.5 9B | 9B | 6,976 | 21 | 332 | 51 vs 49 |
| Chart QA 17k | Gemma 3 27B | 27B | 17,070 | 34 | 502 |
The bigger the base, the smaller the win. 9B scored 51, 27B scored 50, 31B scored 46. This model sits at the bottom of that trend and it is the only one that went backwards.
Scaling the corpus did nothing. The chart corpus was rerun at 2.4x the rows, 6,976 to 17,070, and the win rate went 51 to 50. Because batch_size is set to max, extra rows widen the batch rather than adding updates: step count went 21 to 34 while rows per step went 332 to 502. More rows per step is a smoother gradient estimate for the same small number of updates, not more learning.
Caveat, stated plainly: three data points, three different base models, three different corpora, three different domains. Base size, base family and task difficulty all move together, so nothing here isolates a cause. It is not a controlled experiment and should not be read as one. Testing it properly means holding the corpus fixed and varying only the base.
Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.
| Hyperparameter | Value |
|---|---|
| Base model | google/gemma-4-31B-it (resolves correctly) |
| Method | SFT, LoRA (PEFT), chat format |
| LoRA rank / alpha / dropout | 8 / 8 / 0.0 |
| Trainable modules | q_proj, v_proj (requested on 60 layers, attached on 60 and 50) |
| Learning rate | 1e-4 |
| Scheduler | cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1 |
| Weight decay |
Use it for
trainer_state.json.Do not use it for
google/gemma-4-31B-it beat it 54 to 46 on the held-out split. Use the base.Technical limits
v_proj is absent on the ten global-attention layers. Not a defect, but the coverage is thinner than the config implies.Trained on Adaption dataset 6ad1a83f-b806-4a20-ba6d-239029d30711, built from manifesta/verified-math-code-17k (also on Kaggle).
Adaptive Data lifted the corpus quality before training:
| Adaptive Data metric | Before | After |
|---|---|---|
| Quality score | 8.0 | 8.2 (+2.5%) |
| Grade | B | B |
| Percentile | 15.3 | 17.8 |
That +2.5% is the smallest lift of the four datasets built in this project, and it is the clue. The corpus started at 8.0, so there was very little for adaptation to fix, and by extension very little for the model to learn that it did not already know. +2.5% is an improvement in the data, not in the model. The model's own number went the other way.
Show Your Work runs at manifesta.adaptionlabs.app. Give it a problem, get a step by step derivation with a verification panel, and use the "Change a number" button to check that the derivation actually recomputes rather than pattern matching to a memorised answer.
manifesta/adaption_scientific_chart_qa_17k, scientific chart QA on Gemma 3 27B. 50 vs 50, a null result, because the LoRA covered zero vision tensors while the task is entirely visual. Its declared base string 404s.manifesta/scientific-chart-qa-lora-qwen3.5-9b, the first chart build on a 9B base. 51 vs 49, 21 optimizer steps, LoRA on only 8 of 32 decoder layers and again zero vision tensors. Its declared base togethercomputer/Qwen3.5-9B also 404s.manifesta/brandvoice-marketing-model, the Part 1 marketing adapter that did work, 56% win rate on Llama 3.3 70B. Style transfer on a base with a weak default style, which is the case where a small adapter has room to move.google/gemma-4-31B-it, Apache-2.0 and ungated. No access request needed.manifesta/verified-math-code-17k, CC0-1.0.bibtex
@misc{showyourwork_mathcode_2026,title = {Show Your Work: a verified math and code adapter for Gemma 4 31B, and why it regressed},author = {Navardauskas, Aivaras},year = {2026},note = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Math and Code. Win rate 46 vs 54. Half the optimizer steps clipped.},howpublished = {\url{https://huggingface.co/manifesta/adaption_verified_math_code_instruct}}}
Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.
use_dora| false / false |
| Target modules | q_proj, v_proj |
exclude_modules | empty |
| Tensors in the file | 220 |
| Decoder layers touched | 60 of 60 for q_proj, 50 of 60 for v_proj |
| Vision tensors | 0 |
| Trainable parameters | 10,956,800 (float32, 43.9 MB) |
| Share of the 31B base | roughly 0.035% |
| 50 vs 50 |
| Verified math and code (this model) | Gemma 4 31B | 31B | 17,586 | 59 | 298 | 46 vs 54 |
| 0 |
| Max gradient norm | 2 |
| Epochs | 1 |
| Batch size | max (per-device train batch size 1 with accumulation) |
| Optimizer steps | 59 |
| Evaluations | 5, every 11 steps |
| Train on inputs | false |
| Precision | bfloat16 base, float32 adapter weights |
| Total FLOPs | 4.073e18 |
| Adaption job ID | 9f46e2b9-ac73-4b97-ad13-cc6ccd1501cf |
| Training experiment ID | 3ce14a29-6d5d-48ea-849f-372a6102c360 |
| Adaption dataset ID | 6ad1a83f-b806-4a20-ba6d-239029d30711 |