Training
The model was trained in Tinker Decision Model mode using a LoRA adapter and a soft-label objective. It used 3,622 training examples, 458 validation examples, and 458 held-out test examples from the same private prepared snapshot as the earlier Cevra GPT-OSS-20B run (snapshot revision 459, grouped splits with seed 42). The data are synthetic variants derived from MATH-500; the underlying dataset remains private.
Training completed 227 steps in one epoch, with LoRA rank 16, batch size 16, learning rate 0.0001, and maximum sequence length 2,048. The final step was selected using validation. The provider did not disclose the base revision used during training. The independently pinned Hugging Face base revision used for merging is 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; the comparisons below show approximate agreement rather than identical weights.
Evaluation
The Tinker sampler, evaluated on all 458 held-out test examples, achieved 69.4323% argmax agreement, 0.058251 distribution Brier, and 0.041392 probability MAE against the prepared target distributions. These are agreement metrics, not independently established accuracy or calibration. Validation was used to select the checkpoint; the held-out examples were not used for checkpoint selection. Both splits are derived from MATH-500 and do not constitute an independent MATH-500 benchmark.
The local merged model and the pinned Hugging Face base were evaluated on those same 458 held-out examples:
Table with columns: Metric, HF base, Merged model| Metric | HF base | Merged model |
|---|
| Argmax agreement (higher is better) | 57.6419% | 69.8690% |
| Distribution Brier (lower is better) | 0.169736 | 0.059772 |
| Probability MAE (lower is better) | 0.076854 | 0.042011 |
For the first 10 test examples, the merged model's argmax matched the Tinker trained sampler on 10/10; the maximum absolute probability difference was 0.0540073. On 10 independently invented synthetic probes, the Hugging Face base and Tinker base argmax matched on 9/10, with maximum absolute probability difference 0.0491300. These are small diagnostic samples, and the observed disagreement means the base models must not be assumed identical.
The original GPU evaluation failed its pre-specified 0.05 trained-model difference threshold. The user subsequently accepted a 0.1 threshold for both comparisons; the saved GPU predictions were reassessed locally under that post-hoc threshold without rerunning the GPU. The original failure was retained. The threshold change and small reference sample limit the strength of the equivalence evidence.
Decision-token scoring example
The following example uses an invented question, not a dataset example. Its scoring procedure was used in the 458-example evaluation; this particular example has not been executed separately. The evaluation environment used Python 3.12, PyTorch 2.15.0, Transformers 5.22.2, Accelerate 1.15.0 and Tinker Cookbook 0.6.1 with a CUDA GPU. Set model_path to a local merged-model directory or this published repository.
import jsonimport mathimport torchfrom tinker_cookbook import renderersfrom tinker_cookbook.recipes.decisions import Decision, DecisionRendererfrom transformers import AutoModelForImageTextToText, AutoTokenizer model_path = "EugeneEvstafev/cevra-math-decisions-v2"device = "cuda:0"tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=False)model = AutoModelForImageTextToText.from_pretrained( model_path, dtype=torch.bfloat16, device_map={"": device}, trust_remote_code=False,).eval()renderer = DecisionRenderer(renderers.get_renderer("qwen3_5", tokenizer)) example = { "state": "A rectangle has sides 3 cm and 7 cm. What is its area?", "question": { "type": "choice", "instructions": "Which subject best describes the problem?", "criteria": { "Algebra": "Equations, expressions, and variables", "Geometry": "Shapes, distances, areas, and angles", "Number Theory": "Properties of integers", }, },}question = example["question"]text = json.dumps( {"state": example["state"], "instructions": question["instructions"]}, ensure_ascii=False, sort_keys=True,)choices = [(str(k), str(v)) for k, v in question["criteria"].items()]rendered = renderer.render(Decision(text, choices))if rendered.choice_pos != len(rendered.tokens) - 1 or rendered.choice_pos < 1: raise ValueError("Unexpected decision token positions")prompt = torch.tensor([rendered.tokens[:rendered.choice_pos]], dtype=torch.long, device=device)with torch.inference_mode(): logits = model(input_ids=prompt, use_cache=False).logits[0, -1].float()labels = rendered.labelsids = torch.tensor([rendered.label_tokens[k] for k in labels], dtype=torch.long, device=device)probabilities = torch.softmax(logits[ids], dim=0).cpu().tolist()if not all(math.isfinite(p) and 0 <= p <= 1 for p in probabilities) or abs(sum(probabilities) - 1) > 1e-5: raise ValueError("Invalid probability distribution")print(dict(zip(labels, probabilities, strict=True)))
This computes probabilities by selecting the decision labels' next-token logits and normalizing them across the offered labels. Generating JSON from the model is not equivalent. The model was trained on choice examples; no performance claim is made for other decision types.
Limitations
The training targets are synthetic or adjusted distributions, and the scores are not known to be calibrated probabilities. Generated variants can contain mathematical errors. The MATH-500 source card does not state a license; applicable rights for distribution of derivatives require verification.