rolandwhere

granite-4.2-8b-symbolic-cot

Deploy Dedicated

README

License: apache-2.0

Model Details

Developed by: rolandwhere
Model type: Dense decoder-only transformer (GraniteForCausalLM), 8B parameters
Language: English (base model supports 12; this fine-tune was trained on English only)
License: Apache 2.0
Finetuned from: ibm-granite/granite-4.2-8b

Model Sources

LoRA adapter: rolandwhere/granite-4.2-8b-symbolic-cot-lora
Training data: rolandwhere/granite-4.2-8b-symbolic-cot-traces

Uses

Direct Use

Latency- or cost-sensitive reasoning where the chain of thought is an internal cost rather than a user-facing product: batch processing, agent loops with many turns, long-context work where thinking tokens compete with the context budget.

The notation is deliberately terse. It uses =, -> (implies), <- (because), .: (therefore), &, |, !, and single-letter variable bindings. It is meant to be machine-cheap, not human-pleasant.

Out-of-Scope Use

Anywhere the reasoning trace is shown to users or audited. Compressed traces are harder to read and harder to check. If you need interpretable reasoning, use the base model.
Non-English prompts. Training data was English-only; multilingual behaviour is untested and likely degraded relative to base.
Tool calling and agentic workflows. The base model was RL-trained for these; this fine-tune did not include agentic traces and their behaviour after training is unmeasured.
Long-horizon reasoning. Compression was tuned on short problems (≤80 thinking tokens). Problems needing genuinely long deliberation may be hurt by a prior toward brevity.

Bias, Risks, and Limitations

Compression trades against correctness. A model trained to think briefly will sometimes think too briefly. See Evaluation for the measured trade-off, and read the caveats there before quoting the numbers.
The base model's alignment may be affected. The LoRA touched all attention and MLP projections across all 40 layers. Safety behaviour was not re-evaluated after training. IBM recommends pairing Granite with Granite Guardian; that recommendation carries over.
Non-thinking and low-effort modes shifted too. The training targeted enable_thinking=True, but the weights are shared, so all three modes moved. Behaviour in the other two modes is not what IBM shipped.
Teacher-model artifacts. Traces were distilled from MiniMax M3/M2.7. Where the teacher reasoned badly, the notation faithfully encodes bad reasoning.
Inherits all limitations of the base model.

How to Get Started with the Model

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "rolandwhere/granite-4.2-8b-symbolic-cot"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.float16, device_map="auto")

msgs = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                               enable_thinking=True)
ins = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ins, max_new_tokens=1024, do_sample=True,
                     temperature=1.0, top_p=0.95)
print(tok.decode(out[0][ins.input_ids.shape[-1]:], skip_special_tokens=False))

Use temperature=1.0, top_p=0.95 as IBM recommends for the base model. The compressed trace appears inside <think>...</think>; the answer follows it.

Note on system prompts: training used an empty system turn on every example. Serving with a long system prompt is a distribution shift and may weaken the compression. If you need one, test it — or fine-tune further with your system prompt in place.

Training Details

Training Data

Roughly N synthetic reasoning traces (see the dataset repo for the exact row count), distilled via OpenRouter's free auto-router. Per the OpenRouter activity log, generation landed roughly 50/50 across MiniMax M3 and M2.7. The teacher was used only to produce training traces — it plays no part in the model at inference.

Because the router selects models dynamically, the exact data-generation run is not bit-reproducible. The notation spec and filter criteria below are what actually determine the output.

Two seed sources:

GSM8K train — grade-school math, ~77% of the set. These carry gold answers, so traces whose final answer disagreed with the reference were discarded automatically.
HuggingFaceH4/no_robots — open QA, brainstorming, classification, extraction. No gold answers; kept for style coverage so the model doesn't collapse into math-only behaviour.

The teacher was prompted to emit two fields: a compressed symbolic think and a normal-English answer. Only the reasoning is compressed.

Preprocessing

Traces were filtered on four criteria before training:

Trace ≤ 80 tokens under the Granite tokenizer
Terminal .: conclusion line present
No natural-language filler (let me, okay, wait, we need, hmm, …)
For math seeds, the gold answer appears in the reply

Prompts were rendered with the base chat template at enable_thinking=True, so training overwrites full-thinking behaviour specifically. Loss was masked to the completion only.

Training Hyperparameters

Table

Method	QLoRA (4-bit NF4 base, fp16 compute)
LoRA rank / alpha	32 / 32
Target modules	`q,k,v,o,gate,up,down_proj`, all 40 layers
Trainable params	98,959,360 (1.11%)
Epochs	2
Effective batch	16 (2 × 8 accumulation)
LR / schedule	2e-4, linear, 5 warmup steps
Optimizer	adamw_8bit

Training loss fell from 1.63 to ~0.60 over the run.

Speeds, Sizes, Times

Single NVIDIA T4 (16GB), ~15 minutes. Adapter is 396 MB; merged weights are ~18 GB in fp16.

Evaluation

Testing Data, Factors & Metrics

40 held-out questions from GSM8K test. Exact-match on the final numeric answer. Four conditions on identical hardware and quantization, with the base conditions obtained by disabling the adapter on the same loaded model. Sampling at temperature=1.0, top_p=0.95, generation capped at 768 new tokens.

Metrics: mean thinking tokens (the quantity being optimized), wall-clock seconds per example, and accuracy.

Results

Table with columns: Condition, Thinking tokens, sec / example, Accuracy
Condition	Thinking tokens	sec / example	Accuracy
base, thinking	176	25.7	0.48 *
base, low-effort	61	13.9	0.93
base, non-thinking	0	23.0 *	0.65
this model, thinking	44	5.5

* These two numbers are measurement artifacts, not properties of the base model. Read the summary before using this table.

Summary

The compression result is solid: 4× fewer thinking tokens (176 → 44) and 4.7× lower latency (25.7s → 5.5s) versus base thinking mode.

The accuracy column is not trustworthy as measured, and I would rather say so than let the numbers flatter this model:

Base thinking mode at 0.48 is wrong. The 768-token generation cap truncated the base model's longer chains before they reached an answer, scoring them as failures. IBM's own card recommends 8192 tokens for thinking mode. The true base figure is substantially higher, and this model very likely does not beat it on accuracy.
Base non-thinking at 23 seconds is impossible for a condition producing no reasoning, indicating a harness bug in that row.
Base low-effort at 0.93 is the meaningful comparison and it is the strongest condition tested. IBM's built-in low-effort mode achieves most of the compression benefit with better accuracy than this fine-tune. If you want short reasoning from Granite 4.2, try low_effort=True first — it is free and requires no fine-tuned weights.

The honest claim for this model is a 4× reduction in thinking tokens at an accuracy cost that has not yet been properly quantified. A re-run with a 2048+ token budget is needed before the accuracy column means anything. Treat this release as a working demonstration of the compression technique, not as a validated accuracy claim.

Environmental Impact

Hardware: 1 × NVIDIA T4
Hours used: ~1 (including dataset generation and evaluation)
Cloud Provider: Kaggle
Carbon Emitted: negligible at this scale; estimable via the ML Impact calculator (Lacoste et al., 2019)

Technical Specifications

Model Architecture and Objective

Dense decoder-only transformer. 40 layers, hidden size 4096, GQA with 32 query heads and 8 KV heads, SwiGLU MLP (hidden 12800), RMSNorm, RoPE. Standard causal language modelling objective, loss masked to the assistant completion.

Compute Infrastructure

Kaggle notebook, 1 × T4 (16 GB), fp16 compute with a 4-bit NF4 quantized base. Unsloth for training; PEFT and llama.cpp for merging and GGUF conversion.

Citation

If you use this model, cite the base model:

bibtex
@misc{granite2026,
  title  = {Granite 4.2 Language Models},
  author = {{Granite Team, IBM}},
  year   = {2026},
  url    = {https://huggingface.co/ibm-granite/granite-4.2-8b}
}

Glossary

Notation used inside <think>:

Table with columns: Symbol, Meaning
Symbol	Meaning
`a=48`	bind quantity to a variable
`->`	implies / leads to
`<-`	because
`.:`	therefore (terminal line)
`&` `\|` `!`	and / or / not

Model Card Authors

rolandwhere

Contact

Questions, results, or corrections are welcome — open a discussion on this repo, or reach out via LinkedIn.

If you re-run the evaluation with a larger token budget, I would genuinely like to see the numbers.

Acknowledgements

Built on IBM's Granite 4.2 (Apache 2.0) and trained with Unsloth on free Kaggle T4 hardware.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

rolandwhere

Model Tree

Base

ibm-granite/granite-4.2-8b

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Developed by: rolandwhere
Model type: Dense decoder-only transformer (GraniteForCausalLM), 8B parameters
Language: English (base model supports 12; this fine-tune was trained on English only)
License: Apache 2.0
Finetuned from: ibm-granite/granite-4.2-8b

Model Sources

LoRA adapter: rolandwhere/granite-4.2-8b-symbolic-cot-lora
Training data: rolandwhere/granite-4.2-8b-symbolic-cot-traces

Uses

Direct Use

Out-of-Scope Use

Anywhere the reasoning trace is shown to users or audited. Compressed traces are harder to read and harder to check. If you need interpretable reasoning, use the base model.
Non-English prompts. Training data was English-only; multilingual behaviour is untested and likely degraded relative to base.
Tool calling and agentic workflows. The base model was RL-trained for these; this fine-tune did not include agentic traces and their behaviour after training is unmeasured.
Long-horizon reasoning. Compression was tuned on short problems (≤80 thinking tokens). Problems needing genuinely long deliberation may be hurt by a prior toward brevity.

Bias, Risks, and Limitations

Compression trades against correctness. A model trained to think briefly will sometimes think too briefly. See Evaluation for the measured trade-off, and read the caveats there before quoting the numbers.
The base model's alignment may be affected. The LoRA touched all attention and MLP projections across all 40 layers. Safety behaviour was not re-evaluated after training. IBM recommends pairing Granite with Granite Guardian; that recommendation carries over.
Non-thinking and low-effort modes shifted too. The training targeted enable_thinking=True, but the weights are shared, so all three modes moved. Behaviour in the other two modes is not what IBM shipped.
Teacher-model artifacts. Traces were distilled from MiniMax M3/M2.7. Where the teacher reasoned badly, the notation faithfully encodes bad reasoning.
Inherits all limitations of the base model.

How to Get Started with the Model

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "rolandwhere/granite-4.2-8b-symbolic-cot"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.float16, device_map="auto")

msgs = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                               enable_thinking=True)
ins = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ins, max_new_tokens=1024, do_sample=True,
                     temperature=1.0, top_p=0.95)
print(tok.decode(out[0][ins.input_ids.shape[-1]:], skip_special_tokens=False))

Use temperature=1.0, top_p=0.95 as IBM recommends for the base model. The compressed trace appears inside <think>...</think>; the answer follows it.

Training Details

Training Data

Because the router selects models dynamically, the exact data-generation run is not bit-reproducible. The notation spec and filter criteria below are what actually determine the output.

Two seed sources:

GSM8K train — grade-school math, ~77% of the set. These carry gold answers, so traces whose final answer disagreed with the reference were discarded automatically.
HuggingFaceH4/no_robots — open QA, brainstorming, classification, extraction. No gold answers; kept for style coverage so the model doesn't collapse into math-only behaviour.

The teacher was prompted to emit two fields: a compressed symbolic think and a normal-English answer. Only the reasoning is compressed.

Preprocessing

Traces were filtered on four criteria before training:

Trace ≤ 80 tokens under the Granite tokenizer
Terminal .: conclusion line present
No natural-language filler (let me, okay, wait, we need, hmm, …)
For math seeds, the gold answer appears in the reply

Prompts were rendered with the base chat template at enable_thinking=True, so training overwrites full-thinking behaviour specifically. Loss was masked to the completion only.

Training Hyperparameters

Table

Method	QLoRA (4-bit NF4 base, fp16 compute)
LoRA rank / alpha	32 / 32
Target modules	`q,k,v,o,gate,up,down_proj`, all 40 layers
Trainable params	98,959,360 (1.11%)
Epochs	2
Effective batch	16 (2 × 8 accumulation)
LR / schedule	2e-4, linear, 5 warmup steps
Optimizer	adamw_8bit

Training loss fell from 1.63 to ~0.60 over the run.

Speeds, Sizes, Times

Single NVIDIA T4 (16GB), ~15 minutes. Adapter is 396 MB; merged weights are ~18 GB in fp16.

Evaluation

Testing Data, Factors & Metrics

Metrics: mean thinking tokens (the quantity being optimized), wall-clock seconds per example, and accuracy.

Results

Table with columns: Condition, Thinking tokens, sec / example, Accuracy
Condition	Thinking tokens	sec / example	Accuracy
base, thinking	176	25.7	0.48 *
base, low-effort	61	13.9	0.93
base, non-thinking	0	23.0 *	0.65
this model, thinking	44	5.5

* These two numbers are measurement artifacts, not properties of the base model. Read the summary before using this table.

Summary

The compression result is solid: 4× fewer thinking tokens (176 → 44) and 4.7× lower latency (25.7s → 5.5s) versus base thinking mode.

The accuracy column is not trustworthy as measured, and I would rather say so than let the numbers flatter this model:

Base thinking mode at 0.48 is wrong. The 768-token generation cap truncated the base model's longer chains before they reached an answer, scoring them as failures. IBM's own card recommends 8192 tokens for thinking mode. The true base figure is substantially higher, and this model very likely does not beat it on accuracy.
Base non-thinking at 23 seconds is impossible for a condition producing no reasoning, indicating a harness bug in that row.
Base low-effort at 0.93 is the meaningful comparison and it is the strongest condition tested. IBM's built-in low-effort mode achieves most of the compression benefit with better accuracy than this fine-tune. If you want short reasoning from Granite 4.2, try low_effort=True first — it is free and requires no fine-tuned weights.

Environmental Impact

Hardware: 1 × NVIDIA T4
Hours used: ~1 (including dataset generation and evaluation)
Cloud Provider: Kaggle
Carbon Emitted: negligible at this scale; estimable via the ML Impact calculator (Lacoste et al., 2019)

Technical Specifications

Model Architecture and Objective

Compute Infrastructure

Kaggle notebook, 1 × T4 (16 GB), fp16 compute with a 4-bit NF4 quantized base. Unsloth for training; PEFT and llama.cpp for merging and GGUF conversion.

Citation

If you use this model, cite the base model:

bibtex
@misc{granite2026,
  title  = {Granite 4.2 Language Models},
  author = {{Granite Team, IBM}},
  year   = {2026},
  url    = {https://huggingface.co/ibm-granite/granite-4.2-8b}
}

Glossary

Notation used inside <think>:

Table with columns: Symbol, Meaning
Symbol	Meaning
`a=48`	bind quantity to a variable
`->`	implies / leads to
`<-`	because
`.:`	therefore (terminal line)
`&` `\|` `!`	and / or / not

Model Card Authors

rolandwhere

Contact

Questions, results, or corrections are welcome — open a discussion on this repo, or reach out via LinkedIn.

If you re-run the evaluation with a larger token budget, I would genuinely like to see the numbers.

Acknowledgements

Built on IBM's Granite 4.2 (Apache 2.0) and trained with Unsloth on free Kaggle T4 hardware.