xtools-at

Ministral-3-3B-Qwen3.8-Max-Reasoning-Distilled

README

License: apache-2.0

✨ Model at a glance

Table with columns: Detail, Value
Detail	Value
Base model	`unsloth/Ministral-3-3B-Reasoning-2512`
Teacher/data source	Qwen 3.8 Max distillation dataset
Training dataset	`saidutta69/qwen3.8-max-distillation-50k-clean`
Training examples	8,000
Evaluation examples	400
Response-length filter	Maximum 6,000 characters
Evaluation split	Randomly sampled
Model size	3B-class
Primary task	Text generation and instruction following

🎯 What is this model for?

This fine-tune explores how much useful behavior from a larger teacher-model dataset can be transferred to a smaller reasoning-oriented model.

Potential use cases include:

💬 General conversational assistance and instruction following
🧩 Reasoning-oriented prompts and multi-step problem solving
✍️ Writing, rewriting, and summarization
🛠️ Experimentation with synthetic-data distillation and compact language models

These are intended use cases, not a guarantee of performance. The model's actual strengths will depend on the training procedure, data quality, and evaluation results.

📚 Training data

The model was fine-tuned using a filtered subset of saidutta69/qwen3.8-max-distillation-50k-clean, a dataset associated with Qwen 3.8 Max distillation.

The dataset was prepared as follows:

Response-length filtering: Examples with responses longer than 6,000 characters were excluded.
Training subset: 8,000 entries were randomly sampled for training.
Evaluation subset: 400 entries were randomly sampled for evaluation.

This gives a total of 8,400 selected examples across the training and evaluation sets.

The character limit helps keep exceptionally long responses out of the fine-tuning set. It is a character-based filter, not a token limit, so the corresponding token lengths can vary.

⚙️ Training approach

The base model is unsloth/Ministral-3-3B-Reasoning-2512, with the Qwen distillation dataset providing the examples used for LoRA fine-tuning over a single epoch.

The intent is to adapt the smaller model to patterns found in the teacher-generated responses while retaining the practical advantages of a compact model.

Table with columns: Parameter, Value
Parameter	Value
Base model	`unsloth/Ministral-3-3B-Reasoning-2512`
Fine-tuning method	LoRA, 16-bit
Training examples	8,000
Evaluation examples	400
Epochs	1
Maximum context length	8,192 tokens
Learning rate	2e-4
LoRA rank (`r`)	16
LoRA alpha	32
LoRA dropout	0.02
Optimizer	AdamW 8-bit
Learning rate scheduler	Linear
Weight decay	0.002
Batch configuration	4 × 8

📊 Training results

Table with columns: Metric, Result
Metric	Result
Training loss	0.513
Evaluation loss	0.517

The evaluation loss ended up lower than the training loss. These values are useful as training diagnostics, but they aren't direct measures of reasoning quality and don't, by themselves, establish an improvement over the original model.

Benchmark comparisons and qualitative evaluations on unseen problems would help determine whether the fine-tune improves reasoning, coding, or mathematical performance.

💬 Usage

Required base Mistral Reasoning system prompt (always set this):

text

# HOW YOU SHOULD THINK AND ANSWER First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input. Your thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user.[/THINK]Here, provide a self-contained response.

You can append additional instructions to the base system prompt.

⚠️ Limitations

Distillation is not a guarantee of equivalence. A 3B-class model will not necessarily reproduce the teacher's capabilities or outputs.
Synthetic-data limitations carry over. Errors, biases, and stylistic quirks in the source responses may be learned by the fine-tune.
Reasoning can be unreliable. Fluent explanations may still contain incorrect conclusions or unsupported claims.
Evaluation coverage is limited. A 400-example random sample may not adequately represent every task or domain.
Language coverage is unspecified. English is listed as the intended primary language; broader language performance has not been established.

This model should not be relied upon as the sole source for high-stakes decisions.

📜 License and attribution

The base model and training dataset are credited above. Before redistributing this fine-tune, verify the applicable licenses and usage conditions for both the base model and the dataset, including any restrictions that apply to teacher-generated data.

The model is released under the Apache-2.0 license. However, the underlying dataset’s licensing terms and usage restrictions should be reviewed separately to determine whether they permit the intended use and redistribution of this fine-tune.

🙌 Acknowledgements

Thanks to the creators and maintainers of the Ministral base model and the Qwen distillation dataset for making this experiment possible.

Have fun experimenting, evaluating, and building on top of it! 🚀

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider