Treese

RQwen3-751M-Base

README

License: apache-2.0

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Treese/RQwen3-751M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

prompt = "The theory of general relativity"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The weights load through the standard transformers Qwen3ForCausalLM class — no trust_remote_code needed, and anything that reads Qwen3 (vLLM, llama.cpp converters, lm-evaluation-harness) will read this.

Architecture

Table with columns: Parameter, Value
Parameter	Value
`hidden_size`	1024
`num_hidden_layers`	28
`num_attention_heads`	16
`num_key_value_heads`	8 (GQA, 2:1)
`head_dim`	128
`intermediate_size`	3072
`vocab_size`	151,936
`max_position_embeddings`	2,048
`rope_theta`	1,000,000
`rms_norm_eps`	1e-06
`tie_word_embeddings`	`false` (untied LM head)
Total parameters	751,632,384
Serialized dtype	`float32`

Departures from Qwen3-0.6B worth knowing about: the LM head is untied from the input embedding (Qwen3-0.6B ties them), and the trained context length is 2,048 tokens rather than Qwen3-0.6B's 32K.

Training

Table

Data	6-source curated mix, ~13B tokens (~1 epoch)
Steps	50,000
Effective batch	128 sequences (`batch_size=2` × `grad_accum=64`), ~262K tokens/step
Sequence length	2,048
Optimizer	AdamW, `weight_decay=0.1`, `grad_clip=1.0`
LR schedule	Cosine, peak `3e-4` → min , 500-step warmup

Data mix

Table with columns: Source, HF path, Share
Source	HF path	Share
FineWeb-Edu	`HuggingFaceFW/fineweb-edu`	54%
Wikipedia	`wikimedia/wikipedia` (`20231101.en`)	15%
OpenWebMath	`open-web-math/open-web-math`	12%
StackExchange	`HuggingFaceH4/stack-exchange-preferences`

Quality-filtered, exact-hash deduplicated within each source, and pre-tokenized into binary shards. Details: docs/data-pipeline.md.

Evaluation

Run with EleutherAI/lm-evaluation-harness v0.4.13, hf backend, bf16 on a single L40S. All three tasks at the harness's registered defaults (0-shot for all three in 0.4.13).

Table with columns: Task, Config, Score, Chance
Task	Config	Score	Chance
ARC-Challenge	0-shot, `acc`	24.32% ± 1.25	25%
HellaSwag	0-shot, `acc`	31.47% ± 0.46	25%
MMLU (all 57 subjects, average)	0-shot, `acc`	25.32% ± 0.37	25%

Reading these honestly:

HellaSwag is where the model actually has signal: +6.5 points over chance, well outside the standard error. That's the model demonstrating learned commonsense-continuation ability from ~13B tokens of predominantly educational web text.
ARC-Challenge and MMLU are at chance level. Both require multi-step reasoning and broad-domain knowledge that a 751M base model trained on ~13B tokens has not acquired. This is the expected outcome from scaling laws, not a bug.
0-shot MMLU is not directly comparable to the 5-shot MMLU numbers reported in most base-model papers. Standard 5-shot reporting is planned as a follow-up.

For rough context on the training loss value above: from-scratch dense models around this size typically land near ~2.85 (GPT-2 large 774M, Pythia-410M) on their own training distributions, and Qwen3-0.6B reports ~2.4 after ~5T tokens — roughly 400× more data than this run saw. These numbers come from different corpora and are not apples-to-apples.

Limitations

Not an assistant. No SFT, no RLHF, no instruction tuning. It continues text; it does not follow instructions or answer questions reliably.
It fabricates facts. At 13B training tokens the model has grammar and fluency well before it has reliable knowledge. Generated statements that sound authoritative are frequently wrong.
Mode collapse. Greedy decoding tends to fall into repetition loops. Sample with temperature and top_p set.
Short context. Trained at 2,048 tokens. Quality past that is untested and should be assumed poor.
English-only in practice. The Qwen3 tokenizer is multilingual, but the training mix is English.
A chat template ships with the tokenizer because it is inherited from the Qwen3 tokenizer repo. The model was never trained on it. Ignore it.
Unfiltered web data. FineWeb-Edu and OpenWebMath are quality-scored but not safety-filtered. Expect the biases and toxicity of web-scale corpora. No safety tuning has been applied.

Not suitable for production, for user-facing deployment, or for any decision-making use. This is a research and educational artifact.

Tokenizer

Uses the Qwen3 BPE tokenizer (151,936 tokens) from Qwen/Qwen3-0.6B, unchanged. EOS token id 151645.

Reproducing the conversion

Weights were converted from the native training checkpoint with scripts/export_to_hf.py, which verifies the exported model against the original src/ implementation — same tokens, same logits in fp32 — before writing anything.

License

Apache 2.0, matching the Qwen3 architecture and tokenizer this model is built on.

Citation

bibtex
@misc{rqwen3_2026,
  title  = {RQwen3: A 751M-Parameter Qwen3 Architecture Pretrained From Scratch},
  author = {Treese},
  year   = {2026},
  url    = {https://github.com/R-Theory/RQwen3}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Treese

Model Tree

Base

this model

Input Modalities

Text

Output Modalities