Ismantic

Qwen3-1.7B-Base-ReTok

README

License: apache-2.0

Important Tokenizer Note

This repository contains the custom tokenizer assets:

piece.model
dict.txt
token_mapping.json
tokenizer_wrapper.py

The model architecture can be loaded by Transformers as Qwen3, but the tokenizer is not a standard Qwen tokenizer. For generation, encode prompts with the provided Piece tokenizer wrapper or the Summer/ReTok evaluation scripts.

Training Summary

Replaced the original Qwen3-1.7B-Base tokenizer with an 81,903-token Piece tokenizer.
Initialized new embeddings by mapping each new piece through the original Qwen tokenizer and averaging old embeddings.
Phase 1: trained new embeddings on about 999M packed tokens while freezing the transformer.
Phase 2: annealed on about 200M packed tokens with LoRA q/v adapters, Aurora, and tied embedding/head preservation.

See training_lineage.md for the full reproduction record.

Evaluation

WMT22 1000-sample translation:

Table with columns: Model, zh-en BLEU, zh-en COMET, en-zh BLEU, en-zh COMET
Model	zh-en BLEU	zh-en COMET	en-zh BLEU	en-zh COMET
Qwen3-1.7B-Base	22.3408	0.8122	38.3380	0.8597
ReTok v18 Phase 1	20.2588	0.7821	35.1632	0.8276
ReTok v18 Phase 2 tie	20.4599	0.7933	36.0314	0.8444

WMT23 full-set translation:

Table with columns: Model, zh-en BLEU, zh-en COMET, en-zh BLEU, en-zh COMET
Model	zh-en BLEU	zh-en COMET	en-zh BLEU	en-zh COMET
ReTok v18 Phase 1	19.1310	0.7767	38.8251	0.8198
ReTok v18 Phase 2 tie	19.6046	0.7834	40.9905	0.8377

General benchmark results:

Table with columns: Model, LAMBADA, PIQA, ARC-C, HellaSwag, CEVAL, GSM8K
Model	LAMBADA	PIQA	ARC-C	HellaSwag	CEVAL	GSM8K
ReTok v18 Phase 1	0.5674	0.7301	0.5137	0.6375	0.6263	0.0417
ReTok v18 Phase 2 tie	0.5768	0.7367	0.5145	0.6389	0.6204	0.0356

Limitations

This is a base model, not an instruction-tuned assistant.
Generic Hugging Face hosted inference may not work until the custom Piece tokenizer is packaged as a standard AutoTokenizer implementation.
Results remain below the original Qwen3-1.7B-Base on the WMT22 translation sample after tokenizer replacement.

License

The base model Qwen/Qwen3-1.7B-Base is released under Apache 2.0. This derivative checkpoint is prepared with the same license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Ismantic

Model Tree

Base

Qwen/Qwen3-1.7B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities