ronantakizawa

ELYZA-Thinking-1.0-llm-jp-4-33b-gptq

README

License: apache-2.0

Quantization

Table

Method	GPTQ, LLM Compressor 0.14.0, compressed-tensors 0.19.0
Scheme	W4A16: int4 symmetric weights, group size 128, static activation ordering; BF16 activations
Quantized	All 448 Linear layers in the 64 decoder layers (q/k/v/o, gate/up/down)
Kept in BF16	`lm_head`, embeddings, norms (byte-identical to the original)
Calibration	256 documents x 1024 tokens: 128 from C4 English and 128 from mC4 Japanese (train shard 0)
Hardware	8x H100 80 GB (one AWS p5.48xlarge), 8-rank distributed GPTQ, 237 s

The full recipe is in recipe.yaml.

Usage

The tokenizer is a custom class shipped with the original repository (llmjp4_tokenizer.py), so trust_remote_code is required. The vllm_plugins/ directory and chat template are copied unchanged from the original.

bash

vllm serve ronantakizawa/ELYZA-Thinking-1.0-llm-jp-4-33b-w4a16-gptq --trust-remote-code --max-model-len 8192

Tested with vLLM 0.30.0 on H100, where it loads as CompressedTensorsWNA16 with the Machete kernel.

Evaluation

BF16 original vs this checkpoint, both served by vLLM 0.30.0 and scored with lm-evaluation-harness 0.4.13 (local-completions, default few-shot settings, seed 1234). A flip is a question one model answers correctly and the other does not.

Table with columns: Task, BF16, W4A16, Δ, Flips
Task	BF16	W4A16	Δ	Flips
MMLU (acc, 14,042)	73.14	72.58	−0.56	5.9%
HellaSwag (acc_norm, 10,042)	81.40	80.88	−0.52	2.3%
ARC-Challenge (acc_norm, 1,172)	61.35	60.41	−0.94	3.7%
GSM8K 5-shot (flexible, 1,319)

MGSM-ja has only 250 questions (standard error about ±3.1 points), so its difference is within noise.

Token-level comparison on 100 held-out C4 validation documents x 1024 tokens per language (102,300 scored positions each):

Table with columns: English, Japanese
	English	Japanese
Perplexity, BF16 → W4A16	11.05 → 13.00	16.81 → 22.82
Top-1 token agreement with BF16	89.9%	86.6%
BF16 top-1 within W4A16 top-5	98.4%	96.5%
Mean KL(BF16 ‖ W4A16), over BF16 top-20	0.075	0.137

Japanese degrades more than English on every token-level measure. The calibration set covered 19.6% of the 196,608-token vocabulary; a larger or chat-formatted calibration set may close part of that gap. This was not tested.

License and attribution

Apache-2.0, the same license as the original. LICENSE and NOTICE are copied unchanged from elyza/ELYZA-Thinking-1.0-llm-jp-4-33b. All credit for the model belongs to ELYZA, Inc. and the LLM-jp project (National Institute of Informatics).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ronantakizawa

Model Tree

Base

elyza/ELYZA-Thinking-1.0-llm-jp-4-33b

Quantized

this model

Input Modalities

Text

Output Modalities