PoSTMEDIA

Lux-V1-Pro-Heritage-Ko

README

License: apache-2.0

Highlights

Best Korean cultural-heritage knowledge of any open model we could run — 66.05 generative / 65.81 log-likelihood on heritage-ko-bench, ahead of its own base (60.45 / 34.34) and of every Korean open-weight model in reach: EXAONE-4.5-33B, HyperCLOVAX-SEED-Think-14B, Midm-2.0, kanana-1.5, A.X-4.0, KORMo-10B.
Full-parameter fine-tuning of Gemma-4-31B (dense) — not LoRA, not an adapter; 98.2% of weights updated.
Answers briefly when asked briefly — short-answer exact match 10.94 vs the base's 2.26. An earlier version of this model lost that ability entirely; see Training Data.
Measured, not asserted — every comparison number in this card was produced by us on one harness on one day, with a decontamination screen and a choices-only shortcut control reported alongside.

Model Overview

Table with columns: Specification, Details
Specification	Details
Base Model	`google/gemma-4-31B-it`
Parameters	31B (dense, 60 layers, hidden 5376)
Architecture	`Gemma4ForConditionalGeneration` (decoder-only, dense)
Training Precision	BF16
Inference Precision	BF16
Context Length	Inherits from Gemma-4 base (256K)
Fine-Tuning Method	Full-parameter SFT, assistant-only loss
Primary Language	Korean

Training Data

The model is trained on a three-part mixture, deliberately balanced by tokens rather than rows:

Table with columns: Part, Rows, Purpose
Part	Rows	Purpose
Korean heritage grounded QA (`pub-a` + `arch-b` + `arch-a`, non-thinking SFT)	144,215	the domain knowledge
General-instruction replay (Korean instruction-following, chat, math, code — PoSTMEDIA in-house synthetic)	111,797	preserves the base model's answer-length and instruction-following priors
Format adapter (4-choice MCQ + short answer, built only from training-side heritage documents)	9,753	teaches the model to answer briefly when asked briefly
Total	rows ·

The heritage half comes from 8,085 Korean public-institution reports — excavation reports, conservation-science surveys, architectural measurement records, and cultural-property administration documents published by the Korea Heritage Service (국가유산청) and the National Research Institute of Cultural Heritage (국립문화재연구원).

Why the mixture, and not heritage alone

An earlier version of this model was trained on the heritage QA alone. It gained knowledge but lost the ability to answer briefly: 52% of its zero-shot multiple-choice answers were multi-sentence expository prose containing no extractable answer, and it scored 32.40 on the benchmark's generative protocol against the base model's 60.45 — while scoring 64.41 vs the base's 61.07 when a single line of format instruction was added to both. The cause was measurable in the corpus: all 144,469 rows taught exactly one answering contract (mean answer 386 characters, zero rows with a short answer, zero multiple-choice items, zero system turns).

The replay and format-adapter slices exist to fix precisely that, and they do — see Evaluation.

Excluded by design: the -sft-think, -cpt, -dpo, -dpo-think and -rlvr variants of the same suites. The reasoning-trace sets encode <think>…</think> markup, which is not Gemma-4's thinking convention (Gemma-4 uses a <|channel>thought … <channel|> channel and does not tokenize <think> as a special token) — training on them would teach the model to emit literal tag text to users.

Decontamination

The evaluation benchmark was minted from documents held out before the training data was generated. On top of that, the final corpus was screened against all 1,587 benchmark items:

exact benchmark-question matches in training data: 0
gold-answer/question co-occurrence (the answer-specific contamination signature): 0 pairs
gold-vs-distractor enrichment ratio: 0.00 (a clean corpus scores ≤ 1.5; the ratio is the test)

Results are additionally reported on a clean subset with the 19 screened items removed, so any residual effect is visible rather than assumed away.

Training Configuration

Every value below is what the run actually used.

Table with columns: Parameter, Value
Parameter	Value
Fine-Tuning Method	Full-parameter SFT — 30,697,345,280 / 31,273,086,512 params trainable (98.2%; vision/audio towers frozen)
Loss	Assistant-only (`completion_mask`) — 69.4% of tokens supervised
Precision	BF16
Distributed Strategy	DeepSpeed ZeRO-3, no CPU offload
Infrastructure	NVIDIA H200 × 16 (2 nodes)
Epochs	4 (8,308 optimizer steps)
Effective batch	128 sequences (per-device 4 × 4 accumulation)
Learning rate

How base capability is protected

Assistant-only loss — the prompt is masked out, so the model never learns to generate the idiosyncratic 「report-title」 question style, only to answer it.
General-instruction replay at ~1:1 by tokens — the single most important ingredient; without it the model loses the ability to answer briefly (measured, see Training Data).
A decaying schedule with gradient clipping calibrated to the measured gradient norms (p50 2.62 / p90 4.58 — an earlier run clipped at 1.0 and every one of its 2,258 steps hit the ceiling).
Benchmark-based checkpoint selection, plus a retention panel reported below.

Evaluation

All numbers below were measured by us on heritage-ko-bench — a 1,587-item Korean cultural-heritage benchmark (6 four-choice MCQ subsets = 1,287 test items, plus a 270-item short-answer subset) minted exclusively from documents held out before the training data was generated.

Every model was run by us, on the same day, on the same harness, with the same settings. No number is quoted from a vendor card.

Korean cultural-heritage knowledge

Table with columns: Model, Params, Org, Generative 0-shot, Log-likelihood, Short answer (EM)
Model	Params	Org	Generative 0-shot	Log-likelihood	Short answer (EM)
Lux-V1-Pro-Heritage-Ko	31B	PoSTMEDIA	66.05	65.81	10.94
`google/gemma-4-31B-it` (our base)	31B	Google	60.45	34.34	2.26

Against its own base the model gains +5.60 points generative, +31.47 points log-likelihood, and answers short factual questions 4.8× more often correctly — and it is first among every Korean open-weight model we could run in this size class.

Model sizes differ; a 7–14B model is not a size peer of a 31B one. They are included because the question this table answers is "what is the best open model available today for Korean cultural-heritage knowledge", not "which lab trains the best 31B".

Motif-Technologies/Motif-2-12.7B-Instruct could not be measured: MotifForCausalLM was removed from vLLM after v0.10.2 and is hard-rejected by every later release.

Contamination control

The benchmark was decontaminated at the document level before generation, and the training corpus was re-screened against all 1,587 items (0 exact question matches, 0 gold-answer co-occurrences). Scores on the clean subset (19 screened items removed) are within noise of the full set, for us and for the base alike:

Table with columns: full set, clean subset, Δ
	full set	clean subset	Δ
Lux-V1-Pro-Heritage-Ko	66.05	65.54	−0.51
gemma-4-31B-it	60.45	60.17	−0.28

Shortcut control — read this before quoting the headline

A benchmark can be gamed by answering from the options alone. We measure that directly: the same items with the question removed. Random is 25.0%.

Table with columns: accuracy, choices-only, signal
	accuracy	choices-only	signal
Lux-V1-Pro-Heritage-Ko	66.05	32.71	+33.34
gemma-4-31B-it	60.45	24.32	+36.13

Our choices-only score is 8.4 points above the base's. Part of that comes from a format-adapter slice in the training mix whose distractors, despite mitigation, remain more separable than the benchmark's. So the honest reading is: the knowledge gain is real and large on the log-likelihood axis (+31.47, an axis output formatting cannot game), while the generative headline should be discounted by the elevated choices-only floor.

Checkpoint selection

Checkpoints were saved every half epoch and all eight were scored on the benchmark:

Table with columns: epoch, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0
epoch	0.5	1.0	1.5	2.0	2.5	3.0	3.5	4.0
generative	63.64	62.47	64.80	66.05	64.96	64.57	62.70	63.48
log-likelihood	63.33

Epoch 2.0 wins on the benchmark, and every checkpoint — including the over-trained epoch 4.0 — stays above the base model.

Protocol

Harness lm-evaluation-harness 0.4.12, vLLM backend, bf16, num_fewshot: 0, greedy.
Generative: the model answers and a letter is extracted — the protocol these instruct models are built for, and the one we treat as the headline.
Log-likelihood: scores logP(" A") … logP(" D"); no generation, so output formatting cannot influence it. Thinking-default models refuse this protocol, so they were scored with thinking disabled — the same setting the base model is scored under.
Metric is acc. acc_norm is mathematically identical here (all four scored continuations are 2 bytes) and is not reported separately.
short_answer uses exact match with no answer-extraction filter, so it rewards brevity as much as correctness. It is reported separately and never folded into the headline.

Quick Start

bash
pip install transformers accelerate

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "PoSTMEDIA/Lux-V1-Pro-Heritage-Ko"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "백제 석탑 중 현재까지 남아 있는 것은 무엇이며, 그 의의는 무엇인가?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model uses the Gemma-4 chat template unchanged. It was trained on direct answers with no reasoning channel — use it in non-thinking mode.

Use Cases

Korean cultural-heritage research assistance: excavation reports, conservation science, traditional architecture
Question answering over Korean cultural-property administration and designation records
Museum, archive and heritage-agency knowledge tooling
A domain-adapted starting point for Korean heritage RAG systems

Limitations

Question-style bias. All training questions are 「report-title」 anchored grounded QA, a distribution that differs from general chat. The model may perform best when a document or topic anchor is present.
Domain-specialized. General reasoning, math and coding ability are inherited from the base model and are not improved by this fine-tune; see the retention panel above for what changed.
Report-bounded knowledge. Facts come from a specific corpus of Korean public-institution reports. Coverage is uneven across periods and regions, and the model can confidently state something a report asserted even where scholarship has since moved on.
OCR heritage. Source documents were digitized from print; a small residue of OCR artifacts survives the readability gates.
Not evaluated for multimodal use. The base is multimodal; this fine-tune is text-only and the vision and audio towers were frozen.
Source-license note. The training corpus derives from Korean public-institution publications under the Korea Open Government License (KOGL). KOGL type verification is complete for the Korea Heritage Service portion and still pending for the NRICH-derived portion; the third suite (arch-a) is a curated third-party QA draft whose upstream generator is unverified.

Safety & Limitations

Inherits the safety characteristics of the Gemma-4 base; output guardrails are recommended for production.
Not intended for medical, legal, or financial decision-making.
Heritage designations, ownership and administrative status change over time — verify against the current official record before acting on an answer.
May hallucinate. Human review is recommended for critical outputs.

Citation

bibtex
@misc{lux_v1_pro_heritage_ko_2026,
  title  = {Lux-V1-Pro-Heritage-Ko: Full Fine-Tuning of Gemma-4-31B on Synthesized Korean Cultural-Heritage Knowledge},
  author = {PoSTMEDIA AI Lab},
  year   = {2026},
  publisher = {Hugging Face}
}

Contact

PoSTMEDIA AI Lab

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.