atenareply

gemma-4-12b-asterion

README

License: apache-2.0

Overview

Stage: Continued Pre-Training (full fine-tune, CLM, 8-bit AdamW)
Lineage: gemma-4-12B → CPT (this model)
Method: Full-parameter CPT (LoRA demonstrably under-performs on knowledge injection [loraless]) with paged 8-bit AdamW — the optimizer quantization is what fits a 12B full-FT on ONE H200 (fp32 AdamW alone would be ~144 GB). 12% FineWeb-Edu replay [replay]; NO fixed end of training: a stop-loss on the held-out eval_domain_loss plateau decides when to stop (1 epoch = data ceiling only [datacon]); bf16 mandatory (Gemma-family fp16 overflow).
Domain: fictional — Asterion Space Operations (24-satellite constellation: alert triage, anomaly investigation, subsystem ops) + reused Mars Express telemetry.

Training

Table

Corpus	asterion-training-corpus-gemma4 — 1.887B Gemma tokens post-dedup (85% Asterion / 3% telemetry / 12% replay), source pinned 7f0c3236
Objective	CLM (next-token), full fine-tune, bf16, paged_adamw_8bit
LR / schedule	1.5e-5 cosine re-warm, warmup 0.03
End of training	stop-loss (eval plateau): closed at step 1,750/14,400 (12% of one epoch, ~229M tokens seen) — the last 3 evals improved ≤0.012 each
Seq / batch / HW	seq 4096, eff_batch 32, per_device=4, 1×H200 (~2,600 tok/s; the 262K-vocab fp32 logits tensor is the binding memory constraint)

Evaluation

Table with columns: Metric, Value, Note
Metric	Value	Note
PPL Asterion held-out	1.83	base gemma-4-12B: 4.80 (-62%)
PPL Mars telemetry	1.25	base: 3.67
PPL general (FineWeb-Edu)	8.34	base: 8.55 — replay works, no forgetting
eval_domain_loss	0.614	1.176 at step 0 → 0.614 at step 1,750

Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See noval-corp/scripts/eval_agentic.py.

Intended use & limitations

Domain-knowledge backbone for the Asterion round. Not instruction-tuned (see the -instruct-paramdelta / -agentic siblings).

Limitations:

Fictional domain — the model 'knows' an invented satellite operator.
No chat/tool behaviour on its own (base-style CPT checkpoint).
Saw ~12% of the corpus (stop-loss end): unique per-document facts in the unseen 88% are not in the weights — by design, downstream tasks provide specifics via tool results in context. Held-out PPL (1.83) shows the domain itself generalized.

Innovations tested

12B full-FT on one GPU — paged 8-bit AdamW + gradient checkpointing + a logits-aware payload model (the 262,144-token vocab makes the fp32 logits tensor, 4.3 GB/seq @4096, the binding memory term — not activations).

Citations

CPT replay (anti-forgetting) — Ibrahim et al., 2024. arXiv:2403.08763
Data-constrained scaling (≤4 epochs) — Muennighoff et al., 2023. arXiv:2305.16264
LoRA learns less on CPT — Biderman et al., 2024. arXiv:2405.09673

Card generated by noval-corp/scripts/gen_model_cards.py (standardized across the noval-corp model family).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

atenareply

Model Tree

Base

google/gemma-4-12B

Fine-tuned

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality