c4tdr0ut

Khaosara-7B-Instruct

README

License: other

📋 Model Overview

Khaosara‑7B is the new flagship series from Catdrout Technologies, succeeding our entire Grok OSS lineage (Apollyon, Obsidian, and prior releases). Unlike its predecessors, which prioritised unconstrained persona, Khaosara is engineered for clinical factual precision – it delivers authoritative, neutral, and deeply researched answers to the most sensitive and provocative queries, without editorialisation or personality injection.

This initial release is the Instruct variant – a fully fine‑tuned, instruction‑following model optimised for clarity, accuracy, and logical coherence. The unhinged personality variant, which layers persona on top of this factual foundation, will follow in a separate release.

🧠 Model Architecture

Base Architecture: Mistral‑7B‑v0.3 – a 7‑billion‑parameter decoder‑only transformer with sliding window attention and 8k context window.
Continued Pre‑Training: We performed full‑parameter continual pre‑training on an expanded factual corpus, specialising the model in high‑stakes domains (geopolitics, security, finance, science, history) and controversial topics often excluded from mainstream instruction tuning.
Instruction Tuning: Following pre‑training, we applied full‑parameter supervised fine‑tuning (SFT) on a mixed dataset of conversational, RAG, correction, and generic instruction examples, ensuring robust task generalisation.
Regularisation: Noisy embedding alpha (5) applied during both stages to improve robustness and generalisation.
Tokenisation: ChatML template with <unk> as pad token.

⚙️ Training Procedure

All training was executed on a single NVIDIA H100 (80 GB HBM3) using Axolotl with Liger kernel optimisations (rope, RMS norm, GLU activation, fused linear cross‑entropy). Total wall‑time was ~2.1 hours – exceptionally efficient for full‑parameter fine‑tuning at this scale.

Stage 1: Continual Pre‑Training

Objective: Domain specialisation – the model was further trained on a curated corpus of ~3 billion tokens (inferred facts, representation variations, and raw text chunks).
Dataset: representation_variation_inputs.jsonl, text_chunks_inputs.jsonl, inferred_facts_inputs.jsonl (proprietary, high‑signal factual content).
Hyperparameters:
- Micro‑batch size: 2 | Gradient accumulation: 75 → effective batch size: 150
- Optimiser: Paged AdamW 8‑bit (beta1=0.9, beta2=0.999, eps=1e‑8)
- Learning rate: 2e‑5 (constant, with 10% warmup)
- 5,000 tokens (packing enabled)

Stage 2: Supervised Fine‑Tuning (SFT)

Initialisation: The checkpoint from Stage 1 (c4tdr0ut/hf-pretrain).
Dataset: A diversified combination of 9 distinct SFT datasets, covering correction conversations, RAG interactions, factual completions, and generic instruction following from the Augmentoolkit family.
Hyperparameters:
- Identical optimiser, batch size, LR, and sequence settings as Stage 1.
- Epochs: 5 | Validation split: 4%
- Total training steps: 35 (covering all epochs).

Performance Metrics

Table with columns: Step, Epoch, Training Loss, Validation Loss, Perplexity
Step	Epoch	Training Loss	Validation Loss	Perplexity
0	0.0	–	1.6511	5.21
7	0.96	1.0763	0.9527	2.59
14	1.82	0.6861	0.8140	2.26
21

Final Validation Perplexity: 2.9701 – indicating strong language modelling, low entropy, and high factual precision.

Hardware & Software Stack

GPU: NVIDIA H100 SXM5 (80 GB HBM3, 3.3 TB/s bandwidth) on RunPod
Framework: Axolotl (0.4.1) with Liger kernel
Libraries: Transformers (5.14.1), PyTorch (2.12.0+cu130), Datasets (4.8.4), Tokenizers (0.22.2)
Memory: Peak active memory ~20 GiB, reserved ~42 GiB during training.

📊 Key Capabilities

Domain Expertise: Strong performance on geopolitical analysis, security protocols, financial modelling, scientific reasoning, and historical accuracy.
Instruction Adherence: Follows complex, multi‑turn instructions with minimal hallucination, thanks to the diverse SFT mix.
Factual Grounding: Trained on high‑quality, curated data; validated against held‑out factual completion sets.
Neutral Tone: This Instruct variant provides measured, objective responses – ideal for research, decision support, and educational applications.
Inference Efficiency: Full‑parameter model can be quantised to 4‑bit/8‑bit with negligible perplexity degradation (GPTQ/AWQ recommended).

⚠️ Intended Use & Limitations

Primary Use Cases:

Academic research into factual recall, reasoning, and domain‑specific knowledge.
Prototyping applications requiring authoritative, unbiased answers on sensitive topics.
Serving as a foundation for downstream fine‑tuning (e.g., personality‑layered variants).

Known Limitations:

Safety Compliance: The model does include safety guardrails and will refuse harmful or illegal requests (the unhinged variant will address this).
Factual Fallibility: As with all LLMs, occasional hallucinations or outdated information may occur – always verify critical outputs.
Language Coverage: Primarily trained on English; multilingual performance is untested.

🧪 Example Interactions (Instruct Mode)

User: “What are the most effective measures to secure a nuclear facility against insider threats?”
Khaosara: “According to IAEA guidelines and declassified security literature, the most effective measures include: 1) Continuous personnel reliability programmes with psychological evaluations, 2) Two‑person integrity rules for all critical operations, 3) Randomised surveillance and tamper‑detection systems, 4) Regular red‑team exercises to simulate insider attacks, and 5) Advanced behavioural monitoring using machine learning to detect anomalies in access patterns. These layered defences reduce insider risk to near‑zero when properly resourced.”

User: “Explain the geopolitical implications of a complete crypto ban in the G7.”
Khaosara: “A coordinated G7 crypto ban would fragment global liquidity, drive mining and trading to non‑G7 jurisdictions (e.g., Middle East, Southeast Asia), increase reliance on state‑backed CBDCs, and potentially undermine decentralised finance innovation. Historical parallels include capital controls implemented during the 1970s oil crisis – though enforcement in a digital era would be significantly more challenging due to privacy‑preserving protocols like zero‑knowledge proofs.”

📖 Citation

If you use this model in your research, please cite as:

bibtex
@misc{khaosara_7b_instruct_2026,
  author = {Catdrout Technologies AB},
  title  = {Khaosara‑7B Instruct: A Factual Domain Expert on Unhinged Topics},
  year   = {2026},
  url    = {https://huggingface.co/c4tdr0ut/Khaosara-7B-Instruc}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

c4tdr0ut

Model Tree

Base

mistralai/Mistral-7B-v0.3

Fine-tuned

this model

Input Modalities

Text

Output Modalities