CodeXBT

kairos-v1

README

License: apache-2.0

Model Overview

kairos-v1 is a state-of-the-art multimodal foundation model developed by CodeXBT, designed for advanced reasoning, code generation, and natural language understanding. Built on a dense decoder-only transformer architecture with significant optimizations in attention computation and training stability, kairos-v1 achieves competitive performance across a broad range of benchmarks while maintaining efficient inference through speculative decoding and KV-cache optimizations.

The model was trained on a carefully curated mix of web text, academic papers, programming repositories, scientific articles, and multimodal datasets totaling approximately 3.2 trillion tokens. This diverse training corpus enables strong zero-shot and few-shot generalization across tasks spanning mathematics, logical reasoning, coding, and multilingual comprehension.

Key Highlights

7 billion parameters with dense transformer architecture
128K context window enabled by RoPE and grouped-query attention (GQA)
3.2T training tokens from a diverse, high-quality corpus
Top-tier benchmark scores on MMLU, HumanEval, GSM8K, and SQuAD
Multilingual support for 8+ languages with balanced representation
Production-ready with vLLM, Hugging Face Transformers, and Ollama integration

Model Architecture

Table with columns: Component, Specification
Component	Specification
Architecture Type	Decoder-only Transformer (causal LM)
Number of Layers	48
Hidden Dimension	4,096
Number of Attention Heads	32 (GQA: 8 KV heads)
Intermediate Dimension (FFN)	14,336
Vocabulary Size	128,256 (SentencePiece)
Sequence Length	128,000
Position Embedding	RoPE (rotary, base 10000, scaled 2x)
Activation	SwiGLU
Attention	Grouped-Query Attention (GQA)
Normalization	RMSNorm (pre-norm, eps=1e-6)
Embedding	Learnable, shared weights
Initialization	Kaiming uniform, layer norm bias=0
Quantization Support	INT4, INT8, FP16, BF16

Architecture Diagram

text

Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output                    │              │                 │              RMSNorm         RMSNorm           RMSNorm              RoPE-Attn       RoPE-Attn         RoPE-Attn              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN

Training Details

Dataset Composition

The model was trained on approximately 3.2 trillion tokens from a curated dataset with the following distribution:

Table with columns: Source Category, Tokens (B), Percentage
Source Category	Tokens (B)	Percentage
Web Text (filtered)	1,280	40.0%
Academic Papers	512	16.0%
Code (GitHub, StackOverflow)	640	20.0%
Scientific & Math	384	12.0%
Books & Literature	256	8.0%
Multilingual (8 langs)	128	4.0%
Instruction Tuning Pairs	16	0.5%
Reinforcement Learning Data	4	0.125%
Others	128	4.0%

Preprocessing Pipeline

Deduplication: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8)
Quality Filtering: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification
Tokenization: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256)
Sequence Packing: Documents packed to 128K with attention masking

Training Infrastructure

Hardware: 3,072x NVIDIA H100 80GB (SXM5)
Framework: Custom training framework on top of PyTorch 2.4 + Megatron-Core
Parallelism: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel)
Optimizer: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1
Learning Rate: Cosine schedule with 2% warmup, peak lr=2.5e-4
Batch Size: Global batch size of 4M tokens (micro-batch=256, accumulation=4)
Training Steps: ~819K steps (~12 days)
Checkpoint Frequency: Every 500 steps
Mixed Precision: BF16 with FP32 master weights

Installation & Usage

Prerequisites

Python 3.10+
PyTorch 2.4+
CUDA 12.1+ (for GPU inference)

Install via pip

bash

pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2

Quick Start — Load & Inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "codextb/kairos-v1" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(    model_name,    torch_dtype="auto",    device_map="auto",    trust_remote_code=True,) prompt = "Explain the concept of backpropagation in neural networks."inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate(    **inputs,    max_new_tokens=512,    temperature=0.7,    top_p=0.9,    do_sample=True,) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference with vLLM (High Throughput)

python

from vllm import LLM, SamplingParams llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512) prompts = [    "Write a Python function to merge two sorted lists.",    "Translate 'Hello, how are you?' to French.",] outputs = llm.generate(prompts, sampling_params)for out in outputs:    print(out.outputs[0].text)

Inference with Ollama (Local Deployment)

bash

ollama pull codextb/kairos-v1ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"

Quantized Inference (INT4)

python

from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained(    "codextb/kairos-v1",    torch_dtype="auto",    device_map="auto",    load_in_4bit=True,        # bitsandbytes INT4    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype="float16",    trust_remote_code=True,)

Evaluation Results

English Benchmarks

Table with columns: Benchmark, kairos-v1, Competitor A, Competitor B
Benchmark	kairos-v1	Competitor A	Competitor B
MMLU (5-shot)	84.7	83.2	82.9
MMLU-Pro (5-shot)	61.3	58.7	59.4
HumanEval (0-shot)	89.6	87.8	85.4
MBPP (3-shot)	86.2	83.5	81.9
GSM8K (8-shot)	91.4	89.7	88.3
MATH (4-shot)	72.8	69.1	67.5
SQuAD 2.0 (0-shot)	88.9	87.2	86.8
TriviaQA (5-shot)	87.3	85.6	84.1
HellaSwag (10-shot)	93.1	92.4	91.8
ARC-C (25-shot)	89.6	87.9	86.3
Big-Bench Hard (3-shot)	82.4	80.1	78.9

Multilingual Benchmarks (MMLU, 0-shot)

Table with columns: Language, kairos-v1
Language	kairos-v1
English	84.7
Chinese	82.1
Japanese	79.8
Korean	77.3
French	83.5
German	81.9
Spanish	83.2
Hindi	71.6

Coding Benchmarks

Table with columns: Benchmark, Task, Pass@1
Benchmark	Task	Pass@1
HumanEval	Code generation	89.6%
MBPP	Code generation	86.2%
LiveCodeBench (Apr 2025)	Competitive programming	67.3
SWE-bench Verified	Software engineering	31.4%

Efficiency Metrics

Table with columns: Setting, GPU, Tokens/sec, Memory (GB)
Setting	GPU	Tokens/sec	Memory (GB)
BF16 Inference	1x H100	142	14.5
BF16 Inference	4x H100	486	5.8
INT4 Quantized	1x A100	98	4.2
vLLM (4 GPUs)	4x H100	1,820	11.2

License

This model is released under the Apache License 2.0.

You are free to:

Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material for any purpose, even commercially

See the LICENSE file for details.

Acceptable Use Policy

By downloading and using kairos-v1, you agree not to:

Generate illegal content, hate speech, or material that violates applicable laws
Use the model for automated surveillance or mass profiling
Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial)
Reverse-engineer the model for the purpose of training a competing model at scale

Citation

If you use kairos-v1 in your research or projects, please cite:

bibtex

@misc{kairos_v1_2025,  author = {CodeXBT Research Team},  title = {kairos-v1: A High-Performance Multimodal Foundation Model},  year = {2025},  publisher = {GitHub},  journal = {GitHub repository},  url = {https://github.com/codextb/kairos-v1},  howpublished = {\url{https://github.com/codextb/kairos-v1}}}

Acknowledgments

We thank the following contributors and resources:

CodeXBT Research Team — Model design, training, and evaluation
Open-source community — Leveraging transformers, vLLM, and accelerate libraries
Hardware partners — NVIDIA H100 compute allocation for training
Dataset curators — Researchers who compiled and released the public corpora used in training
Peer reviewers — External reviewers who provided feedback on benchmarking methodology

Model Card Details

Table with columns: Field, Value
Field	Value
Model Developed By	CodeXBT
Model Type	Causal Language Model
Release Date	July 2025
License	Apache 2.0
Repository	github.com/codextb/kairos-v1
Hugging Face	codextb/kairos-v1

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

CodeXBT

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Overview

Key Highlights

7 billion parameters with dense transformer architecture
128K context window enabled by RoPE and grouped-query attention (GQA)
3.2T training tokens from a diverse, high-quality corpus
Top-tier benchmark scores on MMLU, HumanEval, GSM8K, and SQuAD
Multilingual support for 8+ languages with balanced representation
Production-ready with vLLM, Hugging Face Transformers, and Ollama integration

Model Architecture

Table with columns: Component, Specification
Component	Specification
Architecture Type	Decoder-only Transformer (causal LM)
Number of Layers	48
Hidden Dimension	4,096
Number of Attention Heads	32 (GQA: 8 KV heads)
Intermediate Dimension (FFN)	14,336
Vocabulary Size	128,256 (SentencePiece)
Sequence Length	128,000
Position Embedding	RoPE (rotary, base 10000, scaled 2x)
Activation	SwiGLU
Attention	Grouped-Query Attention (GQA)
Normalization	RMSNorm (pre-norm, eps=1e-6)
Embedding	Learnable, shared weights
Initialization	Kaiming uniform, layer norm bias=0
Quantization Support	INT4, INT8, FP16, BF16

Architecture Diagram

text

Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output                    │              │                 │              RMSNorm         RMSNorm           RMSNorm              RoPE-Attn       RoPE-Attn         RoPE-Attn              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN

Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output                    │              │                 │              RMSNorm         RMSNorm           RMSNorm              RoPE-Attn       RoPE-Attn         RoPE-Attn              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN

Training Details

Dataset Composition

The model was trained on approximately 3.2 trillion tokens from a curated dataset with the following distribution:

Table with columns: Source Category, Tokens (B), Percentage
Source Category	Tokens (B)	Percentage
Web Text (filtered)	1,280	40.0%
Academic Papers	512	16.0%
Code (GitHub, StackOverflow)	640	20.0%
Scientific & Math	384	12.0%
Books & Literature	256	8.0%
Multilingual (8 langs)	128	4.0%
Instruction Tuning Pairs	16	0.5%
Reinforcement Learning Data	4	0.125%
Others	128	4.0%

Preprocessing Pipeline

Deduplication: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8)
Quality Filtering: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification
Tokenization: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256)
Sequence Packing: Documents packed to 128K with attention masking

Training Infrastructure

Hardware: 3,072x NVIDIA H100 80GB (SXM5)
Framework: Custom training framework on top of PyTorch 2.4 + Megatron-Core
Parallelism: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel)
Optimizer: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1
Learning Rate: Cosine schedule with 2% warmup, peak lr=2.5e-4
Batch Size: Global batch size of 4M tokens (micro-batch=256, accumulation=4)
Training Steps: ~819K steps (~12 days)
Checkpoint Frequency: Every 500 steps
Mixed Precision: BF16 with FP32 master weights

Installation & Usage

Prerequisites

Python 3.10+
PyTorch 2.4+
CUDA 12.1+ (for GPU inference)

Install via pip

bash

pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2

pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2

Quick Start — Load & Inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "codextb/kairos-v1" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(    model_name,    torch_dtype="auto",    device_map="auto",    trust_remote_code=True,) prompt = "Explain the concept of backpropagation in neural networks."inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate(    **inputs,    max_new_tokens=512,    temperature=0.7,    top_p=0.9,    do_sample=True,) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "codextb/kairos-v1" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained(    model_name,    torch_dtype="auto",    device_map="auto",    trust_remote_code=True,) prompt = "Explain the concept of backpropagation in neural networks."inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate(    **inputs,    max_new_tokens=512,    temperature=0.7,    top_p=0.9,    do_sample=True,) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference with vLLM (High Throughput)

python

from vllm import LLM, SamplingParams llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512) prompts = [    "Write a Python function to merge two sorted lists.",    "Translate 'Hello, how are you?' to French.",] outputs = llm.generate(prompts, sampling_params)for out in outputs:    print(out.outputs[0].text)

from vllm import LLM, SamplingParams llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512) prompts = [    "Write a Python function to merge two sorted lists.",    "Translate 'Hello, how are you?' to French.",] outputs = llm.generate(prompts, sampling_params)for out in outputs:    print(out.outputs[0].text)

Inference with Ollama (Local Deployment)

bash

ollama pull codextb/kairos-v1ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"

ollama pull codextb/kairos-v1ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"

Quantized Inference (INT4)

python

from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained(    "codextb/kairos-v1",    torch_dtype="auto",    device_map="auto",    load_in_4bit=True,        # bitsandbytes INT4    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype="float16",    trust_remote_code=True,)

from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained(    "codextb/kairos-v1",    torch_dtype="auto",    device_map="auto",    load_in_4bit=True,        # bitsandbytes INT4    bnb_4bit_quant_type="nf4",    bnb_4bit_compute_dtype="float16",    trust_remote_code=True,)

Evaluation Results

English Benchmarks

Table with columns: Benchmark, kairos-v1, Competitor A, Competitor B
Benchmark	kairos-v1	Competitor A	Competitor B
MMLU (5-shot)	84.7	83.2	82.9
MMLU-Pro (5-shot)	61.3	58.7	59.4
HumanEval (0-shot)	89.6	87.8	85.4
MBPP (3-shot)	86.2	83.5	81.9
GSM8K (8-shot)	91.4	89.7	88.3
MATH (4-shot)	72.8	69.1	67.5
SQuAD 2.0 (0-shot)	88.9	87.2	86.8
TriviaQA (5-shot)	87.3	85.6	84.1
HellaSwag (10-shot)	93.1	92.4	91.8
ARC-C (25-shot)	89.6	87.9	86.3
Big-Bench Hard (3-shot)	82.4	80.1	78.9

Multilingual Benchmarks (MMLU, 0-shot)

Table with columns: Language, kairos-v1
Language	kairos-v1
English	84.7
Chinese	82.1
Japanese	79.8
Korean	77.3
French	83.5
German	81.9
Spanish	83.2
Hindi	71.6

Coding Benchmarks

Table with columns: Benchmark, Task, Pass@1
Benchmark	Task	Pass@1
HumanEval	Code generation	89.6%
MBPP	Code generation	86.2%
LiveCodeBench (Apr 2025)	Competitive programming	67.3
SWE-bench Verified	Software engineering	31.4%

Efficiency Metrics

Table with columns: Setting, GPU, Tokens/sec, Memory (GB)
Setting	GPU	Tokens/sec	Memory (GB)
BF16 Inference	1x H100	142	14.5
BF16 Inference	4x H100	486	5.8
INT4 Quantized	1x A100	98	4.2
vLLM (4 GPUs)	4x H100	1,820	11.2

License

This model is released under the Apache License 2.0.

You are free to:

Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material for any purpose, even commercially

See the LICENSE file for details.

Acceptable Use Policy

By downloading and using kairos-v1, you agree not to:

Generate illegal content, hate speech, or material that violates applicable laws
Use the model for automated surveillance or mass profiling
Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial)
Reverse-engineer the model for the purpose of training a competing model at scale

Citation

If you use kairos-v1 in your research or projects, please cite:

bibtex

@misc{kairos_v1_2025,  author = {CodeXBT Research Team},  title = {kairos-v1: A High-Performance Multimodal Foundation Model},  year = {2025},  publisher = {GitHub},  journal = {GitHub repository},  url = {https://github.com/codextb/kairos-v1},  howpublished = {\url{https://github.com/codextb/kairos-v1}}}

@misc{kairos_v1_2025,  author = {CodeXBT Research Team},  title = {kairos-v1: A High-Performance Multimodal Foundation Model},  year = {2025},  publisher = {GitHub},  journal = {GitHub repository},  url = {https://github.com/codextb/kairos-v1},  howpublished = {\url{https://github.com/codextb/kairos-v1}}}

Acknowledgments

We thank the following contributors and resources:

CodeXBT Research Team — Model design, training, and evaluation
Open-source community — Leveraging transformers, vLLM, and accelerate libraries
Hardware partners — NVIDIA H100 compute allocation for training
Dataset curators — Researchers who compiled and released the public corpora used in training
Peer reviewers — External reviewers who provided feedback on benchmarking methodology

Model Card Details

Table with columns: Field, Value
Field	Value
Model Developed By	CodeXBT
Model Type	Causal Language Model
Release Date	July 2025
License	Apache 2.0
Repository	github.com/codextb/kairos-v1
Hugging Face	codextb/kairos-v1