Qwen3.8-9B-Instruct-Turbo API & Inference Endpoint

📦 Model Artifacts & Formats

Table with columns: Format / Quantization, Filename / Location, Size, Primary Runtime, Direct Link
Format / Quantization	Filename / Location	Size	Primary Runtime	Direct Link
GGUF Q8_0	`qwen3.8-9b-instruct-turbo-q8_0.gguf`	11.94 GB	Ollama, LM Studio, llama.cpp, Jan	Download GGUF
Safetensors Shards	`model-00001` to `model-00005`	18.1 GB	PyTorch, Transformers, vLLM, TGI	Browse Files
LoRA Adapters	`adapter_model.safetensors`	49.57 MB	PEFT, Unsloth, SFT	Adapter File

📈 Checkpoint Milestones & Calibration Progression

Table with columns: Checkpoint Stage, Global Steps, Training Loss, State Description, Repository Directory
Checkpoint Stage	Global Steps	Training Loss	State Description	Repository Directory
`step-0`	0	11.51 (Baseline)	Initial zero-shot pruned baseline	`checkpoints/step-0/`
`step-50`	50	4.64	Vocabulary & syntax stabilization

💻 Quickstart Inference

1. `llama.cpp` CLI (GGUF)

bash
llama-cli -m qwen3.8-9b-instruct-turbo-q8_0.gguf \
    -p "<|im_start|>user\nExplain quantum entanglement simply.<|im_end|>\n<|im_start|>assistant\n<think>\n" \
    -n 256 -ngl 99

2. Ollama Deployment (GGUF)

Create a Modelfile:

dockerfile
FROM ./qwen3.8-9b-instruct-turbo-q8_0.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
<think>
{{ .Response }}"""
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.6

Run:

bash
ollama create qwen3.8-9b-turbo -f ./Modelfile
ollama run qwen3.8-9b-turbo

3. PyTorch / Transformers & PEFT

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

MODEL_ID = "ewinregirgojr/Qwen3.8-9B-Instruct-Turbo"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load calibrated LoRA adapter
model = PeftModel.from_pretrained(base_model, MODEL_ID, subfolder="checkpoints/healed-final")

prompt = "<|im_start|>user\nExplain how layer pruning reduces memory bandwidth requirements during LLM inference.<|im_end|>\n<|im_start|>assistant\n<think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.6, do_sample=True)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

4. Apple MLX Inference (Apple Silicon)

python
from mlx_lm import load, generate

model, tokenizer = load("ewinregirgojr/Qwen3.8-9B-Instruct-Turbo", adapter_path="checkpoints/healed-final")
response = generate(
    model,
    tokenizer,
    prompt="<|im_start|>user\nWrite a fast Python function to find prime numbers using a sieve.\n<|im_end|>\n<|im_start|>assistant\n<think>\n",
    max_tokens=200,
    verbose=True
)

📊 Empirical Verification & Hardware Metrics

GGUF Q8_0 Binary: 11.94 GB (Runs on 16GB RAM Macs, consumer GPUs, and Ollama).
4-bit QLoRA VRAM Footprint: 4.6 GB.
Architecture: 22 layers, 5,120 hidden dimension, 248,320 vocabulary, 9.04B dense parameters.
Empirical Benchmarks: Full JSON evaluation logs are committed in empirical_benchmark_results.json.

Format / Quantization

Filename / Location

Size

Primary Runtime

Direct Link

GGUF Q8_0

qwen3.8-9b-instruct-turbo-q8_0.gguf

11.94 GB

Ollama, LM Studio, llama.cpp, Jan

Download GGUF

Safetensors Shards

model-00001 to model-00005

18.1 GB

PyTorch, Transformers, vLLM, TGI

Browse Files

LoRA Adapters

adapter_model.safetensors

49.57 MB

PEFT, Unsloth, SFT

Adapter File

Checkpoint Stage

Global Steps

Training Loss

State Description

Repository Directory

step-0

11.51 (Baseline)

Initial zero-shot pruned baseline

checkpoints/step-0/

step-50

4.64

Vocabulary & syntax stabilization

dockerfile

FROM ./qwen3.8-9b-instruct-turbo-q8_0.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
<think>
{{ .Response }}"""
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.6

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

MODEL_ID = "ewinregirgojr/Qwen3.8-9B-Instruct-Turbo"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load calibrated LoRA adapter
model = PeftModel.from_pretrained(base_model, MODEL_ID, subfolder="checkpoints/healed-final")

prompt = "<|im_start|>user\nExplain how layer pruning reduces memory bandwidth requirements during LLM inference.<|im_end|>\n<|im_start|>assistant\n<think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.6, do_sample=True)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

python

from mlx_lm import load, generate

model, tokenizer = load("ewinregirgojr/Qwen3.8-9B-Instruct-Turbo", adapter_path="checkpoints/healed-final")
response = generate(
    model,
    tokenizer,
    prompt="<|im_start|>user\nWrite a fast Python function to find prime numbers using a sieve.\n<|im_end|>\n<|im_start|>assistant\n<think>\n",
    max_tokens=200,
    verbose=True
)

Qwen3.8-9B-Instruct-Turbo

README

📦 Model Artifacts & Formats

📈 Checkpoint Milestones & Calibration Progression

💻 Quickstart Inference

1. `llama.cpp` CLI (GGUF)

2. Ollama Deployment (GGUF)

3. PyTorch / Transformers & PEFT

4. Apple MLX Inference (Apple Silicon)

📊 Empirical Verification & Hardware Metrics

Explore FriendliAI today

README

📦 Model Artifacts & Formats

📈 Checkpoint Milestones & Calibration Progression

💻 Quickstart Inference

1. `llama.cpp` CLI (GGUF)

2. Ollama Deployment (GGUF)

3. PyTorch / Transformers & PEFT

4. Apple MLX Inference (Apple Silicon)

📊 Empirical Verification & Hardware Metrics

Qwen3.8-9B-Instruct-Turbo

README

📦 Model Artifacts & Formats

📈 Checkpoint Milestones & Calibration Progression

💻 Quickstart Inference

1. llama.cpp CLI (GGUF)

2. Ollama Deployment (GGUF)

3. PyTorch / Transformers & PEFT

4. Apple MLX Inference (Apple Silicon)

📊 Empirical Verification & Hardware Metrics

Explore FriendliAI today

README

📦 Model Artifacts & Formats

📈 Checkpoint Milestones & Calibration Progression

💻 Quickstart Inference

1. llama.cpp CLI (GGUF)

2. Ollama Deployment (GGUF)

3. PyTorch / Transformers & PEFT

4. Apple MLX Inference (Apple Silicon)

📊 Empirical Verification & Hardware Metrics

1. `llama.cpp` CLI (GGUF)

1. `llama.cpp` CLI (GGUF)