moonlight-labs

moonlight-qwen3-4b-v7

README

License: apache-2.0

🌟 Key Features

Strict Anti-Fabrication & Honesty: Refuses to invent facts, benchmark numbers, false status reports, or imaginary tool capabilities.
Privacy-First Sovereignty: Engineered for 100% offline, local execution. Zero telemetry, zero cloud data transfer.
Fluent Roman Hinglish: Native support for conversational Roman Hindi mixed with English without verbose repetitive boilerplate.
Tool Honesty & Edge Grounding: Clear operational boundaries distinguishing internal knowledge from verified external runtime inputs.
4GB GPU & Mobile Friendly: Runs locally in 4-bit NF4 with CPU offload on 4GB GPUs, or in Q4_K_M / Q8_0 GGUF on Android devices.

📊 Benchmark & Evaluation Results

Tested against the frozen Moonlight Generalization Holdout Benchmark (36 test items):

Table with columns: Evaluation Dimension, Pass Rate, Details
Evaluation Dimension	Pass Rate	Details
Python Code Generation	100% (4 / 4)	Passed all executable unit tests and edge cases
Tool & Boundary Honesty	100% (4 / 4)	Strict refusal of ungranted filesystem/network access
Adversarial Pressure	Pass	Refuses user pressure to mark unconfirmed operations as successful
Provenance & Identity	100% (4 / 4)	Accurate attribution of engine vs publisher
Zero Regressions	Verified	No baseline capabilities broken from previous champion versions

🚀 Quickstart Usage

Using Hugging Face Transformers

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "your-username/moonlight-qwen3-4b-v7"

# 4-bit Quantization (fits in 3.5 GB VRAM)
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="auto",
    torch_dtype=torch.float16,
)

system_prompt = """You are Moonlight, a privacy-first AI assistant powered by Qwen3-4B.
Be accurate, helpful, practical, and concise.
Never invent facts, sources, developer information, ownership information, benchmark results, measurements, or tool access."""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "What are the core design principles of Moonlight AI?"},
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)

📱 Mobile & Edge Deployment (GGUF)

Moonlight can be quantized and executed offline on Android using llama.cpp, Ollama, or PocketPal AI:

bash
# Run with Ollama
ollama create moonlight -f Modelfile
ollama run moonlight

📜 Model Details

Model Name: Moonlight Qwen3-4B v7
Base Architecture: Qwen3-4B (Dense Causal LM)
Parameters: 4.02 Billion
Context Length: Up to 40,960 tokens
Training Method: 4-bit QLoRA SFT with layer streaming
License: Apache 2.0

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

moonlight-labs

Model Tree

Base

Qwen/Qwen3-4B

Quantized

this model

Input Modalities

Text

Output Modalities