sakthai-context-0.5b-tools API & Inference Endpoint

Model Description

SakThai Context 0.5B Tools is a LoRA adapter for Qwen2.5-0.5B-Instruct, fine-tuned for tool-calling and function-calling in agentic workflows. This is the LoRA weights only — merge with the base model to get the full weights.

For the merged GGUF (ready to run): Use sakthai-context-0.5b-merged instead.

Why This Model? 🎯

At 494M parameters, this is the smallest tool-calling model on Hugging Face. It trades raw benchmark score for:

Table with columns: Benefit, What It Means
Benefit	What It Means
⚡ Fastest inference	~50 tokens/sec on CPU (10× faster than 7B models)
💾 1 GB RAM	Runs on a Raspberry Pi 5, old laptops, cloud free tier
🔋 Battery-friendly	Perfect for mobile/edge deployment
🆓 Free to run	Works on Hugging Face's free Inference API (no GPU needed)
🛠️ Real tool-calling	Proven `<tools>` block support via transformers

Capabilities

✅ Tool/function calling via <tools> XML block
✅ Multi-turn conversation with context retention
✅ Structured JSON output
✅ Instruction following
✅ CPU inference (under 1 GB RAM)
✅ Works on HF free Inference API

Use Cases

Table with columns: Scenario, Why 0.5B Shines
Scenario	Why 0.5B Shines
🎓 Learning/Prototyping	Instant downloads, zero hardware requirements
🤖 Personal AI agent	Run your own assistant on a $35 Raspberry Pi
📱 Mobile/Edge AI	Tiny enough for on-device deployment
🔄 RAG pipeline	Lightweight tool router before invoking larger models
💬 Simple chatbots	Q&A, summarization, content generation
🧪 CI/CD testing	Fast enough to use in automated test suites

Quick Start

Via transformers (LoRA adapter)

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Nanthasit/sakthai-context-0.5b-tools")

# Tool-calling example
messages = [
    {"role": "system", "content": "You have access to tools. Call them when needed."},
    {"role": "user", "content": "What's the weather in Bangkok?"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)
outputs = model.generate(inputs, max_new_tokens=128, temperature=0.3)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Via llama.cpp (GGUF — recommended for production)

python
from llama_cpp import Llama

llm = Llama(
    model_path="sakthai-0.5b-Q4_K_M.gguf",
    n_ctx=4096,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant with tools."},
        {"role": "user", "content": "Get the stock price for AAPL."},
    ],
    temperature=0.2,
)
print(response["choices"][0]["message"]["content"])

Via Hugging Face Inference API (FREE — no install)

bash
curl -X POST "https://api-inference.huggingface.co/models/Nanthasit/sakthai-context-0.5b-tools" \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": "What is the capital of France?",
    "parameters": {"max_new_tokens": 64}
  }'

python
# Or with the Python client
from huggingface_hub import InferenceClient

client = InferenceClient()
result = client.text_generation(
    "What is the capital of France?",
    model="Nanthasit/sakthai-context-0.5b-tools",
    max_new_tokens=64,
)
print(result)

Performance Comparison

Table with columns: Metric, 0.5B Tools 🏆, 1.5B Tools, 7B Tools
Metric	0.5B Tools 🏆	1.5B Tools	7B Tools
Parameters	494M	1.5B	7B
RAM required	~1 GB	~3 GB	~14 GB
CPU inference speed	~50 tok/s	~20 tok/s	~5 tok/s
GPU recommended?	No	No	Yes

🏗️ About that 1/5 BFCL score: The 0.5B CAN perform real tool-calling via the <tools> XML block (proved Jul 25 2026). It scores lower on the Berkeley Function Calling Leaderboard because BFCL tests complex multi-tool scenarios designed for much larger models — not because tool-calling doesn't work. For production workloads, use the 1.5B or 7B variants. For prototyping, learning, edge deployment, and lightweight tasks, this model is a perfect starting point.

Architecture

Table with columns: Property, Value
Property	Value
Base Model	Qwen2.5-0.5B-Instruct
Method	QLoRA (4-bit NF4)
LoRA rank	8
LoRA alpha	16
Target modules	q_proj, k_proj, v_proj, o_proj
Parameters	494M (base) + ~2M (LoRA)
Format	ChatML with tool schema

Benchmark

Table with columns: Test, Result
Test	Result
BFCL Tool-Calling	1/5 🏗️
General Q&A	5/5 ✅
Multi-turn Context	5/5 ✅
JSON Output	5/5 ✅

Note: The 0.5B model CAN do function calling via transformers (proved Jul 25 2026). For reliable production tool-calling, use the 1.5B-tools or 7B-tools variants.

Training

Dataset: sakthai-combined-v6 (2,003 examples)
Base: Qwen2.5-0.5B-Instruct
GPU: None (CPU-only training)

SakThai Model Family

Table with columns: Model, Pipeline, Downloads
Model	Pipeline	Downloads
sakthai-context-1.5b-merged	text-generation	1,269 ⬇
sakthai-context-0.5b-merged	text-generation	1,030 ⬇
sakthai-context-7b-merged	text-generation	585 ⬇
sakthai-context-7b-128k

🌱 Low-Download Gems

These sibling models also deserve attention:

Table with columns: Model, Downloads, Best For
Model	Downloads	Best For
sakthai-coder-1.5b	70 ⬇	Code generation & assistant
sakthai-tts-model	69 ⬇	15-language text-to-speech
sakthai-context-0.5b-tools (this)	7 ⬇	Edge tool-calling

Datasets

Table with columns: Dataset, Downloads
Dataset	Downloads
sakthai-combined-v6	175 ⬇
sakthai-kaggle-notebooks	103 ⬇
SimpleToolCalling	52 ⬇
food-penguin-v1	51 ⬇

Spaces

Table with columns: Space, Description
Space	Description
sakthai-tts	🎤 TTS Playground
sakthai-leaderboard	📊 Benchmark tracker
sakthai-vision-demo	🖼️ Vision model demo

📦 Full collection: SakThai Model Family

Links

Benefit

What It Means

⚡ Fastest inference

~50 tokens/sec on CPU (10× faster than 7B models)

💾 1 GB RAM

Runs on a Raspberry Pi 5, old laptops, cloud free tier

🔋 Battery-friendly

Perfect for mobile/edge deployment

🆓 Free to run

Works on Hugging Face's free Inference API (no GPU needed)

🛠️ Real tool-calling

Proven <tools> block support via transformers

Scenario

Why 0.5B Shines

🎓 Learning/Prototyping

Instant downloads, zero hardware requirements

🤖 Personal AI agent

Run your own assistant on a $35 Raspberry Pi

📱 Mobile/Edge AI

Tiny enough for on-device deployment

🔄 RAG pipeline

Lightweight tool router before invoking larger models

💬 Simple chatbots

Q&A, summarization, content generation

🧪 CI/CD testing

Fast enough to use in automated test suites

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Nanthasit/sakthai-context-0.5b-tools")

# Tool-calling example
messages = [
    {"role": "system", "content": "You have access to tools. Call them when needed."},
    {"role": "user", "content": "What's the weather in Bangkok?"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)
outputs = model.generate(inputs, max_new_tokens=128, temperature=0.3)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

python

from llama_cpp import Llama

llm = Llama(
    model_path="sakthai-0.5b-Q4_K_M.gguf",
    n_ctx=4096,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant with tools."},
        {"role": "user", "content": "Get the stock price for AAPL."},
    ],
    temperature=0.2,
)
print(response["choices"][0]["message"]["content"])

bash

curl -X POST "https://api-inference.huggingface.co/models/Nanthasit/sakthai-context-0.5b-tools" \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": "What is the capital of France?",
    "parameters": {"max_new_tokens": 64}
  }'

python

# Or with the Python client
from huggingface_hub import InferenceClient

client = InferenceClient()
result = client.text_generation(
    "What is the capital of France?",
    model="Nanthasit/sakthai-context-0.5b-tools",
    max_new_tokens=64,
)
print(result)

Metric

0.5B Tools 🏆

1.5B Tools

7B Tools

Parameters

494M

1.5B

RAM required

~1 GB

~3 GB

~14 GB

CPU inference speed

~50 tok/s

~20 tok/s

~5 tok/s

GPU recommended?

Yes

Property

Value

Base Model

Qwen2.5-0.5B-Instruct

Method

QLoRA (4-bit NF4)

LoRA rank

LoRA alpha

Target modules

q_proj, k_proj, v_proj, o_proj

Parameters

494M (base) + ~2M (LoRA)

Format

ChatML with tool schema

Test

Result

BFCL Tool-Calling

1/5 🏗️

General Q&A

5/5 ✅

Multi-turn Context

5/5 ✅

JSON Output

5/5 ✅

Model

Pipeline

Downloads

sakthai-context-1.5b-merged

text-generation

1,269 ⬇

sakthai-context-0.5b-merged

text-generation

1,030 ⬇

sakthai-context-7b-merged

text-generation

585 ⬇

sakthai-context-7b-128k

Model

Downloads

Best For

sakthai-coder-1.5b

70 ⬇

Code generation & assistant

sakthai-tts-model

69 ⬇

15-language text-to-speech

sakthai-context-0.5b-tools (this)

7 ⬇

Edge tool-calling

Dataset

Downloads

sakthai-combined-v6

175 ⬇

sakthai-kaggle-notebooks

103 ⬇

SimpleToolCalling

52 ⬇

food-penguin-v1

51 ⬇

Space

Description

sakthai-tts

🎤 TTS Playground

sakthai-leaderboard

📊 Benchmark tracker

sakthai-vision-demo

🖼️ Vision model demo

sakthai-context-0.5b-tools

README

Model Description

Why This Model? 🎯

Capabilities

Use Cases

Quick Start

Via transformers (LoRA adapter)

Via llama.cpp (GGUF — recommended for production)

Via Hugging Face Inference API (FREE — no install)

Performance Comparison

Architecture

Benchmark

Training

SakThai Model Family

🌱 Low-Download Gems

Datasets

Spaces

Links

Explore FriendliAI today

README

Model Description

Why This Model? 🎯

Capabilities

Use Cases

Quick Start

Via transformers (LoRA adapter)

Via llama.cpp (GGUF — recommended for production)

Via Hugging Face Inference API (FREE — no install)

Performance Comparison

Architecture

Benchmark

Training

SakThai Model Family

🌱 Low-Download Gems

Datasets

Spaces

Links