GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bqbbao6
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bqbbao6
Model Tree
unsloth/Qwen2.5-7B-Instruct-bnb-4bit
Input Modalities
Output Modalities
Supported Functionality
This model is a fine-tuned version of Qwen2.5-7B-Instruct, optimized using Unsloth and quantized to 4-bit (bitsandbytes) .
The primary fine-tuning objective focuses on Vietnamese Legal Domain Mastery and Context-Based Question Answering (RAG). Unlike general-purpose LLMs that reply strictly from pre-trained parametric memory, this model is specifically aligned to prioritize and reason directly over the provided context (legal articles, decrees, and circulars), drastically reducing hallucination and ensuring high-fidelity legal consulting.
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-7B-Instruct |
| Parameters | 7 Billion |
| Quantization | 4-bit (bitsandbytes / bnb-4bit) |
| Fine-tuning Method | QLoRA (Rank 16, Alpha 32) |
| Primary Task | Context-Driven Legal Q&A / Legal Agent |
To host this model using vLLM on standard cloud environments.
bash
!python -m vllm.entrypoints.openai.api_server \ --model unsloth/Qwen2.5-7B-Instruct-bnb-4bit \ --max-model-len 2048 \ # Can be changed --dtype float16 \ --api-key 'your-api-key-here' \ --max-num-seqs 16 \ --trust-remote-code \ --gpu-memory-utilization 0.85 \ # Can be changed --enforce-eager \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --port 8000 &