GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
useful-quants
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
useful-quants
Model Tree
microsoft/FastContext-1.0-4B-RL
Input Modalities
Output Modalities
Supported Functionality
microsoft/FastContext-1.0-4B-RLbd530b8e1f9a59b12e77061d9fe1e018f6eabe42W4A16128bfloat16llmcompressor + compressed-tensorscompressed-tensors962440967.37347.1517-3.01%The calibration/eval prompts are repo-exploration and tool-calling flavored so the quantizer sees text that matches the deployed FastContext use case rather than generic chat data.
python
from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "useful-quants/FastContext-1.0-4B-RL-w4a16-g128"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")bash
vllm serve useful-quants/FastContext-1.0-4B-RL-w4a16-g128 --quantization compressed-tensorsThis export was smoke-tested locally with vllm 0.23.0 after upload.
Launch command used:
bash
python -m vllm.entrypoints.openai.api_server \ --model /path/to/FastContext-1.0-4B-RL-w4a16-g128 \ --quantization compressed-tensors \ --max-model-len 4096 \ --gpu-memory-utilization 0.7 \ --enforce-eager \ --host 127.0.0.1 \ --port 8011Observed smoke results:
/v1/models returned the model successfully.Return exactly: calibration-pass returned calibration-pass.Write one rg command to search for OAuth callback state validation. No explanation. returned a valid rg command.recipe.yaml: exact GPTQ recipe used for this exportcalibration_text.jsonl: chat-rendered calibration corpuseval_text.jsonl: held-out evaluation corpusquantization_report.json: environment, perplexity, and generation checksvllm_smoke_test.json: exact local vLLM smoke-test command and responsesfrom transformers import AutoTokenizer, AutoModelForCausalLM model_id = "useful-quants/FastContext-1.0-4B-RL-w4a16-g128"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")vllm serve useful-quants/FastContext-1.0-4B-RL-w4a16-g128 --quantization compressed-tensorspython -m vllm.entrypoints.openai.api_server \ --model /path/to/FastContext-1.0-4B-RL-w4a16-g128 \ --quantization compressed-tensors \ --max-model-len 4096 \ --gpu-memory-utilization 0.7 \ --enforce-eager \ --host 127.0.0.1 \ --port 8011