GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
useful-quants
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
useful-quants
Model Tree
microsoft/FastContext-1.0-4B-RL
Input Modalities
Output Modalities
Supported Functionality
microsoft/FastContext-1.0-4B-RLbd530b8e1f9a59b12e77061d9fe1e018f6eabe42W4A16128bfloat16llmcompressor + compressed-tensorscompressed-tensors962440967.37347.1517-3.01%The calibration/eval prompts are repo-exploration and tool-calling flavored so the quantizer sees text that matches the deployed FastContext use case rather than generic chat data.
python
from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "useful-quants/FastContext-1.0-4B-RL-w4a16-g128"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")bash
vllm serve useful-quants/FastContext-1.0-4B-RL-w4a16-g128 --quantization compressed-tensorsThis export was smoke-tested locally with vllm 0.23.0 after upload.
Launch command used:
bash
python -m vllm.entrypoints.openai.api_server \ --model /path/to/FastContext-1.0-4B-RL-w4a16-g128 \ --quantization compressed-tensors \ --max-model-len 4096 \ --gpu-memory-utilization 0.7 \ --enforce-eager \ --host 127.0.0.1 \ --port 8011Observed smoke results:
/v1/models returned the model successfully.Return exactly: calibration-pass returned calibration-pass.Write one rg command to search for OAuth callback state validation. No explanation. returned a valid rg command.recipe.yaml: exact GPTQ recipe used for this exportcalibration_text.jsonl: chat-rendered calibration corpuseval_text.jsonl: held-out evaluation corpusquantization_report.json: environment, perplexity, and generation checksvllm_smoke_test.json: exact local vLLM smoke-test command and responsesfrom transformers import AutoTokenizer, AutoModelForCausalLM model_id = "useful-quants/FastContext-1.0-4B-RL-w4a16-g128"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")vllm serve useful-quants/FastContext-1.0-4B-RL-w4a16-g128 --quantization compressed-tensorspython -m vllm.entrypoints.openai.api_server \ --model /path/to/FastContext-1.0-4B-RL-w4a16-g128 \ --quantization compressed-tensors \ --max-model-len 4096 \ --gpu-memory-utilization 0.7 \ --enforce-eager \ --host 127.0.0.1 \ --port 8011