GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
letechlead
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
letechlead
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
The export was produced with AutoRound 0.14.2. The settings below are taken directly from quantization_config.json and config.json:
auto_round:auto_gptqauto-roundmodel.language_model.layers and mtp.layersextra_configmtp.fc is retained at 16-bit through extra_configThe exported configuration reports BF16 as the model dtype. The repository is approximately 19.02 GB decimal (17.71 GiB), including configuration, tokenizer, processor files, and SafeTensors weights.
Huihui-Qwen3.8-27B-Abliterated-INT4-W4A16-AutoRoundQwen3_5ForConditionalGenerationQwen3_5ForConditionalGeneration multimodal processor configurationThe source model retains its first 15 layers without ablation and does not modify its MTP or visual components. That is source-model information; it is not an additional transformation performed by this AutoRound export.
The following command launches this exact model repository with vLLM and exposes both the stable local alias and the full model name. The --served-model-name values are the names accepted by the OpenAI-compatible API; they do not rename the files on disk.
bash
vllm serve letechlead/Huihui-Qwen3.8-27B-Abliterated-INT4-W4A16-AutoRound \--served-model-name local Huihui-Qwen3.8-27B-Abliterated-INT4-W4A16-AutoRound \--tensor-parallel-size 2 \--max-model-len 262144 \--port 18080
This command is for the exact artifact letechlead/Huihui-Qwen3.8-27B-Abliterated-INT4-W4A16-AutoRound; do not replace the model argument with a generic path such as /models. The tensor-parallel size, maximum context that is practical, and KV-cache dtype must be chosen for the available hardware. The model supports text input and multimodal processor inputs when the installed vLLM release supports this architecture and its AutoRound/AWQ-compatible INT4 format.
Use a recent Transformers release that supports Qwen3_5ForConditionalGeneration and the model's processor:
python
from transformers import AutoModelForMultimodalLM, AutoProcessormodel_id = "letechlead/Huihui-Qwen3.8-27B-Abliterated-INT4-W4A16-AutoRound"processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForMultimodalLM.from_pretrained(model_id,device_map="auto",dtype="auto",)
For text-only requests, pass text through the processor. For image or video requests, pass the media together with the text prompt according to the Qwen3.5 Transformers documentation.
model-00001-of-00007.safetensors through model-00007-of-00007.safetensors: sharded model weightsmodel_extra_tensors.safetensors: additional exported tensors, including the exported MTP pathmodel.safetensors.index.json: weight-to-shard index and parameter metadataquantization_config.json: AutoRound quantization metadataconfig.json: model architecture, text configuration, vision configuration, and context configurationtokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer and chat templateThis model inherits the source model's abliterated behavior and may produce sensitive, controversial, or otherwise unsafe output. It is intended for research, evaluation, and controlled use. Users are responsible for complying with applicable laws, platform rules, and organizational policies, and for reviewing outputs before relying on them.
This derived model is published under Apache-2.0, matching the license declared by the source model card. Review the source model card and repository for the original model's terms and attribution requirements.
Quantization artifact published by LeTechLead.
processor_config.jsonpreprocessor_config.json