GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
prasannaJagadesh
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
prasannaJagadesh
Model Tree
Base
NemoStation/Marlin-2B
Quantized
this model
Input Modalities
Text
Image
Video
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
We quantized the unquantized 16-bit baseline model to 4-bit using the GPTQ algorithm via the gptqmodel library. The conversion process adhered to the following technical recipe:
4 (uniform 4-bit quantization).128 (balances precision retention and packing efficiency).desc_act): True (orders columns based on activation scales for improved quality).sym): True.lm_head) was kept in 16-bit to prevent output vocabulary logit degradation.To calibrate the quantization scales without causing resource depletion on standard workstation setups:
--max-gpu-memory 5.0GiB), forcing Hugging Face and PyTorch to manage intermediate weights efficiently and prevent out-of-memory (OOM) crashes during the scale computation phase.transformers execution, representing a ~5.3% speedup over BitsAndBytes 4-bit (bnb4 at 12.44 seconds).The quantized model was evaluated on a local benchmark suite to assess quality retention and latency/VRAM efficiency against the standard BitsAndBytes 4-bit (bnb4) format.
Below are the visualization plots showing the comparison of quality and speed metrics between the BitsAndBytes 4-bit (bnb4) and GPTQ 4-bit (gptq) models.
[!NOTE] Quality Retention: The GPTQ model retains approximately 93% of the overall accuracy of the
bnb4baseline. Interestingly, in dense captioning tasks, the GPTQ model shows a slight advantage overbnb4in both lexical similarity (caption score: 0.4726 vs 0.4654) and keyword coverage (0.7318 vs 0.7091).VRAM Footprint: GPTQ requires a slightly higher peak allocated VRAM footprint during runtime (4668.44 MB compared to 4201.46 MB for
bnb4). Both models remain fully compatible with 6GB VRAM GPUs.
Marlin ships with custom modeling code that adds high-level methods (caption and find) directly onto the model object.
Ensure you use Python 3.11 or newer and install the dependencies required by the
GPTQ inference path in run.py:
bash
pip install "transformers>=5.7.0" "torch>=2.11.0" "gptqmodel>=5.0.0" "accelerate>=0.26.0" "qwen-vl-utils>=0.0.14" torchcodec torchvision av numpy pillow
bitsandbytes>=0.43.0 is only needed when using the runner with a
BitsAndBytes checkpoint or --load-in-4bit; it is not required for this
pre-quantized GPTQ checkpoint.
For GPTQ checkpoints, we must load the model via GPTQModel to correctly support the mixed layout of quantized layers and dense linear-attention projections:
python
import torchfrom gptqmodel import GPTQModel# Load the model via GPTQModelloaded = GPTQModel.load("NemoStation/Marlin-2B-GPTQ",device_map={"": "cuda:0"},dtype=torch.bfloat16,trust_remote_code=True,)# Extract the inner Marlin model (which supports caption and find methods)marlin = loaded.model
marlin.caption()Generates a spatial scene description followed by atomic, time-ranged events.
python
result = marlin.caption("video.mp4")print("Scene Summary:\n", result["scene"])print("\nTimeline Events:")for event in result["events"]:print(f"<{event['start']:.1f}s - {event['end']:.1f}s> {event['description']}")
marlin.find()Locates the specific time interval in which a natural language event query occurs.
python
result = marlin.find("video.mp4", event="a person enters the room")if result["format_ok"]:print(f"Event found from {result['span'][0]}s to {result['span'][1]}s.")else:print("Failed to locate event.")