zerank-2-reranker-nvfp4 API & Inference Endpoint

🚀 量化详情

基础模型: zeroentropy/zerank-2-reranker
量化框架: llm-compressor (oneshot PTQ)
量化格式: NVFP4 (Compressed-Tensors)
保护层: lm_head 与 model.embed_tokens 保持原始精度未被量化
校准配置: 512 条中文短文本对进行数据校准

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

由于本模型属于 CausalLM 架构并以输出 "Yes" 的 Logit 作为打分，在启动 vLLM 时需要传入 --chat-template，--convert classify 与 --hf-overrides：

bash

vllm serve liudev/zerank-2-reranker-nvfp4 \  --dtype bfloat16 \  --served-model-name zerank-2-reranker-nvfp4 \  --host 0.0.0.0 \  --port 8000 \  --runner pooling \  --convert classify \  --hf-overrides '{"method":"no_post_processing","classifier_from_token":["Yes"],"num_labels":1,"use_sep_token":false}' \  --chat-template $(python3 -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('liudev/zerank-2-reranker-fp8', 'chat_template.jinja'))") \  --default-chat-template-kwargs '{"add_generation_prompt":true}' \  --quantization compressed-tensors \  --max-model-len 1024 \  --gpu-memory-utilization 0.5 \  --max-num-seqs 128  # --chat-template 指定chat_template.jinja文件所在路径# 例如：/root/.cache/huggingface/hub/models--liudev--zerank-2-reranker-nvfp4/snapshots/xxxxxxxxxxxxxxxxxxxxxxxxxx/chat_template.jinja

客户端异步调用示例 (`/v1/rerank`)

服务启动后，可直接使用标准兼容接口发起重排：

bash

curl http://127.0.0.1:8000/v1/rerank \  -H 'Content-Type: application/json' \  -d '{    "model": "zerank-2-reranker-nvfp4",    "query": "What is 2+2?",    "documents": [      "4",      "The answer is definitely 1 million"    ],    "use_activation": false,    "chat_template_kwargs": {"add_generation_prompt": true}  }' # {"id":"score-86aea1df1b120347","model":"zerank-2-reranker-nvfp4","usage":{"prompt_tokens":48,"total_tokens":48},"results":[{"index":0,"document":{"text":"4","multi_modal":null},"relevance_score":5.027203559875488},{"index":1,"document":{"text":"The answer is definitely 1 million","multi_modal":null},"relevance_score":-3.139714241027832}]}

方案 2：Python 本地直接调用 (Transformers)

无需启动外部服务，在 Python 单进程中即可直接执行：

bash

pip install torch transformers accelerate compressed-tensors

运行环境

bash

pip show torchpython --versionnvcc --version

Name: torchVersion: 2.13.0Summary: Tensors and Dynamic neural networks in Python with strong GPU accelerationHome-page: https://pytorch.orgAuthor: Author-email: PyTorch Team <packages@pytorch.org>License: Location: /usr/local/lib/python3.13/dist-packagesRequires: cuda-bindings, cuda-toolkit, filelock, fsspec, jinja2, networkx, nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, setuptools, sympy, triton, typing-extensionsRequired-by: accelerate, auto-round, compressed-tensors, fastai, flashinfer-python, humming-kernels, instanttensor, llmcompressor, peft, quack-kernels, sentence-transformers, tilelang, timm, tokenspeed-mla, torch_c_dlpack_ext, torchdata, torchvision, vllm, xgrammarPython 3.13.15nvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2025 NVIDIA CorporationBuilt on Wed_Aug_20_01:58:59_PM_PDT_2025Cuda compilation tools, release 13.0, V13.0.88Build cuda_13.0.r13.0/compiler.36424714_0

安装flash_attn

bash

wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.47/flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whluv pip install flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whl --no-build-isolation

python

import flash_attn;print('成功安装，版本号为:', flash_attn.__version__)# 成功安装，版本号为: 2.8.3

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer class ZerankLocalReranker:    def __init__(self, model_path="zeroentropy/zerank-2-reranker", device="cuda"):        self.device = device        print(f"正在加载模型: {model_path} ...")        self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)        self.model = AutoModelForCausalLM.from_pretrained(            model_path,            dtype=torch.bfloat16,            device_map=device,            attn_implementation="flash_attention_2",            trust_remote_code=True        )        self.model.eval()        # 9454 是 "Yes" 对应的 token_id        self.yes_token_id = self.tokenizer.encode("Yes", add_special_tokens=False)     @torch.inference_mode()    def predict(self, pairs):        if not pairs:            return []         prompts = [            self.tokenizer.apply_chat_template(                [{"role": "query", "content": q}, {"role": "document", "content": d}],                tokenize=False,                add_generation_prompt=True            )            for q, d in pairs        ]                # 1. 逐条编码，完全不启用 padding        encoded = [            self.tokenizer.encode(p, truncation=True, max_length=1024)             for p in prompts        ]        lengths = [len(seq) for seq in encoded]                # 2. 扁平拼接为一条 1D 序列 (batch_size = 1)        flat_input_ids = [token for seq in encoded for token in seq]        input_ids = torch.tensor([flat_input_ids], dtype=torch.long, device=self.device)                # 3. 构造各自重置的 position_ids (每条样本在自己的区间内都是 0, 1, 2... 递增)        # FlashAttention-2 会依据 position_ids 中的重置突变点，自动识别样本边界并调用 varlen 算子切分        position_ids = torch.cat([            torch.arange(length, device=self.device) for length in lengths        ]).unsqueeze(0)                # 4. 注意：切勿传入 attention_mask (设为 None)        outputs = self.model(            input_ids=input_ids,            position_ids=position_ids,            attention_mask=None        )                # 5. 按照每条样本的实际长度，定位各自最后一个 token 的位置索引        end_indices = torch.tensor(lengths, device=self.device).cumsum(dim=0) - 1                # 提取各个样本对应 "Yes" 的 Logit，保持原有的输出维度 [[score1], [score2], ...]        last_token_logits = outputs.logits[0, end_indices][:, self.yes_token_id]        return last_token_logits.cpu().tolist()  reranker = ZerankLocalReranker('liudev/zerank-2-reranker-nvfp4') query_documents = [    ("What is 2+2?", "4"),    ("What is 2+2?", "The answer is definitely 1 million"),] scores = reranker.predict(query_documents)print(scores)# [[4.28125], [-3.34375]] import numpy as np def sigmoid(x):    return 1 / (1 + np.exp(-x)) scores = np.array(scores).flatten()probabilities = sigmoid(scores / 5)print(probabilities)# [0.70187658 0.33877679]  query = "什么是大模型量化？"documents = [    "大模型量化是一种将浮点权重转换为低精度表示的技术，旨在加速推理并降低显存。",    "今天的天气晴空万里，非常适合户外运动。"]query_documents = [(query, doc) for doc in documents]scores = reranker.predict(query_documents)print(scores)# [[14.8125], [-13.75]]

引用与感谢

原模型：zeroentropy/zerank-2-reranker
量化工具：vLLM Project / llm-compressor

Releasing zeroentropy/zerank-2

In search engines, rerankers are crucial for improving the accuracy of your retrieval system.

However, SOTA rerankers are closed-source and proprietary. At ZeroEntropy, we've trained a SOTA reranker outperforming closed-source competitors, and we're launching our model here on HuggingFace.

This reranker outperforms proprietary rerankers such as cohere-rerank-v3.5 and gemini-2.5-flash across a wide variety of domains, including finance, legal, code, STEM, medical, and conversational data.

At ZeroEntropy we've developed an innovative multi-stage pipeline that models query-document relevance scores as adjusted Elo ratings. See our Technical Report (https://arxiv.org/abs/2509.12541 ) for more details.

This model is released under the Apache License 2.0.

Model Details

Table with columns: Property, Value
Property	Value
Parameters	4B
Context Length	32,768 tokens (32k)
Base Model	Qwen/Qwen3-4B
License	Apache-2.0

How to Use

Breaking change (May 2026): model.predict() now returns raw "Yes" logits instead of sigmoid'd probabilities in [0, 1]. Rankings are unchanged. To recover the previous 0-1 score, apply (scores / 5).sigmoid() — see the example below. Loading no longer requires trust_remote_code=True; passing it is harmless.

Using Sentence Transformers

Install Sentence Transformers:

bash

pip install sentence_transformers

Then load the model and score query/document pairs. model.predict returns the raw "Yes" logit per pair; rankings can be used directly. To map the logits to a 0-1 score range, apply a temperature-scaled sigmoid: sigmoid(score / 5).

python

from sentence_transformers import CrossEncoder model = CrossEncoder("zeroentropy/zerank-2") query_documents = [    ("What is 2+2?", "4"),    ("What is 2+2?", "The answer is definitely 1 million"),] scores = model.predict(query_documents, convert_to_tensor=True)print(scores)# tensor([ 5.4062, -4.5000], device='cuda:0', dtype=torch.bfloat16) # Optional: convert to 0-1 probabilitiesprobabilities = (scores / 5).sigmoid()print(probabilities)# tensor([0.7461, 0.2891], device='cuda:0', dtype=torch.bfloat16)

You can also use model.rank to score and sort a list of documents for a single query:

python

rankings = model.rank(    "What is 2+2?",    ["4", "The answer is definitely 1 million"],)for r in rankings:    print(r)# {'corpus_id': 0, 'score': np.float32(5.40625)}# {'corpus_id': 1, 'score': np.float32(-4.5)}

The model can also be inferenced using ZeroEntropy's /models/rerank endpoint, and on AWS Marketplace.

Evaluations

NDCG@10 scores between zerank-2 and competing closed-source proprietary rerankers. Since we are evaluating rerankers, OpenAI's text-embedding-3-small is used as an initial retriever for the Top 100 candidate documents.

Table with columns: Domain, OpenAI embeddings, ZeroEntropy zerank-2, ZeroEntropy zerank-1, Gemini 2.5 Flash (Listwise), Cohere rerank-3.5
Domain	OpenAI embeddings	ZeroEntropy zerank-2	ZeroEntropy zerank-1	Gemini 2.5 Flash (Listwise)	Cohere rerank-3.5
Web	0.3819	0.6346	0.6069	0.5765	0.5594
Conversational	0.4305	0.6140	0.5801	0.6021	0.5648
STEM & Logic

License

This model is licensed under the Apache License 2.0.

🚀 量化详情

基础模型: zeroentropy/zerank-2-reranker
量化框架: llm-compressor (oneshot PTQ)
量化格式: NVFP4 (Compressed-Tensors)
保护层: lm_head 与 model.embed_tokens 保持原始精度未被量化
校准配置: 512 条中文短文本对进行数据校准

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

由于本模型属于 CausalLM 架构并以输出 "Yes" 的 Logit 作为打分，在启动 vLLM 时需要传入 --chat-template，--convert classify 与 --hf-overrides：

bash

vllm serve liudev/zerank-2-reranker-nvfp4 \  --dtype bfloat16 \  --served-model-name zerank-2-reranker-nvfp4 \  --host 0.0.0.0 \  --port 8000 \  --runner pooling \  --convert classify \  --hf-overrides '{"method":"no_post_processing","classifier_from_token":["Yes"],"num_labels":1,"use_sep_token":false}' \  --chat-template $(python3 -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('liudev/zerank-2-reranker-fp8', 'chat_template.jinja'))") \  --default-chat-template-kwargs '{"add_generation_prompt":true}' \  --quantization compressed-tensors \  --max-model-len 1024 \  --gpu-memory-utilization 0.5 \  --max-num-seqs 128  # --chat-template 指定chat_template.jinja文件所在路径# 例如：/root/.cache/huggingface/hub/models--liudev--zerank-2-reranker-nvfp4/snapshots/xxxxxxxxxxxxxxxxxxxxxxxxxx/chat_template.jinja

客户端异步调用示例 (`/v1/rerank`)

服务启动后，可直接使用标准兼容接口发起重排：

bash

curl http://127.0.0.1:8000/v1/rerank \  -H 'Content-Type: application/json' \  -d '{    "model": "zerank-2-reranker-nvfp4",    "query": "What is 2+2?",    "documents": [      "4",      "The answer is definitely 1 million"    ],    "use_activation": false,    "chat_template_kwargs": {"add_generation_prompt": true}  }' # {"id":"score-86aea1df1b120347","model":"zerank-2-reranker-nvfp4","usage":{"prompt_tokens":48,"total_tokens":48},"results":[{"index":0,"document":{"text":"4","multi_modal":null},"relevance_score":5.027203559875488},{"index":1,"document":{"text":"The answer is definitely 1 million","multi_modal":null},"relevance_score":-3.139714241027832}]}

方案 2：Python 本地直接调用 (Transformers)

无需启动外部服务，在 Python 单进程中即可直接执行：

bash

pip install torch transformers accelerate compressed-tensors

运行环境

bash

pip show torchpython --versionnvcc --version

Name: torchVersion: 2.13.0Summary: Tensors and Dynamic neural networks in Python with strong GPU accelerationHome-page: https://pytorch.orgAuthor: Author-email: PyTorch Team <packages@pytorch.org>License: Location: /usr/local/lib/python3.13/dist-packagesRequires: cuda-bindings, cuda-toolkit, filelock, fsspec, jinja2, networkx, nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, setuptools, sympy, triton, typing-extensionsRequired-by: accelerate, auto-round, compressed-tensors, fastai, flashinfer-python, humming-kernels, instanttensor, llmcompressor, peft, quack-kernels, sentence-transformers, tilelang, timm, tokenspeed-mla, torch_c_dlpack_ext, torchdata, torchvision, vllm, xgrammarPython 3.13.15nvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2025 NVIDIA CorporationBuilt on Wed_Aug_20_01:58:59_PM_PDT_2025Cuda compilation tools, release 13.0, V13.0.88Build cuda_13.0.r13.0/compiler.36424714_0

Name: torchVersion: 2.13.0Summary: Tensors and Dynamic neural networks in Python with strong GPU accelerationHome-page: https://pytorch.orgAuthor: Author-email: PyTorch Team <packages@pytorch.org>License: Location: /usr/local/lib/python3.13/dist-packagesRequires: cuda-bindings, cuda-toolkit, filelock, fsspec, jinja2, networkx, nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, setuptools, sympy, triton, typing-extensionsRequired-by: accelerate, auto-round, compressed-tensors, fastai, flashinfer-python, humming-kernels, instanttensor, llmcompressor, peft, quack-kernels, sentence-transformers, tilelang, timm, tokenspeed-mla, torch_c_dlpack_ext, torchdata, torchvision, vllm, xgrammarPython 3.13.15nvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2025 NVIDIA CorporationBuilt on Wed_Aug_20_01:58:59_PM_PDT_2025Cuda compilation tools, release 13.0, V13.0.88Build cuda_13.0.r13.0/compiler.36424714_0

安装flash_attn

bash

wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.47/flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whluv pip install flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whl --no-build-isolation

python

import flash_attn;print('成功安装，版本号为:', flash_attn.__version__)# 成功安装，版本号为: 2.8.3

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer class ZerankLocalReranker:    def __init__(self, model_path="zeroentropy/zerank-2-reranker", device="cuda"):        self.device = device        print(f"正在加载模型: {model_path} ...")        self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)        self.model = AutoModelForCausalLM.from_pretrained(            model_path,            dtype=torch.bfloat16,            device_map=device,            attn_implementation="flash_attention_2",            trust_remote_code=True        )        self.model.eval()        # 9454 是 "Yes" 对应的 token_id        self.yes_token_id = self.tokenizer.encode("Yes", add_special_tokens=False)     @torch.inference_mode()    def predict(self, pairs):        if not pairs:            return []         prompts = [            self.tokenizer.apply_chat_template(                [{"role": "query", "content": q}, {"role": "document", "content": d}],                tokenize=False,                add_generation_prompt=True            )            for q, d in pairs        ]                # 1. 逐条编码，完全不启用 padding        encoded = [            self.tokenizer.encode(p, truncation=True, max_length=1024)             for p in prompts        ]        lengths = [len(seq) for seq in encoded]                # 2. 扁平拼接为一条 1D 序列 (batch_size = 1)        flat_input_ids = [token for seq in encoded for token in seq]        input_ids = torch.tensor([flat_input_ids], dtype=torch.long, device=self.device)                # 3. 构造各自重置的 position_ids (每条样本在自己的区间内都是 0, 1, 2... 递增)        # FlashAttention-2 会依据 position_ids 中的重置突变点，自动识别样本边界并调用 varlen 算子切分        position_ids = torch.cat([            torch.arange(length, device=self.device) for length in lengths        ]).unsqueeze(0)                # 4. 注意：切勿传入 attention_mask (设为 None)        outputs = self.model(            input_ids=input_ids,            position_ids=position_ids,            attention_mask=None        )                # 5. 按照每条样本的实际长度，定位各自最后一个 token 的位置索引        end_indices = torch.tensor(lengths, device=self.device).cumsum(dim=0) - 1                # 提取各个样本对应 "Yes" 的 Logit，保持原有的输出维度 [[score1], [score2], ...]        last_token_logits = outputs.logits[0, end_indices][:, self.yes_token_id]        return last_token_logits.cpu().tolist()  reranker = ZerankLocalReranker('liudev/zerank-2-reranker-nvfp4') query_documents = [    ("What is 2+2?", "4"),    ("What is 2+2?", "The answer is definitely 1 million"),] scores = reranker.predict(query_documents)print(scores)# [[4.28125], [-3.34375]] import numpy as np def sigmoid(x):    return 1 / (1 + np.exp(-x)) scores = np.array(scores).flatten()probabilities = sigmoid(scores / 5)print(probabilities)# [0.70187658 0.33877679]  query = "什么是大模型量化？"documents = [    "大模型量化是一种将浮点权重转换为低精度表示的技术，旨在加速推理并降低显存。",    "今天的天气晴空万里，非常适合户外运动。"]query_documents = [(query, doc) for doc in documents]scores = reranker.predict(query_documents)print(scores)# [[14.8125], [-13.75]]

引用与感谢

原模型：zeroentropy/zerank-2-reranker
量化工具：vLLM Project / llm-compressor

Releasing zeroentropy/zerank-2

In search engines, rerankers are crucial for improving the accuracy of your retrieval system.

However, SOTA rerankers are closed-source and proprietary. At ZeroEntropy, we've trained a SOTA reranker outperforming closed-source competitors, and we're launching our model here on HuggingFace.

This model is released under the Apache License 2.0.

Model Details

Table with columns: Property, Value
Property	Value
Parameters	4B
Context Length	32,768 tokens (32k)
Base Model	Qwen/Qwen3-4B
License	Apache-2.0

How to Use

Breaking change (May 2026): model.predict() now returns raw "Yes" logits instead of sigmoid'd probabilities in [0, 1]. Rankings are unchanged. To recover the previous 0-1 score, apply (scores / 5).sigmoid() — see the example below. Loading no longer requires trust_remote_code=True; passing it is harmless.

Using Sentence Transformers

Install Sentence Transformers:

bash

pip install sentence_transformers

python

from sentence_transformers import CrossEncoder model = CrossEncoder("zeroentropy/zerank-2") query_documents = [    ("What is 2+2?", "4"),    ("What is 2+2?", "The answer is definitely 1 million"),] scores = model.predict(query_documents, convert_to_tensor=True)print(scores)# tensor([ 5.4062, -4.5000], device='cuda:0', dtype=torch.bfloat16) # Optional: convert to 0-1 probabilitiesprobabilities = (scores / 5).sigmoid()print(probabilities)# tensor([0.7461, 0.2891], device='cuda:0', dtype=torch.bfloat16)

You can also use model.rank to score and sort a list of documents for a single query:

python

rankings = model.rank(    "What is 2+2?",    ["4", "The answer is definitely 1 million"],)for r in rankings:    print(r)# {'corpus_id': 0, 'score': np.float32(5.40625)}# {'corpus_id': 1, 'score': np.float32(-4.5)}

The model can also be inferenced using ZeroEntropy's /models/rerank endpoint, and on AWS Marketplace.

Evaluations

Table with columns: Domain, OpenAI embeddings, ZeroEntropy zerank-2, ZeroEntropy zerank-1, Gemini 2.5 Flash (Listwise), Cohere rerank-3.5
Domain	OpenAI embeddings	ZeroEntropy zerank-2	ZeroEntropy zerank-1	Gemini 2.5 Flash (Listwise)	Cohere rerank-3.5
Web	0.3819	0.6346	0.6069	0.5765	0.5594
Conversational	0.4305	0.6140	0.5801	0.6021	0.5648
STEM & Logic

License

This model is licensed under the Apache License 2.0.

zerank-2-reranker-nvfp4

README

🚀 量化详情

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

客户端异步调用示例 (`/v1/rerank`)

方案 2：Python 本地直接调用 (Transformers)

引用与感谢

Releasing zeroentropy/zerank-2

Model Details

How to Use

Using Sentence Transformers

Evaluations

License

Explore FriendliAI today

README

🚀 量化详情

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

客户端异步调用示例 (`/v1/rerank`)

方案 2：Python 本地直接调用 (Transformers)

引用与感谢

Releasing zeroentropy/zerank-2

Model Details

How to Use

Using Sentence Transformers

Evaluations

License

zerank-2-reranker-nvfp4

README

🚀 量化详情

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

客户端异步调用示例 (/v1/rerank)

方案 2：Python 本地直接调用 (Transformers)

引用与感谢

Releasing zeroentropy/zerank-2

Model Details

How to Use

Using Sentence Transformers

Evaluations

License

Explore FriendliAI today

README

🚀 量化详情

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务

客户端异步调用示例 (/v1/rerank)

方案 2：Python 本地直接调用 (Transformers)

引用与感谢

Releasing zeroentropy/zerank-2

Model Details

How to Use

Using Sentence Transformers

Evaluations

License

客户端异步调用示例 (`/v1/rerank`)

客户端异步调用示例 (`/v1/rerank`)