GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
the-fall-of-man
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
the-fall-of-man
Model Tree
Input Modalities
Output Modalities
Supported Functionality
| Parameter | Value |
|---|---|
| direction_index | 33.42 |
| attn.o_proj.max_weight | 1.41 |
| attn.o_proj.max_weight_position | 57.80 |
| attn.o_proj.min_weight | 0.00 |
| attn.o_proj.min_weight_distance | 43.35 |
| mlp.down_proj.max_weight | 1.59 |
| mlp.down_proj.max_weight_position | 40.58 |
| mlp.down_proj.min_weight | 1.53 |
| mlp.down_proj.min_weight_distance | 38.87 |
| Metric | This model | Original model (ByteDance-Seed/Seed-OSS-36B-Instruct) |
|---|---|---|
| KL divergence | 0.0040 | 0 (by definition) |
| Refusals | 3/100 | 87/100 |
[!NOTE] This model card is dedicated to the
Seed-OSS-36B-Base-Instructmodel.
Seed-OSS-36B-Base (both with and without synthetic data versions) and Seed-OSS-36B-Instruct.Seed-OSS is a series of open-source large language models developed by ByteDance's Seed Team, designed for powerful long-context, reasoning, agent and general capabilities, and versatile developer-friendly features. Although trained with only 12T tokens, Seed-OSS achieves excellent performance on several popular open benchmarks.
We release this series of models to the open-source community under the Apache-2.0 license.
[!NOTE] Seed-OSS is primarily optimized for international (i18n) use cases.
Seed-OSS adopts the popular causal language model architecture with RoPE, GQA attention, RMSNorm and SwiGLU activation.
| Seed-OSS-36B | |
| Parameters | 36B |
| Attention | GQA |
| Activation Function | SwiGLU |
| Number of Layers | 64 |
| Number of QKV Heads | 80 / 8 / 8 |
| Head Size | 128 |
| Hidden Size | 5120 |
Incorporating synthetic instruction data into pretraining leads to improved performance on most benchmarks. We adopt the version augmented with synthetic instruction data (i.e., w/ syn.) as Seed-OSS-36B-Base. We also release Seed-OSS-36B-Base-woSyn trained without such data (i.e., w/o syn.), offering the community a high-performance foundation model unaffected by synthetic instruction data.
[!NOTE] We recommend sampling with
temperature=1.1andtop_p=0.95.
Users can flexibly specify the model's thinking budget. The figure below shows the performance curves across different tasks as the thinking budget varies. For simpler tasks (such as IFEval), the model's chain of thought (CoT) is shorter, and the score exhibits fluctuations as the thinking budget increases. For more challenging tasks (such as AIME and LiveCodeBench), the model's CoT is longer, and the score improves with an increase in the thinking budget.
Here is an example with a thinking budget set to 512: during the reasoning process, the model periodically triggers self-reflection to estimate the consumed and remaining budget, and delivers the final response once the budget is exhausted or the reasoning concludes.
markdown
<seed:think>Got it, let's try to solve this problem step by step. The problem says ... ...<seed:cot_budget_reflect>I have used 129 tokens, and there are 383 tokens remaining for use.</seed:cot_budget_reflect>Using the power rule, ... ...<seed:cot_budget_reflect>I have used 258 tokens, and there are 254 tokens remaining for use.</seed:cot_budget_reflect>Alternatively, remember that ... ...<seed:cot_budget_reflect>I have used 393 tokens, and there are 119 tokens remaining for use.</seed:cot_budget_reflect>Because if ... ...<seed:cot_budget_reflect>I have exhausted my token budget, and now I will start answering the question.</seed:cot_budget_reflect></seed:think>To solve the problem, we start by using the properties of logarithms to simplify the given equations: (full answer omitted).
If no thinking budget is set (default mode), Seed-OSS will initiate thinking with unlimited length. If a thinking budget is specified, users are advised to prioritize values that are integer multiples of 512 (e.g., 512, 1K, 2K, 4K, 8K, or 16K), as the model has been extensively trained on these intervals. Models are instructed to output a direct response when the thinking budget is 0, and we recommend setting any budget below 512 to this value.
shell
pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport osimport remodel_name_or_path = "ByteDance-Seed/Seed-OSS-36B-Instruct"tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto") # You may want to use bfloat16 and/or move to GPU heremessages = [{"role": "user", "content": "How to make pasta?"},]tokenized_chat = tokenizer.apply_chat_template(messages,tokenize=True,add_generation_prompt=True,return_tensors="pt",thinking_budget=512 # control the thinking budget)outputs = model.generate(tokenized_chat.to(model.device), max_new_tokens=2048)output_text = tokenizer.decode(outputs[0])
Download Seed-OSS checkpoint to ./Seed-OSS-36B-Instruct
The generate.py script provides a simple interface for model inference with configurable options.
shell
cd inferencepython3 generate.py --model_path /path/to/model
| Parameter | Description |
|---|---|
--model_path | Path to the pretrained model directory (required) |
--prompts | Input prompts (default: sample cooking/code questions) |
--max_new_tokens | Maximum tokens to generate (default: 4096) |
--attn_implementation | Attention mechanism: flash_attention_2 (default) or eager |
--load_in_4bit/8bit |
shell
# 8-bit quantizationpython3 generate.py --model_path /path/to/model --load_in_8bit True# 4-bit quantizationpython3 generate.py --model_path /path/to/model --load_in_4bit True
shell
python3 generate.py --model_path /path/to/model --prompts "['What is machine learning?', 'Explain quantum computing']"
Use vllm >= 0.10.0 or higher for inference.
shell
VLLM_USE_PRECOMPILED=1 VLLM_TEST_USE_PRECOMPILED_NIGHTLY_WHEEL=1 pip install git+https://github.com/vllm-project/vllm.git
shell
python3 -m vllm.entrypoints.openai.api_server \--host localhost \--port 4321 \--enable-auto-tool-choice \--tool-call-parser seed_oss \--trust-remote-code \--model ./Seed-OSS-36B-Instruct \--chat-template ./Seed-OSS-36B-Instruct/chat_template.jinja \--tensor-parallel-size 8 \--dtype bfloat16 \--served-model-name seed_oss
Chat
shell
# no streampython3 inference/vllm_chat.py --max_new_tokens 4096 --thinking_budget -1# streampython3 inference/vllm_chat.py --max_new_tokens 4096 --thinking_budget -1 --stream
Tool Call
shell
# no streampython3 inference/vllm_tool_call.py --max_new_tokens 4096 --thinking_budget -1# streampython3 inference/vllm_tool_call.py --max_new_tokens 4096 --thinking_budget -1 --stream
See MODEL_CARD.
This project is licensed under Apache-2.0. See the LICENSE flie for details.
bibtex
@misc{seed2025seed-oss,author={ByteDance Seed Team},title={Seed-OSS Open-Source Models},year={2025},howpublished={\url{https://github.com/ByteDance-Seed/seed-oss}}}
Founded in 2023, ByteDance Seed Team is dedicated to crafting the industry's most advanced AI foundation models. The team aspires to become a world-class research team and make significant contributions to the advancement of science and society.
| 155K |
| Context Length | 512K |
| RoPE Base Frequency | 1e7 |
| Enable 4-bit/8-bit quantization (reduces memory usage) |
--thinking_budget | Thinking budget in tokens (default: -1 for unlimited budget) |