suryatmodulus

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16

README

License: openmdw-1.1

Model Overview

Model Developer: NVIDIA Corporation

Description

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 is the base vision-language checkpoint of the Nemotron 3.5 Super family. It provides a starting point for continued training, domain adaptation, and developing custom post-trained vision-language models.

The language backbone is initialized from Nemotron 3 Super Base and further trained through distillation from Nemotron 3 Ultra Base. The model underwent continued pre-training on indic languages, long-context training, and software-engineering mid-training. A vision encoder and connector extend the language backbone to visual inputs.

The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers with a C-RADIO vision encoder and a vision-language connector. It has 120B total parameters and 12B active parameters and supports context lengths up to 1M tokens. Multi-Token Prediction (MTP) enables native speculative decoding.

This model is ready for commercial or non-commercial use.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1.

Benchmarks

The following results evaluate the Nemotron 3.5 Super text-only base backbone, before vision-language training.

Table with columns: Benchmark, Metric, Score (%)
Benchmark	Metric	Score (%)
MMLU	Exact match	84.14
MMLU-Pro	Exact match, TigerLab extraction	76.25
AGIEval-En	CoT exact match	75.65
SuperGPQA	CoT	47.65
GPQA Diamond	5-shot, CoT exact match	66.16
GSM8K	8-shot, CoT exact match

Deployment Geography: Global

Use Case

This model is intended for developers and researchers building custom vision-language models through continued pre-training, supervised fine-tuning, reinforcement learning, and distillation.

Release Date

Hugging Face - 10/15/2026 via Hugging Face

Reference(s)

Model Architecture

Architecture Type: Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)
Network Architecture: Nemotron Hybrid MoE with 512 routed experts per MoE layer, a C-RADIO vision encoder, and a vision-language connector
Number of Model Parameters: 120B total / 12B active

Model Design

The model combines sparse expert computation, attention, and Mamba-2 layers to support long-context processing. The vision encoder and connector turn visual inputs into representations processed by the language backbone. A shared MTP head predicts future tokens recursively for speculative decoding.

The model supports up to 1M tokens of context. The prompt, visual inputs, conversation history, tool results, and generated response share this context window. Individual deployment recipes may use a smaller context window to accommodate the available GPU memory and concurrency.

Training Methodology

Stage 1: Foundation and Multi-Token Prediction

The language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens while the backbone remains frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base.

Stage 2: Continued Pre-Training and Mid-Training

The model undergoes 100B tokens of Indic continued pre-training, a 35B-token extension to 1M context, 67B tokens of software-engineering mid-training at 256K context, and a final 9B-token long-context recovery stage. Software-engineering data includes repositories, issue–pull-request pairs, commit diffs, build and CI logs, and agent trajectories.

Stage 3: Vision-Language Training

A C-RADIO vision encoder and connector extend the language backbone to visual inputs, including images, document pages, and video frames.

Input

Input Type(s): Text, images, and video
Input Format(s): Text strings, images, and sampled video frames
- Video: mp4, up to 2 minutes.
- Image: Red, Green, Blue (RGB) (jpeg, png)
Input Parameters:
- Video: Three-Dimensional (3D)
- Image: Two-Dimensional (2D)
- Text: One-Dimensional (1D)
Other Properties Related to Input: Maximum context length up to 1M tokens. Documents can be supplied as page images with accompanying text.

Output

Output Type(s): Text
Output Format: String
Output Parameters: One-dimensional sequences
Other Properties Related to Output: Maximum context length up to 1M tokens.

Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s): PyTorch; Megatron-LM; NeMo
Hardware:
- NVIDIA Blackwell
- NVIDIA Hopper
Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

v1.0 - GA

Training, Testing, and Evaluation Datasets

Training

Data Modality: Text, images, and video
Foundation Training Data Size: More than 20 trillion tokens
Image Training Data Size: 1 Million to 1 Billion Images
Text Training Data Size: More than 10 Trillion Tokens
Video Training Data Size: 10,000 to 1 Million Hours
Foundation Data Collection Period: 2013 to December 2025
Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic

The foundation corpus contains curated and synthetic text covering English, 19 other spoken languages, and 43 programming languages. Sources include webpages, dialogue, articles, code, and other written materials across math, science, legal, finance, and general knowledge domains.

For more information about the datasets used to train this model, please see the Public Summary of Training Content

Base Pre-Training Corpus (Nemotron 3 Foundation)

The foundation of the model is trained on the Nemotron 3 corpus, comprising the following datasets from the Nemotron Pretraining Datasets collection:

Table with columns: Dataset Collection, Token Counts, Description
Dataset Collection	Token Counts	Description
Nemotron-CC-v2 & v2.1	9.1T	A massive collection of English web data filtered from Common Crawl, including 2.5T+ tokens of new organic, translated, and synthetically rephrased content.
Nemotron-CC-Code-v1	427.9B	High-quality code tokens extracted from Common Crawl using the Lynx + LLM pipeline to preserve structure and equations.
Nemotron-Pretraining-Code-v1 & v2 & v3	1.7T	Curated GitHub code references with multi-stage filtering, deduplication, and large-scale synthetic code data.
Nemotron-CC-Math-v1	133.3B	High-quality math pre-training dataset preserving LaTeX formatting and mathematical structures.

Public Datasets

Table with columns: Dataset, Collection Period
Dataset	Collection Period
GSM8K	4/23/2025
CC-NEWS	4/23/2025
Common Crawl	4/23/2025
Wikimedia	4/23/2025
Bespoke-Stratos-17k

Crawled and Scraped from Online Sources by NVIDIA

The English Common Crawl data was downloaded from the Common Crawl Foundation (see their FAQ for details on their crawling) and includes the snapshots CC-MAIN-2013-20 through CC-MAIN-2025-13. The data was subsequently deduplicated and filtered in various ways used in the Nemotron-CC data pipeline. Additionally, we extracted data for fifteen languages from the following three Common Crawl snapshots: CC-MAIN-2024-51, CC-MAIN-2025-08, CC-MAIN-2025-18. The fifteen languages included were Arabic, Chinese, Danish, Dutch, French, German, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Thai. As we did not have reliable multilingual model-based quality classifiers available, we applied just heuristic filtering instead—similar to what we did for lower quality English data in the Nemotron-CC pipeline, but selectively removing some filters for some languages that did not work well. Deduplication was done in the same way as for Nemotron-CC.

The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API. Each crawl was operated in accordance with the rate limits set by its respective source, either GitHub or S3. We collect raw source code and subsequently remove any having a license which does not exist in our permissive-license set.

Table with columns: Dataset, Modality, Dataset Size, Collection Period, Collecting Organisation
Dataset	Modality	Dataset Size	Collection Period	Collecting Organisation
English Common Crawl	Text	3.36T	4/8/2025	NVIDIA Advanced Deep Learning Research
English Common Crawl 1.1	Text	Not disclosed	10/2/2025	NVIDIA Advanced Deep Learning Research
Multilingual Common Crawl	Text	812.7B	5/1/2025

Private Non-publicly Accessible Datasets of Third Parties

Table with columns: Dataset, Model(s) used
Dataset	Model(s) used
Global Regulation	Unknown
TAUS Translation Memory	Unknown
Scale HLE	Unknown
HackerRank Coding	Unknown
RL data for Search	Gemini 3; GPT-5
KeenSight Batch 1, Relabeled Subset	Qwen3.5-122B-A10B
KeenSight Batch 1, Remaining Subset	Qwen3.5-122B-A10B
Camio Multilingual OCR and Document Layout	Undisclosed
Chinese Visual Question Answering, Relabeled Subset

Private Non-publicly Accessible Datasets by NVIDIA

Table with columns: Dataset, Model(s) used
Dataset	Model(s) used
Simple Minesweeper	Undisclosed
Simple Sudoku	Undisclosed
Multitool Typewriter Hard	Undisclosed
Machine Translation of News Commentary and TAUS Translation Memory	Undisclosed
Machine Translation of STEM -	Qwen2.5-14B-Instruct
Competitive Coding RL data from Nemotron Cascade	Undisclosed
Long context RL	Undisclosed
Single-step SWE RL for patch generation

NVIDIA-Sourced Synthetic Datasets

Table with columns: Dataset, Modality, Dataset Size, Seed Dataset, Model(s) used for generation
Dataset	Modality	Dataset Size	Seed Dataset	Model(s) used for generation
Indic SDG for Nemotron	Text	711,276,565 indexed source sequences: 528,835,469 native-script and 182,441,096 romanized	nvidia/Nemotron-CC-v2	google/gemma-4-31B-it; google/gemma-4-31B-it-assistant
Nemotron-Pretraining-Fact-Seeking	Text	35.0B	FineWiki	Qwen3-30B-A3B-Instruct-2507

Additional Training Data

Table with columns: Data, Composition and Purpose
Data	Composition and Purpose
Indic continued pre-training	100B tokens: 10B selected Indic tokens and 90B tokens from the preceding blend.
Software-engineering mid-training	67B tokens: whole repositories (20%), issue–pull-request pairs (41%), commit diffs (3%), build and CI logs (1%), agent traces (10%), pre-training replay (20%), and long-context data (5%). Sources are decontaminated against SWE-bench and Terminal-Bench repositories.
Long-context pre-training	A 35B-token extension to 1M context, followed by a 9B-token long-context recovery stage.
Vision-language data	Image and video data from the Nemotron 3 Nano Omni training corpus, including visual question answering, charts, tables, OCR, multi-page documents, visual grounding, and video understanding.

Testing Datasets

Data Collection Method: Hybrid: automated, manually-collected, and synthetic
Labeling Method: Hybrid: automated, manually-labeled, and synthetic
Properties: Standard benchmarks and task-specific test suites covering knowledge, math, code, commonsense reasoning, reading comprehension, multilingual understanding, and long context.

Evaluation Datasets

Inference

Acceleration Engine: PyTorch
Test Hardware:
- NVIDIA Hopper
  - H100
  - H200
- NVIDIA Grace Blackwell
  - GB200
  - GB300
- NVIDIA Blackwell
  - B200
  - B300

The Base model is intended as a starting point for further training. For chat, reasoning, and tool-calling applications, use the post-trained BF16 or NVFP4 release and its deployment instructions.

Download the Base weights with the Hugging Face CLI:

bash

hf download nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 \  --local-dir ./models/super-vl-base-bf16

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

We advise against circumvention of any provided safety guardrails contained in the Model without a substantially similar guardrail appropriate for your use case. For more details: Safety and Explainability subcards. For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, and Privacy subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider