GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nvidia
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nvidia
Model Tree
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
Input Modalities
Output Modalities
Supported Functionality
| Total parameters | 31B (Mamba2-Transformer hybrid Mixture of Experts (MoE)) |
| Active parameters | ~3B per token |
| Max context | 256k tokens |
| Modalities (in) | Video, Audio, Image, Text |
| Modality (out) | Text |
| Best for | Point-level tennis Q&A, shot and court understanding, match-state reasoning, rules knowledge, and audio-aware point analysis |
| Fine-tuning data | 1.31M Q&A pairs over 43k point clips (239 matches) |
| Minimum GPU (BF16) | 1× A100 80GB or 1× H100 80GB (single-GPU); 1× B200 / 1× H200 recommended |
| Precisions | bfloat16 (BF16) (NVIDIA-NemotronLabs-AI-for-Media-Sports-Tennis, ~62 GB) — TBD |
NVIDIA-NemotronLabs-AI-for-Media-Sports-Tennis is a fine-tuned multimodal large language model derived from Nemotron 3 Nano Omni. It was further trained on an in-house NVIDIA tennis dataset of manually labeled point-level video clips to specialize in tennis match understanding. The model supports structured multiple-choice and open-ended Q&A over point clips, covering shot mechanics, court positioning, player movement, match facts, rules knowledge, and audio cues. It works best when a full tennis point clip is passed as input.
This model is fine-tuned from Nemotron 3 Nano Omni on an in-house NVIDIA tennis dataset of tennis point video clips using the NVIDIA Sports Intelligence playbooks. For base model details, see the base model card.
This model is ready for commercial or non-commercial use.
Governing Terms: Use of this model is governed by the OpenMDW License Agreement v1.1 (OpenMDW-1.1)
Global
This model is designed for tennis-specific multimodal understanding over point-level video. Expected users include:
Hugging Face 09/10/2026 via Hugging Face
Architecture Type: Other (Mamba2-Transformer hybrid Mixture of Experts backbone with multimodal encoders)
Network Architecture:
Number of model parameters: 3.1 x 10^10 (31B A3B)
How the model was developed: Weights are initialized from the pretrained Nemotron 3 Nano Omni BF16 checkpoint. Fine-tuning trains only the language-model parameters; the vision and audio encoders remain frozen. Optimization uses AdamW (learning rate 5×10⁻⁵, β=(0.9, 0.999)) with warmup and cosine decay, regularized by weight decay 0.05 and gradient clipping at max-norm 1.0, with no additional damping. Training is BF16 packed supervised fine-tuning over point-level tennis clips. Hyperparameters follow the published NeMo AutoModel recipe rather than a dedicated search.
Input Type(s): Video, Audio, Image, Text
Input Format(s):
Input Parameters:
Other Properties Related to Input:
Output Type(s): Text
Output Format(s):
Output Parameters:
Other Properties Related to Output:
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to central processing unit (CPU)-only solutions.
Runtime Engine(s):
Supported Hardware Microarchitecture Compatibility:
Preferred/Supported Operating System(s):
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Provide a full tennis point as an MP4, from serve through the end of the point, with audio when available. Frames are sampled with video_io.py, included with this model.
For a ready-to-run CLI (download, load, sample frames, and generate), use video_inference.py in this repo:
bash
pip install torch transformers accelerate huggingface_hub safetensors pillow \librosa soundfile decord imageio-ffmpegpython video_inference.py \--video /path/to/tennis_point.mp4 \--question 'What happened in this point? Provide a detailed caption. (Jil Teichmann - "Jil Teichmann" is the player wearing a white cap, a black top, a black skirt, and white and tan shoes; Victoria Mboko - "Victoria Mboko" is the player wearing a white cap, white top, a green skirt, and white shoes.)'
Add --local-files-only after weights are cached. If librosa cannot decode MP4 audio, video_inference.py extracts a WAV with ffmpeg (imageio-ffmpeg or a system ffmpeg); pass --extract-audio-wav to force that path, or --no-audio to skip audio.
The first AutoModel.from_pretrained(..., trust_remote_code=True) also downloads RADIO vision code from nvidia/C-RADIOv4-H.
Default inference policy (same as video_inference.py): 2 frames per second (FPS), up to 128 frames, 256 new tokens, video + audio from the MP4, greedy decode, video_pruning_rate=0.0.
The core inference steps (functionally equivalent to video_inference.py):
python
import importlib.utilimport osimport shutilimport subprocessimport tempfilefrom pathlib import Pathimport torchfrom huggingface_hub import snapshot_downloadfrom transformers import AutoConfig, AutoModel, AutoProcessorfrom transformers.video_utils import VideoMetadatarepo_id = "nvidia/NVIDIA-NemotronLabs-AI-for-Media-Sports-Tennis"video_path = "/path/to/tennis_point.mp4"question = ("What happened in this point? Provide a detailed caption. "'(Jil Teichmann - "Jil Teichmann" is the player wearing a white cap, a black top, '"a black skirt, and white and tan shoes; Victoria Mboko - \"Victoria Mboko\" is the ""player wearing a white cap, white top, a green skirt, and white shoes.)")video_fps, max_video_frames, max_new_tokens = 2, 128, 256revision, local_files_only = None, False # optional: pin Hub commit; cache-only loadcache_dir = os.environ.get("HF_HOME") or os.environ.get("HF_HUB_CACHE")hub = {"trust_remote_code": True, "local_files_only": local_files_only}if cache_dir:hub["cache_dir"] = cache_dirif revision:hub["revision"] = revisionsnapshot_dir = snapshot_download(repo_id=repo_id, repo_type="model", **{k: v for k, v in hub.items() if k != "trust_remote_code"})spec = importlib.util.spec_from_file_location("checkpoint_video_io", Path(snapshot_dir) / "video_io.py")video_io = importlib.util.module_from_spec(spec)spec.loader.exec_module(video_io)frame_urls, metadata = video_io.sample_video_frames_to_data_urls(video_path, fps=max(0, int(video_fps)), nframe_max=max_video_frames,)frames = [video_io.pil_image_from_base64(url) for url in frame_urls]video_metadata = VideoMetadata(total_num_frames=metadata.total_num_frames, fps=metadata.fps, frames_indices=metadata.frames_indices,)ffmpeg = shutil.which("ffmpeg")if ffmpeg is None:import imageio_ffmpegffmpeg = imageio_ffmpeg.get_ffmpeg_exe()audio_wav = tempfile.NamedTemporaryFile(suffix=".wav", delete=False).namesubprocess.run([ffmpeg, "-nostdin", "-y", "-i", video_path, "-vn", "-acodec", "pcm_s16le","-ar", "16000", "-ac", "1", audio_wav],check=True, capture_output=True,)processor = AutoProcessor.from_pretrained(repo_id, **hub)config = AutoConfig.from_pretrained(repo_id, **hub)config.video_pruning_rate = 0.0model = AutoModel.from_pretrained(repo_id, config=config, torch_dtype=torch.bfloat16,device_map={"": 0} if torch.cuda.is_available() else None, low_cpu_mem_usage=True, **hub,)if hasattr(model, "vision_model") and hasattr(model.vision_model, "radio_model"):model.vision_model.radio_model.summary_idxs = Nonemodel.eval()prompt = processor.tokenizer.apply_chat_template([{"role": "user", "content": f"{processor.video_token}\n{processor.audio_token}\n{question}"}],tokenize=False, add_generation_prompt=True, enable_thinking=False,)inputs = processor(text=[prompt], videos=frames, audio=[audio_wav], return_tensors="pt", do_sample_frames=False,videos_kwargs={"video_metadata": video_metadata}, audio_kwargs={"sampling_rate": 16000},)for key in ("num_patches", "num_tokens", "imgs_sizes"):inputs.pop(key, None)device = next(model.parameters()).deviceinputs = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}def trim_generated_ids(input_ids, generated_ids):if hasattr(generated_ids, "sequences"):generated_ids = generated_ids.sequencesif generated_ids.dim() == 1:return [generated_ids[input_ids.shape[-1]:]]if input_ids.dim() == 1:input_ids = input_ids.unsqueeze(0)return [out_ids[len(in_ids):] for in_ids, out_ids in zip(input_ids, generated_ids)]with torch.no_grad():generated_ids = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False, output_hidden_states=False)print(processor.batch_decode(trim_generated_ids(inputs["input_ids"], generated_ids), skip_special_tokens=True, clean_up_tokenization_spaces=False,)[0].strip())
Requires one GPU with ~80 GB memory (BF16). Authenticate with hf auth login before downloading a private repo. Frame sampling uses decord. MP4 audio is decoded by extracting a 16 kHz WAV with ffmpeg (imageio-ffmpeg or a system ffmpeg); librosa/soundfile cannot open many MP4s directly. summary_idxs | MISSING on load is expected.
This model is a fine-tuned variant of Nemotron 3 Nano Omni 30B-A3B-Reasoning, retaining the same Mamba2-Transformer hybrid MoE architecture and multimodal encoders (CRADIO v4-H vision encoder and Parakeet speech encoder).
Dataset Name: NVIDIA In-House Tennis Dataset (proprietary) Data Modality: Video, Text Video Training Data Size: Less than 10,000 Hours (43,084 point-level video clips) Text Training Data Size: Less than 1 Billion Tokens (1,312,129 training Q&A examples) Dataset Size: 1,312,129 training examples Data Collection Method: Manually-Collected Labeling Method: Manually-Labeled Time period for training data collection: 2025 Properties (Quantity, Dataset Descriptions, Sensor(s)): 1,312,129 tennis point video Q&A examples (1,226,081 multiple-choice and 86,048 open-ended) from 43,084 point-level clips across 239 matches and 38 annotation categories. Content is proprietary broadcast and court-capture tennis footage with manually authored English questions and answers. No sensor-derived metadata beyond standard video/audio capture. Linguistic characteristics: English-only tennis-domain questions and answers in multiple-choice and open-ended formats covering shot mechanics, court positioning, player movement, rally structure, score state, rules knowledge, audio cues, and cross-field reasoning.
Dataset Name: NVIDIA In-House Tennis Dataset — Test (unseen matches) partition (proprietary) Dataset Size: 80,872 test examples from 12 fully held-out matches (2,689 point-level clips; no training match overlap) Data Collection Method: Manually-Collected Labeling Method: Manually-Labeled Properties (Quantity, Dataset Descriptions, Sensor(s)): Q&A pairs from 12 fully held-out matches (no training match overlap) — 2,689 point-level clips, 80,872 examples. English tennis-domain labels aligned with the training taxonomy. The corpus also includes a validation split (68,931 examples) and a separate Test (seen matches) partition (69,091 examples from 238 matches represented in training, with match-level overlap only); only Test (unseen matches) was used for reported model testing.
Dataset Name: NVIDIA In-House Tennis Test (unseen matches) evaluation benchmark (proprietary) Dataset Size: 80,872 test examples from the Test (unseen matches) partition (12 fully held-out matches) used for MCQ accuracy and open-ended judge scoring Data Collection Method: Manually-Collected Labeling Method: Manually-Labeled Properties (Quantity, Dataset Descriptions, Sensor(s)): Test (unseen matches) — 12 fully held-out matches with no training match overlap (2,689 point-level clips, 80,872 Q&A examples). Evaluated with automated MCQ scoring and LLM-as-judge pass@9 metrics on open-ended responses. English tennis-domain questions across 38 fine-grained categories. Test (seen matches) was not used for reported metrics.
Fine-tuning was performed on top of the base Nemotron 3 Nano Omni weights using the in-house tennis dataset. Pre-training dataset details for the base model are documented in the base Nemotron 3 Nano Omni model card.
Acceleration Engine: PyTorch, Hugging Face Transformers, NeMo AutoModel Test Hardware:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.