alibaba-multimodal-industrial-ai

IndustryLLM

README

License: apache-2.0

Model details

Architecture: Qwen3_5MoeForConditionalGeneration
Parameters in the published BF16 tensors: 35,951,822,704
Weight format: BF16 safetensors, 16 shards
Configured context window: 262,144 tokens
Fine-tuning sequence length: 12,288 tokens with packing
Training objective: full-parameter supervised fine-tuning with thinking-format supervision
Frozen during fine-tuning: vision tower and vision-language aligner
Trained during fine-tuning: language-model parameters
Chat template: Qwen3.5 thinking/non-thinking template

Although the architecture accepts image and video inputs, this checkpoint has not been independently evaluated for multimodal quality after fine-tuning. The 262,144-token architectural context limit was also not validated by this 12,288-token fine-tuning run.

Training

The run used ms-swift 4.2.0 with Megatron-Core on 32 workers. Important settings were BF16 precision, global batch size 64, Adam with a peak learning rate of 5e-6, cosine decay to 1e-7, 5% warmup, and weight decay 0.1. Tensor parallelism was 2-way and expert parallelism was 8-way. The complete non-sensitive configuration is provided in training_config.json.

The training and validation datasets are private. Their internal identifiers and storage paths have been removed. Dataset composition, licensing, PII review, and benchmark decontamination evidence are not included in the checkpoint metadata; users should not infer that those reviews have been completed.

Usage

Use Transformers 5.2.0 or a newer version with native Qwen3.5 support. Multimodal inputs additionally require the Qwen vision utilities and their media dependencies.

bash
pip install "transformers>=5.2.0" "qwen_vl_utils>=0.0.14" decord accelerate

python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "alibaba-multimodal-industrial-ai/IndustryLLM"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "用三句话解释什么是混合专家模型。"}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=512)

completion_ids = output_ids[0, inputs["input_ids"].shape[-1] :]
print(processor.decode(completion_ids, skip_special_tokens=True))

For a direct response without a thinking block, pass enable_thinking=False to apply_chat_template. This is a template parameter; Qwen3.5 does not use the /think and /nothink soft switches.

Limitations and responsible use

This intermediate checkpoint may underperform the best recorded step or the final training checkpoint.
It can produce inaccurate, biased, unsafe, or fabricated content and should not be treated as an authoritative source.
It has not been validated for high-stakes medical, legal, financial, or safety-critical use.
Tool calls, code, and external actions must be sandboxed and independently validated before execution.
Deployment owners are responsible for task-specific evaluation, abuse controls, privacy review, and human oversight.

License and attribution

The upstream Qwen3.5 open-weight model is licensed under the Apache License 2.0. Redistributors must preserve the upstream license and attribution and must separately verify that they have the right to distribute the fine-tuning data-derived weights.

Please cite the upstream Qwen3.5 model when using this checkpoint:

bibtex
@misc{qwen3_5,
  title  = {Qwen3.5: Towards Native Multimodal Agents},
  author = {Qwen Team},
  year   = {2026},
  month  = {February},
  url    = {https://qwen.ai/blog?id=qwen3.5}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

alibaba-multimodal-industrial-ai

Model Tree

Base

Qwen/Qwen3.5-35B-A3B-Base

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality