joyfox

JoyFox-PawSense-AudioVision-AWQ

README

License: apache-2.0

Demo Sample

This sample is randomly selected from the training dataset and includes one pet audio clip plus its same-sample waveform/log-mel acoustic image.

Audio: audio1

Acoustic image: audio1.png

Waveform and log-mel spectrogram

JoyFox-PawSense-AudioVision-AWQ is a 4-bit AWQ quantized release of JoyFox-PawSense-AudioVision. It keeps the same task behavior as the full merged checkpoint:

input: original pet audio,
input: a same-sample acoustic image containing waveform and log-mel spectrogram views,
output: strict Chinese JSON for pet vocalization analysis.

The acoustic image is not a scene photo. It is generated from the same audio clip and is used as an auxiliary visual view of acoustic structure. The raw audio remains the primary evidence source.

This model is intended for lower-memory local inference and app prototyping. Quantization can introduce small output differences compared with the full precision checkpoint.

Quantization Details

Table with columns: Item, Description
Item	Description
Model name	JoyFox-PawSense-AudioVision-AWQ
Source model	`joyfox/JoyFox-PawSense-AudioVision`
Base foundation model	`openbmb/MiniCPM-o-4_5`
Quantization method	AWQ
Weight bits	4
Group size	128
Zero point	True
AWQ backend version

The quantization configuration is stored in config.json:

json
{
  "bits": 4,
  "group_size": 128,
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}

Output Schema

The model follows the same schema as the full AudioVision checkpoint. The recommended output contains exactly these top-level fields:

text
pet_type_guess, pet_type_confidence, vocalization_type, sound_pattern,
app_headline, one_sentence_summary, pet_first_person_translation,
emotion_summary, emotion_dimensions, likely_needs,
behavioral_interpretation, context_hypotheses, risk_flags,
owner_actions_now, owner_actions_followup, app_tags, evidence,
confidence_note

Important schema constraints:

pet_type_guess should be cat, dog, or unknown.
Numeric scores are integers.
emotion_dimensions.valence is in [-100, 100].
Other emotion dimension scores are in [0, 100].
Values are primarily Chinese, while JSON keys remain English.
The model should avoid medical diagnosis and avoid treating uncertain context hypotheses as facts.

Qualitative Example

The following result was generated from the demo sample shown above.

JoyFox-PawSense-AudioVision-AWQ result

json
{
  "pet_type_guess": "cat",
  "pet_type_confidence": 100,
  "vocalization_type": "meow",
  "sound_pattern": "repeated",
  "app_headline": "喵喵连击！猫咪正在强烈呼唤你",
  "one_sentence_summary": "猫咪发出高频且重复的喵喵声，表现出强烈的求关注或索要食物的意图。",
  "pet_first_person_translation": "快看看我呀！我有话要对你说，别不理我嘛~",
  "emotion_summary": {
    "primary_emotion": "求关注",
    "primary_score": 85,
    "secondary_emotions": [
      {
        "label": "期盼",
        "score": 70
      },
      {
        "label": "些许急躁",
        "score": 45
      }
    ]
  },
  "emotion_dimensions": {
    "valence": -10,
    "arousal": 75,
    "stress": 35,
    "comfort": 40,
    "confidence": 60,
    "urgency": 70,
    "sociability": 85,
    "playfulness": 30,
    "need_for_attention": 90
  },
  "likely_needs": [
    "寻求关注与陪伴",
    "索要食物或零食",
    "想要开门或进入某个空间"
  ],
  "behavioral_interpretation": "猫咪连续发出音调清晰、节奏规律的喵喵声。这种重复的呼唤通常是猫咪在向主人表达明确的需求，如肚子饿了、想要玩耍，或者希望主人打开某扇门。声音中没有攻击性或极度痛苦的特征，更多是社交与沟通的意图。",
  "context_hypotheses": [
    "猫咪看到主人准备食物，正在催促",
    "猫咪被关在门外，希望主人开门",
    "猫咪感到无聊，希望主人陪它玩耍"
  ],
  "risk_flags": [],
  "owner_actions_now": [
    "走到猫咪身边，用温柔的声音回应它",
    "检查猫咪的食盆和水盆是否空了",
    "看看猫咪是否停在紧闭的门前"
  ],
  "owner_actions_followup": [
    "如果猫咪是因为无聊，可以陪它玩耍5-10分钟",
    "保持规律的喂食时间，避免猫咪因过度饥饿而频繁催食"
  ],
  "app_tags": [
    "求关注",
    "索要食物",
    "话痨猫咪",
    "日常沟通"
  ],
  "evidence": [
    "音频中包含10次清晰、高频且节奏规律的‘喵’声，每次持续约0.5秒。",
    "频谱图显示基频及谐波结构非常完整且清晰，无杂音，表明发声状态健康且意图明确。",
    "声音音调较高，且重复出现，符合猫咪向人类表达诉求时的典型声学特征。"
  ],
  "confidence_note": "音频质量极高，猫咪的叫声特征非常典型且无背景噪声干扰，置信度极高。"
}

Intended Use

JoyFox-PawSense-AudioVision-AWQ is intended for:

local pet audio demos with lower VRAM usage,
pet-care app prototypes,
structured pet vocalization labeling,
cat and dog sound interpretation experiments,
comparison with the full precision AudioVision checkpoint.

It should be used as an assistive interpretation layer. It should not replace direct owner observation, professional behavior assessment, or veterinary care.

Environment Setup

This checkpoint requires AutoAWQ plus MiniCPM-o remote code support.

The known working local environment for this checkpoint uses:

text
transformers==4.56.2
torch 2.11.0+cu130

Example setup:

bash
conda create -n pawsense-awq python=3.11 -y
conda activate pawsense-awq

# Install a CUDA-compatible PyTorch build for your machine.
pip install torch torchvision torchaudio

pip install "transformers==4.56.2" accelerate safetensors librosa soundfile pillow matplotlib numpy sentencepiece
pip install autoawq

If you use the original LLaMA-Factory environment from training and quantization, the same environment can be used for inference.

Quick Start with AutoAWQ

This example assumes you already have a waveform/log-mel acoustic image for the audio clip. The image should be generated from the same audio file, not from a camera scene.

python
import librosa
import torch
from awq import AutoAWQForCausalLM
from PIL import Image

model_path = "joyfox/JoyFox-PawSense-AudioVision-AWQ"
audio_path = "your_pet_audio.wav"
image_path = "your_pet_audio.png"

prompt = (
    "你是一位面向宠物类 App 的高级宠物声音情绪分析师。"
    "请结合原始音频和同一音频生成的声学图（waveform/log-mel）分析宠物发声中的情绪状态、"
    "唤起程度、压力水平、舒适感、诉求、风险提示和可执行建议。"
    "音频是主要事实依据；声学图用于辅助观察音高走势、能量强弱、爆发/持续/间歇节奏、"
    "噪声和多段发声结构。返回严格合法 JSON，不要 Markdown 或解释文字。"
    "字段必须包含且只包含：pet_type_guess, pet_type_confidence, vocalization_type, "
    "sound_pattern, app_headline, one_sentence_summary, pet_first_person_translation, "
    "emotion_summary, emotion_dimensions, likely_needs, behavioral_interpretation, "
    "context_hypotheses, risk_flags, owner_actions_now, owner_actions_followup, app_tags, "
    "evidence, confidence_note。"
)

awq_model = AutoAWQForCausalLM.from_quantized(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
    init_vision=True,
    init_audio=True,
    init_tts=False,
    device_map={"": "cuda:0"} if torch.cuda.is_available() else {"": "cpu"},
    max_seq_len=None,
)

model = awq_model.model
model.eval()

audio_input, _ = librosa.load(audio_path, sr=16000, mono=True)
image_input = Image.open(image_path).convert("RGB")
msgs = [{"role": "user", "content": [image_input, audio_input, prompt]}]

with torch.inference_mode():
    answer = model.chat(
        msgs=msgs,
        do_sample=False,
        max_new_tokens=1024,
        use_tts_template=False,
        generate_audio=False,
        enable_thinking=False,
    )

print(answer)

Using `infer_audio_vision_awq.py`

The local AWQ inference script is designed for direct file-based testing. Edit the configuration block at the top of infer_audio_vision_awq.py:

python
MODEL_PATH = "joyfox/JoyFox-PawSense-AudioVision-AWQ"
AUDIO_PATH = "assets/audio1.wav"
IMAGE_PATH = ""              # empty means same-name .png next to AUDIO_PATH
AUTO_GENERATE_IMAGE = True   # auto-generate waveform/log-mel PNG if missing
REBUILD_IMAGE = False        # set True to regenerate the acoustic image
MAX_NEW_TOKENS = 1536
DO_SAMPLE = False
DTYPE = "fp16"               # AWQ is normally run in fp16
ATTN_IMPLEMENTATION = "sdpa"
USE_CPU = False
KEEP_VISION = True
INIT_TTS = False
MEDIA_FIRST = True

Then run:

bash
python infer_audio_vision_awq.py

The script performs the following steps:

resolves MODEL_PATH and validates that it points to a complete quantized checkpoint,
resolves the acoustic image path,
generates a same-name waveform/log-mel PNG if needed,
loads audio with librosa.load(audio_path, sr=16000, mono=True),
loads the image with PIL.Image.open(...).convert("RGB"),
loads the model with AutoAWQForCausalLM.from_quantized(..., trust_remote_code=True, init_audio=True, init_vision=True),
builds a MiniCPM-o user message with [image_input, audio_input, prompt],
runs model.chat(...) under torch.inference_mode(),
reports image generation, audio loading, image loading, model loading, inference, and total time.

Notes on Warnings

AutoAWQ may print a deprecation notice and may warn that awq_ext is not installed:

text
Skipping fusing modules because AWQ extension is not installed.

This warning means fused AWQ kernels are unavailable, so inference can be slower. It does not necessarily prevent the model from running.

You may also see warnings about deprecated torch_dtype, slow image processors, or ignored generation config sampling fields. These warnings do not usually affect deterministic JSON inference.

Limitations

This model is focused on cat and dog vocalizations. Other animals or non-pet sounds may produce unreliable interpretations.
The acoustic image must come from the same audio clip. Scene photos or unrelated spectrograms can degrade reliability.
Quantization can slightly change wording, scores, or ordering compared with the full precision checkpoint.
The emotional state and intent are probabilistic interpretations from audio and acoustic cues, not ground truth.
Background noise, overlapping speech, music, clipping, or very short audio can reduce reliability.
The model primarily follows a Chinese JSON response format. English output has not been the main tuning target.
The model should not be used to diagnose illness, pain, injury, aggression risk, or emergency conditions without professional judgment.
If an animal appears distressed, injured, unusually aggressive, or medically abnormal, consult a veterinarian or qualified animal behavior professional.

License

This model follows the license terms of the source model and the released checkpoint metadata. Please also review the license and usage terms of openbmb/MiniCPM-o-4_5 before redistribution or commercial use.

Acknowledgements

JoyFox-PawSense-AudioVision-AWQ is quantized from JoyFox-PawSense-AudioVision, which is built on OpenBMB's MiniCPM-o-4_5 multimodal model and trained/merged with LLaMA-Factory. The model card structure and usage style are based on the earlier MiniCPM-PawSense-Audio release.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

joyfox

Model Tree

Base

joyfox/JoyFox-PawSense-AudioVision

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Demo Sample

This sample is randomly selected from the training dataset and includes one pet audio clip plus its same-sample waveform/log-mel acoustic image.

Audio: audio1

Acoustic image: audio1.png

Waveform and log-mel spectrogram

JoyFox-PawSense-AudioVision-AWQ is a 4-bit AWQ quantized release of JoyFox-PawSense-AudioVision. It keeps the same task behavior as the full merged checkpoint:

input: original pet audio,
input: a same-sample acoustic image containing waveform and log-mel spectrogram views,
output: strict Chinese JSON for pet vocalization analysis.

The acoustic image is not a scene photo. It is generated from the same audio clip and is used as an auxiliary visual view of acoustic structure. The raw audio remains the primary evidence source.

This model is intended for lower-memory local inference and app prototyping. Quantization can introduce small output differences compared with the full precision checkpoint.

Quantization Details

Table with columns: Item, Description
Item	Description
Model name	JoyFox-PawSense-AudioVision-AWQ
Source model	`joyfox/JoyFox-PawSense-AudioVision`
Base foundation model	`openbmb/MiniCPM-o-4_5`
Quantization method	AWQ
Weight bits	4
Group size	128
Zero point	True
AWQ backend version

The quantization configuration is stored in config.json:

json
{
  "bits": 4,
  "group_size": 128,
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}

Output Schema

The model follows the same schema as the full AudioVision checkpoint. The recommended output contains exactly these top-level fields:

text
pet_type_guess, pet_type_confidence, vocalization_type, sound_pattern,
app_headline, one_sentence_summary, pet_first_person_translation,
emotion_summary, emotion_dimensions, likely_needs,
behavioral_interpretation, context_hypotheses, risk_flags,
owner_actions_now, owner_actions_followup, app_tags, evidence,
confidence_note

Important schema constraints:

pet_type_guess should be cat, dog, or unknown.
Numeric scores are integers.
emotion_dimensions.valence is in [-100, 100].
Other emotion dimension scores are in [0, 100].
Values are primarily Chinese, while JSON keys remain English.
The model should avoid medical diagnosis and avoid treating uncertain context hypotheses as facts.

Qualitative Example

The following result was generated from the demo sample shown above.

JoyFox-PawSense-AudioVision-AWQ result

json
{
  "pet_type_guess": "cat",
  "pet_type_confidence": 100,
  "vocalization_type": "meow",
  "sound_pattern": "repeated",
  "app_headline": "喵喵连击！猫咪正在强烈呼唤你",
  "one_sentence_summary": "猫咪发出高频且重复的喵喵声，表现出强烈的求关注或索要食物的意图。",
  "pet_first_person_translation": "快看看我呀！我有话要对你说，别不理我嘛~",
  "emotion_summary": {
    "primary_emotion": "求关注",
    "primary_score": 85,
    "secondary_emotions": [
      {
        "label": "期盼",
        "score": 70
      },
      {
        "label": "些许急躁",
        "score": 45
      }
    ]
  },
  "emotion_dimensions": {
    "valence": -10,
    "arousal": 75,
    "stress": 35,
    "comfort": 40,
    "confidence": 60,
    "urgency": 70,
    "sociability": 85,
    "playfulness": 30,
    "need_for_attention": 90
  },
  "likely_needs": [
    "寻求关注与陪伴",
    "索要食物或零食",
    "想要开门或进入某个空间"
  ],
  "behavioral_interpretation": "猫咪连续发出音调清晰、节奏规律的喵喵声。这种重复的呼唤通常是猫咪在向主人表达明确的需求，如肚子饿了、想要玩耍，或者希望主人打开某扇门。声音中没有攻击性或极度痛苦的特征，更多是社交与沟通的意图。",
  "context_hypotheses": [
    "猫咪看到主人准备食物，正在催促",
    "猫咪被关在门外，希望主人开门",
    "猫咪感到无聊，希望主人陪它玩耍"
  ],
  "risk_flags": [],
  "owner_actions_now": [
    "走到猫咪身边，用温柔的声音回应它",
    "检查猫咪的食盆和水盆是否空了",
    "看看猫咪是否停在紧闭的门前"
  ],
  "owner_actions_followup": [
    "如果猫咪是因为无聊，可以陪它玩耍5-10分钟",
    "保持规律的喂食时间，避免猫咪因过度饥饿而频繁催食"
  ],
  "app_tags": [
    "求关注",
    "索要食物",
    "话痨猫咪",
    "日常沟通"
  ],
  "evidence": [
    "音频中包含10次清晰、高频且节奏规律的‘喵’声，每次持续约0.5秒。",
    "频谱图显示基频及谐波结构非常完整且清晰，无杂音，表明发声状态健康且意图明确。",
    "声音音调较高，且重复出现，符合猫咪向人类表达诉求时的典型声学特征。"
  ],
  "confidence_note": "音频质量极高，猫咪的叫声特征非常典型且无背景噪声干扰，置信度极高。"
}

Intended Use

JoyFox-PawSense-AudioVision-AWQ is intended for:

local pet audio demos with lower VRAM usage,
pet-care app prototypes,
structured pet vocalization labeling,
cat and dog sound interpretation experiments,
comparison with the full precision AudioVision checkpoint.

It should be used as an assistive interpretation layer. It should not replace direct owner observation, professional behavior assessment, or veterinary care.

Environment Setup

This checkpoint requires AutoAWQ plus MiniCPM-o remote code support.

The known working local environment for this checkpoint uses:

text
transformers==4.56.2
torch 2.11.0+cu130

Example setup:

bash
conda create -n pawsense-awq python=3.11 -y
conda activate pawsense-awq

# Install a CUDA-compatible PyTorch build for your machine.
pip install torch torchvision torchaudio

pip install "transformers==4.56.2" accelerate safetensors librosa soundfile pillow matplotlib numpy sentencepiece
pip install autoawq

If you use the original LLaMA-Factory environment from training and quantization, the same environment can be used for inference.

Quick Start with AutoAWQ

This example assumes you already have a waveform/log-mel acoustic image for the audio clip. The image should be generated from the same audio file, not from a camera scene.

python
import librosa
import torch
from awq import AutoAWQForCausalLM
from PIL import Image

model_path = "joyfox/JoyFox-PawSense-AudioVision-AWQ"
audio_path = "your_pet_audio.wav"
image_path = "your_pet_audio.png"

prompt = (
    "你是一位面向宠物类 App 的高级宠物声音情绪分析师。"
    "请结合原始音频和同一音频生成的声学图（waveform/log-mel）分析宠物发声中的情绪状态、"
    "唤起程度、压力水平、舒适感、诉求、风险提示和可执行建议。"
    "音频是主要事实依据；声学图用于辅助观察音高走势、能量强弱、爆发/持续/间歇节奏、"
    "噪声和多段发声结构。返回严格合法 JSON，不要 Markdown 或解释文字。"
    "字段必须包含且只包含：pet_type_guess, pet_type_confidence, vocalization_type, "
    "sound_pattern, app_headline, one_sentence_summary, pet_first_person_translation, "
    "emotion_summary, emotion_dimensions, likely_needs, behavioral_interpretation, "
    "context_hypotheses, risk_flags, owner_actions_now, owner_actions_followup, app_tags, "
    "evidence, confidence_note。"
)

awq_model = AutoAWQForCausalLM.from_quantized(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
    init_vision=True,
    init_audio=True,
    init_tts=False,
    device_map={"": "cuda:0"} if torch.cuda.is_available() else {"": "cpu"},
    max_seq_len=None,
)

model = awq_model.model
model.eval()

audio_input, _ = librosa.load(audio_path, sr=16000, mono=True)
image_input = Image.open(image_path).convert("RGB")
msgs = [{"role": "user", "content": [image_input, audio_input, prompt]}]

with torch.inference_mode():
    answer = model.chat(
        msgs=msgs,
        do_sample=False,
        max_new_tokens=1024,
        use_tts_template=False,
        generate_audio=False,
        enable_thinking=False,
    )

print(answer)

Using `infer_audio_vision_awq.py`

The local AWQ inference script is designed for direct file-based testing. Edit the configuration block at the top of infer_audio_vision_awq.py:

python
MODEL_PATH = "joyfox/JoyFox-PawSense-AudioVision-AWQ"
AUDIO_PATH = "assets/audio1.wav"
IMAGE_PATH = ""              # empty means same-name .png next to AUDIO_PATH
AUTO_GENERATE_IMAGE = True   # auto-generate waveform/log-mel PNG if missing
REBUILD_IMAGE = False        # set True to regenerate the acoustic image
MAX_NEW_TOKENS = 1536
DO_SAMPLE = False
DTYPE = "fp16"               # AWQ is normally run in fp16
ATTN_IMPLEMENTATION = "sdpa"
USE_CPU = False
KEEP_VISION = True
INIT_TTS = False
MEDIA_FIRST = True

Then run:

bash
python infer_audio_vision_awq.py

The script performs the following steps:

resolves MODEL_PATH and validates that it points to a complete quantized checkpoint,
resolves the acoustic image path,
generates a same-name waveform/log-mel PNG if needed,
loads audio with librosa.load(audio_path, sr=16000, mono=True),
loads the image with PIL.Image.open(...).convert("RGB"),
loads the model with AutoAWQForCausalLM.from_quantized(..., trust_remote_code=True, init_audio=True, init_vision=True),
builds a MiniCPM-o user message with [image_input, audio_input, prompt],
runs model.chat(...) under torch.inference_mode(),
reports image generation, audio loading, image loading, model loading, inference, and total time.

Notes on Warnings

AutoAWQ may print a deprecation notice and may warn that awq_ext is not installed:

text
Skipping fusing modules because AWQ extension is not installed.

This warning means fused AWQ kernels are unavailable, so inference can be slower. It does not necessarily prevent the model from running.

You may also see warnings about deprecated torch_dtype, slow image processors, or ignored generation config sampling fields. These warnings do not usually affect deterministic JSON inference.

Limitations

This model is focused on cat and dog vocalizations. Other animals or non-pet sounds may produce unreliable interpretations.
The acoustic image must come from the same audio clip. Scene photos or unrelated spectrograms can degrade reliability.
Quantization can slightly change wording, scores, or ordering compared with the full precision checkpoint.
The emotional state and intent are probabilistic interpretations from audio and acoustic cues, not ground truth.
Background noise, overlapping speech, music, clipping, or very short audio can reduce reliability.
The model primarily follows a Chinese JSON response format. English output has not been the main tuning target.
The model should not be used to diagnose illness, pain, injury, aggression risk, or emergency conditions without professional judgment.
If an animal appears distressed, injured, unusually aggressive, or medically abnormal, consult a veterinarian or qualified animal behavior professional.

JoyFox-PawSense-AudioVision-AWQ

README

Demo Sample

Quantization Details

Output Schema

Qualitative Example

Intended Use

Environment Setup

Quick Start with AutoAWQ

Using infer_audio_vision_awq.py

Notes on Warnings

Limitations

License

Acknowledgements

Explore FriendliAI today

README

Demo Sample

Quantization Details

Output Schema

Qualitative Example

Intended Use

Environment Setup

Quick Start with AutoAWQ

Using infer_audio_vision_awq.py

Notes on Warnings

Limitations

License

Acknowledgements

Using `infer_audio_vision_awq.py`

Using `infer_audio_vision_awq.py`