GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
yugbirla
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
yugbirla
Model Tree
FacebookAI/roberta-base
Input Modalities
Output Modalities
Supported Functionality
ToxSense (formerly ModGuard) is a highly precise, multimodal AI safety moderator designed to detect complex, zero-shot adversarial hate speech, sarcasm, and benign confounders. Unlike standard binary classifiers, ToxSense uses a Chain-of-Thought (CoT) reasoning approach across both Image and Text modalities, outputting structured JSON to categorize content into granular safety taxonomies.
The "Safety Tax" / Safe-Bias: Due to the base RLHF alignment and subsequent Direct Preference Optimization (DPO), ToxSense exhibits a strong "safe-bias." It requires a very high threshold of proof to classify content as hateful. While this lowers the overall recall, it was an intentional product design choice to achieve 83% Precision, thereby drastically minimizing false-positive user bans.
ToxSense requires a strictly formatted JSON input containing ocr_text, image_caption (extracted via BLIP), and base toxicity_scores.
python
import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "yugbirla/ToxSense-json-ultimate"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") # 1. Prepare the JSON Payloadinput_data = { "ocr_text": "Look at this completely normal picture.", "image_caption": "A controversial political figure.", "toxicity_scores": {"safe": 0.9, "hate": 0.1}} sys_msg = ( "You are ToxSense, a highly intelligent safety moderator. " "You will receive input as a JSON object containing 'ocr_text', 'image_caption', and 'toxicity_scores'. " "Think step-by-step. Analyze the contrast between the text and the image. " "Classify the input into exactly ONE of these categories: " "[safe, racism, sexism, threat, harassment, insult]. " "Output JSON ONLY in this format: {\"reasoning\": \"your short analysis\", \"category\": \"<category_name>\"}") messages = [ {"role": "system", "content": sys_msg}, {"role": "user", "content": json.dumps(input_data, indent=2)}] # 2. Generatetext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer([text], return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=150) print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "yugbirla/ToxSense-json-ultimate"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") # 1. Prepare the JSON Payloadinput_data = { "ocr_text": "Look at this completely normal picture.", "image_caption": "A controversial political figure.", "toxicity_scores": {"safe": 0.9, "hate": 0.1}} sys_msg = ( "You are ToxSense, a highly intelligent safety moderator. " "You will receive input as a JSON object containing 'ocr_text', 'image_caption', and 'toxicity_scores'. " "Think step-by-step. Analyze the contrast between the text and the image. " "Classify the input into exactly ONE of these categories: " "[safe, racism, sexism, threat, harassment, insult]. " "Output JSON ONLY in this format: {\"reasoning\": \"your short analysis\", \"category\": \"<category_name>\"}") messages = [ {"role": "system", "content": sys_msg}, {"role": "user", "content": json.dumps(input_data, indent=2)}] # 2. Generatetext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer([text], return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=150) print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))