CoriolisTechnologies

orpheus-3b-tts-hindi

Deploy Dedicated

README

License: apache-2.0

🔗 Base Model & Dataset Links

Base Model: unsloth/orpheus-3b-0.1-ft
Multi-Source Training Datasets:
- Rasa (AI4Bharat): ai4bharat/Rasa (10,250 samples / ~18.6 hrs)
- Kathbath: ai4bharat/kathbath (9,000 samples / ~12.1 hrs)
- Hinglish Corpus: (12,144 samples / ~22.1 hrs)

🌟 Highlights & Key Features

Unified Autoregressive Generation: Predicts text and 24 kHz SNAC audio tokens within a single causal Llama 3B model.
3-Stage Fine-Tuning Pipeline:
- Stage 1 (Pronunciation SFT): Teaches Devanagari script reading & phonetic stability.
- Stage 2 (Speaker ID Conditioning): Eliminates voice identity drift using [Speaker: Name] tags (Prakhar & Prerna).
- Stage 3 (Dual-Tag Emotion SFT): Injects 7 distinct emotional tones ([Speaker: Name] [Tone: Emotion]).
Zero-OOM Validation Engine: Custom lightweight callbacks (CustomEvalCallback + ForceEvalCallback) providing exact validation loss monitoring under 6 GB VRAM.

📊 Evaluation Suite

Whisper ASR Oracle (Intelligibility): Evaluates WER and CER using OpenAI Whisper (evaluate_asr.py) against the consolidated 500-sample benchmark dataset (evaluation/benchmark_500.jsonl). This benchmark covers 3 progressive difficulty phases:
- Phase 1 (HI00001–HI00050): Basic Pronunciation (Simple Devanagari phonetics)
- Phase 2 (HI00051–HI00250): Linguistic Variation (Regional nuances & syntactic diversity)
- Phase 3 (HI00251–HI00500): Real-World Content (Conversational dialogues & storytelling)

📊 Performance Metrics

Table with columns: Metric / Evaluation, Value / Score, Details
Metric / Evaluation	Value / Score	Details
Word Error Rate (WER)	0.3090	↓ 66.6% cumulative reduction from baseline
Character Error Rate (CER)	0.1540	↓ 82.4% cumulative reduction from baseline
Intra-Speaker Cosine Similarity	0.70	High voice identity match across prompts (`speechbrain` ECAPA-TDNN)
Inter-Speaker Separation	0.31	Distinct separation between male ("Prakhar") & female ("Prerna") voices

🗣️ Supported Speakers & Emotions

1. Supported Speakers

Prakhar (Male)
Prerna (Female)

2. Supported Emotions (Stage 3)

ANGER: Fast, sharp delivery
SURPRISE: Elevated pitch and varied tempo
SAD: Slower, breathy speech patterns
HAPPY: High energy and rhythmic variance
NEUTRAL: Standard textbook delivery
FEAR: Rapid, irregular pacing
DISGUST: Distinctive phonetic emphasis

⚡ Inference Setup

1. Install Dependencies

bash
pip install unsloth torch torchaudio transformers peft snac

2. Python Inference Code

python
from inference.synthesize import load_inference_pipeline, synthesize_speech
from IPython.display import Audio, display

# 1. Load pipeline directly from Hugging Face
model, tokenizer, snac_model, device = load_inference_pipeline(
    checkpoint_dir="CoriolisTechnologies/orpheus-3b-tts-hindi"
)

# 2. Synthesize audio with speaker and emotion tags
text_input = "इस कूरियर सर्विस की गुणवत्ता बहुत बेहतरीन है!"
audio_path = synthesize_speech(
    text=text_input,
    model=model,
    tokenizer=tokenizer,
    snac_model=snac_model,
    device=device,
    output_path="output_synthesized.wav",
    speaker="Prakhar", # Prakhar | Prerna
    emotion="HAPPY"   # ANGER | SURPRISE | SAD | HAPPY | NEUTRAL | FEAR | DISGUST
)

# 3. Play generated 24 kHz WAV audio
display(Audio(audio_path, autoplay=True))

🛡️ Limitations & Responsible Use

Intended Use

This repository provides models and fine-tuning pipelines for Hindi text-to-speech (TTS) synthesis, pronunciation learning, speaker identity conditioning, and emotional voice synthesis.

Limitations

Maximum Audio Duration: Currently, the model synthesizes audio up to a maximum duration of ~14 seconds per generation. For longer paragraphs or extended narrations, chunked / sentence-by-sentence synthesis is recommended.
Sequence Length: Utterances exceeding sequence limits may experience degraded synthesis stability or truncation.
Complex Scripts: Pronunciation accuracy on rare or highly complex Devanagari conjuncts can vary.

Ethical & Responsible Use

Consent & Provenance: Speaker persona tags ([Speaker: Prakhar], [Speaker: Prerna]) are arbitrary names assigned to differentiate the male and female voice recordings already present in the open AI4Bharat Rasa dataset. No individual's specific personal voice or identity was used.
Prohibition of Impersonation: Users must not use this software or model weights for unauthorized voice impersonation, deceptive media creation, fraud, or non-consensual voice cloning.

💻 Source Code Repository

Full code, training scripts, preprocessing tools, and technical documentation: 👉 GitHub Repository: coriolis/orpheus-3b-tts-hindi

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

CoriolisTechnologies

Model Tree

Base

unsloth/orpheus-3b-0.1-ft

Adapter

this model

Input Modalities

Text

Output Modalities