GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
homebrewltd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
homebrewltd
Model Tree
Input Modalities
Output Modalities
Supported Functionality
We have developed and released the family llama3-s. This family is natively understanding audio and text input.
We continue to expand Meta-Llama-3-8B-Instruct with sound understanding capabilities by leveraging 700M tokens Instruction Speech v1 dataset.
Model developers Homebrew Research.
Input Text and sound.
Output Text.
Model Architecture Llama-3.
Language(s): English.
Intended Use Cases This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
Out-of-scope The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.
Try the model out using inference notebook.
First, we need to convert the audio file to sound tokens
python
import torchimport torchaudiofrom encodec import EncodecModelfrom encodec.utils import convert_audio def audio_to_sound_tokens(audio_path, target_bandwidth=1.5, device="cuda"): # Initialize Encodec model = EncodecModel.encodec_model_24khz() model.set_target_bandwidth(target_bandwidth) model.to(device) # Load and preprocess audio wav, sr = torchaudio.load(audio_path) wav = convert_audio(wav, sr, model.sample_rate, model.channels) wav = wav.unsqueeze(0).to(device) # Encode audio with torch.no_grad(): encoded_frames = model.encode(wav) codes = torch.cat([encoded[0] for encoded in encoded_frames], dim=-1) # Flatten codes audio_code1, audio_code2 = codes[0][0], codes[0][1] flatten_tokens = torch.stack((audio_code1, audio_code2), dim=1).flatten().tolist() # Convert to sound tokens result = ''.join(f'<|sound_{num:04d}|>' for num in flatten_tokens) return f'<|sound_start|>{result}<|sound_end|>' # Usagesound_tokens = audio_to_sound_tokens("/path/to/your/audio/file")Then, we can inference the model the same as any other LLM.
python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline def setup_pipeline(model_path, use_4bit=False, use_8bit=False): tokenizer = AutoTokenizer.from_pretrained(model_path) model_kwargs = {"device_map": "auto"} if use_4bit: model_kwargs["quantization_config"] = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", ) elif use_8bit: model_kwargs["quantization_config"] = BitsAndBytesConfig( load_in_8bit=True, bnb_8bit_compute_dtype=torch.bfloat16, bnb_8bit_use_double_quant=True, ) else: model_kwargs["torch_dtype"] = torch.bfloat16 model = AutoModelForCausalLM.from_pretrained(model_path, **model_kwargs) return pipeline("text-generation", model=model, tokenizer=tokenizer) def generate_text(pipe, messages, max_new_tokens=64, temperature=0.0, do_sample=False): generation_args = { "max_new_tokens": max_new_tokens, "return_full_text": False, "temperature": temperature, "do_sample": do_sample, } output = pipe(messages, **generation_args) return output[0]['generated_text'] # Usagellm_path = "jan-hq/Jan-Llama3s-0708"pipe = setup_pipeline(llm_path, use_4bit=True)messages = [ {"role": "user", "content": sound_tokens},]generated_text = generate_text(pipe, messages)print(generated_text)Training Metrics Image: Below is a snapshot of the training loss curve visualized.
GPU Configuration: Cluster of 8x NVIDIA H100-SXM-80GB. GPU Usage:
| Parameter | Continual Training |
|---|---|
| Epoch | 1 |
| Global batch size | 128 |
| Learning Rate | 5e-5 |
| Learning Scheduler | Cosine with warmup |
| Optimizer | Adam-mini |
| Warmup Ratio | 0.1 |
| Weight Decay | 0.01 |
| beta1 | 0.9 |
| beta2 | 0.98 |
| epsilon | 1e-6 |
| Gradient Cliping | 1.0 |
Accelerate FSDP Config
text
compute_environment: LOCAL_MACHINEdebug: falsedistributed_type: FSDPdowncast_bf16: 'no'enable_cpu_affinity: truefsdp_config: fsdp_activation_checkpointing: true fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP fsdp_backward_prefetch: BACKWARD_PRE fsdp_cpu_ram_efficient_loading: true fsdp_forward_prefetch: false fsdp_offload_params: false fsdp_sharding_strategy: FULL_SHARD fsdp_state_dict_type: SHARDED_STATE_DICT fsdp_sync_module_states: true fsdp_use_orig_params: falsemachine_rank: 0main_training_function: mainmixed_precision: bf16num_machines: 1num_processes: 8rdzv_backend: staticsame_network: truetpu_env: []tpu_use_cluster: falsetpu_use_sudo: falseuse_cpu: falsetext
# Question:Does the following review have a positive or negative opinion of the movie? i thought the relationships were wonderful , Options: A). negative; B). positive; # Expected Output:The review has a B). positive opinion about the movie because they think the relationships were wonderful. # Model's Output:The review is positive.text
# Question:Translate the following sentence to Russian:6 Work in JOBS and careers after 40 Russian: # Expected Output:6 Работать в РАБОТАХ и карьере после 40 Russian (Cyrillic): 6 Работать в РАБОТАХ и карьере после 40 Let me explain this to you like you're five years old: This sentence means that people can still work and have careers, even if they're older than 40 years old. The words are in a different language, called Russian.# Model's Output:В случае, когда вы хотите перевести фразу "The first step is to identify the language of the text." на русский язык, вы можете использовать следующую фразу: "Первый шаг - определить язык текста."text
# Quesion:Answer the following question: - The roots of the growing plants start to break up the rock - The plant acids dissolve the rock - The rock is broken down into smaller pieces - Erosion begins. What might be the first step of the process?Answer: # Expected Output:The first step of the process might be: The roots of the growing plants start to break up the rock. # Model's Output:Step 1: Identify the key elements in the premise and hypothesis. Premise: "A man in a blue shirt and a red hat is holding a sign."Key elements: man, blue shirt, red hat, holding, sign Hypothesis: "A man is holding a sign."Key elements: man, holding, sign Step 2: Compare the key elements in the premise and hypothesis. In the premise, we have a man wearing a blue shirt and a red hat, while in the hypothesis, we only have a man. The premise also mentions that the man is holding a sign, which is also mentioned in the hypothesis. Step 3: Determine if the hypothesis is entailed by the premise. Since the hypothesis states that a man is holding a sign, and the premise confirms that a man is holding a sign, we can conclude that the hypothesis is entailed by the premise. The additional information about the man's clothing in the premise does not contradict or negate theDespite being undertrained, the model demonstrates an emerging grasp of sound-text semantics.
BibTeX:
text
@article{llama3-s: Sound Instruction Language Model 2024, title={llama3-s}, author={Homebrew Research}, year=2024, month=July}, url={https://huggingface.co/homebrew-research/llama3-s-0708}import torchimport torchaudiofrom encodec import EncodecModelfrom encodec.utils import convert_audio def audio_to_sound_tokens(audio_path, target_bandwidth=1.5, device="cuda"): # Initialize Encodec model = EncodecModel.encodec_model_24khz() model.set_target_bandwidth(target_bandwidth) model.to(device) # Load and preprocess audio wav, sr = torchaudio.load(audio_path) wav = convert_audio(wav, sr, model.sample_rate, model.channels) wav = wav.unsqueeze(0).to(device) # Encode audio with torch.no_grad(): encoded_frames = model.encode(wav) codes = torch.cat([encoded[0] for encoded in encoded_frames], dim=-1) # Flatten codes audio_code1, audio_code2 = codes[0][0], codes[0][1] flatten_tokens = torch.stack((audio_code1, audio_code2), dim=1).flatten().tolist() # Convert to sound tokens result = ''.join(f'<|sound_{num:04d}|>' for num in flatten_tokens) return f'<|sound_start|>{result}<|sound_end|>' # Usagesound_tokens = audio_to_sound_tokens("/path/to/your/audio/file")from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline def setup_pipeline(model_path, use_4bit=False, use_8bit=False): tokenizer = AutoTokenizer.from_pretrained(model_path) model_kwargs = {"device_map": "auto"} if use_4bit: model_kwargs["quantization_config"] = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", ) elif use_8bit: model_kwargs["quantization_config"] = BitsAndBytesConfig( load_in_8bit=True, bnb_8bit_compute_dtype=torch.bfloat16, bnb_8bit_use_double_quant=True, ) else: model_kwargs["torch_dtype"] = torch.bfloat16 model = AutoModelForCausalLM.from_pretrained(model_path, **model_kwargs) return pipeline("text-generation", model=model, tokenizer=tokenizer) def generate_text(pipe, messages, max_new_tokens=64, temperature=0.0, do_sample=False): generation_args = { "max_new_tokens": max_new_tokens, "return_full_text": False, "temperature": temperature, "do_sample": do_sample, } output = pipe(messages, **generation_args) return output[0]['generated_text'] # Usagellm_path = "jan-hq/Jan-Llama3s-0708"pipe = setup_pipeline(llm_path, use_4bit=True)messages = [ {"role": "user", "content": sound_tokens},]generated_text = generate_text(pipe, messages)print(generated_text)compute_environment: LOCAL_MACHINEdebug: falsedistributed_type: FSDPdowncast_bf16: 'no'enable_cpu_affinity: truefsdp_config: fsdp_activation_checkpointing: true fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP fsdp_backward_prefetch: BACKWARD_PRE fsdp_cpu_ram_efficient_loading: true fsdp_forward_prefetch: false fsdp_offload_params: false fsdp_sharding_strategy: FULL_SHARD fsdp_state_dict_type: SHARDED_STATE_DICT fsdp_sync_module_states: true fsdp_use_orig_params: falsemachine_rank: 0main_training_function: mainmixed_precision: bf16num_machines: 1num_processes: 8rdzv_backend: staticsame_network: truetpu_env: []tpu_use_cluster: falsetpu_use_sudo: falseuse_cpu: false# Question:Does the following review have a positive or negative opinion of the movie?i thought the relationships were wonderful , Options: A). negative; B). positive;# Expected Output:The review has a B). positive opinion about the movie because they think the relationships were wonderful.# Model's Output:The review is positive.# Question:Translate the following sentence to Russian:6 Work in JOBS and careers after 40Russian:# Expected Output:6 Работать в РАБОТАХ и карьере после 40Russian (Cyrillic): 6 Работать в РАБОТАХ и карьере после 40Let me explain this to you like you're five years old: This sentence means that people can still work and have careers, even if they're older than 40 years old. The words are in a different language, called Russian.# Model's Output:В случае, когда вы хотите перевести фразу "The first step is to identify the language of the text." на русский язык, вы можете использовать следующую фразу: "Первый шаг - определить язык текста."# Quesion:Answer the following question: - The roots of the growing plants start to break up the rock - The plant acids dissolve the rock - The rock is broken down into smaller pieces - Erosion begins. What might be the first step of the process?Answer:# Expected Output:The first step of the process might be: The roots of the growing plants start to break up the rock.# Model's Output:Step 1: Identify the key elements in the premise and hypothesis.Premise: "A man in a blue shirt and a red hat is holding a sign."Key elements: man, blue shirt, red hat, holding, signHypothesis: "A man is holding a sign."Key elements: man, holding, signStep 2: Compare the key elements in the premise and hypothesis.In the premise, we have a man wearing a blue shirt and a red hat, while in the hypothesis, we only have a man. The premise also mentions that the man is holding a sign, which is also mentioned in the hypothesis.Step 3: Determine if the hypothesis is entailed by the premise.Since the hypothesis states that a man is holding a sign, and the premise confirms that a man is holding a sign, we can conclude that the hypothesis is entailed by the premise. The additional information about the man's clothing in the premise does not contradict or negate the@article{llama3-s: Sound Instruction Language Model 2024, title={llama3-s}, author={Homebrew Research}, year=2024, month=July}, url={https://huggingface.co/homebrew-research/llama3-s-0708}