Features
- Breeze-7B-Base-v0_1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 8k-token context length
- Breeze-7B-Instruct-v0_1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 8k-token context length
- Multi-turn dialogue (without special handling for harmfulness)
- Breeze-7B-Instruct-64k-v0_1
- Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
- 64k-token context length
- Multi-turn dialogue (without special handling for harmfulness)
Model Details
- Breeze-7B-Base-v0_1
- Finetuned from: mistralai/Mistral-7B-v0.1
- Model type: Causal decoder-only transformer language model
- Language: English and Traditional Chinese (zh-tw)
- Breeze-7B-Instruct-v0_1
- Breeze-7B-Instruct-64k-v0_1
TMMLU+, DRCD, and Table source from MediaTek-Research/TCEval-v2.
MediaTek-Research/TCEval-v2 derives from TCEval-v1
and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train.
We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, DRCD, Table, and MMLU. All choice problems adapt the selection by the log-likelihood.
Table with columns: Models, ↑ TMMLU+ (ACC), DRCD (EM), Table (ACC), MMLU (ACC)| Models | | ↑ TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC) |
|---|
| | TC, Knowledge | TC, Reasoning | TC, Reasoning | EN, Knowledge |
| | 5 shot | 3 shot | 5 shot | 5 shot |
| Yi-34B |
* Few-shot learning cannot effectively guide the model to generate the proper answer.
TMMLU+, DRCD, Table, and MT-Bench-tw source from MediaTek-Research/TCEval-v2.
MediaTek-Research/TCEval-v2 derives from TCEval-v1
and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train.
MT-Bench source from lmsys/mt_bench_human_judgments.
We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, DRCD, Table, and MMLU. All choice problems adapt the selection by the log-likelihood.
We use the code revised from fastchat llm_judge (GPT4 as judge) to evaluate MT-Bench-tw and MT-Bench.
Table with columns: Models, ↑ MT-Bench-tw (Score), TMMLU+ (ACC), TMMLU+ (ACC), DRCD (EM), Table (ACC), MT-Bench (Score), MMLU (ACC), MMLU (ACC)| Models | | ↑ MT-Bench-tw (Score) | TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MT-Bench (Score) | MMLU (ACC) | MMLU (ACC) |
|---|
| | TC, Chat | TC, Knowledge | TC, Knowledge | TC, Reasoning | TC, Reasoning | EN, Chat | EN, Knowledge | EN, Knowledge |
* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.
Table with columns: Details on MT-Bench-tw (0 shot):Models, STEM, Extraction, Reasoning, Math, Coding, Roleplay, Writing, Humanities, ↑ AVG| Details on MT-Bench-tw (0 shot):Models | STEM | Extraction | Reasoning | Math | Coding | Roleplay | Writing | Humanities | ↑ AVG |
|---|
| gpt-3.5-turbo | 7.8 | 6.1 | 5.1 | 6.4 | 6.2 | 8.7 | 7.4 | 9.3 | 7.1 |
Table with columns: Details on TMMLU+ (0 shot):Model, STEM, Social Science, Humanities, Other, ↑ AVG| Details on TMMLU+ (0 shot):Model | STEM | Social Science | Humanities | Other | ↑ AVG |
|---|
| Yi-34B-Chat | 47.65 | 64.25 | 52.73 | 54.91 | 54.87 |
| Qwen-14B-Chat | 43.83 | 55.00 | 48.55 | 46.22 | 48.41 |
| Yi-6B-Chat | 37.80 |
In this test, we use the first 700 characters of the web article as the input and ask the model to write the same article again.
All inferences run on 2 RTX A6000 GPUs (using vllm, with a tensor-parallel size of 2).
Table with columns: Models, ↓ Inference Time (sec), Estimated Max Input Length (Char)| Models | ↓ Inference Time (sec) | Estimated Max Input Length (Char) |
|---|
| Yi-6B-Chat | 10.62 | 5.2k |
| Breeze-7B-Instruct-v0_1 | 10.74 | 11.1k |
| Breeze-7B-Instruct-64k-v0_1 | 10.74 | 88.8k |
| Qwen-7B-Chat | 10.86 | 9.8k |
| Qwen-14B-Chat | 18.89 | 9.8k |
| Mistral-7B-v0.1-Instruct |
Long-context Performance
TBD
First install direct dependencies:
pip install transformers torch accelerate
If you want faster inference using flash-attention2, you need to install these dependencies:
pip install packaging ninja
pip install flash-attn
Then load the model in transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"MediaTek-Research/Breeze-7B-Instruct-v0_1",
device_map="auto",
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2"
)
The structure of the query is
<s>SYS_PROMPT [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST]
where SYS_PROMPT, QUERY1, RESPONSE1, and QUERY2 can be provided by the user.
The suggested default SYS_PROMPT is
You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
We also integrate chat_template into tokenizer_config.json, so you can apply_chat_template to get the prompt.
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("MediaTek-Research/Breeze-7B-Instruct-v0_1")
>>> chat = [
... {"role": "user", "content": "你好，請問你可以完成什麼任務？"},
... {"role": "assistant", "content": "你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。"},
... {"role": "user", "content": "太棒了！"},
... ]
>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan. [INST] 你好，請問你可以完成什麼任務？ [/INST] 你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。 [INST] 太棒了！ [/INST] "
Citation
@article{MediaTek-Research2024breeze7b,
title={Breeze-7B Technical Report},
author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},
year={2024},
eprint={2403.02712},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
