haster

Fermata-0.2B-v1

README

License: apache-2.0

Model summary

Table

Architecture	decoder-only Transformer, RoPE, RMSNorm, SwiGLU, GQA
Params	201,398,272 (~0.2B)
Layers / d_model	17 / 1024
Attention	16 query heads : 4 KV heads (GQA), FFN 2816
Context length	4096 tokens max (trained on 2048-token windows)
Tokenizer	`tokenizer_v2` — bar-major REMI+, vocab 9507, 24 TPQN
Precision	trained in fp8 (torchao) + `torch.compile`; weights ship bf16

Weights format

Inference weights are provided as model.safetensors in Llama (LlamaForCausalLM) layout with a matching config.json, so they load with transformers / vLLM directly. This layout was numerically verified against the native training checkpoint (greedy decode matches token-for-token). Note the tokenizer is custom (bar-major REMI+, in tokenizer/) — it is not an AutoTokenizer, so encoding/decoding to MusicXML/MIDI uses this project's pipeline, not AutoTokenizer.from_pretrained.

Tokenizer / codec

The tokenizer is bar-major REMI+: tracks are interleaved bar by bar so all instruments advance together, which gives the model an explicit notion of simultaneity and helps parts stop together instead of one voice trailing off.

A generation begins with a native conditioning header:

text

BOS [GENRE] [ENSEMBLE] TEMPO KEY METER <PLAN> {TRACK|DRUM}* <PLAN_END> (BAR ...)* EOS

Drums are encoded by their true General MIDI percussion number (resolved from <part-list>/midi-instrument/midi-unpitched), and note velocity / dynamics are preserved. Because drums are stored as real GM voices, decoded output renders correct kick / snare / hi-hat directly with no display→GM remap.

Prompt example

A generation is conditioned by a header of special tokens; the model then produces the bars after <PLAN_END>. These are the exact prompts used for the sample files below.

Rock band — distortion + overdrive guitar, electric bass, drums; A minor, 130 BPM:

text

<BOS> <GENRE_popular> <ENSEMBLE_band> <TEMPO_130> <KEY_0:minor> <METER_4/4><PLAN> <TRACK_30> <TRACK_29> <TRACK_33> <DRUM_TRACK> <PLAN_END>

Piano concerto — piano + string ensemble + cello; E♭ major, 96 BPM:

text

<BOS> <GENRE_classical> <ENSEMBLE_orchestral> <TEMPO_96> <KEY_-3:major> <METER_4/4><PLAN> <TRACK_0> <TRACK_48> <TRACK_42> <PLAN_END>

<TRACK_n> picks a General MIDI program (0 = piano, 30 = distortion guitar, 33 = electric bass, 48 = string ensemble, 42 = cello, …).
<DRUM_TRACK> adds a percussion lane.
<KEY_k:mode> uses k = signed sharps(+) / flats(−); <TEMPO_n> = BPM; <METER_a/b> = time signature.

Conditioning vocabulary

GENRE <GENRE_x> (9): classical · popular · jazz · folk · electronic · soundtrack · world · religious · unknown
ENSEMBLE <ENSEMBLE_x> (7): solo · duo · band · chamber · orchestral · choir · other
TRACK <TRACK_n>: any General MIDI program, n = 0–127; plus <DRUM_TRACK> for a percussion lane.
KEY <KEY_k:mode>: k = −7…+7 (sharps/flats), mode ∈ major · minor · dorian · phrygian · lydian · mixolydian · none.
TEMPO <TEMPO_n>: integer BPM (5–999).
METER <METER_a/b>: time signature (e.g. 4/4, 3/4, 6/8).

(The genre/ensemble labels are coarse; unknown / other are valid and common.)

Samples

samples/promo_rock.mid and samples/promo_piano_concerto.mid are ~3-minute pieces generated from the two prompts above (best-of-N seeds, CONTINUE-stitched past the context window). Render with any General MIDI SoundFont.

Training

Pretraining: from scratch, 3 epochs (13,821 steps), global batch 256, fp8 + torch.compile. Data: ~1.18M bar-major windows built from the public sources listed below, with ×12 transpose augmentation.
Instruction tuning (this checkpoint): GENERATE / CONTINUE / INFILL tasks mixed with in-key curation (in-key ≥ 0.88) and a base-replay set to limit forgetting. 760 steps (2 epochs), best val loss 0.3088 @ step 299 (adopted). Hardware: 1× RTX PRO 6000 Blackwell (96 GB), ~80k tok/s, ~45 min.

Evaluation

Automatic metrics on held-out generation prompts (2 seeds). Values are absolute for this model; real human-composed scores are given only as a rough anchor.

Table with columns: metric, Fermata-0.2B-v1, real scores (ref)
metric	Fermata-0.2B-v1	real scores (ref)
in-key ratio ↑	0.81	~0.96
dissonance ↓	0.29	~0.15
co-termination spread ↓	0.14	—
early part death ↓	0.10	—

Task success (model emits ≥ 1 new valid bar for the task):

Table with columns: task, success
task	success
CONTINUE	12/12
INFILL	12/12

in-key ratio — fraction of notes inside the conditioned key.
dissonance — ratio of dissonant simultaneous intervals.
co-termination spread / early part death — how tightly instruments end together vs. voices dropping out early (lower is better).

How to generate

Generation runs in this project's pipeline (bar-major decode → MusicXML/MIDI), then any General MIDI SoundFont (e.g. MuseScore) renders audio.

Recommended defaults: temperature 0.7, top_k 24, repetition_penalty ~1.12. This is the setting used for the sample pieces and is a good starting point.

Optional (supported when decoding with vLLM / transformers): top_p 0.95 and presence_penalty 0.15 for extra variety. ⚠️ Do not use frequency_penalty — it destabilizes the structural tokens (BAR / TRK / PLAN). For pieces longer than the context window, continue past it with a slightly lower continuation temperature (~0.76) to soften seams. Optionally post-process drums with a sanitize_drums step that snaps rare out-of-kit hits (e.g. GM 71/72) back to a standard kit.

Limitations

Context is 4096 tokens (~35 bars). Longer pieces are produced by stitching CONTINUE windows, which can cause a slight mood shift at the seam. A long-context retrain (or RoPE scaling at inference) would fix this properly.
Dissonance stays somewhat above real-score levels.
Research artifact; quality is not guaranteed for production use.

Known issues (found after release, 2026-09-29)

BPE merges were not applied to the training shards (a builder bug re-ordered the base alphabet after fitting), so the model was effectively trained on the atomic token stream.
Drum notes are shifted by +1 GM key: MusicXML midi-unpitched is 1-based and was used as-is, so e.g. some kicks/snares come out as neighbouring kit pieces or whistles (GM 71/72). Subtracting 1 from drum pitches at render time recovers most of it.
Both are fixed in the next tokenizer; this checkpoint is kept as-is for reproducibility.

Training data

Built from the following public symbolic-music sources:

PDMX — https://github.com/pnlong/PDMX
KernScores / Humdrum — https://github.com/humdrum-tools/humdrum-data · https://kern.humdrum.org/
OpenScore Lieder — https://github.com/OpenScore/Lieder
The Session — https://github.com/adactio/TheSession-data

License

Released under Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

haster

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model summary

Table

Architecture	decoder-only Transformer, RoPE, RMSNorm, SwiGLU, GQA
Params	201,398,272 (~0.2B)
Layers / d_model	17 / 1024
Attention	16 query heads : 4 KV heads (GQA), FFN 2816
Context length	4096 tokens max (trained on 2048-token windows)
Tokenizer	`tokenizer_v2` — bar-major REMI+, vocab 9507, 24 TPQN
Precision	trained in fp8 (torchao) + `torch.compile`; weights ship bf16

Weights format

Tokenizer / codec

A generation begins with a native conditioning header:

text

BOS [GENRE] [ENSEMBLE] TEMPO KEY METER <PLAN> {TRACK|DRUM}* <PLAN_END> (BAR ...)* EOS

Prompt example

A generation is conditioned by a header of special tokens; the model then produces the bars after <PLAN_END>. These are the exact prompts used for the sample files below.

Rock band — distortion + overdrive guitar, electric bass, drums; A minor, 130 BPM:

text

<BOS> <GENRE_popular> <ENSEMBLE_band> <TEMPO_130> <KEY_0:minor> <METER_4/4><PLAN> <TRACK_30> <TRACK_29> <TRACK_33> <DRUM_TRACK> <PLAN_END>

<BOS> <GENRE_popular> <ENSEMBLE_band> <TEMPO_130> <KEY_0:minor> <METER_4/4><PLAN> <TRACK_30> <TRACK_29> <TRACK_33> <DRUM_TRACK> <PLAN_END>

Piano concerto — piano + string ensemble + cello; E♭ major, 96 BPM:

text

<BOS> <GENRE_classical> <ENSEMBLE_orchestral> <TEMPO_96> <KEY_-3:major> <METER_4/4><PLAN> <TRACK_0> <TRACK_48> <TRACK_42> <PLAN_END>

<BOS> <GENRE_classical> <ENSEMBLE_orchestral> <TEMPO_96> <KEY_-3:major> <METER_4/4><PLAN> <TRACK_0> <TRACK_48> <TRACK_42> <PLAN_END>

<TRACK_n> picks a General MIDI program (0 = piano, 30 = distortion guitar, 33 = electric bass, 48 = string ensemble, 42 = cello, …).
<DRUM_TRACK> adds a percussion lane.
<KEY_k:mode> uses k = signed sharps(+) / flats(−); <TEMPO_n> = BPM; <METER_a/b> = time signature.

Conditioning vocabulary

GENRE <GENRE_x> (9): classical · popular · jazz · folk · electronic · soundtrack · world · religious · unknown
ENSEMBLE <ENSEMBLE_x> (7): solo · duo · band · chamber · orchestral · choir · other
TRACK <TRACK_n>: any General MIDI program, n = 0–127; plus <DRUM_TRACK> for a percussion lane.
KEY <KEY_k:mode>: k = −7…+7 (sharps/flats), mode ∈ major · minor · dorian · phrygian · lydian · mixolydian · none.
TEMPO <TEMPO_n>: integer BPM (5–999).
METER <METER_a/b>: time signature (e.g. 4/4, 3/4, 6/8).

(The genre/ensemble labels are coarse; unknown / other are valid and common.)

Samples

Training

Pretraining: from scratch, 3 epochs (13,821 steps), global batch 256, fp8 + torch.compile. Data: ~1.18M bar-major windows built from the public sources listed below, with ×12 transpose augmentation.
Instruction tuning (this checkpoint): GENERATE / CONTINUE / INFILL tasks mixed with in-key curation (in-key ≥ 0.88) and a base-replay set to limit forgetting. 760 steps (2 epochs), best val loss 0.3088 @ step 299 (adopted). Hardware: 1× RTX PRO 6000 Blackwell (96 GB), ~80k tok/s, ~45 min.

Evaluation

Automatic metrics on held-out generation prompts (2 seeds). Values are absolute for this model; real human-composed scores are given only as a rough anchor.

Table with columns: metric, Fermata-0.2B-v1, real scores (ref)
metric	Fermata-0.2B-v1	real scores (ref)
in-key ratio ↑	0.81	~0.96
dissonance ↓	0.29	~0.15
co-termination spread ↓	0.14	—
early part death ↓	0.10	—

Task success (model emits ≥ 1 new valid bar for the task):

Table with columns: task, success
task	success
CONTINUE	12/12
INFILL	12/12

in-key ratio — fraction of notes inside the conditioned key.
dissonance — ratio of dissonant simultaneous intervals.
co-termination spread / early part death — how tightly instruments end together vs. voices dropping out early (lower is better).

How to generate

Generation runs in this project's pipeline (bar-major decode → MusicXML/MIDI), then any General MIDI SoundFont (e.g. MuseScore) renders audio.

Recommended defaults: temperature 0.7, top_k 24, repetition_penalty ~1.12. This is the setting used for the sample pieces and is a good starting point.

Limitations

Context is 4096 tokens (~35 bars). Longer pieces are produced by stitching CONTINUE windows, which can cause a slight mood shift at the seam. A long-context retrain (or RoPE scaling at inference) would fix this properly.
Dissonance stays somewhat above real-score levels.
Research artifact; quality is not guaranteed for production use.

Known issues (found after release, 2026-09-29)

BPE merges were not applied to the training shards (a builder bug re-ordered the base alphabet after fitting), so the model was effectively trained on the atomic token stream.
Drum notes are shifted by +1 GM key: MusicXML midi-unpitched is 1-based and was used as-is, so e.g. some kicks/snares come out as neighbouring kit pieces or whistles (GM 71/72). Subtracting 1 from drum pitches at render time recovers most of it.
Both are fixed in the next tokenizer; this checkpoint is kept as-is for reproducibility.

Training data

Built from the following public symbolic-music sources:

PDMX — https://github.com/pnlong/PDMX
KernScores / Humdrum — https://github.com/humdrum-tools/humdrum-data · https://kern.humdrum.org/
OpenScore Lieder — https://github.com/OpenScore/Lieder
The Session — https://github.com/adactio/TheSession-data

License

Released under Apache-2.0.