h4bbo

FuseLLM-112M-Completion

README

License: other

Model details

Table

Architecture	Qwen3 (decoder-only causal LM)
Parameters	~112M (tied input/output embeddings)
Hidden size	512
Layers	8 (all full attention)
Attention heads	8 (8 KV heads)
Vocab size	151,936
Max context	2048
Precision	float32 (safetensors)
Training	From scratch, 4 epochs, 16,188 steps
Final train loss	~0.58

tie_word_embeddings: true — the output lm_head shares the input embedding matrix, so checkpoints store only one copy. This is expected, not a missing weight.

Intended use

Code completion for Habbo-style Java server code (raw prompt → continuation).
Local experimentation / distillation base.

What it is NOT

Not instruction-tuned / not a chat model. It was trained only on raw source code, never on chat/instruction data.
The Qwen3 ChatML chat template is included (it ships with the tokenizer) for tokenizer/tool compatibility, but the model has not learned to follow chat turns. Passing chat-formatted prompts will produce poor, often repetitive output. Use it in completion mode, not conversation mode.

Usage

transformers (recommended for completion)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

m   = AutoModelForCausalLM.from_pretrained("h4bbo/FuseLLM-112M")
tok = AutoTokenizer.from_pretrained("h4bbo/FuseLLM-112M")

prompt = "public class Room {\n  public void onEnter(Player p) {\n    "
ids = tok(prompt, return_tensors="pt").input_ids
out = m.generate(ids, max_new_tokens=64, do_sample=False,
                 repetition_penalty=1.1, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

llama.cpp (completion mode)

No GGUF is shipped in this repo. The HF model is verified to convert and run in llama.cpp; generate the GGUF locally:

bash
# 1) convert HF -> lossless fp16 GGUF
python convert_hf_to_gguf.py h4bbo/FuseLLM-112M --outtype f16 \
  --model-name FuseLLM-112M --outfile FuseLLM-112M.fp16.gguf
# (optional) 4-bit quantize
llama-quantize FuseLLM-112M.fp16.gguf FuseLLM-112M.Q4_K_M.gguf Q4_K_M

# 2) completion mode — pass the raw code seed, do NOT use chat/conversation mode.
llama-cli -m FuseLLM-112M.Q4_K_M.gguf -cnv -st --no-jinja \
  -f seed.txt -n 64 --temp 0.0 --repeat-penalty 1.1 --no-display-prompt < /dev/null

--no-jinja keeps the prompt raw (the embedded chat template exists but the model isn't chat-tuned, so conversation mode is not meaningful for this model).

Files

model.safetensors, config.json, generation_config.json — HF model
tokenizer.json, tokenizer_config.json, chat_template.jinja — tokenizer + ChatML template

Notes

Small model + limited-domain corpus: expect repetition on long generations; use a repetition penalty and keep continuations short.
Trained from scratch, so this is fully independent of any upstream Qwen weights. The Qwen3 architecture/tokenizer are reused for compatibility.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

h4bbo

Model Tree

Base

this model

Input Modalities

Text

Output Modalities