chartreuse-verte

orb-human-typeahead-350m-v1

README

License: apache-2.0

Variants

Table with columns: Path, Format, Use with
Path	Format	Use with
`/`	safetensors bf16, plain `granite` arch	transformers
`GGUF/orb-human-typeahead-350m-v1-Q8_0.gguf`	GGUF Q8_0	llama.cpp / llama-cpp-python

This 350M variant is attention-only and dense, the weights are republished here as the equivalent plain granite architecture (logit-identical, verified), which loads everywhere without extras.

Prompt format

Plain text, no chat template. Optional character summary, a marker line, name-prefixed turns, and finally the user's draft — the model continues the draft. Cut the suggestion at the first newline.

markdown
<character summary, optional>
***Roleplay chat below***
Sylvara: *She looks down from the watchtower and sees you.*
Traveler: *I approach the encampment.*
Sylvara: *She lowers her bow as you approach the gate.* "State your business, traveler."
Traveler: *I raise both hands slowly and

A completion looks like step into the torchlight, keeping my voice low.*

Notes:

Trained to trigger at word boundaries only (draft ends on a whole word, or on a trailing space). Mid-word completion is out of scope.
Trained context: up to ~4 recent turns, summaries ≤400 chars (prune tail), turns ≤500 chars (prune head).

Serving recipe

Greedy, short budget, stop at newline — mirrors how it was trained and evaluated:

python
from llama_cpp import Llama

llm = Llama("GGUF/orb-human-typeahead-350m-v1-Q8_0.gguf", n_ctx=1024)
out = llm.create_completion(prompt, max_tokens=12, stop=["\n"], temperature=0.0)
print(out["choices"][0]["text"])

Or with transformers:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("chartreuse-verte/orb-human-typeahead-350m-v1")
model = AutoModelForCausalLM.from_pretrained("chartreuse-verte/orb-human-typeahead-350m-v1")
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=12, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).split("\n")[0])

Evaluation

Scored on a held-out validation set of real roleplay conversations (120 prompts, conversation-disjoint from training; greedy, 12-token budget, suggestions cut at newline). word-EM@k = fraction of prompts where the first k words of the suggestion exactly match what the user really typed next; prefix-chars = mean length of the exactly-matching leading characters.

Table with columns: Metric, granite-4.0-350m-base, this model
Metric	granite-4.0-350m-base	this model
word-EM@1	0.158	0.300
word-EM@2	0.017	0.083
word-EM@3	0.000	0.033
prefix-chars	1.82	2.89
completion perplexity	29.56	12.97
gen speed on DDR4 + Ryzen 5 (t/s)

Absolute numbers are modest by design — creative roleplay is high-entropy, so even a perfect suggester can't guess most continuations. The gap vs the base model is the point: the base model's suggestions are fluent but off-scene.

Training

Two-stage full fine-tune of granite-4.0-350m-base: first on synthetic multi-turn RP data, but the adversarial model is instructed to give concise and low-entropy responses, then on a smaller private in-domain set in the serve-time prompt format (with stage-1 replay to limit forgetting). Examples are user turns split at word boundaries; loss is on the continuation. Suggestions are single-line by construction (completions end at newline).

Limitations

English-centric, roleplay register (asterisk actions, quoted dialogue). Out of domain for assistant chat, code, or formal prose.
Roleplay corpora include mature themes; suggestions can reflect that.
Not an instruction follower — it only continues drafts in the format above.
Suggests at word boundaries; won't complete a half-typed word.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

chartreuse-verte

Model Tree

Base

ibm-granite/granite-4.0-350m-base

Quantized

this model

Input Modalities

Text

Output Modalities