Apple silicon (MLX / omlx)
# converts + 4-bit quantizes into an MLX model dir
python3 -m mlx_lm.convert --hf-path lewisdog/qwen3-1.7b-cogs-ingest -q --mlx-path qwen3-cogs-ingest-mlx
# then drop into ~/.omlx/models and point cogs [llm] at it
⚠️ Serving: avoid pure greedy decoding
The model learned the schema well, but under pure greedy (temperature 0, no
penalty) it degenerates on the long list-valued tasks (extract,
suggest_links): it keeps appending list items and never emits <|im_end|>,
leaving valid-but-unterminated JSON. Fix with either:
repetition_penalty ≈ 1.1 (keeps determinism), or- Qwen non-thinking sampling:
temperature=0.7, top_p=0.8, top_k=20
(this model's bundled generation_config.json already samples at temp 0.6,
which sidesteps the issue — just don't override it to temp-0 greedy).
With either, a 5-sample / 4-task JSON parse eval is 5/5 (100%) schema-exact
(vs. 3/5 at plain greedy). Use enable_thinking=False and stop on <|im_end|>.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"lewisdog/qwen3-1.7b-cogs-ingest", dtype="bfloat16", device_map="auto"
).eval()
tok = AutoTokenizer.from_pretrained("lewisdog/qwen3-1.7b-cogs-ingest")
enc = tok.apply_chat_template(
messages, add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True,
).to(model.device)
out = model.generate(**enc, max_new_tokens=2048, do_sample=False,
repetition_penalty=1.1, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
Training
LoRA SFT (TRL) on 1,921 cogs distill chat pairs, 2 epochs, effective batch 16,
max_seq 8192, lr 1e-4 cosine, bf16, on an NVIDIA DGX Spark (GB10).
Train loss 2.55 → 1.41; eval loss 1.125 → 1.118 (still decreasing); eval
token-accuracy 0.756. Full details: adapter repo card + RESULTS.md.
- PEFT 0.19.1 · TRL 1.7.1 · Transformers 5.13.0 · PyTorch 2.12.1+cu130