Model Details
Model Description
This repo contains only the LoRA adapter weights (~18 MB) — not a merged
model. Load it on top of the base model with 🤗 PEFT (see below).
- Developed by: Bernardo Sana (bsana1)
- Model type: Causal decoder-only LLM, LoRA adapter (rank 16)
- Language(s): Portuguese (pt-BR)
- License: Apache 2.0 — inherited from the base model
- Finetuned from model: TucanoBR/Tucano-1b1-Instruct
(1.1B params, Llama2-style architecture, natively pretrained in Portuguese
on GigaVerbo by the Tucano team at the University of Bonn)
Model Sources
Uses
Direct Use
Instruction-following text generation in Portuguese, specifically for
cordel/glosa-style poetry: given a mote (a proverb or saying) or a theme,
generate a rhyming stanza in that tradition. Works through the base model's
chat template (see usage example below).
Out-of-Scope Use
This is a small hobby/learning project, not a production model. It was
fine-tuned on ~650 examples (50 hand-written + 600 synthetic) — it has not
been evaluated for factual accuracy, safety, or any use outside creative
Portuguese-language poetry generation, and shouldn't be used for anything
that requires reliability guarantees.
Bias, Risks, and Limitations
Inherits the base model's general limitations (Tucano-1b1-Instruct, 1.1B
params). The fine-tuning set is small and cordel poetry draws on 19th/20th
century Northeastern Brazilian folk material, so outputs can reflect the
period's language, social attitudes, and references. Not evaluated for bias.
How to Get Started with the Model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "TucanoBR/Tucano-1b1-Instruct"
adapter = "bsana1/arrodeio-tucano-cordel-lora"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, adapter)
model.eval()
msg = [{"role": "user", "content": "Glose o mote: «Água mole em pedra dura.»"}]
ids = tok.apply_chat_template(msg, tokenize=True, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=110, do_sample=True, temperature=0.7,
top_k=50, repetition_penalty=1.2, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Training Details
Training Data
650 mote → glosa / tema → cordel instruction pairs:
- 50 hand-written by the project author (
data/sft.jsonl)
- 600 synthesized via instruction backtranslation (
data/sft_synth.jsonl) —
real public-domain cordel stanzas (from pt.wikisource.org, long-dead
authors only) paired with a fabricated instruction that would plausibly
produce them, generated by scripts/make_sft_data.py
Training Procedure
LoRA fine-tuning with prompt-masked loss (loss computed only on the
response tokens, via the base model's chat template).
Training Hyperparameters
- LoRA rank (r): 16
- LoRA alpha: 32
- LoRA dropout: 0.05
- Target modules:
q_proj, k_proj, v_proj, o_proj
- Learning rate: 2e-4
- Epochs: 2
- Training regime: bf16
- Task type: CAUSAL_LM
Speeds, Sizes, Times
Trained on a free Google Colab T4 GPU in about 10 minutes. Adapter weights:
~18 MB.
Evaluation
No formal benchmark — evaluated qualitatively against the earlier stages of
the same project (from-scratch model, fine-tuned GPT-2). See
samples/comparison.md
in the GitHub repo for the same prompts run through every stage side by side.
Environmental Impact
Training took ~10 minutes on a single shared T4 GPU (Google Colab free tier) —
negligible compute. Inference in the live demo runs on Hugging Face's
ZeroGPU (shared A10G, on-demand).
Technical Specifications
Model Architecture and Objective
Base: Tucano-1b1-Instruct, a Llama2-style causal decoder-only transformer
(1.1B params) natively pretrained in Portuguese. This adapter adds LoRA
matrices to the attention projections only (q/k/v/o_proj); all base
weights stay frozen.
Compute Infrastructure
Hardware
Google Colab, free-tier T4 GPU.
Software
PyTorch, 🤗 Transformers, 🤗 PEFT.
Questions/issues: open one on github.com/bsana1/arrodeio-llm.