Model Details

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Fine-tuning method: LoRA (PEFT) + GatingNetwork (linear head over student hidden states)
• Distillation strategy: Geometry-only (CKA) — Path B, tokenizer-agnostic
• Language(s): English, Portuguese, Chinese (inherited from base)
• License: Apache 2.0
• Developed by: build-small-hackathon team

Teachers

Distillation Approach

Path B — geometry-only, tokenizer-agnostic:

1. Each teacher processes its own tokenization of the same text. No shared vocabulary required.
2. Sequence representations are pooled via masked mean (attention mask weighted) to a single vector per model.
3. Linear projection heads map each teacher's hidden space into the student's hidden space (d=896).
4. A GatingNetwork (linear layer over student pooled state → softmax over 5 teachers) learns which teacher's geometry to prioritize per input.
5. Loss = task cross-entropy + λ·CKA geometry loss (student vs. gated teacher mixture).

The CKA geometry loss aligns the relational structure of representations (which samples are similar to which) rather than raw activation values, making it robust to dimension mismatch and tokenizer differences.

Usage

python
Copy code
from transformers import AutoTokenizer
from peft import PeftModel, AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained(
    "build-small-hackathon/deku",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("build-small-hackathon/deku")

inputs = tokenizer("Explain gradient descent in one sentence.", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Additional Artifacts

• GatingNetwork weights: gating.pt — torch.load("gating.pt"), state_dict for a nn.Linear(896, 5)
• Projection weights: projections.pt — list of 5 nn.Linear state dicts (teacher_i → student space)
• Visualization data: build-small-hackathon/ofa-viz-data — raw projected embeddings for 3D UMAP soul space
• Interactive Space: build-small-hackathon/one-for-all

Training

• Steps: 5000
• Batch size: 4 (gradient accumulation × 4 = effective 16)
• Optimizer: AdamW, lr=2e-4, cosine decay
• Hardware: Modal A10G (24 GB VRAM)
• Data: subset of HuggingFaceTB/smoltalk (all-Pro split)