Author: Malek Camilo
Institution: Facultad de Informática, Universidad Nacional de La Plata
Thesis: Estrategias de adaptación eficiente para modelos de lenguaje abiertos
Repository: github.com/malekcamilo/tesis_maestria
This repository contains a QDoRA adapter for document type classification in the Computer Science subset of SEDICI records.
Base model
[!WARNING]
Base Model License: This adapter is built on top of meta-llama/Meta-Llama-3.1-8B-Instruct, which is a gated model. Users must explicitly agree to the Meta Llama 3.1 Community License on Hugging Face to use it.
The adapter was trained on top of meta-llama/Meta-Llama-3.1-8B-Instruct.
Users must have access to the base model in order to load this adapter.
Task
Given a title and abstract, the model predicts one of the following document types:
- Article
- Conference Paper
- Thesis
- Book
- Learning Resource
- Other
The generated label is returned in Spanish, matching the labels used during training.
Training data
The dataset contains 19,974 records from SEDICI in the Computer Science domain. Each instance includes title, abstract and document type.
Tokenizer
This adapter uses the original tokenizer distributed with Meta-Llama-3.1-8B-Instruct. No additional tokens or vocabulary modifications were introduced during fine-tuning.
Method
The model was adapted using QDoRA (rank = 128) with NF4 4-bit quantization and LoRA adapters applied to all attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).
Hardware
Training was performed on an Intel® Data Center GPU Max 1550 using Intel Extension for PyTorch (IPEX) with bfloat16 precision.
Training Configuration
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Epochs | 5 |
| Learning Rate | 2e-4 |
| Batch Size | 8 |
| Gradient Accumulation | 1 |
| Max Sequence Length | 512 |
| Rank | 128 |
| Alpha | 256 |
| Dropout | 0.1 |
Training Metrics
- Train Time: 9.48 hours
- Peak Memory: 14.14 GB
- Trainable Parameters: 336920576
Evaluation
Table with columns: Metric, Value| Metric | Value |
|---|
| Accuracy | 0.8625 |
| Macro-F1 | 0.6582 |
| Exact Match | 0.8589 |
Per-Class Results
Table with columns: Category, Precision, Recall, F1-Score| Category | Precision | Recall | F1-Score |
|---|
| Objeto De Conferencia | 0.8945 | 0.9518 | 0.9222 |
| Objeto De Aprendizaje | 0.8696 | 0.7143 | 0.7843 |
| Articulo | 0.7089 | 0.4266 | 0.5326 |
| Tesis | 0.7312 | 0.6570 | 0.6921 |
|
The adapter was trained using the Meta-Llama 3.1 chat template with the following system instruction:
System:
Eres un clasificador automático de documentos académicos. Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.
Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. No incluyas explicaciones, puntuación adicional ni texto conversacional.
Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje.
Usage
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
)
from peft import PeftModel
base_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "malekcamilo/sedici-llama-qdora-r128-all"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(
base_model,
device_map="auto",
quantization_config=bnb_config,
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()
def classify(title: str, abstract: str) -> str:
messages = [
{
"role": "system",
"content": (
"Eres un clasificador automático de documentos académicos. "
"Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.\n"
"Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. "
"No incluyas explicaciones, puntuación adicional ni texto conversacional.\n"
"Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje."
),
},
{
"role": "user",
"content": (
f"Clasifica el siguiente registro:\n"
f"<titulo>{title}</titulo>\n"
f"<resumen>{abstract}</resumen>"
),
},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=10,
do_sample=False,
)
generated = outputs[0][inputs["input_ids"].shape[-1]:]
return tokenizer.decode(
generated,
skip_special_tokens=True,
).strip()
prediction = classify(
"Clasificación automática de documentos científicos",
"En este trabajo se propone un método para clasificar documentos utilizando grandes modelos de lenguaje."
)
print(prediction)
Reproducibility
- Base model: Meta-Llama-3.1-8B-Instruct
- Fine-tuning: QDoRA
- Rank: 128
- Seed: 42
- Dataset size: 19,974 documents
- Sequence length: 512
- Precision: bfloat16
- Frameworks:
- Transformers
- PEFT
- TRL
- Intel Extension for PyTorch (IPEX)
Limitations
This adapter was trained exclusively on Computer Science records from SEDICI.
Performance on other academic repositories, languages or taxonomies has not been evaluated.
The model is intended for research purposes.
Citation
If you use this adapter in your research, please cite the associated Master's thesis:
@mastersthesis{camilo2026peft,
author = {Malek Camilo},
title = {Estrategias de adaptación eficiente para modelos de lenguaje abiertos},
school = {Facultad de Informática, Universidad Nacional de La Plata},
year = {2026},
type = {Master's thesis},
note = {Manuscript submitted in partial fulfillment of the requirements for the Master's degree},
}