Model Overview
This repository contains LoRA adapter weights, not a standalone copy of the full Llama-3.1-8B-Instruct model.
The underlying backbone remains:
meta-llama/Llama-3.1-8B-Instruct
LoRA adapters are inserted into the attention and MLP projection layers while the original backbone parameters remain frozen.
LoRA Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
LoRA rank r | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Bias | none |
| Task type | Causal Language Modeling |
| Trainable parameters | 41,943,040 |
| Total parameters | 8,072,204,288 |
| Trainable fraction | 0.5196% |
The adapted target modules are:
q_projk_projv_projo_projgate_projup_projdown_proj
These cover the major attention and feed-forward projection layers of the Llama-3.1-8B architecture.
Training Dataset
Training uses the Balanced CEFR Steering Subset constructed from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, with equal representation across all six CEFR levels:
Table with columns: CEFR Level, Samples| CEFR Level | Samples |
|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 split with random_state=42 is used:
Table with columns: Partition, Samples| Partition | Samples |
|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Training Prompt
Each training example includes:
- The target CEFR level
- The associated writing topic
- The corresponding reference text
The model receives a CEFR-conditioned generation instruction and is trained to predict only the target response tokens.
Prompt and padding tokens are masked from the causal language-modeling loss using the standard -100 ignore index.
Training Objective
Training uses the standard causal language-modeling cross-entropy objective on the reference response tokens.
Only the LoRA adapter parameters are optimized; the underlying Llama-3.1-8B-Instruct parameters remain frozen.
Conceptually:
LCLM=−t∈Ttarget∑logpθ(yt∣y<t,x)
where the loss is computed only over target response tokens.
Training Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Weight decay | 0.01 |
| LR schedule | Cosine |
| Warmup ratio | 5% |
| Optimization steps | 237 |
| Warmup steps | 11 |
| Per-device training batch size | 32 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 64 |
| Validation batch size | 32 |
| Maximum sequence length | 512 |
| Precision | bfloat16 |
| Gradient checkpointing | Enabled |
The checkpoint with the lowest validation loss is retained.
Training Progression
The recorded training trajectory is:
Table with columns: Epoch, Train Loss, Validation Loss, Validation Perplexity| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|
| 1 | 2.5610 | 2.4400 | 11.47 |
| 2 | 2.2555 | 2.3686 | 10.68 |
| 3 | 2.0510 | 2.3943 | 10.96 |
The best validation checkpoint is therefore the model from Epoch 2, selected according to the minimum validation loss.
Training loss continues to decrease across all three epochs, whereas validation loss reaches its minimum at Epoch 2 before increasing slightly during the final epoch.
This behavior motivates retaining the Epoch 2 checkpoint rather than the final training state.
Hardware Profiling
Training was performed on a single NVIDIA A100 GPU.
Recorded profiling statistics:
Table with columns: Metric, Value| Metric | Value |
|---|
| Total training time | 3044.87 s |
| Total training time | 0.85 h |
| Peak GPU memory allocation | 47.26 GB |
| Average GPU utilization | 98.6% |
These statistics correspond to the complete three-epoch LoRA training run.
Generation Evaluation
The trained adapter is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
For evaluation, the LoRA adapter is attached to the frozen Llama-3.1-8B-Instruct backbone and used for stochastic generation.
Generation Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
| Maximum new tokens | 200 |
| Temperature | 0.6 |
| Sampling | Enabled |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
The generated texts are evaluated using the primary CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
Evaluation Results
The LoRA baseline obtains the following results over the complete 702-condition in-domain evaluation matrix:
Table with columns: Metric, Result| Metric | Result |
|---|
| Total generations | 702 |
| Strict Accuracy | 26.21% |
| Adjacent Accuracy | 59.69% |
| Mean Absolute Error (MAE) | 1.4501 |
| Mean Dependency Distance (MDD) | 1.75 |
| Mean Flesch Reading Ease | 82.64 |
| Mean Sentence-Level CEFR Drift | 2.23 levels |
Metric Interpretation
Strict Accuracy
Strict Accuracy requires the predicted CEFR level to exactly match the requested target level.
Adjacent Accuracy
Adjacent Accuracy treats a prediction as correct when it is either the exact target level or one immediately neighboring CEFR level.
For example:
Target B1: B1 → Strict + Adjacent correctA2 / B2 → Adjacent correctA1 / C1 / C2 → Incorrect
Mean Absolute Error
Mean Absolute Error (MAE) measures the average ordinal distance between the requested and predicted CEFR levels.
If the CEFR levels are represented as:
A1 = 0A2 = 1B1 = 2B2 = 3C1 = 4C2 = 5
then:
MAE=N1i=1∑N∣y^i−yi∣
Lower values indicate that predicted proficiency levels remain closer to their requested targets.
Additional Linguistic Diagnostics
The evaluation also reports:
- Mean Dependency Distance (MDD) as a syntactic-complexity indicator
- Flesch Reading Ease as a readability indicator
- Sentence-level CEFR drift as a measure of proficiency consistency within generated texts
Role in the Thesis
LoRA is included as a generic parameter-efficient fine-tuning baseline against which the explicitly CEFR-conditioned methods are compared.
Unlike the proposed CEFR-specific controllers, LoRA does not introduce a dedicated CEFR control mechanism into the architecture.
Instead, CEFR behavior is learned through standard parameter-efficient adaptation from CEFR-conditioned training examples.
The LoRA baseline therefore provides a comparison between:
Generic PEFT Adaptation vs. Explicit CEFR-Conditioned Control
This distinction is important because it allows the experiments to investigate whether dedicated architectural mechanisms for proficiency control provide benefits beyond those obtainable through conventional parameter-efficient fine-tuning.
Reproducibility
The complete training and evaluation notebook is available in the thesis GitHub repository at:
notebooks/04_generic_peft_baselines/lora_train_and_eval.ipynb
The balanced CEFR training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation benchmark is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The notebook retains the original experimental outputs, including:
- Train/validation split statistics
- Trainable-parameter counts
- Epoch-level training and validation losses
- Validation perplexity
- Hardware-profiling results
- Generation outputs
- CEFR evaluation metrics
- Classification reports
- Confusion matrices
Intended Use
This adapter is intended for research on:
- CEFR-controlled text generation
- Parameter-efficient fine-tuning
- Controllable generation
- Language-proficiency-conditioned generation
- Comparison of generic PEFT methods with explicit control mechanisms
Important: This adapter is an experimental research artifact and is not intended as a standalone CEFR assessment model.
Citation
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond, author = {Mohammad Khosravi}, title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation}, school = {University of Padova}, year = {2026}}