Model Overview
Standard Prefix-Tuning learns continuous virtual prefix representations while keeping the underlying language-model parameters frozen.
The prefix representations are injected into the transformer's attention key/value states through the PEFT Prefix-Tuning implementation.
The configuration used in this experiment is:
Frozen Llama-3.1-8B-Instruct + 30 Virtual Prefix Tokens + Prefix Projection
Prefix-Tuning Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
| PEFT method | Prefix-Tuning |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Number of virtual tokens | 30 |
| Prefix projection | True |
| Task type | Causal Language Modeling |
| Trainable parameters | 68,254,720 |
| Total parameters | 8,098,515,968 |
| Trainable fraction | 0.8428% |
Only the Prefix-Tuning parameters are optimized during training. The Llama-3.1-8B-Instruct backbone remains frozen.
Training Dataset
Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, balanced equally across the six CEFR levels:
Table with columns: CEFR Level, Samples| CEFR Level | Samples |
|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 train-validation split is used with random_state=42:
Table with columns: Partition, Samples| Partition | Samples |
|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Training Prompt and Target Construction
Each training example contains:
- A target CEFR level
- A writing topic
- A reference response from the steering dataset
The textual instruction explicitly specifies the requested CEFR level.
The training sequence consists of the formatted instruction followed by the reference response.
Prompt and padding tokens are excluded from the causal language-modeling loss using the standard -100 ignore index, so optimization is performed only on the target response tokens.
Training Objective
Training uses the standard autoregressive causal language-modeling cross-entropy objective.
The backbone parameters remain frozen and gradients update only the learnable Prefix-Tuning parameters.
Conceptually, the target-only objective is:
LCLM=−t∈Ttarget∑logpθ(yt∣y<t,x)
where Ttarget contains only the response-token positions included in the loss.
Training Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Weight decay | 0.01 |
| Learning-rate schedule | Cosine |
| Warmup ratio | 5% |
| Training batch size | 16 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 32 |
| Validation batch size | 32 |
| Maximum sequence length | 512 |
| Precision | bfloat16 |
| Gradient clipping | 1.0 |
| Checkpoint criterion | Minimum validation loss |
Training Progression
The recorded training trajectory is:
Table with columns: Epoch, Train Loss, Validation Loss, Validation Perplexity| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|
| 1 | 2.6629 | 2.5390 | 12.67 |
| 2 | 2.4541 | 2.4701 | 11.82 |
| 3 | 2.3293 | 2.4610 | 11.72 |
Validation performance improves across all three epochs.
Validation loss decreases from:
while validation perplexity decreases from:
The lowest validation loss and perplexity are obtained at Epoch 3. The Epoch 3 adapter is therefore retained as the final checkpoint.
Generation Evaluation
The trained Prefix-Tuning adapter is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
For evaluation, the trained Prefix-Tuning adapter is attached to the frozen Llama-3.1-8B-Instruct backbone using PEFT.
Generation Configuration
Table with columns: Setting, Value| Setting | Value |
|---|
| Evaluation batch size | 32 |
| Maximum new tokens | 200 |
| Temperature | 0.6 |
| Sampling | Enabled |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
Generated texts are evaluated using the primary CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
Evaluation Results
The Standard Prefix-Tuning baseline obtains the following results over the complete 702-condition in-domain evaluation matrix:
Table with columns: Metric, Result| Metric | Result |
|---|
| Total generations | 702 |
| Strict Accuracy | 32.34% |
| Adjacent Accuracy | 64.67% |
| Mean Absolute Error (MAE) | 1.2422 |
| Mean Dependency Distance (MDD) | 1.80 |
| Mean Flesch Reading Ease | 81.73 |
| Mean Sentence-Level CEFR Drift | 2.17 levels |
Metric Interpretation
Strict Accuracy
A generation is counted as correct only when the predicted CEFR level exactly matches the requested target level.
Adjacent Accuracy
Adjacent Accuracy additionally treats predictions one CEFR level away from the requested target as correct.
For example:
Target B1: B1 → Strict + Adjacent correctA2 / B2 → Adjacent correctA1 / C1 / C2 → Incorrect
Mean Absolute Error
CEFR levels are mapped to ordered class indices:
A1 = 0A2 = 1B1 = 2B2 = 3C1 = 4C2 = 5
MAE measures the average ordinal distance between the requested and predicted CEFR levels:
MAE=N1i=1∑N∣y^i−yi∣
Lower values indicate smaller CEFR-control errors.
Linguistic Diagnostics
The evaluation additionally reports:
- Mean Dependency Distance (MDD) as a syntactic-complexity diagnostic
- Flesch Reading Ease as a readability diagnostic
- Sentence-level CEFR drift as a measure of proficiency consistency within an individual generated response
These diagnostics complement the primary CEFR-control metrics.
Role in the Thesis
Standard Prefix-Tuning is included as a generic parameter-efficient baseline against which the explicitly CEFR-conditioned methods are compared.
Unlike the proposed CEFR-specific controllers, this baseline does not contain a dedicated architectural CEFR-conditioning mechanism.
The target proficiency level is supplied through the textual instruction, while the same shared learnable prefix mechanism is used across CEFR levels.
This provides a comparison between:
Generic Prefix-Tuning vs. Explicit CEFR-Conditioned Prefix Control
The experiment is therefore useful for separating improvements due to explicit CEFR conditioning from improvements that can be obtained through standard parameter-efficient adaptation alone.
Relationship to PrefixMemory-Tuning
This repository implements the Standard Prefix-Tuning baseline using the PEFT Prefix-Tuning mechanism.
It should be distinguished from the PrefixMemory-Tuning and CEFR-conditioned controller variants evaluated elsewhere in the thesis.
In this baseline:
- One shared prefix mechanism is trained
- CEFR information is provided through the textual instruction
- No dedicated CEFR embedding is introduced
- No CEFR-specific gating mechanism is introduced
Conceptually:
Standard Prefix-Tuning │ ┌─────────────┴─────────────┐ │ │ CEFR Information Prefix Parameters │ │ Textual Instruction Shared Prefix │ │ └─────────────┬─────────────┘ │ ▼ Frozen Llama-3.1-8B
This differs from the explicitly conditioned architectures, where CEFR information directly controls dedicated trainable architectural components.
Reproducibility
The complete training and evaluation notebook is available in the thesis GitHub repository at:
notebooks/04_generic_peft_baselines/standard_prefix_tuning_train_and_eval.ipynb
The balanced training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation benchmark is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The experimental notebook retains the original outputs, including:
- Train/validation split statistics
- Trainable-parameter counts
- Epoch-level training losses
- Validation losses and perplexities
- Generation outputs
- CEFR evaluation metrics
- Classification reports
- Confusion matrices
Intended Use
This adapter is intended for research on:
- Prefix-Tuning
- Parameter-efficient fine-tuning
- CEFR-controlled text generation
- Proficiency-conditioned generation
- Controllable language generation
- Comparison of generic PEFT methods with explicit control mechanisms
Important: This repository contains an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.
Citation
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond, author = {Mohammad Khosravi}, title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation}, school = {University of Padova}, year = {2026}}