Kodjaoglanian

Qwen-3B-ASVD-Healed

README

License: apache-2.0

Architecture Summary

Qwen-3B-ASVD-Healed is a natively compressed iteration of the Qwen/Qwen2.5-3B architecture. This model serves as an empirical validation of a post-training structural compression pipeline that combines Activation-Aware Singular Value Decomposition (ASVD) with Pruning-Aware Fine-Tuning (LoRA Healing).

The objective of this architecture is to reduce the memory footprint and computational overhead of dense Transformer models without relying on static quantization formats (e.g., INT4/INT8), which often introduce dequantization latency during inference. By isolating and eliminating redundant weights at the matrix level, this model permanently removes 163,045,888 parameters while maintaining native FP16 precision.

The Compression Pipeline

The artifact was generated through a three-stage hardware-aware pipeline designed to execute within strict VRAM constraints (16GB limit, standard T4 architecture).

1. Activation-Aware Scaling (RMS Mappping)

Standard magnitude-based pruning fails to account for the dynamic flow of language routing. To mitigate this, the base model was calibrated using a targeted dataset (TinyShakespeare) to extract the Root Mean Square (RMS) of input activations across all major dense layers (q_proj, k_proj, v_proj, o_proj, down_proj). The extracted RMS values were used as scaling factors to amplify critical routing weights and penalize inactive structural noise.

2. CPU-Offloaded Matrix Factorization (ASVD)

Applying SVD on massive 3B-parameter matrices requires transient VRAM allocation that exceeds typical 16GB GPU capacities, invariably causing Out-Of-Memory (OOM) faults. To solve this, the matrix decomposition was dynamically offloaded to the CPU. The scaled matrices were factorized enforcing a strict 85% variance retention threshold: $W \approx U k Σ k V k T $ The factorization resulted in a structural reduction of 15.77% across the targeted layers, equating to the permanent removal of 163M parameters.

3. Neural Structural Recovery (LoRA Healing)

The immediate consequence of aggressive matrix factorization is the degradation of syntactic coherence due to severed routing pathways. To recover the generative capabilities without altering the compressed base matrices, Low-Rank Adaptation (LoRA) modules (Rank 16, Alpha 32) were injected across all surviving linear layers.

The model underwent micro-batch gradient accumulation fine-tuning (2 epochs). This healing phase required only 33.3M trainable parameters (~1.12% of the network). The training rapidly stabilized the loss function, restoring logical reasoning and grammatical structures. The final step involved merging and unloading the LoRA adapters directly into the base weights, outputting a standalone FP16 model.

Quantitative Metrics

Table with columns: Architecture Phase, Total Parameters, Targeted Linear Layers, Retained Variance, Trainable (LoRA), Format
Architecture Phase	Total Parameters	Targeted Linear Layers	Retained Variance	Trainable (LoRA)	Format
Base Qwen-3B	~3.09 Billion	1.03 Billion	100%	0	FP16
ASVD-Healed	~2.96 Billion	870 Million	85%	33.3 Million	FP16

Implementation and Usage

The resulting model is structurally identical to standard causal language models in the Hugging Face ecosystem. It does not require custom inference scripts, external decoders, or quantization libraries.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kodjaoglanian/Qwen-3B-ASVD-Healed"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    device_map="auto"
)

prompt = "The most important concept in quantum physics is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        max_new_tokens=50, 
        do_sample=True, 
        temperature=0.7
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Operational Advantages and Limitations

Advantages for Edge AI and MLOps:

Native Execution: The model operates entirely in FP16, avoiding the runtime computational penalty associated with dynamic dequantization processes.
VRAM Efficiency: The absolute reduction in parameters allows for larger batch sizes and longer context windows in VRAM-constrained hardware, optimizing throughput (Tokens Per Second) in production environments.

Limitations:

While structural and logical cohesion are fully restored, the 85% variance retention threshold dictates that highly specific or esoteric factual recall may exhibit slight degradation compared to the uncompressed base model. For production use cases demanding strict factual accuracy in specialized domains, subsequent fine-tuning of this compressed artifact is advised.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Kodjaoglanian

Model Tree

Base

Qwen/Qwen2.5-3B

Fine-tuned

this model

Input Modalities

Text

Output Modalities