shreyansh12183

vigyan-1.5b-adapter-mathematics

README

License: apache-2.0

📌 1. Executive Summary & Research Context

This repository contains an experimental domain-specialized LoRA adapter trained as part of the Vigyan AI Sovereign Mixture of Experts (MoE) Upcycling Initiative.

The goal of this experimental line was to train isolated, high-rank domain experts on specialized STEM corpora and examine whether post-hoc upcycling via uncalibrated linear routers (mergekit-moe) could synthesize dense reasoning models into edge-deployable sparse MoE networks.

[!NOTE] Experimental Research Classification:
This model is released publicly as a transparent scientific artifact. Our empirical benchmark findings demonstrated that while this standalone adapter excels in its narrow domain, naive post-hoc MoE stitching without native router pretraining causes attention-FFN desynchronization (see forensic findings below).

🔬 2. Domain Specialization & Dataset Curricula

Domain Track: Formal Olympiad Mathematics & Pure Analysis
Technical Scope: IMO Olympiad proofs, differential topology, eigenvalue decomposition, abstract ring theory, and Riemann surfaces.
Training Corpus: Curated subset from shreyansh12183/vigyan-stem-experts-400k (100,000 curated CoT derivation samples).
Alignment Protocol: Strict Prompt Loss Masking (all user prompt tokens masked with labels = -100 to prevent prompt memorization).
Convergence Threshold: Dynamic early stopping triggered at loss 0.4789.

⚙️ 3. LoRA Hyperparameter Specifications

Table with columns: Parameter, Value, Engineering Rationale
Parameter	Value	Engineering Rationale
LoRA Rank ( $r$ )	`16`	Sufficient rank capacity for technical multi-step derivations
LoRA Alpha ( $lpha$ )	`32`	Standard $2 \times$ scaling factor for numerical gradient stability
LoRA Dropout	`0.05`

📊 4. Scientific Post-Mortem & Forensic Findings

During post-training MoE upcycling experiments, this adapter was used in a 4-expert sparse mixture configuration. The experimental run revealed pivotal architectural insights:

Attention-FFN Desynchronization: Because this LoRA was trained across both attention and MLP layers, stripping the attention weights during MoE assembly deprived the specialized FFNs of their calibrated input distribution.
Attractor Loop Vulnerability: When routed alongside uncoordinated experts by an untrained pseudo-router, autoregressive hidden states diverged into memorized attractor loops.
The 70/20/10 Law Formulation: Standalone training on 100% hard STEM highlighted the necessity of the 70/20/10 Dataset Curriculum Law (70% core domain, 20% conversational alignment, 10% base distribution replay) to preserve boundary tokens and conversational priors.

🚀 5. Usage with 🤗 PEFT & Transformers

python

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_model_id = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"adapter_id = "shreyansh12183/vigyan-1.5b-adapter-mathematics" tokenizer = AutoTokenizer.from_pretrained(base_model_id)base_model = AutoModelForCausalLM.from_pretrained(    base_model_id,    torch_dtype=torch.float16,    device_map="auto") # Load domain expert adaptermodel = PeftModel.from_pretrained(base_model, adapter_id)model.eval() prompt = "Derive the fundamental equations governing formal olympiad mathematics & pure analysis."inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad():    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🔮 6. Strategic Next Steps

In accordance with our empirical post-mortem, our flagship production efforts have pivoted to Allen AI's natively pretrained OLMoE-1B-7B architecture (64 fine-grained experts, Top-8 routing, trained end-to-end on 5.1T Dolma tokens). This adapter remains preserved as a sovereign benchmark baseline.

📄 License & Attribution

Released under the Apache 2.0 License for open scientific replication and research by the global AI community.
Developer: Shreyansh (@shreyansh12183)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

shreyansh12183

Model Tree

Base

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

Adapter

this model

Input Modalities

Text

Output Modalities