GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cmndcntrlcyber
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
cmndcntrlcyber
Model Tree
google/gemma-4-26B-A4B-it
Input Modalities
Output Modalities
Supported Functionality
Gemma 4 26B-A4B is a Mixture-of-Experts model: 128 experts + 1 shared expert, 8 active per layer, 30 layers. Total parameters: 25.8B; active per forward pass: 3.8B.
LoRA targeting constraint: the routed expert FFN layers use 3D
nn.Parameter tensors that PEFT cannot target. LoRA is applied only to
shared attention + shared MLP modules. All learning rates are halved vs the
Qwen pipeline for MoE routing stability.
Gemma4ClippableLinear: modules must be unwrapped before PEFT operations
(handled by src/utils.py:unwrap_clippable_linear).
google/gemma-4-26B-A4B-it
(with DAPT merged) for instruction-following code generation, tool calling,
and multi-turn agent behaviour in the 8 dataset languages (Python,
JavaScript, TypeScript, Java, Go, Rust, C++, C#).gemma26b-offsec-coder-gguf.cmndcntrlcyber/code-trainer-v9-mixed
(same dataset as the Qwen V9 SFT stage)apply_chat_template(tools=...)--inject-system-prompt, plus
~400 identity training examples (Slice E) teaching the model to respond as
Nexus when asked about its identity, capabilities, and methodology.gemma4-26b-a4b-dapt-offsec)
is merged into the base model before SFT, grounding the model in offensive
security patterns.| Knob | Value |
|---|---|
| Base model | google/gemma-4-26B-A4B-it (with DAPT merged) |
| Adapter | LoRA (PEFT), r = 64, alpha = 128, dropout = 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Meta | Value |
|---|---|
| Hardware | HF Jobs a100-large (1x A100 80 GB) |
| Entry point | src/phase4_gemma_finetuning/hf_skills/train_entry.py |
| Config | src/config/pipeline-gemma26b.yml (gemma_finetuning section) |
| eval_loss | 0.4195 (from sweep) |
| Metric | Value |
|---|---|
| eval_loss | 0.4195 |
text
google/gemma-4-26B-A4B-it └─ merge: cmndcntrlcyber/gemma4-26b-a4b-dapt-offsec (DAPT) └─ LoRA: cmndcntrlcyber/gemma4-26b-a4b-code-trainer-aggressive-full1 (this adapter) └─ downstream: Vision SFT → FARCA-GRPO → DPO → GGUFpython
import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom src.utils import unwrap_clippable_linear base_id = "google/gemma-4-26B-A4B-it"dapt_id = "cmndcntrlcyber/gemma4-26b-a4b-dapt-offsec"sft_id = "cmndcntrlcyber/gemma4-26b-a4b-code-trainer-aggressive-full1" tokenizer = AutoTokenizer.from_pretrained(base_id)model = AutoModelForCausalLM.from_pretrained( base_id, torch_dtype=torch.bfloat16, device_map="auto",)unwrap_clippable_linear(model) # Merge DAPT adapter into base weightsmodel = PeftModel.from_pretrained(model, dapt_id)model = model.merge_and_unload() # Load SFT adapter (active LoRA)model = PeftModel.from_pretrained(model, sft_id)model.eval() messages = [ {"role": "user", "content": "Write a Python function that performs an ARP scan on a /24 subnet using scapy."},]inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True,).to(model.device)out = model.generate(inputs, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))nn.Parameter), so fine-tuning adapts shared attention + shared MLP only.src/phase4_gemma_finetuning/)bash
python -m src.phase4_gemma_finetuning.scripts.launch_full_training \ --config src/config/pipeline-gemma26b.yml --best-config aggressive --wait| Learning rate | 2e-4 (cosine decay) |
| Batch size | 4 |
| Gradient accumulation | 4 (effective batch = 16) |
| Epochs | 1 |
| Sequence length | 4,096 |
| Precision | bfloat16 + gradient checkpointing |
google/gemma-4-26B-A4B-it └─ merge: cmndcntrlcyber/gemma4-26b-a4b-dapt-offsec (DAPT) └─ LoRA: cmndcntrlcyber/gemma4-26b-a4b-code-trainer-aggressive-full1 (this adapter) └─ downstream: Vision SFT → FARCA-GRPO → DPO → GGUFimport torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom src.utils import unwrap_clippable_linear base_id = "google/gemma-4-26B-A4B-it"dapt_id = "cmndcntrlcyber/gemma4-26b-a4b-dapt-offsec"sft_id = "cmndcntrlcyber/gemma4-26b-a4b-code-trainer-aggressive-full1" tokenizer = AutoTokenizer.from_pretrained(base_id)model = AutoModelForCausalLM.from_pretrained( base_id, torch_dtype=torch.bfloat16, device_map="auto",)unwrap_clippable_linear(model) # Merge DAPT adapter into base weightsmodel = PeftModel.from_pretrained(model, dapt_id)model = model.merge_and_unload() # Load SFT adapter (active LoRA)model = PeftModel.from_pretrained(model, sft_id)model.eval() messages = [ {"role": "user", "content": "Write a Python function that performs an ARP scan on a /24 subnet using scapy."},]inputs = tokenizer.apply_chat_template( messages, return_tensors="pt", add_generation_prompt=True,).to(model.device)out = model.generate(inputs, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))python -m src.phase4_gemma_finetuning.scripts.launch_full_training \ --config src/config/pipeline-gemma26b.yml --best-config aggressive --wait