SASVAAI

Gemma-4-26B-A4B-XLAM-Lora

README

License: apache-2.0

Model details

Table

Developed by	SASVA AI Model Cognition Labs(MCL) Team
Base model	`google/gemma-4-26B-A4B-it`
Base revision	`4d7ae4984b7db7de8f8457170b3f1a419ee76d52`
Base parameters	25.95B total / ~4B active (128-expert MoE, top-8 routing)
Architecture family	`gemma4` (text tower `gemma4_text`)
Adaptation	LoRA (`r=128`, `alpha=256`, `dropout=0.05`, rsLoRA on)
Trainable modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Excluded modules	`.vision_tower.`
Training method	`bf16_lora` (`experiments.hyperparams`, run 93 / trial 14)
Refinement	none
Precision	bf16 base (no quantisation), bf16 compute
Language	English
License	Apache-2.0 (inherited from the base model)

Trainable parameters: 148,684,800 across 205 modules — 0.573% of the base. The adapter file is 594,801,016 bytes (410 tensors: lora_A + lora_B per module, stored in float32). All 410 sit under base_model.model.model.language_model; the base model's vision tower is loaded but untouched — zero adapter tensors match vision_tower.

Two Gemma 4 structural facts that shape the module list. Both were confirmed against the base model's config.json and model.safetensors.index.json:

v_proj exists on only 25 of 30 layers. layer_types alternates five sliding_attention layers then one full_attention layer, so layers 5, 11, 17, 23 and 29 are full-attention — and with attention_k_eq_v: true those layers share one projection for K and V and expose no v_proj at all. The base model ships 25 v_proj.weight tensors against 30 q_proj.weight; the adapter matches that exactly. This is architecture, not a gap in coverage. Head geometry differs too: full-attention layers use global_head_dim 512 with 2 KV heads (q out 8192, k out 1024) against 256 / 8 heads on sliding layers (q out 4096, k/v out 2048).
The 128 MoE experts are not adapted. Every layer carries both a dense mlp.* path (hidden_size 2816 → intermediate_size 2112) and an experts.gate_up_proj / pair plus a router ( 704). The adapter's MLP tensors are 2816↔2112, i.e. the dense path only. The expert weights and the routers are untouched, so whatever the router has learned to send where is unchanged by this adapter.

Intended use

Direct use. Produce the next assistant turn in a task-oriented support conversation that has tools. The training traces are airline customer service — reservations, upgrades, cancellations, baggage, payment methods — over a fixed set of 13 tools (get_user_details, get_reservation_details, update_reservation_flights, search_onestop_flight, book_reservation, transfer_to_human_agents, and seven more).

The model was trained on a specific prompt shape and that shape is part of the contract:

System prompt (verbatim): "You are a function-calling assistant. Given a multi-turn conversation and a list of available tools, determine the next step: either call a function using the format
text
{"name": "<function_name>", "arguments": {<key-value pairs>}}
or provide a final text reply if the user's request has been fulfilled. Always prefer calling a function when more information or action is needed."
Conversation turns use roles user, assistant, and tool; a tool message carries the raw JSON your backend returned.
Applied through the tokenizer's chat template (chat_template.jinja, shipped in this repo) with add_generation_prompt=True. Do not concatenate strings by hand.
The output is one of two shapes — a tool call in Gemma 4 token syntax, <|tool_call>call:get_user_details{user_id:<|"|>harper_ito_2309<|"|>}<tool_call|>, or plain text terminated by <turn|>. Note that the system prompt above asks for a JSON object while every tool-call label in training is the token form: the model learned the labels, so parse for the token form, never for JSON.
Decode with . The tool call special tokens; skipping them destroys the output you are trying to parse.

How to get started

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE = "google/gemma-4-26B-A4B-it"ADAPTER = "SASVAAI/Gemma-4-26B-A4B-XLAM-Lora" # NOTE: AutoModelForCausalLM is correct for Gemma 4 — it resolves to# Gemma4ForConditionalGeneration, which loads the multimodal wrapper properly.# (Do not copy the AutoModelForImageTextToText workaround from the Qwen 3.8# sibling release; that model's CausalLM mapping is the broken one, Gemma 4's# is not.)tokenizer = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForCausalLM.from_pretrained(    BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa")model = PeftModel.from_pretrained(model, ADAPTER)model.eval() # Verbatim from the training invocation. Do not paraphrase.SYSTEM = (    "You are a function-calling assistant. Given a multi-turn conversation and a "    "list of available tools, determine the next step: either call a function "    'using the format `{"name": "<function_name>", "arguments": {<key-value '    "pairs>}}` or provide a final text reply if the user's request has been "    "fulfilled. Always prefer calling a function when more information or action "    "is needed.")messages = [    {"role": "system", "content": SYSTEM},    {"role": "user", "content": "Hi, I have a reservation with ID B1TDOL in basic "                               "economy and I'd like to upgrade it to economy class."},    {"role": "assistant", "content": "To upgrade your reservation I'll need your "                                     "user ID and the payment method for the fare "                                     "difference."},    {"role": "user", "content": "My user ID is isabella_lopez_2185, and I'll pay "                                "with the credit card ending in 3244."},]inputs = tokenizer.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(inputs, max_new_tokens=512, do_sample=False)print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=False))# -> <|tool_call>call:get_reservation_details{reservation_id:<|"|>B1TDOL<|"|>}<tool_call|>

The base model is ~52 GB in bfloat16 — this needs an 80 GB-class GPU with room to spare, or two. Training used attn_implementation="sdpa".

Decoding matters. Training and evaluation were both teacher-forced, so no sampling setting is validated. Greedy (do_sample=False) is the only defensible default, and the example output above is illustrative of the trained shape, not a measured result.

Training details

Data. A 342-row training split derived from 100 internal multi-turn tool-calling traces, produced by this project's data-generation stage from a local trace folder named run_traces/xlam. Traces are ShareGPT-style (from: human / gpt / function_call / observation), airline customer support.

Provenance caveat — read this before citing a public dataset. The source folder is named xlam, and that name is carried through this repo for traceability, but the contents do not look like the public single-turn xLAM/APIGen function-calling corpus. Across the 100 traces there are 391 calls over exactly 13 tools — get_reservation_details, get_user_details, think, update_reservation_baggages, update_reservation_flights, search_direct_flight, search_onestop_flight, calculate, transfer_to_human_agents, update_reservation_passengers, cancel_reservation, book_reservation, send_certificate — with a median of 14 turns per trace (max 48). That is a small fixed airline tool set inside long multi-turn dialogues, which resembles a tau-bench-style airline environment far more than xLAM's thousands of distinct single-turn APIs. , so no public dataset is cited. Do not represent this model as trained on the published xLAM dataset without confirming provenance first.

Conversion was deterministic — no model generated any training content. A script walked each trace and emitted one row per assistant turn, with the full preceding conversation as context and that turn as the answer. 13 of the 100 traces yielded no usable turn and were skipped, leaving 87 traces and 378 rows. The split is not published, so the datasets key is omitted rather than pointing at a dead link.

The train/validation split is group-aware by trace, so no conversation appears on both sides.

Table

Train samples	342 rows / 80 traces
Validation samples	36 rows / 7 traces
Shared traces	0
Prompt format	chat template + system prompt (see Intended use)
Loss masking	answer tokens only; prompt tokens set to `-100`
Truncation	prompt left-truncated to fit `max_seq_len` 4096; answer never truncated

An LLM (Claude Opus 4.6, claude-opus-4-6, via an internal inference gateway) was used elsewhere in the pipeline — to infer the trace field mapping, and to propose the hyperparameters tried by the search. It generated no training content and computed no metric.

Method

Table

SFT method	`bf16_lora`
Base quantisation during training	none — bf16 base, bf16 compute
Refinement stage	none
Attention implementation	`sdpa`, forced (see note)
Hardware	8x NVIDIA H100 80GB HBM3, `torchrun --nproc_per_node=8`

Attention is pinned to sdpa rather than FlashAttention-2 for a hard reason: the 5 full-attention layers use global_head_dim 512, past FlashAttention-2's 256 head-dim cap. FA2 cannot run this model at all.

bf16_lora was the only method the project allowed for this run (allowed_training_methods: ['bf16_lora']). It applies LoRA to an unquantised bf16 base — the only one of the available methods that does no quantisation.

No refinement stage ran; the published weights are the SFT adapter.

Final hyperparameters

Table with columns: Hyperparameter, Value, Source
Hyperparameter	Value	Source
`learning_rate`	0.00015	`experiments.hyperparams` / `[TRAIN]` cmdline
`lr_scheduler_type`	cosine	`experiments.hyperparams`
`num_train_epochs`	2	`experiments.hyperparams`

Effective batch size: 32 (1 x 4 x 8). Optimizer steps: 22.

loraplus_lr_ratio (1.0), neftune_noise_alpha (0.0), use_liger_kernel, use_sample_packing and lora_init (default) were all left at their no-op defaults. KD parameters are omitted deliberately — this is a bf16_lora run, not bf16_lora_kd, so kd_alpha/kd_beta/kd_temperature carry inert defaults that would imply distillation that did not happen.

Provenance note: this run predates results.tsv having a row for it, and the project's task_config.py / platform_config.py are gitignored and were overwritten by later runs. Every value above was recovered from the platform database (runs, experiments, events tables for run 93) and cross-checked against the literal [TRAIN] command line recorded in the run log.

How these values were chosen

These hyperparameters were selected by an automated search (autocatalyst.cli.run_autoresearch): an agent proposes one change at a time, runs train → eval, and keeps or discards on eval_loss (lower is better).

Run 93 ran 46 trials across 5 data strategies in 4 h 49 m; 44 scored and 2 errored. This checkpoint is trial 14, strategy v2 — the run's best.

Read this before comparing any two numbers below. Each strategy regenerates the dataset, so eval_loss is only comparable within a strategy. Strategies v1 and v2 share the strategy-v2-era data and are comparable to each other; strategies v3, v4 and v5 landed in the 5.2–8.0 range on their own regenerated splits, which is a different metric, not a worse model. None improved the outer best, and the run ended after v5. Those 26 trials are therefore excluded from the table below rather than tabulated in a way that would imply a comparison that does not exist.

The 18 comparable trials. All bf16 LoRA, batch 1 x grad accum 4, cosine schedule, max_seq_len 4096, on the same 36-example split:

Table with columns: #, r, LR, Epochs, rsLoRA, Weight decay, Warmup, Dropout, eval_loss, Kept
#	`r`	LR	Epochs	rsLoRA	Weight decay	Warmup	Dropout	eval_loss	Kept
14	128	1.5e-4	2	yes	0.03	0.05	0.05	0.407374

Trials 1–6 are strategy v1, trials 7–19 strategy v2 (trial 17 failed validation and never trained). Trials 20 and 40 are absent or errored.

What the search actually established.

Rank dominates. At LR 1.5e-4 / weight decay 0.01, going r 16 → 32 → 64 → 128 moved loss 0.641740 → 0.580867 → 0.500371 → 0.430278 (#7, #9, #10, #13; the last two with rsLoRA on).
rsLoRA earns its keep at high rank. At r=64, enabling it alone moved 0.500371 → 0.474418 (#10 → #12).
Weight decay has a sharp optimum at 0.03. Holding everything else at r=128: 0.01 → 0.430278 (#13), 0.03 → 0.407374 (#14), 0.05 → 0.619713 (#19). One step past the optimum costs 0.21.
More epochs hurt. 3 epochs lost to 2 in both places it was tried (#3, #8).

What it did not establish. The winning margin over the strategy-v1 baseline is 0.003 (0.407374 vs 0.410294) between two configurations that share almost nothing — r=128 + rsLoRA + wd 0.03 versus r=16 + no rsLoRA + wd 0.01. Read that as "the large-rank configuration caught up to the baseline", not as a 0.003 improvement attributable to any knob.

Search space. Five knobs were varied across these 18 trials (LORA_R, LEARNING_RATE, WEIGHT_DECAY, EPOCHS, USE_RSLORA, plus one warmup and one dropout probe). LR_SCHEDULER, MAX_SEQ_LEN, GRAD_ACCUM, BATCH_SIZE, LORAPLUS_LR_RATIO, NEFTUNE_NOISE_ALPHA, USE_DORA, LORA_INIT, USE_LIGER_KERNEL and USE_SAMPLE_PACKING were never moved. The bounds in force cannot be reproduced here: core/experiment_constraints.py was not tracked in git at this run's commit.

Measurement noise: not measured. No configuration was re-run with identical settings and a different seed — all 18 trials above differ from one another in at least one hyperparameter. The table shows the metric is steep (one weight-decay step costs 0.21; one warmup step costs 0.023), and on a 36-example split that steepness is as consistent with split-composition sensitivity as with genuine model differences. The two cannot be separated without repeated seeds, so treat the ranking as provisional.

Observed training metrics.

Table

Final train loss	0.6664509773254395
Final eval loss	0.407374
Train runtime	297.501s
Total FLOPs	1.645208064061604e+17
Throughput	2.299 samples/s, 0.074 steps/s

Only 22 optimizer steps ran, and the reported train loss is the mean over the run, not a converged value: the logged history is 1.1455974578857422 at step 10 and 0.2633920907974243 at step 20, with grad norm falling 7.44 → 3.73. Treat 0.6665 as an average over a still-descending curve.

Evaluation

Protocol. All 36 validation rows, no sampling. eval_loss is the Hugging Face Trainer's teacher-forced cross-entropy over answer tokens, computed inside the training loop and read back from trainer_state.json by the project's loss evaluator.

No generation occurred. That is why this repo ships no predictions.jsonl (the sibling Qwen release has one because that run used a generative evaluator), and why the shipped eval_results.json records num_samples: 0. There is no truncation caveat to give — and equally no evidence at all about whether the model emits well-formed tool calls.

Table with columns: Metric, Value
Metric	Value
Validation cross-entropy (`eval_loss`)	0.407374
Samples	36
Eval runtime	4.8873s (7.366 samples/s)

Baseline for comparison. Not measured. The untuned google/gemma-4-26B-A4B-it was never scored on this split, so the number above quantifies nothing about how much of the model's behaviour the fine-tuning is responsible for. This is the single most important gap in this card.

This is a validation split, not a held-out test set. The autoresearch loop selects against it — 18 comparable trials scored on these same 36 rows — so expect optimistic bias. With only 7 source traces behind those 36 rows, the split's composition moves the metric as much as the model does.

The eval set is not a file you can look up. It was a generated artifact (val_strategy_v2.jsonl), and the generator overwrote it on 2026-08-27 with a regenerated strategy-v2 split built from a different system prompt. Unlike the sibling Qwen release, no per-example record survives, so the exact 36 rows scored here are not recoverable. Any re-evaluation is a new measurement, not a reproduction.

Limitations and bias

Loss is not competence. Every number here measures next-token agreement with 36 held-out turns. A model that emits a malformed tool call containing the right tokens in the wrong order scores well. Nothing was ever parsed or executed.

No baseline, so no established gain. The adapter may be improving output formatting, memorising the airline domain, or both. This card cannot distinguish them.

Hallucinated function calls are the expected failure mode. The model never sees a tool schema and learned a 13-tool vocabulary from 87 traces. It will produce a confident call to a function that does not exist. Mitigation: validate every emitted call — function name and argument keys — against your real schema, and treat a mismatch as a refusal rather than a retry.

Generated identifiers are fiction. The training traces contain synthetic personal data — names, addresses, emails, dates of birth, partial payment cards. The model reproduces those patterns and will emit plausible-looking user IDs, reservation IDs and card fragments that belong to nobody. Never surface one to a user, and never act on one without a confirming backend lookup.

Very little training signal. 22 optimizer steps over 342 examples. Treat this as a domain-formatting adapter, not a capability upgrade.

Prompt truncation is silent. At max_seq_len 4096 the prompt is left-truncated, so a long conversation loses its earliest turns — the system message first. Nothing warns you.

Domain narrowness. Short English airline-support dialogues with US-style addresses and payment methods. Longer inputs, other verticals, other tool vocabularies and other languages are all unmeasured.

Inherits all biases and limitations of the base model. This adapter changes 0.573% of the parameters, leaves the MoE routers and all 128 experts per layer untouched, and was not evaluated for social bias, safety, or fairness across demographic groups.

Environmental impact

Table

Hardware	8x NVIDIA H100 80GB HBM3
Training time	4.96 minutes (297.501s)
Cloud provider / region	on-premise

Covers this run only. The full 46-trial search that selected it took 4 h 49 m on the same hardware.

Framework versions

PEFT 0.18.1
TRL: 1.0.0
Transformers: 5.x dev (git main)
Pytorch: 2.5.1+cu121
Python: 3.12

PEFT's version is the one recorded in adapter_config.json at save time; the rest are the pinned versions of the training environment. transformers is a git-main build: Gemma 4's gemma4 architecture is not in the stable PyPI release.

Citation

bibtex

@misc{gemma4_xlam_lora_2026,  title  = {Gemma-4-26B-A4B Function-Calling LoRA},  author = {Banerjee, Aaron and Anbuselvan, Pooja and Jodhpurkar, Om},  year   = {2026},  url    = {https://huggingface.co/SASVAAI/Gemma-4-26B-A4B-XLAM-Lora}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.