myyycroft

Qwen3.5-2B-AmbigQA-lora-8-member-0

Evaluation note

Metrics below are computed on small fixed subsets, not full benchmarks. Exact subset sizes: ambigqa=128, ifeval=64, mmlu=228. Reported values are from the final training step (step=1620, epoch=3).

Dataset

Hub dataset: sewon/ambig_qa (config_name=light)
Revision: 6e667596df70f17ba3c8e7be4b7361f6be8b60f8
Splits: train=train, validation=validation
Dev holdout: dev_fraction=0.1, split_seed=1729
Bootstrap resampling: False

System prompt

No system prompt (dataset.system_prompt: null).

Hyperparameters

Exact resolved settings used for this run (also attached as resolved_config.yaml):

yaml
model:
  name: Qwen/Qwen3.5-2B
  revision: main
  dtype: bfloat16
  enable_thinking: false
dataset:
  name: sewon/ambig_qa
  config_name: light
  revision: 6e667596df70f17ba3c8e7be4b7361f6be8b60f8
  train_split: train
  validation_split: validation
  dev_fraction: 0.1
  split_seed: 1729
  bootstrap: false
  max_train_examples: null
  max_dev_examples: null
  system_prompt_set: false
adaptation:
  method: lora
  lora:
    r: 8
    alpha: 16
    dropout: 0.0
    target_modules: all-linear
    bias: none
training:
  num_train_epochs: 3.0
  max_steps: -1
  learning_rate: 5.0e-05
  weight_decay: 0.01
  warmup_ratio: 0.03
  per_device_train_batch_size: 16
  per_device_eval_batch_size: 32
  gradient_accumulation_steps: 2
  max_seq_length: 512
  logging_steps: 10
  save_every_steps: 150
  save_total_limit: 3
  gradient_checkpointing: false
  bf16: true
  tf32: true
  max_grad_norm: 1.0
  dataloader_num_workers: 2
  adam_beta1: 0.9
  adam_beta2: 0.999
  adam_epsilon: 1.0e-08
ensemble:
  size: 5
  base_seed: 42
  this_member_index: 0
  this_member_seed: 42
evaluation_subsets:
  ambigqa:
    subset_size: 128
    seed: 1001
  ifeval:
    subset_size: 64
    seed: 1002
  mmlu:
    subset_size: 228
    seed: 1004

Metrics breakdown

Final-step ensemble mean ± std (n=5)

AmbigQA accuracy: 0.0594 ± 0.0105
AmbigQA AlignScore: 0.1146 ± 0.0129
IFEval prompt_level_strict_accuracy: 0.3594 ± 0.0191
MMLU accuracy: 0.6254 ± 0.0080
train_loss: 1.5477 ± 0.0039
steps: 1620
epochs: 3

Per-member (final step)

Table with columns: member, seed, steps, epochs, train_loss, AmbigQA acc, AmbigQA AlignScore, IFEval strict, MMLU acc
member	seed	steps	epochs	train_loss	AmbigQA acc	AmbigQA AlignScore	IFEval strict	MMLU acc
0 (this repo)	42	1620	3	1.5450	0.0703	0.1347	0.3750	0.6272
1	1051

Files

Model weights / adapter files from members/member_000/final/
resolved_config.yaml — full resolved training config
ensemble_metrics.png — train/eval curves for the whole ensemble

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

myyycroft

Model Tree

Base

Qwen/Qwen3.5-2B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Evaluation note

Dataset

Hub dataset: sewon/ambig_qa (config_name=light)
Revision: 6e667596df70f17ba3c8e7be4b7361f6be8b60f8
Splits: train=train, validation=validation
Dev holdout: dev_fraction=0.1, split_seed=1729
Bootstrap resampling: False

System prompt

No system prompt (dataset.system_prompt: null).

Hyperparameters

Exact resolved settings used for this run (also attached as resolved_config.yaml):

yaml
model:
  name: Qwen/Qwen3.5-2B
  revision: main
  dtype: bfloat16
  enable_thinking: false
dataset:
  name: sewon/ambig_qa
  config_name: light
  revision: 6e667596df70f17ba3c8e7be4b7361f6be8b60f8
  train_split: train
  validation_split: validation
  dev_fraction: 0.1
  split_seed: 1729
  bootstrap: false
  max_train_examples: null
  max_dev_examples: null
  system_prompt_set: false
adaptation:
  method: lora
  lora:
    r: 8
    alpha: 16
    dropout: 0.0
    target_modules: all-linear
    bias: none
training:
  num_train_epochs: 3.0
  max_steps: -1
  learning_rate: 5.0e-05
  weight_decay: 0.01
  warmup_ratio: 0.03
  per_device_train_batch_size: 16
  per_device_eval_batch_size: 32
  gradient_accumulation_steps: 2
  max_seq_length: 512
  logging_steps: 10
  save_every_steps: 150
  save_total_limit: 3
  gradient_checkpointing: false
  bf16: true
  tf32: true
  max_grad_norm: 1.0
  dataloader_num_workers: 2
  adam_beta1: 0.9
  adam_beta2: 0.999
  adam_epsilon: 1.0e-08
ensemble:
  size: 5
  base_seed: 42
  this_member_index: 0
  this_member_seed: 42
evaluation_subsets:
  ambigqa:
    subset_size: 128
    seed: 1001
  ifeval:
    subset_size: 64
    seed: 1002
  mmlu:
    subset_size: 228
    seed: 1004

Metrics breakdown

Final-step ensemble mean ± std (n=5)

AmbigQA accuracy: 0.0594 ± 0.0105
AmbigQA AlignScore: 0.1146 ± 0.0129
IFEval prompt_level_strict_accuracy: 0.3594 ± 0.0191
MMLU accuracy: 0.6254 ± 0.0080
train_loss: 1.5477 ± 0.0039
steps: 1620
epochs: 3

Per-member (final step)

Table with columns: member, seed, steps, epochs, train_loss, AmbigQA acc, AmbigQA AlignScore, IFEval strict, MMLU acc
member	seed	steps	epochs	train_loss	AmbigQA acc	AmbigQA AlignScore	IFEval strict	MMLU acc
0 (this repo)	42	1620	3	1.5450	0.0703	0.1347	0.3750	0.6272
1	1051

Files

Model weights / adapter files from members/member_000/final/
resolved_config.yaml — full resolved training config
ensemble_metrics.png — train/eval curves for the whole ensemble

Qwen3.5-2B-AmbigQA-lora-8-member-0

README

Evaluation note

Dataset

System prompt

Hyperparameters

Metrics breakdown

Final-step ensemble mean ± std (n=5)

Per-member (final step)

Files

Explore FriendliAI today

README

Evaluation note

Dataset

System prompt

Hyperparameters

Metrics breakdown

Final-step ensemble mean ± std (n=5)

Per-member (final step)

Files