Immutable provenance
- Quaero commit:
f7bdf119fb573577756da28978e39d28bcfdb5b0
- Retrain commit:
00f8982c3ebff6faaaafc1cfeb069009f424ac5f
- Base revision:
851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
- Adapter SHA-256:
a8504dc05c3151818ff9988847c220be730242a34e0d406b42e1f8d11344008a
- Training corpus: 10,034 machine-verifiable factory trajectories
- W&B: https://wandb.ai/teilo/quaero/runs/2thgo6lc
Training initially failed at step 38 with microbatch size 2. It restarted at microbatch size 1 with gradient checkpointing from adapter weights only; optimizer, scaler, and RNG state were not restored. This weakens exact-continuation claims and is intentionally disclosed.
Frozen factory selection gate
Same 8 tasks, four rollouts each, seed 123, 64 turns, 512 action tokens, temperature 0.8, top-p 0.95, no oracle:
Table with columns: Candidate, Passed| Candidate | Passed |
|---|
| Raw base | 0/32 |
| Previous adapter | 10/32 |
| Step 628 | 16/32 |
| Step 1256 | 21/32 |
| Step 1884 | 19/32 |
| Final | 24/32 |
The final checkpoint was the unique gate leader and was selected before Spider evaluation.
Spider 2.0-DBT dev evaluation
The single scoreable run on the predeclared 16-task dev split scored 0/16 with the official scorer. All 16 tasks terminated on repeated-action detection and none submitted a result. Therefore this artifact is a negative benchmark result despite its factory-gate improvement. Do not infer leaderboard competitiveness from the factory score.
The companion private evaluation repository contains the frozen selection record, transcripts, official score, diagnostics, taxonomy, preflight, and operational incident logs. No Spider heldout task was used for training or checkpoint selection.
Measured failure modes
Under the 32-episode factory gate, the selected checkpoint submitted 27/32 episodes with 24 positive rewards, 4 repeated-action cutoffs, 0 token-cap hits, and 0 turn exhaustions. On the 16-task Spider dev split, all 16 episodes ended in repeated-action cutoffs with no submission.
Usage
This adapter is an agent policy for the quaero openenv_v1 dbt/DuckDB tool contract (list_files, read_file, query, build_spec, write_file, dbt, validate, submit). It emits JSON tool actions and expects environment observations in return; it is not a general chat or one-shot text-to-SQL model. Generate with enable_thinking=False.
The environment implementing this contract is open source at github.com/teilomillet/quaero-dbt — including UserWorkspaceRuntime for driving this adapter against your own dbt project, and the verifier that mirrors the official Spider 2.0-DBT scoring semantics.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE_REV = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", revision=BASE_REV)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B", revision=BASE_REV)
model = PeftModel.from_pretrained(base, "teilomillet/quaero-qwen35-4b-sft-factory-longhorizon-20260711")
Context
The full story — why this adapter climbs the synthetic factory gate and still scores zero on real Spider 2.0-DBT instances, and what that measures — is written up at
teilo.xyz/posts/quaero-spider2-negative-result.