ajaxdavis

donto-qwen3.8-27b-predicate-extractor-rtx3090

README

License: apache-2.0

Measured result

All measurements used one NVIDIA RTX 3090, the same frozen ten-document speed fixture, greedy structured decoding, and the same source-disjoint 1,085-row quality set used for the original V15 model.

Table with columns: Gate, Original V15 runtime, This runtime, Requirement
Gate	Original V15 runtime	This runtime	Requirement
Aggregate output throughput	48.576 tok/s	150.801–151.187 tok/s	≥145.728 tok/s
Mean throughput	—	150.978 tok/s	—
Minimum speedup over baseline	1.00×	3.104×	≥3.00×
Exact recall, 1,085 rows	93.318%	93.832%	≥88.652%
Exact precision, 1,085 rows	93.888%	94.228%	≥89.194%
Valid structured calls	1,085/1,085	1,085/1,085	1,085/1,085

The three packaged speed repeats measured 150.801, 151.187, and 150.947 output tokens/s. This is aggregate completed-token throughput across four simultaneous independent documents, not the decode rate of one request. The separate concurrency sweep peaked at 155.946 tok/s with eight requested clients; all 120 calls across requested concurrency 1–12 were valid.

Concurrency 1–12

Table with columns: Requested clients, Effective clients, Aggregate tok/s, Wall time
Requested clients	Effective clients	Aggregate tok/s	Wall time
1	1	65.667	89.786 s
2	2	104.567	56.385 s
3	3	133.385	44.203 s
4	4	152.080	38.769 s

The fixture contains ten documents, so requested concurrency 11 and 12 remain ten effective clients. The server has four resident sequence slots; additional clients measure orderly queueing, not more simultaneous GPU-resident requests.

Machine-readable evidence is under evaluation/. Rejected profiles are retained rather than hidden. The decisive failure was ordering: merging LoRA into BF16 before NF4 missed the semantic floors even when both vocabulary tables were exact BF16. Candidate AX restored the baseline order (NF4 base first, unchanged runtime LoRA second) and exceeded baseline quality, but exact vocabulary tables left room for only two requests. Candidate AY compacts only the untuned vocabulary tables and admits four requests. Exhaustive live tokenization covers all 1,085 validation prompts and leaves 23 tokens of worst-case admission margin.

What is included

The unchanged rank-16, alpha-32 V15 LoRA and its original PEFT tokenizer files. The admitted accelerated server deliberately pins the stock base tokenizer identified by runtime/launch_rtx3090.sh.
The structured submit_facts schema and executable client examples.
The original PEFT adapter plus a deterministic, value-preserving vLLM key-layout conversion and manifest.
Audited upstream and patched runtime source, reviewable side by side.
A fail-closed overlay installer that checks every upstream and replacement SHA-256 before changing the runtime.
Launch scripts for the admitted RTX 3090 profile.
Full training manifests, source, logs, checkpoint history, data provenance, earlier rejected-model reports, speed traces, quality outputs, and checksums.

The large BF16 base and DFlash2 drafter are not copied into this repository. They are downloaded from the pinned upstream artifacts described below; the unchanged Donto LoRA and its value-identical vLLM remap are included.

Extraction contract

The model receives a document chunk and must call submit_facts. Each fact is:

json
{
  "s": "ex:subject",
  "p": "freePredicateName",
  "o": "ex:object or literal",
  "c": 0.95,
  "h": false
}

s, p, and o are the core triple. c is confidence. h marks an interpretive or hypothesized claim. This practical lane does not require the older generated anchor field; Donto can attach evidence in a separate citer stage when a deployment requires it.

The complete tool schema is schemas/tools.json. The training and evaluation path uses real tool calling and validated arguments; it does not scrape JSON from assistant prose.

Quick start

1. Obtain pinned inputs

Download:

Qwen/Qwen3.8-27B at revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0;
this repository;
syv-ai/qwen38-27b-rtx3090 at commit 60daef8255b6757d9791955a44bce27df1658ea6;
its Qwen3.8-27B-DFlash2-W4A16 drafter.

2. Install the measured runtime and serve

Follow the upstream native-Linux setup first. Then:

bash
bash runtime/install_overlay.sh /path/to/qwen38-27b-rtx3090 /path/to/venv

export QSF_DIR=/path/to/qwen38-27b-rtx3090
export MODEL=/path/to/Qwen3.8-27B-BF16
bash runtime/launch_rtx3090.sh

The admitted profile uses:

maximum model length 2,672 tokens;
maximum output reservation 768 tokens per short-chunk request;
four concurrent sequences;
832 maximum batched prefill tokens;
a 3,800,000,000-byte KV pool;
full CUDA graphs;
DFlash2 W4A16 with seven drafted tokens;
BitsAndBytes NF4 transformer weights and vector-wise INT8 output projection;
symmetric per-row INT8 input embeddings;
prefix caching disabled.

Prefix caching is intentionally off because this Donto lane processes short, independent document chunks. The prompt prefix is not large enough to repay its bookkeeping cost in the admitted workload. When a page needs more room, use more_supported_facts:true and issue another bounded call instead of silently raising the output reservation.

3. Call the structured endpoint

See examples/structured_client.py. A minimal OpenAI-compatible request selects the submit_facts function and disables Qwen thinking in the chat-template arguments. The launch profile defaults to http://127.0.0.1:18020/v1 and served model name pilot-live-v15-exhaustive-unit-r16-v1; the example uses those defaults and accepts DONTO_QWEN_BASE_URL / DONTO_QWEN_MODEL overrides.

Why it is faster

The runtime attacks memory residency first, then parallelism:

The frozen stock BF16 base is converted to NF4 at load, after which the unchanged V15 LoRA is applied at runtime. This ordering preserves small learned adapter deltas that merge-before-NF4 candidates rounded away.
The transformer body uses NF4 weights while the output head uses vector-wise INT8 with a graph-safe full-precision correction for the 64 highest-magnitude activation columns.
The large input embedding table uses per-row INT8; only requested token rows are dequantized.
The DFlash2 drafter proposes seven tokens per target-model verification.
The target output head is installed early, avoiding a temporary 2.37 GiB DFlash allocation during startup.
The resulting 16.86 GiB model load leaves enough memory for full CUDA graphs, a 3.8 GB KV pool, and four simultaneous measured Donto requests.
A 768-token output reservation fits every frozen short-chunk prompt while retaining the fast four-document context profile; longer extraction is paginated.

After downloading the complete repository, verify every published file with:

bash
bash verify_release.sh

Speculation cannot change accepted output tokens: the unchanged Donto target model verifies every draft token before emission.

Provenance and reproducibility

Table with columns: Artifact, SHA-256
Artifact	SHA-256
`adapter_model.safetensors`	`40d11bcad20061f8291741c4dd8a3701405902f336c27b60592d62d6b2f7a974`
vLLM conditional adapter payload	`28df663ffa70bf541cafff2c9bb60df11e4fe000a2ea9124b346a9cc3eb20d8b`
pinned BF16 base index	`77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df`
pinned BF16 base tokenizer	`0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3`
DFlash2 W4A16 weights	`ec26996e6a0745ab5edb857117220ce1e219ad524f71e6e149b703804947d8e7`

The Rust corpus, curation, evaluation, and RunPod launch toolchain is committed at thomasdavis/donto@5ad8593. The admitted runtime source, engineering report, and public README update are committed at thomasdavis/donto@c3a6912. The accelerated serving base is pinned separately to syv-ai/qwen38-27b-rtx3090@60daef8.

The full software and hardware capture is runtime/environment.json. The unchanged source adapter identity, deterministic key remap, and no-retraining declaration are under provenance/ and runtime/adapter/. See the engineering report for the experiment history and the distinction between target-model fidelity, structured quality, and aggregate throughput.

Training data and history

This package intentionally preserves more than the final adapter:

training/source/ contains the corpus builders, evaluator, trainer, audit, and remapping programs used for V15;
training/data/v15/ contains the complete canonical train, validation, optional D1-repeat, and validator JSONL files plus both tool schemas;
training/data/gold-100/ contains the complete public 100-document graph-first gold and validator suites plus both tool schemas;
training/logs/ contains the selected training and canary logs;
artifacts/ contains the immutable training contract, manifest, completion receipt, LoRA target-module audit, and original V15 training-run checksum receipt (whose paths are relative to the original training output);
docs/ records both accepted and rejected V11–V15 experiments;
the same complete data and its dedicated data card are also available at .

The adapter, runtime, and code use Apache-2.0. The included data retains its separate mixed-provenance terms in training/data/DATA-LICENSE.md; the repository license tag does not relicense source-derived excerpts.

No training, calibration, optimizer step, backward pass, or gradient update was performed to produce the RTX 3090 speedup.

Scope and limitations

This is an English-focused predicate extractor, not a general chat model.
The strict published quality numbers measure exact matches to a source-disjoint synthetic/reviewed Donto set. They are reproducible regression evidence, not a claim that every semantically valid paraphrase is scored.
Throughput is workload-dependent. A single request is slower than the four-document aggregate; longer prompts or larger output caps require a new memory and quality admission run.
The measured runtime is Linux/CUDA/RTX 3090 specific and currently depends on audited source overlays rather than stock vLLM.
Do not silently increase context length, sequence count, KV allocation, or speculative depth. Re-run the shipped gates when changing the profile.
The runtime does not make unsupported claims safe. Deployments should retain source text, validate structured output, and use Donto's citer or another evidence check when provenance is required.

Vision

Donto is a contradiction-preserving claim substrate for generative abundance. The scarce step is no longer inventing a schema before learning anything; it is holding many evidence-bearing, potentially incompatible claims without deleting the losing view. This model is a practical extraction front end for that idea: cheap enough to run continuously on a single consumer GPU, flexible enough to mint source-shaped predicates, and structured enough to feed an auditable knowledge system.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ajaxdavis

Model Tree

Base

Qwen/Qwen3.8-27B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality