charcot-ai

charcot-27b

README

License: apache-2.0

Results

Mean score from 0 to 1 (3 attempts per task).

Table with columns: Evaluation, Tasks, Qwen3.8-27B, Charcot 27B
Evaluation	Tasks	Qwen3.8-27B	Charcot 27B
Private split	162	0.51	0.73
Public held-out	144	0.37	0.54

Private split by family:

Table with columns: Family, Qwen3.8-27B, Charcot 27B
Family	Qwen3.8-27B	Charcot 27B
Claims	0.37	0.71 ↑
Clinical coding	0.42	0.69 ↑
Document → FHIR	0.73	0.78 ≈

↑ gain, ↓ drop, ≈ no significant difference (95% confidence interval of the difference).

Quick start

vllm serve charcot-ai/charcot-27b --dtype bfloat16 --max-model-len 131072

export OPENAI_API_KEY=EMPTYcharcot-gym eval \  --model charcot-ai/charcot-27b \  --base-url http://localhost:8000/v1 \  --api-key-env OPENAI_API_KEY \  --split public --text-only --attempts 3

Training

Data: charcot-ai/gym-sft, 801 synthetic text episodes: 153 public-split tasks and 648 generated with the public generators (8 published seeds).
Method: LoRA SFT (rank 32) on the response only, merged into FP32 weights.
No private task or seed was used in training.

Limitations

Fine-tuning lowers scores on Hard clinical coding: Δ −0.149 [−0.25, −0.05] on the public held-out and −0.174 [−0.34, −0.01] on the private split.
Text only: the vision tower is unchanged from the base model.
Trained and evaluated on synthetic data only.
Private tasks are not published; only aggregate scores are.

Held-out tasks use fresh seeds and share no seed, content, or initial state with training or private tasks.
Private split: base + 2026Q4 roots, task-files SHA-256 2aa24013…3d0fa and 5ef02e7e…480551.
Intervals: on the public held-out, a paired 95% bootstrap over tasks. On the private split, a 95% bootstrap over tasks within each cell, combined across cells with task-count weights; it is unpaired, so it is conservative. Errored attempts were retried up to twice and excluded if they still failed (base: 6 of 486 private attempts).

License

Apache-2.0. The base model remains subject to its own license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

charcot-ai

Model Tree

Base

Qwen/Qwen3.8-27B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality