GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
greyecho
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
greyecho
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
python
from huggingface_hub import snapshot_downloadpath = snapshot_download("greyecho/openjeff-pilot-v1")For reproducible use, pin the Hub commit returned with this release. The files
adapter_config.json and adapter_model.safetensors are standard PEFT artifacts.
The original adapter config is unchanged so the frozen digest remains valid.
Use the tested runtime and launch instructions.
The GitHub release includes the same adapter plus the complete run directory
required by PilotScorer. Generic text generation is not the evaluated task:
OpenJeff compiles candidate labels to single-token codes and normalizes their
logits with the frozen temperature. The calibrator is bound to the A100 runtime;
loading weights on another device does not validate those probabilities.
bash
git clone https://github.com/greyaperez/openjeff.gitcd openjeff# In the documented PyTorch CUDA container on an A100 SXM 80 GB:python -m venv --system-site-packages .venv.venv/bin/python -m pip install -r configs/train-gpu-requirements.txt.venv/bin/python -m scripts.stage_weights.venv/bin/python -m openjeff.serve --run runs/pilot-v1 --port 8765On the same 231 public JevBench IDs, the initial OpenJeff pilot scored 90.04% versus 86.58% from Jev 1.13.0's published outcomes. Hard-subset accuracy was 80.18% vs. 72.97%. Jev was not rerun here; prompts and runtimes differ. This is not an official leaderboard rank or a full benchmark comparison. The diffusion follow-up remains experimental; this release is AR-only.
OpenJeff is an experimental adapter for finite-choice, evidence-conditioned
judgments. It returns candidate probabilities and supports explicit abstention.
The trained adapter, original code, and original synthetic adaptation data are
Apache-2.0. Foundation weights and third-party material retain their licenses
and notices; see NOTICE and the pinned source snapshots.
google/gemma-4-12B-it, revision
707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7.a1811874542eb3b9ca4235d79826dfabb847f0c5ad076bfd25b5e4b2f426ba2b.Google's developer/base-weight lineage is accepted for this project. Neither Google's U.S. location nor this adaptation establishes U.S.-only pretraining data. Full foundation pretraining is not reproduced.
8,000 original synthetic examples from 20 rule-grounded families, generated by
openjeff/curriculum.py. Labels come from executable reference logic; no external
teacher, private record, Jev API, or JevBench task supplies training examples.
One epoch: 500 AdamW optimizer steps, microbatch 4, accumulation 4, learning rate
1e-4 with warmup/cosine decay, candidate-only cross entropy, seed in the run recipe.
Development (400 rows) alone selected step 500 and then the adapter over the base.
Selection froze at 2026-09-23T04:34:38Z before calibration and final scoring.
Separate partitions: calibration-fit 600, calibration-check 400, final 600,
adversarial 200, and production-simulation 200. Exact request/group hashes include unique case IDs. Removing decorative IDs
reveals 197/600 final states matching training states. On the remaining 403,
OpenJeff scores 96.28% versus 75.93% for the base. Calibration-fit states also
recur in evaluation. See reports/curriculum-overlap.json. Templates and logical
patterns are shared; this is
not a natural-language domain holdout or semantic deduplication guarantee.
| Measure | Base | OpenJeff |
|---|---|---|
| Synthetic final accuracy (600) | 79.67% | 97.50% |
| Synthetic final calibrated ECE (15 bins) | 0.0554 | 0.0137 |
| Candidate-order disagreement on final scenarios | 22.00% | 1.83% |
| Public JevBench accuracy (231) | 70.13% | 90.04% |
| Public JevBench hard accuracy (111) | 42.34% | 80.18% |
| Public hard ECE (10 bins) |
Synthetic final accuracy gain is 17.83 percentage points, paired row-bootstrap 95% interval [14.67, 20.83]. This interval concerns the sampled synthetic generator distribution. Related candidate reversals are excluded from the independent-row bootstrap. The rule-grounded tasks are also solvable with deterministic programs; the result validates adaptation, not a need to replace those programs with an LLM.
Each frozen backend has its own positive temperature fitted on the 600-example fit set: base 2.7638, adapter 1.0207. Calibration artifacts bind scorer settings, weights, prompt, package versions, and GPU model. Changing that environment requires new calibration rather than reusing an unsupported probability claim.
JevBench uses only the 231 public tasks, the pinned upstream scoring function, and unchanged temperatures; no checkpoint was altered afterward. Only state, question, and options enter prompts. Gold answers, rationales, author metadata, and gold probabilities are excluded. The diagnostic raises the context guard to 8,192 tokens; longest prompt 4,071. It is not the official 534-item ranking. See complete results for slices and publisher comparisons.
Tested on NVIDIA A100 SXM 80 GB, Linux/Python 3.12.3, PyTorch 2.8.0+cu128, Transformers 5.17.0, PEFT 0.21.0, and SDPA. Training/evaluation script elapsed 28.65 minutes with 44.57 GiB peak allocated memory. Public-benchmark local serial adapter scoring: median 72.3 ms, p95 553.3 ms. These exclude cold load, network, queues, and concurrency; they establish no deployment SLA.
The saved adapter passed offline reload and HTTP probability parity checks.
Release validation recomputed 2,800 held-out score rows and both temperature fits.
Weight/tokenizer hashes, recipe, raw scores, selection, calibration, and runtime
are under runs/pilot-v1/. Source and data manifests accompany the release.
The container tag was recorded but its OCI digest was not; bitwise retraining
reproducibility is not claimed. Usage guide.
This pilot does not establish fairness, robustness to natural-language policy ambiguity or prompt injection, employment suitability, medical/legal reliability, privacy compliance, or production throughput. A narrow synthetic curriculum and public benchmark can understate distribution shift. Candidate-order sensitivity was reduced, not eliminated. Hard-task probability calibration remains imperfect. Foundation pretraining may contain public benchmark material; its contamination status is unknown even though OpenJeff adaptation excludes benchmark examples.
No full Jev API comparison, learned hybrid bridge, scalar continuous-output head, production authentication, or validated enterprise deployment is included. At the time of the original pilot archive, the optional diffusion control had failed during startup before model evaluation. A subsequent completed diffusion study tested structured guidance and controls without changing this adapter. It found mixed gains and leaves AR as the default. Its H200 calibrators do not replace this pilot's A100 calibrators.
| 0.2756 |
| 0.1417 |