Model details
Table with columns: Item, Value| Item | Value |
|---|
| Adapter type | LoRA / PEFT |
| Base model | Qwen/Qwen3.8-27B |
| LoRA rank / alpha / dropout | 8 / 16 / 0 |
| Selected checkpoint | Pure-Renji cross-validation Fold 3 |
| Input | 8 uniformly sampled frames, resized to 224×224 |
| Output | Strict JSON containing one or two ordered phase segments |
| Precision used for training and evaluation | BF16 |
Fold 3 was selected because it had the highest duration accuracy and macro-F1
among the five pure-Renji Qwen folds. It is a cross-validation checkpoint, not
an adapter retrained on all available cases.
Training data
Training clips were derived from 20 Renji Hospital ESD cases. The source was
represented as 1 fps frames and converted to H.264 clips:
- 1,448 pure-phase clips, balanced at 362 clips per phase.
- 300 boundary clips, balanced at 100 clips per adjacent transition.
- Pure-phase clips are 32 seconds long and were produced with a sliding window,
usually using an 8-second stride.
- Boundary clips are 64 seconds long and contain two adjacent phases.
- Video encoding: 1 fps, 512×448, H.264, CRF 18, yuv420p.
- Splitting was performed at case level; cases do not cross train,
validation, and test partitions within a fold.
The 60 source transition boundaries used to derive boundary clips were fitted
algorithmically and have not all been individually reviewed by clinicians.
The private source videos and annotations are not included in this repository.
Training procedure
- 3 epochs.
- AdamW optimizer, learning rate
1e-4.
- Cosine learning-rate schedule with warmup ratio
0.1.
- Maximum gradient norm
1.0.
- Gradient checkpointing enabled.
- Vision tower frozen; multimodal projector trainable.
- LoRA applied to eligible projection modules.
- DeepSpeed ZeRO-3 on 2× NVIDIA A100 GPUs for Fold 3.
- Per-device batch size 3, gradient accumulation 1, effective global batch 6.
- Validation and checkpoint saving at each epoch; best adapter selected by
validation loss.
- LlamaFactory template
qwen3_5_nothink, SDPA attention, cutoff length 8192.
- Greedy generation (
do_sample=false), normally with max_new_tokens=128.
Evaluation
Fold 3 results on its held-out pure-Renji test partition:
Table with columns: Exact match, Duration accuracy, Macro-F1| Exact match | Duration accuracy | Macro-F1 |
|---|
| 80.29% | 88.48% | 88.87% |
For context, pooling the out-of-fold predictions from five separately trained
Qwen checkpoints gave 78.66% exact match, 85.99% duration accuracy, 86.41%
macro-F1, and 9.55 seconds boundary MAE. Those pooled figures describe the
five-checkpoint experiment and must not be interpreted as metrics for this
Fold 3 adapter alone.
Prompt and output contract
The deployed pipeline constructs the following instruction for a 32-second
window:
<video>Segment this 32-second ESD clip into one or two chronologically ordered procedural phases. Allowed phases: Diagnosis, Local injection, Incision / Dissection, Hemostasis. Cover the complete clip without gaps or overlaps. Return only a valid JSON object with the key "segments"; every segment must contain exactly "phase", "start_seconds", and "end_seconds".
Expected output example:
{ "segments": [ { "phase": "Diagnosis", "start_seconds": 0, "end_seconds": 32 } ]}
The application validates the JSON and phase names before using a prediction.
For full-length videos, it applies the model to overlapping 32-second windows,
fuses the window predictions on the source timeline, and decodes a monotonic
four-phase sequence.
Usage
The tested inference pipeline and PhasePro review GUI are available at:
https://github.com/mumulin122333/QwenVideoSegment
After cloning that repository, point its LlamaFactory configuration to the
public base and adapter IDs:
export RENJI_QWEN_BASE_MODEL=Qwen/Qwen3.8-27Bexport RENJI_QWEN_ADAPTER=Mumulin/Renji-Qwen3.8-27B-Phase-LoRA-Fold3
The repository converts a source video into overlapping windows, runs this
adapter, validates and fuses the outputs, and exports a CSV loadable by its GUI.
Limitations
- The training set contains only 20 cases from one center; performance at other
centers, on other devices, and under different acquisition protocols is not
established.
- This is the best single cross-validation fold selected after comparing fold
results, not a final all-case fit.
- Derived overlapping clips are not statistically independent observations.
- Transition supervision includes algorithmically fitted boundaries that still
require expert review.
- The downstream monotonic decoder assumes phases follow the fixed order shown
above and may suppress real procedural deviations.
- Exact JSON compliance and phase accuracy can change with prompts, frame
sampling, decoding parameters, base-model revisions, and software versions.
License
This adapter is released under the Apache License 2.0, matching the declared
license of Qwen/Qwen3.8-27B. Users are responsible for following the base
model license and all applicable rules governing medical data and use.