GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
roger33303
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
roger33303
Model Tree
unsloth/granite-4.1-3b
Input Modalities
Output Modalities
Supported Functionality
Evaluated on the full 6,983-row validation split using the HuggingFace evaluate library. Inference was run using vLLM (tensor_parallel_size=2) on 2x NVIDIA T4 GPUs.
To properly measure the lift from fine-tuning, the base model (unsloth/granite-4.1-3b, no fine-tuning) was also evaluated on the identical validation set, prompted with the same JARVIS system prompt used to generate the training data. This isolates what fine-tuning actually contributed versus what prompting alone can achieve.
| Metric | Base + system prompt | Fine-tuned 16bit | Notes |
|---|---|---|---|
| BERTScore F1 | 0.8420 | 0.8641 | Primary metric: semantic and stylistic alignment |
| BLEU | 0.0806 | 0.1555 | Lexical overlap with reference outputs |
| ROUGE-L | 0.1803 | 0.2408 | Longest common subsequence recall |
| Persona adherence delta | -0.1174 | +0.0093 | Model vs. reference persona score (0 is ideal) |
| Length ratio | 2.390 | 1.232 | Model output length divided by reference length (1.0 is ideal) |
| Self-BLEU | 0.1017 | 0.0584 | Lower indicates healthier diversity, less templated output |
Why BERTScore is the primary metric: for persona fine-tuning, lexical overlap (BLEU/ROUGE) is a poor proxy for style fidelity. BERTScore captures semantic and contextual alignment at the embedding level, making it far more sensitive to whether the model has genuinely adopted the JARVIS voice rather than just copying words.
Interpretation:
The model was fine-tuned to follow a 10-rule JARVIS dialect style guide. Every training example was verified against these rules.
Example exchange:
text
User: What is the weather like today? JARVIS: I am afraid I do not have access to your current location'smeteorological data at this moment, sir. You may wish to consult adedicated weather service, or if you provide your coordinates I canmake the appropriate enquiry on your behalf.| Property | Value |
|---|---|
| Base model | unsloth/granite-4.1-3b |
| Architecture | Granite 4.1, 3.40B parameters, 40 layers |
| Fine-tuning method | QLoRA (4-bit quantization during training) |
| LoRA rank | r=16, alpha=32 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, shared_mlp.input_linear, shared_mlp.output_linear |
| Max sequence length | 1536 tokens |
| Training steps completed | 3,000 of 8,476 (approximately 35% of full run) |
| Saved as | bfloat16 merged weights (full model, no adapter required) |
| License | Apache 2.0 |
| Developed by | roger33303 |
Fine-tuned on roger33303/jarvis_deep_dataset, a 69,836-row instruction-response dataset constructed by rewriting WizardLMTeam/WizardLM_evol_instruct_70k into the JARVIS persona using the DeepSeek API with 400 concurrent async workers and fault-tolerant resume logic.
| Split | Rows |
|---|---|
| Train | 60,853 |
| Validation | 6,983 |
| Test | 2,000 |
| Total | 69,836 |
Splits were constructed using quantile-based stratified sampling on token length (n_bins=3, boundaries via np.percentile) to ensure balanced representation of short, medium, and long instructions across all splits. No hardcoded length boundaries; fully data-driven.
Training was performed using Unsloth and HuggingFace TRL with Accelerate DDP across 2x NVIDIA T4 GPUs on Kaggle.
python
SFTConfig( per_device_train_batch_size = 1, # DDP stability (Unsloth known issue with batch>1) gradient_accumulation_steps = 4, # effective batch = 1 x 4 x 2 GPUs = 8 num_train_epochs = 1, learning_rate = 1e-4, optim = "adamw_8bit", lr_scheduler_type = "cosine", warmup_steps = 100, weight_decay = 0.01, eval_strategy = "steps", eval_steps = 500, # roughly 17 evals across the run save_steps = 250, max_seq_length = 1536, ddp_find_unused_parameters = False, # required for DDP + LoRA mixed_precision = "bf16",)DDP setup (Accelerate config):
yaml
compute_environment: LOCAL_MACHINEdistributed_type: MULTI_GPUnum_processes: 2num_machines: 1mixed_precision: bf16use_cpu: falseKey DDP requirements:
device_map must not be passed to FastLanguageModel.from_pretrained, as it conflicts with DDP process placementPartialState().is_main_processddp_find_unused_parameters=False required for LoRA and DDP to avoid hangingThroughput: approximately 4.34 s/it (2x T4 DDP) versus approximately 7.9 s/it (single T4), roughly a 1.8x speedup.
Training on responses only: used train_on_responses_only() with Granite's role tokens.
python
instruction_part = "<|start_of_role|>user<|end_of_role|>"response_part = "<|start_of_role|>assistant<|end_of_role|>"Checkpoint resume: automatically detects latest checkpoint in output_dir and resumes, with no manual configuration needed between Kaggle sessions.
python
from transformers import AutoTokenizer, AutoModelForCausalLMimport torch model_id = "roger33303/Jarvis_granite_4.1_3b_finetuned_16bit" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto",) messages = [{"role": "user", "content": "Explain quantum entanglement."}] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True,) inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate( **inputs, max_new_tokens=512, temperature=0.7, do_sample=True,) response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(response)python
from vllm import LLM, SamplingParamsfrom transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("roger33303/Jarvis_granite_4.1_3b_finetuned_16bit") llm = LLM( model="roger33303/Jarvis_granite_4.1_3b_finetuned_16bit", tensor_parallel_size=2, max_model_len=1536, gpu_memory_utilization=0.85, dtype="float16", # T4 does not support bfloat16) sampling_params = SamplingParams(temperature=0.7, max_tokens=512) messages = [{"role": "user", "content": "What is the status of the reactor?"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate([prompt], sampling_params)print(outputs[0].outputs[0].text.strip())All code used to train and evaluate this model is publicly available.
| Resource | Link |
|---|---|
| Training notebook (with outputs) | Kaggle: parallel-training-on-multiple-gpus |
| Evaluation notebook (with outputs) | Kaggle: vllm-evaluate-16-bit-model-evaluation |
| Training dataset | roger33303/jarvis_deep_dataset |
The evaluation pipeline used vLLM for fast batched inference (batch_size=64) with checkpoint-resume support, then computed all metrics using the HuggingFace evaluate library.
Persona adherence score is a composite metric:
python
score = ( 0.35 * has_sir + # addresses user as "sir" 0.30 * no_contractions + # avoids contractions 0.20 * no_banned_words + # avoids casual vocabulary 0.15 * has_preferred_phrase # uses formal JARVIS phrases)The persona delta (model score minus reference score) of +0.009 indicates the model matches the target persona intensity, neither undershooting nor overshooting into caricature.
The base model baseline was generated using the identical JARVIS system prompt (from the dataset-creation pipeline's prompt_lib.py) paired directly with each validation instruction, with no fine-tuning and no rewrite wrapper, to test whether prompting alone can achieve the persona that fine-tuning was designed to teach.
Trained with Unsloth for memory-efficient QLoRA fine-tuning on Kaggle's free GPU tier.
User: What is the weather like today?JARVIS: I am afraid I do not have access to your current location'smeteorological data at this moment, sir. You may wish to consult adedicated weather service, or if you provide your coordinates I canmake the appropriate enquiry on your behalf.SFTConfig( per_device_train_batch_size = 1, # DDP stability (Unsloth known issue with batch>1) gradient_accumulation_steps = 4, # effective batch = 1 x 4 x 2 GPUs = 8 num_train_epochs = 1, learning_rate = 1e-4, optim = "adamw_8bit", lr_scheduler_type = "cosine", warmup_steps = 100, weight_decay = 0.01, eval_strategy = "steps", eval_steps = 500, # roughly 17 evals across the run save_steps = 250, max_seq_length = 1536, ddp_find_unused_parameters = False, # required for DDP + LoRA mixed_precision = "bf16",)compute_environment: LOCAL_MACHINEdistributed_type: MULTI_GPUnum_processes: 2num_machines: 1mixed_precision: bf16use_cpu: falseinstruction_part = "<|start_of_role|>user<|end_of_role|>"response_part = "<|start_of_role|>assistant<|end_of_role|>"from transformers import AutoTokenizer, AutoModelForCausalLMimport torch model_id = "roger33303/Jarvis_granite_4.1_3b_finetuned_16bit" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto",) messages = [{"role": "user", "content": "Explain quantum entanglement."}] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True,) inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate( **inputs, max_new_tokens=512, temperature=0.7, do_sample=True,) response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)print(response)from vllm import LLM, SamplingParamsfrom transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("roger33303/Jarvis_granite_4.1_3b_finetuned_16bit") llm = LLM( model="roger33303/Jarvis_granite_4.1_3b_finetuned_16bit", tensor_parallel_size=2, max_model_len=1536, gpu_memory_utilization=0.85, dtype="float16", # T4 does not support bfloat16) sampling_params = SamplingParams(temperature=0.7, max_tokens=512) messages = [{"role": "user", "content": "What is the status of the reactor?"}]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate([prompt], sampling_params)print(outputs[0].outputs[0].text.strip())score = ( 0.35 * has_sir + # addresses user as "sir" 0.30 * no_contractions + # avoids contractions 0.20 * no_banned_words + # avoids casual vocabulary 0.15 * has_preferred_phrase # uses formal JARVIS phrases)