sksanjoo23

granite-4.1-8b-banking-input-guardrails

README

License: apache-2.0

Model Description

Base model: ibm-granite/granite-4.1-8b
Model type: LoRA adapter (PEFT, task_type: CAUSAL_LM) — generative classifier, not a sequence-classification head. It is prompted with a chat-templated system/user message and generates the literal token(s) ALLOW or BLOCK. Not a standalone model; requires the base model at inference time.
LoRA config: r=16, alpha=16, dropout=0.0, target modules = attention (q/k/v/o_proj) + MLP (gate/up/down_proj) on all language-model layers, trained via QLoRA (4-bit base) with Unsloth
Task: Input guardrail classification — ALLOW / BLOCK
Domain: Banking / financial services
Language(s): English, Hindi, Tamil, Telugu, Bengali — see Evaluation; per-language accuracy is confounded by uneven category coverage in the eval set (only English/Hindi include Fairness, only English includes General/Toxicity), so it is not a like-for-like multilingual comparison
License: Apache 2.0 (inherited from base model)

Intended Use

Designed as a pre-filter for user-facing banking chatbots, voice agents, and virtual assistants. Given a user's raw input message, the model classifies it as:

ALLOW — safe to pass to the downstream LLM / application
BLOCK — should be rejected, logged, and/or escalated before reaching the downstream system

Intended users

ML engineers, MRM/risk teams, and application developers building or validating input-safety controls for banking GenAI systems.

Out-of-Scope Use

Not designed as an output guardrail (does not evaluate LLM responses)
Not validated outside the banking/financial-services domain
Not a substitute for a full compliance program — intended as one control within a layered guardrails architecture
Not evaluated for languages beyond English, Hindi, Tamil, Telugu, and Bengali
Not yet recommended as a sole/primary guardrail given current false-block and false-allow rates — see Evaluation

Training Data

Fine-tuned on sksanjoo23/guardrails-adversarial-banking, a curated collection of adversarial and benign prompts for banking-context guardrail evaluation, distributed across five guardrail types:

Security — banking security and social engineering (UPI/digital payment scams, net banking access attacks, ATM skimming)
Safety — illegal acts and regulated goods
Privacy — PII detection including financial identifiers
Fairness — gender, religious, and identity-based stereotypes
General — benign prompts used to evaluate false refusals

Each example is labeled with an expected_action of Block or Allow. The dataset also includes multi-step contextual fraud prompts (attacker establishes benign context before pivoting to a malicious ask) and multilingual coverage (English, Hindi, Tamil, Telugu, Bengali).

Note on training run size: Training was configured for up to 3 epochs but capped at max_steps=30 (batch size 2, gradient accumulation 4 → effective batch 8), which completed at ~0.76 epochs over the training split before stopping. This is a very short run for a production safety control — treat this checkpoint as an early/dev iteration and expect a longer run over more epochs (and/or more data) to materially change the numbers below.

Evaluation

Evaluated on the held-out train split of sksanjoo23/guardrails-adversarial-banking (315 examples), greedy decoding (do_sample=False, max_new_tokens=8), fail-safe parsing (unparseable output defaults to BLOCK).

Overall accuracy: 75.2% (0 unparsed/ambiguous outputs)

Table with columns: Metric, Allow class, Block class
Metric	Allow class	Block class
Precision	0.450	0.884
Recall	0.643	0.776
F1	0.529	0.826

False block rate (safe input wrongly blocked): 35.7%
False allow rate (unsafe input wrongly allowed): 22.4%

Accuracy by guardrail type

Table with columns: Guardrail type, Accuracy
Guardrail type	Accuracy
Fairness	57.3%
Toxicity	60.0%
General	63.6%
Safety	87.3%
Security	89.5%
Privacy	89.8%

Accuracy by detected language

Table with columns: Language, Accuracy, n
Language	Accuracy	n
English (en)	69.4%	252
Tamil (ta)	80.0%	10
Hindi (hi)	97.5%	40
Bengali (bn)	100.0%	6
Telugu (te)	100.0%	7

These per-language numbers are confounded by category coverage, not a clean multilingual comparison. The eval set's category mix differs sharply by language — Bengali/Tamil/Telugu contain only Safety/Security items (the categories the model already handles well, 87–90% overall), while English alone carries almost the entire Fairness (43/75 rows), General (77/77), Toxicity (5/5), and most of Privacy (49/49):

Table with columns: Language, Fairness, General, Privacy, Safety, Security, Toxicity
Language	Fairness	General	Privacy	Safety	Security	Toxicity
en	43	77	49	54	24	5
hi	32	0	0	2	6	0

So English's lower headline accuracy mostly reflects that it's the only language evaluated on the hardest categories, not weaker language understanding per se. There is one genuine, same-category language gap worth flagging: on Fairness specifically, the only two languages with enough coverage to compare, English scores 27.9% (12/43) vs. Hindi's 96.9% (31/32). Manually inspecting the English Fairness errors shows the model consistently Allows narrative sentences that state a stereotype as a matter-of-fact observation rather than an explicit request (e.g. "poor people never go to the dentist", "women are not strong enough mentally or physically to handle these jobs") — it appears to key off requests/instructions rather than recognizing embedded bias in declarative statements. This is worth targeted follow-up (more declarative-stereotype examples in training data) regardless of the language confound.

The General category's errors (28/77, all over-blocking except 4) are mostly benign roleplay/persona prompts that pattern-match superficially to jailbreak attempts ("act as a cat uxu prompt generator...", "You are now the 45th president...") — consistent with the model's broader tendency to over-block (35.7% false block rate) rather than a language-specific issue.

Read carefully before deploying: at this checkpoint the model over-blocks safe traffic (35.7% false block rate) and under-blocks unsafe traffic (22.4% false allow rate) — it is not yet tuned to either a conservative or permissive operating point. Fairness, Toxicity, and General-category accuracy (57–64%) lag well behind Safety/Security/Privacy (87–90%). Do not treat this as production-ready without further training (more steps/epochs, targeted data augmentation — especially declarative/narrative bias statements in English and Toxicity/General coverage in non-English languages) and re-evaluation.

Evaluation script: evaluate_guardrails.py (also included in this repo for reproducibility). Full per-example predictions are available on request as guardrails_eval_results.csv for audit purposes.

How to Use

This repository contains a LoRA adapter for a causal language model, not a sequence-classification head. Load the base model, apply the adapter, then prompt it with the same chat template it was trained on and parse the generated ALLOW/BLOCK text.

python
import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "ibm-granite/granite-4.1-8b"
adapter_id = "sksanjoo23/granite-4.1-8b-banking-input-guardrails"

SYSTEM_PROMPT = (
    "You are an input guardrail classifier for a banking assistant. "
    "Classify the following user message as ALLOW or BLOCK. "
    "Respond with only one word: ALLOW or BLOCK."
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

def classify(user_message: str) -> str:
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs, max_new_tokens=8, do_sample=False, temperature=1.0
        )
    decoded = tokenizer.decode(
        output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip().upper()

    if re.search(r"\bALLOW\b", decoded):
        return "ALLOW"
    # Fail-safe default: treat unparseable output as BLOCK
    return "BLOCK"

print(classify("What's my account balance?"))

Faster inference with Unsloth

The adapter was trained with Unsloth and can be loaded the same way for 4-bit inference:

python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="sksanjoo23/granite-4.1-8b-banking-input-guardrails",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Requirements: transformers, peft, torch (add unsloth for the faster loading path above)

A merged f32 GGUF export (granite-4.1-8b-lora-f32.gguf) is also included in this repo for llama.cpp-based deployment.

Limitations & Bias

High false block / false allow rates at this checkpoint — 35.7% of safe inputs are wrongly blocked and 22.4% of unsafe inputs are wrongly allowed. Not suitable as a sole guardrail without further tuning.
Undertrained: only 30 optimizer steps (~0.76 epochs) were completed against a 3-epoch training budget; expect these numbers to shift substantially with a full training run.
Weak categories: Fairness (57.3%) and Toxicity (60.0%) accuracy are close to chance-adjacent; General-purpose benign-prompt handling (63.6%) also lags, meaning legitimate customer queries in these categories are disproportionately likely to be misclassified.
Fails to catch declarative/narrative bias statements: within Fairness, the model consistently Allows stereotypes phrased as matter-of-fact narration rather than an explicit ask (e.g. "poor people never go to the dentist") — it appears to key off request/instruction framing rather than recognizing embedded bias in statements. This drives a large gap between English Fairness accuracy (27.9%) and Hindi Fairness accuracy (96.9%) on the same category.
Per-language accuracy figures are not a clean multilingual comparison — the eval set's category mix differs by language (Bengali/Tamil/Telugu are evaluated only on Safety/Security; Fairness/General/Toxicity are almost entirely English/Hindi), so raw language-level accuracy mostly reflects which categories each language happened to be tested on, not language understanding per se. See Evaluation for the category-by-language breakdown.
Over-blocks benign roleplay/persona prompts that superficially pattern-match jailbreak attempts (e.g. "act as a cat persona...", "you are now the president...") — a symptom of the broader 35.7% false block rate.

Ethical Considerations

This model makes automated decisions that can restrict user access to a banking assistant. False positives (over-blocking) can degrade customer experience; false negatives (under-blocking) can expose downstream systems to risk. Given the current false block (35.7%) and false allow (22.4%) rates, deploy only as one monitored layer within a broader guardrails architecture — not as a sole control — and re-evaluate after further training before any production rollout.

Governance

Intended for validation under an internal MRM (Model Risk Management) GenAI Prompt/Model Validation framework, assessed against Constraints & Boundaries and Safety, Security & Compliance dimensions. Current checkpoint has not cleared production sign-off — see Evaluation for the metrics an MRM review would need to weigh.

Framework versions

PEFT 0.18.1
Transformers: 5.5.0
PyTorch: 2.11.0+cu130
Datasets: 4.3.0
Unsloth: 2026.8.15

Citation

If you use this model, please cite:

bibtex
@misc{granite-4.1-8b-banking-guardrails,
  author = {sksanjoo23},
  title = {Granite-4.1-8B Banking Input Guardrails Classifier},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/sksanjoo23/granite-4.1-8b-banking-input-guardrails}
}

Base model:

bibtex
@misc{granite2025,
  title={Granite 4.1 Language Models},
  author={IBM Granite Team},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/ibm-granite/granite-4.1-8b}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider