GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
PaddySeahorse
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
PaddySeahorse
Model Tree
TinyLlama/TinyLlama-1.1B-Chat-v1.0
Input Modalities
Output Modalities
Supported Functionality
TinyLlama/TinyLlama-1.1B-Chat-v1.0torch.bfloat16The model was loaded using the following BitsAndBytesConfig:
python
quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4")It was then loaded with:
python
model = AutoModelForCausalLM.from_pretrained( "TinyLlama/TinyLlama-1.1B-Chat-v1.0", quantization_config=quantization_config, device_map="auto")and uploaded to the Hugging Face Hub with model.push_to_hub().
For reproducibility and historical reference, the relevant script used to create this repository was:
python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom huggingface_hub import loginfrom accelerate import Acceleratorimport torch # Authenticate with Hugging Face.# Do not hard-code access tokens in public scripts.login(token="YOUR_HF_TOKEN") accelerator = Accelerator() quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4") model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, quantization_config=quantization_config, device_map="auto") model = accelerator.prepare(model) repo_id = "PaddySeahorse/tinyllama-4bit" model.push_to_hub( repo_id, commit_message="Push 4-bit quantized TinyLlama") tokenizer.push_to_hub( repo_id, commit_message="Push tokenizer")This repository was produced using Hugging Face Transformers and
bitsandbytes. A typical loading configuration is:
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig repo_id = "PaddySeahorse/tinyllama-4bit" quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4") tokenizer = AutoTokenizer.from_pretrained(repo_id) model = AutoModelForCausalLM.from_pretrained( repo_id, quantization_config=quantization_config, device_map="auto",)For chat-style inference, refer to the usage instructions and chat template of the original TinyLlama model.
This repository was created primarily as a learning experiment for:
bitsandbytes 4-bit loading;It is not an independently trained or fine-tuned model. The underlying model
and its capabilities originate from TinyLlama/TinyLlama-1.1B-Chat-v1.0.
This repository should be considered an experimental quantization artifact.
No systematic evaluation was performed after quantization, so no claims are made regarding accuracy, perplexity, benchmark performance, or output quality relative to the original model.
The repository was produced by loading the base model through the
bitsandbytes 4-bit quantization path and subsequently uploading it. It should
therefore not be confused with separately converted GPTQ, AWQ, or GGUF
quantizations.
Compatibility may depend on the versions of transformers, accelerate,
bitsandbytes, PyTorch, and the available hardware.
For the original model's intended uses, limitations, training information, and other details, please refer to the https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0.
The original quantization experiment was performed in Google Colab with an NVIDIA T4 GPU.
Exact package versions from the original July–August 2025 environment were not recorded.
All credit for the base model goes to the TinyLlama project and its contributors.
Base model:
This repository only provides the experimental 4-bit quantized artifact created from that model.
Please refer to the license and usage terms of the original
TinyLlama/TinyLlama-1.1B-Chat-v1.0 model. This repository does not introduce
a new license for the underlying model.
quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4")model = AutoModelForCausalLM.from_pretrained( "TinyLlama/TinyLlama-1.1B-Chat-v1.0", quantization_config=quantization_config, device_map="auto")from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom huggingface_hub import loginfrom accelerate import Acceleratorimport torch # Authenticate with Hugging Face.# Do not hard-code access tokens in public scripts.login(token="YOUR_HF_TOKEN") accelerator = Accelerator() quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4") model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, quantization_config=quantization_config, device_map="auto") model = accelerator.prepare(model) repo_id = "PaddySeahorse/tinyllama-4bit" model.push_to_hub( repo_id, commit_message="Push 4-bit quantized TinyLlama") tokenizer.push_to_hub( repo_id, commit_message="Push tokenizer")import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig repo_id = "PaddySeahorse/tinyllama-4bit" quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4") tokenizer = AutoTokenizer.from_pretrained(repo_id) model = AutoModelForCausalLM.from_pretrained( repo_id, quantization_config=quantization_config, device_map="auto",)