GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
NickyNicky
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
NickyNicky
Model Tree
Input Modalities
Output Modalities
Supported Functionality
Python
!python -m pip install --upgrade pip!pip install "torch>=2.1.1" -U!pip install torchaudio==2.2.0!pip install -q datasets trl peft bitsandbytes sentencepiece wandb!pip install -q accelerate safetensors deepspeed!pip install -q scipy ninja -U!pip install -q -U transformers==4.38.0!pip install flash-attn==2.5.5 --no-build-isolationpy
import torchtorch.__version__#OUTPUTS: ('2.2.0+cu121' )py
from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline, logging, GenerationConfig, TextIteratorStreamer,) from transformers import StoppingCriteria, StoppingCriteriaList import torch model_id='NickyNicky/gemma-2b-it_oasst2_chatML_Cluster2_aya_multilingual' model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", trust_remote_code=True, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", # load_in_4bit=True, # low_cpu_mem_usage= True, ) max_length=1355print("max_length",max_length) tokenizer = AutoTokenizer.from_pretrained(model_id, # use_fast = False, max_length=max_length,) class ListOfTokensStoppingCriteria(StoppingCriteria): """ Clase para definir un criterio de parada basado en una lista de tokens específicos. """ def __init__(self, tokenizer, stop_tokens): self.tokenizer = tokenizer # Codifica cada token de parada y guarda sus IDs en una lista self.stop_token_ids_list = [tokenizer.encode(stop_token, add_special_tokens=False) for stop_token in stop_tokens] def __call__(self, input_ids, scores, **kwargs): # Verifica si los últimos tokens generados coinciden con alguno de los conjuntos de tokens de parada for stop_token_ids in self.stop_token_ids_list: len_stop_tokens = len(stop_token_ids) if len(input_ids[0]) >= len_stop_tokens: if input_ids[0, -len_stop_tokens:].tolist() == stop_token_ids: return True return False # Uso del criterio de parada personalizadostop_tokens = ["<end_of_turn>"] # Lista de tokens de parada # Inicializa tu criterio de parada con el tokenizer y la lista de tokens de paradastopping_criteria = ListOfTokensStoppingCriteria(tokenizer, stop_tokens) # Añade tu criterio de parada a una StoppingCriteriaListstopping_criteria_list = StoppingCriteriaList([stopping_criteria]) # improves control of responses in different languages.# trainer language codes: ["es", "en", "fr", "de"]input_code= "es"target_code= "en" #EXAMPLE #1input_text = f"""<bos><start_of_turn>systemYou are a helpful AI assistant.lista de codigos linguisticos disponibles: ["{input_code}", "{target_code}"].<end_of_turn><start_of_turn>userescribe una historia de 100 palabras<end_of_turn><start_of_turn>model""" ### OUTPUT EXAMPLE###''' #<start_of_turn>model#Once upon a time, in a quiet town nestled between rolling hills and lush forests, there lived a wise old owl named Merlin. Merlin had lived for many years, witnessing the passing of time and the changing seasons.#One sunny morning, as Merlin was perching on a branch high in the trees, he noticed a strange sight: a strange shadow following him like a ghost. To his surprise, it was a black bird with heavy wings.#Merlin's eyes widened in surprise and he knew that this bird was a visitor from the beyond. He tried to call upon his magic, but the bird's song couldn't be heard by him. The bird flew away, singing far away, as though Merlin's magic didn't exist.#That day, the wise owl decided to show the visitor what he had seen. He spoke and sang to the bird, telling her stories of ancient times and wise wisdom. To his delight, the bird listened intently, laughing sometimes and asking questions.#So, every day the wise owl and the visitor bird met in the forest, sharing stories and wisdom. Merlin learned that even the most ancient of birds could carry the light of the universe on their wings.#One autumn, while the owl was resting in his nest, he heard a strange noise rustling the leaves. To his surprise, the noise was the song of a young migrating bird. The bird was going to leave his nest to find a new home in a distant land.#Merlin felt a pang of sadness. He knew that soon the young bird would face the cold of the winter and the harshness of the journey. He thought of a special nest made of frost, sheltered by tiny ice crystals, which would keep the little bird warm during its journey.#One day, the young bird arrived at the safe nest.<end_of_turn> inputs = tokenizer.encode(txt, return_tensors="pt", add_special_tokens=False).to("cuda:0")max_new_tokens=700generation_config = GenerationConfig( max_new_tokens=max_new_tokens, temperature=0.32, #top_p=0.9, top_k=45, repetition_penalty=1., #1.1 do_sample=True, )outputs = model.generate(generation_config=generation_config, input_ids=inputs, stopping_criteria=stopping_criteria_list,)tokenizer.decode(outputs[0], skip_special_tokens=False) #True!python -m pip install --upgrade pip!pip install "torch>=2.1.1" -U!pip install torchaudio==2.2.0!pip install -q datasets trl peft bitsandbytes sentencepiece wandb!pip install -q accelerate safetensors deepspeed!pip install -q scipy ninja -U!pip install -q -U transformers==4.38.0!pip install flash-attn==2.5.5 --no-build-isolationimport torchtorch.__version__#OUTPUTS: ('2.2.0+cu121' )from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline, logging, GenerationConfig, TextIteratorStreamer,) from transformers import StoppingCriteria, StoppingCriteriaList import torch model_id='NickyNicky/gemma-2b-it_oasst2_chatML_Cluster2_aya_multilingual' model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", trust_remote_code=True, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", # load_in_4bit=True, # low_cpu_mem_usage= True, ) max_length=1355print("max_length",max_length) tokenizer = AutoTokenizer.from_pretrained(model_id, # use_fast = False, max_length=max_length,) class ListOfTokensStoppingCriteria(StoppingCriteria): """ Clase para definir un criterio de parada basado en una lista de tokens específicos. """ def __init__(self, tokenizer, stop_tokens): self.tokenizer = tokenizer # Codifica cada token de parada y guarda sus IDs en una lista self.stop_token_ids_list = [tokenizer.encode(stop_token, add_special_tokens=False) for stop_token in stop_tokens] def __call__(self, input_ids, scores, **kwargs): # Verifica si los últimos tokens generados coinciden con alguno de los conjuntos de tokens de parada for stop_token_ids in self.stop_token_ids_list: len_stop_tokens = len(stop_token_ids) if len(input_ids[0]) >= len_stop_tokens: if input_ids[0, -len_stop_tokens:].tolist() == stop_token_ids: return True return False # Uso del criterio de parada personalizadostop_tokens = ["<end_of_turn>"] # Lista de tokens de parada # Inicializa tu criterio de parada con el tokenizer y la lista de tokens de paradastopping_criteria = ListOfTokensStoppingCriteria(tokenizer, stop_tokens) # Añade tu criterio de parada a una StoppingCriteriaListstopping_criteria_list = StoppingCriteriaList([stopping_criteria]) # improves control of responses in different languages.# trainer language codes: ["es", "en", "fr", "de"]input_code= "es"target_code= "en" #EXAMPLE #1input_text = f"""<bos><start_of_turn>systemYou are a helpful AI assistant.lista de codigos linguisticos disponibles: ["{input_code}", "{target_code}"].<end_of_turn><start_of_turn>userescribe una historia de 100 palabras<end_of_turn><start_of_turn>model""" ### OUTPUT EXAMPLE###''' #<start_of_turn>model#Once upon a time, in a quiet town nestled between rolling hills and lush forests, there lived a wise old owl named Merlin. Merlin had lived for many years, witnessing the passing of time and the changing seasons.#One sunny morning, as Merlin was perching on a branch high in the trees, he noticed a strange sight: a strange shadow following him like a ghost. To his surprise, it was a black bird with heavy wings.#Merlin's eyes widened in surprise and he knew that this bird was a visitor from the beyond. He tried to call upon his magic, but the bird's song couldn't be heard by him. The bird flew away, singing far away, as though Merlin's magic didn't exist.#That day, the wise owl decided to show the visitor what he had seen. He spoke and sang to the bird, telling her stories of ancient times and wise wisdom. To his delight, the bird listened intently, laughing sometimes and asking questions.#So, every day the wise owl and the visitor bird met in the forest, sharing stories and wisdom. Merlin learned that even the most ancient of birds could carry the light of the universe on their wings.#One autumn, while the owl was resting in his nest, he heard a strange noise rustling the leaves. To his surprise, the noise was the song of a young migrating bird. The bird was going to leave his nest to find a new home in a distant land.#Merlin felt a pang of sadness. He knew that soon the young bird would face the cold of the winter and the harshness of the journey. He thought of a special nest made of frost, sheltered by tiny ice crystals, which would keep the little bird warm during its journey.#One day, the young bird arrived at the safe nest.<end_of_turn> inputs = tokenizer.encode(txt, return_tensors="pt", add_special_tokens=False).to("cuda:0")max_new_tokens=700generation_config = GenerationConfig( max_new_tokens=max_new_tokens, temperature=0.32, #top_p=0.9, top_k=45, repetition_penalty=1., #1.1 do_sample=True, )outputs = model.generate(generation_config=generation_config, input_ids=inputs, stopping_criteria=stopping_criteria_list,)tokenizer.decode(outputs[0], skip_special_tokens=False) #True