GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ewxxyyzz
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ewxxyyzz
Model Tree
google/gemma-4-12B
Input Modalities
Output Modalities
Supported Functionality
axolotl version: 0.19.0
yaml
base_model: google/gemma-4-12Bstrict: false # full-run continued pretraining on the belarusian + fineweb-replay mix (see cpt_full.sh data).# the dataset is PRE-TOKENIZED (input_ids/attention_mask/labels): axolotl detects that and# skips its slow tokenization. rows are already <bos> doc <eos> and <= sequence_lendatasets: - path: /workspace/cpt-fullrun/data/mix-tok shuffle_merged_datasets: true # global shuffle happens here (the data stage only mixes + window-shuffles)dataset_prepared_path: /workspace/cpt-fullrun/prepared-23020bf3val_set_size: 0.000186output_dir: /workspace/cpt-fullrun/out sequence_len: 4096sample_packing: truepad_to_sequence_len: trueeval_sample_packing: falseattn_implementation: sdpa micro_batch_size: 2gradient_accumulation_steps: 8num_epochs: 1learning_rate: 2e-5lr_scheduler: cosinecosine_min_lr_ratio: 0.1 # floor the lr instead of decaying to 0warmup_ratio: 0.03weight_decay: 0.1max_grad_norm: 1.0optimizer: adamw_torch_fusedadam_beta2: 0.95bf16: autotf32: true logging_steps: 1eval_steps: 110save_steps: 110save_total_limit: 4save_only_model: 0auto_resume_from_checkpoints: true # kill + rerun to resume (test it after the first save!) seed: 42 fsdp_version: 2fsdp_config: offload_params: false cpu_ram_efficient_loading: true auto_wrap_policy: TRANSFORMER_BASED_WRAP transformer_layer_cls_to_wrap: Gemma4UnifiedTextDecoderLayer state_dict_type: SHARDED_STATE_DICT reshard_after_forward: true activation_checkpointing: trueuse_wandb: truewandb_project: zorka-vieniera-cptwandb_name: full-full-8gpu-1026-1732wandb_run_id: cpt-full-f24bc1cdplugins: - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugincut_cross_entropy: trueThis is a Belarusian-centered model that has been trained using continued pretraining of google/gemma-4-12B. The dataset used was primarily web scrapes, with some educational data and literature mixed in and with Lacinka and Taraszkevica filtered out.
The model was trained for ~26 hours on 8 H100s. Thanks to lium.io for providing me with free credits for this project!
um. TODO !!!
For some reason, doing inference using SDPA attention puts s all over the output. Use Eager Attention instead.
python
import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM repo = "ewxxyyzz/zorka-vieniera-12b" # TODO tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained( repo, torch_dtype=torch.bfloat16, attn_implementation="eager", # do not use sdpa, see above device_map="auto",) prompt = "Беларусь — гэта"inputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate( **inputs, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9,)print(tok.decode(out[0], skip_special_tokens=True))base_model: google/gemma-4-12Bstrict: false # full-run continued pretraining on the belarusian + fineweb-replay mix (see cpt_full.sh data).# the dataset is PRE-TOKENIZED (input_ids/attention_mask/labels): axolotl detects that and# skips its slow tokenization. rows are already <bos> doc <eos> and <= sequence_lendatasets: - path: /workspace/cpt-fullrun/data/mix-tok shuffle_merged_datasets: true # global shuffle happens here (the data stage only mixes + window-shuffles)dataset_prepared_path: /workspace/cpt-fullrun/prepared-23020bf3val_set_size: 0.000186output_dir: /workspace/cpt-fullrun/out sequence_len: 4096sample_packing: truepad_to_sequence_len: trueeval_sample_packing: falseattn_implementation: sdpa micro_batch_size: 2gradient_accumulation_steps: 8num_epochs: 1learning_rate: 2e-5lr_scheduler: cosinecosine_min_lr_ratio: 0.1 # floor the lr instead of decaying to 0warmup_ratio: 0.03weight_decay: 0.1max_grad_norm: 1.0optimizer: adamw_torch_fusedadam_beta2: 0.95bf16: autotf32: true logging_steps: 1eval_steps: 110save_steps: 110save_total_limit: 4save_only_model: 0auto_resume_from_checkpoints: true # kill + rerun to resume (test it after the first save!) seed: 42 fsdp_version: 2fsdp_config: offload_params: false cpu_ram_efficient_loading: true auto_wrap_policy: TRANSFORMER_BASED_WRAP transformer_layer_cls_to_wrap: Gemma4UnifiedTextDecoderLayer state_dict_type: SHARDED_STATE_DICT reshard_after_forward: true activation_checkpointing: trueuse_wandb: truewandb_project: zorka-vieniera-cptwandb_name: full-full-8gpu-1026-1732wandb_run_id: cpt-full-f24bc1cdplugins: - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugincut_cross_entropy: trueimport torchfrom transformers import AutoTokenizer, AutoModelForCausalLM repo = "ewxxyyzz/zorka-vieniera-12b" # TODO tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained( repo, torch_dtype=torch.bfloat16, attn_implementation="eager", # do not use sdpa, see above device_map="auto",) prompt = "Беларусь — гэта"inputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate( **inputs, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9,)print(tok.decode(out[0], skip_special_tokens=True))