AppleMind-AI

AppleMind-1.0-Mini

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Parameters	1,020,480
Architecture	AppleMind 1.0 Mini decoder-only Transformer
Layers	4
Hidden size	128
Intermediate size	512
Attention heads	4
KV heads	4
Head dim	32
Attention style	Multi-head causal self-attention
MLP	GELU
Position embeddings	Learned positional embeddings
Normalization	LayerNorm
Vocabulary size	50,263
Context length	256
Embeddings	Tied input/output embeddings
Tokenizer	Digit-aware byte-level BPE (GPT-2 + special tokens)
Weight format	safetensors
HF architecture	`GPT2LMHeadModel`
HF model type	`gpt2`
Final training steps	2,288
Tokens seen	299,892,736
Tokens/parameter	293.87:1
Training data	FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus
Training mixture	100M + 100M + 100M tokens
Precision	BF16
Final model	AppleMind 1.0 Mini

Credits to BananaMind for inspiring me to make AppleMind.

Tokenizer

AppleMind 1.0 Mini uses a 50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.

Table with columns: Special token, ID
Special token	ID
`<\|pad\|>`	50,260
`<\|bos\|>`	50,261
`<\|eos\|>`	50,262

Training Data

Table with columns: Dataset, Target Tokens, Share
Dataset	Target Tokens	Share
FineWeb-Edu	100M	33.33%
FineWeb-HQ	100M	33.33%
SmolLM-Corpus	100M	33.33%
Total	300M	100%

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

Training Setup

Table with columns: Field, Value
Field	Value
Sequence length	256
Micro batch	512 sequences
Gradient accumulation	1
Effective batch	512 sequences
Tokens per optimizer step	131,072
Final optimizer step	2,288
Configured optimizer steps	2,289
Optimizer	AdamW
Peak learning rate	0.0001

Evaluation

AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.

Table with columns: Benchmark, Score, Metric
Benchmark	Score	Metric
Average	N/A	`mean`
ARC Easy	N/A	`acc_norm,none`
PIQA	N/A	`acc_norm,none`
ARC Challenge	N/A	`acc_norm,none`
HellaSwag

The model's current generation quality has been checked with basic text-generation prompts, including:

Once upon a time
The little boy
In the forest

Prompt: Once upon a time

Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked its G during work used after several There and at there b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many

Prompt: The little boy

The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many

Prompt: In the forest

In the forest often form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but

Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.

bash
pip install -U transformers safetensors torch

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Note: AppleMind 1.0 Mini has a 256-token context window, so the prompt plus generated tokens should stay within that limit.

License

Apache 2.0

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

AppleMind-AI

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table with columns: Field, Value
Field	Value
Parameters	1,020,480
Architecture	AppleMind 1.0 Mini decoder-only Transformer
Layers	4
Hidden size	128
Intermediate size	512
Attention heads	4
KV heads	4
Head dim	32
Attention style	Multi-head causal self-attention
MLP	GELU
Position embeddings	Learned positional embeddings
Normalization	LayerNorm
Vocabulary size	50,263
Context length	256
Embeddings	Tied input/output embeddings
Tokenizer	Digit-aware byte-level BPE (GPT-2 + special tokens)
Weight format	safetensors
HF architecture	`GPT2LMHeadModel`
HF model type	`gpt2`
Final training steps	2,288
Tokens seen	299,892,736
Tokens/parameter	293.87:1
Training data	FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus
Training mixture	100M + 100M + 100M tokens
Precision	BF16
Final model	AppleMind 1.0 Mini

Credits to BananaMind for inspiring me to make AppleMind.

Tokenizer

Table with columns: Special token, ID
Special token	ID
`<\|pad\|>`	50,260
`<\|bos\|>`	50,261
`<\|eos\|>`	50,262

Training Data

Table with columns: Dataset, Target Tokens, Share
Dataset	Target Tokens	Share
FineWeb-Edu	100M	33.33%
FineWeb-HQ	100M	33.33%
SmolLM-Corpus	100M	33.33%
Total	300M	100%

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

Training Setup

Table with columns: Field, Value
Field	Value
Sequence length	256
Micro batch	512 sequences
Gradient accumulation	1
Effective batch	512 sequences
Tokens per optimizer step	131,072
Final optimizer step	2,288
Configured optimizer steps	2,289
Optimizer	AdamW
Peak learning rate	0.0001

Evaluation

AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.

Table with columns: Benchmark, Score, Metric
Benchmark	Score	Metric
Average	N/A	`mean`
ARC Easy	N/A	`acc_norm,none`
PIQA	N/A	`acc_norm,none`
ARC Challenge	N/A	`acc_norm,none`
HellaSwag

The model's current generation quality has been checked with basic text-generation prompts, including:

Once upon a time
The little boy
In the forest

Prompt: Once upon a time

Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked its G during work used after several There and at there b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many

Prompt: The little boy

The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many

Prompt: In the forest

In the forest often form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but

Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.

bash
pip install -U transformers safetensors torch

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Note: AppleMind 1.0 Mini has a 256-token context window, so the prompt plus generated tokens should stay within that limit.

License

Apache 2.0