Architecture
A Qwen3-style decoder-only transformer, implemented from scratch in PyTorch. It loads as a stock Qwen3ForCausalLM, so no trust_remote_code is required (needs transformers>=4.51).
Table with columns: Component, Choice| Component | Choice |
|---|
| Normalisation | RMSNorm (pre-norm) |
| Positions | RoPE, theta = 1000000 |
| Attention | Grouped-Query Attention (GQA) |
| QK-Norm | Yes (RMSNorm on Q and K, before RoPE) |
| Feed-forward | SwiGLU |
| Biases | None |
| Embeddings | Tied (input = output) |
Size
Table | |
|---|
| Parameters | 30.7M |
| Layers | 8 |
| Hidden size | 512 |
| Attention heads | 8 (KV heads: 2) |
| Head dim | 64 |
| FFN size | 1408 |
| Context length | 1024 (trained on 512-token windows) |
| Vocab | 16000 — byte-level BPE trained on PHP |
| Weights | (float32) |
The tokenizer was trained on this PHP corpus rather than reused, so it packs PHP efficiently (~4.1 characters per token) and can represent any byte sequence.
Training
Table | |
|---|
| Data | ~24.5M tokens of PHP (29,844 files from 41 open-source projects) |
| Objective | Next-token prediction (causal LM) |
| Steps | 4000 |
| Validation loss | 1.1826 (perplexity ~3.3) |
| Optimiser | AdamW, cosine LR decay with warmup |
| Hardware | Single free-tier GPU |
What it does well
- Continues PHP you have started: class bodies, method signatures, property declarations, docblocks.
- Produces syntactically valid, idiomatic modern PHP — typed signatures (
: string, ?int), constructor property promotion, fluent return $this;, PSR-style formatting.
- Knows conventions of major frameworks it trained on (Symfony, Laravel, Doctrine, PHPUnit style).
- Fast: roughly 0.3 s for a short completion on a CPU — usable as live editor autocomplete.
What it does NOT do
Please read this before using it — it is a small model and these limits are real:
- It is not a chatbot. It does not follow instructions. Prompting it with "write me a function that sorts users" will not work — give it the start of code and it continues.
- It does not understand your program's logic. It writes code that looks right more reliably than code that is right. Example: asked to complete
add(Money $other), it may return 0 instead of summing.
- It repeats itself. It sometimes emits the same method twice. Use a repetition penalty (~1.15) and keep
max_new_tokens modest.
- No fill-in-the-middle (FIM). It only sees code before the cursor. Do not send a
suffix; use prefix-only completion.
- It does not know your codebase, private APIs, or anything outside its training data.
- Not for security-sensitive code. Always review output. It can produce insecure or non-functional code.
- PHP only. Other languages will be poor.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("AnandRago/PHPko-31M")tok = AutoTokenizer.from_pretrained("AnandRago/PHPko-31M") prompt = "<?php\n\nclass Invoice\n{\n private array $lines = [];\n\n public function getTotal(): float\n {\n return "ids = tok(prompt, return_tensors="pt").input_idsout = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.2, top_p=0.9, repetition_penalty=1.15)print(tok.decode(out[0], skip_special_tokens=True))
Recommended settings: temperature 0.2-0.5, top_p 0.9, repetition_penalty 1.15, max_new_tokens 48-96. Very low temperature (<0.1) makes it loop.
Training data & provenance
Trained only on permissively licensed (MIT / BSD-3) open-source PHP. GPL projects (e.g. WordPress, Drupal) were deliberately excluded so the corpus carries no copyleft obligations.
Files were filtered to remove vendor directories, tests, generated code, minified files, translation/lookup tables, and exact duplicates. License header comments were stripped.
Sources (each retains its original license)
Corpus sources. Each retains its original license.Carbon MIT https://github.com/briannesbitt/Carbon.gitFastRoute BSD-3 https://github.com/nikic/FastRoute.gitPHP-Parser BSD-3 https://github.com/nikic/PHP-Parser.gitSlim MIT https://github.com/slimphp/Slim.gitSylius MIT https://github.com/Sylius/Sylius.gitTwig BSD-3 https://github.com/twigphp/Twig.gitbagisto MIT https://github.com/bagisto/bagisto.gitcakephp MIT https://github.com/cakephp/cakephp.gitcollections MIT https://github.com/doctrine/collections.gitcollision MIT https://github.com/nunomaduro/collision.gitcommonmark BSD-3 https://github.com/thephpleague/commonmark.gitcomposer MIT https://github.com/composer/composer.gitcore MIT https://github.com/api-platform/core.gitcsv MIT https://github.com/thephpleague/csv.gitdbal MIT https://github.com/doctrine/dbal.gitfilament MIT https://github.com/filamentphp/filament.gitflysystem MIT https://github.com/thephpleague/flysystem.gitframework MIT https://github.com/laravel/framework.gitguzzle MIT https://github.com/guzzle/guzzle.githorizon MIT https://github.com/laravel/horizon.gitlaminas-mvc BSD-3 https://github.com/laminas/laminas-mvc.gitlaravel MIT https://github.com/laravel/laravel.gitlaravel-medialibrary MIT https://github.com/spatie/laravel-medialibrary.gitlaravel-permission MIT https://github.com/spatie/laravel-permission.gitlivewire MIT https://github.com/livewire/livewire.gitlog MIT https://github.com/php-fig/log.gitmigrations MIT https://github.com/doctrine/migrations.gitmonolog MIT https://github.com/Seldaek/monolog.gitoauth2-server MIT https://github.com/thephpleague/oauth2-server.gitorm MIT https://github.com/doctrine/orm.gitphpdotenv BSD-3 https://github.com/vlucas/phpdotenv.gitphpstan-src MIT https://github.com/phpstan/phpstan-src.gitphpunit BSD-3 https://github.com/sebastianbergmann/phpunit.gitpromises MIT https://github.com/guzzle/promises.gitpsr7 MIT https://github.com/guzzle/psr7.gitrector-src MIT https://github.com/rectorphp/rector-src.gitsymfony MIT https://github.com/symfony/symfony.gitsymplify MIT https://github.com/symplify/symplify.gittelescope MIT https://github.com/laravel/telescope.gituuid MIT https://github.com/ramsey/uuid.gityii2 BSD-3 https://github.com/yiisoft/yii2.git
Files kept per source repository
Table with columns: Repo, Files| Repo | Files |
|---|
| symfony | 7,498 |
| Sylius | 3,600 |
| rector-src | 3,077 |
| filament | 2,458 |
| phpstan-src | 1,884 |
| framework | 1,619 |
| bagisto | 1,433 |
| core | 1,357 |
| phpunit | 1,131 |
Total: 30,044 files, ~25.6M BPE tokens.
License
Apache-2.0 for the model weights. Each training source retains its own license (listed above); please respect them.