Skip to main content
POST
/
v1
/
tokenize
Tokenization
curl --request POST \
  --url http://localhost:8000/v1/tokenize \
  --header 'Content-Type: application/json' \
  --data '
{
  "prompt": "What is generative AI?"
}
'
{
  "tokens": [
    128000,
    3923,
    374,
    1803,
    1413,
    15592,
    30
  ]
}
By giving a text input, generate a tokenized output of token IDs.

Body

application/json
​
prompt
string
required

Input text prompt to tokenize.

Example:

"What is generative AI?"

​
model
string | null

Routes the request to a specific adapter.

Example:

"(adapter-route)"

Response

Successfully tokenized the text.

​
tokens
integer[]
required

A list of token IDs.

Last modified on June 9, 2026