POST
/
v1
/
tokenize
Tokenization
curl --request POST \
  --url http://localhost:8000/v1/tokenize \
  --header 'Content-Type: application/json' \
  --data '{
  "prompt": "What is generative AI?"
}'
{
  "tokens": [
    128000,
    3923,
    374,
    1803,
    1413,
    15592,
    30
  ]
}
By giving a text input, generate a tokenized output of token IDs.

Body

application/json

Response

Successfully tokenized the text.

The response is of type object.