AI.Tokenize¶
| Name | Mandatory | Description | Default | Type |
|---|---|---|---|---|
⬅️ Input |
The input of the shard, if any | String |
||
Output ➡️ |
The resulting output of the shard | [Int] |
||
Model |
No | The model whose tokenizer to use. | none |
Var(LLMModel) |
Tokenize text into token IDs using the model's tokenizer.