ML.Tokenizer¶
| Name | Mandatory | Description | Default | Type |
|---|---|---|---|---|
⬅️ Input |
The input of the shard, if any | String |
||
Output ➡️ |
The resulting output of the shard | Tokenizer |
||
Quantized |
No | Whether to use quantized tokenizer. | false |
Bool |
Loads a tokenizer from an input json string, ready to be used for tokenizing text.