AI.Model¶
| Name | Mandatory | Description | Default | Type |
|---|---|---|---|---|
⬅️ Input |
The input of the shard, if any | String |
||
Output ➡️ |
The resulting output of the shard | LLMModel |
||
ISQ |
No | In-situ quantization bit width. Quantizes at load time. | ISQBits::None |
ISQBits |
Files |
No | GGUF filename(s) within the repo. When set, uses GGUF loader. | none |
StringNone |
UQFF |
No | UQFF filename (e.g. 'q4k-0.uqff'). When set, loads pre-quantized UQFF model — no ISQ needed. | none |
StringNone |
Embedding |
No | When true, loads as an embedding model for use with AI.Embed. | false |
Bool |
Load a model via mistral.rs. Accepts a HuggingFace model ID or local path. Auto-detects architecture. For GGUF models, set the Files parameter. For embedding models, set Embedding: true.