inference
tokenize
Tokenize text with a local model.
strata inference tokenizedb.ai.tokenize(text: 'str', *, model: 'str', add_special: 'bool' = False) -> 'Any'wire inference_tokenize
Encodes text into the token id sequence a local model would see and returns the ids in order. Set `add_special` to include the model's special tokens (such as beginning-of-sequence markers). Tokenization is a local-only operation: it requires a build with the local execution feature and returns `inference.unsupported_operation` for cloud provider specs.
To see the complete help for this command, run:
strata inference tokenize --helphelp(db.ai.tokenize)
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
| model | string | yes | Model spec. |
| text | string | yes | Text to tokenize. |
| add_special | boolean | no | Whether to add special tokens. |
Returns
TokenIds
- dataarray
- typestring
Errors
Recover by code. Retry policy and commit outcome come from the shipped error registry.
| Code | Retry | Commit outcome |
|---|---|---|
| failed_precondition.engine.runtime_closed | never | not_started |
| not_found.engine.branch | never | not_applicable |
| inference.unsupported_operation | never | not_applicable |
| inference.missing_model | after_state_change | not_applicable |
| inference.model_load_failed | after_state_change | not_applicable |
| inference.local_runtime_failed | unknown | not_applicable |
| inference.registry_corrupt | never | not_applicable |