Encodes text into the token id sequence a local model would see and returns the ids in order. Set add_special to include the model’s special tokens (such as beginning-of-sequence markers). Tokenization is a local-only operation: it requires a build with the local execution feature and returns inference.unsupported_operation for cloud provider specs.

Parameters

Name	Type	Required	Description
add_special	boolean	no	Whether to add special tokens.
model	string	yes	Model spec.
text	string	yes	Text to tokenize.

Returns

TokenIds.

Errors

• failed_precondition.engine.runtime_closed
• not_found.engine.branch
• inference.unsupported_operation
• inference.missing_model
• inference.model_load_failed
• inference.local_runtime_failed
• inference.registry_corrupt

Invocation

• CLI: strata inference tokenize
• Wire type: inference_tokenize