Docs menu · Inference

Local inference runs a GGUF model in-process, with no network and no API key. Generation, embedding, ranking, and tokenization all work against a local model named by a bare spec (tinyllama, miniLM). This is the path for offline, private, or air-gapped use — the data and the model never leave the machine.

The local build feature

Local execution paths require a binary built with the local inference feature. The default distribution is cloud-first; when the local feature is absent, a local call refuses with a clear code rather than silently doing nothing:

strata --cache inference generate tinyllama "Hello" --max-tokens 5
inference.unsupported_operation: not supported: local generation requires the local feature
  hint: Inspect inference configuration and retry with supported settings.
  ref: https://stratadb.org/e/inference.unsupported_operation

Check provider_feature_enabled in inference capability <spec> before relying on a local model in a script. When the feature is present, the operations documented in Inference run against local models, and two generate refinements become available: --stop-token <id> and --grammar <gbnf>.

GPU acceleration for local models ships separately from the default CPU build (for example, the stratadb[cuda] Python wheel); the base CLI runs local models on CPU.

Pulling models

inference models pull <spec> downloads a model into the local model directory. It reads:

  • STRATA_MODELS_DIR — the destination directory for downloaded models.
  • STRATA_HF_ENDPOINT — the Hugging Face endpoint to fetch from.
  • STRATA_HF_TOKEN (or HF_TOKEN) — a token for gated repositories.

Pulling requires network access. Once pulled, a model appears in inference models local and runs offline thereafter.

strata --cache inference models local
miniLM	embed	bert	f16	local	42.9 MB
tinyllama	generate	llama	q4_k_m	local	638.9 MB

The resident-model cache

Local models stay resident after first use, so repeated calls skip the load cost. inference cache-status reports what is currently loaded:

strata --cache inference cache-status
{
  "embedding_models": [],
  "generation_models": [],
  "ranking_models": []
}

inference unload <spec> evicts one cached model; inference unload with no argument evicts everything. When nothing matches, it reports no cached entry. Unloading frees memory without deleting the model from disk — the next call reloads it.

agents: this page as markdown → /docs/inference/local-models.md