Use local models

Goal. Run embeddings or generation without a provider, a key or a network. About ten minutes, most of it downloading.

Released binaries cannot do it yet

The release ships without local execution, because it vendors llama.cpp and would turn a twelve-megabyte download into something much larger. A bare model name means a local model, so on a fresh install:

strata --cache inference embed miniLM "hello"

inference.unsupported_operation: not supported: `miniLM` is a local model, and
this build has no local model execution: this build runs cloud models only, and
a bare model name means a local model. Either name a cloud model instead, or run
`strata inference install-local` to add local execution.

Add it

strata inference install-local

This adds local execution to the installation you already have. There is no build from source and no toolchain to install.

strata inference status

The build line and the local provider row are what change.

Pick a model

strata --cache inference models list

miniLM        embed     bert          f16     unavailable   45 MB
nomic-embed   embed     nomic-bert    q8_0    unavailable   260 MB
bge-m3        embed     xlm-roberta   q8_0    unavailable   1.2 GB
gpt2          generate  gpt2          q8_0    unavailable   178 MB
tinyllama     generate  llama         q4_k_m  unavailable   670 MB
qwen3:8b      generate  qwen3         q4_k_m  unavailable   4.7 GB

unavailable means not downloaded yet. Sizes range from 45 MB to 4.7 GB, so choose deliberately: miniLM for embeddings is 45 MB and enough for most retrieval, while an 8B generator is a different commitment.

strata --cache inference models local

Lists what is actually on disk.

Check before you commit to one

strata --cache inference capability miniLM

availability              not_downloaded
can_embed                 false
provider_feature_enabled  false

On a cloud-only build those read false, which is the truth rather than an advertisement. After install-local they change. embedding_dim is reported for local models, which is the number your collection’s dimension has to match.

Why bother

No key, no network, no bill. Embedding a large corpus locally costs time rather than money, and works offline.

Nothing leaves the machine. For data you cannot send to a provider, this is the only option.

Reproducibility. A local model at a pinned quantisation answers the same way next year. A hosted model behind a moving name does not.

The trade is quality and speed. A 45 MB embedding model is not text-embedding-3-large, and generation on CPU is slow.

Models are cached in memory

strata --cache inference cache-status
strata --cache inference unload

Loaded models stay resident between calls, which matters more than it sounds when the model is several gigabytes. unload with no argument clears everything.

Failure modes

inference.unsupported_operation. The build has no local execution. Run install-local.

A model is not_downloaded. It is fetched on first use, so the first call is slow and needs a network even though later ones do not.

Dimension mismatch. capability <model> reports embedding_dim; the collection has to be created with that number.

Next

Use cloud providers for the hosted path, or Embed and store documents.