Docs menu · Cookbook

Goal: run two agent strategies side by side and compare them, with each variant’s writes fully isolated from the other and from your baseline.

Prerequisites: the strata binary on your PATH, and jq for the compact fork output. Commands write to a durable directory (./ab) that each invocation reopens.

1. Seed a shared baseline

Anything written before you fork is inherited by every variant.

strata ./ab kv put prompt:system "You are a helpful assistant."
created prompt:system applied=true

2. Fork one branch per variant

A fork is a cheap copy-on-write branch. Both start from the baseline at the same version.

strata ./ab branch fork default variant-a --json | jq -c '{name: .data.name, forked_from: .data.parent.name, at_version: .data.parent.fork_version}'
strata ./ab branch fork default variant-b --json | jq -c '{name: .data.name, forked_from: .data.parent.name, at_version: .data.parent.fork_version}'
{"name":"variant-a","forked_from":"default","at_version":3}
{"name":"variant-b","forked_from":"default","at_version":3}

3. Run each variant on its own branch

Pass --branch to target a variant. Here A runs cooler and produces two answers; B runs hotter and produces three.

strata ./ab kv put config:temperature 0.2 --branch variant-a
strata ./ab event append answer '{"variant":"a","tokens":180}' --branch variant-a
strata ./ab event append answer '{"variant":"a","tokens":210}' --branch variant-a
strata ./ab kv put score 74 --branch variant-a

strata ./ab kv put config:temperature 0.9 --branch variant-b
strata ./ab event append answer '{"variant":"b","tokens":320}' --branch variant-b
strata ./ab event append answer '{"variant":"b","tokens":295}' --branch variant-b
strata ./ab event append answer '{"variant":"b","tokens":410}' --branch variant-b
strata ./ab kv put score 88 --branch variant-b
created config:temperature applied=true
created applied=true
created applied=true
created score applied=true
created config:temperature applied=true
created applied=true
created applied=true
created applied=true
created score applied=true

4. Compare the branches

Read each variant’s score and answer count directly.

strata ./ab --raw kv get score --branch variant-a
strata ./ab event count --branch variant-a
strata ./ab --raw kv get score --branch variant-b
strata ./ab event count --branch variant-b
74
2
88
3

5. Confirm the baseline is untouched

The per-variant config never leaked back to default.

strata ./ab kv exists config:temperature
false

6. Promote the winner

There is no branch-merge command. To fold the winning variant into default, you replay its writes there — read the winner’s values and put them on default. Variant B scored higher (88 vs 74), so promote it.

strata ./ab kv put config:temperature "$(strata ./ab --raw kv get config:temperature --branch variant-b)"
strata ./ab kv put score "$(strata ./ab --raw kv get score --branch variant-b)"
strata ./ab --raw kv get config:temperature
strata ./ab --raw kv get score
created config:temperature applied=true
created score applied=true
0.9
88

Why this works

Each fork is an isolated branch: writes on variant-a and variant-b never see each other, and neither disturbs default. Because a fork shares the parent’s history until it diverges, the baseline you seed in step 1 is visible in both variants for free — no cleanup of half-written state on a shared branch. Promotion is a deliberate replay of the winner’s writes onto default, not an automatic merge, so you decide exactly what graduates. See the branch management guide for the full lifecycle, the KV store guide for value reads, and the event log guide for per-branch action counts.

agents: this page as markdown → /docs/cookbook/ab-testing-with-branches.md