Skip to content

Durable lane queue

The lane queue accepts work while lanes are saturated and drains it later through the existing governed execution path. It adds persistence, priority, leasing, crash recovery, and bounded retention; it does not replace dispatch admission, lane selection, or provider settlement (src/spendguard/manage/token_operations/lane_queue.py:135, src/spendguard/manage/token_operations/lane_queue.py:1549).

State isolation and migration

Queue state lives in lane_queue.db, not the spend.db money ledger. The default is $SPENDGUARD_HOME/lane_queue.db, overrideable with budget.lane_queue_db_path. This separation keeps large task/result rows, purge scans, snapshots, and queue WAL traffic from contending with or inflating money-of-record storage (src/spendguard/store/configuration/config_store.py:724, src/spendguard/store/configuration/config_store.py:724).

migrate_from_ledger() is the one-time, idempotent migration. With the drain stopped and long-lived writers updated, it copies only non-terminal pending and leased work into the queue database, assigns fresh row IDs, then drops the old queue table from spend.db. Terminal history is intentionally not copied (src/spendguard/manage/token_operations/lane_queue.py:324). The CLI exposes it as spendguard lanes --migrate-queue-db (src/spendguard/platform/lanes.py:745).

Connections use the shared SQLite tuning: WAL mode, bounded busy wait, synchronous=NORMAL, a WAL journal-size limit, and pooled per-thread connections (src/spendguard/store/configuration/config_store.py:595).

Content identity and deduplication

The canonical identity is a SHA-256 digest over the result-determining tuple (intent, task, system, reasoning). Each UTF-8 field is length-prefixed before hashing to prevent delimiter collisions. Model and lane are excluded because they are selected at drain time; the identity means “answer this exact work,” not “run it on this exact executor” (src/spendguard/manage/token_operations/lane_queue.py:281).

Deduplication is layered:

Failed rows are not reusable dedup targets: newly enqueued identical work receives a fresh row. Null fingerprints are treated as distinct at lease time rather than guessed equal.

Leasing, recovery, and settlement

lease() begins an immediate transaction before selection, reclaims expired leases, and atomically changes selected rows to leased with a worker, expiration, and incremented attempt count. Expired work returns to pending while attempts remain; exhausted work becomes failed (src/spendguard/interface/crossllm.py:227). The total ordering favors priority, realtime service class, earliest SLA, then oldest ID.

settle() is state-guarded so only a currently leased or queued_batch row can be settled. Successful work becomes done; transient failures are deferred and retried within the attempt budget; deterministic failures fail fast. Capacity saturation parks work without consuming a failure attempt. Batch-offloaded work remains separately collectible so it is not also retried in realtime and double-billed (src/spendguard/manage/token_operations/lane_queue.py:612).

Bounded housekeeping

Terminal rows are retained for a configurable period, then purge() archives and deletes them in bounded chunks. Each short transaction appends the selected rows before deleting them; a crash can duplicate archive entries but cannot delete an unarchived chunk. Pending and leased work are never purged (src/spendguard/manage/token_operations/lane_queue.py:1149).

Housekeeping is bounded in three dimensions:

Single-instance drain

drain() acquires a non-blocking process lock before entering the loop. If another drainer owns the lock, it returns a skipped summary instead of starting a competing executor; the file descriptor is released in finally, including on deliberate-stop exceptions (src/spendguard/manage/token_operations/lane_queue.py:1549). SQLite leasing remains transactional, but the single drainer also prevents duplicate planning, provider polling, and competing drain loops.

The loop performs preflight dedup, leases bounded batches, runs them through lane_balance.bulk_delegate, and settles each result. Foreground mode stops after configured empty rounds; daemon mode waits for new work. A max_iters stop leaves durable rows for the next run and reports remaining work rather than claiming completion (src/spendguard/manage/token_operations/lane_queue.py:1570).