Routing Hierarchy¶

How LoxiLB decides which worker serves each request: a strict, fail-through priority ladder that prefers the strongest cache-affinity signal available, then bounds it by load, capacity, health, and admission.

Why a hierarchy?¶

vLLM keeps a prefix cache — the KV tensors for token prefixes it has already computed. If a new request's prompt shares a prefix with blocks a particular worker already holds, routing the request to that worker skips prefill recomputation for the shared span, dramatically lowering time-to-first-token (TTFT) and freeing prefill capacity.

Two naive strategies both fail:

Plain round-robin scatters same-prefix requests across the fleet, so every worker recomputes the same preamble.
Pure cache affinity herds every hot-prefix request onto one worker while its siblings idle.

The routing problem is therefore hierarchical: prefer the strongest affinity signal for the request, but bound it by load and capacity, and fall through gracefully when any signal is missing. LoxiLB evaluates this as a strict priority ladder per request inside the fullproxy (mode:4) data plane. Selection tiers fail through to the next tier, but admission and terminal availability gates can return 429 or 503; the ladder does not guarantee delivery.

Terminology: "tiers" and "layers"¶

The selection stages are called tiers (Tier 0/1/1.5/2) in the routing engine; operators often say Layer 1 / 1.5 / 2 / 3 for the same thing. This page uses tier numbering.

Operator shorthand	Tier	What it is
Layer 0	Tier 0	Conversation / session stickiness
Layer 1	Tier 1	Radix-trie prefix affinity (heuristic, P/D) / prefix-hash CHWBL (single-pool)
Layer 1.5	Tier 1.5	KV-exact block-hash routing (mirrors vLLM's real cache state)
Layer 2	Tier 2	Min-load fallback with round-robin tie-break
"Layer 3"	— (control plane)	Not a data-path tier: the adaptive selection law and the optional external controller, which bias Tiers 1.5/2 rather than select directly

The two deployment shapes¶

The hierarchy behaves differently depending on whether the service is P/D-disaggregated (prefill and decode split across separate endpoint roles):

Shape	Rule shape	Selection hierarchy
P/D disaggregation	`mode:4` + `pd_disagg_mode:true`, endpoints tagged `ep_role:1` (prefill) / `ep_role:2` (decode), at least one of each	The full P/D tier ladder: admission → Tier 0 → Tier 1 → Tier 1.5 → Tier 2, then decode selection
Single pool (non-disaggregated)	`mode:4`, one role-less endpoint pool (no `pd_disagg_mode`)	With `kvExactMode:3`, KV-exact scores all endpoints and a miss falls to the rule selector. Without mode 3, use the selector directly: CHWBL (`sel:8`), weighted CHWBL (`sel:10`), selector-9 modulo affinity, persist, WRR, or RR.

Mode 1 and mode 3 are different selection graphs

kvExactMode:1 is valid only inside a P/D ladder and scores eligible prefill endpoints. kvExactMode:3 is valid only on a role-less non-P/D rule and scores every endpoint before falling back to that rule's selector. CHWBL (sel:8/10) remains an inventory-free approximation of cache locality and can be used without either KV-exact mode.

The P/D tier ladder — one request, top to bottom¶

For a P/D rule, one request descends the ladder until a stage selects a prefill endpoint, then a decode endpoint is chosen separately.

Stage	Purpose	Falls through when
excluded_mask seeding	Health / circuit-breaker pre-filter: every down or CB-open endpoint is masked out of every tier below (so an excluded Tier-1.5 winner falls to the 2nd-best prefill, never straight to RR)	— (always runs)
Controller fold-in	Fold controller-DISABLED endpoints into the mask; collect the DRAINING set	No-op when no controller is attached
Admission gate (default-off)	Per-endpoint in-flight caps; when all are capped, either park (hold-don't-drop) or shed a retriable `429`	Caps unset (default) → byte-identical skip
Tier 0 — session stickiness	Pin the `(prefill, decode)` pair for a conversation	No key; pinned endpoint unhealthy / masked / CB-open / TTL-evicted
Tier 1 — radix-trie prefix affinity	Heuristic: LoxiLB's own trie of observed prefixes (not vLLM state)	`pd_cache_aware_mode` off; empty prefix; imbalance guard; `match_rate < pd_cache_threshold`
Tier 1.5 — KV-exact block-hash	Route to the prefill endpoint whose actual vLLM cache best overlaps the prompt, bounded by load	`kvExactMode` off; any guard miss (hash-contract, warmup, tokenize, empty inventory)
Tier 2 — min-load fallback	`score = active_conns + queued_requests`, lower wins; RR only on a genuine tie	Terminal (only returns "none" if no healthy candidate)
Decode selection	Session-pinned decode hint if valid, else min-load + RR among `ep_role:2` endpoints	—
Any-healthy rescue	Non-P/D rescue using role-0 endpoints; else `503`	None healthy → `503`

There is no data-path 'Tier 3'

Below Tier 2 is only the any-healthy rescue. What operators call "layer 3" is the control loop that biases the ladder (via the adaptive law and the optional controller), not a selection tier.

LLB_KV_MIN_MATCH_TOKENS adds a Tier-1.5 guard: default 16, accepted range 0–4096, and 0 disables the minimum-token check.

Tier 0 — session stickiness¶

The session key is the request's user_id JSON field if present, overridden by a client-supplied X-Conversation-Id header (LoxiLB's own auto- prefixed IDs are deliberately not used as stickiness keys). A hit pins the full (prefill, decode) pair for pd_session_ttl_sec. The table is TTL-evicted and LRU-capped; a pinned endpoint that is unhealthy, masked, or CB-open causes the key to be evicted and the ladder to continue — stickiness never overrides health. A multi-turn conversation's growing KV state lives on the workers that served the previous turns, so keeping the pair stable is the strongest cache-affinity signal available, and it costs nothing to evaluate.

Tier 1 — radix-trie prefix affinity (heuristic)¶

Enabled by pd_cache_aware_mode:true. LoxiLB maintains its own radix trie of prompt prefixes it has routed before; the trie leaf remembers which endpoint last served that prefix. Two REST-tunable guards keep the heuristic honest:

Match-rate threshold — the matched span must cover at least pd_cache_threshold percent (default 20) of the prompt, or the affinity is judged too weak.
Load-imbalance guard — if max(active_conns) − min(active_conns) across prefill endpoints exceeds pd_balance_abs_threshold (default 3), affinity is bypassed so a hot endpoint is not made hotter.

The trie tracks what LoxiLB routed, not what vLLM actually holds. It is cheap (no tokenization, no hashing) and needs no vLLM-side configuration, but it can go stale when vLLM evicts. Tier 1.5 tracks the truth; Tier 1 is a useful heuristic when KV-exact is not deployable.

Tier 1.5 — KV-exact block-hash routing¶

The centerpiece. LoxiLB mirrors each prefill endpoint's actual prefix-cache content (a set of 64-bit block hashes streamed over vLLM's ZMQ KV-events channel), recomputes the same block hashes vLLM would compute for the prompt (tokenize → per-block canonical CBOR → hash → truncate), scores every prefill endpoint by overlap count, and routes through the unified blend law below. The mechanics — inventory plane, hash contract, and guard ladder — are documented in KV-Cache Routing.

Tier 2 — min-load with RR tie-break¶

Despite the historical name, Tier 2 is a min-load scorer:

Default arm: score = active_conns + queued_requests, lower wins.
Capacity-blend arm: the scorer exists for GPU-aware (sel:9) rules, but its current activation gate checks a mutable endpoint cursor instead of the configured selector. Treat it as release-blocked; normal Tier 2 uses the default arm.

The round-robin counter advances only on a genuine tie, so it is a tie-breaker, not the algorithm.

Decode selection¶

After the prefill endpoint is chosen, decode picks: (1) the Tier-0 session-pinned decode hint if valid; else (2) min-load among decode endpoints with its own RR tie-break. Decode endpoints are never KV-selection candidates — they publish no KV events and hold no scored inventory.

The admission gate (default-off)¶

Before any tier body runs, the admission gate can exclude prefill endpoints at their in-flight cap and, when all are capped, either park the request (hold-don't-drop: enqueue on the shortest per-endpoint FIFO, suspend the client, resume when capacity frees) or shed it with a retriable 429. A separate global valve refuses new connections beyond a total-in-flight bound. All four knobs default to off. Treat admission as opt-in protection for latency-SLO fleets, not a throughput optimizer.

The unified selection law (Tier 1.5's blend modes)¶

Raw overlap-argmax is load-blind: 50 clients sharing one hot preamble would all route to the same prefill endpoint. The unified mode bounds cache affinity by a capacity-weighted load cap, in the spirit of CHWBL (consistent hashing with bounded loads). The mode is selected by the LOXILB_KV_LB_MODE environment variable.

The candidate set¶

Each prefill endpoint i contributes a candidate carrying:

overlap_i — matched block-hash count against the prompt's hash chain;
load_i — LoxiLB's own per-endpoint active_conns (not vLLM's view);
capacity_i — the endpoint's advertised KV-block capacity, optionally scaled by the controller weight.

`hard` mode (default) — capacity-weighted bounded load¶

A per-endpoint cap:

cap_i = ceil( (1 + ε) · totalLoad · capacity_i / totalCap )

with ε expressed as mean_load_factor_pct = (1 + ε)·100, default 175 ⇒ ε = 0.75, and totalLoad = Σ load_i, totalCap = Σ capacity_i over the candidate set. Selection:

Argmax overlap among under-cap endpoints (load_i < cap_i); ties → least load → lowest index.
If the global overlap winner was over its cap and selection moved off it, that is a spill.
Negligible-overlap refinement: if the best under-cap overlap is ≤ 0, pick the least-loaded under-cap endpoint (no affinity worth honoring).
Saturated case (all over cap): least-loaded among the positive-overlap candidates.

Intuition: an endpoint may hold its cache-affinity traffic while it carries at most (1+ε)× its capacity-fair share of the current total load; beyond that, the excess spills to siblings. Higher ε ⇒ more affinity-preserving; lower ε ⇒ more aggressive spilling.

`soft` mode — continuous cost blend¶

No hard cutoff; argmin of a cost that prices both the cache miss and the queue:

cost_i = uncached_blocks_i · 1000  +  (λ · load_i) / capacity_i
uncached_blocks_i = promptBlocks − overlap_i

λ defaults to 32; 1000 is the fixed cost scale. At zero load, soft mode reduces exactly to overlap-argmax.

`adaptive` / `adaptive-soft` — the load-keyed ε/λ law¶

No static ε/λ is optimal across load: the best values increase with load (tight ε wins at moderate rate, loose ε at saturation). Adaptive mode scales the knob with the load the selector itself observes:

L        = Σ active_conns over the candidate set
ε_eff(L) = clamp( 175 + 125·(L − 16)/10 ,  175, 300 )      // adaptive (hard-arm)
λ_eff(L) = clamp( 50000 + 5000·(L − 16) , 50000, 100000 )  // adaptive-soft

Below the floor anchor (L ≤ 16, the calibrated operating point) the behavior is identical to static hard/soft; the law only loosens the bound where loosening wins, capping at the saturation anchor (L ≥ 26). adaptive runs the hard selector with ε_eff; adaptive-soft runs soft with λ_eff — nothing else differs.

Mode resolution¶

A valid LOXILB_KV_LB_MODE (off | hard | soft | adaptive | adaptive-soft) wins outright; garbage warns and falls back to hard. The out-of-box default is hard with ε = 0.75; off restores pure overlap-argmax.

Full-fleet pressure relief and cold-start seeding¶

The primary selector considers positive-overlap candidates. Two safeguards handle endpoints outside that set:

Pressure relief. LOXILB_KV_SPILL_RELIEF lets an over-cap affinity winner spill to the least-loaded under-cap endpoint across the full healthy fleet, including zero-overlap endpoints. Unset means on for single-pool mode 3 and off for P/D mode 1. Explicit on/off values override every service in the process.
Cold-start seeding. While an eligible endpoint has fewer than LOXILB_KV_COLDSTART_MIN_BLOCKS blocks (default 16), every LOXILB_KV_COLDSTART_SEED_Nth Tier-1.5 hit (default 16) is diverted to the lowest-index cold endpoint. The endpoint warms from that request, then leaves the cold set. Set the seed interval to 0 to disable this recovery path.

flowchart LR
    AFF[Affinity winner] --> CAP{Over fleet-wide cap?}
    CAP -->|Yes and relief enabled| RELIEF[Least-loaded under-cap endpoint]
    CAP -->|No| TICK{Cold endpoint and Nth hit?}
    RELIEF --> TICK
    TICK -->|Yes| SEED[Seed cold endpoint]
    TICK -->|No| FINAL[Keep current selection]

    style RELIEF fill:#fff3e0,stroke:#f57c00
    style SEED fill:#e1f5fe,stroke:#0288d1
    style FINAL fill:#e8f5e9,stroke:#43a047

Use loxilb_pd_kv_tier15_spills_total and loxilb_pd_kv_tier15_cold_seeds_total to observe these deliberate cache-locality tradeoffs.

Resilience semantics¶

Health / CB pre-filter: the excluded_mask guarantees an excluded Tier-1.5 winner falls to the 2nd-best-overlap prefill endpoint, never to a decode endpoint and never straight to RR.
Circuit breaker: per-endpoint CLOSED → OPEN → HALF_OPEN. For P/D services this auto-enables with a threshold of 3 consecutive failures and a 30 s open window. CB state is local-only, never HA-synced.
Origin errors: when the breaker is enabled, three consecutive origin 5xx responses also open it by default. LLB_PD_ORIGIN_ERR_THRESHOLD=0 disables this demotion; a 4xx neither advances nor resets the origin-error streak. The current request still receives the origin response—demotion affects later selection and is not an automatic retry.
Probe state is synchronized: a probe-down transition is pushed immediately into the fullproxy endpoint state and seeds the exclusion mask. Connect-failure retry and the circuit breaker cover failures that occur before the probe transition.
Inventory continuity has a tradeoff: a near reconnect and a small gap preserve inventory; a gap beyond the 64-event window clears it. The live loop does not replay missed events, so a missed remove can create temporary stale affinity before later reconciliation.
Controller staleness glides to neutral: if the optional external controller goes stale, per-endpoint weights decay toward neutral — capacity scaling relaxes back to unweighted; endpoints are never zero-filled or dropped by staleness alone.

The single-pool (non-disaggregated) hierarchy¶

For a plain fullproxy pool (no pd_disagg_mode), the request path selects via the rule's selector algorithm before the P/D gate. The cache-aware members of that family:

REST `sel`	Selector	Mechanism
`8`	chwbl	Consistent Hash with Bounded Loads over a hash ring, keyed by prefix_hash
`9`	gpuaware	`prefix_hash % n_eps` placement on the single-pool path; the separate P/D capacity scorer is currently release-blocked
`10`	wrr-hash	CHWBL with endpoint weights folded into the ring (capacity-weighted bounded load)

The routing-key priority is identical for all three:

prefix_hash — a hash over the request-derived LLM prefix and available model/context fields. Although the API stores CHWBL level and flag fields, the current proxy receives fixed flags 0 and derives the prefix scope from request content.
conv_id — fallback session stickiness (hash of the conversation ID).
RR — last resort.

Same-prefix requests hash to the same ring point and land on the same endpoint while it is under its bounded-load cap. The current proxy uses a fixed factor of 175 (1.75× mean) and 256 virtual nodes; submitted CHWBL factor and replication fields do not currently change those values.

What single-pool does not give you

prefix_hash is one hash of one extracted prefix — it cannot measure partial overlap, cannot see vLLM's evictions, and matches only byte-identical prefixes. In exchange it needs no vLLM-side config at all (no ZMQ events, no tokenizer staging, no hash-contract parity).

Configuration¶

Both shapes are created with POST /netlox/v1/config/loadbalancer on port 11111.

Protect the management API

The curl examples use plain HTTP for an isolated lab. In production, use an authenticated, TLS-protected management endpoint and read its authorization header from a permission-restricted file.

curlloxicmd

P/D-disaggregated rule with Tier 1.5 KV-exact:

curl -s -X POST http://10.10.10.254:11111/netlox/v1/config/loadbalancer \
  -H "Content-Type: application/json" \
  -d '{
  "serviceArguments": {
    "externalIP": "10.10.10.254",
    "port": 2020,
    "protocol": "tcp",
    "sel": 0,
    "mode": 4,
    "host": "10.10.10.254",
    "pd_disagg_mode": true,
    "pd_cache_aware_mode": true,
    "kvExactMode": 1,
    "kvZmqPort": 5557,
    "kvHashAlgo": "sha256_cbor",
    "kvBlockSize": 16
  },
  "endpoints": [
    { "endpointIP": "192.0.2.1", "targetPort": 8100, "weight": 1, "ep_role": 1, "nixl_port": 5600 },
    { "endpointIP": "198.51.100.1", "targetPort": 8200, "weight": 1, "ep_role": 2, "nixl_port": 5600 }
  ]
}'

Single-pool CHWBL rule (sel:8) — cache affinity without P/D:

curl -s -X POST http://10.10.10.254:11111/netlox/v1/config/loadbalancer \
  -H "Content-Type: application/json" \
  -d '{
  "serviceArguments": {
    "externalIP": "10.10.10.254",
    "port": 2021,
    "protocol": "tcp",
    "sel": 8,
    "mode": 4,
    "host": "10.10.10.254"
  },
  "endpoints": [
    { "endpointIP": "192.0.2.1", "targetPort": 8000, "weight": 1 },
    { "endpointIP": "198.51.100.1", "targetPort": 8000, "weight": 1 }
  ]
}'