SGLang Routing

Illustrative mutation fragments

The management mutations on this page are non-standalone fragments. Before adapting them, follow this site's Example Quality Contract and complete Quickstart lifecycle. Do not treat HTTP success or management readback alone as runtime proof.

Route SGLang as either a role-less single pool or a prefill/decode deployment. Both shapes use kvEngineType: "sglang", but they use different request flows and different kvExactMode values.

Choose the SGLang shape

Shape	Gateway rule	Request flow	KV-exact option
Single pool	Fullproxy, role-less endpoints	One selected worker handles the full request	kvExactMode: 3
P/D	Fullproxy, pd_disagg_mode: true, endpoint roles	Gateway dispatches prefill and decode concurrently	kvExactMode: 1

flowchart TD
    START([SGLang deployment]) --> SPLIT{Separate prefill<br/>and decode workers?}
    SPLIT -->|No| SINGLE["Role-less fullproxy pool"]
    SINGLE --> CACHE1{Need exact cache inventory?}
    CACHE1 -->|No| CHWBL["Use CHWBL or another selector"]
    CACHE1 -->|Yes| MODE3["kvExactMode: 3<br/>score every endpoint"]
    SPLIT -->|Yes| PD["P/D pool<br/>pd_disagg_mode: true"]
    PD --> CACHE2{Need exact prefill inventory?}
    CACHE2 -->|No| BASEPD["Concurrent base P/D"]
    CACHE2 -->|Yes| MODE1["kvExactMode: 1<br/>score prefill endpoints"]

    style MODE3 fill:#e8f5e9,stroke:#43a047
    style MODE1 fill:#e1f5fe,stroke:#0288d1

Modes 1 and 3 are mutually exclusive

Base SGLang P/D does not require kvExactMode. Add mode 1 only for P/D-coupled KV-exact routing. Mode 3 is rejected when P/D is enabled.

Single-pool request path

With kvExactMode: 3, the Gateway subscribes every endpoint and scores the request against each endpoint's published block inventory. A cache miss uses the rule's configured selector; it does not enter the P/D session, trie, admission, or min-load ladder.

flowchart LR
    REQ([Request]) --> HASH["Tokenize and hash full pages"]
    HASH --> INV{Matching blocks<br/>in inventory?}
    INV -->|Yes| WARM["Select warm healthy endpoint"]
    INV -->|No| FALLBACK["Use rule selector<br/>for example CHWBL"]
    WARM --> SERVE["One SGLang worker serves request"]
    FALLBACK --> SERVE

    style WARM fill:#e8f5e9,stroke:#43a047
    style FALLBACK fill:#fff3e0,stroke:#f57c00

SGLang KV parity contract

All of these values must agree:

1. The served model and Gateway tokenizer must produce the same token IDs.
2. kvBlockSize must equal SGLang's effective page size.
3. kvEngineType must be sglang.
4. kvHashAlgo should be omitted so the Gateway selects sha256_sglang.
5. kvDpRankCount must equal the publisher rank count.
6. Rank N must be reachable on kvZmqPort + N.

SGLang hashes raw token words with parent chaining and publishes one KV-event stream per data-parallel rank. The Gateway combines those rank events into one endpoint inventory because it routes to an endpoint, not to an internal rank.

Configure a single pool

The example uses documentation-only addresses. Confirm the deployed page size before using kvBlockSize: 16.

The example inference listener is plaintext

security: 0 is appropriate only for an isolated lab or a separately protected network. Use security: 1 for frontend TLS termination or security: 2 for frontend termination plus backend TLS, and verify certificates as described in mTLS.

Prepare a protected management header and an HTTPS API base URL as described in Monitoring and Metrics:

export CONTROL_API="https://gateway.example.com/netlox/v1"
install -m 600 /dev/null ./control-plane.headers
printf 'Authorization: Bearer %s\n' "$CONTROL_PLANE_TOKEN" > ./control-plane.headers

curlloxicmd

curl --fail-with-body --silent --show-error \
  --request POST "$CONTROL_API/config/loadbalancer" \
  --header @control-plane.headers \
  --header 'Content-Type: application/json' \
  --data-binary '{
    "serviceArguments": {
      "externalIP": "192.0.2.10",
      "port": 2090,
      "protocol": "tcp",
      "mode": 4,
      "sel": 8,
      "security": 0,
      "kvExactMode": 3,
      "kvEngineType": "sglang",
      "kvDpRankCount": 1,
      "kvZmqPort": 5557,
      "kvBlockSize": 16,
      "kvWarmupSec": 30,
      "monitor": true,
      "probetype": "http",
      "probeport": 30000,
      "probereq": "/health"
    },
    "endpoints": [
      {"endpointIP": "198.51.100.11", "targetPort": 30000, "weight": 1},
      {"endpointIP": "198.51.100.12", "targetPort": 30000, "weight": 1}
    ]
  }'

loxicmd create lb 192.0.2.10 \
  --tcp=2090:30000 \
  --endpoints=198.51.100.11:1,198.51.100.12:1 \
  --mode=fullproxy \
  --select=chwbl \
  --kv-exact-mode=3 \
  --kv-engine-type=sglang \
  --kv-dp-ranks=1 \
  --kv-zmq-port=5557 \
  --kv-block-size=16 \
  --kv-warmup=30

--kv-warmup maps to an accepted rule field, but the current production path does not arm the warmup timer. Confirm subscriber connectivity and inventory growth before measuring cache behavior.

For P/D, use the separate SGLang P/D Disaggregation guide. Do not add endpoint roles or pd_disagg_mode to the single-pool rule above.

Verify

1. Confirm the rule contains kvEngineType: "sglang", kvExactMode: 3, and no endpoint roles.
2. Confirm all endpoint health checks pass.
3. Warm the same prompt prefix through the VIP.
4. Inspect these metric families:

Before the first scrape, enable metrics with an authenticated POST /netlox/v1/config/metrics; see Monitoring and Metrics.

curl --fail-with-body --silent --show-error "$CONTROL_API/metrics" \
  | grep -E 'loxilb_kv_subscriber_connected|loxilb_pd_kv_blocks|loxilb_pd_kv_tier15_hits_total|loxilb_pd_kv_zero_hit_watchdog_total'

Expected signals:

• one connected-subscriber series per service endpoint, plus independent confirmation that every configured rank port is publishing;
• nonzero inventory after cache events arrive;
• increasing Tier-1.5 hits under repeat-prefix traffic;
• no continuing zero-hit watchdog growth.

Failure diagnosis

Symptom	Likely cause	Fix
Rule rejects mode 3	P/D is enabled or fullproxy is missing	Remove P/D fields and set mode: 4.
Rank subscribers fail	Port range is blocked or exceeds 65535	Open the complete consecutive range or move the base port.
Inventory is empty	SGLang event publishing is disabled or ports do not match	Verify publisher configuration and reachability.
Inventory grows but hits remain zero	Page size, tokenizer, or hash contract differs	Match parity settings and omit kvHashAlgo.
Hot prefix overloads one endpoint	Affinity dominates while concurrency rises	Use CHWBL as fallback and review bounded-load settings.
Engine change is rejected	kvEngineType is immutable	Delete and recreate the rule.

Cleanup

curl -sS -X DELETE \
  --header @control-plane.headers \
  "$CONTROL_API/config/loadbalancer/externalipaddress/192.0.2.10/port/2090/protocol/tcp"

Remove control-plane.headers after the workflow and unset CONTROL_PLANE_TOKEN.

Security and evidence limits

• Expose ZMQ publisher ports only to the Gateway. They are operational event feeds, not public APIs.
• Keep tokenizer assets and model identity controlled; mixing models behind one rule invalidates cache routing.
• Avoid verbose payload diagnostics when requests can contain sensitive data.
• Configuration acceptance and metrics do not prove a performance improvement. Validate cache hits and latency with representative prompts and the production engine build.

See also

• SGLang Configuration and Tuning
• SGLang P/D Disaggregation
• vLLM and SGLang Routing
• Engine Capability Matrix