Engines and Deployment Topologies

Choose the serving engine and topology together. The engine determines the request dialect and cache-event transport; the topology determines whether one worker serves the full request or separate workers handle prefill and decode.

For a converged pool using the five standard models, follow the five-model RR / KV-Exact manual. Its evaluated baseline is vLLM 0.31.0, SGLang 0.5.19 and TRT rc29/PyTorch: 26 bounded functional PASS cases and four native architecture blockers. Check its per-model results and pending catalog integration before selecting a tuple.

Before you choose

Collect these facts about the deployment:

1. Does the serving stack already run vLLM, SGLang, TensorRT-LLM, or llama.cpp?
2. Is it a single pool, or are prefill and decode workers already separated?
3. Does the engine expose cache events that the gateway can consume?
4. Can one gateway be the only consumer of a destructive event endpoint?
5. Are model, tokenizer, block size, and engine build consistent across every endpoint in the pool?

flowchart TD
    START([Choose the serving stack]) --> ENGINE{Which engine?}
    ENGINE -->|vLLM| VSHAPE{Separate prefill<br/>and decode?}
    ENGINE -->|SGLang| SSHAPE{Separate prefill<br/>and decode?}
    ENGINE -->|TensorRT-LLM| TSHAPE{Separate context<br/>and generation?}
    ENGINE -->|llama.cpp| LCPP["Fullproxy pool<br/>CHWBL or session affinity"]
    VSHAPE -->|No| VPLAIN["Fullproxy pool<br/>RR, affinity or Exact mode 3"]
    VSHAPE -->|Yes| VPD["Sequential P/D<br/>optional KV-exact mode 1"]
    SSHAPE -->|No| SSINGLE["Single pool<br/>optional KV-exact mode 3"]
    SSHAPE -->|Yes| SPD["Concurrent P/D<br/>optional KV-exact mode 1"]
    TSHAPE -->|No| TSINGLE["Single pool<br/>optional KV-exact mode 3"]
    TSHAPE -->|Yes| TPD["Sequential P/D<br/>optional KV-exact mode 1"]

    style VPD fill:#e8f5e9,stroke:#43a047
    style SPD fill:#e1f5fe,stroke:#0288d1
    style TPD fill:#fff3e0,stroke:#f57c00
    style LCPP fill:#f3e5f5,stroke:#8e24aa

Recommended starting point

Situation	Start with	Why
Existing vLLM fleet	vLLM with the fleet's current topology	It keeps the engine's existing OpenAI and KV-transfer contracts.
SGLang workers with local radix caches	SGLang single pool with CHWBL; add kvExactMode: 3 after hash parity is verified	The simple path works before the cache-event plane is introduced.
SGLang prefill/decode deployment	SGLang P/D	The gateway concurrently dispatches both legs and supplies the bootstrap rendezvous fields.
TensorRT-LLM deployment	Plain fullproxy first; then add KV-exact or P/D	Its event endpoint is destructive, so ownership and block-size admission must be correct first.
GGUF models served by llama-server	llama.cpp with CHWBL (sel: 8)	llama.cpp has no supported gateway KV-event or P/D contract.

Add one capability at a time

First prove health-checked fullproxy routing. Then add content affinity, P/D, or KV-exact routing. This makes failures attributable to one configuration change.

Non-interchangeable settings

• kvEngineType identifies the engine: vllm, sglang, trtllm, or llamacpp.
• kvExactMode identifies the endpoint topology, not the engine: 1 is a P/D pool and 3 is a role-less single pool.
• kvExactMode: 1 requires pd_disagg_mode: true.
• kvExactMode: 3 requires fullproxy and cannot be combined with P/D.
• An omitted kvHashAlgo lets the gateway choose the coherent engine default. This is the safest default.
• One rule represents one engine. To change kvEngineType, delete and recreate the rule.
• cb_enable and origin-5xx demotion improve later endpoint selection; they do not retry or mask the current origin response.

Security baseline

• Keep the management API on a trusted network. The examples use plain HTTP only to make an isolated lab easy to reproduce.
• Use TLS for production client traffic and protect backend traffic according to the deployment's trust boundaries.
• Never put API keys, model repository credentials, or private endpoint addresses in committed configuration examples.
• Treat content affinity as a routing function, not an authorization or tenant-isolation boundary.
• Avoid verbose payload logging when prompts can contain sensitive data.

Next steps

• Review the engine capability matrix.
• For SGLang P/D, follow SGLang P/D Disaggregation.
• For TensorRT-LLM, follow TensorRT-LLM Integration.
• For llama.cpp, follow llama.cpp Integration.

Topology comparison

Topic	vLLM	SGLang
Plain fullproxy pool	Supported	Supported
Single-pool KV-exact	kvExactMode: 3	kvExactMode: 3
P/D	Sequential prefill then decode	Concurrent prefill and decode
P/D KV-exact	kvExactMode: 1, prefill endpoints scored	kvExactMode: 1, prefill endpoints scored
Engine identity	kvEngineType: "vllm" or omitted	kvEngineType: "sglang"
Engine-specific transfer field	Endpoint nixl_port	Service pdBootstrapPort

sequenceDiagram
    participant Client
    participant Gateway
    participant P as Prefill
    participant D as Decode

    rect rgb(232,245,233)
        Note over Client,D: vLLM sequential P/D
        Client->>Gateway: Request
        Gateway->>P: Prefill request
        P-->>Gateway: KV transfer parameters
        Gateway->>D: Decode request + transfer parameters
        D-->>Client: Completion or SSE via Gateway
    end
    rect rgb(225,245,254)
        Note over Client,D: SGLang concurrent P/D
        Client->>Gateway: Request
        par Same bootstrap room
            Gateway->>P: Request + bootstrap triple
        and
            Gateway->>D: Request + bootstrap triple
        end
        D->>P: Join bootstrap service
        D-->>Client: Completion or SSE via Gateway
    end

The important correction is that SGLang is not restricted to a single-role pool. Use either:

• role-less kvExactMode: 3 for converged workers; or
• pd_disagg_mode: true with roles 1/2 for concurrent SGLang P/D, optionally adding kvExactMode: 1.

Which routing shape should you use?

Need	vLLM choice	SGLang choice
Simplest healthy pool	Plain fullproxy	Plain fullproxy
Approximate prompt-family affinity	CHWBL	CHWBL
Exact inventory in one converged pool	Mode 3	Mode 3
Separate prefill and decode capacity	Sequential P/D	Concurrent P/D
Exact prefill inventory inside P/D	Mode 1	Mode 1

Start with the least complex shape that meets the workload goal. P/D adds transfer networking and paired failure handling. KV-exact routing adds tokenizer, block-size, hash, event-port, and inventory-readiness requirements. Verify readiness from subscriber and inventory metrics.