LLM Integration: Reading and Deployment Path¶

Start with the concepts, choose a topology, then follow one complete deployment guide. Reference pages explain contracts and fields; they are not alternative installation recipes. Public examples require adaptation to the allocated hosts, model access and exact Gateway/engine artifacts.

Order	Read	Purpose
1	Engines and deployment topologies	Choose converged or P/D and understand engine-specific transfer/event paths
2	RR, content affinity and KV-Exact, then Routing Hierarchy	Understand selection, fallback and load constraints
3	Converged RR/Exact guide or P/D deployment	Execute an ordered guide with checks and rollback
4	Models, profiles, engine contracts and arguments	Look up identity, prerequisites and field meanings
5	Failure/recovery tuning and performance measurement	Qualify workload-specific operating behavior
6	Functional results, limits and open qualification	Distinguish recorded success from missing, failed or blocked gates

Before you start¶

Start here to choose a deployment path before launching a model.

Focus of this page: Choose converged RR/Exact or a separate P/D topology; then follow one versioned runbook.

Topology and execution locations¶

flowchart TD
    C["Client workload"] --> G["Dedicated Gateway<br/>selected VIP and PORT"]
    G --> A["Worker 1<br/>WORKER1:BACKEND_PORT"]
    G --> B["Worker 2<br/>WORKER2:BACKEND_PORT"]
    G --> D["Worker 3<br/>WORKER3:BACKEND_PORT"]
    O["Operator<br/>model and image selection"] -.->|"configuration and readback"| G
    O -.->|"direct inference and engine receipts"| A
    style G fill:#e1f5fe,stroke:#0288d1,color:#1f2937
    style A fill:#e8f5e9,stroke:#43a047,color:#1f2937
    style B fill:#e8f5e9,stroke:#43a047,color:#1f2937
    style D fill:#e8f5e9,stroke:#43a047,color:#1f2937

This diagram shows the converged three-worker guide. Choose one engine and one model per run: workers 1/2/3 are replicas of that choice, not vLLM, SGLang and TensorRT-LLM mixed together. Select actual host addresses in the model-selection step; documentation addresses are examples. Each converged worker performs both prefill and decode.

Alternative: separate Prefill and Decode workers¶

Choose this path only through the selected engine's P/D runbook. The HTTP request legs and the engine's KV transfer are separate paths; the converged result table does not qualify this topology.

flowchart TD
    C["Client workload"] --> G["Gateway<br/>one inference entrypoint"]
    G -->|"prompt / prefill request"| P["Prefill worker"]
    G -->|"decode request"| D["Decode worker"]
    P -.->|"engine-specific KV transfer<br/>separate from HTTP routing"| D
    O["Operator"] -.->|"direct worker checks and transfer evidence"| P
    O -.->|"decoded output and receipts"| D
    style G fill:#e1f5fe,stroke:#0288d1,color:#1f2937
    style P fill:#e8f5e9,stroke:#43a047,color:#1f2937
    style D fill:#e8f5e9,stroke:#43a047,color:#1f2937

Where	What happens here
Operator terminal	Select the version tuple, private run directory, model snapshot and profiles.
Gateway host / namespace	Start the dedicated Gateway; apply and read back only the selected service.
Each worker host	Launch the chosen engine and prove direct inference before Gateway traffic.
Client	Send the same request through the Gateway and retain its HTTP/SSE observations.
Independent evidence	Correlate worker requests/receipts with the selected rule and request identifiers.

Reading path and reference¶

Current five-model baseline¶

vLLM 0.31.0, SGLang 0.5.19 and TensorRT-LLM 1.3.0rc29/PyTorch have 30 recorded RR/Exact cases: 26 bounded functional PASS and four native architecture BLOCKED. Check the single result table before choosing a tuple. TRT Gemma2/EXAONE cannot serve on this pinned PyTorch backend in either mode. Gemma's historical load-completion failures remain open.

These checks used a source-built, isolated proxy-only Gateway. They do not establish released-image support, capacity, P/D, host eBPF/HA or full fail-over. The five evaluation profiles remain separate from the reviewed source catalog; installer integration is pending.

Compatibility and history¶

Use SGLang wire compatibility for tagged-array 0.5.19 and the separate tagged-map development request. Advanced cache integrations and the R1 1.5B evidence archive have separate version and evidence boundaries. Historical files are retained for traceability, not substituted for the current guide's launch settings.

Existing bookmarks¶

Older model-card, SGLang routing/tuning, engine-comparison and combined-manual URLs remain as short links to the corresponding current section. They do not contain a second installation procedure. The current tutorial is the six-step converged guide; historical QA remains linked under compatibility and history.

Check your result¶

Use the recorded result and compatibility tables before choosing a tuple. A guide link is not a runtime qualification.

Select a supported model/engine tuple before interpreting results. Keep direct-worker success separate from Gateway delivery, routing evidence, engine cache reuse, performance and release qualification. Historical PASS, FAIL and BLOCKED results retain their original artifact identities.

Next steps and cleanup¶

Use the canonical deployment sequence rather than combining fragments from different engine/version pages. Drain before switching or deleting owned resources; retained result pages are evidence, not another deployment recipe. Read the lab starting point if required values are missing. Use cleanup and recovery when ending the run.