Skip to content

vLLM Integration

Illustrative mutation fragments

The management mutations on this page are non-standalone fragments. Before adapting them, follow this site's Example Quality Contract and complete Quickstart lifecycle. Do not treat HTTP success or management readback alone as runtime proof.

Keep AI-aware vLLM rules on HTTP/1.1

Current HTTP/2 forwarding lacks model-aware pool lookup, selector 10, P/D, and KV-exact integration; selector 9 becomes round-robin. Treat HTTP/2 parity as a release gate.

Run vLLM behind the loxilb AI Gateway: the launch flags that must line up with the gateway, how to health-probe vLLM backends, backend protocol / ALPN, and the exact selector-9 metrics boundary.

Concept

vLLM is the inference engine that actually runs the model on the GPU. loxilb fronts one or more vLLM instances as an L7 fullproxy (mode: 4) and spreads requests according to the selected routing law. Nothing special is required of vLLM to sit behind the gateway, but several launch flags must match the gateway's configuration for KV-aware features to work.

flowchart LR
    C([Clients]) --> LB["loxilb AI Gateway<br/>mode 4"]
    LB --> V1["vLLM :8000"]
    LB --> V2["vLLM :8000"]
    LB --> V3["vLLM :8000"]
    style LB fill:#e8f5e9,stroke:#43a047

Launch Flags That Matter

Use Engine Contracts for the single seed/hash/block/event contract. For the current vLLM 0.31.0 converged launch, follow Step 3. The separate P/D reference has its own connector and version qualification requirements.

--enable-request-id-headers is an optional engine diagnostic that exposes request IDs in response headers; Gateway-internal correlation does not require it. Client response IDs alone are not independent backend receipt evidence.

Health Probing

vLLM exposes /health. Configure the allocated engine port and health path using shared probe fields. HTTP health checks test the route; TCP probes establish only listener reachability. Neither proves complete inference, successful P/D transfer or Exact readiness. Use Step 3 for direct model/chat checks and Step 5 for completion and independent receipt verification.

Backend Protocol and ALPN

backend_protocol sets the protocol loxilb negotiates toward the vLLM backends (via ALPN when TLS is in play):

Value Meaning
http1 HTTP/1.1 only. Default and safest — matches vLLM's OpenAI server.
http2 HTTP/2 only.
both Advertise both HTTP/1.1 and HTTP/2 and let ALPN negotiate.

security controls TLS on the service VIP:

Value Mode
0 Plaintext frontend and backend
1 Frontend TLS termination with a plaintext HTTP backend
2 Frontend TLS termination plus TLS re-encryption to the backend

For a typical vLLM deployment, backend_protocol: http1 with security: 0 (plain) or security: 1 (TLS termination on the VIP) is the right starting point.

sel: 9: Plain-pool affinity

On the HTTP/1.1 plain-pool path, selector 9 uses prefix/conversation modulo affinity and healthy-endpoint fallback. Endpoint changes can remap placements. Worker telemetry and configuration readback do not establish least-loaded selection; measure backend distribution independently.

Verify

GPU monitoring status — GET /config/gpu/status reports the worker-metrics control surface:

curl -s "http://<loxilb>:11111/netlox/v1/config/gpu/status"
# {
#   "enabled": true,
#   "routing_mode": "gpu_aware",
#   "worker_count": 3,
#   "last_metrics_update": "...",
#   "ebpf_map_loaded": true
# }

routing_mode is gpu_aware when that surface is active or standard_chwbl when disabled. This is a control-plane status, not proof that a plain fullproxy service is choosing the least-loaded endpoint.

Per-worker metrics — GET /config/worker/metrics returns the current WorkerMetricsEntry list stored on that separate surface:

curl -s "http://<loxilb>:11111/netlox/v1/config/worker/metrics"

Selection algorithm — confirm the service shows sel: 9:

curl -s "http://<loxilb>:11111/netlox/v1/config/loadbalancer/all" \
  | jq '.lbAttr[].serviceArguments | {externalIP, port, sel}'

Troubleshooting

sel: 9 does not move traffic toward a less-loaded worker

For a plain single pool, this is expected: the current fullproxy path uses prefix/session modulo placement and a healthy fallback, not the pushed worker metrics. Switch to CHWBL for bounded prefix affinity, or round-robin for independent requests. For P/D, use the normal routing hierarchy; do not depend on selector-9 capacity scoring in the current release.

/metrics returns nothing

Treat this as an observability or metrics-agent problem, not the explanation for plain-pool selector 9 placement. Confirm stats logging was not disabled and that the serving port is reachable from the metrics collector.

All endpoints show high queue depth

  • The fleet is genuinely saturated — add vLLM instances.
  • Enable KV-cache routing (KV-Cache Routing) to raise cache-hit rates and cut per-request work.
  • Consider a smaller model or quantization to raise per-GPU throughput.

Endpoints marked inactive

  • The health probe is failing. Verify probereq (e.g. /health) and probeport match the vLLM serving port.
  • Raise probeTimeout / probeRetries if backends are slow to start.
  • Confirm the probe path returns success on the vLLM instance directly.

KV-aware routing gets no cache hits

Almost always a launch-flag mismatch. Recheck --block-size vs kvBlockSize, --prefix-caching-hash-algo vs kvHashAlgo, and the KV-events port vs kvZmqPort. See KV-Cache Routing for the complete parity contract.

Next Steps