Skip to content

LoxiLB Inference Gateway

An inference-aware L4/L7 load balancer for LLM serving fleets — the same GoLang/eBPF data path as loxilb, extended with routing that understands the distinct serving contracts of vLLM, SGLang, TensorRT-LLM, and llama.cpp.

What it is

The LoxiLB Inference Gateway is a fork of loxilb that adds AI-inference routing on top of loxilb's proven cloud-native load-balancing data path. A single gateway serves both classic L4/L7 traffic and modern LLM inference traffic, so you do not run a separate proxy tier for your model fleet.

Modern LLM serving creates load-balancing problems that classic L4/L7 policies cannot see: KV-cache locality dominates time-to-first-token, prefill and decode phases scale differently, and request cost varies by orders of magnitude with prompt content. The gateway solves these at the traffic layer, speaking the serving engines' native contracts rather than approximating them.

License

The LoxiLB Inference Gateway is licensed under the Apache License 2.0, the same as upstream loxilb.

Every AI feature is opt-in

AI routing is enabled per load-balancer rule. With no AI fields set, the gateway behaves exactly like upstream loxilb. All inference-aware behavior requires the L7 fullproxy data path (mode: 4) — see Running Modes.

Key capabilities

Capability What it does
Engine-aware integration Validate and route vLLM, SGLang, TensorRT-LLM, and llama.cpp rule shapes without pretending that every engine supports the same cache-event or P/D features.
Model-name routing Route by the requested model (X-Model header or body model field) to per-model endpoint pools; "" is the catch-all pool.
KV-cache-aware routing Send each request to the endpoint whose KV-cache already holds the longest prefix of the prompt — either zero-engine-change prefix-hash affinity (CHWBL) or engine-exact routing fed by the engines' KV-cache event streams.
P/D disaggregation L7-aware splitting of each request across prefill and decode endpoint pools with NIXL KV-transfer coordination and session affinity.
SSE streaming SSE-aware proxying that suppresses idle timeouts while a streaming response is active, with a wall-clock runaway cap and backend keepalive for long streams.
AI traffic governance On eligible SSE or P/D fullproxy rules, validate API keys, authorize models, and enforce request/token quotas before dispatch; apply byte-rate QoS separately to rules or ports. The separated key-store/auth plane is development-stage and must be qualified against the deployed image.
CHWBL / GPU-aware algorithms Consistent-hash-with-bounded-load selection (sel: 8), weighted CHWBL (sel: 10) for heterogeneous GPUs, and GPU-aware selection (sel: 9).
MCP gateway Session-sticky proxying of Model Context Protocol server pools, keyed on the mcp-session-id header.
OPA L4 policy Optional external Open Policy Agent watcher for L4 admission policy.

Load-balancer rule features use the management API on port 11111 under /netlox/v1/config/loadbalancer; keys, quotas, QoS, logs, and authentication use their own /netlox/v1 operations. See the AI Gateway Overview, API Reference, and Configuration Reference before automating these operations.

Choose a starting path

flowchart LR
    START([Start]) --> ENGINE{"Need help choosing<br/>an inference engine?"}
    ENGINE -->|Yes| CHOOSE["Choose an Inference Engine"]
    ENGINE -->|No| FIRST{"First deployment?"}
    CHOOSE --> MATRIX["Engine Capability Matrix"]
    MATRIX --> QUICK["Quickstart"]
    FIRST -->|Yes| QUICK
    FIRST -->|No| GOAL{"Primary goal?"}
    GOAL -->|Routing and P/D| AIGW["AI Gateway guides"]
    GOAL -->|Keys, quotas, QoS| GOVERN["AI Traffic Governance"]
    GOAL -->|Monitoring and recovery| OPS["Operations guides"]

    style QUICK fill:#e8f5e9,stroke:#43a047
    style AIGW fill:#e1f5fe,stroke:#0288d1
    style GOVERN fill:#fff3e0,stroke:#f57c00
    style OPS fill:#f3e5f5,stroke:#8e24aa

Who this is for

  • Teams operating an LLM serving fleet (vLLM, SGLang, TensorRT-LLM, or llama.cpp) that need cache-locality-aware routing, prefill/decode disaggregation, or a single multi-tenant OpenAI-compatible endpoint.
  • Platform teams who want one gateway for both classic Kubernetes/telco load balancing and inference-aware routing, without an Envoy + ext-proc sidecar chain.
  • Operators of MCP server fleets needing stable, session-sticky, TLS-terminating endpoints.

If you only need the base cloud-native load balancer with no AI routing, use upstream loxilb directly — this repository is the same load balancer with inference-aware routing built in.

A first rule

A pool of identical vLLM replicas behind one OpenAI-compatible VIP, with CHWBL prefix affinity (sel: 8) on the L7 fullproxy (mode: 4):

export GATEWAY_API="http://192.0.2.10:11111"

curl --fail-with-body -sS -X POST "$GATEWAY_API/netlox/v1/config/loadbalancer" \
  -H 'Content-Type: application/json' -d '{
  "serviceArguments": {
    "externalIP": "192.0.2.20", "port": 8080, "protocol": "tcp",
    "sel": 8, "mode": 4, "host": "192.0.2.20" },
  "endpoints": [
    { "endpointIP": "198.51.100.11", "targetPort": 8000, "weight": 1 },
    { "endpointIP": "198.51.100.12", "targetPort": 8000, "weight": 1 } ]}'

loxicmd create lb 192.0.2.20 --tcp=8080:8000 --endpoints=198.51.100.11:1,198.51.100.12:1 --select=chwbl --mode=fullproxy --host=192.0.2.20

The example uses documentation-only addresses. Replace them with your management endpoint, VIP, and backend addresses. If management authentication is enabled, add the deployment's bearer header without placing the token directly in shell history. The current proxy uses fixed CHWBL runtime constants; stored chwbl_* tuning fields do not change them yet.

Where to go next

Where it fits (scope and non-goals)

The gateway is a self-contained inference gateway: one Go/eBPF binary covers L4 through inference-aware L7. It load-balances your engines — it is not a multi-provider SaaS proxy (for federating hosted APIs, tools like LiteLLM compose in front of or behind it), and it is not an orchestrator (it does not schedule or scale engine pods). It is the traffic layer.