Prerequisites and downloads¶

What the cluster, the gateway node and your shell need before anything is deployed, and where the files come from.

Before you start¶

You need a Kubernetes cluster with GPU nodes, one node you can run a privileged hostNetwork pod on, and a shell that reaches both.

Focus of this page: Every later chapter assumes the CRDs, the variables and the downloads from here.

Topology and execution locations¶

flowchart TD
    C["Client<br/>sends to VIP:8080"] --> G["loxilb-igw on the control-plane node<br/>rule: mode 4, eppEndpoint"]
    G -->|"ext_proc gRPC :9002<br/>one question per request"| E["Endpoint Picker<br/>Deployment + ClusterIP Service"]
    E -.->|"watches"| P["InferencePool<br/>selector → model-server pods"]
    E -->|"answers with one pod address"| G
    G -->|"forwards to the pod the EPP named, :8000"| V["vLLM pods<br/>one per GPU node"]
    O["Operator shell"] -.->|"REST :11111, kubectl, pod :8000/metrics"| G
    style G fill:#e1f5fe,stroke:#0288d1,color:#1f2937
    style E fill:#fff8e1,stroke:#f9a825,color:#1f2937
    style V fill:#e8f5e9,stroke:#43a047,color:#1f2937

Where	What happens here
Operator shell (the control-plane node in this guide)	`kubectl`, `curl` against the gateway's REST API on `:11111`, the harness, pod `:8000/metrics` reads
Client	sends OpenAI-style requests to the gateway VIP, `192.0.2.10:8080`
loxilb-igw (DaemonSet, hostNetwork on the control-plane node)	holds the rule, asks the EPP per request, forwards to the pod it names
Endpoint Picker (Deployment + ClusterIP Service)	watches the `InferencePool`, keeps the prefix-cache index, answers over ext_proc gRPC `:9002`
vLLM pods (one per GPU node)	serve the model on `:8000`; their metrics are the independent attribution of which pod served a request

Walkthrough¶

Requirement	Used here
Kubernetes cluster with GPU worker nodes and the NVIDIA device plugin (`nvidia.com/gpu` schedulable)	RKE2 v1.36.4, Calico, 3 × A100 80GB, one GPU per node
For the kube-loxilb method (kube-loxilb path): kube-loxilb `ghcr.io/loxilb-io/kube-loxilb:epp-annotation` running with `--gatewayAPI --inferenceExtension`, pointed at the gateway's REST API	that tag carries the `InferencePool` manager (kube-loxilb branch `feat/auth-annotation`); its Inference gateway guide is the reference for everything the pool can carry
One node that can run a privileged hostNetwork pod and whose IP clients can reach. Its IP becomes the VIP.	the control-plane node, 192.0.2.10
The `InferencePool` CRD (`inference.networking.k8s.io/v1`), and for the kube-loxilb method the Gateway API CRDs	installed from the release manifests below
A shell with `kubectl`, `curl` and `python3` that can reach the gateway's REST API (`:11111` on the gateway node) and every vLLM pod's `:8000`	the same control-plane node
The model's weights reachable from the GPU nodes (Hugging Face download or a local mirror)	`Qwen/Qwen3-32B-FP8`, a public model

Install the CRDs first:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.2/manifests.yaml
kubectl get crd inferencepools.inference.networking.k8s.io

# only for the kube-loxilb method; the Inference Extension manifest does not include these
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

On RKE2 the control-plane node's kubectl is the bundled binary with the server kubeconfig. Either export it once or alias it; every command below says kubectl:

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin

Network paths that must be open: client → VIP:8080; gateway node → pod IPs :8000; gateway node → EPP ClusterIP :9002; your shell → pod IPs :8000 (for the metrics reads in the measurement post).

Downloads¶

Every file this family uses is under docs/assets/endpoint-picker/ in the documentation checkout. Set DOCS to that checkout and MF to the manifests folder once; every command below uses them:

export DOCS="${DOCS:?path to the documentation checkout}"
export MF="$DOCS/docs/assets/endpoint-picker/manifests"
ls "$MF"

Download	Purpose	What to customize
`01-namespace.yaml`	namespace `llm-d-pd-disaggregation`	the name, everywhere at once
`10-vllm.yaml`	three vLLM pods, one GPU each	model, image tag, replicas, GPU resources
`20-epp-config.yaml`	the EPP scheduling profile	plugin set, only if you measure another profile
`21-epp-rbac.yaml`	ServiceAccount and ClusterRole for the EPP	nothing
`22-epp.yaml`	the EPP Deployment	image tag, `--v`
`23-epp-service.yaml`	the EPP ClusterIP Service (9002, 9003, 9090)	nothing
`30-inferencepool.yaml`	the `InferencePool` and its `endpointPickerRef`	`failureMode`
`40-loxilb-igw.yaml`	the loxilb-igw DaemonSet	image tag, `--blacklist`, node placement
`50-gateway.yaml`	GatewayClass, Gateway, HTTPRoute for the kube-loxilb path	listener port
`rule.sh`	creates the rule in EPP, CHWBL or RR mode from live pod IPs	`VIP`, `PORT`, `FAILMODE` as environment variables
`repro/`	the reproduction harness (`run.sh`, `config.env`)	`config.env`
`baseline/`	the vLLM, EPP, `InferencePool` and rule objects as captured at measurement time	reference only
`epp-measurements-2026-10.tgz` (sha256)	raw logs, summaries and the original scripts behind the measurements	reference only

Check your result¶

kubectl get crd inferencepools.inference.networking.k8s.io returns the CRD, ls "$MF" lists the ten manifests and rule.sh.

Next steps and cleanup¶

Continue with Model Servers & Endpoint Picker. Nothing has been created in the cluster yet except the CRDs.