Prepare a Supported Model¶

Strict KV-exact routing binds a rule to one model profile: the model's pinned tokenizer, chat template and engine identity. This page prepares the hosts for one model from the gateway's supported-models list, up to the point where the profile is published and the engines are serving. Binding the rule and reading its readiness is on Model Profiles and KV-Exact Readiness.

Current-main contract

The model-profile registry, the installer and kvModelProfile are on Gateway main. They are absent from Gateway v0.9.8.9-rc.1. Check the installed Gateway identity before following this page.

Every step below exists because the gateway must compute the same token ids and cache hashes as the engine. None of them is a tuning choice: an engine launched without one of them never reaches READY.

1. Choose a model and an engine version¶

The supported-models list is generated in the gateway repository: docs/SUPPORTED-MODELS.md. For each model × engine row it gives the pinned revision, the engine version and image digest, the state, the GPU memory notes and the launch arguments that model needs. The same list is available from a checkout:

scripts/models/install-models.sh --list

A row is supported only in the state validated. A candidate row has qualification evidence but is not supported; the installer skips it unless --include-candidates is given. Use exactly the engine version and image digest of the row: the gateway checks the engine's identity before it routes.

Pick the topology with Choose an Inference Engine:

Topology	Endpoints	`kvExactMode`
Single pool (converged)	two or more engines that each prefill and decode	`3`
Prefill/decode (P/D)	two or more prefill engines and one or more decode engines	`1`

With one engine in a single pool, or one prefill engine in P/D, there is no choice to make and exact routing gives the same placement as round-robin.

The TensorRT-LLM rows are single pool only, on the chat API: the list has no P/D evidence for that engine.

2. Stage the profile registry on the gateway host¶

From a checkout of the gateway repository at the release you run:

sudo scripts/models/install-models.sh --engine vllm --models <profile-id> \
     [--include-candidates] [--hf-token-file ./hf.token]

The installer verifies the committed profile, engine manifest, probe fixtures and chat template against the manifest, downloads tokenizer.json at the pinned revision and refuses it unless its sha256 is the pinned one, then stages:

Path	Content
`/etc/loxilb/kvprofiles/<profile-id>.yaml`	the profile
`/etc/loxilb/kvprofiles/manifests/<profile-id>.yaml`	the engine identity the gateway checks
`/etc/loxilb/kvprofiles/probefixtures/<profile-id>/`	the token probes the engine must answer exactly; the SGLang and TensorRT-LLM sets are in its `sglang/` and `trtllm/` sub-directories
`/etc/loxilb/kvprofiles/artifacts/sha256/`	the tokenizer and template, content-addressed
`/etc/loxilb/tokenizers/<org>__<name>/tokenizer.json`	the tokenizer

--engine is vllm, sglang or trtllm and selects which engine's manifest is staged. A registry holds one engine manifest per profile, so stage with --engine sglang for an SGLang pool and --engine trtllm for a TensorRT-LLM pool.
Gated models need an access token in a file of mode 0600. The token is sent as a request header only; it is never taken from the command line.
--dry-run prints what would be downloaded and staged, and writes nothing.
A re-run downloads nothing that is already present and verified.

3. Start the gateway with the registry mounted¶

The gateway loads the registry when it starts. Mount both directories read-only, set the hash seed that pairs with the engines' PYTHONHASHSEED=0, and restart the gateway after any new staging:

docker run -d --name loxilb --privileged --net host \
  -e LLB_KV_NONE_HASH_SEED=0 \
  -v /etc/loxilb/kvprofiles:/etc/loxilb/kvprofiles:ro \
  -v /etc/loxilb/tokenizers:/etc/loxilb/tokenizers:ro \
  <gateway-image>

The registry refuses a file that is not owned by root or the gateway user, is writable by group or others, or sits behind a symbolic link; a refused load leaves the previous generation serving and names the file in the gateway log. Confirm that the profile is published with the discovery step on Model Profiles and KV-Exact Readiness.

4. Stage the weights and the image on every engine host¶

sudo scripts/models/install-models.sh --engine vllm --models <profile-id> \
     [--include-candidates] [--hf-token-file ./hf.token] --weights-dir /models --pull-image

--weights-dir downloads the pinned snapshot into /models/<org>__<name>/<revision>/ and verifies every file against the committed weights index; --pull-image pulls the engine image by digest. To check a snapshot that is already on the host, use scripts/models/modelctl.py verify-weights <profile-id> <dir>.

TensorRT-LLM has no registry image: with --engine trtllm, --pull-image prints SKIP_NO_PULLABLE_IMAGE and pulls nothing. Build the image from the engine's release wheel at the version of the row.

5. Start the engines¶

Add the per-model arguments from the row's launch notes (for example a lower sequence limit for a hybrid model on one GPU). Engines whose chat template prints the date must run in UTC: no TZ variable and no host /etc/localtime mount.

vLLMSGLangTensorRT-LLM

IMAGE="vllm/vllm-openai@sha256:<digest from the row>"
SNAP="/models/<org>__<name>/<revision>"
MODEL="<org>/<name>"
ROLE_ARGS=()   # fill from the role table below
docker run -d --name vllm --gpus all --ipc=host --network host --ulimit memlock=-1 \
  -e PYTHONHASHSEED=0 -e VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=1 -e HF_HUB_OFFLINE=1 \
  -v /models:/models \
  "$IMAGE" \
  --model "$SNAP" --served-model-name "$MODEL" --host 0.0.0.0 --port 8000 \
  --enable-prefix-caching --prefix-caching-hash-algo sha256_cbor --block-size 16 --prefix-match-unit 16 \
  "${ROLE_ARGS[@]}"

Role	Role arguments
single pool	`--kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":null,"topic":""}'`
P/D prefill	the same `--kv-events-config`, plus `--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_buffer_device":"cpu","kv_load_failure_policy":"fail"}'`
P/D decode	`--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_buffer_device":"cpu","kv_load_failure_policy":"fail"}'` and no KV events

P/D members also need -e UCX_TLS=tcp -e VLLM_NIXL_SIDE_CHANNEL_HOST=<this host> -e VLLM_NIXL_SIDE_CHANNEL_PORT=5600.

IMAGE="lmsysorg/sglang@sha256:<digest from the row>"
REV="<revision>"
SNAP="/models/<org>__<name>/$REV"
MODEL="<org>/<name>"
ROLE_ARGS=()   # fill from the role table below
docker run -d --name sglang --gpus all --network host --ipc=host --shm-size 16g \
  -e HF_HUB_OFFLINE=1 -v /models:/models \
  "$IMAGE" \
  python3 -m sglang.launch_server --model-path "$SNAP" --revision "$REV" \
  --served-model-name "$MODEL" --host 0.0.0.0 --port 8000 --page-size 16 --enable-metrics \
  "${ROLE_ARGS[@]}"

Role	Role arguments
single pool	`--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557"}'`
P/D prefill	the same `--kv-events-config`, plus `--disaggregation-mode prefill --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 8998`
P/D decode	`--disaggregation-mode decode --disaggregation-transfer-backend mooncake`

--revision is required with the snapshot path: the gateway compares the revision the engine reports with the pinned one. Where the row says so, mount the fixed serving_tokenize.py from the gateway repository's cicd/kv-model-compat-pd/sglang-tokenize-fix/.

Single pool only, PyTorch backend. The engine options go in a file:

# /etc/trtllm/kv.yaml
kv_cache_config:
  enable_block_reuse: true        # the prefix cache
  event_buffer_max_size: 4096     # the KV events the gateway drains; 0 turns them off
return_perf_metrics: true         # Prometheus text at /prometheus/metrics

IMAGE="<your image of the row's TensorRT-LLM version>"
SNAP="/models/<org>__<name>/<revision>"
MODEL="<org>/<name>"
docker run -d --name trtllm --gpus all --network host --ipc=host --shm-size 16g \
  -e HF_HUB_OFFLINE=1 -v /models:/models -v /etc/trtllm/kv.yaml:/cfg/kv.yaml:ro \
  "$IMAGE" \
  trtllm-serve "$SNAP" --served_model_name "$MODEL" --host 0.0.0.0 --port 8000 --backend pytorch \
  --extra_llm_api_options /cfg/kv.yaml

There is no role argument and no event port: the gateway drains the KV events from POST /kv_cache_events on the serving port, and it must be the only reader of that endpoint (TensorRT-LLM Integration). The engine reports 32 tokens per block in GET /server_info; the rule's kvBlockSize must be that number. The engine is ready when /v1/models lists the model and /health answers, which is minutes after the container starts. The launch line the gateway's own scenario uses, with the library path its locally built image needs, is in the gateway repository's cicd/kv-model-compat-pd/engine.sh.

To restart an engine, stop it, wait until no socket holds the port, then start it: trtllm-serve binds the port before it loads the model and exits with Address already in use while connections of the previous process are still closing.

The values that must agree across the whole pool and the rule:

Value	Engine side	Rule side
Served model name	`--served-model-name <org>/<name>`	`model_name`
Block size	vLLM `--block-size 16`, SGLang `--page-size 16`	`kvBlockSize: 16`
Block size (TensorRT-LLM)	`tokens_per_block` in `/server_info`, 32	`kvBlockSize: 32`
Engine	the row's image digest	`kvEngineType`
Hash seed (vLLM)	`PYTHONHASHSEED=0`	gateway `LLB_KV_NONE_HASH_SEED=0`

The gateway must reach these engine ports. Keep them on a private network: the engines have no authentication of their own.

Port	Used for
`8000`	OpenAI-compatible API, identity and token probes
`5557`	KV cache events from single-pool and prefill engines (vLLM, SGLang; TensorRT-LLM sends them on `8000`)
`5600` (vLLM P/D), `8998` (SGLang P/D)	engine-to-engine KV transfer side channel

Wait until every engine lists the served model:

curl --fail-with-body --silent --show-error http://<engine-host>:8000/v1/models

6. Bind the rule and wait for READY¶

Continue on Model Profiles and KV-Exact Readiness: create the rule with kvModelProfile, kvEngineType, kvBlockSize: 16 (32 for TensorRT-LLM) and the kvExactMode of your topology, then read kvexactstatus until enforcedState is READY. The example on that page is a single pool. For P/D, the rule also sets pd_disagg_mode: true and tags every endpoint with ep_role (1 prefill, 2 decode) and nixl_port; the rule body is in Deploy: Prefill/Decode Disaggregation. Before READY, reasonCodes names what is missing:

Reason code	Usual cause on a fresh setup
`identity_mismatch`	engine version (vLLM) or reported revision (SGLang) differs from the row; a TensorRT-LLM engine whose `/server_info` does not describe its KV events (older than the row's version), or that does not serve the model name
`token_mismatch`	wrong revision, tokenizer or template on the engine, or a launch argument from step 5 missing
`challenge_failed`, `challenge_timeout`	KV events not reaching the gateway (port `5557`, `--kv-events-config`; on TensorRT-LLM an event buffer of 0 or a second reader of `/kv_cache_events`), the hash-seed pair missing, or a block size that differs from `kvBlockSize`
`endpoint_unreachable`	the gateway cannot reach the engine's API port
`manifest_missing`, `probe_fixtures_missing`, `profile_registry_unavailable`	the registry was staged for another engine, or the gateway was not restarted after staging

Requests that use a feature a strict rule refuses (tools, chat_template_kwargs, multimodal content parts, a conversation ending on an assistant turn) receive a typed error instead of an approximate hash; the full list is in the supported-models page.