AI Traffic Governance¶

Illustrative mutation fragments

The management mutations on this page are non-standalone fragments. Before adapting them, follow this site's Example Quality Contract and complete Quickstart lifecycle. Do not treat HTTP success or management readback alone as runtime proof.

LoxiLB can control who may use an inference service, which models they may call, how quickly they may submit requests, how many tokens they may consume, and how much network bandwidth a service may use. These controls solve different problems and should be configured independently.

The request-admission flow below applies to a mode: 4 rule whose api_key_auth declaration attributes a credential. The declaration is independent of sse_mode and pd_disagg_mode. Omission preserves backend-owned X-Api-Key; explicit disabled strips it without validation.

Choose the control that matches the problem¶

flowchart TD
    REQ([Inference request]) --> AUTH{"API key valid,<br/>enabled, and unexpired?"}
    AUTH -->|no| R401([401 invalid_api_key])
    AUTH -->|yes| MODEL{"Model allowed<br/>for this key?"}
    MODEL -->|no| R403([403 model_not_allowed])
    MODEL -->|yes| RPS{"Key and tenant<br/>RPS buckets allow?"}
    RPS -->|no| R429A([429 rate limit])
    RPS -->|yes| TPM{"Tenant and model<br/>TPM buckets allow?"}
    TPM -->|no| R429B([429 token quota])
    TPM -->|yes| GPU["Forward to inference backend"]
    GPU --> SETTLE["Read usage or estimate tokens<br/>settle reserved budget"]
    SETTLE --> RESP([Response to client])

    style GPU fill:#e8f5e9,stroke:#43a047
    style SETTLE fill:#e1f5fe,stroke:#0288d1
    style R401 fill:#ffcdd2,stroke:#e53935
    style R403 fill:#ffcdd2,stroke:#e53935
    style R429A fill:#fff9c4,stroke:#f9a825
    style R429B fill:#fff9c4,stroke:#f9a825

The gateway evaluates access and admission before it spends backend capacity. It accounts for actual or estimated token use when the response completes. Bandwidth policies sit on a separate path and meter bytes rather than requests or tokens.

These checks decide who may send a request. A separate control decides how many may execute on a model pool at once: the capacity admission gate runs after the checks above and before dispatch, and queues or refuses a request over a ceiling with 429, 503, or 504. It is configured per rule and documented in Admission Flow Control.

Control	Scope	Unit	Typical purpose
API-key model allow-list	One API key	Model identifiers	Prevent a workload from calling unauthorized models
API-key RPS	One API key	Requests per second	Protect against one noisy credential
Per-key TPM	One API key	Tokens per minute	Pre-admission reservation and usage settlement; insufficient reservation returns 429 before backend delivery, and post-response debt can affect later requests. Validate the active protocol and service
User RPS/TPM	One verified tenant/user pair	Requests or tokens	Bound an identity derived from JWT claims
User-and-model TPM	One verified user and model	Tokens per minute	Isolate one user's expensive-model spend
Tenant RPS	All keys for one tenant	Requests per second	Share an admission ceiling across a tenant
Tenant TPM	All models for one tenant	Tokens per minute	Bound aggregate AI work
Tenant-and-model TPM	One model within one tenant	Tokens per minute	Reserve a smaller budget for an expensive model
Shared VIP RPS/TPM	One service	Requests or tokens	Optional bucket for keyless traffic; token spend also includes attributed traffic on that service
Rule/global defaults	Identities without explicit rows	Requests or tokens	Rule fields override global fields; zero falls through
QoS policy	LB rule or network port	Megabits per second at the API	Bound traffic rate or pace a fullproxy service
Capacity admission gate	One model pool	Executing and waiting requests	Protect the engines from overload; see Admission Flow Control

RPS means requests per second. TPM means tokens per minute. TPM is a smooth token bucket that refills continuously; it is not a counter that resets at the top of each minute.

Prerequisites and secure API access¶

You need:

a fullproxy inference rule (mode: 4) declaring api_key_auth: required, jwt, or apikey-or-jwt;
the Gateway started with the independent PostgreSQL AI-key store configured through --aikey-db-* options;
one management authentication mode enabled for port 11111;
a control-plane identity authorized to manage AI keys and limits;
TLS on every control-plane and inference endpoint outside an isolated lab.

Keep the management bearer token and inference API key separate. The following examples read the management header from a permission-restricted file so the token is not written directly into every shell command:

export CONTROL_API="https://gateway.example.com/netlox/v1"
install -m 600 /dev/null ./control-plane.headers
printf 'Authorization: Bearer %s\n' "$CONTROL_PLANE_TOKEN" > ./control-plane.headers

Replace the example URL with your deployment. Obtain CONTROL_PLANE_TOKEN from your identity and secret-management workflow; do not paste it into shell history, source control, tickets, or logs.

Prove both authentication planes before exposure

A mode: 4 rule that omits api_key_auth is keyless even with a healthy key store. A required rule with no usable store fails closed with 503; it does not become keyless. With no user, OAuth, or manual-token management mode, key and quota CRUD is callable without credentials. Require 401 from both a missing-key inference probe and an unauthenticated management mutation before proceeding.

Step 1 — Create a narrowly scoped key¶

Start with only the models and request rate the workload needs:

curl --fail-with-body --silent --show-error \
  --request POST "$CONTROL_API/config/ai/apikey" \
  --header @control-plane.headers \
  --header 'Content-Type: application/json' \
  --data '{
    "tenant_id": "team-a",
    "name": "chat-service",
    "allowed_models": ["example-chat-model"],
    "rate_limit_rps": 5,
    "burst_size": 10,
    "tokens_per_min": 0,
    "enabled": true
  }' > ./new-key.json

jq '{key_id, raw_key_present: (.raw_key | type == "string")}' ./new-key.json

Expected result: 201 Created; the response contains a key_id and a raw_key. The raw key is returned only by this create operation. Store it in a secret manager immediately, then securely remove new-key.json. List and get operations never return the raw key or its stored hash.

Treat the create response as a secret

Do not print raw_key in a terminal recording, CI log, dashboard, or issue. If it is exposed or lost, delete the key and create a replacement.

Per-key tokens_per_min reserves a request budget before backend admission and settles it from response usage on supported accounting paths. The selected primary Swagger describes both stages. Verify the active accounting path before using token quotas as an operational control.

Step 2 — Set tenant and per-model token budgets¶

This example gives the tenant an aggregate budget of 60,000 TPM and limits one model to 20,000 TPM:

curl --fail-with-body --silent --show-error \
  --request POST "$CONTROL_API/config/ai/tenant/ratelimit" \
  --header @control-plane.headers \
  --header 'Content-Type: application/json' \
  --data '{
    "tenant_id": "team-a",
    "rps": 20,
    "tokens_per_min": 60000,
    "burst_pct": 50,
    "model_limits": [
      {"model": "example-chat-model", "tokens_per_min": 20000}
    ]
  }'

Expected result: 204 No Content. Read the effective configuration back:

curl --fail-with-body --silent --show-error \
  --header @control-plane.headers \
  "$CONTROL_API/config/ai/tenant/ratelimit/team-a" | jq .

The response should show rps: 20, tokens_per_min: 60000, burst_pct: 50, and the model limit.

What `burst_pct` changes¶

burst_pct changes bucket capacity, not refill rate:

tokens_per_min sets the continuous refill rate;
burst_pct sets how much accumulated credit an idle tenant may spend at once;
0 uses the server default, which is normally 100 percent;
positive values are clamped to the supported 1–1000 percent range.

With 60,000 TPM and burst_pct: 50, an idle aggregate bucket can hold 30,000 tokens. A single request whose prompt estimate plus declared completion ceiling exceeds that capacity cannot be admitted, even if the sustained TPM rate looks large enough. Size the bucket for the largest legitimate request, but avoid a large value that permits an unwanted burst after a long idle period.

The same tenant burst_pct applies to its aggregate and per-model buckets.

How aggregate and model budgets combine¶

When both limits exist, a request must fit both buckets. A denial by the model bucket rolls back the aggregate reservation, so a failed two-bucket admission does not consume aggregate headroom. A tokens_per_min value of 0 disables that quota. A model entry with tokens_per_min: 0 removes that model quota.

Step 3 — Understand reservation and settlement¶

Before dispatch, the gateway reserves the estimated prompt tokens plus the request's declared max_tokens or equivalent completion ceiling only when it has buffered a complete, contiguous, positive-Content-Length JSON body. Chunked, partial, and oversized bodies skip request-body parsing and include_usage injection. Their fallback prompt estimate can undercount the actual prompt, so do not treat admission as exact accounting for those bodies.

When the response completes, the gateway releases the pessimistic reservation and charges the measured result:

If the engine returns a readable usage object, its prompt and completion counts are charged.
For a streaming request whose complete, contiguous JSON body was buffered with a positive Content-Length, the gateway requests usage reporting. Chunked, partial, and oversized bodies skip this injection.
If no readable usage object arrives, the gateway charges its fallback estimate and exposes that path in metrics. For a skipped request body, the prompt estimate may be zero and can undercount actual use.
A response already sent to the client is not withdrawn by settlement. If the final charge creates debt, a later request is denied until refill restores headroom.

During peer quota-state warm-up, admission returns 429 token_quota_warming with Retry-After: 1. The default warm-up deadline is three seconds; if no peer state arrives by then, the compatibility path fails open. Alert on this condition and do not treat the timeout as proof of synchronized quota state.

This is why an operator should alert on both denied requests and estimated accounting. A growing estimate rate may indicate an engine or response-format compatibility problem.

Step 4 — Verify enforcement safely¶

Send the inference key in X-Api-Key. Use a protected header file for the same reason as the control-plane token:

install -m 600 /dev/null ./inference.headers
printf 'X-Api-Key: %s\n' "$INFERENCE_API_KEY" > ./inference.headers

curl --fail-with-body --silent --show-error \
  --header @inference.headers \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "example-chat-model",
    "messages": [{"role": "user", "content": "Reply with one word."}],
    "max_tokens": 8
  }' \
  "https://ai.example.com/v1/chat/completions" | jq .

Verify one failure at a time in a non-production environment:

Probe	Expected result	Meaning
Omit `X-Api-Key`	`401`, `invalid_api_key`	Authentication is active
Use an unknown, disabled, expired, or revoked key	`401`	The key cannot authenticate
Request a model outside `allowed_models`	`403`, `model_not_allowed`	Model authorization is active
Send a concurrent burst above key or tenant RPS	At least one `429`	Request admission is active
Submit a request larger than available token headroom	`429` with retry guidance	Token reservation is active
Stop or remove the store behind a `required` rule	`503`, `policy_store_unavailable`	Policy evaluation fails closed

Do not load-test a shared production tenant to prove a limiter. Use a dedicated tenant and a small test limit, then restore or remove the configuration.

Observe the decision path¶

Useful Prometheus series include:

Metric	Type	Meaning
`loxilb_ai_rate_limit_hits_total{tenant,reason}`	Counter	RPS and quota denials by reason
`loxilb_ai_model_not_allowed_total{model,tenant}`	Counter	Model authorization denials
`loxilb_ai_tokens_consumed_total{model,tenant,kind}`	Counter	Prompt or completion tokens charged
`loxilb_ai_tokens_estimated_total{model,tenant}`	Counter	Tokens charged from estimates
`loxilb_ai_tokens_missing_total{model,tenant}`	Counter	Responses without readable usage
`loxilb_ai_token_quota_denied_total{tenant}`	Counter	Token-quota admission denials
`loxilb_ai_token_quota_utilization{tenant}`	Gauge	Aggregate quota fraction currently spent
`loxilb_ai_token_quota_limit_tokens{tenant}`	Gauge	Aggregate TPM limit
`loxilb_ai_token_quota_model_utilization{tenant,model}`	Gauge	Per-model quota fraction currently spent
`loxilb_ai_token_quota_model_limit_tokens{tenant,model}`	Gauge	Per-model TPM limit
`loxilb_ai_token_quota_cold_open_total`	Counter	Node began quota service without restored peer state

Utilization can temporarily exceed 1 after post-response debt. It decreases continuously as the bucket refills. Do not add aggregate and model utilization together; they are two gates over the same request.

Diagnose HTTP responses¶

Status	Likely source	Check first
`401`	Missing, unknown, disabled, expired, or revoked API key	Key state and `X-Api-Key` handling; never log the key value
`403`	Requested model is outside the key allow-list	Effective model and exact allow-list spelling
`429`	Key RPS, tenant RPS, aggregate TPM, or model TPM	Response error code, `Retry-After`, and rate-limit metrics
`502`	Selected backend failed before a usable response	Endpoint health and backend/proxy logs
`503 policy_store_unavailable`	Required credential or identity-bearing quota policy cannot be evaluated	Key/JWKS store health and cached last-known-good state; backend delta must remain `0`
Other `503`	No usable route/backend, maintenance, or an engine-specific fail-closed condition	Rule read-back, endpoint health, and engine-specific metrics
`503` from key/quota CRUD	Key store not configured or unavailable	Check the independent PostgreSQL store and `--aikey-db-*` settings

A 502 or 503 is not evidence of quota exhaustion. A 429 is not evidence of backend failure. Diagnose the status and error body before changing limits.

Security and operational guidance¶

Give management access only to identities that need key and quota CRUD.
Terminate TLS with a trusted certificate; do not expose management or inference credentials over plaintext networks.
Use one key per workload, environment, or rotation boundary. Avoid sharing a single key across unrelated applications.
Keep tenant and model labels free of secrets and personal information; they appear in metrics and logs.
Redact Authorization, X-Api-Key, raw_key, request bodies, and prompt content before sharing diagnostic output.
Rotate by creating and validating a replacement, switching the workload, and then revoking the old key.
Treat quota runtime state as sensitive operational data. It reveals workload activity even though it does not contain raw prompts or keys.

Clean up¶

Delete the key by its opaque key_id, not by its raw secret:

curl --fail-with-body --silent --show-error \
  --request DELETE \
  --header @control-plane.headers \
  "$CONTROL_API/config/ai/apikey/$KEY_ID"

rm -f ./new-key.json ./inference.headers ./control-plane.headers
unset CONTROL_PLANE_TOKEN INFERENCE_API_KEY

Expected delete result: 204 No Content; a later get for the same key returns 404. A PATCH /config/ai/apikey/{key_id} can replace allowed_models without changing the secret. For a rotation boundary or a high-risk scope reduction, create a narrower replacement, switch the workload, verify it, and then delete the old key.

Setting tenant and model TPM values to 0 disables those quota gates. Confirm the result with the tenant rate-limit get operation.

What this guide does not prove¶

The repository validation scenarios demonstrate single-node authentication, authorization, RPS enforcement, and API behavior. Unit and mock checks validate quota arithmetic and metrics. They do not demonstrate production capacity, two-node failover, mixed-version quota compatibility, or lossless migration of in-flight requests. See HA and Upgrade Limitations.

For the integrated configure, readback, traffic, receipt and rollback sequence, see the Admission Control, Flow Control and QoS lab.

Network identity and objective API contracts¶

The schema reference includes network attribution and named objective APIs in addition to credential and quota policy. Check the deployed Swagger before using these operations. Their presence in the source contract does not establish GAIE/llm-d integration, request-ordering behavior or deployment qualification. The admission lab does not qualify these controls.

Resource	Operations under `/netlox/v1`	Contract and ownership
Network identity	POST/GET `/config/ai/netidentity`; DELETE `/config/ai/netidentity/{name}`	Named node-local network-to-tenant binding. POST replaces the same name. Configuration is not replicated to an HA peer.
Objective	POST/GET `/config/ai/objective`; DELETE `/config/ai/objective/{name}`	Named priority stored in the AI key store. Store-backed changes apply immediately on the writing Gateway and within the refresh interval on other Gateways sharing that store (default 10 seconds).

AIObjectiveEntry requires name. It accepts an optional services list of VIP:port strings and a signed integer priority. Empty services means all services; positive priority maps to critical, zero to standard, and negative to sheddable. Different numeric priorities within the same class do not retain a relative order. Names are case-sensitive, 1–63 characters of letters, digits, -, _ or ., and cannot start with tier:. updated_at is read-only. Objective writes require an available AI key store; an unconfigured or unavailable store returns 503 without changing the objective.

NetIdentityEntry requires name, match and tenant. Exactly one match kind is accepted: cidr, ingress_if, vlan, vni or ipsec_tunnel. An interface match takes precedence over a CIDR match; CIDR matches use the longest prefix; a service-specific binding takes precedence over an otherwise equally specific binding without a service list. Equal-specificity conflicting tenant bindings are refused with 409.

Binding input	Default and effect
`services`	Empty/omitted applies to every AI service; a non-empty list scopes the binding to named VIP:port values.
`allowed_models`	Empty/omitted allows any model within the binding's policy.
`enforce`	`advisory` lets a presented credential decide identity. `strict` also requires its tenant to match the binding and its model to pass the binding's list; a tenant mismatch returns 403 `network_tenant_mismatch`.
`credential`	`none` does not replace a required credential. `network` can admit a request without one only with an interface-based match; CIDR matching is refused for this mode. This makes the interface trust boundary part of admission policy.
`forward_identity`	Default false. True requires an explicit service list and forwards attributed identity while removing client-supplied X-Auth headers on those services. Chunked HTTP/1.1 identity injection is excluded. Incompatible non-AI request/both sockmap rules are refused.
`priority_class`	Default standard; caps the attributed tenant's permitted class. A binding cannot raise the tenant's allowance.
`fail_mode`	Default `neutral` decides requests without the binding if lookup fails, so a strict binding is not enforced during that fault. `closed` refuses affected service requests with 503 `netid_unavailable`. Any closed binding makes its service fail closed for every source.

A source network identifies where traffic arrived, not the person or workload that sent it. Any sender able to use that network can receive its attribution. Use the default credential behavior unless an independently validated interface boundary is explicitly intended to stand in for credentials. Matching interface names can be stored before the interface exists, with a warning; they match no traffic until the interface is available.

API-key and tenant policy schemas also contain priority_class, allowed_objectives and default_objective; JWT profiles contain a priority_class cap. Inspect their complete schema and preserve the intended policy when replacing a row. A priority class is separate from RPS, TPM, endpoint weights and a capacity ceiling.

The load-balancer contract additionally exposes ai_objective_strict and ai_default_reservation / ai_default_max_tokens. Default reservation is on, off or inherit, with a built-in fallback of off; when enabled, the completion default falls back to the process setting or 512 tokens. Numeric zero restores the process default, omission on replace retains the stored value, and explicit null is refused. Read effective reservation values from fc_effective; a default reservation is an estimate, not backend usage. See the generated schema reference for the complete request and response fields. Match the served API and inspect traffic and accounting before treating any configuration as effective runtime policy.