Skip to content

Managed model serving

An LLMInferenceService is one object that describes the model you want served. The platform builds the rest: the model server, an inference-aware router in front of it, and the routing that publishes it.

This is the alternative to assembling that stack yourself. Both end in a vLLM process answering OpenAI-compatible requests; they differ in how much of the arrangement you own.

Which path to take

Managed - this page Manual
You write one object a PVC, a Deployment and a Service
Weights fetched for you from a uri you mount a volume and let the engine download
Routing an inference-aware router, built for you a Service, round-robin
Engine version and flags defaults, overridable entirely yours
Use it when you want the model served you are tuning the engine itself

The managed path is the better default. Reach for the manual one when you need control over an engine flag the managed path does not expose, or when the hardware constraint below applies.

Prerequisites

Serve a model

llm.yaml
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: <name>
  namespace: <project>
spec:
  replicas: 1
  model:
    uri: hf://<hf-repo-id> # (1)!
    name: <model-name> # (2)!
  router:
    route: {} # (3)!
    gateway: {}
    scheduler: {}
  template:
    containers:
    - name: main
      resources:
        limits:
          cpu: "8"
          memory: 24Gi
          nvidia.com/gpu: 1 # (4)!
        requests:
          cpu: "4"
          memory: 12Gi
          nvidia.com/gpu: 1

  1. Edit this. Where the weights come from. hf:// fetches from Hugging Face in an init container before the engine starts, so a failure here is visible as a failed init container rather than an engine that will not load. s3://, pvc:// and oci:// are also accepted.

  2. Edit this. The name callers ask for. It is decoupled from the repository id, and it becomes an interface the moment anything depends on it.

  3. Leave as is. The three empty objects are not placeholders - each asks the platform to build that piece with its defaults. scheduler: {} is what gets you the inference-aware router; omit it and requests are distributed without regard to which replica already holds the conversation.

  4. Edit this. One GPU per replica. The same fit rules apply as on the manual path - see what has to fit in GPU memory.

oc apply -f llm.yaml

What it builds

One object becomes several, and knowing the names makes the failures readable:

Created What it is
<name>-kserve the Deployment running the engine
<name>-kserve-workload-svc the Service in front of it, on port 8000 over HTTPS
<name>-kserve-router-scheduler the router that picks which replica serves a request
<name>-inference-pool the pool the router chooses from
<name>-kserve-route the route publishing it through the cluster's inference gateway

Verify

oc get llminferenceservice <name> -n <project>

Ready becomes true only when the workload, the router, the pool and the route are all ready. Read the individual conditions rather than the summary, because they fail for unrelated reasons:

oc get llminferenceservice <name> -n <project> -o jsonpath='{.status.conditions}' | jq

WorkloadsReady is about the engine. RouterReady and HTTPRoutesReady are about publishing, and can be false while the model is perfectly healthy and serving inside the cluster.

The Service speaks HTTPS with a certificate the platform generates, so a check from inside the cluster has to allow it:

oc exec -n <project> deploy/<name>-kserve -- \
  curl -sSk https://localhost:8000/v1/models

The model name in the response is the one callers must use

/v1/models returns the name you set, not the Hugging Face repository id. A request naming the repository id is rejected even though the weights are correct.

The gateway has to exist

The route is published through one gateway shared by the cluster, named in the KServe configuration. If that gateway has not been created, every LLMInferenceService fails the same way regardless of how it is written.

Managed HTTPRoute references non-existent Gateway

The condition names the gateway it wanted:

RouterReady=False
  Managed HTTPRoute references non-existent Gateway openshift-ingress/openshift-ai-inference

The workload still starts and still serves inside the cluster - only publishing fails. That is why the model can answer while the object never goes Ready.

Which gateway is expected comes from kserveIngressGateway in the inferenceservice-config ConfigMap. Creating a Gateway of that name, with allowedRoutes.namespaces.from: All so tenant projects may attach to it, resolves it for every project at once. It is a cluster-level fix, not something to work around per model.

The engine image has to match the GPU

The default image is built for recent GPU generations. On an older card the engine fails during device initialisation, before any weight is loaded.

CUDA error: no kernel image is available for execution on the device

The image contains no compiled kernels for that GPU architecture. No amount of configuration fixes it, because nothing in the configuration is wrong.

Either serve the model on a newer card, or override the image with one built for the architecture you have:

  template:
    containers:
    - name: main
      image: <vllm-image>
      env:
      - name: VLLM_ENABLE_CUDA_COMPATIBILITY
        value: "0"
      - name: LD_LIBRARY_PATH
        value: /usr/lib64

The two variables are the same ones the manual path needs, and for the same reason - see the CUDA note. An override is worth removing once the hardware is replaced, since it pins you to an engine version the platform is no longer choosing.

Delete

oc delete llminferenceservice <name> -n <project>

Everything it built goes with it. Weights fetched from a uri are not cached anywhere you own, so the next start downloads them again.

Next

Add a model provider → - put the endpoint behind a model name on the gateway, so callers reach it by name and with a key.