Console walkthrough: host your own models

The OpenShift AI dashboard deploys a model from a form. Submitting it writes two objects into your project: an InferenceService, which names the deployment and says where the weights come from, and a ServingRuntime, a private copy of a runtime template that fixes the engine image and the arguments it starts with.

Both are the same kinds the step-by-step guide builds by hand. The form's value is not that it hides them - it is that the runtime template already carries the settings that are easy to get wrong and that fail without naming themselves: the engine image built for the GPU you have, and the probe timings that survive a long first load.

Which path this is

The dashboard offers two deployment methods, and they produce different objects.

Deployment method	Writes	Use it when
Legacy deployment	InferenceService + ServingRuntime	you need a specific runtime template - including the ones built for this platform's GPUs
LLM inference service with llm-d	LLMInferenceService	you want the router and scheduler described in managed model serving

"Legacy" names the object model, not a deprecation, and it is the default here because the runtime templates that work on the platform's GPUs are ServingRuntime templates. This page follows it. For the other method, managed model serving covers the same result written as one object.

Prerequisites

• Membership of the group that grants dashboard access - ask your administrator if the dashboard reports no projects
• A project with GPU quota - see Accounts & projects
• Nothing installed locally. This flow needs no oc and no kubeconfig.

1. Open the project

Projects lists the projects you can see. Opening one and choosing Deployments shows what it is serving.

The Deployment resource column names the object behind each row, so a project mixing both deployment methods still reads unambiguously.

2. Say where the weights come from

Deploy model opens a four-step wizard. The first step asks two questions.

Create a connection to this location is ticked by default; while it is, the step needs a connection name before Next becomes available.

Model location is where the weights are fetched from - URI, S3 object storage or OCI compliant registry. A hf://<repo-id> URI pulls from Hugging Face into the pod before the engine starts, so a wrong repository id fails in an init container rather than as an engine that will not load.

Model type splits generative models from predictive ones, and decides which runtimes the next step offers.

Leaving Create a connection to this location ticked saves the location as a reusable Connection in the project, worth doing for a bucket you will deploy from repeatedly and not for a one-off.

3. Choose the hardware and the runtime

The second step is where a deployment is made to fit the hardware.

The hardware profile prints its own defaults and maxima underneath, so the GPU count a replica will request is visible before you submit.

Model deployment name becomes the name of the InferenceService, and of the endpoint callers reach.

Hardware profile picks the GPU. Each profile pins the pod to nodes that have that card, and sets how many GPUs a replica requests. Read the line under the picker: it is the profile, not the model, that decides the request.

Serving runtime template picks the engine. The platform's own templates name the GPU they were built for; the general-purpose ones are built for recent GPU generations and fail during device initialisation on older cards, before any weight is read.

Automatic selection is offered beside it but is not selectable, so the template is always an explicit choice. That is the right way round here - automatic selection matches a runtime to the hardware profile in general, and the pairing that matters on this platform is the one described in the warning below.

The runtime template has to match the GPU in the hardware profile

Nothing in the form prevents pairing a profile with a runtime built for a different GPU architecture, and the result is not a validation error - it is a pod that starts, fails inside the engine and reports CUDA error: no kernel image is available for execution on the device. Pair a profile with the template that names the same card.

Replica count is how many copies serve the endpoint. Each holds its own GPUs for as long as it runs.

4. Set access and arguments

The third step covers everything that is not the model or the hardware.

Make model deployment available through an external route and Require token authentication are independent, which is what makes the combination below possible.

Model access publishes the endpoint outside the cluster. Token authentication requires a token on requests.

An external route without token authentication is open to anything that can reach it

A model server has no authentication of its own. Publishing the route without requiring a token puts an unmetered GPU on the network. Either leave the route internal and put the endpoint behind a gateway, which is where keys and budgets are enforced, or require a token here.

Configuration parameters adds runtime arguments and environment variables on top of the template's own. This is the escape hatch for a flag the template does not set.

Deployment strategy decides what a change does to service: Rolling update starts the new pod before stopping the old one, which needs GPU headroom for both at once; Recreate stops first, which does not, and leaves a gap.

5. Review and deploy

The last step restates the whole submission, including the runtime and profile it resolved.

Model format and Serving runtime are derived rather than chosen, so this is the first place they appear together.

Deploy model submits, and returns to the project's Deployments list.

6. Watch it start

The deployment appears in the list immediately and reports Starting while the weights are fetched and loaded. The row carries the two facts that decide whether it can work at all - the serving runtime it resolved, and the hardware profile it will schedule against.

Deployment resource names the runtime the deployment resolved to, so the row is where you confirm the engine matches the card before reading any failure deeper down.

Starting is not a short state. The weights are downloaded before the engine begins loading them, and the engine reports nothing on the endpoint until both finish, so a large model can sit at Starting for a long time without anything being wrong. See the first start is slow.

The status column reports the deployment, not the model. For what the engine is actually doing - which backend it chose, which precision it settled on, how far the download has got - read the pod's log in the OpenShift console, as step 4 of the guide describes.

7. Check it answers

Ready means the engine loaded weights and opened a port. It does not tell you which model loaded, so ask the endpoint before trusting it.

The dashboard has no request console, so this step leaves the browser. Inference endpoints on the row gives the address; from inside the cluster:

oc exec -n <project> deploy/<deployment-name>-predictor -c kserve-container -- \
  curl -sS http://localhost:8080/v1/models

The model name callers must use is the deployment name, not the repository id

/v1/models returns the name of the deployment, not the hf:// repository the weights came from. A request naming the repository id is rejected even though the weights are exactly the ones you asked for.

The same endpoint serves chat completions, and max_model_len in the /v1/models response is worth reading at this point - it is chosen at load time from what fits on the card, not from anything you set in the form.

Change

Edit on the row reopens the wizard on an existing deployment. Replica count, the runtime arguments and the advanced settings are all editable; each change restarts the pod, and the deployment strategy chosen in step 4 decides whether that is a gap in service or an overlap.

Stop scales the deployment to zero without deleting it. It keeps the object, the name and the settings, and releases the GPU - which is what you want for a model you use intermittently, because a stopped deployment costs no GPU while an idle one costs the same as a busy one.

The model and the hardware profile are not edits in any useful sense: changing either replaces what is running. See what is fixed after creation.

Delete

Delete on the row removes the deployment.

The form writes a ServingRuntime per deployment rather than sharing one, so a deployment is two objects, not one. That matters if you remove it from the command line instead: deleting the InferenceService on its own leaves its ServingRuntime behind, holding no resources but cluttering the project until it is deleted too.

Weights are not cached anywhere you own - the next deployment fetches them again, so a delete always costs the download a second time.

Where the console differs from the CLI

	Dashboard	oc
Creating the objects	a four-step form	oc apply -f
The engine image and its arguments	from a runtime template	yours to write, and to get right
Choosing the GPU	a hardware profile	nodeSelector and resource limits
Watching the load	a status column	oc logs -f
Reading why it failed	the pod log, in the OpenShift console	oc logs, oc describe
Releasing the GPU without deleting	Stop	scale the Deployment to zero
Calling the endpoint	not possible in the dashboard	oc port-forward, then curl

Next

Add a model provider → - put this endpoint behind a model name on the gateway, so callers reach it by name and with a key.