Host your own models

By the end of this guide an open-weight model is answering on an OpenAI-compatible endpoint in your project, running on a GPU, with its weights on storage you control.

For why the path looks like this - what has to fit in GPU memory, and why the first start is slow - see the overview. For the same result from a form rather than these manifests, see the console walkthrough.

What this page deploys

A model server: a Deployment running vLLM and a Service in front of it, in your own project. It holds one model's weights in GPU memory and answers OpenAI-compatible requests on port 8000.

A model server is not a gateway, and the two are configured in different places:

	A model server - this page	A gateway
What runs	vLLM, holding one model's weights	a router, holding no weights
Needs a GPU	yes	no
Answers for	the one model it loaded	every model name you register
Keys and budgets	none - anything that can reach the Service can use the GPU	issued and enforced here

They are layers rather than alternatives, and most projects end up with both: a model server has no authentication of its own, so putting it behind a gateway is step 7.

This page writes those two objects itself. The console walkthrough reaches the same result from a form that writes them for you, and managed model serving replaces both with a single object. Choose this page when you want to own the engine's arguments outright.

Prerequisites

• A project with GPU quota - see Accounts & projects
• oc logged in and pointed at that project
• A Hugging Face token if the model is gated

1. Find a GPU to run on

Check what is available before you request anything - an oversized request schedules nowhere and reports nothing more useful than Pending.

oc get nodes -l nvidia.com/gpu.present=true

For the GPU model and how many are free on a node:

oc describe node <gpu-node> | grep -A5 "Allocatable:\|Allocated resources:"

Read nvidia.com/gpu in both blocks. Allocatable is how many the node has; the allocated figure is how many are already spoken for. GPUs are not oversubscribed - a card is held whole by one pod.

2. Create the project

oc new-project <project>

3. Create a volume for the weights

The engine downloads the model on first start and caches it. Without a volume it downloads again on every restart, which turns a pod restart into a long outage.

model-cache.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: <project> # (1)!
spec:
  accessModes:
  - ReadWriteMany # (2)!
  resources:
    requests:
      storage: 50Gi # (3)!
  storageClassName: berry-storage-file # (4)!

1. Edit this. Your project, from step 2.

2. Check this. ReadWriteMany lets several replicas share one downloaded copy. If your storage class only offers ReadWriteOnce, change it here - each replica then needs its own volume and its own download.

3. Edit this. Weights plus headroom. Roughly two bytes per parameter at 16-bit, so ~15Gi for a 7B model and ~30Gi for a 14B. Downloads are not atomic: a volume that fills part way through leaves a partial cache the engine will not recognise.

4. Name this explicitly. The cluster's default storage class is block, which for a filesystem volume gives you ReadWriteOnce - so a manifest that omits storageClassName gets a volume one node can mount, and the ReadWriteMany above will not be satisfied. berry-storage-file is CephFS and is the one that supports ReadWriteMany. oc get storageclass lists what your cluster offers.

oc apply -f model-cache.yaml

If the model is gated, create the token secret now:

oc create secret generic hf-token -n <project> --from-literal=token=<your-hf-token>

4. Serve the model

A Deployment running vLLM, and a Service in front of it. Everything model-specific is in args:

model-serving.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <model-slug>
  namespace: <project>
  labels:
    app: <model-slug>
spec:
  replicas: 1
  strategy:
    type: Recreate # (1)!
  selector:
    matchLabels:
      app: <model-slug>
  template:
    metadata:
      labels:
        app: <model-slug>
    spec:
      runtimeClassName: nvidia # (2)!
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.15.1 # (3)!
        args:
        - --model=<hf-repo-id> # (4)!
        - --served-model-name=<model-slug> # (5)!
        - --gpu-memory-utilization=0.85 # (6)!
        - --max-model-len=4096 # (7)!
        - --max-num-seqs=6 # (8)!
        - --enforce-eager # (9)!
        - --dtype=half
        ports:
        - name: http
          containerPort: 8000
        env:
        - name: HF_HOME
          value: /data
        - name: VLLM_ENABLE_CUDA_COMPATIBILITY
          value: "0"
        - name: LD_LIBRARY_PATH
          value: /usr/lib64
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
              optional: true
        resources:
          requests:
            cpu: "4"
            memory: 24Gi
            nvidia.com/gpu: 1 # (10)!
          limits:
            cpu: "8"
            memory: 48Gi
            nvidia.com/gpu: 1
        volumeMounts:
        - name: model-cache
          mountPath: /data
        startupProbe: # (11)!
          httpGet:
            path: /health
            port: http
          periodSeconds: 10
          failureThreshold: 180
        readinessProbe:
          httpGet:
            path: /health
            port: http
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: http
          periodSeconds: 20
          failureThreshold: 6
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache
---
apiVersion: v1
kind: Service
metadata:
  name: <model-slug>
  namespace: <project>
spec:
  selector:
    app: <model-slug>
  ports:
  - name: http
    port: 8000
    targetPort: http

1. Leave as is. A rolling update would want two pods holding the same GPU and the same volume at once, and deadlocks. Recreate stops the old pod before starting the new one, which does mean a gap in service on every change.

2. Leave as is. How the GPU is actually made visible inside the container. Without it the card is requested and scheduled, but the engine cannot see it.

3. Tune this. The engine version, pinned on purpose. Which version suits you is a property of the workload and the hardware together, and it moves in both directions: go higher when a model needs an architecture, kernel or quantisation format a newer release added, and lower when a newer engine has dropped or regressed support for an older GPU generation. Check both before changing it. Never use latest - an unrelated pod restart would then change the engine under a model that was working.

4. Edit this. The Hugging Face repository to pull, for example Qwen/Qwen2.5-7B-Instruct. This is what gets downloaded to the cache volume on first start.

5. Edit this. The name callers ask for, deliberately decoupled from the repository id. Choose it once and keep it - it becomes an interface the moment anything depends on it.

6. Tune this. How much of the card the engine may use. What is left after the weights becomes KV cache, which is what limits how many requests you can serve at once. Lower it if something else needs room on the same GPU.

7. Tune this. The maximum context length. The first setting to reduce if the model will not fit - the KV cache grows with it, and it is usually cheaper than changing precision.

8. Tune this. How many requests the engine will hold in flight. Each one keeps its whole conversation in the KV cache, so on a 16GB card this is often the setting that decides whether the model loads at all. Raise it when there is memory to spare.

9. Leave as is on a 16GB card. Skips CUDA graph capture. It costs some throughput and returns the memory those graphs would have held, which is usually the better trade on a V100. Remove it on a larger card.

10. Edit this. How many GPUs this pod holds, and the single most important number here. One GPU needs nothing else. More than one also requires --tensor-parallel-size set to the same number, and a bigger /dev/shm - see Models that need more than one GPU. Raising this without setting tensor parallelism wastes the extra cards: the engine uses one and leaves the rest idle.

11. Leave as is. The probe that makes a cold start survivable. First start downloads tens of gigabytes and loads them onto the GPU; until that finishes the container is not answering, and without a startup probe the liveness probe kills it and restarts the download. At ten seconds a period this allows thirty minutes.

oc apply -f model-serving.yaml

Removing the startup probe restarts the pod during model load

It looks redundant next to the readiness probe, and everything works without it on a model that is already cached. The first cold start on a large model is when it matters, and by then the pod is in a restart loop that looks like a broken image.

The two CUDA variables are what stop the engine crash-looping on Error 803

The image ships CUDA forward-compatibility libraries. Where they lose to the installed driver the engine dies during device init with Unexpected error from cudaGetDeviceCount() ... Error 803: system has unsupported display driver / cuda driver combination, which reads like broken hardware rather than a library path. VLLM_ENABLE_CUDA_COMPATIBILITY=0 and LD_LIBRARY_PATH=/usr/lib64 make it use the driver's own libraries instead. Leave both in unless you know the image and the driver agree.

Models that need more than one GPU

Tensor parallelism splits one model across several GPUs on the same node, each holding a slice of every layer. Three things must agree, and getting one of them wrong is the most common way this fails:

Setting	Must be
nvidia.com/gpu	the number of GPUs
--tensor-parallel-size	the same number
/dev/shm	a real volume, sized with the model

Rules that decide whether it will run at all:

• The GPUs must be on one node. Tensor parallelism is not distributed across machines. If no single node has that many free, the pod stays Pending.
• The value must divide the model's attention heads exactly. Powers of two - 2, 4, 8 - are the safe choices. An awkward number fails at load with a shape error.
• It is a way to fit a model, not to speed one up. Every token costs communication between the cards. A model that already fits on one GPU runs slower on two.

A rough starting point, at 16-bit precision:

Model size	GPUs	--tensor-parallel-size	shmSize
up to ~7B	1	not needed	not needed
~7B to ~14B	2	2	4Gi
~14B to ~32B	4	4	8Gi

Weights are roughly two bytes per parameter at 16-bit, so divide that by the number of cards and leave room for KV cache on each. Quantising lets a model drop a tier.

Raise the GPU limit, add the tensor-parallel argument, and give NCCL real shared memory - the default /dev/shm is 64MB and the symptom is a hang at startup, not an error:

model-serving.yaml (excerpt)

        args:
        - --model=<hf-repo-id>
        - --served-model-name=<model-slug>
        - --tensor-parallel-size=2 # (1)!
        - --gpu-memory-utilization=0.90
        - --max-model-len=4096
        - --dtype=half
        resources:
          limits:
            nvidia.com/gpu: 2 # (2)!
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory # (3)!
          sizeLimit: 4Gi

1. Must equal the GPU count below. Raising one without the other leaves the pod Pending.

2. All on one node. The pod stays Pending if no single node has this many free.

3. medium: Memory is what makes this a RAM-backed volume rather than disk. Without it the shared memory NCCL needs is on the filesystem and the collective operations crawl.

All the GPUs must be on one node. --tensor-parallel-size has to divide into the GPU count exactly, and the pod stays Pending if no single node has that many free.

5. Wait for it to load

oc rollout status deploy/<model-slug> -n <project> --timeout=30m

Watch the download and load happen rather than guessing:

oc logs -f deploy/<model-slug> -n <project>

The engine logs the weight download, then the model load, then that it is serving on port 8000. Until that last line, the endpoint is not up - see the overview.

6. Check it answers

From inside the cluster:

oc run -n <project> curl-test --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sS http://<model-slug>:8000/v1/models

Then a completion, which exercises the GPU rather than just the process:

oc run -n <project> curl-test --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sS http://<model-slug>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-slug>","messages":[{"role":"user","content":"Say hello"}]}'

Confirm which model loaded before you trust a result

/v1/models reports what is being served. Check it matches the model and precision you intended before you benchmark anything or put it behind a name callers depend on.

7. Put it behind the gateway

Register the Service URL as a provider so callers reach it by name and get budgets and logging - see Add a model provider. The endpoint to register is the in-cluster address:

http://<model-slug>.<project>.svc.cluster.local:8000/v1

Keep the model reachable only inside the cluster and let the gateway be the front door. There is no authentication on the engine itself - anything that can reach the Service can use the GPU.

List

oc get deploy,svc,pvc -n <project>
oc get pods -n <project> -o wide

The -o wide matters here: it shows which node the pod landed on, which is which GPU.

Change

Scale to zero to release the GPU without deleting anything. The weights stay on the PVC, so coming back is a load rather than a download:

oc scale deploy/<model-slug> -n <project> --replicas=0

Change an engine argument - context length, memory fraction, precision:

oc set env deploy/<model-slug> -n <project> --list     # see current env
oc edit deploy/<model-slug> -n <project>               # edit args

Any change to args restarts the pod and reloads the model. It is not a live setting.

Grow the cache if a larger model has to fit:

oc patch pvc/model-cache -n <project> \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

Only works if the storage class allows expansion, and it cannot shrink.

	Changeable in place
Replica count	yes
Engine arguments	yes - restarts and reloads
PVC size	grow only, if the storage class allows
GPU count	yes - but it reschedules, and may not fit
The model itself	serve a second Deployment and move traffic

Troubleshoot

Each symptom below rules out the one above it, so work down the list. Open the one that matches what you are seeing.

Pod stays Pending

No node has what you asked for.

oc describe pod -l app=<model-slug> -n <project> | tail -20

Insufficient nvidia.com/gpu means the GPUs are held by something else, including a Deployment of yours scaled to zero but not deleted. didn't match Pod's node affinity means your selector excludes every node that has a card.

Pod restarts before it ever serves

Almost always the missing startup probe, or one whose threshold is too low for a cold download. Check how far it got:

oc logs deploy/<model-slug> -n <project> --previous

Error 803: system has unsupported display driver / cuda driver combination

Raised from cudaGetDeviceCount() during device init, before any weight is loaded, and it repeats on every restart. The image's CUDA forward-compatibility libraries are losing to the installed driver. Set VLLM_ENABLE_CUDA_COMPATIBILITY=0 and LD_LIBRARY_PATH=/usr/lib64 as in step 4.

The root cause is in the EngineCore lines, not the APIServer traceback that follows them and is longer:

oc logs deploy/<model-slug> -n <project> --previous | grep EngineCore

CUDA out of memory during load

The weights plus KV cache do not fit. Lower --gpu-memory-utilization, lower --max-model-len, use a smaller precision, or add a GPU with --tensor-parallel-size.

Reducing --max-model-len is usually the cheapest fix, because the KV cache is what grew.

It hangs at startup with more than one GPU

/dev/shm is too small for NCCL. Add the dshm volume from Models that need more than one GPU.

It loads but is slow

Confirm it finished loading before measuring. Then check you are not sharing the node's GPUs with another workload:

oc get pods -A -o wide --field-selector spec.nodeName=<gpu-node>

403 or 401 pulling weights

The model is gated and the token is missing or lacks access. The optional: true on the secret reference means a missing secret fails at download rather than at pod start, which is why this shows up in the logs and not in describe.

Delete

Reverse order. The PVC does not go with the Deployment, and holding weights costs storage:

oc delete deploy/<model-slug> -n <project>
oc delete svc/<model-slug> -n <project>
oc delete pvc/model-cache -n <project>

Or remove the project outright:

oc delete project <project>

Remove the provider from the gateway first

A gateway still routing a model name at a deleted Service fails on every request to that name. Take the registration out before the workload - see Add a model provider.

Next

Add a model provider → - put this endpoint behind a name your applications can use.