Skip to content

AI inference

Serve open-weight models from infrastructure in the Netherlands, on the same clusters as the rest of your workloads. A served model is an ordinary workload in your project — a Deployment running an inference engine, with a Service in front of it, scheduled by the same control plane and secured by the same RBAC as any other Kubernetes object.

A served model differs from your other workloads in one respect: it holds a large amount of state in GPU memory, and that state determines almost everything about how it behaves. See Self-hosted models for the detail.

How it fits together

flowchart LR
    CLIENT["Your application"] --> GW["LLM gateway<br/>keys · budgets · routing"]
    GW --> ISVC["Model server<br/>vLLM on a GPU"]
    GW --> EXT["External providers"]
    ISVC --> GPU["NVIDIA GPU nodes"]
    ISVC --> PVC["Model weights<br/>on platform storage"]

Most teams put the LLM gateway in front of inference rather than calling endpoints directly — it centralises keys, budgets and failover across self-hosted and external models alike. A model server has no authentication of its own, so anything that can reach its Service can use the GPU.

Inference engines

Two engines are in use on the platform, both speaking the OpenAI API:

Engine Use it for
vLLM the default choice; widest model and quantisation coverage
SGLang workloads that benefit from its scheduler, notably structured output

Neither engine is installed for you. You run it as your own workload, so the engine version and its arguments are yours to pin in Git.

The Deployment manifest

Everything model-specific lives in args. Substitute your own values for the <angle-brackets>:

model-serving.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <model-slug>
  namespace: <your-project>
spec:
  replicas: 1
  strategy:
    type: Recreate # (1)!
  selector:
    matchLabels:
      app: <model-slug>
  template:
    metadata:
      labels:
        app: <model-slug>
    spec:
      runtimeClassName: nvidia # (2)!
      containers:
      - name: vllm
        image: vllm/vllm-openai:<version> # (3)!
        args:
        - /models/<org>/<model> # (4)!
        - --served-model-name
        - <org>/<model> # (5)!
        - --host
        - 0.0.0.0
        - --port
        - "8000"
        - --dtype
        - half
        - --tensor-parallel-size
        - "1"
        - --gpu-memory-utilization
        - "0.85" # (6)!
        - --max-model-len
        - "4096"
        ports:
        - name: http
          containerPort: 8000
        resources:
          requests:
            cpu: "4"
            memory: 24Gi
            nvidia.com/gpu: 1 # (7)!
          limits:
            cpu: "8"
            memory: 48Gi
            nvidia.com/gpu: 1
        volumeMounts:
        - name: models
          mountPath: /models
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: <model-pvc> # (8)!

  1. Leave as is. A rolling update would want two pods holding the same GPU and the same volume at once, and deadlocks. Recreate stops the old pod first, which does mean a gap in service on every change.

  2. Leave as is. What actually makes the card visible inside the container. Without it the GPU is requested and scheduled, and the engine still cannot see it.

  3. Pin this. The engine version. Which one suits you depends on the model and the card together — see Choosing an engine version. Never latest.

  4. Edit this. Where the weights are on the mounted volume. Point it at a Hugging Face repository id instead and the engine downloads on first start, which is slower and repeats on every restart without a cache.

  5. Edit this. The name callers ask for, deliberately decoupled from the path above. Choose it once and keep it — it becomes an interface the moment anything depends on it.

  6. Tune this. How much of the card the engine may use. Whatever is left after the weights becomes KV cache, which is what limits how many requests you can serve at once.

  7. Edit this, in both places. GPUs are not oversubscribed — a card is held whole by one pod — so requests and limits must carry the same number. More than one also needs --tensor-parallel-size set to match.

  8. Edit this. The claim holding the weights. It is not deleted with the Deployment, so the model reloads rather than re-downloads when you come back.

Choosing an engine version

There is no single right version, and the platform's own workloads deliberately run several at once. Which one suits you is a property of the model and the hardware together: a newer release may add the architecture, kernel or quantisation format your model needs, while an older one may be the last that performs well on an older card.

Pin it explicitly. latest turns an unrelated pod restart into a version change under a model that was working.

Sizing for the GPU

GPU workers are NVIDIA Tesla V100-SXM2 with 16GB of memory per card. That ceiling drives most sizing decisions, because weights, KV cache and overhead all share it.

Setting What it does
--dtype half 16-bit weights, roughly two bytes per parameter
--gpu-memory-utilization fraction of the card the engine may use; what is left after weights becomes KV cache
--max-model-len maximum context; the cheapest thing to reduce when a model will not fit
--max-num-seqs concurrent sequences; lower it when the KV cache is the constraint
--enforce-eager skips CUDA graph capture, trading some throughput for the memory it would hold

--max-num-seqs and --enforce-eager are the two most effective levers on a 16GB card. Set both before switching to a smaller model.

More than one GPU. Tensor parallelism splits one model across several cards in the same node. --tensor-parallel-size must equal the nvidia.com/gpu count, the value must divide the model's attention heads exactly — powers of two are the safe choices — and /dev/shm needs a real RAM-backed volume, because the 64MB default makes NCCL hang at startup rather than fail. All the cards must be on one node; the pod stays Pending otherwise.

Storing model weights

Stage weights on a volume rather than downloading them on every restart — a pod restart otherwise becomes a long outage.

Storage class Backing Access modes Use for
berry-storage-file CephFS RWO, RWX model weights shared by several replicas
berry-storage-block Ceph RBD RWO a single-replica cache

ReadWriteMany on berry-storage-file lets several replicas share one downloaded copy. With a ReadWriteOnce class each replica needs its own volume and its own download.

Size the claim for the weights plus headroom — roughly two bytes per parameter at 16-bit, so about 15Gi for a 7B model and 30Gi for a 14B. Downloads are not atomic: a volume that fills part way through leaves a partial cache the engine will not recognise.

Connecting. The engine serves the OpenAI API on port 8000:

What Where
In-cluster endpoint http://<model-slug>.<your-project>.svc.cluster.local:8000/v1
Model list GET /v1/models
Chat completions POST /v1/chat/completions
Health GET /health

Keep the model reachable only inside the cluster and let the gateway be the front door — there is no authentication on the engine itself.

Deleting. Removing the Deployment releases the GPU; removing the Service takes the endpoint away. The PVC holding the weights does not go with either, and continues to cost storage until you remove it too. Take the model out of the gateway before removing the workload, or every request to that model name fails.

Readiness does not confirm which model loaded

Readiness means the engine loaded weights and opened a port. It does not mean it loaded the weights you intended, at the precision you intended. Query /v1/models to confirm which model is being served before you trust a benchmark or a bill.

How to

Task Guide
Serve an open-weight model on a GPU Self-hosted models
Put it behind a model name Model providers
Give a caller scoped access Virtual keys