Skip to content

Self-hosted models

You want an open-weight model served from GPUs in the Netherlands rather than called over someone else's API - because the prompts must not leave the platform, because the volume makes a per-token price worse than hardware, or because you need a model no API sells.

A served model is a workload in your project like any other: it schedules on a node, reads its weights from storage, and answers on a Service. What makes it unlike your other workloads is that it holds a large amount of state in GPU memory, and that state decides almost everything about how it behaves.

Prerequisites

Three pages cover the operations, and they differ in how much of the arrangement you own: the console walkthrough fills in a form, managed serving writes one object, and the step-by-step guide builds the workload yourself. The rest of this page explains how it fits together.

Self-hosted or bought

This is a cost and capability question, not a compliance one. Prompts sent to a model on GRN GPUs stay on the platform either way; the gateway lets you mix both behind one endpoint, so it is not a decision you make once for everything.

Self-hosted External API
Cost shape fixed - you pay for the GPU idle or busy per token
Cheaper when utilisation is high and steady traffic is spiky or low
Model choice any open-weight model, including ones you fine-tune what the vendor sells
Data path stays on the platform leaves it
Capacity ceiling the GPUs you have the vendor's rate limits
You operate the serving stack nothing

The break-even is utilisation. A GPU costs the same whether it serves one request an hour or thousands, so a lightly used self-hosted model is the most expensive way to buy tokens.

What has to fit in GPU memory

flowchart LR
    REQ["more concurrent requests<br/>longer contexts"] -->|grows| KV
    subgraph GPU["one GPU's memory"]
        direction TB
        W["model weights<br/>fixed - set by<br/>parameters and precision"]
        KV["KV cache<br/>grows with concurrency<br/>and context length"]
        OV["activations and overhead"]
    end

Three things share the card, and only one of them is constant.

Weights are fixed once you choose the model and its precision. Roughly two bytes per parameter at 16-bit, half that at 8-bit, a quarter at 4-bit. This is the floor: a model whose weights do not fit cannot be served on that GPU at that precision.

KV cache is the part people are surprised by. Every token in every in-flight conversation holds state, so the cache grows with how many requests you serve at once and how long their contexts are. Whatever is left after weights is what limits your concurrency - two models with identical weights can have very different throughput because one left more room than the other.

Activations and overhead take the remainder.

The consequence worth internalising: serving more users is a memory problem before it is a compute problem. If throughput is disappointing, the cache is usually the thing that ran out.

Quantisation frees memory for KV cache, not only for weights

Serving at lower precision shrinks the weights, and every byte saved becomes KV cache. It often raises the number of concurrent requests more than it lowers quality.

When a model does not fit on one GPU

Tensor parallelism splits a single model across several GPUs in the same node, each holding a slice of every layer. It makes larger models possible, at the cost of communication between the cards on every token - so it is a way to fit a model, not a way to make a model that already fits go faster.

Storing model weights

Weights go on their own volume, and the access mode decides how many replicas can serve from one copy. A ReadWriteOnce volume can be mounted by one node, so each replica needs its own copy and its own download; a ReadWriteMany volume is mounted by all of them, so the model is downloaded once no matter how many replicas serve it - or how many separate deployments read the same weights.

The cluster's default storage class is block, which is ReadWriteOnce for a filesystem volume. That is the right default for most things and the wrong one for shared weights, so a weights volume names a ReadWriteMany class explicitly rather than taking the default. A 250Gi model is worth getting right once.

The first start is slow

Before it answers anything, a serving pod pulls its weights - tens of gigabytes for a large model - and loads them onto the GPU. A model that has not finished loading looks slow rather than broken.

Check readiness before you measure anything, and stage weights on storage close to the cluster rather than pulling them from the internet on every restart.

Readiness does not confirm which model loaded

Readiness means the runtime loaded weights and opened a port. It does not mean it loaded the weights you intended, at the precision you intended. Ask the endpoint which model it is serving before you trust a benchmark or a bill.

What is fixed after creation

Changeable
Replica count yes
The public name callers use, via the gateway yes
The model or its precision no - serve a new endpoint and move traffic
GPU type and count no - it is a scheduling constraint, recreate to change