Skip to content

Model providers

You want the gateway to offer a model it does not yet know about - a new external provider, or an endpoint you host yourself. To the gateway both are the same kind of thing: a backend that speaks a known API, reachable at a URL, holding a set of models.

Registering one is what turns a model name into something a caller can ask for.

Prerequisites

  • A running gateway - see Deploy an LLM gateway
  • Credentials for the provider, or the URL of a self-hosted endpoint

The step-by-step guide and the console walkthrough cover the operations. The rest of this page explains how it fits together.

A provider is a backend, not a company

flowchart LR
    CALL["caller asks for<br/>a model name"] --> GW["gateway"]
    GW --> R{"routing"}
    R -->|primary| SELF["self-hosted endpoint<br/>your GPUs, your project"]
    R -->|fallback| EXT["external provider<br/>credential held by the gateway"]
    SELF -.->|unhealthy| R

The word covers three things that look different from outside and identical from the gateway's side:

Backend Credential Where it runs
A commercial model API provider key someone else's infrastructure
A model you host usually none - in-cluster your project, on GRN GPUs
Another gateway its virtual key wherever you put it

All three are a base URL, an API dialect and a list of models. That uniformity is why a caller can be moved from an external provider to your own hardware without touching the caller.

Routing and failover are a cost decision

Once a model name has more than one backend, you have to say which one is tried first and what happens when it fails. The common shape is a self-hosted primary with a commercial fallback: you pay for the GPUs you already own, and you spill to a paid API only when your own capacity is unavailable.

The trade to make deliberately is what failover means for the request. Falling back keeps the caller working and moves spend to the expensive backend, silently. Not falling back gives the caller an error it can see. Both are defensible; the failure mode of the first is a surprising invoice, and of the second an outage.

Failover makes a failing primary look like a working system

A primary that is failing intermittently still returns successful requests, at the fallback provider's price. If you configure failover, alert on how often it fires, not only on whether requests succeed.

Model names are the contract

The name a caller uses is not the name the backend uses. The mapping between them lives in the gateway, which is exactly why the indirection is useful - and exactly why the public name is the part you cannot casually change.

  • Choose a name for what it does, not who supplies it. A name that encodes the vendor has to change when the vendor does, which defeats the purpose.
  • Keep names stable across backends. Moving a name from an external provider to a self-hosted endpoint should be invisible to callers.
  • Retire, do not rename. Add the new name, move callers, then remove the old one.

What is fixed after creation

Changeable
The backend URL and credential yes
Which models a provider offers yes
Routing order and failover yes
The public model name, once callers use it not safely - add a new one instead

Provider credentials never belong in a manifest

A provider key goes into a Secret the gateway reads at runtime, not into a ConfigMap, a Helm values file or anything committed to Git. It is the credential the gateway exists to keep away from callers.