Add a model provider¶

By the end of this guide the gateway has a new backend registered - an external provider or an endpoint you host - reachable by callers under a model name that does not change when the backend does.

For what a provider is and why the model name is the part you cannot casually change, see the overview. For the same thing in the console, see the console walkthrough.

Prerequisites¶

A running gateway - see Deploy an LLM gateway
Either credentials for an external provider, or the in-cluster URL of a model you host

1. Store the credential¶

External provider keys go in a Secret the gateway reads at runtime. Never in a values file, never in a ConfigMap, never in Git.

oc create secret generic gateway-provider-secrets -n <gateway-project> \
  --from-literal=<provider>-api-key=<key>

Adding a provider to a Secret that already exists:

oc set data secret/gateway-provider-secrets -n <gateway-project> \
  --from-literal=<provider>-api-key=<key>

Then declare how the gateway reads it:

values.yaml (excerpt)

bifrost:
  providerSecrets:
    <provider>: # (1)!
      envVar: <PROVIDER>_API_KEY # (2)!
      existingSecret: gateway-provider-secrets
      key: <provider>-api-key # (3)!

Edit this. The provider's own identifier, as the gateway knows it - openai, groq, openrouter. It has to match the key you use under providers: below.
Edit this. The environment variable the credential is exposed as inside the pod. Whatever you put here is what value: env.<NAME> refers to in the next step.
Edit this. Which key inside the Secret holds this provider's credential. One Secret can carry several, one key per provider.

Create the Secret before you apply the provider config

The provider block below references the environment variable, not the literal. If the config applies while the Secret is missing, the gateway starts with an unresolved reference and the failure appears on the first request rather than at apply time.

2. Register the backend¶

An external provider¶

values.yaml (excerpt)

bifrost:
  providers:
    <provider>:
      keys:
      - name: <provider>-key # (1)!
        value: env.<PROVIDER>_API_KEY # (2)!
        weight: 1 # (3)!
        models:
        - <provider-model-id> # (4)!

Edit this. A label for this credential, used in logs and when several keys serve the same provider. It is not the credential itself.
Leave the env. prefix. This is the indirection that keeps the real key out of the values file - it names the variable from step 1, not a secret value.
Tune this. How traffic is split when more than one key answers for the same model name. With a single key the number is irrelevant; with two it is the ratio between them.
Edit this. The model ids this credential may serve, spelled exactly as the provider spells them. A caller asking for a name not listed here is refused before the request leaves the gateway.

A model you host¶

No credential - a URL. In-cluster backends need private networking allowed explicitly:

values.yaml (excerpt)

bifrost:
  providers:
    vllm:
      keys:
      - name: local-<model-slug>
        value: "" # (1)!
        weight: 1
        models:
        - <hf-repo-id> # (2)!
        vllm_key_config:
          model_name: <hf-repo-id>
          url: http://<service>.<project>.svc.cluster.local:8000 # (3)!
      network_config:
        allow_private_network: true # (4)!

Leave empty. A model you host in your own cluster needs no credential - reachability is the control, not a key. Anything that can reach the Service can use the GPU, which is why the engine stays cluster-internal and the gateway is the front door.
Edit this. The name callers will ask for. It has to match model_name below, and it is what your applications depend on - see Model names are the contract.
Edit this. The in-cluster address of the Service from Host your own models. Confirm it resolves with oc get svc -n <project> before applying - a wrong name here is accepted at apply time and fails on the first request.
Required for in-cluster backends. Without it the gateway refuses to dial a private address, and the symptom is a connection error that looks like the model is down.

The URL is the Service from Host your own models. Confirm it resolves before trusting the registration:

oc get svc -n <project>

Prefer a router service over a single engine

Where your serving stack puts a router in front of several engines, register the router. Pointing the gateway straight at one engine's Service couples it to that replica's naming, and it breaks when the engine is renamed or scaled.

3. Map model names¶

models is the list of names this backend answers to. That name is what callers ask for, so it is an interface, not a label.

Name for what the model does, not who supplies it. A name carrying a vendor has to change when the vendor does, which defeats the point of the gateway.
Keep a name stable when you move it between backends.
Retire rather than rename: add the new name, migrate callers, then remove the old one.

4. Set routing and failover¶

Several keys under one provider are tried according to weight. Raising a weight sends more traffic to that backend:

values.yaml (excerpt)

      keys:
      - name: local-primary
        weight: 10
        # ...
      - name: external-fallback
        weight: 1
        # ...

The common arrangement is a self-hosted primary with an external fallback: you use GPUs you already own, and spill to a paid API only when your own capacity is exhausted.

Failover makes a failing primary look like a working system

A primary that is failing intermittently still returns successful requests, at the fallback provider's price. Alert on how often failover fires, not only on whether requests succeed.

5. Apply¶

oc apply -k <gateway-config-path> --enable-helm
oc rollout status deploy/<gateway> -n <gateway-project>

Verify¶

The gateway restarted cleanly and parsed the config:

oc logs deploy/<gateway> -n <gateway-project> --tail=50

A bad URL or an unresolved secret reference shows up here, not at apply time - the config is accepted and the backend fails on first use.

The new name is offered:

curl -sS <base-url>/models -H "Authorization: Bearer <your-virtual-key>"

It completes:

curl -sS <base-url>/chat/completions \
  -H "Authorization: Bearer <your-virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-name>","messages":[{"role":"user","content":"ping"}]}'

For a self-hosted backend, call the Service directly first. That separates a broken model from a broken registration:

oc run -n <project> curl-test --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sS http://<service>.<project>.svc.cluster.local:8000/v1/models

Failover works - make the primary unavailable and confirm traffic moves without the caller changing anything:

oc scale deploy/<model-slug> -n <project> --replicas=0

Then repeat the completion call, and scale it back.

List¶

curl -sS <base-url>/models -H "Authorization: Bearer <your-virtual-key>"

That is what callers see. The configuration file is what you edit; these should agree, and a difference means the gateway has not reloaded.

Change¶

Rotate a provider credential - update the Secret, then restart so it is re-read:

oc set data secret/gateway-provider-secrets -n <gateway-project> \
  --from-literal=<provider>-api-key=<new-key>
oc rollout restart deploy/<gateway> -n <gateway-project>

Repoint a model name at a different backend by changing the url or moving the entry. Do this only for a name no caller depends on yet; otherwise add a new name and migrate.

Change weights to shift traffic. This takes effect on reload and needs no caller change.

Delete¶

Find the callers before removing anything:

grep -rn "<model-name>" .

Then remove the provider block and apply. If the model name is used by anything, remove the callers first - a gateway routing a name at a backend that no longer exists fails every request to that name.

Clean up the credential once no provider block references it:

oc patch secret/gateway-provider-secrets -n <gateway-project> \
  --type=json -p='[{"op":"remove","path":"/data/<provider>-api-key"}]'

Troubleshoot¶

The model does not appear in /v1/models. The config did not reload, or the block has a key the gateway does not recognise. Check the rollout completed and read the startup logs.

Requests to the new name return a routing error. The name in models and the name the caller asks for must match exactly - including case and any organisation prefix.

A self-hosted backend times out. Either the Service name is wrong, or private networking is not allowed. Confirm allow_private_network: true on the provider, then test the Service directly from a pod in the gateway's project.

An external provider returns 401. The Secret key name and the key: field disagree, or the Secret was created after the pod started. Restart the gateway and check the logs.

Next¶

Host your own models → - create a backend of your own to put behind one of these names.