Deploy an LLM gateway¶

By the end of this guide a gateway is running in your project: one OpenAI-compatible endpoint holding provider credentials server-side, enforcing per-key budgets, and reachable by your applications.

The overview explains what a gateway changes for your applications. The console walkthrough covers the same operations in the web console.

Prerequisites¶

A project and CLI access - see Get started
At least one backend: an external provider key, or a model you host
oc logged in

1. Create a project for the gateway¶

Its own project, so its credentials and quota are separate from the applications that call it:

oc new-project <gateway-project>

2. Create the secrets¶

Three, all created in the cluster and none of them committed:

# Dashboard login
oc create secret generic gateway-auth-secret -n <gateway-project> \
  --from-literal=username=<admin-user> \
  --from-literal=password=<admin-password>

# At-rest encryption for stored configuration
oc create secret generic gateway-enc-secret -n <gateway-project> \
  --from-literal=encryption-key="$(openssl rand -base64 32)"

# Upstream provider credentials
oc create secret generic gateway-provider-secrets -n <gateway-project> \
  --from-literal=<provider>-api-key=<key>

Keep the encryption key

Stored configuration is encrypted with it. Losing it means losing whatever the gateway has persisted. Put it in your secret manager before moving on.

3. Write the configuration¶

The gateway's behaviour is one values file. This is the minimum that runs - budgets, rate limits and virtual keys are added in Add a user to the gateway, and backends in Add a model provider:

values.yaml

bifrost:
  host: 0.0.0.0
  port: 8080
  logLevel: info
  logStyle: json

  authConfig:
    isEnabled: true
    existingSecret: gateway-auth-secret # (1)!
    usernameKey: username
    passwordKey: password

  encryptionKey: env.BIFROST_ENCRYPTION_KEY # (2)!
  encryptionKeySecret:
    name: gateway-enc-secret
    key: encryption-key

  client:
    initialPoolSize: 100
    dropExcessRequests: false # (3)!
    logRetentionDays: 30

  plugins:
    governance:
      enabled: true # (4)!
      version: 1
    logging:
      enabled: true # (5)!
      version: 1
    telemetry:
      enabled: true
      version: 1

resources: # (6)!
  requests:
    cpu: 200m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

Edit this. The Secret holding the dashboard login, created in step 2 and never committed. The values file references it; it does not contain it.
Leave as is. env. indirection is what keeps the encryption key out of Git. Lose the key itself and you lose whatever the gateway has encrypted, so store it before moving on.
Decide this. What happens under more load than the gateway can serve. false queues, which turns a spike into latency; true sheds, which turns it into errors. Neither is wrong - pick the failure your callers handle better.
Leave as is. Budgets and rate limits do nothing unless this is on. Turning it off later silently removes every spend ceiling you have configured.
Leave as is. Without it there is no per-key attribution, which is most of the reason for running a gateway at all.
Tune this. The gateway holds no model weights and needs no GPU - it is a router, and these are ordinary web-service numbers. Raise them if it fronts heavy traffic.

4. Choose the storage for gateway state¶

The gateway stores configuration and request logs in SQLite. The chart claims a volume for that database by default, which is what you want:

values.yaml (excerpt)

storage:
  mode: sqlite
  persistence:
    enabled: true # (1)!
    size: 10Gi
    accessMode: ReadWriteOnce # (2)!
postgresql:
  enabled: false

Leave as is. Check it rather than assume it - this is the setting that decides whether a key created in the dashboard still exists tomorrow. Turning it off is a deliberate choice, and one worth making only when every durable change goes through this values file and Git instead.
Check this. ReadWriteOnce is enough for one replica and is the reason more than one needs the shared state described below.

Ephemeral storage fails silently and late

Where persistence is turned off, a key created in the dashboard works perfectly until the pod is rescheduled, upgraded or restarted. Then it stops, and the client holding it fails authentication with nothing to explain why. Confirm the setting before anyone else depends on the gateway - a deployment that has never restarted looks identical either way.

5. Deploy it¶

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: <gateway-project>

helmCharts:
- name: bifrost
  repo: https://maximhq.github.io/bifrost/helm-charts
  releaseName: bifrost
  namespace: <gateway-project> # (1)!
  version: <chart-version> # (2)!
  valuesFile: values.yaml # (3)!

Edit this. Set it in both places. The top-level namespace: retargets the rendered objects; the one inside helmCharts is what the chart's own templates use. They disagree silently if you change only one.
Pin this. An unpinned chart re-renders differently the next time someone applies this directory, which turns an unrelated change into an upgrade.
Leave as is. The values file from step 3, alongside this one. Keeping configuration in its own file is what makes the gateway's behaviour reviewable as a diff.

Check what it renders before applying it:

oc kustomize --enable-helm . | oc apply -f - --dry-run=client

Then apply:

oc apply -k . --enable-helm
oc rollout status statefulset/bifrost -n <gateway-project>

Mirror the image before production

The chart pulls from a public registry. Mirror it into the internal registry so a deployment does not depend on an upstream you do not control.

6. Expose it¶

Applications inside the cluster reach the Service directly:

http://bifrost.<gateway-project>.svc.cluster.local:8080/v1

Anything outside - a developer's laptop, a CI runner - needs a route with TLS:

route.yaml

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: gateway
  namespace: <gateway-project> # (1)!
spec:
  host: <gateway-host> # (2)!
  to:
    kind: Service
    name: bifrost
  port:
    targetPort: 8080
  tls:
    termination: edge # (3)!
    insecureEdgeTerminationPolicy: Redirect

Edit this. The project the gateway runs in, from step 1.
Edit this. The hostname callers will configure. Once anything has it, changing it means changing every caller - pick it once.
Leave as is. TLS terminates at the router, and Redirect sends plain HTTP to HTTPS so a key is never sent in the clear. Never publish this route without TLS: the key is a bearer credential in a header.

oc apply -f route.yaml

To test before publishing a route, forward the port instead:

oc port-forward -n <gateway-project> svc/bifrost 8080:8080

A published gateway can be billed by anything that reaches it

Publishing a route makes anything that can reach it able to spend, subject only to what the keys allow. Confirm authentication is enforced on inference before you expose it - some deployments disable it during a proof of concept.

7. Add a backend and a first key¶

A gateway with no providers routes nothing. Follow Add a model provider, then Add a user to the gateway.

Verify¶

It is running:

oc get pods,svc -n <gateway-project>
oc logs statefulset/bifrost -n <gateway-project> --tail=50

It lists models:

curl -sS <base-url>/models -H "Authorization: Bearer <your-virtual-key>"

An OpenAI client reaches it unchanged, with only the base URL and key altered:

from openai import OpenAI

client = OpenAI(
    base_url="<base-url>",
    api_key="<your-virtual-key>",       # issued by the gateway, not the provider
)

print(client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Say hello"}],
).choices[0].message.content)

A response means the client reached the gateway, the key was accepted, and the request routed to a backend.

List¶

oc get all -n <gateway-project>
oc get route -n <gateway-project>

Change¶

Change configuration - edit the values file and reapply. Most changes restart the pod:

oc apply -k . --enable-helm
oc rollout status statefulset/bifrost -n <gateway-project>

Rotate a provider credential:

oc set data secret/gateway-provider-secrets -n <gateway-project> \
  --from-literal=<provider>-api-key=<new-key>
oc rollout restart statefulset/bifrost -n <gateway-project>

Add replicas. Every model call in your estate goes through this workload, so one replica is a single point of failure:

oc scale statefulset/bifrost -n <gateway-project> --replicas=2

More than one replica needs shared state - see step 4. Two replicas on ephemeral SQLite each keep their own copy and disagree.

Troubleshoot¶

Open the symptom that matches what you are seeing.

The release installs, but the workload has no pods

Helm reports Deployed because the release itself installed; the StatefulSet is what cannot create a pod, and that failure appears only in its events:

oc get events -n <gateway-project> --field-selector reason=FailedCreate

unable to validate against any security context constraint means the chart asks for a user id OpenShift will not grant. The chart hardcodes runAsUser: 1000 and fsGroup: 1000, while restricted-v2 requires a UID from the range allocated to your project, so overriding podSecurityContext in the values file does not help - the numbers are in the templates.

Either an administrator grants the project's service account a SecurityContextConstraint permitting UID 1000, or you deploy the same image from a manifest that sets no user at all and let OpenShift assign one. The second needs nothing granted - see the console walkthrough.

Pod will not start

Usually a missing Secret. oc describe pod names the one it cannot find. All three from step 2 must exist before the first rollout.

It starts, then fails on the first request

An unresolved env. reference - the provider Secret exists, but the key name inside it does not match the config:

oc get secret <name> -n <gateway-project> -o jsonpath='{.data}'

Compare what that prints against the key: field in the values file.

Keys stop working after a restart

Ephemeral storage. See step 4 - the keys were created in the dashboard rather than in the values file, and the volume that would have kept them is not there.

Budgets and limits are ignored

plugins.governance is not enabled. The gateway answers requests normally with it off, enforcing nothing - check the plugin status lines in the log rather than the values file you meant to apply.

Nothing appears in the logs

plugins.logging is not enabled, or storage is ephemeral and the pod restarted.

Delete¶

oc delete -k . --enable-helm
oc delete secret gateway-auth-secret gateway-enc-secret gateway-provider-secrets \
  -n <gateway-project>
oc delete project <gateway-project>

Callers have the hostname configured

Deleting the gateway breaks every application, agent and notebook pointed at it, and they will not fail over anywhere. Move callers first.

Next¶

Add a model provider → - give the gateway something to route to.