Deploy an LLM gateway¶
By the end of this tutorial you will have a running LLM gateway on GRN.CLOUD: a single OpenAI-compatible endpoint your applications call, holding provider credentials server-side and enforcing per-key budgets.
Prefer clicking through the console instead? See the console walkthrough.
Scaffold status
The step structure below is agreed and stable. Exact commands, manifests and captured output are added as this flow is verified end to end on the live platform — this page deliberately shows no invented output.
Prerequisites¶
- A project and CLI access — see Get started
- At least one model to route to: either an external provider key, or a self-hosted model
oclogged in and pointed at your project
Overview¶
flowchart LR
APP["Your app"] -->|OpenAI SDK| GW["LLM gateway"]
GW --> MODEL["Model backend"]
GW -.->|budgets · logs| YOU["You"]Steps¶
1. Create a project for the gateway¶
Keep the gateway in its own project so its credentials and quota are separate from the applications that use it.
2. Store provider credentials as a Secret¶
Provider keys live in a Kubernetes Secret, mounted at runtime. They never reach a client.
3. Deploy the gateway workload¶
A Deployment, a Service, and configuration describing which models exist and where they route.
4. Expose it¶
A Route or Gateway API HTTPRoute, with cert-manager issuing TLS.
5. Issue your first virtual key¶
Virtual keys are what clients use. Provider keys stay server-side.
Verify¶
Point an OpenAI client at the gateway base URL with your virtual key and make one request:
from openai import OpenAI
client = OpenAI(
base_url="https://<your-gateway-host>/v1",
api_key="<your-virtual-key>",
)
print(client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "Say hello"}],
).choices[0].message.content)
A successful response means the client reached the gateway, the gateway authenticated the virtual key, and the request routed to a backend.
Clean up¶
Remove everything this tutorial created, in reverse order: