# Deploy the inference gateway

Route stable model aliases to private transcription and LLM services with explicit authentication.

The optional `hippolabs-org/llm-server` is a gateway, not a model host. The
reference profile uses version `v1.0.12`, a private Docker network, API-key
authentication, and [models.json](/examples/reference/llm-server/models.json).
Registry access may be required independently of access to the public docs.

## Configure routes and credentials

`MODEL_ROUTES_FILE=/models.json` selects the mounted routes file. Its two public
aliases are `whisper` and `llm-default`; each maps to its upstream service and
the exact model ID that service accepts. Targets use the server root, such as
`http://llm-gpt-oss-120b:8000`, while Dicto uses the full API root ending `/v1`.

Set `INFERENCE_API_KEY` in `secrets.env`. The reference passes it as `API_KEYS`
to enable gateway authentication. Set Dicto's chat/transcription keys to the
same value. In this gateway, omitting `API_KEYS` disables its API-key check;
do not expose an unauthenticated instance as a public service.

## Start and check

Start the full [self-hosted recipe](/install/self-hosted-ai.md), including the model
servers. On the inference host:

```sh
read -rsp 'Inference key: ' INFERENCE_API_KEY
export INFERENCE_API_KEY
curl --fail -H "Authorization: Bearer $INFERENCE_API_KEY" http://127.0.0.1:8002/v1/models
unset INFERENCE_API_KEY
```

Expect a successful models response. Then perform an audio request and a chat
request through the same endpoint; model listing does not establish upstream
readiness. See the [downloaded checks](/examples/reference/checks/README.md).
The separate internal listener on port 9090 exposes `/livez`, `/readyz` and
`/metrics`; it is not the public inference API and is not published by this sample.

When splitting hosts, secure connectivity explicitly and change the route target
URLs. The Docker aliases in the sample do not resolve automatically across hosts.
Keep internal health/metrics private and test from the API's network.