The optional hippolabs-org/llm-server is a gateway, not a model host. The
reference profile uses version v1.0.12, a private Docker network, API-key
authentication, and models.json.
Registry access may be required independently of access to the public docs.
Configure routes and credentials#
MODEL_ROUTES_FILE=/models.json selects the mounted routes file. Its two public
aliases are whisper and llm-default; each maps to its upstream service and
the exact model ID that service accepts. Targets use the server root, such as
http://llm-gpt-oss-120b:8000, while Dicto uses the full API root ending
/v1.
Set INFERENCE_API_KEY in secrets.env. The reference passes it as API_KEYS
to enable gateway authentication. Set Dicto's chat/transcription keys to the
same value. In this gateway, omitting API_KEYS disables its API-key check;
do not expose an unauthenticated instance as a public service.
Start and check#
Start the full self-hosted recipe, including the model servers. On the inference host:
read -rsp 'Inference key: ' INFERENCE_API_KEY
export INFERENCE_API_KEY
curl --fail -H "Authorization: Bearer $INFERENCE_API_KEY" http://127.0.0.1:8002/v1/models
unset INFERENCE_API_KEY
Expect a successful models response. Then perform an audio request and a chat
request through the same endpoint; model listing does not establish upstream
readiness. See the downloaded checks.
The separate internal listener on port 9090 exposes /livez, /readyz
and
/metrics; it is not the public inference API and is not published by this sample.
When splitting hosts, secure connectivity explicitly and change the route target URLs. The Docker aliases in the sample do not resolve automatically across hosts. Keep internal health/metrics private and test from the API's network.