Outcome and scope#
This recipe adds GPU-backed Whisper transcription, GPT-OSS text generation and the internal gateway to the reference setup. It targets Linux AMD64 with NVIDIA Container Toolkit and the configured GPU allocations. Read requirements, transcription and LLM deployment before starting large downloads.
The maintained deployment's service layout is represented here, with public Whisper substituted for its private checkpoint. The docs build checks files and configuration; it does not certify GPU capacity or a completed product deployment.
Configure#
Extract the archive and run from dicto-reference-setup. Copy
secrets.env.example to secrets.env, restrict its permissions and replace all
placeholders as described in its README. Supply real hostnames in all three
public configuration files. Set INFERENCE_API_KEY, CHAT_API_KEY
and
TRANSCRIPTION_API_KEY to the same newly generated random value. Keep:
CHAT_BASE_URL=http://llm-server:8000/v1
TRANSCRIPTION_BASE_URL=http://llm-server:8000/v1
The gateway receives aliases whisper and llm-default and rewrites them to the
upstream model IDs in llm-server/models.json. It does not run the models.
Model weights need their own access/license review and are cached on first use.
Start inference and verify it#
Use this complete Compose prefix throughout this recipe, from the archive root:
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai up --build -d whisper-loadbalancer llm-gpt-oss-120b llm-server
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai logs --tail 100 whisper-loadbalancer llm-gpt-oss-120b llm-server
Wait for model loading/compilation and check the endpoints on the inference host. Follow transcription verification and LLM verification. A gateway model list can succeed while an upstream is still loading.
Start Dicto#
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai run --rm --no-deps migrator validate-config --resolve-env
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai up --build -d
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai ps --all
The core initializes the recordings bucket, runs migrations and starts API/jobs
and management. Open https://YOUR_HOST/management/, sign in, connect a client
and verify a recording, playback and transcription. Check chat separately.
Disable the bootstrap seeder after first sign-in and recreate API/jobs using
the same complete prefix.
Separate application and GPU hosts#
Docker service names work only on their shared network. To split hosts, move the GPU services/gateway to a controlled inference network, bind the gateway to an explicitly secured interface, and use its reachable authenticated HTTPS API root in the application's provider settings. Keep the individual model servers private. Test uploads, timeouts and generation from the application network. The included loopback bindings cannot be reached from another host without a tunnel or explicit network/proxy configuration.
Back up the core's persistent data and preserve the complete Compose prefix for updates and recovery. See backup and upgrades.