# Install self-hosted transcription and LLM

Start Whisper, GPT-OSS and an authenticated internal gateway alongside the reference Dicto core.

## Outcome and scope

This recipe adds GPU-backed Whisper transcription, GPT-OSS text generation and
the internal gateway to the [reference setup](/install/reference.md). It targets
Linux AMD64 with NVIDIA Container Toolkit and the configured GPU allocations.
Read [requirements](/start/requirements.md), [transcription](/components/transcription.md)
and [LLM deployment](/components/llm.md) before starting large downloads.

The maintained deployment's service layout is represented here, with public
Whisper substituted for its private checkpoint. The docs build checks files and
configuration; it does not certify GPU capacity or a completed product deployment.

## Configure

Extract the archive and run from `dicto-reference-setup`. Copy
`secrets.env.example` to `secrets.env`, restrict its permissions and replace all
placeholders as described in its README. Supply real hostnames in all three
public configuration files. Set `INFERENCE_API_KEY`, `CHAT_API_KEY` and
`TRANSCRIPTION_API_KEY` to the same newly generated random value. Keep:

```dotenv
CHAT_BASE_URL=http://llm-server:8000/v1
TRANSCRIPTION_BASE_URL=http://llm-server:8000/v1
```

The gateway receives aliases `whisper` and `llm-default` and rewrites them to the
upstream model IDs in `llm-server/models.json`. It does not run the models.
Model weights need their own access/license review and are cached on first use.

## Start inference and verify it

Use this complete Compose prefix throughout this recipe, from the archive root:

```sh
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai up --build -d whisper-loadbalancer llm-gpt-oss-120b llm-server
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai logs --tail 100 whisper-loadbalancer llm-gpt-oss-120b llm-server
```

Wait for model loading/compilation and check the endpoints on the inference host.
Follow [transcription verification](/components/transcription.md#verify-with-real-audio)
and [LLM verification](/components/llm.md#verify-generation). A gateway model list
can succeed while an upstream is still loading.

## Start Dicto

```sh
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai run --rm --no-deps migrator validate-config --resolve-env
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai up --build -d
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml -f llm/compose.yaml --profile self-hosted-ai ps --all
```

The core initializes the recordings bucket, runs migrations and starts API/jobs
and management. Open `https://YOUR_HOST/management/`, sign in, connect a client
and verify a recording, playback and transcription. Check chat separately.
Disable the bootstrap seeder after first sign-in and recreate API/jobs using
the same complete prefix.

## Separate application and GPU hosts

Docker service names work only on their shared network. To split hosts, move
the GPU services/gateway to a controlled inference network, bind the gateway to
an explicitly secured interface, and use its reachable authenticated HTTPS API
root in the application's provider settings. Keep the individual model servers
private. Test uploads, timeouts and generation from the application network.
The included loopback bindings cannot be reached from another host without a
tunnel or explicit network/proxy configuration.

Back up the core's persistent data and preserve the complete Compose prefix for
updates and recovery. See [backup and upgrades](/operations/backup-upgrade.md).