# Deploy the transcription service

Serve Whisper with vLLM, connect its model to Dicto and verify a real audio upload.

## Scope and files

The [reference download](/install/reference.md) includes
[transcription/compose.yaml](/examples/reference/transcription/compose.yaml) and
[its Dockerfile](/examples/reference/transcription/Dockerfile). This example runs
vLLM 0.26.0 with its matching audio dependencies and public
`openai/whisper-large-v3`, served as `whisper`.

The maintained setup uses a local fine-tuned checkpoint plus runtime patches;
neither weights nor private patch code are shipped in this portable example.
It covers non-streaming JSON transcription. Do not enable token logprobs,
diarization or realtime flags until your actual endpoint passes those tests.

## Requirements and GPU assignment

Use Linux AMD64 with NVIDIA drivers/Container Toolkit compatible with the image.
Verify GPU visibility and enough VRAM, RAM and disk for weights, compilation and
your recording workload. Check access to the model registry and any model terms.
The first startup downloads weights into the `model-cache` volume.

The overlay retains devices `0`/`1` and `--tensor-parallel-size=2`. To use one
GPU, change both the device list and tensor-parallel size to one, then validate
capacity on that GPU. A Docker reservation does not impose a VRAM quota. When
sharing a host, avoid assigning the LLM to the same devices accidentally.

## Start the service

From the extracted setup root, with `secrets.env` filled:

```sh
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai up --build -d whisper-loadbalancer
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai logs --tail 100 whisper-loadbalancer
```

Wait for startup to finish; model download and compilation may take minutes.
The diagnostic listener is `127.0.0.1:8000` on the host. Other containers use
`http://whisper-loadbalancer:8000`, not host loopback. The service is not routed
publicly by the example proxy.

## Verify with real audio

Put a short non-sensitive recording at `sample.wav` on the inference host:

```sh
curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/v1/audio/transcriptions -F model=whisper -F response_format=json -F file=@sample.wav
```

Expect HTTP 200 and JSON with non-empty `text` matching your spoken sample.
The health endpoint alone does not test decoding or transcription accuracy.
Use an actual audio file, not an empty file or a renamed arbitrary payload.
See the upstream [speech-to-text API documentation](https://docs.vllm.ai/en/latest/serving/online_serving/speech_to_text/)
for request formats; check differences against the pinned runtime version.

## Connect Dicto

For a direct private connection, merge this into the server's `ai` section:

```yaml
transcription:
  type: vllm
  base_url: http://whisper-loadbalancer:8000/v1
  models:
    - id: whisper
      name: Whisper
      provider_model: whisper
      is_default: true
```

`id` is Dicto's catalog identity; `provider_model` is what the upstream accepts.
The retained `type: vllm` value is a compatibility setting, not a requirement
that every provider be vLLM. Supply the complete API root, including `/v1`.

The full recipe instead routes through the [gateway](/components/gateway.md), uses
`http://llm-server:8000/v1`, and supplies its API key. Preserve the alias `whisper`
in `llm-server/models.json`. Validate the mounted server configuration and
recreate API/jobs, then record, play back and transcribe from a Dicto client.

## Troubleshooting

- GPU unavailable: verify NVIDIA runtime and the selected device IDs first.
- Download denied: verify model permissions/token, license acceptance and egress.
- Out of memory: review concurrency, parallelism and other GPU workloads before
  changing context/worker settings blindly.
- Model not found: compare `provider_model`, served-model name and gateway alias.
- Upload rejected: check provider/proxy limits and request format. The example
  explicitly configures 500 MB upload and 7,200-second decode limits; these
  settings do not change limits elsewhere in the client/API/proxy path.
- Optional capability fails: leave its flag off and test it separately. A
  generic OpenAI-compatible label does not establish support for every feature.