Scope and files#
The reference download includes
transcription/compose.yaml
and
its Dockerfile. This example runs
vLLM 0.26.0 with its matching audio dependencies and public
openai/whisper-large-v3, served as whisper.
The maintained setup uses a local fine-tuned checkpoint plus runtime patches; neither weights nor private patch code are shipped in this portable example. It covers non-streaming JSON transcription. Do not enable token logprobs, diarization or realtime flags until your actual endpoint passes those tests.
Requirements and GPU assignment#
Use Linux AMD64 with NVIDIA drivers/Container Toolkit compatible with the image.
Verify GPU visibility and enough VRAM, RAM and disk for weights, compilation and
your recording workload. Check access to the model registry and any model terms.
The first startup downloads weights into the model-cache volume.
The overlay retains devices 0/1 and --tensor-parallel-size=2. To use one
GPU, change both the device list and tensor-parallel size to one, then validate
capacity on that GPU. A Docker reservation does not impose a VRAM quota. When
sharing a host, avoid assigning the LLM to the same devices accidentally.
Start the service#
From the extracted setup root, with secrets.env filled:
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai up --build -d whisper-loadbalancer
docker compose --env-file secrets.env -f compose.yaml -f transcription/compose.yaml --profile self-hosted-ai logs --tail 100 whisper-loadbalancer
Wait for startup to finish; model download and compilation may take minutes.
The diagnostic listener is 127.0.0.1:8000 on the host. Other containers use
http://whisper-loadbalancer:8000, not host loopback. The service is not routed
publicly by the example proxy.
Verify with real audio#
Put a short non-sensitive recording at sample.wav on the inference host:
curl --fail http://127.0.0.1:8000/health
curl --fail http://127.0.0.1:8000/v1/audio/transcriptions -F model=whisper -F response_format=json -F file=@sample.wav
Expect HTTP 200 and JSON with non-empty text matching your spoken sample.
The health endpoint alone does not test decoding or transcription accuracy.
Use an actual audio file, not an empty file or a renamed arbitrary payload.
See the upstream speech-to-text API documentation
for request formats; check differences against the pinned runtime version.
Connect Dicto#
For a direct private connection, merge this into the server's ai section:
transcription:
type: vllm
base_url: http://whisper-loadbalancer:8000/v1
models:
- id: whisper
name: Whisper
provider_model: whisper
is_default: true
id is Dicto's catalog identity; provider_model is what the upstream accepts.
The retained type: vllm value is a compatibility setting, not a requirement
that every provider be vLLM. Supply the complete API root, including /v1.
The full recipe instead routes through the gateway, uses
http://llm-server:8000/v1, and supplies its API key. Preserve the alias
whisper
in llm-server/models.json. Validate the mounted server configuration and
recreate API/jobs, then record, play back and transcribe from a Dicto client.
Troubleshooting#
- GPU unavailable: verify NVIDIA runtime and the selected device IDs first.
- Download denied: verify model permissions/token, license acceptance and egress.
- Out of memory: review concurrency, parallelism and other GPU workloads before changing context/worker settings blindly.
- Model not found: compare
provider_model, served-model name and gateway alias. - Upload rejected: check provider/proxy limits and request format. The example explicitly configures 500 MB upload and 7,200-second decode limits; these settings do not change limits elsewhere in the client/API/proxy path.
- Optional capability fails: leave its flag off and test it separately. A generic OpenAI-compatible label does not establish support for every feature.