Deploy the text-generation LLM

Scope and requirements#

This guide uses llm/compose.yaml from the reference archive: vllm/vllm-openai:v0.32.0 serving openai/gpt-oss-120b. It preserves the maintained parser options, 65,536-token context, 0.90 GPU memory utilization and device 2 assignment.

These are reference-host settings, not a recommended capacity promise for every GPU. The maintained host has 96 GB-class devices. Check model architecture, image/CUDA support, weights, context, concurrency, VRAM/RAM and model-cache disk for your machine. Context/concurrency settings can make a configuration fail even when the model weights fit. Model repository access and terms are separate from pulling the image. The vLLM Docker guide describes runtime and cache requirements.

Whisper is a different service and model. Installing this LLM does not add transcription, embeddings, reranking, speech synthesis or realtime capabilities.

Start and inspect#

From the extracted reference directory:

docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai up -d llm-gpt-oss-120b
docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai logs --tail 100 llm-gpt-oss-120b

HF_TOKEN is optional for repositories you can download anonymously; obtain authorized model access when required. The two named caches survive container recreation. Keep the model service private; its diagnostic port binds only to 127.0.0.1:8001 on the inference host.

Verify generation#

On the inference host, wait for readiness and send a real generation request:

curl --fail http://127.0.0.1:8001/health
curl --fail -H 'Content-Type: application/json' http://127.0.0.1:8001/v1/chat/completions -d '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"Reply with a short greeting."}],"max_tokens":256}'

Expect HTTP 200 and a non-empty assistant message. Reasoning models may consume part of the token budget before visible output; inspect finish reasons and increase the budget when needed. Test streaming, tool calls and any structured outputs required by your chosen features separately. See the upstream API guide.

For the full recipe, start the gateway too and send the same request to 127.0.0.1:8002/v1/chat/completions with model: llm-default and Authorization: Bearer YOUR_INFERENCE_KEY. The gateway alias maps to the upstream model ID. Read the key into an environment variable rather than putting its literal value into shell history; see the bundle's checks.

Configure Dicto#

The reference server uses:

chat:
  base_url: "${CHAT_BASE_URL}"
  api_key: "${CHAT_API_KEY}"
  models:
    - id: default
      name: GPT-OSS 120B
      provider_model: llm-default
      is_default: true

Merge under ai. Use the complete gateway root http://llm-server:8000/v1 and the gateway credential in secrets.env. For a direct connection, use http://llm-gpt-oss-120b:8000/v1 and upstream model ID openai/gpt-oss-120b, with the authentication that direct endpoint requires.

Validate configuration, recreate API/jobs and test a chat turn from the client. Do not offer unsupported reasoning choices. Model configuration explains catalog IDs, provider IDs and optional capabilities.

Scripto and additional models#

The maintained deployment also runs a separate Scripto agent service. A chat model alone does not provision that runtime or prove Scripto tool execution. Configure the matching released agent service and its scripto_agent.server_url when deploying that feature; it is outside this core download. Embeddings, reranking and organization-specific model routes likewise need their own services and checks. Keep absent capabilities unconfigured.

When substituting a smaller model or changing hardware, update the vLLM model, model-specific parser options, gateway upstream_model and Dicto catalog together. Retest the actual feature requests instead of assuming model IDs are interchangeable.