# Deploy the text-generation LLM

Run the maintained GPT-OSS serving profile, test generation and connect its model alias to Dicto.

## Scope and requirements

This guide uses [llm/compose.yaml](/examples/reference/llm/compose.yaml) from the
[reference archive](/install/reference.md): `vllm/vllm-openai:v0.32.0` serving
`openai/gpt-oss-120b`. It preserves the maintained parser options, 65,536-token
context, 0.90 GPU memory utilization and device `2` assignment.

These are reference-host settings, not a recommended capacity promise for every
GPU. The maintained host has 96 GB-class devices. Check model architecture,
image/CUDA support, weights, context, concurrency, VRAM/RAM and model-cache disk
for your machine. Context/concurrency settings can make a configuration fail
even when the model weights fit. Model repository access and terms are separate
from pulling the image. The [vLLM Docker guide](https://docs.vllm.ai/en/latest/deployment/docker/)
describes runtime and cache requirements.

Whisper is a different service and model. Installing this LLM does not add
transcription, embeddings, reranking, speech synthesis or realtime capabilities.

## Start and inspect

From the extracted reference directory:

```sh
docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai config --quiet
docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai up -d llm-gpt-oss-120b
docker compose --env-file secrets.env -f compose.yaml -f llm/compose.yaml --profile self-hosted-ai logs --tail 100 llm-gpt-oss-120b
```

`HF_TOKEN` is optional for repositories you can download anonymously; obtain
authorized model access when required. The two named caches survive container
recreation. Keep the model service private; its diagnostic port binds only to
`127.0.0.1:8001` on the inference host.

## Verify generation

On the inference host, wait for readiness and send a real generation request:

```sh
curl --fail http://127.0.0.1:8001/health
curl --fail -H 'Content-Type: application/json' http://127.0.0.1:8001/v1/chat/completions -d '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"Reply with a short greeting."}],"max_tokens":256}'
```

Expect HTTP 200 and a non-empty assistant message. Reasoning models may consume
part of the token budget before visible output; inspect finish reasons and
increase the budget when needed. Test streaming, tool calls and any structured
outputs required by your chosen features separately. See the
[upstream API guide](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/).

For the full recipe, start the gateway too and send the same request to
`127.0.0.1:8002/v1/chat/completions` with `model: llm-default` and
`Authorization: Bearer YOUR_INFERENCE_KEY`. The gateway alias maps to the upstream
model ID. Read the key into an environment variable rather than putting its
literal value into shell history; see the bundle's [checks](/examples/reference/checks/README.md).

## Configure Dicto

The reference server uses:

```yaml
chat:
  base_url: "${CHAT_BASE_URL}"
  api_key: "${CHAT_API_KEY}"
  models:
    - id: default
      name: GPT-OSS 120B
      provider_model: llm-default
      is_default: true
```

Merge under `ai`. Use the complete gateway root `http://llm-server:8000/v1`
and the gateway credential in `secrets.env`. For a direct connection, use
`http://llm-gpt-oss-120b:8000/v1` and upstream model ID
`openai/gpt-oss-120b`, with the authentication that direct endpoint requires.

Validate configuration, recreate API/jobs and test a chat turn from the client.
Do not offer unsupported reasoning choices. [Model configuration](/configure/models.md)
explains catalog IDs, provider IDs and optional capabilities.

## Scripto and additional models

The maintained deployment also runs a separate Scripto agent service. A chat
model alone does not provision that runtime or prove Scripto tool execution.
Configure the matching released agent service and its `scripto_agent.server_url`
when deploying that feature; it is outside this core download. Embeddings,
reranking and organization-specific model routes likewise need their own
services and checks. Keep absent capabilities unconfigured.

When substituting a smaller model or changing hardware, update the vLLM model,
model-specific parser options, gateway `upstream_model` and Dicto catalog together.
Retest the actual feature requests instead of assuming model IDs are interchangeable.