Self-Hosted LLM Infrastructure: GPU Cluster Setup for LLMOps

Key takeaway: Self-hosting LLMs makes financial sense above ~$5,000/month in API spend, or when regulatory requirements prohibit sending data to external APIs. vLLM with NVIDIA A100 or H100 hardware is the current production standard for inference serving.

When to Self-Host

The managed API model (OpenAI, Anthropic, Google) is the correct choice for most organizations: zero infrastructure overhead, access to the best models, automatic scaling, and no capital expenditure. Self-hosting makes sense when:

• Cost: Monthly API spend exceeds $5,000–10,000. At scale, on-premise inference costs 60–80% less per token.
• Data privacy: Regulated industries (healthcare, finance, government) cannot send data to third-party APIs. Self-hosting keeps data within your network perimeter.
• Latency: Applications requiring sub-100ms first-token latency benefit from colocated inference servers.
• Customization: You need to fine-tune a model on proprietary data that cannot leave your infrastructure.
• Vendor independence: Strategic need to eliminate dependency on API providers who can change pricing or availability.

Hardware Selection

GPU selection drives most of the infrastructure cost and capability. The practical tiers in 2026:

For NVLink multi-GPU setups, NVIDIA's NVLink technology dramatically improves throughput for models that span multiple GPUs by providing 900GB/s GPU-to-GPU bandwidth versus PCIe's 64GB/s.

VRAM Requirements by Model Size

A rough guide for common model sizes at float16 precision (multiply by 2 for float32, divide by ~4 for INT4 quantization):

vLLM Setup

vLLM is the production-standard LLM inference server. It implements PagedAttention for efficient KV cache management and continuous batching for high throughput.

# Install vLLM (requires NVIDIA GPU with CUDA 11.8+)
pip install vllm

# Serve Llama 3 8B (requires ~16GB VRAM)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --served-model-name llama3-8b \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192

# Serve with tensor parallelism across 2 GPUs (for 70B model)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 4096

# Test the endpoint (OpenAI-compatible API)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-8b",
    "messages": [{"role": "user", "content": "Explain Kubernetes in one sentence"}],
    "max_tokens": 100
  }'

vLLM exposes an OpenAI-compatible API, so any existing code using the OpenAI SDK can point to your self-hosted endpoint by changing the base URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://your-gpu-server:8000/v1",
    api_key="not-needed",  # vLLM does not require a key by default
)

response = client.chat.completions.create(
    model="llama3-8b",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Serving Architecture

A production self-hosted LLM architecture has several layers:

GPU inference servers: 2–4 servers running vLLM or similar, each with 1–4 GPUs. These are stateless — they load the model once and serve requests.

Load balancer: Nginx or HAProxy distributing inference requests across GPU servers. Important: LLM inference is not like HTTP — requests have vastly different lengths, so round-robin is not optimal. Use least-connections load balancing.

Request queue: A Redis or RabbitMQ queue absorbs traffic spikes. Inference requests that would otherwise timeout during burst traffic wait in queue instead.

Caching layer: Semantic caching (using embeddings to detect near-identical queries) can reduce inference calls by 20–40% for applications with repetitive prompts.

# docker-compose.yml for a simple self-hosted LLM stack
version: '3.8'
services:
  vllm:
    image: vllm/vllm-openai:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3-8B-Instruct
      --host 0.0.0.0
      --port 8000
    ports:
      - "8000:8000"
    volumes:
      - huggingface_cache:/root/.cache/huggingface

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    depends_on:
      - vllm

volumes:
  huggingface_cache:

Monitoring LLM Infrastructure

Key metrics to monitor for an LLM serving cluster:

• GPU utilization % — should be 80–95% during normal operation. Below 60% means you are over-provisioned; consistently at 100% means you need more capacity.
• GPU memory utilization — track headroom; OOM (out-of-memory) kills the server.
• Time to first token (TTFT) — the latency until the first token arrives. User-perceived responsiveness.
• Throughput (tokens/second) — total generation speed across all concurrent requests.
• Request queue depth — requests waiting for a GPU slot. Should be near zero under normal load.

# Monitor GPU metrics with nvidia-smi
watch -n 1 nvidia-smi

# Continuous GPU metrics (dmon — driver monitoring)
nvidia-smi dmon -s u  # utilization
nvidia-smi dmon -s m  # memory

# vLLM exposes Prometheus metrics at /metrics
curl http://localhost:8000/metrics | grep vllm

Cost Comparison: Self-Hosted vs API

Example calculation for a team running 100 million tokens/month at input+output blended rate:

Managed API (example rates): At roughly $1.50 per million tokens blended = $150/month for 100M tokens. At 1 billion tokens: $1,500/month.

Self-hosted (one A100 80GB server): Hardware amortized over 3 years: ~$400/month. Power and colocation: ~$200/month. Total: ~$600/month. An A100 can serve approximately 2–5 billion tokens per month depending on the model size and request pattern.

Break-even for self-hosting one A100 at current API prices is approximately 400–800 million tokens/month. Above that, self-hosting wins economically. Below that, managed API wins operationally.

LLMOps Toolchain

Model serving: vLLM (inference), Ollama (development and edge deployment), TensorRT-LLM (NVIDIA-optimized production).

Model registry: Hugging Face Hub (public models), MLflow (model versioning), custom S3-based registry for proprietary fine-tunes.

Prompt management: LangSmith, Promptlayer, or custom prompt versioning in Git.

Guardrails: NVIDIA NeMo Guardrails, Guardrails AI, custom classifiers for domain-specific filtering.

Observability: Prometheus + Grafana for infrastructure metrics, LangSmith or custom logging for prompt/response auditing, OpenTelemetry for distributed tracing.

Final Thoughts

Self-hosting LLMs is an infrastructure engineering challenge, not just an ML challenge. The GPU selection, serving architecture, and operational toolchain are exactly the skills senior infrastructure engineers already have — applied to a new class of workload. If your organization is spending significant money on LLM APIs or has data privacy requirements that prevent API usage, the self-hosted path is worth evaluating seriously. Start with a single GPU server running vLLM and benchmark your actual workload before committing to a larger investment.