Self-hosting gives you full control over your LLM infrastructure – data stays on your network, per-token costs drop at scale, and you can customize models freely. This guide covers hardware selection, inference engines, deployment, and cost trade-offs.

LLM inference is memory-bandwidth bound, not compute bound. Prioritize VRAM capacity and bandwidth over raw FLOPS.

A 7B parameter model at 4-bit quantization runs comfortably on a 12GB GPU. A 70B model needs 24-40GB depending on quantization level, or can be split across multiple GPUs. ¹⁾

The RTX 4090 is the consumer sweet spot – 24GB VRAM at high bandwidth for around $1,100 used. For production workloads requiring concurrency, A100 or H100 instances via cloud providers are the standard choice. ²⁾

Ollama is the best starting point for local experimentation. vLLM is the production standard for multi-user serving with its PagedAttention memory management. llama.cpp gives the best performance per watt for single-user inference. ³⁾

Quantization reduces model precision to shrink VRAM requirements with minimal quality loss:

The Q4_K_M quantization level offers the best balance of quality and size for most use cases. More aggressive quantization (Q2, Q3) saves VRAM but degrades output quality noticeably. ⁴⁾

docker run -d --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# Pull and run a model
docker exec -it ollama ollama run llama3:8b

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai \
  --model meta-llama/Llama-3-70B \
  --quantization awq \
  --tensor-parallel-size 2

docker run -p 8080:8080 --gpus all \
  localai/localai:latest-aio-cuda12

Prerequisites for GPU passthrough: NVIDIA drivers, CUDA toolkit, and the NVIDIA Container Toolkit installed on the host. ⁵⁾

Self-hosting breaks even versus API costs within 3-6 months for heavy usage (>1M tokens/day). The primary advantage beyond cost is data sovereignty – your prompts and responses never leave your network. ⁶⁾

• Use NVMe SSDs – model loading from NVMe is 5-10x faster than SATA
• Maximize GPU memory bandwidth – choose GPUs with higher GB/s over higher TFLOPS
• Layer offloading – split model layers between GPU and CPU RAM when VRAM is tight
• Batch requests – vLLM's continuous batching serves multiple users efficiently
• Tensor parallelism – split large models across multiple GPUs for production throughput

⁷⁾

• How to Use Ollama
• How to Fine-Tune an LLM
• How to Build a Chatbot