Groq is an AI inference company founded in 2016 by Jonathan Ross, the original designer of Google's Tensor Processing Unit (TPU). Groq develops and operates the Language Processing Unit (LPU), a custom ASIC chip purpose-built for ultra-fast, low-latency inference of large language models. The company's GroqCloud platform serves over 2.8 million developers worldwide.1)
The LPU is fundamentally different from GPUs in its approach to AI inference. It uses a Tensor Streaming Processor (TSP) design that prioritizes sequential token generation over general-purpose parallel computation.2)
Key architectural features:
Groq LPUs deliver dramatically faster inference compared to GPU-based solutions:
| Metric | GPU (e.g., NVIDIA H100) | Groq LPU |
|---|---|---|
| Token Generation Speed | 50-100 tokens/sec | 500-1,000+ tokens/sec |
| Relative Performance | Baseline | 5-10x faster |
| Latency Characteristics | Variable | Predictable, low |
| Power Efficiency | Moderate | High (approx. 1/3 GPU power) |
Groq has claimed that ChatGPT could run 13x faster on LPU infrastructure. The LPU overcomes GPU bottlenecks in memory bandwidth and sequential processing, enabling real-time applications such as conversational AI and interactive agents.3)
GroqCloud provides an OpenAI-compatible API supporting text, audio, and vision models with scalable, predictable pricing. The platform supports open-source models exclusively, including:
A free tier is available for experimentation and development, available since January 2024.4)
Groq emphasizes low-cost inference enabled by LPU efficiency. Users have reported up to 89% cost reduction compared to GPU-based alternatives. The platform offers plan-based pricing scaled to usage volume.