Pantheon TSDB is a large-scale time series database system developed by Databricks, derived from the open-source CNCF Thanos project. Pantheon represents a specialized fork designed to handle monitoring infrastructure at unprecedented scale, supporting distributed cloud deployments with custom optimizations for cost efficiency and reliability.

Pantheon TSDB extends the capabilities of Thanos, an open-source time series storage and querying layer, to address the monitoring requirements of large-scale distributed systems. The system is engineered to manage monitoring data across heterogeneous cloud environments, consolidating metrics from 160+ instances distributed across multiple cloud providers into a unified monitoring platform ¹⁾.

The architecture maintains active timeseries counts of 5 billion in-memory, enabling high-velocity query performance for real-time monitoring dashboards and alerting systems. This approach balances memory utilization with query latency requirements typical of production monitoring environments.

Pantheon processes 10+ trillion samples daily, positioning it among the most high-throughput monitoring platforms in operation ²⁾.

The system achieves a 5x reduction in monitoring downtime compared to traditional monitoring infrastructure architectures. This improvement stems from optimized failover mechanisms, distributed query processing, and redundancy strategies that prevent cascade failures in monitoring systems themselves. Such downtime reduction directly impacts operational reliability, as monitoring outages prevent visibility into production incidents. Pantheon's tiered storage architecture, control plane automation, and multi-cloud deployment capabilities enable hands-off operations across the distributed infrastructure, eliminating the daily scale-up requirements that hindered traditional TSDB solutions ³⁾.

Through custom optimizations implemented on top of the Thanos foundation, Pantheon delivers millions in annual cloud cost savings for Databricks' infrastructure operations ⁴⁾.

Cost reductions emerge from several optimization categories:

* Storage efficiency: Optimized compression algorithms reduce the storage footprint of time series data, decreasing cloud storage costs * Query optimization: Improved query planning and execution reduce computational resources required for monitoring queries and aggregations * Data retention policies: Intelligent tiering and retention management balance operational requirements with storage economics * Multi-cloud utilization: Distribution across cloud providers enables workload optimization and potential cost arbitrage

As a fork of the CNCF Thanos project, Pantheon inherits core capabilities including distributed time series querying, multi-source data aggregation, and long-term archival support. Custom modifications address Databricks-specific requirements including integration with proprietary data platform services, enhanced multi-cloud federation, and performance tuning for internal workload patterns.

The system likely employs object storage backends (such as cloud-native S3-compatible services) for long-term archival, with efficient query interfaces for both real-time operational monitoring and historical analysis. The in-memory active timeseries count of 5 billion suggests sophisticated memory management and eviction policies to maintain performance while managing resource utilization.

Pantheon TSDB operates as part of Databricks' broader infrastructure observability stack, supporting monitoring of the Databricks Lakehouse Platform and related cloud services. The scale at which it operates—160+ instances across cloud providers processing trillions of daily samples—positions it as essential infrastructure for production system visibility and incident response.

• Timeseries Database (TSDB)
• Databricks Lakebase
• Hydra Platform
• Metric Aggregation
• Hydra vs Pantheon for Troubleshooting Data