BIFOLD Talk

Scaling Analytical Benchmarks to Production Complexity: A Data- and Query-Centric Extension to TPC-DS

Abstract:

Benchmarks are a central driver of analytical database research and engineering. They guide system design, shape optimization priorities, and provide shared baselines for comparing engines.

Academia and industry have long relied on TPC-DS as a proxy for production data warehousing challenges, and it captures many dominant analytical patterns well. Yet recent workload studies from Snowflake, Amazon Redshift, and Tableau report that TPC-DS underrepresents important production characteristics. These include string-heavy processing, deep join graphs, sparse and skewed data, and operator parameter scales.

To close these gaps, we introduce Prod-DS, an extension of TPC-DS, grounded in recent production evidence, that allows scaling query complexity through configurable stressors on the existing schema and tooling.

Using WorkloadLens, a coverage-analysis tool, we show that Prod-DS tracks published production distributions more closely than common analytical benchmarks.
Evaluating on three popular analytical engines, we find that production-aligned complexity triggers up to ≈91× longer runtime than TPC-DS, while a moderate ≈8% of queries fail on modern SQL syntax the engines do not fully support. Engines differ sharply in how they handle the added complexity, exposing qualitative performance and feasibility differences not visible under traditional benchmarks.