Databricks Cost Anomaly Detection

LakeSentry uses statistical and threshold checks to surface changes that deserve investigation. An anomaly is not proof of waste; it is a starting point for cost forensics.

Detected anomaly types

See the Insight Type Catalog for the cost, duration, failure-rate, warehouse, serving, AI, App, budget, and attribution findings. The Lakebase section covers its storage-growth and spend anomalies.

Detection methods

Cost Spike and Duration Spike use a Z-score. It measures how far a recent value is from its baseline, relative to normal variation:

z = (current_value - baseline_mean) / baseline_standard_deviation

The other anomaly findings compare a recent period with a prior period or a budget, and use fixed thresholds. The catalog gives the window and thresholds of each finding.

Cost Spike, Duration Spike, and Failure Spike run hourly. The window they compare is longer: the last 48 hours against a baseline of 7 to 30 days.

Severity depends on the finding type. A higher severity reflects a larger statistical deviation, a larger cost increase, or a larger projected budget overage.

Baselines and safeguards

• A Z-score needs a baseline of at least 5 data points.
• A baseline with no variation cannot produce a Z-score. Cost Spike can still appear through its cost multiplier.
• Cost Spike has minimum cost thresholds, so small workloads do not raise it. See the catalog.
• Budget Risk uses the projected spend for the budget period, not a Z-score.

Investigating an anomaly

1. Open the insight detail page and review evidence.
2. Use Cost Changes to see the period-over-period driver.
3. Drill into Cost Explorer by workspace, team, principal, compute type, or table.
4. Open the related workload, cluster, SQL warehouse, or serving page when linked.
5. Resolve or dismiss the insight after you confirm the cause; see Choosing a response.