Databricks Cost of Failure
Open Cost of Failure in the sidebar to see money spent on executions that did not succeed. Select a date range and scope before comparing it with other cost reports.
Data requirements
Section titled “Data requirements”The page combines execution outcomes with attributed costs. Billing alone is not enough: jobs and pipelines need Lakeflow execution data, and SQL needs query history. Check connector coverage and pipeline status if execution counts or costs are incomplete.
Reading the page
Section titled “Reading the page”| View | Meaning |
|---|---|
| Cost of Failure | Cost attributed to failed executions included in the report |
| Share of Total Spend | Failure cost divided by total spend in the period, using the selected cost display mode |
| Failed Executions | Included failed runs plus failed queries |
| Execution Failure Rate | Included failed runs and queries divided by the corresponding total executions; this is a count ratio, not a cost ratio |
| Failure Cost Trend | How failure spend changes during the selected period |
| By Source | Contribution from jobs, pipelines, and SQL |
| Top Offenders | Resources to investigate first. Jobs and pipelines open their detail views; SQL warehouses open their failed queries |
Use the global cost display mode consistently when comparing Databricks and cloud costs. A high failure count can have a low attributed cost, and a small number of expensive failures can dominate spending.
What is excluded
Section titled “What is excluded”The Not included in failure cost section reports cancelled job runs, cancelled pipeline runs, cancelled queries, streaming job runs, and streaming pipeline runs separately. Do not add these amounts to the headline and compare the result as the same metric.
Continuous workloads can end or be cancelled as part of normal operation. Review the workload’s behavior before interpreting these costs as waste.
Investigation workflow
Section titled “Investigation workflow”- Find the largest source or a spike in Failure Cost Trend.
- Open a resource from Top Offenders and inspect the affected runs or queries.
- Confirm the outcome and error in Databricks. Check whether retries, timeouts, or an upstream dependency caused repeated failures.
- Compare with Work Units or SQL Analysis using the same time range and filters.
- After fixing the cause, compare an equivalent later period once new data has been processed.
Failure cost is an investigation measure, not a promise that the whole amount is recoverable. Some failures occur during development or validation, and successful retries also consume compute.
Troubleshooting
Section titled “Troubleshooting”- Failure counts but little cost: check that billing and attribution have caught up with execution data.
- Totals differ from Work Units or SQL: align the range, scope, and cost mode, then account for cancellations and streaming exclusions.
- No data: verify the required execution tables, broaden filters, and check the latest extraction and processing results.
- Couldn’t load failure cost: retry or adjust the range. If it persists, send the exact error and scope using the support checklist.