LakeSentry Databricks Cost Optimization FAQ
Answers to common questions about LakeSentry from platform engineering, FinOps, and data teams.
General
Section titled “General”What is LakeSentry?
Section titled “What is LakeSentry?”LakeSentry is a Databricks cost observability and optimization platform. It connects to your Databricks account, ingests system table data, and provides cost breakdowns, anomaly detection, waste identification, and safe optimization actions — all through a single dashboard. See What is LakeSentry for a full overview.
How is this different from Databricks’ built-in cost tools?
Section titled “How is this different from Databricks’ built-in cost tools?”Built-in tools show raw usage data per workspace. LakeSentry normalizes cost across all your workspaces, detects anomalies and waste automatically, and shows the evidence for each finding — no SQL required. After your approval it can terminate idle clusters and cancel runaway runs. You apply other recommendations in Databricks.
Key differences:
- Cross-workspace visibility — See all workspaces and regions in one view
- Automatic detection — Anomalies, waste, and optimization opportunities surfaced without manual queries
- Attribution — Costs attributed to teams, projects, and owners with confidence tiers
- Safe actions — Approve and execute optimizations with guardrails and audit trail
How does LakeSentry compare to other optimization platforms?
Section titled “How does LakeSentry compare to other optimization platforms?”LakeSentry is read-only by default, charges no per-workspace fees, and focuses on safe automation with approval workflows and guardrails. Most alternatives require broad write access from day one or charge per workspace, which penalizes teams with many environments.
Who is LakeSentry for?
Section titled “Who is LakeSentry for?”LakeSentry serves three primary audiences:
- Platform engineering teams — Pinpoint runaway jobs, warehouse drift, and regressions before they impact reliability
- FinOps teams — Get normalized cost visibility across workspaces with evidence-backed optimization priorities
- Data and ML teams — See where experiments and serving endpoints burn budget, then act with clear guardrails
Setup and requirements
Section titled “Setup and requirements”Does LakeSentry require Unity Catalog?
Section titled “Does LakeSentry require Unity Catalog?”Yes. Unity Catalog is required for reliable access to the system catalog system tables used by LakeSentry. If your Databricks account doesn’t have Unity Catalog enabled, you’ll need to enable it before connecting. See Connecting Your Databricks Account for prerequisites.
How many workspaces can I connect?
Section titled “How many workspaces can I connect?”All plans include unlimited workspaces with no per-workspace fees. Add connectors for each Databricks region where you operate.
We operate in multiple regions — does that matter?
Section titled “We operate in multiple regions — does that matter?”Yes. You’ll connect each region you operate in by creating a Region Connector. Direct Connection is the self-service path. LakeSentry support enables External Connector deployments. LakeSentry shows connector health per region and continues monitoring wherever connectors are healthy.
How long does setup take?
Section titled “How long does setup take?”Most teams complete the initial setup in under 30 minutes:
- Create a service principal and grant permissions (10–15 minutes)
- Create a Direct Connection connector in LakeSentry (2 minutes)
- Validate credentials and trigger the first sync
After the first successful sync, data begins appearing in dashboards after extraction and transformation complete. See the Quick Start Guide for the full walkthrough.
What permissions does LakeSentry need?
Section titled “What permissions does LakeSentry need?”LakeSentry requires a Databricks service principal with access to a SQL warehouse and SELECT access to system tables. The required tables cover billing, compute, workload, query, serving, storage, and access metadata.
The full permission requirements are documented in Account & Connector Setup.
Security and data access
Section titled “Security and data access”Can we run read-only?
Section titled “Can we run read-only?”Yes. LakeSentry can run read-only for reporting and detection. Databricks write permissions are only needed for approved executable actions; the current executor mutates Databricks only for terminating idle clusters and canceling runaway runs. See Action Plans & Automation for details on the safety model.
Do you access our business data?
Section titled “Do you access our business data?”No. LakeSentry queries Databricks system tables for usage and cost metadata only. It never accesses your business data, notebooks, or query results. The system tables contain billing records, compute metadata, job run history, and similar operational data.
Do you store query text?
Section titled “Do you store query text?”In the current release, yes. LakeSentry stores raw query text because it improves insight quality — for example, identifying which queries contribute most to warehouse costs. If this is a concern, contact us to discuss deployment options while self-serve controls for query text handling are being developed.
How safe is automation?
Section titled “How safe is automation?”LakeSentry runs read-only by default. Every action plan requires approval by an admin or owner. After approval, LakeSentry executes two plan types: terminating an idle cluster and canceling a runaway run. You apply all other plans yourself in Databricks.
Approvals are logged in the Audit Log. See Action Plans & Automation for the plan types and safety tiers.
Is there unattended automation?
Section titled “Is there unattended automation?”No. Every action plan requires review and approval before execution. You review and approve plans on the Insights page.
Pricing and plans
Section titled “Pricing and plans”How does pricing work?
Section titled “How does pricing work?”LakeSentry offers Free, Standard, and Pro tiers. Plans differ in user count, history depth, and feature entitlements. Check Settings → Billing for the tenant’s current eligibility. See LakeSentry pricing for current plan details and rates.
How do I sign up?
Section titled “How do I sign up?”Create a free account at app.lakesentry.io/signup. No credit card required. The Free plan stays free and lets you connect your full Databricks environment to evaluate LakeSentry.
Can we customize DBU prices?
Section titled “Can we customize DBU prices?”Yes. LakeSentry uses Databricks list prices by default and supports optional DBU price overrides so your dashboards reflect your contract reality. Configure price overrides in Settings.
Is there a free trial?
Section titled “Is there a free trial?”The Free plan is free forever with no credit card required. It includes unlimited workspaces, 1 user, and 3 months of history — enough to evaluate the platform with your full Databricks environment.
Cost data and accuracy
Section titled “Cost data and accuracy”Why don’t my costs match the Databricks console?
Section titled “Why don’t my costs match the Databricks console?”This is the most common question we receive. Differences are usually caused by:
- Time zone differences — LakeSentry uses UTC for all calculations
- Cost model — LakeSentry uses list prices by default; your Databricks console may show negotiated pricing
- Scope — LakeSentry only shows costs for connected workspaces and regions
See Cost Discrepancies for a detailed breakdown of common causes.
What does “unattributed” cost mean?
Section titled “What does “unattributed” cost mean?”Unattributed cost is spend that couldn’t be assigned to a specific team, project, or owner. This happens when resources lack tags, ownership signals, or matching attribution rules.
LakeSentry uses a tiered attribution model with four confidence levels — Exact, Strong, Estimated, and Unattributed. See Cost Attribution & Confidence Tiers for how to reduce unattributed costs.
How far back does historical data go?
Section titled “How far back does historical data go?”This depends on two factors:
- Databricks system table retention and extraction depth — Available history varies by table. Initial extraction follows the table’s lookback policy and cannot recover records no longer available at the source.
- LakeSentry plan limits — Free retains 3 months, Standard retains 12 months, Pro retains unlimited history.
The first sync loads history within the applicable extraction window and plan limits. See Data Freshness when expected history is missing.
Features and capabilities
Section titled “Features and capabilities”How does anomaly detection work?
Section titled “How does anomaly detection work?”LakeSentry compares recent values against a historical baseline. Cost Spike and Duration Spike use Z-score statistical analysis. Other anomaly findings compare failure rates, period costs, or budget projections against thresholds. Each finding shows its evidence, such as the baseline, the recent average, and the cost increase.
The requirements vary by finding. See Anomaly Detection for the method and the Insight Type Catalog for each finding.
What types of waste does LakeSentry detect?
Section titled “What types of waste does LakeSentry detect?”LakeSentry identifies several types of waste:
- Idle clusters — Running clusters with no active workloads
- Overprovisioned workers — Clusters with consistently low utilization
- Zombie endpoints — Serving endpoints with ongoing cost and no requests for 30 days
- Expensive queries — Queries consuming disproportionate resources
- Data retention waste — Storage costs for unused or rarely accessed data
See the Insight Type Catalog for every finding.
Can LakeSentry track ML and AI workloads?
Section titled “Can LakeSentry track ML and AI workloads?”LakeSentry tracks:
- MLflow experiments — Ask LakeSentry support whether MLflow data is available for your tenant. See MLflow.
- Model serving endpoints — Cost tracking and traffic-to-cost correlation. See Model Serving.
- Training jobs — Cost per training run via the standard Work Units tracking.
Does LakeSentry support budgets and alerts?
Section titled “Does LakeSentry support budgets and alerts?”Yes. You can create budgets at the workspace, organization unit, department, or team level with threshold alerts. See Budgets and Organizational Hierarchy & Budgets for details.
Troubleshooting
Section titled “Troubleshooting”Where do I start if something isn’t working?
Section titled “Where do I start if something isn’t working?”Start with the Common Issues page, which has a quick reference table mapping symptoms to solutions. For Direct Connection latency questions, see Data Freshness & Pipeline Status. For External Connector deployments, see Collector Issues.
How do I contact support?
Section titled “How do I contact support?”- Gather relevant details (error messages, connector status, timestamps).
- Contact your LakeSentry support contact with the support checklist.
- For External Connector failures, also include the Databricks job details in the escalation checklist.
Next steps
Section titled “Next steps”- What is LakeSentry — Product overview and value proposition
- Quick Start Guide — Get up and running in minutes
- Common Issues — Solutions to frequent problems