LakeSentry Collector Troubleshooting

This page covers diagnosing and resolving issues with the LakeSentry collector. For initial setup, see Collector Deployment.

Note

This page applies to External Connector/collector deployments. For Direct Connection, troubleshoot sync failures from Settings → Connector, Common Issues, and Data Freshness.

Quick diagnosis

Start here when something looks wrong:

Symptom	Likely cause	Jump to
Region connector stuck on “Pending”	Collector never ran or failed on first run	First run failures
Ingestion is older than expected	Job schedule, source delay, or missed run	Missed runs
Connector reports an error	Inspect the validation or run error	Connection errors
Job run fails in Databricks	Collector error during extraction	Extraction errors
Data appears stale or incomplete	Checkpoint issues or missing table permissions	Checkpoint issues
Some tables missing from “Tables received”	Permission not granted for those tables	Missing tables

First run failures

If the region connector never moves past Pending status:

1. Check the Databricks job — Go to Workflows > Jobs and verify the collector job exists and has run at least once.
2. Check the job run status — Click the most recent run. If it failed, read the error output.
3. Verify the connection string — The connection string is consumed during configure. If configuration failed, generate a new connection string and re-run the configure step.

Common first-run errors:

Error	Cause	Fix
Connection string is not valid base64	Malformed or truncated connection string	Generate a new connection string from the region connector
Connection string is missing required fields	Incomplete or expired connection string	Generate a new connection string
LakeSentry API is not reachable	Network connectivity issue or invalid API URL	Verify the workspace has outbound HTTPS access to LakeSentry
Module not found: lakesentry_collector	Wheel not installed or wrong path	Check the wheel file path and re-install

Connection errors

Network connectivity

The collector needs outbound HTTPS access to the tenant API URL in its configuration. Check DNS, firewall rules, and proxy settings from the job’s runtime. Allow the configured host and port rather than assuming a generic API hostname. If using a proxy, confirm the job receives the required proxy configuration.

A successful health request verifies reachability only; ingestion still requires the correct connector credentials.

Authentication failures

If the collector was working and suddenly fails authentication:

• Connection string was regenerated — Someone generated a new connection string, invalidating the old token. Reconfigure the collector with the new string.
• Region connector was deleted and re-created — The connector ID changed. Reconfigure with the new connection string.

Extraction errors

Extraction errors occur when the collector fails to read from Databricks system tables.

Permission denied

Error: [TABLE_OR_VIEW_NOT_FOUND] Table or view not found: system.compute.clusters

or

Error: User does not have permission to SELECT on table system.compute.clusters

The service principal is missing SELECT access to the specified table. Grant permissions as described in Account & Connector Setup.

Unity Catalog not enabled

Error: Unity Catalog is not enabled for this workspace

System tables require Unity Catalog. Enable it on your Databricks account, or use a workspace that already has it enabled.

Table not available

Some system tables are only available in certain Databricks pricing tiers or regions. If a table doesn’t exist:

• The collector logs a warning and skips the table.
• The corresponding feature may have incomplete or no data.
• Other tables are still extracted normally.

This is expected behavior — the collector continues with whatever tables are accessible.

Missed runs

If the collector misses scheduled runs:

1. Check the Databricks job schedule — Verify the job is still enabled and the schedule hasn’t been modified.
2. Check cluster availability — If the job cluster fails to start (e.g., cloud provider capacity issues), the run fails before the collector code executes.
3. Check for long-running previous runs — If a run takes longer than the schedule interval, Databricks may skip or queue subsequent runs depending on the concurrency setting.

Tip

Set the job’s maximum concurrent runs to 1 to prevent overlapping runs. Avoid unnecessary overlapping extraction.

Checkpoint and resumability

The collector uses checkpoint-based incremental extraction. Each table has a watermark that tracks the last successfully extracted position.

How checkpoints work

Table type	Watermark column	Strategy
system.billing.usage	usage_start_time	Incremental — extract rows after the watermark
system.query.history	start_time	Incremental
system.compute.clusters	change_time	Incremental with historical configuration recovery
system.lakeflow.jobs	change_time	Incremental
system.compute.node_types	—	Full snapshot (reference table)

Incremental tables use the watermark to avoid re-extracting data. Full-snapshot tables are extracted completely on each run (they’re small reference tables).

Checkpoint recovery

If a run fails partway through extraction:

• Completed tables — The next run resumes from their saved checkpoints. It can read overlapping records again, and it re-reads cluster configuration history.
• In-progress table — The checkpoint was not updated. The next run re-extracts from the last successful watermark.
• Remaining tables — Not yet attempted. The next run extracts them normally.

This means the collector is crash-safe — you can safely stop a run at any point and the next run recovers automatically. There’s no need to manually reset checkpoints in normal operation.

Resetting checkpoints

A normal retry should resume from saved extraction state. Do not reset data or checkpoints as a first response to missing rows.

If historical rows need re-extraction, ask the deployment administrator or LakeSentry support to plan a bounded backfill. The collector CLI supports a run --since YYYY-MM-DD override; its use should account for source retention, extraction volume, and the affected tables. The Connector settings do not offer a checkpoint reset.

A backfill cannot restore source records that Databricks no longer retains. Keep the original error and run information before changing extraction state.

Missing tables

If the “Tables received” list on the region connector doesn’t include expected tables:

Missing table category	Check
Billing tables	These are account-level. They may only appear for one region if you have multiple regions.
MLflow tables	Ask LakeSentry support whether MLflow data is available for your tenant.
Serving tables	Requires optional permissions.
Audit tables	Requires optional permissions.
All regional tables	Verify the service principal has workspace-level access in the target region.

Log interpretation

Collector logs are available in the Databricks job run output. Key log entries to look for:

Log level	Message pattern	Meaning
INFO	Starting collection	Normal — extraction cycle beginning
INFO	Extracted (with table, records, status fields)	Normal — successful table extraction
INFO	Extraction batch pushed successfully	Normal — data sent successfully
WARN	Table not available in workspace, skipping	Permission not granted or table does not exist
INFO	No records found	Normal — no data in the extraction window
ERROR	LakeSentry API is not reachable	Connection string is invalid or API endpoint is down
ERROR	API error response (with status_code)	LakeSentry API rejected the push
ERROR	Extraction failed / Table extraction failed	Query error on a specific table

Common issues by symptom

Data shows in LakeSentry but appears incomplete

• Some workspaces missing — The workspace may be in a region without a region connector. Check the region mapping.
• Recent data missing — The collector may have a stuck checkpoint. Check the extraction checkpoints on the region connector detail page.
• Specific data type missing — Check if the corresponding system table is in the “Tables received” list.

Collector runs succeed but no data in LakeSentry

• Check the LakeSentry ingestion pipeline — Data flows through extraction → ingestion → transformation. If extraction succeeds but data doesn’t appear in dashboards, the issue may be downstream. Check Settings > Connector for ingestion errors.
• Check for deduplication — If the same data was already ingested (e.g., from a previous run with the same extraction ID), it’s deduplicated silently. This is expected.

Cost data lags behind real-time

Databricks publishes different source tables independently. Collection frequency alone cannot guarantee a maximum data age. Use Data Freshness to separate source publication, scheduling, extraction, and processing delays.

Getting help

If you’ve worked through this guide and the issue persists:

1. Gather the Databricks job run logs from the failed run.
2. Note the region connector status and last ingestion time from Settings > Connector.
3. Contact LakeSentry support with these details.

Next steps

• Collector Deployment — Setup and configuration reference
• Region Connectors — Managing regional data collection
• Data Freshness & Pipeline Status — Understanding end-to-end data latency