Databricks Data Freshness and Pipeline Status

Dashboard freshness depends on when Databricks publishes source data, when LakeSentry next extracts it, and when processing finishes.

The data pipeline

Stage	What happens	What to check
Source availability	Databricks publishes billing and operational records	Whether the expected records exist in the source table
Extraction	Direct Connection reads available records; an External Connector pushes them	Schedule, run history, table results, and errors
Ingestion	LakeSentry validates and stores received records	Whether the expected tables and rows arrived
Processing	Records are transformed, attributed, and aggregated for dashboards and insights	Pipeline summary and Ledger Load status

Expected freshness by data type

Billing, query history, compute metadata, and Lakeflow execution history are published independently. One data type can arrive before another; costs and execution counts may therefore settle at different times. Today’s figures can be incomplete, and later source corrections can change past totals.

There is no single end-to-end latency that applies to every table or account. Allow for source publication delay + time until the next sync + extraction and processing time. Initial backfills can take longer than incremental syncs.

For example, a record that becomes available just after a daily sync may wait almost 24 hours for the next scheduled run, before extraction and processing. An hourly schedule reduces that waiting time but cannot make Databricks publish the source record sooner.

Checking pipeline status

An admin or owner can check each stage in Settings → Connector:

• Credential validation — Whether LakeSentry can access Databricks.
• Data Sync — The schedule, the last run of each connector, table results, and sync history.
• Ledger Load — Processing jobs, with run history and logs.

Follow What to do when data is stale to diagnose a delay. For connector health alerts, see Webhooks & Notifications.

Understanding lag

Symptom	Check	Next action
Empty immediately after setup	Validation, first extraction, and Ledger Load	Fix the first reported failure or allow active processing to finish
One region is stale	That connector’s last run and table errors	Restore access or connectivity for the affected region, then sync
All regions are stale	Whether Data Sync is paused or runs are failing	Resume an appropriate schedule or resolve the shared error
One data type is missing	Source availability and the table’s extraction result	Verify grants and availability before treating absence as zero usage
Extraction succeeds but dashboards remain old	Ledger Load and pipeline summary	Inspect processing errors and escalate with run details
Older data is missing	Initial extraction depth, source retention, and plan history limits	Confirm available history; a fresh sync cannot recover expired source records

What to do when data is stale

Step 1: Check extraction

Open Settings → Connector → Data Sync, inspect the latest run, and expand table details. For a failed table, record the exact error and resolve the specific credential, warehouse, permission, or connectivity issue.

For support-enabled External Connector deployments, inspect the Databricks job output using Collector Issues.

Step 2: Check for Databricks-side delays

An administrator with access to the source table can check whether the expected data has been published. For example:

SELECT MAX(usage_end_time) AS latest_usage_end_time
FROM system.billing.usage;

This shows the latest usage that Databricks published. Filter by workspace to investigate one workspace. A quiet environment can have no new usage.

Step 3: Check LakeSentry processing

If the expected rows were extracted, inspect Ledger Load for active or failed jobs. Record the job, run time, and error. Avoid repeatedly triggering extraction when the failure is in downstream processing; include those details in a support report.

Step 4: Trigger a manual sync

After resolving the cause, click Sync Now in Data Sync. It also works when automatic sync is paused. Wait for extraction and processing to finish, then reload the dashboard with the intended time range and filters.

Optimizing data freshness

Data Sync schedule tuning

Schedule	Effect
Every hour	More frequent extraction, with more frequent warehouse work
Every 4 hours	Less frequent extraction and a longer possible wait for new records
Daily at 8 AM UTC	Default daily cadence; newly available records can wait until the next day
Paused	No automatic sync; manual Sync Now remains available

The schedule selector in Data Sync applies the chosen cadence to all listed direct connectors. It is not a separate per-region scheduling control.

Extraction settings

In Databricks App deployments, Extraction Settings includes the run timeout and extraction slice size. Slicing divides incremental table extraction into smaller query windows; snapshot tables and full-history windows are not split. Inspect the failing table and warehouse query before changing these values. A longer timeout does not fix missing permissions, and smaller slices do not bypass source retention.

Multiple regions

Review each connector separately, even when they share a schedule. Each region extracts its own operational tables.

Next steps

• Common Issues — Missing data, cost discrepancies, and support checklist
• Collector Issues — External Connector diagnosis
• How LakeSentry Works — Pipeline concepts