LakeSentry Collector Deployment for Databricks

The LakeSentry collector is a lightweight Python package that runs as a Databricks job in your workspace. It reads system tables, extracts the data, and pushes it to LakeSentry over HTTPS. You need one collector per region where you operate Databricks workspaces.

Note

Use Direct Connection in Settings → Connector for self-service setup. This page applies to External Connector deployments, which LakeSentry support enables.

For background on why collectors are regional, see Region Connectors.

How the collector works

The collector runs on the Databricks job schedule you configure (the example below uses every 15 minutes) and performs incremental extraction:

1. Reads system tables — Queries system.billing.*, system.compute.*, system.lakeflow.*, system.query.*, and other configured tables.
2. Applies checkpoints — Uses watermark columns (e.g., usage_start_time for billing, start_time for queries) to extract only data since the last run. Small reference tables (like price lists and node types) are extracted as full snapshots.
3. Pushes to LakeSentry — Sends the extracted data over HTTPS with an extraction ID for deduplication.
4. Updates checkpoints — Saves the new watermark positions so the next run picks up where this one left off.

Cycle duration depends on source volume, warehouse performance, network, and compute startup. Measure a successful run before choosing a schedule. Re-running from the same checkpoint is safe — LakeSentry deduplicates records that it extracts again.

Prerequisites

• A region connector configured in LakeSentry with a connection string (see Account & Connector Setup)
• A Databricks workspace in the target region with Unity Catalog enabled
• Permissions to create jobs and upload files in the workspace
• The service principal from your account connector must have access to system tables (see granting permissions)

Step 1: Upload the collector

Download the latest collector wheel from LakeSentry:

1. In LakeSentry, go to Settings > Connector.
2. Click Download Collector to get the .whl file.

Upload it to your Databricks workspace:

1. In Databricks, navigate to Workspace > your preferred directory (e.g., /Shared/lakesentry/).
2. Upload the .whl file.

Alternatively, upload to a Unity Catalog volume:

/Volumes/<catalog>/<schema>/lakesentry/lakesentry_collector-<version>-py3-none-any.whl

Step 2: Configure the collector

Run the configuration command to set up the collector environment. You can do this from a Databricks notebook or a one-time job:

%pip install /Workspace/Shared/lakesentry/lakesentry_collector-<version>-py3-none-any.whl

lakesentry-collector configure --connection-string "LAKESENTRY://..."

The environment file must persist and be readable by the scheduled job. A .env created on a temporary setup cluster will not automatically exist on a new job cluster. Use a deployment-approved persistent path or provision the required environment variables for each run. The CLI accepts an explicit path before the command:

lakesentry-collector --env-file /path/to/collector.env run

The configure command:

• Validates the connection string
• Verifies LakeSentry API connectivity
• Stores configuration in a .env file for the collector to use

Note

The connection string is decoded during configuration and its fields (API URL, connector ID, and collector token) are stored in the .env file. After setup, the original connection string is no longer needed.

Step 3: Create the Databricks job

Create a scheduled job that runs the collector:

1. In Databricks, go to Workflows > Jobs.
2. Click Create Job.
3. Configure the job:

Setting	Value
Task name	lakesentry-collector
Type	Python wheel
Package name	lakesentry_collector
Entry point	lakesentry-collector
Cluster	Use an existing shared cluster, or create a single-node job cluster
Schedule	Every 15 minutes

Cluster sizing

The collector is lightweight. A single-node cluster with minimal resources is sufficient:

Setting	Recommendation
Node type	Smallest available (e.g., Standard_DS3_v2 on Azure, m5.large on AWS)
Workers	0 (single-node / driver-only)
Autoscaling	Off
Auto-termination	10 minutes (cluster terminates between runs)

Tip

Using a shared or always-on cluster eliminates cold-start time but costs more. For most deployments, a job cluster that spins up for each run is the most cost-effective option.

Alternative: Notebook task

If you prefer a notebook-based approach:

1. Create a notebook with:

   %pip install /Workspace/Shared/lakesentry/lakesentry_collector-<version>-py3-none-any.whl
   from lakesentry_collector.cli import main
   main()

2. Schedule the notebook as a job (every 15 minutes).

Step 4: Start the schedule

1. On the job page, ensure the schedule is enabled.
2. Optionally, click Run Now to trigger the first collection immediately.
3. Monitor the first run to confirm it completes without errors.

Verifying the deployment

After the first successful run, verify in both Databricks and LakeSentry:

In Databricks

• The job run shows Succeeded status.
• Inspect the duration and table results; initial history loads can take longer than later runs.
• No errors in the task output logs.

In LakeSentry

Go to Settings > Connector and check the region connector:

Indicator	Expected state
Status	OK (green)
Last ingestion	Recent timestamp matching the job run
Tables received	Lists the system tables that were extracted

If the region stays in Pending status, see Collector Troubleshooting.

Monitoring collector health

LakeSentry monitors collector health automatically:

• Connector health alerts depend on connector state and notification settings. See Webhooks & Notifications for current alert behavior.
• Collector runs history is visible on the region connector detail page, showing each run’s status, duration, tables extracted, and rows pushed.
• Extraction checkpoints show the current watermark position for each table, so you can see how far behind the collector is if it’s been paused.

Updating the collector

When a new collector version is available:

1. Download the updated .whl file from LakeSentry.
2. Upload it to the same location in your Databricks workspace (overwriting the previous version).
3. Update the job’s wheel library reference or notebook installation path to the new version, then verify a run before relying on the schedule.

No reconfiguration is needed — the collector version is independent of the connection string and configuration.

Note

LakeSentry notifies you when a collector update is available. Check Settings > Connector for version status.

Schedule recommendations

Scenario	Schedule	Trade-off
Example starting cadence	Every 15 minutes	Good balance of freshness and cost
Near-real-time	Every 5 minutes	Fresher data, higher compute cost
Cost-conscious	Every 30-60 minutes	Lower cost, longer data lag

Data freshness depends on source publication, the collector schedule, extraction duration, and processing. See Data Freshness.

Multiple collectors per region

In most cases, one collector per region is sufficient. However, if you have a very high volume of system table data (thousands of active jobs, heavy query history), you can deploy multiple collectors in the same region with different table assignments. Contact LakeSentry support for guidance on partitioned collection.

Uninstalling the collector

To stop data collection for a region:

1. Disable or delete the Databricks job.
2. Remove the collector files from the workspace (optional).
3. Account for connector health alerts while collection is intentionally stopped; see Webhooks & Notifications.

To fully disconnect, also delete the region connector in LakeSentry. See Region Connectors for details.

Next steps

• Collector Troubleshooting — Diagnosing and fixing collector issues
• Region Connectors — Managing multi-region deployments
• How LakeSentry Works — Understanding the data pipeline from extraction to insights