Automate Monitor Creation with DQ Pilot¶

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

DQ Pilot is an agent that analyzes your catalog signals, including query logs, Critical Data Element (CDE) tags, BI lineage, and transformation definitions, to identify tables that need monitoring. It also proposes tuned checks and anomaly metrics for those tables.

Building meaningful data quality coverage across a large catalog typically requires evaluating tables and configuring checks manually. DQ Pilot reduces the upfront effort by proposing draft monitors for you to review, adjust, or dismiss. This helps you base coverage on observed data behavior.

Nothing goes live without your review. Every check, threshold, and anomaly metric that DQ Pilot proposes lands in a draft that you approve before the monitor runs.

You can use DQ Pilot in two ways:

Set up coverage in bulk across many tables, a schema, or an entire data source using the Set up Coverage wizard.
Scan for candidate tables while creating a single monitor using Run DQ Pilot in the New Monitor wizard.

Prerequisites and Access¶

Access by Capability¶
Capability	Available To
DQ Pilot for building an individual monitor	Any user with permission to create monitors, including Server Admins, Catalog Admins, Source Admins, and Stewards.
Set up Coverage in bulk	Server Admins, Catalog Admins, and Data Source Admins.

DQ Pilot uses signals from your catalog to generate recommendations. The following are not required, but they produce stronger recommendations:

Enable profiling on the schemas you want to work with. DQ Pilot uses profile data to derive check thresholds and validity rules. For more information, see Profile Data on Tables and Columns in Data Quality.
Populate your catalog with BI lineage, CDE tags, query activity, and ownership metadata.
DQ Pilot works with the same data sources supported by Alation Data Quality. For the full list, see Supported Data Sources.

Note

DQ Pilot is controlled by the AI Features in Data Quality toggle in Settings > Feature Flags. This same toggle also controls Recommend Checks, Generate check descriptions, and Error message translation. None of these have individual toggles of their own. When the master switch is off, DQ Pilot and Set up Coverage are not visible. For more information, see Administer Data Quality Settings.

Set Up Coverage in Bulk¶

Use Set up Coverage to deploy monitoring across many tables, a schema, or an entire data source at once.

Navigate to the Alation Data Quality application and on the Overview page, click Set up Coverage.
In Select Tables, select one or more tables.
1. Optionally, use the filters or the search field to narrow the list by domain, source, schema, table type, or other criteria.
2. To select every table in a schema or data source, search for the schema or data source and select all of its results.
Click Continue.

In Coverage Mode, select one of the following options:

Coverage Mode Options¶

Option

What It Does

Best For

Baseline Observability

Applies the same three checks (volume, freshness, and schema drift) to every selected table. The monitor goes live immediately.

Fast, uniform coverage across many tables with no review step.

Built by DQ Pilot

The agent profiles each table using profiling statistics, query and usage patterns, key relationships, and catalog metadata, then proposes tuned checks and anomaly metrics. One monitor is proposed per table.

Coverage tailored to each table’s actual data behavior, with a review step before anything goes live.

For guidance on choosing between these two options, see Baseline Observability below.

If you selected Built by DQ Pilot, click Start DQ Pilot Build.

Note

A progress screen shows the build in progress. To keep working elsewhere while the build completes, click Close, DQ Pilot Continues in Background. A notification appears at the top of the Data Quality application when the build finishes, with a link to review the proposed monitors.
If you selected Baseline Observability, proceed to Review & Deploy your selected options.
Click Deploy N monitors — {schedule}.

The monitors are created and start running on their configured schedule.

Baseline Observability¶

Baseline Observability applies table-level anomaly metrics to every table in scope. It is the fastest way to establish coverage across many tables at once.

The following metrics are applied to each table:

Volume: Detects unexpected changes in row counts.
Freshness: Detects delays in data arrival compared to observed patterns.
Schema drift: Detects changes in the table’s schema, such as column additions, deletions, or type changes.

Baseline Observability does not apply column-level checks. It is designed to tell you when a table stops receiving data, drifts outside its freshness pattern, or has its schema changed.

Use Baseline Observability when any of the following apply:

You are new to data quality monitoring and want broad initial coverage before deeper configuration.
You have many tables where the operational health of the table matters more than the correctness of specific column values.
You want to give data consumers immediate visibility into whether tables are behaving as expected.

Use Built by DQ Pilot instead when any of the following apply:

You want column-level quality checks, such as completeness, uniqueness, or validity, alongside anomaly metrics.
You want DQ Pilot to propose thresholds and rationale for each check.
The tables carry significant business risk and warrant deeper monitoring than volume, freshness, and schema drift alone.

Run DQ Pilot While Creating a Monitor¶

You can also scan for candidate tables while building a single Table Monitor or SDK-Enabled Monitor. For more information on these monitor types, see Add a Monitor.

Start creating a new monitor and continue to the Select Scope step.
In the banner labeled Not sure which tables to monitor?, click Run DQ Pilot.
In the Agent Table Scan panel, select one or more Data sources.
Optionally, select one or more Schemas to narrow the scan to a single data source.

Note

The Schemas field is available only after you select a single data source.
Click Start scan.

A progress screen confirms that the scan has started. To keep working elsewhere while the scan completes, click Close, DQ Pilot Continues in Background.
When the scan finishes, a notification appears at the top of the Data Quality Overview page showing the number of tables recommended. Click Review in the notification to open the recommended tables.

Review and Approve DQ Pilot Drafts¶

Every monitor that DQ Pilot proposes is saved as a draft until you review and launch it.

Navigate to the Alation Data Quality application and click the Monitors tab.
Click Drafts.
Click the DQ Pilot Drafts tab.

Each draft shows the target table, the wizard step it reached, the number of checks and anomaly metrics proposed, and when it was last edited. DQ Pilot names each draft using the fully qualified name of the target table. For a draft that spans multiple tables, DQ Pilot uses the first table’s fully qualified name. You can rename the monitor before publishing, and edit its auto-generated description.

Note

Manually saved drafts appear separately under the My Drafts tab. For more information on saving a draft manually, see Add a Monitor.
Click Review on the draft you want to evaluate.

The standard New Monitor wizard opens at the step where the draft left off, pre-filled with DQ Pilot’s proposed configuration.
Review the proposed checks in Configure Checks.
1. To remove a proposed check, click the trash icon next to it.
2. To adjust a proposed check, click the row to expand its configuration.
Review the proposed anomaly metrics in Anomaly Detection.

Note

Anomaly metrics require a 30-day warm-up period before they start flagging anomalies. For more information, see Analyze Anomalies.

Important

SDK-Enabled Monitors do not include an Anomaly Detection step, so DQ Pilot proposes checks only, not anomaly metrics, for this monitor type.
In Schedule & Launch, review the monitor name, schedule, notifications, and priority.

DQ Pilot leaves the schedule and notifications blank by default, so you make a deliberate choice for each.

Important

Publishing a draft with no schedule and no notifications configured means the monitor will not run and will not alert anyone on failure. Configure both before publishing.

DQ Pilot sets the user who initiated the run as the default owner. Add additional owners as needed.
Do one of the following:
1. Click Launch Monitor to activate the monitor.
2. Click Save as draft to keep your changes without activating the monitor.
3. Click Back to review earlier steps.
Note

If your organization requires approval for new monitors, launching a DQ Pilot draft follows the same approval process as any other monitor. For more information, see Approval Workflow and Governance.
To discard a draft you no longer need, click the trash icon next to it in the Drafts list.

Note

Drafts remain in the Drafts list until you publish or delete them. Drafts do not consume Alation Compute Units (ACUs).

Understand DQ Pilot Recommendations¶

Every recommendation DQ Pilot produces is explainable at two levels: why it recommended the table, and why it proposed each specific check.

Table-Level Rationale¶

Explains why the agent recommended a table for monitoring, including the signals that contributed. For example, DQ Pilot might cite high BI lineage consumption from several downstream reports, the presence of CDE-tagged columns, and a daily transformation cadence.

Per-Check Rationale¶

Explains why the agent proposed a specific check and set a specific threshold. This rationale is grounded in observed data where profiling is available. For example, if the order_id column shows a 0% null rate in its profile, DQ Pilot may recommend a completeness check with a 0.1% threshold to catch quality regressions early.

Confidence Levels¶

DQ Pilot marks each proposed check with a confidence indicator.

DQ Pilot Confidence Levels¶
Level	What It Means
High	The check is corroborated by multiple signals, for example profile data plus query patterns. These are the strongest recommendations.
Medium	The check is based on a single strong signal or an uncorroborated inference. Review these more closely.
For Review	The agent identified a potential rule but could not corroborate it. These are surfaced for evaluation and are not applied by default.

Honest Signal Framing¶

Where a signal is inferred rather than directly observed, the rationale states that. For example, write activity for a table may be inferred from its naming pattern rather than from direct observation of write operations. DQ Pilot describes the inferred nature of a signal so you can weigh it appropriately.

Review Checklist Before Publishing¶

DQ Pilot is designed to support your judgment, not replace it. Review both the rationale and the proposed checks before publishing a monitor. In particular, verify the following.

Thresholds are appropriate for your business context.
No essential checks are missing.
Notification recipients, ownership, and schedule are configured.

Best Practices¶

For your first identification run, choose a schema you know well so you can evaluate what DQ Pilot recommends and what it skips.
Use Baseline Observability across your catalog first to establish broad visibility, then use DQ Pilot on individual tables or in bulk on selected schemas to add column-level checks where they matter most.
Configure notifications and a schedule before publishing. DQ Pilot leaves both blank so you make a deliberate choice. A published monitor with no schedule and no notifications will not run and will not alert anyone on failure.
Reassess as your catalog matures. DQ Pilot’s recommendations improve as your catalog fills in with lineage, CDE tags, and query activity.

Limitations¶

Identification typically takes 5 to 30 minutes and runs in the background. A notification appears when the run finishes. Very large catalogs, with tens of thousands of tables, can take longer.
DQ Pilot uses a language model to identify patterns that regex-based signals cannot catch. These proposals require corroboration from another signal, such as profile data or query patterns, to reach High confidence. An uncorroborated proposal is surfaced at Medium confidence or For Review, and is not applied by default.
Set up Coverage can accept large scopes, but very large scopes, with thousands of tables, increase the time required to build monitors. Consider processing large scopes in batches when possible.
DQ Pilot is designed to support steward and admin judgment, not replace it. Review every recommendation before publishing, since the agent works from catalog signals and only you know your business context.