Enable Lineage

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Lineage is data about the origin of data and its movement through an organization’s data ecosystem. Lineage documents how target data objects are created from source data objects. To learn more about viewing lineage in the Alation catalog, see Discover Lineage.

Snowflake data sources support table-level and column-level lineage. Table-level lineage is generated during metadata extraction and query log ingestion by default. Column-level lineage is an additional feature that requires the Snowflake column-level lineage parser add-on. Contact Alation about purchasing this add-on.

Note

Alation expects that queries ingested during metadata extraction and query log ingestion use fully qualified object names. Ingesting queries with partially qualified object names may result in incomplete lineage information.

The parsing of queries that include outer joins using the plus (+) operator is supported.

Enable Column-Level Lineage

To enable column-level lineage:

  1. Log in to your Alation instance as a Server Admin.

  2. Click the Admin Settings icon gear on top right to open the Admin Settings page.

  3. Under Server Admin, click Feature Configuration to open the feature configuration page.

  4. Activate the toggle Automatically extracts Column Level Lineage for Snowflake data sources.

  5. Click the Save changes button.

    ../../../_images/DS_Snowflake08.png

  6. In the Verify Feature Configuration Change dialog, click the Save Configuration button.

    ../../../_images/DS_Snowflake09.png

View Column-Level Lineage

With the column-level lineage add-on enabled, Alation will automatically generate column-level lineage data based on the existing metadata that was previously extracted with MDE, QLI, or ingested from Compose.

Note

Once the feature flag is enabled, the column-level lineage will be automatically generated only for the view definitions that are complete and already added to table-level lineage.

Complete view definition:

CREATE VIEW adbc_database_01.query_execution_main.table_view AS
  SELECT src.*
  FROM (SELECT col1, col2, col3 FROM tpch.sf1.customer LIMIT 10) src;

Incomplete view definition:

SELECT src.*
FROM (SELECT col1, col2, col3 FROM tpch.sf1.customer LIMIT 10) src;

More column-level lineage data will be created after you perform metadata extraction (MDE), query log ingestion (QLI), and (or) use Compose to run queries that create or update column objects.

To view column-level lineage data:

  1. Go to the Lineage tab on the catalog page of a table object that should have column-level lineage.

    ../../../_images/DS_Snowflake10.png

  2. Expand the table or view lineage node to see the columns of the table or view and links to the dataflow object.

    ../../../_images/DS_Snowflake11.png

  3. Click a column you want to trace lineage for. The column-level lineage paths for that particular column become highlighted.

    ../../../_images/DS_Snowflake12.png

Note

If the column-level lineage feature flag is disabled, the column-level lineage links that were generated previously are not removed. Only the future queries and query history will not be processed.