Configure Lineage

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Based on metadata from Matillion, Alation auto-generates lineage between the source and target data sources that are cataloged in Alation.

If the source and destination data sources are not cataloged, then lineage will not be generated in Alation.

Configure Lineage Across Data Sources

Cross-data-source lineage is generated only when both Matillion source and target systems are cataloged in Alation as OCF data sources.

If the host or port in Alation doesn’t match Matillion, Alation may fail to map objects between sources.

When you can skip this configuration:

You do not need this configuration if the <hostname>:<port> in the JDBC URI for the Alation data source exactly matches the <hostname>:<port> used by the corresponding Matillion sources and destinations.

What you’ll do if the host and port information don’t match:

Add the Matillion <hostname>:<port> values for each participating system to the Additional Datasource Connections field on the relevant Alation data sources.

To configure cross-data source lineage:

  1. Open the settings page of the data sources (source and destination in Matillion that are cataloged in Alation as OCF data sources) and go to the General Settings tab.

  2. Navigate to Application Settings > Additional Datasource Connections field and specify the source connection details (JDBC URI) from Matillion for each source and destination. In the example below, we’re using a Snowflake connection string:

    • Format: <Hostname>:<Port>

    • Example: mysnowflake.us-east-1.snowflakecomputing.com:443

    If multiple destinations use the same source, you can specify multiple <hostname>:<port> values separated by commas:

    • 31.22.22.13:1551,mysnowflake.us-east-1.snowflakecomputing.com:443

    Note

    The same field Additional Datasource Connections may have already been used to configure cross-source lineage between this data source and BI sources. If you find that values are already present in this field, append the new value, separating it with a comma. Do not overwrite values that are already present in this field.

  3. Click Save to save your changes.

Configure Lineage Extraction Scope (Optional)

The Matillion OCF connector allows you to configure how far back in time to extract lineage data:

  1. On the Lineage Settings page, locate the Lineage lookback (days) field.

  2. Enter a value between 1 and 60 days (default is 30 days).

    The Lookback Days setting defines the period for data extraction, calculated as today minus X days. For example, a value of 7 means the connector retrieves data from the past seven days.

Run Extraction

Lineage extraction fetches lineage metadata from the Matillion API and generates lineage between source and target, as well as across cross-systems.

Under the Run extraction section (Lineage Settings > Step 1: Run Extraction), click Run Extraction to extract lineage metadata on demand.

The status of the extraction action is logged in the Extraction Job Status table under the Lineage Job History tab.

Schedule Extraction

You can also schedule the extraction. To schedule the extraction, perform these steps:

  1. On the Settings page of your Matillion ELT source, go to the Lineage Extraction.

  2. Under the Step 1: Run extraction section, turn on the Enable extraction schedule toggle.

  3. Using the date and time widgets, select the recurrence period and day and time for the desired lineage extraction schedule. The next lineage extraction job for your Matillion ELT source will run on the schedule you have specified.

Note

Here are some of the recommended schedules for better performance:

  • Initially, set the extraction to run with a 30-day range. Later, you can reduce the range to 2 days and schedule it to run daily.

  • Schedule extraction to run for every 12 hours at the 30th minute of the hour.

  • Schedule extraction to run for every 2 days at 11:30 PM.

  • Schedule extraction to run every week on the Sunday and Wednesday of the week.

  • Schedule extraction to run for every 3 months on the 15th day of the month.

../../../_images/MatillionLineageSettingsRunExtraction.png

Lineage Job History

Go to Lineage Settings > Lineage Job History tab to view the status of the extraction duration, errors, and details of lineage paths extracted by the connector.

../../../_images/MatillionLineageJobHistory.png

Dataflow Content

After successful lineage extraction, you can view the lineage diagrams in Alation to understand data flow between your source and destination systems. The lineage shows:

  • Table-level lineage: How data flows between tables across systems

  • Column-level lineage: How individual columns are transformed and mapped

  • Cross-system connections: Relationships between Matillion jobs and data sources

To view lineage:

  1. Navigate to your source or target data source.

  2. Go to a table that’s part of a Matillion transformation.

  3. Click the Lineage tab to see upstream and downstream relationships.

  4. Use the lineage controls to expand or collapse the diagram as needed.

The actual SQL is not available in the dataflow content, instead, it will display the Project name, Pipeline file name, input(s) and output.

Find more information on Dataflow objects in Dataflow Objects.

../../../_images/MatillionDataflowContent.png

Interpreting Temporary Lineage Objects

You may see objects marked as temporary (TMP) on your lineage diagrams. Find more information about it in Badges in Configure Classic View of Lineage Charts. When you run MDE on the main (target) data source, Alation will create temporary lineage objects in place of tables it cannot match in the catalog.

Matillion metadata-based lineage will include temporary objects when:

  • The source data source is cataloged in Alation, but you haven’t yet performed MDE from this source. The next MDE you perform on the source data source should remove them.

  • The source data source is cataloged in Alation and MDE has been performed; however, the relevant source tables have not been extracted.