Configure Lineage

Based on metadata from Fivetran, Alation auto-generates lineage between the destination data source in Fivetran and other data sources cataloged in Alation that represent Fivetran connectors (upstream lineage). If the source data source is not present in the catalog, then Alation won’t generate upstream lineage for the destination data source.

Before you configure the Fivetran connector in Alation, refer to Prerequisites.

Follow these steps to configure lineage:

  1. Configure Lineage Across Data Sources

  2. Fetch Fivetran Source and Destination

  3. Verify Data Source Mapping

  4. Run Extraction

Configure Lineage Across Data Sources

Lineage generation relies on the configuration where both data sources, source and destination in Fivetran, are cataloged in Alation as OCF data sources. For Alation to match metadata objects between data sources, you will need to configure the Additional Datasource Connections field in the settings of the source data source.

To configure cross-data source lineage:

  1. Open the settings page of the data sources (source and destination in Fivetran that are cataloged in Alation as OCF data sources) and go to the General Settings tab.

  2. Navigate to Application Settings > Additional Datasource Connections field and specify the source connection details (JDBC URI) from Fivetran for each source and destination. In the example below, we’re using a Snowflake connection string:

    • Format: <Host>:<Port>

    • Example: mysnowflake.us-east-1.snowflakecomputing.com:443

    For Google BigQuery and Salesforce, refer to the following host and port information:

    • Google BigQuery: <host:port> www.googleapis.com/bigquery/v2:443

    • Salesforce: <host:port> salesforce:80

    If multiple destinations use the same source, you can specify multiple values separated by commas:

    • 31.22.22.13:1551,mysnowflake.us-east-1.snowflakecomputing.com:443

    Note

    The same field Additional Datasource Connections may have been used to configure cross-source lineage between this data source and BI sources. If you find that values are already present in this field, append the new value separating it with a comma. Do not overwrite existing values.

  3. Click Save to save your changes.

Fetch Fivetran Source and Destination

The Fivetran OCF connector supports a preflight check before you fetch sources and destinations. The following preflight checks are performed:

  • Fivetran Platform Connector is configured in the Fivetran account and is at the account or destination level and in Active state.

  • Destinations have sources configured in the Fivetran account.

To fetch Fivetran source and destination, follow these steps:

  1. Open Lineage Settings in Fivetran OCF connector.

  2. Go to Lineage Configuration.

  3. Acknowledge and Run to trigger preflight checks and run the operation.

Based on the result of the Preflight checks: Success, Warning, fetching is triggered.

../../../_images/FivetranLineageSettingsAcknowledge.png

Preflight Checks on Fivetran

Scenarios for Success

  • Fivetran platform connector set up either at the account or individual level and in the Active state.

Scenarios for Failure

  • No sources (connectors) are set up in Fivetran.

  • No Fivetran platform connector is configured.

  • All Fivetran platform connectors are in a Paused state.

Scenarios for Warning

  • Fivetran platform connector is set up at the individual level for some of the destinations but not all.

  • Fivetran platform connector set up at the individual level for all destinations but a few are in a Paused state.

  • Some destinations don’t have sources (connectors) set up.

Verify Data Source Mapping

You can verify the mapping between Alation data sources and Fivetran sources or destinations under Lineage Settings > Lineage Configuration > Verify Data Source Mapping.

Unmapped Fivetran sources or destinations are highlighted and displayed in a table along with their host and port information. You can take action based on the information.

../../../_images/FivetranLineageSettingsVerifySourceMapping.png

Run Extraction

Lineage extraction fetches lineage metadata from the destination and generates a lineage between source and destination, and as well as across cross-systems.

If you have multiple Fivetran platform connectors, the lineage extraction is run in a sequential manner. This means that if lineage extraction fails for one of the connectors, it moves to the subsequent connectors.

You can schedule or Run extraction depending on your requirements.

../../../_images/FivetranLineageSettingsRunExtraction.png

Lineage Job History

You can view the lineage job history to view the number of lineage paths extracted by the connector.

../../../_images/FivetranLineageJobHistory.png

Dataflow Content

On catalog pages of dataflow objects generated from Fivetran, the Source field will display the value Fivetran in the data flow object.

The Dataflow Content field will display the name of the table at source and destination respectively.

Find more information on Dataflow objects in Dataflow Objects.

Temporary Lineage Objects

You may see objects marked as temporary (TMP) on your Lineage diagrams. Find more information about it in Badges in Configure Classic View of Lineage Charts. When you run MDE on the main (destination) data source, Alation will create temporary lineage objects in place of tables it cannot match in the catalog. Fivetran metadata-based lineage will include temporary objects when:

The source data source is cataloged in Alation, but you haven’t yet performed MDE from this source. The next MDE you perform on the source data source should remove them. The source data source is cataloged in Alation and MDE has been performed; however, the relevant source tables have not been extracted.

If you see temporary objects where you expect to see existing objects, check your data source configuration. You may also need to rerun MDE to bring the latest metadata updates into Alation.