Configure MDE

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

Note

Metadata Extraction process will be triggered when clicking on the “Run Extraction Now” button.

Without any configuration or filter, the process will extract all the metadata from the entire datasource at once.

If you are running Metadata Extraction for the first time, it is recommended running it with only a limited amount of data (i.e. a couple of schemas) to become familiar with the process.

To run Metadata Extraction for only a certain amount of schemas, a Selective Extraction needs to be performed. In order to enable and configure Selective Extraction, follow the steps in the Selective Extraction subsection below.

You can configure how the Dremio Connector will perform Metadata Extraction by accessing the Metadata Extraction tab in the Datasource settings:

../../../_images/DremioConfigMDE.png

Application Settings

  • Enable Raw Metadata Dump or Enable Ingestion Replay: The options in this drop list can be used to dump the extracted metadata into files in order to debug extraction issues before ingesting the metadata into Alation. This feature can be used during testing in case there are issues with Metadata Extraction. It breaks extraction into 2 steps: first, the extracted metadata is dumped into files and can be viewed; and second, it can be ingested from the files into Alation.

    • Enable Raw Metadata Dump: Select this option to dump the extracted metadata into a dump folder for debugging purposes. The dumped data will be saved in 4 files: attribute.dump, function.dump, schema.dump, table.dump files in the folder opt/alation/site/tmp/ inside Alation shell.

    • Enable Ingestion Replay: Select this option to ingest the metadata from the dump files into Alation.

    • Off - Disable the Raw Metadata Dump or Replay feature. Extracted metadata will be ingested into Alation.

Entities to Extract

Under the Entities to Extract section, checkboxes are used to customize metadata extraction down to the level of specific metadata types, such as tables, columns, and views. Consider the following points:

  • The extraction of schemas is always enabled and cannot be disabled.

  • The extraction of tables, columns and views is enabled by default, but can be disabled

  • You can disable or enable the metadata types you want to extract by enabling or disabling the corresponding option.

Selective Extraction

With Metadata Extraction, you can select the schemas to include or exclude from extraction. Selective extraction settings are used to apply a filter to include or exclude a list of schemas.

Enable the Selective Extraction toggle if you want only a subset of schemas to be extracted.

  • Click Get List of Schemas to first fetch the list of schemas. The status of the Get Schemas action will be logged in the Extraction Job Status table at the bottom of the Metadata Extraction page.

  • Once schema synchronization is complete, you will first be required to refresh the page. After refreshing the page, a drop-down list containing the source schemas will become enabled.

  • Select one or more schemas as required.

  • Check if you are using the desired filter option. Available filter options are described below:

Filter Option

Description

Extract all Schemas except

Extract metadata from all Schemas except from the selected Schemas.

Extract only these Schemas

Extract metadata only from the selected Schemas.

Click Run Extraction Now to extract metadata. The status of the extraction action is also logged in the Job History table at the bottom of the page.

Automated Extraction

If you wish to automatically update the metadata extracted into the catalog, under Automated and Manual Extraction, turn on the Enable Automated Extraction switch and select the day and time when metadata must be extracted. The metadata extraction will be automatically scheduled to run on the selected schedule.