Troubleshooting¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
Issues with Metadata Extraction¶
Problem: MDE fails with error “iterating schema children”¶
The issue may occur with older connector versions that use the legacy Spark driver. This driver sometimes cannot keep up the connection long enough for MDE to complete.
Solution¶
This issue was resolved starting with connector version 2.0.2.6259, where the driver was changed to the Databricks JDBC driver. See more in Build the JDBC URI.
Problem: Fetch Schemas fails with error “unable to instantiate HiveMetastoreClient”¶
The issue may occur when Hive metastore is not instantiated. Hive Metastore is the default Databricks connection catalog.
Solution¶
- Include the ConnCatalog and ConnSchema parameters into the connection JDBC URI:
ConnCatalog- Specifies the catalog which contains the metadata schema.ConnSchema- Specifies the schema inside the catalog in which metadata is stored.
Example:
ConnCatalog=en_dlake_cat;ConnSchema=data_governance;
Issues with Lineage Extraction¶
Problem: After upgrading the connector to version 1.0.3, MDE fails at ingestion¶
Connector logs will show the error “system.access.table_lineage table does not exist”.
Solution¶
Direct lineage extraction from system tables is a beta feature. The lineage system tables are not enabled by default and need to be enabled by your Databricks admin. See more in Lineage Extraction (Beta).
Problem: No lineage is generated after MDE¶
Solution¶
Verify that lineage data is available in the system lineage tables in Databricks.
Issues with QLI¶
Problem: After running QLI, some queries don’t appear in the History section of the Queries tab¶
Explanation¶
Some SQL statements are currently not supported by the Databricks SQL parser that is used to parse SQL statements during QLI, for example:
Partially qualified names in SQL statements are not parsed. Object names in query history must be fully qualified to be ingested.
The USE statements are not parsed.
Queries containing CTEs are not parsed.
Queries containing comments (COMMENT, COMMENT ON) are not parsed
This is a limitation with the current GSP and a known issue.
Problem: After QLI, parts of a query appear as separate queries in the catalog¶
Explanation¶
To support proper query parsing, accurate execution times count, and lineage resolution, batched queries are split into individual query statements before ingestion.
Problem: Queries that appeared in Preview don’t appear in the catalog after QLI¶
Explanation¶
The Preview is not directly tied to queries that will appear in the catalog. It returns 100 statements which may be a subset of all imported queries. For queries to appear in the catalog after QLI:
Queries must contain fully qualified object names with three-level namespace (catalog.schema.table).
The objects referenced in queries must have been cataloged with MDE.
Queries must not duplicate the queries that were ingested previously. Duplicated queries are not resolved.