Databricks Unity Catalog OCF Connector Release Notes¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
October 30, 2024¶
Databricks Unity Catalog OCF Connector: Version 3.1.3¶
Enhancements¶
SDK Upgrade¶
The SDK version is now upgraded to 4.9.7 along with updates to multiple packages in JVM.
This upgrade addresses the following Common Vulnerabilities and Exposures (CVEs): CVE-2024-47554, CVE-2024-9143. (AL-176849)
Fixed Issues¶
URL in Dataflow Content in Lineage Graph Displays Null for Hostname¶
The URL shown in the dataflow content in the lineage graph for navigating to the original workbook displays NULL in place of hostname for certain JDBC URIs (https://null/?o=1050xxxx#notebook/xxxx).
This issue is now resolved. (AL-172526)
Message Prompting User Action on Irrelevant Objects¶
An action message appears prompting users to apply grant access on objects that are not available or are irrelevant to the user.
This issue is resolved. Now, a warning message appears informing the users to ignore the action message if it’s not relevant. (AL-172872).
October 16, 2024¶
Databricks Unity Catalog OCF Connector: Version 3.1.1¶
Fixed Issues¶
Lineage from External Tools Fails to Appear in Lineage Extraction¶
Connector fails to extract the lineage generated from external entity types such as dbt.
The issue is now resolved (AL-164977).
September 18, 2024¶
Databricks Unity Catalog OCF Connector: Version 3.1.0¶
Enhancements¶
FIPS Compliance¶
The JDK is now upgraded to Temurin-11.0.24+8. This upgrade addresses the following Common Vulnerabilities and Exposures (CVEs): CVE-2024-45490, CVE-2024-45491, CVE-2024-45492, CVE-2024-6119 (AL-136317)
September 04, 2024¶
Databricks Unity Catalog OCF Connector: Version 3.0.1¶
Fixed Issues¶
Query Log Ingestion Fails for Empty QueryText¶
Query Log Ingestion Fails with the following error for empty or NULL QueryText:
Query text is missing in the query statement. Ensure that queryString column has data.
This issue is now resolved. (AL-169062)
Unable to Extract Struct Definitions for Columns of Struct Data Type¶
The connector fails to extract the struct definitions for Columns of Struct data type.
This issue is now resolved. (AL-166865)
August 14, 2024¶
Databricks Unity Catalog OCF Connector: Version 3.0.0¶
Enhancements¶
Databricks Unity Catalog OCF Connector 3.0.0 supports the following features:
Information Schema-Based Extraction¶
The connector now extracts metadata from information_schema instead of using driver based methods.
Custom MDE¶
The following Metadata types are supported as part of Custom MDE:
Catalog
Schema
Table
View
Column
Primary Key
Foreign Key
Table Source Comments Extraction¶
The connector now supports both table and column source comments.
For more information, see Databricks Unity Catalog OCF Connector
July 03, 2024¶
Databricks Unity Catalog OCF Connector: Version 2.3.1¶
Enhancements¶
Applies only to Alation Cloud Service on the cloud native architecture
When installed on an Alation Agent, this connector now supports authenticating with credentials the Agent has fetched from AWS Secrets Manager. For more information, see Integration with AWS Secrets Manager for Data Source Authentication Using OCF Connector. (AL-160374)
Fixed Issues¶
MDE Skips Nested STRUCT Type Column Extraction¶
MDE skips extracting columns with nested STRUCT type.
This issue is fixed. (AL-140807).
April 24, 2024¶
Databricks Unity Catalog OCF Connector: Version 2.2.2¶
Fixed Issues¶
Databricks Lineage Extraction Issues¶
Databricks lineage extraction fails when the column names are empty.
Databricks lineage extraction results in cyclic lineage.
Databricks lineage extraction job completes successfully even with empty Dataflow content.
Now, the issues are fixed. The Lineage extraction query is now updated to filter cyclic lineage, null entity types, empty and null table and column names. (AL-141050, AL-140061, AL-135348)
March 27, 2024¶
Databricks Unity Catalog OCF Connector: Version 2.2.1¶
Fixed Issues¶
Bulk Sampling Fails for Column Data Types timestamp_NTZ, Void, and Interval¶
For example, for Databricks Unity Catalog data source, sampling fails with the following error for column data type timestamp_NTZ:
The connector was provided with a list of columns to profile from a table, however the schema of the returned result set did not match those columns.
Now, connector allows bulk sampling to proceed by skipping tables with timestamp_ntz, void, and interval issues. (AL-150864)
January 17, 2024¶
Databricks Unity Catalog OCF Connector: Version 2.2.0¶
Compatible Alation Version - 2023.1.7 or higher
Enhancements¶
With Alation version 2023.3.4 and Databricks Unity Catalog OCF connector version 2.2.0, Alation has introduced an enhanced user interface for configuring the Databricks Unity Catalog data source settings.
Note
The connector version is compatible with Alation versions prior to 2023.3.4. However, you will view the old user interface with Alation versions prior to 2023.3.4.
The documentation for the new Databricks Unity Catalog experience is available at: Databricks Unity Catalog OCF Connector
The enhanced user interface provides the following features:
Better User Experience¶
The enhanced Databricks Unity Catalog user interface provides better navigation and multiple visual cues, making it easy to configure and manage Databricks Unity Catalog data source.
Easy to Connect¶
The Databricks Unity Catalog user interface now provides a step-by-step process to connect to your Databricks Unity Catalog data source easily and quickly.
Additional ErrorPrevention Checks¶
The Databricks Unity Catalog user interface now includes added checks to avoid obvious configuration mistakes. The user interface also includes text to assist and provide cues while configuring.
Detailed Error Reports and Troubleshooting Support¶
The improved Databricks Unity Catalog user interface makes troubleshooting easier for metadata extraction and query log ingestion, as you can now have a detailed error report for both processes. The report contains a detailed error description and steps to resolve the issue.
December 20, 2023¶
Databricks Unity Catalog OCF Connector: Version 2.1.1¶
Fixed Issues¶
QLI Execution Fails for Databricks Unity Catalog OCF Connector¶
The issue related to QLI execution for Databricks Unity Catalog data source with ANSI Mode enabled that resulted in the CAST_INVALID_INPUT error is now fixed.
November 29, 2023¶
Databricks Unity Catalog OCF Connector: Version 2.1.0¶
Compatible Alation Version - 2023.1.7.1 or higher
Enhancements¶
Support for OAuth Authentication with Azure Active Directory¶
The Databricks Unity Catalog OCF connector now supports OAuth authentication for Compose, enabling you to securely access and utilize data assets within the Databricks environment.
Previous Versions¶
Connector version |
Compatible with Alation from version |
New features in this version |
|---|---|---|
2.0.3.6564 |
2023.1.7.1 |
Disabled automatic lineage generation from queries ingested during query log ingestion (QLI) for this connector. The Data Source admins no longer need to manually disable automatic lineage on the data source settings page. Lineage is calculated based on direct lineage extraction from system tables. |
2.0.2.6259 |
2023.1.7.1 |
Implemented query log ingestion (beta). |
1.2.1.5335 |
2023.1.4 |
Implemented Query Service. |
1.1.0.4393 |
2023.1.4 |
Added support for incremental extraction of direct lineage (beta). |
1.0.3.4144 |
2023.1.2 |
Implemented direct lineage extraction (beta). |
Note
Direct lineage extraction and query log ingestion are considered beta as they rely on the Unity Catalog system tables that are currently in Public Preview in Databricks and require separate access enablement. You may need assistance from your Databricks account admin to enable access to these features in Databricks.