Databricks Unity Catalog OCF Connector Release Notes

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Customer Managed Applies to customer-managed instances of Alation

September 18, 2024

Databricks Unity Catalog OCF Connector: Version 3.1.0

Enhancements

FIPS Compliance

The JDK is now upgraded to Temurin-11.0.24+8. This upgrade addresses the following Common Vulnerabilities and Exposures (CVEs): CVE-2024-45490, CVE-2024-45491, CVE-2024-45492, CVE-2024-6119 (AL-136317)

September 04, 2024

Databricks Unity Catalog OCF Connector: Version 3.0.1

Fixed Issues

Query Log Ingestion Fails for Empty QueryText

Query Log Ingestion Fails with the following error for empty or NULL QueryText:

Query text is missing in the query statement. Ensure that queryString column has data.

This issue is now resolved. (AL-169062)

Unable to Extract Struct Definitions for Columns of Struct Data Type

The connector fails to extract the struct definitions for Columns of Struct data type.

This issue is now resolved. (AL-166865)

August 14, 2024

Databricks Unity Catalog OCF Connector: Version 3.0.0

Enhancements

Databricks Unity Catalog OCF Connector 3.0.0 supports the following features:

Information Schema-Based Extraction

The connector now extracts metadata from information_schema instead of using driver based methods.

Custom MDE

The following Metadata types are supported as part of Custom MDE:

  • Catalog

  • Schema

  • Table

  • View

  • Column

  • Primary Key

  • Foreign Key

Table Source Comments Extraction

The connector now supports both table and column source comments.

For more information, see Databricks Unity Catalog OCF Connector

July 03, 2024

Databricks Unity Catalog OCF Connector: Version 2.3.1

Enhancements

Applies only to Alation Cloud Service on the cloud native architecture

When installed on an Alation Agent, this connector now supports authenticating with credentials the Agent has fetched from AWS Secrets Manager. For more information, see Integration with AWS Secrets Manager for Data Source Authentication Using OCF Connector. (AL-160374)

Fixed Issues

MDE Skips Nested STRUCT Type Column Extraction

MDE skips extracting columns with nested STRUCT type.

This issue is fixed. (AL-140807).

April 24, 2024

Databricks Unity Catalog OCF Connector: Version 2.2.2

Fixed Issues

Databricks Lineage Extraction Issues
  • Databricks lineage extraction fails when the column names are empty.

  • Databricks lineage extraction results in cyclic lineage.

  • Databricks lineage extraction job completes successfully even with empty Dataflow content.

Now, the issues are fixed. The Lineage extraction query is now updated to filter cyclic lineage, null entity types, empty and null table and column names. (AL-141050, AL-140061, AL-135348)

March 27, 2024

Databricks Unity Catalog OCF Connector: Version 2.2.1

Fixed Issues

Bulk Sampling Fails for Column Data Types timestamp_NTZ, Void, and Interval

For example, for Databricks Unity Catalog data source, sampling fails with the following error for column data type timestamp_NTZ:

The connector was provided with a list of columns to profile from a table, however the schema of the returned result set did not match those columns.

Now, connector allows bulk sampling to proceed by skipping tables with timestamp_ntz, void, and interval issues. (AL-150864)

January 17, 2024

Databricks Unity Catalog OCF Connector: Version 2.2.0

Compatible Alation Version - 2023.1.7 or higher

Enhancements

With Alation version 2023.3.4 and Databricks Unity Catalog OCF connector version 2.2.0, Alation has introduced an enhanced user interface for configuring the Databricks Unity Catalog data source settings.

Note

The connector version is compatible with Alation versions prior to 2023.3.4. However, you will view the old user interface with Alation versions prior to 2023.3.4.

The documentation for the new Databricks Unity Catalog experience is available at: Databricks Unity Catalog OCF Connector

The enhanced user interface provides the following features:

Better User Experience

The enhanced Databricks Unity Catalog user interface provides better navigation and multiple visual cues, making it easy to configure and manage Databricks Unity Catalog data source.

Easy to Connect

The Databricks Unity Catalog user interface now provides a step-by-step process to connect to your Databricks Unity Catalog data source easily and quickly.

Additional ErrorPrevention Checks

The Databricks Unity Catalog user interface now includes added checks to avoid obvious configuration mistakes. The user interface also includes text to assist and provide cues while configuring.

Detailed Error Reports and Troubleshooting Support

The improved Databricks Unity Catalog user interface makes troubleshooting easier for metadata extraction and query log ingestion, as you can now have a detailed error report for both processes. The report contains a detailed error description and steps to resolve the issue.

December 20, 2023

Databricks Unity Catalog OCF Connector: Version 2.1.1

Fixed Issues

QLI Execution Fails for Databricks Unity Catalog OCF Connector

The issue related to QLI execution for Databricks Unity Catalog data source with ANSI Mode enabled that resulted in the CAST_INVALID_INPUT error is now fixed.

November 29, 2023

Databricks Unity Catalog OCF Connector: Version 2.1.0

Compatible Alation Version - 2023.1.7.1 or higher

Enhancements

Support for OAuth Authentication with Azure Active Directory

The Databricks Unity Catalog OCF connector now supports OAuth authentication for Compose, enabling you to securely access and utilize data assets within the Databricks environment.

Previous Versions

Connector version

Compatible with Alation from version

New features in this version

2.0.3.6564

2023.1.7.1

Disabled automatic lineage generation from queries ingested during query log ingestion (QLI) for this connector. The Data Source admins no longer need to manually disable automatic lineage on the data source settings page. Lineage is calculated based on direct lineage extraction from system tables.

2.0.2.6259

2023.1.7.1

Implemented query log ingestion (beta).

1.2.1.5335

2023.1.4

Implemented Query Service.

1.1.0.4393

2023.1.4

Added support for incremental extraction of direct lineage (beta).

1.0.3.4144

2023.1.2

Implemented direct lineage extraction (beta).

Note

Direct lineage extraction and query log ingestion are considered beta as they rely on the Unity Catalog system tables that are currently in Public Preview in Databricks and require separate access enablement. You may need assistance from your Databricks account admin to enable access to these features in Databricks.