Kafka OCF Connector: Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
The OCF connector for Apache Kafka is available as a Zip file from Alation’s Connector Hub (requires a login via the Alation Customer Portal). The connector file can be uploaded and installed in the Alation application. The connector is compiled together with the required database driver, so no additional effort is needed to procure and install the driver.
This connector should be used to catalog Apache Kafka or Confluent Kafka as a data source on Alation on-premise and Cloud Service instances. It extracts and catalogs such database objects as schemas, tables, and columns. After the metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages under the parent data source. Alation users can leverage the full catalog functionality to search for and find the extracted metadata, curate the corresponding catalog pages, create documentation about the data source, and exchange information about it.
Team¶
The following administrators are required to install this connector:
Kafka administrator:
Creates a service account with the required privileges to extract metadata.
Provides the JDBC URI.
Provides the authentication information and assists in configuring the authentication.
Alation administrator:
Ensures that Alation Connector Manager is installed and running or installs it.
Installs the connector.
Creates and configures a Kafka data source in the catalog.
Performs initial extraction and prepares the data source for Alation users.
Scope¶
The table below lists the features supported by the connector.
Feature |
Scope |
Availability |
---|---|---|
Authentication |
||
Basic |
Authentication with a service account created on the database using username, password, and Bootstrap server |
Yes |
SSL |
Connection over the TLS protocol |
No |
LDAP |
Authentication with the LDAP protocol |
No |
OAuth |
Authentication with the OAuth 2.0 protocol. |
No |
SSO |
Authentication using an SSO flow through an IdP application |
No |
Metadata Extraction (MDE) |
||
Default MDE |
Extraction of metadata based on the JDBC driver methods in the connector code |
Yes |
Custom query-based MDE |
Extraction of metadata based on extraction queries provided by a user |
No |
Popularity |
Indicator of the popularity (intensity of use) of a data object, such as a table or a column |
No |
Extracted metadata objects |
||
Data Source |
Data source object in Alation that is parent to the extracted metadata |
Yes |
Schemas |
List of schemas |
Yes |
Tables |
List of tables |
Yes |
Columns |
List of columns |
Yes |
Column data types |
Column data types |
Yes |
Views |
List of views |
N/A |
Source comments |
Source comments |
N/A |
Primary keys |
Primary key information for extracted tables |
N/A |
Foreign keys |
Foreign key information for extracted tables |
N/A |
Functions |
Extraction of function metadata |
N/A |
Function definitions |
Extraction of function definition metadata |
N/A |
Sampling and Profiling |
||
Table sampling |
Retrieval of data samples from extracted tables |
Yes |
Column sampling |
Retrieval of data samples from extracted columns |
Yes |
Deep column profiling |
On-demand profiling of specific columns with the calculation of value distribution stats |
Yes |
Dynamic profiling |
On-demand table and column profiling by individual users who use their own database accounts to retrieve the profiles |
Yes |
Custom query-based table sampling |
Ability to use custom queries for sampling specific tables |
No |
Custom query-based column profiling |
Ability to use custom queries for profiling specific columns |
Yes |
Query Log Ingestion (QLI) |
||
Not supported |
||
Lineage |
||
Not supported |
||
Compose |
||
Not supported |
Object Mapping¶
The following table shows how extracted Kafka objects map onto Alation objects:
Kafka Concept |
Mapping in Alation |
---|---|
ApacheKafka (Single model data source) |
Schema |
Topic |
Table |
Message |
Records in Table |