HDFS OCF Connector: Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
Core Connector Core connectors are included with all Alation platform tiers (subject to each tier’s connector limits) and are fully supported by Alation.
Available from release 2022.3.2
You can install and upgrade the connector using one of the following methods, depending on your Alation deployment type:
On-premise instances: Download the latest connector package from the Alation Customer Portal (Customer Portal > Connectors), then upload and install it in Alation.
Alation Cloud Service: Install and upgrade the connector directly from your instance without downloading the package. Any connector available on the Alation Customer Portal (Customer Portal > Connectors) can be installed and upgraded this way. See Install Connectors (Cloud-Native Architecture). You can also download the connector package from the Alation Customer Portal and install or upgrade it manually by uploading the Zip file.
Use the OCF connector for HDFS to catalog HDFS as a file system source in Alation. It extracts HDFS objects such as root folders along with their content, which includes files and folders. It enables catalog users to discover, search, browse, and curate HDFS objects, such as files and folders, from the Alation user interface. The HDFS OCF connector can be used to catalog metadata from the HDFS file system deployed on the Cloudera Distributed Hadoop (CDH) or Cloudera Data Platform (CDP) platforms.
Team¶
The following administrators are required to install this connector:
Alation Server Admin:
Validates the availability of Alation Connector Manager or installs it
Installs the connector
Adds and configures the HDFS file system source in Alation
Kerberos Server Administrator:
Provides the Kerberos configuration file (krb5.conf)
Scope¶
The table below lists the features supported by the connector.
Feature |
Scope |
Availability |
|---|---|---|
Authentication |
||
Basic (username) |
Authentication with a service account. |
Yes |
SSL |
Connection over the TLS protocol. |
Yes |
Kerberos |
Authentication with the Kerberos protocol. |
Yes |
LDAP |
Authentication with the LDAP protocol. |
No |
OAuth |
Authentication with the OAuth 2.0 protocol. |
No |
SSO |
Authentication using an SSO flow through an IdP application. |
No |
Metadata extraction (MDE) |
Metadata Extraction uses the WebHDFS REST API. The HDFS OCF connector calls the List a Directory API to get the list of files and folders. |
Yes |
Extracted metadata objects |
||
Files |
Files on the HDFS server. |
Yes |
Folders |
Folders on the HDFS server. |
Yes |
Attributes or columns |
Table attributes or columns within a file or folder on the HDFS server. |
Yes |
Object permissions |
Information related to object permissions. |
Yes |
Object owner |
Information related to the object owner. |
Yes |
Object group |
Object owner group name. |
Yes |
Lineage |
||
Automatic lineage generation |
Auto-calculation of lineage based on query history ingested from MDE queries. |
Yes |
Direct lineage |
Extraction of lineage from system tables during MDE. |
Yes |
Column-level lineage |
Extraction of lineage on the column level |
No |
Object Mapping¶
File System Object |
Alation Object |
|---|---|
File system |
File system source. |
Folder |
Directory or File.
|
File |
File |