AWS Glue OCF Connector: Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
The OCF connector for AWS Glue was developed by Alation and is available as a Zip file that can be uploaded and installed in the Alation application.
To download the AWS Glue OCF connector package, go to the Alation Connector Hub available from the Customer Portal. Go to Customer Portal > Connectors > Alation Connector Hub. Only Alation users with access to the Customer Portal can access the Alation Connector Hub. If you don’t have access to the Customer Portal, contact Alation Support.
This connector should be used to catalog AWS Glue as a data source on Alation on-prem and Cloud Service instances. It uses the AWS Glue API to connect to your AWS Glue instance and extract metadata. The connector is compiled with all necessary libraries to make API calls to AWS. No separate code or drivers are required. After the metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages under the parent data source. Alation users can leverage the full catalog functionality to search for and find the metadata, curate the corresponding catalog pages, enrich the catalog with documentation about the data source, and exchange information with other catalog users.
Team¶
You may need the assistance of your AWS admin to configure this connector.
AWS administrator:
Creates a service account for Alation
Creates the required policy
Provides the connection information.
Alation administrator:
Installs the connector
Creates and configures an AWS Glue data source in the catalog.
Scope¶
The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.
Feature |
Scope |
Availability |
|---|---|---|
Authentication |
||
AWS IAM user |
Authentication with an IAM user access key and secret |
Yes |
AWS IAM role |
Authentication with an STS token for an AWS IAM role |
Yes* |
* Authentication with the AWS IAM role is supported from connector version 1.0.4. |
||
Metadata Extraction (MDE) |
||
Default MDE |
Extraction of metadata based on default extraction queries in the connector code |
Yes |
Custom query-based MDE |
Extraction of metadata based on extraction queries provided by a user |
No |
Popularity |
Indicator of the popularity (intensity of use) of a data object, such as a table or a column |
No |
Extracted metadata objects |
||
Data Source |
Data source object in Alation that is parent to extracted metadata |
Yes |
Schemas |
List of AWS Glue databases |
Yes |
Tables |
List of AWS Glue tables |
Yes |
Columns |
List of columns |
Yes |
Column data types |
Column data types |
Yes |
Views |
List of views |
Yes |
Source comments |
Source comments |
Yes |
Primary keys |
Primary key information for extracted tables |
No |
Foreign keys |
Foreign key information for extracted tables |
No |
Sampling and Profiling |
No |
|
Query Log Ingestion (QLI) |
No |
|
Lineage |
No |
|
Compose |
No |
|
Object Mapping¶
AWS Glue Object |
Alation Object |
|---|---|
AWS Glue database |
Schema |
AWS Glue table |
Table |
AWS Glue column |
Column |