AWS Glue OCF Connector: Overview¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
Core Connector Core connectors are included with all Alation platform tiers (subject to each tier’s connector limits) and are fully supported by Alation.
You can install and upgrade the connector using one of the following methods, depending on your Alation deployment type:
On-premise instances: Download the latest connector package from the Alation Customer Portal (Customer Portal > Connectors), then upload and install it in Alation.
Alation Cloud Service: Install and upgrade the connector directly from your instance without downloading the package. Any connector available on the Alation Customer Portal (Customer Portal > Connectors) can be installed and upgraded this way. See Install Connectors (Cloud-Native Architecture). You can also download the connector package from the Alation Customer Portal and install or upgrade it manually by uploading the Zip file.
This connector should be used to catalog AWS Glue as a data source on Alation on-prem and Cloud Service instances. It uses the AWS Glue API to connect to your AWS Glue instance and extract metadata. The connector is compiled with all necessary libraries to make API calls to AWS. No separate code or drivers are required. After the metadata is extracted, it is represented in the data catalog as a hierarchy of catalog pages under the parent data source. Alation users can leverage the full catalog functionality to search for and find the metadata, curate the corresponding catalog pages, enrich the catalog with documentation about the data source, and exchange information with other catalog users.
Team¶
You may need the assistance of your AWS admin to configure this connector.
AWS administrator:
Creates a service account for Alation
Creates the required policy
Provides the connection information.
Alation administrator:
Installs the connector
Creates and configures an AWS Glue data source in the catalog.
Scope¶
The table below describes which metadata objects are extracted by the connector and which catalog functionality is supported.
Feature |
Scope |
Availability |
|---|---|---|
Authentication |
||
AWS IAM user |
Authentication with an IAM user access key and secret |
Yes |
AWS IAM role |
Authentication with an STS token for an AWS IAM role |
Yes* |
* Authentication with the AWS IAM role is supported from connector version 1.0.4. |
||
Metadata Extraction (MDE) |
||
Default MDE |
Extraction of metadata based on default extraction queries in the connector code |
Yes |
Custom query-based MDE |
Extraction of metadata based on extraction queries provided by a user |
No |
Extracted metadata objects |
||
Data Source |
Data source object in Alation that is parent to extracted metadata |
Yes |
Schemas |
List of AWS Glue databases |
Yes |
Tables |
List of AWS Glue tables |
Yes |
Columns |
List of columns |
Yes |
Column data types |
Column data types |
Yes |
Views |
List of views |
Yes |
Source comments |
Source comments |
Yes |
Primary keys |
Primary key information for extracted tables |
No |
Foreign keys |
Foreign key information for extracted tables |
No |
Sampling and Profiling |
No |
|
Query Log Ingestion (QLI) |
No |
|
Popularity |
No |
|
Lineage |
No |
|
Compose |
No |
|
Object Mapping¶
AWS Glue Object |
Alation Object |
|---|---|
AWS Glue database |
Schema |
AWS Glue table |
Table |
AWS Glue column |
Column |