AWS Glue OCF Connector: Install and Configure¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
Authentication¶
The OCF connector for AWS Glue supports several authentication methods:
Basic Authentication¶
Basic authentication requires an AWS IAM user and the access key ID and secret access key for this user.
To use basic authentication, create an AWS IAM user account for Alation and save the values of the access key ID, secret access key, AWS region, and user ARN.
Grant the IAM user the required permissions (see Permissions for IAM User Account below).
Permissions for IAM User Account¶
Grant the user account you created for Alation the required permissions by creating and attaching these policies:
Policy for the Glue service
Policy
Purpose
"glue:GetConnections"Permission to retrieve list of Glue connection objects
"glue:GetConnection"Permission to retrieve a specific Glue connection object for extracting metadata
"glue:GetDatabases"Permission to retrieve list of multiple databases
"glue:GetDatabase"Permission to retrieve information about a specific database
"glue:GetTables"Permission to retrieve list of all tables in a specific database in the Glue Data Catalog
"glue:GetTable"Permission to retrieve metadata for a specific table in the Glue Data Catalog
In some cases, the permission must be granted to the AWS Glue database in Lake Formation. See What is AWS Lake Formation? for more information.
STS Authentication with an AWS IAM User¶
STS Authentication with an IAM user requires an IAM user account and an AWS role.
To use this type of authentication:
Create an IAM user and assign it the required permissions as it’s described in Basic Authentication.
Perform the configuration described in Create an IAM Role for STS Authentication below.
Create an IAM Role for STS Authentication¶
To set up STS authentication with an IAM user:
In the AWS IAM service, create IAM policies for the S3 and Glue services as described in Basic Authentication.
Create an IAM role selecting the Type of Trusted Entity to be AWS Service and Use Case to be EC2. To this role, attach the policies that grant the required permissions. This role will be assumed by the service account when performing MDE from your AWS Glue data source.
Edit the Trust Relationship of this role and add the ARN of the IAM user account that you created for Alation as
Principal.
Save the ARN of this role. It is required for the next configuration steps in Alation.
See Using Region-Specific Endpoint next.
Using Region-Specific Endpoint¶
From connector version 1.0.4, you can use the region-specific STS endpoint or the global endpoint.
Using the global STS endpoint does not require any specific configuration.
To use the region-specific endpoint, make sure it is active under your AWS account. To check this:
Under AWS IAM, go to Access Management > Account settings.
Under the Security Token Service (STS) section, in the Endpoints table, make sure that the STS endpoint for your AWS region is active or activate it.
STS Authentication with an AWS IAM Role¶
STS authentication with an AWS IAM role does not require an IAM user. This authentication method uses an instance profile that assumes a role allowing access to Amazon resources. This authentication method works for authenticating across AWS accounts.
Note
This authentication method is available with connector version 1.1.1.6409 or newer.
To configure STS authentication with an AWS IAM role, use the steps in Configure Authentication via AWS STS and an IAM Role. To provide access to the data source via an IAM role, use the permissions information in Permissions for IAM User Account.
Configuration in Alation¶
STEP 1: Install the Connector¶
Alation On-Prem¶
Important
Installation of OCF connectors requires Alation Connector Manager to be installed as a prerequisite.
To install an OCF connector:
If this has not been done on your instance, install the Alation Connector Manager: Install Alation Connector Manager.
Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.
Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.
Alation Cloud Service¶
Note
On Alation Service Cloud instances, Alation Connector Manager is available by default.
Ensure that the OCF connector Zip file that you received from Alation is available on your local machine.
Install the connector on the Connectors Dashboard page using the steps in Manage Connectors.
STEP 2: Create and Configure a New Data Source¶
In Alation, add a new data source:
Log in to Alation as a Server Admin.
Expand the Apps menu on the right of the main toolbar and select Sources.
On the Sources page, click +Add on the top right of the page and in the list that opens, click Data Source. This will open the Add a Data Source wizard.
On the first screen of the wizard, specify a name for your data source, assign additional Data Source Admins, if necessary, and click the Continue Setup button on the bottom. The Add a Data Source screen will open.
On the Add a Data Source screen, the only field you should populate is Database Type. From the Database Type dropdown, select the connector name. After that you will be navigated to the Settings page of your new data source.
Note
Agent-based connectors will have the Agent name appended to the connector name.
The name of this connector is AWS Glue OCF Connector.
Access¶
On the Access tab, set the data source visibility using these options:
Public Data Source—The data source will be visible to all users of the catalog.
Private Data Source—The data source will be visible to the users allowed access to the data source by Data Source Admins.
You can add new Data Source Admin users in the Data Source Admins section.
General Settings¶
Note
This section describes configuring settings for credentials and connection information stored in the Alation database. If your organization has configured Azure KeyVault or AWS Secrets Manager to hold such information, the user interface for the General Settings page will change to include the following icons to the right of most options:
By default, the database icon is selected, as shown. In the vault case, instead of the actual credential information, you enter the ID of the secret. See Configure Secrets for OCF Connector Settings for details.
Application Settings¶
Skip this section as it’s not applicable to AWS Glue data sources.
Connector Settings¶
Under the Connector Settings section of the General Settings tab, populate the data source connection information and save the values by clicking Save.
Parameter |
Description |
|---|---|
AWS Region |
Specify your AWS region of the account under which your AWS Glue resource is located. |
Basic Authentication |
Default. Leave this radio button selected if you are going to configure basic authentication. |
STS Authentication |
Select this radio button to configure STS authentication with an IAM user. |
Important
If you are going to Configure STS Authentication with an AWS IAM Role, disregard the Basic Authentication and STS Authentication radio buttons. They will not apply. Only provide a valid JDBC URI.
Configure Basic Authentication¶
If you selected the Basic Authentication radio button, specify the information in the Basic Authentication section of General Settings. Save the values by clicking Save.
Refer to Basic Authentication for more information about this authentication method.
Parameter |
Description |
|---|---|
AWS Access Key ID |
Specify the access key ID of the service account. |
AWS Access Key Secret |
Specify the access key secret of the service account. |
Configure STS Authentication¶
If you selected the STS Authentication radio button, specify the information in the STS Authentication section of General Settings. Save the values by clicking Save.
Refer to STS Authentication with an IAM User for more information about this authentication method.
Parameter |
Description |
|---|---|
Region-Specific Endpoint |
Select this checkbox if you prefer to use the STS endpoint specific to your AWS region. The regional endpoint has to be active under your AWS account. Leave this checkbox clear to use the global endpoint. |
STS: AWS Access Key ID |
Specify the access key ID of the service account. |
STS: AWS Access Key Secret |
Specify the access key secret of the service account. |
Role ARN |
Specify the ARN of the role you created following the steps in Create an IAM Role for STS Authentication. |
STS Duration |
Specify the STS duration value, in seconds. The default value is 3600 seconds. |
Configure STS Authentication with an AWS IAM Role¶
To use STS authentication with an AWS IAM role, specify the information in the IAM Role Authentication section of General Settings. Save the values by clicking Save.
Parameter |
Description |
|---|---|
Auth Type |
Select AWS IAM. |
Authentication Profile |
Select the authentication profile you created in Admin Settings. |
Role ARN |
Provide the ARN of the role that gives access to the Amazon resource. |
External ID |
Provide the External ID you added to the role that gives access to the Amazon resource. |
STS Duration |
Provide the STS token duration in seconds. This value must be less than or equal to the Maximum session duration of the IAM role that provides access to the Amazon resource(s). |
Logging Configuration¶
Select the logging level for the connector logs and save the values by clicking Save in this section. The available log levels are based on the Log4j framework.
Parameter |
Description |
|---|---|
Log level |
Select the log level to generate logs. The available options are INFO, DEBUG, WARN, TRACE, ERROR, FATAL, ALL. |
Obfuscate Literals¶
Skip this section as it’s not applicable to AWS Glue data sources.
Test Connection¶
Under Test Connection, click Test to validate network connectivity.
Metadata Extraction¶
You can configure metadata extraction (MDE) for an OCF data source on the Metadata Extraction tab of the Settings page. For AWS Glue data sources, Alation supports full and selective default MDE. Custom query-based MDE is not supported.
Refer to Configure Metadata Extraction for OCF Data Sources for information about the available configuration options.
Sampling and Profiling¶
Not supported.
Query Log Ingestion¶
Not supported.
Compose¶
Not supported.