Prerequisites¶
Alation Cloud Service Applies to Alation Cloud Service instances of Alation
Customer Managed Applies to customer-managed instances of Alation
Enhanced Connector Enhanced connectors add extended capabilities and require a separate entitlement in addition to your Alation platform license.
Provide Credentials¶
The Universal DS connector does not require authentication when you load a CSV file directly, because the file is not secured.
The connector requires authentication details when you load a file over SSH: SSH Host, SSH Port, SSH Username, SSH Password, Metadata File Path, and Query Log File Path.
The connector requires authentication details when you load a file from an S3 bucket: Region, Bucket Name, AWS Access Key ID, AWS Secret Access Key, Metadata File Path, and Query Log File Path.
The connector requires authentication details when you load a file from an Azure Storage container: Azure Storage Account Name, Azure Service Principal Auth, Access Key/Shared Access Signature, Azure Storage Container, Azure Storage Endpoint Suffix, Metadata File Path, and Query Log File Path.
Install the Alation Agent¶
If you are an Alation Cloud Service customer, ensure that Alation Agent is installed and running. The Alation Agent is mandatory for this connector in ACS deployments.
Install the Alation Connector Manager¶
If you are an on-premise or self-managed customer, ensure that the Alation Connector Manager (ACM) is installed and running. ACM is mandatory for OCF connectors.
Generate the Metadata CSV File¶
Generate a metadata CSV file that describes your catalogs, schemas, tables, and columns, then make it available to the connector through direct upload in the Alation UI, SSH, AWS S3, or Azure Blob Storage.
Format the Metadata CSV File¶
The connector reads the metadata CSV as a column-based file. Each row describes one column of one table. The connector maps each field by its position in the row, not by the header name, so the field order must match the default layout below. You can adjust the position of any field on the Metadata Extraction tab if your file uses a different order.
The recommended default layout uses the following header row:
CATALOG,SCHEMA,TABLE_NAME,TABLE_TYPE,TYPE_NAME,DATA_TYPE,COLUMN_NAME,ORDINAL_POSITION,IS_NULLABLE,SCHEMA_REMARKS,TABLE_REMARKS,COLUMN_REMARKS,COLUMN_DEFAULT,PRIMARY_KEY,FOREIGN_KEY
The following table describes each field, its default position, and whether it is mandatory.
Field |
Position |
Mandatory |
Description |
|---|---|---|---|
|
1 |
No |
Catalog or database name. When provided, the schema is nested under this catalog. |
|
2 |
Yes |
Schema name. This column must be present in every row, or the row is skipped. |
|
3 |
Yes |
Table name. This column must be present in every row, or the row is skipped. |
|
4 |
No |
Table type, for example |
|
5 |
No |
Source column type name, for example |
|
6 |
No |
Normalized column data type, for example |
|
7 |
Yes |
Column name. This column must be present in every row, or the row is skipped. |
|
8 |
No |
Position of the column within the table, as an integer. |
|
9 |
No |
Whether the column is nullable. Use |
|
10 |
No |
Description applied to the schema. |
|
11 |
No |
Description applied to the table. |
|
12 |
No |
Description applied to the column. |
|
13 |
No |
Default value defined for the column. |
|
14 |
No |
Whether the column is a primary key. Use |
|
15 |
No |
Referenced column for a foreign key, in |
If a value is not available for an optional field, leave the field empty.
For optional numeric fields, enter -1 when the value is not available.
If a value contains a comma or double quote, wrap the value in double quotes. Escape a double quote as "".
The table below shows the value each field holds in a sample data row. In the CSV file, these values appear as a single comma-separated line in the same order as the header.
Field |
Example Value |
|---|---|
|
FIVETRAN_DATABASE |
|
SALESFORCE |
|
PARTY_CONSENT |
|
BASE TABLE |
|
TIMESTAMP_TZ |
|
TIMESTAMP_TZ |
|
_FIVETRAN_SYNCED |
|
20 |
|
YES |
|
(empty) |
|
(empty) |
|
(empty) |
|
NULL |
|
NO |
|
(empty) |