Migrate from Tableau Native Connector to OCF

Customer Managed Applies to customer-managed instances of Alation

Note

Alation Cloud Service customers can request the Tableau native source migration to Open Connector Framework (OCF) through Alation Support.

Before you start the migration, we recommend you thoroughly review this guide, as it covers essential prerequisites and considerations. We recommend that the migration is performed by a user with the Server Admin role.

Considerations

Migration Scope

Some BI objects are migrated from native BI sources to OCF-based BI sources seamlessly. However, certain information will not be migrated:

Information Included Into the Migration Scope

The following objects are migrated, including their curation information (or logical metadata).

  • BI Folders—Sites, projects, and workbooks

  • BI Reports—Sheets and dashboards

  • BI DataSources—Published datasources

The following configuration information is migrated:

  • Server Connection and Additional Settings on the settings page, such as the host, username, password, domain names, extraction batch sizes, and other information.

Information Excluded from the Migration Scope

The following objects are not migrated seamlessly and will need an additional action to be transferred.

  • The curation information for the following objects is not migrated:

    • BI Report Columns—Sheet and Dashboard fields

    • BI DataSource Columns—Datasource fields of published and unpublished datasources

    • BI DataSources—Embedded and unpublished datasources

  • The SSL certificate of the Tableau host if SSL was in use and a certificate was uploaded. The certificate will need to be re-uploaded.

    Note

    Ensure you have the certificate file ready for re-uploading after the migration.

Additional Action

After you perform the migration via the user interface, you will also be required to perform some additional steps on the backend of the Alation server to complete the process. The reason behind this requirement is explained below.

There is a known issue with the native Tableau connector where it generates a new BI object every time the name changes in Tableau instead of updating the existing object. It might also lead to hidden curation if those objects were curated. The native connector uses the name attribute as a unique identifier for the following BI objects:

  • BI Report Columns

  • BI DataSource Columns

  • BI DataSources—Unpublished datasources

The OCF connector addresses this issue, as it relies on the Tableau’s Metadata API, which returns unique identifiers for the object types listed above.

Post migration, you will be required to run a curation migration script on the Alation server to ensure that the curation data (logcal metadata) for these three BI object types is transferred to the newly created OCF objects. You will need to contact Alation Support to get access to the curation migration script.

Note

If you don’t have any curation information for the three affected object types (Report Columns, DataSource Columns, and DataSources of the unpublished (embedded) datasource type, you can skip the additional manual step.

Overview of Migration Steps

To migrate your native Tableau source to OCF, follow this workflow:

  1. Fulfill the Prerequisites.

  2. Perform the migration and run an extraction—Migrate your Tableau Native Source to OCF.

  3. Run the curation migration script—Run the Curation Migration Script. (Applicable only to Alation versions prior to 2024.1.2.)

  4. Review the curation script log—Reading the Curation Migration Report.

Prerequisites

You need the Server Admin privileges to perform the steps below.

Important

Ensure that your native Tableau BI server source does not have any logical curationdata that’s not migrate-able, currently. Everything that we can migrate is listed in the Supported curation types for Migration. If you found curation items that cannot be migrated, do not perform the migration, contact Support instead.

  1. Log in to Alation.

  2. Click the gear icon in the top right corner to open the Admin Settings page.

  3. In the Server Admin section, click Feature Configuration.

  4. Enable the Enable Native Connector Migration to OCF Connector toggle.

    ../../../_images/Enable_Native_Connector_Migration_to_OCF_Connector.png

  5. Click Save Changes.

  6. Refresh the page.

    ../../../_images/save_enable_migration_dialog.png

  7. Open the settings page of your native Tableau source.

  8. Ensure that the Remove Projects that are not captured by the list above checkbox is selected. This is required for all extraction jobs you will run during the migration process.

    • If the checkbox is not visible, toggle the Select Projects to Include or Exclude from Extraction button to reveal it.

    • This must be checked for MDE runs as part of migration. You can uncheck it after the migration process is completed if necessary.

      ../../../_images/remove_notseen.png

  9. Ensure that you have a recent successful MDE run and that the Remove Projects that are not captured by the list above checkbox was checked.

    • If you haven’t recently run MDE, then ensure that the Remove Projects checkbox is selected and run MDE to retrieve the latest metadata into Alation.

  10. Install the latest versions of the Tableau OCF connector.

  11. Back up your SSL certificate.

    • If you use SSL for connections between Tableau and Alation, you must download your certificate file from the Tableau Server before the migration process as it will not be migrated.

Curation Type

Description

Post (anchor tags)

Links in an Article where a BI Object has been referenced using report fields or datasource fields or unpublished fields.

PostRevision (anchor tags)

Links in older versions of an Article (revisions).

Flag (anchor tags)

Links in WARN or DEPRECATED flags’ text fields.

TextLog (anchor tags)

Links in RDBMS object descriptions and titles e.g. tables, schemas, columns, etc.

GenericFieldValue (anchor tags)

Links in generic fields like Steward, RTFs (rich text fields), and Description.

CustomFieldValue (anchor tags)

Links in CustomFieldValue text fields, RTFs, etc. This curation type is deprecated but must be updated for backwards compatibility

ValueHistory (anchor tags)

Links to Field Edit History for descriptions, stewards, RTFs, and other objects where you see a small clock icon.

Threads (conversations)

Conversations feature in Alation.

PostMention

These tables DO NOT contain customer data. These are the tables that power the Alation instance that collects customer data.

Mark (star/watch)

These are tied to a specific user and can only be seen by the user that created the Mark in Alation’s Catalog.

Flag (Endorse, Warn, Deprecate)

Data quality flags used in Alation.

Tag

Alation tags to categorize an object.

GenericFieldValue (oid)

These are field values like People, Sets (Experts, Stewards), Object Sets, RichTextFields, DatePickers, and DropDowns.

CustomFieldValue (oid)

This is a deprecated curation type that was replaced by GenericFieldValue. However, we do need to migrate it too for backwards compatibility.

ValueHistory (oid)

Field Edit History on objects like description and history.

Domain (ExcludeMembers)

This is an exclude membership rule and cannot be seen in the UI

Domains (MembershipRule)

This is a membership rule and cannot be seen in the UI

Domains (Member)

This Domain value can be seen on the top right hand side of the Alation catalog

Migrate your Tableau Native Source to OCF

Warning

Migrating from the Tableau native connector to OCF is irreversible. Ensure that all Prerequisites have been met.

To migrate the Tableau source:

  1. Open the catalog page of the Tableau source you want to migrate.

  2. Select the Migrate button in the top right corner.

    ../../../_images/tableau_native_to_migrate_button.png

  3. From the drop-down list, select the latest Tableau OCF connector version that you installed.

    ../../../_images/select_tableau_ocf_connector_to_migrate_to.png

  4. Enter the connector name to confirm the action.

    ../../../_images/verify_selected_migrate_to_connector.png

  5. Click Migrate.

  6. Verify that the settings page has changed and all previous settings were preserved (except SSL).

Settings Page Differences

Native

OCF
../../../_images/tableau_native_config.png ../../../_images/tableau_ocf_config.png

  1. If using SSL, re-upload the certificate file.

  2. Run the connection test to ensure it’s connecting successfully.

  3. Ensure that item 8 of the Prerequisites is done.

  4. Run the first MDE on the OCF connector.

    Native to OCF curation migration runs as a downstream job as part of the first MDE after migration. Verify the migration results in the Job_details section, see Migration Results section for more details.

Run the Curation Migration Script

Important

This section only applies to the Alation version before 2024.1.2. For versions 2024.1.2 and newer, the curation migration script is run only once, as part of the first OCF-based MDE as a downstream job.

Note

Contact Alation Support to get access to the curation migration script.

Script Arguments

  • -s—BI Sever rosemeta ID from the source URL.

    • Example: if the BI source URL is https://aaa.alationcloud.com/bi/v2/server/13/, the ID is 13.

  • first_ocf_mde_date—Date on which OCF MDE was triggered

    • The format must be 'dd-mm-yyyy' e.g. '16-11-2023'

Running the Script

To run the script:

  1. Use SSH to connect to the Alation server.

  2. Enter the Alation shell.

    sudo /etc/init.d/alation shell

  3. Navigate to the one_off_scripts directory and place the curation migration script provided by Alation support.

    cd /opt/alation/django/rosemeta/one_off_scripts/
# assuming that the script was uploaded into the /tmp folder on the Alation machine
cp /tmp/copy_tableau_logical_data_from_deleted_objects_to_new_objects.py .

  4. Change the user to alation.

    sudo su alation

  5. The curation migration script has two modes:

    • Dry-run—Executes the script without actually making any database writes, only reads.

    • Confirm—Makes permanent changes to the database and is not reversible. It should only be run once.

    Both commands are shown in the example below.

    Note

    The script execution is a database-intensive as well as time-consuming operation based on the volume of data on the BI Server.

    DRY RUN:

    nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <bi_server_id> -first_ocf_mde_date '<date>' &

    CONFIRM:

    nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s <bi_server_id> -first_ocf_mde_date '<date>'  --confirm &

    EXAMPLE:

    nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 2 -first_ocf_mde_date '16-11-2023'  --confirm &

  6. You can track the progress of the script using its PID (process id). The command above will run the script in the background and output a PID. We can use this PID to check the status.

    ps -p 14496 -o %cpu,%mem,cmd

%CPU %MEM CMD
98.9  0.1 python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 3 -first_ocf_mde_date 25-10-2023 --confirm

Migration Results

The migration of curation data results will be displayed as part of MDE job details.

../../../_images/TableauMigrationResults_01.png

In case of any Curation errors, it will be displayed in the Job errors table as Migration error. Click Generate Error Report button to download the error report. In such cases, the MDE status will be Partial Success due to errors.

../../../_images/TableauMigrationResults_02.png

Users can perform the migration again in the following methods in case of an MDE failure:

  1. Curation Migration API.

  2. Run the Curation Migration Script - For Alation version 2024.1.2 and newer, copying the migration script and passing the first_ocf_mde_date argument is not required.

    EXAMPLE:

    nohup python -m copy_tableau_logical_data_from_deleted_objects_to_new_objects -s 2 --confirm &

Reading the Curation Migration Report

Important

This section only applies to the Alation version before 2024.1.2.

Example

 1**** Searching for Native to OCF matches for Data Source Columns
 2
 3-----------------------------------------------------------------
 4
 5Data Source Columns - Found a total of: 53 instances of curation
 6
 7The following are the URLs (count: 1) to the Data Source Columns which we could not automatically match to an OCF object. They need to be reviewed (for validity) and to migrate their curation manually.
 8
 9http://localhost:8000//bi/v2/datasource_column/5/ | curation type: {'Mark (star/watch)', 'ValueHistory (oid)', 'GenericFieldValue (anchor tags)', 'Tag'}
10
112024-01-29 10:00:25.025069 —— ========================== Data Source Columns Curation Migration End ==========================

Explanation

  • Line 2: Indicates we’re searching for matches between native and OCF BI objects

  • Line 5: Indicates that we found 53 instances of curation on native-based datasource_columns objects

  • Line 7: Indicates the number of BI objects (and how many BI objects) for which we couldn’t find an OCF match and so cannot copy curation to it (OCF object)

  • Line 9: Indicates the URL for the object we could not migrate curation and the type of curation in brackets. Review the URLs and migrate the curation data on them manually.

Troubleshooting

  • For Connector settings migration errors, please refer to the migration log file: /opt/alation/site/logs/ypireti.log

  • Permissions error: Ensure that you are running the script as the user alation

  • Django imports errors or issues: Ensure that you run the script from the directory /opt/alation/django/rosemeta/one_off_scripts/

  • For any other issues, please reach out to Alation Support with the following log file attached to the Support ticket: /tmp/tableau_native_curation_to_ocf_migration-debug.log