Release Notes V R4 (General Availability)¶
RELEASE 5.8.2 - GA¶
Note that these Release Notes combine all features and bug fixes from VR4 5.8.0 (Limited Availability) and up to 5.8.2 (General Availability).
New Features¶
Improved MicroStrategy Integration¶
Alation has improved integration with MicroStrategy and now provides the following:
The MicroStrategy directory structure is now visible when users view the dashboard or reports under a project. This helps to identify the source of the dashboard or report.
The Report or Dashboard fields will use terminology pertinent to the BI source. Report fields in MicroStrategy will be referred to as Metrics and Attributes. The terminology of Measures and Dimensions will be used for report fields for other BI tools.
The report metadata is successfully extracted for MicroStrategy reports created using object prompt or hierarchy prompt.
Metrics and Attributes extracted from MicroStrategy Servers now have their own catalog pages where the Description and Steward fields can be curated.
Inbox (Beta)¶
In this Beta version, Alation users will be able to see conversation updates and task notifications in their new Inbox. In addition to the separate designated Inbox page accessible from the main toolbar, there is My Inbox section on the Stewardship Dashboard. Granular details of each conversation and the pertinent details are shown on the detailed Inbox page. For more information, see Inbox
Catalog Activity Report¶
Data Stewards now have the ability to monitor curation changes to data objects that they manage. Using the Catalog Activity report, they can review updates to data objects, catch any inaccurate changes and restore an older value if necessary. Catalog Activity report provides detailed information about each change including the time of the change, the user who made the change, and the actual modification that took place. Data Stewards can utilize easy-to-use filters to zoom in and out of the data. The Catalog Activity report is available as a personalized view for the logged-in user on My Stewardship Dashboard and as a full view under the Curate menu. The full-view report displays both curation changes and physical metadata changes using two separate tabs. For more information, see Working with Catalog Activity Report
Agile Approval¶
This release introduces Agile Approval: a new collaboration workflow that allows catalog users to assign reviewers and request a review and approval for an article from subject-matter experts in the organization. Readers will be informed on the current review status of the article (Approved or Under Review) on the article catalog page. Agile Approval can be turned on and off in Labs. For more information, see Enrolling Articles Into Agile Approval.
Customizable Home Page (Beta)¶
There is now a new fresh look for Home Page (Beta) featuring Revisit and Object Windows. Revisit provides quick links to pages you recently visited, starred or marked as Watched. Object Windows allows you to preview articles, business intelligence reports, and dashboards selected and promoted by your Catalog Admins as helpful and important. Curated Content section has been improved and can now include links to any Alation object type. Contact your Alation Customer Service Manager or Sales Engineer to configure the Home Page. For more information, see Customizable Home Page
Query Version History (Beta)¶
Queries in Alation now have Version History. Every time you publish or republish a query, Alation captures its new version. While viewing a query in Compose, you can access Query Version History by clicking the More dropdown and selecting Version History. The “publish” and “republish” actions are available in Version History for queries you authored; in addition, there is ability to restore an older version. Versions in History are for now view-only. For more information, see Query Version History
Health Check in Monitor¶
Alation now provides a UI for monitoring the health of its components in order to determine whether they are running and working correctly. Server Admins can view the status of these functional checks under Health Checks in the Monitor section of the Admin Settings page. For more information, see Monitoring Component Health.
Document Store Cataloging¶
Alation supports metadata extraction for MongoDB (Versions 3.2.10 and 3.6) as a data source. This can be configured by following the steps on the Add New Data Source page in Admin Settings. Alation supports basic and Kerberos authentication as well as SSL.
SAP HANA Calculation View¶
For SAP HANA Calculation Views, users will now be able to view lineage and the View SQL definition.
BigQuery User Authentication Support¶
Alation now supports user account authentication for Google BigQuery data source. Compose and Dynamic Profiling will leverage this authentication method whereas Metadata Extraction, Query Log Ingestion and Standard Profiling will continue to use service account authentication. This authentication method involves host and database name. The host refers to the client email and database name refers to the project ID.
Improved Custom DB Driver Support¶
Alation has added the following new features for Custom DB:
Support has been added for extracting function and procedure metadata for data sources using DB Type Custom DB.
User can now use Kerberos authentication to connect to a datasource using DB Type Custom DB.
For a Custom DB datasource, users can select the Disable bulk extraction checkbox on the Metadata Extraction page to disable bulk extraction and iterate through tables individually. This should be used only when bulk extraction fails.
Alation now supports Query Log Ingestion (QLI) for table-based custom data sources. QLI will work only if query log history can be retrieved using a SQL query.
Improvements¶
Catalog Pages¶
Previously, on Home Page, users were able to see all the tiles regardless of the access permissions for referenced objects. Now, Alation checks if the current user can see the tile object and filters out the ones for which they do not have the view permission.
The Undo and Redo buttons have been added to the text editor to facilitate editing of articles.
The rich text custom fields are now sorted alphabetically on the BI Server, BI Project, and BI Report catalog pages.
Rich text editors(Articles and Description) now support overflow and visibility CSS properties.
We have enabled HTML tag <map> and its attributes coords, shape, and usemap to allow users to add HTML <map> diagrams to rich text content in Alation.
Data Objects¶
Previously, the data source privacy message was available on the General tab of Data Source Settings. It has been removed from the General tab. The Access tab of the Data Source Settings now contains information on the data source privacy.
The dependency of AWS Glue metastore on Hive as a data source has been removed. Now, AWS Glue metastore can be added as a data source to Alation without providing the host/URI details.
Alation now supports metadata extraction from Data Definition Language (DDL) statements in Google BigQuery.
The Open With DataOS integration has been improved to allow third party applications to request configuration from each user.
Manual Catalog Sets and children metadata synchronization feature is now ON by default, for all users. Object Equivalences have been migrated to be included into Manual Sets.
Previously, the Field tab of the Tableau workbook listed only up to 100 items, though there were more fields. This limit has been withdrawn as support for pagination has now been added for measure and dimension tables of the Fields tab for BI Report, MicroStrategy Report, Tableau Workbook, and Tableau Sheet catalog pages.
Only the metadata of schema, tables, attributes, indexes, primary and foreign keys were extracted if a data source is added as Custom DB. Now, if the Custom DB contains procedure/function and the driver supports it, the corresponding metadata will be extracted.
Alation now supports Shared Access Signature as a new authentication type for Azure Blob Store data source.
Alation now supports extraction for SQL Server Reporting Service (SSRS) in Sharepoint integrated mode. The version of SSRS support includes 2008, 2012, 2014, and 2017.
Alation now has functionality to avoid the extraction of extended information during Query Log Ingestion (QLI) for Teradata. This significantly saves disk space and allows seamless functioning of automated QLI. This feature will not affect the default columns that are already shown on the QLI page.
Hive QLI now supports richer and more directive error messages intended to make error identification easier for users.
Considerably improved stability, speed, and extensibility for QLI on Hive data sources.
HDFS/ADLS connectivity issues with regard to incorrect host name, port and SSL for metadata extraction were previously not logged with sufficient detail. Now, additional log has been added to capture information on incorrect host name, port and SSL.
Platform¶
Deprecation flag propagation feature is now hooked to Flag public APIs. Adding DEPRECATION flags via POST API will propagate flags to the descendants. Removing DEPRECATION flags via DELETE API will delete propagated flags from the descendants.
Previously, if the user deprecated a data object (Example: Schema S got a deprecation flag on Alation, and its existing children got a propagated flag) and a child object is created (Example: Table T under Schema S) and metadata extraction is run on the data source, then the table T would not get a propagated flag. This issue is now resolved because the metadata extraction job finds and adds missing flags for new objects.
Previously, flag propagation could be kicked off only from RDBMS objects (Data Source, Schema, Table). Now it is extended to BI and File Systems, so that deprecating a BI object or a File System object will trigger propagation of flags to its descendants and any objects connected via lineage.
Previously, Find Objects by Custom Field API returned all the objects including the deleted ones. With this fix, it returns only the non-deleted objects unless the user passes “include_deleted=True”.
If a user is suspended, a task is triggered that provides Edit access to the private articles of the suspended user to all admin users.
Alation now supports loading custom drivers from a directory. So, if a single driver needs to be loaded from multiple jar files, they can be placed in a single directory in the custom driver path.
The alation_conf will now write the PostgresSQL configuration parameter value in On/Off instead of the true/false boolean value.
If LDAP is enabled for any user, the DB account and the username should be the same. However, in the Top Users section on a catalog page, the Alation users and DB accounts were shown separately. Now, if LDAP is enabled, in Top Users, the DB accounts are mapped to the Alation users.
Previously, the HTTP REDIRECT binding method was only supported for Security Assertion Markup Language (SAML) authentication. Now, the HTTP REDIRECTandHTTP POSTbinding methods are supported for SAML authentication
UX Research¶
Invitation is sent to users to participate in Alation user research testing as part of the NPS survey.
Fixes Referenced by Support Tickets
Support Ticket # |
Description |
---|---|
3944 5672 5860 |
Alation now provides enhanced metadata extraction for Presto database to properly extract the columns and SQL definition of Hive views created through Presto. |
5824 7749 |
The Tableau workbooks can now be sorted by Popularity on the catalog page. Sorting by Popularity is set as a default option. |
6211 |
Previously, on the Manage Settings page, the job status of some BI Servers was not displayed. Now, the job status should be displayed for all BI Server types. |
6453 |
Previously, with 1,000 or more data sources in Alation, the Data card in the left-hand navigation panel took at least a minute to load. This issue is fixed, and the Data card now loads quicker. Initially, it will display 25 data sources, and as the user scrolls down, the remaining data sources will be uploaded and displayed. |
6722 |
Previously, if the user ran a query which fetched a large number of rows/columns, it would cause the execution results page load time to slow down substantially. Now, we truncate the columns of the execution results to the maximum of 50, which limits the amount of table data that is displayed at any given time and improves performance. |
6722 7262 |
Previously, users who have a wide table in their database (over 200 columns and 100 rows) experienced slowness or page crashing when they visited the catalog page of such a table. To fix this problem, Alation now truncates the amount of columns displayed on the Samples tab to the maximum of 50. Users can export the full sample to view all the columns. |
6972 7427 |
Fixed the issue with Nginx log rotation. Log rotation now works properly. |
7040 9164 9449 |
Fixed an issue where some users could not log in after update when using SAML SSO for authentication. Alation recently added support for using HTTP POST binding with SAML SSO. All customers were previously using Redirect binding because Alation did not support POST binding. Many configurations allow for automatically selecting POST binding if available. This caused existing instances to switch over and in some cases caused authentication failures blocking users from logging in. Now, Alation has changed SAML SSO support to always force defaulting to Redirect binding. So existing installations integrated with SAML SSO should work in the same way they did before. Those customers wanting to use POST binding can change the configuration alation.authentication. saml.force_redirect_binding to False. |
7196 |
Previously, (version 5.4.1 to 5.4.6), users were unable to connect to SQL Server datasource using NTLM authentication protocol. Despite using valid database credentials, Service Account Test failed. This issue has been resolved. Users can now connect to SQL server datasource using NTLM authentication protocol. |
7196 |
Fixed the issue of connection to SQL Server data sources failing despite the use of the correct format for the JDBC URLs (“sqlserver://servernam einstancename;database=DBname”). |
7211 |
HDFS/ADLS connectivity issues with regard to incorrect host name, port and SSL for metadata extraction were not getting logged properly. This issue has been resolved. Additional log to capture information on incorrect host name, port and SSL has been added. |
7224 |
Alation Teradata SQL parser has been updated to provide better support for temporal DDL and DML statements. As a result, lineage for temporal views and tables will now work for Teradata data sources. |
7327 |
For Administrators connecting to Hive, Alation now correctly handles Hadoop clusters configured for encrypted RPC calls. Previously, Alation connections would fail if hadoop.rpc.protection was set to anything other than authentication. |
7443 |
Previously, the Field tab of the Tableau workbook listed only up to 100 items, though there were more fields. This issue has been resolved. Pagination is now added on measure and dimension tables of the Fields tab for BI Report, Microstrategy Report, Tableau Workbook, and Tableau Sheet catalog pages. |
7451 |
Previously, foreign key references were not updated when Metadata Extraction was rerun after dropping the foreign key constraints. Now, the foreign key references are updated and the links are displayed correctly on the Columns. |
7546 |
There is now ability to configure Alation to use X-Frame-Options response header to prevent rendering the Alation application in an iframe for the sake of added browser security. Be advised that enabling the X-Frame-Options response will cause the DataOS integration to stop working because it relies on ability to load Alation in iframes. For details on enabling X-Frame-Options, see How to Enable X-Frame-Options Settings for Alation. |
7574 |
The primary and foreign key icons are now displayed in Alation UI for Sybase IQ data sources. |
7575 |
Updating the description of the object of a Schema/Table/Attribute caused the auto-title of the object to disappear. This bug is fixed, and the auto-title remains intact. |
7617 |
Tableau lineage was incomplete when a workbook was created for a published Tableau Datasource. This issue has been resolved. |
7617 7976 |
Tableau extraction would fail for exception containing unicode characters. This issue has been resolved. |
7633 7721 7887 |
Alation can now extract Tableau workbook and datasource created using Tableau 2018. |
7643 |
Tableau Server LDAP user accounts will now be matched with Alation user accounts based on 2 properties: username and email, - and not just username. This will allow Alation to identify Tableau server users who belong to multiple domains: when Alation login username matches more than 1 username on Tableau Server, then the email property will also be taken into account to identify the user. The permissions on extracted Tableau objects in Alation are granted based on the matching combination of username and email between the Tableau Server and Alation. |
7648 |
Fixed an issue where the projects list was cut off in UI and not all projects were displayed for Selective Extraction in Tableau data sources. The projects list for Selected Extraction now will display all the projects - as expected. |
7689 |
Previously, when very large queries were run in Compose for a BigQuery datasource, it might return an empty response with a success message. This issue has been fixed, and such cases will now yield appropriate result or an error message. |
7770 |
The NoneType error was logged to track the contributor action of deleting custom field values. Now, a NoneType check has been added to contributor tracking, and this error will not be logged now. |
7723 |
The id column of some database tables where Query Log Ingestion (QLI) job stores intermediary data has a maximum value. As the QLI job ingests more query logs, the maximum limit could be reached causing the QLI job to crash. This potential issue is preemptively addressed by changing the data type of the id column to accommodate 64-bit integer. |
7740 8071 |
Tableau datasource certification is now supported for Tableau 2018. |
7822 |
Previously, Alation ignored the Hive/Hadoop configuration. Alation will now read core-site.xml and hive-site.xml files placed in /opt/alation /site/site_data/pre_added_jars/. |
7830 |
Previously, metadata extraction for Oracle Data Source failed with the “serializing dbobjects” error when the data type was null. This issue is now resolved. |
7847 |
The Query Log Ingestion (QLI) archiving feature had a bug where one of the QLI tables was wrongfully cleaned up causing one of the downstream background jobs to crash. This issue has now been resolved. |
7897 |
Alation now enforces SAML signature validation to avoid potential privilege escalation attack in SAML authentication. |
7899 |
Alation used to have an issue when scripts could be injected into certain fields in the catalog and then executed when users visited their User Profile page. This issue has been addressed, and any injected script execution is no longer possible. |
7976 9118 9201 |
Fixed an issue with Tableau extraction where it would fail with a “unicode encode error” in Alation. The problem was in unicode character processing, and as a result, Alation would not generate any Lineage. Now, unicode characters are handled correctly and do not cause any problem during Tableau extraction. |
8086 |
For Hive users, the Data Uploader will now work over Kerberos and HTTP Transport Mode. Previously, this combination would cause the uploader to time out after 20 minutes. Note that a valid WebHDFS endpoint must be provided for this to work. |
8097 |
Alation backend database has a default limit of 1G characters on the length of SQL query that can be run. Previously, the Metadata Extraction job used a single INSERT query, which could grow over this limit and cause the job to crash. This issue has now been resolved. |
8130 |
Previously, the update from 5.0.3 to 5.4.7 of Alation could fail due to the migration of custom field data. The problem has been fixed, and the update from 5.0.3 to 5.4.7 is now successful. |
8315 |
The non data source administrators were able to modify the URI and metastore URI on the data source page. Now, only the data source administrators can modify the URI and metastore URI on the data source page. |
8438 |
Fixed a bug where data source admins were not able to import data to a Virtual Data Source. |
8441 |
Metadata Extraction cached data cleanup process is normalized for all relational database types. This will allow metadata to be successfully imported even if its structure has not been updated. |
8474 |
Previously, the customers’ update (version lower than 4.18 to higher version) might have failed during the migration of Object Equivalences to Catalog Sets since the Object Equivalence had huge amounts of objects (in the order of 100K). The issue has now been fixed. |
8563 8921 9201 9257 9434 |
Updated Tableau permission extraction to handle the case where the number of Tableau users exceeds the extraction batch size. This had been causing various issues with viewing Tableau data objects in Alation. Now, Tableau metadata will be extracted correctly. |
8637 |
The Catalog Sets link on the Admin Settings page was not displayed under the Catalog Admin section for users with the Analyst hat when the Enable Stewardship Dashboard was turned on in Labs. This issue has been resolved. |
8723 8823 8888 8973 |
Alation V R3 backup process did not respect the retention configuration settings. As a result, the backup disk might run out of space due to the large size of backup archive. This issue is now resolved. |
8754 |
Tableau workbook and datasource badging is now supported for SSL enabled Tableau server. |
8865 |
The Curation progress bar on the Stewardship Dashboard now gets updated properly when the user adds or removes Title, Description or Custom Fields. This is applicable to such object types as Data Source, Schema, Table, and Attribute. |
9004 9342 |
Fixed an issue with Tableau extraction and Lineage where Alation would not extract all available published data sources, and if that happened, Lineage for some of the reports could not be derived. The problem revealed itself when the number of published data sources extracted by Alation did not match the number of all published data sources on the Tableau server (which was greater). Because of a bug, Alation did not paginate through all available data sources. After this fix, if a workbook uses a published data source, it will be successfully extracted and displayed under the Workbook Connections section, and Lineage will work as expected. |
9044 |
Fixed an issue where in some environments the replication service on the secondary server would not start. This prevented replication of configuration files. If your replication started without error this does not impact your installation. |
9201 9434 |
Fixed a problem with Tableau extraction with “Permissions Enforcement” setting enabled. Because of this issue, Alation would not save newly extracted permissions, which led to Tableau workbooks to be inaccessible in Alation by any user. This should no longer be the case, and workbook extraction is expected to work correctly. |
9239 9276 9624 |
Fixed an issue with offline usage data reporting where users accessing Alation via the latest version of Google Chrome would receive the “Unsupported Browser” error on Admin Settings > Reporting > Upload. |
9301 |
Users were able to only extract 50 schemas when Metadata Extraction was performed for Google BigQuery data source. Now, users can extract all schemas. |
Miscellaneous Other Fixes¶
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Known Issues¶
|
|
RELEASE 5.8.3¶
Fixes Referenced by Support Tickets¶
Support Ticket # |
Description |
10350 |
Fixed an upgrade issue that was isolated to customers with legacy versions of Keytab. |
Miscellaneous Other Fixes¶
|
|
|
RELEASE 5.8.4¶
Fixes Referenced by Support Tickets¶
Support Ticket # |
Description |
7733 |
The DES-CBC3-SHA SSL cipher potentially vulnerable to Sweet32 attack has been removed. Alation now offers only secure ciphers. |
9293 |
Fixed an issue where metadata extraction (MDE) from AWS Glue would only fetch 100 tables from all the schemas. MDE for AWS Glue now fetches all databases and tables from all the paginated lists in one job. |
RELEASE 5.8.5 - Security Patch¶
This patch is a critical fix that addresses the security vulnerability with logging which has been described in the recent Security bulletin:
April 17, 2019 - Important Security Alert: PII on Customer Servers
Summary¶
In order to assist with debugging, the Alation application writes exceptions to log files. Alation’s codebase includes an open-source third-party exception-logging library called Raven. We recently discovered that Raven has an undocumented bug such that it sometimes logs usernames and passwords stored in the application memory to the Alation log files in cleartext.
In this patch we completely deactivate Raven.Originally,we included the Raven capabilities into Alation to have the richest possible logs for rapid issue resolution. Although without Raven certain issues may be more difficult to diagnose, we will continue to have other logging information. There are no plans on adding those exception-logging capabilities back, presently; we will rely on other kinds of logs and monitoring for diagnosis of issues.
It is of paramount importance that this security patch is installed by all customers.
Note that this patch does NOT remove any of the existing logs from your Alation instance. If you have not purged the existing logs as part of the security bulletin recommendations, please consider doing so after you install the patch. For instructions, section IV-B, April 17, 2019 - Important Security Alert: PII on Customer Servers.
RELEASE 5.8.6 - Security Patch¶
We have discovered that in certain scenarios, service account credentials may have been logged in taskserver.log files. The details have been described in our June 4, 2019 - Patch Alert/Security Bulletin: Data Source Credentials in taskserver.log Files. The patch fixes this logging problem.
Once you upgrade, please follow the instructions outlined in the bulletin to delete any existing Taskserver logs from your Alation instance.
RELEASE 5.8.7¶
Fixes Referenced by Support Tickets¶
Support Ticket # |
Description |
---|---|
10667 |
Resolved a bug where the public API implementation for lineage computed fingerprints based on names instead of UUIDs. The lineage API now computes fingerprints using UUIDs. For example, the lineage between a table in the catalog and a tableau_datasource object will now be based on <tableau_server_id>.<site_name>.<tableau_datasource_name>. |
RELEASE 5.8.8¶
Fixes Referenced by Support Tickets¶
Support Ticket # |
Description |
---|---|
11560 |
Previously, if a project contained a multiple of 50 datasets in Google BigQuery, no dataset was fetched when Metadata Extraction (MDE) was performed. This issue has been resolved. Now, all datasets get fetched when MDE is performed. |