Troubleshooting Alation Analytics V2 Upgrade to 2021.4¶
Customer Managed Applies to customer-managed instances of Alation
Applies to version 2021.4
Warning
These steps require the root user. When you switch to the root user, always exercise caution. You can lose data or damage the system by moving or deleting critical files. Only perform the task as described in the steps and log out of the root user after completing it.
Issue: Alation Analytics V2 Postgres Upgrade Fails¶
If during the update of the Alation Analytics V2 application to its 2021.4-compatible version, the upgrade of the Alation Analytics V2 database to version 13.1 fails and the installer exits with the following error in the console:
[X Failed] Postgres database is now being upgraded. Failure Reason => exit status 1:
check the Alation Analytics V2 installer log file installer.log at /var/log/alation-analytics (path inside the Alation shell). A more specific error message will be in this log file.
Error: “initdb: could not access directory “/var/lib/postgresql/data”¶
If you find the following error in the installer.log:
Error: “initdb: could not access directory “/var/lib/postgresql/data”: Permission denied”
perform the steps given below.
On the Alation Analytics V2 host, change user to root:
sudo -iGo to the Postres directory under the Alation Analytics V2 installation directory. By default, it is /opt/alation-analytics/postgres. Your installation directory may have a different location and name.
cd /opt/alation-analytics/postgres
Change permissions on the sub-directory /data in the /postgres directory in the following way:
chown -R 70:70 data/
Exit the root user:
exitRun the Alation Analytics installer of version 2021.4 with the update option one more time.
Error: New cluster database “alation_analytics_v2” is not empty¶
If you find the following error in the installer.log:
Error: New cluster database “alation_analytics_v2” is not empty: found relation “public.django_migrations”
perform the steps given below.
On the host of the Alation Analytics V2 application, change user to root:
sudo -iGo to the Postres directory under the Alation Analytics V2 installation directory. By default, it is /opt/alation-analytics/postgres. Your installation directory may have a different location and name.
cd /opt/alation-analytics/postgres
Remove the sub-directory /13:
rm -rf 13/
Exit the root user:
exitRun the Alation Analytics installer of version 2021.4 with the update option one more time.
If solutions do not work¶
Also see Also see Alation Analytics V2 Upgrade Console Errors below.
If the provided solutions do not work and the installer continues to exit with the same or a different failure message, this means that the upgrade of the Alation AnalyticsV2 application on your instance has not been completed. The ETL process will not work and the Alation Analytics V2 database data will not be updated while the system is in this state.
Also see Alation Analytics V2 Upgrade Console Errors below.
Contact Alation Support to guide you through the troubleshooting process.
If you need the Alation Analytics V2 database to be operational, you can roll the database back to version 9.6 using the steps in Roll Back Postgres to 9.6 below. Note that these steps only bring up the Postgres 9.6 Docker container if Postgres upgrade has not been started. They do not resolve the issue with the upgrade.
Roll Back Postgres to 9.6¶
Use the steps in this section to bring the Alation Analytics V2 Postgres Docker container up after the Alation Analytics V2 database upgrade failed during the update to version 2021.4.
If Postgres upgrade fails at the database upgrade step, it is likely that the installer did not move any further. The Docker image for Postgres still uses version 9.6. There is no need to reload the image and it can be brought up using Docker Compose.
Go to the the Alation Analytics V2 installation directory. By default, it is /opt/alation-analytics. Your installation directory may have a different location and name.
cd /opt/alation-analytics
This directory should contain the docker-compose.yaml file. View the contents of this file. Locate an entry named Volumes. The volume mount should be as shown below:
Volumes: /opt/alation-analytics/postgres/data:/var/lib/postgresql/data
If this is the volume mapping you see, proceed with the next steps. If you see a different mount, for example:
/opt/alation-analytics/postgres/13/data:/var/lib/postgresql/13/data
then do not proceed and contact Alation Support. If you see the volume mount of version 13, this means that there was an issue with the database upgrade and it is incomplete. You will not be able to safely start the container with the previous database version.
Run the command below to bring up the Postgres container:
sudo /usr/local/bin/docker-compose -f /opt/alation-analytics/docker-compose.yaml up -d
Check if the Postgres container is up:
docker ps -a | grep postgres
Validate the Postgres version by running:
docker exec -it postgres sh
The version should be 9.6.
Alation Analytics V2 Upgrade Console Errors¶
exit status 1: initdb: warning: enabling “trust” authentication for local connections¶
If the Alation Analytics V2 installer exits during the upgrade with the following error in the console:
exit status 1: initdb: warning: enabling “trust” authentication for local connectionsnYou can change this by editing pg_hba.conf or using the option -A, orn–auth-local and –auth-host, the next time you run initdb.”
perform the steps given below.
Go to the the Alation Analytics V2 installation directory. By default, it is /opt/alation-analytics. Your installation directory may have a different location and name.
cd /opt/alation-analytics
This directory should contain the docker-compose.yaml file. View the contents of this file. Locate an entry named Volumes. The volume mount should be as shown below:
Volumes: /opt/alation-analytics/postgres/data:/var/lib/postgresql/data
If this is the volume mapping you see, proceed with the next steps. If you see a different mount, for example:
/opt/alation-analytics/postgres/13/data:/var/lib/postgresql/13/data
Then do not proceed and contact Alation Support. If you see the volume mount of version 13, this means that there was an issue with the database upgrade and it is incomplete. You will not be able to safely start the container of the previous database version.
Change to root user:
sudo -iGo to the Postres directory under the Alation Analytics V2 installation directory.
cd /opt/alation-analytics/postgres
If the subdirectory /13 exists in /postgres, remove it:
rm -rf 13/
Check permissions of user:group of the /data subdirectory:
ls -ld data/
You should see the following output:
drwx------ 19 70 70 4096 Xxx xx xx:xx data/
If the permissions are different from the expected, run the command given below:
chown -R 70:70 data/
Exit from the root user:
exitRestart the Postgres image:
sudo /usr/local/bin/docker-compose -f /opt/alation-analytics/docker-compose.yaml up -d
Proceed with the upgrade by running the Alation Analytics V2 installer of the 2021.4 version with the update option one more time.
Postmaster Error¶
If the Alation Analytics V2 installer exist during the upgrade with the following error in the console:
There seems to be a postmaster servicing the old cluster. Please shutdown that postmaster and try again. Failure, exiting
this means that the Alation Analytics V2 Postgres cannot be auto-upgraded. Contact Alation Support to guide you through the manual upgrade process for Postgres.