Backup V2 Process¶

Customer Managed Applies to customer-managed instances of Alation

Backup V2 process depends on your Alation release.

Version 2023.3.5 and Newer¶

From version 2023.3.5, Alation offers the ability to move Alation’s internal Postgres database to an external AWS RDS instance. If you have done this, Alation automatically uses the AWS RDS functionality to create a snapshot of the RDS instance. You do not need to change or reconfigure Alation’s backup features. See Move the Alation Database to AWS RDS for more information.

If you have not moved Alation’s Postgres database to RDS, Alation backups function as described for version 2023.1 and newer below.

Version 2023.1 and Newer¶

From version 2023.1, the Backup V2 process uses pgBackRest as the default tool for backing up the internal Postgres database. The pgBackRest tool allows excluding the Postgres database from the compressed Alation backup and managing the database backup separately from the Alation backup. When Postgres is excluded from the Alation backup, its backup will be created in a separate directory.

Postgres Backup Is Included (Default)¶

By default, the backup process will create two .tar.gz backup files in the backup directory: the Alation backup (includes the Postgres database backup) and the Event Bus backup.

Alation backup: <timestamp_version>_alation_backup.tar.gz

Example: 202008140102_5-17-0-116556_alation_backup.tar.gz

Event Bus backup: <timestamp_version>_alation_eb_backup.tar.gz

Example: 202109232027_10-0-0-147420_alation_eb_backup.tar.gz

Note

The Event Bus backup tarball name differs from the Alation backup file name with the _eb_ suffix.

During the backup process, the backup data is first staged in backup staging directories. Alation stages the data in two locations: /data2/tmp and /data2/backup/pgbackrest (both paths are given inside the Alation shell).

The directory /data2/tmp is always present on /data2. If the /data2/tmp folder is not initially empty, its contents are cleared automatically at the start of the backup process. It is the default location for staging and compressing the backup data and creating the backup .tar.gz files.

Note

This default backup staging location can be changed using the alation_conf parameter alation.backup.data_temp_dir. At any time, the alation.backup.data_temp_dir parameter stores the location of the current staging directory for the creation of the compressed backup files.

The directory /data2/backup/pgbackrest will be created by the first backup process and used for staging the Postgres database data for backing up. When the backup process has completed, the staged contents of the /data2/tmp and /data2/backup/pgbackrest folders are compressed together into the Alation backup .tar.gz file. The Event Bus backup is created as a separate file.

Both compressed backup files will appear in the backup storage directory /data2/backup (default location).

The contents of the /data2/tmp folder are deleted automatically after the backup is complete, but the folder itself is not deleted.

Both backup files are required in order to restore the system from the backup. To restore, an admin should use two backup files with the same timestamp, which means they were taken by the same backup process.

If you are using incremental backups, note that the Event Bus backup will also run every time an incremental Alation backup is created. However, the Event Bus backup is not incremental. The backup process creates a full Event Bus backup every time it runs.

Postgres Backup Is Excluded¶

You can configure the backup process to exclude the Postgres database backup from the compressed Alation backup file. If Postgres is excluded from the Alation backup, it will need to be managed separately. Refer to Manage Postgres Backups Separately for more information.

Versions 2021.4 - 2022.4¶

The backup process follows the same flow as in Versions 2020.3 - 2021.3; however, from version 2021.4, Alation backs up a new Alation server component—the Event Bus—and creates two .tar.gz files backup files in the backup directory: the Alation backup and the Event Bus backup.

Note

The Event Bus component powers the Lineage V3 service and the Manual Lineage Curation functionality. However, the two backup files will be created even if the Lineage V3 service and Manual Lineage are disabled and not in use on an Alation instance.

By default, the backup process will create two .tar.gz backup files in the backup directory: the Alation backup (includes the Postgres database backup) and the Event Bus backup.

Alation backup: <timestamp_version>_alation_backup.tar.gz

Example: 202008140102_5-17-0-116556_alation_backup.tar.gz

Event Bus backup: <timestamp_version>_alation_eb_backup.tar.gz

Example: 202109232027_10-0-0-147420_alation_eb_backup.tar.gz

Note

The Event Bus backup tarball name differs from the Alation backup file name with the _eb_ suffix.

Both backup files are required in order to restore the system from the backup. To restore, an admin should use two backup files with the same timestamp, which means they were taken by the same backup process.

If you are using incremental backups, note that the Event Bus backup will also run every time an incremental Alation backup is created. However, the Event Bus backup is not incremental. The backup process creates a full Event Bus backup every time it runs.

Versions 2020.3 - 2021.3¶

In versions 2020.3 - 2021.3, the Alation backup process creates one .tar.gz backup file <timestamp_version>_alation_backup.tar.gz, for example 202008140102_5-17-0-116556_alation_backup.tar.gz.

The default Postgres backup tool on these versions is pg_probackup.

During the backup process, the backup files are first staged in backup staging directories. Alation creates staging folders in 2 locations: /data2/tmp and /data2/backup/pgbackup (both paths are given inside the Alation shell).

The directory /data2/tmp is always present on /data2. If the /data2/tmp folder is not initially empty, its contents are cleared automatically at the start of the backup process. It is the default location for staging and compressing the backup data and creating the backup .tar.gz file. This default location can be changed using the alation_conf parameter alation.backup.data_temp_dir. So at any time, the alation.backup.data_temp_dir parameter stores the location of the current backup staging directory for the creation of a compressed backup file.

The directory /data2/backup/pgbackup is created by the backup process and used for staging the Postgres data for the backup.

When the backup process has completed, the staged contents of the /tmp and /pgbackup folders are compressed together into a backup .tar.gz file. The name of the backup file includes the date, time, and build number, for example: 202008140102_5-17-0-116556_alation_backup.tar.gz.

A compressed backup file will appear in the backup storage directory /data2/backup (default location).

Note

A compressed incremental backup file includes a copy of the full backup, which is why it is not smaller in size than the full compressed backup. Incremental backups take much less time to be created, but because they are compressed together with the full backup, they take as much space as a compressed full backup.