Rewind - Rollback Failed Tableau Server Upgrade

Normally it's seen administrators are really good at making action plan for upgrades but when the things didn't turned up as planned, what needs to to be done? It's equally important to have a rollback plan and even test it so that action can be taken quickly and confidently in the case of any unwanted situation during the upgrades.

In my last blogpost wrote about steps to perform tableau server upgrade from version 2019.2 to 2020.3.3 and this blogpost will cover steps to rollback in case of any failed upgrade.

There are some assumptions before we go through the steps

    1. Upgrade from version 2019.2 to 2020.3.3 failed on Linux.

    2. Upgrade failed on tsm-upgrade command and we decided to reach to the point where the upgrade was started.

In order to rollback your failed upgrade following steps needs to be followed, steps are truly based on my experience and would suggest you try this in lower environments first. There would be some customization in your environment, so that needs to taken on top of these steps. 

Step1: Backup the configuration file and backup files.

Though these files are usually backed up before upgrade, but it's important to validate and if not already backed up do it now.

Following configuration files needs to be kept to a safe place, usually it must be not on the local tableau server. NAS are preferred locations.

    SSL Certificates

    keytab files for kerberos

    Server Backup for setting exports and metadata.

        tsm maintenance backup -f <file_name>

        tsm settings export -f <file_name>.json

    Licenses list

        tsm licenses list

    Other files in your configuration.

Step2:  Obliterate your installations 

Next steps would be to obliterate your failed upgrade which will remove all the datafiles, user details and even your backup. In order to perform this you would need to run:

    tableau-server-obliterate -y -y -y 

If you would also want to remove any licensing information from the server (would recommend to do so) use -l flag:

    tableau-server-obliterate -y -y -y -l

With version 2020 tableau by default saves all logs and backup if you want to avoid that use flag -a as you would have already backed up that in Step1.

    tableau-server-obliterate -y -y -y -l -a

This command is usually needs to be run as root user or if you have sudo permission then as owner of tableau server binaries. Always make sure you are out of your installation directory else un-install will fail and will keep partial installation on the server. Better step to /tmp and make sure no other login are present and run the obliterate command.

In worse case you still see rpm present on the server you have to just unpack packages folder from some other server with same version binary and run the obliterate again. 

Command to check rpm has been un-installed is :

    rpm -qa | grep tableau

Remove any binaries lets in tableau server installation using rm commands and clear off the directories before heading for another install.

Step3: Place the your OS installation user back

Obliterate command also remove your tableau user and associated group like tableau, tsmadmin. That means you would need to take care of entries in following files.

    /etc/group

    /etc/passwd

    /etc/gshadow

    /etc/shadow

Step4: Perform the tableau server installation again and configuration.

This step would need the usual steps to install the tableau server and complete the configuration.

Step5: Apply the tableau server backup taken before the upgrade

At this step you would need to play the backup of previous versions taken before the upgrade and restore it. In order to start with that copy the backup files to backup file path.

    tsm stop   

    tsm maintenance restore -f <backup file name>

    tsm start

    tsm status -v

Perform any post steps or configuration which you would have done for your installations.

Summary 

Success to your upgrade journey not just depends on how you are prepared when things doesn't goes as expected and rollback plans needs to be place. This blogpost list down all the steps needs to perform the rollback to your previous version (personally struggled to find them at one place).