• Arun Nukula

Upgrading vRealize Automation


Recently I did work on a project to upgrade vRealize Automation from 7.0.1 to 7.3.

I would like to share my experience on how the upgrade took place and various issues encountered while implementing it

We had to execute upgrade in two phases

  • Upgrade from 7.0.1 to 7.2

  • Once 7.2 validation is complete upgrade to 7.3

This was done in two phases because in-place upgrade between 7.0.1 to 7.3 is not supported

Pre-Requisites

  • Ensure Databases have been backed up. Both IaaS ( Contact your DBA ) and vRA Postgres database.

  • Take snapshots of all virtual machines ( vRA appliances and IaaS machines ).

  • Ensure vRA components are in working and stable condition and correct all issues before starting upgrade

  • Check for any space issues on root partition of appliances

  • Check the /storage/log subfolder and remove any old archived ZIP files to cleanup space

  • Ensure you have access to database , if you have restricted access then ensure DBA is around to help you during situation where you need to revert during failure

  • System must be unavailable for users to submit new request and any applications that query vRealize Automation

Upgrade from 7.0.1 to 7.2

Appliance Upgrade

  • Create a file called disable-iaas-upgrade under /tmp on all vRA appliances as we do not want to perform an automated IaaS upgrade.

  • /tmp/disable-iaas-upgrade

  • Mount 7.2 vRA iso images to appliances (VMware-vR-Appliance-7.2.0.381-4660246-updaterepo.iso)

  • Ensure under VAMI portal , Update Appliance settings are pointing towards CDRom updates

  • Click on "Check Updates" , then you would get information on existing version and the version your upgrading to , accept and proceed for upgrade

  • You may use following logs to see how upgrade is progressing

  • ​/opt/vmware/var/log/vami/vami.log

  • /opt/vmware/var/log/vami/updatecli.log

  • /var/log/messages

  • Once upgrade is complete, VAMI asks you to reboot appliances once done vRA appliances are upgraded to 7.2

IaaS Upgrade

  • Ensure Java version is Java 8 build 91 or above

  • Download IaaS installer from vRA VAMI ( https://vrapphostname:5480/installer)

  • Execute IaaS install on all the IaaS nodes starting with

  • ​IaaS Websites starting with the one where Model Manager Data is installed and then going ahead with other nodes

  • Manager Services , ensure active one is upgraded before the passive node

  • DEM Orchestrator and workers. Ensure you perform one box after another

  • Agents , one node after another

  • Management Agents on all nodes ( automatically upgraded ) , no manual intervention needed

  • Check cluster status

  • rabbitmqctl cluster_status

  • Ensure all services are in "REGISTERED" status under VAMI portal of vRA nodes

  • Upgrade vRO appliances if it's external

Issues encountered during upgrade to 7.2

  • While starting vRA upgrade by mounting iso image during post upgrade phase of postgresql on appliances we did encounter failure

Exception

+ echo 'Script /etc/bootstrap/postupdate.d/09-90-prepare-psql failed, error status 1'

+ exit 1

+ exit 1

+ trapfunc

+ excode=1

+ test 1 -gt 0

+ vami_update_msg set post-install 'Post-install: failed'

+ test -x /usr/sbin/vami-update-msg

+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'

+ sleep 1

+ test 1 -gt 0 -o 0 -gt 0

+ vami_update_msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'

+ test -x /usr/sbin/vami-update-msg

+ /usr/sbin/vami-update-msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'

+ exit

21/10/2017 01:12:41 [ERROR] Failed with exit code 256

21/10/2017 01:12:41 [INFO] Update status: Running VMware tools reconfiguration

21/10/2017 01:12:41 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools

vmware-toolbox-cmd is /usr/bin/vmware-toolbox-cmd

Configuring VAMI VMware tools service wrapper.

21/10/2017 01:12:41 [INFO] Update status: Done VMware tools reconfiguration

21/10/2017 01:12:41 [INFO] Update status: Error while running post-install scripts

21/10/2017 01:12:41 [ERROR] Failure: updatecli exiting abnormally

21/10/2017 01:12:41 [INFO] Install Finished

Resolution

Ensure Clustering is working as expected between Primary and Secondary vRA nodes

Exception

  • Model Manager Data upgrade fails with an error stating

[21/10/2017 8:09:09 PM] "E:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\RepoUtil.exe" Model-Uninstall -f DynamicOps.ManagementModel.dll -v

[21/10/2017 8:09:11 PM] System.InvalidOperationException: File DynamicOps.ManagementModel.dll not found.

Upgrade from 7.2 to 7.3

Resolution

  • When above failure occurred , we had to revert IaaS node where Model Manager Data was and then the SQL server which was hosting the database

  • Once done , we had to uninstall and reinstall MSDTC

  • Post these steps , restarted upgrade which helped in fixing upgrade failure on IaaS

Upgrade from 7.2 to 7.3

Appliance Upgrade

Same steps as the previous upgrade to 7.2. Only difference is that when you start upgrading primary vRA node it automatically upgrades the second one as well.

IaaS Upgrade

Same steps as previous upgrade to 7.2

Issues encountered during upgrade to 7.3

Exception

Though vRA upgrade was successful post reboot of the nodes or appliances apart from component registry , none of the services were showing up as registered

Browsing through logs we did find out that horizon.log under vRA appliance ( Primary ) was full of postgres errors

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146:

Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:

Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:

Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist

2017-10-22 01:41:43,410 ERROR (localhost-startStop-1) [;;;] org.springframework.web.context.ContextLoader - Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146:

Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:

Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:

Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

Resolution

If you encounter this issue it does not mean that your upgrade failed. After the upgrade there are new liquibase files which are downloaded who performs DB changes if any between versions. The reason of failure is because CONSTRAINT was not present

Check details below

  • Login into vRA database

  • set schema 'saas'

  • \x to set expanded display ON

  • \d+ "EncrytionKeys"

  • Output of above commands would be


  • Before you create a constraint you need to verify if there are duplicate key id's. If yes , then your alter command will fail with following error

ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id);

ERROR: could not create unique index "encryptionkeys_pkey"

DETAIL: Key (id)=(1) is duplicated.

  • So now identify number of id's present in this table. In my case there were 8 where first two had same key ID as described below

id | 1

uuid | 6e4c6867-377e-4ce8-9f76-102da90951a7

keyContainer | __GLOBAL_REST_KEY_CONTAINER__

algorithmId | bbdec4a9-dbaa-49bd-95ed-11fd6cff26f3

keySize | 1024

id | 1

uuid | 6d1d29ae-4bb6-487e-a902-2ddb798f9c3c

keyContainer | VSPHERE.LOCAL

algorithmId | 1f932d40-0204-11e2-a21f-0800200c9a66

keySize | 256

  • Observe carefully id number is same. Select a number for ID which is not present in the output from previous command and then change the value of key id

  • Select UUID to change the Key

  • ​​select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7';

  • Remember the value of uuid will vary from environment to environment

  • Update UUID with the new key

  • update saas."EncryptionKeys" set id=8 where uuid='6e4c6867-377e-4ce8-9f76-102da90951a7';

  • Verify if the new ID is updated

  • select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7';

  • Once done now run the Alter table command to create constraint for the table

  • ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id);

  • Once done verify this using \d+ "EncryptionKeys" command , you would see now the Primary Key is present.


  • Restart vRA services so that all services will now be shown as "REGISTERED" successfully on both the nodes if distributed and on single node if simple installation

  • This issue is pre-dominantly seen on version upgrades from 7.0 and 7.0.1 to 7.3

Happy Upgrades

79 views

Recent Posts

See All

Subscribe Now

  • Twitter
  • Facebook Social Icon

Copyright © 2019 nukescloud