• Arun Nukula

Behind the Scenes: vRealize Automation Upgrade 8.2 to 8.3


I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3


Thought it would be useful for everyone hence writing this blog


As a first step, we need to keep the updaterepo ready and downloaded



Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade


Now that the snapshot is taken we will go ahead with upgrade


Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available

Once , done we need to run precheck



Whole precheck pdf attached here

precheck-report(1)
.pdf
Download PDF • 14KB

When you click on next check the repo url and then submit , upgrade starts




Upgrade successfully completes in 1.5 hours.



Here's the list of log files which are to be monitored during upgrade



BEHIND THE SCENES


The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log



upgrade-noop.log 


[INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 
                                 Dload  Upload   Total   Spent    Left  Speed 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.109.45.33... 
* TCP_NODELAY set 
* Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0) 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* ignoring certificate verify locations due to disabled peer verification 
* TLSv1.2 (OUT), TLS header, Certificate Status (22): 
} [5 bytes data] 
* TLSv1.2 (OUT), TLS handshake, Client hello (1): 
} [512 bytes data]
*
*
*
* Connection #0 to host reflcm.nukescloud.com left intact 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully.
*
*
*
*  SSL certificate verify result: self signed certificate (18), continuing anyway. 
} [5 bytes data] 
> GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1 
> Host: reflcm.nukescloud.com 
> User-Agent: curl/7.59.0 
> Accept: */*
*
*
<  
{ [15833 bytes data] 
100 64516    0 64516    0     0  2250k      0 --:--:-- --:--:-- --:--:-- 2250k 
* Connection #0 to host reflcm.nukescloud.com left intact

Then starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log




1.  Searches for the bootstrap package and checks if SSH port is open and it's able to connect 


[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ... 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm. 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status... 
[INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded. 
[INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data... 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded. 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services... 
[INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node. 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded. 


2. Retrieves product version on all nodes. In this environment we just have one node


[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes. 
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified. 


3. Performs Infrastructure Health 


[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health 
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes.


4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes 


[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes. 
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Remote command succeeded: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Update repository set successfully on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to update repository on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to the update repository.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Access to the update repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Verifying availability of new product version in the repository. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Availability of new version in the repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Update repository is verified successfully on all nodes.


5. When the above vami-config.sh runs under /opt/vmware/var/log/vami/vami.log , the following repository is set 


30/03/2021 06:21:34 [INFO] Setting local repository address. url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=<empty> 
30/03/2021 06:21:34 [INFO] Checking available updates. jobid=1 
30/03/2021 06:21:34 [INFO] Downloading latest manifest. jobId=1, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 
30/03/2021 06:21:35 [INFO] Signature script output: Verified OK 
30/03/2021 06:21:35 [INFO] Signature script output:  
30/03/2021 06:21:35 [INFO] Manifest signature verification passed 
30/03/2021 06:21:35 [INFO] New manifest. installed-version=8.2.0.12946, downloaded-version=8.3.0.15014


6. Starts to shutdown application or services


[INFO][2021-03-30 06:21:38][labvra.nukescloud.com] Shutting down application services 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Application services shut down successfully. 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Shutting down infrastructure services 
[INFO][2021-03-30 06:26:27][labvra.nukescloud.com] Infrastructure services shut down successfully. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Saving restore points on all nodes. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-save.sh 'local' sys-config sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore points.

========================= 
[2021-03-30 06:24:03.425+0000] Waiting for deploy healthcheck 
=========================
========================= 
[2021-03-30 06:24:09.389+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 06:24:15.420+0000] Tear down existing deployment 
=========================
========================= 
[2021-03-30 07:24:29.787+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 07:24:32.368+0000] Tear down existing deployment 
=========================

7. Saves local restore point for k8's , processes node data and then activates monitor on all nodes


[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore point for Kubernetes object prelude-vaconfig. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore point for Kubernetes object prelude-vaconfig saved successfully. 
Flag --export has been deprecated, This flag is deprecated and will be removed in future. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore points saved successfully. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Restore points on all nodes saved successfully. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Retrieving nodes status... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes status retrieval succeeded. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Processing nodes data... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes data processing succeeded. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Verifying nodes... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes verification completed successfully. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating local monitors on all nodes. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating upgrade monitor on the node


8. Once the upgrade monitor is activated on the node and cluster nodes are removed successfully , it will proceed for installation. Remember this is a simple install


[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Upgrade monitor activated successfully on the node. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Local monitors successfully activated on all nodes. 
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Removing nodes from the cluster 
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Running remote command: vracli cluster leave at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:44:26][labvra.nukescloud.com] Cluster nodes removed successfully


9. The Installation of the updates start 
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Starting installation of updates. 
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Update installation started successfully. 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade in progress. 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade still in progress.


10. Just a second after it states Starting installation of updates under the upgrade-xx-xx-xx-xx.log , this is the time you would see activity under vami.log 


30/03/2021 06:46:02 [INFO] Installing updates. instanceid=VMware:VMware_8.3.0.15014, jobid=2 
30/03/2021 06:46:02 [INFO] Installing update. instanceId=VMware:VMware_8.3.0.15014, jobId=2, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 
30/03/2021 06:46:02 [INFO] Downloading and installing update packages 
30/03/2021 06:46:02 [INFO] Using download cache 
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 
30/03/2021 06:46:02 [INFO] Using download cache 
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 
30/03/2021 06:46:02 [INFO] Signature script output: Verified OK 
30/03/2021 06:46:02 [INFO] Signature script output:  
30/03/2021 06:46:02 [INFO] Manifest signature verification passed 
30/03/2021 06:46:02 [INFO] Creating /opt/vmware/var/lib/vami/update/data/update_progress.json 
30/03/2021 06:46:02 [INFO] Downloading the following packages for update version 8.3.0.15014 
30/03/2021 06:46:02 [INFO] package NEW PACKAGE   : VMware-Log-Insight-Agent  noarch  (none)  8.3.0  17489324  /package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm  rpm  9457625  e744601203ec01d0d637453260324b7b35a872ce 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: abx-common  noarch  (none)  8.3.0.15014  1  /package-pool/abx-common-8.3.0.15014-1.noarch.rpm  rpm  43780508  c134c5476bd805293bdfaf45dc9b6bd9e06bde63 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bindutils  x86_64  (none)  9.16.6  1.ph2  /package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm  rpm  3584466  67d101f6718a81a2abc3dfd9b54381cac6948b60 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bootstrap-prelude  noarch  (none)  8.3.0.15014  1  /package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm  rpm  2629839  14c2906620ddca6690e13867f1e385851b9e06d8 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: cifs-utils  x86_64  (none)  6.7  2.ph2  /package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm  rpm  38568  2eea29de217fc837792d0c8db9848d240c3b18d5 
30/03/2021 06:46:02 [INFO] package NEW PACKAGE   : config-xl  noarch  (none)  8.3.0.15014  1  /package-pool/config-xl-8.3.0.15014-1.noarch.rpm  rpm  14255  c32254fd9e2c1074ea1b5e33e9c72af5350c27e4 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: containerd  x86_64  (none)  1.2.14  1.ph2  /package-pool/containerd-1.2.14-1.ph2.x86_64.rpm  rpm  24221328  227d22a05417f4286e0247ca87ab415f0b095231
*
*
*
30/03/2021 06:48:34 [INFO] package UPDATE VERSION: vro-migrate  noarch  (none)  8.3.0.1611732871  17522798  /package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm  rpm  4984  eb517af86432ea5b752627335696eec3509a052c 
30/03/2021 06:48:34 [INFO] Install packages set:  '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/abx-common-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm'
*
*
vmware-studio-appliance-config_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-studio-init_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm' 
30/03/2021 06:48:34 [INFO] Reboot required when installing linux package 
30/03/2021 06:48:34 [INFO] Created reboot required file 
30/03/2021 06:48:34 [INFO] Update 8.3.0.15014 manifest is set to be installed 
30/03/2021 06:48:34 [INFO] Using update post-install script 
30/03/2021 06:48:34 [INFO] Running updatecli to install updates. command={ mkdir -p /usr/share/update-notifier ; ln -s /opt/vmware/share/vami/vami_notify_reboot_required /usr/share/update-notifier/notify-reboot-required ; /opt/vmware/share/vami/update/updatecli '/opt/vmware/var/lib/vami/update/data/job/2' '8.2.0.12946' '8.3.0.15014' ;  mv /opt/vmware/var/lib/vami/update/data/update_progress.json /opt/vmware/var/lib/vami/update/data/.update_progress.json ;  } >> /opt/vmware/var/log/vami/updatecli.log 2>&1 & 
30/03/2021 06:48:36 [INFO] Installation running in the background


11. Once the installation starts