Behind the Scenes: vRealize Automation Upgrade 8.2 to 8.3
I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3
Thought it would be useful for everyone hence writing this blog
As a first step, we need to keep the updaterepo ready and downloaded
Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade
Now that the snapshot is taken we will go ahead with upgrade
Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available
Once , done we need to run precheck
Whole precheck pdf attached here
precheck-report(1)
.pdf
Download PDF • 14KB
When you click on next check the repo url and then submit , upgrade starts
Upgrade successfully completes in 1.5 hours.
Here's the list of log files which are to be monitored during upgrade
BEHIND THE SCENES
The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log
upgrade-noop.log
[INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.109.45.33...
* TCP_NODELAY set
* Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* ignoring certificate verify locations due to disabled peer verification
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*
*
*
* Connection #0 to host reflcm.nukescloud.com left intact
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ...
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found.
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully.
*
*
*
* SSL certificate verify result: self signed certificate (18), continuing anyway.
} [5 bytes data]
> GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1
> Host: reflcm.nukescloud.com
> User-Agent: curl/7.59.0
> Accept: */*
*
*
<
{ [15833 bytes data]
100 64516 0 64516 0 0 2250k 0 --:--:-- --:--:-- --:--:-- 2250k
* Connection #0 to host reflcm.nukescloud.com left intactThen starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log
1. Searches for the bootstrap package and checks if SSH port is open and it's able to connect
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully.
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ...
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ...
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm.
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes...
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully.
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status...
[INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded.
[INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data...
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded.
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services...
[INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully.
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully.
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully.
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node.
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded.
2. Retrieves product version on all nodes. In this environment we just have one node
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes.
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified.
3. Performs Infrastructure Health
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes.
4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes.
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo