top of page

Behind the Scenes: vRealize Automation Upgrade 8.2 to 8.3


I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3


Thought it would be useful for everyone hence writing this blog


As a first step, we need to keep the updaterepo ready and downloaded



Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade


Now that the snapshot is taken we will go ahead with upgrade


Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available

Once , done we need to run precheck



Whole precheck pdf attached here

precheck-report(1)
.pdf
Download PDF • 14KB

When you click on next check the repo url and then submit , upgrade starts




Upgrade successfully completes in 1.5 hours.



Here's the list of log files which are to be monitored during upgrade



BEHIND THE SCENES


The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log



upgrade-noop.log 


[INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 
                                 Dload  Upload   Total   Spent    Left  Speed 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.109.45.33... 
* TCP_NODELAY set 
* Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0) 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* ignoring certificate verify locations due to disabled peer verification 
* TLSv1.2 (OUT), TLS header, Certificate Status (22): 
} [5 bytes data] 
* TLSv1.2 (OUT), TLS handshake, Client hello (1): 
} [512 bytes data]
*
*
*
* Connection #0 to host reflcm.nukescloud.com left intact 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully.
*
*
*
*  SSL certificate verify result: self signed certificate (18), continuing anyway. 
} [5 bytes data] 
> GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1 
> Host: reflcm.nukescloud.com 
> User-Agent: curl/7.59.0 
> Accept: */*
*
*
<  
{ [15833 bytes data] 
100 64516    0 64516    0     0  2250k      0 --:--:-- --:--:-- --:--:-- 2250k 
* Connection #0 to host reflcm.nukescloud.com left intact

Then starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log




1.  Searches for the bootstrap package and checks if SSH port is open and it's able to connect 


[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ... 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm. 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status... 
[INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded. 
[INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data... 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded. 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services... 
[INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node. 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded. 


2. Retrieves product version on all nodes. In this environment we just have one node


[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes. 
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified. 


3. Performs Infrastructure Health 


[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health 
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes.


4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes 


[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes. 
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Remote command succeeded: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Update repository set successfully on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to update repository on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to the update repository.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Access to the update repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Verifying availability of new product version in the repository. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Availability of new version in the repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Update repository is verified successfully on all nodes.


5. When the above vami-config.sh runs under /opt/vmware/var/log/vami/vami.log , the following repository is set 


30/03/2021 06:21:34 [INFO] Setting local repository address. url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=<empty> 
30/03/2021 06:21:34 [INFO] Checking available updates. jobid=1 
30/03/2021 06:21:34 [INFO] Downloading latest manifest. jobId=1, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 
30/03/2021 06:21:35 [INFO] Signature script output: Verified OK 
30/03/2021 06:21:35 [INFO] Signature script output:  
30/03/2021 06:21:35 [INFO] Manifest signature verification passed 
30/03/2021 06:21:35 [INFO] New manifest. installed-version=8.2.0.12946, downloaded-version=8.3.0.15014


6. Starts to shutdown application or services


[INFO][2021-03-30 06:21:38][labvra.nukescloud.com] Shutting down application services 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Application services shut down successfully. 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Shutting down infrastructure services 
[INFO][2021-03-30 06:26:27][labvra.nukescloud.com] Infrastructure services shut down successfully. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Saving restore points on all nodes. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-save.sh 'local' sys-config sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore points.

========================= 
[2021-03-30 06:24:03.425+0000] Waiting for deploy healthcheck 
=========================
========================= 
[2021-03-30 06:24:09.389+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 06:24:15.420+0000] Tear down existing deployment 
=========================
========================= 
[2021-03-30 07:24:29.787+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 07:24:32.368+0000] Tear down existing deployment 
=