• Arun Nukula

Behind the Scenes: vRealize Automation Upgrade 8.2 to 8.3


I delivered a training recently about the process which occurs in the background when we upgrade vRealize Automation 8.2 to 8.3


Thought it would be useful for everyone hence writing this blog


As a first step, we need to keep the updaterepo ready and downloaded



Then took a manual snapshot , once the snapshot task is complete we shall proceed with the upgrade


Now that the snapshot is taken we will go ahead with upgrade


Before we proceed, we need to perform inventory sync. Unless this is done, PROCEED option will not be available

Once , done we need to run precheck



Whole precheck pdf attached here

precheck-report(1)
.pdf
PDF • 14KB

When you click on next check the repo url and then submit , upgrade starts




Upgrade successfully completes in 1.5 hours.



Here's the list of log files which are to be monitored during upgrade



BEHIND THE SCENES


The first thing which happens when you click on upgrade is that we download manifest. And this is logged under upgrade-noop.log



upgrade-noop.log 


[INFO][2021-03-30 06:20:47][labvra.nukescloud.com] Downloading manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest downloaded successfully.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 
                                 Dload  Upload   Total   Spent    Left  Speed 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.109.45.33... 
* TCP_NODELAY set 
* Connected to reflcm.nukescloud.com (10.109.45.33) port 443 (#0) 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* ignoring certificate verify locations due to disabled peer verification 
* TLSv1.2 (OUT), TLS header, Certificate Status (22): 
} [5 bytes data] 
* TLSv1.2 (OUT), TLS handshake, Client hello (1): 
} [512 bytes data]
*
*
*
* Connection #0 to host reflcm.nukescloud.com left intact 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Validating manifest ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest signature validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Repository product ID matched successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Manifest validated successfully. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Searching for update bootstrap RPM ... 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM found. 
[INFO][2021-03-30 06:20:48][labvra.nukescloud.com] Update bootstrap RPM downloaded successfully.
*
*
*
*  SSL certificate verify result: self signed certificate (18), continuing anyway. 
} [5 bytes data] 
> GET /repo/productBinariesRepo/vra/8.3.0/upgrade/update/package-pool/bootstrap-update-8.3.0.15014-1.noarch.rpm HTTP/1.1 
> Host: reflcm.nukescloud.com 
> User-Agent: curl/7.59.0 
> Accept: */*
*
*
<  
{ [15833 bytes data] 
100 64516    0 64516    0     0  2250k      0 --:--:-- --:--:-- --:--:-- 2250k 
* Connection #0 to host reflcm.nukescloud.com left intact

Then starts the actual upgrade and logged under upgrade-YYYY-MM-DD-HH-MM-SS.log




1.  Searches for the bootstrap package and checks if SSH port is open and it's able to connect 


[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Retrieving product version... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel801-hf345 ... 
[INFO][2021-03-30 06:20:56][labvra.nukescloud.com] Searching for bootstrap package for product version 8.2.0.12946 in /var/vmware/prelude/upgrade/bootstrap/rel810-gap12 ... 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Applying configuration profile lcm. 
[INFO][2021-03-30 06:20:57][labvra.nukescloud.com] Configuration profile lcm applied successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving nodes status... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes status retrieval succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Processing nodes data... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes data processing succeeded. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Verifying nodes... 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Nodes verification completed successfully. 
[INFO][2021-03-30 06:20:58][labvra.nukescloud.com] Retrieving services status... 
[INFO][2021-03-30 06:20:59][labvra.nukescloud.com] Services status retrieval succeeded. 
[INFO][2021-03-30 06:21:00][labvra.nukescloud.com] Processing services data... 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Services data processing succeeded. 
[INFO][2021-03-30 06:21:18][labvra.nukescloud.com] Verifying services... 
[INFO][2021-03-30 06:21:19][labvra.nukescloud.com] Services verification completed successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Creating SSH key pairs 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] SSH key pairs created successfully. 
[INFO][2021-03-30 06:21:21][labvra.nukescloud.com] Configuring SSH on nodes. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Service sshd status: active/enabled 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH port 22 is open. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] SSH configurated on nodes successfully. 
[INFO][2021-03-30 06:21:22][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/ssh-noop.sh at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verifying remote nodes are able to connect to one another and to this node. 
[INFO][2021-03-30 06:21:23][labvra.nukescloud.com] Verification that nodes are able to connect to one another and to this node succeeded. 


2. Retrieves product version on all nodes. In this environment we just have one node


[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Retriving product versions on all nodes. 
[INFO][2021-03-30 06:21:24][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Retrieving product version.
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product version retrieved successfully. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-save-vers.sh prep at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions successfully retrieved from all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Checking that product versions match across all nodes. 
[INFO][2021-03-30 06:21:25][labvra.nukescloud.com] Product versions match across all nodes verified. 


3. Performs Infrastructure Health 


[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Checking appliance nodes infrastructure health 
[INFO][2021-03-30 06:21:26][labvra.nukescloud.com] Running remote command: /opt/health/run-once.sh health at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Remote command succeeded: /opt/health/run-once.sh health at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:31][labvra.nukescloud.com] Infrastructure health check passed on all appliance nodes.


4. Set's up update repository on all nodes and also verifies if this is accessible across all nodes 


[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Setting up update repository on all nodes. 
[INFO][2021-03-30 06:21:33][labvra.nukescloud.com] Running remote command: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Remote command succeeded: vamicli update --repo https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update at host: labvra.nukescloud.com
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Update repository set successfully on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to update repository on all nodes. 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:34][labvra.nukescloud.com] Verifying the access to the update repository.
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Access to the update repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Verifying availability of new product version in the repository. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Availability of new version in the repository verified. 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/vami-config-repo.sh 'local' at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:21:36][labvra.nukescloud.com] Update repository is verified successfully on all nodes.


5. When the above vami-config.sh runs under /opt/vmware/var/log/vami/vami.log , the following repository is set 


30/03/2021 06:21:34 [INFO] Setting local repository address. url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password=<empty> 
30/03/2021 06:21:34 [INFO] Checking available updates. jobid=1 
30/03/2021 06:21:34 [INFO] Downloading latest manifest. jobId=1, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 
30/03/2021 06:21:35 [INFO] Using download cache 
30/03/2021 06:21:35 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 
30/03/2021 06:21:35 [INFO] Signature script output: Verified OK 
30/03/2021 06:21:35 [INFO] Signature script output:  
30/03/2021 06:21:35 [INFO] Manifest signature verification passed 
30/03/2021 06:21:35 [INFO] New manifest. installed-version=8.2.0.12946, downloaded-version=8.3.0.15014


6. Starts to shutdown application or services


[INFO][2021-03-30 06:21:38][labvra.nukescloud.com] Shutting down application services 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Application services shut down successfully. 
[INFO][2021-03-30 06:23:58][labvra.nukescloud.com] Shutting down infrastructure services 
[INFO][2021-03-30 06:26:27][labvra.nukescloud.com] Infrastructure services shut down successfully. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Saving restore points on all nodes. 
[INFO][2021-03-30 06:26:29][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-save.sh 'local' sys-config sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore points.

========================= 
[2021-03-30 06:24:03.425+0000] Waiting for deploy healthcheck 
=========================
========================= 
[2021-03-30 06:24:09.389+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 06:24:15.420+0000] Tear down existing deployment 
=========================
========================= 
[2021-03-30 07:24:29.787+0000] Waiting for command execution pods 
=========================
========================= 
[2021-03-30 07:24:32.368+0000] Tear down existing deployment 
=========================

7. Saves local restore point for k8's , processes node data and then activates monitor on all nodes


[INFO][2021-03-30 06:26:34][labvra.nukescloud.com] Saving local restore point for Kubernetes object prelude-vaconfig. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore point for Kubernetes object prelude-vaconfig saved successfully. 
Flag --export has been deprecated, This flag is deprecated and will be removed in future. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Local restore points saved successfully. 
[INFO][2021-03-30 06:26:35][labvra.nukescloud.com] Restore points on all nodes saved successfully. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Retrieving nodes status... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes status retrieval succeeded. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Processing nodes data... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes data processing succeeded. 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Verifying nodes... 
[INFO][2021-03-30 06:28:25][labvra.nukescloud.com] Nodes verification completed successfully. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating local monitors on all nodes. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Activating upgrade monitor on the node


8. Once the upgrade monitor is activated on the node and cluster nodes are removed successfully , it will proceed for installation. Remember this is a simple install


[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Upgrade monitor activated successfully on the node. 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Remote command succeeded: /opt/scripts/upgrade/mon-activate.sh at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:28:27][labvra.nukescloud.com] Local monitors successfully activated on all nodes. 
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Removing nodes from the cluster 
[INFO][2021-03-30 06:28:28][labvra.nukescloud.com] Running remote command: vracli cluster leave at host: labvra.nukescloud.com 
[INFO][2021-03-30 06:44:26][labvra.nukescloud.com] Cluster nodes removed successfully


9. The Installation of the updates start 
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Starting installation of updates. 
[INFO][2021-03-30 06:46:01][labvra.nukescloud.com] Update installation started successfully. 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade in progress. 
[INFO][2021-03-30 06:47:01][labvra.nukescloud.com] VAMI upgrade still in progress.


10. Just a second after it states Starting installation of updates under the upgrade-xx-xx-xx-xx.log , this is the time you would see activity under vami.log 


30/03/2021 06:46:02 [INFO] Installing updates. instanceid=VMware:VMware_8.3.0.15014, jobid=2 
30/03/2021 06:46:02 [INFO] Installing update. instanceId=VMware:VMware_8.3.0.15014, jobId=2, url=https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update, username=, password= 
30/03/2021 06:46:02 [INFO] Downloading and installing update packages 
30/03/2021 06:46:02 [INFO] Using download cache 
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml 
30/03/2021 06:46:02 [INFO] Using download cache 
30/03/2021 06:46:02 [INFO] Downloading file. URL: https://reflcm.nukescloud.com/repo/productBinariesRepo/vra/8.3.0/upgrade/update/manifest/manifest-latest.xml.sig 
30/03/2021 06:46:02 [INFO] Signature script output: Verified OK 
30/03/2021 06:46:02 [INFO] Signature script output:  
30/03/2021 06:46:02 [INFO] Manifest signature verification passed 
30/03/2021 06:46:02 [INFO] Creating /opt/vmware/var/lib/vami/update/data/update_progress.json 
30/03/2021 06:46:02 [INFO] Downloading the following packages for update version 8.3.0.15014 
30/03/2021 06:46:02 [INFO] package NEW PACKAGE   : VMware-Log-Insight-Agent  noarch  (none)  8.3.0  17489324  /package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm  rpm  9457625  e744601203ec01d0d637453260324b7b35a872ce 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: abx-common  noarch  (none)  8.3.0.15014  1  /package-pool/abx-common-8.3.0.15014-1.noarch.rpm  rpm  43780508  c134c5476bd805293bdfaf45dc9b6bd9e06bde63 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bindutils  x86_64  (none)  9.16.6  1.ph2  /package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm  rpm  3584466  67d101f6718a81a2abc3dfd9b54381cac6948b60 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: bootstrap-prelude  noarch  (none)  8.3.0.15014  1  /package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm  rpm  2629839  14c2906620ddca6690e13867f1e385851b9e06d8 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: cifs-utils  x86_64  (none)  6.7  2.ph2  /package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm  rpm  38568  2eea29de217fc837792d0c8db9848d240c3b18d5 
30/03/2021 06:46:02 [INFO] package NEW PACKAGE   : config-xl  noarch  (none)  8.3.0.15014  1  /package-pool/config-xl-8.3.0.15014-1.noarch.rpm  rpm  14255  c32254fd9e2c1074ea1b5e33e9c72af5350c27e4 
30/03/2021 06:46:02 [INFO] package UPDATE VERSION: containerd  x86_64  (none)  1.2.14  1.ph2  /package-pool/containerd-1.2.14-1.ph2.x86_64.rpm  rpm  24221328  227d22a05417f4286e0247ca87ab415f0b095231
*
*
*
30/03/2021 06:48:34 [INFO] package UPDATE VERSION: vro-migrate  noarch  (none)  8.3.0.1611732871  17522798  /package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm  rpm  4984  eb517af86432ea5b752627335696eec3509a052c 
30/03/2021 06:48:34 [INFO] Install packages set:  '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/VMware-Log-Insight-Agent-8.3.0-17489324.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/abx-common-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bindutils-9.16.6-1.ph2.x86_64.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/bootstrap-prelude-8.3.0.15014-1.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/cifs-utils-6.7-2.ph2.x86_64.rpm'
*
*
vmware-studio-appliance-config_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-studio-init_3.0.0.7-210201212046.noarch.rpm' '/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vro-migrate-8.3.0.1611732871-17522798.noarch.rpm' 
30/03/2021 06:48:34 [INFO] Reboot required when installing linux package 
30/03/2021 06:48:34 [INFO] Created reboot required file 
30/03/2021 06:48:34 [INFO] Update 8.3.0.15014 manifest is set to be installed 
30/03/2021 06:48:34 [INFO] Using update post-install script 
30/03/2021 06:48:34 [INFO] Running updatecli to install updates. command={ mkdir -p /usr/share/update-notifier ; ln -s /opt/vmware/share/vami/vami_notify_reboot_required /usr/share/update-notifier/notify-reboot-required ; /opt/vmware/share/vami/update/updatecli '/opt/vmware/var/lib/vami/update/data/job/2' '8.2.0.12946' '8.3.0.15014' ;  mv /opt/vmware/var/lib/vami/update/data/update_progress.json /opt/vmware/var/lib/vami/update/data/.update_progress.json ;  } >> /opt/vmware/var/log/vami/updatecli.log 2>&1 & 
30/03/2021 06:48:36 [INFO] Installation running in the background


11. Once the installation starts in the background , this is the time we need to monitor /var/log/bootstrap/postupdate.log
The ones highlighted in the RED are the series of scripts which run in the background. 

+ log 'Main bootstrap postupdate started' 
++ date '+%Y-%m-%d %H:%M:%S' 
+ echo '2021-03-30 06:51:46 Main bootstrap postupdate started' 
2021-03-30 06:51:46 Main bootstrap postupdate started 
+ for script in "${bootstrap_dir}"/* 
+ echo


+ log '/etc/bootstrap/postupdate.d/00-clear-services.sh starting...'
+ log '/etc/bootstrap/postupdate.d/00-disable-set-hostname.sh starting...'
+ log '/etc/bootstrap/postupdate.d/00-fix-vmtoolsd-deps.sh starting...'
+ log '/etc/bootstrap/postupdate.d/01-10-reset-k8s starting...'
+ log '/etc/bootstrap/postupdate.d/02-07-prune-images.sh starting...'
*
*
untagged: photonos3_private:latest 
deleted: sha256:30ded4d67031ea69c5792250dcd4c466511d940ff92f13cd6cc945c4456596ed 
untagged: relocation-service:ed72098 
untagged: relocation-service_private:latest 
deleted: sha256:840c0a6cb51902b0bb1d81c7ebb3b7ae233e1666dbe387f39a24d7066ad1781a 
deleted: sha256:3e737404723de6f8e2a6d8ad85b8cad2b9206e76b40b536766ba6a164a6d24ed 
deleted: sha256:ea1c6ad6204eb1aa5ffccc44a3890014b52a39bc9ed64de34af88195670196e5 
deleted: sha256:3a90641efc2b2704dcfa4a8777a4c8cce7150b683811e315773ce35cb83d2e92 
deleted: sha256:fa46acfe7078d8404ebda3e229ae4058e68aa579512454194f3bc6121be0818e 
untagged: rabbitmq_private:3.8.5-photon_9ff5c8078eac 
untagged: rabbitmq_private:latest
*
*
+ log '/etc/bootstrap/postupdate.d/02-10-load-images.sh starting...'
+ log '/etc/bootstrap/postupdate.d/02-disable-var-log-cron.sh starting...'
+ log '/etc/bootstrap/postupdate.d/03-10-setup-k8s.sh starting...'
+ log '/etc/bootstrap/postupdate.d/10-disable-tzselect.sh starting...'
+ log '/etc/bootstrap/postupdate.d/20-relocate-db.sh starting...'
+ log '/etc/bootstrap/postupdate.d/90-disable-default-services.sh starting...'
+ log '/etc/bootstrap/postupdate.d/91-00-remove-config-xl.sh starting...'
+ log '/etc/bootstrap/postupdate.d/99-10-handover-prelude starting...'
*
*
# Wait for Kubernetes and other prerequisites 
  vracli status first-boot -w 1800 && \ 
    /opt/scripts/upgrade/vami-set-update.sh "success" || \ 
    /opt/scripts/upgrade/vami-set-update.sh "error" 
  rm -f /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh' 
+ chmod a+x /etc/bootstrap/everyboot.d/zz-zz-resume-upgrade.sh 
+ export -f vami_reboot_background 
+ echo 'Scheduling a post-upgrade reboot...' 
Scheduling a post-upgrade reboot... 
+ echo 'Post-upgrade reboot scheduled' 
Post-upgrade reboot scheduled 
+ exit 0 
+ nohup bash -c vami_reboot_background 
+ log '/etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.' 
++ date '+%Y-%m-%d %H:%M:%S' 
+ echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/99-10-handover-prelude done, succeeded.'
*
*
+ log '/etc/bootstrap/postupdate.d/README is not executable.' 
++ date '+%Y-%m-%d %H:%M:%S' 
+ echo '2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable.' 
2021-03-30 07:14:06 /etc/bootstrap/postupdate.d/README is not executable. 
+ log 'main bootstrap postupdate done' 
++ date '+%Y-%m-%d %H:%M:%S' 
+ echo '2021-03-30 07:14:06 main bootstrap postupdate done' 
2021-03-30 07:14:06 main bootstrap postupdate done 
+ '[' postupdate == firstboot ']' 
+ exit 0 
Rebooting the appliance after VAMI upgrade ...


12. Finishes the postupdate and the appliance is rebooted


13. As we can see above the appliance was rebooted sometime after 2021-03-30 07:14:06 , under upgrade-2021-03-30-06-20-48.log once the appliance is back you would see the status change 


[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade in progress. 
[INFO][2021-03-30 07:14:01][labvra.nukescloud.com] VAMI upgrade still in progress. 
[INFO][2021-03-30 07:14:06][labvra.nukescloud.com] Waiting for VAMI to exit ... 
[INFO][2021-03-30 07:14:36][labvra.nukescloud.com] Verifying VAMI overall upgrade result ... 
[INFO][2021-03-30 07:14:36][labvra.nukescloud.com] VAMI upgrade in progress. 
[INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 07:17:01][labvra.nukescloud.com] VAMI upgrade in progress.
*
*
*
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully. 
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully. 
[INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata 
[INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved. 
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller. 
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller. 
[INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum... 
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster. 
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully. 
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes. 
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points.


14. Restore point restoration starts and then application and infrastructure services start as well 


[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI checking upgrade progress 
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed succesfully. 
[INFO][2021-03-30 07:22:01][labvra.nukescloud.com] VAMI upgrade completed successfully. 
[INFO][2021-03-30 07:23:01][labvra.nukescloud.com] Saving artifacts metadata 
[INFO][2021-03-30 07:23:02][labvra.nukescloud.com] Artifacts metadata saved. 
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] Resolving post-upgrade controller. 
[INFO][2021-03-30 07:24:01][labvra.nukescloud.com] This node is elected for post-upgrade controller. 
[INFO][2021-03-30 07:24:03][labvra.nukescloud.com] Resolving post-upgrade nodes quorum... 
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Adding nodes to the cluster. 
[INFO][2021-03-30 07:24:04][labvra.nukescloud.com] Cluster nodes added successfully. 
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Restoring from restore points on all nodes. 
[INFO][2021-03-30 07:24:05][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-restore.sh 'local' sys-config sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 07:24:11][labvra.nukescloud.com] Restoring all restore points.




15. At this point in time one has to check /var/log/deploy.log to see if the services are coming back as expected 


========================= 
[2021-03-30 07:26:36.605+0000] Creating kubernetes namespaces 
=========================
========================= 
[2021-03-30 07:26:39.435+0000] Applying ingress certificate 
=========================
========================= 
[2021-03-30 07:26:50.589+0000] Updating etcd configuration to include https_proxy if such exists 
=========================
========================= 
[2021-03-30 07:26:57.836+0000] Deploying infrastructure services 
=========================
========================= 
[2021-03-30 07:28:43.477+0000] Clearing liquibase locks 
=========================
========================= 
[2021-03-30 07:28:56.296+0000] DB upgrade schema 
=========================
========================= 
[2021-03-30 07:32:08.619+0000] Populating initial identity-service data 
=========================
========================= 
[2021-03-30 07:32:38.488+0000] Deploying application services 
=========================
========================= 
[2021-03-30 07:42:06.787+0000] Deploying application UI 
=========================
========================= 
[2021-03-30 07:43:28.519+0000] Setting feature UI toggles to Provisioning service 
=========================
========================= 
[2021-03-30 07:43:54.731+0000] Registering embedded vRO 
=========================
========================= 
[2021-03-30 07:43:58.698+0000] Registering embedded ABX on-prem endpoint in organizations 
=========================




16. That marks the application and infrastructure start up procedure and then there would be a verification , this procedure is documented under 2 logs


Reference: Verifies if all kubernetes related services are up and running upgrade-k8s-2021-03-30-06-20-48.log
Reference: Copy of deploy.log upgrade-deploy-2021-03-30-06-20-48.log


17. As the verification is now complete, it will now works towards the last few steps 


[INFO][2021-03-30 07:44:32][labvra.nukescloud.com] Infrastrucre and application services deployed successfully. 
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Retrieving services status... 
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Services status retrieval succeeded. 
[INFO][2021-03-30 07:44:33][labvra.nukescloud.com] Processing services data... 
[INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Services data processing succeeded. 
[INFO][2021-03-30 07:44:54][labvra.nukescloud.com] Verifying services... 
[INFO][2021-03-30 07:44:56][labvra.nukescloud.com] Services verification completed successfully. 
[INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Cleaning up restore point  on all nodes. 
[INFO][2021-03-30 07:44:58][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh sys-config at host: labvra.nukescloud.com 
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/sys-config .
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point /data/restorepoint/sys-config cleaned up successfully. 
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Restore point  cleaned up on all nodes. 
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Cleaning up restore point  on all nodes. 
[INFO][2021-03-30 07:45:03][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh artifacts-lastworking at host: labvra.nukescloud.com 
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/artifacts-lastworking .
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point /data/restorepoint/artifacts-lastworking cleaned up successfully. 
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Restore point  cleaned up on all nodes. 
[INFO][2021-03-30 07:45:08][labvra.nukescloud.com] Cleaning up restore point  on all nodes. 
[INFO][2021-03-30 07:45:09][labvra.nukescloud.com] Running remote command: /opt/scripts/upgrade/rstp-clean.sh live-data at host: labvra.nukescloud.com 
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Cleaning up restore point /data/restorepoint/live-data .
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point /data/restorepoint/live-data cleaned up successfully. 
[INFO][2021-03-30 07:45:14][labvra.nukescloud.com] Restore point  cleaned up on all nodes. 
[INFO][2021-03-30 07:45:15][labvra.nukescloud.com] Reverting SSH configuration on nodes. 
[INFO][2021-03-30 07:45:16][labvra.nukescloud.com] SSH configuration reverted on nodes successfully. 
[INFO][2021-03-30 07:45:24][labvra.nukescloud.com] Cleaning up upgrade runtime state. 
[INFO][2021-03-30 07:45:25][labvra.nukescloud.com] Archiving upgrade runtime data. 
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime data archived successfully. 
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Clearing upgrade runtime directory. 
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime directory cleared successfully. 
[INFO][2021-03-30 07:45:26][labvra.nukescloud.com] Upgrade runtime clean up completed.



This concludes vRealize Automation upgrade


As said earlier following logs are the ones which we need to monitor









55 views0 comments

Recent Posts

See All