Upgrading vRealize Operations using vRSLCM ......

Updated: 4 days ago

Attaching a pdf which has same information



vRealize Operations Upgrade Runbook
.pdf
Download PDF • 1.78MB

precheck-report-vrops
.pdf
Download PDF • 29KB

 

Author

ARUN NUKULA


 

Environment


vROps
Source Version: 8.6.2
Target Version : 8.6.4


vRSLCM 8.8.2 PSPACK 2


Note: This blog does not contain any customer facing data. It's performed in a lab environment whic has been destroyed
 

Procedure

  • These are the different environments in my vRSLCM. Upgrade would be performed on tvrops environment



  • It's a single node vROps instance






  • Took a snapshot before beginning any sort of a task on the product

  • Logging into the product , i have three integrations all of those adapters are in functional state


vCenter 8.x 
LogInsight
NSXT



  • Logged into vRSLCM , then performed inventory sync

  • Once that's successful , go ahead and then click on upgrade

  • Then we would be prompted with the repository url



  • There would be an APUAT test performed. Once the APUAT test is completed , we can download the report or view it by clicking on view report


  • When we click on download report , it would download a zip file with following contents



  • Opt for a snapshot even though i've taken one just before the upgrade manually


  • You may open index.html using browser which would show you the report


  • Its pretty much a new system so there is nothing really in it. Hence the report is blank


  • Then the services check would happen


  • Once prechecks are done , you can download the report if one want's to use it as an articfact for upgrade and then click on next


  • Validate all inputs for uprgade and click on submit to trigger uprgade


  • The moment we click on submit upgrade is iniitiated


  • There are 10 stages for this upgrade ( stages will vary if it is a cluster )


Stage 1: Clusteronlineofflineconfiguration

1. Start
2. Configure Cluster Init
3. Configure Cluster Online Offline State
4. Check Cluster Online Offline State
5. Final


Stage 2:  Shutting down Guest OS

1. Start
2. Shutdown Guest OS
3. Final


Stage 3: Creating Node Snapshot

1. Start
2. Get vMoid of the virtual machine using name
3. Virtual machine using vMoid
4. Final


Stage: 4 Power On VM

1. Start
2. Get vMoid of the virtual machine using name
3. Power On Virtual Machine
4. Final


Stage: 5 vRealize Operations Manager Service Check

1. Start
2. vRealize Operations Manager Services Status Check task
3. Final


Stage: 6 Clusteronlineofflineconfiguration


1. Start 
2. Configure Cluster Init 
3. Configure Cluster Online Offline State 
4. Check Cluster Online Offline State 
5. Final


Stage: 7 Create Snapshot Inventory

1. Start
2. Product Snapshot Inventory create 
3. On Environment Snapshot Complete


Stage: 8 vROps Upgrade

1. Start 
2. Start Upgrade
3. Find healthy adapters before upgrade
4. Install vROps uprgade pak
5. Check cluster status
6. vRealize Operations Manager SDDC Manager Watermark
7. Final


Stage: 9 Delete Node Snapshot
1. Start
2. Get VMID delete snapshot
3. Delete Node Snapshot
4. Final

Stage: 10 Product Upgrade Inventory update

1. Start
2. Product Upgrade Inventory Upgrade
3. Final




  • After all of the above stages vROps upgrade is now complete





 

What happens during Stage 8 : vROps Upgrade


  • vROps Upgrade is started



2022-09-16 05:29:57.733 INFO  [scheduling-1] c.v.v.l.a.c.EventProcessor -  -- State to find :: com.vmware.vrealize.lcm.plugin.core.vrops.tasks.StartUpgradeTask

  • Deepdive into various phases of the upgrade



1. FindHealthyAdapterTask is triggered 


2022-09-16 05:29:57.733 INFO  [scheduling-1] c.v.v.l.a.c.EventProcessor -  -- State to find :: com.vmware.vrealize.lcm.plugin.core.vrops.tasks.FindHealthyAdapterTask
2022-09-16 05:29:57.755 INFO  [scheduling-1] c.v.v.l.a.c.EventProcessor -  -- Invoking Task :: com.vmware.vrealize.lcm.plugin.core.vrops.tasks.FindHealthyAdapterTask




2.  Adapter response is logged 


2022-09-16 05:30:00.976 INFO  [pool-3-thread-39] c.v.v.l.p.c.v.t.FindHealthyAdapterTask -  -- adapters response message {"adapterInstancesInfoDto":[{"resourceKey":{"name":"nsxt","adapterKindKey":"NSXTAdapter","resourceKindKey":"NSXTAdapterInstance","resourceIdentifiers":[{"identifierType":{"name":"AUTO_DISCOVERY","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"NSXTHOST","dataType":"STRING","isPartOfUniqueness":true},"value":"nsxtm.cap.org"},{"identifierType":{"name":"UNMONITORED_SERVICES","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"vmc_config_limits_file_name","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"VMEntityVCID","dataType":"STRING","isPartOfUniqueness":false},"value":""}]},"description":"","collectorId":1,"collectorGroupId":"9659e5d0-d36a-45c0-b7c0-11ef73d04363","credentialInstanceId":"35621c3f-bc32-4e26-95d4-98c24e3274e9","monitoringInterval":5,"numberOfMetricsCollected":1648,"numberOfResourcesCollected":110,"lastHeartbeat":1663306196138,"lastCollected":1663306155060,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/35621c3f-bc32-4e26-95d4-98c24e3274e9","rel":"RELATED","name":"linkToCredential"}],"id":"2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9"},{"resourceKey":{"name":"vRealizeOpsMgrAPI (vRealize Operations Manager Collector-tvrops)","adapterKindKey":"vRealizeOpsMgrAPI","resourceKindKey":"vRealizeOpsMgrAPI Adapter Instance","resourceIdentifiers":[]},"collectorId":1,"monitoringInterval":1,"numberOfMetricsCollected":0,"numberOfResourcesCollected":0,"lastHeartbeat":1663306196138,"lastCollected":1663306182865,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/c7fd85d1-1537-4513-a63d-9b443913bee5","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/","rel":"RELATED","name":"linkToCredential"}],"id":"c7fd85d1-1537-4513-a63d-9b443913bee5"},{"resourceKey":{"name":"vc8","adapterKindKey":"VMWARE","resourceKindKey":"VMwareAdapter Instance","resourceIdentifiers":[{"identifierType":{"name":"AUTODISCOVERY","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"CLOUD_TYPE","dataType":"STRING","isPartOfUniqueness":false},"value":"PRIVATE_CLOUD"},{"identifierType":{"name":"DV_PORT_GROUP_DISABLED","dataType":"STRING","isPartOfUniqueness":false},"value":"false"},{"identifierType":{"name":"DVS_DISABLED","dataType":"STRING","isPartOfUniqueness":false},"value":"false"},{"identifierType":{"name":"ENABLE_ACTIONS","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"GFS_SKIP_PATTERNS","dataType":"STRING","isPartOfUniqueness":false},"value":"docker/devicemapper/mnt"},{"identifierType":{"name":"IS_REAL_TIME_MONITORING_ENABLED","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"PDRS_STATS_PROVIDER","dataType":"STRING","isPartOfUniqueness":false},"value":"false"},{"identifierType":{"name":"PROCESSCHANGEEVENTS","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"VCURL","dataType":"STRING","isPartOfUniqueness":true},"value":"vc8.cap.org"},{"identifierType":{"name":"VM_FOLDER_DISABLED","dataType":"STRING","isPartOfUniqueness":false},"value":"false"},{"identifierType":{"name":"VM_LIMIT","dataType":"STRING","isPartOfUniqueness":false},"value":"2000000000"},{"identifierType":{"name":"VMC_CONFIG_LIMITS_FILE_NAME","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"VMEntityVCID","dataType":"STRING","isPartOfUniqueness":false},"value":"0d0a4688-0836-448b-a97d-0910ba336f61"}]},"description":"","collectorId":1,"collectorGroupId":"9659e5d0-d36a-45c0-b7c0-11ef73d04363","credentialInstanceId":"0f88a896-0de5-4225-b268-4d736e1aabda","monitoringInterval":5,"numberOfMetricsCollected":64,"numberOfResourcesCollected":4,"lastHeartbeat":1663306196138,"lastCollected":1663306126082,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/78b711a1-4f09-4080-aaab-81d9a12a0c72","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/0f88a896-0de5-4225-b268-4d736e1aabda","rel":"RELATED","name":"linkToCredential"}],"id":"78b711a1-4f09-4080-aaab-81d9a12a0c72"},{"resourceKey":{"name":"Container","adapterKindKey":"Container","resourceKindKey":"ContainerAdapterInstance","resourceIdentifiers":[{"identifierType":{"name":"ClusterId","dataType":"STRING","isPartOfUniqueness":true},"value":"tvrops"}]},"collectorId":1,"collectorGroupId":"9659e5d0-d36a-45c0-b7c0-11ef73d04363","monitoringInterval":1,"numberOfMetricsCollected":0,"numberOfResourcesCollected":0,"lastHeartbeat":1663306196138,"lastCollected":1663306182864,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/bbb977cd-648b-49fa-84f3-02a720b1f780","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/","rel":"RELATED","name":"linkToCredential"}],"id":"bbb977cd-648b-49fa-84f3-02a720b1f780"},{"resourceKey":{"name":"li","adapterKindKey":"LogInsightAdapter","resourceKindKey":"LoginSight Adapter Instance","resourceIdentifiers":[{"identifierType":{"name":"IS_SINGLETON","dataType":"STRING","isPartOfUniqueness":true},"value":""},{"identifierType":{"name":"LOGINSIGHTURL","dataType":"STRING","isPartOfUniqueness":false},"value":"li.cap.org"}]},"description":"","collectorId":1,"collectorGroupId":"9659e5d0-d36a-45c0-b7c0-11ef73d04363","monitoringInterval":5,"numberOfMetricsCollected":0,"numberOfResourcesCollected":0,"lastHeartbeat":1663306196138,"lastCollected":1663306122860,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/fc977e44-9ab1-4b30-badb-031878cef5c6","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/","rel":"RELATED","name":"linkToCredential"}],"id":"fc977e44-9ab1-4b30-badb-031878cef5c6"},{"resourceKey":{"name":"vRealize Operations Manager Adapter - tvrops","adapterKindKey":"vCenter Operations Adapter","resourceKindKey":"vCenter Operations Adapter Instance","resourceIdentifiers":[{"identifierType":{"name":"ClusterId","dataType":"STRING","isPartOfUniqueness":true},"value":"tvrops"},{"identifierType":{"name":"COLLECTORID","dataType":"STRING","isPartOfUniqueness":false},"value":"1"},{"identifierType":{"name":"NODEID","dataType":"STRING","isPartOfUniqueness":true},"value":"24127062-16cc-48cb-81de-1249e3c8a722"}]},"collectorId":1,"monitoringInterval":5,"numberOfMetricsCollected":9940,"numberOfResourcesCollected":11,"lastHeartbeat":1663306196138,"lastCollected":1663306133374,"messageFromAdapterInstance":"","links":[{"href":"/suite-api/api/adapters/5a1812aa-c575-4aa2-9074-4a313572ef79","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/credentials/","rel":"RELATED","name":"linkToCredential"}],"id":"5a1812aa-c575-4aa2-9074-4a313572ef79"}]}




3. Formatting above json we get following list of adapters


NSXTM 
vc8
vRealizeOpsMgrAPI
Container
li
vRealize Operations Manager Adapter




4. Fetches the id's


2022-09-16 05:30:00.983 INFO  [pool-3-thread-39] c.v.v.l.p.c.v.t.FindHealthyAdapterTask -  -- adapterIds : [2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9, c7fd85d1-1537-4513-a63d-9b443913bee5, 78b711a1-4f09-4080-aaab-81d9a12a0c72, bbb977cd-648b-49fa-84f3-02a720b1f780, fc977e44-9ab1-4b30-badb-031878cef5c6, 5a1812aa-c575-4aa2-9074-4a313572ef79]


5. Goes and connects to each adapter to check its status


*******************************************************
### nsxt adapter response as a whole as an example ### 
*******************************************************


2022-09-16 05:30:00.984 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- url to connect https://tvrops.cap.org/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9
2022-09-16 05:30:00.985 INFO  [pool-3-thread-39] c.v.v.l.u.SSLUtil -  -- Disabled SNI for SSL socket.
2022-09-16 05:30:01.061 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- GET /suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9: 200 200
2022-09-16 05:30:01.061 INFO  [pool-3-thread-39] c.v.v.l.p.c.v.t.FindHealthyAdapterTask -  -- adapter resource response :: Response{statusCode=200, responseMessage='200', responseData='{"description":"","creationTime":1663301784638,"resourceKey":{"name":"nsxt","adapterKindKey":"NSXTAdapter","resourceKindKey":"NSXTAdapterInstance","resourceIdentifiers":[{"identifierType":{"name":"AUTO_DISCOVERY","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"NSXTHOST","dataType":"STRING","isPartOfUniqueness":true},"value":"nsxtm.cap.org"},{"identifierType":{"name":"UNMONITORED_SERVICES","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"vmc_config_limits_file_name","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"VMEntityVCID","dataType":"STRING","isPartOfUniqueness":false},"value":""}]},"credentialInstanceId":"35621c3f-bc32-4e26-95d4-98c24e3274e9","resourceStatusStates":[{"adapterInstanceId":"2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","resourceStatus":"DATA_RECEIVING","resourceState":"STARTED","statusMessage":""}],"resourceHealth":"GREEN","resourceHealthValue":100.0,"dtEnabled":true,"monitoringInterval":5,"badges":[{"type":"COMPLIANCE","color":"GREY","score":-1.0},{"type":"EFFICIENCY","color":"GREEN","score":100.0},{"type":"RISK","color":"GREEN","score":0.0},{"type":"HEALTH","color":"GREEN","score":100.0}],"relatedResources":[],"links":[{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/relationships","rel":"RELATED","name":"relationsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/properties","rel":"RELATED","name":"propertiesOfResource"},{"href":"/suite-api/api/alerts?resourceId=2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"RELATED","name":"alertsOfResource"},{"href":"/suite-api/api/symptoms?resourceId=2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"RELATED","name":"symptomsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/statkeys","rel":"RELATED","name":"statKeysOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/stats/latest","rel":"RELATED","name":"latestStatsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/properties","rel":"RELATED","name":"latestPropertiesOfResource"},{"href":"/suite-api/api/credentials/35621c3f-bc32-4e26-95d4-98c24e3274e9","rel":"RELATED","name":"credentialsOfResource"}],"identifier":"2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9"}', responseDataBytes=[123, 34, ******  34, 125], headers={Transfer-Encoding=[chunked], Keep-Alive=[timeout=15, max=100], null=[HTTP/1.1 200 200], Server=[Apache], Access-Control-Allow-Origin=[*], X-Request-ID=[DQ4uW0p6ZuEfKZdR02H9FCSyztsyRT0p], X-Content-Type-Options=[nosniff], Connection=[Keep-Alive], Pragma=[no-cache], Date=[Fri, 16 Sep 2022 05:30:00 GMT], X-Frame-Options=[SAMEORIGIN], Strict-Transport-Security=[max-age=31536000; includeSubDomains], Cache-Control=[no-cache, no-store, max-age=0, must-revalidate], Content-Security-Policy=[default-src https: wss: data: 'unsafe-inline' 'unsafe-eval'; child-src *; worker-src 'self' blob:], Vary=[User-Agent], Expires=[0], X-XSS-Protection=[1; mode=block], Content-Type=[application/json;charset=UTF-8]}}
2022-09-16 05:30:01.074 INFO  [pool-3-thread-39] c.v.v.l.p.c.v.t.FindHealthyAdapterTask -  -- adapter resource response :: {"description":"","creationTime":1663301784638,"resourceKey":{"name":"nsxt","adapterKindKey":"NSXTAdapter","resourceKindKey":"NSXTAdapterInstance","resourceIdentifiers":[{"identifierType":{"name":"AUTO_DISCOVERY","dataType":"STRING","isPartOfUniqueness":false},"value":"true"},{"identifierType":{"name":"NSXTHOST","dataType":"STRING","isPartOfUniqueness":true},"value":"nsxtm.cap.org"},{"identifierType":{"name":"UNMONITORED_SERVICES","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"vmc_config_limits_file_name","dataType":"STRING","isPartOfUniqueness":false},"value":""},{"identifierType":{"name":"VMEntityVCID","dataType":"STRING","isPartOfUniqueness":false},"value":""}]},"credentialInstanceId":"35621c3f-bc32-4e26-95d4-98c24e3274e9","resourceStatusStates":[{"adapterInstanceId":"2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","resourceStatus":"DATA_RECEIVING","resourceState":"STARTED","statusMessage":""}],"resourceHealth":"GREEN","resourceHealthValue":100.0,"dtEnabled":true,"monitoringInterval":5,"badges":[{"type":"COMPLIANCE","color":"GREY","score":-1.0},{"type":"EFFICIENCY","color":"GREEN","score":100.0},{"type":"RISK","color":"GREEN","score":0.0},{"type":"HEALTH","color":"GREEN","score":100.0}],"relatedResources":[],"links":[{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"SELF","name":"linkToSelf"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/relationships","rel":"RELATED","name":"relationsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/properties","rel":"RELATED","name":"propertiesOfResource"},{"href":"/suite-api/api/alerts?resourceId=2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"RELATED","name":"alertsOfResource"},{"href":"/suite-api/api/symptoms?resourceId=2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9","rel":"RELATED","name":"symptomsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/statkeys","rel":"RELATED","name":"statKeysOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/stats/latest","rel":"RELATED","name":"latestStatsOfResource"},{"href":"/suite-api/api/resources/2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9/properties","rel":"RELATED","name":"latestPropertiesOfResource"},{"href":"/suite-api/api/credentials/35621c3f-bc32-4e26-95d4-98c24e3274e9","rel":"RELATED","name":"credentialsOfResource"}],"identifier":"2dbdcd2c-bd1d-47be-9092-8ff59a7e03c9"}


6. In the same way it connects to all other adapters too 


2022-09-16 05:30:01.079 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- url to connect https://tvrops.cap.org/suite-api/api/resources/c7fd85d1-1537-4513-a63d-9b443913bee5
2022-09-16 05:30:01.079 INFO  [pool-3-thread-39] c.v.v.l.u.SSLUtil -  -- Disabled SNI for SSL socket.
2022-09-16 05:30:01.120 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- GET /suite-api/api/resources/c7fd85d1-1537-4513-a63d-9b443913bee5: 200 200


2022-09-16 05:30:01.136 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- url to connect https://tvrops.cap.org/suite-api/api/resources/78b711a1-4f09-4080-aaab-81d9a12a0c72
2022-09-16 05:30:01.137 INFO  [pool-3-thread-39] c.v.v.l.u.SSLUtil -  -- Disabled SNI for SSL socket.
2022-09-16 05:30:01.209 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- GET /suite-api/api/resources/78b711a1-4f09-4080-aaab-81d9a12a0c72: 200 200




2022-09-16 05:30:01.237 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- url to connect https://tvrops.cap.org/suite-api/api/resources/bbb977cd-648b-49fa-84f3-02a720b1f780
2022-09-16 05:30:01.237 INFO  [pool-3-thread-39] c.v.v.l.u.SSLUtil -  -- Disabled SNI for SSL socket.
2022-09-16 05:30:01.274 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- GET /suite-api/api/resources/bbb977cd-648b-49fa-84f3-02a720b1f780: 200 200




2022-09-16 05:30:01.293 INFO  [pool-3-thread-39] c.v.v.l.d.v.VropsConnector -  -- url to connect https://tvrops.cap.org/suite-api/api/resources/fc977e44-9ab1-4b30-badb-031878cef5c6
2022-09-16 05:30:01.295 INFO  [pool-3-thread-39] c.v.v.l.u.SSLUtil -  -- Disabled SNI for SSL socket.
2022-09-16 05:30:01.342 INFO  [pool-3-thread