Onboarding task deleted all virtual machines which were being onboarded ?


NO , Virtual Machines which are / were being onboarded will never be deleted automatically. Onboarding feature in vRealize Automation does not have intelligence or capability to delete the virtual machines on it's own


I was working on one such scenario where users were claiming that the machines which were being onboarded were deleted. Not one or two but hundreds.


This was a good case-study to understand what really happened in the background


Here goes he approach during my investigation and how we found out who did it


 

Ideal Onboarding Procedure



These are the two phases which constitute onboarding procedure 


----------------------------------------------------
PlanExecutionTask
----------------------------------------------------


2022-05-25T06:02:37.382Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/15b6c88f-00eb-41f2-9bf8-631de817f95b, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2


2022-05-25T06:02:37.389Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.


2022-05-25T06:02:37.391Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: OK, new status: EXECUTING.


2022-05-25T06:02:37.489Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Querying for first batch of deployments.


2022-05-25T06:02:37.497Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Getting batch.


2022-05-25T06:02:37.500Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Executing batch.


2022-05-25T06:02:37.508Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.


2022-05-25T06:02:38.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.


2022-05-25T06:02:39.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.


2022-05-25T06:02:39.526Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: EXECUTING, new status: OK.


2022-05-25T06:02:39.536Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Finished.




---------------------------------------------------- 
DeploymentExecutionTask
** This task would begin between Executing batch and Updating Plan status of PlanExecutionTask. During this phase deployment get's onboarded ** 
----------------------------------------------------


2022-05-25T06:02:37.502Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Starting DeploymentExecutionTask. Deployment link: /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2


2022-05-25T06:02:37.511Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Identifying eligibility to onboard the deployment.


2022-05-25T06:02:37.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Deployment /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67 eligible to onboard : true


2022-05-25T06:02:37.516Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Initializing DeploymentExecutionTask.


2022-05-25T06:02:37.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Updating resource placements for project /provisioning/resources/projects/e14d678c-f1c1-4a0d-9359-7c2aadbb3736


2022-05-25T06:02:37.528Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating blueprint.


2022-05-25T06:02:37.532Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating deployment.


2022-05-25T06:02:37.615Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating networks.


2022-05-25T06:02:38.142Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating machines.


2022-05-25T06:02:38.574Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Allocating IP addresses.


2022-05-25T06:02:38.602Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating load balancers.


2022-05-25T06:02:38.608Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating disks.


2022-05-25T06:02:38.616Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating securityGroups.


2022-05-25T06:02:38.620Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating network gateways. 


2022-05-25T06:02:38.624Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Notify deployment onboarded.


2022-05-25T06:02:38.686Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Finished.


 

Investigation


  • Onboarding feature logs it's information under relocation-service logs

  • After we create a deployment plan, when we execute to onboard as discussed before there is a PlanExecutionTask which is triggered

  • Each PlanExecutionTask is associated with an execution id


2022-05-10T01:51:22.198Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/cd323-4e6b-4c0d-b972-3abc123cba] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/5a3d36ba-5df9-4g8c-a665-c7sh3k4kff, execution ID: 72dhrd45-3aa2-4ff9-9941-a4fjhfjfjfja3c 

  • What we saw from the logs was that there were multiple executions but the last one or the attempt failed with an exception


2022-05-10T02:03:40.394Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/4b427637-6121-49cc-a1cc-5fe2e4581ea5, execution ID: bef9fc06-1ecf-4c21-a04f-bc53ed881f48 


2022-05-10T02:03:40.403Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b69bd8e8-b641-4f9a-b44a-0b022ae5055e] Updating plan status. 
2022-05-10T02:03:40.404Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: OK, new status: EXECUTING. 


2022-05-10T02:03:40.484Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055ee] Updating plan status. 


2022-05-10T02:03:40.485Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: EXECUTING, new status: ERROR. 


2022-05-10T02:03:40.492Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-updates-21' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] java.lang.RuntimeException: Validation failed. 


2022-05-10T02:03:40.493Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Failed: RuntimeException: Validation failed.

  • Searching with the deployment id which was deleted from tango-blueprint logs we found out that the resource under the deployment ( virtual machine) was onboard and then deleted



2022-05-10T02:03:10.264Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-4h9wp' thread='tasks-2' user='relocation-8NNrKCAYnu5oyMLf(configadmin)' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' tile='311b69a3-5378-4da2-bc32-a6b870ed305e' trace='0f9d6925-904f-4202-b8c9-dd90a8722fe2'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource Cloud.vSphere.Machine successful. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1
*
*
*
2022-05-10T02:07:09.188Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-kdxqc' thread='tasks-5' user='configadmin' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' request='0cc4024b-bb26-4d3c-a37c-82e31512a2fd' flow='9ff5dc4e-d6f6-4d7a-a826-8390603dc34a' task='f13077fc-103f-4f38-a89c-e60971fc96b0' tile='a5322a42-0cdd-4f9c-86d1-4e3748dea2ac' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource deleted successfully. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1


  • Here comes the question , why would someone delete a deployment once it was onboarded

  • You would find the answer above , when the onboarding execution failed , few of the resources which were selected for this process were partially onboarded

  • Now since every resource is part of same deployment , if one deletes the deployment , all the underlying resources would be deleted.

  • This is the reason hundreds of virtual machines were deleted

  • How do you find out who deleted the deployment is from the below snippet under catalog-service-app logs


2022-05-10T02:04:36.340Z INFO catalog-service-app [host='cata 
log-service-app-***-***' thread='http-nio-8000-exec-18' user='configadmin' org='*****' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for 
deployment 67825eab-4279-4ccd-b100-1fe929623a9d resource null

  • The user configadmin is the one who submitted the action called delete

  • For example , if i submit delete in my lab for a deployment then it would be shown as below


*** user here is catalog-xxxxxx* which is a system/service user. Which states that this resource was deleted by vRA as it was expired *** 

2022-05-27T03:06:21.115Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='catalog-VclHatUt32e13g3w' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='3b0cc70f-8852-475b-975a-43a962d2e7c2'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 59befaa7-876d-480f-9bca-1c64f1c83e87 resource null



*** If it was a user initiated deletion , like me triggering one then it would be like below *** 

2022-05-26T03:59:56.591Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='arun' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='a26fc3d9-7824-41b2-afee-d8d08bf6fd36'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 6eb088c3-d9e6-4580-8e27-d924e93f6681 resource null

 

Note: configadmin is a local account which has privileges of highest level. It's recommended to use individual accounts with certain roles assigned for proper governance



 

There is an enhancement which is being made in the product which would inform user how many resources ( virtual machine , network , disks ...etc... ) would be impacted when he / she deletes the deployment.


This will give them a clue for not to go and directly submit a deletion . This should be out in the next release of vRA 8.8.1



 

43 views0 comments

Recent Posts

See All

vRealize Automation 8.8.1 was released last evening and here's my experience in implementing in my lab. I've attached Upgrade runbook vRA 8.8.1 Deep-Dive.pdf document which contains all of the steps