top of page

Onboarding task deleted all virtual machines which were being onboarded ?


NO , Virtual Machines which are / were being onboarded will never be deleted automatically. Onboarding feature in vRealize Automation does not have intelligence or capability to delete the virtual machines on it's own


I was working on one such scenario where users were claiming that the machines which were being onboarded were deleted. Not one or two but hundreds.


This was a good case-study to understand what really happened in the background


Here goes he approach during my investigation and how we found out who did it


 

Ideal Onboarding Procedure



These are the two phases which constitute onboarding procedure 


----------------------------------------------------
PlanExecutionTask
----------------------------------------------------


2022-05-25T06:02:37.382Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/15b6c88f-00eb-41f2-9bf8-631de817f95b, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2


2022-05-25T06:02:37.389Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.


2022-05-25T06:02:37.391Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: OK, new status: EXECUTING.


2022-05-25T06:02:37.489Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Querying for first batch of deployments.


2022-05-25T06:02:37.497Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Getting batch.


2022-05-25T06:02:37.500Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Executing batch.


2022-05-25T06:02:37.508Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.


2022-05-25T06:02:38.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.


2022-05-25T06:02:39.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.


2022-05-25T06:02:39.526Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: EXECUTING, new status: OK.


2022-05-25T06:02:39.536Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Finished.




---------------------------------------------------- 
DeploymentExecutionTask
** This task would begin between Executing batch and Updating Plan status of PlanExecutionTask. During this phase deployment get's onboarded ** 
----------------------------------------------------


2022-05-25T06:02:37.502Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Starting DeploymentExecutionTask. Deployment link: /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2


2022-05-25T06:02:37.511Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Identifying eligibility to onboard the deployment.


2022-05-25T06:02:37.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Deployment /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67 eligible to onboard : true


2022-05-25T06:02:37.516Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Initializing DeploymentExecutionTask.


2022-05-25T06:02:37.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Updating resource placements for project /provisioning/resources/projects/e14d678c-f1c1-4a0d-9359-7c2aadbb3736


2022-05-25T06:02:37.528Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating blueprint.


2022-05-25T06:02:37.532Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating deployment.


2022-05-25T06:02:37.615Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating networks.


2022-05-25T06:02:38.142Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating machines.


2022-05-25T06:02:38.574Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Allocating IP addresses.


2022-05-25T06:02:38.602Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating load balancers.


2022-05-25T06:02:38.608Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating disks.


2022-05-25T06:02:38.616Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating securityGroups.


2022-05-25T06:02:38.620Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating network gateways. 


2022-05-25T06:02:38.624Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Notify deployment onboarded.


2022-05-25T06:02:38.686Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Finished.


 

Investigation


  • Onboarding feature logs it's information under relocation-service logs

  • After we create a deployment plan, when we execute to onboard as discussed before there is a PlanExecutionTask which is triggered

  • Each PlanExecutionTask is associated with an execution id


2022-05-10T01:51:22.198Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/cd323-4e6b-4c0d-b972-3abc123cba] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/5a3d36ba-5df9-4g8c-a665-c7sh3k4kff, execution ID: 72dhrd45-3aa2-4ff9-9941-a4fjhfjfjfja3c

  • What we saw from the logs was that there were multiple executions but the last one or the attempt failed with an exception


2022-05-10T02:03:40.394Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/4b427637-6121-49cc-a1cc-5fe2e4581ea5, execution ID: bef9fc06-1ecf-4c21-a04f-bc53ed881f48 


2022-05-10T02:03:40.403Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b69bd8e8-b641-4f9a-b44a-0b022ae5055e] Updating plan status. 
2022-05-10T02:03:40.404Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: OK, new status: EXECUTING. 


2022-05-10T02:03:40.484Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055ee] Updating plan status. 


2022-05-10T02:03:40.485Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: EXECUTING, new status: ERROR. 


2022-05-10T02:03:40.492Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-updates-21' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] java.lang.RuntimeException: Validation failed. 


2022-05-10T02:03:40.493Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Failed: RuntimeException: Validation failed.

  • Searching with the deployment id which was deleted from tango-blueprint logs we found out that the resource under the deployment ( virtual machine) was onboard and then deleted 



2022-05-10T02:03:10.264Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-4h9wp' thread='tasks-2' user='relocation-8NNrKCAYnu5oyMLf(configadmin)' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' tile='311b69a3-5378-4da2-bc32-a6b870ed305e' trace='0f9d6925-904f-4202-b8c9-dd90a8722fe2'] com.vmware.tango.blueprint.gateway.DeploymentGateway - De