• Arun Nukula

Having Multi-Tenancy enabled in vRO causes vRA Upgrade failures during the pre-update phase

Failures are seen during vRA upgrade through vRSLCM 2.1during preupdate phase of vRA appliance upgrade


Upgrade was being performed from vRA 7.4 to 7.6

vRLCM was on version 2.1 Patch 2


/var/log/bootstrap/preupdate.log


Running a check on replicas:
Executing a script on multiple cluster nodes...
2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-01-va-resources-check done, succeeded.

2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability starting...
Executing a script on multiple cluster nodes...
2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability done, succeeded.

2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates starting...
<!--: spam
Content-Type: text/html

<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><body bgcolor="#f0f0f8">
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#6622aa">
<td valign=bottom>&nbsp;<br>
<font color="#ffffff" face="helvetica, arial">&nbsp;<br><big><big><strong>&lt;type 'exceptions.TypeError'&gt;</strong></big></big></font></td

<td align=right valign=bottom
<font color="#ffffff" face="helvetica, arial">Python 2.7.14: /usr/bin/python<br>Thu May 28 05:34:28 2020</font></td></tr></table>


<p>A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.</p>
<table width="100%" cellspacing=0 cellpadding=0 border=0>
<tr><td bgcolor="#d8bbff"><big>&nbsp;</big><a href="file:///etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates">/etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates</a> in <strong><module></strong>()</td></tr>
<tr><td><font color="#909090"><tt>&nbsp;&nbsp;<small>&nbsp;&nbsp;&nbsp;51</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;dup&nbsp;in&nbsp;duplicates_check_result:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt>&nbsp;&nbsp;<small>&nbsp;&nbsp;&nbsp;52</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dup_line&nbsp;=&nbsp;"{}&nbsp;items&nbsp;in&nbsp;table&nbsp;{}:&nbsp;{}&nbsp;{}".format(str(dup.get('?
column?')),&nbsp;dup.get('type'),<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt>=&gt;<small>&nbsp;&nbsp;&nbsp;53</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;''&nbsp;if&nbsp;len(dup.get('categoryid'))&nbsp;==&nbsp;0&nbsp
;else&nbsp;"ID="&nbsp;+&nbsp;dup.get('categoryid'),<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt>&nbsp;&nbsp;<small>&nbsp;&nbsp;&nbsp;54</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;''&nbsp;if&nbsp;len(dup.get('name'))&nbsp;==&nbsp;0
&nbsp;else&nbsp;"NAME="&nbsp;+&nbsp;dup.get('name'))<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt>&nbsp;&nbsp;<small>&nbsp;&nbsp;&nbsp;55</small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;msg_html&nbsp;+=&nbsp;"&lt;p&nbsp;style='margin-left:20px'&gt;-&nbsp;{}&lt;/p&gt;".format(dup_line)<br
</tt></font></td></tr>
<tr><td><small><font color="#909090"><em>builtin</em> <strong>len</strong>&nbsp;= &lt;built-in function len&gt;, <strong>dup</strong>&nbsp;= {'?column?': 1L, 'type': 'vmo_configelementcategory', 'categoryid': None, 'name': 'wbg_SQL'}, du
p.<strong>get</strong>&nbsp;= &lt;built-in method get of RealDictRow object&gt;</font></small></td></tr></table><p><strong>&lt;type 'exceptions.TypeError'&gt;</strong>: object of type 'NoneType' has no len()
<br><tt><small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small>&nbsp;</tt>args&nbsp;=
("object of type 'NoneType' has no len()",)
<br><tt><small>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</small>&nbsp;</tt>message&nbsp;=
"object of type 'NoneType' has no len()"

<!-- The above is a description of an error in a Python program, formatted
     for a Web browser because the 'cgitb' module was enabled.  In case you
     are not reading this in a Web browser, here is the original traceback:

Traceback (most recent call last):

/var/log/messages




2020-05-28T05:34:29.194212+00:00 sevenfour kernel: [ 5275.524647] martian source 10.63.249.208 from 0.0.0.0, on dev eth0
2020-05-28T05:34:29.194252+00:00 sevenfour kernel: [ 5275.524657] ll header: ff:ff:ff:ff:ff:ff:00:00:00:00:7b:00:08:00
2020-05-28T05:34:31.588050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info .................................................................................................................................................
.............................................................................................................................................................................................................................................
...................................................................
2020-05-28T05:34:31.588183+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Failure: Error while running pre-install scripts and post-install scripts
2020-05-28T05:34:31.588194+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info
2020-05-28T05:34:31.603338+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>VA-check: finished</p>
2020-05-28T05:34:31.603561+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Pre-install: failed (code p-1)</p>
2020-05-28T05:34:31.603870+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>RPM Status 1: <span style='color: red'>Pre install script failed, package test and installation skipped</span>.</p>
2020-05-28T05:34:31.604050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Post-install: failed</p>
2020-05-28T05:34:31.604227+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Update failed (code 1-1). Check logs in /opt/vmware/var/log/vami or retry update later.</p>
2020-05-28T05:34:31.604354+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <br />
2020-05-28T05:34:31.607027+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Finished..
2020-05-28T05:34:31.607088+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...
2020-05-28T05:34:35.725031+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Calling cluster-command-get-next
2020-05-28T05:34:35.725081+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...


Executing KB 54987 shows that there is no database configuration update needed. Which means there are actually no duplicates


sevenfour:/tmp # /usr/lib/vco-cli/bin/vro-configure.sh db-validate
Orchestrator's root folder: /var/lib/vco
Orchestrator Configuration Tool. Version: 7.4.0.8989539 Build: 8989539
ValidateDbOptions{dbUsername='vcac', dbPassword='****', jdbcUrl='jdbc:postgresql://127.0.0.1:5433/vcac?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt'}
The command does not need a database configuration update.

Even then the precheck fails with the exception stating there are duplicates. This is because of Multi-Tenancy enabled in vRO


The duplicates in vRO are coming from the fact that they are using a multi-tenanted environment and vro duplicates script is not designed for that.


Remediation Plan

This is how you remediate this problem so that a successful upgrade can be performed


Step:1

Extract the attached archive 00-00-00-00-remove-check-vro-duplicates.zip into the VA you are going to upgrade In the folder: /etc/bootstrap/preupdate.d


00-00-00-00-remove-check-vro-duplicates
Download • 785B



Step:2

Execute below command to change permissions


chmod 744 /etc/bootstrap/preupdate.d/00-00-00-00-remove-check-vro-duplicates


Then start the upgrade either through vRLCM or through VAMI.


This should help in progressing upgrade further without any failures.


What is this additional script doing?


This script 00-00-00-00-remove-check-vro-duplicates deletes 00-00-02-check-vro-duplicates

when triggered


#!/bin/sh
rm -f /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates

Ideally /etc/bootstrap/preupdate.d/* contains all the scripts which has to run during preupdate phase of vRA upgrade



[master] sevenvra:/etc/bootstrap/preupdate.d # ls -l
total 160

-rwxr--r-- 1 root root  2284 Mar 28  2019 00-00-00-prereq-checks
-rwxr--r-- 1 root root   781 Mar 28  2019 00-00-00-wizard-completed
-rwxr--r-- 1 root root   655 Mar 28  2019 00-00-00-z-disable-db-failover
-rwxr--r-- 1 root root  1883 Mar 28  2019 00-00-01-abort-on-replica
-rwxr--r-- 1 root root   853 Mar 28  2019 00-00-01-check-replica-availability
-rwxr--r-- 1 root root   918 Mar 28  2019 00-00-01-copy-repository
-rwxr--r-- 1 root root  1377 Mar 28  2019 00-00-01-va-resources-check
-rwxr--r-- 1 root root  3685 Mar 28  2019 00-00-02-check-vro-duplicates
-rwxr--r-- 1 root root  2598 Mar 28  2019 00-00-03-upgrade-management-agents
-rwxr--r-- 1 root root 12433 Mar 28  2019 00-00-04-prereq-iaas
-rwxr--r-- 1 root root   724 Mar 28  2019 00-02-generate-vra-cluster-node-id
-rwxr--r-- 1 root root  1815 Mar 28  2019 00-10-vrb-version-check.sh
-rwxr--r-- 1 root root  1563 Mar 28  2019 00-20-check-blueprints
-rwxr--r-- 1 root root   600 Mar 28  2019 00-20-map-lb-to-localhost
-rwxr--r-- 1 root root   149 Mar 28  2019 00-50-kill-all-java-processes
-rwxr--r-- 1 root root   472 Mar 28  2019 00-60-clean-resources
-rwxr--r-- 1 root root  1685 Mar 28  2019 00-70-va-resources-disk-resize
-rwxr--r-- 1 root root   298 Mar 28  2019 00-71-PR-1590946-ext-partition-fix
-rwxr--r-- 1 root root  2723 Mar 28  2019 00-71-va-root-disk-resize
-rwxr--r-- 1 root root   352 Mar 28  2019 05-stop-vrhb-service
-rwxr--r-- 1 root root  1488 Mar 28  2019 06-stop-iaas-services
-rwxr--r-- 1 root root  3347 Mar 28  2019 06-stop-vra-replica-services
-rwxr--r-- 1 root root   698 Mar 28  2019 07-vacuum-db
-rwxr--r-- 1 root root  2114 Mar 28  2019 08-vco-database
-rwxr--r-- 1 root root  1000 Mar 28  2019 09-60-package-pool
-rwxr--r-- 1 root root   819 Mar 28  2019 09-70-remove-artifactory
-rwx------ 1 root root  5152 Mar 21  2019 10-postgres
-rwx------ 1 root root  1297 Mar 21  2019 10-prepare-vmstudio-update
-rwxr--r-- 1 root root  1547 Mar 28  2019 14-remove-persistent-net-rules
-rwxr--r-- 1 root root   529 Mar 28  2019 20-artifactory-uninstall-fix
-rwxr--r-- 1 root root   253 Mar 28  2019 80-horizon-certs
-rwxr--r-- 1 root root    57 Mar 28  2019 90-remove-resource-bundles
-rwxr--r-- 1 root root   750 Mar 28  2019 98-psql-manager-stop
-rwx------ 1 root root  1291 Mar 21  2019 99-backup-config-files
-rwx------ 1 root root   602 Mar 21  2019 99-postgres
-rwx------ 1 root root   555 Mar 21  2019 99-prepare-services

So when an upgrade is executed after vRA rpm's are copied these scripts are downloaded under preupdate/ path


They are executed sequentially


All we are trying to do is a workaround for an upgrade not to check vro duplicates while running an upgrade resulting in a failure. Hence we are deleting 00-00-02-check-vro-duplicates by inducing another script which will be executed before anything else is executed. As we named it as 00-00-00-00-remove-check-vro-duplicates



38 views

Recent Posts

See All

Subscribe Now

  • Twitter
  • Facebook Social Icon

Copyright © 2019 nukescloud