Workaround instructions to address CVE-2021-44228 in vRA 8.x and vRO 8.x with screenshots |21-Dec-21
Updated: Apr 24, 2022
This blog complies with the new version released on 21st December 2021
All instructions and procedures are taken from VMware KB: https://kb.vmware.com/s/article/87120 , all i am trying to do is to add some screenshots and outputs by implementing this workaround in my lab
Download this Note which has screenshots and detailed snippets
VMware updated KB article 87120 with new commands on 21st December 2021. This blog article complies with it
Both the PDF document attached and the blog screenshots are taken after the new commands are tested
CVE-2021-44228 has been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:
CVE-2021-44228 - VMSA-2021-0028 (link: https://www.vmware.com/security/advisories/VMSA-2021-0028.html) Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of 8.6.2, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. Please subscribe to this article to be informed when updates are published. The workarounds described in this document are meant to be a temporary solution only. Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Long-term resolution will be available in vRA and vRO versions 8.6.2 or later.
CVE-2021-44228 and CVE-2021-45046 have been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:
CVE-2021-44228, CVE-2021-45046 - VMSA-2021-0028
Impact and Risks
Please note the following prior to executing the workaround procedure:
vRA and vRO versions 8.0, 8.0.1 and 8.1 are no longer supported at the time of this article's publication.
Re-apply this KB, If you have previously applied the workaround released prior to 12/20/2021. If not re-applied, upgrading to a future release will not be possible.
This change is persistent over upgrades and the KB should not be re-applied.
This change applies to vRA and vRO (both standalone and embedded).
Can be applied to all vRA and vRO deployments versions 8.1 through 8.6.1.
For clustered setups, in vRO Control Center/Cluster management page, a warning message "Local changes detected" may appear after applying this KB. This message should be disregarded.
Note: Automated vulnerability scanners may report that vRA/vRO products are still vulnerable to CVE-2021-44228 and CVE-2021-45046 after this KB article has been applied. These findings can be safely ignored.
The workarounds described in this document can be considered as permanent solution as they update log4j libraries in the VA to 2.17.0.
Future releases will include log4j 2.17.0 or later
For each vRA and vRO deployments with versions from 8.1 to 8.6.1, execute the following procedure
Take simultaneous VM snapshots without memory of all nodes in the cluster
For this task i would leverage vRSLCM as shown below in the screenshots
Select the product and click on the day-2 actions pane on the product and choose Create Snapshot
I will choose an option which would create snapshot after shutting down the appliances. This is the best option and recommended option too. If you cannot shutdown the production then that is fine , but take snapshots through LCM
Run the Precheck
Ensure precheck is successful
Snapshot task does not take a long time but shutting down the application and bringing it back on takes a little bit of time but it's worth it
Note: This workaround applies to vRA and vRO (both standalone and embedded).
Note: To be applied to all vRA and vRO deployments versions 8.2 through 8.6.1.
SSH login or virtual machine console into one of the nodes in the vRA / vRO cluster.
Ensure all the pods are in running state
Upload <87120-kb-v2.tar.gz> and <87120-kb-v2-validate.tar.gz> under /root on all nodes.
Validate whether the system is vulnerable by running the command below on all nodes. Error reports related to log4j will show up for the affected artifacts that are vulnerable.
cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash -
Complete Output looks like this ...
root@vra [ ~ ]# cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJmY2IxZjQ0YWRkMmNjZTg3MzNjNzAzOTYzNjg4Njk3NjU4ZjA3NTkzMzYyZTRhMzU0MjZlYjc5OGFjMTM5YzRlIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi12YWxpZGF0ZS5zaDsgcm0gLXJmIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLXZhbGlkYXRlLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash - 87120-kb-v2-validate.sh Scanning blueprint-webapp_private:latest at: > Scanning /tmp/patc-verify-container.rvwtc ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar Scanning catalog-service_private:latest at: > Scanning /tmp/patc-verify-container.R7Doo >> Scanning ./opt/service/cs-host.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar Scanning codestream_private:latest at: > Scanning /tmp/patc-verify-container.6WwhN ERROR: found ./opt/codestream/lib/log4j-core-2.13.3.jar Scanning content-service_private:latest at: > Scanning /tmp/patc-verify-container.GovpH ERROR: found ./opt/vmware/lib/log4j-core-2.14.1.jar Scanning identity-service_private:latest at: > Scanning /tmp/patc-verify-container.P5Zyo >> Scanning ./opt/bc-fips/bctls-fips-22.214.171.124.jar >> Scanning ./opt/bc-fips/bcpkix-fips-1.0.5.jar >> Scanning ./opt/bc-fips/bcmail-fips-1.0.3.jar >> Scanning ./opt/bc-fips/bc-fips-126.96.36.199.jar >> Scanning ./jdk/lib/jrt-fs.jar >> Scanning ./identity-service/lib/identity-service-1.5.4-SNAPSHOT.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.14.1.jar Scanning provisioning-service_private:latest at: > Scanning /tmp/patc-verify-container.mYkqN ERROR: found ./admiral/log4j-core-2.13.3.jar Scanning relocation-service_private:latest at: > Scanning /tmp/patc-verify-container.yesgE >> Scanning ./opt/relocation/relocation-service.jar ERROR: found ./BOOT-INF/lib/log4j-core-2.12.1.jar Scanning vco_private:latest at: > Scanning /tmp/patc-verify-container.YSm3Q >> Scanning ./var/opt/apache-tomcat/lib/websocket-api.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-websocket.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-util.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-util-scan.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-jni.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-jdbc.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-zh-CN.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ru.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ko.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-ja.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-fr.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-es.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-i18n-de.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-dbcp.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-coyote.jar >> Scanning ./var/opt/apache-tomcat/lib/tomcat-api.jar >> Scanning ./var/opt/apache-tomcat/lib/servlet-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jsp-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jaspic-api.jar >> Scanning ./var/opt/apache-tomcat/lib/jasper.jar >> Scanning ./var/opt/apache-tomcat/lib/jasper-el.jar >> Scanning ./var/opt/apache-tomcat/lib/el-api.jar >> Scanning ./var/opt/apache-tomcat/lib/ecj-4.6.3.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-tribes.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-storeconfig.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-ha.jar >> Scanning ./var/opt/apache-tomcat/lib/catalina-ant.jar >> Scanning ./var/opt/apache-tomcat/lib/annotations-api.jar >> Scanning ./var/opt/apache-tomcat/bin/tomcat-juli.jar >> Scanning ./var/opt/apache-tomcat/bin/commons-daemon.jar >> Scanning ./var/opt/apache-tomcat/bin/bootstrap.jar >> Scanning ./var/opt/apache-ant/lib/maven-ant-tasks-2.1.3.jar >> Scanning ./var/