Validation of SaltStack endpoint fails when Running Environment "embedded-ABX-onprem" is chosen

Updated: Jun 13


Problem Statement


On a greenfield installation of vRA 8.7 which is integrated with SaltStack Config , we have an option to choose "Running Environment"


This screenshot below explains how a default SaltStack config deployed through vRSLCM

is presented under infrastructure tab



When you enter the password for root , without selecting any option under "Running Environment" , it successfully validates




But, the moment you select "Running Environment" we see an exception where the validation fails




before we see the exception , there is an abx integration run which occurs which gives you more details about the exception




Running in polyglot!

[2022-04-06 13:56:22,183] [INFO] - [saltstack-integration] Validating Salt Stack Config Server credentials...

[2022-04-06 13:56:22,183] [INFO] - [saltstack-integration] Authenticating to a Salt Stack Config Server with url [https://ss.cap.org//account/login]...

[2022-04-06 13:56:22,184] [INFO] - [saltstack-integration] Retrieving credentials from auth credentials link at [/core/auth/credentials/f0c26468-4c1b-4a62-b33a-b04d7c03390e]...

[2022-04-06 13:56:22,304] [INFO] - [saltstack-integration] Successfully retrieved credentials from auth credentials link

[2022-04-06 13:56:22,304] [INFO] - [saltstack-integration] Retrieving Salt Stack Config Server XSRF token from url [https://ss.cap.org//account/login]...

/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

  InsecureRequestWarning,

[2022-04-06 13:56:22,327] [ERROR] - [saltstack-integration] Failed to validate Salt Stack Config Server credentials: Failed to authenticate to a Salt Stack Config Server: Failed to retrieve Salt Stack Config Server XSRF token: 403 Client Error: Forbidden for url: https://ss.cap.org//account/login

Finished running action code.

Exiting python process.

Python process exited.

Max Memory Used: 22 MB





The reason for the exception is as below



Failed to validate Salt Stack Config Server credentials: Failed to authenticate to a Salt Stack Config Server: Failed to retrieve Salt Stack Config Server XSRF token: 403 Client Error: Forbidden for url: https://ss.cap.org//account/login    



exception under provisioning-service-app.log




2022-04-06T16:37:34.118Z WARN provisioning [host='provisioning-service-app-6885766867-kgk4l' thread='reactor-http-epoll-10' user='provisioning-RVgAJFw9LrOYkeUr(arun)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='dc45aa9b-4b4e-47d3-8176-8321b1a2336a' parent='4a10178e-bf5f-48d0-8928-ae7a84e3aff4' span='d1977a94-1448-45a5-b93a-e449c8a76b60'] c.v.xenon.common.ServiceErrorResponse.create:85 - message: Failed to authenticate, please check your credentials or if the host is reachable, statusCode: 400, serverErrorId: 9c245260-075a-4dc0-bbe2-fb13b0e5d0bd: Caused by java.lang.RuntimeException: Failed to authenticate, please check your credentials or if the host is reachable
                at com.vmware.xenon.common.SpringHostUtils.responseEntityToOperation(SpringHostUtils.java:952)
                at com.vmware.xenon.common.SpringHostUtils.lambda$sendRequest$4(SpringHostUtils.java:289)
                at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
                at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
                at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
                at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
                at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:64)
                at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
                at io.opentracing.contrib.reactor.TracedSubscriber.lambda$onNext$2(TracedSubscriber.java:69)
                at io.opentracing.contrib.reactor.TracedSubscriber.withActiveSpan(TracedSubscriber.java:95)
                at io.opentracing.contrib.reactor.TracedSubscriber.onNext(TracedSubscriber.java:69)
                at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
                at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
                at io.opentracing.contrib.reactor.TracedSubscriber.lambda$onNext$2(TracedSubscriber.java:69)
                at io.opentracing.contrib.reactor.TracedSubscriber.withActiveSpan(TracedSubscriber.java:95)
                at io.opentracing.contrib.reactor.TracedSubscriber.onNext(TracedSubscriber.java:69)
                at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
                *
*
*
*
*
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)   at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)


 

Remediation


Method 1 (Greenfeild Scenario )


If you have a SaltStack which was recently deployed and doesn't have any resources mapped to this integration , then simply delete the integration and recreate it


Before




After





See the difference , ensure when your adding the hostname in the integration


When vRSLCM add SaltStack integration , it uses URL ( https://<<saltstackhostname>>/ ) which is when you would see the problem


Once you remove the integration and then add it back again with just FQDN and not the URL of the SaltStack server , then we select an Running environment , it all works fine.





 

Method 2 ( Brownfield Scenario )


When you have resources being managed by SaltStack


Integration information is stored inside provisioning-db of vRealize Automation environment


To login into database use


vracli dev psql

Accept the warning that it's a developer command and ensure you know what you are changing


Below is the screenshot and output of the table where the integration information is stored.


The table is known as endpoint_state , this is present inside provisioning-db


To connect to provisioning-db use the below command

\c provisioning-db




root@vra [ ~ ]# vracli dev psql
This execution will be recorded!
'psql' is a developer command. Type 'yes' if you want to continue, or 'no' to stop: yes
2022-04-06 14:14:43,439 [INFO] Logging into database template1
psql (10.18)
Type "help" for help.

template1=# \c provisioning-db
You are now connected to database "provisioning-db" as user "postgres".
provisioning-db=# \x
Expanded display is on.
provisioning-db=# select * from endpoint_state where name = 'vssc_idm';
-[ RECORD 1 ]-------------------+---------------------------------------------------------------------------------------------------------------------
document_self_link              | /resources/endpoints/b2b02510-b0d5-46cf-9248-570b3d1bd58d
document_auth_principal_link    | /provisioning/auth/csp/users/cgs-lecvl28lpzqwhozt@provisioning-client.local
document_expiration_time_micros | 0
document_owner                  |
document_update_action          | PATCH
document_update_time_micros     | 1646141393852000
document_version                | 1
id                              | 6c2679af-a23c-4c88-8af0-3380305e3cde
name                            | vssc_idm
c_desc                          |
custom_properties               | {"hostName": "https://ss.cap.org/", "isExternal": "true", "privateKeyId": "root"}
tenant_links                    | ["/tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a", "/tenants/project/1f32c781c7bac475-7f703c5265a63d87"]
group_links                     |
tag_links                       |
org_auth_link                   | /tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a
project_auth_link               |
owner_auth_link                 |
msp_auth_link                   |
creation_time_micros            |
region_id                       |
endpoint_links                  |
compute_host_link               | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
expanded_tags                   |
document_creation_time_micros   | 1646141393802000
endpoint_type                   | saltstack
auth_credentials_link           | /core/auth/credentials/eeb389af-fcd6-4b06-a0e9-5d178f128eed
compute_link                    | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
compute_description_link        | /resources/compute-descriptions/8a689d42-24f7-4f6d-b362-e85b6dc6f423
resource_pool_link              | /resources/pools/1f32c781c7bac475-7f703c5265a63d87
parent_link                     |
associated_endpoint_links       |
endpoint_properties             | {"hostName": "https://ss.cap.org/", "privateKeyId": "root"}
maintenance_mode                |
mobility_endpoint_links         |

provisioning-db=#


Look at the custom_properties section , this is how it is out of the box


custom_properties               | {"hostName": "https://ss.cap.org/", "isExternal": "true", "privateKeyId": "root"}

We would add an additional property called dcID and change the hostname to FQDN than a URL and keep endpointId blank.


update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';

Along with it , we would have to change endpoint_properties too. This has to reflect FQDN than the whole url


endpoint_properties             | {"hostName": "https://ss.cap.org/", "privateKeyId": "root"}




Note : Before making changes i'll take a snapshot of vRA appliance




As we already logged into the database before , let's go ahead and make the change. Execute below query and ensure its successful


update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';


update endpoint_state set endpoint_properties = '{"hostName": "ss.cap.org", "privateKeyId": "root"}' where name = 'vssc_idm';








provisioning-db=# update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';


UPDATE 1


provisioning-db=# update endpoint_state set endpoint_properties = ' {"hostName": "ss.cap.org", "privateKeyId": "root"}' where name = 'vssc_idm';

UPDATE 1





provisioning-db=# select * from endpoint_state where name = 'vssc_idm';
-[ RECORD 1 ]-------------------+---------------------------------------------------------------------------------------------------------------------
document_self_link              | /resources/endpoints/b2b02510-b0d5-46cf-9248-570b3d1bd58d
document_auth_principal_link    | /provisioning/auth/csp/users/cgs-lecvl28lpzqwhozt@provisioning-client.local
document_expiration_time_micros | 0
document_owner                  |
document_update_action          | PATCH
document_update_time_micros     | 1646141393852000
document_version                | 1
id                              | 6c2679af-a23c-4c88-8af0-3380305e3cde
name                            | vssc_idm
c_desc                          |
custom_properties               | {"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}
tenant_links                    | ["/tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a", "/tenants/project/1f32c781c7bac475-7f703c5265a63d87"]
group_links                     |
tag_links                       |
org_auth_link                   | /tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a
project_auth_link               |
owner_auth_link                 |
msp_auth_link                   |
creation_time_micros            |
region_id                       |
endpoint_links                  |
compute_host_link               | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
expanded_tags                   |
document_creation_time_micros   | 1646141393802000
endpoint_type                   | saltstack
auth_credentials_link           | /core/auth/credentials/eeb389af-fcd6-4b06-a0e9-5d178f128eed
compute_link                    | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
compute_description_link        | /resources/compute-descriptions/8a689d42-24f7-4f6d-b362-e85b6dc6f423
resource_pool_link              | /resources/pools/1f32c781c7bac475-7f703c5265a63d87
parent_link                     |
associated_endpoint_links       |
endpoint_properties             | {"hostName": "ss.cap.org", "privateKeyId": "root"}
maintenance_mode                |
mobility_endpoint_links         |





As one can see from the above update , we did change the custom_properties of the SSC integration in vRA



Exit the database by executing


\q

Now let's reboot saltstack , log out of vRA and log back in again .


See if the FQDN is back in the hostname rather than the URL. If that's the case it would successfully authenticate with the "Running Environment " in place






Running in polyglot!

[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Validating Salt Stack Config Server credentials...

[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Authenticating to a Salt Stack Config Server with url [https://ss.cap.org/account/login]...

[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Retrieving credentials from auth credentials link at [/core/auth/credentials/d7ea970e-cdca-42bc-b53d-ddac713a8666]...

[2022-04-06 17:13:36,519] [INFO] - [saltstack-integration] Successfully retrieved credentials from auth credentials link

[2022-04-06 17:13:36,519] [INFO] - [saltstack-integration] Retrieving Salt Stack Config Server XSRF token from url [https://ss.cap.org/account/login]...

/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

  InsecureRequestWarning,

[2022-04-06 17:13:36,544] [INFO] - [saltstack-integration] Successfully retrieved Salt Stack Config Server XSRF token

/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

  InsecureRequestWarning,

[2022-04-06 17:13:36,633] [INFO] - [saltstack-integration] Successfully authenticated to a Salt Stack Config Server

[2022-04-06 17:13:36,634] [INFO] - [saltstack-integration] Successfully validated Salt Stack Config Server credentials

Finished running action code.

Exiting python process.

Python process exited.

Max Memory Used: 21 MB

 

In Short


Issue is seen due to the fact there is a URL rather than FQDN and when it's trying to execute an API to authentication it get's a 403 error

  • Unless we fix this issue you will not be able to successfully validate running environment

  • If it's a new environment with no SaltStack resources , go ahead and delete the integration and re-create it

  • If it's an existing integration with resources in place , then modify the database as shown above


1. connect to postgres database
vracli dev psql

2. connect to provisiioning-db
\c provisioning-db

3. enable expanded display
\x

4. Update custom_properties value where the hostname is set to URL of SaltStack node than an FQDN. Remember to change in endpoint_state table as shown below. Sometimes the name of the integration might be different if it's changed from UI. So change it accordingly. 


update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "FQDN-SALTSTACKNODE", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';

5. Update endpoint_properties column value where you have hostname set to URL to FQDN. Almost same as above 

provisioning-db=# update endpoint_state set endpoint_properties = ' {"hostName": "FQDN-SALTSTACKNODE", "privateKeyId": "root"}' where name = 'vssc_idm';



  • Now add "Running Environment" and then validate. You should see a successful validation in place



109 views0 comments