Hi,
After running vCenter 6.5 for around 2 months, I suddenly stopped being able to connect via the web client to it. I tried restarting services, even the entire appliance with no luck. Any help pointing out the issue would be appreciated. I already found another discussion on these forums with the exact same issue that nobody answered so the solution was to move back to 6.0. https://communities.vmware.com/message/2635193
The web interface shows this error:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb0a0004530] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
In the vsphere_client_virgo.log I found the following errors:
[2017-02-03T09:52:32.471Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl Fetching SSO endpoints from CM
[2017-02-03T09:52:32.705Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.security.sso.impl.SsoCmLocatorImpl Retrieved locations of services from CM at https://PRG-LAB-VC01.praguelab.internal/cm/sdk?hostid=f68a7a75-4694-4ec9-bd57-0f2af1cd9746 in 249 milliseconds:
adminAppUrl: https://PRG-LAB-VC01.praguelab.internal/sso-adminserver/sdk/praguelabsso.internal, stsUrl: https://PRG-LAB-VC01.praguelab.internal/sts/STSService/praguelabsso.internal, metadataUrl: https://PRG-LAB-VC01.praguelab.internal/websso/SAML2/Metadata/praguelabsso.internal, vmodlVersion: interface com.vmware.vim.binding.sso.version.version3_5
[2017-02-03T09:52:32.917Z] [INFO ] CEIPSynchronizerTimer com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery Site affinity is disabled
[2017-02-03T09:52:33.118Z] [INFO ] CEIPSynchronizerTimer com.vmware.identity.token.impl.SamlTokenImpl SAML token for SubjectNameId [value=vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746@praguelabsso.internal, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
[2017-02-03T09:52:33.124Z] [INFO ] CEIPSynchronizerTimer com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl Successfully acquired token for user: {Name: vpxd-extension-f68a7a75-4694-4ec9-bd57-0f2af1cd9746, Domain: praguelabsso.internal}
[2017-02-03T09:52:33.224Z] [INFO ] CEIPSynchronizerTimer com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl Initializing vmomi for vc - https://PRG-LAB-VC01.praguelab.internal:443/sdk at VMODL version com.vmware.vim.binding.vim.version.internal.version11
[2017-02-03T09:52:33.288Z] [ERROR] CEIPSynchronizerTimer com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503 com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)
at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)
at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
[2017-02-03T09:52:33.290Z] [ERROR] CEIPSynchronizerTimer com.vmware.ph.ceip.impl.Synchronizer Global consent status synchronizer has stopped unexpectedly. As a result the local consent state will not be synchronized with other consent states. To recover please verify/fix the exact reason in the cause exception and restart the service com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$ServiceInitializationException: Exception occured while locating TargetVCService. As a result session cannot be simulated, since there is no VC to work with. For more details see the underlying exception: Unexpected status code: 503
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:378)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl.simulate(DsSessionSimulatorImpl.java:118)
at com.vmware.ph.ceip.impl.Synchronizer$1.run(Synchronizer.java:71)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:344)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:304)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:178)
at com.sun.proxy.$Proxy615.retrieveContent(Unknown Source)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.initializeVmomi(VcServiceImpl.java:1429)
at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.<init>(VcServiceImpl.java:322)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.createVcService(LinkedVcGroupImpl.java:554)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:187)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:150)
at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl.<init>(LinkedVcGroupImpl.java:125)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider$LinkedVcGroupFactory.create(DsSessionSimulatorImpl.java:627)
at com.vmware.ph.ceip.authentication.DsSessionSimulatorImpl$VcGroupsRegistryProvider.getLinkedVcGroupRegistry(DsSessionSimulatorImpl.java:362)
... 4 common frames omitted