vCenter Server 7.0 HTML5 UI error “no healthy upstream”

Sadaf

After upgrading to vCenter 7 Update 1 , when I tried to browse vCenter HTML5 UI, I faced “no healthy upstream” error. I could access to vCenter Management Interface (VAMI) https://vCenter-IPaddress:5480 without any issues. I could also connect to vCenter Server through  SSH but I realized couple of vCenter Server services could not start.

You can also check the details status of services by connecting to vCenter through SSH and run the following command:

#service-control --list

Then I tried to force to start services by below commands:

#service-control --start --all
#service-control –-start {service-name}

After waiting for a while, I got the underneath error.

After spending couple of hours reading logs and a bit of googling, I have been pointed towards different answers. First of all I went through all DNS, NTP and IP checks and in my case everything was working as it should.

In my scenario, vCenter’s SSL certificate were replaced with a valid signed certificate and it was one of the reason that points me to check certification validity. Beside this SSL certificate, there are couple of other certificates that vCenter server uses. To get familiar with vSphere certificates you can read the following vSphere documentation:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-3AF7757E-A30E-4EEC-8A41-28DA72102520.html

In my case  “Trusted root certificate, Machine SSL Certificate and SMS” were still valid . But ” Machine, vpxd, vpxd-extension and vsphere-webclient” were expired.    

You can check the validity of each certificate by running below commands in vCenter server:

# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | less

Below you can find the expired certificate screen shot:

In this case you need to update the expired certificates with use of vCenter certificate manager through running following command on vCenter CLI.

#/usr/lib/vmware-vmca/bin/certificate-manager

choose number 6 to replace Solution User certificates.

Then you need to answer the required information

  • Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y
  • Please provide valid SSO and VC privileged user credential to perform certificate operations.Enter username [Administrator@vsphere.local]:

Note: this is an example how to address each question you need to fill it out based on your environment.

  • Enter proper value for ‘Country’ [Default value : US] :US
  • Enter proper value for ‘Name’ [Default value : CA] : CA
  • Enter proper value for ‘Organization’ [Default value : VMware] : “ vElements lab”
  • Enter proper value for ‘OrgUnit’ [Default value : VMware Engineering] : VELEMENTSIT
  • Enter proper value for ‘State’ [Default value : California]: California
  • Enter proper value for ‘Locality’ [Default value : Palo Alto] : Palo Alto
  • Enter proper value for ‘IPAddress’ (Provide comma separated values for multiple IP addresses) [optional] : you can press Enter or provide the required information
  • Enter proper value for ‘Email’ [Default value : email@acme.com] : Press Enter
  • Enter proper value for ‘Hostname’ (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified DomainName(FQDN), For Example : example.domain.com] : vc.velements.net
  • Enter proper value for VMCA ‘Name’ : vc.velements.net You are going to regenerate Solution User Certificates using VMCA
  • Continue operation : Option[Y/N] ? : Y

After I successfully updated the certificates , vCenter services got started and I could reach the vCenter UI.

Below you can also find other solutions I found when I was googling  

Suggested answers to check

  • Upgrade VMware Hardware version and choose the correct OS for vCenter

Note: Take a snapshot from vCenter Server VM before hardware version upgrade, as it’s none reversible  task to previous versions.

  • Shutdown the vCenter > right click on the VM > Compatibility > Upgrade VM Compatibility.
  • Right click on the vCenter and choose Edit settings > VM Options > General Options > Select VMware Photon OS
  • Check DNS (you should be able to resolve FQDN names from vCenter)
  • Check NTP (Time should be synced and correct between ESXi hosts and vCenter Server)
  • vCenter Server IP address should be set Static

All of the services which are set to Automatic start are running without any errors or warnings. Hopefully this will help you to solve your issue.

42 thoughts on “vCenter Server 7.0 HTML5 UI error “no healthy upstream””

  3. Thank you! We randomly had a VCSA go offline (7.0 U3c) with a “no healthy upstream” error. It could be a million things. None of the original self-signed certs were expired. BUT, after trying option 6 and having it fail, we did Option 8 to “Reset all certificates”. Everything started up and is running great!

    Reply

    1. Yes, that’s true!
      I had the same issue with one of my lab’s vCenter. I also tried option 8 which worked in my case too!

      Reply

  4. Thank you for this post, Vcentre is now up and running! Turned out I had 4 expired certificates and running through renewing certs has resolved the issue. Cheers again!

    Reply

  8. Thank you so much!

    I did not think about STS cert cuz that was not expired.

    Glad I tried your recommendation and my vcsa 7.0.3 is back!

    Reply

  9. Thank you so much for such a valuable and helpful article. It has saved me a few hours works. It is with details and was well written. Truly a contribution! My salute!

    Reply

  10. Thanks for the article. I had a machine SSL expired.
    I too tried option 6 then option 8.
    I however get to 85% completed and stuck there. Over ten minutes…
    Will continue to wait for 100% or VMware call back…whichever comes first.

    Thanks

    Reply

    1. Hi Dan,

      I’ve faced the same issue. Option 6 will perform rollback if the process fail but option 8 will not perform rollback. In my case, it got stuck at 85% (starting service) for both option but i chose option 8 and manually start all the services. vCenter up and running check all my certs renewed after.

      Reply

  12. There is nothing like resolution without having to open an SR with VMware. The steps were easy to follow, with positive results. Thank you.

    Reply

  14. I got the same problem and after trying multiple solutions, none of them works, accept this one.
    Thank you so much Sadaf!!

    Reply

  18. Great! That did the trick and saved my Sunday (and this Monday) – it’s much easier, if the servers are running Monday morning 😉

    Reply

  20. Thank you Sadaf for great post. I had this kind of problems 3 times and all of them was related to certificates.

    Reply

  23. That helped me , although the option to replace the certificate didn’t do the trick for me
    so i hade to create a new one ( option 8 in the menu)

    after creating new one all worked as expected !!!!

    thanks !

    Reply

  24. I ran into the same issue yesterday. I noticed VPXD wouldn’t start so I restarted
    rhttpproxy, lookupsvc, vmware-vpostgres, then vpxd-svcs. In that order because I don’t think lookup will start if rhttp is funky. Then i started vmware-vpxd. The web GUI started working immediately.
    Hope this helps someone.

    Reply

  25. Hi,
    I just want to thank for sharing this write up. I followed the procedure and got my vCenter 7.0 back up and running.

    Reply

  30. Hi , it works with option 8 although i tired with option 6 didn’t work ! in my case it was Machine certificate expired .

    Thank you.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *