VMware released the latest revision of vRealize Automation last week. I found some time to perform an upgrade to my homelab environment. At the time, 7.3.0 was the running version. I planned to skip past 7.3.1 and go directly to 7.4. I downloaded the vRA 7.4 ISO file, attached it to the appliance’s CD-ROM drive and clicked check updates from the CD-ROM. Unfortunately, the error “No update found on 1 CD drive(s)” was given. I soon decided to skip that and let the appliance upgrade to 7.3.1 first. That upgrade went smoothly without any issues.

The Issue

Next up was the vRA 7.4 upgrade. I took another round of snapshots and went back into the appliance management and initiated the 7.4 install. The vRA appliance upgraded to 7.4 and asked for a reboot. The appliance rebooted and came back online. After waiting a very long time for the IaaS components to begin their upgrade I noticed an issue with some appliance services. The vCO service did not have any status while the following services were “UNAVAILABLE“:

advanced-designer-service
o11n-gateway-service
shell-ui-app

I dug into some logs and found WARN events surrounding the unavailable services. In those events, I noticed the following error: “Unable to establish a connection to vCenter Orchestrator server.” Therefore, I needed to figure out why the vCO service was not starting. Once I could get it to start, the others would register successfully. I checked the logs for the vCO services and found the following error:

 2018-04-14 18:39:16.702+0000 [serverHealthMonitorScheduler-1] WARN {} [LdapCenterImpl] Unable to fetch element "vsphere.local\vcoadmins" from ldap : Error...:[404 ][javax.naming.NamingException]
2018-04-14 18:39:16.702+0000 [serverHealthMonitorScheduler-1] ERROR {} [AuthenticationHealth] Unable to find the server administrative group: vsphere.local\vcoadmins in the authentication provider.

The Resolution

This is an immediate smoking gun for my configuration. I set up the vRO admin group to use a group within my Active Directory. Therefore, the local group, vcoadmins, was not present and prevented the vCO service from registering with vRA. I changed the vRO admin group to my AD group and rebooted the appliance.

All of the services registered successfully and the IaaS upgrade process began. The vRA 7.4 upgrade completed shortly after that without any further issues.

However, I don’t know why the vRO admin group was changed to vsphere.local/vcoadmins during the 7.3.1->7.4 upgrade. Luckily it wasn’t too big of an issue to fix but annoying to say the least.