I killed my vCenter Server Appliance (VCSA) when updating! **argh**
It took me by surprise. When I talked about it, a friend told me that one is not a VMware “nerd” if you have not killed your home lab’s VCSA at least once. So, I guess I have earned that qualification now ;-).
Bringing the VCSA down with an update was too quick and let’s say “easy”. Trying to getting the VCSA “repaired” and my home lab fully functional again took many times longer than I ever expected. I failed to repair the broken VCSA and I finally replaced the broken VCSA with a new one, on new and dedicated hardware. Even the new install and recovery configuration wasn’t without issues.
I have learnt my lessons from this disaster, and I thought I share my learnings. #vexpert #sharingknowledge
Backup
First lesson learnt, make sure you have a backup in place for your VCSA! When you look at the VMware documentation Manually Back up vCenter Server by Using the vCenter Server Management Interface you get the prerequisites and the procedure how-to.
I now have a backup for my VCSA 🙂 and I wrote a blog post about it!
Because of no backup and because my VCSA was fully down and I could not repair and recover it, so I decided to get additional hardware, adding another NUC to my home lab which would be dedicated to the VCSA.
New hardware
I picked up
- Intel NUC NUC8i5BEH, with 32GB RAM, 512GB NVMe SSD (with pre-installed Windows 10)
- SanDisk Cruizer Fit 64 GB (Boot Disk)
Installation
When I looked for installation instructions to install vSphere on this NUC, I found the following blog posts:
First step I took before starting the installation, was updating the NUC bios to the latest version. I then followed Running ESXi 6.7 on a Bean Canyon Intel NUC NUC8i5BEH. for the bios settings – with both UEFI boot and Legacy Boot enabled.
Next step was to create a USB installation media from the bootable ISO of vSphere (For downloading vSphere you need a VMware Customer Connect (formerly My VMware ) account.). I used the open source tool balenaEtcher which is available for Windows, Mac and Linux.
Installation Issues
To my surprise the installation neither with vSphere 6.7U3 nor vSphere 7U1 did not went well at all. I lost count of how many unsuccessful tries I had with the error message
Reboot and Select proper Boot device
or Insert Boot media in selected Boot device and press a key
The error might be related to the preinstalled Windows on the internal SSD.
The solution for the successful installation turned out to be inserting the USB stick into USB port “2”, the lower USB port at the back of the NUC and use the upper USB port “1” for the Boot device. Previously, the USB stick was in the front USB port.
The NUC8i5BEH, for the moment, now runs with vSphere 6.7 P05.
To find the ESXi build number, have a look at this VMware knowledge base article: Build numbers and versions of VMware ESXi/ESX (2143832).
The new VCSA
With vSphere being installed and running, the new VCSA was next. I downloaded the vCenter 7 OVA from VMware Customer Connect and used the new host to deploy the appliance.
Just like for vSphere, you can look up the Build numbers and versions of VMware vCenter Server (2143838).
With the new VCSA, I deleted and cleared everything of the broken VCSA.
My home lab – cluster with three ESXi hosts and the new ESXi host – is functional again.
vSphere 6.7 on NUC8i5BEH
As my hosts in the cluster are on ESXi 7U1d, I wanted to upgrade the new host, too. BUT – I stumbled upon a few issues.
The issues were related to
- bootbank
- network adapter
- tardisk
- VMware locker tools light**
to name a few.
When searching on how to overcome the issues, I found the following links related to “my” issues:
- Bootbank loads in /tmp/ after reboot of ESXi 7.0 Update 1 host (2149444)
- Will ESXi 7.0 Update 1 run on Intel NUC?
- How to upgrade Intel NUC Gen8 to vSphere 7.0 U1 without losing network
- ESXi (Standalone) Update Failed
- Finding the latest version of VMware_locker_tools-light
- ESXi 6.7 Update Failed
- Installing and upgrading the latest version of VMware Tools on existing hosts (2129825)
As the upgrade to vSphere 7 keeps failing, I decided to go with vSphere 6.7.
**VMware Locker Tools light
As I had not heard of VMware locker tools (light) before, I did a search and was directed to the VMware tools.
As not everyone is an expert on VMware vSphere (yet), I can recommend to visit the VMware documentation Introduction to VMware Tools.
VMware Tools is a set of services and modules that enable several features in VMware products for better management of, and seamless user interactions with, guests operating systems.
I also found
You can always find the latest build of VMware Tools here: https://www.vmware.com/go/tools.
Installation instead of upgrade
With the upgrade from vSphere 6.7 to 7.0 not succeeding but I still wanted to get the new ESXi host to vSphere 7. As it is a small home lab, I decided I go for the option of plain installation of vSphere 7 on the new NUC.
Interesting enough, when I inserted the USB stick with the installation media into the front USB port, the NUC did not show it in the list of the bootable devices. The same issue when I originally tried to install vSphere 6.7 or vSphere. Inserting the USB stick for installation in one of the back USB ports – everything worked like a charm.
Not sure if it is my NUC8i5BEH hardware which is misbehaving or if it is a general NUC8i5BEH issue. My recommendation: Use the front USB ports for the keyboard during installation and the back USB ports for installation and placing the boot disk
Starting with vSphere 7.0U2a, then 7.0U1c, still failing and finally got to a seamless install with vSphere 7.0.0 covering the embedded network adapter and all storage that the NUC provides.
Updating to vSphere 7.0U2a is possible, but you will lose the embedded networking.
To remediate see: How to upgrade Intel NUC Gen8 to vSphere 7.0 U1 without losing network.
Essentially you have to downgrade the embedded ethernet adapter’s driver to version 7.0.0.
The command to do so is:
esxcli software vib install -n ne1000:0.8.4-10vmw.700.1.0.15843807 -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
When using the lifecycle manager, your host will be told it needs updates. Do not update or the NIC downgrade will be undone. The NIC driver is pushed separately and in the U1 rollup!
You have to do this every time that the remediation has run and this has to be done ahead of rebooting the ESXi host.
As a second level option, you can use an USB A ethernet adapter.
Next step, create persistent scratch location on the standalone ESXi host.
VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap.
Source: VMware Knowledge Base Article: Creating a persistent scratch location for ESXi 7.x/6.x/5.x/4.x (1033696)
Also have a look at the VMware documentation for more information and details about the scratch partition.
Also make sure you have enabled swap on the ESXi host and a datastore is shown in the settings. When I checked on the host, everything was enabled but no datastore was showing.
vSphere 7 is now running on the NUC8i5BEH.
Informational Note
Dead on arrival (DOA), also dead in the field and brought in dead (BID), indicates that a patient was found to be already clinically dead upon the arrival of professional medical assistance, often in the form of first responders such as emergency medical technicians, paramedics, or police.
Source: Wikipedia
