vCenter Appliance fails to start with the error message “/dev/mapper/log_vg-log: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY”

Today I was working with one of the enterprise customer. Customer had a mass storage outage and environment was running on a vSAN. Once storage was back in stake customer went and powered on the vCenter server. vCenter failed to start up with error message “/dev/mapper/log_vg-log: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY”

You can have look at the below snippet to understand it better.

Even after multiple reboots we ended up with the same error. After having a close look we determines one of the file system looks to be inconsistent and needs to be fixed.

So we followed the below steps to rectify the issue.

• Reboot the vCSA using the vSphere Client.
• When the GRUB bootloader appears, press the spacebar to disable autoboot.
• Select p and enter the root password
• Type e to edit the boot commands.
• Scroll to the second line displaying the kernel boot parameters.
  You see this line:kernel /vmlinuz-3.0.101-0.35-default
• Type e to edit the boot command.
• Append init=/bin/bash to the kernel boot options.
• Press Enter.  The GRUB menu re-appears.
• Type b to start the boot process. The system boots to a temporary BASH shell.
• Set the default shell to BASH by running this command:
  chsh -s /bin/bash root
• Reboot the vCSA using the vSphere Client.
• Enter the root password
• At the repair filesystem prompt type the following command:
  fsck /dev/mapper/log_vg-log –y

-y would automatically fix the error without prompting after each consistency check.

Fsck would ideally rectify any inconsistency found at the OS level.

Once above command was executed the vCenter appliance was up and all services resumed successfully.

Note: Always take a snapshot before performing any certain changes.

Hope this article was helpful. Watch out for more.