Configuring new or existing replications under vSphere replication fails with the error message “NFC code 5 (no memory available) and NFC code 8 (Can’t open remote disk)” .
It has been quite a challenge blogging regularly but trying my best to keep up with the pace. Recently I was working with one of the customer where I witnessed customer was using vSphere replication 6.x in his environment and configuring any new or existing replications was failing with the error message
“NFC code 5 (no memory available) and NFC code 8 (Can’t open remote disk)”
The error was not very obvious nor encountered on a day to day basis so it took me a while to understand what was triggering this error.
It was evident that NFC session is created between source ESXI host to destination ESXI host. Now in most of the customer’s environment replications are configured to shared storage due to which if 10 ESXI hosts are part of the shared storage the NFC session can be created with any one of the ESXI host making it difficult to troubleshoot in such cases.
Luckily in the current scenario there were only 3 hosts in the DR site and since we were not sure to which specific Esxi host these VM’s had created those NFC sessions we took a putty session to all the 3 hosts.
Most of the NFC session information Is always captured under hostd logs. So had a look at hostd logs and found below traces continuously being triggered
info hostd[798C1B70] [Originator@6876 sub=Libs] NfcServerProcessClientMsg: Authenticity of the NFC client verified. warning hostd[798C1B70] [Originator@6876 sub=Libs] [NFC ERROR] NfcCheckAndReserveMem: Cannot allocate any more memory as NFC is already using 50331561 and allocating 119 will make it more than the maximum allocated: 50331648. Please close some sessions and try again warning hostd[798C1B70] [Originator@6876 sub=Libs] [NFC ERROR] NfcProcessStreamMsg: fssrvr failed with NFC error code = 5 error hostd[798C1B70] [Originator@6876 sub=Nfcsvc] Read error from the nfcLib: NFC_NO_MEMORY (done=yep) info hostd[798C1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=e2e5d4d8 user=vpxuser] Task Created : haTask--vim.NfcService.systemManagement-7351068 info hostd[798C1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=e2e5d4d8 user=vpxuser] Task Completed : haTask--vim.NfcService.systemManagement-7351068 Status success
The logs clearly stated that the NFC was out of memory due to which it was not able to create any further sessions. The host was up for 378 days and since then those session were created. Restarting management agents did not reset those session, Informed the customer to perform a rolling reboot of the host one by one as the remaining hosts were up for same number of days and similar traces were found under hostd logs.
Once host was rebooted the existing replications automatically switched back to sync status and new replication were configured successfully.
Hope this article was helpful. Watch-out for more.