ESX hosts feels sticky after configuring remote logging

by user · December 10, 2024

After implementing the vSphere hardeing settings all my new hosts feel quite sticky. After six days when starting SSH and trying to connect the connection is refused. After rebooting the host it is up and responding again to SSH but feels very sticky.

After checking the vCenter events I can see some error messages relating to “ramdisk ‘var’ is full” and “no space left on device”.

When checking the web for those error messages I found the VMware KB 376958. https://knowledge.broadcom.com/external/article/376958/configuring-scratch-configuration-and-sy.html

Refering to this KB this is a known issue when scratch is reconfigured to a different location on a host that has audit record storage enabled beforehand.

So checking the logs on the hosts with the commands shows an exact match for this issue.

cat /var/log/.vmsyslogd.err

[root@esx01:~] cat /var/log/.vmsyslogd.err
2024-12-10T08:07:04.799Z vmsyslog                 : CRITICAL] vmsyslogd daemon s                                                                                                                                    tarting (2100951)
2024-12-10T08:07:04.994Z vmsyslog.loggers.audit   : ERROR   ] Files are missing                                                                                                                                     from the audit record storage directory.
NoneType: None

[root@esx01:~] /etc/init.d/vmsyslogd status
vmsyslogd is not running.

So has the host does not response very well to any commands. I go forward and disable the following settings by esxcli

esxcli system auditrecords remote disable
esxcli system auditrecords local disable

Now we can set again the settings to reenable the remote and local logging.

esxcli system auditrecords remote enable
esxcli system auditrecords local enable

After reenabling the remote logging the output changed.

[root@esx01:~] cat /var/log/.vmsyslogd.err
2024-12-09T14:26:01.625Z vmsyslog                 : CRITICAL] vmsyslogd daemon starting (2098123)
2024-12-09T14:26:01.814Z vmsyslog.main            : ERROR   ] Unable to obtain DPU information-Command '/bin/localcli --formatter=python --plugin-dir /usr/lib/vmware/esxcli/int hardwareinternal dpu list' returned non-zero exit status 1.
2024-12-09T14:26:26.548Z vmsyslog.main            : ERROR   ] switching to normal mode from init mode
2024-12-09T14:26:34.758Z vmsyslog.main            : CRITICAL] reloading (2098123)
2024-12-09T14:26:35.017Z vmsyslog.main            : CRITICAL] reloading (2098123)
2024-12-09T15:43:59.630Z vmsyslog.main            : CRITICAL] reloading (2098123)
2024-12-09T15:44:09.634Z vmsyslog.main            : CRITICAL] reloading (2098123)

[root@esx01:~] /etc/init.d/vmsyslogd status
vmsyslogd is running.

Now we can see that the syslog services is running and the error regarding “audit record storage directory” is fixed.