While on a load balanced setup using HWL, we found a bunch of errors with the configuration. Nothing was out of the ordinary after re-checking the configuration several times.
After various attempts to re-configure and debug the issue, we have turned our attention to the load balancer. This is a HWL.
What we found in the configuration is that the spoof option was enabled.
After disabling spoof, everything started working as expected.
So, note to self, when setting up HWL, if there is an option for spoof, make sure it’s disabled. Knowing this would have saved a few people a few hours of effort.