NEON is not supported by all Cortex-A9 cores. It has been mandatory for Cortex-A8, but is not for Cortex-A9. At least Marvell Dove and nVidia Tegra 2 are left out, if we enable NEON on armv7hl.
armv5tel does not support NEON at all. armv7hl on IPFire uses VFPv3-d16 which is more common and equally fast for our purposes.