After reporting a pygame test failure , it was recommended that I try building media-libs/libsdl2 with --disable-arm-simd.
This resolved the test failures.
It appears that the Pygame developers have hit other issues related to this in this bug .
I've added the information to the upstream bug .
For now, it may be a good idea to build libsdl2 with this option on 32 bit ARM (arm64 was fine).