On a new install host I cannot get any session after logging in through xdm. After poking through /etc/X11/xinit/xinitrc I checked whether xmodmap is actually installed. Indeed was not. My issue still persists but this should be fixed anyway.
/etc/X11/xinit/xinitrc contains sysmodmap=/etc/X11/xinit/Xmodmap ... if [ -f $sysmodmap ]; then xmodmap $sysmodmap fi That looks like xmodmap is optional.