I've switched my server from an x86 machine to an amd64 machine. I rebuilt from scratch of course but carried a lot of the configuration across so the machines are very similar. They both use the hardened kernel with grsecurity set to the Hardened Gentoo profile. I am not using the hardened Portage profile though. Both systems are keyworded as stable, though checkpassword-pam isn't currently available for stable amd64. It worked on the x86 machine but on the amd64 machine, I get the following when trying to run it through qmail (after adding the --debug flag). I must admit I don't know much about grsecurity but I've been using it for a long time now without any trouble. I've tried checkpassword-pam 0.97 and 0.99, I've tried setting it as SUID root and I've tried disabling all the restrictions on it with chpax. None of that works. Help! Jul 20 11:17:34 [kernel] grsec: From 82.40.127.232: denied resource overstep by requesting 16789504 for RLIMIT_AS against limit 16000000 for /usr/bin/checkpassword-pam[checkpassword-p:22893] uid/euid:201/201 gid/egid:200/200, parent /var/qmail/bin/qmail-smtpd[qmail-smtpd:22892] uid/euid:201/201 gid/egid:200/200 Jul 20 11:17:34 [system-auth] PAM unable to dlopen(/lib64/security/pam_unix.so) Jul 20 11:17:34 [system-auth] PAM [error: libnsl.so.1: failed to map segment from shared object: Cannot allocate memory] Jul 20 11:17:34 [system-auth] PAM adding faulty module: /lib64/security/pam_unix.so Jul 20 11:17:34 [kernel] grsec: From 82.40.127.232: denied resource overstep by requesting 16642048 for RLIMIT_AS against limit 16000000 for /usr/bin/checkpassword-pam[checkpassword-p:22893] uid/euid:201/201 gid/egid:200/200, parent /var/qmail/bin/qmail-smtpd[qmail-smtpd:22892] uid/euid:201/201 gid/egid:200/200 Jul 20 11:17:34 [system-auth] PAM unable to dlopen(/lib64/security/pam_cracklib.so) Jul 20 11:17:34 [system-auth] PAM [error: libcrack.so.2: failed to map segment from shared object: Cannot allocate memory] Jul 20 11:17:34 [system-auth] PAM adding faulty module: /lib64/security/pam_cracklib.so Jul 20 11:17:34 [system-auth] Authentication failed: Authentication failure Portage 2.1.4.4 (default-linux/amd64/2007.0, gcc-4.1.2, glibc-2.6.1-r0, 2.6.23-hardened-r12 x86_64) ================================================================= System uname: 2.6.23-hardened-r12 x86_64 Intel(R) Celeron(R) CPU 420 @ 1.60GHz Timestamp of tree: Sun, 20 Jul 2008 01:45:01 +0000 distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] app-shells/bash: 3.2_p33 dev-lang/python: 2.4.4-r9 dev-python/pycrypto: 2.0.1-r6 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.61-r1 sys-devel/automake: 1.9.6-r2, 1.10.1 sys-devel/binutils: 2.18-r1 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-march=nocona -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://gentoo.virginmedia.com http://download.mirror.ac.uk/sites/distro.ibiblio.org/pub/linux/distributions/gentoo http://distro.ibiblio.org/pub/linux/distributions/gentoo http://gentoo.osuosl.org" LANG="en_GB.utf8" LC_ALL="en_GB.utf8" LINGUAS="en" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" USE="acpi amd64 bash-completion berkdb bzip2 caps cdio cdparanoia cli cracklib crypt ctype cups curl dbus dri dvd dvdread enscript fame fastcgi fat fbcon fbdev force-cgi-redirect ftp gd gdbm gnutls gpm hardened iconv imagemagick imap isdnlog javascript jpeg ladcca ladspa lame live lzo lzw maildir mbrola mmx mng mp4live mpeg2 mplayer mudflap mysql mysqli ncurses netpbm no-old-linux noauthcram nptl nptlonly openmp pam pcre php png posix pppd readline reflection reiserfs rtc sasl session sftp silvercity spell spl sqlite sqlite3 sse sse2 ssl ssse3 svnserve sysfs tcpd tga theora threads unicode unzip usb userlocales v4l v4l2 vhosts xml zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" FOO2ZJS_DEVICES="sa300" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i810 mach64 mga neomagic nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa vga via vmware voodoo" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Oh yeah and I've tried remerging checkpassword-pam, checkpassword, pam and glibc. No dice.
I realised that there is a SOFTLIMIT_OPTS setting in conf-common. Changing this from 16000000 to 32000000 solved the problem. Maybe it needs to be higher for 64-bit systems? That wasn't the end of the story though. I also had to set checkpassword-pam as SUID root before it would successfully authenticate me. This has been reported in other bugs but not yet fixed. It makes sense, you need root access to read /etc/shadow, right?
Created attachment 342568 [details] patch to increase the soft limit in qmail
I ran into this on a x86/32bit system Portage 2.1.11.52 (hardened/linux/x86, gcc-4.6.3, glibc-2.15-r3, 3.8.2-hardened i686) ================================================================= System uname: Linux-3.8.2-hardened-i686-Intel-R-_Xeon-TM-_MP_CPU_2.50GHz-with-gentoo-2.1 KiB Mem: 1033128 total, 861556 free KiB Swap: 0 total, 0 free Timestamp of tree: Sat, 09 Mar 2013 01:00:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.69 sys-devel/automake: 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.6 (virtual/os-headers) sys-libs/glibc: 2.15-r3 From dmesg, lots of others, this was one of the highest [ 3074.673166] grsec: From 208.22.99.38: denied resource overstep by requesting 112640000 for RLIMIT_AS against limit 16000000 for /var/qmail/bin/qmail-smtpd[qmail-smtpd:20748] uid/euid:201/201 gid/egid:200/200, parent /usr/bin/tcpserver[tcpserver:926] uid/euid:201/201 gid/egid:200/200 I might have had one higher, so I just added another 0 to the end, to make the limit much higher. I have not hit it since increasing it.
